Introduction to Hadoop

Dear Friends,

Couple of months back I've published a post on SQL Server 2016 new features here.

Meanwhile, let me introduce you to Hadoop. We will learn this as a series of inter-related post. So don't miss any post in between and read it serially. Let's make it fun and interesting to learn Hadoop.

So, lets understand what are the formats of data that we handle in real word.

• Flat File
• Rows  and Columns
• Images and Document
• Audio and Video
• XML Data and many more......

Big Data is an ocean of data which an organization stores. These data come in three V's i.e. Volume, Velocity and Variety.

Now-a-days huge Volume of data are getting generated by many sources such as Facebook,Whatsapp, E-commerce sites, etc, etc, etc. These huge volume of data are getting generated with high Velocity, can say it is multiplying every seconds, every minute, every hour. Along with the huge Volume and high Velocity numerous Variety of data is generated in different forms.

These data can be in any format i.e. structure, semi-structure as well as unstructured. Data stored in the form of row and column can be well defined as structured data whereas data in form of document, image, sms, video, audio,etc can be categories into unstructured and data in html or in XML format can be semi-structure data.

Q. I am sure you must be thinking that then how does a RDBMS handles these kind of unstructured or semi-structure data in there Database?

A. Well, to handle these kind of data's we have special data type such as Varbinary(Max), XML. Drawback of this is, if we are storing an image, it is stored in binary format within the database; whereas actual image is stored in Filestream or the Server itself. Hence there is an performance impact during storing and retrieving Petabyte of data's.

Moreover to this, Big Data is not just about maintaining and growing the data year on year, but it is also about how you manages these data's to make an informative decision. Data in Big Data can also comes in various complex format, to manage and process these type of data we need large set of cluster servers.

With this introduction to Big Data, now let me introduce you to Hadoop. 

Hadoop is a large set of cluster servers which is built to process large set of data. It has two main core component i.e. 'Hadoop MapReduce' (Processing Part) and 'Hadoop Distributed File System' (Storage Part). Hadoop project comes under Apache and that is why it is called as 'Apache Hadoop'. The idea behind these two core component came into existence when Google has released there two white paper of there project on 'MapReduce' and 'Google File System (GFS)' in the year 2004.
Hadoop was created by Doug Cutting in 2005. Cutting, He was working at Yahoo! at the time he build the software, it was named after his son's toy elephant.

Wikipedia defines Hadoop as "an open-source software framework written in Java for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware"

Hadoop is an open source framework available in free as well as commercial use under Apache license. It allows distributing and processing of dataset across the large Cluster set. On top of this, there are lot more application build by other organisation who use Hadoop or continuously work on this product which all comes under 'Hadoop Ecosystem'. Check here to find the list of projects under Hadoop Ecosystem. As we move further we will see post on the important projects under Hadoop Ecosystem.

Apache Hadoop Architecture consist of following components:

• Hadoop Common: It contains the libraries and other utilities needed by other module of Hadoop.
• Hadoop Distributed File System (HDFS): It is cluster of Servers with commodity storage which is used for data storage across the cluster.
• Hadoop YARN: This component is used for Job scheduling and Resource management in Cluster.
• Hadoop MapReduce: The processing part of the data is done by this component. 

At least now, I can consider that you are having a fair enough idea about this technology. But how can or who is the best person to get into Hadoop??

Technical answer for that would be those who are interested to learn this technology can get in two ways:

• As a Developer
• As a Administrator

1. As a Developer: Hadoop is a framework which is built in JAVA language. So having JAVA background can get easy access to become a Hadoop Developer. Since the growing popularity of Hadoop, now a days this is the most common designation you can find in Job sites. 
2. As a Administrator: Most organisation with Hadoop installation prefer for a Part time or a

   

   
   

   Full time Administrator to manage there Hadoop Clusters. It is not compulsion that the admin should have the knowledge of JAVA to learn this technology. Indeed! they should have some basics for troubleshooting. Candidate those who are having knowledge with Database Admin (SQL Server, Oracle, etc) background who already have troubleshooting, Server maintenance, Disaster Recovery knowledge are preferred or anyone with Network or Storage or Server Admin (Windows\Linux) skills  can be the other best choice. Here in this post it is mentioned in detail who suits best for Hadoop.     

Following might be the questions in your mind if we want to get start with Hadoop Admin:

1. Do we need any DBA skills? Of course Yes; If we need to Admin the Hadoop Cluster (Maintaining, Monitoring, Configuration, Troubleshooting,etc). 
2. Do we need to learn Java? Yes; At least some basics to understand the Java errors while troubleshooting any issue.
3. Do we need to understand Non-RDBMS? Yes; Hadoop understand both SQL and NoSQL (Not only SQL). So having knowledge on Non-RDBMS product is most important.
4. Do we need to learn Linux too? Yes;  at least the basics.

In our next post we will see the concept of HDFS (Hadoop Distributed File Structure).

Interested in learning SQL Server Clustering check here. Stay tuned for many more updates...

Keep Learning and Enjoy Learning!!!

Error - While Putting Database from Single to Multi User Mode

Dear Friends,

Click here to check how to start the SQL Server without TempDB Database. Let's learn what to do or how simple it can be to change the Database Mode from Multi User to Single User Mode or vice-versa?

Indeed! it is simple with the following command:

a. Alter Database Out set Single_User 
b. Alter Database Out set Multi_User

If we don't mention any termination clause like above it will run until the statements get completed.

Suppose there are n numbers of users connected to the Database and you executed the above command it will take hell lot of time to complete.
So rather, you can force disconnect the users to put the Database in Single User mode you have to fire the below command:

Alter Database out set Single_user with Rollback After 30 -- After 30 Seconds it will cancel and Rollback the Query

Alter Database out set Single_user with Rollback  Immediate -- It will immediately cancel and Rollback the Query

Alter Database out set Single_user with No_Wait -- If  there is any incomplete transaction No_Wait will Error

But again it might get horror if the Databases is in Single User and you cannot access the Database because only one connection can be made at a time and just think that connection is taken by the system i.e. SQL Server.

In this situation you are locked out, reason you cannot access the Database. Like you can see in the snapshot the Database Out is used by the system i.e. it is used by the Background process.

If you try to bring back the Database again in Multi User mode system will throw the following error:

Another possibility you can try would be detaching the Database. But when I tried detaching the Database, SQL Server will first kill the connections to the Database. Rather I should frame it as SQL Server will kill only the User connect and not the system connection.

After loads of struggle we were back to square one, that our Database was not getting back to Multi User mode. Seems it was like a deadlock between System SPID with Out Database. So we enabled the trace flag and checked the Error Log file. So following snapshot confirms that there was an deadlock:

DBCC TRACEON (1204,1222,-1)

Deadlock Graph

If we try to Alter the Database to put it in Multi User mode. We will get a deadlock and since our Alter Statement is having low priority it will fail with the error message:

After random tries we tried the following command and it saved us. We have to set the deadlock priority high and then execute the Multi User mode query like below:

Set Deadlock_Priority High
Go
Alter Database Out Set Multi_User

So what this will do is it will set the Dead Lock priority High and Alter the Database to Multi User mode.

Guys do share the feedback about this article and of course about the blog too.

Want to start learning SQL Server Clustering?? Check here the three part series on SQL Server Clustering.

Do you know MS SQL Server 2016 is ready to launch?? Check here the two part series on New Features of SQL Server 2016.

Keep Learning and Enjoy Learning!!!

Error 233 - No process is on the other end of the pipe

Hi Guys,

As we have seen here an error related to SQL Server Restart. Today we will see another error i.e. Error 233 "A connection was successfully established with the server, but then an error occurred during the login process."

You might have faced this common issue in your DBA career and yes the solution is relatively simple.

So what I was doing, I was connecting to the SQL Server from Windows Login but during the login process it failed and prompted the below error message:

Error 233 in SQL Server

As I said this seems an common problem so there might be multiple workaround for this.

Now, for me the solution was to start the SQL Server management Studio (SSMS) under "Run as Administrator" (When I was fresher, I use to always wonder what difference it makes if I start any program under Run as Administrator?? Let's find out the reason for that).

Reason for this error was:

a. When we are running an application under "Run As Administrator" we get extra privileges which we may not have under the local user account.

b. So running the program under Run As Administrator will grant extra rights to the account (but it should have admin rights in Active Directory).

Once I started the SSMS under Run as Administrator and connected to SQL Server it succeeded.

There might be chances that this may not solve this  issue, check here to find more work around on the issue.

Do you know?? We should properly set the Auto Growth parameter in Database else it would end up with consuming extra space. Check here.

Thanks,
Vikas B Sahu

Keep Learning and Enjoy Learning!!!

How to Start SQL Server Instance when TempDB is unavaliable

Hi Guys,

Here is my last post on error while starting the SQL Server Analysis Services (SSAS) and here I've shared earlier on a issue, I request you please read that before proceeding.

Today we will learn a fact on the internal of SQL Server Tempdb (which was little unusual for me before I actually faced it. So I must say it was a learning experience).

Following are some questions that comes to my mind when I think about TempDB:

1. Where and how exactly the TempDB Database is used?

2. What exactly happened when your TempDB is corrupted?

3. Have you ever though of restarting the SQL Server Services without TempDB? Is it possible to do that?

4. Is it possible to Backups (Full, Differential or Log) and Restore the TempDB Database?

Let's see the possible answers to these questions:

Q 1. Where and how exactly the TempDB Database is used?

Ans. i. It is a System Database. Database ID for TempDB is 4. 
        ii. It is use to store temporary Data. Whenever we create a hash table (#abc) it gets created in TempDB.  
        iii. It also used for Sorting of Data (for e.g. If we use Order by clause in a query it uses TempDB to sort).
        iv. Also used for Row-Versioning.
        v. Recovery model of the TempDB is "Simple". 
        vi. We cannot run DBCC CheckDB on TempDB. 
        vii. We cannot detach the TempDB files. (Rather I must say one cannot detach the system Database files) 
        viii. Most important it is re-created every time whenever SQL Server is restarted. By checking the "Last Creation Date" we can see when was the last SQL Server was restarted (can expect as an interview question).

2. What exactly happened when your TempDB is corrupted?

Ans. i. Since we cannot run DBCC CheckDB; we cannot identify if there is any corruption on TempDB Database.
        ii. The only way to get rid of  corruption is to restart the SQL Server Services. Since it will recreate the Database files again.
        iii. Without TempDB Database SQL Server Services cannot be started. 
        iv. Model Database acts as the template for all the user created Database as well as for TempDB. So if model Database is unavailable TempDB will not get create and hence SQL Server  Service cannot be started.

TempDB Cycle

3. Is it possible to Backups (Full, Differential or Log) and Restore the TempDB Database?

Ans. i. Since TempDB is recreated every time when ever we restart the instance. We cannot Backup this Database.

4. Have you ever though of restarting the SQL Server Services without TempDB? Is it possible to do that?

Ans. (Now here is the question what made to write this post)

i. As said above, TempDB is restarted every time when a SQL Server is restarted. (This will create both the mdf and ldf files on the location present for TempDB in the Master Database). 

ii. But what if there is a disk level corruption or the defined location is not present due to xyz reason.

iii. Now here comes my question: Will the SQL Server Services will start??? So, answer for this is YES!!! it is possible (Till now it was NO for me). 

iv. If  you will start the SQL Services normally (assuming there is a disk level corruption on the disk were the TempDB was placed) the disk is unavailable you will get the below error message:

TempDB Error Log

v. So basically starting the SQL Services normally will not help you. Therefore we have to start the SQL Server Instance in Single User Mode as well as Minimally Configured. 

Adding Parameter in Startup parameter

vi. You can do that by adding -m (Single User Mode) and -f (Minimal Configuration) parameter in the startup parameter in the SQL Server Instance.

vii. This will start your SQL Server in minimal configuration (of course TempDB files is created) placing the TempDB files i.e. Data file on the User Database Data file location and the Log file on the User Log file location.

viii. And by this way your TempDB files are placed on different location other than the actual location for TempDB files.

ix. But this different location is temporary and only if you will start your SQL Server Instance in minimal configuration. So it is important to change the TempDB location to another drive by using the following command:


USE master;
GO
ALTER DATABASE tempdb
MODIFY FILE
(NAME = tempdev, FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQL2008R2\MSSQL\DATA\TEMPDB\Tempdb.mdf');
GO
ALTER DATABASE tempdb
MODIFY FILE
(NAME = templog, FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQL2008R2\MSSQL\DATA\TEMPDB\Tempdb.ldf');
GO

SELECT name, physical_name AS Location
FROM sys.master_files
WHERE database_id = DB_ID(N'tempdb');
GO

Hope this post will give you a different angle on concept of TempDB. Further we will see so more findings on TempDB. Share your comment on this to discuss more on it. You might be aware on SQL Server 2016 check here for the new features in it.

Thank You !!!