Introduction to Hive – week 2

Key Takeaways

• Hive is a very powerful infrastructure tool that allows us to query large datasets in an interactive environment sitting on top of Hadoop.
• Hive allows us to query using SQL like commands very large structured and unstructured data.
• The commands are easy to use and do not require programming.
• We used Hive to analyze data about bike share program in San Francisco. This is the first time we imported public data in form of CSV files and created tables in Hive. We used the Hue environment. 

Hive is an infrastructure tool that allows us to process structured and unstructured data in Hadoop. Hive sits on top of HDFS. It was originally created at Facebook. It provides an interactive environment to Extract Transform and Load (ETL) Big Data.

Why Hive?

If you were running MapReduce on HDFS – you have to run it in batch.  The map and reduce functions are written in Java and the process is slow and not interactive. Hive provides an SQL type of environment with simple commands, that can be run interactively by non Java programmers who are more interested in looking at data than programming mappers. Hive is written for On Line Analytical Processing (OLAP). It is fast, scalable and extensible. It allows custom mappers and reducers if required.

We discussed the sequence of execution steps in Hive from the UI to the Driver to the Compiler to the Metastore and Execution engine. The execution engine interacts with Hadoop and returns the results. Hive can be run in high level languages or local and MapReduce. Hive can be run interactively or in batch mode.

Hive Data structure

Hive tables are HDFS directories. Within HDFS directories are sub-directories which are called partitions in Hive. Buckets are files under each partition. By defining data in buckets you are able to sample large data sets and significantly reduce CPU usage.

It supports primitive types such as integer, floats, strings, doubles, list and structures.

Hive comes with a number of Table commands such as CREATE, SHOW, ALTER, DROP TABLE.  It allows to load data utilizing HDFS or load data into a local file. Hive allows you to do JOIN – one JOIN command can replace 60 or more lines of Java code. Join is easy to use – it combines tables with specific fields based on a common value.

We did a number of Hive Assignment. We used an example data table in the Hue environment. We used SQL like queries to select fields from a Table based on a certain criteria (like Price > 10) and ordered them in descending order. We used these interactive queries to find the highest salary, the lowest salary or the most product you can buy for a $10 etc.

We then went to a more elaborate exercise where we loaded public data of San Francisco bike-share program.  We used command such as count and calculate the number of people borrowing the bike by station. The assignments were relatively easy after going through the Map and Reduce programs in Python in a previous course. See: http://www.bayareabikeshare.com/datachallenge

Big Data Analytics – Week 1.

Key Takeaways

1. Data analysis requires you to access, manipulated/transform and query/explore data.
2. For data analysis you need something more interactive and versatile than MapReduce.
3. There are a number of higher level tools that sit on top of Hadoop HDFS such as HBase, Hiave, Pig, Spark and Splunk.
4. In week 1 we went into great detail in HBase, why it is relevant and what is its architecture. We did some data exploring using Hue that uses HBase to explore a large dataset1.    

 General Introduction

Data Analysis involves analyzing data, gaining insights and drawing inferences/decisions and then ultimately taking some actions. So data analysis requires searching for meaningful things. This requires easy access to data, good data manipulation functions and ability to explore/query the data. Basic MapReduce framework doesn’t allow us to do that. So we need higher level tools that sit on toop of Hadoop to allow us interactively work with the data.  Data base management system (DBMS) is useful for structured data, for transactional uses and for SQL. Hadoop is used for unstructured data, where the data is growing and allows flexible mapping. Over time there is an overlap of traditional tools and Hadoop overlay tools. This course is about tools other than MapReduce that will help us access, transform and query/explore Big Data.

HBase – stores data in ‘Big Tables’ based on Google. HBase organizes data into column oriented format with indexing done on a row and column basis instead of with a key.

Hive - is a metastore where it holds the schedmatics of data from HDFS. In that way it is like DBMS and allows for an SQL like interface.

Pig – is a scripting language that provides more flexibility than SQL.

Spark - is a engine for programming with an interface that works directly with HDFS and provides a richer set of programming functions other than MapReduce.

Splunk - is useful for working with machine data, such as web logs etc.

HBase

Hadoop is good for sequential access and batch processing. HBase allows fast random access of big data and random access and allows updates. HBase files are stored on HDFS. The data is organized in big rows and column tables. HBase is not a relational database, sparse and highly distributed. HBase data can be 100 TB to 1 PB or 1 ZB. HBase is organized along 3 dimensions.

• -       Rows
• -       Columns
• -       Time Stamp (stored changes over time)

 HBase Data Model in detail

Columns are grouped in column families. The rows run across the column families. Several tables – each table has multiple column families. Designed to run on 10s if not 1000s of servers and is designed to deal with distributed computing challenges – coordination, management, data distribution and network latency. There are three components of the architecture

• -       Regions – a subset of table rows, horizontal range
• -       Region server – slaves
• -       Master Server co-ordinates actions of region servers.
• 
  

  HBase depends on zookeeper. Most basic unit is a column. Distinct values for each time stamp. Column. keys are grouped as families. Column family is a stored as a HFile. A cell can hold multiple versions of data. Data stored in such a way such that recent data is fasted to read.

We looked at a large dataset which had web visits to various domains (rows) in different countries (columns). We were asked to select data, identify the countries or domains that have highest or lowest number of  visitors. This was an exercise to familiarize ourselves with data selection/exploration in HBase.

Week 5: Introduction to Spark - and its (almost) over!

Ok to be honest, I am pretty tired writing weekly summaries of what I learned, especially with the heavy programming workload the past two weeks have entailed. But I think for the sake of completeness I will post a few highlights.

1. Spark is an interactive environment that is more interactive than Hadoop Streaming and provides a superior framework than MapReduce (which we learnt last week). It is especially true when there are iterative algorithms involved such as for machine learning. It also supports a number of languages such as Python, Scala etc. rather than just Java.
2. We looked at Spark architecture in a Cloudera VM on Amazon EC3 for e.g. There is a cluster manager and a bunch of worker nodes which has many partition of Python executors.
3. We learnt to fire off PySpark the interactive Spark environment with Python Command Line Interpreter.
4. Lesson 2 was all about Resilient Distributed Datasets (RDD). These are Spark data constructs which store blocks of data. They have resiliency as such that they can be recreated if a node fails.  RDDs are immutable. One creates a new  RDD by performing transform operations.
5. We talked about transform operations or transformations. There are narrow transformations which are not very heavy on network usage such as map, flatmap, coalesce, select etc. There are wide transformations such as ‘groupByKey’, ‘reduceByKey’ and repartition. These cause huge movement of data between nodes and are memory/compute intensive. As part of this lecture’s homework we did a simple ‘Join’ operation of two sets of data in Spark using the PySpark environment. I had no trouble passing this.
6. The third lesson in the series was all about jobs scheduling, actions, caching data and use of shared variables. We learned about the ‘Directed Acyclic Graph Scheduler (DAG) and how it allows Spark to keep track of execution pipeline. Interestingly most Spark transformations are negative in the sense they are not triggered until certain actions such as collect().
7. The final assignment was a programming assignment where we had to go back to the channel, shows, viewers data from week 4 and redo the assignment using the Spark interactive framework. This assignment was tough and took me a better part of my Sunday. But in the end I was able to upload my answer – total viewership for a particular channel and get 100/100. In getting this one write I had to be shameless and try a lot of things many times ending in ‘epic-fails.’ I also had to repeated go back to past lessons and identify course material relevant to the problem at hand. In the end this assignment was way more completed than should be expected for a business leader to solve – but I am glad I raised my work to meet the expectations of this week.
8. I still have a final quiz to pass on the Section 3. After numerous tries (and repeated review of course notes and videos) I am only able to get 4 out of 6 right. I need to get at least 5/6 right.  I have (I think) until the end of the week to get this right so that I can pass the course  with 16/16 assignments. So far I have 15/16 done.

Overall Observation

Overall I learnt a lot in this course and really pushed my ability to code and apply the course material to actual programming problems. Even after such grueling assignments it is humbling to know that I have far from a working knowledge to use Spark/Hadoop for rigorous data modeling. I just know what questions to ask!

Week 4: Map Reduce and the Challenging yet Successful week coding Mapper/Reducer for a JOIN operation

Week 4 – MapReduce

KEY TAKEAWAYS

1. Learned about the MapReduce framework on Hadoop.
2. We looked at a simple word count program to understand how a mapper works and how a reducer works and how you get different results with different number of reducers.
3. We learnt about JOIN where you collect two sets of data and join them by finding common key values and get meaningful insights out of the data.
4. There was a (what now feels like) pretty easy assignment that was used to demonstrate a JOIN for calculating network viewership by show. We had to code the mapper and reducer in Python and also device the algorithm. After several days of struggle I was finally able to make it work. 
5. Sometimes bugs/difficulties are a blessing as it really forces you to learn all the tricks of the trade on how to debug code, really understand what each line in example code works and also marshal resources.
6. There isn’t a lot of TA support on the Coursera online courses 

General Info on MapReduce

MapReduce is a key framework for handling and processing large amounts of unconnected data that is relatively static and does not change. Hadoop does that by mapping the data into a number of nodes and performing a mapping function – which is assigning a key and a value to create a pair.

We studied in this lesson how map function works. Then after all this data is mapped how a reducer work which shuffles all the data and sums up the values for all keys. We applied this for a simple example of word counts for words in a Star Wars line.

We ran the Word Count program by writing the mapper and reducer in Python and specifying the mapper and reducer as the python files in Hadoop streaming commands.

We did an example for a word count with a single reducer, zero reducer (it just does mapping) and two reducers (generates two output files).

I had reviewed the lectures on Monday and ran the code on Thursday. I took the quiz on Thursday and was happy to see retaining most of my knowledge and getting 7/7 write.

Time to fool myself is over – onward to more complex examples of Map/Reduce.

 JOIN and Programming Assignment

We learnt about how to join data from two separate data sets using one common key to get some meaningful insight. As part of the assignment we ran an example program with a mapper and a reducer to practice learning testing of individual mapper and reducer code and also understand Hadoop Streaming command syntax. The final assignment was to do a join that built on previous join examples. There were two sets of data – one set showed a set of TV programs with viewership data. The second set showed TV programs for many channels. We had to count the total viewership by channel and then summarize it for a particular channel in this case “ABC.”

It took me a couple of days (many hours each day) just to understand the assignment and figure out the structure of the mapper and reducer programs and how they function in the map reduce framework. I had a very busy weekend and the assignment was due at midnight on Sunday. I worked till midnight but still couldn’t get the totals of the first two shows to match up with the hint provided in the assignment. It was a long and frustrating evening but I also learnt a lot.

On Monday I was a day late but I kept plodding but my totals kept coming short. Finally on Tuesday I created the data file and mapper output and output it to excel and verified that the totals from my reducer were the same as what excel was calculating.

I then made sure that the data generated (using a pseudo-random number data generator) was identical no matter what platform I used. My nephew Pranav Dandekar(who I suspect has done a lot of map reduce work in his life) also volunteered to verify the data generator on his machine as well as AWS. So I ran the data on my Virtual Machine (VM), my Mac and also on my Raspberry PI. Same data – so I know the problem was with my mapper. After looking at my code for the 1000^th time I realized that I was only outputting a 3 digit channel viewership as a result the mapper was not outputting lines that had 4 digit viewer numbers. Once I fixed it everything went like clockwork. I submitted the result and got 100/100 instantly. All that is well, ends well!

(See picture below from a blog of http://xiaochongzhang.me that demonstrates the MapReduce steps for a simple word count)