Interesting use case about migrating away from SQL to Hadoop and NoSQL

Paytronix analyzes data from 8,000 restaurants that adds up to a few tens of terrabytes of data. Not that complex in terms of volume, but there are a lot of data fields and potential reports. They migrated from MS SQL Sever and constantly evolving ETL jobs to Hadoop and MongoDB with a lot of success.

source:

• http://www.informationweek.com/software/information-management/making-the-case-for-hadoop-variety-not-volume/d/d-id/1112894

Summary of Terradata’s big data approach

• Terradata Aster 6 platform
• Includes graph analysis engine (visualization), in addition to traditional rows/columns.
• Enables execution of SQL across multiple NoSQL repositories
• Integrates with multiple 3rd parties for solutions such as analytical workflow (Alteryx), advanced analytics algorithms (Fuzzy Logix).
• Cloud services at comparable cost to on-premises

Source

• http://www.information-management.com/blogs/teradata-establishes-trust-in-big-data-technology-10025111-1.html

 

Databricks to commercialize Spark and Shark in-memory processing

Shark utilizes in-memory SQL queries for complex analytics, and is Apache Hive compatible. The name “Shark” is supposed to be short hand for “Hive on Spark”. This seems to be a competitor to Cloudera Impala or the Hortonworks implementation of Hive.

Apache Spark utilizes APIs (Python, Scala, Java) for in-memory processing with very fast reads and writes, claiming to be 100x faster than disk-based MapReduce. Spark is the engine behind Shark. Spark can be considered as an alternative to MapReduce, not an alternative to Hadoop.

Scala is an interesting language being used by companies such as Twitter as both higher performance and easier to write than Java. Some companies that had originally developed using Rails or C++ are migrating to Scala rather than to Java.

Source:

• http://gigaom.com/2013/09/25/databricks-raises-14m-from-andreessen-horowitz-wants-to-take-on-mapreduce-with-spark/
• http://spark.incubator.apache.org/index.html
• https://amplab.cs.berkeley.edu/projects/shark-making-apache-hive-run-at-interactive-speeds/
• http://databricks.com/company.html
• http://www.scala-lang.org/

10 Key/Value Store, Distributed, Open Source Databases

Riak

• HTTP API
• Master-less, so remains operational even if multiple nodes fail
• Near linear scalability
• Architecture same of both large and small clusters
• Key/value model, flat namespace, can store anything

Redis

• Key/value. Can store data types such as sets, sorted lists, hashes and do operations on them such as set intersection and incrementing the value in a hash.
• In-memory dataset
• Easy to setup, master/slave replication

Hibari

• Very simple data model with 5 attributes: keys, values, timestamps, expiry date, flags for metadata
• Chain replication across nodes that are geographically dispersed. Not single points of failure
• Excellent performance for large batches (~200k) read/write operations
• Runs on commodity hardware or blades. Does not require SAN

Hypertable

• High performance, massively scalable, modeled after Google’s Bigtable
• Runs on top of a distributed file system such as Apache Hadoop DFS, GlusterDS, or Kosmos File System
• Data model is a traditional, but huge table, that is physically stored in sort order of the primary key

Voldemort

• High scalability due to allowing only very simple key/value data access.
• Used by LinkedIn
• Not an object or a relational database. Just a big, distributed, fault-tolerant, persistent hash table
• Includes in-memory caching, so separate caching tier isn’t required

MemcacheDB

• High performance persistent storage that’s compatible with Memcache protocol

Tarantool

• NoSQL database with messaging server
• All data maintained in RAM. Persistence via a write ahead log.
• Asynchronous replication and hot standby
• Supports stored procedures
• Data model: tuples (unique key plus any number of other fields); spaces (multiple tuples)

Apache Cassandra

• Can use massive cluster of commodity servers with no single point of failure. Can be deploy across multiple data centers.
• Was used by Facebook for Inbox Search until 2010
• Read/write scales linearly with number of nodes
• Data replicated across multiple nodes
• Supports MapReduce, Pig, and Hive
• Has SQL-like CQL providing for a hybrid between key/value and tabular database

HyperDex

• NoSQL key/value that provides lower latency and higher throughput than some alternatives
• Replicates data to multiple nodes
• Very easy to administer and maintain
• Data model: key plus zero or more attributes

Lightcloud

• Great performance even on small clusters with millions of keys
• Nodes replicated via master-to-master replication.  Hot backups and restores
• Very small client footprint
• Built on top of Tokyo Tyrant

Sources:

• http://www.toolsjournal.com/integrations-articles/item/1852-10-key-value-store-distributed-open-source-databases-for-your-bigdata-needs
• http://basho.com/riak/
• http://redis.io/
• https://github.com/hibari/hibari
• http://hypertable.com/
• http://www.project-voldemort.com/voldemort/
• http://memcachedb.org/
• http://tarantool.org/
• http://cassandra.apache.org/
• http://hyperdex.org/

Cassandra – NoSQL database to use in conjunction with Hadoop

Some use cases feed data directly into Hadoop from their source (such as web server logs), but others feed into Hadoop from a database repository. Still others have use cases in which there is a massive output of data that needs to be stored somewhere for post-processing. One model for handling this dataset is a NoSQL database, as opposed to SQL or flat files.

Cassandra is an Apache project that is popular for its integration into the Hadoop ecosystem. It can be used with components such as Pig, Hive, and Oozie. Cassandra is often used as a replacement for HDFS and HBase since Cassandra has no master node, so eliminates a single point of failure (and need for traditional redundancy). In theory, its scalability is strictly linear; doubling the number of nodes will exactly double the number of transactions that can be processed per second. It also supports triggers; if monitoring detects that triggers are running slowly, then additional nodes can be programmatically deployed to address production performance problems.

Cassandra was first developed by Facebook. The primary benefit of its easily distributed infrastructure is the ability to handle large amount of reads and writes. The newest version (2.0) solves many of the usability problems encountered by programmers.

DataStax provides a commercially packaged version of Cassandra.

MongoDB is a good non-HBase alternative to Cassandra.

Sources:

• http://www.arnnet.com.au/article/525504/apache_cassandra_2_0_brings_old_school_sql_functionality/
• http://wiki.apache.org/cassandra/HadoopSupport
• http://www.dbta.com/Articles/Columns/Notes-on-NoSQL/Cassandra-and-Hadoop—Strange-Bedfellows-or-a-Match-Made-in-Heaven-75890.aspx
• http://www.datastax.com/big-data-challenges
• http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html

HortonWorks trying to make Hive faster, contrasting it to Impala

Hive was invented by Facebook as a data warehouse layer on top of Hadoop, and has been adopted by HortonWorks. The benefit of Hive is that it enables programmers, with years of experience in relational databases, to write MapReduce jobs using SQL. The problem is that MapReduce is slow, and Hive slows it down even further.

HortonWorks is pushing for optimization (via project Stinger) of the developer friendly toolset provided by Hive. Cloudera has abandoned Hive in favor of Impala. Rather than translate SQL queries into MapReduce, Impala implements a massively parallel relational database on top of HDFS.

Sources:

• http://gigaom.com/2013/09/06/hortonworks-effort-to-speed-up-hive-is-coming-along-nicely/
• http://gigaom.com/2013/05/29/why-hortonworks-is-riding-a-faster-hive-to-the-bitter-end/
• http://hortonworks.com/stinger/
• http://gigaom.com/2013/04/30/with-impala-now-ga-clouderas-ceo-sizes-up-the-sql-on-hadoop-market/

What’s the root of the differences been Big Data and Relational Database

Database World

• Encode the relationships between objects in tables, and use keys to link the tables together
• Standard query language (with emphasis on standard, applying to all database vendors, versions, implementations, programmers) relies on the relationship encoding and vendor architecture for optimization/efficiency
• Algorithms rely on a single pass execution, using operations such as Joins and Group Bys and Counts.

Big Data World

• Based on linear algebra and probability theory
• Encode objects using a property list
• Data  stored as a matrix, similar to relational tables, except that the intersection of multiple matrices does not imply relationships
• Algorithms have iterative solutions with multiple steps each of which store results that are used as input by the next step, which is very inefficient to execute in SQL
• Indices are not needed, since massively scaled hardware will be used to process the entire data set by brute force or by intelligent jobs (on the front side in Map or the back side in Reduce).

Either you structure your data ahead of time so that SQL algorithms will work, or you break down your algorithms in to algebra (MapReduce jobs) in order to process semi-structured data.

Where does this leave systems like Hive, that enable programmers to write something that looks like SQL and is transformed on the backend into MapReduce jobs? Maybe purists don’t like Hive because it’s used by people on the fence between Database and Big Data, instead of those who have fully converted to Big Data?

Systems similar yet different from Hadoop/MapReduce. They claim to be Big Data, but have roots in the database world.

• Twitter’s Storm/Summingbird is event driven (not batch) so can target real time applications
• Spark uses iterative algorithms and in-memory processing with the goal of being a few orders of magnitude faster than MapReduce

Source:

• http://blog.mikiobraun.de/2013/09/data-base-vs-data-science.html

Hive can be used to program MapReduce using a subset of SQL

Hive enables MapReduce to be programmed using something that looks like SQL, instead of a procedural language like Java or Python. This is useful if a team of database, as opposed to application, programmers are called upon to program MapReduce.

Using Hive tables requires defining a schema.

The SQL-like language (called HiveQL) is converted to a MapReduce job.

Hue is a browser based GUI within which you can do Hive work. You type your query and see tabular results. Hue has ODBC drivers, and can export a CSV to Excel.

The Apache page for Hive calls it “a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets.” I’m not sure how the data warehouse piece applies.

Source:

• http://hive.apache.org