FUSE security

CDH enables use of Kerberos to securely mount a filesystem via FUSE

Source
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_12.html

HDFS NFS Proxy is an NFS4 server for HDFS

Clients should access an HDFS mount using Fuse for Production use, but the NFS proxy gets you initially deployed faster. Here’s how.

Source:
https://github.com/cloudera/hdfs-nfs-proxy/wiki

In-memory Hadoop – use it when speed matters

GridGain has a 100% HDFS compatible RAM solution that it claims is 10x faster for IO and network intensive MapReduce processing. I understand the IO, but am not sure why it work help with network intensive operations.  It can be used standalone or along with disk based HDFS as a cache. It is compatible with all Hadoop distributions as well as standard tools like HBase, Hive, etc.

Source:

• http://ctovision.com/2013/09/in-memory-hadoop-use-it-when-speed-matters/

Email indexing using Cloudera Search

This article from Cloudera offers up use cases (such as customer sentiment) and a tutorial for using Apache Flume for near-real-time indexing (as emails arrive on your mail server) or MapReduce (actually MapReduceIndexerTool) for batch indexing of email archives. The two methods can be combined if you decide to do real-time, but later decide to add another MIME header field into the index.

Cloudera Search is based on Apache Solr (which contains components like Apache Lucene, SolrCloud, Apache Tika, and Solr Cell).

The email (including the MIME header) is parsed (with the help of Cloudera Morphlines), then uses Flume to push the messages into HDFS, as Solr intercepts and indexes the contents of the email fields.

Searching and viewing the results can be done using the Solr GUI or Hue’s search application.

Sources:

• http://blog.cloudera.com/blog/2013/09/email-indexing-using-cloudera-search/
• http://www.cloudera.com/content/cloudera/en/products/cdh/search.html
• http://lucene.apache.org/solr/
• http://lucene.apache.org/
• http://wiki.apache.org/solr/SolrCloud
• http://tika.apache.org/
• http://wiki.apache.org/solr/ExtractingRequestHandler
• http://cloudera.github.io/hue/

eBay discusses failover and time to recovery with HBase containing tens of petabytes of data

eBay worked with HortonWorks and ScaledRisk to improve Mean Time to Recovery (MTTR). Not only did this require faster recovery time, but also faster detection of failures.

The types of failures considered included the following, but only Node/Region server failures were included in the tests. The HBase tables contained 900 million rows.

• Node/Region server failed while writing
• Node/Region server failed while reading
• Rack failure
• Whole cluster failure
• Machine reboot (due to CPU temperature)
• NIC speed steps down to 100Mb/s from gigabit speeds

The tests had favorable results, with improvements submitted (some implemented, some proposed) into Apache HBase and HDFS.

Sources:

• http://hortonworks.com/blog/hortonworks-scaled-risk-and-ebay-collaborate-to-improve-hbase-mean-time-to-recovery-mttr/
• http://www.scaledrisk.com/

HortonWorks tutorial on streaming server log data into HDFS using Flume

Source:

• http://hortonworks.com/blog/hadoop_tutorial_visualizing_server_logs/

Cloudera Search Engine

Cloudera has announced a realtime search engine running on top of HBase and HDFS, enabling natural language keyword searches.

Indices are stored in HDFS and indexing takes place in batches using MapReduce. Realtime indexing happens via Flume and the Lily HBase indexer.

Source:

• http://www.drdobbs.com/database/first-hadoop-search-engine-launched-by-c/240161081

Cassandra – NoSQL database to use in conjunction with Hadoop

Some use cases feed data directly into Hadoop from their source (such as web server logs), but others feed into Hadoop from a database repository. Still others have use cases in which there is a massive output of data that needs to be stored somewhere for post-processing. One model for handling this dataset is a NoSQL database, as opposed to SQL or flat files.

Cassandra is an Apache project that is popular for its integration into the Hadoop ecosystem. It can be used with components such as Pig, Hive, and Oozie. Cassandra is often used as a replacement for HDFS and HBase since Cassandra has no master node, so eliminates a single point of failure (and need for traditional redundancy). In theory, its scalability is strictly linear; doubling the number of nodes will exactly double the number of transactions that can be processed per second. It also supports triggers; if monitoring detects that triggers are running slowly, then additional nodes can be programmatically deployed to address production performance problems.

Cassandra was first developed by Facebook. The primary benefit of its easily distributed infrastructure is the ability to handle large amount of reads and writes. The newest version (2.0) solves many of the usability problems encountered by programmers.

DataStax provides a commercially packaged version of Cassandra.

MongoDB is a good non-HBase alternative to Cassandra.

Sources:

• http://www.arnnet.com.au/article/525504/apache_cassandra_2_0_brings_old_school_sql_functionality/
• http://wiki.apache.org/cassandra/HadoopSupport
• http://www.dbta.com/Articles/Columns/Notes-on-NoSQL/Cassandra-and-Hadoop—Strange-Bedfellows-or-a-Match-Made-in-Heaven-75890.aspx
• http://www.datastax.com/big-data-challenges
• http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html

HortonWorks trying to make Hive faster, contrasting it to Impala

Hive was invented by Facebook as a data warehouse layer on top of Hadoop, and has been adopted by HortonWorks. The benefit of Hive is that it enables programmers, with years of experience in relational databases, to write MapReduce jobs using SQL. The problem is that MapReduce is slow, and Hive slows it down even further.

HortonWorks is pushing for optimization (via project Stinger) of the developer friendly toolset provided by Hive. Cloudera has abandoned Hive in favor of Impala. Rather than translate SQL queries into MapReduce, Impala implements a massively parallel relational database on top of HDFS.

Sources:

• http://gigaom.com/2013/09/06/hortonworks-effort-to-speed-up-hive-is-coming-along-nicely/
• http://gigaom.com/2013/05/29/why-hortonworks-is-riding-a-faster-hive-to-the-bitter-end/
• http://hortonworks.com/stinger/
• http://gigaom.com/2013/04/30/with-impala-now-ga-clouderas-ceo-sizes-up-the-sql-on-hadoop-market/

Good article about how the NSA replicates data from Yahoo and Google Hadoop file systems

Most of the discussions about NSA data collection are devoid of technical facts. The media just likes to throw around the word “metadata” as if that means nothing to those of us who work all day with nothing other than metadata.

Here’s an article that doesn’t talk down to us, but explains how simple it is to replicate the HDFS nodes from Yahoo and Google data centers.

The problem seems to be that Yahoo and Google encrypt data in motion, but not data at rest. Would Accumulo solve the encryption problem for data at rest? However,  Accumulo was originally developed for the NSA, who can likely break the encryption using the processing power of huge Hadoop clusters.

Source:

• http://www.cringely.com/2013/11/02/google-file-system-makes-nsas-hack-blatantly-illegal-know/
• http://accumulo.apache.org/