Hadoop news

Hadoop

hadoop Authors: Liz McMillan, Greg Schulz, Jim Kaskade, Jnan Dash, Dana Gardner

Related Topics: Big Data on Ulitzer, Hadoop

BigData: Article

Oracle Database and Big Data: A Powerful Combination

Use Oracle database/Big Data combo to process huge volumes of data, or power of Google at your fingertips
By Ranko Mosic
Article Rating:
October 27, 2011 02:57 PM EDT
Reads:
 10,210

Ever wondered how it is possible that Google searches through so much data with such speed and precision ?

Part of the answer is MapReduce, Google technology for processing and generating large data sets.

Apache Hadoop is open source software that can process petabytes of data in parallel on hundreds and thousands of commodity hardware nodes. It was inspired by Google MapReduce. Oracle corporation is acknowledging the power of Oracle/Hadoop combination by announcing Big Data Appliance - essentially Hadoop/Oracle database software/Oracle hardware bundle, to be available next year.

Oracle database is an RDBMS which can be very slow when processing really big volumes of data. If tables become larger then couple of dozen GBs then you have to start using partitioning, index very carefully, get to know inner workings of query optimizer so that queries can be structured in a proper way, maybe use query hints to improve performance or process data in parallel. Sometimes, no matter what you do, performance will not improve. Batch processes will break through the allocated processing window, queries will take too long to execute and users are not going to be happy.

Analytics or big data processing activities can be performed much more efficiently using Hadoop. You can extract data from Oracle database into Hadoop where it can be efficiently processed in parallel ( MapReduce ). End results can then be uploaded back into Oracle database.

Another case where Hadoop/Oracle can be useful is if you have to process big volumes of raw, unstructured data. Raw data should be preprocessed in Hadoop before results are loaded into Oracle for querying purposes.

A tool named Sqoop can move data between Oracle database and Hadoop. Sqoop dumps data from an Oracle database into Hadoop file system, or exports data from Hadoop into Oracle. Oracle company announced their own version of Loader for Hadoop will be released some time next year ( 2012 ). Quest is offering free Hadoop loader for Oracle.

You can get access to Hadoop cluster on Amazon Web Services ( AWS Elastic MapReduce service ). AWS instant provisioning capabilities make it possible to start hundreds of Hadoop servers to execute data processing job in parallel,  then shut them down once processing is completed, thus enabling large scale computation in a very economical way.

Oracle databases can either reside in your own, private data center and Cloudburst into AWS, or they can be hosted by AWS.

Oracle/Hadoop is a very powerful combination that opens new frontiers in data warehousing, and is available on AWS right now.

Published October 27, 2011 – Reads 10,210
Copyright © 2011 Ulitzer, Inc. — All Rights Reserved.
Syndicated stories and blog feeds, all rights reserved by the author.

More Stories By Ranko Mosic

Ranko Mosic, BScEng, is specializing in Big Data/Data Architecture consulting services ( database/data architecture, machine learning ). His clients are in finance, retail, telecommunications industries. Ranko is welcoming inquiries about his availability for consulting engagements and can be reached at 408-757-0053 or [email protected]