Big Data and Hadoop

HDFS Architecture

Deprecated Properties The following table lists the configuration property names that are deprecated in this version of Hadoop, and their replacements. Deprecated property name New property name create.empty.dir.if.nonexist mapreduce.jobcontrol.createdir.ifnotexist dfs.access.time.precision dfs.namenode.accesstime.precision dfs.backup.address dfs.namenode.backup.address dfs.backup.http.address dfs.namenode.backup.http-address dfs.balance.bandwidthPerSec dfs.datanode.balance.bandwidthPerSec dfs.block.size dfs.blocksize dfs.data.dir dfs.datanode.data.dir dfs.datanode.max.xcievers dfs.datanode.max.transfer.threads dfs.df.interval fs.df.interval dfs.encryption.key.provider.uri hadoop.security.key.provider.path dfs.federation.nameservice.id dfs.nameservice.id dfs.federation.nameservices dfs.nameservices dfs.http.address dfs.namenode.http-address dfs.https.address dfs.namenode.https-address dfs.https.client.keystore.resource dfs.client.https.keystore.resource dfs.https

For getting Information related to Unix Shell used in Hadoop or HDFS operation, get in touch with this link :- Unix Shell Guide : - HADOOP

FOR YOU ALL , I HAVE DEVELOPED ONE ANDROID APP FOR THIS BLOG WHICH WILL GIVE YOU DIRECTLY UPDATES RELATED TO THIS BLOG. SO PLEASE DOWNLOAD FROM HERE AND USE THIS APP. ANDROID APP APK FOR BIG DATA HADOOP FOR U

VI EDITOR COMMANDS VI COMMANDS BIBLE OF HADOOP : THE DEFINITIVE GUIDE OF HADOOP V4 MY BDHAD LOGO

REF :- HADOOP : THE DEFINITIVE GUIDE :- BY TOM WHITE ( THE BIBLE OF HADOOP ) [ Relational Database Management Systems Why can’t we use databases with lots of disks to do large-scale analysis? Why is Hadoop needed? The answer to these questions comes from another trend in disk drives: seek time is improving more slowly than transfer rate. Seeking is the process of moving the disk’s head to a particular place on the disk to read or write data. It characterizes the latency of a disk operation, whereas the transfer rate corresponds to a disk’s bandwidth. If the data access pattern is dominated by seeks, it will take longer to read or write large portions of the dataset than streaming through it, which operates at the transfer rate. On the other hand, for updating a small proportion of records in a database, a traditional B-Tree (the data structure used in relational databases, which is limited by the rate at which it can perform seeks) works well. For updating the majority of a datab

FIRST THING IN OUR MIND THAT WHERE TO USE THIS HADOOP   Let us see and consider one use case. Suppose You are the Manager of an Organisation, and you want how many times these words like POWER FAILURE, CSAT NO , REFUND used by his customers in their emails. If we had structured data than it can easily managable without using Hadoop provided Volume warehouse management proportions, but we have Unstructured data  in emails, so we will have to solve this problem using Hadoop. Solutions would be. As we know that Hadoop Components have two things. 1) HDFS ( HADOOP DISTRIBUTED FILE SYSTEM ) for Distributed storage. 2) MAPREDUCE for prarallel data processing. So execution steps. Load the data into HDFS. Analyse the data in Parallel using MAP-REDUCE. Store results back into HDFS. Retrieve results and Visualize. So, We need to know deeply that What is HDFS exactly and how is it processed and stored using its Architecture. Next➱

Seeds of Hadoop had planted in 2002 for better understanding and exploring Open Source Technology in Big Data world. Hadoop had started by SIR DOUG CUTING AND MIKE CAFARELLA who were the Graduate student of University of Washington for the project NUTCH which was the sub project of APACHE LUCENE.