Taking a Look at the HDFS File System

Storage of data plays a major role in improving the performance of a company and this can happen either offline or online and in various formats.

Parallel clustering: the database application is run parallel on both hosts. The difficulty in implementing parallel clusters is providing some form of distributed locking mechanism for files on the shared disk.

When a file is written in HDFS, it is divided into fixed size blocks. The client first contacts the NameNode, which get the list of DataNode where actual data can be stored. The data blocks are distributed across the Hadoop cluster. Figure \ref{fig.clusternode} shows the architecture of the Hadoop cluster node used for both computation and storage. The MapReduce engine (running inside a Java virtual machine) executes the user application. When the application reads or writes data, requests are passed through the Hadoop \textit{org.apache.hadoop.fs.FileSystem} class, which provides a standard interface for distributed file systems, including the default HDFS. An HDFS client is then responsible for retrieving data from the distributed file system by contacting a DataNode with the desired block. In the common case, the DataNode is running on the same node, so no external network traffic is necessary. The DataNode, also running inside a Java virtual machine, accesses the data stored on local disk using normal file I/O

HDFS uses NameNode operation to realize data consistency. NameNodes utilizes a transactional log file to record all the changes of

There are two ways in which the cluster programming can oversee access to the information on the disk.

GFS: Google File System is a distributed file system which is developed by Google in order to provide efficient, reliable access to data. . It is designed and implemented inorder to meet the requirements provided by Google’s data processing. The file system consists of hundreds of storage machines to provide inexpensive parts and it is accessed by different client machines. Here the search engine is providing huge amounts data that should be stored. GFS has 1,000 nodes with 300TB disk storage.

HDFS is Hadoop’s distributed file system that provides high throughput access to data, high-availability and fault tolerance. Data are saved as large blocks making it suitable for applications

Hadoop1 provides a distributed filesystem and a framework for the analysis and transformation of very large data sets using the MapReduce [DG04] paradigm. While the interface to HDFS is patterned after the Unix filesystem, faithfulness to standards was sacrificed in favor of improved performance for the applications at hand.

HFS+ is file system developed by apple to replace their Hierarchical file system as the primary file system used in Mac computers It is also used by IPod and it is referred to as Mac OS extended.

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a Parallel and distributed computing environment. It makes Use of the commodity hardware Hadoop is Highly Scalable and Fault Tolerant. Hadoop runs in cluster and eliminates the use of a Super computer. Hadoop is the widely used big data processing engine with a simple master slave setup. Big Data in most companies are processed by Hadoop by submitting the jobs to Master. The Master distributes the job to its cluster and process map and reduce tasks sequencially.But nowdays the growing data need and the and competition between Service Providers leads to the increased submission of jobs to the Master. This Concurrent job submission on Hadoop forces us to do Scheduling on Hadoop Cluster so that the response time will be acceptable for each job.

Listed below are the concepts that should be learned in order to properly understand HADOOP technology.

Hadoop is a great data storage choice and Hadoop Distributed File System (HDFS) or Hive is often used to store transactional data in its raw state. The map-reduce processing supported by these Hadoop frameworks can deliver great performance, but it does not support the same specialized query optimization that mature relational database technologies do. Improving query performance, at this time, requires acquiring query accelerators or writing code. Every company who chose to use Hadoop needs to optimize their architecture in a way compatible to Hadoop.

Abstract - Hadoop Distributed File System, a Java based file system provides reliable and scalable storage for data. It is the key component to understand how a Hadoop cluster can be scaled over hundreds or thousands of nodes. The large amounts of data in Hadoop cluster is broken down to smaller blocks and distributed across small inexpensive servers using HDFS. Now, MapReduce functions are executed on these smaller blocks of data thus providing the scalability needed for big data processing. In this paper I will discuss in detail on Hadoop, the architecture of HDFS, how it functions and the advantages.

Hierarchical File System (HFS) is a proprietary file system developed by Apple Inc. for use in computer systems running Mac OS. Originally designed for use on floppy and hard disks, it can also be found on read-only media such as CD-ROMs. HFS is also referred to as Mac OS Standard (or, erroneously, "HFS Standard"), while its successor, HFS Plus, is also called Mac OS Extended

Storage Modeling: MapR has distributed namenode architecture, which removes the single point of failure that plagues HDFS. MapR’s Lockless Storage Services layer results in higher MapReduce throughput than competing distributions. It has ability