QuickSpecs

ManualsBrandsCompaq ManualsComputer equipmentProLiant SL4540

Technical white paper

HP Reference Architecture for

Hortonworks Data Platform on

HP ProLiant SL4540 Gen8

Server

HP Converged Infrastructure with the Hortonworks Data Platform for Apache Hadoop

Table of contents

Executive summary ...................................................................................................................................................................... 2

HP SL4500 Series ..................................................................................................................................................................... 3

Hortonworks Data Platform overview ...................................................................................................................................... 3

High-availability considerations ................................................................................................................................................. 5

Pre-deployment considerations ................................................................................................................................................ 6

Operating system ...................................................................................................................................................................... 6

Computation .............................................................................................................................................................................. 6

Memory ....................................................................................................................................................................................... 6

Storage ........................................................................................................................................................................................ 7

Network ....................................................................................................................................................................................... 7

Switches ...................................................................................................................................................................................... 8

HP Insight Cluster Management Utility ..................................................................................................................................... 8

Server selection ........................................................................................................................................................................... 10

Management nodes ................................................................................................................................................................ 10

Worker nodes ........................................................................................................................................................................... 14

Reference Architectures ............................................................................................................................................................ 17

Single Rack configuration ...................................................................................................................................................... 17

Multi-Rack Cluster configuration ......................................................................................................................................... 20

HP Vertica and Hadoop .............................................................................................................................................................. 22

Summary ....................................................................................................................................................................................... 22

For more information ................................................................................................................................................................. 23

Summary of content (23 pages)

PAGE 1
Technical white paper HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server HP Converged Infrastructure with the Hortonworks Data Platform for Apache Hadoop Table of contents Executive summary ...................................................................................................................................................................... 2 HP SL4500 Series ...............................................................................................
PAGE 2
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Executive summary HP and Apache Hadoop allow you to derive new business insights from Big Data by providing a platform to store, manage and process data at scale. However, Apache Hadoop is complex to deploy, configure, manage and monitor.
PAGE 3
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server – The HP iLO Management Engine on the servers is a comprehensive set of embedded management features, including HP Integrated Lights-Out 4 (iLO 4), Agentless Management, Active Health System, and Intelligent Provisioning which reduces node and cluster level administration costs for Hadoop.
PAGE 4
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server The Hortonworks Data Platform consists of the essential set of Apache Hadoop projects including Hortonworks MapReduce, Hadoop Distributed File System (HDFS), HCatalog, Pig, Hive, HBase, ZooKeeper and Ambari. Hortonworks is the major contributor of code and patches to many of these projects.
PAGE 5
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Typically, Hadoop clusters are used for batch MapReduce analysis of data or they are used to run HBase, which is an online distributed store for reading and writing structured data. The Hadoop Master services (NameNode, JobTracker and HBase Master) are installed and configured on HP DL360p head nodes.
PAGE 6
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Pre-deployment considerations There are a number of key factors you should consider prior to designing and deploying a Hadoop Cluster. The following subsections articulate the design decisions in creating a balanced baseline configuration for the reference architectures.
PAGE 7
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server In addition, the Hortonworks HDP cluster can also be used for HBase which is very memory intensive, which is why we recommend 96GB of RAM per server. Storage Fundamentally, Hortonworks Data Platform (HDP) is designed to achieve performance and scalability by moving the compute activity to the data.
PAGE 8
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server amount of remote-rack accesses increases for larger clusters, it is expected to put a relatively small load on the ToR and core switches. The section on Staging data within the Reference Architectures section of the document provides a number of options for network configurations for ingress and egress.
PAGE 9
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Note HP Insight CMU allows you to easily correlate Hortonworks Data Platform (HDP) metrics with cluster infrastructure metrics, such as CPU Utilization, Network Transmit/Receive, Memory Utilization, Power Consumption, Ambient Node and Rack temperature and I/O Read/Write.
PAGE 10
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Figure 5. HP Insight CMU Interface – Time View Server selection Depending on the size of the cluster, a Hadoop deployment consists of one or more nodes running management services and a quantity of worker nodes.
PAGE 11
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Server platform: HP ProLiant DL360p Gen8 The HP ProLiant DL360p Gen8 (1U) is an excellent choice as the server platform for the management nodes. Figure 6.
PAGE 12
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Best practice For each management server we recommend bonding and cabling two of the 10GbE NICs to create a single bonded pair which will provide 20GbE of throughput as well as a measure of NIC redundancy. The Management node in the cluster can be leveraged for ingress and staging data. In the reference configurations later in the document you will notice that we use two IRF Bonded switches.
PAGE 13
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Note: The Hortonworks Management Center enables the deployment of the cluster as well as which additional components to install. Please see the following link for deploying Apache Pig and Apache Hive with HMC, http://docs.hortonworks.com/CURRENT/index.htm#Deploying_Hortonworks_Data_Platform/Using_HMC/Configuring_And_ Deploying_The_Cluster/Configuring_and_Deploying_the_Cluster.
PAGE 14
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server NameNode server The NameNode server contains the following software. Please see the following link for more information on installing and configuring the NameNode, http://docs.hortonworks.com/CURRENT/index.htm#Deploying_Hortonworks_Data_Platform/Using_HMC/Configuring_And_ Deploying_The_Cluster/Configuring_and_Deploying_the_Cluster.htm Table 10.
PAGE 15
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Figure 7. HP ProLiant SL4540 Gen8 (3 x 15) Server Processor configuration The configuration for each server node in the SL4540 features two sockets with 8 core processors and the Intel C600 Series chipset which provide 16 physical or 32 Hyper-Threaded cores per server.
PAGE 16
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Best practice It is important to saturate all the memory channels available to ensure optimal use of the memory bandwidth.
PAGE 17
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Best practice Customers have the option of purchasing a second power supply for additional power redundancy. This is especially appropriate for single rack clusters where the loss of a node represents a noticeable percentage of the cluster. Table 12.
PAGE 18
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Best practice Once a Hortonworks HDP cluster is isolated, the users of the cluster will still need a way to access the cluster and submit jobs to it. To achieve this we recommend multi-homing the Management node so that it participates in both the cluster subnet and a subnet belonging to the users of the cluster.
PAGE 19
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Power and cooling In planning for large clusters, it is important to properly manage power redundancy and distribution. To ensure the servers and racks have adequate power redundancy we recommend that each server have a backup power supply, and each rack have at least two Power Distribution Units (PDUs). There is an additional cost associated with procuring redundant power supplies.
PAGE 20
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Figure 10. Single Rack Hortonworks HDP Configuration – Rack Level View Management Node 1 x HP ProLiant DL360p Gen8 with 12 cores 2 x E5-2667 (Intel Xeon 2.9GHz, 15M Cache) 64 GB - 8 x HP 8GB 1Rx4 PC3-12800R 7.2 TB - 8 x HP 900GB 6G SAS 10K 2.
PAGE 21
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server Figure 11. Large Cluster Hortonworks HDP Configuration Figure 12.
PAGE 22
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server HP Vertica and Hadoop Relational database management systems such as HP Vertica excel at analytic processing for big volumes of structured data including call detail records, financial tick streams and parsed weblog data. HP Vertica is designed for high speed load and query when the database schema and relationships are well defined.
PAGE 23
Technical white paper | HP Reference Architecture for Hortonworks Data Platform on HP ProLiant SL4540 Gen8 Server For more information Hortonworks, hortonworks.com HP Solutions for Apache Hadoop, hp.com/go/hadoop HP Insight Cluster Management Utility (CMU), hp.com/go/cmu HP ProLiant SL4540 Gen8, hp.com/servers/sl4540 HP ProLiant DL380e Gen8, hp.com/servers/dl380e HP ProLiant servers, hp.com/go/proliant HP Enterprise Software, hp.com/go/software HP Networking, hp.