HDP Enabling the Modern Data Architecture

Transcription

1 HDP Enabling the Modern Data Architecture Herb Cunitz President, Hortonworks Page 1

2 Hortonworks enables adoption of Apache Hadoop through HDP (Hortonworks Data Platform) Founded in 2011 Original 24 architects, developers, operators of Hadoop from Yahoo! We are leaders in Hadoop community 500+ employees Customer Momentum 300+ customers in seven quarters, growing at 75+/quarter Two thirds of customers come from F1000 Hortonworks and Hadoop at Scale HDP in production on largest clusters on planet Multiple node clusters, including 35,000 nodes at Yahoo!, 800 nodes at Spotify Page 2

3 Analysts consensus estimates enterprise data growth of! 2013 Digital universe! 2.3 Zettabytes 50x year over year through Digital universe! 40 Zettabytes & Hadoop Market $50B 85% of growth from new types of data with machine-generated data increasing 15x Page 3 1 Zettabyte (ZB) = 1 million Petabytes (PB); Sources: IDC and IDG Enterprise

4 Big Data & Hadoop Market Drivers and Opportunities $ Business Drivers From reactive analytics to proactive customer interaction Insights that drive competitive advantage & optimal returns Technical Drivers Data is growing exponentially & existing systems overwhelmed Predominantly driven by NEW types of data that can inform analytics Financial Drivers Cost of data systems, as % of IT spend, continues to grow Cost advantages of commodity hardware & open source software Page 4

5 A shift from reactive to proactive interactions A shift in Advertising From mass branding A shift in Financial Services From Educated Investing A shift in Healthcare From mass treatment A shift in Retail From static branding A shift in Telco From break then fix to 1x1 Targeting to Automated Algorithms to Designer Medicine to Real-time Personalization to repair before break HDP and Hadoop allow organizations to shift interactions from Reactive Post Transaction Proactive Pre Decision Page 5

6 HDP Realized Cost Savings with EDW Optimization Hadoop Helps you optimize and reduce costs associated with your EDW ANALYTICS Data Marts Business Analytics Visualization & Dashboards Archive Data away from EDW Move cold or rarely used data to Hadoop as active archive Store more of data longer DATA SYSTEMS Systems of Record RDBMS ERP CRM Other Enterprise Data Warehouse Hot HDP 2.2 Cold Data, Deeper Archive & New Sources ELT N Offload costly ETL process Free your EDW to perform high-value functions like analytics & operations, not ETL. Use Hadoop for advanced ETL Optimize the value of your EDW NEW SOURCES Clickstream Web & Social Geoloca3on Sensor & Machine Server Logs Unstructured Use Hadoop to refine new data sources, such as web and machine data for new analytical context Page 6

7 Realize dramatic savings for cost of storage Hadoop Enables Scalable Compute & Storage at a Compelling Cost Structure Cost Efficiencies Reduce costs associated with expensive archive systems Utilize existing relationships with hardware vendors Open Source Software Active Archive Provide access to archived data not just collect dust Cloud Storage HADOOP NAS Engineered System MPP SAN Storage Costs/Compute Costs from $19/GB to $0.23/GB $0 $20,000 $40,000 $60,000 $80,000 $180,000 Fully-loaded Cost Per Raw TB of Data (Min Max Cost) Page 7

8 Unlock New Applications from New Types of Data INDUSTRY USE CASE Sentiment & Web Clickstream & Behavior Machine & Sensor Geographic Server Logs Structured & Unstructured Financial Services Telecom Retail Manufacturing Healthcare Pharmaceuticals Oil & Gas Government New Account Risk Screens Trading Risk Insurance Underwriting Call Detail Records (CDR) Infrastructure Investment Real-time Bandwidth Allocation 360 View of the Customer Localized, Personalized Promotions Website Optimization Supply Chain and Logistics Assembly Line Quality Assurance Crowd-sourced Quality Assurance Use Genomic Data in Medial Trials Monitor Patient Vitals in Real-Time Recruit and Retain Patients for Drug Trials Improve Prescription Adherence Unify Exploration & Production Data Monitor Rig Safety in Real-Time ETL Offload/Federal Budgetary Pressures Sentiment Analysis for Government Programs Page 8

9 End Game: Data Lake - An architectural shift SCALE Unlocking the Data Lake RDBMS MPP EDW Governance & Integration HDP 2.1 Data Access YARN Data Management Security Operations Data Lake Enabled by YARN Single data repository, shared infrastructure Multiple biz apps accessing all the data New Analytic Apps or IT Optimization Enable a shift from reactive to proactive interactions Gain new insight across the entire enterprise SCOPE Page 9

10 Case Study: 12 month Hadoop evolution at TrueCar 12 Month Results at TrueCAR Six Production Hadoop Applications Sixty nodes/2pb data Storage Costs/Compute Costs from $19/GB to $0.23/GB May 14 IPO Data Platform Capabilities We addressed our data platform capabilities strategically as a pre-cursor to IPO. June 2013 Begin Hadoop Execution July 2013 Hortonworks Partnership Aug 2013 Training & Dev Begins Nov 2013 Production Cluster 60 Nodes 2 PB Dec 2013 Three Production Apps (3 total) Jan % Dev Staff Perficient Feb 2014 Three More Production Apps (6 total) 12 months execution plan Page 10

11 Enterprise Goals for the Modern Data Architecture APPLICATIONS Business Analytics Custom Applications Packaged Applications Consolidate siloed data sets structured and unstructured Central data set on a single cluster DATA SYSTEM RDBMS EDW MPP CRM ERP Other Batch Interactive Real-Time YARN: Data Operating System HDFS (Hadoop Distributed File System) 1 1 N Multiple workloads across batch interactive and real time Central services for security, governance and operation Preserve existing investment in current tools and platforms SOURCES EXISTING Systems Clickstream Web &Social Geoloca3on Sensor & Machine Server Logs Unstructured Single view of the customer, product, supply chain Page 11

12 HDP delivers a comprehensive data management platform GOVERNANCE Hortonworks Data Platform 2.2 BATCH, INTERACTIVE & REAL-TIME DATA ACCESS SECURITY OPERATIONS YARN is the architectural center of HDP Data Workflow, Lifecycle & Governance Falcon Sqoop Flume Kafka NFS WebHDFS Script Pig Tez SQL Hive Tez Java Scala Cascading Tez NoSQL HBase Accumulo Slider Stream Storm Slider In-Memory Spark YARN: Data Operating System (Cluster Resource Management) Search Solr Others ISV Engines Authentication Authorization Accounting Data Protection Storage: HDFS Resources: YARN Access: Hive, Pipeline: Falcon Cluster: Knox Cluster: Ranger Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie Enables batch, interactive and real-time workloads Provides comprehensive enterprise capabilities 1 HDFS (Hadoop Distributed File System) Deployment Choice Linux Windows On-Premises Cloud The widest range of deployment options Delivered Completely in the OPEN Page 12

13 HDP is deeply integrated in the data center APPLICATIONS DATA SYSTEM BusinessObjects BI RDBMS EDW MPP HANA Governance & Integration HDP 2.1 Data Access YARN Data Management Security Operations DEV & DATA TOOLS OPERATIONAL TOOLS INFRASTRUCTURE Deep Partnerships Hortonworks engages in deep engineered relationships with the leaders in the data center, such as Microsoft, Teradata, Redhat, HP, SAS & SAP Broad Partnerships Over 600 partners work with us to certify their applications to work with Hadoop so they can extend big data to their users SOURCES EXISTING Systems Clickstream Web &Social Geoloca3on Sensor & Machine Server Logs Unstructured Page 13

14 Our Mission To enable Apache Hadoop to be the enterprise data platform that powers the modern data architecture and processes half the worlds data Page 14

15 Our Strategy: A Commitment to Enterprise Hadoop 1 Innovate the Core 2 Extend Hadoop as an Enterprise Data Platform 3 Enable the Ecosystem Architect and build innovation at the core of Hadoop YARN transformed Hadoop to enable multiple workloads across a multi-tenant architecture Extend Hadoop with enterprise capabilities for governance, security & operations Apply enterprise software rigor to the open source development process Enable the leaders in the data center to easily adopt & extend their platforms Establish Hadoop as standard component of a modern data architecture Joint engineering Batch Interactive Real-Time HDP 2.1 YARN: Data Operating System HDFS (Hadoop Distributed File System) 1 N Governance & Integration Data Access YARN Data Management Security Operations Page 15

16 4 all done completely in Open Source Innovating within the community for the enterprise Open Source: fastest path to innovation for a platform technology Complete open source platform speeds enterprise and ecosystem adoption and minimizes lock in Enables the market to function much bigger much faster Batch Map Reduce Script Pig SQL Hive/Tez, HCatalog NoSQL HBase Accumulo Stream Storm Memory Spark YARN : Data Opera3ng System HDFS (Hadoop Distributed File System) Contributes more to the Apache Hadoop ecosystem in the ASF than any other vendor Governance & Integration HDP 2.1 Data Access YARN Data Management Security Operations Driving our innovation through Apache Software Foundation Projects Apache Project Committers PMC Members Hadoop Tez Hive 15 5 HBase 7 3 Pig 5 5 Accumulo 2 2 Flume 1 0 Storm 2 2 Sqoop 1 0 Ambari Oozie 3 2 Zookeeper 2 1 Knox 6 6 Falcon 3 3 TOTAL Page 16

17 A Leader in Hadoop Hortonworks loves and lives open source innovation The Forrester Wave Big Data Hadoop Solutions Q World Class Support and Services. Hortonworks' Customer Support received a maximum score and was significantly higher than both Cloudera and MapR Page 17

18 Batch Interactive Real-Time YARN: Data Operating System HDFS (Hadoop Distributed File System) 1 1 N Thank You, Questions Page 18

19 2. Or to realize a dramatic cost savings EDW Optimization OPERATIONS 50% ANALYTICS 20% ETL PROCESS 30% Current Reality EDW at capacity: some usage from low value workloads Older data archived, unavailable for ongoing exploration Source data often discarded OPERATIONS 50% ANALYTICS 50% Hadoop Parse, Cleanse Apply Structure, Transform Augment w/ Hadoop Free up EDW resources from low value tasks Keep 100% of source data and historical data for ongoing exploration Mine data for value after loading it because of schema-on-read Commodity Compute & Storage Hadoop Enables Scalable Compute & Storage at a Compelling Cost Structure Cloud Storage HADOOP NAS Engineered System MPP Storage Costs/Compute Costs from $19/GB to $0.23/GB SAN $0 $20,000 $40,000 $60,000 $80,000 $180,000 Fully-loaded Cost Per Raw TB of Data (Min Max Cost) Page 19