What Next for DBAs in the Big Data Era

Transcription

1 What Next for DBAs in the Big Data Era February 21 st, 2015 Copyright Apps Associates LLC. 1

2 Satyendra Kumar Pasalapudi Associate Practice Director Apps Associates Co Founder & President of Copyright Apps Associates LLC. 2

3 Agenda Technology Trends Big Data Overview Hadoop Basics NoSQL Databases Big Data Sql What Next for DBAs Copyright Apps Associates LLC. 4

4 Big data Challenge Cost effectively manage and analyze all available data in its native form unstructured, structured, streaming Website Social Media Billing ERP CRM RFID Network Switches Copyright Apps Associates LLC. 5

5 Trend 1 The end of one size fits all Copyright Apps Associates LLC. 7

6 History of databases Magnetic tape flat (sequential) files IDMS Access HBase Pre-computer technologies: Magnetic Disk ADABAS System R Oracle V2 Postgres MySQL Dynamo MongoDB Redis Printing press Dewey decimal system Punched cards VoltDB Neo4J Relational Model defined IMS Network Model Hierarchical model Indexed-Sequential Access Mechanism (ISAM) SQL Server Sybase Informix Ingres DB2 dbase Aerospike Hana Riak Cassandra Vertica Hadoop

7 Why? Why 3 rd Platform drives new demands on the database: Global High Availability Data volumes Unstructured data Transaction rates Latency A single architecture cannot meet all those demands

8 Enterprise Big data Architecture In-memory processing (Spark) Analytic/BI software (SAS, Tableau Web Server Data Warehouse RDBMS (Oracle, Terradata ) In-memory Analytics (HANA, Exalytics ) Hadoop Web DBMS (MySQL, Mongo, Cassandra) Operational RDBMS (Oracle, SQL Server, ) ERP & inhouse CRM

9 Oracle Engineered Systems

10 Trend #2: Big Data and Hadoop

11 Biggest IT inflection point in our generation Cloud Social Mobile Big Data

12 Characteristics of Big Data Copyright Apps Associates LLC. 14

13 The instrumented human Bluetooth Personal Area Network 3G/WiFi Wide Area Network GPS Storage Compass Camera Mike/earphones Heads up display Emotion/Attention monitor Pulse, temp monitor Silent alarms Pedometer, sleep monitoring

14 Operational vs. Analytical Databases Copyright Apps Associates LLC. 16

15 Google Software Architecture (circa 2005) Reduce Google Applications Google File System (GFS) BigTable

16 Reduce Start Reduce

17 Hadoop Design Principles System shall manage and heal itself Automatically and transparently route around failure Speculatively execute redundant tasks if certain nodes are detected to be slow Performance shall scale linearly Proportional change in capacity with resource change Compute should move to data Lower latency, lower bandwidth Simple core, modular and extensible Copyright Apps Associates LLC. 19

18 Hadoop History Dec 2004 Google GFS paper published July 2005 Nutch uses Reduce Feb 2006 Starts as a Lucene subproject Apr 2007 Yahoo! on 1000-node cluster Jan 2008 An Apache Top Level Project Jul 2008 A 4000 node test cluster May 2009 Hadoop sorts Petabyte in 17 hours Copyright Apps Associates LLC. 20

19 ZooKeeper (Coordination) Hadoop Ecosystem Chukwa (Monitoring) Client Access Hue Hive(Sql) Pig(Pl/Sql) Data Access Sqoop Flume Data Mining Mahout Orchestration Oozie Networking Reduce (Job Scheduling/Execution System) HBase (key-value store) (Streaming/Pipes APIs) HDFS (Hadoop Distributed File System) Java Virtual Machine OS Redhat, Suse, Ubuntu,Windows Commodity Hardware Copyright Apps Associates LLC. 21

20 HDFS Distributions Copyright Apps Associates LLC. 22

21 Hadoop 2.0 Copyright Apps Associates LLC. 23

22 Hadoop at Yahoo 2010(biggest cluster): 4000 nodes 16PB disk 64 TB of RAM 32,000 Cores 2014: 16 Clusters 32,500 nodes

23 Oracle Big Data with Oracle Exadata Copyright Apps Associates LLC. 25

24 Trend #3: NoSQL

25 Database Market Disruption $30B Database Market Being Disrupted Copyright Apps Associates LLC. 27

26 Operational vs. Analytical Databases Copyright Apps Associates LLC. 28

27 BigTable Data Model NameId Name 1 Dick 2 Jane SiteId SiteName 1 Ebay 2 Google 3 Facebook 4 ILoveLarry.com 5 MadBillFans.com Name Site Counter Dick Ebay 507,018 Dick Google 690,414 Jane Google 716,426 Dick Facebook 723,649 Jane Facebook 643,261 Jane ILoveLarry.com 856,767 Dick MadBillFans.com 675,230 NameId SiteId Counter , , , , , , ,230 Id Name Ebay Google Facebook (other columns) MadBillFans.com 1 Dick 507, , , ,230 Id Name Google Facebook (other columns) ILoveLarry.com 2 Jane 716, , ,767

28 Big Data /Hadoop Use Cases Financial services Internet retailer Retailers Drug discovery Healthcare Mobile telecom IT technical support Scientific research Internet travel Discover fraud patterns based on multi-years worth of credit card transactions and in a time scale that does not allow new patterns to accumulate significant losses. Measure transaction processing latency across many business processes by processing and correlating system log data. Discover fraud patterns in Internet retailing by mining Web click logs. Assess risk by product type and session/internet Protocol (IP) address activity. Perform sentiment analysis by analyzing social media data. Perform large-scale text analytics on publicly available information sources. Analyze medical insurance claims data for financial analysis, fraud detection, and preferred patient treatment plans. Analyze patient electronic health records for evaluation of patient care regimes and drug safety. Discover mobile phone churn patterns based on analysis of CDRs and correlation with activity in subscribers networks of callers. Perform large-scale text analytics on help desk support data and publicly available support forums to correlate system failures with known problems. Analyze scientific data to extract features (e.g., identify celestial objects from telescope imagery). Improve product ranking (e.g., of hotels) by analysis of multi-years worth of Web click logs.

29 Document databases Structured documents XML and JSON (JavaScript Object Notation) become more prevalent within applications Web programmers start storing these in BLOBS in MySQL Emergence of XML and JSON databases

30 MemchacheD B MongoDB Key Value Oracle NoSQL Voldemort JSON based CouchDB Dynamo DynamoDB Document RethinkDB Riak XML based MarkLogic BerkeleyDB XML Cassandra Hbase Neo4J Table Based BigTable HyperTable Graph Database Infinite Graph Accumulo FlockDB

31 How Do You Take This Growth? Copyright Apps Associates LLC. 33

32 Scaling Out RDBMS Copyright Apps Associates LLC. 34

33 RDBMS are Not Enough? Copyright Apps Associates LLC. 35

34 NoSQL Technology Scales Out Copyright Apps Associates LLC. 36

35 A New Technology Copyright Apps Associates LLC. 37

36 Use Cases Copyright Apps Associates LLC. 38

37 Brewer's CAP Theorem Copyright Apps Associates LLC. 39

38 Brewer's CAP Theorem Copyright Apps Associates LLC. 40

39 NoSQL Technology Spectrum Copyright Apps Associates LLC. 41

40 No Means Yes!

41 Copyright Apps Associates LLC. 43 Big Data Architecture D A T A S O U R C E S DATA LAKE On AWS Big Data Infra (Optrion2) DATA CONNECTORS A N A L Y T I C S DATA LAKE on Oracle Big data Appliance (Option1) DATA LAKE On Premise Hadoop Infra(Option3) D A T A L A K E

42 On Premise Hadoop as RDBMS active archive Unstructured + Structured Data Analytics from Apps Hive provides an SQLlike query layer over Hadoop and Reduce Structured Data Analytics from Apps Hadoop for Structured Archive and Unstructured data SALES 2011 SALES 2010 SALES 2013 SALES 2012 SALES 2011 SALES 2010 Oracle Database Copyright Apps Associates LLC. 44

43 AWS EMR as RDBMS active archive Unstructured + Structured Data Analytics from Apps Hive provides an SQLlike query layer over Amazon EMR Structured Data Analytics from Apps AWS EMR for Structured Archive and Unstructured data SALES 2011 SALES 2010 SALES 2013 SALES 2012 SALES 2011 SALES 2010 Amazon Elastic Reduce (Amazon EMR) Oracle Database Copyright Apps Associates LLC. 45

44 Oracle Database Support for All Data Structured Data Numeric, String, Date, Row and column formats Unstructured Data LOB Text XML JSON Spatial Graph

45 Oracle Support for Any Data Management System Hadoop Change the Business Scale-out, low cost store Collect any data -reduce, SQL Analytic applications NoSQL Scale the Business Scale-out, low cost store Collect key-value data Find data by key Web applications Relational Run the Business Scale-out and scale-up Collect any data SQL Transactional and analytic applications for the enterprise Secure and highly available

46 Big Data SQL SELECT FROM WHERE AND w.sess_id, c.name web_logs w, customers c w.source_country = Brazil w.cust_id = c.customer_id; SQL Push Down in Big Data SQL WEB_LOGS Big Data SQL B B B Hadoop Cluster Hadoop Scans on Unstructured Data WHERE Clause Relevant Evaluation SQL runs on BDA nodes Column Projection Bloom Filters for Better Join Performance JSON Parsing, 10 s of Gigabytes Data Mining of Data Model Evaluation Only columns and rows needed to answer query are returned Oracle Database CUSTOMERS

47 Separate silos of information to analyze Data Analytics Challenge

48 Separate data access interfaces Data Analytics Challenge

49 SQL on Hadoop is Obvious Stinger

50 Data Analytics Challenge No comprehensive SQL interface across Oracle, Hadoop and NoSQL

51 Oracle Big Data Management System Rich, comprehensive SQL access to all enterprise data NoSQL

52 What Does Unified Query Mean for You? PhD Before Data Science Anyone After???

53 What Does Unified Query Mean for You? Before Application Development After

54 Big Data SQL : A New Hadoop Processing Engine Reduc e and Hive Processing Layer Spark Impala Search Big Data SQL Resource Management (YARN, cgroups) Storage Layer Filesystem (HDFS) NoSQL Databases (Oracle NoSQL DB, Hbase)

55 What Next for DBA s in Big Data Era? NoSQL Hadoop Big data Sql 12c New Features on Big data Engineered Systems Knowledge

56 Connect with Us Web: YouTube: LinkedIn: Facebook: Copyright Apps Associates LLC. 58

57 @pasalapudi Thank You!