Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy. Derrick Kondo INRIA, France

Transcription

1 Volunteer Computing, Grid Computing and Cloud Computing: Opportunities for Synergy Derrick Kondo INRIA, France

2 Outline Cloud Grid Volunteer Computing

3 Cloud Background Vision Hide complexity of hardware and software management from a user by offering computing as a service Benefits Pay as you go Scale up, down dynamically No hardware management, less software management

4 Platform Performance vs. Costs high Supercomputer Grid Cluster Cost low low Cloud Performance What is the relationship? BOINC BOINC + Cloud? high

5 Clouds versus Volunteer Computing Performance tradeoffs Monetary tradeoffs Client hosting Server hosting

6 Method Use real performance measurements Exported BOINC project data Use real costs Large/small BOINC projects / XtremLab) Amazon Elastic Computing Cloud (EC2)

7 Stages of Project & Application Platform Construction Application Deployment Application Execution Application Completion

8 Platform Construction Application Deployment Application Execution Application Completion How long before I get X TeraFLOPS? Strategy: Add to BOINC project list Press releases Forum Announcements Google Ad Sense Respond to users (leverage volunteers) Number of cloud nodes Can get over 20 TeraFLOPS within 6 months Months for registration TeraFLOPS

9 Platform Construction Application Deployment Application Execution Application Completion How long to deploy my batch of tasks needing fast response time? Strategy: Specify lower latency bounds [Heien et al.] 7,82/9*,34.4:*,.;*:3)4,6<!" '!" &!"!!" "!"!!.!"".4=6>6!""".4=6>6!"""".4=6>6 For 1000 tasks, ~10 minutes with 10 5 hosts!" #!" $!" % ()*+,-./0.1/2)34,,-.3/5,6.

10 Platform Construction Application Deployment Application Execution Application Completion How many volunteer nodes are equivalent to 1 cloud node? 2.8 active volunteer hosts per 1 cloud node. (Total performance still orders of magnitude better) Strategy: Use statistical prediction of availability

11 Platform Construction Application Deployment Application Execution Application Completion How long should I wait for task completion? Strategy: See BOINC Catalog for typical deadlines and compute/comm/mem ratios. Median project latency bound: 9 days for 3.7 hour work unit (on 3GHz host). Ratio of lat. bound / exec time > 5. Good success rates: 96.1% of WCG tasks met out of 227,000 tasks

12 Monetary Tradeoffs Client hosting on cloud Not worth it and never will Server hosting on the cloud Possible solution

13 Monthly Project Costs Text 1K Students 1K 1K 1K

14 EC2 Pricing

15 What about Client Hosting on the Cloud?

16 Cost of Clouds versus BOINC When is BOINC more cost effective than Clouds? 7,8./9,)/:;<( #!!## #!!#$ #!!#% #!!#' F#'!'/G*HI8.,,)/8*3,5 2'J&/-H*I3/8*3,56 '"&$/G*HI8.,,)/8*3,5 2#&#$/-H*I3/8*3,56 A7$ / #!!#" After ~13 days #!!#& /! " #! #" $! $" %! ()*+,-./01,/230456

17 Cost of Clouds versus BOINC (2) 8 x 104 What are total costs over time? Total cost Within <3 days, BOINC is cheaper EC2 100n EC2 1000n EC n XtremLab SETI@home Platform age (days)

18 Cost of Clouds versus BOINC (3) How many months of BOINC can X months of Cloud buy me? Months with VC Xtremlab / 1000n EC2 SETI@home / 1000n EC2 Xtremlab / 10000n EC2 SETI@home / 10000n EC2 12 months of Cloud can pay for > 125 BOINC years. BOINC way more sustainable Months with EC2

19 Equivalent Clouds given BOINC Costs Given BOINC costs, what size Cloud can I buy? 1.75 TeraFLOPS, 7 TeraBytes 2 orders of magnitude lower than BOINC.75 TeraFLOPS, 3 TeraBytes

20 What about Server Hosting on the Cloud?

21 Number of Hosts over Time Load variation exits w/ publicity, projects run out of work, etc. Clouds take care of server management. Potential to exploit clouds.

22 Server Costs on a Cloud How much to host BOINC server on cloud? times cheaper to host on Cloud, but bandwidth is expensive. (versus 43K +12K/month) (versus {4,1}K + {5,1}K/month)

23 Cloud Resources with Given Budget How big of a server can I support with given budget? Many project servers are sustainable on the cloud 4 Mbps, 2 TB 12 Mbps, 2 TB

24 Summary of Cloud Versus Volunteer Computing Performance tradeoffs 20 BOINC TeraFLOPS within 6 months Monetary tradeoffs Client hosting After 13 days, BOINC more cost effective Server hosting BOINC server on cloud is cost-effective Best for small-medium sized projects Savings of at least 40%

25 Outline Cloud Grid Volunteer Computing

26 Volunteer Computing (Compared to Grids) User Not computer savy Cause errors (accidentaly) (or bring benefits) Shuts BOINC client machine on/off Doesn t have or wants to avoid root access Software Heterogeneous OS (most are Windows) Resource contention Firewalls Hardware Heterogeneous size and type Low bandwidth network, firewalls

27 Synergy Reasons Ways Use storage as cache to put data closer to end nodes Why Grids should use volunteer computing Most Grids jobs are embarrassingly parallel Order of magnitudes more compute power and storage and more pervasive Overloaded during peak periods Why volunteer computing should use Grids Utilization is still low Results are deterministic (relatively) Performance is deterministic (relatively) Use storage as archival Rerun computation on the fly Offload all runnable jobs to volunteer computing system New hybrid algorithms for non-uniform platforms: allow for global search on volunteers and refined local on Grids Increased compute/storage power Check results (spot checking w/ replication) Replication with reliable performance for failed jobs Support fast turnaround jobs

28 Bridging Volunteer Computing and Grids Standard API Servers that act as bridge 3 central points of failure 3 job queues: high latency Scaling issues: 1 bridge for M Grids and N BOINC projects Pilot jobs VM for system checkpointing, security, and linux support

29 Outline Cloud Grid Volunteer Computing

30 Grids versus Clouds Clouds built for web services Grids built for data-intensive applications Petabytes of data Files used by groups of users

31 Issues with Cloud data management [Iamnitchi08] Current cost model One size fits all (same true for comp) Data durability: existence Data availability: accessibility Access performance E.g. 1: archiving: need durability E.g. 2: caching: need availability, acces perf.

32 Summary Cloud vs Volunteer Computing Server hosting is cost-effective Volunteer Computing vs Grid Pilot jobs seem like the ideal approach Cloud vs Grid Clouds must provide different costs models for different qualities of service

33 Current and Future Work Bring Volunteer Computing to the Cloud C3: Cloud Computational Co-op RightScale Grant NFSC-ANR Grants between China and France Bring the Cloud to Volunteer Computing Bridging Grids and BOINC EDGES project (Peter Kacsuk, SZTAKI, Hungary)