雲計算與分散式系統

《雲計算與分散式系統：從並行處理到物聯網》是一本全面而新穎的教材，內容覆蓋高性能計算、分散式與雲計算、虛擬化和格線計算。作者將套用與技術趨勢相結合，揭示了計算的未來發展。無論是對在校學生還是經驗豐富的實踐者，本書都是一本優秀的讀物。

基本介紹

• 中文名：雲計算與分散式系統
• 外文名：Distributed and Cloud Computing: From Parallel Processing to the Internet of Things
• 類別：書籍
• 涉及方面：計算機和數學
• 內容：介紹雲計算和分散式

編輯推薦

“格線計算、對等計算、雲計算這些新興領域近幾年日益受到學術界和工業界的關注。預計這些新技術將對商業、科學和工程及社會等眾多方面產生巨大影響。本書的及時出版將會幫助讀者了解分散式計算領域的最新技術。”

——YiPan,喬治亞州立大學

——ThomasJ.Hacker,普度大學

“你是學分散式系統或分散式計算課程的學生嗎？如果是，那么這本書是你的最佳選擇。本書作者做了一個傑出的工作，在書中講述了在硬體和軟體、系統體系結構、新的編程範式和生態系統方面的最新進展，既關注速度和性能最佳化，又考慮能源效率與節能。

作者首先評價了過去30年計算和IT趨勢中的演變，並討論了集群計算系統和大規模並行處理器。接著，針對雲計算套用，他們介紹了虛擬機、虛擬機線上遷移、虛擬集群構建、資源供應、虛擬配置適配和虛擬數據中心設計。然後，作者介紹了雲體系結構和數據中心設計的設計原理與使能技術，探究了兩種主要的分散式服務（REST和Web服務）以及Web服務在格線系統中的擴展。此外，作者還介紹了主要的雲編程範式：MapReduce、BigTable、Twister、Dryad、DryadLINQ、Hadoop、Sawal和PigLatin。而且書中還介紹了計算/數據格線中的設計原理、平台體系結構、中間件支持、資源管理和服務標準。之後，作者描述了非結構化、結構化和混合式覆蓋網路。最後，他們檢測了IBM、Salesforce、SGI、Manijarsoft、NASA和CERN構建的一些公有雲和私有雲。”

本書的目的是將傳統的多處理器和多計算機集群轉換成Web規模格線、雲。也許更重要的是，本書關注未來網際網路中廣泛在使用的對等網路，以及快速發展的大型社會網路和物聯網。

——摘自Amazon讀者評論

內容簡介

隨著信息技術的廣泛套用和快速發展，雲計算作為一種新興的商業計算模型日益受到人們的廣泛關注。本書是一本完整講述雲計算與分散式系統基本理論及其套用的教材。書中從現代分散式模型概述開始，介紹了並行、分散式與雲計算系統的設計原理、系統體系結構和創新套用，並通過開源套用和商業套用例子，闡述了如何為科研、電子商務、社會網路和超級計算等創建高性能、可擴展的、可靠的系統。

【本書特色】

全面覆蓋現代分散式計算技術，包括集群、格線、面向服務的體系結構、大規模並行處理器、對等網路和雲計算。

提供的案例研究來自主流分散式計算供應商，如亞馬遜、微軟、谷歌等。

解釋如何利用虛擬化來促進管理、調試、遷移和災難恢復。

專為本科生或研究生的分散式系統課程而設計——每章後都配有習題和進一步閱讀建議，並為教師提供配套的PPT等教輔資源。

作者簡介

KaiHwang（黃鎧）美國南加州大學電子工程與計算機科學教授，網際網路/雲計算研究實驗室主任；清華大學IV客座講席教授；IEEE終身會士。他擁有加州大學伯克利分校EECS博士學位，主要研究領域為雲計算、分散式系統、高性能計算、普適計算、信任格線計算等。現已發表論文220多篇，出版8本計算機體系結構、數字運算、並行處理、分散式系統、網際網路安全和雲計算方面的相關著作。他還創建了《theJournalofParallelandDistributedComputing》，並獲得了中國計算機學會2004傑出成就獎、IEEE2011IPDPS創立者獎。

GeoffreyFox美國印第安那大學計算機科學、信息與物理學傑出教授，社會格線實驗室主任。之前曾在加州理工和錫拉丘茲大學任教，並領導多個研究組。他擁有英國劍橋大學的博士學位。Fox在並行體系結構、分散式編程、格線計算、Web服務和網際網路套用方面做了廣泛的工作並發表了大量作品。

JackDongarra美國田納西大學電子工程與計算機科學傑出教授，橡樹嶺國家實驗室傑出研究員，曼徹斯特大學TurningFellow。他是ACM/IEEE/SIAM/AAAS會士，是超級計算機基準測試、數值分析、線性代數解算器和高性能計算領域的先驅。多年以來，他都在負責Top500最快計算機的Linpack基準測試評估。基於他在超級計算和高性能領域的巨大貢獻，他被評為美國國家工程院院士。

目錄

Contents

Preface.............................. vii

AbouttheAuthors..

PART1SYSTEMSMODELING,CLUSTERING,

ANDVIRTUALIZATION1

CHAPTER1DistributedSystemModelsandEnablingTechnologies...... ............3

Summary.................4

1.1ScalableComputingovertheInternet.................4

1.1.1TheAgeofInternetComputing...............4

1.1.2ScalableComputingTrendsandNewParadigms..........8

1.1.3TheInternetofThingsandCyber-PhysicalSystems......11

1.2TechnologiesforNetwork-BasedSystems................13

1.2.1MulticoreCPUsandMultithreadingTechnologies........14

1.2.2GPUComputingtoExascaleandBeyond...............17

1.2.3Memory,Storage,andWide-AreaNetworking...........20

1.2.4VirtualMachinesandVirtualizationMiddleware.........22

1.2.5DataCenterVirtualizationforCloudComputing.........25

1.3SystemModelsforDistributedandCloudComputing....27

1.3.1ClustersofCooperativeComputers.............28

1.3.2GridComputingInfrastructures...............29

1.3.3Peer-to-PeerNetworkFamilies...............32

1.3.4CloudComputingovertheInternet..............34

1.4SoftwareEnvironmentsforDistributedSystemsandClouds.... ..........36

1.4.1Service-OrientedArchitecture(SOA)...................37

1.4.2TrendstowardDistributedOperatingSystems............40

1.4.3ParallelandDistributedProgrammingModels...........42

1.5Performance,Security,andEnergyEfficiency............44

1.5.1PerformanceMetricsandScalabilityAnalysis............45

1.5.2FaultToleranceandSystemAvailability................48

1.5.3NetworkThreatsandDataIntegrity............49

1.5.4EnergyEfficiencyinDistributedComputing.............51

1.6BibliographicNotesandHomeworkProblems............55

Acknowledgments.......56

References..............56

HomeworkProblems....58

Foreword.........v

......................x

CHAPTER2ComputerClustersforScalableParallelComputing.....65

Summary...............66

2.1ClusteringforMassiveParallelism................66

2.1.1ClusterDevelopmentTrends................66

2.1.2DesignObjectivesofComputerClusters................68

2.1.3FundamentalClusterDesignIssues................69

2.1.4AnalysisoftheTop500Supercomputers................71

2.2ComputerClustersandMPPArchitectures...............75

2.2.1ClusterOrganizationandResourceSharing..............76

2.2.2NodeArchitecturesandMPPPackaging................77

2.2.3ClusterSystemInterconnects................80

2.2.4Hardware,Software,andMiddlewareSupport...........83

2.2.5GPUClustersforMassiveParallelism......... ..83

2.3DesignPrinciplesofComputerClusters.............. 87

2.3.1Single-SystemImageFeatures...............87

2.3.2HighAvailabilitythroughRedundancy..................95

2.3.3Fault-TolerantClusterConfigurations................ 99

2.3.4CheckpointingandRecoveryTechniques...............101

2.4ClusterJobandResourceManagement.................104

2.4.1ClusterJobSchedulingMethods................... 104

2.4.2ClusterJobManagementSystems...................107

2.4.3LoadSharingFacility(LSF)forClusterComputing.....109

2.4.4MOSIX:AnOSforLinuxClustersandClouds.........110

2.5CaseStudiesofTopSupercomputerSystems............112

2.5.1Tianhe-1A:TheWorldFastestSupercomputerin2010......... ....112

2.5.2CrayXT5Jaguar:TheTopSupercomputerin2009.....116

2.5.3IBMRoadrunner:TheTopSupercomputerin2008......119

2.6BibliographicNotesandHomeworkProblems..........120

Acknowledgments......121

References.............121

HomeworkProblems.........................122

CHAPTER3VirtualMachinesandVirtualizationofClustersandDataCenters...........129

Summary..............130

3.1ImplementationLevelsofVirtualization.................130

3.1.1LevelsofVirtualizationImplementation.............130

3.1.2VMMDesignRequirementsandProviders.............133

3.1.3VirtualizationSupportattheOSLevel............. 135

3.1.4MiddlewareSupportforVirtualization............. 138

3.2VirtualizationStructures/ToolsandMechanisms.........140

3.2.1HypervisorandXenArchitecture..................140

3.2.2BinaryTranslationwithFullVirtualization.............141

3.2.3Para-VirtualizationwithCompilerSupport.............143

xiiContents

3.3VirtualizationofCPU,Memory,andI/ODevices.......145

3.3.1HardwareSupportforVirtualization................ 145

3.3.2CPUVirtualization........................147

3.3.3MemoryVirtualization.....................148

3.3.4I/OVirtualization.........................150

3.3.5VirtualizationinMulti-CoreProcessors............. 153

3.4VirtualClustersandResourceManagement.............155

3.4.1PhysicalversusVirtualClusters................. ..156

3.4.2LiveVMMigrationStepsandPerformanceEffects......159

3.4.3MigrationofMemory,Files,andNetworkResources....162

3.4.4DynamicDeploymentofVirtualClusters..............165

3.5VirtualizationforData-CenterAutomation..............169

3.5.1ServerConsolidationinDataCenters..............169

3.5.2VirtualStorageManagement...............171

3.5.3CloudOSforVirtualizedDataCenters................172

3.5.4TrustManagementinVirtualizedDataCenters.........176

3.6BibliographicNotesandHomeworkProblems..........179

Acknowledgments......179

References.............180

HomeworkProblems.........................183

PART2COMPUTINGCLOUDS,SERVICE-ORIENTED

ARCHITECTURE,ANDPROGRAMMING189

CHAPTER4CloudPlatformArchitectureoverVirtualizedDataCenters. .............191

Summary..............192

4.1CloudComputingandServiceModels..................192

4.1.1Public,Private,andHybridClouds.............. 192

4.1.2CloudEcosystemandEnablingTechnologies...........196

4.1.3Infrastructure-as-a-Service(IaaS)............... ..200

4.1.4Platform-as-a-Service(PaaS)andSoftware-as-a-Service(SaaS)... ............203

4.2Data-CenterDesignandInterconnectionNetworks......206

4.2.1Warehouse-ScaleData-CenterDesign..................206

4.2.2Data-CenterInterconnectionNetworks.................208

4.2.3ModularDataCenterinShippingContainers...........211

4.2.4InterconnectionofModularDataCenters...............212

4.2.5Data-CenterManagementIssues....................213

4.3ArchitecturalDesignofComputeandStorageClouds... ............215

4.3.1AGenericCloudArchitectureDesign.................215

4.3.2LayeredCloudArchitecturalDevelopment.............218

4.3.3VirtualizationSupportandDisasterRecovery...........221

4.3.4ArchitecturalDesignChallenges................... 225

Contentsxiii

4.4PublicCloudPlatforms:GAE,AWS,andAzure........227

4.4.1PublicCloudsandServiceOfferings..................227

4.4.2GoogleAppEngine(GAE)................229

4.4.3AmazonWebServices(AWS)..................... 231

4.4.4MicrosoftWindowsAzure.................233

4.5Inter-cloudResourceManagement...................234

4.5.1ExtendedCloudComputingServices..................235

4.5.2ResourceProvisioningandPlatformDeployment........237

4.5.3VirtualMachineCreationandManagement............243

4.5.4GlobalExchangeofCloudResources..................246

4.6CloudSecurityandTrustManagement..................249

4.6.1CloudSecurityDefenseStrategies.................. 249

4.6.2DistributedIntrusion/AnomalyDetection...............253

4.6.3DataandSoftwareProtectionTechniques..............255

4.6.4Reputation-GuidedProtectionofDataCenters..........257

4.7BibliographicNotesandHomeworkProblems..........261

Acknowledgements.....261

References.............261

HomeworkProblems.........................265

CHAPTER5Service-OrientedArchitecturesforDistributedComputing ..............271

Summary..............272

5.1ServicesandService-OrientedArchitecture..............272

5.1.1RESTandSystemsofSystems.....................273

5.1.2ServicesandWebServices.................277

5.1.3EnterpriseMultitierArchitecture..................282

5.1.4GridServicesandOGSA..................283

5.1.5OtherService-OrientedArchitecturesandSystems.......287

5.2Message-OrientedMiddleware................289

5.2.1EnterpriseBus......289

5.2.2Publish-SubscribeModelandNotification..............291

5.2.3QueuingandMessagingSystems.....................291

5.2.4CloudorGridMiddlewareApplications...............291

5.3PortalsandScienceGateways................294

5.3.1ScienceGatewayExemplars................295

5.3.2HUBzeroPlatformforScientificCollaboration..........297

5.3.3OpenGatewayComputingEnvironments(OGCE)......301

5.4Discovery,Registries,Metadata,andDatabases.........304

5.4.1UDDIandServiceRegistries...............304

5.4.2DatabasesandPublish-Subscribe................307

5.4.3MetadataCatalogs........................308

5.4.4SemanticWebandGrid...................309

5.4.5JobExecutionEnvironmentsandMonitoring...........312

xivContents

5.5WorkflowinService-OrientedArchitectures.............314

5.5.1BasicWorkflowConcepts..................315

5.5.2WorkflowStandards.......................316

5.5.3WorkflowArchitectureandSpecification...............317

5.5.4WorkflowExecutionEngine................319

5.5.5ScriptingWorkflowSystemSwift.....................321

5.6BibliographicNotesandHomeworkProblems..........322

Acknowledgements.....324

References.............324

HomeworkProblems.........................331

CHAPTER6CloudProgrammingandSoftwareEnvironments.......335

Summary..............336

6.1FeaturesofCloudandGridPlatforms..............336

6.1.1CloudCapabilitiesandPlatformFeatures..............336

6.1.2TraditionalFeaturesCommontoGridsandClouds......336

6.1.3DataFeaturesandDatabases...............340

6.1.4ProgrammingandRuntimeSupport...................341

6.2ParallelandDistributedProgrammingParadigms........343

6.2.1ParallelComputingandProgrammingParadigms........344

6.2.2MapReduce,Twister,andIterativeMapReduce.........345

6.2.3HadoopLibraryfromApache....................355

6.2.4DryadandDryadLINQfromMicrosoft................359

6.2.5SawzallandPigLatinHigh-LevelLanguages...........365

6.2.6MappingApplicationstoParallelandDistributedSystems...... .........368

6.3ProgrammingSupportofGoogleAppEngine...........370

6.3.1ProgrammingtheGoogleAppEngine.................370

6.3.2GoogleFileSystem(GFS).................373

6.3.3BigTable,Google’sNOSQLSystem...................376

6.3.4Chubby,Google’sDistributedLockService............379

6.4ProgrammingonAmazonAWSandMicrosoftAzure............ .....379

6.4.1ProgrammingonAmazonEC2....................380

6.4.2AmazonSimpleStorageService(S3)..................382

6.4.3AmazonElasticBlockStore(EBS)andSimpleDB......383

6.4.4MicrosoftAzureProgrammingSupport................384

6.5EmergingCloudSoftwareEnvironments................387

6.5.1OpenSourceEucalyptusandNimbus..................387

6.5.2OpenNebula,Sector/Sphere,andOpenStack............389

6.5.3ManjrasoftAnekaCloudandAppliances...............393

6.6BibliographicNotesandHomeworkProblems..........399

Acknowledgement.....399

References.............399

HomeworkProblems.........................405

Contentsxv

PART3GRIDS,P2P,ANDTHEFUTUREINTERNET413

CHAPTER7GridComputingSystemsandResourceManagement................415

Summary..............416

7.1GridArchitectureandServiceModeling................416

7.1.1GridHistoryandServiceFamilies.................416

7.1.2CPUScavengingandVirtualSupercomputers..........419

7.1.3OpenGridServicesArchitecture(OGSA)..............422

7.1.4Data-IntensiveGridServiceModels.................425

7.2GridProjectsandGridSystemsBuilt................427

7.2.1NationalGridsandInternationalProjects...............428

7.2.2NSFTeraGridintheUnitedStates..................430

7.2.3DataGridintheEuropeanUnion................... 431

7.2.4TheChinaGridDesignExperiences...................434

7.3GridResourceManagementandBrokering..............435

7.3.1ResourceManagementandJobScheduling.............435

7.3.2GridResourceMonitoringwithCGSP.................437

7.3.3ServiceAccountingandEconomyModel..............439

7.3.4ResourceBrokeringwithGridbus................. 440

7.4SoftwareandMiddlewareforGridComputing..........443

7.4.1OpenSourceGridMiddlewarePackages...............444

7.4.2TheGlobusToolkitArchitecture(GT4)................446

7.4.3ContainersandResources/DataManagement...........450

7.4.4TheChinaGridSupportPlatform(CGSP)..............452

7.5GridApplicationTrendsandSecurityMeasures.........455

7.5.1GridApplicationsandTechnologyFusion..............456

7.5.2GridWorkloadandPerformancePrediction............457

7.5.3TrustModelsforGridSecurityEnforcement...........461

7.5.4AuthenticationandAuthorizationMethods.............464

7.5.5GridSecurityInfrastructure(GSI)..................466

7.6BibliographicNotesandHomeworkProblems..........470

Acknowledgments......471

References.............471

HomeworkProblems.........................473

CHAPTER8Peer-to-PeerComputingandOverlayNetworks........479

Summary..............480

8.1Peer-to-PeerComputingSystems...................480

8.1.1BasicConceptsofP2PComputingSystems............480

8.1.2FundamentalChallengesinP2PComputing............486

8.1.3TaxonomyofP2PNetworkSystems..................490

8.2P2POverlayNetworksandProperties..................492

8.2.1UnstructuredP2POverlayNetworks...................492

xviContents

8.2.2DistributedHashTables(DHTs)....................496

8.2.3StructuredP2POverlayNetworks.................. 498

8.2.4HierarchicallyStructuredOverlayNetworks............501

8.3Routing,Proximity,andFaultTolerance................505

8.3.1RoutinginP2POverlayNetworks....................505

8.3.2NetworkProximityinP2POverlays...................507

8.3.3FaultToleranceandFailureRecovery.................509

8.3.4ChurnResilienceagainstFailures..................512

8.4Trust,Reputation,andSecurityManagement............514

8.4.1PeerTrustandReputationSystems.................514

8.4.2TrustOverlayandDHTImplementation...............517

8.4.3PowerTrust:AScalableReputationSystem.............520

8.4.4SecuringOverlaystoPreventDDoSAttacks............522

8.5P2PFileSharingandCopyrightProtection..............523

8.5.1FastSearch,Replica,andConsistency.................524

8.5.2P2PContentDeliveryNetworks....................529

8.5.3CopyrightProtectionIssuesandSolutions..............533

8.5.4CollusivePiracyPreventioninP2PNetworks...........535

8.6BibliographicNotesandHomeworkProblems..........538

Acknowledgements.....538

References.............539

HomeworkProblems.........................541

CHAPTER9UbiquitousCloudsandtheInternetofThings.........545

Summary..............546

9.1CloudTrendsinSupportingUbiquitousComputing.....546

9.1.1UseofCloudsforHPC/HTCandUbiquitousComputing... ..........546

9.1.2Large-ScalePrivateCloudsatNASAandCERN........552

9.1.3CloudMashupsforAgilityandScalability.............555

9.1.4CloudletsforMobileCloudComputing................558

9.2PerformanceofDistributedSystemsandtheCloud......561

9.2.1ReviewofScienceandResearchClouds...............562

9.2.2Data-IntensiveScalableComputing(DISC).............566

9.2.3PerformanceMetricsforHPC/HTCSystems............568

9.2.4QualityofServiceinCloudComputing................572

9.2.5BenchmarkingMPI,Azure,EC2,MapReduce,andHadoop.............574

9.3EnablingTechnologiesfortheInternetofThings........576

9.3.1TheInternetofThingsforUbiquitousComputing.......576

9.3.2Radio-FrequencyIdentification(RFID)................580

9.3.3SensorNetworksandZigBeeTechnology..............582

9.3.4GlobalPositioningSystem(GPS)................ 587

9.4InnovativeApplicationsoftheInternetofThings........590

9.4.1ApplicationsoftheInternetofThings.............591

Contentsxvii

9.4.2RetailingandSupply-ChainManagement..............591

9.4.3SmartPowerGridandSmartBuildings................594

9.4.4Cyber-PhysicalSystem(CPS)....................595

9.5OnlineSocialandProfessionalNetworking.............597

9.5.1OnlineSocialNetworkingCharacteristics..............597

9.5.2Graph-TheoreticAnalysisofSocialNetworks...........600

9.5.3CommunitiesandApplicationsofSocialNetworks......603

9.5.4Facebook:TheWorld’sLargestSocialNetwork........608

9.5.5TwitterforMicroblogging,News,andAlertServices....611

9.6BibliographicNotesandHomeworkProblems..........614

Acknowledgements.....614

References.............614

HomeworkProblems.........................618

Index........623