Intermediate Parallel and Distributed Computing

By: Elizabeth Shoop (CSinParallel project and Macalester College)

With assistance and inspiration from Joel Adams, Richard Brown, and Suzanne Matthews (The CSinParallel project team)

This book is designed as a follow-on from our PDC for Beginners book. If you already have some experience with OpenMP, MPI, and CUDA coding, then you may be able to start with this book.

A hallmark of this book, like PDC for Beginners, is that the programs presented here can be run directly from this book, enabling learners to experience PDC concepts immediately.

All of the code in this book can be found on a GitHub repo for the CSInParallel Project. Please feel free to explore that if you want to have your students try the code on lab machines.

NOTE: THIS BOOK IS A WORK IN PROGRESS! However, there are eight solid chapters available. We hope to add further examples and suggestions for projects based on starting point code representing real-world high-performace computing problems. So please stay tuned!

Table of Contents

• About Intermediate Parallel and Distributed Computing
  • How to use this book
• CHAPTER 1: PDC Patterns
  • Parallel Programming Patterns
• CHAPTER 2: Shared Memory Patterns using OpenMP
  • Shared Memory Program Structure and Coordination Patterns
  • Data Decomposition Algorithm Strategies and Related Coordination Strategies
  • Patterns used when threads share data values
  • Task Decomposition Algorithm Strategies
• CHAPTER 3: Random Number Generators for PDC
  • 3.1 A new random number generator for Parallel and Distributed systems
  • 3.2 Basic Patterns of Sequential and Parallel PRNG use
• Chapter 4: Message Passing Parallel Patternlets
  • Program Structure: SPMD, Conductor-Worker, and Parallel Loops
  • Coordination Patterns in MPI: Basic message passing
  • Collective Communication Patterns in MPI: Broadcast
  • Collective Communication Patterns
  • Barrier Synchronization and Timing MPI Code
• Chapter 5: Message Passing: combining patterns
  • 5.1 MPI Tags as Sequence Numbers
  • 5.2 Broadcast and Data Decomposition with Parallel for Loop
  • 5.3 Scatter, Data Decomposition with Parallel for Loop, then Gather
  • 5.4 Scatter and Gather with any size array and odd or even number of processes
• Chapter 6: Message Passing Example Applications
  • 6.1 A simple Monte Carlo simulation
  • 6.2 Trapezoidal Rule Integration with MPI
• Chapter 7: OpenACC Basics
  • 7.0 OpenACC : A new standard and compiler
  • 7.1 Sequential versions with 2 compilers
  • 7.2 OpenMP code with gcc and pgcc
  • 7.3 Fast multicore code with new pgcc compiler
  • 7.4 GPU device code using OpenACC pragmas with the pgcc compiler
• Chapter 8: OpenACC Next steps
  • 8.1 The collapse() pragma directive on parallel for loop
  • 8.2 Reduction with a parallel for loop
  • 8.3 A classic linear algebra example: matrix multiply

To Cite:

Shoop E., and members of CSInParallel. Intermediate PDC, 1.0 Edition. Available online

Acknowledgements

This work is sponsored in part by

U.S. National Science Foundation (NSF) Collaborative Research Grants DUE-1822480/1822486/1855761. Collaborative Research: CSinParallel: Experiential Learning of Parallel and Distributed Computing through Sight, Sound, and Touch.

This work is also sponsored in part by:

United States Military Academy (USMA) Award Number W911NF-23-2-0044. USMA, as the Federal awarding agency, reserves a royalty-free, nonexclusive and irrevocable right to reproduce, publish, or otherwise use this software for Federal purposes, and to authorize others to do so in accordance with 2 CFR 200.315(b).

License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

You have attempted of activities on this page