2.3 Splitting loops: the parallel for pattern

Most programs have loops, and many programs use loops to calculate an approximation, such as an integral, or to work on individual elements of a lark array of data. At the core of every loop is each iteration. Arguably the most common pattern in parallel computing with MPI is the parallel for loop pattern, where iterations of a loop are split between processes, each of which computes a portion of the loop simultaneously.

from mpi4py import MPI def main(): comm = MPI.COMM_WORLD id = comm.Get_rank() #number of the process running the code numProcesses = comm.Get_size() #total number of processes running myHostName = MPI.Get_processor_name() #machine name running the code REPS = 8 if ((REPS % numProcesses) == 0 and numProcesses <= REPS): # How much of the loop should a process work on? chunkSize = int(REPS / numProcesses) start = id * chunkSize stop = start + chunkSize # do the work within the range set aside for this process for i in range(start, stop): print("On {}: Process {} is performing iteration {}"\ .format(myHostName, id, i)) else: # cannot break into equal chunks; one process reports the error if id == 0 : print("Please run with number of processes divisible by \ and less than or equal to {}.".format(REPS)) ########## Run the main function main()

Run this code and envision that this is how the loop of 8 repetitions is being split by 4 processes:

Exercises:

1. Compare source the code to the output from running it.

2. Run, using these numbers of processes, -np: 1, 2, 4, and 8

3. Change REPS to 16, save, rerun, varying -np again.

4. Explain how this pattern divides the iterations of the loop among the processes.

The last exercise above is important: be sure that you can envision and explain how the loop was divided up. We call this decomposing the problem into smaller parts that each process can complete independently. This is the basis of the parallel for loop pattern.

In the next section we will combine what we have seen so far to solve a simple application.

You have attempted of activities on this page