An abstraction of
Find the parallelism explicitly and coordinating this.
MPI is standard that a message-passing library interface should adhere to.
Q: Why use MPI instead of simply making a huge address space? A: Extent of memory contention will scale with address space
No shared address space, each process is on a a different processor with distributed address space.
There is a communicator in every MPI program.
In the communicator, each process has
MPI_Comm_size
)MPI_Comm_rank
)int main(int argc, char **argv)
{
/* Runs on ONE PROCESSOR */
int rank, size;
char hostname[256];
/* Runs on EACH PROCESSOR (multiple) */
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
memset(hostname, 0, sizeof(hostname));
int sc_status = gethostname(hostname, sizeof(hostname)-1);
if (sc_status) {
perror("gethostname");
return sc_status;
}
/* From here on, each process is free to execute its own code */
/* Rank can be used to identify processes. */
if (rank == 0) {
printf("%s: I am the first process!\n", hostname);
// size == number of processes.
for (i = 1; i < size; i++) {
MPI_Send(hostname, 256, MPI_CHAR, i, tag, MPI_COMM_WORLD);
}
} else {
MPI_Status status;
char recvHost[256];
int rc = MPI_Recv(recvHost, 256, MPI_CHAR, 0, tag, MPI_COMM_WORLD, &status);
printf("%s: recved msg from %s\n", hostname, recvHost);
}
printf("Hello world from process %d out of %d on host %s\n", rank, size, hostname);
MPI_Finalize();
/* Anything else here runs on host processor again. */
return 0;
}
Blocking | Non-Blocking | |
---|---|---|
Send | MPI_Send | MPI_ISend |
Recv | MPI_Recv | MPI_IRecv |
SendRecv | MPI_SendRecv | MPI_ISendRecv |
MPI is about communication. If our processes are independent,we can just run srun
/sbatch
slurm
Handles job scheduling, distribution of processes across nodes/cores.
MPI: used forcommunication between processes.
srun -n 192 -N 12 /nfs/home/$USER/hello
--overcommit