Wednesday, May 28, 2008

MPI with Python on EC2

For such a seemingly straightforward tutorial on setting up MPI to run on EC2 with Python wrappers (I *really* don't feel like writing glue code in C for lightly communicating processes), I struggled for way too long to get it to actually work.

So... the magic incantation is:


mpdboot -n 5 -f mpd.hosts

python /usr/local/bin/mpirun.py -n 5 pyMPI -c "import os; import mpi; print(mpi.rank); os.system('hostname')"


This assumes you made a mpd.hosts file listing the internal IP addresses of 5 running instances with MPICH2 and pyMPI installed and opened all your ports. You should see the numbers 0 through 4 as well as the machine names. Your mileage may vary - I clearly suck at this.

Hat tip: establishing connections between machines the 1st 1-2 times around is very slow. I suggest running a script, before MPI or anything else, that consumes your mpd.hosts file and SSHs, in both directions, between all pairs of instances. Twice. Running it periodically during your job may help too, but I'm not at that point yet in my development.
Post a Comment