Developer Forum

Hi all,

I need your help.

I have a Matlab program, which uses the Matlab symbolic toolbox and does
exclusively symbolic computations. It calls Maple's library internally. My
program is currently non-parallel. I need to run it on a super computer. The
super computer does not have Matlab and it is Linux based and supports MPI.

I didn't figure out how to convert non-parallel program into parallel
exectuables on Linux with MPI.

Since I failed to get a nice solution of the above problem, now I am willing
to make a comprise, I am going to make it coarse-grained distrubuted
computing:

(1) It is easy to compile Matlab non-parallel program into non-parallel
exectuables;
(2) I am willing to launch 100 independent copies fo these exectuables on
100 CPUs;

Is there a mechanism in Linux, or MPI, or third party applications that can
facilitate luanching of such 100 programs and finally automatically collect
back the 100 results from these 100 runs?

I am thinking this is too hard to do well by me myself manually. I need some
kind of automation to manage this whole thing.

Please shed some lights on me! Thanks a lot!

Yes, it's called a loop, and the shell can do it. As in:

for i in 1 2 3 4 5 6 7 8 9
do
./myprogram $i &
done

Or:

while read line
do
./myprogram $line &
done < my_file_with_parameters_for_each_desired_instance


Sure, collecting the results from 100 programs is easy. If they all
write everything to standard output, you'd just do something like
this:

while read outputfile param1 param2
do
./myprogram $param1 $param2 > $outputfile &
done < my_file_with_output_filenames_and_parameters

# now wait for them *all* to finish
wait

echo "all of the programs are done now!"

Of course, you can put all that in a file and have the shell run it
as a script so you don't have to type it all interactively (and so
you can do it again if you make an error or something).

- Logan

Logan, thanks for your help!

Do you mean I just need to launch the 100 programs using a loop and then the
supercomputer will automatically dispatch the jobs to 100 CPUs?

Your shell script looks serial to me... (as you see, I've got no previous
experience with parallel computing...)

Thanks!

Check my werbsite for the MPI tutorial link to learn about MPI itself.
Basically there are only 6 functions you need to know.

http://www.yqcomputer.com/ ~bgy1mm


Under MPI, each program is identical, but they send messages to each other,
so the data and the execution statye can be different on each node.
Generally, however, all nodes will communicate at the same point.

So your algorithm is

lauch program with all problems stored on node 0
divide problem into 100 chunks, send data to all nodes.
calculate chunk on each node
send solutions back to node 0
print out result.

step 2 may be easy or difficult, depending on your problem.

The problem is that the code is in Matlab whilst MPI provides C and Fortran.
The easiest solution, conceptually, is to translate the code from Matlab to
C. I am sure there will be some tool to enable you to this without rewriting
everything manually, however it is likely to be difficult to use.

Mike" < XXXX@XXXXX.COM > writes:


the '&' operator starts a process in the background and the shell
continues without waiting it to return.

look at these examples:

phix@igloo:~> { sleep 10 ; echo I am done } &
[1] 10674
phix@igloo:~> echo test
test
phix@igloo:~> I am done

[1] + done {; sleep 10; echo I am done; }
phix@igloo:~>
phix@igloo:~> { { sleep 5; echo last } & echo first } > out ; wait ; cat out
[2] 10710
[2] + done {; sleep 5; echo later; }
first
last

--
Philippe Amarenco, aka Phix
epita 2007 - EpX - ex-{GISTR,ACU,LSE}

"Mike
"


Hi,try this way:
oh,I'm not an English native speaker,so please ignore the spelling
errors if they occur :)
1,you must find a proper way to split your problem into small pieces
(it has nothing to do with MPI. This is a key step and it's just a math
problem)
2, rewrite your matlab programm, make it can work with some parameters
to solve a part of your problem...(the parameters it received decide
which part it should to compute) I mean, after this step ,you should be
able to solve the problem by repeated calling the programme,each time
with a different parameter.
3. Compile your matlab programme into .EXE file. There are several ways
to do that. Maybe LCC can work.
4.Write a wrapper tool for your programme.The wrapper should use MPI to
interact with each other,and make it call your .EXE programme with
parameters it received from the center control node.
5. Dispatch your wrappers and .EXE programmes

P.S MPI itself can dispatch and collect back results perfectly.No
third party application is needed.Google MPI_Broadcast and MPI_Wait
for more.

Of course, .EXE programs do not run in Linux.

--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key: 9A2FC99A Registered Machine 241939.
/( )\ Shrewsbury, New Jersey http://www.yqcomputer.com/
^^-^^ 08:45:01 up 78 days, 11:18, 3 users, load average: 4.36, 4.36, 4.32

You are probably looking for IPC, InterProcessCommunication.

Check out Chapter 5 in
http://www.yqcomputer.com/

Personally I would do it with Perl, and Perl's IPC modules,
like IPC::Open3
Perl makes it easy to setup socket connections, the IPC, etc.
and combine them into a single program.

One of the things to consider is whether you can collect results
all at the end-of-run in a results-file, or whether you need to collect
the results in real-time as they are returned. Possibly you need an even
more complex system, where results are returned in realtime to the
collecting system, decisions are made, and new commands are sent out
to the clients, so they can alter their computations.

In other words, it may be a bit more complex than shell scripts can
cleanly handle, and you might want to look into Perl, Python, etc.

I would ask this question at http://www.yqcomputer.com/ for some advice
from a wide variety of systems programmers, it's free, and probably
would set you on the right track.




--
I'm not really a human, but I play one on earth.
http://www.yqcomputer.com/

Good point. Obviously, it will depend on the specifics of the computer.

But in most cases, it probably will be good enough. There is no guarantee
that the job will get a fixed assignment to one of the 100 CPUs, but it
very well may not matter if a job moves between one CPU and another.
The scheduler makes this decision for you.

Some (most?) schedulers in multiprocessor systems have a "processor
affinity" feature where they note which processor a job most recently
ran on and prefer to schedule it on that processor again in the future.
That does not create a fixed binding, but it does reduce the (already
fairly small) overhead of switching CPUs a bit.

The real answer to this question depends on what type of multiprocessor
architecture that system has.

Is it:

(1) an SMP machine (SMP stands for symmetric multiprocessor and basically
means a machine where all processors are identical and all share the
same physical memory; with SMP, there is very little penalty for
switching around between processors)?
(2) a NUMA (NUMA is like SMP except that some processors have local memory
they can access faster than others; this is harder to program for
since it's very important which processor is running what, but it
can get higher performance)?
(3) actually not truly a single multiprocessor machine but instead a
cluster of completely separate machines (each with their own
process, memory, and instance of the operating system) connected
together by a network, i.e. a Beowulf cluster?
(4) something else?

The right thing to do will depend on the type of machine. In fact,
if it is a cluster of machines, then launching all the jobs locally
with a simple loop is probably precisely the *wrong* thing to do.
I looked up MPI, and it seems there are MPI implementations that can
support different architectures, so the fact that you mentioned MPI
doesn't really seem to imply what kind of architecture you have.

So, I suggest you should talk to the support staff for the computer
you will be using. They probably have a web page that you can use
to read up about how to do all this, and they are surely used to
questions from people in exactly your position, i.e. people who've
never run anything on a 100-processor system before, which would
be the majority of the people in the world. :-)

- Logan

Did you consider to use "Matlab Distributed Computing Toolbox 3.0"?

--
This signature intentionally left almost blank.

Did you see I said the super-computer does not have Matlab installed?

I like your suggestion - learning 6 functions is good. But in your tutorial
you said MPI doesnot work with C++?


I don't have to divide the program. Matlab allows me to compile a
non-parallel Matlab program into non-parallel c program and executables.

So I need to dispatch 100 copy of such executables and hopefully they will
run on 100 cpus simultaneously.

That could be changed, couldn't it? Maybe you can bring your own Matlab
in libraries?

I interpret your "answer" as "No, I did not yet consider using it.".

If you see no way to use the parallelization features that come with
the programming environment of your choice then it can get very hard.

So let me ask me another question:

At the level of control that you currently have in your program, could you
split the problem in about 100 nearly equaly time consuming parts?
If the answer is yes then you could really let run 100 instances of your
program via sh/rsh/ssh (whatever suits the given super computer).

If the answer is no, then the question becomes: can you convert the program
to get more control? Answering this question is very tough without knowing
the details. If you do not provide the details, nobody can help.
Did you try to consult some newsgroups that are centered around Matlab like
comp.soft-sys.matlab or cern.matlab?

--
This signature intentionally left almost blank.

"Mike" < XXXX@XXXXX.COM > wrote
You've got to divide the problem, not the program.
If matlab will spit out a C program that will take as a parameter 1/100 th
of your problem and return 1/100th of the answer, then you've got it made.
Simply compile the C under MPI, then in your main function write a little
bit of code to send 1/100th of the problem to each node, collect the
answers, and probaly you'll need a bit of code to collate them.

Eg you want to calcuate an average of 10000 numbers in parallel.

matlab spits out

double sum(double *x, int N)
{
int i;
doubel answer = 0;
for(i=0;i<N;i++)
answer += x[i];
return answer;
}

pur MPI program goes something like

#include "mpi.h"

int main(int argc, char **argv)
{
double array[10000];
int rank;
MPI_Init(argc, argv);
MPI_Rank(MPI_COMM_WORLD, &rank);
readinarray(array);
if(rank == 0)
{
for(i=1;i<100;i++)
/* haven't got the parameters quite right */
MPI_Send(array + i * 100, MPI_DOUBLE, 100, i, MPI_COMM_WORLD):
}
else
{
MPI_Recv(array, MPI_DOUBLE,100, 0, MPI_COMM_WORLD);
}
/* calcuate a partial answer */
answer = sum(array, 100);
if(rank != 0)
MPI_Send(&answer, MPI_DUBLE, 1, 0, MPI_COMM_WORLD):
else
{
double total = answer;
double partial;
for(i=1;i<100;i++)
{
MPI_Recv(&partial, MPI_DOUBLE, 1, i, MPI_COMM_WORLD);
total += partial;
}
printf("average is %g\n", total/10000);
}

MPI_Finalize();
}

This is only untested code, but it gives you the general idea. An average is
extremely easy to break up. Other functions might be a lot harder.

ike wrote:


Note that (a) Matlab is available for linux and (b) GNU Octave
is available for linux, if your code can run under octave (okay probably
not true in your case given maple usage, but hey), it can
help skirt the licensing issues that can crop up running
closed-source stuff like matlab on large clusters.


There are a variety of systems that both real supercomputers and
ordinary linux clusters run that manage such things (as previously
hinted by other posters).

*** You'll just have to *ask the local systems guy* what they're running
(or read the local documentation), if anything. What you want to do
sounds like a typical use case for a mid-range linux batch cluster to
munch through, though - Don't be too proud to just ask the guys running
the system you have access to, rather than random folk on usenet, they
probably know the system in question far better than anyone out here!

[Following list is non-exhaustive]

(i) batch systems:

Many, perhaps most (at least in academic/research environments),
supercomputers and linux clusters run batch job processing software.
Such software includes "Torque", "PBS", "Sun Grid Engine". In this
model, one writes a job script, submits it, it's put in a queue, and
scheduled for execution (by a batch scheduler like "maui"). Batch
processing allows fair sharing of a limited computational resource
while providing guaranteed run times of jobs and exclusive locks on
nodes.

It feels a bit 1970s I guess, but unless users tolerate highly variable
run times due to users' processes slowing eachother down (and most
_don't_ at the high end), batch processing remains the choice.

In your case, you might submit 100 jobs. The batch+scheduler system,
usually implemented as a collection of daemon processes, looks after
spawning your processes on available nodes (or reserving nodes for your
use later if need be according to some priority scheme), tracking
stdio, staging in input data and staging back out outputs, and so on,
according to your specifications laid out in the job submission
script(s) for your jobs. (you can typically pass parameters or at least
env vars into the script, thus reusing the same basic script for 1000s
of similar but nonidentical runs)

[Note 1] Alternatively you might structure your run as a single 100-proc
parallel job, perhaps with the help of an MPI implementation or a ready
made intra-batch-job task farming script.

Here are the instructions for a pretty typical batch facility:
http://www.ichec.ie/user_documentation
(I'm not affiliated with them, I run a smaller 256-proc system elsewhere
in Ireland).


(ii) Process migrating linux clusters:

In these clusters, which typically involve a patched kernel, you just
spawn everything on the head node, and the system farms out (with
varying degrees of efficiency and post-migration performance issues -
check out "kerrighed" for a particularly interesting one) to worker
nodes. These *may* _also_ be batch systems, but typically you see them
where a cluster is used by a single department or research group, where
squabbling over "hogging" of resources is less likely. The naive "kick
stuff off in a loop in shell script with &" may well work on such
clusters.

(To people in the field "beowulf" suggests a particular kind of process
migrating linux cluster, though laymen tend to use the word
in a much looser sense, thanks in no small part to trolls on /. ...)