Developer Forum

Hello,

Briefly, here is a bug that I may be confronted with and told to
spend some time on.

There are several embedded "time critical" programs on different
machines (SGI, and Linux [PC's and mainframes]) that communicate
through a shared memory file to a server program The server is
currently on a SGI system. But, any changes to the code should be
accomplished, so that the source code is transportable to a Linux
machine. All the processes run at the same speed, and were completely
written by individuals where I work.

Questions:

(1) Explicitly, what instructions can I use to prevent all the
clients from clobbering each other in their desire to use the single
communication link? I.E: The big problem is sometimes these client
programs overwrite the communication area at about the same time and
thus the network hangs up! So, when the network becomes useless like
this, I have to restart the server which is not a good thing since all
previous data is gone.

(2) Via a shared memory location (using the same shared memory file
as above and memory mapping technique), could I define a mutex to guard
the critical section of server code? Are there any problems in doing
this (such as speed reduction), and explicitly how would I fix them? I
have never implemented a mutex like this before.

(3) Are there any other alternatives?

(4) What are the advantages and disadvantages of any choice?

(5) What is my best option?

Help,
Christopher Lusardi

In article < XXXX@XXXXX.COM >,



It sounds to me as if your needs would be well served by a semaphore.
semaphores have been around a long time, and should by now be
implemented in any operating system that supports shared memory.
ipcrm semop semctl semget -- or the posix versions that start
with sem_* such as sem_init sem_wait



If you are already using POSIX Threads (pthreads), then see
pthread_mutex_init pthread_mutex_lock and kin.


Perhaps I am reading too much into your words, but it seems to me
that you were looking for a way to implement mutexes yourself,
based upon some kind of code directly accessing one of the shared
locations.

Unfortunately, the C programming language does not provide atomic
operations -- no "test and set" operation for example. The closest C
comes to that is that it does provide sig_atomic_t as a type that can
be safely accessed (i.e., read from) even in the presence of
interrupts.

If you chase down the various clauses in the C standard, about all you
can do is use a volatile sig_atomic_t variable manipulated only within
signal handlers that get invoked when you raise() signals... and even
then, as C was not designed specifically to be thread-safe, you can't
be sure that two different processors cannot enter the same signal
handler code at the same time. Trying to implement mutexes via signal
handlers is the best you can do in standard C, but it doesn't always
work in practice -- and it is almost certain to be slow.

So... you are better off finding a library function (perhaps such as
the pthread mutexes) that is implemented by all of the operating
systems that are of interest to you. If you are not using pthreads
already, you will find semaphores to be lower coding overhead to add in.

First, I am not using threading at all.

O'kay you may be helping to unconfuse me and I like your conversation!

But, it surely can not be possible to use one semaphore and no "shared
memory file" and
no memory mapping across multible computers. My goal in life is to have
each computer ask for service from another routine on a different
computer or it may be on the same computer without each request being
overwritten by other requests.

As it stands now, I would create a shared memory file, and memory map a
semaphore to that file. Different programs on different computers would
do this in a very simliar fashion.

Confused still,
Christopher Lusardi

In article < XXXX@XXXXX.COM >,


: As it stands now, I would create a shared memory file, and memory map a
: semaphore to that file. Different programs on different computers would
: do this in a very simliar fashion.

Are you trying to use a shared memory zone across multiple computers? There's a
number of very good reasons why that won't work.


Cheers - Tony 'Nicoya' Mantler :)

--
Tony 'Nicoya' Mantler -- Master of Code-fu -- XXXX@XXXXX.COM
-- http://www.yqcomputer.com/ ://www.ubb.ca/ --

The shared file is on a shared drive.

Chris Lusardi

In article < XXXX@XXXXX.COM >,
XXXX@XXXXX.COM writes:


NFS? Don't.
I would change the whole approach and transfer the data by other means.

An answer, below, is an answer given with an attempt at a simple
description! First, I already have one shared memory file which I use
for communication! It's fine.



So no, I woun't get the o'kay to stop using my already existing shared
memory file. For now, it's use the shared memory file on the shared
disk as a means of communication between separate programs on different
computers. I do this by memory mapping a C struct to that shared memory
file.

But, to install a new semaphore, I may have to create a new shared
memory file on the shared drive and then memory map all the program's
new internal semaphore to this shared memory file. All programs will do
this so that the semaphore is viable across all computers. I currently
am not using a semaphore so I have a clobbering problem that I want to
fix with the new semaphore.

In detail with maybe a simple useful pseudo program, how would you
change the whole approach of using a semaphore across SGI, Linux (and
later possibly Window) machines?

Chris Lusardi

In article < XXXX@XXXXX.COM >,



: > In article < XXXX@XXXXX.COM >,

: >
: > : As it stands now, I would create a shared memory file, and memory map a
: > : semaphore to that file. Different programs on different computers would
: > : do this in a very simliar fashion.
: >
: > Are you trying to use a shared memory zone across multiple computers?
: > There's a
: > number of very good reasons why that won't work.
:
: The shared file is on a shared drive.

There is really only one situation where you should be using a file on a shared
drive to transfer data between cluster nodes. That is when you have large
amounts of bulk data, and you're using a clustered file system on a SAN that's
significantly faster than your network.

HOWEVER.

You should never attempt to use such a shared file as a primary communications
channel. It should only be considered a side channel, to be used not unlike the
"data" channel in an FTP connection.

As an example:

Cluster node A,B,C connect to server node X via TCP sockets. All cluster nodes
share a cxfs file system mounted as /clusterdata.

Node X fills the file "/clusterdata/NodeA-Raw" with raw data from some data
source. Once filled, node X then sends a message over the TCP socket to node A
telling it to read and process "/clusterdata/NodeA-Raw".

Node A processes "/clusterdata/NodeA-Raw" into "/clusterdata/NodeA-Cooked". Once
completed, node A sends a message over the TCP socket to node X telling it to
read and integrate "/clusterdata/NodeA-Cooked", and to refill
"/clusterdata/NodeA-Raw" with new data.

Meanwhile, node X also fills "/clusterdata/NodeB-Raw", "/clusterdata/NodeC-Raw"
with data, separately signals B and C via TCP to start processing, etc etc.

You could also supplement this arrangement with *read only* shared data, like
"/clusterdata/Shared-TransformMatrix", which would *only* be updated by Node X,
and would *only* be updated when all nodes are idle; either by waiting until all
nodes have processed and returned all current data, or by signalling through the
TCP channel that the nodes should abort current processing and wait for new data.


Now, if you don't have large amounts of bulk data or if your network is
sufficiently fast to pass them that way, then I would strongly encourage you to
use a network-only approach. I would also strongly suggest a network-only
approach if you don't have a clustered FS SAN at all and are using a network
filesystem like CIFS, SMB or NFS.

If you need a high level of inter-node communication (like, if node calculations
are interdependant on the results of other node calculations) then you'll need a
different approach. You should post more details of the problem you're trying to
calculate if this is the case.


Cheers - Tony 'Nicoya' Mantler :)

--
Tony 'Nicoya' Mantler -- Master of Code-fu -- XXXX@XXXXX.COM
-- http://www.yqcomputer.com/ ://www.ubb.ca/ --

Do you have any simple implementations that you can refer me to?


For the foreseeable future, it appears that we will be keeping our
original shared file on the shared drive to do communication between
separate programs on different computers (SGI and Linux). But, to come
up with a temporary fix for our clobbering problem, is it o'kay to
create a new shared memory file on a shared drive and memory map
everybody's semaphone to it? This new file will only be used for the
semaphore. In simple English, please explain to me what specific
problems will I run into?

Christopher Lusardi

In < XXXX@XXXXX.COM >, on
02/10/2006
at 07:09 AM, XXXX@XXXXX.COM said:


If the file is mapped into different machines then it is *NOT* shared
memory and you need to worry about synchronization. But it's not my
dog.

--
Shmuel (Seymour J.) Metz, SysProg and JOAT < http://www.yqcomputer.com/ ~shmuel>

Unsolicited bulk E-mail subject to legal action. I reserve the
right to publicly post or ridicule any abusive E-mail. Reply to
domain Patriot dot net user shmuel+news to contact me. Do not
reply to XXXX@XXXXX.COM

In article < XXXX@XXXXX.COM >,



: > Now, if you don't have large amounts of bulk data or if your network is
: > sufficiently fast to pass them that way, then I would strongly encourage you to
: > use a network-only approach. I would also strongly suggest a network-only
: > approach if you don't have a clustered FS SAN at all and are using a network
: > filesystem like CIFS, SMB or NFS.
:
: Do you have any simple implementations that you can refer me to?

Most people use MPI or something similar. < http://www.yqcomputer.com/ ;


: For the foreseeable future, it appears that we will be keeping our
: original shared file on the shared drive to do communication between
: separate programs on different computers (SGI and Linux). But, to come
: up with a temporary fix for our clobbering problem, is it o'kay to
: create a new shared memory file on a shared drive and memory map
: everybody's semaphone to it? This new file will only be used for the
: semaphore. In simple English, please explain to me what specific
: problems will I run into?

The problem you'll run into is that semaphores are not guaranteed to work
between multiple systems. In fact, mmap() isn't guaranteed to be coherent across
systems either without careful use of msync() and some sort of mutex-like
exclusion to prevent multiple writers.

What you are asking to do is simply not viable nor advisable.


Cheers - Tony 'Nicoya' Mantler :)

--
Tony 'Nicoya' Mantler -- Master of Code-fu -- XXXX@XXXXX.COM
-- http://www.yqcomputer.com/ ://www.ubb.ca/ --

In article < XXXX@XXXXX.COM >,



You latched on to my mention of semaphores a bit too soon ;-)
I did not realize at the time that your "shared memory" consisted
of a common nfs mounted filesystem.

If you have a shared file and you want to ensure that only one
process at a time is writing to it, then you should use rpc.lockd .
You could just request locking the entire file, or if you can
divide your work more finely, you could lock particular records.

If for some reason you cannot use rpc.lockd then you can fake
a lock by using open() with O_CREAT|O_EXCL in the option and
naming a file in a shared nfs directory. If another process already
has the file open then the open() will fail, so your process could
sleep and try again later. If the open() succeeds then you own
the lock and can go ahead and write to the shared mmap()'d file.

The make-shift lock will persist as long as the file exists with
that name. If the writing process is somehow able to detect
that it is now safe for other processes to write to the shared
area, then the writing process could be the one that removes
the file. It might, though, turn out to make more sense to
have the reading server process remove the lock file when it has finished
processing the data. You will, though, need to find some way for
the writing process to signal to the reading process that it has
finished putting in the request.

So, to clarify, rpc.lockd will allow different programs on different
computers (Linux, SGI, and Windows) to an equivalent of a semaphore
(mutex specifically)?

Are there any better alternatives to rpc.lockd?

Thanks,
Christopher Lusardi

So, to clarify, rpc.lockd will allow different programs on different
computers (Linux, SGI, and Windows) to do an equivalent of a semaphore
(mutex specifically)? Where one program can lock a file and other
programs on different computers would wait until the file becomes
unlocked. Only when the file becomes unlocked will a program continue.

Questions: With rpc.lockd is it possible to have a timeout on the
amount of waiting before the file becomes unlocked. For example, if a
program doesn't have to have the locked file then it can give up trying
to access it.

Are there any better alternatives to rpc.lockd?


Thanks,
Christopher Lusardi