Developer Forum

Obviously, yes. The script is some 150 lines long. But the hang-up
occurs because of the forked Java process, not the other lines.
>> If not, you could just invoke java directly from Python. Also, gt; > you set stdin=PIPE - is your java process expecting some input? you're not >
> writing anything to stdin.

It does not expect input from stdin. However, this does not affect
any OTHER scripts or commands I run.

Let's remember to look at the objective facts: for shell scripts that
launch child processes of their own, Python hangs. For all other
types of commands, it works 100% as expected. >
> Anyway, it's better to use the communicate method instead (it uses select> > to read from both stdout and stderr):

That doesn't help me.>
> See ttp://docs.python.org/library/subprocess.html#subprocess.Popen.commu...

I have.

OK, so, I'm very confused as to why this would matter.

I removed the stdin=PIPE argument, and this works. Many thanks for
bringing this to my attention.

But, my question now is, WHY is this an issue? If the launched
process doesn't read from its stdin, why would it block? Hence my
question here:


Meaning, if I launched the Java process directly, it works fine. If I
launch it from the shell script WITHOUT background execution, it works
fine. But when I launch it WITH background execution (e.g., with the
& suffix), then it blocks.

Any ideas, so I can write this into my log for future reference?

Thanks.

OK, I am confused. After observing a bug where the code works "every
other time", like clockwork, I've used strace to figure out what is
going on. In the times when "it works," it's only the shell script
complaining that it's already running.

So, I'm right back to square one. Even _without_ setting stdin=PIPE,
it fails to unblock. I'm left utterly bewildered.

The only thing I can think of is that the JVM's stdout is tied to the
shell script's stdout, because that's the only way it can remain open
upon the termination of the child process. I suppose, at this point,
the next place to look is in shell script syntax to find out how to
detach the JVM from the script's process group.

> > Is your shell script doing something else, apart from invoking the java >>>> process? >> >> Obviously, yes. he script is some 150 lines long. ut the hang-up >> occurs because of the forked Java process, not the other lines. >> >>>> If not, you could just invoke java directly from Python. Also, gt; > > you set stdin=PIPE - is your java process expecting some input? you're not > >> > writing anything to stdin>
> It does not expect input from stdin. owever, this does not affec>
> any OTHER scripts or commands I run>
> Let's remember to look at the objective facts: for shell scripts tha>
> launch child processes of their own, Python hangs. or all othe>
> types of commands, it works 100% as expected>
> >> > Anyway, it's better to use the communicate method instead (it uses select>gt; > > to read from both stdout and stderr>: >>
> That doesn't help m>. >>> > > See ttp://docs.python.org/library/subprocess.html#subprocess.Popen.commu.>. >>
> I have.

You should be nicer to Gabriel. He is a guru.

En Wed, 08 Oct 2008 15:24:39 -0300, Samuel A. Falvo II
< XXXX@XXXXX.COM > escribi

>>> you set stdin=PIPE - is your java process expecting some input? you're >>> not writing anything to stdin. >> >> It does not expect input from stdin. However, this does not affect >> any OTHER scripts or commands I run.

But it *does* affect how subprocess handles the child process internally.
>> Let's remember to look at the objective facts: for shell scripts that >> launch child processes of their own, Python hangs. For all other >> types of commands, it works 100% as expected.

Don't conclude too fast... I've tested a variant (using a Perl script as a
replacement instead of a Java program), and it worked fine:
<
gabriel@sleipnir:~$ cat foo.sh
#!/bin/sh
perl foo.pl&
echo End of foo.sh

gabriel@sleipnir:~$ cat foo.pl
#!/usr/bin/perl

for ($i=5; $>>0; $i--) {
print "$i seconds remaining...\n";
sleep(1);
}
print "End of foo.pl\n";

gabriel@sleipnir:~$ cat foo.py
#!/usr/bin/python2.5

import subprocess

command = "./foo.sh"

p = subprocess.Popen(command,
shell=True,
stdin=None,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
close_fds=True
)
print "py: before p.communicate"
output = p.communicate()[0]
print "py: after p.communicate"
print "py: returncode=",p.returncode
print "py: read",len(output),"bytes"
print "py: output=", repr(output)
print "py: End of foo.py"


gabriel@sleipnir:~$ ./foo.py
py: before p.communicate
py: after p.communicate
py: returncode= 0
py: read 143 bytes
py: output= 'End of foo.sh\n5 seconds remaining...\n4 seconds
remaining...\n3 seconds remaining...\n2 seconds remaining...\n1 seconds
remaining...\nEnd of foo.pl\n'
py: End of foo.py
gabriel@sleipnir:~$ uname -a
Linux debian 2.6.18-4-486 #1 Mon Mar 26 16:39:10 UTC 2007 i686 GNU/Linux
gabriel@sleipnir:~$ <

This was tested both with python 2.4.4 and 2.5.2
I don't think it's a Python problem, but something related to java and how
it handles stdin/stdout. Try with another program.
>> But, my question now is, WHY is this an issue? If the launched >> process doesn't read from its stdin, why would it block?

Several reasons - the child process might send enough text to stderr to
fill its buffer, and if the parent process doesn't read from the other end
of the pipe, the child blocks. That's why I suggested to use communicate
instead of stdout.read() - it takes care of such cases.
>> The only thing I can think of is that the JVM's stdout is tied to the >> shell script's stdout, because that's the only way it can remain open >> upon the termination of the child process. I suppose, at this point, >> the next place to look is in shell script syntax to find out how to >> detach the JVM from the script's process group.

Can't you redirect to a file instead?
java foo.jar >>/tmp/foo.log >>&1 &

--
Gabriel Genellina

On Tue, Oct 07, 2008 at 05:43:41PM -0700, Samuel A. Falvo II wrote:

You call read with no arguments. This is highly problematic in the
context of interprocess communication, unless you can be 100% positive
that none of the children will write anywhere besides STDOUT, and
won't try to read from STDIN in the meanwhile.

Python's read() with no args reads until the end of file, which in IPC
contexts is bad... Normally the child process won't close stdout
until it exits. So, if it did any other output in between, say, to
STDERR, the child will block waiting for the parent to read STDERR,
meanwhile the parent is blocked waiting for input from the child's
STDOUT, which results in a deadlock. Both processes sleep forever.
The exact same thing can happen if either the shell script or a
process started by the shell script tries to read from STDIN.

Since Java is launched by the shell script, it inherits the shell
script's STDIN, STDOUT, and STDERR file descriptors (i.e. the two
processes share the same STDIO). Thus if the java process writes to
STDERR, that also could be causing your deadlock.


On Wed, Oct 08, 2008 at 11:24:39AM -0700, Samuel A. Falvo II wrote:

It's far from obvious. I can't count the number of scripts I've seen
whose sole purpose was to launch a Java program (with the right
environment)... You would do well to avoid being dismissive and
listen to those who respond with help, when you are the one who
obviously doesn't understand the behavior you're getting, and you're
the one asking for help.
>> The script is some 150 lines long. But the hang-up >> occurs because of the forked Java process, not the other lines.

I'm betting it's because the Java program is writing warnings to
STDERR (more data than will fit in the STDIO buffer), which you're not
reading...
>>>> If not, you could just invoke java directly from Python. Also, gt; >>
> It does not expect input from stdin. However, this does not affec>
> any OTHER scripts or commands I run.

Irrelevant... Unless "any OTHER scripts" encompases all possible
combinations of process interactions, and you can demontstrate that it
does so, this proves nothing. >
> Let's remember to look at the objective facts: for shell scripts tha>
> launch child processes of their own, Python hangs. For all othe>
> types of commands, it works 100% as expected.

These are not facts which are in evidence. We don't know what your
script is doing, and it's clear that you yourself are not able to
explain the behavior you are seeing, therefore there is no reason for
us to conclude that the above statements are true and correct. Most
likely, they are not. > >> > Anyway, it's better to use the communicate method instead (it uses select>gt; > > to read from both stdout and stderr>:
> That doesn't help me.

Please explain why it doesn't.

The most likely cause of the behavior you are seeing is the deadlock I
described, above. Using select() (i.e. using communicate()) should
generally fix about half the cases... The rest would be fixed by
redirecting STDIN of the child (or at least the Java process) from
/dev/null, most likely.

Then of course, there could be other explanations. But this being
overwhelmingly the most likely one, if you don't try those solutions,
there's no point in trying to help you further...

--
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D