Massive performance loss from OS::sleep hack

Massive performance loss from OS::sleep hack

Post by kris » Mon, 17 Sep 2007 02:55:33


Hi,

I have been running the volano java benchmark
( http://www.yqcomputer.com/ ) on an 8-core i386 system, and
out of the box jdk15 on FreeBSD performs extremely poorly. The system
is more than 90% idle, and profiling shows that the ~800 threads in the
benchmark are spending most of their time doing short nanosleep() calls.


I traced it to the following FreeBSD-specific hack in the jdk:

// XXXBSD: understand meaning and workaround related to yield
...
// XXXBSD: done differently in 1.3.1, take a look
int os::sleep(Thread* thread, jlong millis, bool interruptible) {
assert(thread == Thread::current(), "thread consistency check");
...

if (millis <= 0) {
// NOTE: workaround for bug 4338139
if (thread->is_Java_thread()) {
ThreadBlockInVM tbivm((JavaThread*) thread);
// BSDXXX: Only use pthread_yield here and below if the system thread
// scheduler gives time slices to lower priority threads when yielding.
#ifdef __FreeBSD__
os_sleep(MinSleepInterval, interruptible);
#else
pthread_yield();
#endif

When I removed this hack (i.e. revert to pthread_yield()) I got an
immediate 7-fold performance increase, which brings FreeBSD performance
on par with Solaris.

What is the reason why this code is necessary? Does FreeBSD's
sched_yield() really have different semantics to the other operating
systems, or was this a libkse bug that was being worked around?

Kris
_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by deische » Mon, 17 Sep 2007 03:47:05


It's certainly not a libkse bug, at least with scope process threads
libkse does the right thinng. For scope system threads and all libthr
threads, it probably depends on what scheduler you are using since
it's essentially a __sys_sched_yield().

On a side note, I think pthread_yield() is deprecated and not in the
latest POSIX spec. sched_yield() is in the spec and is specified
to account for behavior in a threaded environment.

--
DE
_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "

 
 
 

Massive performance loss from OS::sleep hack

Post by kris » Mon, 17 Sep 2007 04:13:50


Yeah, libthr's pthread_yield just calls sched_yield. Anyway, it seems
to me that the decision of what thread to run next is a scheduler
decision, and if it is inappropriate for some reason for a scheduler to
possibly choose to reschedule the same thread that just yielded, this is
something that should be addressed in the scheduler rather than by
adding hacks to the application.

Kris

_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by freebs » Mon, 17 Sep 2007 05:16:49

I looked up the bug # in Sun's bug database, and although it isn't there
anymore, there is another bug marked as a duplicate:

http://www.yqcomputer.com/

I don't pretend to understand the details, but it looks like that bit of
code was introduced to work around a Solaris kernel bug related to threads
not yielding/suspending.

I wonder if that might have been a problem a long time ago with libc_r
that no longer troubles us. Fortunately, the bug indicates that it is
very easy to repeat (though that may be more true on Solaris than
FreeBSD), so it could be easy to test whether just using pthread_yield()
is now sufficient.

Nick




--
"Courage isn't just a matter of not being frightened, you know. It's being
afraid and doing what you have to do anyway."
Doctor Who - Planet of the Daleks
This message has been brought to you by Nick Johnson 2.3b1 and the number 6.
http://www.yqcomputer.com/ ://morons.org/ http://www.yqcomputer.com/
_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by kurt » Mon, 17 Sep 2007 11:56:21

n Saturday 15 September 2007 01:50:13 pm Kris Kennaway wrote:

Hello Kris,

I recall why I added the os_sleep() call. While working on the certification
testing one of the JCK tests was deadlocking. The test itself was faulty in
that it created a high priority thread in a tight yield loop. Since
pthread_yield() on a single processor system will not yield to lower
priority threads, the higher priority thread effectively blocked the
lower priority thread from running and the test deadlocked.

I filed a formal appeal to have the JCK test corrected. However since the
api states:

http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Thread.html#yield()
"Causes the currently executing thread object to temporarily pause
and allow other threads to execute."

the appeal was denied. I further argued that many publications written
by or or-authored by Sun employees state that Thread.yield() can not
be relied upon in this fashion:

"However, I believe there are other less authoritative sources
of information that support the position that yield() can not
expected to allow lower priority threads to execute in all cases.
For example the following Sun document describes yield() as
advisory only and not to depend on it for correctness:

http://java.sun.com/j2se/1.5.0/docs/guide/vm/thread-priorities.html#general

The document also refers to a Sun published book, "Effective
Java Programming Language Guide" that describes yield()
should not be relied on for correctness and portability.

Another book I have specifically addresses the topic. It states
that most schedulers do not stop the yielding thread from running
in favor of a thread of lower priority. This is from the Sybex
"Complete Java 2 Certification Study Guide" co-authored by
a Sun employee.

The publications I have referred to above present a case that the
api description of yield() doesn't fully describe the expected behavior
of the function when combined with thread priorities. While they are
not the authoritive publications on the function, I think they represent
the general historical interpretation of the expected behavior."

In the end I was not able to convince Sun to change the JCK test
so the os_sleep() call was added.

I see in my notes that kse on a single cpu system had the issue
but thr didn't (this is on 6.1-release). Perhaps now that thr is the
default on 7.0 this hack can be made conditional and only applied
to < 7.0.

The following are programs I wrote when I isolated the problem.
If the c program runs ok on 7.0 for both single cpu and mp then
remove the os_sleep() and try the java program. If that works too
then you're clear to make the os_sleep() hack only apply to <
7.0 and still be able to pass the certification tests.

Regards,
-Kurt

-----------
#include <pthread.h>
#include <unistd.h>
#include <signal.h>
#include <inttypes.h>
#include <time.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

volatile int init=0;
volatile int interrupt=0;

static void *
yielder(void *arg)
{
init = 1;
while (1) {
pthread_yield();
}
}

static void
sighandler(int sig)
{
interrupt = 1;
printf("sighandler\n");
static void
sighandler(int sig)
{
interrupt = 1;
printf("sighandler\n");
}

static void
waitForInit() {
 
 
 

Massive performance loss from OS::sleep hack

Post by kurt » Mon, 17 Sep 2007 11:56:28

n Saturday 15 September 2007 01:50:13 pm Kris Kennaway wrote:

Hello Kris,

I recall why I added the os_sleep() call. While working on the certification
testing one of the JCK tests was deadlocking. The test itself was faulty in
that it created a high priority thread in a tight yield loop. Since
pthread_yield() on a single processor system will not yield to lower
priority threads, the higher priority thread effectively blocked the
lower priority thread from running and the test deadlocked.

I filed a formal appeal to have the JCK test corrected. However since the
api states:

http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Thread.html#yield()
"Causes the currently executing thread object to temporarily pause
and allow other threads to execute."

the appeal was denied. I further argued that many publications written
by or or-authored by Sun employees state that Thread.yield() can not
be relied upon in this fashion:

"However, I believe there are other less authoritative sources
of information that support the position that yield() can not
expected to allow lower priority threads to execute in all cases.
For example the following Sun document describes yield() as
advisory only and not to depend on it for correctness:

http://java.sun.com/j2se/1.5.0/docs/guide/vm/thread-priorities.html#general

The document also refers to a Sun published book, "Effective
Java Programming Language Guide" that describes yield()
should not be relied on for correctness and portability.

Another book I have specifically addresses the topic. It states
that most schedulers do not stop the yielding thread from running
in favor of a thread of lower priority. This is from the Sybex
"Complete Java 2 Certification Study Guide" co-authored by
a Sun employee.

The publications I have referred to above present a case that the
api description of yield() doesn't fully describe the expected behavior
of the function when combined with thread priorities. While they are
not the authoritive publications on the function, I think they represent
the general historical interpretation of the expected behavior."

In the end I was not able to convince Sun to change the JCK test
so the os_sleep() call was added.

I see in my notes that kse on a single cpu system had the issue
but thr didn't (this is on 6.1-release). Perhaps now that thr is the
default on 7.0 this hack can be made conditional and only applied
to < 7.0.

The following are programs I wrote when I isolated the problem.
If the c program runs ok on 7.0 for both single cpu and mp then
remove the os_sleep() and try the java program. If that works too
then you're clear to make the os_sleep() hack only apply to <
7.0 and still be able to pass the certification tests.

Regards,
-Kurt

-----------
#include <pthread.h>
#include <unistd.h>
#include <signal.h>
#include <inttypes.h>
#include <time.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

volatile int init=0;
volatile int interrupt=0;

static void *
yielder(void *arg)
{
init = 1;
while (1) {
pthread_yield();
}
}

static void
sighandler(int sig)
{
interrupt = 1;
printf("sighandler\n");
static void
sighandler(int sig)
{
interrupt = 1;
printf("sighandler\n");
}

static void
waitForInit() {
 
 
 

Massive performance loss from OS::sleep hack

Post by list » Mon, 17 Sep 2007 12:10:38


Sorry copy/paste glitch. He is the c program again:

#include <pthread.h>
#include <unistd.h>
#include <signal.h>
#include <inttypes.h>
#include <time.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

volatile int init=0;
volatile int interrupt=0;

static void *
yielder(void *arg)
{
init = 1;
while (1) {
pthread_yield();
}
}

static void
sighandler(int sig)
{
interrupt = 1;
printf("sighandler\n");
}

static void
waitForInit() {
struct timespec t, rt;

while (init == 0) {
t.tv_sec = 0;
t.tv_nsec = 100000;
nanosleep(&t, &rt);
}
}

static void
waitForInterrupt() {
struct timespec t, rt;

while (interrupt == 0) {
t.tv_sec = 0;
t.tv_nsec = 100000;
nanosleep(&t, &rt);
}
}

int
main(int argc, char *argv[])
{
pthread_t yldr;
pthread_attr_t attr;
struct sigaction act;

/* Install a signal handler for SIGUSR1 */
sigemptyset (&act.sa_mask);
sigaddset (&act.sa_mask, SIGUSR1);
act.sa_handler = sighandler;
act.sa_flags = 0;
sigaction (SIGUSR1, &act, NULL);

pthread_attr_init(&attr);
pthread_attr_setschedpolicy(&attr, SCHED_FIFO);

pthread_create(&yldr, &attr, yielder, NULL);
pthread_setprio(yldr, 16);
waitForInit();
if(pthread_kill(yldr, SIGUSR1) != 0)
printf("pthread_kill failed with errno = %d\n", errno);
waitForInterrupt();
}
_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by deische » Mon, 17 Sep 2007 13:53:15


[ ... ]

It's odd that Sun would take this position since the POSIX behavior
for sched_yield() is:

The sched_yield() function shall force the running thread to
relinquish the processor until it again becomes the head of its
thread list. It takes no arguments.

The "its thread list" mentioned above is the list of threads for its
given priority. POSIX has a notion of a list of threads for each
given priority; in SCHED_RR and SCHED_FIFO scheduling the higher
priority threads will always run before lower priority threads.

Sun's defined Java behavior, or their interpretation, of Thread.yield()
is not obtainable on a POSIX compliant system using sched_yield()
(or pthread_yield()). Even using scope system threads or libthr,
the kernel scheduler should run the higher priority threads before
lower priority threads.

I suspect that Sun will eventually be forced to change their
documented behavior for Thread.yield().

--
DE
_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by deische » Mon, 17 Sep 2007 14:21:22


[ ... ]

It's odd that Sun would take this position since the POSIX behavior
for sched_yield() is:

The sched_yield() function shall force the running thread to
relinquish the processor until it again becomes the head of its
thread list. It takes no arguments.

The "its thread list" mentioned above is the list of threads for its
given priority. POSIX has a notion of a list of threads for each
given priority; in SCHED_RR and SCHED_FIFO scheduling the higher
priority threads will always run before lower priority threads.

Sun's defined Java behavior, or their interpretation, of Thread.yield()
is not obtainable on a POSIX compliant system using sched_yield()
(or pthread_yield()). Even using scope system threads or libthr,
the kernel scheduler should run the higher priority threads before
lower priority threads.

I suspect that Sun will eventually be forced to change their
documented behavior for Thread.yield().

--
DE
_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by kip.mac » Mon, 17 Sep 2007 14:59:17

Or more likely they'll continue to maintain a sched_yield that isn't
posix compliant. We may just want to add some sort of interface so the
jvm can tell the kernel that sched_yield should be non-compliant for
the current process.





_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by kip.mac » Mon, 17 Sep 2007 14:59:27

Or more likely they'll continue to maintain a sched_yield that isn't
posix compliant. We may just want to add some sort of interface so the
jvm can tell the kernel that sched_yield should be non-compliant for
the current process.





_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by kris » Mon, 17 Sep 2007 19:22:35


Thanks, I'll test this out and see where we stand.

Kris

_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by deische » Tue, 18 Sep 2007 00:51:47


I don't think that is a good idea, it seems like too much of a hack.
The scheduler(s) should schedule threads the way they are designed
to, either obeying a threads priority, using it as a hint, or totally
ignoring it.

If the JVM kept track of the thread priorities in use, I suppose
Thread.yield() could first lower the current thread's priority to
the next used priority and then yield, raising the priority back after
the yield. This isn't perfect, there are race conditions, the next
highest priority thread(s) could be blocked and not runnable, etc.
Maybe just lowering the priority to min or default priority would work
well enough.

This test would fail even on Solaris if you use SCHED_RR or SCHED_FIFO
since it is POSIX compliant for those scheduling classes.

--
DE
_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by deische » Tue, 18 Sep 2007 00:52:38


I don't think that is a good idea, it seems like too much of a hack.
The scheduler(s) should schedule threads the way they are designed
to, either obeying a threads priority, using it as a hint, or totally
ignoring it.

If the JVM kept track of the thread priorities in use, I suppose
Thread.yield() could first lower the current thread's priority to
the next used priority and then yield, raising the priority back after
the yield. This isn't perfect, there are race conditions, the next
highest priority thread(s) could be blocked and not runnable, etc.
Maybe just lowering the priority to min or default priority would work
well enough.

This test would fail even on Solaris if you use SCHED_RR or SCHED_FIFO
since it is POSIX compliant for those scheduling classes.

--
DE
_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "
 
 
 

Massive performance loss from OS::sleep hack

Post by kip.mac » Tue, 18 Sep 2007 05:02:11


Yes, if we could hack the jvm to work around this without calling
sleep that would be better yet. However, making java work well is more
important than keeping the interface clean.


I honestly don't think it matters. The JCK using it implies that there
are probably java apps that rely on this defective behavior. I would
be willing to bet that Solaris and Java developers at Sun already had
this argument and the Java developers won :-(.

-Kip
_______________________________________________
XXXX@XXXXX.COM mailing list
http://www.yqcomputer.com/
To unsubscribe, send any mail to " XXXX@XXXXX.COM "