Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3

Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3

Post by Omar Kilan » Fri, 13 Aug 2004 22:00:09


ist,

I'm doing some "benchmarking" using ApacheBench against a bunch of
different operating systems and kernel versions, and noticed what
appears (by appears I mean "this may be intended behavior that I am
not aware of") a severe performance degradation between 2.4 and 2.6.

In this case, I'm testing the same version of Apache (2.0.50) on the
same hardware, using my own distributions (one is 2.4 based and uses
the RHEL 2.4.21-15.0.3 kernel, the other is 2.6 based and uses
2.6.8-rc3, although 2.6.8-rc4-bk1 exhibits the same issues.)

Hardware: Pentium 4 2.8GHz HT (running the SMP kernel) with 512MB of
RAM, and an 80Gb WD HDD.

Some numbers:

2.4.21-2ts:

[root@rio root]# hdparm -T -t /dev/hda

/dev/hda:
Timing buffer-cache reads: 2316 MB in 2.00 seconds = 1158.00 MB/sec
Timing buffered disk reads: 176 MB in 3.01 seconds = 58.47 MB/sec

[root@rio i686-pc-linux-gnu]# ./bw_tcp -s
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 1284.40 MB/sec
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 828.20 MB/sec
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 1346.99 MB/sec
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 1372.90 MB/sec
[root@rio i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 871.46 MB/sec

[root@rio root]# ab -n 100000 -c 2 http://localhost/index.html
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests


Server Software: Apache/2.0.50
Server Hostname: localhost
Server Port: 80

Document Path: /index.html
Document Length: 51200 bytes

Concurrency Level: 2
Time taken for tests: 34.37600 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 852584179 bytes
HTML transferred: 825083904 bytes
Requests per second: 2937.93 [#/sec] (mean)
Time per request: 0.681 [ms] (mean)
Time per request: 0.340 [ms] (mean, across all concurrent requests)
Transfer rate: 24461.21 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 0 0.0 0 5
Waiting: 0 0 0.0 0 2
Total: 0 0 0.0 0 5

Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 5 (longest request)

Multiple runs of ab produce similar results.

2.6.8-rc4-bk1:

[root@odin vm]# hdparm -T -t /dev/hda

/dev/hda:
Timing buffer-cache reads: 2384 MB in 2.00 seconds = 1192.18 MB/sec
Timing buffered disk reads: 124 MB in 3.01 seconds = 41.20 MB/sec

[root@odin i686-pc-linux-gnu]# ./bw_tcp -s
[root@odin i686-pc-linux-gnu]# ./bw_tcp 127.0.0.1
0.065536 404.62 MB/sec

(Ran this 5 times with no real difference between runs.)

[root@odin ~]# ab -n 100000 -c 2 http://localhost/ind
 
 
 

Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3

Post by Omar Kilan » Fri, 13 Aug 2004 22:40:10

Maciej,

On Thu, 12 Aug 2004 15:09:22 +0200, Maciej Soltysiak


This halves all my performance values.

I've just tried 2.6.8-rc4-mm1 with the same results.

I also tested the 2.6.8-rc4-mm1 kernel on a dual P4 Xeon 3.2GHz with
2MB Cache (with HT, so 4 logical processors) using 15K RPM U320
Fujitsu SCSI drives.

[root@minbar root]# hdparm -T -t /dev/sda

/dev/sda:
Timing buffer-cache reads: 1852 MB in 2.00 seconds = 927.07 MB/sec
Timing buffered disk reads: 218 MB in 3.02 seconds = 72.29 MB/sec

[root@minbar root]# ./bw_tcp 127.0.0.1
0.065536 333.85 MB/sec

Ran this 5 times with no real difference in value.

[root@minbar root]# ab -n 100000 -c 2 http://localhost/index.html
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.yqcomputer.com/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.yqcomputer.com/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests


Server Software: Apache/2.0.50
Server Hostname: localhost
Server Port: 80

Document Path: /index.html
Document Length: 51200 bytes

Concurrency Level: 2
Time taken for tests: 43.787872 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 852032704 bytes
HTML transferred: 825032704 bytes
Requests per second: 2283.74 [#/sec] (mean)
Time per request: 0.876 [ms] (mean)
Time per request: 0.438 [ms] (mean, across all concurrent requests)
Transfer rate: 19002.13 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 0 0.1 0 12
Waiting: 0 0 0.0 0 1
Total: 0 0 0.1 0 12

Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 12 (longest request)

Ran this 5 times with no real difference in performance.

So a single 2.8GHz P4 HT with a 2.4 kernel has better performance than
a dual Xeon 3.2GHz HT with a 2.6 kernel. I think the SCSI drives
eliminate the IDE results as a factor, so this looks like it's a net
loopback performance issue. Maybe. :)


Regards,
Omar Kilani
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/

 
 
 

Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3

Post by Omar Kilan » Fri, 13 Aug 2004 23:10:03

eplying to self,

Some additional stats. I ran 'vmstat 1' during the ab run. Results:

2.4.21-2ts (RHEL 2.4.21-15.0.3):

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 304852 18760 142880 0 0 0 0 104 22327 39 50 10 0
2 0 0 304256 18760 143196 0 0 0 0 103 22740 38 50 12 0
1 0 0 303176 18760 143504 0 0 0 0 106 22306 41 47 12 0
3 0 0 302808 18760 143824 0 0 0 0 102 22410 36 50 15 0
1 0 0 302548 18768 144140 0 0 0 1612 131 22384 39 50 11 0
3 0 0 302224 18768 144456 0 0 0 0 107 22707 34 57 9 0
3 0 0 301932 18768 144776 0 0 0 0 104 22399 42 44 14 0
3 0 0 301784 18768 145092 0 0 0 0 103 22635 36 52 11 0
2 0 0 301164 18768 145408 0 0 0 0 102 22613 32 57 11 0
2 0 0 301064 18776 145724 0 0 0 1608 134 22595 38 49 13 0
1 0 0 300672 18776 146040 0 0 0 0 102 22444 38 51 11 0
1 0 0 300088 18776 146356 0 0 0 0 103 21374 40 50 10 0
2 0 0 299628 18776 146668 0 0 0 0 109 20670 31 52 17 0
2 0 0 299440 18780 146988 0 0 0 0 103 21666 44 46 10 0
3 0 0 299100 18788 147304 0 0 0 1664 138 22305 36 51 12 0
2 0 0 298664 18788 147620 0 0 0 0 108 22328 36 49 15 0
2 0 0 298264 18788 147936 0 0 0 0 105 22404 34 50 17 0
1 0 0 297792 18788 148248 0 0 0 0 105 22415 37 50 13 0
1 0 0 297412 18788 148564 0 0 0 0 105 22480 37 50 13 0
2 0 0 296560 18804 148876 0 0 0 1640 138 22471 35 51 14 0
1 0 0 296040 18804 149188 0 0 0 0 104 22395 36 51 13 0
2 0 0 296048 18804 149504 0 0 0 0 104 22405 32 52 16 0
2 0 0 295440 18804 149820 0 0 0 0 109 22162 37 50 13 0
1 0 0 294676 18804 150136 0 0 0 0 106 22317 37 48 15 0
1 0 0 294384 18812 150452 0 0 0 1580 132 22068 41 46 12 1
1 0 0 294208 18812 150768 0 0 0 0 104 22168 42 49 9 0
3 0 0 293796 18816 151084 0 0 0 0 109 22345 40 49 12 0
2 0 0 293300 18816 151396 0 0 0 0 104 22793 37 49 14 0
1 1 0 292828 18816 151712 0 0 0 1568 106 22518 36 52 11 0
3 0 0 292680 18824 152028 0 0 0 40 139 22407 41 47 12 0
2 0 0 291952 18824 152344 0 0 0 0 105 21948 39 51 10 0
2 0 0 291092 18824 152668 0 0 0 0 104 20059 35 50 15 0
1 0 0 289944 18824 152908 0 0 0 0 114 13542 39 39 22 0
0 0 0 293840 18832 152908 0 0 0 1236 140 53 12 0 84 4
0 0 0 293840 18832 152908 0 0 0 0 103 11 0 0 100 0


2.6.8-rc4-mm1:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 0 0 396092 8300 55140 0 0 0 0 1014 13901 23 65 12 0
2 0 0 396796 8300 55140 0 0 0 0 1007 11587 30 66 4 0
1 0 0 395964 8300 55660 0 0 0 0 1005 14045 23
 
 
 

Performance Degradation: 2.6.8-rc4-bk1 vs RHEL 2.4.21-15.0.3

Post by Nick Piggi » Sat, 14 Aug 2004 10:00:12


Take a profile of each kernel. Boot with profile=1, then run:

echo 0 > /proc/profile ; ab --foo --bar ; readprofile | sort -nr |
head -n30 > profile.out

And send in the profiles for both kernels.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/