Linux disk performance.

Linux disk performance.

Post by Manish Reg » Tue, 19 Dec 2006 13:20:04


Hi all,
I was working in one application that requires heavy disk
writes, I noticed some inconsistencies in the write timing.
We are using raw reads to bypass filesystem overhead.

Firstly i tried open("/dev/hda",O_RDWR) i.e without O_DIRECT option.
I saw that after some writes 1 write took too much time.

the results are for writing 128KB data in MIPS 400mhz
sequence channel time (in microseconds)
0 1 1675
0 2 1625
0 3 1836
...
0 16 3398
0 63 1678
1 0 1702
1 1 1845
.....
3 46 17875 // large value
...
4 13 17142 ///
...
4 44 18711 /// large value

Is this behaviour ok?
I beleive this is due to deep request queue.

But when i used O_DIRECT. I got a little higher write times but it
also had such time bumps but at smaller rate.
-----------------------------------------
0 0 3184
0 1 3165
0 2 3126
...
0 52 10613 // large value
0 60 19004 // large value

results similar with O_DIRECT|O_SYNC


Can we achieve smooth write times in Linux?

I am using 2.6.10 the results are moreover same (i dont mean
numerically same but i am getting thiming difference) in both P4 3 GHZ
512MB ram and MIPS. Disk is working in UDMA 5.

--
---------------------------------------------------------------
regards
Manish Regmi

---------------------------------------------------------------
UNIX without a C Compiler is like eating Spaghetti with your mouth
sewn shut. It just doesn't make sense.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Arjan van » Tue, 19 Dec 2006 20:50:09


if you want truely really smooth writes you'll have to work for it,
since "bumpy" writes tend to be better for performance so naturally the
kernel will favor those.

to get smooth writes you'll need to do a threaded setup where you do an
msync/fdatasync/sync_file_range on a frequent-but-regular interval from
a thread. Be aware that this is quite likely to give you lower maximum
performance than the batching behavior though.

--
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.yqcomputer.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/

 
 
 

Linux disk performance.

Post by Manish Reg » Tue, 19 Dec 2006 21:50:14


Thanks...

But isn't O_DIRECT supposed to bypass buffering in Kernel?
Doesn't it directly write to disk?
I tried to put fdatasync() at regular intervals but there was no
visible effect.

--
---------------------------------------------------------------
regards
Manish Regmi

---------------------------------------------------------------
UNIX without a C Compiler is like eating Spaghetti with your mouth
sewn shut. It just doesn't make sense.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Nick Piggi » Tue, 19 Dec 2006 22:10:10


I don't know exactly how to interpret the numbers you gave, but
they look like they might be a (HZ quantised) delay coming from
block layer plugging.

O_DIRECT bypasses caching, but not (all) buffering.

Not sure whether the block layer can handle an unplug_delay set
to 0, but that might be something to try (see block/ll_rw_blk.c).

--
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://www.yqcomputer.com/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Erik Mou » Tue, 19 Dec 2006 22:20:10


It is.


Yes, but it still uses an IO scheduler.


In your first message you mentioned you were using an ancient 2.6.10
kernel. That kernel uses the anticipatory IO scheduler. Update to the
latest stable kernel (2.6.19.1 at time of writing) and it will default
to the CFQ scheduler which has a smoother writeout, plus you can give
your process a different IO scheduling class and level (see
Documentation/block/ioprio.txt).


Erik
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Manish Reg » Wed, 20 Dec 2006 15:30:09


<...snip...>

Ok. but i also tried with noop to turnoff disk scheduling effects.
There was still timing differences. Usually i get 3100 microseconds
but upto 20000 microseconds at certain intervals. I am just using
gettimeofday between two writes to read the timing.




Thanks... i will try with CFQ.



Nick Piggin:

Sorry i didn understand what you mean.

To minimise scheduling effects i tried giving it maximum priority.


--
---------------------------------------------------------------
regards
Manish Regmi

---------------------------------------------------------------
UNIX without a C Compiler is like eating Spaghetti with your mouth
sewn shut. It just doesn't make sense.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Nick Piggi » Wed, 20 Dec 2006 15:50:08

This is a multi-part message in MIME format.
Send instant messages to your online friends http://www.yqcomputer.com/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/



When you submit a request to an empty block device queue, it can
get "plugged" for a number of timer ticks before any IO is actually
started. This is done for efficiency reasons and is independent of
the IO scheduler used.

Use the noop IO scheduler, as well as the attached patch, and let's
see what your numbers look like.

Thanks,
Nick

--
SUSE Labs, Novell Inc.
 
 
 

Linux disk performance.

Post by Arjan van » Wed, 20 Dec 2006 21:20:10


however the O_DIRECT codepath unplugs the queues always immediately..



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Manish Reg » Thu, 21 Dec 2006 20:20:08


Thanks for the information..


Unfortunately i got the same results even after applying your patch. I
also tried putting
q->unplug_delay = 1;
But it did not work. The result was similar.

--
---------------------------------------------------------------
regards
Manish Regmi

---------------------------------------------------------------
UNIX without a C Compiler is like eating Spaghetti with your mouth
sewn shut. It just doesn't make sense.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Bill David » Fri, 22 Dec 2006 07:30:10


Just to say it another way.
That's correct. But it doesn't put your write at the head of any queue,
it just doesn't buffer it for you.

Also correct, when it's your turn to write to disk...

Quite honestly, the main place I have found O_DIRECT useful is in
keeping programs doing large i/o quantities from blowing the buffers and
making the other applications run like crap. If you application is
running alone, unless you are very short of CPU or memory avoiding the
copy to an o/s buffer will be down in the measurement noise.

I had a news (usenet) server which normally did 120 art/sec (~480 tps),
which dropped to about 50 tps when doing large file copies even at low
priority. By using O_DIRECT the impact essentially vanished, at the cost
of the copy running about 10-15% slower. Changing various programs to
use O_DIRECT only helped when really large blocks of data were involved,
and only when i/o clould be done in a way to satisfy the alignment and
size requirements of O_DIRECT.

If you upgrade to a newer kernel you can try other i/o scheduler
options, default cfq or even deadline might be helpful.

--
bill davidsen < XXXX@XXXXX.COM >
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Manish Reg » Fri, 22 Dec 2006 15:10:07


But the only process accessing that disk is my application.


Yes... my application does large amount of I/O. It actually writes
video data received from ethernet(IP camera) to the disk using 128 K
chunks.


I tried all disk schedulers but all had timing bumps. :(



--
---------------------------------------------------------------
regards
Manish Regmi

---------------------------------------------------------------
UNIX without a C Compiler is like eating Spaghetti with your mouth
sewn shut. It just doesn't make sense.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Daniel Che » Fri, 22 Dec 2006 16:20:09


[...]

Did you try to disable the on disk write cache?

man hdparm(8)

-W Disable/enable the IDE drives write-caching
feature


--

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Erik Mou » Fri, 22 Dec 2006 22:30:17


Bursty video traffic is really an application that could take advantage
from the kernel buffering. Unless you want to reinvent the wheel and do
the buffering yourself (it is possible though, I've done it on IRIX).

BTW, why are you so keen on smooth-at-the-microlevel writeout? With
real time video applications it's only important not to drop frames.
How fast those frames will go to the disk isn't really an issue, as
long as you don't overflow the intermediate buffer.


Erik

--
They're all fools. Don't worry. Darwin may be slow, but he'll
eventually get them. -- Matthew Lammers in alt.sysadmin.recovery
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Bhanu Kaly » Sat, 23 Dec 2006 09:20:10


I am assuming that your program is not seeking inbetween writes.

Try disabling the Disk Cache, now-a-days some disks can have as much
as 8MB write cache. so the disk might be buffering as much as it can,
and trying to write only when it can no longer buffer. Since you have
an app which continously write copious amounts of data, in order,
disabling write cache might make some sense.

Bhanu


--
There is only one success - to be able to spend your life in your own way.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/
 
 
 

Linux disk performance.

Post by Bhanu Kaly » Sat, 23 Dec 2006 14:40:05


Performance degradation is expected. But the point is - did the
anomaly, that you have pointed out, go away? Because if it did, then
it is the disk cache which is causing the issue, and you will have to
live with it. Else you will have to look elsewhere.



--
There is only one success - to be able to spend your life in your own way.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to XXXX@XXXXX.COM
More majordomo info at http://www.yqcomputer.com/
Please read the FAQ at http://www.yqcomputer.com/