200MB and bigger flat files

200MB and bigger flat files

Post by U2luaXNhIE » Thu, 21 Dec 2006 00:56:01


Hi, I am doing some proof of concept tests to find best solution to process
flat files that are very large (from BizTalk perspective): 200MB and bigger.
One solution is to split file into smaller chunks in the receive pipeline by
inserting carriage return (before its converted to XML). Right now I am doing
such test with the 100MB flat file that will be split into 29 pieces each
with the size around 3.5MB 9I am gradualy increasing file size). I am doing
tests with various batch sizes and have found that bigger batch size
generally cuts processing time. I am doing splitting only and routing
outgoing XML into separate send folder. As I have expected processing is slow
for such large input file but I wonder why is not faster because I have
noticed that CPU is almost not used after file is taken and split into
chunks. Memory also stays flat all the time (1.4GB from the total of 2GB, 4
logical processors). It looks processing is IO bound and not CPU or memory
bound. I wonder if processing could be shortened by adding more messagebox
databases or maybe tuning some parameters. It does not look even that SQL
Server is bottleneck. More looks like the problem is how BizTalk is using SQL
Server. Any suggestion would be welcome. Thanks.

Sinisa Catic
 
 
 

200MB and bigger flat files

Post by Paul Somer » Tue, 09 Jan 2007 18:38:27

Are you using BizTalk 2006?

I have been able to process 200 mb flat files through BizTalk 2006
successfully, it did take a bit of time I recall, I did need to ensure that
I had sufficient memory on the BizTalk server, even though with 2006 it
caches things to disk.