CSV Import

CSV Import

Post by Kerr » Sat, 23 Jul 2005 18:30:47



Hi all,
Using VB.NET i have a collection of 12 fairly large (largest file is
100MB) CSV files. I have a SQL Server data structure that mimics the
structure of these csv files. I need a method in code of getting the
data from these CSV files into my SQL Server.

Slight addition to this problem is that these files are not using pure
comma delimiters. They have a custom delimiter so I need code in which I
can specify a delimiter.

Any help on this would be appreciated.

Cheers

Kerr


*** Sent via Developersdex http://www.yqcomputer.com/ ***
 
 
 

CSV Import

Post by shrio » Sun, 24 Jul 2005 02:35:24

I need some more info. The quickest way to get the data into SQL server
is using a DTS package, which you can create ahead of time, but then
execute from VB.Net code. You can also use a CSV parser like the one I
sell, http://www.yqcomputer.com/ , to loop over each line in the CSV
files and do whatever with them, import them, convert them, etc. Both
of these methods might have issues with the custom delimiter depending
on how custom it is. Neither methods handle crazy delimiters I've seen
like <!> or !#! type stuff, so in that case, depending on your data,
you may have to run a raw conversion run over the file first before
attempting to import.

 
 
 

CSV Import

Post by Kerr » Sun, 24 Jul 2005 22:20:45

Thanks for the reply. I have tried using a DTS package but the import
process takes too long which is why I wnat to do it in code.
Unfortunately I don't have a budget for the application I am building so
purchasing your parser is not an option.
All I am looking for is a method of getting the data from these CSV
files into SQL Server.

I am hoping that once i import the CSV files into a .net component i can
then merge into SQL through code and improve my performance issues.

any articles!!

*** Sent via Developersdex http://www.yqcomputer.com/ ***
 
 
 

CSV Import

Post by Isaa » Tue, 26 Jul 2005 04:19:47


I would guess that doing it via .net and then into SQL would be even
slower than using DTS - baring in mind that DTS is designed to do
exactly this sort of thing...

Are you doing any complex processing on the DTS task (e.g. ActiveX
scripts) or is it just straight column-copying?

You could always try kicking off the DTS task from your .net app on a
separate thread, this way at least you'd keep your app responsive.