actually, we want to see as less errors as we can (more automation).
in the other words, we try to insert as much as we can. If the lookup
key is not found then insert it too.
the only thing I worry about is "time," it may take a long time to
distint each column and compare and then insert only un matching ones.
The flat raw data is like below table (of course way more columns). it
takes long time to distint each column (I have 7 millions records now)
and then check the lookup tables then insert the missing ones. this
may take long time.
item,custmer, address, vender
TV, custmer1, address1, vender1
DVD, custmer2, address1, vender1
TV, custmer1, address2, vender2