Duplicate Data Question

[GraffixNyc]

Hi, We have a SSIS package that goes out to the web and downloads a CSV file
and then imports it into SqlServer. Now the file we download daily is a
file that gets appended. so when we download the file today we get all of
todays data plus the same data we already imported. The data we are
importing has a unique ID. What steps(or tasks) do we need so that the
package will NOT import the duplicate data but only take the new data. I
know in SQL this would be an insert where the unique ID is NOT in the table
already. But how do we go about doing this is SSIS? I would assume we
would need some kind of task before the data importing task. we are very
new to SSIS so any help would be greatly apperciated

 

[manmaria]

First load this datainto a temp table and then a sql task to identify the dups and keep the latest data then load into your target.

HTH

 

[dhulbert]

Loa dthe data into a temp - staging table as manmaria said but then either insert into your table where the unique id is not in the table already
or
delete the data from the existing table then insert all the rows from your stage table,

All depends if the existing data is likely to have changed.

I love deadlines. I like the whooshing sound they make as they fly by
Douglas Adams
(1952-2001)