ETL on SAP BEX report - missing fields

[AndrewZX]

I've done basic table loads with SSIS and had success but now have a complicated report to load.
The format is similar to this:

TransactionID UserID  firstname lastname itemID itemdescription itemcost
100001        1001    Joe       Jones     abc    black shoes    $50.
100002        1002    Jane      Johnson   bcd    red shoes      $60.
                                          cde    green shoes    $61.
                                          def    blue shoes     $60.
100003        1003    Jay       Jackson   efg    orange shoes   $50.

The problem is to impute or fill in fields missing from the report, to fill in all fields
during the ETL. This would be a simple programming task in Perl or Python, but how
do you approach this in SSIS?

Any pointers or recommendations greatly appreciated!

 

[SQLScholar]

Where does this missing data come from?

----------------------------------------

http://www.fountain.me.uk

Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind - Bernard Baruch

Computer Science is no more about computers than astronomy is about telescopes - EW Dijkstra
----------------------------------------

 

[fredericofonseca]

I'm assuming the missing data is from the previous line on the report that has those filled.

Pure SSIS - don't advise it if volumes are big.

Load the file to a recordset object

On a dataflow add script as source, and on this script you read the recordset above in a loop, storing the values of each row and sending the values of the previous row to the output collection if they are empty

mix of SSIS/T-SQL.

Load the file onto a staging table adding a row number while doing it (either identity column or a row generator on SSIS)
Then using t-sql perform a update of the rows with the required

Regards

Frederico Fonseca
SysSoft Integrated Ltd

http://www.syssoft-int.com

FAQ219-2884
FAQ181-2886

 

[AndrewZX]

Frederico,

Thanks, yes missing fields come from a previous record. OK, I will look at recordset object + T-SQL.