Data Cleansing

[eric102269]

Data Cleansing/Scrubbing

I have just been given the task of cleansing an MS SQL Server 7 data warehouse.

Question: Are there any good books, white papers, docs, etc that can help me figure out where to start? Other resources?

I have found info on 'why' you should, but not on 'how'.


Thanks,
Eric

 

[jnicho02]

A couple of resources....

http://www.dmreview/com/whitepaper/dqa.pdf

http://www.dmreview.com/editorial/dmdirect/dmdirect_article.cfm?EdID=2342&issue=062300&record=2

A couple of tips....

* Run reports showing values and counts for each field.(e.g. Sex: M=10,000 F=9,945 U=1,000 Q=20 N=35)
* Identify the set of allowable values. (e.g. Sex = M,F,U)
* Compare the two and give the field a data quality score. ( 55/21,000 * 100/1 %)
* Rank the fields by importance.
* Identify any important fields that have bad quality scores.
* Track the errors back to the original source and have them fix their system.
* Identify correctable errors (e.g. N is probably a misskey of M)
* Look for linked fields (e.g. number of credit cards and age...we had a few children with high incomes, cars and credit cards...they may be a bit olde than the db thinks!)
My Home -->

http://www.geocities.com/jnicho02/

 

[MattDowd]

Vality Technology has been re-engineering data for over 13 years and INTEGRITY is ranked as the industry leader in data quality by The Gartner Group.

Vality's online media kit has white papers, customer profiles and industry specific information. I would point you in the direction of the White paper "The 5 Data Contaminants You will Encounter" This will not tell you how but will give you a roadmap as to what to look out for.

Matthew Dowd
Vality Technology
mdowd@vality.com

http://www.vality.com