Skip to main content

Posts

Showing posts from July 6, 2008

Approach to Start the Data Profiling

So you got the large / medium enterprise legacy systems or may be a ERP system in your organization and decided to profile the data you have. The first step is to decide what all data you are going to work with. Normally spend analysis has to be done on your procurement, material data. So material master, vendor master, MRV, PO, part master are the ideal candidate to start. While extracting the data you need to very careful as if you miss some critical fields or required fields for analysis - which you will come to know at very later stage and then everything starts from the scratch. E in the ETL process is a large subject in itself to talk. So we will not get into that right now. Here I will assume that, you are there. You identified the fields correctly and then went ahead with the generating those text files - with proper delimiters. :-). And you are ready for running profiling. Here we have two options. A semi automatic database driven approach and fully automatic tool dependant ap

Why should I Profile my Data?

You have got an enterprise system for your organization since long time. Do you know how much data quality issues you may have in your data? What does that mean ? 1. You may have different date formats - like mmddyyyy, ddmmyyyy, dd-mon-yyyy and so on. 2.You may have unit of measurement (UOM) inconsistencies issues. - Somebody putting the data in inches, one in cms and one in feet. 3. Have you considered the currency conversion and exchange rate issues. 4. And how about the material code ? One coding it as ABC123, other as ABC-123 and another as ABC 123. All are same, but when you search you dont find the material ABC123 and you order another 1000 quanitities, when same material named as ABC 123 is there in your warehouse. 5. Do you know the vendor ABC, AB-C, AB Corp and AB Ltd are same under one group? All these issues are eating in your money. Directly or indirectly. Just under your nose, because they are the issues, you are going through everyday, but just ignoring it due to invisibi