Data intake and cleaning
Data arrives from suppliers, legacy systems, web forms and colleagues’ spreadsheets. It is rarely consistent. Before anything useful can happen, someone has to decide what valid looks like and apply that decision every time.
Your decision: You decide what happens to rejected records: fixed at source, corrected by rule, or excluded.
Method
- Profile sample files: columns, types, null rates, duplicates, encodings
- Agree written validation rules with your data owner
- Implement rules as a repeatable cleaning step
- Produce a rejection report showing each failed record and the rule it broke