November 6, 2009 at 9:37 am
I have some conerns with my SSIS package regarding the data validation. My package do the data validation according to the business rules to the files that we get from various clients.
My concern is In my package as I am validating data with various rules the record that does not pass the valdation are kicked of out of the file. Now For example if I am validating my Column1 with some rule like It sholud not be NULL, after the validation is completed the record that pass the validation are sent to next task, and the records that does not pass the validation are redirected to error file with a error description column added to it. Now my problem over here is Once my 1st validation is done my second validation task will run and so on, but for each validation the rows that does not pass the validation rule will be redirected to the error folder , so in the next validation step I'll be not able to validate the second validation rule to the columns which are redirected to error folder in the previous validation task.U may be wondering why I need to validate the error rows for every validation rule as It coludnt pass even 1 validation, this is because we are going to manually go through the error file re-evaluate and make cooresponding changes to the error records if it is possible.
So when the data analyst see's the error file, it checks the error message for that record makes the changes to the column in that record which couldm't pass the validation.So one record may be invalid for multiple vaidation rules.So, By using my package it is not possible to check all the records for all the validation rules because as the error records are kicked off for each validation task and remining records are passed to next alidation, I cannot perform all validation tasks on all the records......
I thought of using multi-cast as a solution for this problems so I can send the same records to all validation task and merge them at the end and remove the duplicates. But I don't think this is advicable....So If any one has goe through this kind of situation or any one have any suggestions please help me....Ur help is highly appreciated.......
November 9, 2009 at 9:28 pm
Try Post #805312.
Viewing 2 posts - 1 through 1 (of 1 total)
You must be logged in to reply to this topic. Login to reply