Post reply

to resist the duplicates in the ssis packages

  • January 3, 2011 at 8:19 pm

    #388545

    i have a SSIS package that extracts data from three different sources and then then there is a unionall transformation applied on the columns from where the three data sources are coming from after that theres a slowely changing dimension applied

    now the problem is , every day when the package is updated , the duplicates are being appended to the

    destination table

    can any body help me with this please

    thanks

    regards

    greg

  • January 4, 2011 at 12:02 am

    This was removed by the editor as SPAM

  • January 4, 2011 at 12:32 am

    #1268775

    Be aware that the Sort component is a blocking component in SSIS and that it can drastically affect performance if you transfer a lot of rows. If this becomes an issue, you can try to clean the duplicates in the sources themselves or to drop the data in a staging table and do the clean-up there.

    On a side-note: if you are using the built-in Slowly Changing Dimension component, this one has performance issues as well, as a OLE DB command is issued for every row. If you have a lot of updates, performance will go down. Since you posted this question in a 2008 forum, I assume you're working with SSIS 2008. Performance wise I would recommand the MERGE (T-SQL) operator.

    Need an answer? No, you need a question
    My blog at https://sqlkover.com.
    MCSE Business Intelligence - Microsoft Data Platform MVP

  • January 4, 2011 at 1:15 am

    This was removed by the editor as SPAM

Viewing 4 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply