What's the best way of getting this to work?

  • Hi All,

    I'm working on an SP that will be run as part of an SSIS package and I'm not sure of the best way of going about getting it working. and I hope I can get some more help up here.

    The basic idea is we'll have a table that holds queries that the user runs for data validation (After ETL) and queries that we run as part of of our data validations (After ETL) I want to read the queries table and using the query text dynamically execute the queries and then store the results in a separate results table.

    I know clear as mud. Here's the table structures and the script of the SP:

    The SP has some sections commented out as those were alternate paths I was trying to go down that didn't work either.


    CREATE TABLE [val].[AuditQueries](
        [AuditQueryID] [INT] IDENTITY(1,1) NOT NULL,
        [AuditItemID] [INT] NOT NULL,
        [DbConnectionName] [VARCHAR](25) NOT NULL,
        [AuditQueryText] [VARCHAR](2000) NULL,
        [AuditQueryNotes] [VARCHAR](200) NULL,
        [RecordAddedDate] [DATETIME2](0) NOT NULL DEFAULT (SYSDATETIME()),
    CONSTRAINT [pk_AuditQueries] PRIMARY KEY CLUSTERED
    (
        [AuditQueryID] DESC
    )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
    ) ON [PRIMARY]


    CREATE TABLE [val].[AuditRunResults](
        [AuditRunResultId] [INT] IDENTITY(1,1) NOT NULL,
        [AuditItemId] [INT] NOT NULL,
        [DbConnectionName] [VARCHAR](25) NOT NULL,
        [AuditRunId] [INT] NOT NULL,
        [AuditQueryText] [NVARCHAR](2000) NULL,
        [AuditQueryGroupBy] [VARCHAR](50) NOT NULL,
        [QueryResult] [DECIMAL](29, 0) NOT NULL,
        [RecordAddedDate] [DATETIME2](0) NOT NULL DEFAULT (SYSDATETIME()),
    CONSTRAINT [PK_AuditRunResults] PRIMARY KEY CLUSTERED
    (
        [AuditRunResultId] DESC
    )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
    ) ON [PRIMARY]


    DECLARE @SQL AS NVARCHAR(MAX),
          @dbConnection AS NVARCHAR(100),
            @AuditItemID AS INT,
            @AuditQueryID AS INT,
            @GroupBy AS VARCHAR(150),
            @Result AS VARCHAR(MAX),
            @RecordAddedDate AS DATETIME,
            @ConnStr nVarchar(MAX);

    IF OBJECT_ID('tempdb..#ResultSet') IS NOT NULL
         /*Then it exists*/
    DROP TABLE #Resultset

    CREATE TABLE #Resultset(
        [AuditItemId] [INT] ,
        [DbConnectionName] [VARCHAR](25) ,
        [AuditRunId] [INT] ,
        [AuditQueryText] [NVARCHAR](2000) ,
        [AuditQueryGroupBy] [VARCHAR](50) ,
        [QueryResult] NVARCHAR(MAX), --[DECIMAL](29, 0) ,
        [RecordAddedDate] [DATETIME2](0)
        )

    Select
    @AuditItemID = AuditItemID
    ,@dbConnection = AuditQueries.DbConnectionName
    ,@AuditQueryID = AuditQueryID
    ,@SQL = AuditQueryText
    ,@GroupBy = 0 -- GROUP By
    ,@Result = ''
    --,@ConnStr = DbConnectionString
    ,@RecordAddedDate = SYSDATETIME() -- RecordAddedDate - datetime2(0)

    FROM [Dwbi_Metadata].[val].[AuditQueries]
    INNER JOIN val.AuditDbConnections ON AuditDbConnections.DbConnectionName = AuditQueries.DbConnectionName

    INSERT INTO #Resultset
    Select
    @AuditItemID
    ,@dbConnection
    ,@AuditQueryID
    ,@SQL
    ,@GroupBy
    ,@Result
    --,@ConnStr = DbC
    ,@RecordAddedDate

    SELECT * FROM #Resultset

    SELECT * into #T1 EXECUTE sp_executesql @SQL

    EXECUTE sp_executesql @SQL, N'@Result Varchar(max) output', @Result output

    SELECT @Result

    The Problem is I can't seem to get the results from the EXECUTE sp_executesql @SQL, N'@Result Varchar(max) output', @Result output into either a variable or a temp table to insert into the Runresults table. The Execute sp_executesql does not return any column headers to work with. 

    Here's an example of the query I'm executing and trying to get the results from. And yes the query does work.

    SELECT 'warehouse' AS AuditDataSource,2 AS AuditItemID,1 AS AuditRunID,Fill_Date_Month_SurrogateKey AS QueryGroupBy,SUM(Claim_Paid_Amount)AS QueryResult
    FROM warehouse.vw_Fact_Pharmacy_Claim
    WHERE Fill_Date>='20120101'
    GROUP BY Fill_Date_Month_SurrogateKey;

    Any help is appreciated.

  • craig.bobchin - Tuesday, January 31, 2017 9:49 AM

    Hi All,

    I'm working on an SP that will be run as part of an SSIS package and I'm not sure of the best way of going about getting it working. and I hope I can get some more help up here.

    The basic idea is we'll have a table that holds queries that the user runs for data validation (After ETL) and queries that we run as part of of our data validations (After ETL) I want to read the queries table and using the query text dynamically execute the queries and then store the results in a separate results table.

    I know clear as mud. Here's the table structures and the script of the SP:

    The SP has some sections commented out as those were alternate paths I was trying to go down that didn't work either.


    CREATE TABLE [val].[AuditQueries](
        [AuditQueryID] [INT] IDENTITY(1,1) NOT NULL,
        [AuditItemID] [INT] NOT NULL,
        [DbConnectionName] [VARCHAR](25) NOT NULL,
        [AuditQueryText] [VARCHAR](2000) NULL,
        [AuditQueryNotes] [VARCHAR](200) NULL,
        [RecordAddedDate] [DATETIME2](0) NOT NULL DEFAULT (SYSDATETIME()),
    CONSTRAINT [pk_AuditQueries] PRIMARY KEY CLUSTERED
    (
        [AuditQueryID] DESC
    )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
    ) ON [PRIMARY]


    CREATE TABLE [val].[AuditRunResults](
        [AuditRunResultId] [INT] IDENTITY(1,1) NOT NULL,
        [AuditItemId] [INT] NOT NULL,
        [DbConnectionName] [VARCHAR](25) NOT NULL,
        [AuditRunId] [INT] NOT NULL,
        [AuditQueryText] [NVARCHAR](2000) NULL,
        [AuditQueryGroupBy] [VARCHAR](50) NOT NULL,
        [QueryResult] [DECIMAL](29, 0) NOT NULL,
        [RecordAddedDate] [DATETIME2](0) NOT NULL DEFAULT (SYSDATETIME()),
    CONSTRAINT [PK_AuditRunResults] PRIMARY KEY CLUSTERED
    (
        [AuditRunResultId] DESC
    )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
    ) ON [PRIMARY]


    DECLARE @SQL AS NVARCHAR(MAX),
          @dbConnection AS NVARCHAR(100),
            @AuditItemID AS INT,
            @AuditQueryID AS INT,
            @GroupBy AS VARCHAR(150),
            @Result AS VARCHAR(MAX),
            @RecordAddedDate AS DATETIME,
            @ConnStr nVarchar(MAX);

    IF OBJECT_ID('tempdb..#ResultSet') IS NOT NULL
         /*Then it exists*/
    DROP TABLE #Resultset

    CREATE TABLE #Resultset(
        [AuditItemId] [INT] ,
        [DbConnectionName] [VARCHAR](25) ,
        [AuditRunId] [INT] ,
        [AuditQueryText] [NVARCHAR](2000) ,
        [AuditQueryGroupBy] [VARCHAR](50) ,
        [QueryResult] NVARCHAR(MAX), --[DECIMAL](29, 0) ,
        [RecordAddedDate] [DATETIME2](0)
        )

    Select
    @AuditItemID = AuditItemID
    ,@dbConnection = AuditQueries.DbConnectionName
    ,@AuditQueryID = AuditQueryID
    ,@SQL = AuditQueryText
    ,@GroupBy = 0 -- GROUP By
    ,@Result = ''
    --,@ConnStr = DbConnectionString
    ,@RecordAddedDate = SYSDATETIME() -- RecordAddedDate - datetime2(0)

    FROM [Dwbi_Metadata].[val].[AuditQueries]
    INNER JOIN val.AuditDbConnections ON AuditDbConnections.DbConnectionName = AuditQueries.DbConnectionName

    INSERT INTO #Resultset
    Select
    @AuditItemID
    ,@dbConnection
    ,@AuditQueryID
    ,@SQL
    ,@GroupBy
    ,@Result
    --,@ConnStr = DbC
    ,@RecordAddedDate

    SELECT * FROM #Resultset

    SELECT * into #T1 EXECUTE sp_executesql @SQL

    EXECUTE sp_executesql @SQL, N'@Result Varchar(max) output', @Result output

    SELECT @Result

    The Problem is I can't seem to get the results from the EXECUTE sp_executesql @SQL, N'@Result Varchar(max) output', @Result output into either a variable or a temp table to insert into the Runresults table. The Execute sp_executesql does not return any column headers to work with. 

    Here's an example of the query I'm executing and trying to get the results from. And yes the query does work.

    SELECT 'warehouse' AS AuditDataSource,2 AS AuditItemID,1 AS AuditRunID,Fill_Date_Month_SurrogateKey AS QueryGroupBy,SUM(Claim_Paid_Amount)AS QueryResult
    FROM warehouse.vw_Fact_Pharmacy_Claim
    WHERE Fill_Date>='20120101'
    GROUP BY Fill_Date_Month_SurrogateKey;

    Any help is appreciated.

    You need to do INSERT INTO, not SELECT INTO.  The latter is for when the table does not already exist (and doesn't work with stored procedure results sets anyway).

    John

  • Assuming your queries always return results in the same format (ie, matching Column name, Column data type, Column order, number of columns), there's a way to do this.

    Read your queries into an SSIS Object variable (oQueries, say)
    Loop round a foreach loop (for each row in oQueries)
    Execute a data flow task which uses the 'current' query from oQueries as its data source and has the Results table as its source

    Job done. No separate proc needed.

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • John Mitchell-245523 - Tuesday, January 31, 2017 10:00 AM

    craig.bobchin - Tuesday, January 31, 2017 9:49 AM

    Hi All,

    I'm working on an SP that will be run as part of an SSIS package and I'm not sure of the best way of going about getting it working. and I hope I can get some more help up here.

    The basic idea is we'll have a table that holds queries that the user runs for data validation (After ETL) and queries that we run as part of of our data validations (After ETL) I want to read the queries table and using the query text dynamically execute the queries and then store the results in a separate results table.

    I know clear as mud. Here's the table structures and the script of the SP:

    The SP has some sections commented out as those were alternate paths I was trying to go down that didn't work either.


    CREATE TABLE [val].[AuditQueries](
        [AuditQueryID] [INT] IDENTITY(1,1) NOT NULL,
        [AuditItemID] [INT] NOT NULL,
        [DbConnectionName] [VARCHAR](25) NOT NULL,
        [AuditQueryText] [VARCHAR](2000) NULL,
        [AuditQueryNotes] [VARCHAR](200) NULL,
        [RecordAddedDate] [DATETIME2](0) NOT NULL DEFAULT (SYSDATETIME()),
    CONSTRAINT [pk_AuditQueries] PRIMARY KEY CLUSTERED
    (
        [AuditQueryID] DESC
    )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
    ) ON [PRIMARY]


    CREATE TABLE [val].[AuditRunResults](
        [AuditRunResultId] [INT] IDENTITY(1,1) NOT NULL,
        [AuditItemId] [INT] NOT NULL,
        [DbConnectionName] [VARCHAR](25) NOT NULL,
        [AuditRunId] [INT] NOT NULL,
        [AuditQueryText] [NVARCHAR](2000) NULL,
        [AuditQueryGroupBy] [VARCHAR](50) NOT NULL,
        [QueryResult] [DECIMAL](29, 0) NOT NULL,
        [RecordAddedDate] [DATETIME2](0) NOT NULL DEFAULT (SYSDATETIME()),
    CONSTRAINT [PK_AuditRunResults] PRIMARY KEY CLUSTERED
    (
        [AuditRunResultId] DESC
    )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
    ) ON [PRIMARY]


    DECLARE @SQL AS NVARCHAR(MAX),
          @dbConnection AS NVARCHAR(100),
            @AuditItemID AS INT,
            @AuditQueryID AS INT,
            @GroupBy AS VARCHAR(150),
            @Result AS VARCHAR(MAX),
            @RecordAddedDate AS DATETIME,
            @ConnStr nVarchar(MAX);

    IF OBJECT_ID('tempdb..#ResultSet') IS NOT NULL
         /*Then it exists*/
    DROP TABLE #Resultset

    CREATE TABLE #Resultset(
        [AuditItemId] [INT] ,
        [DbConnectionName] [VARCHAR](25) ,
        [AuditRunId] [INT] ,
        [AuditQueryText] [NVARCHAR](2000) ,
        [AuditQueryGroupBy] [VARCHAR](50) ,
        [QueryResult] NVARCHAR(MAX), --[DECIMAL](29, 0) ,
        [RecordAddedDate] [DATETIME2](0)
        )

    Select
    @AuditItemID = AuditItemID
    ,@dbConnection = AuditQueries.DbConnectionName
    ,@AuditQueryID = AuditQueryID
    ,@SQL = AuditQueryText
    ,@GroupBy = 0 -- GROUP By
    ,@Result = ''
    --,@ConnStr = DbConnectionString
    ,@RecordAddedDate = SYSDATETIME() -- RecordAddedDate - datetime2(0)

    FROM [Dwbi_Metadata].[val].[AuditQueries]
    INNER JOIN val.AuditDbConnections ON AuditDbConnections.DbConnectionName = AuditQueries.DbConnectionName

    INSERT INTO #Resultset
    Select
    @AuditItemID
    ,@dbConnection
    ,@AuditQueryID
    ,@SQL
    ,@GroupBy
    ,@Result
    --,@ConnStr = DbC
    ,@RecordAddedDate

    SELECT * FROM #Resultset

    SELECT * into #T1 EXECUTE sp_executesql @SQL

    EXECUTE sp_executesql @SQL, N'@Result Varchar(max) output', @Result output

    SELECT @Result

    The Problem is I can't seem to get the results from the EXECUTE sp_executesql @SQL, N'@Result Varchar(max) output', @Result output into either a variable or a temp table to insert into the Runresults table. The Execute sp_executesql does not return any column headers to work with. 

    Here's an example of the query I'm executing and trying to get the results from. And yes the query does work.

    SELECT 'warehouse' AS AuditDataSource,2 AS AuditItemID,1 AS AuditRunID,Fill_Date_Month_SurrogateKey AS QueryGroupBy,SUM(Claim_Paid_Amount)AS QueryResult
    FROM warehouse.vw_Fact_Pharmacy_Claim
    WHERE Fill_Date>='20120101'
    GROUP BY Fill_Date_Month_SurrogateKey;

    Any help is appreciated.

    You need to do INSERT INTO, not SELECT INTO.  The latter is for when the table does not already exist (and doesn't work with stored procedure results sets anyway).

    John

    How do I do the Insert INTO when there's no column names in the result set?

  • Phil Parkin - Tuesday, January 31, 2017 10:03 AM

    Assuming your queries always return results in the same format (ie, matching Column name, Column data type, Column order, number of columns), there's a way to do this.

    Read your queries into an SSIS Object variable (oQueries, say)
    Loop round a foreach loop (for each row in oQueries)
    Execute a data flow task which uses the 'current' query from oQueries as its data source and has the Results table as its source

    Job done. No separate proc needed.

    I'll have to give this a try. Thanks

  • craig.bobchin - Tuesday, January 31, 2017 10:21 AM

    Phil Parkin - Tuesday, January 31, 2017 10:03 AM

    Assuming your queries always return results in the same format (ie, matching Column name, Column data type, Column order, number of columns), there's a way to do this.

    Read your queries into an SSIS Object variable (oQueries, say)
    Loop round a foreach loop (for each row in oQueries)
    Execute a data flow task which uses the 'current' query from oQueries as its data source and has the Results table as its source

    Job done. No separate proc needed.

    I'll have to give this a try. Thanks

    You probably noticed, but I just noticed a typo in my text. It should have stated '... Results table as its destination'.

    Post back if you have any follow-up questions. I've done this type of thing before & should be able to get you over the line.

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • Phil Parkin - Tuesday, January 31, 2017 10:43 AM

    craig.bobchin - Tuesday, January 31, 2017 10:21 AM

    Phil Parkin - Tuesday, January 31, 2017 10:03 AM

    Assuming your queries always return results in the same format (ie, matching Column name, Column data type, Column order, number of columns), there's a way to do this.

    Read your queries into an SSIS Object variable (oQueries, say)
    Loop round a foreach loop (for each row in oQueries)
    Execute a data flow task which uses the 'current' query from oQueries as its data source and has the Results table as its source

    Job done. No separate proc needed.

    I'll have to give this a try. Thanks

    You probably noticed, but I just noticed a typo in my text. It should have stated '... Results table as its destination'.

    Post back if you have any follow-up questions. I've done this type of thing before & should be able to get you over the line.

    Yeah, I saw that and got the meaning.

  • Phil Parkin - Tuesday, January 31, 2017 10:43 AM

    craig.bobchin - Tuesday, January 31, 2017 10:21 AM

    Phil Parkin - Tuesday, January 31, 2017 10:03 AM

    Assuming your queries always return results in the same format (ie, matching Column name, Column data type, Column order, number of columns), there's a way to do this.

    Read your queries into an SSIS Object variable (oQueries, say)
    Loop round a foreach loop (for each row in oQueries)
    Execute a data flow task which uses the 'current' query from oQueries as its data source and has the Results table as its source

    Job done. No separate proc needed.

    I'll have to give this a try. Thanks

    You probably noticed, but I just noticed a typo in my text. It should have stated '... Results table as its destination'.

    Post back if you have any follow-up questions. I've done this type of thing before & should be able to get you over the line.

    Okay, so here's what I've gotten so far, I have the SSIS package and the control flow has an execute SQL task that runs this query:


    SELECT [AuditQueryID]
      ,[AuditItemID]
      ,[DbConnectionName]
      ,[AuditQueryText]
      ,[AuditQueryNotes]
      ,[RecordAddedDate]
    FROM [val].[AuditQueries]

    The result set is loaded to the oQueries Object variable.
    What I can't figure out how to do is set the oVariable as the data source for the data flow. I have it set as the Foreach From Variable Enumerator of the Foreach Loop container. 

    How do I get the Querytext as the data source for the data flow?

  • craig.bobchin - Tuesday, January 31, 2017 12:22 PM

    Okay, so here's what I've gotten so far, I have the SSIS package and the control flow has an execute SQL task that runs this query:


    SELECT [AuditQueryID]
      ,[AuditItemID]
      ,[DbConnectionName]
      ,[AuditQueryText]
      ,[AuditQueryNotes]
      ,[RecordAddedDate]
    FROM [val].[AuditQueries]

    The result set is loaded to the oQueries Object variable.
    What I can't figure out how to do is set the oVariable as the data source for the data flow. I have it set as the Foreach From Variable Enumerator of the Foreach Loop container. 

    How do I get the Querytext as the data source for the data flow?

    You now need to 'shred' the object variable. What does that mean? It means you need to break it apart using a foreach loop such that
    a) Every loop iteration corresponds to a single row in the object variable
    b) The values in the columns of the result set are mapped to scalar variables in SSIS.
    This page describes how to do that.
    Once you have got the shredding working, we can move on to how you dynamically configure your source.

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

  • Phil Parkin - Tuesday, January 31, 2017 1:20 PM

    craig.bobchin - Tuesday, January 31, 2017 12:22 PM

    Okay, so here's what I've gotten so far, I have the SSIS package and the control flow has an execute SQL task that runs this query:


    SELECT [AuditQueryID]
      ,[AuditItemID]
      ,[DbConnectionName]
      ,[AuditQueryText]
      ,[AuditQueryNotes]
      ,[RecordAddedDate]
    FROM [val].[AuditQueries]

    The result set is loaded to the oQueries Object variable.
    What I can't figure out how to do is set the oVariable as the data source for the data flow. I have it set as the Foreach From Variable Enumerator of the Foreach Loop container. 

    How do I get the Querytext as the data source for the data flow?

    You now need to 'shred' the object variable. What does that mean? It means you need to break it apart using a foreach loop such that
    a) Every loop iteration corresponds to a single row in the object variable
    b) The values in the columns of the result set are mapped to scalar variables in SSIS.
    This page describes how to do that.
    Once you have got the shredding working, we can move on to how you dynamically configure your source.

    Got it shredded in the data flow task into variables. So I have the query text in one variable and the actual data connection string from a second table in another variable. 

    What I need to get done now is execute the query text and return the results of that to a table (May be one row or multiple rows returned per query executed). I presume I do that in the Foreach loop container? 

    How do I dynamically set a connection manager's connection string since this is supposed to run automated and I don't want to have to create a separate connmgr for each query. That could get into the hundreds.

  • craig.bobchin - Tuesday, January 31, 2017 10:20 AM

    How do I do the Insert INTO when there's no column names in the result set?

    The same way you would if there were column names.  Of course, you need to make sure that the destination table has the correct number of columns in the correct order with the correct data types.

    DECLARE @john-2 table (IntCol int)
    INSERT INTO @john-2
    SELECT 1 -- run this line on its own to see there's no column header
    SELECT IntCol FROM @john-2

    John

  • craig.bobchin - Tuesday, January 31, 2017 5:32 PM

    Got it shredded in the data flow task into variables. So I have the query text in one variable and the actual data connection string from a second table in another variable. 

    What I need to get done now is execute the query text and return the results of that to a table (May be one row or multiple rows returned per query executed). I presume I do that in the Foreach loop container? 

    How do I dynamically set a connection manager's connection string since this is supposed to run automated and I don't want to have to create a separate connmgr for each query. That could get into the hundreds.

    OK, I presume that you are familiar with data flows. Create one inside your FEL with a source joined to a destination (your results table). Set the data access mode for your source to be 'SQL command from variable' and set Variable Name to be the variable containing the dynamic SQL query text.

    Set the SQL query text variable up so that its default value is a query which works, using a connection manager which also works. Call the connection manager something generic, like 'Source'. Doing this means that you can create your mappings on the destination component.

    Now, click on the source connection manager and find the Expressions property. Add a ConnectionString expression and map your variable to it (assuming that it contains a connection string in the right format, that is).

    That's a condensed version of what's required. Hope it makes some sense, good luck.

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.

Viewing 12 posts - 1 through 11 (of 11 total)

You must be logged in to reply to this topic. Login to reply