Aggrigation of 200 Mil Data records

  • Hi All,

    We are having a requirement to Aggregate the data and create LY, CY data across 5 Metrics.

    the Volume of Data will be 200 Mil - 250 Mil

    Server Configuration:

    Memory: 32 GB

    Processors: 16

    need help to understand the bench mark configuration needed for the server & any hints on better aggregation methods & LY / CY data

  • sudhirnune (7/7/2015)


    Hi All,

    We are having a requirement to Aggregate the data and create LY, CY data across 5 Metrics.

    the Volume of Data will be 200 Mil - 250 Mil

    Server Configuration:

    Memory: 32 GB

    Processors: 16

    need help to understand the bench mark configuration needed for the server & any hints on better aggregation methods & LY / CY data

    How should we know? Just knowing that there are 200 to 250 million records and that such records encompass 2 years worth of data, and that there are 5 metrics that require aggregation, doesn't really say much. How big is any given row? How, exactly, is each aggregate derived (meaning is just using SUM or AVG or MIN or MAX sufficient) ? Are there any complex calculations required for each metric? We have nowhere near enough info to even start down the road...

  • You should start by reading this article. http://weblogs.sqlteam.com/jeffs/archive/2008/05/13/question-needed-not-answer.aspx

    Then keeping that article in mind, read this article and follow the advice found within. http://spaghettidba.com/2015/04/24/how-to-post-a-t-sql-question-on-a-public-forum/[/url]

    _______________________________________________________________

    Need help? Help us help you.

    Read the article at http://www.sqlservercentral.com/articles/Best+Practices/61537/ for best practices on asking questions.

    Need to split a string? Try Jeff Modens splitter http://www.sqlservercentral.com/articles/Tally+Table/72993/.

    Cross Tabs and Pivots, Part 1 – Converting Rows to Columns - http://www.sqlservercentral.com/articles/T-SQL/63681/
    Cross Tabs and Pivots, Part 2 - Dynamic Cross Tabs - http://www.sqlservercentral.com/articles/Crosstab/65048/
    Understanding and Using APPLY (Part 1) - http://www.sqlservercentral.com/articles/APPLY/69953/
    Understanding and Using APPLY (Part 2) - http://www.sqlservercentral.com/articles/APPLY/69954/

  • Sounds like the kind of project that several of my colleagues do. I know some are between projects at the moment, and we definitely have the skills. Drop me a PM and I'll have my boss contact your boss to discuss rates and specifics.

    Gail Shaw
    Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
    SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

    We walk in the dark places no others will enter
    We stand on the bridge and no one may pass
  • Piling on to Steve's reply, how, what and on what are you trying to do this? Please give us all relevant information as holding back is a waist of everyone's time.

    😎

    Year on year revenue calculation is trivial if the right tools are used and that is essentially what a last year-current year calculation is.

    The server's configurations / specs look rather meagre but then again it all depends.

  • sudhirnune (7/7/2015)


    Hi All,

    We are having a requirement to Aggregate the data and create LY, CY data across 5 Metrics.

    the Volume of Data will be 200 Mil - 250 Mil

    Server Configuration:

    Memory: 32 GB

    Processors: 16

    need help to understand the bench mark configuration needed for the server & any hints on better aggregation methods & LY / CY data

    The DBA at your company should be able to help with this.

    --Jeff Moden


    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.
    "Change is inevitable... change for the better is not".

    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)
    Intro to Tally Tables and Functions

Viewing 6 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic. Login to reply