April 2, 2008 at 5:52 am
I'm not sure if this should be in the strategies forum or here but here goes.
We have a data framework system that loads data from other systems. It isn't overly large but it is big enough. It processes files it receives in a batch type mode each day.
Its built on SQL 2000 and mainly uses DTS to load the data. The company/people who built it and implemented DTS made some bad decisions and we've been struggling with DTS ever since, especially when we move the packages between environments. (Wish they'd have heard of dynamic properties tasks!)
I'm working on convincing our management to upgrade to SQL Server 2005. I want to migrate the packages from DTS to SSIS. (I've found an interesting conversion tool but thats another post.) One of the developers feels that re-wrrewritingDTS packages in pure VB.NET would be the best way to go, rather than migrating to SSIS. From a developer's perspective I can see why this is an attractive option. Lots of control, flexibility and less interaction with the pesky DBA. As a DBA this looks like a nightmare to me for a whole lot of reasons.
My gut feeling is that this if coded right, loading and processing data in .NET could be the fastest way to load the data however knowing our management, our developers won't be given the time to go through the iterations to write really efficient code. They'll be told to "git 'r done!" as they always are.
I'm thinking that with less effort, we can develop much faster running code in SSIS easier than in .Net and even leverage .NET in SSIS, where it makes sense but I'm going to need to make some compelling arguments to defend my recommendation. (And I want to make sure this really is the best route)
What's your experience with this? Have you had this debate with your developers? Did anyone go this route and what was your experience?
cdp
Chris.
April 2, 2008 at 7:16 am
These arguments always drive me crazy, so this will be my only post on the subject.
I have always bucketed people into "programmers" and "developers".
My idea of a programmer is someone that can write code - some of it great code - that will do what you need it to.
My idea of a developer is a programmer that only writes what they have to and uses what is already written when possible.
You may have developers that can write more efficient ETL code than the (rather large) group of developers at Microsoft that are working on SSIS. It is the case that the code your developers will write will probably be more focused (or less flexible depending on your perspective) and may end up faster (or less thorough). At the end of the day, however, I usually like to ask myself if writing an application is the best use of my time. Now when you have what is a pretty good 2nd generation ETL tool like SSIS basically free (I assume you already need SQL Server for something) it seems like a waste of resources to write another one that will be very likely to be inferior in almost every way.
In addition to this, anything written custom will have to be maintained forever by your in-house developers. How much are you paying these developers? It gets expensive - fast - to pay developers to maintain something that acts as a communication layer. Just keeping up with .Net versions will cost you regularly.
If you do not think SSIS is the best ETL product for your company, look for another product before building one that will, in the end, be inferior to something that you could buy for far less money than you can build it for.
April 2, 2008 at 7:24 am
Agreed - This argument IS driving me nuts.
I'm not certain my original post indicated this so I'll restate it if I wasn't clear: I think SSIS is a great tool and I really want them to adopt it. I think MS did a fantastic job creating it.
Thanks a lot Michael, those are excellent points.
cdp
Chris.
April 2, 2008 at 7:34 am
Ok - one more post.
I didn't mean to post that you did not think SSIS was any good. I also did not mean to post that it is a perfect product in any way.
I have seen a lot of developers build something that was already there (ETL tools, Replication, Encryption, etc.) all claiming they could build something better. None of them have been right and they have usually built something inferior, unmaintainable, and ultimately in need of replacement.
Now if you are a software company building a tool to try to support the masses, this is absolutely possible. If you are a corporate IT shop, my suggestion is to stick to writing applications that are specific to your business - anything with a customer base of 1. If there is something that you need that matches with a horizontal market it is always cheaper and faster to buy it from a software company that focuses on the product. This includes CRM software, ETL software, database engines, workflow engines, etc. It seems crazy to build one of these unless you intent to sell it (realistically).
Viewing 4 posts - 1 through 3 (of 3 total)
You must be logged in to reply to this topic. Login to reply