February 18, 2012 at 9:48 am
lozis6947 (2/17/2012)
I would say that your #2 is a bit off
Care to elaborate?
--Jeff Moden
February 20, 2012 at 6:30 am
PaulB-TheOneAndOnly (2/17/2012)
GSquared (2/17/2012)
PaulB-TheOneAndOnly (2/17/2012)
Jeff Moden (2/11/2012)
PaulB-TheOneAndOnly (2/11/2012)
Just loading a table hardly qualifies as ETL.Seriously? You don't think that taking a mega-row text file from something like Double-Click.Net and loading it into a table qualifies as at least the first step of and ETL process? Loading a warehouse from OLTP data is just one form of ETL.
well... it turns out that you are talking a more complex table load that involves moving text from an external data source to a data warehouse - in this case it does qualifies as ETL.
Having said that, I stick to my arms "the (simple) load of a table hardly qualifies as ETL".
I disagree. Complexity isn't what makes it ETL or not. Neither does scope. Type of process is what makes it ETL.
Importing a .txt file with rows and columns defined by line-returns and pipe-characters, into a table, is what if not ETL?
Extract obtain something from source: to obtain something from a source, usually by separating it out from other material
Transform change something dramatically: to change somebody or something completely, especially improving their appearance or usefulness
Load transitive verb put something in machine: to put into a machine the items that it will work on, e.g. clothes for washing
To load a table from a text file, you have to extract the data from the text file, including conversion of delimiters or spacings (fixed-width material) into columnar format.
Then there's transform: conversion of storage format (external .txt file to internal binary representation in the database), validation of length and characteristics of the data to conform to the table definition (typed instead of free-text data), etc.
Then it's actually loaded into the table, exactly as per the definition of "load".
What else would it be, if not ETL?
If I'm reading it correctly - as per your own definition of ETL Jeff's scenario qualifies as ETL, scenario calls for:
- Extract data from a plain text external datasource
- Transform that text (at least) mapping it to the target columns
- Load the target table which by Jeff's scenario is part of a data warehouse.
Yep. Jeff's would.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
February 21, 2012 at 5:26 am
up vote 2 down vote favorite
1
share [g+] share [fb] share [tw]
I'm fully convinced that a significant part of the work I'm doing falls into this domain of ETL, but I didn't even know the term existed before 3 months ago. I've found SSIS to be a bit of a mismatch for my skillset, i.e. my instincts are that writing C# code in a well thought out way will give me the result I need (also my employer doesn't own it). I started looking at WF because if seemed logical, but I came back to the original conclusion that I really need to understand the fundamentals of the problem domain, and when I do that it will make the most sense to leverage my experience and code the solution in .net/c# (I'm a one man team and that doesn't seem to be changing). So far I have a sort of hodge-podge of syncher utilities, and it was the difficultly that began arising in managing them all that led to seek out this knowledge.
QUESTION 1 is: is there a resource for me to get some examples of how it's all put together for things like:
extracting from REST services with usage limits --> loading to databases for purposes of (as close to) real time (as possible) synchronization
extracting from in-house 3rd party apps like QuickBooks --> loading to databases
monitoring for changes in database and updating external systems in carefully tracked batches (i.e. the same information that was extracted is changed by an LOB app and then needs to be pushed back)
QUESTION 2 is: I've yet to grasp where the T part will come into play. Thus far I've been pulling the information that represents logical entities in one system and pushing them into another.
http://www.stylusstudio.com/etl/
Search Engine Optimizing | Search Engine Marketing | Social Media Marketing | Pay Per Clicks
February 21, 2012 at 6:27 am
carmellabanker (2/21/2012)
...QUESTION 2 is: I've yet to grasp where the T part will come into play. Thus far I've been pulling the information that represents logical entities in one system and pushing them into another.
The "T" in ETL can be as simple as format transformation, and can be as complex as breaking up data into snowflake schemas and things like that in multiple data warehouses. Usually, it's somewhere in between those two extremes.
A very common T would be taking flat-file data (.txt, .csv, .xls) and normalizing it into a relational structure in a database, split into multiple tables and possibly with string values replaced by keys from lookup tables. That kind of thing.
Another is converting data from relational format to snowflake or star schemas for loading into a warehouse.
Does that help?
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
Viewing 4 posts - 16 through 18 (of 18 total)
You must be logged in to reply to this topic. Login to reply