November 11, 2006 at 5:33 am
Hi all,
I recently started playing around the the Parallel Process utility seeing as my server has 8 procs. It looks fantastic in that it cuts my cube processing time (which is about 6 hours!) in half but there is one slight issue I need to find a way around...
The cubes I have inherited have a bit of a flaw. There are many dimension key errors (for example people in my fact table who have been deleted from the hierarchy) on many of them. I can not control the quality of this data due to external systems and other boring political bits and bobs within my company so we have had to turn off in Analysis Manager "stop on key errors".
Parallel Process doesn't seem to like key errors regardless of what this setting is. It will spend an hour processing the partition and runs through all the rows, builds all the aggregates etc and displays which members have errors but when it's finished doing all that it ends, reports an error in the log and the partition is still marked as UnProcessed in Analysis Manager.
Does anyone know of any way to tell the utility to not care and finish off, leaving them processed and able to be queried?
Alternatively does anyone know of a better way of processing 132 partitions spread over 33 cubes that can process more than 1 partition at a time (or otherwise speed the whole process up a lot).
Cheers,
Mike
November 11, 2006 at 9:00 am
If you're inclined (and you have access to the VB6 dev environment) you could open the src folder in the install directory, open the project and then step through the code when processing a single partition. This could let you see what's happening in the code to set/not set the 'stop on error keys' flag.
Steve.
November 11, 2006 at 9:22 am
I had a quick look through (ie not comprehensive) but it doesn't look like the ProcessingKeyErrorLimit value is set anywhere for the partition, so this would probably leave it at the default which I think is 0 which means it accepts none.
Steve.
November 13, 2006 at 1:22 am
Thank you very much for that Steve, unfortunately we're not allowed VB6 in this environment. I have .net express at home I keep meaning to look into but I would imagine that differs vastly and might be beyond my tiny brain at the moment?
Have you used anything else simliar? I keep thinking I can't be the only one out there wanting something a bit cleverer than MSAS' sequential process followed by a complete rollback at the first sign of trouble... wiping out 6 hours of process for the tinniest of errors in the Virtual Cube at the end which doesn't hurt any of the previous objects.
We've created a looping DTS that processes them one at a time and commits each one but it still takes just as long.
Cheers again,
Mike
November 13, 2006 at 7:45 am
Sorry, am working almost exclusively with 2005 now which has this built in. You're right, /net express (which is VB.net) is quite different to the old VB6 code.
Just a thought though, why not have your DTS package build several in parallel (e.g. 4)? You would have to add a little more to the package. If you had a persisted DB table to hold the partition names and a 'processing' flag, then added 4 processing tasks, put a sql task before these to read off the next partition off the stack (ie out of the table), update the flag to indicate that that partition has started (so the other processes don't try to process the same one, they pull another off the stack), then post processing you update the flag again to indicate completion. I've tried to depict this (loosely) below. Forgot to mention, you'd need some dynamic properties set in there too after you pull the partition name off the table you need to update the task to process only that partition.
start
read partiton read partition read partition
process process process
update flag update flag update flag
back to loop start back to loop start back to loop start
Cheers,
Steve.
November 13, 2006 at 9:13 am
Great Minds! I did actually try that on Saturday... created 4 seperate process tasks, and then tried:
a) All of them reliant on a SQL task (which just contained a comment)
b) None of them reliant on anything
Only one at a time ran when I did A... even when I deleted and recreated the logic a few times, checked in properties that it was happy running 4 things at once... all to nought.
When I tried B none of them ran.
This is probably more fixable than the utility though, unless it's something deep within the bowels of the MSAS custom task?
Cheers again,
Mike
Viewing 6 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply