One's Missing
-
heh... she said 'sucks'..
-
They never really start out as 5 hour programs... they tend to start out as 20 min programs... then a year later they're 40 min... and the next year they're running for 80 min... and the data keeps piling up.... I've been there and seen it... The program I mentioned in the last post I'm sure started out at a perfectly acceptable level of performance... when there were a couple hundered records... now... when there got to be millions... things changed...
How to tell when a program is working? I deal with as400 programs all the time that sit in delay statuses. it's all a matter of peeking at the data to see if things are changing or not. (you need to know what the program does in order to do this, but most batch processing systems read a record, do some work, and then either mark the record processed, or delete it. In many cases with a date or time stamp... so, you can see what's going on...)
I can see the 5 hour jobs seeming sane... financial institutions and the like that may be processing millions and millions of transactions in a single batch... Maybe the designers wrote the program, and figured .25 seconds per transaction was good. when they were processing 1000 transactions per day it took a little over 4 min to run. no big deal. but, several years later, they're processing 72,000 transactions per day, and the job is running for 5 hours...
Things can change considerably when you think about scalability instead of just here and now. unluckily a lot of us think about the here and now. we deal with making this process run acceptably. Then... several years later, the DB admin needs to come along, and fix that flaw that makes the transaction take .25 seconds (no big deal right?) and get it down to where it should be in the range of .01 seconds or less. And maybe re-design some company logic to process multiple batches a day instead of trying to do everything in one run.
oh... and I wasn't talking about you... I was referring to my time. 4-6 hours, and we would probably have something useable running... just a matter of getting those 4-6 hours. :/ and I've been SWAMPED lately... (of course, part of that's my own fault... but...)
-
I guess I've let my experiences get the better of me. Yes, I can think of plenty of applications where a long run time is acceptable. Just not the same ones where most people seem to think it's acceptable. It really comes down to how much there is to process. My problem isn't with the application that really does something complex, but with the app that gets run daily, processes somewhere under 100,000 transactions, and takes forever to run when there's no real reason for it to do so other then the designer did not think ahead.
It's just very rare to come across a non academic situation that really requires that type of processing time (where appropriate funding and hardware to divide and conquer isn't also available)
-
sorry . i meant he said 'sucks'
-
no, you got it right... I hit the wrong quote button.... doing too many things at once at work :/
-
And my final conclusion is that whatever tool is best suited to the task is the one I'll use.
-
Steve is right .. if a developer is not good in writing queries he will have to depend on others for writing queries .. this is not a good sign .. hence it is essential to learn SQL by all the developers ..
-
A final thought on SQL. A course on the syntax is not enough; it should include relational algebra first, so that the logic behind the SQL is understood. More deep magic - but without it I don't think the programmer really understands what he is doing.
-
For what this is doing, yes, 11 minutes is very fast...
🙂
Steve G.
-
I can't even fathom what you'd build in 'vbscript' or 'javascript' that would call for a 5 hour run time... to be honest... .... it sounds like bullshit..i might be wrong.. i can't even after searching the planet .... find an example of code that might pull that long of a query... so i'm calling you on it...i think your bullshitting us...
my tests of apples for apples... .net c# iis apps or otherwise vb.net run a touch slower against ANYTHING such as ASP / JSP even dcom ... but the difference is marginal... ie not a consideration..
its the precompiled and management overhead that does it...
I saw hestitation when i was at the MS seminar I was at last summer... designed by a lead designer and the site was http://localhost//
and that was in C#....
i do not think were talking about the same thing....
VBScript / Javascript from the good people at wiki
VBScript is interpreted by a script engine vbscript.dll, which can be invoked by ASP engine asp.dll in a web environment, wscript.exe in a Windows GUI environment, and cscript.exe in a command-line environment. When VBScript source code is contained in stand-alone files, they have the file extension
.vbs.further.... there are alot smarter people than i out there I'll admit that.... but my pencil aint that dull either....
but according to this guy ... who i do not know, though i think his thoughts on exposing asp within asp.net are interesting and btw the only way I've found how to do so...
My short analysis is certainly not "Lab Quality", but I do believe it is not flawed either. There appears to be some performance hit using ASP .NET vs Classic ASP especially when using the asp
ataGrid control."ASP.NET vs CLASSIC ASP SCALABILITY / LOAD TESTS TEST DETAILS: 10THREADS - NO RANDOM DELAY - ONE MINUTE TEST TEST NAME HITS Code: 200 TTLB Content Length Req. /Sec. DataSetRenderDataGrid 1080 1080 343.86 36077 17.95 ASP:RecordSetToTable 1581 1581 204.11 22700 26.20 DataSetMakeTable 1111 1111 269.30 36781 18.46 DataSetMakeTable(SB) 1114 1114 272.45 36781 18.50 TEST DETAILS: 100 THREADS - NO RANDOM DELAY - ONE MINUTE TEST TEST NAME HITS Code: 200 TTLB Content Length Req. /Sec. DataSetRenderDataGrid 1145 1145 3867.19 36077 18.94 ASP:RecordSetToTable 1626 1626 1493.85 22700 27.13 DataSetMakeTable 1139 1139 1548.58 36781 19.14 DataSetMakeTable(SB) 1192 1192 1620.94 36781 19.29 http://www.eggheadcafe.com/articles/20010920.asp
Now MY POINT is they render results AROUND THE SAME TIME... before I get flamed...
now if asp and asp.net run about the same speed... and ... and vb.net runs about 2x in best case scenario's faster than asp.net and c#.net is shown only to be "by microsoft's own reports" 20% faster than VB.NET... the idea being VB.NET aint a shitty way to code... how is it possible to go from 5 hours .. unless the code was shit to begin with to 11 minutes...
I can see 5 to 1 hour jump .... if all is equal... and that is being incredibly generous on my part...
But what your really saying is I, as in you, not I as in me, poorly programmed the original code... and came up with a better structure....
and kudos for that...
-
Nice analysis, but you're missing the point.
What I'm saying is that an experienced programmer who understands how these machines work and what people really use them for, wouldn't be questioning the existence of an 11-hour process. We are talking about processes that are going to take a long time, no matter how the process is coded. You're calling this guy a liar because of your own ignorance. You don't believe there's any need in the world for an 11-minute process that used to take 5 hours, so you're calling the guy a liar, because in your mind, no such process could possibly exist?
You go on to say that a change from VBScript to C# wouldn't make a big increase, but you're wrong about that too. Again, because your mind is closed inside the box. Your analysis is for the web page parser in classic ASP vs. ASP.Net. You know there are other things on computers besides web sites? There is this old-skool idea called applications (we used to call them programs.) That's what we're talkin about here, so your comparison of two web server technologies doesn't apply and was a total waste of your time.
This is about changing an application from VBScript (which was capable, but not optimised for applications), to C# (which is optimised for applications and compiled). Strictly speaking VBScript and C# are both interpreted languages, but VBScript is completely interpreted, meaning during execution, the parser is actually reading and parsing the source code. In C#, the source code is compiled into MSIL, and the MSIL is run through the interpreter. This change to MSIL, which is closer to machine code than C#, is what makes C# so much faster. From 5 hours to 11 minutes is only a 27x increase in speed, and when going from an interpreted language to a compiled language, increases of 1000x or more are possible.
When I talk in my previous post about people not knowing the basics... you are the person I'm talking about. Most programmers these days don't know how computers work, aren't aware of applications outside their little world, have no problem-solving ability, and are basically no more than parts-assemblers. You obviously don't understand the difference between an interpreted language and a compiled one, and that is a huge thing, something that should be covered in the first few days of a programming class, in any language.
If you're gonna come in here and say people don't know what they're talkin about, and make snide remarks about my gender choices, then you better be damn sure you are correct.
Without searching the web these are the things I can think of that would take a really long time:
Preparing the bill for all Verizon Wireless customers from data about indvidual calls. (This operation can barely be completed in a month's time before the next cycle begins. BTW, this is absolutely as fast as it could be done with 2001 AS/400 hardware.)
Predicting the weather.
Finding huge prime numbers.
Estimating molecular structures for large proteins.
Cracking the enemy's encryption
Fluidic motion simulations, used in engineering applications... notorious for being slow.
Basically anything with a lot of calculations is going to take a long time. Don't go around insulting people's coding ability when you don't even have the imagination or experience to think of these very obvious cases.
-
I aint insult anything or anyone. But 'i come in here and say alot worse more of than that.....'
I did make an assuption though; I assumed Jasmine meant you were a girl.
I corrected myself.
I further called him on his miraculous code generation in vbscript, who built something that takes 5 hours to run.
Those programming challenges you brought up Preparing the bill for all Verizon Wireless customers from data about indvidual calls. (This operation can barely be completed in a month's time before the next cycle begins. BTW, this is absolutely as fast as it could be done with 2001 AS/400 hardware.)
Predicting the weather.
Finding huge prime numbers.
Estimating molecular structures for large proteins.
Cracking the enemy's encryption
Fluidic motion simulations, used in engineering applications... would NOT be done in vbscript...not even from the get go.... so talking about it jsut plain stupid.
It's his words.
Lets see the code.
If it's true, there should be no problem releasing the source for it.
I do not believe it. It's bullshit It is though, alot like someone saying they are something, or putting on that they are something, when they are not!
Maybe in assembly; but nothing in managed .NET can improve that much. Unless it's that poorly written to begin with.
And its "TOM CRUISE:..... You don't even know what Ritalin is...."
So stop flirting with me. And learn to read.
-
Everyone here needs to take a deep breath..
IN ...
OUT ...
OK, Now that we've got that out of the way, let's get a little more accurate information. Why was my program taking 5 hours to run? Simply put - good old fashioned page thrashing. It used to run in 20 minutes. Yes, it was slow, but it was written in JScript so I didn't expect it to be fast and it ran overnight so it didn't matter. One fine day we hit the 'magic number' in the amount of data being processed and JScript's exceedingly lousy (IMHO) memory management started biting us in the .
I rewrote the thing in something that had better memory management (C#) and process times went way down. Everyone's happy, I get a gold star and we're on to the next thing.
To answer a couple of questions:
1) Why did I write it in JScript to start with? Several years ago that was the only tool available to me. Seems absurd, but that's the way it was.
2) What the $%^& is this thing doing anyway? It's processing data from one very fixed format database to another very fixed format database. The program does a large number of rather complicated regular expression pattern replacements on a couple of fields for each of about 700,000 records. Frankly, I'm amazed that it runs in only 10 minutes.
3) Has it been optimized to the N'th degree? No. Simply put, I have better things to do. Trying to get my 10 minute program down to 5 or 4 minutes when I have an open 4 hour window is not the best use of my time.
Hope this all helps, and y'all have a good weekend!
Steve G.
-
I know Steve, that's what I've been saying. For him to say your lying, when anyone with any experience should know exactly what you're talking about, well that's just wrong, and insulting me was pointless and juvenile. I take discrimination against transgender (or any) people very seriously and it's attitudes like that which keep many great minds from achieving their potential. I can't allow that to go unanswered.
-
I'd have to agree with some of the above few posts. I was a bit caught up in what I'm used to, and had forgotten about things such as cracking cryptography, research, and the like. Personal experience, I haven't seen any of those systems behind the scenes, so, I can't say anything as to if they could be optimized further. I've only seen the 2-3+ hour business/accounting job that runs at that speed because someone did some custom work on the code, and broke the index without fixing it, or just simply had really bad programming practices.
I can understand not fixing it if it isn't broken. That was a bad habit I had for a while. I've since discovered at least in my position, it is a bad idea. Where I work, we run a very tight schedule. We have about a 3 hour window to process nightly backups, all the previous day's transactions, inventory comparisons, and the like between our warehousing, and accounting system. A large portion of my job has become simply to optimize everything I can find that's misbehaving. Since we have a very seasonal work flow, volume breaks everything, and what doesn't seem like a problem one month, suddenly balloons into a massive headache the next. While our business has increased considerably, changes I've made have allowed our as400s processor to be used considerably less.
As for the gender discrimination, I don't think I saw it in the thread. (Unless I'm really missing something) I believe I saw both me and GPF2^192 either being a bit confused as to who we were talking to, or simply misunderstanding who we were talking to based on making an assumption. I didn't even know, or care if anyone here was transgender until the last couple posts. Honestly, when you're behind a terminal, most times all you have to go on is what a person's name is. What I do know is that doesn't really have a place in this conversation. The root of this entire conversation is that SQL should be just as big of a requirement for most positions as the programming language the person is working in. Not just a basic understanding of SQL, but a complete one. The reason, that 10-20 min to execute query can very quickly become a 5 hour query, and it's far better to plan ahead, and build it to be scalable the first time instead of someone else needing to revisit it several years later, and basically start over from scratch. More important, monitoring for those heavy duty jobs that run for too long should at least occupy a small portion of someone's time, because just spending a few hours here and there can end up making a massive difference in overall system performance.
Really the point we should be making here is back to the list of 10 programming languages... SQL should be first on the list, simply because it is so central to the majority of programming jobs, and it crosses over most platforms.
There was a great post a few days ago on another site I read, it was about a developer who didn’t understand SQL, and submitted a table for creation consisting of nothing but varchar(4000) columns, then resubmitted as varchar(255), and finally after several back and forths submitted a real table design with appropriately typed columns. Stuff like that is the reason that developers need to know SQL, if not a solid grounding, they should know at the very least, the basics of good database design, and how to make things at least marginally scaleable.
Viewing 15 posts - 16 through 30 (of 33 total)
You must be logged in to reply to this topic. Login to reply