February 3, 2012 at 8:25 am
We are pretty small but still have a basic change control process. Before we deploy any code (database or .NET), we deploy and test it on a staging server. We use SQL Compare to keep the database schemas in sync. Once the testing has been accomplished on the staging system we then have a meeting with all of the developers in the same room and push out the database changes (using SQL Compare) and the .NET changes (either via copy or by using the built in VS deployment tools). It isn't perfect but it works pretty well and at least makes people think a bit before their code is put into production.
February 3, 2012 at 8:53 am
At this large corporation, we have a Process Change system that we adhere to rather strictly for .net application development, at least in my smallish team (12) in a department of about 100. I'm also the document custodian for our group meaning I have to sit in audits and explain things when the procedures aren't followed, so I tend to keep after my coworkers about change control. Trouble is, we have to work with several different, non-integrated systems, one for Change Control, one for time tracking, one for logging, test plans, and MS SourceSafe for some of our source control. None of them are tied together electronically. There are a lot of manual steps to document everything, and there are plenty of ways to cheat the system. I think our developers pretty much follow the procedures, though. Once you know the process it doesn't really take that much effort. I kick up a stink when I find someone has been playing with something without leaving the paper trail.
Our MS SQL Server environment is a different story with few controls, and all of our developers are sysadmins (don't get me started...) so keeping the databases clean is a bit of a challenge. Sometimes I don't know objects and code have been installed until I start getting alert email that jobs are failing. I would like to tighten up change control on the DB's, but my complaints mostly fall on deaf ears...my team lead is one of the worst offenders...
No good deed goes unpunished.
February 3, 2012 at 9:19 am
CM is what it is, and most everyone knows that, amd most everyone knows that it is REQUIRED in most organizations nowadays too. So, managers that make you hurry up and wait, knowing that the CM approval process is right down at the end the hallway with a door with a keypad on it, is just playing head games with their people IMHO. They know software does not go into production unless it goes through the QA/CM approval process first, but that still does not stop them from making their people code software in three days anyway and work crazy hours doing it. Personally, I think its a power trip that a lot of managers (not all) get off on doing to their people.:w00t:
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
February 3, 2012 at 9:29 am
My company has about 2000 employees, 150 work in I.T.
Yes, we follow strict change management at least 90% of the time for production changes. We only started doing so about 8 years ago. Back then, we had a custom-written in-house change management system that worked pretty well -- not much red tape at all. But then management wanted an off-the-shelf ITIL software package, and the red tape just exploded. Everyone hates it because it's SO hard to move anything to production (data entry is very counter-intuitive compared to the custom in-house solution), but management likes it because now the software will tell them all systems that might be impacted by a change, or so they think. The reliability of the "systems impacted" report is only as good as the CMDB (change management data base), and that is really hard to keep up-to-date.
February 3, 2012 at 10:03 am
Ken Stelter (2/3/2012)
... Our MS SQL Server environment is a different story with few controls, and all of our developers are sysadmins (don't get me started...) so keeping the databases clean is a bit of a challenge. Sometimes I don't know objects and code have been installed until I start getting alert email that jobs are failing. I would like to tighten up change control on the DB's, but my complaints mostly fall on deaf ears...my team lead is one of the worst offenders...
We have a Change Control process, and I had similar problems to what Ken describes. Until the evening the developers put in an "invisible" change (outside the process) and essentially took down a client's online enrollment system. It was several hours until we knew there was a problem, then several more before the changes could be rolled back and confirmed - fortunately they had implemented the change using the code mover so rollback was relatively simple once they got to a computer.
The client managers wanted to know what happened and how, since they had to explain it to the client, plus contact the folks who had a bad online enrollment and ask them to put it in again. I documented the error, how it occurred, the steps to recovery, and the cost. I was VERY clear on what our normal procedures are, which steps of the normal process could have caught the error before it hit production, and how circumventing that process made it impossible to have caught the error. If the normal process had been followed, there were four distinct places where we could have found the error and fixed it before the move to production.
Haven't had a problem getting that lead to follow standard process since. 😀
To answer Steve's question, though, I can say that the systems I work with use Change Control 100% of the time. We even have a process for putting in "quick fixes" - with a fast-tracked approval process. Change Control doesn't have to hurt.
(Full Disclosure: I maintained the in-house written Change Control system at a prior company, and wrote major enhancements for it, so I'm a fan. I've seen the value...)
Steph Brown
February 3, 2012 at 10:21 am
Yes, 90% of the time I follow a change process. It is not a strict formal one but change has to be tracked and verified on some level. The ripple effect of the change determines the level of the change control.
February 3, 2012 at 10:36 am
I used work for a S&P500 financial institution in their asia branch. Change management is on every layer from dev machine configuration to code deploy on production. Change control is big part of IT audit and result of audit is part of KPI, so like one reply above, if you failed seriously in complying with the change control policy you are gone.
On some degree it is good because it force you to be meticulous - if you only find out the typos after releasing to production it takes you another week or two to make it right again.
February 3, 2012 at 11:49 am
Our MS SQL Server environment is a different story with few controls, and all of our developers are sysadmins
I was once told this in a interview by the interviewer and I terminated the interview right there. This is a HUGE red flag that screams STAY AWAY!!! You shouldn't walk away, you should run. This is a disaster waiting to happen, trust me, and when it happens, and it will, you will be the one cleaning it up at 3AM in the morning and explaining it in the CIO's office the next day with your job on the line. DBA's are the sole caretakers of production databases for a reason folks. Too many fingers in the pie can only result in one outcome, a messed up pie.:-D
"Technology is a weird thing. It brings you great gifts with one hand, and it stabs you in the back with the other. ...:-D"
February 3, 2012 at 11:55 am
We have very strict change management: first, there must be a triaged change request in TFS.
Developers make changes and check them into TFS, ops deploy the changes to test; if/when QA approves, ops deploy to UAT; i/when users approve, then ops deploy to production.
In case of any glitch we start from testing in the dev environment.
When the change works in prod for a week, PM closes the original TFS item or bug.
To answer Steve's question: this is followed 100 percent of the time.
February 3, 2012 at 12:01 pm
If you work for a publicly traded company in the US, then a compliant change management process is required by SOX.
I think that those who work in an organization where there is no change management process, and who think they don't need one, simply have a short memory when it comes to recalling all the messes they've made in the past. We developers tend to forget things like that unless someone in the organization is keeping score.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
February 3, 2012 at 12:07 pm
Eric M Russell (2/3/2012)
. . . We developers tend to forget things like that unless someone in the organization is keeping score.
That's what TFS does - if you use it.
February 3, 2012 at 12:21 pm
You're absolutely correct TravisDBA.
Our last 'DBA' left after a nervous breakdown; I'm an accidental DBA.
But I'm only the DBA when the servers are on their knees, spilling ones and zeros on the floor.
Those are 'I told you so' moments for me. But that doesn't help getting back my lost sleep...
No good deed goes unpunished.
February 3, 2012 at 12:30 pm
Revenant (2/3/2012)
We have very strict change management: first, there must be a triaged change request in TFS.Developers make changes and check them into TFS, ops deploy the changes to test; if/when QA approves, ops deploy to UAT; i/when users approve, then ops deploy to production.
In case of any glitch we start from testing in the dev environment.
When the change works in prod for a week, PM closes the original TFS item or bug.
To answer Steve's question: this is followed 100 percent of the time.
Question for you, and maybe a little off topic. How you maintain a reasonably up to date UAT environment ? We're essentially a data warehouse or datamart. I can't see replicating all our morning loads in UAT as well as production. So what happens is that when a new round of UAT comes along we'll move a couple months of data from prod into UAT but you always miss something. If UAT is really bad we'll take a backup of prod and copy into UAT; not horrible.
Ken
February 3, 2012 at 12:58 pm
Couldn't resist posting this: when it comes specifically to SQL Server, there's one comprehensive version-control solution out there that doesn't require setting any 'processes' , or any effort on the part of the developers. It simply scans your SQL Servers in the background, and saves what was changed, when and by whom. Its a 'set it and forget it' type thing, where you can be relaxed that a full version control information is always there:
February 3, 2012 at 1:08 pm
We are running the classic BI stack: DataStaging and DataMart databases that feed cubes that are accesed by SharePoint and can be accessed aby authorized users via PowerPivot. We keep all DataStaging and DataMart data for 13 months. Relevant PROD mart data older than 13 months is kept in archive.
Our biggest tables have ~4.5 G rows and grow 10-50 M rows per day.
UAT runs in parallel with PROD with a lag of about 8 hours to avoid overload of the servers from which we ingest data. That means that depending on time of the day, our UAT lags 16 hours behind PROD or is 8 hours ahead. Our users are aware of that and take it into account for their QA.
On the weekend following the end of the month we update the partitioning schemes and reload the partitioned tables. That is done on UAT first, then on PROD.
We run TFS schema diffs on both UAT and PROD. Then we restore PROD backups on UAT and re-apply changes that were done on UAT but not yet promoted to PROD.
Ops love it because it is a known PITA and they get paid overtime.
Viewing 15 posts - 16 through 30 (of 32 total)
You must be logged in to reply to this topic. Login to reply