November 22, 2009 at 5:35 pm
Can you share one of the most challenging downtime you handled working in SQL Server, and how did you manage yourself, your boss,colleagues, users; and also finally overcame the situation?
For example, in my case, one of the Admins had taken off the database offline and by mistake deleted the log file. Thankfully, it happened in off-peak hours, and we were able to use checkdb to fix (build) it again, but it was nerve-wracking, as we appeased our panic-mania manager on call while working at it.
It will be interesting to know other's recent challenging experience, and also prepare us of what all may be the extreme situations any one of us can face at any unexpected time...
November 22, 2009 at 11:15 pm
We run an ancient VB6 based app, and level 80 DB on SQL 2005 STD.
Someone enabled automatic updates on all servers, and the patches ans SP's came in from hell like never before.
Suddenly we are sitting with a VB6 app that doesnt want to connect, because it is pending restart the whole time.
After given the restarts, the SQL instance corrupted. Kicked over to DR mode..
costed us reinstalling and restoring from backups.
....
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This thing is addressing problems that dont exist. Its solution-ism at its worst. We are dumbing down machines that are inherently superior. - Gilfoyle
November 23, 2009 at 3:32 am
Other DBA added an index(which is added in all other client sites), went out and I was not aware of it, suddenly the server was very slow and the its like everything is halted, it took some time for me to find it and everything went fine, when the index was deleted...!
Manager was aware of it, but he was not behind me..Lots of calls from end users.
November 23, 2009 at 8:36 am
I wouldnt say it was that challenging or anything special, but I fixed a permissioning issue that came close to losing a 400 million $ trade. Also lost all the control files for an oracle production box which I had to fix quickly. had production clusters stop working for billion dollar trading systems. etc etc
Being a DBA is an every day challenge, nuff said...
--------------------------------------------------------------------------------------
[highlight]Recommended Articles on How to help us help you and[/highlight]
[highlight]solve commonly asked questions[/highlight]
Forum Etiquette: How to post data/code on a forum to get the best help by Jeff Moden[/url]
Managing Transaction Logs by Gail Shaw[/url]
How to post Performance problems by Gail Shaw[/url]
Help, my database is corrupt. Now what? by Gail Shaw[/url]
November 23, 2009 at 10:25 am
One of the member truncated large tables data in one of the environment without intimating the concerned team due to space issues in the server. Team member thought that its junk data. But concerned team escalated as there was no data for their testing and its their test data.
We tried to restore the database with last run backup, but as I said due to space issue we could not able to restore successfully. And also one more thing is we don't have RDP access to box as we are sharing the server with other group.
Then we tried to access each drive and each folder with xp_cmdshell 'dir C:\' to view the contents of the folders and drives. At last we moved some of the backupfiles to other server and made the restore to complete successfully.
November 24, 2009 at 7:25 am
2 years ago I had a client running year end tax stuff (payroll processing client) against 4900 or so databases on a single server (which was manually report shipped from the production server). The process initiated updates back to the production server too, which then refreshed those databases back to the report server. Well, the process had a glitch and hung, leaving over 700 databases with bad updates and refreshes. It took about 2.5 hours to figure out what the extent of the issues were and what damage was done. I was then able to develop some scripts to automate the restore of over 700 databases (with 6 backup files each) to the production server and get those then refreshed over to the report server while another person unwound the 'global' updates to things like invoicing, ACH, etc. That took another incredibly intense 3.5 hours or so. Thankfully this was discovered early on a Sunday morning!! My poor client really thought it was the end of the world. :w00t:
Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
Viewing 6 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply