August 5, 2013 at 2:30 pm
Hello,
I have an interesting issue, the cause of which remains a little elusive. We have recently had multiple failures in our transaction log backups, each time seems to result in a corrupt tlog in a similar sequence of events. We are on SQL Server 2012 Enterprise (recently upgraded), running on top of Windows Server 2008R2 Enterprise and are using Red Gate as our third-party backup solution. Twice now, we have suffered a break in our tlog chain, only to find that the tlog which was the breaking point was 'successfully' created, but with issues that are only logged in the Red Gate error log. The error apparently does not get passed up high enough to be trapped in our SQL error checking embedded in the script which does the backup, therefore the backup job does not fail. The error we are seeing is:
----------------------------- ERRORS AND WARNINGS -----------------------------
8/4/2013 12:00:14 PM:
8/4/2013 12:00:14 PM: Verifying files:
8/4/2013 12:00:14 PM: K:\Backups\[ServerName]_XXXXX\Tranlogs\FileNameX\LOG_NameX_20130804120000.sqb
8/4/2013 12:00:14 PM:
8/4/2013 12:00:14 PM: Thread 0 error:
Process terminated unexpectedly. Error code: -2139684860 (An abort request is preventing anything except termination actions.)
8/4/2013 12:00:14 PM:
8/4/2013 12:00:14 PM: SQL error 3013: SQL error 3013: VERIFY DATABASE is terminating abnormally.
8/4/2013 12:00:14 PM: SQL error 3254: SQL error 3254: The volume on device 'SQLBACKUP_0C2E9FDE-8A43-41AC-B7C1-2CBE70956DA2' is empty.
8/4/2013 12:00:14 PM: Validation of all backup files failed.
----------------------- PROCESSES COMPLETED SUCCESSFULLY --------------------
8/4/2013 12:00:00 PM: Backing up XXXXX (transaction log) on InstNameX instance to:
8/4/2013 12:00:00 PM: K:\Backups\InstNameX\Tranlogs\FileNameX\LOG_NameX_20130804120000.sqb
8/4/2013 12:00:00 PM: BACKUP LOG [NameX] TO VIRTUAL_DEVICE = 'SQLBACKUP_8798FF6C-E2C5-4FCB-8728-EB49BA1335B0' WITH BUFFERCOUNT = 6, BLOCKSIZE = 65536, MAXTRANSFERSIZE = 1048576, NAME = N'Database (NameX), 8/4/2013 12:00:00 PM', DESCRIPTION = N'Backup on 8/4/2013 12:00:00 PM Server: NameX\NameX Database: NameX', FORMAT
8/4/2013 12:00:13 PM: Backup data size : 15.625 MB
8/4/2013 12:00:13 PM: Compressed data size: 2.790 MB
8/4/2013 12:00:13 PM: Compression rate : 82.15%
8/4/2013 12:00:13 PM: Processed 1203 pages for database 'NameX', file 'NameXlog' on file 1.
8/4/2013 12:00:13 PM: BACKUP LOG successfully processed 1203 pages in 0.609 seconds (15.420 MB/sec).
The only pattern we can find is a concurrent SQLVDI error which is appearing in the servers Event Logs, which specifically states:
SQLVDI: Loc=SVDS::Open. Desc=Bad State. ErrorCode=(-1). Process=564. Thread=11324. Server. Instance=NameX. VD=Global\SQLBACKUP_A9FCA3B4-CB68-4C45-985F-DB89C6E60D6A_SQLVDIMemoryName_0.
When the tlog attempts to restore, SQL Server generates the error:
The volume on device 'SQLBACKUP_59E6E383-98CA-4179-8721-FB8379EB6817' is empty.
...which appears to indicate corruption in the tlog because it is NOT empty and is the correct and expected size. We have ruled out corruption during the network copy of the tlog to the log-shipped server, since the error is happening closer to home as the backup is actually occurring. Other than the SQLVDI error above, we have not been able to locate any other errors at the SAN, Server, or OS level.
The box this is running on is a monster with 1 TB of RAM that is not remotely over-stressed, so I don't believe any sort of memory pressure is the culprit, outside of a buried config that we haven't found yet. The occurrences of this issue have also not followed a day/time pattern and do not coincide with any obvious maintenance or business processes that might be a source of interference.
Has anyone experienced similar issues, particularly related to SQL Server 2012 and SQLVDI errors? Would love to hear any experiences at all that might help shed some light on this odd issue.
Thank you!
------------------------
David Caughill
Production DBA
Grad Student
Coffee Enthusiast
June 12, 2015 at 2:42 am
Check this link out, it does apply to different version of SQL, but the scenario could have been the same.
Viewing 2 posts - 1 through 1 (of 1 total)
You must be logged in to reply to this topic. Login to reply