Veeam Backup not backing up all disks

[dspjones]

Veeam Support - Case # 00729572

I have an Exchange 2010 DAG being backed up by Veeam. Up until a few days ago it was backing up just fine for over 2 years. Minor hiccups here and there but never anything like this. The job is configured to run every 4 hours and includes 4 servers

My 3 CAS servers
vm-cas02
vm-cas03
vm-cas04

and 1 of the 3 mailbox database servers

vm-mbdb01 (it doesn't matter which server I use the issue occurs on all of them)

the CAS servers all have a single C:\ drive (i.e. one vmdk per server)
the mailbox server has a C:, D: and E: drive

No drives are excluded in the job

when the job runs here is what I get from the mailbox server:
1/16/2015 10:38:02 AM :: Hard disk 1 (40.0 GB) 21.2 GB read at 29 MB/s [CBT]
1/16/2015 10:38:03 AM :: Hard disk 2 (198.0 GB) 33.7 GB read at 46 MB/s [CBT]
1/16/2015 10:38:03 AM :: Hard disk 3 (0.0 KB) 0.0 KB read at 0 KB/s [CBT] <------- THIS DISK NEVER BACKS UP

Thus the job never finishes and in fact in order to get Veeam to stop the job I have to actually go in and stop the Veeam Backup Service.

This also means I have to go into vSphere and manually remove the snapshot on this box and further it means that the logs are never truncated for exchange. Has anyone had this issue before. So far support has no answers although they provided a workaround using Windows native backup to at least get the logs truncated.

While waiting for an answer from support I decided to delete the original backup job and all of the backups as well and create a completely new job. The hope is that maybe there was something up with the job itself. But the new job is doing the same thing. I also chose a different mailbox database server to use instead of the one I mentioned above. This time instead of Hard Disk 3 not being read, it is Hard Disk 2 that is not being read. So I get the following:

1/17/2015 12:38:02 AM :: Hard disk 1 (40.0 GB) 21.2 GB read at 29 MB/s [CBT]
1/17/2015 12:38:03 AM :: Hard disk 2 (0.0 KB) 0.0 KB read at 0 KB/s [CBT] <------- THIS DISK NEVER BACKS UP
1/17/2015 12:38:03 AM :: Hard disk 3 (87.0 GB) 31.4 GB read at 30 MB/s [CBT]



Thanks,
Jones

[dspjones]

Going through old forum posts and came across this one:

http://forums.veeam.com/veeam-backup-re ... 18176.html

Looks very similar to our problem. When I first saw this occur, the backup was still running. It had been sitting at 74% complete for over 24 hours.

[Vitaliy S.]

Were there any changes made to these VMs recently? What happens if you exclude disks that are stuck in the backup job? Also I wonder what is written to the job debug log for all these cases. What was your support engineer response on this issue?

[dspjones]

I think I found the root cause and have resolved the problems.... In a nutshell here is a condensed version of the issue and resolution. Earlier this week (Sunday 1/11/15 and again on Wednesday 1/14/15) we experienced crashes of the Veeam SQL Database. This has happended in the past and we recovered from it without problems. However these 2 particular times the crash of the Veeam SQL database occurred during our exchange backup job runs. For some reason that I still can't nail down, the backup proxy servers (2 of 4 used for these jobs) had an issue reconnecting the iscsi volume of the mail log volume and the mail database volume on 2 of the exchange mailbox servers. After initially contacting Veeam Support last night with the problems I began investigating all the various pieces involved, everything from the mail servers, exchange, backup servers, network settings, vsphere - you name it. While doing that an interesting thing happened. When I logged into the NAS device where our backups are stored an alert popped up telling me the prior shutdown was not done properly. WHAT? Prior shutdown??? So I investigated the NAS logs and sure enough the device has rebooted itself for an as yet undetermined reason 3 times in the past 7 days coinciding perfectly with our Veeam issues on Sunday and Wednesday. In addition to storing the backups, the NAS also provides the location for the install drive for Veeam. If the NAS dies, so does Veeam etc.... So I ran disk checks on the device, rebooted it properly. Then I also rebooted all of my Backup proxy servers for good measure. I ran the Exchange backups and so far they are back to running normally. In fact the backups are running faster than every before. I have opened a case with our NAS vendor for closer inspection of the hardware which I believe is the root cause and I asked for support to close the case.

I do want to make a note about support's response on this issue. When I called it was 10:37PM CST - a full 2 hours outside of our standard support time. The guy I spoke with mentioned that and then asked if the issue was critical. While it was not at the specific time, I told him that because this was an Exchange backup issue, if it did not get resolved today that by the time we got around to our support window we would probably be in full crisis mode when the log folders overflowed because the exchange logs had not been truncated. Veeam Support stepped up and helped me address the issue, he took logs and provided a workaround method for getting our logs truncated while he took a look at everything. Surely he could have told me to call back later or whatever but I wanted to say that the engineer I spoke with got an A+ for service beyond expectation. In the end, it is the customer experience that keeps winning us over with Veeam. Great product - BUT EVEN BETTER SUPPORT AND CUSTOMER FOCUS. Thanks guys!!! Keep it UP!