Backup failed: fault.InvalidDiskFormat.summary

[kjstech]

I got the error on one of my VM's on last nights backup fault.InvalidDiskFormat.summary. The VM is a critical VM and has 902.23GB of data on it. It backed up fine the night before.

I looked at the vm's settings and I see it running on -00001.vmdk's, but there are no snapshots for it. I have vSphere 5.0 and it says consolidation needed. I'm not going to touch anything now while its production hours, but once everyone goes home and I do the "Consolidate" option in vSphere client, do you think this will resolve the issue so it backs up tonight?

Storage views under Datastores views in vSphere client is wrong too. It shows 79.00 B for Snapshot space, but if I go into the datastore browser both disks show a 3 and 12gb snapshot file.

This used to happen a lot with Acronis vmProtect 6,7 and 8. It didn't happen so much with version 9, but I had enough of it from previous versions so we switched to Veeam. Please don't tell me we are going to start having the same problems with this software.

[foggy]

Looks like this one is a "hidden" snapshot (not visible within vSphere Client) that must be consolidated.

[kjstech]

Well I tried the consolidate option and I got the same error that Veeam got in the vSphere 5.0 client. fault.InvalidDiskFormat.summary.

So next I tried creating a snapshot with the ultimate goal of being able to do a Delete All. It was thinking for awhile but then it got the same problem fault.InvalidDiskFormat.summary.

So I check the datastore and the original vmdk disk files were last modified 1/20/14 2:38 AM, so that would be right within the backup window. This leads me to believe that this is Veeam's fault. How am I to trust in this product?

I guess I have to open a case with both Veeam and VMWare on high priority and see who gets back to me first. This is a very mission critical server. Needless to say I am pretty irritated so I hope we can get some help here.

Veeam ticket: 00505889 severity 2
VMWare ticket: 14429841301 severity 2

[kjstech]

VMware determined the course of action was to power down the VM and do the consolidation while it was off. I was able to get this done after hours tonight and this worked.

They did see based on the original VMDK last time stamp that this was caused during the backup job. They are going to get a summary of what happened from their perspective and I will attach that to the Veeam case.

I'll see if tonight's backup executes without issue now.

[foggy]

Thanks for the update, I'm sure Veeam support guys will be able to track the issue down with the information provided by VMware.

[kjstech]

Last nights backup went fine but now today its back to running on snapshot disks again. Guess what, I cannot consolidate again!!!!

Adding notes to the case.

I think ultimately instead of screwing around I might have to just delete the backup job in Veeam and recreate it.

[foggy]

I think ultimately instead of screwing around I might have to just delete the backup job in Veeam and recreate it.

Not sure this can help in the long run. Let's wait for what support will come up with after reviewing the case.

[kjstech]

The only thing that vmware can see odd is in the vm's log is this line:

Code: Select all

2014-01-23T06:41:40.424Z| vcpu-0| SNAPSHOT: SnapshotBranchDisk: Failed to branch disk: '/vmfs/volumes/8793dc31-10bb6cf7/R2D2/R2D2-000001.vmdk' -> '/vmfs/volumes/8793dc31-10bb6cf7/R2D2/R2D2-000002.vmdk' : Change tracking target file already exists (3900)

There was a 000002-ctk.vmdk with a last modified date back in October of last year so we removed that. Now after hours we will power off the VM and run a consolidate. Then when finished ssh into the vm host and take a look and make sure there are no 000001 files and no ctk files, if so ensure the vm isn't linked to them and then move them out of that directory.

Finally power the VM back on and see what happens tonight with the backups.

[foggy]

Please keep us posted on how it is going on.