Replica ran out of space - Corrupt Files on Source VM?

[Slappy3243]

Hello,

Please help me understand this as I am lost. The latest replica of a particular VM was created on a datastore which ran out of space. That caused a number of problems but they will not be addressed here. Anyway, while this was occurring, we started to receive complaints from users on the source VM that they were unable to open files such as PDFs because they were receiving error messages that the files had become corrupt. Luckily we had a local backup of the source VM to restore the files from, but this is concerning. Why would the source VM, not the replica, start to have corrupting of files even though only the replica ran out of space on a totally different datastore than where the source VM is located? Could this have something to do with how VMWARE does the Quiescence of the files? This is concerning to me since the source VM is in production. Any ideas? Thanks.

-Adam

[Vitaliy S.]

Hello Adam,

I've never seen a behavior like that, could you please clarify a couple of questions:

1. When did your users begin complaining about files corruption? Did it happen right after the job failure when VM started snapshot consolidation process?
2. What about other files and applications installed inside the VM?
3. Also is it a Windows VM? Were you using application-aware image processing?

Thank you!

[Slappy3243]

Hello,

1. The users began to complain about the inability to open files as the replica was still running, before it officially failed. The problems seemed to be isolated to one shared folder with many sub-folders which all inherited the problem. The files contained within these folders were a mixture of PDF's, Excel Worksheets, and other similar documents. When a user would try to open a file the appropriate program would open but then complain that the file is in an unrecognizable format or corrupt. It seems like the files were encrypted or compressed which is why the files couldn't be opened or recognized correctly.

2. As far as I can tell, no other files or applications were exhibiting this strange behavior. Many people use this VM so if something was wrong we would have likely heard about it from the end-users immediately.

3. It is a Windows VM. It is Windows Server 2003 R2 Enterprise w/SP2. We are not using application-aware image processing for this job.

Thank you for your help!

-Adam

[Vitaliy S.]

1. Veeam is using vStorage API to backup the VMs and does it on image level, not sure what have caused the issues with this particular shared folder... Is there anything special about this share? Have you had a chance to review Windows Event log and VMware VM logs (located next to VM configuration file) at the time when the issue occurred?
3. Why not? Using application-aware image processing is considered to be best practice for all Windows VMs.

[Slappy3243]

1. Veeam is using vStorage API to backup the VMs and does it on image level, not sure what have caused the issues with this particular shared folder... Is there anything special about this share? Have you had a chance to review Windows Event log and VMware VM logs (located next to VM configuration file) at the time when the issue occurred?
3. Why not? Using application-aware image processing is considered to be best practice for all Windows VMs.

Hey,

I don't see anything strange in the Window's Event logs. Could you please elaborate a bit further on how to view the VMware VM logs? Thank you.

-Adam

[foggy]

Adam, I believe VMware support could better assist you in reading VM logs. Thanks.

[Vitaliy S.]

VM logs (*.log) are located in the datastore where VMX file is located, you can review it with a simple notepad. If you don't see anything there, you might want to ask assistance from the VMware team as foggy has suggested.

On the other hand, I doubt that you will find any Guest OS related information there (it was just a guess), Windows Event logs should have the details on the issue occurred.

BTW, is there any difference between the share you had file issues with and other shares?