V6 - Impossible to backup a certain VM

[stevenrodenburg1]

Hello,

Trying to backup a specific VM but it always fails with the same error. All other VM's are backup without problems. It's just this one, single VM.

Does anyone know what this error means?

"Error: Client error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"

The VM is powered off so my first thoughts of "VSS Agent insertion problems" cannot be the cause. CBT is disabled and VMtools quesce or application-aware-image-processing(VSS) is not configured (not needed on a mostly powered-off VM).

It's a clone of a server i want to make a backup off. It was cloned with the vCenter cloning function.

In vcenter, i see that the snapshot is made succesfully, then the hot-add seems to hang for a while. Then, before hot-add finishes, the snapshot is deleted and the job ends up as failed.

Another thought i had was that with "connected party" the assigned proxy or ESXi server was meant but at the same time other jobs run fine. And it always happens (goes wrong, says: "Error: Client error: A connection attempt failed")

Running v6.0.0.158 (V6 + Hotfix 1)


Disclaimer: i started this topic before. It was slightly different and was deleted as a consequence of forum rules violation. I hope it does not violate them now. I'm not seeking to open a support-case (it's too early for that).
If it is felt that i should open up a case with support, please tell me instead of just deleting it and not saying a word

[Vitaliy S.]

Try using other backup modes (Network), if it doesn't help, then please contact our support team for assistance directly. Thanks!

[Gostev]

Error: Client error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

From the error, it sounds like generic networking issue (connection drop, or firewall blocking communications). I get these once every few weeks with the job report when WAN link between our sites drops while the backup is running (we do remote backups over Atlantic).

[Gostev]

Or may be this.

[stevenrodenburg1]

Just checked: Network-mode runs fine. It only affects Hot-Add Mode and is 100% reproducible.

The VM is a clone, it's 400GB in size (single VMDK). It's powered-off. Job has CBT and application-aware-image-processing(VSS) is not configured (not needed on a mostly powered-off VM).

Before i contact support, where are the logs stored so i can take a look myself?

[Vitaliy S.]

Log files can be found under Help > Support information. Hope this helps!

[Gostev]

Just checked: Network-mode runs fine. It only affects Hot-Add Mode and is 100% reproducible.

Then this is the issue that topic is talking about... no need to contact support

[stevenrodenburg1]

Yeah i looked at the logs and i sure looks like "that" topic indeed.

Never got these problems in V5 though. Backed up monster VM's with Hot-Add Mode. Wierd.

I'll keep an eye on that topic.

[Gostev]

Did you possibly upgrade VMware vSphere recently?

[stevenrodenburg1]

Yes and no. V6 was a complete fresh install on all new VM's. So technically it's not an upgrade.

Update: i just created a new VM (cloned it from the same template as the "problem VM"). I then copied the 400GB VMDK from the "problem VM" to the folder of the new VM and added it to the new VM as it's second VMDK.
So now this new VM has a 30GB VMDK (C: Drive, OS) and 400GB second VMDK.

The VM was not powered on during the backup, just like it's "no i don't wanna be backed-up cousin" who caused me to start this topic.

No problems with backing this guy up...

I noticed that this new VM was backed-up by another proxy. At the moment of writing, i powered-down the other proxies to force the assignment of the proxy #1 which did that new VM succesfully.
And as i'm typing this, the "problem VM" is being backed-up with hot-mode by proxy #1 even though it always failed on proxy #2.

So it seems that a certain proxy, the #2, has a issue with (only) this VM even though all other jobs, both backup and replica, never have a problem with #2. Wierrrrrrrrd....

[stevenrodenburg1]

Update:

Confirmed:
Proxy #1 has no problems backing this "problem VM" up (the VM causing me to start this topic).
Proxy #2 cannot back it up

I deleted proxy #2 from Veeam.
I then removed all Veeam Software from proxy #2
Then i rebooted #2 and had Veeam install the transport-components etc. and register it as a proxy again.

Result: nothing has changed. #2 can still not back this VM up even though #1 has no issues at all. #2 can backup all other VM's.

#2 fails to backup the "new" large VM as well. Same error.

Both proxies are cloned from the same template and had Veeam software installed. Nothing else was done to either of them. I checked them and they are still the same.

I'm lost.....

[Gostev]

Sorry, I don't quite get your response. Did you upgrade VMware vSphere recently? Your replied "Yes and No"

As for the proxies, most likely, Proxy #2 uses hot add processing mode - and Proxy #1 does not, which is why it works. Anyway, don't bother playing with Veeam components - as "that topic" explains, this is VMware hot add functionality issue, and it is reproducible even without Veeam present in the environment.

[stevenrodenburg1]

"Yes and no" meant: Yes we upgraded from v5 to v6 but No, it was not a real upgrade because V6 was installed from scratch on all new VM's and everything.

Both proxies use hot-mode (they are configured for Virtual appliance mode and failback to network-mode is disabled).

These two large VM's did not exist in the V5 era but much larger ones did (which do not exist anymore now btw.)


In the logs, from jobs of these large VM's backed-up on #2, i see lines like "Client error: Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown. (System.Exception)".
As far as i'm capable of interpreting those logs, connections then start to die off (explaining the "A connection attempt failed because the connected party did not properly respond after a period of time" errors).

[Gostev]

But I have been asking about VMware vSphere upgrade, not about Veeam B&R upgrade?

Yes, this is expected error caused by long VM stun (so long that OS automatically drops TCP/IP connection after set amount of keep-alives).

If both proxies are using hot add, and one of them works, this means that you simply never backed up the proxy VM that works with Veeam, so it still does not have CBT enabled in the VM settings. As "that topic" explains, the issue sits in VMware CBT logic implementation, and how it works together with hot add.

I think it will help if we lock this topic and move to the original discussion, otherwise we will keep wasting our time discussing things that are already explained there (for example, all scenarios when the issue surfaces). And even if we find some new scenario or something we do not know, it would be good to have everything in one place.

[JWester]

I also have several VMs which I cannot backup with hotadd:

* Proxy tries to "hotadd" the VM disk
* Proxy is unreachable (tested with ping) for some time
* error message "A connection attempt failed because the connected party did not properly respond ..."
* VM backup failed
* backup with NBD works fine

I can narrow it down to VMs with non-Windows/Linux filesystems on the disks. E. g. FreeBSD UFS, Oracle ASM. Probably the Windows-Proxy "freezes" when those disks are added.

Case with Veeam is open.

[Gostev]

I also have several VMs which I cannot backup with hotadd:

* Proxy tries to "hotadd" the VM disk
* Proxy is unreachable (tested with ping) for some time
* error message "A connection attempt failed because the connected party did not properly respond ..."
* VM backup failed
* backup with NBD works fine

I can narrow it down to VMs with non-Windows/Linux filesystems on the disks. E. g. FreeBSD UFS, Oracle ASM. Probably the Windows-Proxy "freezes" when those disks are added.

Case with Veeam is open.

No real need for support case, as this specific issue is covered in the Known Issues sticky topic in the main forum.

[JWester]

Support helped me.
Deleting proxy CBT-files and excluding it from backup resolved the problem.