Host-based backup of VMware vSphere VMs.
Post Reply
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

6.1 Hot Add issue

Post by lobo519 »

I have noticed an issue with the hot add – it hangs (10-15 minutes) when trying to mount the snap shot to the proxy server for several minutes. There is no task in vCenter to reconfigure the virtual machine while its hanging. I see the same type of delay in the cleanup tasks ie. reconfiguring to remove the snapshot and Removing the snapshot from the VM

Support ID:5199329
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 6.1 Hot Add issue

Post by Gostev »

Well, removing snapshot does take long time due to data commit to VMDK, this is just how VMware snapshots work.
Hot add taking 1-2 minutes is almost normal (it is pretty slow operation), but 10-15 minutes is not normal.

Anything obviously wrong about VM which disks are being attached (many existing snapshots), or the issue is for every VM?

I am willing to bet the issue is not 6.1 specific, but let's see what support finds.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: 6.1 Hot Add issue

Post by lobo519 »

Not the time it takes to run - The task in vcenter literally doesn't show up for 10 - 15 minutes - its happening on every vm

I might be on to something - possibly this behavior starts after doing a file level restore.
I was able get it running normally again after a reboot.

I will do more testing tomorrow and I too am interested what support has to say
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: 6.1 Hot Add issue

Post by lobo519 »

I did a few tests today - The issue definitely presents itself after a file level restore and a restart fixes it.

With a replication job, it only affects the source Proxy/vCenter (My Veeam Server also). The destination runs all the vsphere tasks in a timely fashion. I have sent an update to support but they haven't even looked at my logs yet.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 6.1 Hot Add issue

Post by Gostev »

Then you should be able to easily work around this by setting up a proxy on a different VM, and then disabling the proxy on your backup server.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: 6.1 Hot Add issue

Post by lobo519 »

Gostev wrote:Then you should be able to easily work around this by setting up a proxy on a different VM, and then disabling the proxy on your backup server.
Working around it isn't really an issue - The much easier solution (for me) is to just restart the server after a file level restore (not a frequent task).

Regardless - I felt it deserved attention.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 6.1 Hot Add issue

Post by Gostev »

It definitely does worth investigation.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: 6.1 Hot Add issue

Post by lobo519 »

The latest from support is that VM Tools not being up to date is causing this issue?? They requested that I try doing the tools upgrade on a VM that is in an affected job.

No sure I understand how this could really be the issue..
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: 6.1 Hot Add issue

Post by Vitaliy S. »

Hmm...If server reboot resolves the issue, then I do not understand how VMware Tools version might affect HotAdd operation either.

As a further troubleshooting step could you please create a brand new "HotAdd" proxy server (any existing Windows server will work fine) and run a test backup job with one of the affected VMs? It would help us to narrow down this issue either to the backup proxy, backup server or infrastructure itself.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: 6.1 Hot Add issue

Post by lobo519 »

When I tried to move one of my jobs to new proxy (windows 7) I got the following error " Error: Client error: VDDK thread terminated." after "Preparing replica VM"

Moved it back to my original proxy (2008 R2, Backup Server)and it works fine. There's nothing special about the new Windows 7 proxy, it just running SpiceWorks and I turned the firewall off.

Any thoughts?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: 6.1 Hot Add issue

Post by Vitaliy S. »

Have you informed your support engineer about this error? Unfortunately, it's hard to say what could have caused this error message without looking at the related VM processing steps. Could you please update your ticket with the latest info and let our support team investigate it for you?
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: 6.1 Hot Add issue

Post by lobo519 »

will do - thanks
Jamie Pert
Enthusiast
Posts: 68
Liked: 2 times
Joined: Jun 14, 2012 10:56 am
Full Name: Jamie Pert
Location: twitter.com/jam1epert
Contact:

Re: 6.1 Hot Add issue

Post by Jamie Pert »

Did you ever fix this lobo?
@jam1epert on Twitter
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: 6.1 Hot Add issue

Post by lobo519 »

still working on it - just got off the phone with level 2 - looks like it maybe anti-virus related...
Jamie Pert
Enthusiast
Posts: 68
Liked: 2 times
Joined: Jun 14, 2012 10:56 am
Full Name: Jamie Pert
Location: twitter.com/jam1epert
Contact:

Re: 6.1 Hot Add issue

Post by Jamie Pert »

Interesting - I'm not sure if it's related but a very low-end server is being used as a proxy and is throwing-up that error, we're trying to determine whether this is the sole reason for the VDDK error, or if something else may be causing it.

The proxy runs McAfee, but it's disabled.
@jam1epert on Twitter
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: 6.1 Hot Add issue

Post by lobo519 »

Jamie Pert wrote:Interesting - I'm not sure if it's related but a very low-end server is being used as a proxy and is throwing-up that error, we're trying to determine whether this is the sole reason for the VDDK error, or if something else may be causing it.

The proxy runs McAfee, but it's disabled.
sorry - I didnt know you were referring to the VDDK error - I was referring to the original issue. No I did not get anywhere with that error.
RPatel
Novice
Posts: 5
Liked: never
Joined: Aug 22, 2012 4:22 pm
Full Name: Rajesh
Contact:

Client error: VDDK thread terminated

Post by RPatel »

Hi,
My Veeam backup is no longer running "Client error: VDDK thread terminated"
Please advise
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: 6.1 Hot Add issue

Post by Vitaliy S. »

Rajesh, with all technical issues please contact our support team for assistance. It is almost impossible to troubleshoot technical issues via forums without reviewing full debug log files. Thanks!
RPatel
Novice
Posts: 5
Liked: never
Joined: Aug 22, 2012 4:22 pm
Full Name: Rajesh
Contact:

Re: 6.1 Hot Add issue

Post by RPatel »

How do i go about logging technical issue?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 6.1 Hot Add issue

Post by foggy »

You can open a case using our support portal (step-by-step instructions are also provided when you click New Topic).
RPatel
Novice
Posts: 5
Liked: never
Joined: Aug 22, 2012 4:22 pm
Full Name: Rajesh
Contact:

Backup Failing Due to VDDK Thread Terminated

Post by RPatel »

I have logged the support call since yesterday and still waiting for Support:(
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 6.1 Hot Add issue

Post by foggy »

Rajesh, as I can see from your support case, you already have been contacted. Please note, that support is provided according to our support policy, where response times are specified depending on your support level (Standard or Premium) and case severity. Thanks.
maikelfalcon
Lurker
Posts: 1
Liked: 1 time
Joined: Oct 24, 2012 9:06 am
Full Name: Maikel
Contact:

Re: 6.1 Hot Add issue

Post by maikelfalcon » 1 person likes this post

We've had this error come up. The problem here was that the Veeam proxy server couldn't reach our Veeam server on FULL dns name.
On the proxy server we could ping HOSTNAME but not HOSTNAME.DOMAIN.COM. (We recently did a rendom on our DNS servers).
After adding HOSTNAME.DOMAIN.COM IP-adrress to the hosts file on our Veeam proxy server, the backup started functioning again.
Post Reply

Who is online

Users browsing this forum: No registered users and 94 guests