VDDK Thread Terminated

[rubeng]

Hi,

One of our customers has recently bought a VMware ESXi 4.1 U2 host and a Windows Server 2008 R2 running Veeam Backup & Replication 6.5. He wants all of his VMs replicated to the new VMware ESXi. These two servers is stored in one of our datacenters for Disaster Recovery. They are placed in a removable rack in case of a disaster, and the servers need to be moved to another location.

His VMs are inside our Datacenter, but as I mentioned, if anything happens, he wants the opportunity to bring all of his VMs to another location if necessary.

So to do this I created a replication job on our main Veeam Backup & Replication server, and chose the remote/new Veeam server as both Source and Target Proxy. But when the remote/new Veeam server is chosen as Source Proxy, the error code "VDDK Thread Terminated" appears. If I set the Source Proxy back to our main Veeam server, it works brilliant, but then all the load is on the Source Proxy, and not our remote/new Veeam server.

We want our main Veeam server to be the brains, to report problems and run all the jobs. But the remote/new Veeam server is gonna take care of the heavy work. That should be able to work right?
I've added our Virtual Center on the remote/new Veeam, and I've added our main Veeam server on the remote/new Veeam. I've also added the remote/new Veeam server and the new VMware host on our main Veeam server.

Edit: Forgot to mention that all servers have been added by IP, and not hostname/dns.

Is there anyone who knows what might cause this error?

[Vitaliy S.]

Hi Ruben,

If you want to replicate within the remote ESXi hosts, why do you need a secondary Veeam backup server? Why can't you just use backup proxy server on the DR location?

Thanks!

[rubeng]

Hi Vitaly, thanks for answering me this fast.

I probably didnt explain myself clear enough! Look here:

Lets call our main Datacenter for Datacenter 1, and our second for Datacenter 2.

Datacenter 1 is located in a basement in a building, and in Datacenter 1, all our ESXi hosts, SAN and Primary Veeam Backup & Replication Server are located. This is where all our customers VMs are hosted.
Datacenter 2 is located in the same building as our office, a different than Datacenter 2. There we have 3 ESXi host, for our internal systems. No customers VMs are located in Datacenter 2.

And our customer wants us to put up a small rack with wheels, containing an ESXi host and a Veeam Backup & Replication Server, for Disaster Recovery. He also wants a Veeam server so that he can easily restore if something happens to OUR systems. (Note: We have our own DR-site and all this, but he wants an exclusive solution just for him!)

So what we want to do, is to use OUR primary Veeam Backup & Replication Server to run replication jobs, and replicate all of his VMs inside Datacenter 1, to the rack on wheels in Datacenter 2. This is because if anything should happen, he can just move that rack out of the building and to another location. The Veeam server in Datacenter 2 is a really heavy server, so we want that server to do all the work.


If all this is necessary is not for me to decide, we are just doing what the customer ask us to do!


Hopefully this explains the situation a litte better


Thanks!

[rbrambley]

rubeng,

I'm not sure what the VDDK thread terminated means. My guess is that something broke Veeam's ability to access the VM files on your host's datastores. This could be as simple as the Proxy system's hostname resolution (yes you used IPs but VMware is still heavily dependent on name resolution and the proxy is getting a VMware object's name via the API - make sure your Proxy can resolve all ESX hostnames on all sides), ESX vCenter permissions or it could be an external issue like a network or firewall blocked ports. If you continue to experience this please contact Veeam Support so they can "log dive" and hopefully pinpoint what is happening when it happens.

Second topic is your replication design.

The Veeam Proxy is the piece of the architecture that does the work. Vitally suggests you don't need a whole other Veeam install, but instead just to create a windows VM on your DR hosts and add that system as a Veeam Proxy. Use this Proxy VM as your source and target. Simple design that accomplishes what you want - all load is on your DR hosts except for the Veeam server.

However, when I think about a DR event and fail over scenario I lean back towards you using a Veeam server installed on your DR systems. Different from what you are trying now, I would create my replication jobs on the DR Veeam and do not add/include the primary Veeam server to it. Keep it completely independent of your primary environment as well. In fact, don't even join the domain with the DR Veeam server.

BTW, if you are not doing any backups you don't even need a primary environment Veeam server. Just go with the single install on the DR side.

Here's some reasons why replication jobs should be run from Veeam on the DR hosts:

- if you lost your Veeam server (with the jobs) in the DR event you will only have the option to start VMs with the vSphere client. A second Veeam server that did not run the jobs is no use.

- Starting VMs with the vSphere Client will cause you to have to recreate your replication jobs once all is fixed again. Having the Veeam server that runs the jobs means you will be able to preserve your jobs and even sync the changes made back to the source vms. Always fail over with the Veeam wizard.

- You can even do test fail overs by isolating the DR rack. Once the rack is isolated use the DR side Veeam to fail over, check everything, and then fail back without saving any changes - without taking the source VMs offline.

- having the Veeam server around also eases the ability to choose from multiple restore points for fail over

- You can re ip, change vSwitches, do file level recovery from guest VM files (yes, even from replicas) if ever needed.

- if you need to take advantage of our Proxy to Proxy data compression between sites you can always deploy a windows VM as a proxy on the source side, but it doesn't sound like you need this now.

This design accomplishes putting ALL the load on the DR hosts, but running the jobs from a Veeam server on the DR side also gives you so much more.

[rubeng]

Hi rbrambley,

Your thoughts on VDDK sounds very correct. We have a lot of VLANs and different networks in our environment. And we have only opened traffic between the DR-site and to Virtual Center, not all ESXi hosts.

And your suggestions around our design is probably the best way to approach this. The only reason we want our Primary Veeam is because its easier to monitor the job and make sure everything works. Your reasons for running the job on the DR Veeam is true, and seems to be the best solution considering all the possibilities you miss out on if you run the job on our Primary Veeam. We decided to keep the DR Veeam outside the domain, to keep it independent, like you mentioned here.


Thanks for taking the time to answering me and provide me with this information. I will bring this to my colleagues on monday, and we will most likely design this the way you described here.


Best Regards,

Ruben.

[rbrambley]

Don't forget to use the Veeam Enterprise Manager (EMS) to monitor both your Veeam Servers. It's included when you bought Veeam. it's the "web server" (not sure of exact file name) installation file in the download. I would add IIS to your production Veeam VM and then install the EMS there. You can then go to the EMS web portal, add both Veeam servers in the configuration section, and set global alerts and reports.

Check out this 1 min lesson from our free Online University on working with the EMS here: http://www.veeam.com/university/emeu.swf

Veeam has you covered!

[zak2011]

i am getting error " client error vddk thread terminated". This happened after VMs have been storage vmotioned to the new SAN.
Any particular reason why this could be happening.

Thanks

[foggy]

Arun, have you opened the case for this issue already? It's hard to say what could have caused this error without looking at the VM processing steps outlined in the log. Thanks!

[zak2011]

Will open the case and post the results then.

Thanks

[zak2011]

I have opened the case. The technician said there was a patch for this and tried installing it. However the same error keeps coming. After couple of retries, the job completes.
Another SE might have a look into it today.

[zak2011]

Support resolved the issue.
1) The VBR database contained some reference to duplicate hosts. This happened when a copy of the exisiting virtual vcenter was restored to another host and vcenter was run from the new copy. So the database was cleaned up.
2) Out of the four proxy servers used, one of the backup proxy servers had dns resolution problems. This proxy vm had to be temporarily disabled.

After these two steps were done, the jobs are all working well!

Thanks,
Arun