Does anyone have replication with Veeam6 functional?

[vertices]

We have been trying for weeks to get this to work. Everything was fine with Veeam5. Upgraded to Veeam6 and nothing but nightmares. So far have got nowhere with my support case (ID#5172531) so posting here in hopes anyone has any suggestions. All ESX hosts are ESXi 4.1. Veeam 6 is patched with latest patch as of a couple days ago.

We have 2 sites with a 20mbps pipe between them, I'll simply call them source and destination. Source site has 3 ESX hosts connected to a SAN. Also has a physical Veeam server that only performs backups. All servers are joined to the same Windows domain and all are using the same domain admin credentials. This physical Veeam server has no problem backing up the VMs in the source site. It backs them up between 5:30pm and 8:00pm. Everything is perfect with this.

In the destination site we have another physical Veeam server with 6 core, 16GB of RAM, nice system, and there is a single ESXi host connected to a small iSCSI SAN. The Veeam server has a fresh install of Veeam 6 on it (not upgraded from 5). We also have a fresh dedicated 2008R2 VM at the source site I'll call prox1. The replication jobs on the Veeam server in the destination site are configured to use prox1 as the source proxy, and the local server as the destination proxy. I have it set to use existing replicas as a seed and the automap just fine. Everything is left to automatic, other than which proxies to use.

So 2 Veeam servers, identical configs. One is for backing up in the source site and works fine, the other is for pulling replicas to the destination site which doesn't work at all.

We are plagued with nondescript failures such as "An existing connection was forcibly closed by the host" and "An established connection was aborted by the software in your host machine".

I watched the jobs run one night and saw some things. Almost always when I see a failure with "An established connection was aborted by the software in your host machine" it fails at the snapshot removal point, either during or just after. I also sometimes see an error in VMware about "Unable to access file <unspecified filename> since it is locked" in regards to a snapshot removal. However using the same credentials that Veeam is configured with, I can manually remove it just fine.

I have done everything support has wanted me to do, recreating jobs, deleting all vbk files in the replica repository. Nothing seems to improve the situation at all. I am strongly considering deeming Veeam6 not ready for production and going back to 5 which worked just fine, albeit not very efficient.


Does anyone have replication working or any suggestions for us? So far we have had our case open for 9 days, and no DR site for 14 days. I can't let this go on much longer without dropping down to Veeam5 to fix things.

By the way my case# is 5172531in case anyone from Veeam chimes in and wants to take a look.

[SunkistDavid]

Is the remote DR ESXi server added in your server list? Check its credentials since its an ESXi server? Are you using root or the equivalant for username?

Also try

Please connect to the host\VC that the guest OS sits on WITH Vsphere FROM the Veeam Server USING the soap connection provided to Veeam for connection to the above host\vc and try downloading server.nvram from the same datastore that the guest is on to the Veeam server. If this does not work its a permission issue as the remote proxy is calling back to the local Veeam server and doesn't have the correct permissions.

Also please verify that port 902 is open open on the VC\Host and Veeam Server

[SunkistDavid]

Also if you are unable to download the nvram then contact support and inform them of case number 5167894 to use as reference to help you finish solving problem.

[vertices]

All ESXi hosts are licensed hosts and part of the same vCenter.

These VMs do occasionally successfully replicate. Like maybe once a week each VM is randomly successful which leads me to believe it's not a permissions issue being that it does sometimes work. It just usually doesn't. To be honest, I didn't quite follow 100% what you want me to do in regards to:

"Please connect to the host\VC that the guest OS sits on WITH Vsphere FROM the Veeam Server USING the soap connection provided to Veeam for connection to the above host\vc and try downloading server.nvram from the same datastore that the guest is on to the Veeam server. If this does not work its a permission issue as the remote proxy is calling back to the local Veeam server and doesn't have the correct permissions."

[SunkistDavid]

Connect to your remote Veeam Server. From that server connect to your ESX server that is being backed-up locally and download the VIClient. Run the VI client and connect to the local host, browse the datastore of the VM that is being backed-up, select the nvram file and select it to be uploaded to your local file. If it errors out then its a permission issue. This is bug VMWare found. BUT I don't think this is your issue as you indicate they do occassionally replicate.

Do you have a firewall between the sites that may be closing down the connection?

[kellison]

I had the same problem when I upgraded to Ver 6. I was trying to replicate VM's from my production site over to the DR site and sometimes it would work but most of the time it would not and I saw the same type of errors. I also tried everything under the sun with no luck.

But - instead of running the replication from my production Veeam server using the "push" method, I setup the replications on my Veeam server over at my DR site and have not had a problem since.. Guess here you can say it uses the "pull" method.

Both my Veeam server's are VM's, Windows 2008 R2 with 16GB of RAM and 8 CPU's allocated to each. My backups running at both sites are working fine. Maybe try to run the replication from your DR site????? Thanks Ken >

[marty9876]

No, and I'm completely sick of this. I'm really close to throwing in the towel and being done with Veeam.

Bought matching firewalls for VPN tunnels, sunk the money into a DR ESXi host to replicate to in a colo and it's been a complete bust. $15k+ and hundreds of hours of my time during nights and weekends to get this working. Just a total failure.

Well back to seeding a USB drive to ship off to the colo since I can't replicate over WAN or use the seed I shipped the server with. What's that they say about doing the same thing over again expecting a different result...rhymes with Veeam I think...

[SunkistDavid]

Marty, I have a pretty complicated network at our DR site that includes 3 different subnets with two matching my primary site but blocked by FW. It took awhile but its working flawlessly. Take a deep breath and by the way I don't work for Veeam. Check your firewall and make sure there is a rule with a route back from the Veeam Proxy at the remote site to your primary site. Also did you test the NVRAM process that I recommended. This is what help me determine I had a firewall issue between remote site and primary site.

[marty9876]

For me I've had a few different problems to address. Yes, networking was an issue. Brand new Watchguard XTM's have a bug in the latest version which stops the Veeam SOAP traffic at a given point.

I setup a virtual pfsense firewall now (on the host) and use that for a VPN endpoint. The small replication jobs are ok I think, the large job fails every time. Sometimes after 36 hours, sometimes after 48 hours (I now have the latest Veeam patch 4 with the 48 hour bug fixed) and then throws the error. Most of the time bombes out after a few hours. I had to grow the drives on the big job (600 GB VM) and that seems to have invalided the seed because things blow up after the first pass of the seed job (inflates from backup, the actual replication pass to grab the changes).

It's been 2.5 months of just grinding away at this. I've had firewall bugs, Veeam bugs and shot myself in the foot bugs all stacked on each other.

I was able to grab the nvram file no problem.

[vertices]

Ok well now I'm being told to contact VMware to find out what is going on. Which strikes me as odd as we've had no VMware troubles at all, and again, everything was fine with Veeam5. All the troubles started the day we put in Veeam6. What a mess. I don't think my client is going to enjoy Veeam anymore after all of this work at $150 an hour with no resolution in sight. I wouldn't be surprised if he ditched Veeam Enterprise altogether.

[SunkistDavid]

What is the reason given that it was pushed to VMWare?

[vertices]

"If that's the case (and it was what I was afraid was going to happen), then I'll have to request that you contact VMWare directly to have them look into what is happening on the host. Something is being triggered in the host that is making it drop the communication."

Which I also find odd because a Veeam6 server at the source site has NO PROBLEM backing up all VMs that are having replication problems. I mean VMware as well as Veeam6 are working perfectly together in the source site. It's only when I put Veeam6 in a destination site and tell it to replicate instead of backup that it fails, and even then, it worked fine with Veeam5.

[marty9876]

I feel like the client, at a certain point I need to respect the business needs and move on to something else. Regardless of it's the products issues or my ability to get the product working, at the end of the day the business's needs are not being met.

[vertices]

I wonder if there is any possibility that the seeds are not working out right due to the fact that they being created by a different Veeam 6 server.

I think I'm going to try one last thing. We have a new physical Veeam server at the source site for backups that is not in production yet. I'm thinking I will make this server simply a proxy and repository and have the jobs themselves managed by the Veeam server in the destination site. If I can get that working with backups then I have completed isolated it to everything fully functioning with the single Veeam Management server. Then I will use those same backup jobs as seeds. This way the Veeam server in the management site controls all jobs, both backup and replication, and if backups work but not replication it will be easier to point direclty at Veeam without this runaround and trying to push me back to VMware.

[Jfmoots]

Rob,

Check your PMs.

I'm currently working a problem with similar traits and I'd like to compare notes to see how similar they really are.