Testing Veeam Replication

[samuk]

Hi,

We are using V8 of veeam backup, and would like to test veeam replication.

Backups run at the moment every evening, and full backup on Friday evening.

Can the Veeam backups run along side the replication jobs? or will this cause issues?

as we are testing, we did not want to disable the backup jobs.

Any other pointers appreciated.

Have downloaded the user guide to read trough later today.

Thanks

[PTide]

Hi,

If you have two jobs trying to either backup or replicate the same VM at the same time, one of the jobs will wait until the first one ends. Please consider using post job script in order to avoid that problem.

Thanks.

[samuk]

Okay, thanks.

I have failed over, made a change and then failed back to production - onto existing VM.

We have 1 Veeam server (proxy) the DR ESX Host is in the same switch (on a different Vlan)

The fail back process is really slow - is this normal?

Code: Select all

28/09/2015 10:27:04          Failback started at 28/09/2015 10:27:04
28/09/2015 10:27:42          Queued for processing at 28/09/2015 10:27:42
28/09/2015 10:27:48          Preparing next VM for processing
28/09/2015 10:27:43          Required backup infrastructure resources have been assigned
28/09/2015 10:27:43          No available proxies are running on ESX(i) management interface subnet. Using proxies from a different subnet, performance may be impacted.
28/09/2015 10:27:48          Using source proxy VMware Backup Proxy [nbd]
28/09/2015 10:27:50          Using target proxy VMware Backup Proxy [nbd]
28/09/2015 10:27:52          Preparing original VM
28/09/2015 10:28:01          Creating working snapshot on original VM
28/09/2015 10:35:26          Calculating original signature Hard disk 1 (20.0 GB) 55% at 0 KB/s

[PTide]

How long does it take in total to failback? Please note, that during the failback to the original VM Veeam B&R needs to calculate and synchronize the differences between the original VM and the replica VM. The time required to scan the VM image depends on its size and is comparable to the full backup time of this VM. Also, in your case this

28/09/2015 10:27:43 No available proxies are running on ESX(i) management interface subnet. Using proxies from a different subnet, performance may be impacted.

means that source proxy pulls the data using network mode across WAN, instead of using hotadd mode and transferring data between proxies. Please check both target and source proxy availability.

Thank you.

[samuk]

Fallback took 15mins.

this was a small VM. Would veeam be okay with a 4TB VM?

Can i install a Veeam proxy on the DR VC? is there any licence implications? as the live environment is licensed.

[PTide]

Can i install a Veeam proxy on the DR VC? is there any licence implications? as the live environment is licensed.

Sure, that should speed up the process. The product components (proxies, backup repositories, backup servers) are not licensed separately. You can deploy as many of these components as you like while licensed to the correct number of sockets. So everything should go smooth.

this was a small VM.

How small was it?

]Would veeam be okay with a 4TB VM?

That depends on your connection and on how much data change will have happened to that VM. For example Exchange failback may take hours.

[samuk]

VM is 20GB.

I have now installed a backup proxy.

VM failed over, copied a small folder no more then 100mb, and failed back.

Process took 20 mins.

Code: Select all

28/09/2015 12:39:28          Failback started at 28/09/2015 12:39:28
28/09/2015 12:40:06          Queued for processing at 28/09/2015 12:40:06
28/09/2015 12:40:11          Preparing next VM for processing
28/09/2015 12:40:06          Required backup infrastructure resources have been assigned
28/09/2015 12:40:11          Using source proxy 10.216.4.51 [nbd]
28/09/2015 12:40:13          Using target proxy VMware Backup Proxy [nbd]
28/09/2015 12:40:16          Preparing original VM
28/09/2015 12:40:24          Creating working snapshot on original VM
28/09/2015 12:59:57          Calculating original signature Hard disk 1 (20.0 GB) 
28/09/2015 13:00:13          Replicating restore point for Hard disk 1 (20.0 GB) 767.5 MB processed
28/09/2015 13:00:16          Creating replica restore point
28/09/2015 13:00:56          Replicating changes Hard disk 1 (20.0 GB) 982.0 MB processed
28/09/2015 13:01:07          Removing working snapshot from original VM
28/09/2015 13:01:09          Powering on original VM
28/09/2015 13:01:26          Failback completed at 28/09/2015 13:01:26

[PTide]

28/09/2015 12:40:11 Using source proxy 10.216.4.51 [nbd]
28/09/2015 12:40:13 Using target proxy VMware Backup Proxy [nbd]

For some reason your proxies are still used in Network Block Device mode instead of hotadd. Please check with this article and the link provided in order to troubleshoot.

Thank you.

[samuk]

Okay looking into the link this morning.

I had just changed the proxy to use automatic mode rather then network.

the Failback is quicker then before...8 mins it took, last try was 15 - as 20 the time before that


Also @ the point of Calculating original signature Hard Disk 1 at 0 KB/S is this normal?

Code: Select all

29/09/2015 10:30:06          Failback started at 29/09/2015 10:30:06
29/09/2015 10:30:46          Queued for processing at 29/09/2015 10:30:46
29/09/2015 10:30:51          Preparing next VM for processing
29/09/2015 10:30:46          Required backup infrastructure resources have been assigned
29/09/2015 10:30:51          Using source proxy 10.216.4.51 [nbd]
29/09/2015 10:30:54          Using target proxy VMware Backup Proxy [nbd]
29/09/2015 10:30:56          Preparing original VM
29/09/2015 10:31:04          Creating working snapshot on original VM
29/09/2015 10:39:50          Calculating original signature Hard disk 1 (20.0 GB) 
29/09/2015 10:40:13          Replicating restore point for Hard disk 1 (20.0 GB) 899.0 MB processed
29/09/2015 10:40:16          Creating replica restore point
29/09/2015 10:40:50          Replicating changes Hard disk 1 (20.0 GB) 698.0 MB processed
29/09/2015 10:40:56          Removing working snapshot from original VM
29/09/2015 10:40:58          Powering on original VM
29/09/2015 10:41:16          Failback completed at 29/09/2015 10:41:16

[PTide]

I suppose that failback went faster this time due to a lesser amount of changes that had to be committed to an original VM. As you can see from this log

29/09/2015 10:30:51 Using source proxy 10.216.4.51 [nbd]
29/09/2015 10:30:54 Using target proxy VMware Backup Proxy [nbd]

your proxies are still using NBD mode. Please force both of your proxies to use Virtual Appliance mode so we can check if it's even available in your case. If it's not available then we will attempt to find the reason why.

Thank you.

[samuk]

Cool, just tried Virtual Appliance Mode - still says NBD - even thought network fail over is disabled.

Will carry on working my way through the link.

[PTide]

Cool, just tried Virtual Appliance Mode - still says NBD - even thought network fail over is disabled.

Looks like something went very wrong - please open a case with our support team so they can take a look at your logs and post your case ID here.

Thank you.

[samuk]

Done

[ID# 01069765]

[samuk]

HI Just an update - think i need to explain the current environment:

2 environments SAN's are not linked.

LIVE Environment
3 ESX hosts (vsphere 5.5)
1 VC (VM)
1 Veeam Server & Proxy(Physical)
3 Par SAN

DR Environment
3 ESX Hosts
Veeam Proxy - Physical
EVA 3PAr SAN

Replication is occurring over the network - currently @ 5MB\s

Have exchanged afew emails with Vitaliy - and hot add is only supported if the Veeam proxy is installed on a VM on both locations.

Waiting for confirmation on this, and i will do this if everything else fits and will work.

have pointed him to this thread.

[PTide]

and hot add is only supported if the Veeam proxy is installed on a VM on both locations.

Correct! It's mentioned in this troubleshooting article:

The Proxy being used to back up the VM must be a Virtual Machine on a host that is able to access the datastore where the source VM’s disks are stored.

[samuk]

Okay, so have installed proxy on a VM in the Live environment & in a VM on the DR environment.

Not sure what is meant by : The Proxy being used to back up the VM must be a Virtual Machine on a host that is able to access the datastore where the source VM’s disks are stored.

how do i give the proxy vm access to the datastores?


Also when i try to use the 2 new proxies to replicate i get this

01/10/2015 11:35:49 :: Task failed Error: No proxy available

So i had to disable the other 2 proxies, now is working

Got this even though proxy installed ok - is this to do with the live site or DR?

01/10/2015 11:44:00 :: No available proxies are running on ESX(i) management interface subnet. Using proxies from a different subnet, performance may be impacted.

[PTide]

Not sure what is meant by : The Proxy being used to back up the VM must be a Virtual Machine on a host that is able to access the datastore where the source VM’s disks are stored.

Your source proxy has to reside on the same ESXi host where replicated VM os located or, in case of shared storage, source proxy can reside on some other host that has an access to the storage. you can follow this guide to check if your infrastructure ready.

Both source and target proxy have to see each other in order to perform data transfer. Also each proxy has to be in the same subnet with ESXi host that has an access to replicated VM's vmdk.

[samuk]

HI,

just a quick update and afew questions.

Hot HDD looks like its working on only 1 esx host - confused as the hosts are identical and so are the datastores presented.



I have setup a replication job to happen every 30mins - this contains 6 VM's

Job log says 35 mins for the job to complete for the incremental jobs (Data transferred 1.2GB)

Little confused to if i can set the job to run every 15mins? or will this not work?

Also seeing source is busy 99% for all vm's being replicated..

Speeds are still poor? most of the time is used up processing?

Code: Select all

08/10/2015 10:00:18 :: Job started at 08/10/2015 10:00:17 
08/10/2015 10:00:19 :: Building VM list 
08/10/2015 10:01:23 :: VM size: 866.0 GB 
08/10/2015 10:01:23 :: Changed block tracking is enabled 
08/10/2015 10:01:25 :: Preparing next VM for processing 
08/10/2015 10:01:30 :: Preparing next VM for processing 
08/10/2015 10:01:30 :: Processing Web2 
08/10/2015 10:06:40 :: Processing Srv01 
08/10/2015 10:06:40 :: Preparing next VM for processing 
08/10/2015 10:12:50 :: Processing SRV02 
08/10/2015 10:12:50 :: Preparing next VM for processing 
08/10/2015 10:21:01 :: Preparing next VM for processing 
08/10/2015 10:21:02 :: Processing Srv03 
08/10/2015 10:25:16 :: Processing Srv01 
08/10/2015 10:25:16 :: Preparing next VM for processing 
08/10/2015 10:29:01 :: Processing king8 
08/10/2015 10:29:01 :: All VMs have been queued for processing 
08/10/2015 10:35:10 :: 1 restore point removed by retention policy from VM Web2_replica 
08/10/2015 10:35:18 :: 1 restore point removed by retention policy from VM Srv01_replica 
08/10/2015 10:35:26 :: 1 restore point removed by retention policy from VM SRV02_replica 
08/10/2015 10:35:38 :: 1 restore point removed by retention policy from VM Srv03_replica 
08/10/2015 10:35:44 :: 1 restore point removed by retention policy from VM Srv01_replica 
08/10/2015 10:35:50 :: 1 restore point removed by retention policy from VM king8_replica 
08/10/2015 10:35:59 :: Load: Source 97% > Proxy 30% > Network 0% > Target 23% 
08/10/2015 10:35:59 :: Primary bottleneck: Source 
08/10/2015 10:35:59 :: Job finished at 08/10/2015 10:35:59

[PTide]

Hot HDD looks like its working on only 1 esx host - confused as the hosts are identical and so are the datastores presented.

That's strange. Please keep working with support on that.

Data transferred 1.2GB<...>Little confused to if i can set the job to run every 15mins? or will this not work?

What's he total size of those 6 VMs?

Also seeing source is busy 99% for all vm's being replicated..

What kind of production datastore do you have and how is it connected to ESXi?

[samuk]

Total size of the VM's is 866GB

Have several VMFS5 datastores, connected to the ESX hosts over Fibre (Round Robin configuration within Vsphere 5.5


Is the source the ESX host or the Veeam backup server?

[PTide]

Is the source the ESX host

This. More specifically - the datastore (btw, what's the vendor, model?). Please check your datastore performance.

[foggy]

Sam, what is the processing rate during the initial (full) job run?

[samuk]

HP 3Par SAN -

processing time for the original job - how can i get this out, as the job report does not tell me.



Veeam support advised the hotadd is not in the scope of veeam support - as its working on one host and not the others. they have still offered to help, however our go live date is today - this is now getting urgent.

2 esx hosts not able to replicate.

I have also logged a call with VMware - who said to review this article: http://kb.vmware.com/kb/1008524

So little confused to what and how to get this resolved...


Any thoughts?

[foggy]

You can use arrow keys in the job stats window to switch between job runs.

Why not to try direct SAN access, since you seem to have physical Veeam B&R server?

[PTide]

Hot HDD looks like its working on only 1 esx host - confused as the hosts are identical and so are the datastores presented.

If two identical things work differentIy then I believe that you should to either double check everything, or escalate the case to get it passed to a higher tier. Do to the latter please click "Talk to a Manager" button on the customer support portal. Sorry, but not much we can do on the forum in your case

[samuk]

Indeed you can move through using the arrow keys..

I will report back Monday. I have requested this case be escalated

I am able to download the VMX if i connect to the host directly using the VI client, but not if i connect to the Vcenter server - vmware say this is by design?

Also noted, replication is not working on the other hosts over the network.

I ran the same incremental job over the network, and it seemed to have worked quicker then hotadd?

09/10/2015 16:19:10 :: Load: Source 96% > Proxy 37% > Network 4% > Target 17%
09/10/2015 16:19:10 :: Primary bottleneck: Source

Took 20mins, rather then 35 mins using hottadd..

Will do more testing over the weekend

[foggy]

Sometimes hotadd can take longer due to overhead required to actually hotadd disks.

[samuk]

Still finding Network mode is quicker then Hotadd... Any thoughts?

How many VM's is recommended but replication job? have 25VM's in total, hoping to run replication every 15mins

Also what do you make of this? Says no proxy on the Managment but has it listed as in use

Code: Select all

12/10/2015 15:19:32 :: Queued for processing at 12/10/2015 15:19:32 
12/10/2015 15:19:32 :: Required backup infrastructure resources have been assigned 
12/10/2015 15:19:32 :: [b]No available proxies are running on ESX(i) management interface subnet. Using proxies from a different subnet, performance may be impacted. [/b]
12/10/2015 15:19:37 :: VM processing started at 12/10/2015 15:19:37 
12/10/2015 15:19:37 :: VM size: 20.0 GB 
12/10/2015 15:19:42 :: Discovering replica VM 
12/10/2015 15:19:43 :: Getting VM info from vSphere 
12/10/2015 15:19:52 :: Creating VM snapshot 
12/10/2015 15:19:58 :: Preparing replica VM 
12/10/2015 15:20:15 :: Processing configuration 
12/10/2015 15:20:22 :: Creating helper snapshot 
12/10/2015 15:20:26 :: [b]Using source proxy MS01 for disk Hard disk 1 [nbd] [/b]    (Source Proxy)
12/10/2015 15:20:26 :: Using target proxy VC01 for disk Hard disk 1 [nbd] 
12/10/2015 15:20:31 :: Hard disk 1 (20.0 GB) 24.0 MB read at 16 MB/s [CBT]
12/10/2015 15:21:01 :: Removing VM snapshot... (100% done)

[PTide]

Have you tried to login into proxy and ping your ESXi's management interface from there?

[samuk]

Yes, and it could not ping one of the hosts - this is now resolved and i am no longer getting the NFC error when replicating.

Have afew more questions:

I had setup afew replication jobs to use the Network mode (as hot add is still not working, need to get more logs to support)

I had an issue with replicating a 1.5TB VM:

14/10/2015 16:57:21 :: Processing Parking5 Error: Transmission pipeline hanged, aborting process.

Guessing this is a network issue?

Also on the incremental relication of 2 terminal servers, and users were not able to login for the duration of the replication.

How well does veeam cope with busy servers & network replication?