Slow Replication of a Powered Off VM

[markfellows]

I’m in the process of migrating our small infrastructure to new hardware using Veeam replication.
This is the process I’ve been undertaking:

1. Replicate VM from source infrastructure (Veeam 6.1 ) to new infrastructure storage (Veeam 7.0 R2).
2. Outside business hours (same day as initial replication) power down the source VM.
3. Replicate once more.
4. Failover to new hardware.

The timings for replicating the powered off VM are not what I was expecting though. The example below is the timings from a small (100Gb) VM.

VM powered on (3Gb transferred) - 20 minutes
VM powered off (500Mb transferred) – 45 minutes (This was performed about 2 hours after the previous replication)

The only difference in the entire environment between jobs is the power status of the VM.
Is this normal or would people be expecting replication of the powered off VM to be quicker especially with a minimal amount of changes?

Would switching CBT and Application Aware processing off on the power off VM job make any difference?

Just curious as I’m now moving on to some larger VMs so I was attempting to estimate the time.

P.S I know Quick Migration is an option but I like knowing if/when the server will be down. We have no storage vMotion and with one of the older servers there may be a CPU compatibility issue, ruling out SmartSwitch.

Thanks.

[foggy]

Mark, job stats and probably logs should be reviewed more closely to identify the reason of this behavior (i.e. which operation took most of the time). Btw, is the same transport mode used during both job runs?

[markfellows]

Alexander, I'd already checked this, it seems to be on the reading of the disk.

VM powered on – Disk 1 (55GB) 52.4GB read at 68 MB/s – 13:11, Disk 2 (40GB) read at 85 MB/s -8:02
VM powered off – Disk 1 (55GB) 52.4GB read at 42 MB/s – 21:29, Disk 2 (40GB) read at 48 MB/s -14:08

Transport the same for both.

I am right in thinking with the same (or less changes) the powered off VM should be quicker?

[Gostev]

Just to throw this in here. Back in time, CBT was not supposed to work with powered off VMs due to a VMware limitation. I've never actually seen it NOT work for powered off VMs though, even at that time. Perhaps this was limited to some ESX(i) versions. What is your source infrastructure?

[veremin]

I'm wondering whether the proxy was the same in both cases. Thanks.

[markfellows]

Anton & Vladimir, thanks for the reply. Both answers are definitely plausible.

In the source infrastructure it’s Esxi 4.0 (Veeam 6.1 backup server/proxy)

In the destination environment it’s Esxi 5.1 (Veeam 7 R2 backup server/proxy)

[foggy]

What are the full bottleneck stats and data read/transferred counters for both runs? Are those you posted above from the consecutive full and incremental job runs? And just to confirm, is the same pair of proxy servers selected during those runs and what transport mode do they use (nbd or hotadd)?

Have you already contacted support to investigate the logs?

[markfellows]

VM Powered On [NBD]
Source 99% > Proxy 40% > Network 0% > Target 8%
Processed 95GB
Read 92.4GB
Transferred 2.3GB

VM Powered Off [NBD]
Source 99% > Proxy 21% > Network 0% > Target 8%
Processed 95GB
Read 92.4GB
Transferred 312.4MB

I've not logged the issue, wanted to ensure it was a valid issue first.

[veremin]

You might want to do so, as we can spend weeks of guessing via the forum correspondence. The support team, in contrast, will be able to understand the root cause, once the infrastructure/logs are analyzed.

Thanks.

[Gostev]

VM Powered On [NBD]
Processed 95GB
Read 92.4GB
Transferred 2.3GB

VM Powered Off [NBD]
Processed 95GB
Read 92.4GB
Transferred 312.4MB

This Read values indicate that neither powered on, nor powered off VMs processing leverages CBT. So, your issue is not caused by the fact that the VM is powered off after all. Are there any indication in the VM session log on why CBT is not being leveraged on this VM? If not, debug logs investigation will be required.

[markfellows]

I hadn't noticed that both were NBD (until I had a look for the last post), which makes sense as the destination host for this particular VM doesn't have a proxy. I'll dig into the logs and post any finding/results back.

[markfellows]

I migrated another couple of VMs on Saturday. As before when replicating the powered off VM this took longer than expected. It completed ok but with a warning (Cannot use CBT: Soap fault….).

Just about to delve deeper into the logs but I was thinking for my final VM migration (a small Exchange VM about 300Gb) I may try one of the options below as an alternative (if any). Which would be best?

1. Initiate the replication job from the destination side (Veeam 7 R2) using the Veeam 6.1 VM in the source environment as a proxy. I could even ignore the proxy if there’s a chance this is causing the issues.

2. Leave the VM on and perform the initial replication. Shutdown the Exchange services (to ensure application consistency) and then replicate again while the VM is still on. Then shut down the VM and fail it over to the new infrastructure.

I may not have time to wait for this to be resolved through support hence the questions here. As an aside the second VM from the weekend was just replicated (while on), shutdown and then failed over. This was just a small file and Intranet server and this method worked fine.

As I mentioned before I do like Quick Migration (and have used it before) but for certain servers I prefer control over the potential reboot (due to CPU mismatch).

[foggy]

I migrated another couple of VMs on Saturday. As before when replicating the powered off VM this took longer than expected. It completed ok but with a warning (Cannot use CBT: Soap fault….).

Well, this clearly indicates that CBT was not used during this job, hence longer job duration.

1. Initiate the replication job from the destination side (Veeam 7 R2) using the Veeam 6.1 VM in the source environment as a proxy. I could even ignore the proxy if there’s a chance this is causing the issues.

Actually, you cannot use Veeam B&R components of previous versions with the newer installation, all components are upgraded to the Veeam B&R main console version automatically.

2. Leave the VM on and perform the initial replication. Shutdown the Exchange services (to ensure application consistency) and then replicate again while the VM is still on. Then shut down the VM and fail it over to the new infrastructure.

Pretty common migration scenario is as follows: perform the initial replication, shutdown the VM, perform final replication of the latest changes, and then perform failover.

[markfellows]

Actually, you cannot use Veeam B&R components of previous versions with the newer installation, all components are upgraded to the Veeam B&R main console version automatically.

This may be an issue as the original Veeam server is a 32bit Windows 2003 server.

Pretty common migration scenario is as follows: perform the initial replication, shutdown the VM, perform final replication of the latest changes, and then perform failover.

This is what I had been doing but it's taking a bit too long with this CBT issue.

Presuming it is a CBT issue, what's the best fix? Keep it in mind that the old Veeam server in the source infrastructure is Veeam 6.1 on a 32bit Windows 2003.

[veremin]

This may be an issue as the original Veeam server is a 32bit Windows 2003 server.

Currently, only target (repository) component is 64-bit, while source one (proxy) is still 32-bit. However, the problem is that existing components (6.1) will be updated to the latest version (7.0). This, in its turn, will most likely result in inability to use 6.1 installation any longer.

Presuming it is a CBT issue, what's the best fix? Keep it in mind that the old Veeam server in the source infrastructure is Veeam 6.1 on a 32bit Windows 2003.

You might want to open a ticket with VMware and see whether they are able to address CBT issue.

Thanks.

[Vitaliy S.]

You may want to disable/enable CBT manually, might do the trick.

[GabesVirtualWorld]

Was there a solution? Because we're experiencing the same. Replicating 5 TB without issues and pretty fast over multiple jobs. But as soon as the VM goes down for a last replication, I get the CBT warnings and the time of the job increases. It more than doubles the time.

[foggy]

OP hasn't provided his support case ID, so no way to check. Better create your own one for deeper investigation.

[markfellows]

I never logged it in the end as I didn't think it could be resolved in time for the migration. I do however think I've found our issue (running final migration this weekend to confirm this).

It seems to be an issue with the Veeam Proxy configuration, I’ll try and explain but it’s a bit complicated. The server we’re having an issue with is an Exchange server (we’ll call this EX1). There used to be a Veeam Proxy (we’ll call this server VP1) on the same local storage (on a host we’ll call HOST1) as EX1. A few weeks back we migrated (via replication) VP1 to the new environment (to a host called NEWHOST), again using local storage. The plan was to migrate EX1 to NEWHOST (where VP1 now resides).
When I inspected the logs for the slow replication of EX1 it seems VP1 was being used as the source and destination proxy. What makes this even stranger is that it seemed to be that the old version of VP1 on HOST1 was being used as the source proxy (even though this was switched off). The only reason I think this was that I could see the ‘Reconfigure virtual machine’ in the VMWare Tasks & Events on HOST1.

Removing the old proxy will hopefully sort it, the preliminary tests suggest it will, I’ll post next week to let you know.

Veeam guys does this seem plausible? That a proxy will be attempted to be used if switched off?

Anyway to summarise make sure the proxies are configured correctly if using Automatic mode or just select the proxies manually.

[foggy]

Mark, no way switched off proxy could be used to process VM data. You can check for the proxy server effectively selected for VM processing in the job session log, after selecting the particular VM to the left.

[Vitaliy S.]

Veeam guys does this seem plausible? That a proxy will be attempted to be used if switched off?

I agree with what Alexander has said above, but I would be curious to see what your job log shows for this. Can you send it to our support team and post your case ID, so we could take a look at this?

[markfellows]

Got to be honest I didn't think it would attempt this. Just thought the ‘Reconfigure virtual machine’ messages for the old proxy (powered off) on the old host while the job was running were strange.

I'll log it out of interest though, probably won't get chance this week but I will try and do it next week.

[markfellows]

Also noticed (which I missed from the original post) I was getting this message in the logs of the slow Replication job.

Veeam backup proxy detected, changed block tracking is disabled.

The VM being replicated was not a proxy server though only had Exchange 2010 on there.

This post describes the same issues but there doesn't seem to be a resolution posted.

http://forums.veeam.com/veeam-backup-re ... 11130.html

[veremin]

Veeam backup proxy detected, changed block tracking is disabled.

Seems like there are some remnants of proxy components that prevent VB&R from using CBT on this machine. You can try to add this machine as proxy server to VB&R console, and, then, delete it again. Probably, the present components this time will be deleted successfully and you will be able to utilize CBT functionality.

Thanks.

[markfellows]

No elements of Veeam have ever been installed on this machine, it's only used as an Exchange 2010 server.

Plus if this were the case would we not receive the same message when the VM was being backed up?

[foggy]

Just another reason to involve support.

[markfellows]

Alexander,

I may after the migration but as I mentioned earlier in the thread the migration is taking place this weekend so I doubt support will have a resolution by then.

Out of interest what was the resolution to support case ID 5180520? (from the hyperlink above)

Mark.

[foggy]

Honestly, I cannot make much sense from the case details, at least nothing special that relates to your issue. Logs review will be definitely more effective. So feel free to open a case when you have time for that.

[markfellows]

Just an update I performed the migration a few weeks back and it was as quick as it should be. Looks as though the proxy configuration issue was the culprit.

Perversely the ‘Veeam backup proxy detected, changed block tracking is disabled’ warning did not appear when the replication jobs were run manually only in the schedule replication job (6am every morning).

[Vitaliy S.]

Hi Mark,

That's strange as there is no difference in logging when you manually start the job or do it on schedule. I will double-check with the QC team.

Thanks!