Quick migration bad Processing rate

[Frederic]

Hi,

i am trying to build a backup with Veeam Backup and replication. The infrastructur:

1 ESXI 6.5 on an Fujitsu Pimergy RX300 S7 with 2 SAS HDDs Raid1
1 QNAP TS-432XU with 3 WD Red Pro 4TB Raid 5 and 1 WD Red Pro 4TB HotSpare
both are directly connected via 10GBe SFP+

a Backup from esxi to QNap works with 222MB/s
a Quick migration from QNap to esxi works with 14MB/s with Bottleneck 99% Target (esxi)
Veeam Backup and replication runs in a Server2016 with 4 Cores and 16GB RAM (its only so much for testing)
i got the 10GBe to a virtual switch to the Veeam VM and attached it via iscsi.
i configured Transport Mode Virtual Appliance but it doesn't change the Performance
i can't find any other option to increase the quick migration speed

[foggy]

Hi Frederic, do you mean you have VMFS datastores on both Fujitsu and QNAP and try to migrate the VM from one to another, within the same ESXi host? Does the proxy effectively utilizes hotadd to populate the target datastore during migration (you should see the [hotadd] tag next to the proxy server name in the stats window)?

[Frederic]

esxi is installed on an internal usb device
the internal 900GB Raid1 is the datastore on whitch the VMs operate. on the qnap is a 2TB Partition whitch is in Windows atached via iscsi-Initiator
here a screenshot of the backup job
https://picload.org/view/dcldgdlc/veeam ... 2.png.html

and here a screenshot of the quick migration
https://picload.org/view/dcldgdoc/veeam ... n.png.html

[foggy]

Then I'm not quite understand how QNAP is involved here, since Quick Migration is an operation of moving VM between vSphere hosts and datastores. Anyway, please select the VM name in the left pane of the stats window (on the screenshot) and you should see the transport mode tags for the source and target proxies.

[Frederic]

i am using a backupjob to the qnap and the instant vm recovery from the qnap back to the esxi.

Backup Job looks like that:

Code: Select all

28.11.2018 11:54:47 :: Queued for processing at 28.11.2018 11:54:47  
28.11.2018 11:54:47 :: Required backup infrastructure resources have been assigned  
28.11.2018 11:54:52 :: VM processing started at 28.11.2018 11:54:52  
28.11.2018 11:54:52 :: VM size: 50,0 GB  
28.11.2018 11:54:54 :: Getting VM info from vSphere  
28.11.2018 11:55:03 :: Skipping guest processing (VM is powered off)  
28.11.2018 11:55:03 :: Creating VM snapshot  
28.11.2018 11:55:12 :: Saving [datastore1] Test-SRV01/Test-SRV01.vmx  
28.11.2018 11:55:12 :: Saving [datastore1] Test-SRV01/Test-SRV01.vmxf  
28.11.2018 11:55:13 :: Saving [datastore1] Test-SRV01/Test-SRV01.nvram  
28.11.2018 11:55:14 :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]  
28.11.2018 11:55:42 :: Hard disk 1 (50,0 GB) 11,5 GB read at 239 MB/s [CBT] 
28.11.2018 11:57:04 :: Removing VM snapshot  
28.11.2018 11:57:13 :: Finalizing  
28.11.2018 11:57:15 :: Busy: Source 99% > Proxy 61% > Network 26% > Target 0%  
28.11.2018 11:57:15 :: Primary bottleneck: Source  
28.11.2018 11:57:15 :: Network traffic verification detected no corrupted blocks  
28.11.2018 11:57:16 :: Processing finished at 28.11.2018 11:57:16  

the instant vm recovery looks like that:

Code: Select all

28.11.2018 12:01:46 :: Starting VM Test-SRV01 processing at 28.11.2018 12:01:46  
28.11.2018 12:01:46 :: Migration plan: 
[Mode] SmartSwitch  
[Host] 10.100.0.152 > 10.100.0.152
[Resource pool] Resources > Resources
[VM folder] vm > vm
[Datastore] VeeamBackup_Test-BK01 > datastore1

  
28.11.2018 12:01:47 :: Preparing  
28.11.2018 12:01:54 :: Queued for processing at 28.11.2018 12:01:54  
28.11.2018 12:01:54 :: Required backup infrastructure resources have been assigned  
28.11.2018 12:01:59 :: Using source proxy VMware Backup Proxy [nbd]  
28.11.2018 12:01:59 :: Using target proxy VMware Backup Proxy [nbd]  
28.11.2018 12:02:01 :: Preparing target VM  
28.11.2018 12:02:29 :: Creating VM snapshot  
28.11.2018 12:02:34 :: Copying VM files  
28.11.2018 12:02:35 :: Hard disk 1 (50,0 GB) 11,5 GB read at 14 MB/s 
28.11.2018 12:16:43 :: Copying cache files  
28.11.2018 12:16:44 :: Copying snapshot files  
28.11.2018 12:17:00 :: Renaming source VM  
28.11.2018 12:17:03 :: Renaming target VM  
28.11.2018 12:17:05 :: Removing VM snapshot  
28.11.2018 12:17:17 :: Source VM will not be deleted (heartbeat not detected)  
28.11.2018 12:17:17 :: Updating configuration database skipped (source VM not deleted)  
28.11.2018 12:17:17 :: Finished processing VM Test-SRV01 at 28.11.2018 12:17:17  
28.11.2018 12:18:43 :: Busy: Source 0% > Proxy 6% > Network 1% > Target 99%  
28.11.2018 12:18:43 :: Primary bottleneck: Target  
28.11.2018 12:18:43 :: Network traffic verification detected no corrupted blocks  
28.11.2018 12:18:43 :: Processing finished at 28.11.2018 12:18:43

[foggy]

Ok, got it, you're doing Quick Migration of the instantly recovered VM.

According to the session log, NBD mode is being used on both sides, not hotadd:

28.11.2018 12:01:59 :: Using source proxy VMware Backup Proxy [nbd]
28.11.2018 12:01:59 :: Using target proxy VMware Backup Proxy [nbd]

[Frederic]

i got a little bit progress. I found in the internet Thick-Provision Lazy-Zeroed Virtual Disk is for Veeam bad. I build a new VM with Eager-Zeroed and got a Processing rate of 59MB/s

how can i set up the hotadd mode or what am i doing wrong?

if i go into Backup Infrastructure/Backup Proxies/VMWare Backup Proxy/Properties/ in Transport mode is Virtual appliance and "Failover to network mode if primary mode fails, or is unavailable" chosen

[foggy]

In your case NFS mount server is the proxy server itself (both components are running on the backup server itself) - in this case hotadd is not allowed. If you select any other proxy, hotadd will be possible.

[Frederic]

mhm ok for hot add i need a explicit Veeam Proxy. But is it normal to geht only about 60MB/s while using instant recovery while getting about 200MBs in a Backup Job?

[foggy]

You cannot compare read and write operations performance. Btw, is target still reported as the primary bottleneck for Quick Migration?

[Frederic]

yes the esxi is still the Bottleneck with
Busy: Source 3% > Proxy 13% > Network 9% > Target 99%

i know the quick migration will work slower but i hoped i could boost it a little bit more. In our other Server/Backup Strukture with Veeam, newer servers and SAS we got 400+MBs for Backupjobs and about 200MBs for quick migration so i thought it should be possible to geht 50% of the backup speed for the quick migration.

Btw aktual i try this backup with a Fujitsu Primergy RX300 S7. It is not the newest so i hope with the new server would come more Performance The new Server would come in January

[foggy]

Let's see if hotadd will allow for better performance, when you set up a dedicated proxy server VM for that.

[Frederic]

thank you very much for your help. as i read for hot add i need a didicated server or vm. But at the location this setup should build we haven't enough ressources to do this. The new Hardware should come in january at this moment i can try probably more.

[foggy]

For hotadd you can use any other existing VM as well, just not the same as NFS mount server.