HELP - slow backup performance - no clue why

[geraldb]

hello folks!

we have a really performance problem with our veeam backups and i really don't no why.
I hope you can help us!
(short information: we have a write speed with veeam of ~50MB/s)

ESXi Server:
HP ProLiant ML350 Gen6
CPU: 2* Intel Xeon E5606 with 2.13GHz
RAM: 24GB
HDD: 6*1TB in RAID5 with 512MB cache

our backup Server with Veeam installed on:
Lenovo TS430
CPU: 1* Intel Xeon E3 1240 with 3.30GHz
RAM: 12GB
HDD: 6*1TB in RAID10 - no cache (write through modus)

ESXi-Host and the backup server are connected via lan

On our backup server we have an avarage write speed of 350MB/S (ATTO Disk Benchmark) and ~160MB/s if we copy (read and write) on the local disk in windows)
If we copy files from the ESXi host to the backup server it goes with 100MB/s (limited by the network card on the backup server - because it use only 1 port and that is by 99%workload- but that is an other question ^^)

when the backup is running we have an avarage cpu load of 30-50% and 3-5GB RAM load on the backup server
on the ESXi-Host there is also no high capacity. (much lower as on the backup server)

veeam says:
Busy: Source 99% > Proxy 42% > Network 1% > Target 2%

Veeam is set on:
Compression: optimal
Storage: optimizations: local target

I HAVE REALLY NO CLUE why the veeam is running noly with 50MB/s
.. i would be SO HAPPY(!!) with 100-150mb/s ^^ (150mb/s if we fix the network-card issue gg)

so why do we have only 50MB/s when we do the backup with veeam and how could we increase the speed!
.. i would be happy with 100-150mb/s ^^ (150mb/s if we fix the network-card issue gg)

case #00153723

[foggy]

Not very bad performance actually, considering you are using network processing mode (the data is retrieved through ESXi management interface, you could check transfer speed while downloading files from the same datastore using Datastore Browser in vSphere Client).

"Source 99%" means that source storage data retrieval speed is a bottleneck. Installing a proxy on the VM having access to the datastore with VMs you are backing up, will allow for using hotadd source data retrieval mode, which generally provides better performance.

[geraldb]

ok, I've installed a backup-proxy in the VM and changed the mode to Virtual Appliance.
a few questions to that:

1) Do I have to install a backup-proxy in EVERY VM and pick it as backup-proxy in the settings when I want to change to Virtual Applianc mode or is it only needed in one VM on every ESXi Server ?

2) what is the difference between "porcessing rate:" and "hard disk read at xx MB/s" ?!


3)

veeam is reading now the whole 500GB(!!) and transferes 58gb .. so it takes longer (48min. instead of ~7-8min. befor)
If I connect on the windows machine it uses only 27,7GB. (1 HDD with 500GB and 27,7 in use)
mhhh ok .. veeam has created a snapshot on the machine "VEEAM BACKUP TEMPORARY SNAPSHOT - Please do not delete this snapshot. It is being used by Veeam Backup." which is deleted after the backup ..
is this the disadvantage of the Virtual Appliance mode ? so it is faster but it takes longer ?! because it can't use CBT ???

best regards

[tsightler]

1) Ideally you need at least one virtual proxy on at least one host that has access to all datastores. If using local storage you would need one proxy on each host, if using shared storage, then a single proxy in a cluster would be able to access all shared datastores within the cluster.

2) Processing rate is the aggregate MB/s of all processed data in the entire job, taking into account all overhead. For example, in the screenshot you shared, the job had processed 17.7GB of data in 5:30 seconds so that's 17.7GB/330 seconds = .0536 GB/s = 55MB/s. The "read rate" is rate at which data was retrieved from the disk, for example, in the job above 17.7GB was read in 4:17, so that 17.7GB/257 seconds = .068GBs = 70MB/s. Note that this doesn't necessarily mean that the actual data was transferred at this speed, as there is still overhead in skipping blocks, etc., and this number only indicates the amount of data that was actually retrieved for processing.

3) I can see that you have configured server StriziSQL-TEST as your virtual backup proxy. Unforunately, CBT must be disabled on the virtual backup proxy itself (we do this automatically) so using a system with a large amount of data as a proxy is generally not a good idea as it will not be able to use CBT to backup itself. Most customers either build small, dedicated systems as proxies, or use smaller VMs that already exist in their environment.

[foggy]

3) I can see that you have configured server StriziSQL-TEST as your virtual backup proxy. Unforunately, CBT must be disabled on the virtual backup proxy itself (we do this automatically) so using a system with a large amount of data as a proxy is generally not a good idea as it will not be able to use CBT to backup itself. Most customers either build small, dedicated systems as proxies, or use smaller VMs that already exist in their environment.

In addition to that, hotadd mode cannot be used to backup such VMs (which makes these VMs to be backed up over network), this is a known limitation covered in the release notes.

mhhh ok .. veeam has created a snapshot on the machine "VEEAM BACKUP TEMPORARY SNAPSHOT - Please do not delete this snapshot. It is being used by Veeam Backup." which is deleted after the backup ..
is this the disadvantage of the Virtual Appliance mode ? so it is faster but it takes longer ?! because it can't use CBT ???

Snapshot is always created during backup, irrespective of the backup mode being used. This is just how VMware image level backup solutions work.

[geraldb]

thx for the quick response!

@3
we have 3 ESXi Hosts with local storage
.. so I have to install a VM with a proxy on every ESXi host mhhh

Is there a recommended OS (is Win XP ok?) or numbers of CPU cores / RAM for a backup proxy ?
Does the backup proxy need a lot of cpu / ram power ? or is it is it regardless ?

*edit*
so for the VMs which the backup-proxy is isntalled I have to chose network mode ..?

thx

[foggy]

It's more CPU rather than RAM, 2 cores per job is recommended. Please review the proxy server system requirements outlined in the release notes and user guide.

[veremin]

My two cents for the purpose of complete understanding.

Truth be told, VB&R is image-based solution for backing up and replicating VMs. Thus, it always leverages snapshot functionality, regardless of backup method you use.

Little reference regarding image-based backups (I think, you probably know it) : Image-based backup is a backup process for a computer or virtual machine (VM) that creates a copy of the operating system (OS) and all the data associated with it, including the system state and application configurations. The backup is saved as a single file that is called an image.

As to Virtual Appliance, this mode is recommended and can only be used if the backup proxy is deployed on a VM. The Virtual Appliance mode uses the SCSI hot-add capability of ESX hosts to attach disks of the backed up VM to the backup proxy VM. In this mode, VM data is retrieved directly from storage through the ESX I/O stack, instead of going through the network stack, which improves performance. So, in this case, I would recommend you to deploy backup proxy on virtual machine residing on the same ESXi host. Notice: not on the same VM you’re trying to backup up or replicate.

As far as I can understand, virtual appliance failed, switching to the network method*, due to the fact, that you’d chosen the one machine as source and as a proxy at the same time. It was also the reason for the absence of CBT.

*It can be noticed by looking at the last screenshot - source proxy [nbd]. If virtual appliance had worked properly, you would have seen [hot-add].

In addition, be aware also that before using Virtual Appliance you have to install VMWare Tools on the VM you’re going to use as proxy, working in such
method; otherwise, it will not work.

Thanks.

[tsightler]

thx for the quick response!
@3
we have 3 ESXi Hosts with local storage
.. so I have to install a VM with a proxy on every ESXi host mhhh

Well, you do not "have" to do so, but it's the only way to get the best performance when using local disk since virtual appliance mode is the only way to read data directly from those disk. Other than that, you can use network mode, and, while performance will be slower, especially for full backups, you can compensate for this somewhat by running multiple jobs at the same time, and incremental runs will likely still be reasonably fast in either mode.

[geraldb]

mhh i don't get it ^^
I've removed the proxy from the VM and installed it on another but on the same host ..
but in the middle of the backup veeam writes "hot add is not supported for this disk, failing over to network mode" ... why ?! what kind of disk is "this disk" !?`

[foggy]

There are several limitations of using hotadd mode outlined in the following KB article: http://www.veeam.com/KB1054. Please review, the issue could be in one of them.

[geraldb]

thx for the KB site
and yes - we have restored the VBR server once.


just one short question to that:
on the site it says that i can find the UUID value in: HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VMware Virtual Disk Development Kit\volatile
but such a path does not exist (in Server 2008R2 x64 - Veeam 6.5 x64)
is it: HKEY_LOCAL_MACHINE\SOFTWARE\volatile the "UUIDHost" value ?


another short question:
• LUN where Veeam Backup server resides on should be formatted with proper block size to be able to mount the largest virtual disk of the hot added VMs.
1MB block size – 256GB maximum file size
2MB block size – 512GB maximum file size
4MB block size – 1024GB maximum file size
8MB block size – 2048GB maximum file size

I read that there is only one block size (1MB) since VMFS5.
So is the block size requirement still active ? (in VMFS5)

thx!

[veremin]

just one short question to that:
on the site it says that i can find the UUID value in: HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VMware Virtual Disk Development Kit\volatile
but such a path does not exist (in Server 2008R2 x64 - Veeam 6.5 x64)
is it: HKEY_LOCAL_MACHINE\SOFTWARE\volatile the "UUIDHost" value ?

thx!

Try to check it under wow6432Node. The value you're trying to find for 64-bit systems is being stored under wow6432Node branch of registry.

So, in your case it should be like that : HKEY_LOCAL_MACHINE\SOFTWARE\wow6432Node\VMware, Inc.\volatile

Thanks.

[geraldb]

I searched for it - but it does not exist

[Vitaliy S.]

I read that there is only one block size (1MB) since VMFS5.
So is the block size requirement still active ? (in VMFS5)

No, it's not.

[veremin]

In this case I’d recommend you to contact support team directly and have them take a look at your issue, since, it’s way too hard to investigate such a problem remotely through forum correspondence.

Additionally, support team will shed a light on all dark sides regarding registry keys and so forth. Furthermore, they even will able to provide you full assistance through WebEx session.

After finding solution, don’t hesitate to share it with community.

Thanks.

[aweiss]

Hi,
Did you get any solution to your problem ? I have exactly the same
Veeam 6.5, vSphere 5.1 hotadd mode with proxy in a VM.

CBT is detected but the entire file is read even if their is only 50 Go out of 2 To of changes
About 45 min to scan a 2 To disk with CBT ...

Thks
Rgds
Alex

[ZachW]

I apologize if I am misunderstanding you but if you are saying that while trying to backup a proxy it is not properly utilizing CBT then this is a known issue as seen here: (http://forums.veeam.com/viewtopic.php?f=24&t=15622 ). If I am incorrect then please let me know.

[aweiss]

I apologize if I am misunderstanding you but if you are saying that while trying to backup a proxy it is not properly utilizing CBT then this is a known issue as seen here: (http://forums.veeam.com/viewtopic.php?f=24&t=15622 ). If I am incorrect then please let me know.

I have no problem to backup the proxy as it uses network (proxy is a vm). The problem i have is when my VM Proxy is doing CBT on hot added disks from others VMs. CBT is well detected but it seems that it is not used as i have 100% CPU, 0% network. All IO are done locally on the added drive (I can see around 800Mb/s read in the windows perfmon on that hot added drive) but only some Gigas get transfered to the target repo.

Alex

[foggy]

Alexandre, please review this topic, probably, should clarify the issue for you.