Reverse Incremental - Incredibly Slow

[rich_england]

Hi all,

Firstly want to say I'm by no means a Veeam expert so go easy on me. We recently had a powercut which has caused havoc with our current backup schedule. I'll give you a quick debrief on our system:

4 - ESX 5.5 VMWare hosts

Backup VM has Veeam 6.1 installed - yes, I know this is an outdated version and we are planning on upgrading in the near future.

Backup schedule was a reverse incremental ran every night on our file servers. The backup repository target was the E: Drive on the backup server is a 9TB drive spread across several SANs via SCSI connections. Not an ideal setup but this is something I inherited and plan to reconfigure during next summer however for now this is what we have to work with. The Backup VM has 8 CPU cores and 12GB RAM and also has the VMXNET3 - 10gb NIC. It doesn't seem to run out of resources whilst running the backup.

This backup schedule has run fine and dandy for 2 years on a reverse incremental until the powercut. Since the powercut it could no longer find the CBT file so it was wanting to perform a full backup on each VM. We were advised to copy the files from our repository (vbk, vbm and vrb's) to a new location and add it as a new repository so we can still restore from these files and delete them from the current location (E: Drive) and create a new schedule and basically start again.

Since I inherited the backup system, this is the first time it is wanting to do a full backup of the VMs rather than just copying the changes. This is where our problem starts.

So I've got the schedule set up to backup 5 VMs with a total of 4TB. When I go to perform the backup it is averaging at around 12mb/s. I've never monitored the network speed on the previous schedule because it was only backing up the changes but now it's backing up the full VMs at 4TB, by my calculations it's going to take over 20 days to complete this backup. Is this normal or does this seem incredibly slow? I once backed up a full site using Veeam to a NAS station which was 10TB and it took 4 days so I'm not sure why this is running so slow. I can't believe anyone would buy a product that took over 20 days to backup 4TBs of space...

So my question is guys, does this seem normal? Is there anything I need to look for to improve the data rate. Any hints or tips would be great.

Cheers

[foggy]

Richard, what are the bottleneck stats for this job?

[rich_england]

Hi Foggy,

Ok, so for testing purposes I have narrowed the backup down a single VM which is only 180GB in size as I cannot wait 20+ days for a full backup.

So last night I ran the backup on the 180GB VM which took a full backup of this in just under 8 hours. The processing rate was 12mb/s and the bottleneck was Source.

I ran it again today so it was only backing up the changes which obviously only took a short while (6 minutes) but the processing rate was 443 mb/s. I don't understand how it can run one backup at 12mb/s and then the next at 443 mb/s...surely it's running the same process - hitting the same target using the same agent?

As I said I am a bit of a Veeam novice so expect someone will tell me I'm wrong

[foggy]

Do not compare full and incremental runs processing rate, since it is not throughput, but total VM size divided by total VM processing time.

What transport mode is used to retrieve VM data from source storage?

[rich_england]

Currently just using a default VMWare Proxy with automatic transport mode selected. This does failover to network mode I believe though as it says hot add is not supported.

Block tracking is also enabled.

[foggy]

Network mode uses ESXi management interface, which is throttled by VMware on 1Gb networks, so the speed looks expected in such configuration. Please check this KB article for hotadd limitations that might apply.

[rich_england]

Thanks for your reply.

I seem to have improved the performance of the backup by upgrading our datastores to VMFS 5 from VMFS 3. The backup has taken 4 days to complete 4TB which I didn't think was too bad compared the 28 days it was previously looking at taking.

I can work with this now the incremental's are completing in a few hours. I will now look at upgrading to the latest version of Veeam.

Thanks for your responses.