Veeam 4.1 Slow backup of large VMDK files on VI3.5

[icaltun]

Hi all!

We are implementing Veeam Backup 4.1 for one of our clients.
They use ESX3.5 build 143128 and SANs EMC² CX3-10c.
Veeam is working in VCB mode and all jobs are configured SAN/NBD without CBT (obvious client will migrate to vSphere in 2010).
Powerpath is installed on the VCB/Veeam server. The VMs are hosted on the CXs and Veeam places data in FC to an AX4-5 so write speeds shouldn't be bad at all . The ClariiOns are configured for manual multi-pathing and LUN distribution between the SPs. SP workload peaks at around 60% max.

The Veeam Server is a 8 way server with 6GB of RAM. CPU usage is around 54% and RAM usage 30%

For a typical small VM i get stuff like:

7 of 7 files processed
Total VM size: 166,00 GB
Processed size: 166,00 GB
Processing rate: 121 MB/s
Backup mode: SAN/NBD without changed block tracking
Start time: 26/12/2009 18:27:06
End time: 26/12/2009 18:50:30
Duration: 0:23:23

But for a large VMDK VM it gets a lot more complicated.
7 of 13 files processed

Total VM size: 1,23 TB
Processed size: 252,21 GB
Processing rate: 41 MB/s
Backup mode: SAN/NBD without changed block tracking
Start time: 29/12/2009 12:22:12
Time remaining: 07:10:58

Yes the VMDKs are larger that 256GB. Let's say the biggest VMDK will scale up to 1.2 TB.
The design and implementation was not of my choice and optimizations of the VMs will be done next year when migrating to vSphere.

Still, is there any way I could improve these results? For me the most dangerous situation is an overwhelming snapshot ...

What I posted here is a lucky case too... I opened also a support case as I see a speed degradation over time.
The speed wil reach 10-15MB/s when my backup will be at 60%. I end up with a slow and painful backup and I am now wondering how the incrementals will perform.

These VMs are either file servers or lotus domino servers.

Any ideas or help will be appreciated.


PS: My client is more than interested in a scenario where Veeam will do VCB mode restores as well .
Today they backup every year per total 15TB of data and data growth is estimated to 30-50% per year.
That would be a different post maybe for a nice feature request

Kind regards,

Ioan

[Gostev]

This is interesting situation... I have two ideas:
1. A very large amount of changes on the large VM (will be able to say if this is true by looking at incremental size produced after this run).
2. Source or target storage issues, this is pretty much the only other thing which could explain such a slowdown. But may be devs will find out something researching the issue.

Probably live webex with devs will be best option here, I doubt they would be able to diagnose the issue through the logs and they will only show that everything works okay (just slow).

[icaltun]

Hi,

To answer the questions one by one.

1. Yes. There are several changes but only on some of the VMs : a snapshot on the biggest VM may increase by 1GB/hour. But it is only a peak estimate.

2. Of course we can do a webex whenever you like, just inform me in time so we can be present on the client site for the webex or at least a multiple webex between our team and your guys. A one day notice should be sufficient I believe.


PS: I believe we met at the VMworld in Cannes this year. Not absolutely sure.

Have a great day !

Ioan

[Gostev]

Please suggest the webex to your support engineer, and he or she will coordinate with the development folks so they could attend. Not sure when it would be the best date due to the holiday season though (dev. office has public holidays in the beginning of January).

If you met with someone from Veeam, that was not me for sure.