Backups slowing over time

[donikatz]

Hi, we are excited about vStorage API & CBT because the short backup windows allow us to back up some of our critical servers at several points during production hours. However, we're finding that backup speeds seem to slow over time, where similar small amounts of incremental data may fluctuate between 2-3x longer. We've found that in such cases archiving/deleting the backup file allowing it to create a new one resolved the issue -- the next incremental is back to a small window. I know this is not unusual for non-CBT backups because the vbk file grows and therefore it has to check larger data each time, but why would this be the case for CBT backups? Could this be because of automatic integrity checks coincidentally running during those backups (is that even included in the backup time reported)?

On a related note, are there plans to include the incremental data size of the backup in reports? This would be very useful, rather than just showing the total object size. Per-vmdk would be nice as well.

Thanks!

[Gostev]

Hi Doni, at first, where are you backing up?

There is no difference between CBT and non-CBT other than how incremental changes are calculated. So VBK behavior is the same whether or not CBT is used.

Data integrity check is included in the overall job time and does affect the backup time reported, however the integrity check happens in the beginning of every back cycle, not just once in a while.

We would be interested to investigate logs to see what takes longer with each cycle.

To be perfectly honest, we had similar issue reported once, by one customer, for one large VM, during very early beta for v4 - a few months ago. We could not reproduce the issue after massive long term testing of this scenario, and I believe the actual issue just went away for the customer with later beta code updates (we did not explicitly "fix" anything because we could not even reproduce). We also carefully reviewed the code trying to understand what could cause this, and did not find anything. So we thought it was temporary environment or VM-specific snapshot creation/deletion time growing problem. This is the only time we heard about something like this, so may be there are some VM's specifics which may sometimes lead to such behavior.

[donikatz]

We're backing up to a fiber-attached server with eSATA DAS. Could definitely be environment and/or VM-specific, was just curious if there were any known issues that could be related. Since not, we'll continue to trend it and see how things proceed before any detailed investigation. Since we're seeing fluctuation, could be related to SAN/FC I/O contention and the "fix" is just coincidental. Anyhow, we'll probably just wait for 4.1 before delving deeper.

Thanks for the quick response, Anton! doni

[Gostev]

Yes, please monitor this, and if this appears consistent for you, we can make a webex and have devs investigate this behavior deeper.

[donikatz]

Almost forgot, what about my last question: "On a related note, are there plans to include the incremental data size of the backup in reports? This would be very useful, rather than just showing the total object size. Per-vmdk would be nice as well." Thanks

[Gostev]

I missed it was not in our near-term plans, but will add if this appears easy. Not sure if these stats are preserved anywhere in the session information right now.