v10 takes looong incremental backups

[gerling-sit]

Hello everybody,

we have had problems with Veeam V10 in connection with VMWare vSphere (ESXi) servers since mid-March.

C.a. 2-3 days after updating to Veeam v10 version the incremental backups of the VMs take extremely long (6-10. hours instead of 15- 60 minutes).

Together with the Veeam support we have found out that the "Change Block Tracking (CBT)" does not work and therefore the software includes all the data of the VMs with every incremental backup.

According to Veeam Support the software only accesses the CBT service of VMWare and therefore we should look for the problem here (CBT Reset).
There is also an article about it here:
https://kb.vmware.com/s/article/2139574

Neither the offline variant nor the online variant helped.

Affected are 3 ESXi customers who use the versions:
6.7.0, 8169922
6.7.0, 14320388
In the field.

Anybody here happen to know what the problem is?
In my research work on the net I could not find out anything else so far.

[foggy]

Could you please share the Veeam support case ID?

[gerling-sit]

Hi foggy,

of course

Case 1: 04070341
Case 2: 04069192

[gerling-sit]

Hello, everyone,

I could find out that only servers with vSphere version 6.7 are affected by the described problem.

Versions up to 6.5 are working fine. Has anyone else observed this behavior ?

[sandsturm]

Hi
is this problem widely known or maybe just in specific environments? Some of our esxi hosts do have the version 6.7.0, 14320388 and if this is a serious problem, I'll wait with the upgrade to V10.

thx
sandsturm

[Gostev]

It's not currently a known issue. Basically, this above is the only report after 155K of v10 downloads. Thanks!

[gerling-sit]

Hi, everybody,

@sandstorm
Like I said, I have the problem with three different customers. In total, these are 5 ESXi servers with patch level 6.7.0 U2 - U3.
But you are welcome to try it, @Gostav already mentioned it, 155K v10 downloads without problems.

If there are no problems with your servers after the update then I would be glad about a post.

[xasz]

Hello gerling-sit,

i think i have some kind of the same problem.
We have Upgraded to V10 some days after release and at the beginning it was perfect. Backup Times for Raw Data of 12 TB on one of our Backup Servers in like 1 or 2 Hours ( Talking about Incremental).
We use 10Gbit and an lokal Disk Raid on a Windows Server, No Nas or Something.

I getting problem with Backup times, which are now very long. I have to times up to 20 Hours now and most of the time 8+ hours.
I checked the logs and have found this:

Code: Select all

[21.05.2020 19:30:12] <01> Info     STARTBACKUPJOB 
……..
[21.05.2020 20:39:29] <64> Info           Merging oldest incremental backup into full backup file (1% done)
…..
[22.05.2020 03:55:55] <71> Info           Merging oldest incremental backup into full backup file (99% done)
….
[22.05.2020 03:59:41] <01> Info     ------- Veeam Manager Stopped -------

The merge backup time is like 1 hours and the merge time is like 7.5 hours.
With V9 we never had Backup Times more thatn 4 Hours and at the beginning i thought V10 would be faster, because it was like 1-2 hours at the beginning.
For me it is a problem, because the Backup does not finish in Time for my Tape job to schedule correctly.

Some informations:
Job:

• Incremental forever

• 30 Day Retention Policy

• No Active or Syn Full Backups

VMware:

• vCenter: 6.7.0 (Build 14368073)

• 2x ESX Host 6.7.0 (Build 14320388) - Update 3

Support Case: 04159171

[foggy]

Hi Michael, judging by the provided log messages, your issue is different from the OP's one - related to merging rather than to CBT. Anyway, let's see what our engineers could suggest after looking deeper into your logs.

[xasz]

Hello foggy,

yeah i first thought the same and then i though it could take longer merging times because of CBT problems and somehow data which takes longer to merge, but iam not that into that

[foggy]

CBT issues would lead to longer backups, not longer merges.