Discussions specific to the VMware vSphere hypervisor
Post Reply
gerling-sit
Novice
Posts: 5
Liked: 1 time
Joined: Apr 21, 2020 6:25 am
Contact:

v10 takes looong incremental backups

Post by gerling-sit » Apr 21, 2020 9:37 am

Hello everybody,

we have had problems with Veeam V10 in connection with VMWare vSphere (ESXi) servers since mid-March.

C.a. 2-3 days after updating to Veeam v10 version the incremental backups of the VMs take extremely long (6-10. hours instead of 15- 60 minutes).

Together with the Veeam support we have found out that the "Change Block Tracking (CBT)" does not work and therefore the software includes all the data of the VMs with every incremental backup.

According to Veeam Support the software only accesses the CBT service of VMWare and therefore we should look for the problem here (CBT Reset).
There is also an article about it here:
https://kb.vmware.com/s/article/2139574

Neither the offline variant nor the online variant helped.

Affected are 3 ESXi customers who use the versions:
6.7.0, 8169922
6.7.0, 14320388
In the field.

Anybody here happen to know what the problem is?
In my research work on the net I could not find out anything else so far.

foggy
Veeam Software
Posts: 19054
Liked: 1694 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 takes looong incremental backups

Post by foggy » Apr 21, 2020 11:00 am

Could you please share the Veeam support case ID?

gerling-sit
Novice
Posts: 5
Liked: 1 time
Joined: Apr 21, 2020 6:25 am
Contact:

Re: v10 takes looong incremental backups

Post by gerling-sit » Apr 21, 2020 11:36 am 1 person likes this post

Hi foggy,

of course :)

Case 1: 04070341
Case 2: 04069192

gerling-sit
Novice
Posts: 5
Liked: 1 time
Joined: Apr 21, 2020 6:25 am
Contact:

Re: v10 takes looong incremental backups

Post by gerling-sit » Apr 28, 2020 6:04 am

Hello, everyone,

I could find out that only servers with vSphere version 6.7 are affected by the described problem.

Versions up to 6.5 are working fine. Has anyone else observed this behavior ?

sandsturm
Expert
Posts: 145
Liked: 12 times
Joined: Mar 23, 2015 8:30 am
Contact:

Re: v10 takes looong incremental backups

Post by sandsturm » May 01, 2020 9:45 am

Hi
is this problem widely known or maybe just in specific environments? Some of our esxi hosts do have the version 6.7.0, 14320388 and if this is a serious problem, I'll wait with the upgrade to V10.

thx
sandsturm

Gostev
SVP, Product Management
Posts: 26116
Liked: 4065 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v10 takes looong incremental backups

Post by Gostev » May 01, 2020 3:18 pm

It's not currently a known issue. Basically, this above is the only report after 155K of v10 downloads. Thanks!

gerling-sit
Novice
Posts: 5
Liked: 1 time
Joined: Apr 21, 2020 6:25 am
Contact:

Re: v10 takes looong incremental backups

Post by gerling-sit » May 04, 2020 10:20 am

Hi, everybody,

@sandstorm
Like I said, I have the problem with three different customers. In total, these are 5 ESXi servers with patch level 6.7.0 U2 - U3.
But you are welcome to try it, @Gostav already mentioned it, 155K v10 downloads without problems.

If there are no problems with your servers after the update then I would be glad about a post.

xasz
Novice
Posts: 9
Liked: never
Joined: May 21, 2017 8:03 pm
Full Name: Michael Schneider

Re: v10 takes looong incremental backups

Post by xasz » May 22, 2020 5:58 am

Hello gerling-sit,

i think i have some kind of the same problem.
We have Upgraded to V10 some days after release and at the beginning it was perfect. Backup Times for Raw Data of 12 TB on one of our Backup Servers in like 1 or 2 Hours ( Talking about Incremental).
We use 10Gbit and an lokal Disk Raid on a Windows Server, No Nas or Something.

I getting problem with Backup times, which are now very long. I have to times up to 20 Hours now and most of the time 8+ hours.
I checked the logs and have found this:

Code: Select all

[21.05.2020 19:30:12] <01> Info     STARTBACKUPJOB 
……..
[21.05.2020 20:39:29] <64> Info           Merging oldest incremental backup into full backup file (1% done)
…..
[22.05.2020 03:55:55] <71> Info           Merging oldest incremental backup into full backup file (99% done)
….
[22.05.2020 03:59:41] <01> Info     ------- Veeam Manager Stopped -------
The merge backup time is like 1 hours and the merge time is like 7.5 hours.
With V9 we never had Backup Times more thatn 4 Hours and at the beginning i thought V10 would be faster, because it was like 1-2 hours at the beginning.
For me it is a problem, because the Backup does not finish in Time for my Tape job to schedule correctly.

Some informations:
Job:
  • Incremental forever
  • 30 Day Retention Policy
  • No Active or Syn Full Backups

VMware:
  • vCenter: 6.7.0 (Build 14368073)
  • 2x ESX Host 6.7.0 (Build 14320388) - Update 3

Support Case: 04159171

foggy
Veeam Software
Posts: 19054
Liked: 1694 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 takes looong incremental backups

Post by foggy » May 22, 2020 4:33 pm

Hi Michael, judging by the provided log messages, your issue is different from the OP's one - related to merging rather than to CBT. Anyway, let's see what our engineers could suggest after looking deeper into your logs.

xasz
Novice
Posts: 9
Liked: never
Joined: May 21, 2017 8:03 pm
Full Name: Michael Schneider

Re: v10 takes looong incremental backups

Post by xasz » May 28, 2020 11:41 am

Hello foggy,

yeah i first thought the same and then i though it could take longer merging times because of CBT problems and somehow data which takes longer to merge, but iam not that into that :)

foggy
Veeam Software
Posts: 19054
Liked: 1694 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 takes looong incremental backups

Post by foggy » May 28, 2020 10:17 pm

CBT issues would lead to longer backups, not longer merges. ;)

Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests