Host-based backup of VMware vSphere VMs.
Post Reply
TRE3
Enthusiast
Posts: 32
Liked: 5 times
Joined: Apr 11, 2018 7:47 am
Contact:

Full disk read when a new VMware disk is attached to VM

Post by TRE3 »

Hi Veeam Forum,

one of our customers added a new disk to his file server VM (vSphere) and although CBT is enabled (Veeam shows [CBT] in the job), all the OTHER disks get FULLY read. No other changes have been made (no disk extension). Is this a default behavior? Can we fix this?

Cheers, Thomas

(Case ID: 06129466)
HiHoItsOffToWorkWeGo
Enthusiast
Posts: 45
Liked: 2 times
Joined: Jan 20, 2021 1:06 pm
Full Name: Mark Fellensiek
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by HiHoItsOffToWorkWeGo »

Although the entire source disk gets read, you will find that only the usual tiny portion of changed data gets sent over the network to the repository thanks to CBT still working and doing its thing.
You can confirm this yourself by looking at the backup. You will find that the incremental is the same size as that of the previous day, becuase, just like the previous day, the CBT technology only backs up the portions of the disk that have actually changed.
On the next backup the situation will go back to normal, with only a tiny bit read and sent.
username because using Nutanix Mine with Veeam
TRE3
Enthusiast
Posts: 32
Liked: 5 times
Joined: Apr 11, 2018 7:47 am
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by TRE3 »

Yes, that is what we see. The problem here is: The backup has a speed of 30 MB/s (infrastructure limitation) and it has to read 20 TB of data. So the file server needs several days before the next backup can start. In the near future, there are more disks to be added to this server. So he needs a workaround for the future that really uses CBT and where the backup does not fully read the untouched disks.
HiHoItsOffToWorkWeGo
Enthusiast
Posts: 45
Liked: 2 times
Joined: Jan 20, 2021 1:06 pm
Full Name: Mark Fellensiek
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by HiHoItsOffToWorkWeGo »

Sounds like you might be doing backups using the Veeam agent installed inside the VM.
Have you considered performing an image backup of the VM instead?
You might see much faster read speeds, e.g. over 200MB/s.
At this speed the entire 20TB can be read in just 1 day.
And image level backups are much less susceptible to the problem you mentioned. Changing one disk means only the changed disk needs to be read. The other disks continue having only their changed data read.
username because using Nutanix Mine with Veeam
TRE3
Enthusiast
Posts: 32
Liked: 5 times
Joined: Apr 11, 2018 7:47 am
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by TRE3 »

The customer does VMware based backups (no agent / NBD). SAN-based is not possible.
HiHoItsOffToWorkWeGo
Enthusiast
Posts: 45
Liked: 2 times
Joined: Jan 20, 2021 1:06 pm
Full Name: Mark Fellensiek
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by HiHoItsOffToWorkWeGo »

NBD is what's slowing this down.
Can you use a Veeam proxy with virtual appliance mode (with NBD disabled) to backup that file server instead?
username because using Nutanix Mine with Veeam
TRE3
Enthusiast
Posts: 32
Liked: 5 times
Joined: Apr 11, 2018 7:47 am
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by TRE3 »

Thank you for your suggestions but what the customer wants to clarify is the following:

Is it an expected behaviour that if you add a virtual disk to a VM with CBT enabled, that EVERY UNCHANGED disk gets also fully read instead of using CBT data?
chris.childerhose
Veeam Vanguard
Posts: 589
Liked: 140 times
Joined: Aug 13, 2014 6:03 pm
Full Name: Chris Childerhose
Location: Toronto, ON
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by chris.childerhose » 1 person likes this post

I believe that is the expected behaviour of Veeam, as you are changing the disk configuration for the VM even though you are just adding a new disk. So Veeam does a full read, and CBT still works but kicks in for the subsequent runs after this one.
-----------------------
Chris Childerhose
Veeam Vanguard / Veeam Legend / Veeam Ceritified Architect / VMCE
vExpert / VCAP-DCA / VCP8 / MCITP
Personal blog: https://just-virtualization.tech
Twitter: @cchilderhose
TRE3
Enthusiast
Posts: 32
Liked: 5 times
Joined: Apr 11, 2018 7:47 am
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by TRE3 »

Thanks, I'll submit that info to the customer!
PetrM
Veeam Software
Posts: 3511
Liked: 590 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by PetrM »

Hello,

@TRE3 please wait if it's possible :)

I asked my colleagues from QA to test this scenario in our lab and we clearly see that there is no read of disk (btw we tested NBD only) which had no changes even after a new disk was added to the VM. It looks like that there are some specific conditions, I'll ask our support team to research your case a bit deeper.

May I ask you to clarify which transport mode was used in your test?

Thanks!
TRE3
Enthusiast
Posts: 32
Liked: 5 times
Joined: Apr 11, 2018 7:47 am
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by TRE3 » 1 person likes this post

NBD was used
PetrM
Veeam Software
Posts: 3511
Liked: 590 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by PetrM »

Let's wait for our support team. I'm not ready to confirm that the behavior described in your first post is by design.

Thanks!
Rumple
Service Provider
Posts: 81
Liked: 14 times
Joined: Mar 10, 2010 7:50 pm
Full Name: Mark Hodges
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by Rumple » 1 person likes this post

my thought is..how in the world are they running a vmware server with a virtual 20+TB file server and absolutely no way to install run a helper VM on the same server that could mount the file server disks so it wouldn't use NBD?
All they need is basic vmware licensing (which they need anyway to even do backups) and just a barebones server VM with the backup proxy installed....
TRE3
Enthusiast
Posts: 32
Liked: 5 times
Joined: Apr 11, 2018 7:47 am
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by TRE3 »

The customer just bought a new NIC for the physical backup server that will be used for iSCSI traffic (SAN based backup) in a few days. Nevertheless, when adding new virtual disks to the file server, he cannot be sure that not all the disks get read fully again.
HiHoItsOffToWorkWeGo
Enthusiast
Posts: 45
Liked: 2 times
Joined: Jan 20, 2021 1:06 pm
Full Name: Mark Fellensiek
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by HiHoItsOffToWorkWeGo »

My experience with a 40TB VM with 20x VMDK disks is that if one disk is changed, such as expanded in size, then only that one changed disk gets read in full during the next incremental Veeam backup.
username because using Nutanix Mine with Veeam
TRE3
Enthusiast
Posts: 32
Liked: 5 times
Joined: Apr 11, 2018 7:47 am
Contact:

Re: Full disk read when a new VMware disk is attached to VM

Post by TRE3 »

That's what I also thought would be default behaviour but let's wait what the support team says.
Post Reply

Who is online

Users browsing this forum: No registered users and 39 guests