Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
scidoner
Enthusiast
Posts: 35
Liked: 8 times
Joined: Jul 07, 2013 4:43 am
Contact:

CBT lost on reboot

Post by scidoner »

Hey guys,

Got Agent for Linux running, it backs up about a 5.5TB volume.

Problem is, when the server is rebooted (cleanly even), the CBT data is lost (I assume from the Veeam kernel module), and Veeam takes forever to do the incremental, as it has to manually scan to see what has changed.

Is this by design, or can I do something to avoid this?

Running CentOS 7, Linux data3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Cheers!
nielsengelen
Product Manager
Posts: 5618
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: CBT lost on reboot

Post by nielsengelen »

This is currently by design however we may change it in a later version.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
SteveBLGS
Novice
Posts: 7
Liked: 1 time
Joined: Jan 03, 2018 9:52 pm
Full Name: Steve
Contact:

[FEATURE REQUEST] Persistent CBT

Post by SteveBLGS »

Would it be possible to make the CBT information persistent across reboots? We have a few really large servers that would make it difficult to use VAL on if a reboot in the middle of the week loses the CBT information. In our case, it wouldn't be possible for Veeam re-read the entire server within our backup window during the week. This could prevent us from switching to VAL for these servers. I believe the Windows Agent has this feature now, so I'm hoping it's a planned feature for the Linux Agent.

It would also be nice to retain CBT information if the veeamsnap module is unloaded and reloaded (I've seen recommendations to unload this module when expanding LVs).

Thanks!
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by PTide »

Hi and welcome to the community!

CBT persistence has been requested before, so your voice has been added to the feature request. Also, the bug has been fixed, so starting from 2.0 module unloading is no longer required.

May I ask you what filesystem do you use, and how big is your storage?

Thanks.
SteveBLGS
Novice
Posts: 7
Liked: 1 time
Joined: Jan 03, 2018 9:52 pm
Full Name: Steve
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by SteveBLGS »

Hi,

Right now we're using ext4, but it will likely be xfs in the future. Most of the file systems are using LVM. Our two largest servers are about 13-14 TB each. The majority of it is stored on slower NL-SAS 7.2K drives.

I'm glad to hear that unloading veeamsnap is no longer required with 2.0.

Do you know if CBT persistence is being planned for the next release of the Linux agent?

Thanks!
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by PTide »

Yes, we have that feature in mind, however I can't give any ETA right now because there are other features with higher priorities, and this partucluar feature requires significant amount of research and testing. Right now I think that the best what could be done is to keep CBT data persistent only in case of a normal (expected) reboot. I can't see how it could be done for situations when reboot was unexpected (for example, accidental power-loss). I want to believe that would be an acceptable solution, since for server environment its should be a rare thing. What do you think?

Thanks
SteveBLGS
Novice
Posts: 7
Liked: 1 time
Joined: Jan 03, 2018 9:52 pm
Full Name: Steve
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by SteveBLGS »

I think that would be an acceptable solution.

Thanks!
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by PTide »

Thank you for clarification, request has been noted.
cloggy
Enthusiast
Posts: 40
Liked: 2 times
Joined: Nov 08, 2015 12:22 am
Full Name: Dick Hoogendoorn
Contact:

[MERGED] First incremental backup after reboot takes as much time as a full backup

Post by cloggy »

Hi, I recently installed the latest free version of Veeam Agent Linux (2.0.1.665) on my Linux Mint 19 laptop. The initial full backup takes about 23 minutes and the subsequent incremental backup 33 seconds. After completion of the previous incremental backup I rebooted the laptop and started a new incremental backup and this incremental backup takes now 20 minutes, so almost as much as an full backup. Any incremental after this one completes very quickly again.

My question: why does the first incremental backup after a reboot need as much time as a full backup? Am I missing something?

Note that I have installed VAL on another desktop PC, also running Linux Mint 19 and there I see the same behaviour.

Thanks in advance..
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

[MERGED] Re: First incremental backup after reboot takes as much time as a full backup

Post by PTide »

Hi,

That happens because changed-block tracking data is reset every time the veeamsnap module is unloaded, which is exactly what happens before reboot. In this case VAL has to perform a rescan operation in order to know which blocks to pick for incremental backup. I consider your question to be a feature request, so I've merged the post to the related thread.

Thanks!
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by DGrinev »

Btw, according to the system requirements Linux Mint is unsupported OS.
cloggy
Enthusiast
Posts: 40
Liked: 2 times
Joined: Nov 08, 2015 12:22 am
Full Name: Dick Hoogendoorn
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by cloggy »

Hi, thanks for the quick reply. Why not performing a rescan immediately (or shortly) after a reboot? So, whenever an incremental backup is performed later, then it will not be running so slow...(just a suggestion...).
cloggy
Enthusiast
Posts: 40
Liked: 2 times
Joined: Nov 08, 2015 12:22 am
Full Name: Dick Hoogendoorn
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by cloggy »

DGrinev wrote: Oct 31, 2018 4:27 pm Btw, according to the system requirements Linux Mint is unsupported OS.
Nice, but Linux Mint 19 is based upon Debian and Ubuntu 18.04...which are both supported.... :D
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by PTide »

Hi, thanks for the quick reply. Why not performing a rescan immediately (or shortly) after a reboot? So, whenever an incremental backup is performed later, then it will not be running so slow...(just a suggestion...).
That might be technically possible and looks like another way to resolve the problem, however it might be difficult to compare current data with its previous image while the data is changing while you rescan it. A possible workaround would be to create a snapshot solely for the purpose of rescan, but that would put more IO load on the system. Anyway thank you for the idea, we will look in that direction.

Nice, but Linux Mint 19 is based upon Debian and Ubuntu 18.04...which are both supported.... :D
As almost any other software for Linux, VAL depends on other packages and libraries. If some package/library is in Debian --> it is in Ubuntu --> it is in Mint. However, if something is in Mint, it does not necessarily mean that it is also in either of the previous two. That is, if some library required by VAL is different in Mint/Debian/Ubuntu, things might break. Also, in my book, "supported" means "has been tested", which is not the case with Mint, unfortunately.

Thanks!
cloggy
Enthusiast
Posts: 40
Liked: 2 times
Joined: Nov 08, 2015 12:22 am
Full Name: Dick Hoogendoorn
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by cloggy »

..though not been tested/verified by you I can share that VAL runs fine on my Linux Mint systems.....the only issue is the time needed for the first incremental backup after a reboot (but this is a known issue and I have to live with it for the time being).

Note that I have not only tested backups but also restores and all tests completed very well...so I keep using it... :-)

I was also a user of VAW but, for several reasons, I decided about 6 months ago to throw windows 'out of the window' and moved the Linux, hence I'm now using VAL.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by PTide »

..though not been tested/verified by you I can share that VAL runs fine on my Linux Mint systems.....the only issue is the time needed for the first incremental backup after a reboot (but this is a known issue and I have to live with it for the time being).
I take that as a feature request. Actually, no - two feature requests!

Note that I have not only tested backups but also restores and all tests completed very well...so I keep using it...
Great! Any comments regarding overall usability?

I was also a user of VAW but, for several reasons, I decided about 6 months ago to throw windows 'out of the window' and moved the Linux, hence I'm now using VAL.
Plot-twist! : )

Thank you for your feedback!
Morc
Novice
Posts: 3
Liked: never
Joined: May 09, 2021 10:26 am
Full Name: Gábor 'Morc' KORMOS
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by Morc »

Hi Veeam Team,

Based on https://github.com/veeam/veeamsnap version 4 of the agent added a lot of code regarding this feature. Upon loading, veeamsnap even logs that "Persistent CBT parameters are not set". Is this not available yet due to lack of testing?
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by PTide »

Hi,

That and also the feature is not finalized yet. Moreover, it doesn't quite fit our priorities right now. As you you might have heard already, we are currently battling with kernels 5.8 and higher. As soon as we resolve that one, we can switch to other neat driver-related features.

Thanks!
dejan.ilic@liu.se
Enthusiast
Posts: 29
Liked: never
Joined: Apr 11, 2019 11:37 am
Full Name: Dejan Ilic
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by dejan.ilic@liu.se »

I would like to raise this issue again.
For security reasons we reboot our Linux servers regulary (once a week) to load any kernel related changes.
Veeam agent lose track of changes every time, so next backup on a larger physical fileserver takes aproximately 15 hours and would it not be for it deduplicating away 99% you would essentialy have a full active backup size every week.

Why cannot the Linux Agent keep track if it was a clean shutdown or a dirty shutdown. It seems that the Windows agent can.
Only dirty shutdowns would require a re-read of every single block imho.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by PTide »

Hi,
Veeam agent lose track of changes every time, so next backup on a larger physical fileserver takes aproximately 15 hours and would it not be for it deduplicating away 99%
Small correction - it's not deduplicating. Incrementals after a reboot are possible because VAL does full rescan (rereads blocks and calculates CRCs).

As for the feature requested - I think Linux agent could track if a shutdown was clean or not. However, there are other features with higher priorities for now. Will try to expedite.

Thanks!
dejan.ilic@liu.se
Enthusiast
Posts: 29
Liked: never
Joined: Apr 11, 2019 11:37 am
Full Name: Dejan Ilic
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by dejan.ilic@liu.se »

It seems that this has not been implemented in the agent for Linux v6 that comes with Veeam B&R v12.
Is there any plan for this feature?
Flole
Influencer
Posts: 11
Liked: 1 time
Joined: Feb 06, 2021 2:43 am
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by Flole »

I hope it gets fixed eventually. Just for understanding: Is this part of blksnap? Since for debugging I currently need to load/unload the module quite often to provide logs with various setting, which then triggers the super long backups.
dejan.ilic@liu.se
Enthusiast
Posts: 29
Liked: never
Joined: Apr 11, 2019 11:37 am
Full Name: Dejan Ilic
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by dejan.ilic@liu.se »

Yes, I'm talking about standard "volume backup". Right now a reboot of our server takes 18h of constant disk reading by Veeam to be able resume with normal backups.
And as we reboot Linux servers weekly, this is quite many hours every month.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by PTide »

Hi,

May I ask you what's the reason to reboot the server weekly? Not that I am going to tell you not to do so, but to better understand your use-case.

Thanks!
dejan.ilic@liu.se
Enthusiast
Posts: 29
Liked: never
Joined: Apr 11, 2019 11:37 am
Full Name: Dejan Ilic
Contact:

Re: [FEATURE REQUEST] Persistent CBT

Post by dejan.ilic@liu.se »

-The reboot is part of our security and operations routine.
We then know that we are running the current kernels and any changes that might break anything during startup is at most a week old, especially as we autoupdate our servers to keep all the software up to date.
Post Reply

Who is online

Users browsing this forum: No registered users and 16 guests