[FEATURE REQUEST] Persistent CBT

[scidoner]

Hey guys,

Got Agent for Linux running, it backs up about a 5.5TB volume.

Problem is, when the server is rebooted (cleanly even), the CBT data is lost (I assume from the Veeam kernel module), and Veeam takes forever to do the incremental, as it has to manually scan to see what has changed.

Is this by design, or can I do something to avoid this?

Running CentOS 7, Linux data3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Cheers!

[nielsengelen]

This is currently by design however we may change it in a later version.

[SteveBLGS]

Would it be possible to make the CBT information persistent across reboots? We have a few really large servers that would make it difficult to use VAL on if a reboot in the middle of the week loses the CBT information. In our case, it wouldn't be possible for Veeam re-read the entire server within our backup window during the week. This could prevent us from switching to VAL for these servers. I believe the Windows Agent has this feature now, so I'm hoping it's a planned feature for the Linux Agent.

It would also be nice to retain CBT information if the veeamsnap module is unloaded and reloaded (I've seen recommendations to unload this module when expanding LVs).

Thanks!

[PTide]

Hi and welcome to the community!

CBT persistence has been requested before, so your voice has been added to the feature request. Also, the bug has been fixed, so starting from 2.0 module unloading is no longer required.

May I ask you what filesystem do you use, and how big is your storage?

Thanks.

[SteveBLGS]

Hi,

Right now we're using ext4, but it will likely be xfs in the future. Most of the file systems are using LVM. Our two largest servers are about 13-14 TB each. The majority of it is stored on slower NL-SAS 7.2K drives.

I'm glad to hear that unloading veeamsnap is no longer required with 2.0.

Do you know if CBT persistence is being planned for the next release of the Linux agent?

Thanks!

[PTide]

Yes, we have that feature in mind, however I can't give any ETA right now because there are other features with higher priorities, and this partucluar feature requires significant amount of research and testing. Right now I think that the best what could be done is to keep CBT data persistent only in case of a normal (expected) reboot. I can't see how it could be done for situations when reboot was unexpected (for example, accidental power-loss). I want to believe that would be an acceptable solution, since for server environment its should be a rare thing. What do you think?

Thanks

[SteveBLGS]

I think that would be an acceptable solution.

Thanks!

[PTide]

Thank you for clarification, request has been noted.

[cloggy]

Hi, I recently installed the latest free version of Veeam Agent Linux (2.0.1.665) on my Linux Mint 19 laptop. The initial full backup takes about 23 minutes and the subsequent incremental backup 33 seconds. After completion of the previous incremental backup I rebooted the laptop and started a new incremental backup and this incremental backup takes now 20 minutes, so almost as much as an full backup. Any incremental after this one completes very quickly again.

My question: why does the first incremental backup after a reboot need as much time as a full backup? Am I missing something?

Note that I have installed VAL on another desktop PC, also running Linux Mint 19 and there I see the same behaviour.

Thanks in advance..

[PTide]

Hi,

That happens because changed-block tracking data is reset every time the veeamsnap module is unloaded, which is exactly what happens before reboot. In this case VAL has to perform a rescan operation in order to know which blocks to pick for incremental backup. I consider your question to be a feature request, so I've merged the post to the related thread.

Thanks!

[DGrinev]

Btw, according to the system requirements Linux Mint is unsupported OS.

[cloggy]

Hi, thanks for the quick reply. Why not performing a rescan immediately (or shortly) after a reboot? So, whenever an incremental backup is performed later, then it will not be running so slow...(just a suggestion...).

[cloggy]

Btw, according to the system requirements Linux Mint is unsupported OS.

Nice, but Linux Mint 19 is based upon Debian and Ubuntu 18.04...which are both supported....

[PTide]

Hi, thanks for the quick reply. Why not performing a rescan immediately (or shortly) after a reboot? So, whenever an incremental backup is performed later, then it will not be running so slow...(just a suggestion...).

That might be technically possible and looks like another way to resolve the problem, however it might be difficult to compare current data with its previous image while the data is changing while you rescan it. A possible workaround would be to create a snapshot solely for the purpose of rescan, but that would put more IO load on the system. Anyway thank you for the idea, we will look in that direction.

Nice, but Linux Mint 19 is based upon Debian and Ubuntu 18.04...which are both supported....

As almost any other software for Linux, VAL depends on other packages and libraries. If some package/library is in Debian --> it is in Ubuntu --> it is in Mint. However, if something is in Mint, it does not necessarily mean that it is also in either of the previous two. That is, if some library required by VAL is different in Mint/Debian/Ubuntu, things might break. Also, in my book, "supported" means "has been tested", which is not the case with Mint, unfortunately.

Thanks!

[cloggy]

..though not been tested/verified by you I can share that VAL runs fine on my Linux Mint systems.....the only issue is the time needed for the first incremental backup after a reboot (but this is a known issue and I have to live with it for the time being).

Note that I have not only tested backups but also restores and all tests completed very well...so I keep using it...

I was also a user of VAW but, for several reasons, I decided about 6 months ago to throw windows 'out of the window' and moved the Linux, hence I'm now using VAL.

[PTide]

..though not been tested/verified by you I can share that VAL runs fine on my Linux Mint systems.....the only issue is the time needed for the first incremental backup after a reboot (but this is a known issue and I have to live with it for the time being).

I take that as a feature request. Actually, no - two feature requests!

Note that I have not only tested backups but also restores and all tests completed very well...so I keep using it...

Great! Any comments regarding overall usability?

I was also a user of VAW but, for several reasons, I decided about 6 months ago to throw windows 'out of the window' and moved the Linux, hence I'm now using VAL.

Plot-twist! : )

Thank you for your feedback!

[Morc]

Hi Veeam Team,

Based on https://github.com/veeam/veeamsnap version 4 of the agent added a lot of code regarding this feature. Upon loading, veeamsnap even logs that "Persistent CBT parameters are not set". Is this not available yet due to lack of testing?

[PTide]

Hi,

That and also the feature is not finalized yet. Moreover, it doesn't quite fit our priorities right now. As you you might have heard already, we are currently battling with kernels 5.8 and higher. As soon as we resolve that one, we can switch to other neat driver-related features.

Thanks!

[dejan.ilic]

I would like to raise this issue again.
For security reasons we reboot our Linux servers regulary (once a week) to load any kernel related changes.
Veeam agent lose track of changes every time, so next backup on a larger physical fileserver takes aproximately 15 hours and would it not be for it deduplicating away 99% you would essentialy have a full active backup size every week.

Why cannot the Linux Agent keep track if it was a clean shutdown or a dirty shutdown. It seems that the Windows agent can.
Only dirty shutdowns would require a re-read of every single block imho.

[PTide]

Hi,

Veeam agent lose track of changes every time, so next backup on a larger physical fileserver takes aproximately 15 hours and would it not be for it deduplicating away 99%

Small correction - it's not deduplicating. Incrementals after a reboot are possible because VAL does full rescan (rereads blocks and calculates CRCs).

As for the feature requested - I think Linux agent could track if a shutdown was clean or not. However, there are other features with higher priorities for now. Will try to expedite.

Thanks!

[dejan.ilic]

It seems that this has not been implemented in the agent for Linux v6 that comes with Veeam B&R v12.
Is there any plan for this feature?

[Flole]

I hope it gets fixed eventually. Just for understanding: Is this part of blksnap? Since for debugging I currently need to load/unload the module quite often to provide logs with various setting, which then triggers the super long backups.

[dejan.ilic]

Yes, I'm talking about standard "volume backup". Right now a reboot of our server takes 18h of constant disk reading by Veeam to be able resume with normal backups.
And as we reboot Linux servers weekly, this is quite many hours every month.

[PTide]

Hi,

May I ask you what's the reason to reboot the server weekly? Not that I am going to tell you not to do so, but to better understand your use-case.

Thanks!

[dejan.ilic]

-The reboot is part of our security and operations routine.
We then know that we are running the current kernels and any changes that might break anything during startup is at most a week old, especially as we autoupdate our servers to keep all the software up to date.