Veeam Agent deselected recovery partitions problem

[Anguel]

Hi,

on our physical PCs we have SSDs containig the OS + our data (drive letter C:) but we also have internal hard drives for additional backups (drive letter R:).
We need to backup the complete SSD contents to our central Veeam Repository but we do not want to backup the hard drive R: because it only contains additional backups which are large and will mean a waste of space in the repository.
So the only choice is to use Volume Level Backup in Veeam Agent and to select the C: drive only - this will also include all hidden volumes like recovery partitions on C:.

Unfortunately I notice that on our PCs e.g. after Windows 10 Upgrades (or maybe even Veeam Agent upgrades) some recovery partitions constantly become deselected so Veeam does not back them up anymore.
I need to go to the backup wizard and reselect the C: drive which will also again reselect all hidden volumes, however this is not a solution to the problem at all.
Please address this issue.

EDIT: On some PCs I even got an error that a volume does not exist any more after a Windows 10 Upgrade - probably Windows changed something in the partition and Veeam did not notice it. There must be a way to make Veeam rescan the volume and to backup what is actually there for a particular drive.

Anguel

[Anguel]

I tested a little more and what really scares me is that if I deselect one of the recovery partitions manually (as Veeam Agent does without any notification!) I get this warning:
"You will not be able to use the produced backup for an entire computer recovery."
There must be a way to fix this behavior - it does not appear if we backup the entire computer so it should also be possible to make entire drive backups consistent...

[Anguel]

This is definitely a problem as it appeared on all PCs (different hardware) that do volume level backup of single drives.
Veeam Agent either deselects some of the recovery partitions without noticing it or it points to some old partitions it cannot find anymore which results in an error:
25.07.2018 10:42:11 :: Error: Cannot find volume \\?\Volume{c003ec49-fe2d-48e2-937b-a8f9665ea71f}
IMHO the way Veeam looks at a drive must be changed, so that it sees a drive as a whole (no matter if partitions change) and not as multiple partitions which can obviously change after Windows 10 upgrades.
Again, this problem does not appear with full computer backup, only with volume level backup.
Please pass this information to your developers.

[Dima P.]

Anguel,

What agent version are you running? Thanks.

[Anguel]

Agent version is latest 2.2.0.589 but I observed such behavior in older versions too.
I think this happens after Windows 10 upgrades, eg. 1709 to 1803.

Notice in the screenshot how one of the recovery partitions is not selected, although the C: drive is still selected:
https://drive.google.com/open?id=14LTyE ... 7FtcfyuwUT

In other cases I get the "Cannot find volume" error described above, so I again need to reselect the C: drive manually to fix the problem:
https://drive.google.com/open?id=18vKWJ ... _2BjNiSxZl
After fixing the problem Veeam then notices that the volume bitmap has changed:
https://drive.google.com/open?id=15OtAn ... 6lO-ZPORZM

Again, note that these problems only occur when doing volume level backups of single drives, not when backing up in the whole computer mode.

[Dima P.]

Thanks. It reminds me the problem we had with previous version of Veeam Agent (disk layout was modified during Windows updates and that affected the source objects). This issue should be fixed in latest version, so may I ask you to submit a case and share the ID with me. Thank you in advance.

[Anguel]

Here some more screenshots from a PC I upgraded from Windows 1709 to 1803 yesterday, Agent was latest version 2.2.0.589 in all cases.

Before the upgrade:
https://drive.google.com/open?id=1gsRRj ... cpeMZnMx-7

after the 1803 upgrade Veeam had deselected one partition without notice:
https://drive.google.com/open?id=1VaP5p ... QwJ55YNNzc

This problem occurred on all kind of PCs here and should be therefore reproducible I think. It is only important that the mode used is volume level to reproduce.
I will submit a case and refer to this thread.

[Anguel]

Case ID is 03120814

[Dima P.]

Thanks! Asked QA team to review the logs.

[Anguel]

Now I found this older thread - same problem as here after update to Win 1709, read from here, and my workaround is the same as described in this thread:
https://forums.veeam.com/veeam-agent-fo ... ml#p259245

So the bug still persists. I had forgotten that I wrote about this issue already. Did Veeam actually try to reproduce that?

[Dima P.]

Thanks for the heads up! We are discussing this issue with the team.

[Anguel]

Support confirmed my expectations: Windows is changing the partitions after upgrades, so this has to be expected and I thought that Veeam developers were well aware of this.

I think there are many Veeam customers that just don’t notice that some partitions are not selected anymore. The problem is that a complete drive recovery is not possible if those partitions are missing in case of drive failure for example.

Maybe you can also pass to the developers that in a few cases Veeam backups just fail after the Windows upgrade because it cannot find a partition with a specific ID for the same reason.
I am looking forward to a fix soon, as stated the key is to be able to backup complete drives. Most backup programs offer this option.

[Dima P.]

Hello Anguel,

Thank you for sharing the heads up!

Windows is changing the partitions after upgrades, so this has to be expected and I thought that Veeam developers were well aware of this

That's true.

there are many Veeam customers that just don’t notice that some partitions are not selected anymore. The problem is that a complete drive recovery is not possible if those partitions are missing in case of drive failure for example

That's not true similar to your case job fails whenever Recovery Partition has been modified by Windows Update. This issue affects only Volume level backup and does not affect recovery from restore points created before such job failure.

Maybe you can also pass to the developers that in a few cases Veeam backups just fail after the Windows upgrade because it cannot find a partition with a specific ID for the same reason.

We are discussing the solution. The temporary fix would be to either switch to Entire Computer Backup or File level Backup and select 'Operating System' check box as it dynamically updates the list of system partitions after Windows upgrade.

[Anguel]

That's not true similar to your case job fails whenever Recovery Partition has been modified by Windows Update. This issue affects only Volume level backup and does not affect recovery from restore points created before such job failure.

I am not sure, if I remember correctly in most cases the backup did not fail, it just did not backup the newly created recovery partitions. I think the job only fails if Windows modifies an existing partition. Therefore I think that most customers will not even notice the problem until it's too late.

The temporary fix would be to either switch to Entire Computer Backup or File level Backup and select 'Operating System' check box as it dynamically updates the list of system partitions after Windows upgrade.

Switching to Entire Computer is not possible because of the additional large local disk drives that we do not want to backup. We need complete computer images to restore in the event of failure, so file level backup is not an option.

I am happy that you are now aware of the problem and hope that there will be a solution. Something similar to Entire Computer mode with the possibility to exclude certain drives would be the best I think.

[Anguel]

It looks like this problem / bug has been solved by the Veeam developers meanwhile by adding the "Operating System" selection option:
https://helpcenter.veeam.com/docs/agent ... tml?ver=30

Thank you for this!

Anguel

[Dima P.]

Hello Anguel,

Yes, we've added operating system checkbox to the volume level backup mode to make sure that backup scope is not affected due to the volume modifications during Windows upgrades, so for now it's recommended to keep operating system checkbox check if you want to protect operating system (including the system partitions) via volume level backup. Thanks for the confirmation!