Unusable OS after baremetal restore, backup files missing all files in userprofile

[mlabbe]

Hi,

I had a laptop that glitched and powered off at the wrong moment a couple months ago, and when I tried to do a baremetal restore, the restore operation completed without issue using recovery media but then the OS was unable to boot, reporting error "The User Profile Service failed the sign-in. The User Profile cannot be loaded".

The OS has a single user profile (administrator account disabled by default and no alternate user account created to save the day on moments like this, wrong idea) configured with autologon so there was no way I could reach desktop (not a Veeam issue, but the point is that restoring a "supposedly reliable" backup couldn't get the OS back running as it should).

I tried restoring several different backups from different chains (vib's, vbk's and multiple ad-hoc) and they all had the same result. All of them had been backed up successfully (over several months), showing no error in systray or on report I received via email.

The backup files are stored on a NAS (via SMB share) and the filesystem on this share is BTRFS on RAID5.
Scheduled integrity scans + S.M.A.R.T. tests that both are run every months reported no sign of corruption or drive/filesystem issue (and no error during restore operation), so backups file corruption seems highly unlikely (especially on that number of vbk and ad-hoc).

I ended up being able to restore a very old backup (over 2 years old, backed up using Veeam Endpoint 2.1 instead of VAW 3.0.1) which finally was able to recover the OS booting fine (after dealing with another issue with Samsung's Magician's Rapid mode, but that's another story) and the rest of documents were in more recent backups so I didn't lose too much stuff overall. So the laptop is fixed, but now I'm worried on Veeam backup files's reliability

After further investigation, I realized that all the "non-working" backups don't include any file in %userprofile%. All I see there are empty (sub)folders, which explain why Win10 couldn't load the userprofile at boot.

This laptop wasn't backuped very often (maybe once a month as it's not used regularly) but again, all reports were successful.
Backup task is configured "file-level backup" because I select only a few folders on drive D. However, "Operating System" is selected (so no exclusion on OS drive) and nothing in Advanced File Filters. I really don't see how these files could end up being excluded from the backup files.


I opened a ticket (ref. #03709976) to have someone check the log files and try to explain how this could've happened but it just got automatically closed without being processed, due to high support team load.

I understand support for the free version is based on support team's load, but this looks like some serious bug that I had hoped it would catch support attention.

I still have the "bad" backup files, including VAW logs for investigation.

[Dima P.]

Hello mlabbe,

Thank you for the detailed report. I've asked support team to reopen the case and start the investigation process. Stay tuned.

[mlabbe]

Thanks Dima.

[kgajda]

Dima, could You update status of this issue, please?

[Dima P.]

The case is still in progress, so, unfortunately, I got no information to share.

[kgajda]

Ok. Thanks

[mlabbe]

Dima, can you take another look at the ticket status please?

No rush here, but I haven't had any follow-up for more than 2 weeks.
I pinged for a status update last week and still got no answer.

Thanks.

[Dima P.]

Hello Michel,

I can see that case owner from the support side is currently out of office. I've asked support management to assign another engineer (or escalate the case) as soon as possible. Thank you for bringing this up!

[mlabbe]

Hi again Dima,

I forgot following up here last time, I had checked the thread a couple times for a couple days following my last post, then I actually got a reply before I noticed you had replied to this thread. It's a bit late, but "thanks Dima"


The case was still in progress, but I took a bit too long to reply to the last email, and while replying (another long email) today I got noticed that the case is now closed (my fault, I didn't notice the 48h deadline before archiving mention in the "First Follow up" email and the email subject kinda mislead me into thinking there was a "Second/Final Follow up" email before the case gets closed).

The emails we have been exchanging in this case are very long and technical (on both sides) so we need a couple hours ahead everytime to read carefully, retest and reply something that makes sense. I've been very busy in the last couple weeks but I should've at least replied a quick note that I'd get back later so the case didn't automatically close. Sorry about it.

As you guys showed alot of interest in finding the root cause (as mentioned "to make sure Veeam is a completely reliable product"), is it possible to reactivate the case so we can resume where we were at?

I'm also not sure if the reply I sent today was added to the (closed) case anyway or not.

Thanks.

[Dima P.]

Hello Michel,

Thank you for the update. I'll ask support team to reopen the case, stay tuned!

[mlabbe]

Awesome.

Thanks again Dima.

[Dima P.]

Hello Michel,

I got an update from our support team that the mentioned behavior was caused by UI glitch. During backup job configuration user folder has been excluded but then job was reconfigured to include the operating system volume. Folder remains in the exclusion list, however, it is not reflected in the UI. This bug is marked critical and will be fixed in the upcoming version.

Thank you for your help while we were investigating this issue and sorry for any inconvenience caused. Cheers!