Restore user profile nightmare

[sogapex]

case # 06012526

Hi,

friday, we encounter a problem with a cleaning powershell script = several user profiles were deleted on one of the RDS server. The problem occured at about 13pm and there was a backup at 12h15.
So, I would have think it was simply a matter of restoring the deleted content. I already did a lot of file restore with Veeam (but never directly to the original location), and so I was confident it would not take too long.

So, I launched a restore session, select the first deleted user folder (D:\users\XXX) in the veeam console, and tell it to restore to its original location => in a few minutes, I could see the server remaining disk space shrinking "like hell" (more than 100GB free before the incident) => I was restoring the first user, but there was no room for him ?
I didn't have time to "think" about what was happening (the users were waiting to be able to work). I thought it was because of the fact the volume was "deduplicated" (windows server 2019 dedupe).
In such a case, it is "normal" that the free space would be smaller after restore compare to original size. But not in such a proportion (this first user profil = less than 15GB of data, after restoration = more than 100GB !)
I was in a hurry, I told everybody to quit the server. I stopped it, increased the volume size from 500GB to 800GB and restored all the other impacted users profile. It was quickly for some (a few minutes) and too long for others (more than one hours). I finished at more than 20h this friday => users were not really able to test if everything was ok
There was only 111GB free on the 800GB volume (original = more than 100GB free with 500GB volume)

Saturday : there was a dedupe task during the night, the free space was 218GB this morning.
I searched where all this data was coming from. By selecting the whole volume content and displaying its properties, I got 412GB. But when looking at the disk free space, it told me 218GB (this is a 800GB volume now) which meant 582GB written. So, there was a difference of 582-412=170GB hidden files.
I launched "spacemonger" utility => it gave me the same numbers = 218GB free and about 415GB written.

Sunday :
With little hope, I decided maybe it was a dedupe problem and launched a few chkdsk + dedupe optimization + dedupe garbage collection...
Chkdisk reported more than 2 million files. Whereas I could only see about 1.4 millions
I then decided to "undedupe" the volume => 154GB free
I searched more, and somebody (forum) was saying to use the software named "TreeSize".
I gave it a shot => it told me the "truth" = more than 2 millions files, 508GB for the "users" folder
This tool display the folders by size, and I could see the first folder = the one of the first restored user => 104GB for this user alone !
When looking at the details : there was a "mess" with the "appData", "Application Data" and "Local settings". Thousands of files were duplicated several times.
There was more than 100 000 files for this user alone. (and more than 45 000 folders for the "best" user impacted).
I opened a case on the Veeam website, hoping it was a "known" problem when restoring a user profile, and they would help me with a tool to clean this "mess" before monday, so that my users could work normally => answer = "In order to investigate the issue better we will need to check the log files".
I politely told them to wait for that, my priority = finding a way to repair the damages done. (I was already trying to make a powershell script to help me delete the unwanted folders and create the missing "junction" link files). After some more researches, it was ok. But I needed to apply the script for all users, one by one, since some manual steps were needed.
After running this cleaning : "users" folder was 287GB (more than 570 000 files and 190 000 folders deleted)

Now, I had to connect with every impacted user credential to see if everything was fine : for some, the restore operation was not complete, and the start menu, or the task bar links were not fonctionnal. => In such case, I had to restore again from veeam (but not the whole profile folder = I had to unselect the "junction" link files), run the powershell script again and test with the user credential again. It was a marcellous sunday, starting at 6h30am and finishing at 22h00... (less 15min to have a meal in the whole day).

But today : everything seems fine for my users, nobody is complaining about something related to the "friday incident".

This is a good ending story, but now, I have to document all this problem, find with Veeam the reason why a backup software is not able to properly restore a user profile, and also ask you this question : What is your best practice to restore a whole user profile ?

In our case, we have 2 RDS servers. Some users should never connect to both servers, and some have to.
And so, when doing some cleaning, an administrator could wrongly delete a user from one server => in such case, we are counting on our favorite backup software.

[HannesK]

Hello,
that sounds like trying to restore with symbolic links that did not resolve. My guess is, that it's not really a Veeam issue, but a general challenge when you try to copy & paste something with links between volumes.

One should see the same challenge when doing the same copy & paste with Windows Explorer.

Best regards,
Hannes

[sogapex]

Hi Hannes,
when I try to copy a "JUNCTION link" file from someone profile, nothing happen. I can't open it by double-clicking on it neither. I suppose there is a protection to avoid doing "foolishness" (permission preventing "execute")
I also suppose this is not a "veeam only" problematic. (other backup softwares should present the same issue)
But : as a backup software, I would assume such a problem is "addressed".
I can't believe I am the first one to have to restore a whole windows user profile => What is the best practice (or the right way ?) to restore a whole user profile ?

[sogapex]

So, here is the definitive answer from support =

It can be worked around by using Instant recovery and copying the files from the restored VM as described in the following KB article: https://www.veeam.com/kb1459
Unfortunately, Veeam does not have any predefined best practices, which can be used for the restore of User profiles.

Ok, as far as I can see, we can’t copy these special links from one location to another because there is special permission on them.
So, there are 2 solutions to restore a complete user profile I guess =
1. Remove the user from the server, and then, join the server as if this was the first time, and then, restore with Veeam all the content of the user profile folder, except all these «special » links.
2. restore with Veeam all the content of the user profile folder, except all these «special » links and then, run my powershell script to create the special links.

I will try to remember this "good" lesson if I encounter this scenario one day

[FrancWest]

Why not switch to FSLogix for the user profiles? This way you only have to restore the vhd, which is way faster than restoring thousands of files within the userprofile. Also, symbolic links aren’t an issue with this setup.

[sogapex]

Hi,
I have absolutely no experience with FSLogix (first time I see this name). In my case, it was a matter of keeping what the users have done since the last backup (yes : users were working when the problem occured, and so, I wanted to keep their new files on the desktop, download, documents folders... etc, and add all the files from the backup)
regards,
Michel