Not trustworthy (yet)

[mmark]

Hello,

One of the computers here went into corrupted Windows installation state (Windows 10 if you curious). No backups at all.
I decided to backup user files first then perform fresh install of Win10 then restore files. Simple task you say...
So, I backed the computer using VEB and to just in case using Trueimage. Both on the volume level. No problem whatsoever.

During restore I realized that several quirks/issues/show stoppers, hope anyone can comment that:

1) File restore from backup doesn't exist. Instead it's just backup browser without "restore" button. It was nightmare crawling thru several directories, pressing copy, choosing destination folder and so on. That's really missing functionality.
2) During "restore" process I realized that some of the files are missing! I could find Outlook .OST files as an example, also some databases were missing as well. WTH??? How I can be sure that all else is still in backup???
Issues from general testing:
3) Where is the option to remove any of the backups? During tests I've created unnecessary backup with tons of temp databases, it's something I'd like to get rid of.
4) I deleted last incremental backup file manually, perform another backup (it was full), then tried to restore; only last backup was visible, all previous backups was gone (backup files are there but restore ignores them).

Thanks,
Mark

PS Finally I launched Trueimage and restored everything I needed (together with .OST files)

[Vitaliy S.]

Hi Marek,

1. Can you post a screenshot of the restore browser? Here is a detailed step by step guide that shows how to restore files and folders > File-level restore
2. Where do you store these files? On the local disks? What backup mode did you use for VEB?
3. These backup files will be removed according to your retention policy. However, you can also remove backup files manually if necessary.
4. Can you please tell me what happens if you try to import your existing backup chain to Veeam backup server? Do you also see only latest backup chain?

Thanks for your feedback!

[mmark]

Hi Vitaly,

1) Sure.
2) Local USB drive. As stated in my post; volume level backup. Default settings.
3) When I removed last incremental backup file next time VEB created full backup and "forgot" everything that happened before.
4) Didn't tried that. Will try. Both volume restore and file restore sees only last full backup and nothing happened before that. See above.

Thanks,
Mark

[Vitaliy S.]

2. Ah... that explains it. With VEB you can only backup local disks and disks that are connected via iSCSI. Direct attached disks (USBs) cannot be backed up, see this topic for more info > External Drive not being backed up

[Dima P.]

Hello,

File restore from backup doesn't exist. Instead it's just backup browser without "restore" button. It was nightmare crawling thru several directories, pressing copy, choosing destination folder and so on. That's really missing functionality.

I am not sure I follow and screenshot is not opening for some reason. Kindly take a look at this post - it explains a lot Recovery possibilities in depth: File level recovery. For instance you can use C:\VeeamFLR folder to browse the backup once the file level recovery is started

During "restore" process I realized that some of the files are missing! I could find Outlook .OST files as an example, also some databases were missing as well. WTH??? How I can be sure that all else is still in backup???

OST file is excluded from MS VSS snapshot by default. Actually, this behavior is more likely expected in other backup solutions unless the following workaround is implemented Oops? Windows 7 Backup omits key Outlook mail files and this one might be a good explanation why its happening: Why CompletePC Backup does not include .ost files in the backup image

As we figured out the files stored on USB drives are not included in the backup. I assume you had a symbolic link on the disk, is that correct?

3) Where is the option to remove any of the backups? During tests I've created unnecessary backup with tons of temp databases, it's something I'd like to get rid of.
4) I deleted last incremental backup file manually, perform another backup (it was full), then tried to restore; only last backup was visible, all previous backups was gone (backup files are there but restore ignores them).

No reasonable explanation here, as it does not sound like expected (unless with .vib file, .vbm was accidentally deleted). Please open a support case to investigate this behavior.

[mmark]

2) No. I mean that files are stored on local C: drive (SSD to be precise). Backup file is stored on local USB disk. Sorry for confusion.
Point is still valid. Some files are missing in backup file.

[mmark]

1) Screenshot once again:
2) thanks for links Dima. it explain the issue. Anyway would be better to have a control over excluded files. Competitor software has no problem with this. No, file are store on SSD, no symbolic links here.
3.4) Try it yourself. Delete last incremental backup and see what happened. Will double check again in spare time.

Thanks,
Mark

[Dima P.]

Now I see – thanks Mark! The restore choice is displayed when the File level recovery is launched at the original operating system/machine, so the folder structure is preserved.

That explains only the missing .ost (third party backup software, I believe, just modifies the regkey as described in the first article I’ve shared), but not the files – please open a support case via Endpoint’s Control Panel > Support: support folks will take a look at the logs and such.

Will check the incremental issue you described. Meanwhile, please check might be the files are just not automatically located (again because of non-original location) and must be browsed manually at the first step of the recovery wizard (browse backup)

[mmark]

To sum up; the issues I experienced are somehow show stoppers (at least for me) and I'd like to see two extra features in next builds:

1) Add ability to restore files from backup even if it's not original OS/machine. Scenarios like reinstall OS, setup additional machine for the same user or simple machine upgrade aren't exotic...
2) Let user control the list of excluded files/folders. I'd like to backup .ost files! Restoring 8GB ost file from exchange isn't fast...
BTW What else is excluded from backup by default?

Will do another tests with missing backups in spare time. Will keep you posted.
Mark

PS Only .OST files were missing in VEB backup, I'm so sorry for misleading info. Looks like other mentioned files were moved to another location between VEB and Trueimage backups. Mea culpa.

[Vitaliy S.]

Here is the list of all files that are excluded by default:

Veeam Endpoint Backup automatically adds to the list of exclusions the following Microsoft Windows objects for all computer users: temporary files folder, Recycle Bin, Microsoft Windows pagefile, hibernate file and VSS snapshot files from the System Volume Information folder.

[Dima P.]

Thanks Mark!

Add ability to restore files from backup even if it's not original OS/machine. Scenarios like reinstall OS, setup additional machine for the same user or simple machine upgrade aren't exotic...

By the way, I was explaining why there is no “Restore” choice and only “Copy to” option. Thru copy to you can restore the file from the backup browser UI to the desired location

[mmark]

Thanks Mark! By the way, I was explaining why there is no “Restore” choice and only “Copy to” option. Thru copy to you can restore the file from the backup browser UI to the desired location

Great explanation Kind of "It's not a bug, it's a feature!" Oh, just keep both "Copy" and "Restore". Yin and Yang. Balance. Everyone happy...

BTW Would be great to have checkboxes too and just one restore button. Much more productive in case of restoring many huge files.

Here is the list of all files that are excluded by default:

And .OST Remember that. I will

Cheers guys,
Mark

[Dima P.]

Thanks for sharing your thoughs Marek! We will pay attention to your feedback during next version improvements discussions. Cheers.

[Gostev]

2) Let user control the list of excluded files/folders. I'd like to backup .ost files! Restoring 8GB ost file from exchange isn't fast...

Well, it is definitely configurable via a registry edit (and Dmitry has already provided the link explaining the process earlier), however Microsoft specifically advises against making this change. Just keep in mind that OST skip is not a Veeam specific exclusion, it is a system wide exclusion introduced by Microsoft themselves due to significant impact on disk performance.

PS Only .OST files were missing in VEB backup, I'm so sorry for misleading info. Looks like other mentioned files were moved to another location between VEB and Trueimage backups. Mea culpa.

Sounds good, thanks for coming back on this one.

[snadam]

I'd like to hear from folks that have made the registry change AND had a chance to test restoration in the real world. If you've restored an old OST backed up by Veeam after changing the registry key, did it work, did you encounter any issues, and have you seen any other related issues with backing up Outlook data?

I'm considering the change but want to know what others have experienced before jumping in. I'm testing but only on one machine and want as much input as possible before applying this to production users.

Thanks in advance,
Adam

[Dima P.]

Adam,

Honestly, restoring a really old .ost file makes no sense. Outlook will start to sync .ost with your mailbox after recovery and eventually download all new content from exchange server.

[Qbasix]

I'd like to add my experience just now with Veeam Agent for Microsoft Windows 2.2.0.589 Free Edition

I was doing a bare metal recovery to dissimilar hardware (laptop 1 to laptop 2)

All went well however upon starting Outlook 2016 many emails and folders were missing.

The laptop in question had IMAP configured (not exchange) and the local files were stored in .ost format.

However, Outlook allows storing folders / emails / objects in the .ost file which are marked as "(This computer only)" yet still stored in the .ost file.

The problem here is that these files are NOT on the server, and therefore LOST if the .ost file is not backed up / restored.

Luckily the old laptop was still running so I could manually copy the outlook folder to the new laptop, this fixed the issue.

This would have been a major problem if the source laptop (hard drive) was not available due to crash etc... those files in these "local ost" files would be gone forever.

It's probably not best practice to store files / emails in local .ost files but Outlook permits this by default. So I'm sure many people have local only files stored in .ost files.

This I believe is a serious issue in Veeam backup.

[Qbasix]

Also forgot to add that many of the user settings / preferences are also gone if the .ost and .oab files are not backed up.

If we are doing a volume level backup it should be exact.

[Vitaliy S.]

Wojtek, please take a look at this topic, as not only does it explain why OST files are not backed up, but it also gives you a workaround on how to start backing it up. Hope this help!

[Qbasix]

Hi Vitaliy,

I understand why it does not but believe Veeam should be setting those flags to off / or backup as serious, important data can be lost and this was in turn create distrust in Veeam backup.

In my case the important emails would have forever been lost if the old hard drive was not available. The point is .ost files DO STORE LOCAL data... and this data is gone if the backup does not restore the .ost files.

I just did a default backup using Macrium Reflect free version and it DOES backup .ost files and .oab files. So this can be done, you do not have to just rely or follow VSS snapshot rules set by Microsoft.

[Vitaliy S.]

Wojtek, I certainly understand the concern, so thanks for your feedback and the heads up! I will share this data with the corresponding PM. Thanks!

[ThomasR]

In the microsoft article it says they excluded those files because of IO activity, but with Veeam VSS is only used when the backup is running (not all the time), so it shouldnt matter, right?

[Vitaliy S.]

If you are backing up the entire image, then yes it should not be a problem:

The performance impact doesn’t occur during the image backup itself–the only extra work at backup time is backing up the .ost file as part of the image.

[Dima P.]

Wojtek,

OST file is a copy of your production mailbox. Basically, it contains all the data for Outlook to work in the offline mode correctly and there is no sensitive data that cannot be restored from your exchange mailbox. It's added to the Microsoft predefined VSS exclusion list by default and VAW simply uses that list during backup.

[Qbasix]

Hi Dima,
I believe you are incorrect here. Outlook creates an .ost file for IMAP accounts too and can by default configuration store "local only" copies of folders, emails and possibly other objects that are NOT on the IMAP server. Exchange is a totally different scenario.

So in such a case if the .OST file is NOT backed up, the user will lose those "local only" objects from Outlook.

I have experienced just this recently which prompted me to find this thread and add my opinion.

Honestly I believe Veeam should be by default backing up .ost files as you will sooner or later find customers who lost data and lost trust in Veeam.

[Dima P.]

Wojtek,

Outlook creates an .ost file for IMAP accounts too and can by default configuration store "local only" copies of folders

IMAP leaves copies on the server. If you are exporting data from the server via POP or IMAP its going to be saved as .pst

[Qbasix]

I think we are going around in circles here...

Anyway points are:

1) Outlook .ost files when used with IMAP can store "local only" data that is NOT on the IMAP server and therefore cannot be restored.
2) I encountered this exact problem and luckily had the original data (hard drive) to copy the .ost file back into the restored machine.

I've had to adjust all the registry flags on machines which run Veeam to include .ost files in the backup.

[Dima P.]

Wojtek,

Thank you for the use case explanation! I've added you vote to this feature request. Cheers!

[chaveb]

Wojtek,

IMAP leaves copies on the server. If you are exporting data from the server via POP or IMAP its going to be saved as .pst

I think as Wojtek said, OST files often contain data not synchronised with the server, particularly when configured for IMAP accounts. IMAP is only email, but the OST file created for the IMAP account also contains calendar and contacts data which is NOT synchronised with the server and is lost if that OST file is deleted.

We usually recommend configuring a local PST file to store contacts/calendars for just this reason when people are using IMAP accounts in Outlook.

[techgirl]

"We usually recommend configuring a local PST file to store contacts/calendars for just this reason when people are using IMAP accounts in Outlook."

Yes, this is a poor choice on MS part. Also be aware, should an inexperienced user decide to delete the email account intending to recreate it to start fresh if it wasn't working properly, he/she loses those calendars and contacts as MS deletes those "this PC only" .ost as well when the account is removed.