VEP keep failing + enhancement suggestions

[bhwong]

The rate of a successful backup for VEP is very low in my environment, not to mention that in the rare few times when it completed successfully, it took about 2-3 days for it to complete a backup of folders under 2TB, far slower than the default Windows Server Backup, which took less than 24 hours for the same set of folders. Why is this so?

I have a few suggestions to improve this:

1. The performance should improve if VEP can make use of existing persistent shadow copies instead of creating a temp one every time it started it's backup job.

2. Allow configuration of DataMoverLocalFastPath and EndPointBlockSize from the UI instead of editing the registry. Better if there is a slider for administrator to adjust between performance vs reliability to make this more user friendly.

3. Ability to resume from previous failed backup job, instead of starting a recovery point.

4. WAN accelerator and compression will be helpful.

[Dima P.]

Hi bhwong,

Thanks for the prompt feedback.

successful backup for VEP is very low in my environment, not to mention that in the rare few times when it completed successfully, it took about 2-3 days for it to complete a backup of folders under 2TB

May I ask is that a file level backup or volume level backup? Did you open a support case regarding this issues?

DataMoverLocalFastPath

Connection between VEB components via shared memory is supported out of the box, so I am not sure why you are referencing to this registry value

EndPointBlockSize

Will be available in UI in next version

WAN accelerator and compression will be helpful.

Can you elaborate, need to know your potential use case for WAN accelerator in VEB? Regarding compression – the UI control will become available in next version as well.

[bhwong]

It's file level backup as mentioned "same set of folders".

And yes, I have open a support case where support suggested me to add DataMoverLocalFastPath in the registry to improve reliability for backing up over the WAN connection. It does help as I finally manage to get one successful incremental backup. But the failure rate is still too high. So support suggest that I backup to local NAS and then copy the incremental backup to remote NAS. However, the failure rate for local backup is still very high as well.

I don't mind slow performance in exchange for better reliability. However, VEP is both slower and less reliable than our default Windows Server Backup. The main reason why I wanted to give VEP a try is that it support incremental backup that Windows Server Backup do not. I expect VEP will help to reduce the time and WAN bandwidth required to backup our shared folders in our Windows Storage Server over to our offsite storage, but unfortunately, it keep failing.

[Dima P.]

May I ask why you are using file level backup instead of volume level backup?

[bhwong]

Because I just want to make backup of the frequently changed shared folders without the OS, applications, temp files etc that will consume my backup storage, especially when I intend to keep multiple copies.

I need a backup software that can copy locked file and preserve NTFS permission. Most sync applications I tried do not support these. And Windows Server backup do not support incremental backup.

FYI, previously, even Windows Server backup keep failing. The problem was resolved once I remove Windows Storage Server dedup. It's quite shocking that Microsoft failed to verify that their own backup works with their own deduplication before releasing them.

btw, this is a physical server, not a virtual machine. Will it be able to restore to new server using difference hardware and drivers?

[bhwong]

Veeam Support has told me that it's our unreliable WAN that cause offsite backup to fail despite the fact that our Windows Server Backup has done the same successfully and repeatedly over our WAN connection. So I'm not very convinced with this explanation.

FOr the benefit of doubt, I have even deployed a new NAS from another brand (Synology instead of QNAP) and this is connected directly to the same switch as the server we want to backup, so there is no "excuse" of any unreliable network connectivity issue.

Guess what? Veeam backup keep failing too! I have selected the same set of folders (under 2TB) for both VEP and WSB (Windows Server Backup) and WSB has backed up successfully the next day while Veeam was still at 25% despite both started around the same time. I'm puzzled over the extremely slow performance. And after more than 65 hours later, it still failed to complete a simple local backup job.

Why is such a high failure rate and poor performance be possible? Especially it's from a good reputable backup vendor like Veeam? What could be the other possible cause?

Error: Timed out to wait until the size of mount cache will be less than threshold Agent failed to process method {FileBackup.SyncDirs}.

[Dima P.]

Because I just want to make backup of the frequently changed shared folders without the OS, applications, temp files etc that will consume my backup storage, especially when I intend to keep multiple copies.

Try to run a volume level backup with a system folders being excluded. Due it's nature volume level backup is much faster, specifically when talking about huge amount of data.

Will it be able to restore to new server using difference hardware and drivers?

If you have a recovery media and volume level backup of entire OS, then yes.

Why is such a high failure rate and poor performance be possible?

Unfortunately, support team was right. VEB 1.5 is not designed for large file level backups over slow WAN links (it does not have the required reconnect logic and such). An option to perform a direct backup to the cloud connect repository and backup cache feature are coming in the next version. Both should solve your issue completely.

[bhwong]

Thanks! Glad to hear that our feedbacks are noted and I can't wait for the next version!

FYI, I have also tried local backup with direct connection and it failed too! This is local 1GB ethernet network, not over the WAN.

But I will give the volume backup excluding system folders a try once VEP has complete the local backup I'm retrying. I didn't know that a volume backup can exclude folders.

[Dima P.]

I didn't know that a volume backup can exclude folders

This is a bit tricky: choose File level backup mode, select the volume and then uncheck the folders. Label close to the volume should indicate that you are still using volume level backup.

[bhwong]

I tried volume level backup and it still failed with this error:

Unable to exclude files from snapshot: Shadow Copy Optimization Writer timed out (see Windows event log for more info).

[bhwong]

I also notice that despite I have selected only 3 folders to backup under volume level backup, it still backed up everything in the whole volume!

[Dima P.]

it still backed up everything in the whole volume

Looks like VSS exclude operation has failed. Hold on, I'll check with the QA guys. Will follow up shortly.

[bhwong]

Hi Dima,

I'm still trying to get it to works in my environment and I realize 2 issues:

1. I tried to make it run as volume level backup and I did saw it indicating volume level backup under the Files screen. However, when I reached the Summary screen, it show Backup mode: file level backup. (is it possible to attach my screen capture here?)

2. I have reconfigured my 2TB selected folders backup to 192.168.100.245, but the log shows that the total backup size is 6.46TB and it's still trying to backup to 192.168.100.241 which was my original configuration. No wonder it fail everyday.

[Dima P.]

1. I tried to make it run as volume level backup and I did saw it indicating volume level backup under the Files screen. However, when I reached the Summary screen, it show Backup mode: file level backup. (is it possible to attach my screen capture here?)

Just share the link to the screenshot.

I have reconfigured my 2TB selected folders backup to 192.168.100.245, but the log shows that the total backup size is 6.46TB and it's still trying to backup to 192.168.100.241 which was my original configuration. No wonder it fail everyday.

So, you mean the destination path was not updated in the job settings?

[bhwong]

Here's the screen capture that show volume level backup:


At the end, it show file level backup instead:


The new saved destination path to: 192.168.100.245\backup


The actual path still return to previous destination: 192.168.100.241\veeam:

[Dima P.]

Backup is performed on a volume level, so it looks like the UI glitch, shared with QA guys - they will investigate. Thanks.

Regarding the IP address, that’s an interesting issue. To clarify, you just changed the IP or path to the backup location was also modified?

[bhwong]

I think forcing a volume level backup thru file level backup won't work in my environment because of this: https://www.veeam.com/kb2073

I'm changing both the IP and the path. Support said I might need to reset my database to overcome this.

[Dima P.]

Hi,

This KB is a bit outdated (refers to 1.1.2 version) - we implemented several fixes around the exclude logic. Can you clarify was the backup issue with the destination resolved by resetting the database?

[bhwong]

Veeam support said this: "Shadow Copy Optimization Writer timeout is set by Microsoft, and it's value cannot be changed. That warning message appears because large amount of data is excluded and cannot be processed by Shadow Copy Optimization Writer for it's timeout."

So the technique to perform a volume level backup for file level backup is out.

Why is it so difficult for Veeam to make a simple successful backup?

[Dima P.]

Check your forum's inbox and let me know if the suggested solution works. Thank you.

[bhwong]

Still not working. The error now is: Timed out to wait for traffic control event. Agent failed to process method.

I also notice the total backup size shown in Veeam is wrong: 7.81TB. My selected folders are less than 2TB.

[Dima P.]

So, you've added the timeout value like suggested and that did not help, right?

[bhwong]

Yes, I added the timeout value and that didn't help. But at least I don't see timeout error now.

Veeam support suggest that I either use volume-level backup of that disk without any folders exclusions, or try some high-performance storage device and network connection (like 10 gbps).

The question is, why is it that windows server backup works with file level backup and can even able to complete the backup job over 1gbps network or even the slow WAN connection without the need for any high performance network Nor storage?

Shouldn't Veeam be more superior? Why is it keep failing over such a simple backup job while Windows Server Backup works everytime? I don't understand.

[Dima P.]

Thanks for the heads up. I’ll discuss your issue with our team meanwhile the suggested approach sounds good.

P.S. What is the size of the excluded files?

[bhwong]

The excluded files size is about 1.2 TB.

The volume capacity is 9TB, with 3 TB used. I need to backup just 3 shared main folders of about 1.8TB and not everything.

[Dima P.]

Thanks. I guess that’s too much for VSS exclusion engine, thus file level backup approach is preferable. We will discuss you case with QA team.

[bhwong]

btw, is there a way to control how many full backup copies to keep? My storage was full as VEP has made a second full backup instead of incremental backup.

[Dima P.]

Hi bhwong,

Generally, VEB should have only one full backup for backup chain due the nature of forever forward incremental backup approach. At rare circumstances, it can start a new backup chain, thus new full backup is created. Check the exact reason with support team as log investigation is required.

Additionally, double check that your restore points are operational via File Level Recovery.

[bhwong]

Is there any update to fix the high frequently of backup failure yet?

btw, is it possible to manually trigger it to start a new backup chain so that I can remove all the existing incremental backup files from the old backup chain?

[bhwong]

Support claim that the new 2.0 will solve my problem and unfortunately, the backup continue to fail everytime even after upgraded to 2.0.

On the contrary, Windows Backup has rarely fail backing up the same set of folders. Thus, the reason given that this is too much for VSS exclusion engine to handle, seem to suggest that Veeam has implemented a flawed design?

btw, it will be great if we can merge incremental backups into the full backup instead of creating a new full backup over the WAN.