Standalone backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

VEP keep failing + enhancement suggestions

Post by bhwong »

The rate of a successful backup for VEP is very low in my environment, not to mention that in the rare few times when it completed successfully, it took about 2-3 days for it to complete a backup of folders under 2TB, far slower than the default Windows Server Backup, which took less than 24 hours for the same set of folders. Why is this so?

I have a few suggestions to improve this:

1. The performance should improve if VEP can make use of existing persistent shadow copies instead of creating a temp one every time it started it's backup job.

2. Allow configuration of DataMoverLocalFastPath and EndPointBlockSize from the UI instead of editing the registry. Better if there is a slider for administrator to adjust between performance vs reliability to make this more user friendly.

3. Ability to resume from previous failed backup job, instead of starting a recovery point.

4. WAN accelerator and compression will be helpful.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

Hi bhwong,

Thanks for the prompt feedback.
successful backup for VEP is very low in my environment, not to mention that in the rare few times when it completed successfully, it took about 2-3 days for it to complete a backup of folders under 2TB
May I ask is that a file level backup or volume level backup? Did you open a support case regarding this issues?
DataMoverLocalFastPath
Connection between VEB components via shared memory is supported out of the box, so I am not sure why you are referencing to this registry value
EndPointBlockSize
Will be available in UI in next version
WAN accelerator and compression will be helpful.

Can you elaborate, need to know your potential use case for WAN accelerator in VEB? Regarding compression – the UI control will become available in next version as well.
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

It's file level backup as mentioned "same set of folders".

And yes, I have open a support case where support suggested me to add DataMoverLocalFastPath in the registry to improve reliability for backing up over the WAN connection. It does help as I finally manage to get one successful incremental backup. But the failure rate is still too high. So support suggest that I backup to local NAS and then copy the incremental backup to remote NAS. However, the failure rate for local backup is still very high as well.

I don't mind slow performance in exchange for better reliability. However, VEP is both slower and less reliable than our default Windows Server Backup. The main reason why I wanted to give VEP a try is that it support incremental backup that Windows Server Backup do not. I expect VEP will help to reduce the time and WAN bandwidth required to backup our shared folders in our Windows Storage Server over to our offsite storage, but unfortunately, it keep failing.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

May I ask why you are using file level backup instead of volume level backup?
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Because I just want to make backup of the frequently changed shared folders without the OS, applications, temp files etc that will consume my backup storage, especially when I intend to keep multiple copies.

I need a backup software that can copy locked file and preserve NTFS permission. Most sync applications I tried do not support these. And Windows Server backup do not support incremental backup.

FYI, previously, even Windows Server backup keep failing. The problem was resolved once I remove Windows Storage Server dedup. It's quite shocking that Microsoft failed to verify that their own backup works with their own deduplication before releasing them.

btw, this is a physical server, not a virtual machine. Will it be able to restore to new server using difference hardware and drivers?
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Veeam Support has told me that it's our unreliable WAN that cause offsite backup to fail despite the fact that our Windows Server Backup has done the same successfully and repeatedly over our WAN connection. So I'm not very convinced with this explanation.

FOr the benefit of doubt, I have even deployed a new NAS from another brand (Synology instead of QNAP) and this is connected directly to the same switch as the server we want to backup, so there is no "excuse" of any unreliable network connectivity issue.

Guess what? Veeam backup keep failing too! I have selected the same set of folders (under 2TB) for both VEP and WSB (Windows Server Backup) and WSB has backed up successfully the next day while Veeam was still at 25% despite both started around the same time. I'm puzzled over the extremely slow performance. And after more than 65 hours later, it still failed to complete a simple local backup job.

Why is such a high failure rate and poor performance be possible? Especially it's from a good reputable backup vendor like Veeam? What could be the other possible cause?

Error: Timed out to wait until the size of mount cache will be less than threshold Agent failed to process method {FileBackup.SyncDirs}.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

Because I just want to make backup of the frequently changed shared folders without the OS, applications, temp files etc that will consume my backup storage, especially when I intend to keep multiple copies.
Try to run a volume level backup with a system folders being excluded. Due it's nature volume level backup is much faster, specifically when talking about huge amount of data.
Will it be able to restore to new server using difference hardware and drivers?
If you have a recovery media and volume level backup of entire OS, then yes.
Why is such a high failure rate and poor performance be possible?
Unfortunately, support team was right. VEB 1.5 is not designed for large file level backups over slow WAN links (it does not have the required reconnect logic and such). An option to perform a direct backup to the cloud connect repository and backup cache feature are coming in the next version. Both should solve your issue completely.
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Thanks! Glad to hear that our feedbacks are noted and I can't wait for the next version! :)

FYI, I have also tried local backup with direct connection and it failed too! This is local 1GB ethernet network, not over the WAN.

But I will give the volume backup excluding system folders a try once VEP has complete the local backup I'm retrying. I didn't know that a volume backup can exclude folders.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

I didn't know that a volume backup can exclude folders
This is a bit tricky: choose File level backup mode, select the volume and then uncheck the folders. Label close to the volume should indicate that you are still using volume level backup.
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

I tried volume level backup and it still failed with this error:

Unable to exclude files from snapshot: Shadow Copy Optimization Writer timed out (see Windows event log for more info).
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

I also notice that despite I have selected only 3 folders to backup under volume level backup, it still backed up everything in the whole volume!
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

it still backed up everything in the whole volume
Looks like VSS exclude operation has failed. Hold on, I'll check with the QA guys. Will follow up shortly.
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Hi Dima,

I'm still trying to get it to works in my environment and I realize 2 issues:

1. I tried to make it run as volume level backup and I did saw it indicating volume level backup under the Files screen. However, when I reached the Summary screen, it show Backup mode: file level backup. (is it possible to attach my screen capture here?)

2. I have reconfigured my 2TB selected folders backup to 192.168.100.245, but the log shows that the total backup size is 6.46TB and it's still trying to backup to 192.168.100.241 which was my original configuration. No wonder it fail everyday.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

1. I tried to make it run as volume level backup and I did saw it indicating volume level backup under the Files screen. However, when I reached the Summary screen, it show Backup mode: file level backup. (is it possible to attach my screen capture here?)
Just share the link to the screenshot.
I have reconfigured my 2TB selected folders backup to 192.168.100.245, but the log shows that the total backup size is 6.46TB and it's still trying to backup to 192.168.100.241 which was my original configuration. No wonder it fail everyday.
So, you mean the destination path was not updated in the job settings?
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Here's the screen capture that show volume level backup:
Image

At the end, it show file level backup instead:
Image

The new saved destination path to: 192.168.100.245\backup
Image

The actual path still return to previous destination: 192.168.100.241\veeam:
Image
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

Backup is performed on a volume level, so it looks like the UI glitch, shared with QA guys - they will investigate. Thanks.

Regarding the IP address, that’s an interesting issue. To clarify, you just changed the IP or path to the backup location was also modified?
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

I think forcing a volume level backup thru file level backup won't work in my environment because of this: https://www.veeam.com/kb2073

I'm changing both the IP and the path. Support said I might need to reset my database to overcome this.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

Hi,

This KB is a bit outdated (refers to 1.1.2 version) - we implemented several fixes around the exclude logic. Can you clarify was the backup issue with the destination resolved by resetting the database?
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Veeam support said this: "Shadow Copy Optimization Writer timeout is set by Microsoft, and it's value cannot be changed. That warning message appears because large amount of data is excluded and cannot be processed by Shadow Copy Optimization Writer for it's timeout."

So the technique to perform a volume level backup for file level backup is out. :(

Why is it so difficult for Veeam to make a simple successful backup?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

Check your forum's inbox and let me know if the suggested solution works. Thank you.
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Still not working. The error now is: Timed out to wait for traffic control event. Agent failed to process method.

I also notice the total backup size shown in Veeam is wrong: 7.81TB. My selected folders are less than 2TB.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

So, you've added the timeout value like suggested and that did not help, right?
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Yes, I added the timeout value and that didn't help. But at least I don't see timeout error now.

Veeam support suggest that I either use volume-level backup of that disk without any folders exclusions, or try some high-performance storage device and network connection (like 10 gbps).

The question is, why is it that windows server backup works with file level backup and can even able to complete the backup job over 1gbps network or even the slow WAN connection without the need for any high performance network Nor storage?

Shouldn't Veeam be more superior? Why is it keep failing over such a simple backup job while Windows Server Backup works everytime? I don't understand.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

Thanks for the heads up. I’ll discuss your issue with our team meanwhile the suggested approach sounds good.

P.S. What is the size of the excluded files?
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

The excluded files size is about 1.2 TB.

The volume capacity is 9TB, with 3 TB used. I need to backup just 3 shared main folders of about 1.8TB and not everything.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

Thanks. I guess that’s too much for VSS exclusion engine, thus file level backup approach is preferable. We will discuss you case with QA team.
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

btw, is there a way to control how many full backup copies to keep? My storage was full as VEP has made a second full backup instead of incremental backup.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VEP keep failing + enhancement suggestions

Post by Dima P. »

Hi bhwong,

Generally, VEB should have only one full backup for backup chain due the nature of forever forward incremental backup approach. At rare circumstances, it can start a new backup chain, thus new full backup is created. Check the exact reason with support team as log investigation is required.

Additionally, double check that your restore points are operational via File Level Recovery.
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Is there any update to fix the high frequently of backup failure yet?

btw, is it possible to manually trigger it to start a new backup chain so that I can remove all the existing incremental backup files from the old backup chain?
bhwong
Enthusiast
Posts: 99
Liked: 3 times
Joined: May 24, 2012 9:57 am
Full Name: Boon Hong Wong
Contact:

Re: VEP keep failing + enhancement suggestions

Post by bhwong »

Support claim that the new 2.0 will solve my problem and unfortunately, the backup continue to fail everytime even after upgraded to 2.0.

On the contrary, Windows Backup has rarely fail backing up the same set of folders. Thus, the reason given that this is too much for VSS exclusion engine to handle, seem to suggest that Veeam has implemented a flawed design?

btw, it will be great if we can merge incremental backups into the full backup instead of creating a new full backup over the WAN.
Post Reply

Who is online

Users browsing this forum: No registered users and 33 guests