-
- Service Provider
- Posts: 217
- Liked: 28 times
- Joined: Jan 24, 2012 7:56 am
- Full Name: Massimiliano Rizzi
- Contact:
Weird error messages during daily forward incremental backup
Hello experts,
all of a sudden we have been getting the following error messages on a small production environment where backups are being targeted to a Buffalo TeraStation III Rackmount NAS device equipped with 4 x 1 TB SATA hard drives on a RAID 5 3+1 configuration:
==================================================
- "The process cannot access the file because it is being used by another process. Error code: 32 Failed to open file [forward incremental backup file name] in readonly mode"
- "Error: Client error: The specified network name is no longer available. Failed to read data from the file..."
- "Error: Client error: The specified network name is no longer available. Error code: 64 Failed to flush file buffers. File: [forward incremental backup file name]."
==================================================
The Veeam backup server is a virtual machine configured as follows:
==================================================
- Microsoft Windows Server 2008 R2 Standard Edition SP1 running Veeam Backup & Replication 7.0 R2 update Patch #4
- 4x Intel Xeon E5-2620 v2 vCPUs configured as 1 socket and 4 cores
- 4 GB RAM
- Hot-add is the transport mode that is used
- the Veeam primary backup repository is a CIFS share on the Buffalo TeraStation III Rackmount NAS device
==================================================
The weird thing is that we’ve been getting the above errors starting 2 weeks ago after adding a new Windows file server to the list of VMs being backed up. What’s even weirder is that we’re getting the above errors only during the first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completes successfully.
Before adding the new Windows file server on the 4th of July
==================================================
- the list of VMs being backed up comprised 3 VMs (approximately 120 GB full backup file size)
- we were using reversed incremental backup
- no one backup has been missed starting November 2013
- on the 2nd of July we abandoned the existing backup chain and started a new one using forward incremental backup
- no one backup has been missed from the 2nd of July
==================================================
After adding the new Windows file server on the 4th of July
==================================================
- the list of VMs being backed up now comprises 4 VMs (approximately 700 GB full backup file size)
- we are using forward incremental backup with active full every Saturday
- on the 4th of July we added the new Windows file server to the list of VMs being backed up and. As a result, the new file server has been backed up the first time. The job completed successfully during the first scheduled run
- on the 5th of July (Saturday) the weekly active full completed successfully during the first scheduled run
- from the 6th of July (Sunday) until the 11th of July (Friday) we got the above error during each first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completed successfully.
- on the 12th of July (Saturday) the weekly active full completed successfully during the first scheduled run
- from the 13th of July (Sunday) until today we got the above error during each first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completed successfully.
==================================================
Any help would be greatly appreciated.
Thanks and Regards,
Massimiliano
all of a sudden we have been getting the following error messages on a small production environment where backups are being targeted to a Buffalo TeraStation III Rackmount NAS device equipped with 4 x 1 TB SATA hard drives on a RAID 5 3+1 configuration:
==================================================
- "The process cannot access the file because it is being used by another process. Error code: 32 Failed to open file [forward incremental backup file name] in readonly mode"
- "Error: Client error: The specified network name is no longer available. Failed to read data from the file..."
- "Error: Client error: The specified network name is no longer available. Error code: 64 Failed to flush file buffers. File: [forward incremental backup file name]."
==================================================
The Veeam backup server is a virtual machine configured as follows:
==================================================
- Microsoft Windows Server 2008 R2 Standard Edition SP1 running Veeam Backup & Replication 7.0 R2 update Patch #4
- 4x Intel Xeon E5-2620 v2 vCPUs configured as 1 socket and 4 cores
- 4 GB RAM
- Hot-add is the transport mode that is used
- the Veeam primary backup repository is a CIFS share on the Buffalo TeraStation III Rackmount NAS device
==================================================
The weird thing is that we’ve been getting the above errors starting 2 weeks ago after adding a new Windows file server to the list of VMs being backed up. What’s even weirder is that we’re getting the above errors only during the first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completes successfully.
Before adding the new Windows file server on the 4th of July
==================================================
- the list of VMs being backed up comprised 3 VMs (approximately 120 GB full backup file size)
- we were using reversed incremental backup
- no one backup has been missed starting November 2013
- on the 2nd of July we abandoned the existing backup chain and started a new one using forward incremental backup
- no one backup has been missed from the 2nd of July
==================================================
After adding the new Windows file server on the 4th of July
==================================================
- the list of VMs being backed up now comprises 4 VMs (approximately 700 GB full backup file size)
- we are using forward incremental backup with active full every Saturday
- on the 4th of July we added the new Windows file server to the list of VMs being backed up and. As a result, the new file server has been backed up the first time. The job completed successfully during the first scheduled run
- on the 5th of July (Saturday) the weekly active full completed successfully during the first scheduled run
- from the 6th of July (Sunday) until the 11th of July (Friday) we got the above error during each first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completed successfully.
- on the 12th of July (Saturday) the weekly active full completed successfully during the first scheduled run
- from the 13th of July (Sunday) until today we got the above error during each first scheduled run of a daily forward incremental backup, while in one of the subsequent automatic retries every single job completed successfully.
==================================================
Any help would be greatly appreciated.
Thanks and Regards,
Massimiliano
-
- Chief Product Officer
- Posts: 31809
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Weird error messages during daily forward incremental ba
Hi, Massimiliano. Please include the support case ID for this issue, as requested when you click New Topic. Otherwise, this topic will be removed by moderators eventually. Thank you!
-
- Service Provider
- Posts: 217
- Liked: 28 times
- Joined: Jan 24, 2012 7:56 am
- Full Name: Massimiliano Rizzi
- Contact:
Re: Weird error messages during daily forward incremental ba
Hi Gostev,
thank you for your prompt reply.
I will file a support case and report back its ID in a few minutes.
Thanks !
Massimiliano
Edit // Case # 00603666
thank you for your prompt reply.
I will file a support case and report back its ID in a few minutes.
Thanks !
Massimiliano
Edit // Case # 00603666
-
- Chief Product Officer
- Posts: 31809
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Weird error messages during daily forward incremental ba
Thank you. I will guess here that the issue here is caused by your backup storage misbehaving. Generally speaking, low end NAS devices are the top source of support cases involving backup file corruption or access issue. But let's see what our support says after close review of debug logs. I assume you have already tried rebooting your NAS. Thanks!
-
- Service Provider
- Posts: 217
- Liked: 28 times
- Joined: Jan 24, 2012 7:56 am
- Full Name: Massimiliano Rizzi
- Contact:
Re: Weird error messages during daily forward incremental ba
Hi Gostev,
I apologize for the late reply, I have been very busy lately and, as a result, I have been unable to follow up on this issue immediately.
As part of running through the checklist of KB1735 as requested by the Support Engineer and for performance reasons, I am about to reconfigure the backup job to use a different primary storage that will be responsible for receiving backups from the production VMs, while the Buffalo SMB repository will receive backup copies from the primary backup storage using the BackupCopy feature.
I am thinking of presenting to the Veeam B&R VM a LUN (using a RDM in virtual compatibility mode) large enough to accommodate a backup chain with the number of restore points we want to retain.
Does it make sense to you ?
Thanks again !
I apologize for the late reply, I have been very busy lately and, as a result, I have been unable to follow up on this issue immediately.
I fully agree with you. That was my first thought too.Gostev wrote:I will guess here that the issue here is caused by your backup storage misbehaving. Generally speaking, low end NAS devices are the top source of support cases involving backup file corruption or access issue.
Like you said, the Support Engineer pointed out that the issue may be caused by the Buffalo storage device backup misbehaving especially when used as a SMB repository. The Support Engineer also pointed out that it is not uncommon to have to set an ingestion rate on the repository configuration when using Buffalo storage devices as a SMB repository.Gostev wrote:But let's see what our support says after close review of debug logs.
I confirm I have already tried rebooting the NAS. I also have updated it to the latest product firmware, I have initialized it to its factory default and disabled every useless feature.Gostev wrote:I assume you have already tried rebooting your NAS.
As part of running through the checklist of KB1735 as requested by the Support Engineer and for performance reasons, I am about to reconfigure the backup job to use a different primary storage that will be responsible for receiving backups from the production VMs, while the Buffalo SMB repository will receive backup copies from the primary backup storage using the BackupCopy feature.
I am thinking of presenting to the Veeam B&R VM a LUN (using a RDM in virtual compatibility mode) large enough to accommodate a backup chain with the number of restore points we want to retain.
Does it make sense to you ?
Thanks again !
-
- Chief Product Officer
- Posts: 31809
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Weird error messages during daily forward incremental ba
Sure, as long as this LUN is not located on the same storage that runs your virtual machines
And you don't really need vRDM in this case, you can go with pRDM as well.
In-guest iSCSI is also a common approach in case of iSCSI-based storage.
And you don't really need vRDM in this case, you can go with pRDM as well.
In-guest iSCSI is also a common approach in case of iSCSI-based storage.
-
- Service Provider
- Posts: 217
- Liked: 28 times
- Joined: Jan 24, 2012 7:56 am
- Full Name: Massimiliano Rizzi
- Contact:
Re: Weird error messages during daily forward incremental ba
Hi Gostev,
thank you for your reply.
Unfortunately this LUN is located on the same storage that runs the virtual machines, so I like to just consider it a “helper” primary storage responsible for receiving backups from the production VMs, while the Buffalo SMB repository will receive backup copies from this primary backup storage using the BackupCopy feature right after the the primary backup job completes successfully.
During first run the newly-created Backup Copy job targeted to the Buffalo SMB repository created the first restore point/full backup successfully. From this point on, we'll have to wait and see how it goes with the forever-incremental backup. It has to be said, all weekly active fulls completed successfully during the first scheduled attempt when we were using the Buffalo SMB repository as the primary storage responsible for receiving backups from the production VMs.
Is it correct to say that a BackupCopy job puts less stress on a backup repository rather than a regular backup job ?
Thank you again for your support.
Massimiliano
thank you for your reply.
Unfortunately this LUN is located on the same storage that runs the virtual machines, so I like to just consider it a “helper” primary storage responsible for receiving backups from the production VMs, while the Buffalo SMB repository will receive backup copies from this primary backup storage using the BackupCopy feature right after the the primary backup job completes successfully.
During first run the newly-created Backup Copy job targeted to the Buffalo SMB repository created the first restore point/full backup successfully. From this point on, we'll have to wait and see how it goes with the forever-incremental backup. It has to be said, all weekly active fulls completed successfully during the first scheduled attempt when we were using the Buffalo SMB repository as the primary storage responsible for receiving backups from the production VMs.
Is it correct to say that a BackupCopy job puts less stress on a backup repository rather than a regular backup job ?
Thank you again for your support.
Massimiliano
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Weird error messages during daily forward incremental ba
It's actually the opposite, due to its synthetic nature, backup copy job requires more I/O on the target storage than the forward incremental backup job.massimiliano.rizzi wrote:Is it correct to say that a BackupCopy job puts less stress on a backup repository rather than a regular backup job ?
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Weird error messages during daily forward incremental ba
As always, it depends
Backup Copy jobs uses 2 IO per saved block while running transform operations (the first write operations are 1 IO like forward incremental until it reaches the retention limit and starts transforming).
So, it' less stressful than reversed incremental (3 IO), but more than forward (1 IO).
Let's say it's a great compromise between IO and space savings (it uses only one full backup just like reversed).
Backup Copy jobs uses 2 IO per saved block while running transform operations (the first write operations are 1 IO like forward incremental until it reaches the retention limit and starts transforming).
So, it' less stressful than reversed incremental (3 IO), but more than forward (1 IO).
Let's say it's a great compromise between IO and space savings (it uses only one full backup just like reversed).
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Service Provider
- Posts: 217
- Liked: 28 times
- Joined: Jan 24, 2012 7:56 am
- Full Name: Massimiliano Rizzi
- Contact:
Re: Weird error messages during daily forward incremental ba
Hi guys,
thank you for your replies.
I'd like to take a few minutes to follow up on this issue after taking a deeper look into it with the Support Engineer.
As part of running through the checklist of KB1735 as requested by the Support Engineer, I have checked the system event log on the proxy server and searched for events with Source=MUP // Event ID: 139 and I confirm there have been events with Source=MUP // Event ID: 139 until a day before we configured a helper repository on local drive and the Backup Copy to the Buffalo storage. After targeting the backup job to a helper repository on a local drive/LUN and the Backup Copy to the Buffalo storage I have seen no errors.
The issue was definitely being caused by the Buffalo storage misbehaving when used as primary storage responsible for receiving backups from the production VMs.
So far so good, let’s see if things will go as well when the Backup Copy will reach the retention limit and will start transforming, thus putting more stress on the Buffalo storage.
Thanks everyone for the help.
Massimiliano
thank you for your replies.
I'd like to take a few minutes to follow up on this issue after taking a deeper look into it with the Support Engineer.
As part of running through the checklist of KB1735 as requested by the Support Engineer, I have checked the system event log on the proxy server and searched for events with Source=MUP // Event ID: 139 and I confirm there have been events with Source=MUP // Event ID: 139 until a day before we configured a helper repository on local drive and the Backup Copy to the Buffalo storage. After targeting the backup job to a helper repository on a local drive/LUN and the Backup Copy to the Buffalo storage I have seen no errors.
The issue was definitely being caused by the Buffalo storage misbehaving when used as primary storage responsible for receiving backups from the production VMs.
So far so good, let’s see if things will go as well when the Backup Copy will reach the retention limit and will start transforming, thus putting more stress on the Buffalo storage.
Thanks everyone for the help.
Massimiliano
-
- Product Manager
- Posts: 20410
- Liked: 2299 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Weird error messages during daily forward incremental ba
Thank you Massimiliano for taking your time and updating the topic with the found resolution; we do appreciate it!
Who is online
Users browsing this forum: No registered users and 58 guests