Backup copy jobs holding files open after update 2

[darryl]

We run backup copies to a HP StoreOnce, which then does HP native replication to another StoreOnce.

After applying update 2 to V8, it seems that these backup copy files are held open by Veeam even when the backup copy job is idle. I can see this in ProcessExplorer. I did not investigate this behaviour prior to update 2, since I never had a reason to...

What we're observing - the HP StoreOnces end up pausing the replication of the backup copy job files because they're still in use (by Veeam). This breaks replication between the StoreOnces.

We now have to disable the backup copy jobs in order to allow the StoreOnces to replicate, then re-enable after.

Any thoughts? Is this just happening to us? I did open a case yesterday (00908396) but am still waiting to hear back.

[Dima P.]

Hello darryl,
While we are awaiting for support’s investigation results can you confirm the same behavior with the newly created backup copy jobs?

[bgbGsy]

I have upgraded from version 7 to 8 patch 2. Since the upgrade my copy jobs are failing because of open files. Normally after the transform when the .vbk is renamed / deleted. I have logged a call and provided the engineer with process mon output. The backup jobs have been in place for well over a year without a glitch. In my case the only other process with is looking at the files is explorer.exe. I'm awaiting feedback from the engineer.

[foggy]

Brendan, starting v8 Update 2, target data mover agent caches metadata from backup files and keeps backup copy job chain opened for this purpose. This was done to improve performance of storages that are not capable of high random I/O. If you're satisfied with the backup copy performance, you can disable caching by setting the AgentReadOnlyCache registry value 0 (this, however, will affect all your jobs) or 2 (this will affect only those jobs that have inline deduplication turned off).

[bgbGsy]

Thanks for that Foggy. I have submitted a whole jobs worth of process monitor requests to support this morning (case: # 00908796) It confirmed that only veeamagent.exe and veeam services were 'touching' the backup copy job directory. I'm awaiting a response from the engineer. Moving forward, this change of behaviour will need to be addressed in some way? Is it that the file operations are effectively happening too fast In memory and that's why I'm seeing this error?

Do I apply the setting you gave on the remote repository sever or the veeam server? In my Case I have a physical veeam server with the backup jobs (no issues since upgrade) and a remote physical server repository (Both windows 2008 R2) link together via 2b LACP link. If is the remote repository server doing the copy jobs which is exhibiting the error.

Thanks in advance.

[foggy]

Do I apply the setting you gave on the remote repository sever or the veeam server?

It should go under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication on the Veeam B&R console server.

[darryl]

If you're satisfied with the backup copy performance, you can disable caching by setting the AgentReadOnlyCache registry value 0 (this, however, will affect all your jobs) or 2 (this will affect only those jobs that have inline deduplication turned off).

Thanks for the tip - I disabled the cache, and the files are closed after the backup copy job. The StoreOnce replication is then able to proceed, as it was prior to update 2.

The Veeam - HP white paper says to leave inline data deduplication enabled for HP StoreOnce. This means I can't target only my Backup Copy jobs for StoreOnce by setting AgentReadOnlyCache to 2. So I had to set it to 0.

However, this leaves me totally unable to take advantage of the new cache functionality on any job, so this isn't great either.

For the cache, are the files open for write? I don't think the StoreOnce would have an issue if they were just open for read. Should I consider disabling inline dedupe on the StoreOnce, against the advice of the whitepaper, so I can target these jobs with AgentReadOnlyCache = 2 ?

Is there some other way around getting the advantage of the cache, but not having HP StoreOnce replication break?

[foggy]

While you could disable inline deduplication, this actually would not take effect for backup copy jobs, since cache is always enabled for them, regardless of the deduplication setting (due to the backup copy job architecture specifics). So the only way to disable it, is setting the value to 0.

[mcwill]

# 00910961
After upgrading to Update 2 we are seeing our Backup Copy jobs report failure whenever they attempt to suffix (WMQY) the filename of a synthetic restore point...

Code: Select all

11/05/2015 11:00:31 :: Failed to generate points Error: boost::filesystem::rename: The process cannot access the file because it is being used by another process
Failed to rename file [\\192.168.2.242\public\Exchange Copy_1\Exchange Copy2015-05-04T040000.vbk].
Failed to process [renameFileWithRollback].

Could the problem mentioned above with caching also be causing the Backup Copy job to fail and report the errors we are seeing?

Thanks,
Iain

[foggy]

Logs review is required to tell the actual reason of the failure. Please continue working with your engineer on this.

[mcwill]

Will do, I have also applied the AgentReadOnlyCache = 0 entry above to see if that solves the problem. Although it may be a week before another synthetic restore point is attempted.

[andreash]

Same here:
"Failed to generate points Error: boost::filesystem::rename: ... <File is used by another process>" - The actual message is in german I'm afraid.

I have deleted the backup copy jobs from disk, preserving the archived fulls. Disabling the Cache will be my next option.

My biggest concern is that all these failed jobs are reported as successful in the email reports and the job history. This makes is difficult to spot the problem.

[foggy]

Andreas, please also contact our technical team to spot the problem.

[andreash]

#00916903 - The Job History correctly identifies the Jobs as failed, the email reports incorrectly report success.

[foggy]

# 00910961
After upgrading to Update 2 we are seeing our Backup Copy jobs report failure whenever they attempt to suffix (WMQY) the filename of a synthetic restore point...

This was confirmed to be an issue, GFS mechanism can fail if cache is enabled. Thanks for reporting this.

[andreash]

Is this issue being fixed in the latest Update 2a?

[Dima P.]

Hello Andreas,

The fix is not included in the Update 2a but will be provided as soon as possible. Thank you for your patience and sorry for any inconvenience caused.

[MiniWalks]

Any update to this?
We are noticing this behavior also, not great as a new customer trying to prove veeam over com vault when veeam doesn't work as advertised.
Hope to see it fixed soon

[foggy]

Matt, Andreas, if you're talking about GFS mechanism issues with the cache enabled, then those should be fixed after installing Update 2a. Please contact support if you experience different behavior.

[darryl]

Does update 2a/b address this issue? Only happens with the cache turned on. This is a big deal for us, as the cache appears to help quite a bit, but we can't leave it enabled.

We run backup copies to a HP StoreOnce, which then does HP native replication to another StoreOnce.

After applying update 2 to V8, it seems that these backup copy files are held open by Veeam even when the backup copy job is idle. I can see this in ProcessExplorer. I did not investigate this behaviour prior to update 2, since I never had a reason to...

What we're observing - the HP StoreOnces end up pausing the replication of the backup copy job files because they're still in use (by Veeam). This breaks replication between the StoreOnces.

[Gostev]

My understanding is that this is currently by design. It was explained to me that cache requires that cached files are locked to prevent them from being modified externally (otherwise, cache contents may go out of sync with the actual contents of the files). I have an open discussion on this with the devs, but it needed a bump

[darryl]

I'm going to re-test tonight with the cache re-enabled, but with the cache on, I think we save something like 5 minutes for each and every vmdk being backup copied to the StoreOnce.

But then when the backup job completes it's run and goes idle, the files stay open, and the StoreOnce can't replicate. I believe I even tried setting a blackout window on the backup copy job, and the files still remained open. Ideally it would be nice if when the job went idle, the files would close. Not sure how long it would take to re-seed the cache when the backup copy went active again (?).

[Gostev]

I agree there's a room for improvement. I understand that devs were also limited with the existing caching functionality our data movers already had (which is what allowed us to implement caching so fast, as a part of an update). Generally, developing new functionality within the data mover requires a major release vehicle, as we are essentially getting into the holy grail of the product with a pavement braking hammer.

[darryl]

Thanks for the info.

My testing results - this is for a backup copy job with 40 VMs and around 400 GB changed data per day.

Without cache - 18.5 hours to write to the StoreOnce.

With cache - 3.8 hours to write to the StoreOnce. (Files locked after, StoreOnce native replication fails). The average VMDK with little changed data takes just seconds to process, vs the 6+ minutes with cache disabled.

Both of these are without a merge.

Are there any suggestions for a workaround that would allow me to keep the cache enabled? Disable the backup copy job (script?) each day for a period of time when it's idle? I would really like to take advantage of the cache.

[Gostev]

Wait a second. Even with cache enabled, backup files should still be released after all VMs have been processed for the specific copy interval (which is when you get job's email report). If your backup files remain locked permanently, this is definitely not normal and unexpected, so should be investigated more closely.

[Gostev]

On a separate note, I found that we lock backup files for Shared Read only, which really should not prevent any software from reading them for replication purposes. Unless StoreOnce purposely avoids touching files locked for read (which sounds more like a bug to me, because it does not make any sense).

[darryl]

Wait a second. Even with cache enabled, backup files should still be released after all VMs have been processed for the specific copy interval (which is when you get job's email report). If your backup files remain locked permanently, this is definitely not normal and unexpected, so should be investigated more closely.

You're right - it happens that the job with the bulk of our VMs does not send the Email until the interval closes. The backup job which the backup copy job contains has several templates in it. These don't seem to get processed. The Email is not sent until the backup copy job is basically starting again.

That being said, we have had the issue with other jobs, that for whatever reason don't process all of the VMs that given day. Like there was no new restore point to copy, so the job stays running and the files stay open.

I did end up scripting something to disable the backup copy jobs after they've gone idle, and to start them up at their normal "start" time. This seems to have temporarily resolved the issue for us.

[darryl]

On a separate note, I found that we lock backup files for Shared Read only, which really should not prevent any software from reading them for replication purposes. Unless StoreOnce purposely avoids touching files locked for read (which sounds more like a bug to me, because it does not make any sense).

I had been wondering if they were locked for write... The StoreOnce definitely complains about the file being accessed by a higher priority process. Not sure why it would care about a read lock.

[albertwt]

We run backup copies to a HP StoreOnce, which then does HP native replication to another StoreOnce.

After applying update 2 to V8, it seems that these backup copy files are held open by Veeam even when the backup copy job is idle. I can see this in ProcessExplorer. I did not investigate this behaviour prior to update 2, since I never had a reason to...

What we're observing - the HP StoreOnces end up pausing the replication of the backup copy job files because they're still in use (by Veeam). This breaks replication between the StoreOnces.

We now have to disable the backup copy jobs in order to allow the StoreOnces to replicate, then re-enable after.

Any thoughts? Is this just happening to us? I did open a case yesterday (00908396) but am still waiting to hear back.

Darryl,

Yes I concur, I've got the same problem too after upgrading into Veeam Backup & Replication 8.0.0.2021 or patch 2, somehow the Backup Copy job locks the file in the destination repository which is NTFS LUN presented by HP EVA 6400 Storage Array.

I don't use Storage Array based replication technology but I was using Symantec Backup Exec 15 to write the .VBK files to tape drive. This process has been working flawlessly since VBR 6.5 up until I upgrade it to Patch 2, problem started to happening.

Case number: 00911739, has been closed since the Escalated technician could not resolve or give me the official explanation that this is the new behaviour of Veeam Backup & Replication 8.0.0.2021. Since then my manager has also gave up in writing the backup to tape due to this file lockup issue.