Failed to create backup file because it is already present on the file system

[SignalSeeker]

Hello all, I do not have a Veeam case #. I have a NFR on my home test system.

Out of the blue the backups are failing. No updates have been done that I know of to both Windows and Veeam. Here is what I get, not sure what information will be needed.

Processing SERVER1 Error: Failed to create backup file because it is already present on the file system. Agent link: E:\Backup\BACKUPJOB\BACKUPJOBD2022-06-02T173255_06FE.vib
Processing SERVER2 Error: Failed to create backup file because it is already present on the file system. Agent link: E:\Backup\BACKUPJOB\BACKUPJOBD2022-06-02T173255_06FE.vib
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to create VM (ID: 37bf6a28-de65-4ba4-a3fa-d17fc5ffb83e) recovery checkpoint. Job failed ('Checkpoint operation for 'SERVER3' failed. (Virtual machine ID 37BF6A28-DE65-4BA4-A3FA-D17FC5FFB83E) Checkpoint operation for 'SERVER3' was cancelled. (Virtual machine ID 37BF6A28-DE65-4BA4-A3FA-D17FC5FFB83E) 'SERVER3' could not initiate a checkpoint operation: %%2147754994 (0x800423F2). (Virtual machine ID
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 6/2/2022 5:37:42 PM
Unable to allocate processing resources. Error: Failed to create production checkpoint.

I tested Checkpoints on all 3 servers and it works fine. The host is Server 2016 with Hyper-V installed. The Backup server is a VM, also 2016, using a file share to a USB drive on the host for backup files. The other 2 servers are 2012R2. Veeam is version 11.0.1.1261 P20211211, the newer update is not applied as I didn't appear it would help with this issue.

Thank you for looking!

[Mildur]

Hi

You can also open a support case with a NFR License. Please do so, or this topic could be deleted by a moderator.

The Backup server is a VM, also 2016, using a file share to a USB drive on the host for backup files.

Why not making the Win2016 HyperV a backup repository server and using the usb disk as a direct attached backup repository (rotated drive enabled)? Using a share is already bad for stability and performance. Leaving out the share step makes at least the backup repo more stable.

Veeam is version 11.0.1.1261 P20211211, the newer update is not applied as I didn't appear it would help with this issue.

The newest update from march should be applied, because older versions had a serious security issue. And not all bugfixes are listed in the release notes. So give it a try


Thanks Fabian

[SignalSeeker]

Thank you for the reply Fabian

"You can also open a support case with a NFR License. Please do so, or this topic could be deleted by a moderator."
I will get on that this evening

"Why not making the Win2016 HyperV a backup repository server and using the usb disk as a direct attached backup repository (rotated drive enabled)? Using a share is already bad for stability and performance. Leaving out the share step makes at least the backup repo more stable."
I have it this way so it is more portable, it is running on decent, but older hardware. If it dies, I can grab the USB drive and rebuilt on other computers I have after installing Hyper-V. It is my home servers being backed up, doesn't really need to be fast.

"The newest update from march should be applied, because older versions had a serious security issue. And not all bugfixes are listed in the release notes. So give it a try "
I have it downloaded already, I can do that this evening also.

[Mildur]

Thanks.
Let me know if the update helps with the issue, if not, please go forward with the support case.
If checkpoints are failing, then something must had to be wrong with the HyperV server at the time of the backup. Veeam only gives the command to HyperV todo the checkpoint.

Thanks.
Fabian

[SignalSeeker]

Well I installed the update, now on 11.0.1.1261 P20220302 and have the same issue.

FYI, the Checkpoints are working fine, I tested them before starting this post.

So Windows is all up to date and now Veeam B&R is up to date.

I guess I need to open a support ticket.

I will post the ticket number later today.

Thanks!
Justin

[SignalSeeker]

Ticket #05469115

[Mildur]

Hi Signal

Thanks for the case number.
Can you check the vss writer state? Just open a CMD and use:

Code: Select all

vssadmin list writers

[SignalSeeker]

Here yo go Mildur, thanks!

Code: Select all

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {a1cb98de-0143-4ef5-93b2-88775c80a513}
   State: [1] Stable
   Last error: No error

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {99b3c273-5165-4159-a023-75cf27620076}
   State: [5] Waiting for completion
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {c63e7006-2d73-4b31-83ba-d7854f05788d}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {3f44506e-25fb-48e7-95f1-ec1da1db85ca}
   State: [5] Waiting for completion
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {37fa9cac-99f8-4863-9804-876e457ed038}
   State: [5] Waiting for completion
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {7775829d-3779-466e-a8a4-2b28a79d7c47}
   State: [5] Waiting for completion
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {f59141b1-38ab-4863-b17c-b1086cb20406}
   State: [1] Stable
   Last error: No error

[SignalSeeker]

Any guess how long it takes for a response after creating a ticket? Granted it has only been 3 business days.

[Mildur]

Unfortunately, Free and NFR user are on Best Effort.
If the load on support is to high, chances are that the case will be closed automatically after one week.

https://www.veeam.com/support-policy.html

I assume you have rebooted the server already? If you have done the vssadmin list writers command when no backup was running, then the writers shouldn‘t be in the state Waiting for completion.
I found some other solutions in the forum. Some had to disable veeams Application aware processing and backup has worked. Can you try that? Just as a test.

Thanks
Fabian

[SignalSeeker]

Thank you Fabian, I will try getting the VSS Writers cleared up. Yes, the server has been rebooted more than once.
I will report back as soon as I can.

Thanks!

[SignalSeeker]

I cleaned the VSS Writers, rebooted the Backup server.

Same errors.

Any other ideas?

[SignalSeeker]

You were right Fabian, they closed it without contacting me. Just got an email saying they closed it.

[Mildur]

Unfortunately that can happen with our free products. You can try to open another case.
Have you already tried a job run with disabled application aware processing?
If nothing helps, consider using the Veeam agent as a workaround until you have found a solution to this issue.

[alwitco]

You're not the only one.

Also, I DO NOT have the free version. Our version is licensed and has been for over 10 years.

Today, out of the blue all my backup jobs VMware and Agent backups fail with the same error: Failed to create backup file because it is already present on the file system......
Kind of odd that this just crept up on at least 2 users out of nowhere within a few days.

I created a case this morning but no response yet.

I can't even create a new job without getting an error.
The first run of any new job just hangs up before it even gives the waiting for resources message.
I have to reboot the server to cancel it as even cancel immediately doesn't work.
Then, if I re-run or re-try I get the error about the backup file is already present.

Now that I can't even get a new job to work, this is VERY critical as my production VM's are left without any new backups.

Knowing now that at least 2 users are getting this error, now indicates this is NOT an isolated case.

[alwitco]

Unfortunately that can happen with our free products. You can try to open another case.
Have you already tried a job run with disabled application aware processing?
If nothing helps, consider using the Veeam agent as a workaround until you have found a solution to this issue.

See my post, We just stared getting the same error today on EVERY job. Can not backup nothing !!
We have a fully licensed version and have not had any major issues like this in 8-10 years.

[Mildur]

Hi @alwitco

If you don't get an answer from support within SLA times, please try first to escalate the case.
This way, our support management knows about it and will see, that you get in contact with the engineer.

For the issue itself, I'd like to check if we have similar cases.
Please provide me with your case number.


Thanks
Fabian

[Mildur]

@alwitco
@SignalSeeker

One Question, is there by any chance Crowdstrike installed?
If yes, can you check for any alerts or blocks (it may be informational) in the logs?
And please check your exclusion list: https://www.veeam.com/kb1999

[alwitco]

I'm please to announce that as of 20 minutes ago, the issue has been resolved.

Something I should have expected as a possible cause right off the bat, but never put 2 and 2 together.

At the suggestion of a Veeam tech looking into my case, he asked if we had Malwarebytes or any other AV installed on the backup server.

We had Malwarebytes and Trend-Micro Endpoint protection.

I disabled both, and bingo life is good (or a little better) !!

I never suspected Malwarebytes or Trend-Micro as they were working in harmony for several years on the backup server, and I had all the possible exclusions set.

Now, it makes sense because everything stopped working around midnight, probably when an update to one of the two was sent out.

I can now sleep well again, as last night I had nightmares of coming into the office at 5:00 am and seeing our VMware servers all up in flames, with no recent backups.

[jjpeterspa]

What was the issue with MWB? Should i completely remove it?
Would like to know if there is a security issue or other.

Thanks,
JimP

[Mildur]

@alwitco
I‘m glad that it works now. We suspect, that multiple AV applications have updated their signature and now are blocking our Backup files/processes.

@jjpeterspa
Please check your AV logs if any files are blocked.
And check if you have excluded all veeam processes and veeam files as recommended in our guide.
https://www.veeam.com/kb1999

[david.domask]

To elaborate a bit more on the interaction between AV and backup applications (not just Veeam ), Read on. This applies a lot to firewalls also. Please excuse the wall of text, but it's worth the read to understand why random updates to security software cause this.

It's not so much that the definitions specifically target Veeam, it's more that some benign action Veeam does and has been doing for years suddenly triggers a heuristics rule or a block of data happens to hit a false positive.

Keep in mind that AV often works on heuristics and signature based matching, and this can often lead to false positives. If you were ever into some naughty activity with torrents and keygens and such, you know what I'm talking about

Now, it's very difficult to talk deeply about these subjects because AV is a black box by design; if they're too open on how they flag items, malware authors will just circumvent it, so a lot of this has to be hush hush. But, from observation, we can observe a few things and by logic we can deduce some others:

Signature Matching: The premise is simple -- a database of all naughty files/blocks would be insurmountably big and the processing power to check everything would slow your machines to a crawl. So instead, security software keeps a hash database and does fast-hashing of the data it's checking to see if there's a match (parsing strings of the same predictable length is infinitely faster and can be optimized for, and if there's a match, you then just fetch the necessary user info from elsewhere).

This has a few side effects though: With hashing, you either have strong uniqueness but computationally expensive (read: slow), or weak uniqueness but computationally cheap (fast). The latter is usually chosen so that security software doesn't interfere, but therein introduces a problem: if your uniqueness is weak, you have an increasingly non-zero chance of hash collisions, wherein two unique inputs produce the same output. As backup data is essentially random with billions (if not more) of unique data blocks being produced, you roll the dice billions+ times and eventually you hit a false positive match.

Heuristics: The idea here is much the same, but instead you apply levels of "risk" to processes when looking at it from a high-level. The more risk assigned, the more likely the AV is eager to step in and check it further or just block it outright. Both operations can interfere with normal operations of innocent software, and the worst part is that it's not super feasible to know what specifically triggers the security software. As we can see in the topic, even something benign like making a file gets flagged, but who knows why this is considered "risky." Even worse, sometimes it seems whitelisted directories/processes still get flagged in the spirit of security. A few AV vendors post a general overview of their heuristics scheme if you search "how antivirus heuristics work", so feel free to read on and see the actions they might do if they feel something is "suspicious." I personally feel that such methods are too often specious, but with security, it's a reasonable standpoint to err on the side of security than allow a malicious actor to act.

This isn't to say that it's always security software's fault, but it is why a lot of times it is a usual suspect and the guidance on what specifically to check is not always clear. Security Vendors have a vested interest in keeping their methodology secret (understandably), so almost always it has to be a test of trial and error, just to see what does and does not get triggered. Furthermore, it can be wildly inconsistent between environments.

For example, a case I had to step in on, Windows Defender was clearly blocking the installation of Enterprise Manager. We have tens of thousands of such installations without incident every year, many of them running Windows Defender. The client in the case rightfully wanted to know how to avoid having to disable more advanced Defender features, but we really had no idea how to even discuss such a thing. The client absolutely did not want to raise a case with Microsoft because they believed it would go nowhere, which I completely get, but we're stuck at an impasse then; we have no idea what Defender is doing, why it triggered in _this specific environment and not others_, and really have no better option.

So I know it's a ton of text, but this is generally how it works and why it causes issues. For some of the security vendors that have been very communicative and helpful, they openly admitted they had an overzealous update to a heuristic and a later definition update toned it down a bit (or they already had tricks to do a real whitelist). Others have been completely silent stating it worked "by design." I don't fault the latter, but it ends in a frustrating situation for the users and the application vendor.

[jjpeterspa]

@alwitco
I‘m glad that it works now. We suspect, that multiple AV applications have updated their signature and now are blocking our Backup files/processes.

@jjpeterspa
Please check your AV logs if any files are blocked.
And check if you have excluded all veeam processes and veeam files as recommended in our guide.
https://www.veeam.com/kb1999

Not seeing anything blocked, though last i ran it manually with MWB disabled. Today its enabled and will see.