-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Feb 24, 2014 6:33 pm
- Contact:
Failed to create backup file because it is already present on the file system
Hello all, I do not have a Veeam case #. I have a NFR on my home test system.
Out of the blue the backups are failing. No updates have been done that I know of to both Windows and Veeam. Here is what I get, not sure what information will be needed.
Processing SERVER1 Error: Failed to create backup file because it is already present on the file system. Agent link: E:\Backup\BACKUPJOB\BACKUPJOBD2022-06-02T173255_06FE.vib
Processing SERVER2 Error: Failed to create backup file because it is already present on the file system. Agent link: E:\Backup\BACKUPJOB\BACKUPJOBD2022-06-02T173255_06FE.vib
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to create VM (ID: 37bf6a28-de65-4ba4-a3fa-d17fc5ffb83e) recovery checkpoint. Job failed ('Checkpoint operation for 'SERVER3' failed. (Virtual machine ID 37BF6A28-DE65-4BA4-A3FA-D17FC5FFB83E) Checkpoint operation for 'SERVER3' was cancelled. (Virtual machine ID 37BF6A28-DE65-4BA4-A3FA-D17FC5FFB83E) 'SERVER3' could not initiate a checkpoint operation: %%2147754994 (0x800423F2). (Virtual machine ID
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 6/2/2022 5:37:42 PM
Unable to allocate processing resources. Error: Failed to create production checkpoint.
I tested Checkpoints on all 3 servers and it works fine. The host is Server 2016 with Hyper-V installed. The Backup server is a VM, also 2016, using a file share to a USB drive on the host for backup files. The other 2 servers are 2012R2. Veeam is version 11.0.1.1261 P20211211, the newer update is not applied as I didn't appear it would help with this issue.
Thank you for looking!
Out of the blue the backups are failing. No updates have been done that I know of to both Windows and Veeam. Here is what I get, not sure what information will be needed.
Processing SERVER1 Error: Failed to create backup file because it is already present on the file system. Agent link: E:\Backup\BACKUPJOB\BACKUPJOBD2022-06-02T173255_06FE.vib
Processing SERVER2 Error: Failed to create backup file because it is already present on the file system. Agent link: E:\Backup\BACKUPJOB\BACKUPJOBD2022-06-02T173255_06FE.vib
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to create VM (ID: 37bf6a28-de65-4ba4-a3fa-d17fc5ffb83e) recovery checkpoint. Job failed ('Checkpoint operation for 'SERVER3' failed. (Virtual machine ID 37BF6A28-DE65-4BA4-A3FA-D17FC5FFB83E) Checkpoint operation for 'SERVER3' was cancelled. (Virtual machine ID 37BF6A28-DE65-4BA4-A3FA-D17FC5FFB83E) 'SERVER3' could not initiate a checkpoint operation: %%2147754994 (0x800423F2). (Virtual machine ID
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 6/2/2022 5:37:42 PM
Unable to allocate processing resources. Error: Failed to create production checkpoint.
I tested Checkpoints on all 3 servers and it works fine. The host is Server 2016 with Hyper-V installed. The Backup server is a VM, also 2016, using a file share to a USB drive on the host for backup files. The other 2 servers are 2012R2. Veeam is version 11.0.1.1261 P20211211, the newer update is not applied as I didn't appear it would help with this issue.
Thank you for looking!
-
- Product Manager
- Posts: 9848
- Liked: 2608 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed to create backup file because it is already present on the file system
Hi
You can also open a support case with a NFR License. Please do so, or this topic could be deleted by a moderator.
Thanks Fabian
You can also open a support case with a NFR License. Please do so, or this topic could be deleted by a moderator.
Why not making the Win2016 HyperV a backup repository server and using the usb disk as a direct attached backup repository (rotated drive enabled)? Using a share is already bad for stability and performance. Leaving out the share step makes at least the backup repo more stable.The Backup server is a VM, also 2016, using a file share to a USB drive on the host for backup files.
The newest update from march should be applied, because older versions had a serious security issue. And not all bugfixes are listed in the release notes. So give it a tryVeeam is version 11.0.1.1261 P20211211, the newer update is not applied as I didn't appear it would help with this issue.
Thanks Fabian
Product Management Analyst @ Veeam Software
-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Feb 24, 2014 6:33 pm
- Contact:
Re: Failed to create backup file because it is already present on the file system
Thank you for the reply Fabian
"You can also open a support case with a NFR License. Please do so, or this topic could be deleted by a moderator."
I will get on that this evening
"Why not making the Win2016 HyperV a backup repository server and using the usb disk as a direct attached backup repository (rotated drive enabled)? Using a share is already bad for stability and performance. Leaving out the share step makes at least the backup repo more stable."
I have it this way so it is more portable, it is running on decent, but older hardware. If it dies, I can grab the USB drive and rebuilt on other computers I have after installing Hyper-V. It is my home servers being backed up, doesn't really need to be fast.
"The newest update from march should be applied, because older versions had a serious security issue. And not all bugfixes are listed in the release notes. So give it a try "
I have it downloaded already, I can do that this evening also.
"You can also open a support case with a NFR License. Please do so, or this topic could be deleted by a moderator."
I will get on that this evening
"Why not making the Win2016 HyperV a backup repository server and using the usb disk as a direct attached backup repository (rotated drive enabled)? Using a share is already bad for stability and performance. Leaving out the share step makes at least the backup repo more stable."
I have it this way so it is more portable, it is running on decent, but older hardware. If it dies, I can grab the USB drive and rebuilt on other computers I have after installing Hyper-V. It is my home servers being backed up, doesn't really need to be fast.
"The newest update from march should be applied, because older versions had a serious security issue. And not all bugfixes are listed in the release notes. So give it a try "
I have it downloaded already, I can do that this evening also.
-
- Product Manager
- Posts: 9848
- Liked: 2608 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed to create backup file because it is already present on the file system
Thanks.
Let me know if the update helps with the issue, if not, please go forward with the support case.
If checkpoints are failing, then something must had to be wrong with the HyperV server at the time of the backup. Veeam only gives the command to HyperV todo the checkpoint.
Thanks.
Fabian
Let me know if the update helps with the issue, if not, please go forward with the support case.
If checkpoints are failing, then something must had to be wrong with the HyperV server at the time of the backup. Veeam only gives the command to HyperV todo the checkpoint.
Thanks.
Fabian
Product Management Analyst @ Veeam Software
-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Feb 24, 2014 6:33 pm
- Contact:
Re: Failed to create backup file because it is already present on the file system
Well I installed the update, now on 11.0.1.1261 P20220302 and have the same issue.
FYI, the Checkpoints are working fine, I tested them before starting this post.
So Windows is all up to date and now Veeam B&R is up to date.
I guess I need to open a support ticket.
I will post the ticket number later today.
Thanks!
Justin
FYI, the Checkpoints are working fine, I tested them before starting this post.
So Windows is all up to date and now Veeam B&R is up to date.
I guess I need to open a support ticket.
I will post the ticket number later today.
Thanks!
Justin
-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Feb 24, 2014 6:33 pm
- Contact:
-
- Product Manager
- Posts: 9848
- Liked: 2608 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed to create backup file because it is already present on the file system
Hi Signal
Thanks for the case number.
Can you check the vss writer state? Just open a CMD and use:
Thanks for the case number.
Can you check the vss writer state? Just open a CMD and use:
Code: Select all
vssadmin list writers
Product Management Analyst @ Veeam Software
-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Feb 24, 2014 6:33 pm
- Contact:
Re: Failed to create backup file because it is already present on the file system
Here yo go Mildur, thanks!
Code: Select all
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.
Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error
Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error
Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error
Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {a1cb98de-0143-4ef5-93b2-88775c80a513}
State: [1] Stable
Last error: No error
Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {99b3c273-5165-4159-a023-75cf27620076}
State: [5] Waiting for completion
Last error: No error
Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {c63e7006-2d73-4b31-83ba-d7854f05788d}
State: [1] Stable
Last error: No error
Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {3f44506e-25fb-48e7-95f1-ec1da1db85ca}
State: [5] Waiting for completion
Last error: No error
Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {37fa9cac-99f8-4863-9804-876e457ed038}
State: [5] Waiting for completion
Last error: No error
Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {7775829d-3779-466e-a8a4-2b28a79d7c47}
State: [5] Waiting for completion
Last error: No error
Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {f59141b1-38ab-4863-b17c-b1086cb20406}
State: [1] Stable
Last error: No error
-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Feb 24, 2014 6:33 pm
- Contact:
Re: Failed to create backup file because it is already present on the file system
Any guess how long it takes for a response after creating a ticket? Granted it has only been 3 business days.
-
- Product Manager
- Posts: 9848
- Liked: 2608 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed to create backup file because it is already present on the file system
Unfortunately, Free and NFR user are on Best Effort.
If the load on support is to high, chances are that the case will be closed automatically after one week.
https://www.veeam.com/support-policy.html
I assume you have rebooted the server already? If you have done the vssadmin list writers command when no backup was running, then the writers shouldn‘t be in the state Waiting for completion.
I found some other solutions in the forum. Some had to disable veeams Application aware processing and backup has worked. Can you try that? Just as a test.
Thanks
Fabian
If the load on support is to high, chances are that the case will be closed automatically after one week.
https://www.veeam.com/support-policy.html
I assume you have rebooted the server already? If you have done the vssadmin list writers command when no backup was running, then the writers shouldn‘t be in the state Waiting for completion.
I found some other solutions in the forum. Some had to disable veeams Application aware processing and backup has worked. Can you try that? Just as a test.
Thanks
Fabian
Product Management Analyst @ Veeam Software
-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Feb 24, 2014 6:33 pm
- Contact:
Re: Failed to create backup file because it is already present on the file system
Thank you Fabian, I will try getting the VSS Writers cleared up. Yes, the server has been rebooted more than once.
I will report back as soon as I can.
Thanks!
I will report back as soon as I can.
Thanks!
-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Feb 24, 2014 6:33 pm
- Contact:
Re: Failed to create backup file because it is already present on the file system
I cleaned the VSS Writers, rebooted the Backup server.
Same errors.
Any other ideas?
Same errors.
Any other ideas?
-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Feb 24, 2014 6:33 pm
- Contact:
Re: Failed to create backup file because it is already present on the file system
You were right Fabian, they closed it without contacting me. Just got an email saying they closed it.
-
- Product Manager
- Posts: 9848
- Liked: 2608 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed to create backup file because it is already present on the file system
Unfortunately that can happen with our free products. You can try to open another case.
Have you already tried a job run with disabled application aware processing?
If nothing helps, consider using the Veeam agent as a workaround until you have found a solution to this issue.
Have you already tried a job run with disabled application aware processing?
If nothing helps, consider using the Veeam agent as a workaround until you have found a solution to this issue.
Product Management Analyst @ Veeam Software
-
- Novice
- Posts: 8
- Liked: never
- Joined: Aug 19, 2010 6:49 pm
- Full Name: John Gibel
- Contact:
Re: Failed to create backup file because it is already present on the file system
You're not the only one.
Also, I DO NOT have the free version. Our version is licensed and has been for over 10 years.
Today, out of the blue all my backup jobs VMware and Agent backups fail with the same error: Failed to create backup file because it is already present on the file system......
Kind of odd that this just crept up on at least 2 users out of nowhere within a few days.
I created a case this morning but no response yet.
I can't even create a new job without getting an error.
The first run of any new job just hangs up before it even gives the waiting for resources message.
I have to reboot the server to cancel it as even cancel immediately doesn't work.
Then, if I re-run or re-try I get the error about the backup file is already present.
Now that I can't even get a new job to work, this is VERY critical as my production VM's are left without any new backups.
Knowing now that at least 2 users are getting this error, now indicates this is NOT an isolated case.
Also, I DO NOT have the free version. Our version is licensed and has been for over 10 years.
Today, out of the blue all my backup jobs VMware and Agent backups fail with the same error: Failed to create backup file because it is already present on the file system......
Kind of odd that this just crept up on at least 2 users out of nowhere within a few days.
I created a case this morning but no response yet.
I can't even create a new job without getting an error.
The first run of any new job just hangs up before it even gives the waiting for resources message.
I have to reboot the server to cancel it as even cancel immediately doesn't work.
Then, if I re-run or re-try I get the error about the backup file is already present.
Now that I can't even get a new job to work, this is VERY critical as my production VM's are left without any new backups.
Knowing now that at least 2 users are getting this error, now indicates this is NOT an isolated case.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Aug 19, 2010 6:49 pm
- Full Name: John Gibel
- Contact:
Re: Failed to create backup file because it is already present on the file system
See my post, We just stared getting the same error today on EVERY job. Can not backup nothing !!Mildur wrote: ↑Jun 13, 2022 12:47 pm Unfortunately that can happen with our free products. You can try to open another case.
Have you already tried a job run with disabled application aware processing?
If nothing helps, consider using the Veeam agent as a workaround until you have found a solution to this issue.
We have a fully licensed version and have not had any major issues like this in 8-10 years.
-
- Product Manager
- Posts: 9848
- Liked: 2608 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed to create backup file because it is already present on the file system
Hi @alwitco
If you don't get an answer from support within SLA times, please try first to escalate the case.
This way, our support management knows about it and will see, that you get in contact with the engineer.
For the issue itself, I'd like to check if we have similar cases.
Please provide me with your case number.
Thanks
Fabian
If you don't get an answer from support within SLA times, please try first to escalate the case.
This way, our support management knows about it and will see, that you get in contact with the engineer.
For the issue itself, I'd like to check if we have similar cases.
Please provide me with your case number.
Thanks
Fabian
Product Management Analyst @ Veeam Software
-
- Product Manager
- Posts: 9848
- Liked: 2608 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed to create backup file because it is already present on the file system
@alwitco
@SignalSeeker
One Question, is there by any chance Crowdstrike installed?
If yes, can you check for any alerts or blocks (it may be informational) in the logs?
And please check your exclusion list: https://www.veeam.com/kb1999
@SignalSeeker
One Question, is there by any chance Crowdstrike installed?
If yes, can you check for any alerts or blocks (it may be informational) in the logs?
And please check your exclusion list: https://www.veeam.com/kb1999
Product Management Analyst @ Veeam Software
-
- Novice
- Posts: 8
- Liked: never
- Joined: Aug 19, 2010 6:49 pm
- Full Name: John Gibel
- Contact:
Re: Failed to create backup file because it is already present on the file system
I'm please to announce that as of 20 minutes ago, the issue has been resolved.
Something I should have expected as a possible cause right off the bat, but never put 2 and 2 together.
At the suggestion of a Veeam tech looking into my case, he asked if we had Malwarebytes or any other AV installed on the backup server.
We had Malwarebytes and Trend-Micro Endpoint protection.
I disabled both, and bingo life is good (or a little better) !!
I never suspected Malwarebytes or Trend-Micro as they were working in harmony for several years on the backup server, and I had all the possible exclusions set.
Now, it makes sense because everything stopped working around midnight, probably when an update to one of the two was sent out.
I can now sleep well again, as last night I had nightmares of coming into the office at 5:00 am and seeing our VMware servers all up in flames, with no recent backups.
Something I should have expected as a possible cause right off the bat, but never put 2 and 2 together.
At the suggestion of a Veeam tech looking into my case, he asked if we had Malwarebytes or any other AV installed on the backup server.
We had Malwarebytes and Trend-Micro Endpoint protection.
I disabled both, and bingo life is good (or a little better) !!
I never suspected Malwarebytes or Trend-Micro as they were working in harmony for several years on the backup server, and I had all the possible exclusions set.
Now, it makes sense because everything stopped working around midnight, probably when an update to one of the two was sent out.
I can now sleep well again, as last night I had nightmares of coming into the office at 5:00 am and seeing our VMware servers all up in flames, with no recent backups.
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Jun 17, 2022 5:40 pm
- Full Name: Jim Peters
- Contact:
Re: Failed to create backup file because it is already present on the file system
What was the issue with MWB? Should i completely remove it?
Would like to know if there is a security issue or other.
Thanks,
JimP
Would like to know if there is a security issue or other.
Thanks,
JimP
-
- Product Manager
- Posts: 9848
- Liked: 2608 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed to create backup file because it is already present on the file system
@alwitco
I‘m glad that it works now. We suspect, that multiple AV applications have updated their signature and now are blocking our Backup files/processes.
@jjpeterspa
Please check your AV logs if any files are blocked.
And check if you have excluded all veeam processes and veeam files as recommended in our guide.
https://www.veeam.com/kb1999
I‘m glad that it works now. We suspect, that multiple AV applications have updated their signature and now are blocking our Backup files/processes.
@jjpeterspa
Please check your AV logs if any files are blocked.
And check if you have excluded all veeam processes and veeam files as recommended in our guide.
https://www.veeam.com/kb1999
Product Management Analyst @ Veeam Software
-
- Veeam Software
- Posts: 2125
- Liked: 513 times
- Joined: Jun 28, 2016 12:12 pm
- Contact:
Re: Failed to create backup file because it is already present on the file system
To elaborate a bit more on the interaction between AV and backup applications (not just Veeam ), Read on. This applies a lot to firewalls also. Please excuse the wall of text, but it's worth the read to understand why random updates to security software cause this.
It's not so much that the definitions specifically target Veeam, it's more that some benign action Veeam does and has been doing for years suddenly triggers a heuristics rule or a block of data happens to hit a false positive.
Keep in mind that AV often works on heuristics and signature based matching, and this can often lead to false positives. If you were ever into some naughty activity with torrents and keygens and such, you know what I'm talking about
Now, it's very difficult to talk deeply about these subjects because AV is a black box by design; if they're too open on how they flag items, malware authors will just circumvent it, so a lot of this has to be hush hush. But, from observation, we can observe a few things and by logic we can deduce some others:
Signature Matching: The premise is simple -- a database of all naughty files/blocks would be insurmountably big and the processing power to check everything would slow your machines to a crawl. So instead, security software keeps a hash database and does fast-hashing of the data it's checking to see if there's a match (parsing strings of the same predictable length is infinitely faster and can be optimized for, and if there's a match, you then just fetch the necessary user info from elsewhere).
This has a few side effects though: With hashing, you either have strong uniqueness but computationally expensive (read: slow), or weak uniqueness but computationally cheap (fast). The latter is usually chosen so that security software doesn't interfere, but therein introduces a problem: if your uniqueness is weak, you have an increasingly non-zero chance of hash collisions, wherein two unique inputs produce the same output. As backup data is essentially random with billions (if not more) of unique data blocks being produced, you roll the dice billions+ times and eventually you hit a false positive match.
Heuristics: The idea here is much the same, but instead you apply levels of "risk" to processes when looking at it from a high-level. The more risk assigned, the more likely the AV is eager to step in and check it further or just block it outright. Both operations can interfere with normal operations of innocent software, and the worst part is that it's not super feasible to know what specifically triggers the security software. As we can see in the topic, even something benign like making a file gets flagged, but who knows why this is considered "risky." Even worse, sometimes it seems whitelisted directories/processes still get flagged in the spirit of security. A few AV vendors post a general overview of their heuristics scheme if you search "how antivirus heuristics work", so feel free to read on and see the actions they might do if they feel something is "suspicious." I personally feel that such methods are too often specious, but with security, it's a reasonable standpoint to err on the side of security than allow a malicious actor to act.
This isn't to say that it's always security software's fault, but it is why a lot of times it is a usual suspect and the guidance on what specifically to check is not always clear. Security Vendors have a vested interest in keeping their methodology secret (understandably), so almost always it has to be a test of trial and error, just to see what does and does not get triggered. Furthermore, it can be wildly inconsistent between environments.
For example, a case I had to step in on, Windows Defender was clearly blocking the installation of Enterprise Manager. We have tens of thousands of such installations without incident every year, many of them running Windows Defender. The client in the case rightfully wanted to know how to avoid having to disable more advanced Defender features, but we really had no idea how to even discuss such a thing. The client absolutely did not want to raise a case with Microsoft because they believed it would go nowhere, which I completely get, but we're stuck at an impasse then; we have no idea what Defender is doing, why it triggered in _this specific environment and not others_, and really have no better option.
So I know it's a ton of text, but this is generally how it works and why it causes issues. For some of the security vendors that have been very communicative and helpful, they openly admitted they had an overzealous update to a heuristic and a later definition update toned it down a bit (or they already had tricks to do a real whitelist). Others have been completely silent stating it worked "by design." I don't fault the latter, but it ends in a frustrating situation for the users and the application vendor.
It's not so much that the definitions specifically target Veeam, it's more that some benign action Veeam does and has been doing for years suddenly triggers a heuristics rule or a block of data happens to hit a false positive.
Keep in mind that AV often works on heuristics and signature based matching, and this can often lead to false positives. If you were ever into some naughty activity with torrents and keygens and such, you know what I'm talking about
Now, it's very difficult to talk deeply about these subjects because AV is a black box by design; if they're too open on how they flag items, malware authors will just circumvent it, so a lot of this has to be hush hush. But, from observation, we can observe a few things and by logic we can deduce some others:
Signature Matching: The premise is simple -- a database of all naughty files/blocks would be insurmountably big and the processing power to check everything would slow your machines to a crawl. So instead, security software keeps a hash database and does fast-hashing of the data it's checking to see if there's a match (parsing strings of the same predictable length is infinitely faster and can be optimized for, and if there's a match, you then just fetch the necessary user info from elsewhere).
This has a few side effects though: With hashing, you either have strong uniqueness but computationally expensive (read: slow), or weak uniqueness but computationally cheap (fast). The latter is usually chosen so that security software doesn't interfere, but therein introduces a problem: if your uniqueness is weak, you have an increasingly non-zero chance of hash collisions, wherein two unique inputs produce the same output. As backup data is essentially random with billions (if not more) of unique data blocks being produced, you roll the dice billions+ times and eventually you hit a false positive match.
Heuristics: The idea here is much the same, but instead you apply levels of "risk" to processes when looking at it from a high-level. The more risk assigned, the more likely the AV is eager to step in and check it further or just block it outright. Both operations can interfere with normal operations of innocent software, and the worst part is that it's not super feasible to know what specifically triggers the security software. As we can see in the topic, even something benign like making a file gets flagged, but who knows why this is considered "risky." Even worse, sometimes it seems whitelisted directories/processes still get flagged in the spirit of security. A few AV vendors post a general overview of their heuristics scheme if you search "how antivirus heuristics work", so feel free to read on and see the actions they might do if they feel something is "suspicious." I personally feel that such methods are too often specious, but with security, it's a reasonable standpoint to err on the side of security than allow a malicious actor to act.
This isn't to say that it's always security software's fault, but it is why a lot of times it is a usual suspect and the guidance on what specifically to check is not always clear. Security Vendors have a vested interest in keeping their methodology secret (understandably), so almost always it has to be a test of trial and error, just to see what does and does not get triggered. Furthermore, it can be wildly inconsistent between environments.
For example, a case I had to step in on, Windows Defender was clearly blocking the installation of Enterprise Manager. We have tens of thousands of such installations without incident every year, many of them running Windows Defender. The client in the case rightfully wanted to know how to avoid having to disable more advanced Defender features, but we really had no idea how to even discuss such a thing. The client absolutely did not want to raise a case with Microsoft because they believed it would go nowhere, which I completely get, but we're stuck at an impasse then; we have no idea what Defender is doing, why it triggered in _this specific environment and not others_, and really have no better option.
So I know it's a ton of text, but this is generally how it works and why it causes issues. For some of the security vendors that have been very communicative and helpful, they openly admitted they had an overzealous update to a heuristic and a later definition update toned it down a bit (or they already had tricks to do a real whitelist). Others have been completely silent stating it worked "by design." I don't fault the latter, but it ends in a frustrating situation for the users and the application vendor.
David Domask | Product Management: Principal Analyst
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Jun 17, 2022 5:40 pm
- Full Name: Jim Peters
- Contact:
Re: Failed to create backup file because it is already present on the file system
Not seeing anything blocked, though last i ran it manually with MWB disabled. Today its enabled and will see.Mildur wrote: ↑Jun 17, 2022 6:47 pm @alwitco
I‘m glad that it works now. We suspect, that multiple AV applications have updated their signature and now are blocking our Backup files/processes.
@jjpeterspa
Please check your AV logs if any files are blocked.
And check if you have excluded all veeam processes and veeam files as recommended in our guide.
https://www.veeam.com/kb1999
Who is online
Users browsing this forum: joshima and 16 guests