Windows File Backup Without Snapshot

[BackupBytesTim]

Support mentioned the possibility of solving an issue by switching from Volume-Level snapshot-based backups to File-Level backups and turning the snapshot feature off. At the time this was made it was believed that the computer in question was running Linux. So that was an option, however the computer is actually running Windows so it doesn't have the option to turn off the snapshot feature for File-Level backups to my knowledge, and by default I believe Windows will do snapshots for the regular File-Level backups. Is that accurate? Is there a way to do a Windows backup without a snapshot?

[Mildur]

Hello Tim

I moved your topic, because it's not related to the service provider console.

Veeam Agent for Windows does require VSS snapshots. Snapshot less backups are not possible.
We have a few request for snapshot less Veeam Agent backups.
Managed through a VBR server, you could use storage snapshot integration if Volumes are on supported storage appliances. But it's not possible with the service provider console.

May I ask, what issue are you trying to solve?

Best,
Fabian

[BackupBytesTim]

I have a computer that is failing to complete it's initial full backup in one session, and so when it starts again it starts over completely because the previous snapshot becomes unavailable to resume (at least that's the suggestion from support) so it needs to do a new snapshot and start over.

Alternatively, I'm fine with the snapshot functionality itself, is there a way to accept only a partially transferred snapshot as a complete backup? Treating it as though that was the entire snapshot, so the next one just has whatever's left? (Plus any changes of course.)

Or do you have some alternative suggestion?

Case #06259816

[Mildur]

Hello Tim

Did you see the entire data transferred again?

For Volume level backups, our agent never transfer the entire data in the retry after a backup session has failed. A VSS snapshot of the entire machine will still be created.
Please see the limitations of this in our user guide:
https://helpcenter.veeam.com/docs/agent ... tml?ver=60

A workaround for it is to use backup seeding for the initial backup:
https://www.veeam.com/kb3158

Best,
Fabian

[BackupBytesTim]

It has not completed the transfer. Unfortunately backup seeding isn't an option, even if that solves the issue this moment, that doesn't solve the larger issue of storing a partial backup and the computer is likely to have issues transferring more than a few hundred megabytes at a time due to how often it's online and how it's connected. Also there are other computers that occasionally have the same issue.

Could that be added as a quick feature request? Since the Linux Agent already has the same feature, as does every other backup software I've used significantly. In my opinion the issue is more with Veeam deleting the partial backup. If there was a way to prevent that then I don't see anything wrong with the use of snapshots, only with Veeam deleting incomplete snapshots.

Unless I'm misunderstanding your comment. I'm not actually sure what you mean by "our agent never transfer the entire data in the retry after a backup session has failed. A VSS snapshot of the entire machine will still be created".

[Mildur]

Could that be added as a quick feature request?

Unfortunately not. It's not just a bug to solve.
A new feature may be considered if there are enough requests available. At the moment we have a lot more of other long awaited features we are working on.

Unless I'm misunderstanding your comment. I'm not actually sure what you mean by "our agent never transfer the entire data in the retry after a backup session has failed. A VSS snapshot of the entire machine will still be created".

1.) A job starts on his configured schedule.
2.) The job creates a VSS snapshot
3.) The job sends backup data to the repository
4.) The job fails
5.) The job starts a retry session
6.) The job creates a VSS snapshot
7.) The Job compares the hash values on for data blocks on the source disk and already stored backup blocks on the backup repository
8.) Only not yet uploaded, changed and new blocks will be send to the repository.

Limitations for this procedure is in our user guide. For example, if you use File Level backup jobs, we transfer the complete backup data again. Or if you have started the initial job manually.

Best,
Fabian

[BackupBytesTim]

Does that mean it should be retaining the partially transferred backup, and only transferring new data? That's what it sounds like what you're describing. Which is what I want it to do, however it's not doing that. It's deleting the partially transferred backup data and starting over again each time. Which is what support indicated is the expected behavior. Support suggested as long as it's using a snapshot, it must transfer the entire snapshot in one job session, that it deletes the partial snapshot when it tries again.

Support said that if it were doing a file-level backup (with no snapshots) that it would retain the partially transferred backup data, the files that got transferred. But that there's no way to disable the snapshot feature in Windows, only in Linux, and so I can't implement that as a solution here.

Does that behavior you're suggesting also apply to cache syncing? That's what it's usually trying to do and failing to do the described behavior.

It is an "Entire Computer" volume-level backup, which is just running on schedule. Technically not actually "on" schedule, it's usually offline at the scheduled time and so it runs when comes online, but it's not being started manually.

[BackupBytesTim]

Any suggestions from anyone?

[Dima P.]

Hello Tim,

I believe Windows will do snapshots for the regular File-Level backups. Is that accurate? Is there a way to do a Windows backup without a snapshot?

Veeam Agent for Windows indeed requires VSS snapshot to be created to take an application consistent backup, there is no way to avoid it (otherwise every opened file during backup will be stored with inconsistent backup). We use VSS for all types of backup entire computer, volume level, file level.

is there a way to accept only a partially transferred snapshot as a complete backup?

We have a resume functionality in Windows agent that should continue the failed backup. There are some limitations: this functionality works for the entire computer backup or volume level backup and for the scheduled job runs (i.e. it does not work for manual job start). What are the original Windows job settings (i.e. how the source was configured and what was the job destination)?

Treating it as though that was the entire snapshot, so the next one just has whatever's left? (Plus any changes of course.)

Snapshot is created on the machine and is used for backup however we are not transferring snapshots, just a note to be on the same page.

Support said that if it were doing a file-level backup (with no snapshots) that it would retain the partially transferred backup data, the files that got transferred. But that there's no way to disable the snapshot feature in Windows, only in Linux, and so I can't implement that as a solution here.

Correct. Windows Agent always requires a VSS snapshot to be created, no way to avoid it.

It is an "Entire Computer" volume-level backup, which is just running on schedule. Technically not actually "on" schedule, it's usually offline at the scheduled time and so it runs when comes online, but it's not being started manually.

For entire computer backup and volume level backup you can use backup cache functionality, it creates backup first on a local disk then transfers it to the target periodically when the connection is available. Please try it and let us know if it helps in your scenario.

Case #06259816

Thank you for the case details, we will review it!

[BackupBytesTim]

If you've looked at the case you probably saw, support had the same recommendation regarding use of the backup cache, however in my opinion that makes things more problematic since it still seems it needs to transfer 100% of a restore point in one session, else it restarts. So now it's trying for weeks to transfer a backup that's weeks old, sitting in the cache, it's not actually trying to backup new data. So if/when it eventually finishes uploading a restore point, that restore point is already weeks old and past the desired retention period anyways.

Treating it as though that was the entire snapshot, so the next one just has whatever's left? (Plus any changes of course.)

What I meant with this wasn't asking about transferring the snapshot itself, just if there was a way to like mark the transferred data as a completed restore point, so the next time it runs it assumes the entire contents of the snapshot were transferred and so it just does a new snapshot and creates a new restore point without deleting the existing one.

Veeam Agent for Windows indeed requires VSS snapshot to be created to take an application consistent backup, there is no way to avoid it

The current method doesn't seem to work very well, my understanding is that it relies on the same snapshot being available (and potentially other circumstances as well) in order to resume in the middle of a restore point, so maybe Windows is deleting the restore point when the computer restarts or something, I'm really not entirely sure, but the end result of whatever's happening is that Veeam is starting over each time.

Veeam Agent for Windows indeed requires VSS snapshot to be created to take an application consistent backup, there is no way to avoid it (otherwise every opened file during backup will be stored with inconsistent backup). We use VSS for all types of backup entire computer, volume level, file level.

I am aware that without the snapshot then the volume isn't necessarily consistent to a single point in time for all files, however the majority of things don't care about that, and in this case, whether an application on the customer's computer cares about consistent data like that or not, there's no backup right now. So an inconsistent, snapshotless backup, seems preferable to no backup. So while I understand the reason for using snapshots, it would seem to be beneficial to be able to backup without it. Someone seems to have had this thought when designing the Linux Agent, it would seem the same reasoning would apply to any operating system, not just Linux.

it does not work for manual job start

Does that mean if I start a backup job manually it will not attempt to resume the previous job's restore point (which is my current understanding) or does that mean that if a manually started job gets interrupted, that job will not resume? Or both?

[Dima P.]

Hello Tim,

If you've looked at the case you probably saw, support had the same recommendation regarding use of the backup cache, however in my opinion that makes things more problematic since it still seems it needs to transfer 100% of a restore point in one session, else it restarts

Backup lands to the local cache, then it be slowly synced with the target including resumes on the data transfer for the connection drops.

next time it runs it assumes the entire contents of the snapshot were transferred and so it just does a new snapshot and creates a new restore point without deleting the existing one.

Got it. That's exactly how it should work in Veeam Agent for Windows during entire computer backup and volume level backup for the scheduled backup job runs.

The current method doesn't seem to work very well, my understanding is that it relies on the same snapshot being available (and potentially other circumstances as well)

It does not rely on the VSS. We take VSS snapshot during the job start, start transferring the data to the repository. Backup job fails, say due to the network being disconnected, VSS snapshot is deleted, incomplete backup remains in the repository. Backup job retries, creates new VSS snapshot, starts to compare the changes between existing machine state and what has been transferred to the repository on a intimal run. Similar data blocks are not reuploaded, new data blocks are uploaded.

in order to resume in the middle of a restore point, so maybe Windows is deleting the restore point when the computer restarts or something

In order for resume to work that should be within job retry, job retry is performed when the agent is running but faced an issue such as connection drop, or VSS snapshot creation issue or repository being inactive for some reason. When you shut down the machine or restart it resume wont work since it's possible to retry the previously stated job run.

Does that mean if I start a backup job manually it will not attempt to resume the previous job's restore point (which is my current understanding) or does that mean that if a manually started job gets interrupted, that job will not resume? Or both?

Both because manual job start does not have a retry logic.

[BackupBytesTim]

Backup lands to the local cache, then it be slowly synced with the target including resumes on the data transfer for the connection drops.

That's not happening. It's starting over each time with the backup cache same as it would be with regular jobs direct to the repository.

Got it. That's exactly how it should work in Veeam Agent for Windows during entire computer backup and volume level backup for the scheduled backup job runs.

Well, how it "should" work isn't how it "is" working, so I'm thinking something is broken.

It does not rely on the VSS. We take VSS snapshot during the job start, start transferring the data to the repository. Backup job fails, say due to the network being disconnected, VSS snapshot is deleted, incomplete backup remains in the repository. Backup job retries, creates new VSS snapshot, starts to compare the changes between existing machine state and what has been transferred to the repository on a intimal run. Similar data blocks are not reuploaded, new data blocks are uploaded.

But it's not doing that. That's what I want it to do, but I have multiple devices across multiple customers that aren't doing that. They're deleting the existing incomplete restore point and starting over again, been going on for months at some, weeks at others. With no completed restore points during that time, which I find highly unacceptable.

In order for resume to work that should be within job retry, job retry is performed when the agent is running but faced an issue such as connection drop, or VSS snapshot creation issue or repository being inactive for some reason. When you shut down the machine or restart it resume wont work since it's possible to retry the previously stated job run.

Okay, well... that explains the problem then, so this back to the other problem I have of Veeam doesn't retry after a reboot. Which causes other problems for us just because it creates large gaps in the recovery points a computer has. And now also causing a problem because it cancels the resume functionality.

Realistically, if I somehow prevented Veeam from deleting the partially transferred restore point, would that restore point's data still be recoverable? Or is it compressed in some manner that would render the contents of a partial VIB file useless?

[Dima P.]

Thank you for the feedback!

That's not happening. It's starting over each time with the backup cache same as it would be with regular jobs direct to the repository.

I've asked QA team to doublecheck this scenario.

Realistically, if I somehow prevented Veeam from deleting the partially transferred restore point, would that restore point's data still be recoverable?

Nope, such restore point will be incomplete, within our logic in will be threated as corrupted and wont participate in any recoveries.

[BackupBytesTim]

Nope, such restore point will be incomplete, within our logic in will be threated as corrupted and wont participate in any recoveries.

I know I don't work at Veeam so I'm not really entitled to a highly technical explanation of how it works, but I'm curious at least to know if there's a reason for this, like the way the data is compressed or something makes a partial file useless or if someone just decided that the software shouldn't even try to access data from a partial file and so there's not actually a technical reason for that? Other software I've used can access partial backups, even with image-level backups created via a system snapshot, so I'm curious why Veeam can't. I can accept it may be due to something like the particular method for the data being compressed, as that's obviously a software-specific situation, not all software compresses things the same way, but if it is just that someone decided it shouldn't be a feature, then I'd be curious if that was on the roadmap for future implementation at all?

I've asked QA team to doublecheck this scenario.

Thanks, and to clarify, what I mean is "if it does not transfer a complete restore point from the cache, it deletes the partial restore point and starts over" which is the same behavior for if it backs up directly to the repository without the cache feature at all. As far as I am aware it does keep individual restore points once transferred from the cache, it is not requiring that 100% of the cache be transferred, just 100% of a restore point from the cache. However technically I haven't tested it extensively, to my knowledge there's nothing that indicates the number of restore points stored in the cache.

[Dima P.]

I know I don't work at Veeam so I'm not really entitled to a highly technical explanation of how it works, but I'm curious at least to know if there's a reason for this, like the way the data is compressed or something makes a partial file useless or if someone just decided that the software shouldn't even try to access data from a partial file and so there's not actually a technical reason for that?

When you are backing up a file its being chunked and compressed into data blocks and such data blocks land to the vbk/vib backup file. Imagine you are cutting the connection in the middle of data transfer, how to open such file? When you backup several files, folders or volumes multiply this issue to all the files because data is being processed in parallel, meaning blocks related to different files are transferred at the same time. Now you cut the data transfer with reset or shut down.

How to say what data in the restore point is valid and which data is invalid without actually checking the data and validating it's consistency? That's exactly what's happening on the job retry where we can understand what blocks are missing and upload only the missing data. If the retry is not happening there is no way to compare what's changed, thus incomplete restore point is being removed completely.

Now let's add retention to the picture. You want to keep 7 days of backups - should such incomplete restore point participate in the retention? If yes - at later time you are replacing the complete restore point full of the needed data with incomplete restore point where some data is missing, that would be a data loss. If no it should be put aside and such restore point should not participate in the retention, then the restore point followed by incomplete one should contain all the changes from the last complete job run meaning such incomplete restore point should no longer be a part of the backup chain. That's what we do - we either try to resume it and make incomplete restore point complete or we remove it from backup chain to make the backup chain consistent in terms of retention.

Other software I've used can access partial backups, even with image-level backups created via a system snapshot, so I'm curious why Veeam can't.

You can access incomplete restore points with Veeam B&R console via File level recovery, and possibly restore a file from such backup if it's data was transferred before the connection was terminated, but during the next job run will remove such incomplete backup (unless it's resumed by the job retry).

[BackupBytesTim]

How to say what data in the restore point is valid and which data is invalid without actually checking the data and validating it's consistency?

I can think of multiple ways that could be handled, some sort of hash check to validate the consistency would be an option, but I'd assume it'd be easier just to track what files are completely transferred. I understand the "chunk" methodology, but it seems we can still determine what the contents of the chunks are without needing to transfer all the chunks.

should such incomplete restore point participate in the retention?

EDIT: To be clear, my "common complaint" is that a lot of the things about Veeam seem leftover from the original "vm only" design that it had, like no thought was given to handling workstations any differently than virtual machines, that and numerous other design decisions that just don't seem to make any logical sense to me, like the last point in this post as well.

This question comes back to one of my common complaints about Veeam. It was originally designed to backup virtual machines, typically used in a server environment for hosting different applications. It was not originally designed for backing up individual computers directly. The way Veeam handles "restore points" is leftover from the original design where the ability to restore the entire infrastructure to a single point in time is highly preferable. With individual workstations (though when a company's entire infrastructure is 1 or 2 laptops I debate even calling them "workstations") that's really not necessary. There is rarely ever a need for the entire computer to have a single point in time copy of all the data. I'd rather have the entire computer's contents, but not be consistent, have some documents from Monday, some applications from Tuesday, some emails from Wednesday, the operating system from Thursday, and the local copies of cloud synced files from Friday, then have nothing. Sure getting everything at one single point in time is nice, but I wouldn't describe it as necessary virtually ever, at least not with workstations. With servers that have things like large databases, numerous virtual machines that all need to work together to run the company's infrastructure, that makes sense.

So, I guess the simple answer is I don't believe there should be "restore points" there should just be "files" and we attempt to backup all the files each time, but if all the files were not successfully backed up, the ones that are backed up are still useful and can still be restored. So each file has "versions" that can be restored, and we can note (for the sake of helping the user make an informed decision) whether a particular "restore point" actually contains the entire computer's contents, so the user can decide whether to restore a partial restore point, optionally filling in missing files with their previous versions, or just to restore the last completed restore point.

Ultimately the goal is to have a backup, I'd rather have a partial backup each time vs no backup. Right now I have no backup. (For some customers.) Due to what still seems like a design that's leftover from the original function Veeam served of backing up virtual machines in a large enterprise environment. It seems the ability to back up workstations was sort of stuck on and so we treat workstations like they're a collection of virtual machines in a large enterprise environment that must all be point in time consistent, without the software actually being redesigned to back up workstations.

Whether or not everyone has my issues with getting Veeam to actually create a successful restore point, I'm guessing if we were to poll people, most people would agree that there's no need for workstations to be point in time consistent for most scenarios. I can agree there may be some specific scenarios where that's the case, and I'm not saying Veeam shouldn't try to do that still, but the current behavior of assuming a partial restore point is useless is outdated and causing grief.

we either try to resume it and make incomplete restore point complete or we remove it from backup chain to make the backup chain consistent in terms of retention

I understand if incomplete restore points can't always be made "complete" for the very specific, single reason of the OS deleting the snapshot that was used, and I know I've asked about this recently, but it seems we could still be trying to continue from a previous snapshot the next time a job runs, no? Alternatively, if we go with the "point in time consistency is not required" idea, could we just do snapshot-less backups of the entire computer? I did have a separate thread recently about it, but I believe that's a feature available on the Linux Agent, which is definitely helpful, and I could definitely make use of the feature on the Windows Agent too if it were available.

You can access incomplete restore points with Veeam B&R console via File level recovery, and possibly restore a file from such backup if it's data was transferred before the connection was terminated, but during the next job run will remove such incomplete backup (unless it's resumed by the job retry).

This confuses me, it would seem that incomplete restore points are then definitely accessible, to the extent of the data that was transferred. So then why does Veeam delete it and transfer the same unmodified data again the next time it runs? Not to be overly critical of any particular person who made that decision, but that seems like just poor design, if the data is present on the repository, why do we delete it and transfer it again, especially in a scenario where the data didn't change since the last time it was transferred?