Rolling back changes query

[glamic26]

I've got a quick query about the rolling back changes functionality and whether it is expected behaviour, and if so I then have a feature request!

We have some jobs that have a backup window on them to prevent them from running during working hours. Normally these jobs finish without any issues, however if we have an issue with the environment (rebuilding disk, source storage latency, stuck snapshot etc) then on occasion the job doesn't quite finish and is then stopped by the backup window. This then lands us in a situation where the backup will now never complete due to the requirement to roll back the changes made by the previous incomplete job run. And the close the job was to finishing (and the more data there is therefore to roll back) the longer it takes to do the roll back. The problem is that this roll back doesn't even start until the new scheduled start time of the job. That then eats several hours into the backup window rolling back the changes and the job is then stopped again at the end of the backup window and the same thing happens the next time the job starts and it has to roll back all of the changes again!

The only reason we have a backup window is to prevent anything affecting the production storage during the day. We have no issues with the Veeam infrastructure performing additional IO during the day as it is a self-contained environment in terms of storage and compute. So ideally we would like the changes to be rolled back as soon as the job fails due to the backup window so that at least when the job starts the following night it doesn't have to wait for hours rolling back changes and it should, therefore, be able to complete in time before the backup window closes. Alternatively, it would be good to have a manual option on a job to roll back changes made by incomplete job run whenever we want to from a right click menu on the job. Or a scheduled task to start any required roll backs during the day. Or lastly, if the job can calculate how long it is likely to take for the roll back and start the job that much earlier in order to complete the roll back (this wouldn't be ideal but better than the current situation).

P.S. We cannot change the job to run earlier because the system is needed for production until that time.

[foggy]

Hello, are we talking about reverse incremental jobs?

[glamic26]

Yes, on the jobs where this is occurring for us they are reverse incrementals.

[tdewin]

Have you considered backup io control / latency io control to prevent affecting the production storage? You might be able to disable "backup windows" and still not impact your production
https://helpcenter.veeam.com/docs/backu ... tml?ver=95

This is however limited to reading the data from production, not the backend systems (backup server/repository)

[glamic26]

Hi, yes we do use latency IO control already and this often stops further tasks from running and throttles IO. There does seem to be some issues with it where it seems to think there is latency when there isn't, and it also doesn't seem to return to full speed again once the latency has gone. It actually ends up making the backups take longer because of the throttling, but at least hopefully prevent extremely high IO on the production storage overnight. It doesn't really help us with the above issue though unfortunately.

[foggy]

The described behavior is typical for reverse incremental jobs only, where changes from incomplete VRB are copied back to the full backup file the next time the job starts after the failure. If you switch to forever forward incremental (which is always recommended in terms of backup duration and space consumed), you will not see this anymore.