Feature request: Merge different job data on the fly if same machine is backed up by 2 jobs

[Poweruser]

I backup different Hyper-V machines with 2 jobs.
one backups them offline (shuts them down)
and the normal job backups them online/vss.

now i have 2 full big files per machine and i bet nearly 90% of the data will the identical.
i suggest, that veeam will notice, that it has a double full file of a VM and merge them, so i just have a delta per job but only one full file for both jobs.

currently backing up the same machine with 2 jobs leads to double space wasting.

[david.domask]

Hi Poweruser,

May I ask, why 2 jobs for the machine? I understand you're doing cold vs hot backups here, but what is the benefit of having the additional cold backup?

Right now there is not a means to do this; it sounds like you're imagining Fast Clone being able to use data from a different backup chain, but at this time Fast Clone will only check a current backup chain.

But maybe if you can explain the need for the extra cold backup, we can find a better solution.

[Poweruser]

The reason is, that the vendor says, that the offline backup (cold) is the only method, which will 100% work.
so i have to do this. also i do this befor big updates, so i have a consistent state of all machines without any risks of open files etc.
so the machines can be recovered independently, as they all have no state to each other (or connections etc).
also there is no risk if vss etc is not working fine or some files are open. shutdown and powered off is the cleanest way to backup.

the problem of hot backup is also, that the exchange server will immediately receive new emails which are behind vss. if an error occurs, alls received emails after vss are lost forever, as they are confirmed to the other smtp server as received already.
if its offline, it cant receiver anything. nothing can be changed.

[david.domask]

Understood, the use case is more clear now. Is it just the hypervisor snapshots that cause the issue or it's completely the VSS freeze?

As for the main request, consider it noted and maybe just merging the jobs and powering off the machine for a monthly backup or before change freezes, or even using Quick Backup after powering off the machine to have such a point. You can do Export Backup from such a Quick Backup if you have a requirement for full backups during this time and reduce the total time you need to have the VM offline.

[Poweruser]

Its just a normal office-hours server, so its mostly not important, that the machine is running 24/7.
main office hours and some hours more is enough. maybe this clarifies, that downtime is no real matter for a backup.
the problem is, that the main business software vendor says:
a) do offline backup
b) do also online backup and use our SQL export to file tool while doing so
c) they dont give out sa account... so many things may not work as desired...

so i just do both, to be sure. an offline backup while machine down must be considered as always 100% safe, so this is imho they best option ever.
this is what i do before an bigger update/patchday etc.

normal days just use the online backup. the scripts write down an online backup of the sql to a safe folder on the same machine. after pre job is done, then - when the machine is backed up - these folders are also backed up, as they are inside the vm which is being backed up.

so its quite tricky. also it is never ok to restore or access sql elements manually. veeam may never change anything inside the sql, even if you know the sa account or gain access.
consider the sql database restore/backup as one ACID transaction all-or-nothing.

so we dont know if a vss freeze or whatever will create a consistent state. also sql and filesystem build together the data of the business software. means it must be consistent.
you have a gap between online sql backup job (internal tool of business sw vendor) and the filesystem.
so only the offline backup is 100% safe.
the online backup may be used - but - you are not allowed to restore sql by veeam, you can restore everything else but the sql must be restored using their tool.
theoretically you could restore db files manually, but this is allowed if and only if you stop the sql service first.
so we have a double "online" backup:
first vm is online
second sql inside vm is online

normally i stop the sql server inside the vm to make sure we are in a safe and consistent state. this is the only safe solution except the complete shutdown.
also i do the online (tool) sql backup in addition, just to make sure we have thousands of restore possibilities.

maybe this clarifies a bit more, but you see, the business software is very special..

restoring email to exchange on per element basis is easy, but restoring parts of sql not.
the sql is a big vendor locked in dbms.

[david.domask]

Understood, thank you for the clarification, I have heard of similar issues.

The request is noted, but consider the Quick Backup function with the machine powered off to allow it to be part of the main backup chain; with Fast Clone capable repositories, you will get additional space savings that re-use blocks from the existing backups on top of the deduplication/compression already part of the backup process.

[Poweruser]

okay, maybe i have to look inside this, but for my 2 variants i need different pre and post job scripts.
i must be able to set different scripts to quick and normal backup.

the online job makes: sql check, sql online backup (tool), sql shutdown -> sql start, sql optimizations
the offline job makes: sql check, vm Shutdown in right order -> vm start in right order and delay, big sql optimations

[david.domask]

If you're going the scripting route, I would just set a task in Task Scheduler and use PowerCLI in combination with Veeam cmdlets to start the Quick Backup; if you go with pre/post job scripts, you can do it similarly and write some logic to detect if it should start the shutdown first.

[Poweruser]

in past i did all the jobs local inside the vm using "crons"
but using fixed times and different places leads to problems for maintenance the code over years.
maybe i could provide a variable inside the pre/post job with the job name or sth else?

[david.domask]

Right now there aren't built-in variables for such things, but you can use the following Powershell code to get the job which started the script:

Code: Select all

$ScriptPID = Get-WMIObject win32_process | Where-Object {$_.ProcessID -eq $PID}
$ParentProc = Get-WMIObject win32_process | Where-Object {$_.ProcessID -eq $ScriptPID.ParentProcessId}
$JobID = $ParentProc.CommandLine.Split()[7].Trim('"')
$Job = Get-VBRJob | Where-Object {$_.id -eq $JobID}

$Job will have a CJob object returned by Get-VBRJob, so it will know which job started the script and you can take specific actions based on your pre-job script.

[Poweruser]

Ok this would be an option. i hope veeam will add some more options someday without scripting :-)