From infrastructure function missing in new backup copy job

[EvanWaite]

@Gostev, I realize at some point you need to ship v12 and hard choices like that need to be made. Does Veeam have plans to port "From infrastructure" selection into the new engine in the future? (in the pipeline, I understand you can't give a timeline even if it is).

The impact of not being able to continue using VM tags for retention based copy job selection will require a pretty major change in our primary job layout. This takes us from a very small (12 primary, 20 copy job) layout to over 100 jobs (possibly more). Thankfully we're not forced into that right after we upgrade to v12 so can transition to the new layout over time but it's still a lot of effort. Using tags to abstract all our backup settings into vCenter (aka full policy driven) has been quite useful in our environment.

[Gostev]

So what you're really after is machine-level retention policy? In other words, you don't really care to be able to select machines for Backup Copy jobs from infrastructure (what this topic is about)? Rather, you used this now-missing capability as a workaround because the feature you actually need is missing? Or is there more to it that I am missing? Thanks.

[EvanWaite]

@Gostev, we assign tags to each VM to cover a number of different Veeam related settings (closely following Luca's excellent design from https://www.virtualtothecore.com/using- ... rotection/. Tags come from 4 categories.
* Backup Job (Primary job, based on start time ex: 6pm, 9pm, 1am, 5am, etc)
* Backup Authentication (flips the account for guest processing)
* Backup Options (turns off certain options, controls method of indexing, app aware, db log backup, etc)
* Backup Retention (which drives Backup copy jobs with certain retention ex: 15 Day, 45 Day, 1 Year, etc)

This allows us to have a mix of various retention VMs out of a single primary job. It also allows provisioning scripts or manual configuration by a vCenter administrator to set/change any settings as needed. We have two backup copy destinations and those retention tags handle selection for the associated copy job for both (example a tag "45 Day" would be used in two copy jobs going to two different SOBRs in different locations). If the administrator needed to change the VMs retention, it's just a matter of flipping the tag on the VM in vCenter (which does create another chain but with immutable backups, we already size for this happening occasionally)

We've been using this method in our environment since B&R v9.0. Previously we used a mess of VM folders to select into primary jobs then copy jobs tied to those (with over 100 primary and copy jobs even back then). For us, the new v12 method is moving us closer to the old "lots of jobs" to handle the mix of requirements in our environment.

To answer your question about machine level retention, yes in theory that's what we use tags with copy jobs for right now. A lot of the attraction of tags for us it is delegating to people who have zero access to our Veeam B&R environment and the consistency with other Veeam related settings being driven off tags. The ultimate goal is finding a more elegant solution than creating a new primary job for each start time with different retention (aka 6:00pm 15 Day, 6:00pm 45 Day, 6:00pm 1 Year, 9:00pm 15 Day, 9:00pm 45 Day, etc), if that can happen without tags but in a scalable method (driven by PowerShell or some other method), we're open to ideas.

[dries.vergote]

As an MSP we have the same issue. We have retention based tag that allows us to populate the primary backup job with the specific tag,
Same goes for the backup copy jobs, When a vm get a specific tag it is added to the backup copy job for longer retention to other repositories.

Selection from full hierarchy is possible in the exclusion but not for the include.
So you have to select your full repo and start excluding the tags you don't want then?

Now we have to go back to jobs per customer and link those to a copy job.

Hope there a solution for this fast.

[Spex]

We use tags the same way as EvanWhite wrote and it is very efficent.

@Gostev, we assign tags to each VM to cover a number of different Veeam related settings (closely following Luca's excellent design from https://www.virtualtothecore.com/using- ... rotection/. Tags come from 4 categories.
* Backup Job (Primary job, based on start time ex: 6pm, 9pm, 1am, 5am, etc)
* Backup Authentication (flips the account for guest processing)
* Backup Options (turns off certain options, controls method of indexing, app aware, db log backup, etc)
* Backup Retention (which drives Backup copy jobs with certain retention ex: 15 Day, 45 Day, 1 Year, etc)
...

Because of this, I wish I had the same functionality for agent-based backups too!
This would give the server administrator full control over the backup settings in a controlled manner without having to have access to the B&R environment. Also there would be no need to setup a whole bunch of backup/copyjobs for agent based backups since all jobs would be tag based.

[dvdn61]

We upgraded to Veeam v12 (on vSphere) yesterday and just found out that the "select by tag" and "from infrastructure" features were removed from the copy job settings.
For us, just as for many other businesses using Veeam, this is a big deal. We have an entire policy-driven Veeam environment where all vm's from our many clients are being backed up using vSphere tags in an automated process. The vm retention policy is directly related to the tag being issued.
Our vm's reside in multiple datacenters and are backed up locally with a short retention, due to limited backup storage available. But then tag-based copy jobs take care of moving these backups offsite to a central location with much more available storage. Here retention is based on what our clients need, and this can be anything from 2 weeks to one year (or even longer using tape backups). So the tag-based copy jobs are very important to us, and this change creates quite a challenge. It would definitely have been a no-go if we had known this beforehand. Very strange that the release notes did not mention any of this.
Fortunately copy jobs are currently still working in legacy mode, but we are now unable to add new tag-based copy jobs if the need arises in the near future. @Gostev, will this feature be brought back in a coming update?

[Gostev]

No, as sadly this feature is not a quick fix to implement for the new Backup Copy architecture from what I heard. @Dima P. @Egor Yakovlev we're missing you from this thread, please track this.

[Dima P.]

Hello folks,

Thank you for your feedback! The backup copy communication with infrastructure requires additional processing time which affects the RPO. Simply put: for every query based on tag the job must await for the list of machines from the hypervisor and match those with backups in B&R to start processing the data. Mind me asking a couple of questions regarding the setup with tags to better understand your scenarios.

Say I have a tag ‘short term retention’ such machine should only be processed by the primary backup job which has short term retention tag as a source. However, if have a ‘long term retention’ tag it’s included to the primary backup and the backup copy job source at the same time. Did I get it, right?

If I got it right, can you please elaborate why the workaround to include the 'long term retention' primary job to a 'long term retention' backup copy job is not an option for you? As soon machine is added to the primary backup job included in the backup copy – machine is automatically picked up by the backup copy too. Thank you!

[JaySt]

No, as sadly this feature is not a quick fix to implement for the new Backup Copy architecture from what I heard.

Ok, would it then be a better path to be able to create legacy jobs to have these features back?

[EvanWaite]

Say I have a tag ‘short term retention’ such machine should only be processed by the primary backup job which has short term retention tag as a source. However, if have a ‘long term retention’ tag it’s included to the primary backup and the backup copy job source at the same time. Did I get it, right?

We don't mix the tags used in primary backup jobs with the tags used in backup copy jobs. They're from separate categories. In our environment, a VM contains at least a tag from the "Backup Job" category (used to place it in the primary backup job) and a tag from the "Backup Retention" category which is used to place it in a backup copy job with certain retention. Here's an example. The first 3 tags (from Backup Processing, Backup Options and Backup Job categories) are used with the Veeam primary job. The last option from the "Backup Retention" category is the one tied into a copy job.

Code: Select all

Name                           Category
----                           --------
No App-Aware Proc              Backup Processing
No Guest Indexing              Backup Options
6:00pm                         Backup Job
45 Day Group 1                 Backup Retention

[Dima P.]

JaySt,
You can use it as a temporary solution. However, we are currently not adding support for all new functionality to legacy jobs and we have plans to deprecate legacy backup copy completely some day.

Evan,
Got it, thank you. The combination of first three tags define the primary job where you will land the VM right?

[EvanWaite]

Evan,
Got it, thank you. The combination of first three tags define the primary job where you will land the VM right?

Basically yes, the first tag (Backup Job) controls which primary job it lands in, the options and processing tags (as well as an authentication category I didn't list) controls any job specific options but those are used for any of the primary jobs to toggle things from defaults.

[JaySt]

JaySt,
You can use it as a temporary solution. However, we are currently not adding support for all new functionality to legacy jobs and we have plans to deprecate legacy backup copy completely some day.

yes i understand. but i can still see the ability to create legacy jobs (without new functionality) as pretty handy thing when a deployment requires (because pre-v12 design was based on it) the infrastruture objects selection for BCJ.
Yes, it gets deprecated one day perhaps, but it could be a bit more gradually with this flexibility of being able to create legacy jobs. It caught alot of us off guard this change (combined with the fact your cannot stick to the legacy format when creating new jobs). the way i see it now, legacy jobs are already past "deprecated", given you can't create them anymore. Discouraging the use of the legacy job format (by ways of gui messages, documentation etc.) before preventing the creation of them would have been better imho.

[emachabert]

Just been called on an upgrade issue within one of our production environment where a Legacy Backup Copy Job target could not be upgraded to new chain format because the ops team could not rebuild the new image level job and realized this was related to this regression.
We have machines that need to be copied for compliance to different target/location, they have the same backup policy but different second or third protection level policy, so they share the same backup jobs but not copy jobs.
Quick fix I see is building backup jobs dedicated to the full protection policy (backup+compliance) and moving the VM between jobs but this will add quite a few backup jobs and change the time alignment of backup processing of business related VMs.

keep us updated about the come back of tag based or per item selection.

Thanks.

[AlexandreD]

Hello,

Same observation here, there is no more "from infrastructure" on new backup copy jobs.

We use backup copy job based from infrastructure for Cloud Director environnement.
Tenant cannot create themself backup copy job from Veeam plugin, and we have to create backup copy for them.

So in a first time, we create a backup copy based on Cloud Director Organization (from infrastructure) so every new primary backup jobs are include automatically in a backup copy (I agree it is not optimal because retention doesn't match with retention configure by tenant, but it's a security).

I don't know how other service provider work with Backup copy jobs and Cloud Director.

Alexandre

[dvdn61]

I just found out that the "select by tag" functionality is still available in v12 copy jobs. But only under the "Exclusions" button. So that means that the tagging logic is still available within the code for copy jobs. Why then not for >>including<< vm's based on tags?

[Gostev]

Exclusions are too easy. Inclusions are much harder as there can be multiple backups of the same VM and it's impossible to say which one to pick for copying just from the tag. And the worst part is all the data loss/corruption bugs caused by accidental "jumping" between different backups of the same VM.

[karsten123]

enough functionality for me. thx

[emachabert]

Anton from what you said I understand all the issues that can come from "automatic" discovery based only on tags and it clearly disqualify that kind of selection.

Wouldn't it be "lighter" to add an include logic where we could explicilty select what we want to include from a specific backup job.
So we would have two buttons "include" and "exclude", be default include would have a default checked option "include all". One could uncheck the "include all" option and then make explicit selection (select a backup job and then a VM handled by the backup job to avoid any mismatch), thus avoiding any automatic discovery process ?
Like a rule engine, the exclude option should at any time override any include in case of conflict.

[Dima P.]

Hello Eric,

Right now within the new image-level backup copy job you can include the primary backup jobs as a source and then exclude specific VM underneath that job but that's have to be done manually. I like the idea of the predefined exclude / include, noted! Thank you!

[emachabert]

Yes the issue with manual exclusion as you might guess is the need to do it every days as new VM enter the backup via tagging, then you end-up making some powershell magic.

[BackItUp2020]

Here to report that this is a major bummer for me too. We rely on tags and/or choosing specific datastores and now its going to be messy.

[alec-at-work]

Adding another vote to the pile for this. I've just sat down with a customer to implement some backup copy jobs for offsite copies, and I can't do what we've promised them. I.e. take a subset of VMs from their existing backup jobs and copy them to our offsite repository (either VCC or S3). Now faced with either a really clumsy exclusions based setup that will have new VMs springing up into the copy job, or having to spend tens of hours restructuring their local jobs to make it fit the new v12 logic.
I now need to somehow bust out 40TB of VMs from a 100+TB job, whilst keeping their established 120 day backup chain (and 12x monthly GFS), just so I can put them into a copy job by themselves.

A huge step backwards for us as a VCSP, makes it very very difficult to integrate existing Veeam setups into Cloud Connect now.

[pdxkid]

This issue nailed us as well and it is proving to be a massive problem. To the point our renewal in 118 days is in jeopardy which is very frustrating as I would like to remain in the Veeam family.

We heavily leverage VMware tags across 4 datacenters. These tags control everything from SAN snapshots, reboot schedules, SRM, Veeam backup jobs, and pre-v12 Veeam cloud copy jobs to name a few of the tag driven automations in place.

While Primary Backup Jobs have Backup Copy jobs to create copies of the VM's in remote datacenters, we have a critical VMware Tag used for Cloud Storage. This cherry picks ~50 VM's out of our slew of backup jobs and via a Backup Copy jobs ensures they get into a Cloud Connect bucket.

Since v12 that job does not work due to the lack of Tag support. Having to set a non-cloud Tag on ALL VM's, exclude them from the cloud, and have only non-tagged VM's land in the job is not very feasible, elegant, nor a long term play as my complaints to both Veeam Support and my Veeam Rep have informed me that the exclusion option is short lived and will be removed in future updates.

Removing a feature to rework how backup copy job sucks. It is a big step backwards. I read the reasons in the posts and I don't think they are very valid. Veeam employs a lot of smart people and makes great software. You can figure out how to replace this functionality that we have used for years and years. And while I am on the 'super frustrated with Veeam' train, the communication around this removal was very poor. Veeam is the gold standard for a lot of things but this was a massive misstep that has hurt quite a few customers. I keep hoping Veeam will realize that and put some resources into rectifying this unwanted change so I am posting my experiences and pain here. Thanks for reading.

[Dima P.]

Hello pdxkid,

Thank you for your honest feedback, we will discuss it with the dev team for sure!

Mind me asking why the workaround to use primary jobs with tags AND assign a dedicated backup copy job per source job wont work for your environment? Say:

Tier 1: Primary job 1 (Source: Tag Daily Backup ) > No tailed backup copy
Tier 2: Primary job 2 (Source: Tag Daily Backup & Backup Copy) > Tailed by backup copy job with Primary job 1 added to the source; aimed at HDD repository on-prem
Tier 3: Primary job 3 (Source: Tag Daily Backup & Backup Copy to Cloud Connect ) > Tailed by backup copy job with Primary job 3 added to the source; aimed at VCC repository

When it's required to move the vm between tiers you adjust the tag and all the magic happens automatically. Moreover with v12 new per-vm backup chains & move features you can move VM backup from the Job 1 to Job 3 including the tailed backup copy jobs which eliminates the need of full backup.

[pdxkid]

Thank you Dima, I appreciate it.

Each of our jobs are tailored to types of VMs, criticality, credentials required, retention required, etc. and there isn't a 1:1 mapping of all VMs of job 1, or tier 1, get to go to the cloud. As a financial institution we have lots of corner case policies that govern what can and can not go into the cloud.

[bhrvi]

We got bit by this now, too. I have dozens of legacy backup-copy jobs which source from multiple backups. We moved repositories on some of those backup jobs, which required an active full, which upgraded backup chains, which broke my backup copies. I didn't quite see that sequence of events coming. Has there been any movement on bringing back this functionality?

[BackupBytesTim]

I'll admit this doesn't affect me personally as the majority of our customer's jobs are configured on an individual basis (mostly small business with little to no compliance requirements so they just sort of pick policies based on convenient times and how much storage they want to pay for), but I've been following this and some other issues for a while now and I am curious, the only legitimate sounding explanation I think I've seen anyone at Veeam give was essentially "we didn't get this feature added before the scheduled release for V12, so we released V12 without it". I'll admit, I don't really like that as an explanation, seems it would've made more sense to just not release V12, or take out the new "backup engine" entirely so the old system was all still in place (wouldn't see Microsoft release a new Windows update with the Search feature and be like "ran out of time, oh well"). So, while I understand the explanation and how that does make it not the dev team's fault, it sounds like it's just management's fault then for demanding a version be sent out knowing it was buggy/broken.

That said, I am more curious why, after what's it been, 6 or 7 months now since the V12 release, seems we're no closer to getting this back in, surely if it was originally meant to be in V12 and simply not added because you "ran out of time" then surely we won't be waiting until V13 to get it back?

[Gostev]

If it was a simple feature, then this would be correct: we would add it back quickly, perhaps in one of the cumulative patches even. However, the very reason why it was not added to V12 is because of its high complexity and major risks of data loss bugs explained above. Because of this complexity and particularly high QA costs, it may take significant time before this feature is brought back to the new Backup Copy architecture.

[JaySt]

just challenging that statement:
why not go for providing optionality between legacy and new backup copy architecture/chain format? That way, the new backup copy architecture can still go forward without infrastructure objects and customers heavily leaning on this ability can agree on limitations legacy chains and copy job achitecture will bring.
I would not heavily argue that this functionality should be brought to the new copy job architecture, but i would argue for a better way of phasing this functionality out.