Veeam + Exagrid - Very worried

[zuldan]

Veeam B&R: v9.5 U1
Exagrid: EX13000E v4.8.0.216
Case: 02082443

The case was opened because backups for a Backup Job randomly went missing under Disks -> Backups.

Opening up the status for the job the following warning showed.



The Veeam support person soon realized that our Backup Copy jobs were going to the exact same location (Exagrid share) to their relevant Backup Jobs and was concerned. I said it was configured this way because deduplication on a Exagrid is done by share and because KB2056 recommends it.



Our repository configuration



The Veeam support person did some research and found there was a known major bug between Veeam and Exagrids where backup files would lose their association with a backup job. We then looked at Backup -> Backup (imported) and found it completely flooded with backup jobs (even ones that had run the night before).

He then installed a Veeam patch (file replacement), Veeam.Backup.Core.dll (MD5 1C0BD7AD1153151B1386F8C7C6F2E976).

The backup jobs that ran last night have gone directly to Disk (imported) (as they were doing before), however, this time the backup is also showing in the normal backup area? So I guess the patch made some sort of difference.



The Veeam support person said they think we haven't lost any data however we could not restore the data in the normal way, we have to manually copy the backup files out of the Exagrid repository which is not a great situation if you were planning to use Instant VM Recovery. The boss isn't happy as we have $100,000 in Veeam licensing / support.

I believe the Veeam support person is doing everything possible to help and I have no complaints, I just thought I'd post it here to get some feedback from engineers (any ideas as to what the issue may be?) as I'm stressing about the accessibility of our backup data. Very worried.

[coopsam]

We have exagrids for our veeam backups. Not tried what your doing but I can say that you'll find instant VM restore on Exagrid leaves you with almost unusable VM you do need to fully recover them to your primary storage in order to get them into a workable state. The Exagrids were never designed to function as primary storage for virtual machines.

[foggy]

Basically, the requirement is not to have multiple repositories pointing to exactly the same location, since this can result in multiple issues, including the observed behavior (due to the fact that the same chains are visible on different repos). It is recommended to point the repositories to different subfolders of the same share.

[foggy]

The Exagrids were never designed to function as primary storage for virtual machines.

Unless you're starting IR from the backup that is still in the landing zone.

[tgillispie]

zuldan - as the Director of Engineering here at ExaGrid responsible for interop between ExaGrid's solution and Veeam, I wanted to update you on a few things.
1. We have been working closely with Veeam's support and development organization to fully understand what you are seeing in your previous posts. BTW - thanks for all the details in your posts
2. There are some changes in Veeam B&R v9.5 that are likely to drive a change in the best practices - covered in the KB article you reference - stay tuned.
3. We have confirmed with Veeam (thanks foggy) that if the path to the backups is different from the path to the copy job, all is well. Case (UC vs LC) does not count, but the default behavior of Veeam creating a different folder for each job name and hence a different path existing for backups vs copy jobs should significantly reduce this condition. And, like I said, we may be updating best practices to include adding a "subfolder" for the backup copy job target path.

We noticed you have decided to drop ExaGrid maintenance and support. You will receive updates from Veeam on this issue; while ExaGrid does not anticipate any updates to ExaGrid software related to this issue, but if there are, you should get back in contact with ExaGrid.

Regarding previous comments about Veeam InstantRecovery performance, foggy's comments are right on target. All of our mutual ExaGrid and Veeam customers report industry-leading Instant VM Recovery performance when booting (or SureBackup or Virtual Lab) from the most recent backup or backups from a few days ago - which is what you need 90+% of the time. In this case, ExaGrid is not used as primary storage since Veeam vPower keeps changes on some other (configurable, hopefully flash) datastore, and the booted VM is migrated to production storage, not the ExaGrid.

vPower operations on older backups do trigger rehydration, and won't be as fast as recent backups, but are certainly viable. Finally, recovery of a VM from a remote/DR ExaGrid site requires a full VM restore because the remote/DR site often has no Landing Zone and so populating the Landing Zone during restores of older backups is not done, leading to restricted Instant VM Recovery performance at the remote/DR site.

Hope this helps; let me know via forum reply or private messages if can be of any assistance.

[zuldan]

@foggy, I understand what the problem is now and yes I agree the destination of the Backup Job and Backup Copy Job should be going to the same share but different folders. Unfortunately the KB made no mention of this. The Exagrid admin console does not allow the creation of subfolders so I'm guessing this would have to be done via Veeam. I'll wait for the final solution from the Veeam support person before making any changes to folder structures. Today I disabled all Backup Copy Jobs today to prevent any further destruction of Backup Job association.

@tgillispie, it's great to hear Veeam has such a close relationship with Exagrid. If we were in the market for another deduplication platform, Exagrid would be our first choice. I'm looking forward to the updated KB article with best practices. Regarding VM Instant Recovery, we haven't experienced any performance issues.

On a final note, thank you both foggy and tgillispie for your posts. Your interaction with customers is wonderful and the support is appreciated.

[Delo123]

Am i reading it correctly that backup job and backup copy jobs go to the same share on these things? (aka same underlying storage etc...) so it can de deduped for better dedupe ratio?
Why do backup copies at all if you are placing them on the same set of disks?

[foggy]

The Exagrid admin console does not allow the creation of subfolders so I'm guessing this would have to be done via Veeam.

Correct, you just specify the required path in the backup repository wizard and the folder will be created.

[tgillispie]

@zuldan - thanks for the feedback.

@Delo123 - Disk-based deduplication really pays off when your retention is 4 weeks or longer. ExaGrid customers typically retain backups for many months, if not years. The way Veeam accomplishes extended retention is via Backup Copy jobs - where Veeam nicely manages the "GFS" retention of weekly, monthly, quarterly, yearly - so Veeam BCJs are very common at ExaGrid customers; indeed, targetting the backup and backup copy to the same share maximizes deduplication performance.

[foggy]

I believe Guido's concern here is that all eggs go into the same basket in this setup, which is not inline with the 3-2-1 rule. But I also believe that customers implementing such scenario also have some sort of a secondary storage.

[mkaec]

Am i reading it correctly that backup job and backup copy jobs go to the same share on these things? (aka same underlying storage etc...) so it can de deduped for better dedupe ratio?
Why do backup copies at all if you are placing them on the same set of disks?

Because there is no way to do GFS retention in a primary backup job.

[mkaec]

I believe Guido's concern here is that all eggs go into the same basket in this setup, which is not inline with the 3-2-1 rule. But I also believe that customers implementing such scenario also have some sort of a secondary storage.

The standard ExaGrid setup involves two appliances that replicate between one another with one being off site. I believe this satisfies 3-2-1.

[Gostev]

I believe this satisfies 3-2-1.

No, actually. While a common misperception, this infrastructure does not meet the "2" part (two different medias), and is a fairly common reason for irreversible data loss that we're seeing in our support. Anything bad happening to your backup (deletion or corruption) will be instantly and efficiently propagated to a copy

Two storage devices in storage-based replication effectively makes it the single media, just dispersed (geo-dispersed in your case). So, effectively, you're doing 3-1-½. Ideally, you need to add an offline copy - but at the very least, a Veeam Backup Copy job to a separate storage - this will break that synchronization loop.

[Mike Resseler]

To add to Anton's comment. There are also quite a few people (such as myself) that consider 2 storage devices, from the same vendor cannot be considered as 2 different media. Even if you are not using replication (or something). So even in the case of one <favoritestoragehere> and a BCJ to <favoritestoragehere> is not seen as 2 different media. I had the unfortunate event a few years ago that a bug in a storage device was slowly (took too long to detect) corrupt my backups. The other device of the same vendor had a different firmware (I know, we forgot to update it...), but the bug was present in that older one also. By the time we detected it on the first device, and went looking at the second device, we realized that we got screwed big time (Luckily I still had tapes )

So sometimes it is worth having your primary and secondary storage to be different vendors also

[mkaec]

That's a good point. I think 3-1-½ is unfair, though. The 1 is supposed to protect from site disaster. If an earthquake swallows up your primary site, that's not going to propagate to the DR site. I think swapping out the ExaGrid replication for a Veeam BCJ might be enough for a true "2" unless you subscribe to Mike's two different vendors rule.

[tgillispie]

Its great that adherence to the 3-2-1 rule is a topic for discussion.

@Gostev's insights are (as usual) spot on. Storage from a single vendor/ecosystem is fundamentally "single media".

In the ExaGrid case, I wanted to point out that each ExaGrid appliance's internal storage is logically divided into two areas that are for different purposes. One area is the landing zone, where Veeam backups land unimpeded and without any inline/deduplication processing. The other area holds the deduplicated backups. The most recent Veeam backups will remain in this landing zone/cache until the next week's fulls come along. While they share the underlying physical RAID6 storage, the lifecycle of backups is very different in the two areas. I don't claim the two areas count as "two media", satisfying the 3-*2*-1 rule, but when considering a deduplication appliance for Veeam backup storage, the appliance's architecture needs to be considered in light of the 3-2-1 rule.

[mkaec]

Maybe that's 3-1½-1.

[ViperDaSnake]

zuldan - as the Director of Engineering here at ExaGrid responsible for interop between ExaGrid's solution and Veeam, I wanted to update you on a few things.
2. There are some changes in Veeam B&R v9.5 that are likely to drive a change in the best practices - covered in the KB article you reference - stay tuned.

Came here for this...
We deployed in 2014 and we now need to reconfigure our backups to take advantage of the many changes since then.
We have 4 different sites and each has varying levels of implementation.
It's time to blow them away and rebuild but not sure what that looks like yet?
That KB doesn't seem to exist now. The link doesn't work and I am unable to search for it.
Hopefully that means it is being rewritten for 9.5...?

[Mike Resseler]

Hi,

I'll talk to support to see what is happening with the KB. It might be indeed that it is being rewritten

Stay tuned
Mike

[zuldan]

Hi all, sorry, just bringing the conversation back to the original post.

I thought the final solution might involve configuring repositories to point to sub-folders, however this (below) is the what the Veeam patch does according to Veeam support.

The hotfix doesn't change any folder names. The problem appears because in some certain circumstances Veeam may perform metadata re-sync and as a result re-create .vbm file. Hotfix contains improved logic which should avoid such situations for newly created jobs, however, it' won't fix already affected backups.

So is the problem because we're pointing both Backup Job and Backup Copy Job to the same folder or is it a bug within Veeam? If someone could explain exactly what the root cause is that would at least relieve some of the frustration.

[foggy]

As I've mentioned in my first post above, the general requirement is not to have multiple repositories pointing to exactly the same location, since this can result in multiple issues, including the observed behavior. You've seen one of the issues caused by how metadata handling is currently performed (which is fixed by the hotfix you obtained), but other issues inclined by such configuration are possible as well. So ideally, you should avoid repositories pointing to the same location.

[mkaec]

I definitely recommend creating sub-folders if you are going to point multiple repositories to the same ExaGrid share. Assuming your shares are configured as "Veeam Backup & Replicaiton" type with a transport of "ExaGrid-Veeam Accelerated Data Mover", you'll be able to create the sub-folder in Veeam.

1. Click the FILES tab in the left pane.
2. In the upper portion of the left pane, navigate to Linux > [ExaGrid Server] > home > vveam > [share name].
3. Right click and select "New Folder" from the popup menu.

Then you can go into the repository and set the "Path to folder" to "home/veeam/[share name>]<folder name>". Obviously, the best time to do this is when setting up a new repository. But, I believe you would be able to move the existing files into the new folder. Though, that does get a little more complicated as you've got to let Veeam know that you did this.

[zuldan]

As I've mentioned in my first post above, the general requirement is not to have multiple repositories pointing to exactly the same location

Now that I realize is going on, I totally agree with you.

since this can result in multiple issues, including the observed behavior. You've seen one of the issues caused by how metadata handling is currently performed (which is fixed by the hotfix you obtained)

Ok understood, jobs pointing the exact same share location has caused issues with the metadata which has caused some sort of corruption with my backup job association with files. I wish the KB mentioned creating sub-folders. Come to think of it, I'm regretting finding that KB article and just leaving the Backup Jobs to 20 restore points.

So ideally, you should avoid repositories pointing to the same location.

I would like to point out that the current and previous support person working on the case has not once said anything about pointing the backup jobs to different sub-folders (feel free to scan the case notes). The only thing that has occurred is a patch was provided that has now made everything worse even though Backup Copy Jobs are disabled. I have started including my account manager in the communication going back and forth. I'm sure you can understand my frustration.

Looking forward to sleeping better each night when this is all over and Backup Jobs are back to normal!

@mkaec, thank you for the clear instructions. As soon as Veeam support make this recommendation I will make the changes to the Backup Jobs.

[foggy]

Your frustration is explainable. I will make sure support people are aware of the corrected recommendations.

[jessethomas]

Our ExaGrid support representative made us aware of a Veeam fix for this issue:

https://www.veeam.com/kb2249

[applc]

Thank you for posting this! Do you know if this hotfix overwrites any files in Update 1 or vice versa?

[jessethomas]

appic,

There is no installer, you just stop services and manually replace a single DLL file (Veeam.Backup.Core.dll), which is newer than the version provided in Update 1.

[mkaec]

I still would recommend setting up the repositories in separate sub-folders on the share. Co-mingling the files just doesn't sit well with me.

[ChrisSnell]

I have to agree. It's better house-keeping, if nothing else!

[DerOest]

Hello everyone,
i just skimmed over this thread, and i think the core problem is also to relevant to Netapp users.

Our Veeam Repository is a CIFS share on a Netapp FASxxx. Netapp does Compression/Deduplication on a per-volume-basis.
So to use GFS, i need to set up a volume with 2 separate CIFS shares added as 2 different repositories.

Yes, it works, but it's extra work. Would be nice if it would just be possible to have GFS on the same backup repository.