Veeam + Exagrid - Very worried

Availability for the Always-On Enterprise

Veeam + Exagrid - Very worried

Veeam Logoby zuldan » Thu Mar 02, 2017 5:44 am

Veeam B&R: v9.5 U1
Exagrid: EX13000E v4.8.0.216
Case: 02082443


The case was opened because backups for a Backup Job randomly went missing under Disks -> Backups.

Opening up the status for the job the following warning showed.

Image

The Veeam support person soon realized that our Backup Copy jobs were going to the exact same location (Exagrid share) to their relevant Backup Jobs and was concerned. I said it was configured this way because deduplication on a Exagrid is done by share and because KB2056 recommends it.

Image

Our repository configuration

Image

The Veeam support person did some research and found there was a known major bug between Veeam and Exagrids where backup files would lose their association with a backup job. We then looked at Backup -> Backup (imported) and found it completely flooded with backup jobs (even ones that had run the night before).

He then installed a Veeam patch (file replacement), Veeam.Backup.Core.dll (MD5 1C0BD7AD1153151B1386F8C7C6F2E976).

The backup jobs that ran last night have gone directly to Disk (imported) (as they were doing before), however, this time the backup is also showing in the normal backup area? So I guess the patch made some sort of difference.

Image

The Veeam support person said they think we haven't lost any data however we could not restore the data in the normal way, we have to manually copy the backup files out of the Exagrid repository which is not a great situation if you were planning to use Instant VM Recovery. The boss isn't happy as we have $100,000 in Veeam licensing / support.

I believe the Veeam support person is doing everything possible to help and I have no complaints, I just thought I'd post it here to get some feedback from engineers (any ideas as to what the issue may be?) as I'm stressing about the accessibility of our backup data. Very worried.
zuldan
Enthusiast
 
Posts: 45
Liked: 5 times
Joined: Wed Feb 15, 2017 9:51 am

Re: Veeam + Exagrid - Very worried

Veeam Logoby coopsam » Thu Mar 02, 2017 9:13 am

We have exagrids for our veeam backups. Not tried what your doing but I can say that you'll find instant VM restore on Exagrid leaves you with almost unusable VM you do need to fully recover them to your primary storage in order to get them into a workable state. The Exagrids were never designed to function as primary storage for virtual machines.
coopsam
Influencer
 
Posts: 21
Liked: never
Joined: Wed Apr 30, 2014 11:48 am
Full Name: Sam Cooper

Re: Veeam + Exagrid - Very worried

Veeam Logoby foggy » Thu Mar 02, 2017 11:08 am

Basically, the requirement is not to have multiple repositories pointing to exactly the same location, since this can result in multiple issues, including the observed behavior (due to the fact that the same chains are visible on different repos). It is recommended to point the repositories to different subfolders of the same share.
foggy
Veeam Software
 
Posts: 15078
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam + Exagrid - Very worried

Veeam Logoby foggy » Thu Mar 02, 2017 11:09 am 1 person likes this post

coopsam wrote:The Exagrids were never designed to function as primary storage for virtual machines.

Unless you're starting IR from the backup that is still in the landing zone.
foggy
Veeam Software
 
Posts: 15078
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam + Exagrid - Very worried

Veeam Logoby tgillispie » Thu Mar 02, 2017 4:55 pm 3 people like this post

zuldan - as the Director of Engineering here at ExaGrid responsible for interop between ExaGrid's solution and Veeam, I wanted to update you on a few things.
1. We have been working closely with Veeam's support and development organization to fully understand what you are seeing in your previous posts. BTW - thanks for all the details in your posts
2. There are some changes in Veeam B&R v9.5 that are likely to drive a change in the best practices - covered in the KB article you reference - stay tuned.
3. We have confirmed with Veeam (thanks foggy) that if the path to the backups is different from the path to the copy job, all is well. Case (UC vs LC) does not count, but the default behavior of Veeam creating a different folder for each job name and hence a different path existing for backups vs copy jobs should significantly reduce this condition. And, like I said, we may be updating best practices to include adding a "subfolder" for the backup copy job target path.

We noticed you have decided to drop ExaGrid maintenance and support. You will receive updates from Veeam on this issue; while ExaGrid does not anticipate any updates to ExaGrid software related to this issue, but if there are, you should get back in contact with ExaGrid.

Regarding previous comments about Veeam InstantRecovery performance, foggy's comments are right on target. All of our mutual ExaGrid and Veeam customers report industry-leading Instant VM Recovery performance when booting (or SureBackup or Virtual Lab) from the most recent backup or backups from a few days ago - which is what you need 90+% of the time. In this case, ExaGrid is not used as primary storage since Veeam vPower keeps changes on some other (configurable, hopefully flash) datastore, and the booted VM is migrated to production storage, not the ExaGrid.

vPower operations on older backups do trigger rehydration, and won't be as fast as recent backups, but are certainly viable. Finally, recovery of a VM from a remote/DR ExaGrid site requires a full VM restore because the remote/DR site often has no Landing Zone and so populating the Landing Zone during restores of older backups is not done, leading to restricted Instant VM Recovery performance at the remote/DR site.

Hope this helps; let me know via forum reply or private messages if can be of any assistance.
tgillispie
Technology Partner
 
Posts: 11
Liked: 5 times
Joined: Tue Nov 16, 2010 10:52 pm
Full Name: Tom Gillipsie

Re: Veeam + Exagrid - Very worried

Veeam Logoby zuldan » Fri Mar 03, 2017 9:04 am 1 person likes this post

@foggy, I understand what the problem is now and yes I agree the destination of the Backup Job and Backup Copy Job should be going to the same share but different folders. Unfortunately the KB made no mention of this. The Exagrid admin console does not allow the creation of subfolders so I'm guessing this would have to be done via Veeam. I'll wait for the final solution from the Veeam support person before making any changes to folder structures. Today I disabled all Backup Copy Jobs today to prevent any further destruction of Backup Job association.

@tgillispie, it's great to hear Veeam has such a close relationship with Exagrid. If we were in the market for another deduplication platform, Exagrid would be our first choice. I'm looking forward to the updated KB article with best practices. Regarding VM Instant Recovery, we haven't experienced any performance issues.

On a final note, thank you both foggy and tgillispie for your posts. Your interaction with customers is wonderful and the support is appreciated.
zuldan
Enthusiast
 
Posts: 45
Liked: 5 times
Joined: Wed Feb 15, 2017 9:51 am

Re: Veeam + Exagrid - Very worried

Veeam Logoby Delo123 » Fri Mar 03, 2017 9:30 am

Am i reading it correctly that backup job and backup copy jobs go to the same share on these things? (aka same underlying storage etc...) so it can de deduped for better dedupe ratio?
Why do backup copies at all if you are placing them on the same set of disks?
Delo123
Expert
 
Posts: 351
Liked: 101 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Veeam + Exagrid - Very worried

Veeam Logoby foggy » Fri Mar 03, 2017 11:27 am

zuldan wrote:The Exagrid admin console does not allow the creation of subfolders so I'm guessing this would have to be done via Veeam.

Correct, you just specify the required path in the backup repository wizard and the folder will be created.
foggy
Veeam Software
 
Posts: 15078
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam + Exagrid - Very worried

Veeam Logoby tgillispie » Fri Mar 03, 2017 2:10 pm 1 person likes this post

@zuldan - thanks for the feedback.

@Delo123 - Disk-based deduplication really pays off when your retention is 4 weeks or longer. ExaGrid customers typically retain backups for many months, if not years. The way Veeam accomplishes extended retention is via Backup Copy jobs - where Veeam nicely manages the "GFS" retention of weekly, monthly, quarterly, yearly - so Veeam BCJs are very common at ExaGrid customers; indeed, targetting the backup and backup copy to the same share maximizes deduplication performance.
tgillispie
Technology Partner
 
Posts: 11
Liked: 5 times
Joined: Tue Nov 16, 2010 10:52 pm
Full Name: Tom Gillipsie

Re: Veeam + Exagrid - Very worried

Veeam Logoby foggy » Fri Mar 03, 2017 3:03 pm

I believe Guido's concern here is that all eggs go into the same basket in this setup, which is not inline with the 3-2-1 rule. But I also believe that customers implementing such scenario also have some sort of a secondary storage.
foggy
Veeam Software
 
Posts: 15078
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam + Exagrid - Very worried

Veeam Logoby mkaec » Sun Mar 05, 2017 10:26 pm

Delo123 wrote:Am i reading it correctly that backup job and backup copy jobs go to the same share on these things? (aka same underlying storage etc...) so it can de deduped for better dedupe ratio?
Why do backup copies at all if you are placing them on the same set of disks?


Because there is no way to do GFS retention in a primary backup job.
mkaec
Expert
 
Posts: 185
Liked: 48 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Veeam + Exagrid - Very worried

Veeam Logoby mkaec » Sun Mar 05, 2017 10:31 pm

foggy wrote:I believe Guido's concern here is that all eggs go into the same basket in this setup, which is not inline with the 3-2-1 rule. But I also believe that customers implementing such scenario also have some sort of a secondary storage.

The standard ExaGrid setup involves two appliances that replicate between one another with one being off site. I believe this satisfies 3-2-1.
mkaec
Expert
 
Posts: 185
Liked: 48 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Veeam + Exagrid - Very worried

Veeam Logoby Gostev » Sun Mar 05, 2017 11:16 pm 2 people like this post

mkaec wrote:I believe this satisfies 3-2-1.

No, actually. While a common misperception, this infrastructure does not meet the "2" part (two different medias), and is a fairly common reason for irreversible data loss that we're seeing in our support. Anything bad happening to your backup (deletion or corruption) will be instantly and efficiently propagated to a copy ;)

Two storage devices in storage-based replication effectively makes it the single media, just dispersed (geo-dispersed in your case). So, effectively, you're doing 3-1-½. Ideally, you need to add an offline copy - but at the very least, a Veeam Backup Copy job to a separate storage - this will break that synchronization loop.
Gostev
Veeam Software
 
Posts: 21503
Liked: 2379 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Veeam + Exagrid - Very worried

Veeam Logoby Mike Resseler » Mon Mar 06, 2017 6:01 am 1 person likes this post

To add to Anton's comment. There are also quite a few people (such as myself) that consider 2 storage devices, from the same vendor cannot be considered as 2 different media. Even if you are not using replication (or something). So even in the case of one <favoritestoragehere> and a BCJ to <favoritestoragehere> is not seen as 2 different media. I had the unfortunate event a few years ago that a bug in a storage device was slowly (took too long to detect) corrupt my backups. The other device of the same vendor had a different firmware (I know, we forgot to update it...), but the bug was present in that older one also. By the time we detected it on the first device, and went looking at the second device, we realized that we got screwed big time (Luckily I still had tapes ;-))

So sometimes it is worth having your primary and secondary storage to be different vendors also :-)
Mike Resseler
Veeam Software
 
Posts: 3340
Liked: 379 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Veeam + Exagrid - Very worried

Veeam Logoby mkaec » Mon Mar 06, 2017 2:59 pm

That's a good point. I think 3-1-½ is unfair, though. The 1 is supposed to protect from site disaster. If an earthquake swallows up your primary site, that's not going to propagate to the DR site. I think swapping out the ExaGrid replication for a Veeam BCJ might be enough for a true "2" unless you subscribe to Mike's two different vendors rule.
mkaec
Expert
 
Posts: 185
Liked: 48 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: amannella, lukejf and 43 guests