Veeam and ExaGrid

[Daveyd]

Anyone?

[Gostev]

Yes.

[Daveyd]

Yes.

I assume that is the answer to my question about the high dedupe and compression rations being worse than lower ones? What about my question of why forward incrementals with synthetic fulls are recommended over reversed incrementals when backing up to Exagrid?

[Vitaliy S.]

What about my question of why forward incrementals with synthetic fulls are recommended over reversed incrementals when backing up to Exagrid?

It doesn't necessarily refer to Exagrid, forward incremental mode is at least 3 times faster than reversed incremental because of 3x less I/O on the target storage.

[Daveyd]

It doesn't necessarily refer to Exagrid, forward incremental mode is at least 3 times faster than reversed incremental because of 3x less I/O on the target storage.

What about when synthetic fulls are performed? Won't that use a lit of I/O on the target device? If I have multiple jobs all doing forward incrementals with weekly synthetic fulls, is it recommended to run all the syntectic fulls on the same day or different days?

[Vitaliy S.]

You're right in thinking that you'll have to pay for it during weekends when synthetic full is performed, but based on the feedback coming from our existing customers, that shouldn't be a big deal.

Provided that you don't have target storage that is capable of very high IOPS, you may want to create synthetic fulls on different time schedule during weekends.

[Daveyd]

Thank you for the info. Do you know what the typical IOPS are for a weekly synthetic full? I know there are different variables involved like the size of the vibs but it would be nice to know average IOPS of a weekly syntehtic full. That way I can compare the IOPS from the Exagrid appliance and see if I may need to run synthetic fulls on different days.

So, synthetic fulls have to be created on a weekly basis? How does that work if I want to keep 30 days of incrementials? You cannot create the synthetic full on the 30th day?

[Vitaliy S.]

That's correct, but you may want to replace synthetic fulls with full runs which can be scheduled to run once a month.

As for the IOPs, it's really hard to tell, as you've correctly stated everyting depends on many variables such as backup size, storage disk performance etc. May be someone from our Community could provide you with these numbers.

Basically, your decision should be based on the time spent on synthetic full creation, if it is accepable, then you should be good to go with the existing job schedule.

[Daveyd]

If I chose to go the 30 day retention period with daily incrementals and 1 monthly full backup to the Exagrid, what would my file structure look like? Would I have 29 vibs and 1 vbk the first month then another 29 vibs and 1 vbk the next month for a total of 58 vibs and 1 vbk? Then the 3rd month the oldest vib would get overwritten and so on? So, I would always have 58 vibs and 2 vbks at any given point.

Or am I way off base here?

[tsightler]

You're way off base!

When using forward incremental, VIB/VBK files are never overwritten, only new ones are created and old ones are deleted when no longer required for the retention policy. Thus in your scenario, when the 30th day passes, all the VBK/VIB files not required will be deleted. What you'll find is that you vary somewhere between 30 and 60 backup files at any given time.

I'd strongly suggest NOT going with 30 day incremental. We did this and found that performance for restores was simply not acceptable when you were on the long end of the incremental chain (for example, if you had a full and 28 incrementals). Also, even a single corrupt incremental and your backup chain is no good. Also, if you do 30 days, with a full only every 30 days, you won't want to use synthetic fulls but rather real full backups as the time to roll up 30 backups into a synthetic full is just too long and synthetic full is not designed for this.

I would suggest that you use the product as it was designed, which would be a synthetic full at least every week (or a real full). Since these synthetic fulls will be highly "dedupable" (I guess that's not a word) it should lead to a good storage ratio, good restore performance, and not too high of a risk that a bad incremental can break your restore chain.

[Daveyd]

Thanks for the reply. I figured I was off base

I do understand about the processing time it would take to rollup all daily incrementals into a monthly synthetic full....Is there even an option to do a monthly synthetic?

So, if I wanted to have a 30 day retention period (be able to restore any file on a VM, or the VM itself any day of a month), the best option would be to set the backup job to 30 restore points, use forward incrementals with weekly synthetic full backups, correct? If so, potentially how many files would I see on the Exagrid appliance? I am trying to gauge how much space I am going to need with a 30 day retention period. We are migrating physical servers to VMs. The physical servers were being backed up by Backup Exec with a 30 day retention policy (daily increments/monthly full)

[Vitaliy S.]

Is there even an option to do a monthly synthetic?

Not really, this option can be configured to run on weekly basis only.

So, if I wanted to have a 30 day retention period (be able to restore any file on a VM, or the VM itself any day of a month), the best option would be to set the backup job to 30 restore points, use forward incrementals with weekly synthetic full backups, correct?

Yes. Having weekly synthetic full backups is definitely a good idea.

If so, potentially how many files would I see on the Exagrid appliance? I am trying to gauge how much space I am going to need with a 30 day retention period. We are migrating physical servers to VMs. The physical servers were being backed up by Backup Exec with a 30 day retention policy (daily increments/monthly full)

If you enable a synthetic full backups, you should see somewhere between 30 and 37-38 restore points. If you want to see more details on how this works, please take a look at this topic.

[tgillispie]

With a 30 day retention period, weekly synthetic (or real) fulls, and daily incrementals, the space consumed on the ExaGrid will be something like:

Cached for best possible vPower/InstantRecovery, AIR, SureBackup, other restores, and synthetic fulls:
- The last full backup – real or synthetic – i.e. your .vbk files – with Veeam deduplication applied
- The last set of daily incrementals – i.e. your .vib files – with Veeam deduplication and vSphere change block tracking applied

Protected in deduplicated form both locally and at any remote ExaGrid site:

- All current and retained full backups – real or synthetic – available for file-level, disk-level, VM-level restores (InstantRestore OK too but expect increased RTO)
- All daily incrementals

Indeed, since the scheduling of Veeam synthetic full backups can be done on any day of the week, or no days of the week, the typical pattern is to have a full backup (synthetic or real) on which to build the next week’s worth of (forward) incremental backups, thereby leveraging ExaGrid’s caching of the “most recent backups”.

Hope this helps.

[Daveyd]

With a 30 day retention period, weekly synthetic (or real) fulls, and daily incrementals, the space consumed on the ExaGrid will be something like:

Cached for best possible vPower/InstantRecovery, AIR, SureBackup, other restores, and synthetic fulls:
- The last full backup – real or synthetic – i.e. your .vbk files – with Veeam deduplication applied
- The last set of daily incrementals – i.e. your .vib files – with Veeam deduplication and vSphere change block tracking applied

Protected in deduplicated form both locally and at any remote ExaGrid site:

- All current and retained full backups – real or synthetic – available for file-level, disk-level, VM-level restores (InstantRestore OK too but expect increased RTO)
- All daily incrementals

Indeed, since the scheduling of Veeam synthetic full backups can be done on any day of the week, or no days of the week, the typical pattern is to have a full backup (synthetic or real) on which to build the next week’s worth of (forward) incremental backups, thereby leveraging ExaGrid’s caching of the “most recent backups”.

Hope this helps.

Awesome info...thanks!

When you say Exagrid's caching of the Veeam's backups, are you referring to the entire set of Veeam backup files, lets say 1 week, 6 vib and 1 vbk, sitting in the Exagrid landing zone, which basically only has Veeam's dedupe processing applied and has not yet gone through Exagrid's dedupe and compression?

[romwarrior]

So what is the general thinking regarding doing weekly synthetic fulls vs real fulls as it relates to Exagrid? I've seen the recommendation to do synthetics but there doesn't seem to be much of a time benefit. It also seems to me that if you only ever rely on the incrementals to create your fulls there is a greater possibility of corruption seeping into the file chain. With a real full you start off with a clean slate every week. I recognize the benefits of offloading some of the I/O from the VM's but I'm having a hard time justifying synthetics for fear of corrupted incremental chains.

[Gostev]

I'm having a hard time justifying synthetics for fear of corrupted incremental chains.

You could use SureBackup to overcome your fear there are so many things that can go wrong with backup (no matter what software you use), corrupted incremental chain is actually one of the least likely issues... don't be afraid of a rabbit, as there is a lion behind. Just test your backups, manually or automatically. This will help you neutralize both rabbit, and lion.

Besides, note that our product did not even have real fulls option until the latest versions, with forever-incremental being the only option. Plus, we did not invent the whole approach (one of those few things we did not invent hehe). Many big backup vendors are using it. So more generally speaking, it is quite proven both in our product, and overall as a concept.

That said, my answer above is storage-agnostic. While for ExaGrid or other dedupe targets, because of their specifics, you may find more reasons to use active fulls over synthetic fulls. So, this is totally up to you.

Thanks.

[Daveyd]

I was talking to an Exagrid tech about best practices for synthetic backups to an Exagrid device and could not get a real answer. Synthetic backups are pretty IO intensive and I found out if I run 3-4 simultaneous synthetic fulls, I can bring the Exagrid appliance to its knees

[Beevoir]

My own belief on this (having done the majority of the UK testing with the HP D2D appliances) is that the problem is one of the ASICs within the deduplicating appliances themselves. If you imagine the primary function of the Dedupe appliance, it is to accept streaming, sequential reads and then work their way through them deduplicating, compressing and performing whatever other functions the box is able/required to do. The same is true of restore, they are designed to be streamed sequentially back to a target, or offloaded somewhere else etc.

This means that they are great at handling forward incremental because it is all new data and all sequentially written.

When you start to look at synthetic fulls (think of it as a building/overwriting process of existing data) and Reverse Incremental then you start to encounter random read and write requests from the appliance, which is something that confuses them, as it was they were not really designed to provide this function and this is the basis of our reverse/synthetic full process.

So, definitely work on the principles that Vit/Anton provided in terms of synthetic full time acting as the limiting factor in the backup job schedule and also make full use of Surebackup.

Cheers

[jpaul]

I was talking to an Exagrid tech about best practices for synthetic backups to an Exagrid device and could not get a real answer. Synthetic backups are pretty IO intensive and I found out if I run 3-4 simultaneous synthetic fulls, I can bring the Exagrid appliance to its knees

If the Exagrid appliance is sized properly there should be no issues with Synthetic Full backups. But here is the problem... what they consider sized properly is not enough for Veeam backups.

For Veeam Synthetic full backups on the Exagrid to work well you would need to keep an entire week (assuming your last "full" was 7 days ago) in an undeduped form. So that full backup from last week and then some of the data from each incremental from every night after that full backup .... because to build a new full... you will need parts of each of those files.

SO if the appliance isnt that big what it has to do is undedupe those files on the fly... which slows the synthetic full WAY DOWN.

I worked with a couple of the engineers at exagrid when i was doing my Blog series on the Exagrid with Veeam, and they were kinda struck by the extra size requirements you would need for synthetic fulls.

One work around that I did find is that if you do a synthetic full every Friday and then again on like tuesday... you only have to read files from half the number of days as you would with a 6 days between synthetic fulls.

The other option is to disable fulls and and fun straight full backups.

One of the other issues you will run into if the appliance isn't large enough is that when Veeam un-deduplicates those older files to help build its new synthetic full... it tears up landing pad space in some cases... which reduces the total size you ahve for new backups.... sooo moral of the story is BUY BIG.

Check out my posts for more info http://jpaul.me/?p=1618

[Daveyd]

If the Exagrid appliance is sized properly there should be no issues with Synthetic Full backups. But here is the problem... what they consider sized properly is not enough for Veeam backups.

For Veeam Synthetic full backups on the Exagrid to work well you would need to keep an entire week (assuming your last "full" was 7 days ago) in an undeduped form. So that full backup from last week and then some of the data from each incremental from every night after that full backup .... because to build a new full... you will need parts of each of those files.

SO if the appliance isnt that big what it has to do is undedupe those files on the fly... which slows the synthetic full WAY DOWN.

I worked with a couple of the engineers at exagrid when i was doing my Blog series on the Exagrid with Veeam, and they were kinda struck by the extra size requirements you would need for synthetic fulls.

One work around that I did find is that if you do a synthetic full every Friday and then again on like tuesday... you only have to read files from half the number of days as you would with a 6 days between synthetic fulls.

The other option is to disable fulls and and fun straight full backups.

One of the other issues you will run into if the appliance isn't large enough is that when Veeam un-deduplicates those older files to help build its new synthetic full... it tears up landing pad space in some cases... which reduces the total size you ahve for new backups.... sooo moral of the story is BUY BIG.

Check out my posts for more info http://jpaul.me/?p=1618

Thanks for the info and link

We have a 5TB (5TB landing 5TB retention) Exagrid appliance and are backing up physical servers to it using BackupExec and VM using Veeam. Apparently we really undersized the appliance as we only have 30% available in our landing space with no retention space available. When talking to Exagrid, the rep said they size Veeam according to the actual sum of all vmdks on all VMs. So, in our case we are backing up 70 VM with a total vmdk size of 10TB. Although I do question the validity of that since Veeam dedupes before sending to Exagrid. But whatever. We just received a new 10TB appliance (10TB landing, 10TB retention) to add to the current 5TB "Grid" (try to do that with DataDomain ) just for Veeam backups. Seems a little overkill to me but we should now have the space for Synthetic backups....even though non Synthetic fulls work really well.

[mcrespi]

My name is Marc Crespi, VP of Product Management from ExaGrid. I wanted to reply here to let everyone know that ExaGrid architected our system to perfom well with Veeam 5's default setting of synthetic fulls. We now have close to 100 customers (and rising daily) using the product for that purpose and to date, the vast majority are seeing good performance. If for any reason, anyone sees performance that does not meet their expectations we will resolve the situation aggressively. It is true that the system does need to be sized properly, which is really true of any deduplicating appliance.

I am also happy to speak to any of our joint customers personally. My e-mail is mcrespi@exagrid.com if I can be of any help or if you have any suggestions for our product's feature set.

Sincerely,
Marc Crespi

[ddockter]

[merged]

What is the best "Storage Optimize For" value when using an Exagrid as the backup target? I've seen two documents on the subject. One says to use "LAN target" and the other says to use "local target".

[Gostev]

Generally speaking, this setting is not storage specific, and typically should be set depending on how the storage is connected to Veeam Backup server - whether it is locally attached, or accessible over LAN or WAN.

[mooreka777]

I agree with the d2d 4324. My eval of the unit wasn't very impressive. The AD integration was OK, the number of streams/inject speed was OK, and for the life of me I could NOT get even close to 1G (400-550 TOPS) of throughput without using the 10G interface. I finish up my testing soon, but for simplicity and quick growth the ExaGrid is very impressive.

I still have yet to test the offsite replication and DR restore speeds. That is for a later date as we are waiting for the 1GB links to come in between two of our sites for replication.

[Daveyd]

[merged]

I am looking to getting the fastest throughput rate from my physical Veeam server to my Exagrid appliance. Adding a 10Gb NIC to each is out of the question for now.

My Veeam server has a FC HBA attached to a Clariion CX4 @4Gbps. The Veeam server also has 2 physical 1Gb NICs teamed together using HP's teaming software for backup traffic to the Exagrid appliance. The Team is set to use Transmit Load Balancing with Fault Tolerance (TLB) and Windows reports a 2Gb connection speed. The Exagrid has 2, 1GB NICs each with separate IP addresses.

All the Veeam jobs use the SAN method for backups. Half the jobs are setup to use 1 Exagrid IP address, the other half use the 2nd Exagrid IP address. A lot of the jobs run concurrently. I want Veeam to fully utilize the 2Gb NIC Team during backups and make the backups as fast as possible. Is this the best way to accomplish this?

[Gostev]

Merged to have everything about ExaGrid in one place, plus I believe ExaGrid team is monitoring this topic.

[Daveyd]

From a Veeam perspective, is this an optimal setup? On the Veeam server, since I have a 4Gb FC connection to the Clariion CX4 and a Teamed 2Gb (2x1Gb NICs) to the Exagrid target device, will Veeam fully utilize the Teamed 2Gb connection if I run multiple jobs at once going to different IP addresses on the Exagrid?

[Gostev]

Sure, why not. The setup looks good if your Veeam server is powerful enough.

[Daveyd]

Sure, why not. The setup looks good if your Veeam server is powerful enough.

2x Quad Core Hyperthreaded CPUs with 8GB of RAM.
I am monitoring the the NIC Team interface on the Veeam server to see my actual throughput during the backups. Hopefully Veeam can fully utilize the link(s)

[Gostev]

We are working on much highler level of network stack (on top of OS) and thus don't deal directly with any links, teams, NICs or HBAs. If OS will route the traffic through both links, Veeam will effectively "utilize" both as well.