v6 Beta?

[averylarry]

I'm finding v5 replication across a WAN using ESXi -- let's just say "frustrating". If I use the size of the snapshots as a rough indication of how much changed data there is to replicate, then there's a whole lot more traffic going on than just changed blocks.

But anyway -- I've only been a Veeam customer for a few months and I was wondering if there will be public beta releases of v6? Or if not then could I get on a list of private beta testers? I'm pretty small, though. 3Tb of data, 12 VMs, 10Mb/s WAN link.

[Gostev]

Hi Ted, are you using push or pull? With ESXi target and v5, it is best to install Veeam Backup server in the target site.

No, we are not doing public beta for Veeam Backup for a number of reasons. Private beta is only available to Veeam Backup Focus Group members. Customers are invited to join the focus group primarily based on their forum activity. Based on that, we can judge on a number of factors which affects our decision to invite specific person to join the group.

Thanks.

[averylarry]

I'm doing everything I can find on this forum plus anything else I can think of but it feels pointless to try and use v5 replication with ESXi servers.

Point 1) It's been stated on this forum that ESXi has 3 times as much data transfer for WAN replication compared to ESX. I'm just not realistically able to use ESX.

Point 2) My 10Mb WAN connection is absolutely saturated with replication traffic. I do not have over 700Gb of changes to replicate over the past week, but I have had replication jobs running constantly for a week.

Point 3) Veeam server is in target location -- pull configuration.

Point 4) All pagefiles are on separate disks that are not replicated.

Point 5) All log files (Exchange and SQL) are on separate disks that are not replicated.

Point 6) Exchange maintenance is set to run only once per week instead of daily.

Point 7) Using WAN setting for smallest block size/best deduplication.

Point 8) I've tried different compression settings, but I don't think that matters unless my processors are pegged out.

Point 9) I'm pretty convinced that there's some very significant "extra" traffic as I could have pulled the entire virtual disk across the wan faster than the replication is going.

Point 10) Further investigation seems pointless since v6 is supposed to have major replication improvements including better support for ESXi servers.

Point 11) HyperIP could be a nice possible solution, except the free version is limited to 2Mb bandwidth and I don't know if I'll need it until I see if my replication can keep up when using v6.

[Gostev]

Actually, Point 1) is not applicable when backup server is in target location -- pull configuration.

How do you determine that you do not have over 700Gb of changes to replicate over the past week? I can see that you are replicating Exchange and SQL, and these generate tons of changes on EDB and MDB file blocks.

Basically, in your setup (pull replication), most traffic is coming from changed blocks, there is little to no overhead. There is little v6 could do if the reason for your issue is trying to push too much data through your WAN link. There will be improvements in traffic usage, but they are not by the order of magnitude (which is what it seems you need here).

[mplep]

I believe the worst (or least performing) combination was replicating ESXi to ESXi with Veeam in the target site. Thats based on my experiences, forum feedback and also support advising the traffic is much higher that having ESX or Veeam on the source side. That's the physical combination we have with several hosts across a couple of different sites using (10 and 100MB) leased lines. Averrylarry seems to have a similar setup (ESXi/pull) so this quote confuses me a little:

"Basically, in your setup (pull replication), most traffic is coming from changed blocks, there is little to no overhead"

I'm just wanting to totally understand the situation and especially the difference v6 will make to v5.

Thanks

mplep

[Gostev]

Well, honestly there are about 100 existing topics going into this in details... v5 push replication to ESXi causes 3x more traffic than pull because replica VMDK rebuild is done over WAN. And during rebuild, each changed block causes 3 I/O (read/write/write) - these happen over WAN with push, and locally with pull. Will pull replication all that is going over WAN is 1 I/O per changed block (read).

In v6, over WAN it will be 1 I/O per changed block + traffic compression (not available with v5 pull today) + much less mgmt/aux traffic overhead. Certainly an improvement over v5 pull, but as I've said, not by an order of magnitude for sure.

[mplep]

Hi Anton,

I'm well aware of all the topics and have created some and contributed to many. But I've also had differing advice from Veeam personel on this over time. I'm was really hoping v6 would improve replication performance when compared v5 in my scenario and especially to another block level (guest based however) product we use a lot. You did tell me v6 will deliver "awesome improvements in replication, and ideal support for ESXi targets". I look forward to seeing how it pans out.

Thanks

mplep

[averylarry]

. . .
How do you determine that you do not have over 700Gb of changes to replicate over the past week?
. . .

First, it's not a week's worth of changes -- rather a day or 2 -- it's the replication jobs that are taking that long (I seeded the jobs locally, took it out to my remote site, it worked ok for a few days, and then it took a week). Second, I don't even have 700Gb of data (though the virtual disk sizes are much larger). With log files excluded, I can't see where I would generate more changed blocks than I have data (though I understand it's possible if you have lots of temporary/deleted files that dirty the blocks). Third -- just looking at the sizes of snapshots (since we now have multiple snapshots when backups run daily while a replication takes days/weeks) I would estimate at most 25Gb/day -- and I assume that the snapshots are roughly equivalent to the total of changed blocks that would need to be replicated (probably most correlated if using continuous replication).

But again -- It's really not worth trying to prove what my instincts are telling me given the timeframe on v6 and the expectation of significant changes in replication.

My plan:

Use HyperIP evaluation as long as I can (and hopefully I can get my replica caught up to current).
After making additional reductions in what I replicate, change most of my replications to continuous or hourly instead of once daily (except I can only run 3 simultaneous jobs).
Hope to make it until v6 comes out where I can fully decide if I need HyperIP to make it all work.

[Gostev]

You did tell me v6 will deliver "awesome improvements in replication, and ideal support for ESXi targets".

Michael, that's true, I am only saying that you should not expect WAN traffic reduction by an order of magnitude comparing to pull replication to ESXi with v5, even though the reduction will still be decent (at least in a few times).

Of course, comparing to push replication to ESXi with v5, incremental traffic reduction will be massive (order of magnitude easily) - but this is only because of how push replication is not effective with ESXi targets today.

[Gostev]

Ted, your plan is good. I only wanted to comment that biggest issue in your case seems to be bandwidth. Hyper-IP is mosly about improving unreliable connections, in fact it does this so awesome this results in about 2x effective bandwidth improvement. However, it is important to realize that Hyper-IP does not do traffic dedupe. The later is what can do miracles to bandwidth with Veeam replication, increasing effective bandwidth up to 10 times. Search this forum for Riverbed or Cisco WAAS for some real-world feedback on using Veeam replication with these guys.

Although of course, the price tag for these beasts is much higher than Hyper-IP.

Just some food for your thoughts.

[averylarry]

I used to backup everything except Exchange every night using robocopy. Only took about 6 hours. Sad if robocopy is better than Veeam.

[Gostev]

Not sure you are comparing apples to apples here, since robocopy simply cannot access data on VMFS volumes in the first place, and thus cannot do image-level backup by definition.

[rbrambley]

It sounds like not everyone on this thread is sold on the importance of pull replication, or placing the Veeam server responsible for the replication jobs in the DR site. As Gostev already pointed out:

... v5 push replication to ESXi causes 3x more traffic than pull because replica VMDK rebuild is done over WAN. And during rebuild, each changed block causes 3 I/O (read/write/write) - these happen over WAN with push, and locally with pull. Will pull replication all that is going over WAN is 1 I/O per changed block (read).

In v6, over WAN it will be 1 I/O per changed block + traffic compression (not available with v5 pull today) + much less mgmt/aux traffic overhead. Certainly an improvement over v5 pull, but as I've said, not by an order of magnitude for sure.

Also be aware that changing the target location type in the job settings changes the size of the blocks in flight and could help. Read the description carefully under each option in the drop down box.

Finally, consider this as another important design decision to put the Veeam replication server at the DR site: if you lose your primary site your only choice is to start VMs with the vSphere client at the DR site - even if you build a new Veeam server over there. If the Veeam server is already there you can fail over with the Veeam restore options including previous restore points and FLR.

Keep a Veeam server at HQ for backups only. Use the Veeam Enterprise Manager to connect to both the HQ and DR Veeam servers ( put the EMS at HQ). All pieces are included in your license at no extra charge. You can install as many Veeam servers as you need at no charge too. You could even use a Veeam File Copy job to copy your backup files offsite too.

[tsightler]

I think the thing most people miss is the impact that block sizes have on replication of servers like Exchange and SQL. These types of servers change many small blocks all across the disk, but Veeam can only replicate in 256K blocks at the smallest, so changing even a single 4K block can cause Veeam to replicate a 256K block via the WAN.

Based on what I have heard it doesn't appear that Veeam is addressing this issue in V6, which is a complete shame as it's the most glaring weakness of the replication portion of the product. That being said, I could be wrong and perhaps there are improvements to this issues in V6. This really must be fixed to make Veeam competitive with other replication products on the market.

[mplep]

Regarding block sizes and WAN mode, I recall that if Veeam was offsite then this feature wasn't effective for replication. Was that right and is that changing if so?

[tsightler]

I would think that block sizes would have an impact no matter what the mode. Certainly the best scenario is Veeam to ESX push replication, but for ESXi, where pull replication is pretty much required for WAN setups to be anywhere near efficient, Veeam is still reading smaller blocks from the source side. The main reason this mode is still less efficient that ESX push replication is that there is no network compression available in this mode (without tools like WAN accelerators/optimizers that perform transparent compression at the network level), but block sizes still make a difference.

[Gostev]

Correct, Veeam server location never affected this piece.

[Frosty]

At the risk of making myself look like an idiot (as I have no experience whatsoever with replication) ... I had a once-a-week backup problem with Exchange 2007 whereby my Tues/Wed/Thurs/Fri backups took about 15-30 minutes and generated a diff file of 5-10GB, and my Mon backup took 60-90 minutes with a diff file of 20-40GB ... it turned out to be my Exchange maintenance schedule which ran at the Weekends ... so I re-jigged my backup regime a little and made sure that maintenance on Exchange finished by 6pm Sunday night and then ran a Veeam Backup job ... after doing that all my Mon-Fri jobs on Exchange take roughly the same amount of time.

EDIT: I know you mentioned Exchange maintenance (point #6 in your early post) ... are you *certain* that it is not a factor? ... or is some other process causing a huge volume of updates to the DB (e.g. a journaling mailbox, or something like that?).

[rbrambley]

All of the points about Exchange maintenance and block sizes are valid. Do what you can to reduce the data changes before the job kicks off. Do what you can to reduce block sizes during replication.

Regardless of the efficiency of Veeam's block size options or the amount of data that needs to be replicated,the simple point about the IO "ping pong" of a Veeam server at the Primary site is the point I wanted to stress. It's the same reason Terminal Server app publishing is so effective. If an app has to send data back and forth multi times across the WAN then it will take longer or perform worse then if it can do it's job locally.

Maybe that's oversimplification?

[tsightler]

Hey Rich, I think your point was pretty clear. I should probably point out that while Veeam's replication is perhaps not ideal for replicating Exchange/SQL for sites with relatively low bandwidth links, Veeam's use of a larger block size actually has some performance benefits in environments with relatively high inter-site bandwidth as the larger block sizes actually cause less "ping-pong" than sending many small chuncks. As with everything, it's a balancing act, and choices that are ideal for one setup may not be for another. What I'm hoping to see is continuing improvement in the replication options to allow users to tune Veeam ideally to their environment.