VEEAM with Riverbed

[canotuna]

Has anyone ever tried performing a replication using VEEAM across a slow link with Riverbed? I have a client that would like to replicate often, but only has the option of a single 1.5 T1 for connectivity. It is a small environment with replicating 1 host to 1 host, so there isn't much data. Can anyone provide input if Riverbed has helped? Also, is there a good calculation method for figuring out how much bandwidth I need across a WAN?

Thanks
Dan Sanderson

[Gostev]

Dan, I have not heard any feedback about Riverbed yet (and never heard about them before, to be honest); but I know some Veeam Backup & Replication customers are using HyperIP with great success. Hope this helps.

[Bunce]

We're about to test a couple of Riverbed Steelheads at both ends of our 2Mbit links, will post back the results.

A

[tsightler]

I have not used Riverbed, however, we do use the Cisco WAN acceleration product (WAAS). We see tremendous improvements with WAAS, with "over-the-wire" compression ratios of 90+% and are able to replicate a small group of 5 servers over a 2Mb link from Europe to the US with no issues. The Riverbed appliances are generally considered "better" than the Cisco product so I suspect you would see a significant improvement. Please let us know the results.

[Bunce]

Hey All, thought I'd post some early results of our Veeam trial using replication over Riverbeds.

Environment

• Multi site MPLS VPN - includes HO and DR
• HO Site - 4Mb link
• DR Site - 2Mb link (effectively 2Mbs between them)
• VBR server (in VM) at HO, using Virtual Appliance mode
• Replica job includes 12 VM's located in HO for total of 616GB. Most have multiple disks (system + apps)
• VM's predominantly W2K3, with few W2K / W2K8. Some built from templates others from P2V. Includes Exchange FE and File Server
• vSphere 4, VC, and EMC Clariion AX4-iSCSI (1xGB links - multi-pathed) at each site as VM storage.
• Riverbed Steelheads at both sites to optimise traffic

In setting up the replica job, seeded it to external HD and shipped up to DR site. This is the result of the first 3 runs (including the initial seed):

Code: Select all

Date  	Port	Job    	VMs  Tot   	LAN-if  Veeam   WAN-if   Rvbd 	VRB  	Comment
-     	-   	(Drtn) 	(#)  (GB)  	(GB)	 (%rdn)  (MB) 	(%opt)   (MB)	-
19-May	n/a 	9:48:40	12	616.09	n/a  	n/a  	n/a   	n/a  	n/a  	Manually shipped to DR via external HDD
22-May	2500	9:34:30	12	616.09	35.3 	94%  	4906.1	86%  	14972	CBT not enabled on 1 VM (70GB total / 1.8GB VRB)
22-May	2500	1:40:12	12	616.09	10.2 	98%  	0235.6	98%  	05568	Run directly after initial. CBT enabld for rem VM

A few notes:

• First entry is the seed to external HD. Second is the 1st replication over the WAN. Third was run directly after the second (so should be minimal VM data change)
• Port is the port used by VBR to send data for this job (believe it can change)
• LAN-if and WAN-if measure the traffic received from VBR (port) on internal and external interfaces of the Riverbed (note the different measure, LAN=GB, WAN=MB)
• Veeam %rdn is the difference between total VM size and the traffic received on internal interface (i.e. what's sent out by Veeam). Very nice - however doesn't provide useful stats such as how much is empty disk, what is the actual change rate, compression rate of actual data, etc (like esXpress offers for example) etc.
• Rvbd opt% is the difference betwee LAN and WAN interafce - i.e. the amount of traffic saved over the WAN using its compression/caching technology. This is outstanding for us as allows us to complete a run within a few hours over the 2Mb link. Will be interesting to see how it settles during the week after a normal days production usage.
• Again, as Veeam doesn't provide us with any stats as to what is actually sent out, I totalled up the size of the new VRB files created at the destination. Interestingly, this was roughly only half of what was actually sent from VBR to the Riverbed on the LAN, so there's other traffic at play there somewhere..
• One of the VM's did not have CBT enabled on the initial job. It was enabled on the second but doesn't affect the stats too much.

Summarising:

• Riverbed = WIN
• would be great if Veeam could include detailed staistics of what its actually achieving. Given the importance of determing precise traffic requirements in capacity planning for DR replication, I'm suprised this is not currently available and not everyone can afford a Steelhead! The 'Total Processed Size' metric is pretty pointless.
• Would also be nice if we could manually set the port used by Veeam per job (with Veeam verification its not in use). This would allow far easier configuration and reporting on WAN optimisation devices such as this, as well as assit firewall admins should it be traversing a firewall.
• The first external HD used to seed the replica failed once received at the DR site, meaning we had to reseed to a new USB Hdd. This was not possible using the existing job, and so we had to create a completely new job. Would be good to include a checkbox to 'reseed replica to local storage' or similar for cases such as this.
• About to begin testing the replicas and noticed you can only bring up one at a time?? Surely in a DR situation having to do this manually for what could be hundreds of VM's could cause long delays..
• Overall, the replication feature of Veeam is quite nice and gets a tick over competitors in this area.

Will update the stats later in the week once we get a baseline.

Cheers,
A

[cby]

Bunce

Many thanks for the detailed report -- look forward to follow-up stats. We are seriously looking at this type of replication/DR scenario and as always real-world use uncovers the nitty-gritty!

Dare I ask, how much the Riverbed Steelhead costs?

cby

[Bunce]

still on trial actually

they'll generally give them to you for free on a trial basis after you provide some basic usage patterns, which indicate whether they are likely to be productive.

after xx months you either pay and keep them, or send em back. safe to say we'll be rolling them out to each site rather than upgrading our pipes, but haven't yet got quotes.

[ibogachev]

Cyber Advisors Inc is the Riverbed partner, call us for quotes and support: 952-924-9990 - promise to share partner's discount . I have a few customers using Riverbed Steelheads. My next step is to test a DR scenario with VEEAM/Riverbeds. Please let us know the results - looks like Bunce is a step ahead of me. thanks a lot!

[dreddaway]

Riverbed is the granddaddy of WAN optimization, however my understanding is, Riverbed really performs well with unchanged data - improving on post, first, full replicas, but no dramatic performance changes with incrementals that are just sending changed data without repetitive patterns. Efficiency gains are then accomplished by compression algorithms, which Riverbed has also perfected. However, with all the features supporting repetitive data, Riverbed can be very costly - starting at ~7K per side. BTW - it is the same with most other appliance-based solutions.

This begs the question, are you fixing a symptom or the root cause? I contend the issue is not the data, which Riverbed and SilverPeak address; it's the transport protocol inefficiencies that impair data throughput. TCP/IP is the culprit with the way it accommodates latency, error correction and poor circuits. You can fix TCP/IP for less than 50% the cost of Riverbed or SilverPeak and realize 10X performance gains in VMware backup, recovery and replication even when using the fastest application Veeam.

Check out http://www.netex.com.

[tsightler]

Well, I'm glad that you contend things without knowing. I have no problem with HyperIP, it's a fine product for what it does, however, products like Riverbed, SilverPeak, and Cisco WAAS offer caching which is quite useful for a product like Veeam. When they show "compression" numbers they generally include DRE savings which is basically caching. Veeam is not very efficient as it sends 1MB blocks even when there are small changes. If a single byte is changes in a 1MB block Veeam sends the entire 1MB block. The Riverbed, SilverPeak, and WAAS products would be able to use their cache to send only the changed data. We find that approximately 90% of the WAN savings are from the DRE, not from compression or TCP optimization as Veeam is already pretty good with that anyway.

If Veeam were to optimize their replication to only send the actual changed data within the 1MB blocks then I suspect that HyperIP would be just as useful as Riverbed, SilverPeak or WAAS. I believe that the next version of Veeam will switch to a smaller block size, which will help some, but changing to a hashing algorithm that only sends the actual changes over the wire, basically a block level rsync, would likely provide performance similar to what we're seeing with WAN optimization products even for environments that didn't have them.

In other words, the WAN acceleration products are really highlighting just how inefficient Veeam's replication actually is.

[1-0-1]

[merged]

Looks like we going to start with some tests on Riverbed Steelhead 6050 and 2050L for our replication over our 5Mbps (!) link. Still have to read up on the devices but though, in the meantime, if anybody share their experience on this and what to expect? How does it match against HyperIP? Will backups benefit from this as well seeing they are already compressed and de-duplicated?

Biggest challenge we had with replication is that our servers are all big servers mostly 400-1TB the replication takes anywhere from 8 hours to 110 hours. Exchange is a big headache as it is close to 1TB and even after the seeded replications it still takes several days to replicate.

[Bunce]

See above for my stats. I will update with more recent ones when I get a chance, however traffic reduction is still similar (roughly 90%)

Assume you're not using XGE 2010 as wouldn't make sense to replicate using Veeam when you can use XGE DAG's..

[1-0-1]

Just a quick question - I have noticed we doing all our replication tests with the VEEAM compression on and seeing some positive gains. As far as I understand this should be ideally turned off as the Riverbeds will not be able to effectively optimize compressed data?

[dinger76]

Hi,

Just thought that I would offer some of our experiences with regards to Veeam and Riverbed. We are actually using Veeam v5 and replicating around 50 virtual machines either on a daily basis or weekly basis from our datacenter in the UK over to our DR location in the US. This has been a bit hit and miss, varied performance and issues being seen.
We are also using a similar design for some of our locations across Europe, replicating their virtual machines into our datacenter in the UK... again we were seeing some mixed results. All of these configurations were based on using a Push design, meaning that the Veeam server on the source site was being used to push the data across to the target.

This last week or so we have begun to make some real progress. We have discovered, after lengthy investigation, that the Riverbed is optimizing the traffice pretty well but the transfer speeds were not really increasing that much. Apparently, our network team, say that this is because although the Riverbed is optimizing the traffic and sending it as quick as it can... there is a delay from Veeam between chunks of traffic getting to the Riverbed, sometimes up to 7 or 8 seconds between each chunk.

We have now also just begun performing a Pull replication design, this means that we have the Veeam server on the target site and pull the virtual machines from the source. We have just received information from our network team to say that they are seeing the best optimisation they have ever seen with Veeam traffic through the Riverbed, getting up to around 94% optimisation... plus there doesn't seem to be as much delay between chunks. My feeling is that with the new Veeam v6 software, they have corrected the delay issues between chunks which is why the speed is estimated to be 10x faster, this thrown in with the Riverbed using the Pull design should make for an excellent solution... I just need to get my hands on V6 of the software to prove it.

Cheers

Chris

http://virtualizedworlduk.wordpress.com/

[oyell]

[merged]

Hello,

We are currently testing to use Riverbed (Wan optimization http://www.riverbed.com) together with Veeam Backup & Replication.

I have setup a complete new installation offiste for this testing. and location is Singapore & Norway with a 2Mb/s connection. (Running 12 MB/s on increamental jobs)
What I have done is to install the Veeam server as a virtual machine down in Singapore, mapping a local drive from a server in Norway to backup our VMWare Virtual Machines directly to it. So this is plain CIFS traffic going from Singapore to Norway, becouse of the WAN optimizing unit Riverbed.

The backup job in veeam is setup with Vitual Applicance, Incremental with performance of active full backup jobs periodically. On storage is setup with inline data depuplication, Best compression level, and WAN target.

When looking at the logs for Riverbed, we only gain 8% compression, but since veeam is setup with best compressions, this is likely to be expected.

Question is, should I skip the compression level in Veeam, and let the Riverbed handle this job? As my understanding (I`m not a network specialist, so riverbed setup is not performed by me), cifs is the best compression level for Riverbed to use?

Any maybe to setup the Veeam software in Norway would be the best option, and let it map directly to the storage in Singapore VMWare hosts?

Our end setup will have 13 sites around the world setup with Riverbed, and hopefully Veeam. Most sites will have 2Mb/s connection, or better.

Anyone else have experience to share with Riverbed and Veeam?

Thanks in advance

[Gostev]

Well, compression indeed kills dedupe, so definitely try disabling it first to let Riverbed do traffic dedupe, and see if it helps. Thanks!

[bobegilbert]

Here area details of Riverbed on Veeam on a 2Mbps link

http://blog.riverbed.com/2013/07/riverb ... -link.html

[foggy]

Bob, thanks for sharing the results of this testing with Veeam community members, highly appreciated!

[tuscani]

I am curious.. are you guys using Riverbed with private connections between sites, or site-to-site VPN tunnels over the Internet? We use the latter (no Riverbeds) and the performance is dismal. For example, our data center has a 100mb connection.. and our DR site is 50mb.. we are average about a 1MB transfer speed on backups or replication over the tunnel which seems absurd to me.

[ctvader]

We're trying out Veeam Backup right now and want to send it across our riverbed devices. I've search the veeam posts on Riverbed but can't find how to set up the in-path rules. Do we define rules from ESX host to ESX host? Any specifics on the optimization policy? Thank you.

[Vitaliy S.]

I believe you should set rules between Veeam proxy servers or gateways on both sites. As to the specifics of the configuration, then let's see what our community members can say on using Riverbed. As to Veeam configuration, then it is recommended to uncheck multiple upload streams option in the network throttling settings.