What is the largest environment being backed up

[razorvines]

I have used B&R for all of my clients to backup their virtual environment as well as replicate to an alternate site. They are typically a 3 host environment with 20 - 30 virtual machines. Now I am involved in a much larger internal environment build that will be approximately 10 hosts, 200 VMs, and about 100 TB of data.

What is the largest environment out there being backed up and replicated using Veeam B&R? How many proxies are you using and what kind of bandwidth do you have to DR?

[tsightler]

I work with environments every day that are 1000's of VMs. Rarely do I work with environments that are less than 500 VMs. What I'm a little surprised about with your setup is that you have only 200VMs but 100TB of data. That's a very high ratio, far higher than what I typically see. That means that your average VM size is 500GB, which is pretty huge for an average. More typically I see environments that are 100GB/VM, actually with actual used sizes much smaller than that. Is you data evenly spread across your VMs or do you have a small percentage of very huge VMs that make up the bulk of the data?

[razorvines]

And you manage all of those backups with Veeam and nothing else? 50 to 60GB of that is actual production data, the rest are archives so pretty static.

[tsightler]

Sure, all data managed with Veeam (fair notice, I work for Veeam as the Solutions Architect for large customers). Veeam scales out pretty well, just add enough proxies to handle your data by running enough jobs concurrently. 200 VMs really aren't all that much, though I now see in your initial question that you asked about both backups and replication (sorry, I missed that in the first read). Replication can be a little bit more of a challenge, at least if you are attempting to replicate more often than once a day. Still, it's all about having an infrastructure that can support the required snaps and enough bandwidth to get the data to the remote site. Veeam itself wouldn't have any issues with that many VMs or that much data in general, but there might be some specific things that could trip it up, such as if you have a single server that is very, very large. How big is your largest server?

[razorvines]

2Tb is the largest we have. So are you on named accounts or to a territory????

[Berniebgf]

Tsighter

Can I presume the majority of your backup Targets for Proxies at these large sites are NFS targets? or CIFS targets? Would be interested in how you structure the solutoin (around storage targets/ devices) for these larger multi proxies sites (specifically around backup) when it comes to storage performance... and then movement to tape (if at all)..
Do you more lean towards Multiple proxies (CPU grunt) to Centralised NFS storage? or more distributed model for processing and storage?

Obviously this will depend on site / location and so on.....but lets say for a single LARGE site that requires MANY "Data Mover"........

Bernie.

[tsightler]

I'd say I don't typically see NFS targets, but they can be by using a Linux repository. I actually like that approach, but most clients are more comfortable with Windows, and this more common is to see CIFS or NTFS attached to Windows.

It really boils down to what the customer is wanting to do and what infrastructure they have in place. If the client is using a dedupe appliance (probably the majority) then it's either CIFS directly from multiple proxies, or NFS via a Linux repository. If a client is simply going to disk (probably the next biggest group) then SAN attached disk or locally attachted disk on a system used as a repository is the next best method.

From a performance/cost perspective I personally prefer using dedicated physical servers as proxies with locally attached disks for repositories. This provides a self contained device with a single maintenance contract with a fixed performance ceiling that can be easily defined. When you need more storage you add another repository so you get more processing horses as well. Potentially these can be proxies as well so you effectively get SAN offload and scale forever simply by adding dedicated proxy nodes. The disadvantage of this approach is of course that it requires manual balancing of jobs across the available storage/proxies, somewhat negating the smart load balancing built into the V6 product. That being said, from a scale out performance perspective, it's hard architecture to beat since it guarantees that traffic does not cross the network (direct SAN to local disk).

Many large clients have attempted to build single massive repositories using SAN attached disk. This simplifies job management since there is simply one massive pool of target storage, but has significant performance side effects as all of the I/O is targeted at a single large pool of disks it only takes a few reverse incremental jobs to have long request queues and cause tremendous I/O latency, significantly degrading backup performance. The lure of this "single repository" is strong, but is much more difficult to build at scale with reasonably random I/O performance, especially because they are generally attempting to use low end SAN hardware to do so (small caches that are easy to saturate).

So in other words, just as you said, it varies a lot from site to site, customer to customer, based on their goals and budget.

[Berniebgf]

Agree %100 with NOT having backups on the same SAN Array that you are backing up, even if separate disks, draws or controllers.....
With the workload that Veeam pushes you do not want to double up your IO on the array! (especially if its a Netapp array with iSCSI!!)

Can't beat internal disk to the proxy or DAS or Dedicated Fibre Array, biggest issues I see are IO throughput issues with source and/or destinations.

Must admit I have changed my mindset with the last few install between Reverse Inc and Incremental, doing allot more Incremental solution to reduce workload on the back-end disk systems...

Thanks for the info, I was wondering how you best leverage the "Load Balancing" which I LOVE with the Replica Proxies. But hard to Architect with the backups.....($$$).

Bernie

[tsightler]

Certainly incremental will reduce the workload, but the cost is storage space is significant and thus difficult to scale (assuming no dedupe appliance). Basically you effectively need at least 2x the space of forward incremental, and in many cases (based on retention) 3x or more. Reverse incremental can be designed to perform well for the vast majority of systems, only systems with high change rate are usually an issue. Of course the problem is that most clients don't really know what systems will have high change rate and which ones will not, they want a "single best mode to rule them all" but unfortunately for now it's about tradeoffs between the modes.

Getting the best performance out of reverse incremental as well as synthetic fulls is all about building target storage that is well optimized for the random I/O workload that it will get. So many times I find customers that simply take the default of their RAID or attempt to strip massive bundles of 12-16 disks together creating stripes that are far to small (thus creating too many IOPS), or are way too large (thus creating wasted I/O on every block read/write). Not only that, but having controllers with decent sized, battery-backed write cache is critical.

Also, remember that RAID levels 5/6 have a significant IOP write penalty, which really comes into play with reverse incremental. RAID1 has a write penalty that is half of RAID 5, and thus can generally deliver more IOPS for a mixed R/W workload. Because of this, it can many times be beneficial to use RAID1. Many customers look at this as "wasting" space, but the reality is that this may be the difference between being able to use reverse incremental or not and reverse incremental will easily provide the most space efficient and easiest to manage backup options assuming the IOPS are available to support it.

[bhwong]

Agree %100 with NOT having backups on the same SAN Array that you are backing up, even if separate disks, draws or controllers.....
With the workload that Veeam pushes you do not want to double up your IO on the array! (especially if its a Netapp array with iSCSI!!)

For backing up on the same SAN Array, some SAN vendors provide an API for vCenter so that the data transfer happen internally within the SAN storage without consuming the iSCSI network. Wonder can Veeam make use of this to improve performance?

[Gostev]

You are missing the fact that Veeam backup process involves on-the-fly data deduplication and compression, not just moving unmodified source data to the backup repository.

[bhwong]

You are missing the fact that Veeam backup process involves on-the-fly data deduplication and compression, not just moving unmodified source data to the backup repository.

I see. Will Veeam be able to install an agent to this: http://www.dell.com/us/enterprise/p/pow ... -nx3100/pd that comes with Windows® Storage Server 2008 R2 build-in so that it can act as Proxy and Repository Server without assigning a VM to do so?

[Vitaliy S.]

Yes, it should be possible. For additional info on supported OS for proxy and repository servers, please review system requirements section in the Release Notes document included with your download. Thanks!

[jpeake]

I'm curious how large some of your environments are. Is anyone backing up 1,000+ VM's at a single site?

[v.Eremin]

There is one of the largest clients that I’ve heard about (from Tom Sightler) who backs up over 2000 VMs on a regular basis. This is nothing but an example, and there are others, as well.

Thanks.