Veeam licensing and support for large enviroments

[tomas.olsen]

Hello, two questions:

I am working on a case where we want to backup about 130 TB of data. and we might want to consider veeam (among others) as a solution.
We are designing a environment of about five vSphere hosts with EMC VNX FC SAN, and a dedicated EMC VNX with cheap and large disks, CIFS/NFS and 10gig ethernet. We might also want to store more than 30 days of restore points, and maybe some monthly jobs as well. Will this be a valid design that will work?

Second, Veeam issues socket licenses on backup. that means when ever a vm is backed up, veeam checks for a valid host license and issues a valid license for that particular host.
In the above environment of five vsphere hosts. Let's say I want to have another vcenter environment with vsphere essentials licenses just for lab environment. Do veeam issue licenses if I add the second vCenter, (no backup) and restores a virtual machine there?

Best regards,
Tomas

[tomas.olsen]

VNX with NFS/CIFS is dedicated disk storage for backup and surebackup/instant recovery and so forth. Forgot to mention that in the previous post.

[veremin]

Do veeam issue licenses if I add the second vCenter, (no backup) and restores a virtual machine there?

As it’s mentioned in this FAQ, Veaam Licensing concerns only about “source” host. So, yes you can add aforesaid server and use it as your destination host for restoration jobs without any licensing issues.

Hope this helps.
Thanks.

[Vitaliy S.]

Hi Tomas,

We might also want to store more than 30 days of restore points, and maybe some monthly jobs as well. Will this be a valid design that will work?

Yes, your scenario is absolutely valid. If you want to have a full backup for each month, then you may want to use some kind of script to copy monthly backup files or just run a secondary monthly backup job.

How are you going to present your VNX as a backup repository? Linux box or just CIFS share?

Thanks!

[tomas.olsen]

I was thinking of just using a CIFS share to keep it simple. Monthly job will most likely be a secondary job. I also might enable dedup and or compression on the VNX to save space. There will most likely be many jobs, running simultaneously and veeam only compress and dedup withing each job/file.

[Vitaliy S.]

Looks good to me!

[Yuki]

Uh,

Not sure how storing 130TB worth of backups will work on your "cheap" disk storage. Depending on your backup type selection - you may run into an issue we are seeing. The core of the problem is that the Reverse incremetnals, which we are using, take VERY long time to create backup files. The speed is not limited by network, but more so by storage device, since Veeam seems to be rewriting an entire FULL backup file. The end result is backup that takes many hours and interferes with other jobs.

You could go and do Synthetic or Active fulls, but then with Synthetic (which makes most sense on large data sets) Veeam advises not to have to many incrementals between fulls (no more than a week?). This is because the creation of the synthetic full is going to be so resource intensive that it can run for many hours (if not days in your case). Actives are always an option if you are willing to move these 130TB regularly...

We are doing this on about 12TB of raw space with about 4-4.5TB of actual data being written to a 12 disk NAS via CIFS (SATA drives). I suggest you use trial version to do a test before committing to anything. If your environment can handle it - by all means use Veeam.

[dellock6]

I really cannot see how a VNX storage, even if filled with NL SAS or SATA disks, can be called a cheap storage; if by quoting cheap you mean "a slow and under-powered nas storage with small cpu that can be killed by reverse incremental", well I think a VNX storage is not that...

Also, a reverse incremental DO NOT creates a new full everyday. It updates the only full of the chain with the latest incremental changes. If you have bad performances out of your 12-disk NAS, maybe you can switch to forward incremental. I do not know the size of your full backup, but if that NAS is killed by reverse incremental, it would be even more loaded by running synthetic or transform.

Supposing you are running on raid5 without spare, you have 11 Tb of space. This can hold double the size of your actual backup set and still have space, so the extra full created by forward could be managed. To be sure you can switch, I would need to know the size of your full backup and your retention policy.

Luca.

[Yuki]

I don't know what VNX costs, but if it is SATA based and has low number of disks in RAID5/6 - they could see issues we are seeing. If it has a good controoler, plenty of spindles in RAID10 - then i'm sure it will run smoother.

Dellock - i'm really at a loss why we are seeing low performance... i've gone through so many attempts to fix this, but so far haven't had much luck. I only suspect the issue is with the NAS IOPS. Our main backup file VBK - is 3.4TB and incremental sizes are usually 100-230GB. In terms of time - they take between 6-13 hours for an incremental. We should be able to do it faster on a 12 spindle NAS RAID5 array IMHO.

[tomas.olsen]

I have seen quite low performance from low end nas from qnap, netgear, qnap and so on even though they have several disks to relay on. the problem is that the nas is short on cpu and/or memory and in some cases have just one 1gig ethernet. VNX is a High End SAN produced by EMC that delivers both auto tiering functionality, pooling of different raids, different disks as well as thin provisioning, compression, dedup. The VNX can scale up to hundreds of disk but in this case we might use Larger 2.5inch NL SAS drives divided into several RAID5 groups in one pool with AutoTiering. The idea will be to use the filer option and present nfs or cifs to the veeam backup server and use a LACP channel for two 10Gigabit ethernet channels to boost performance. I am rather confident that this solution will deliver the performance we need, as long as we deploy enough proxies and creates several different jobs to handle the total amount of data.
My only concern is that veeam isn't fast enough to handle this.
Veeam uses a lot of time just preparing the virtual machine for backup, and in a environment with many vm's veeam will perform worse then if you have an environment with fewer but larger vm's.

[Yuki]

Alright, but you should provide more details on your servers (local storage or SAN, if SAN - is it FC, iSCSI, etc), how many VMs, size of VMs and rate of change.

From personal experience (again, in environment that probably doesn't stack up to yours since we run VMs from local drives) - where we see some delays is in removal of snapshots. Especially on large VMs with terrabytes of data. But that is to be expected and is not Veem's issue as much as VMware.

You may also want to use MPIO instead of LACP as the former will allow to increase throughput even for a single system, while the later will only do that during multisystem access. General recommendation is that for a SAN you use MPIO and for NAS you would use LACP.

[dellock6]

Dellock - i'm really at a loss why we are seeing low performance... i've gone through so many attempts to fix this, but so far haven't had much luck. I only suspect the issue is with the NAS IOPS. Our main backup file VBK - is 3.4TB and incremental sizes are usually 100-230GB. In terms of time - they take between 6-13 hours for an incremental. We should be able to do it faster on a 12 spindle NAS RAID5 array IMHO.

Uhm, first of all, depends on the NAS model and raid configuration. By the numbers, 12 sata disks on raid5 can give you a total iops of about 1000, and with a block size of 64k this translates to different numbers based on backup type:
- forward incremental is 100% write, so it gives you 16MBs
- reverse incremental is 33% read and 66% write, so in theory can gives you 32MBs. BUT, since has 3 IOPS per saved bytes, the effective speed is 32/3, so is around 11 MBs

All these numbers are calculated only using disks, without considering cache and storage controllers. Both can increase the actual numbers, so that's why a VNX with the same number and type of disks can be really faster than a consumer grade NAS like Qnap, Synology or other based on Intel Atom and loaded with few GB of ram.

One workaround to increase speed on these kind of NAS is to convert them to raid10. First, to have the same usable space (10 TB) as before, you would need to use 2TB disks. Once you have these disks, the new numbers with raid10 would be:
- forward incremental, around 33 MBs
- reverse incremental, in theory 47 MBs, effective 15.6 MBs

There is a big difference in these numbers. If you can save all your backups in 6 Tb, the quickest solution would be to convert your raid5 in a raid10, this would give you 33 MBs for forward incremental, that is 3 times the speed in regards to the reverse incremental using a raid5.

Hope this helps.
Luca.

[dellock6]

An addendum, your 230 Gb per night, on a reverse incremental are equal to 690 Gb to be read or written to disks (remember, 3 iops per saved byte).
At a speed of 32 Mbs, this equals to almost 6 hours of backup time.
Convert to raid10, and with a speed of 47Mbs the same backup can be completed in less than 2 hours.

Luca.

[Yuki]

Last nights file server backup was about 85GB and it took 6 hours at 4MB/s

anyway, I will see about converting RAID5 to RAID10

[tomas.olsen]

Alright, but you should provide more details on your servers (local storage or SAN, if SAN - is it FC, iSCSI, etc), how many VMs, size of VMs and rate of change.

Sorry if I forgot this Yuki,

environment will be 5 - 6 HP DL380G8 or IBM X3690X5 hosts. 150 - 250 virtual machines. Each host will start at 256GB of RAM, we might double i to 512. all virtual machines running out of an EMC VNX 5300 with about 150 TB of raw disk. SAN is FC. the veeam backup server will be virtual, and we might deploy at least one proxy per host for simultaneous backup jobs from all the hosts in the cluster. An extra VNX 5300 with filer to provide CIFS/NFS as a backup repository. Hosts will have dual 10gig ethernet, and vnx will have at least dual 10gig ethernet for NAS.
this configuration is part of a dual site configuration.
a secondary site will contain at least three hosts and a separate VNX5300 with block replication from the primary site. The VNX dedicated for backup will be at the secondary site.

[tomas.olsen]

Last nights file server backup was about 85GB and it took 6 hours at 4MB/s

anyway, I will see about converting RAID5 to RAID10

Sounds like you have some serious problems, Yuki.
I am doing backup of a few virtual machines in my lab to a single disk nas with really bad performance. It's a domain controller, one file server and a few linux vm's. It only takes 15minutes doing reversed incremental, no indexing. The total amount of disk is about 250 GB.
You need to examine your logs to find out what takes so long time.

[Yuki]

thats what i think, but so far support didn't find any causes and their only suggestion is that the problem is slow NAS. I will be going on-site next week to take a look at the NAS while a backup is running. CPU, RAM, network all seems underutilized.

I'm also backing up same systems to another NAS but a 16 bay one over a gig link, the other difference is that i've separated the larger file server into its own job. Right now going to 16 bay nas in reverse incremental i'm getting 11MB/s. The first run, when there is no existing CBT data or need to read/write to recreate VBK - i get 100-116MB/s. It's a 2.5TB file on a VM that spans 5x2TB VMDK files.

Veeam support told me the problem is probably low throughput on the NAS when rewriting VBK file.

The setup is definitely not performing to the expectations I've had for it.

[cronosinternet]

OK my take on it (I am familiar with all of this since we are running 12 x 256GB Dell R620s with VNX as the back-end storage. 10Gb iSCSI)

1. I don't think EMC VNX is particularly suitable for this. Behind all the marketing of Unified storage what you actually have is a block device first and foremost, with NAS filers to provide those protocols. So its two bits of hardware that just happen to sit in the same rack with the same bezels Well, there is a little more to it than that, but unless you have a truly massive VNX and a big file requirement, its easier to stick to block. Can always spin up VMs to present storage via CIFS/NFS if you wish.

The other reason is that the price they charge for a 3.5" shelf and the capacity NL-SAS will make for a very expensive backup store.

2. You'll find that going from VNX to VNX using veeam as the middle man won't be particularly efficient with all of those proxies and networking flying around. Ideally you want veeam talking to the SAN directly, not going via VMs - they are terribly CPU intensive when they run.

We use supermicro 2U boxes that have 24 drivebays in them (2.5" tech). We are only using one currently but when its full we'll simply create another. This will lead to a very distributed scale-out architecture. Veeam requires a fair bit of CPU, so every time we add a storage box we're adding another 16 cores of CPU as well as the storage. 24 x 2.5" seagate SAS drives on the back of an LSI 2208 makes for excellent performance, far in excess of what veeam can send its way.

The only downside to this plan is that in your setup you'll need to fit FC cards to each veeam box, and ensure you have enough switching capacity. A better approach would to be to buy the 10Gb iSCSI cards for your VNX (I think you can run iSCSI as well as FC - but pls check!), then you will have a lovely setup. All of your hosts will talk FC to the SAN for primary storage, but Veeam can pull the data off over iSCSI. I have installed/configured Windows MPIO (we're running veeam on server 2012) and I see exactly 50% traffic down each NIC. So with two switches, two NICS per host and two NICS in each of your VNX SPs you've also got complete fault tolerance. I'd recommend Arista switches for 10Gb.

In our setup everything is Direct-SAN access so I see no resource being used on the cluster. I do have a proxy VM installed, but thats more for restores else you'll fall back to rubbish NBD.

More than happy to share any configs or findings.

Can't help on the licensing front as we're a cloud hosting company so part of the relevant program.

cheers
Lee.