Backup of 250 VM’s takes too much time

[tedsteenvoorden]

We currently backup around 250 VM’s (26 TB) with incremental backups during workdays and a synthetic full with transformation in the weekend. These backups are done by three physical proxy/repository servers (Veeam 6.1) each with 20 TB local storage using direct SAN (fibre channel connected).

Currently we have two problems:
- We are running out of the backup window during workdays
- The synthetic full with transformation takes more than 48 hours to finish

To solve this problems I’d like to do the following:
- Implement a NFS/iSCSI based NAS as a central backup repository. This way all backup proxies can access the central backup repository and workloads can be more efficiently distributed over all backup proxies.
- During the weekend only do a synthetic full without transformation. This will result in less processing time, but requires more storage (multiple full backups).

I’d prefer to use virtual backup proxies (using virtual appliance mode), but I know direct SAN is preferred over virtual appliance mode. Virtual backup proxies allows us to more easily scale the backup infrastructure, but I don’t know the exact (performance) penalty by using virtual appliance mode.

Does anybody has implemented his backup infrastructure this way and if so, how does it handle the job? Any advice would be welcome!

For the central backup repository I am considering a Synology RackStation RS10613xs+ with three Synology RX1213sas enclosures (46x 4TB SAS disks) resulting in 92 TB RAID 10 storage.

[Vitaliy S.]

Hi Ted,

Can you please tell me if you have patch 1 installed on your Veeam B&R server? The reason why I ask you about this is that in this patch we have improved performance of synthetic full backup creation, this might solve your problem:

•Increased synthetic full transformation, and reverse incremental backup performance (up to a few times depending on backup storage).

Thanks!

[dellock6]

Hi Ted,
your idea for future design is similar to ours, we have virtual proxies in place, and a shared physical repository used by all the proxies so we can distribute jobs among the proxies and do not pin every job to a specific proxy.
we choose virtual proxies above direct san even knowing about little lower performances so even proxies are protected by our vSphere infrastructure, sorry I do not have numbers to compare with DirectSAN since we went straight to virtual proxies...

About the shared san, be careful about the CIFS storage: I've seen that storage can be raised up to 32 Gb of ram, maybe it could be worth the price, to have better performances on it. We choosed another way, using real X86 servers fully loaded with disks (supermicro 4U servers by the way) so they can have higher ram amounts, faster CPUs, and also by installing Windows Server OS on it you can deploy Veeam repository module on this server. It act way better than a CIFS share...

Luca.

[tedsteenvoorden]

Hi Ted,

Can you please tell me if you have patch 1 installed on your Veeam B&R server? The reason why I ask you about this is that in this patch we have improved performance of synthetic full backup creation, this might solve your problem: Thanks!

Vitaliy S.,

We are indeed running Veeam 6.1 patch 1.

[tedsteenvoorden]

About the shared san, be careful about the CIFS storage: I've seen that storage can be raised up to 32 Gb of ram, maybe it could be worth the price, to have better performances on it. We choosed another way, using real X86 servers fully loaded with disks (supermicro 4U servers by the way) so they can have higher ram amounts, faster CPUs, and also by installing Windows Server OS on it you can deploy Veeam repository module on this server. It act way better than a CIFS share...

Luca.

Hi Luca,

Thanks for your response! I think i will go for the virtual appliance route. But there are still some other choices to be made. We also need to offload all data once a week to tape. If we decide to go with the Synology RackStation RS10613xs+ we will indeed add 32 GB ram. The Synology RackStation RS10613xs+ can both act as NFS or iSCSI storage. Triggered by your response, it may be better to put some Windows repositories before the the Synology RackStation RS10613xs+ and present the backup storage through iSCSI. This makes backup to tape also easier.

[dellock6]

Oh, that's for sure. If you also need tape offload, definitely a Windows Server acting as a frontend repository will also allows you to install a tape backup software on it!

Luca.

[tedsteenvoorden]

Hi Luca,

One more question out of curiosity. How many virtual proxies are you using in your infrastructure and how many VM's do they handle? Also, how many jobs are a single backup repository handling?

[veremin]

Hi, Ted.

While estimating the number of proxies that need to be implemented in your environment, please, keep in mind these proxy resource requirements which are considered to be “best practice”.

Hope this helps.
Thanks.

[dellock6]

Ted, there was a thread some months ago where I described our infrastructure:

http://forums.veeam.com/viewtopic.php?f ... ure#p66822

Since then, we increased to 10 proxies, my rule of thumb is 100 VMs per proxy. For repository, we introduces some of the VeeamPod I talked about there, basically are supermicro servers, 4U, fully loaded with sata disks and configured in one large stripe R5.

Each proxy is 4 vCPU and 4 gb ram, westmere sockets, limited to 2 jobs, and each repository can ingest 4 jobs at a time.

But we are a service provider, I would not use our design as "standard" design, for example we only have 7 days retention....
Luca.

[Jonathan]

About the shared san, be careful about the CIFS storage: I've seen that storage can be raised up to 32 Gb of ram, maybe it could be worth the price, to have better performances on it. We choosed another way, using real X86 servers fully loaded with disks (supermicro 4U servers by the way) so they can have higher ram amounts, faster CPUs, and also by installing Windows Server OS on it you can deploy Veeam repository module on this server. It act way better than a CIFS share...

+ 1

We recently went down the Supermicro road to build a couple of 4U high capacity storage servers and it's been a blast. You really get much more bang for your buck compared to most iSCSI / CIFS filers. Decent hardware RAID controller, tons of RAM, and you get to run Windows Server 2012.

[Bernd]

Syntetic fulls seem to be a nightmare to NAS. We went over to reverse incremental for about 300 VMs using 5 virtual Proxy Server. But maybe thats not a good idea if you want to go to tape.

[Yuki]

Syntetic fulls seem to be a nightmare to NAS. We went over to reverse incremental for about 300 VMs using 5 virtual Proxy Server. But maybe thats not a good idea if you want to go to tape.

Funny you should say that, we are trying the exact opposite - going from Reverse Incremetnals running to NAS to Synthetic Fulls (initially to NAS and if that doesn't work out then to a Win2012 on top of Dell PE720xd with H700 and 1GB RAM) and 12x3TB drives. When we put Dell server as the target, the performance increased substantially and to a point where the bottleneck changed from Target to Proxy. Proxy is a VM sitting right on that Dell Server and has 8 cores and 8GB ram assigned to it. Right now writing to Win2012 share (VM on server) we load the 8 cores to 100% for the entire job.
Later today we will try a reverse incremental on the server and see how that performance, because out 12 and 16 bay QNAPs are very slow at creating reverse incrementals.

We started to run out of time when running reverse incrementals to NAS due to how long it took to re-write the data on the target and create the VBK file. Be careful about using NAS for your primary storage unless you do full active+incremental.

At this point i think we will not be recommending QNAP/Synology as Veeam backup targets will just go back to high disk density servers. The price difference between QNAP and Dell 720xd is about $3-4K, but you get so much more out of the server! 12-16 cores, 32-64GB RAM, can run Win2012 for proxy, backup server or deduped store on top, better IOPS, more flexible networking (FC, 10GBit, Infiniband, MPIO, TCP optimizations, etc). The only place where you take a big hit is the raw storage. With 24x3TB drives in QNAP we get 55TB usable after RAID and format, but RAID5 on NAS shows poor performance. So we would have to convert to RAID10 and that would half the storage to 30TB. At the same time RAID5 on the server gives us 30TB and better performance, so it evens it out at the end (in my eyes at least).

[tedsteenvoorden]

We recently went down the Supermicro road to build a couple of 4U high capacity storage servers and it's been a blast. You really get much more bang for your buck compared to most iSCSI / CIFS filers. Decent hardware RAID controller, tons of RAM, and you get to run Windows Server 2012.

Hi Jonathan, just to be curious, which SuperMicro 4U high capacity storage servers are you using?

[dellock6]

Ted, we are using this one:

http://www.supermicro.com/products/syst ... E1R24L.cfm

filled with LSI MegaRAID SAS 9265-8i controller and 24 Western Digitar RE4 disks.

Luca.

[Yuki]

Hi Luca,
I would like to ask you a few questions about your setup:

1)desktop drives or enterprise?
2) How's the performance with that raid adapter - any issues at all with Synthetic full or reverse incrementals?
3) I assume you are running something like Win2012 on top of the SuperMicro for dedupe features and not a hypervisor?
4) You are using these pods as proxy and repository right? Just SMB/CIFS ?
5) Any issues at all with LSI raid controller? I've heard people complain about RAID dropping and require a rebuild.

[dellock6]

1) WD enterprise grade drives (RE4)
2) never used synthetic full, only reverse. We run at about 90-100 MBs sustained writes
3) Windows 2008 R2 at the moment, no 2012 and absolutely no dedupe on the storage, only dedup coming from Veeam
4) Only Veeam repository installed on board, proxies are virtualized
5) not at the moment, but we know about those issues and we are keeping them under control. I was also thinking about 3ware for our next pods, or Adaptec

[Jonathan]

Hi Jonathan, just to be curious, which SuperMicro 4U high capacity storage servers are you using?

Hi Ted,

I've selected individual Supermicro components and assembled them myself. The main components are:

Supermicro SC846 E26-R1200B chassis
SUPERMICRO X9DR3-LN4F+ mobo
Intel Xeon E5-2620 cpu (1 per server, room for 1 more)
4 x 8GB PC3-10600 RAM (ECC, Reg)
Areca ARC-1882IX-24 RAID controller with BBWC and 4GB cache
Hitachi Ultrastar 7K4000 (12 in each server, 12 empty slots)
The onboard SAS/SATA controllers are used to drive two 120GB Intel 330 SSD's in RAID-1 for the operating system (Windows Server 2012).

As a first attempt at building something completely tailored to our needs, I think they're great. Compared to our previous standard of HP DL380 or BL460 servers, they require a bit more research and configuring hardware monitoring is slightly more time consuming, but you really get much more bang for your buck. Also, HP hasn't got a single-box high volume storage platform, so you're stuck with a Proliant DL and one or more (expensive) external SAS expander boxes. As a bonus, you get to pay a hefty premium for yesterdays rebranded Seagate or WD disks. I reckon the equivalent HP hardware would have cost 2,5 times more.

[tedsteenvoorden]

Everybody thank you for your valuable input. Based on the provided information i've decided to go for the 'pod' concept. Initially we will start with two pods. I will be using the following configuration for a pod:

Supermicro SuperStorage Server 6047R-E1R24N
2x Intel Xeon E5-2620 (6-core 2.00GHz, 15MB Cache, 7.2GT/s, 95W per cpu)
1x Kingston Technology 32GB DDR3 1333MHz Kit
24x Seagate® Constellation ES.3 SAS 6Gb/s 4-TB
1x Intel X520-SR2

The SuperStorage Server is supplied with a hardware raid controller AOC-SAS2LP-Hi8R which is based on the LSI SAS 2108 chip. These pods will functions as backup repositories and will be connected using a 10 Gbit connection. The backup proxies will be virtual machines with 4 vCPU's and 6 GB RAM. Each proxy will be handling two concurrent backup jobs.

The SuperStorage Server will run Windows Server 2008 R2 or Windows Server 2012. I haven't decided on this yet.

[Yuki]

if you are doing forward incremental - you are better off with 2012 for dedupe.

[Jonathan]

if you are doing forward incremental - you are better off with 2012 for dedupe.

+1

Server 2012 is just so damn nice to work with...

@Ted: that looks like a nice setup! The pre-built SuperStorage Server is actually almost the same as my configuration. Keep in mind it hasn't got any internal drive bays by default, so you'll have to order the 2-bay rear hotswap kit to mount your SSD.