How would you configure this environment?

[WellMark]

I've been testing, and researching various backup solutions for a few months, and my brain is fried. I'd really appreciate some help, if someone can spare the time. Our entire IT team left, and I've been dubbed the "backup person". I was trying to work with NetBackup, but I've had it, and want to setup an all new backup solution.

I am looking for some recommendations on configuring the backup environment for my work. I will be replacing our old backup software with Veeam. I've never done this before, and would love some advice from the more experienced. I've been trying to research, but I've come across a few concerns, and since I'll be able to set this up from scratch, I'd like to do it right to begin with.

A lot of this comes from problems I encountered while testing Veeam. When testing between a Windows virtual server running Veeam backing up to another Windows virtual server shared drive, we had some very slow backups and restores with Processing rates of 2 - 19MB/s. The storage was a proxy, using Virtual Appliance. Although the backups were configured to use HotAdd, they kept switching to Network. Maybe some suggestions will help when I setup the production environment?

I've seen several documents about Thin vs Thick provisioned datastores, creating a separate datastore for restoring the virtual machines, using RAID or not, backing up to the SAN first, iSCSI being slow or fast? Some information may be out of date, but it’s hard to tell.



So, How would you configure this environment for Veeam Backup and Replication 9

CURRENT ENVIRONMENT NEEDING BACKUPS:
•VMWare/vCenter
•5 ESXi 5.5 hosts (9 sockets)
•50 Virtual Machines all running Windows
•Virtual Windows Server 2008 R2 with Informix databases
•Virtual Windows Server 2003 SP2 with SQL Server 2005
•Virtual Windows Server 2008 R2 with SQL Server 2008 & SharePoint 2007
•We currently backup approximately 2-3TB

HARDWARE:
•2x 1Gb switches. The ESXi’s, SAN, and Dell PowerEdge are all in the same room.
•Dell PowerEdge R815 for Storage with 6x 2TB Hard Drives. Planning on installing Windows 2012 Standard.
•EMC2 VNXe3300 w/8TB (almost full). Looks like the existing datastores are Thin provisioned. Hopefully being replaced with an EMC VNXe3200 w/Fast Cache 17TB of storage / 3600 IOPS + 10000 supplemental IOPS in the next few months
•We have a DR site where the VNXe SAN is being replicated. The old backup solution is sending monthly backups to a couple of virtual machines at that location. The network connectivity is not great between the sites (approximately 20Mb/s). These backups can take several (to 30) hours to complete.


OTHER QUESTIONS I CAN THINK OF:
1.Would you set the physical Dell PowerEdge as a Proxy? Should there be other virtual proxies (Windows servers) for each of the ESXi hosts? Do they need any particular minimum hardware configurations?


2.If we create new Datastores, should we configure them as Thin, Thick, Lazy, Eager?


3.Should we use RAID on the Dell PowerEdge? If so, RAID 6 or 10?


4.Is there any benefit of using Hyper-V on the Dell PowerEdge?


5.Would you install Veeam backup server on the Dell storage server or as a virtual machine pointing to the physical storage?


6.Is there a physical or other way to improve the connection between the physical storage server and the SAN? Would that help?


7.Any suggestions? Even hardware changes?


this is my first post here, hopefully I put this in the right place

[DaveWatkins]

Are you iSCSI or FC connected to your SAN? I assuming the 815 is going to be the Veeam Repository where all your backups go to?

1. It would depend if it can get block level access to the SAN (FC or iSCSI access direct to the LUN's). If it can, then definitely use it as a proxy, possibly the only proxy
2. Datastores themselves aren't typically provisioned like that, at least not at the VMWare level. I think you're talking about the VMDK's when provisioning new VM's, personally I use Think with Lazy Zero, but plenty of people thin provision. I prefer to know exactly how much free space I have on my LUN's
3. "It depends". The things it depends on is what sort of backup type you're going to be doing, and how much space you're going to need. If you're going to do reverse incrementals, then you're going to want IOPs, you don't mention if the disks are SAS, NL-SAS or SATA which is the other driving factor. If you don't really see your backup size increasing much from the 2-3TB you do currently then you could easily run RAID10 and still have plenty of space and know the performance is going to be the best it can, if you want more space, then you'll be trading IOP's. If they are 7200RPM disks then RAID6 can seem pretty slow at times.
4. Not if you can connect it to the SAN directly. Putting a virtualization layer will stop you being able to do that, also, for what perceived benefit?
5. I like having it as a VM personally, you can replicate it to another site easily and it stays up if you need ot do hardware maintenance
6. More of faster links. If your SAN if FC connected then I doubt that will be a problem anyway. If it's iSCSI then going to 10Gb is the obvious one, but adding more network links might help (depending on how many you already have)

If you get the Ent Plus license you'll have access to the WAN Accelerator, which will allow you to do some backups or replication over the WAN link, but if you're already replicating the whole SAN, that may not be necessary anyway

[WellMark]

Thank you Dave for all your information!!!

Yes, the R815 will be the Repository / Storage for backups.

The hard drives will be 6x 2TB Seagate Enterprise Capacity 6GB/s SATA 7200RPM

1. Right now the physical R815 is not connected to the SAN. I imagine we will be using iSCSI, since we don't have Fiber (not sure how much that would cost to setup?).

2. I had to go search for the topic on Thin/Thick/block zero stuff. This is the thread I was thinking of: vmware-vsphere-f24/slow-restore-speed-2 ... 92-60.html
This is what got me wondering about how to properly configure everything, to try and prevent the slow restores I was getting during testing.

3. I don't see our backups growing too much. I kind of hate to lose the storage space with RAID 10 (6TB) vs RAID 6 (10TB) - if I did the math right. Is there really a noticeable difference in backup/restore speeds?

4. That was my question, lol. Some other backup solution person was suggesting Hyper-V and then present the storage to the hypervisor, or something. Wasn't sure.

5. That's what I was originally thinking, but I was wondering if the Backup Server software was running on the R815 if it would process faster using those resources?

Really appreciate your time on this Dave, many thanks!

[DaveWatkins]

1. FC will cost a lot if you don't have it, it's nice to have but expensive and these days 10Gb iSCSI can compete in most situations.
2. There really is, especially on 7200 RPM drives. RAID 6 (like RAID 5) cuts your write speed down to the speed of a single disk as while you can stripe the writes across all the data disks, you have to wait for the parity calculation and for that to be written, so the calculation isn't the bottleneck so much as how quickly you can get that block written to disk, but that's still a single disk. In the case of RAID 6 you're waiting for 2 parity blocks to be written to 2 disks, that happens in parallel but you've still got to wait for it to happen before you get he OK. RAID 10 you get the full speed of half your total disks. Read speed would probably actually favor the RAID 6 array as there is one more stripe disk.

In saying all that, it may not be an issue for you depending on the speed of your SAN and how fast you can get the data off it. Just be aware reverse incremental backups consume many more IOPS and doing an Instant VM Recovery from the Repo will perform significantly better off a RAID 10. You might decide that's a price worth paying to get the additional 4TB though.

4. Without knowing why they suggested it it's hard to say for sure, but I can't see a good reason to do so.

5. No, the Backup Server is basically a scheduling agent and co-ordinator. The default install loads everything onto it but it's better to move all the other roles off. Move proxy and repository off to your 815 and let the backup server VM manage those roles on the other box. That way you can easily backup or replicate your master server and keep the inventory/catalogue and job settings safe without having to backup the repo or work out how to backup just the master components on a physical server.

So in summary, setup a basic VM for the Veeam Backup Server, remove/disable the proxy role and get your 815 iSCSI connected to your SAN and with the LUN's shared to the 815. Make sure not to ever initialize them on the windows host (it will ask you about that when you go into Disk Management, say NO). Use as many NIC's as you can and MPIO to get better than single Gb throughput. Load the proxy and repo roles on this server. If you're planning on using the WAN acceleration (and have the license) and have all 4 CPU's in the 815 it'll likely handle that job without issue too.

What did Veeam show as the bottleneck for your slow restores? My guess would be the problem is your almost full current SAN, so the bottleneck would be "target" for a restore. A SAN thats almost full is almost always also getting way more IOP's than it was designed for and struggling