Backup Repository Reccomendation?

[mathew.frost]

Hi all,

I'm posting here because i've spent a huge amount of time researching this and still and none the wiser what our best solution would be. I'm hoping someone here could offer some advice?

Our Current VM Environment:
•20 VMWare Virtual Servers
•Running on 3 Clustered Hosts in a vSphere Environment
•Data Storage for the VM’s is a PowerEdge MD3000i iSCSI SAN.
•A full Backup of all VM’s using Veeam is roughly 2.5 Terabytes of Data
•An incremental backup is about 80-200gb



Our current Veeam Backup solution
Comprises of a Virtual Veeam Backup Host using a QNAP TS-453U-RP NAS with 8TB of storage (4 physical Drives) in a Raid 5 configuration. This is accessed through an NFS windows share currently. This solution is proving to have several problems:

a)It’s not a fast enough backup target. A full backup takes 4/5 days, an incremental takes about 18-20 hours.

b)It’s not big enough, we’re really struggling to keep any more than 5/6 days’ worth of backup before manually deleting .vbk (and related .vib) files to allow for the next backup to complete successfully.

c)It cannot run any backups where it is required to merges files (the process is just too IO intensive and would probably take about 2 weeks to complete if we left one running). We are currently relying solely on Full Backups and 3 or 4 incremental – then another full backup.

Unfortunately our budget is not very big at all so it’s even more vital we make the right choice.

Budget: £2,500

 
Options we’ve got:

Option 1
Expand the QNAP device with a QNAP Expansion unit (QNAP REXP-1200U-RP) which can be purchased for ~£1600, then put as maybe SATA drives in as we can afford with the remaining £500. Re configure the QNAP Raid from Raid 5 to Raid 10 to increase the Write Speed.

The worry with this solution is that although we remove the Space problem we probably will not make much of a dent in the IO speed problem we’re having. Plus there is a worry that the expansion device is only connected using USB3 and there doesn’t seem to be much info on speed of access to the expansion devices available.

Option 2
Re-utilise a Redundant Server we have available left over after we virtualised our environment:
Specification
Dell PowerEdge R710
INTEL XEON E5645 PROCESSOR (2.40GHZ, 6Cores)
64GB Memory
2x 100GB SSD’s
6 SATA compatible Hard Drive Bays Available to fill.
Microsoft Windows Server 2008

We could either remove the SSD’s and have 8 available Hard Drive bays, or use the 2 SSD’s as a cache drive (or something else useful?). Then utilise the £2500 by filling the 6 Bays with the biggest and fastest hard drive we can, SAS is maybe a possibility with the budget.

This Solution would allow for a small Raid10 Environment (either 6 or 8 disks depending on whether we utilise the SSD’s or not).

I would hope that the RAM and processing power of the Server would speed things up quite considerably?

Option 3
Purchase another second hand SAN (Possibly a Dell PowerVault MD1000 15 Bay) which can be purchased for ~200. Then Put 8/10/12 or 14 disks (In Raid 10) into the SAN to give as much IO performance as possible and continue to utilise the Virtual Backup Host referencing the new iSCSI SAN.

This solution has the most space for future expansion but with the budget the SAN would not be under support/warranty.

Option 4 – Advice?
We’re incredibly open to advice on other options we may not have considered.

All 3 Options above would be cabled into the environment shown in the Network Diagram in the same location that the current “QNAP” device is.

Any help would be amazingly appreciated!!

Matt

[Homes32]

Hi Matthew,

I recently had similar issues with a QNAP TS-EC1680U-RP. My environment is pretty close to yours 20 vm's in Vsphere approx 2.5-3tb data

Bottom line is that the 7200 RPM drives are going to suck for IO no matter what but here is what I ended up going with to get semi-reasonable performance out of the QNAP.

• 10+1 2TB 7200 WD in RAID10
• iSCSI for block level storage on the QNAP instead of NFS/CIF
• Repository max concurrent tasks set to 1
• Under Job settings > Storage > Advanced Button > Storage Tab > Storage Optimization = Local Target (This gave us a HUGE performance boost as it uses large block sizes when reading/writing to the slow 7200 drives)

Using forever incremental the backup portion takes ~1-1.5hr to complete 100-200gb incremental and the merge between 1.5-2hrs to complete. otherwise you can create periodic active fulls (ie once a week) and get rid of the merge completely.

Personally if I were to do this project over again I would probably built a standalone server filled with 2TB SAS drives in a RAID10 if a SAN wasn't in the budget. the QNAP/7200 RPM drives just don't have the performance.

regards,
Homes32

[sunnydays]

Hi Matthew,

I'm pretty new to the forums and Veeam but thought I would post a reply.
I just had a couple of questions around your setup. How are the Veeam roles deployed? Are you running a single VBR / Proxy? Is it physical or Virtual? and what mode is the proxy configured to use? How many jobs do you have?
Have you checked the backup jobs to see where the bottleneck is? Example Load: Source 92% > proxy 38% > Network 4% > Target 0%
When jobs are happening you can see what the job is waiting for i.e. Waiting for backup resource to be become available

There were a couple of tweaks I have come across with our recent deployment of Veeam. Some of these might help?
- Deduplication is per job not global. Similar grouped VM's will achieve better dedupe ratios.
- A proxy has a default configuration of 4 max concurrent tasks. I found out that a "task" is actually a VMDK. So if you have 1 x VM with 4 x VMDK's, this would max out the proxy.
- A backup repository also has a default configuration of 4 max concurrent tasks, so you may want to adjust accordingly (you may need to test and tune this).
- There is a hard limit on active snapshots per datastore (I think this is 4). I wouldn't increase this too much higher but everyone's environment is different. Here is more info: http://forums.veeam.com/vmware-vsphere- ... 19711.html

Sorry for the ramblings but hope it helps!

In regards to your actual question I would lean towards the Dell PowerEdge R710 and fill the bays with something like 6 x 4TB SATA RAID5 ($200-$250 AUD per drive), giving you around 18TB capacity. This will give you more options around your backup methods (Processing Power + Capacity). Create a NIC team of 2 to 4 NICS for optimal throughput.

I found this great article about I/O profiles: http://www.virtualtothecore.com/en/veea ... torage-io/
There is also a Veeam whitepaper around Backup Repository Design and Planning: http://www.veeam.com/wp-veeam-backup-re ... mance.html

Cheers,
Sunny

[cupplesey]

Hi Matt,

My two cents

I would disagree with any SATA recommendations if its IOPS you want for the primary backup storgae, the best solution here would SAS in a RAID-DP (if supported by the NAS/SAN) setup as RAID5 is not the best choice for the high IOPS due to a higher performance overhead. You need at least RAID6 preferably RAID-DP (RAID6 + performance) and SATA is not the way to go for disk performance, its cheaper for a reason and should be used for longer term backup retention/Tier 2 Backups. As your issue will be a comprimise with the budget you have i would go for IOPS and capacity with as many NICs as possible and team them, a SAN may not be required as long as the NAS can support high performance SAS disks.

Also if you do end up buying a replacement i suggest keeping thie original device and using it for tier 2, keep 2-3 backups on the faster storage and then create a VM copy job for longer backup retention to the slower storage. There is no harm in splitting the the capacity across two or more repository strorage devices, the main backups can run at night and the copy job can then run through the day at a slower speed.

Other things to consider is to VLAN off the SAN and the QNAP backup device (if this has not already been done) to reduce unnecessary network chatter from the ESXi production network. Also how are you taking the backups, are they SAN storage snapshots or are you doing vSphere/VM level backups as this can impact the overall speed of the backup time?

Andy

[PTide]

Hi,

Another 2 cents: it might be worth checking this article when planning a backup repo configuration.

Thanks!

[jveerd1]

Please don't forget your off-site backup. With your budget I would consider a Veaam Cloud Connect provider.

[mathew.frost]

Thanks for the replies everyone! I'll try to answer the questions as best I can!

[*]iSCSI for block level storage on the QNAP instead of NFS/CIF

o We are planning to try converting the QNAP to iSCSI very soon to see how it performs in comparison.

[*]Repository max concurrent tasks set to 1

o Changed from 4 to 1

[*]Under Job settings > Storage > Advanced Button > Storage Tab > Storage Optimization = Local Target (This gave us a HUGE performance boost as it uses large block sizes when reading/writing to the slow 7200 drives)[/list]

o Changed from LAN to Target

How are the Veeam roles deployed?

o One User/Role (Administrators)

Are you running a single VBR / Proxy?

o If I understand you correctly, we’re running a single Backup Proxy (The QNAP Device)

Is it physical or Virtual?

o Our Veeam Backup server is one of our Virtual Servers

and what mode is the proxy configured to use?

o Automatic Selection

How many jobs do you have?

o 1 Job containing all VM’s.

Have you checked the backup jobs to see where the bottleneck is?

o Our Last 5 Full Backup Jobs: (These are full backups so no merging taking place)
o Srce ProxyNetTarget
o 39%49%67%46%
o 11%30%22%52%
o 39%51%57%55%
o 39%48%70%50%
o 18%34%29%40%

- Deduplication is per job not global. Similar grouped VM's will achieve better dedupe ratios.

o So should I create more, smaller jobs? What’s an example of similar VM’s? All our VM Server perform different tasks.

In regards to your actual question I would lean towards the Dell PowerEdge R710 and fill the bays with something like 6 x 4TB SATA RAID5 ($200-$250 AUD per drive), giving you around 18TB capacity. This will give you more options around your backup methods (Processing Power + Capacity). Create a NIC team of 2 to 4 NICS for optimal throughput.

o I’m leaning towards re-utilising the Dell R710 Server, but possibly going for 6x4TB SAS Drives from Western Digital (Model: WD4001FYYG).

the best solution here would SAS in a RAID-DP (if supported by the NAS)

o I think the current solution is just not going to work for us. It has no room for expansion so anything above RAID5 is not possible.

how are you taking the backups, are they SAN storage snapshots or are you doing vSphere/VM level backups as this can impact the overall speed of the backup time?

o I believe they are VM level backups. Is there some performance increase I can get from changing to SAN Storage snapshots? Bearing in mind I have no idea how to do this!


Now for a question of my own...

If I was to re-utilise the Dell Power Edge 710 Server, would there be much benefit in me keeping the 2 existing 100GB SSD's in it or would it be more beneficial for me to remove those 2 (Giving me another 2 Bays (8 in total)) to add 2 more drives to the RAID giving me 8x4TB SAS Drives in Raid 10?

[foggy]

I'd also go with Option 2 (another good topic for you for that matter). However, based on the fact that your Veeam B&R server currently holds both proxy and repository roles and also on your bottleneck stats, I'd also play with the DataMoverLocalFastPath registry key or try to offload processing load to other VMs, leaving only repository role on Veeam B&R VM.

[cupplesey]

Hi Matt

I think the current solution is just not going to work for us. It has no room for expansion so anything above RAID5 is not possible.

Maybe worth looking into a NAS/SAN that can if you are still looking to replace your existing solution. Also see if there is a firmeware upgrade that may provide more functionality for other RAID's.

I believe they are VM level backups. Is there some performance increase I can get from changing to SAN Storage snapshots? Bearing in mind I have no idea how to do this!

Yes there is an improvement in performance if your SAN can do this but is whether Veeam can support the version of your SAN snapshots in the current B&R version 8. For example, if you had something like a NetApp then the snaps are done at the volume level which reduce the overhead of the process and does not require the same freeze/thaw process that VMware does. When the snaps are done a the volume level they are then transferred by Veeam to your backup repository. While your current hardware may not support this then i would highly recommend researching this and what is possible with your budget as this may mean the backup repository does not need such high IOPS to do the backup as they will transfer in the background and therefore a lesser impact on the production server performance.

Now for a question of my own...

If I was to re-utilise the Dell Power Edge 710 Server, would there be much benefit in me keeping the 2 existing 100GB SSD's in it or would it be more beneficial for me to remove those 2 (Giving me another 2 Bays (8 in total)) to add 2 more drives to the RAID giving me 8x4TB SAS Drives in Raid 10?

I most cases such use of the SSD's are for flash pools for faster data access/buffers/caches but if you are looking for capacity with better RAID protection and high performance then yes replacing them would be a good idea. Also correct me if im wrong but you mentioned the Dell 710 was SATA not SAS? Also RAID 6 maybe better as you lose less disks/capacity but this is down to your exact needs and other devices setups.

Option 2
Re-utilise a Redundant Server we have available left over after we virtualised our environment:
Specification
Dell PowerEdge R710
INTEL XEON E5645 PROCESSOR (2.40GHZ, 6Cores)
64GB Memory
2x 100GB SSD’s
6 SATA compatible Hard Drive Bays Available to fill.
Microsoft Windows Server 2008

Either way hope you can do what you need with what you have

Andy

[foggy]

For example, if you had something like a NetApp then the snaps are done at the volume level which reduce the overhead of the process and does not require the same freeze/thaw process that VMware does.

That's not completely true, since Veeam B&R still triggers a VMware snapshot for processed VMs, however it is deleted right after the storage snapshot of the volume holding VMs is created, resulting in less painful commit.

[cupplesey]

I stand corrected