Comprehensive data protection for all workloads
Post Reply
mathew.frost
Lurker
Posts: 2
Liked: never
Joined: Aug 17, 2015 3:19 pm
Contact:

Backup Repository Reccomendation?

Post by mathew.frost »

Hi all,

I'm posting here because i've spent a huge amount of time researching this and still and none the wiser what our best solution would be. I'm hoping someone here could offer some advice?

Our Current VM Environment:
•20 VMWare Virtual Servers
•Running on 3 Clustered Hosts in a vSphere Environment
•Data Storage for the VM’s is a PowerEdge MD3000i iSCSI SAN.
•A full Backup of all VM’s using Veeam is roughly 2.5 Terabytes of Data
•An incremental backup is about 80-200gb

Image

Our current Veeam Backup solution
Comprises of a Virtual Veeam Backup Host using a QNAP TS-453U-RP NAS with 8TB of storage (4 physical Drives) in a Raid 5 configuration. This is accessed through an NFS windows share currently. This solution is proving to have several problems:

a)It’s not a fast enough backup target. A full backup takes 4/5 days, an incremental takes about 18-20 hours.

b)It’s not big enough, we’re really struggling to keep any more than 5/6 days’ worth of backup before manually deleting .vbk (and related .vib) files to allow for the next backup to complete successfully.

c)It cannot run any backups where it is required to merges files (the process is just too IO intensive and would probably take about 2 weeks to complete if we left one running). We are currently relying solely on Full Backups and 3 or 4 incremental – then another full backup.

Unfortunately our budget is not very big at all so it’s even more vital we make the right choice.

Budget: £2,500


Options we’ve got:

Option 1
Expand the QNAP device with a QNAP Expansion unit (QNAP REXP-1200U-RP) which can be purchased for ~£1600, then put as maybe SATA drives in as we can afford with the remaining £500. Re configure the QNAP Raid from Raid 5 to Raid 10 to increase the Write Speed.

The worry with this solution is that although we remove the Space problem we probably will not make much of a dent in the IO speed problem we’re having. Plus there is a worry that the expansion device is only connected using USB3 and there doesn’t seem to be much info on speed of access to the expansion devices available.

Option 2
Re-utilise a Redundant Server we have available left over after we virtualised our environment:
Specification
Dell PowerEdge R710
INTEL XEON E5645 PROCESSOR (2.40GHZ, 6Cores)
64GB Memory
2x 100GB SSD’s
6 SATA compatible Hard Drive Bays Available to fill.
Microsoft Windows Server 2008

We could either remove the SSD’s and have 8 available Hard Drive bays, or use the 2 SSD’s as a cache drive (or something else useful?). Then utilise the £2500 by filling the 6 Bays with the biggest and fastest hard drive we can, SAS is maybe a possibility with the budget.

This Solution would allow for a small Raid10 Environment (either 6 or 8 disks depending on whether we utilise the SSD’s or not).

I would hope that the RAM and processing power of the Server would speed things up quite considerably?

Option 3
Purchase another second hand SAN (Possibly a Dell PowerVault MD1000 15 Bay) which can be purchased for ~200. Then Put 8/10/12 or 14 disks (In Raid 10) into the SAN to give as much IO performance as possible and continue to utilise the Virtual Backup Host referencing the new iSCSI SAN.

This solution has the most space for future expansion but with the budget the SAN would not be under support/warranty.

Option 4 – Advice?
We’re incredibly open to advice on other options we may not have considered.

All 3 Options above would be cabled into the environment shown in the Network Diagram in the same location that the current “QNAP” device is.

Any help would be amazingly appreciated!!

Matt
Homes32
Enthusiast
Posts: 32
Liked: 15 times
Joined: Oct 29, 2013 6:35 pm
Full Name: Jonathan Holmgren
Contact:

Re: Backup Repository Reccomendation?

Post by Homes32 » 1 person likes this post

Hi Matthew,

I recently had similar issues with a QNAP TS-EC1680U-RP. My environment is pretty close to yours 20 vm's in Vsphere approx 2.5-3tb data

Bottom line is that the 7200 RPM drives are going to suck for IO no matter what but here is what I ended up going with to get semi-reasonable performance out of the QNAP.
  • 10+1 2TB 7200 WD in RAID10
  • iSCSI for block level storage on the QNAP instead of NFS/CIF
  • Repository max concurrent tasks set to 1
  • Under Job settings > Storage > Advanced Button > Storage Tab > Storage Optimization = Local Target (This gave us a HUGE performance boost as it uses large block sizes when reading/writing to the slow 7200 drives)
Using forever incremental the backup portion takes ~1-1.5hr to complete 100-200gb incremental and the merge between 1.5-2hrs to complete. otherwise you can create periodic active fulls (ie once a week) and get rid of the merge completely.

Personally if I were to do this project over again I would probably built a standalone server filled with 2TB SAS drives in a RAID10 if a SAN wasn't in the budget. the QNAP/7200 RPM drives just don't have the performance.

regards,
Homes32
sunnydays
Influencer
Posts: 11
Liked: 1 time
Joined: Jun 30, 2015 2:27 am
Full Name: Sunny
Contact:

Re: Backup Repository Reccomendation?

Post by sunnydays »

Hi Matthew,

I'm pretty new to the forums and Veeam but thought I would post a reply.
I just had a couple of questions around your setup. How are the Veeam roles deployed? Are you running a single VBR / Proxy? Is it physical or Virtual? and what mode is the proxy configured to use? How many jobs do you have?
Have you checked the backup jobs to see where the bottleneck is? Example Load: Source 92% > proxy 38% > Network 4% > Target 0%
When jobs are happening you can see what the job is waiting for i.e. Waiting for backup resource to be become available

There were a couple of tweaks I have come across with our recent deployment of Veeam. Some of these might help?
- Deduplication is per job not global. Similar grouped VM's will achieve better dedupe ratios.
- A proxy has a default configuration of 4 max concurrent tasks. I found out that a "task" is actually a VMDK. So if you have 1 x VM with 4 x VMDK's, this would max out the proxy.
- A backup repository also has a default configuration of 4 max concurrent tasks, so you may want to adjust accordingly (you may need to test and tune this).
- There is a hard limit on active snapshots per datastore (I think this is 4). I wouldn't increase this too much higher but everyone's environment is different. Here is more info: http://forums.veeam.com/vmware-vsphere- ... 19711.html

Sorry for the ramblings but hope it helps!

In regards to your actual question I would lean towards the Dell PowerEdge R710 and fill the bays with something like 6 x 4TB SATA RAID5 ($200-$250 AUD per drive), giving you around 18TB capacity. This will give you more options around your backup methods (Processing Power + Capacity). Create a NIC team of 2 to 4 NICS for optimal throughput.

I found this great article about I/O profiles: http://www.virtualtothecore.com/en/veea ... torage-io/
There is also a Veeam whitepaper around Backup Repository Design and Planning: http://www.veeam.com/wp-veeam-backup-re ... mance.html

Cheers,
Sunny
cupplesey
Enthusiast
Posts: 50
Liked: 3 times
Joined: Aug 06, 2015 8:08 am
Full Name: Andy C
Contact:

Re: Backup Repository Reccomendation?

Post by cupplesey »

Hi Matt,

My two cents

I would disagree with any SATA recommendations if its IOPS you want for the primary backup storgae, the best solution here would SAS in a RAID-DP (if supported by the NAS/SAN) setup as RAID5 is not the best choice for the high IOPS due to a higher performance overhead. You need at least RAID6 preferably RAID-DP (RAID6 + performance) and SATA is not the way to go for disk performance, its cheaper for a reason and should be used for longer term backup retention/Tier 2 Backups. As your issue will be a comprimise with the budget you have i would go for IOPS and capacity with as many NICs as possible and team them, a SAN may not be required as long as the NAS can support high performance SAS disks.

Also if you do end up buying a replacement i suggest keeping thie original device and using it for tier 2, keep 2-3 backups on the faster storage and then create a VM copy job for longer backup retention to the slower storage. There is no harm in splitting the the capacity across two or more repository strorage devices, the main backups can run at night and the copy job can then run through the day at a slower speed.

Other things to consider is to VLAN off the SAN and the QNAP backup device (if this has not already been done) to reduce unnecessary network chatter from the ESXi production network. Also how are you taking the backups, are they SAN storage snapshots or are you doing vSphere/VM level backups as this can impact the overall speed of the backup time?

Andy
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup Repository Reccomendation?

Post by PTide »

Hi,

Another 2 cents: it might be worth checking this article when planning a backup repo configuration.

Thanks!
jveerd1
Service Provider
Posts: 52
Liked: 10 times
Joined: Mar 12, 2013 9:12 am
Full Name: Joeri van Eerd
Contact:

Re: Backup Repository Reccomendation?

Post by jveerd1 »

Please don't forget your off-site backup. With your budget I would consider a Veaam Cloud Connect provider.
mathew.frost
Lurker
Posts: 2
Liked: never
Joined: Aug 17, 2015 3:19 pm
Contact:

Re: Backup Repository Reccomendation?

Post by mathew.frost »

Thanks for the replies everyone! I'll try to answer the questions as best I can!
Homes32 wrote: [*]iSCSI for block level storage on the QNAP instead of NFS/CIF
o We are planning to try converting the QNAP to iSCSI very soon to see how it performs in comparison.
Homes32 wrote: [*]Repository max concurrent tasks set to 1
o Changed from 4 to 1
Homes32 wrote: [*]Under Job settings > Storage > Advanced Button > Storage Tab > Storage Optimization = Local Target (This gave us a HUGE performance boost as it uses large block sizes when reading/writing to the slow 7200 drives)[/list]
o Changed from LAN to Target
sunnydays wrote: How are the Veeam roles deployed?
o One User/Role (Administrators)
sunnydays wrote: Are you running a single VBR / Proxy?
o If I understand you correctly, we’re running a single Backup Proxy (The QNAP Device)
sunnydays wrote: Is it physical or Virtual?
o Our Veeam Backup server is one of our Virtual Servers
sunnydays wrote: and what mode is the proxy configured to use?
o Automatic Selection
sunnydays wrote: How many jobs do you have?
o 1 Job containing all VM’s.
sunnydays wrote: Have you checked the backup jobs to see where the bottleneck is?
o Our Last 5 Full Backup Jobs: (These are full backups so no merging taking place)
o Srce ProxyNetTarget
o 39%49%67%46%
o 11%30%22%52%
o 39%51%57%55%
o 39%48%70%50%
o 18%34%29%40%
sunnydays wrote: - Deduplication is per job not global. Similar grouped VM's will achieve better dedupe ratios.
o So should I create more, smaller jobs? What’s an example of similar VM’s? All our VM Server perform different tasks.
sunnydays wrote: In regards to your actual question I would lean towards the Dell PowerEdge R710 and fill the bays with something like 6 x 4TB SATA RAID5 ($200-$250 AUD per drive), giving you around 18TB capacity. This will give you more options around your backup methods (Processing Power + Capacity). Create a NIC team of 2 to 4 NICS for optimal throughput.
o I’m leaning towards re-utilising the Dell R710 Server, but possibly going for 6x4TB SAS Drives from Western Digital (Model: WD4001FYYG).

cupplesey wrote: the best solution here would SAS in a RAID-DP (if supported by the NAS)
o I think the current solution is just not going to work for us. It has no room for expansion so anything above RAID5 is not possible.
cupplesey wrote: how are you taking the backups, are they SAN storage snapshots or are you doing vSphere/VM level backups as this can impact the overall speed of the backup time?
o I believe they are VM level backups. Is there some performance increase I can get from changing to SAN Storage snapshots? Bearing in mind I have no idea how to do this!


Now for a question of my own...

If I was to re-utilise the Dell Power Edge 710 Server, would there be much benefit in me keeping the 2 existing 100GB SSD's in it or would it be more beneficial for me to remove those 2 (Giving me another 2 Bays (8 in total)) to add 2 more drives to the RAID giving me 8x4TB SAS Drives in Raid 10?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Repository Reccomendation?

Post by foggy »

I'd also go with Option 2 (another good topic for you for that matter). However, based on the fact that your Veeam B&R server currently holds both proxy and repository roles and also on your bottleneck stats, I'd also play with the DataMoverLocalFastPath registry key or try to offload processing load to other VMs, leaving only repository role on Veeam B&R VM.
cupplesey
Enthusiast
Posts: 50
Liked: 3 times
Joined: Aug 06, 2015 8:08 am
Full Name: Andy C
Contact:

Re: Backup Repository Reccomendation?

Post by cupplesey »

Hi Matt
mathew.frost wrote: I think the current solution is just not going to work for us. It has no room for expansion so anything above RAID5 is not possible.


Maybe worth looking into a NAS/SAN that can if you are still looking to replace your existing solution. Also see if there is a firmeware upgrade that may provide more functionality for other RAID's.
mathew.frost wrote: I believe they are VM level backups. Is there some performance increase I can get from changing to SAN Storage snapshots? Bearing in mind I have no idea how to do this!
Yes there is an improvement in performance if your SAN can do this but is whether Veeam can support the version of your SAN snapshots in the current B&R version 8. For example, if you had something like a NetApp then the snaps are done at the volume level which reduce the overhead of the process and does not require the same freeze/thaw process that VMware does. When the snaps are done a the volume level they are then transferred by Veeam to your backup repository. While your current hardware may not support this then i would highly recommend researching this and what is possible with your budget as this may mean the backup repository does not need such high IOPS to do the backup as they will transfer in the background and therefore a lesser impact on the production server performance.
mathew.frost wrote: Now for a question of my own...

If I was to re-utilise the Dell Power Edge 710 Server, would there be much benefit in me keeping the 2 existing 100GB SSD's in it or would it be more beneficial for me to remove those 2 (Giving me another 2 Bays (8 in total)) to add 2 more drives to the RAID giving me 8x4TB SAS Drives in Raid 10?
I most cases such use of the SSD's are for flash pools for faster data access/buffers/caches but if you are looking for capacity with better RAID protection and high performance then yes replacing them would be a good idea. Also correct me if im wrong but you mentioned the Dell 710 was SATA not SAS? Also RAID 6 maybe better as you lose less disks/capacity but this is down to your exact needs and other devices setups.
mathew.frost wrote: Option 2
Re-utilise a Redundant Server we have available left over after we virtualised our environment:
Specification
Dell PowerEdge R710
INTEL XEON E5645 PROCESSOR (2.40GHZ, 6Cores)
64GB Memory
2x 100GB SSD’s
6 SATA compatible Hard Drive Bays Available to fill.
Microsoft Windows Server 2008
Either way hope you can do what you need with what you have

Andy
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Repository Reccomendation?

Post by foggy »

cupplesey wrote:For example, if you had something like a NetApp then the snaps are done at the volume level which reduce the overhead of the process and does not require the same freeze/thaw process that VMware does.
That's not completely true, since Veeam B&R still triggers a VMware snapshot for processed VMs, however it is deleted right after the storage snapshot of the volume holding VMs is created, resulting in less painful commit.
cupplesey
Enthusiast
Posts: 50
Liked: 3 times
Joined: Aug 06, 2015 8:08 am
Full Name: Andy C
Contact:

Re: Backup Repository Reccomendation?

Post by cupplesey »

I stand corrected :-)
Post Reply

Who is online

Users browsing this forum: Amazon [Bot], Semrush [Bot] and 225 guests