Attempting to build local storage repository

[bbricker]

For a long time we'd been running our B&R backups against an old EMC CX SAN but it was horribly slow for weekly transforms (24+ hours to do 1TB), and it was running out of space. So since I had available space, I moved the backups to our main production EMC VNX 5300 SAN . Even though Veeam is writing to RDM LUNs that I specified in the VNX to only be lowest-tier-available (NL-SAS), they run really fast. Those same transforms now complete in just a few hours!

But writing my primary backup to the production SAN where all the other data is obviously not ideal even though we also copy the jobs to a cheap iSCSI JBOD at a remote site. If we were to have a major failure of our VNX SAN, it would take forever to get the multiple terabytes of data restored from the remote JBOD, even if we physically moved it back to the data center. Not only that, using 10-15TB of space on my VNX is quite costly!

So I decided on a new course of action and bought 8 x 3TB drives for one of our Dell R720 ESXi hosts, and figured I would just write primary Veeam backups to it. I took the easy route and piled them all into one big RAID5 group on the host, then made a single VMFS volume and created a big 18TB vmdk to mount to my Veeam B&R vm server. I started copying the current backup data from the VNX to the local storage and was very disappointed to see it only copying about 100-130MB/s. Sometimes as slow as 95MB/s. Granted they are only SATA (NL-SAS?) disks in the Dell server, but shouldn't they theoretically achieve 300MB/s or more?

I know a lot of folks here are going the route of using local storage to make affordable backup repositories. Is this the normal kind of performance I should see? I'm afraid that kind of speed just isn't going to be adequate if we have to do a major restore. It might even be slower to write nightly backups to versus our VNX SAN. (about to test that)

thanks for any input or thoughts!
Ben

[dellock6]

Hi Ben,
you are saying you are copying the backup files, so it means it's a file copy operation inside Windows VM, not a Veeam backup job, right?
There are some additional information missing to have a complete idea of what's happening, but those are 7200 rpm disks, and raid5 has a write "penalty" you need to consider. Also, I don't know what kind of raid controller you are using and how it is configured (read-ahead, cachins is enabled or not, block size and so on).

Any chance the vmdk you created is thin? The size increase on a thin disk has many metadata updates in VMFS file system, thus reducing the speed of the operations. I would honestly go for a much simpler path: install windows directly onto the Dell server, and install the repository role on it if you want to keep the virtual proxies. In this way there are 2 less abstraction layers (vmfs + vmdk) between the volume and the windows file system. And if something happens to the vmware infrastructure as you said, your repository is a windows machine separated from production, you can quickly install VBR server on it and start restoring.

[bbricker]

Hi Ben,
you are saying you are copying the backup files, so it means it's a file copy operation inside Windows VM, not a Veeam backup job, right?
There are some additional information missing to have a complete idea of what's happening, but those are 7200 rpm disks, and raid5 has a write "penalty" you need to consider. Also, I don't know what kind of raid controller you are using and how it is configured (read-ahead, cachins is enabled or not, block size and so on).

Any chance the vmdk you created is thin? The size increase on a thin disk has many metadata updates in VMFS file system, thus reducing the speed of the operations. I would honestly go for a much simpler path: install windows directly onto the Dell server, and install the repository role on it if you want to keep the virtual proxies. In this way there are 2 less abstraction layers (vmfs + vmdk) between the volume and the windows file system. And if something happens to the vmware infrastructure as you said, your repository is a windows machine separated from production, you can quickly install VBR server on it and start restoring.

Luca- thanks for the reply. Yes I meant a plain file copy within Windows, not a Veeam backup job. To answer your questions though- Yes they are 7200rpm drives, connected to a good Dell controller: a 512MB H710 PERC. The RAID5 array I made looks to have taken the default 512byte block size and 64K stripe size. It's using Adaptive Read Ahead and Write Back for read/write policies. And yes the vmdk I created is thin. What do you suggest for block/strip/read/write settings? I haven't put any real data on this yet, so I could blow it away and start again.

As far as making it a bare-metal Windows box, I thought about that too. I wouldn't do it with this *particular* host just because it is a top of the line machine with 256GB of ram and 16-core of E5-2690 cpu's that I need to use for running 20+ vm's. But, I could source me a used R720 off eBay that has less CPU/mem specs and just use it as a backup box to put the disks in. I'd go this route if I thought it would be significantly better performance. Do you think I can do much better just by tweaking the items just mentioned in the way the local array was built?

[dellock6]

Since the machine is already in place, I would first try to recreate it as a bare-metal repository, without touching other parameters (always change only one value at the time, otherwise you cannot understand which one made the change for good or bad...). Controller parameters are ok and also blocks and stripe. One way to have even better speed is to move from R5 to R10, so you remove completely the write penalty (especially with Reversed incremental we do 1 read and 2 writes per saved block).

[bbricker]

HI Luca- I figured I would give you an update. I stuck with the B&R vm that is the all in one backup/repository/proxy server. I thought the ~100MB/s local vm disk performance was going to hurt us, but it actually performs just fine, in fact it's not much different than before when it was reading/writing to the VNX SAN (except transforms do take a bit longer to complete now). But as you mentioned, I agree this is not the best scenario, particularly since all the backup data is on an ESX host that is highly utilized and part of the vSphere infrastructure. So I'd like to get your thoughts on an "ideal" setup if you don't mind lending you expertise.

We have our ~80 production vm's backed up over 4 jobs. Of those, 3 run each night, and 1 that is a weekend-only job. Of the first 3 jobs, the 1st has about 30+ vm's that are mostly small application/terminal servers types that we don't need to go back to restore data from. So that 1st job just does reverse incremental and only keeps 2 priors. It's total vbk/vbr use is less than 400GB. The next 2 jobs run 5 days a week as forward incremental and do a full transform on Saturday night, keeping 15 days prior. These are mostly SQL, file, mail, etc, vm servers. There are about 10 vms in each and I separated them mainly because their weekend transform-fulls are about 1.3TB each and when it was one job it was getting hard to manage the large files that were getting close to 3TB. Then as I said, the final 4th job is only run weekly, and there are about 30 vm's in it. It is forward incremental and it's vbk/vibs are about 800GB. We keep 4 priors.

In addition to all that, we have robocopy scripts to copy the backup data after completion to a remote JBOD. We do a bit of manual pruning on the other side and for example only keep the full transform vbk's from the two big 1.3TB jobs after a week. I know I need to move that to use the grandfather-father-son copy function that B&R7 has now, but I haven't gotten around to it. I assume if i did that, I would probably benefit/need a proxy at the remote site to help facilitate that? If not, my next comment might make it more necessary...

In the very near future we will replace that remote JBOD with a refurb EMC CX4 SAN and add some old production ESX hosts, so we can have more of a full DR site with replication instead of just backups. This site has a gigabit connection to it from the prod site. We have considered using Veeam for the replication but haven't played with it. Nor have we tested SAN-2-SAN replication but are partially getting the EMC CX so we have that option with our prod VNX. I have tested vSphere Replication with decent success, but it was only on a small scale and it was over a year ago, so not sure what improvements have been address since then. I'm not sure of the pros and cons of Veeam versus vSphere replication or SAN-2-SAN. And I'm not sure how nightly/weekly backups play into B&R also running replication.

I know that is a lot, but what are your thoughts? It would be greatly appreciated!

thanks,
Ben

PS: is it true that if we move to a physical B&R server(s), that even if it's FC SAN attached, restores still go over the network, and not the SAN? If so, is that a benefit to sticking with a vm as the B&R server? (aside from having 10GB network)

[foggy]

PS: is it true that if we move to a physical B&R server(s), that even if it's FC SAN attached, restores still go over the network, and not the SAN? If so, is that a benefit to sticking with a vm as the B&R server? (aside from having 10GB network)

Yes, it's not possible to use direct SAN for restores, however, you can restore via hotadd if there's a virtual proxy available on the production host.

[dellock6]

Hi Ben,
sorry for delay but I was away for a conference.
Well, I would keep both backups and add replications, they are both valuable solutions for a complete data protections, and they do solve two different issues.

For backups, you are totally right in thinking about switching to backup copy jobs, but instead of using GFS I would try first normal rotation if the retention period is not long. Also because GFS is only made with full backup files, so you will need a bigger storare than a simple rotation. Up to you to evaluate retention and thus the final size of the remote storage you will need.

About replica, yes you cannot touch the same VM at the same time with both backup and replica. But even if the two jobs kick in at the same time, the second one will wait for the VM to be freed by the first job. And with version 8 you will have an additional possiblity we already listed in the new features that are coming (http://go.veeam.com/data-loss-avoidance ... hancements) that is replication from backup, so you will only have to touch the production storage once, and do replica from the backups you just created

Luca.