Replicas from backups, and too many restore points

[roaima]

I'm trying to get my head around a reasonable backup, replica, and archive solution given finite disk storage constraints.

Background:

I have a Production datacentre operated through ISP 1 and a smaller DR datacentre operated through ISP 2. I also have a large Storage server from ISP 1 in a separate location to my Production datacentre that I'm using for long-term GFS archives. My only true downtime for the Production datacentre is across the weekend (we run 24x5), so I need to try to avoid impacting the Production datacentre during the week more than is necessary. We have around 15-20 VM Guests in Production of which around 10 are replication candidates.

In the unlikely event that ISP 1 disappears completely we can cope with losing our archives. It's not great, but it's bearable. I don't have any off-line storage and providing some is not a viable option in our situation. My protection is that the Storage server is read-only for everyone/everything except Veeam B&R and our administrative ssh logins (it's based on Proxmox and Linux, rather than vSphere and Windows), and the Veeam archives themselves have periodic snapshots taken from within the target filesystem.


Requirement:

My target replication frequency from Production to DR is every two hours. I have been advised that to avoid hitting our production site with multiple I/O requests for the same sets of Guest changes I could take an off-site backup every two hours and use that to generate the replicas.

Disk space permitting, I'd like to keep at least five days of backups in my DR centre, but since I have replicas I don't think I need more than one or two backups every day. However, I don't see how I can achieve this while using these backups to update replicas every two hours.


Questions:

1. Is it better to run one repository containing five days of backups at 2 hourly intervals, or two repositories on the same server, one holding a day of 2 hourly backups and the other holding four days of 12 hourly backups?
2. If I generated a few backups on-site, stored on a Veeam Proxy, would it be better (more efficient) to copy those off-site, or just to connect directly to the Production VMware from the DR datacentre as if the on-site backups didn't exist? Both approaches would require two I/O hits per VM guest disk, wouldn't they?
3. What is the difference between Veeam Proxy running on Windows and the Veeam "data mover" utility that runs on demand on a Linux-based storage server?
4. I understand that good practice is to group multiple VMs per backup/replica job. What recommendations are there with regard to choosing which VMs to group together in a single job?

Thanks,
Chris

[PTide]

Hi,

1. That largely depends on your backup RPO requirements. Also I cannot grasp the idea behind having two separate repos on the same server instead of one, what is your plan?

Another concern is that having two separate backup jobs (12 and 2hrs) for the same VMs might lead to both jobs trying to back up (i.e. take a snapshot) of the same VM simultaneously which is not a good thing from I/O perspective.

2. Having a short on-site backup chain sounds like a good idea to me. You can run a backup copy job on it (backup copy won't introduce any production I/O hit) to transfer the backups offsite, and perform replica from backup later. Or you can perform replica from the onsite backup directly without copying it at all.

3. Veeam Proxy is a helper machine that reads data from production storage, processes it, and transmits it further to the final destination (repository). Linux hosts can only serve as repositories at this moment.

4. Generally, the best deduplication ratio can be achieved when VMs with similar operating systems are grouped together.

Thanks!

[roaima]

Hi,

Thanks for the comments. I'll keep the numbering so we can track which comments apply where.

1. I have been advised by Veeam Support that I can reduce the load on my Production VMware servers by creating off-site Replicas from my off-site Backups. (Especially as "off-site" in this context is the same server in my DR datacentre.) But to have two-hourly Replicas I'm obviously going to need to have two-hourly Backups. I'm trying to balance that against my desire to maintain five days worth of twice-daily backups in my off-site DR datacentre.

I don't know which is better:
* to run two repositories in that same off-site datacentre, one containing "a few" two-hourly backups (from which I generate my replicas) and one containing five days of 2x backups/day
* to run one repository containing 5 days of two-hourly backups, i.e. 60 restore points

2. I inadvertently omitted to mention that all my Veeam Proxy servers are virtual (and I know this isn't Best Practice, but it's what I have). I don't have direct access to the SAN in my Production datacentre, but I am able to hot-add disks. I had assumed that in this scenario asking the on-site Proxy for copies of its backups would generate exactly the same I/O load on my Production VMware server as hitting the VMware servers directly. Is this an incorrect assumption?

3. Would it be more efficient (probably in terms of data transfer) to run up a Veeam Proxy as a Windows VM on the Storage server? At the moment I have just the Linux-based data mover. What would installing a Veeam Proxy gain me?

4. Thank you

Regards,
Chris

[PTide]

1. Considering your storage space constraints, two separate repos might work well if you carefully schedule two separate jobs. Just keep in mind that (depending on the channel between sites) sending backups offsite might take some time, so probably it would be better to save 2-hourly backups locally first (just a short backup chain) and THEN create offsite replicas, backup copies (twice a day), and whatnot from that local primary chain. That way, you will have the most recent local copy and save space on the offsite storage (no need to have 60 restore points).

2. Backup copying does not hit production storage as it copies backups that already exist on Veeam repository. However, if you store your backups on VM's disks (which is not a good practice), backup copy process can introduce some I/O on VMware storage system as it will read backups from the VM's disk. The same goes for replica from backup - Veeam reads required data from backup files that already exist without touching production storage.

3. Proxy that reads data from production storage (no matter if it's direct access or hot-add) should reside closer to the production storage. For offsite scenario it is better that the target datamover resides closer to the target repository. As far as I understand, your offsite repository runs Linux, which means that it can run a data mover, therefore you already good to go.

P.S. Here is BP guide, worth checking.

Thanks!

[roaima]

Hello,

Thank you for bearing with me. It's taken me a few days to get the configuration set up and running smoothly.

1 & 2. I've now got 2-hourly backups running from Production VMware to our on-site Veeam Proxy. (I have no option to put the Veeam Proxy anywhere except as a VMware guest on our Production platform. I appreciate this is not BP but I really have no choice.) From this Veeam Proxy I take an immediate copy of selected guests for Replication in our off-site location. I also take a 12-hourly backup of all guests to this same off-site location.

It takes around 40 minutes to complete an incremental backup to the on-site server, and then another 40 minutes or so to replicate that copy off-site. So at the point a replica becomes available it's already over an hour old.

Is this normal?

Although I want to leave this to settle down for a few more days, should I consider taking a 2-hourly copy directly from VMware to my off-site location (i.e. using just the Veeam Proxy located in the off-site centre, and completely ignoring the Veeam Proxy that's on-site) and replicating directly from that off-site copy?

3. Thanks.

Regards,
Chris

[PTide]

1 & 2: The numbers are not whopping and the answer depends on what is a bottleneck, and the overall size of the incremental, and your network speed. Do you have the bottleneck stats?

3: You can do that, although I would not recommend to use a proxy that is distant from the production storage. Keep backup proxy closer to the datastore, and make sure that a target datamover runs close to the repository. Please check this page for better understanding of the architecture. There are two examples: offsite backup with simple backup repo, and offsite backup with repo on a share.

Thanks!