Issues/Concerns with Veeam and Cloud backup

[Nax]

I have a few thoughts/questions/concerns for Veeam and anyone who is looking to use Cloud backups as a secondary point for backups (which everyone should have)!

I want to see what we're doing wrong. We have 47 VMs we're backing up. We're almost completely virtualized with the exception of our SMTP gateway, the server that processes Veeam backups, and of course our servers that host our massive RAID, which all the VMs live daily.

We originally got our Cloud gateway to have a redundant location. We have backups on site and on tape. The tape backups are rotated off site.

We've asked for 3.5 TB of data. The size of the backup file without any restore points is only 2.5 TB - way under our threshold. But when you add the minimum restore point (you can't go below two), this is when things get odd.

We have a daily backup job run every night at 10PM. By the time it's done processing the incremental and backing up to tape, it's 6AM. Then the backup copy job to cloud processes. It has to read the differences in every VM and then push the changes to the cloud. Okay no problem. We have some VMs that are over 1TB in size. Reading disks is very slow, for some reason, the backup proxy only wants to read from SAN on the regular backup job, but not the backup copy job. Okay.

As soon as the daily kicks in again, it stops processing the backup copy job. There are changes to be made. Makes sense, except that the backup copy job got cut off in the process and has to restart to catch changes. This makes a loop in which the backup chain (that has incomplete points on some VMs) gets bigger and bigger until we run out space on the cloud backup. Then backup copy jobs will fail until you make more room. But if you manually delete the incremental chains to make room, you will break the whole backup.

It seems we're screwed no matter what we do. Backup copy job to cloud is too slow even on our 50meg pipe, paying for the extra space doesn't make sense, and there isn't enough time between the daily jobs for the backup copy job to even fully process. Thank goodness this is a last resort option, but the fact that you can't schedule the backup files (aka file copy job) from the daily job to a cloud repository on a schedule sounds...unacceptable. You can only file copy job manually.

Backup copy job to cloud only seems to make sense if:
-You don't have many VMs to process
-If you do have a lot of VMs to process, you need to have an ample downtime period for the backup copy job to fully process
-have a LOT of room for growing backups past what you actually need so that the most recent point can be written and THEN older points will delete. Otherwise you will run into out of space errors.

What can I do here? Why doesn't veeam just dump the full backup file to cloud? It's not like you can live mount from the cloud repo anyway, you still have to download the file locally and only THEN can it be mounted to do file-level/whole VM restores. Restoring from tape is way faster (crazy, right) and it would be nice to justify the very expensive cost/contract investment for these providers you give us as choices to work with.

Thanks for any info/advice you can offer!

[nielsengelen]

Could you give some more information about the cnofiguration on your backup copy jobs? In regarding to copying data offsite it might be useful to use WAN acceleration (you need Enterprise Plus for this).

[chrisdearden]

the backup proxy only wants to read from SAN on the regular backup job, but not the backup copy job. Okay.

The copy job behaves that way by design - the mechanism is very similar to the backup job, but the source is the backups sitting in your on site repository , rather than the actual virtual machines.

it sounds like you have had problems getting the initial full backup to your CC repository - have you looked at seeding options with your service provider ?

[Nax]

@ vmniels:
Hey there! So the backup copy job config is pretty standard. We have the lowest amount of restore points (2) and it runs after our daily backup job is finished processing. The daily backup job only takes maybe 3-4 hours. I haven't even changed any of the default options.

WAN acceleration doesn't really help in our case - we have a very fast connection. The backup copy job will run for 12-17 hours, then cut off when the daily backup job runs. It just takes a ridiculously long time to read through each VM disk for changes. Why doesn't the backup copy job just use the same backup files as the backup job?

@Chrisdearden:
We've sent a full backup seed offsite many times but we still end up running out of space over time and the backup copy job takes too long to catch up. We have 3.5 TB of space. We can throw more space at the situation, but it would be nice if it worked independently of the backup job so it wouldn't cut off. that's very frustrating. How much space should we ask for?

[veremin]

If you have a very fast connection, and don't want for a backup copy job to utilize backup files, why not to use additional backup job pointed to a cloud repository, instead of backup copy one?

[chrisdearden]

How big are your daily incrementals onsite ?

how much of that 50MBps pipe is available during the copy job window ? I assume its not dedicated for copy jobs ?

[Nax]

If you have a very fast connection, and don't want for a backup copy job to utilize backup files, why not to use additional backup job pointed to a cloud repository, instead of backup copy one?

You can do this? I will try it.

[Gostev]

Anyway, back to the main topic (Backup Copy). Most folks here seem to be focused on troubleshooting performance issue - but I think the main issue here is unexpected disk space requirements. And they actually do not look correct to me, as 3.5 TB should be more than enough with 2.5TB VBK and just a few increments.

So first, I recommend installing Update 2, as it fixes some issues which caused full backups of specific virtual disks transferred to the target repository as a part of an incremental backup. After installing Update 2, best would be to delete the existing job and backups (if possible), and start from scratch. Alternatively, you can simply try a local Backup Copy first (using a separate job with the same setting, but local target) to ensure that disk space consumption issue is resolved.

Now, if you target backups will still go over 3.5 TB even with Update 2 installed, then best would be to open a support case for further troubleshooting.

Thanks!

[foggy]

Why doesn't the backup copy job just use the same backup files as the backup job?

Here's a good discussion covering this question in detail, worth reviewing to get better understanding of the backup copy job basics.

We've sent a full backup seed offsite many times...

Just to check, have you performed seeding as it is described in this article?