Stopping backup jobs

[JDP]

Hi

We have a couple dozen backup jobs with have child jobs that are sheculed to be executed after their parent ends. However, I have found myself today with 12 running Synthetic full backup jobs that appear to have started during the weekend and are still running (some of them 54 hours long now).

How bad would it be stopping these synthetic backup jobs? What consequences would it have, consistency wise and in terms of repeating the jobs afterwards?

Is there any setting i have missed so child backup jobs don't get started after I force stop on their parent? If i manually stop a XXXX01 backup job, i don't want XXXX02 to start immediately, and if I stop XXXX02, XXXX03 is launched and so on.

I understand that manually disabling children before stopping the parent is a solution, but hasn't anyone run into this before and thought it should be revised?

Thanks!

[foggy]

First of all, those chained jobs that are not started on the day synthetic full is scheduled to be performed at, will not perform synthetic full, but just a regular incremental run.

Regarding your question, stopping/killing the job will make the latest (currently being created synthetic) full backup file in inconsistent state, however all older restore points will not be affected. The next job run will repair the chain.

Also, what is the reason for using chained jobs in your case? Please note that jobs chaining is not among our best practices as it could also have other undesired consequences. If the goal is to control processing resources load, then limiting max concurrent number of tasks assigned to proxy/repository servers is recommended instead.

[JDP]

Hi

Our main reason to choose backup chaining is to split backup jobs from a role perspective. We mostly have dual VMs for every role in our infrastructure, and load balancers take stunned VMs out of the balancing pool at the time of removing snapshots, however, we cannot permit that in any case, for instance, WEB01 and WEB02 finish their backup jobs at once and become unavailable, even if it is for a 30sec - one minute.

All this Synthetic backup jobs are run in the Veeam backup server which is the local proxy for our backup storage, right? I want to investigate what kind of resource contention we are finding. I can see it says Target, but can't figure out what to look at.

CPU is at 4%
RAM usage is 6.1 out of 7 GB
EMC CPU usage is around 30%
Throughput at disk array is around 2MBytes/sec
I'm not really sure about how to detect latency or iowait times to a CIFS share mounted datastore

Best regards
Javier

[JDP]

I've stopped the synthetic backups because we can't really hold the snapshots too much.

What alternative approaches would you recommend for backing up with the followin scenario?
500-600 VMs
6TB used space in the VMs
3 remote datacenter proxies (4vCPU, 4GB RAM)
1 local veeam backup server (4vCPU, 7GB RAM) proxying an EMC CIFS shared volume (no deduplication on storage)

[foggy]

Synthetic backup activity is performed locally by an agent installed on repository server (in case it is an agent-enabled repository, not CIFS). However in case of CIFS repository mounted to the proxy server, it is performed by an agent on the proxy, over network (which is not optimal, indeed). If target is identified as the bottleneck, then the write speed to the target storage is the weakest point in data processing chain. With 12 parallel synthetic fulls you're almost killing it.

[foggy]

What alternative approaches would you recommend for backing up with the followin scenario?

As the first step, you could set synthetic fulls to be performed on different days for different jobs.

[JDP]

Ok so perhaps using the CIFS share as a backup repository was not the best choice here. What I understand from your message is that CIFS protocol adds much overhead to the process, even if they Host where Veeam backup server and EMC backup repository are connected to the very same network switch.

Veeam Server is running in a VM. Do you think it would be better if we assigned a big HDD (vmdk) (or several) to it in this same EMC storage? That way the backup repository would be local storage from the VM point of view.

Do you think that would improve the performance noticeably?

Thanks

[foggy]

Am I understanding right that EMC storage has two LUNS, one being VMFS datastore holding Veeam B&R VM and another NTFS LUN currently mounted to Veeam B&R via CIFS? Then the more preferred way would be mounting that NTFS LUN directly to Veeam B&R console using an iSCSI initiator.

[JDP]

Hi
The storage has two volumes, VOL1, shared using NFS for development VMs (out of the scope of this backup jobs) and other shared as CIFS, which is used as a "CIFS Shared Folder" repository in Veeam Backup Server.

[foggy]

iSCSI is the preferred way of adding NAS devices as backup repositories. Since Veeam data mover will be installed on the Windows server with the attached NTFS LUN, it will act like a local drive to your repository, so worth trying.

[JDP]

Along with switching to an iSCSI mounted disk in the Veeam backup server for the backup repository, how can I ensure Synthetic full backups don't stack up all together? I can see that they can be scheduled for a certain day, but can't figure out how to make them run on a specified time window out of the normal backup jobs.

Thanks

[foggy]

but can't figure out how to make them run on a specified time window out of the normal backup jobs.

This is not possible, synthetic fulls are always come after the corresponding incremental job run. You can, however, use this script to run it on a separate schedule.

[JDP]

Thanks
Another question. Synthetic full backups are dependant on the datastore concurrent jobs limit?
Our datastore can easily handle 8 concurrent VM backups, but i bet it will have problems handling 8 concurrent and random IO operations such as Synthetic backup.

Can this limit be configured as a separate setting?

[foggy]

No, however, you can control the combined data ingestion rate on the repository (right the next check box to max concurrent tasks limit).

[JDP]

Yes, but if this is random IO, combined data ingestion rate will probably be a low value since disk heads will be jumping around continuously doing R/W operations, right? 10Mbytes/sec in 100% random operation can be quite a hog. In the other hand, 10MBps of sequential streams of data (only writes, even if they are 8 at once) should be much less impacting for the storage.

[foggy]

Correct, so I recommend at least scheduling them on different days and see how this helps.