Advice on synthetic or active full backup/Veeam proxy backup

[omerlin98000]

Hello to all

Running latest 7.0.0.871 (patch 4) B&R, I backup around 12 VM each day.
For that, I have installed one veeam 7 server (with its 2 Embedded/active proxies), and 3 more proxies in three different VM (spread among my 3 ESXi 5.1u2 servers).

1) Each night, start at 7pm, I first launch a backup of these 3 VM containing proxies through the Veeam server proxy itself : doing this will avoid Veeam to use proxies in these 3 VM in a backup job that will be used for backing these very VMs ! In other words, I intend to quickly backup the 3 proxies VM to let them quickly handle without hassle the next jobs...
IS THAT A GOOD IDEA ???
2) after backuping the 3 VM containing proxies, I unleash hell and all other backup jobs are started by scheduling them after this first 3 proxy backup job (in reality, I have from 3 to 5 VM backing up simultaneously)
The first job start at 7pm, the latest backup stop around midnight, all is ok and perfect

On saturday, jobs are set to do their synthetic full, and I have a problem.
My first job (point 1) at 7pm finish rapidly its backup as usual, but begin the synthetic job... that last for 12 hours !!!
As all the other jobs are dependant of the end of job 1, they start sunday around 10am: OK, they did their backup well BUT they have no more synthetic job to do, because we are already on sunday and miss the saturday window.

I'm sure there is a better way to achieve my goal, perhaps I'm taking too more caution on my first job ?
I wonder what happen if a job is backing up a VM that have an active Veeam proxy doing another job ???

[Vitaliy S.]

Hi Olivier,

For that, I have installed one veeam 7 server (with its 2 Embedded/active proxies)

Veeam backup server can have only 1 default proxy that is installed with the backup console, so I assume you have only 1 embedded proxy, right?

1) Each night, start at 7pm, I first launch a backup of these 3 VM containing proxies through the Veeam server proxy itself : doing this will avoid Veeam to use proxies in these 3 VM in a backup job that will be used for backing these very VMs ! In other words, I intend to quickly backup the 3 proxies VM to let them quickly handle without hassle the next jobs...
IS THAT A GOOD IDEA ???

Do you backup these VMs just because they are Veeam proxy servers or because you have some data you want to protect on these VMs?

On saturday, jobs are set to do their synthetic full, and I have a problem.
My first job (point 1) at 7pm finish rapidly its backup as usual, but begin the synthetic job... that last for 12 hours !!!
As all the other jobs are dependant of the end of job 1, they start sunday around 10am: OK, they did their backup well BUT they have no more synthetic job to do, because we are already on sunday and miss the saturday window.

What is your target storage? How did you present it to the backup server? What does job bottleneck stats show as the weakest part in the infrastructure?

I'm sure there is a better way to achieve my goal, perhaps I'm taking too more caution on my first job ?
I wonder what happen if a job is backing up a VM that have an active Veeam proxy doing another job ???

Yes, there is a better way to achieve what you want Please be aware that proxy servers are NOT involved in synthetic full creation, backup proxy servers retrieve data from the VI, send it to the repository and then repository starts to build synthetic fulls. So If I were you, I would just kick off your other jobs at the time you want a synthetic full backup to be created (on Saturday).

On the other hand, given that you synthetic full creation for these 3 proxy VMs (first job) takes 12 hours, I think that your target storage is already saturated and would not be able to handle 4 synthetic full jobs. Probably, It would make sense either to use different schedule for these jobs, for example, split them into 2 groups - one group starts building synthetic full on Saturday, the second one can do that on Sunday, or run active full backups instead.

Thanks!

[omerlin98000]

Hi Vitaliy, many thanks for your quick answer

...
Veeam backup server can have only 1 default proxy that is installed with the backup console, so I assume you have only 1 embedded proxy, right?
...

Yes, badly written , I have one proxy with two max concurrent tasks

...
Do you backup these VMs just because they are Veeam proxy servers or because you have some data you want to protect on these VMs?
...

I have indeed datas to protect (these 3 proxy servers are my AD, my monitoring server and my file server - 2003/2008 servers - not very used after 7pm )

...
What is your target storage? How did you present it to the backup server? What does job bottleneck stats show as the weakest part in the infrastructure?
...

Target storage is a Synology 1813+ NAS with CIFS repository, set up for max 6 concurrent jobs.
Do you mean standard daily job bottleneck ? or saturday synthetic job bottleneck ?

...
Yes, there is a better way to achieve what you want Please be aware that proxy servers are NOT involved in synthetic full creation, backup proxy servers retrieve data from the VI, send it to the repository and then repository starts to build synthetic fulls. So If I were you, I would just kick off your other jobs at the time you want a synthetic full backup to be created (on Saturday).
On the other hand, given that you synthetic full creation for these 3 proxy VMs (first job) takes 12 hours, I think that your target storage is already saturated and would not be able to handle 4 synthetic full jobs. Probably, It would make sense either to use different schedule for these jobs, for example, split them into 2 groups - one group starts building synthetic full on Saturday, the second one can do that on Sunday, or run active full backups instead.

Yes,, I have already thought about modifying the synthetic day on this groups jobs (I have 5 groups).
I miss the point that synth jobs were done by Veeam server itself
Yes, kick off the other jobs immediately will certainly do it but better idea: the backup job of my 3 proxies start at 7pm and always finish around 11pm, so instead of waiting this first group to finish, I can set the other jobs to start at 11pm ? so the 3 proxies will be in good shape (no snapshot commiting, no VSS, etc.) to handle the other jobs ?

But again, is this a good method to first backup the servers that have a Veeam proxy, then wait for the end of these backups to fire other jobs that will use these proxies ???

[Vitaliy S.]

Target storage is a Synology 1813+ NAS with CIFS repository, set up for max 6 concurrent jobs. Do you mean standard daily job bottleneck ? or saturday synthetic job bottleneck ?

Got it! Yes, you can post daily jobs and synthetic full jobs, but I'm pretty sure the bottleneck status will point to target storage. What time does it take you to create an active full backup targeted to this storage?

Yes,, I have already thought about modifying the synthetic day on this groups jobs (I have 5 groups).
I miss the point that synth jobs were done by Veeam server itself
Yes, kick off the other jobs immediately will certainly do it but better idea: the backup job of my 3 proxies start at 7pm and always finish around 11pm, so instead of waiting this first group to finish, I can set the other jobs to start at 11pm ? so the 3 proxies will be in good shape (no snapshot commiting, no VSS, etc.) to handle the other jobs ?

Yes, you can certainly do that.

But again, is this a good method to first backup the servers that have a Veeam proxy, then wait for the end of these backups to fire other jobs that will use these proxies ???

If these three proxy servers are also involved in backing up other VMs, then your strategy looks fine to me.

[omerlin98000]

Thank you for your advices

About bottlenecks.
On Synthetic, here it is:

This is the first backup job of the night backup window, the 3 VM backup in 3h/3h30 as usual each night, THEN begin the synthetic job that last 12h...
During week days, the job lasts the same amount of time (around 3h), the bottleneck is exactly the same: 99% source, around 40/45% proxy (itself).

Strong missed point : my Veeam server use its own proxy (2 concurrent job max setup) to backup these 3 VMs, so the load is all on this server.
This is perhaps the problem ? This VM is W2008-64b, 8GB Ram and 2 vCPU, and is holding: VMware vCenter 5.5, Veeam B&R 7 and VeeamOne... TOO MUCH (for 56VM, 11 are off) ?
RAM seems always fine (below 3GB), cpu is stable around 30-60%, but 80% peek at the synthetic job startup. 4 vCPU will definitely be better ? (because for other jobs, I was counting on the load of the different other proxies, but you teaches me that the synthetic will be handle by this server)

I can also decide to install a separate Veeam 7 VM (on W7 Pro 64b), only if MUCH MUCH better...

[Vitaliy S.]

I have a couple of suggestions to your setup, choose any (depending on available resources):

- Run active full backup instead of synthetic and keep your existing backup infrastructure configuration, It will save you about 10-12 hours of your backup window.

- Remove proxy roles from these three VMs and install proxy server on any other server that you don't need to protect. This will allow you to leverage VMware CBT (Changed Block Tracking) for AD, Files Server and GFI box which is disabled by default sine these are Veeam proxy servers. Currently the entire image of the VMs is read and then only changes are transferred to target repository. Given that you have only 1.4 GBs of changes for all VMs, I expect the incremental job pass to be accomplished within 1 hour.

Can you please tell me why did you install three proxy servers on each ESXi host? Did you want to use hotadd backup mode instead of network? How much time does it take you to backup all other VMs with these three proxy servers?

[omerlin98000]

...
- Run active full backup instead of synthetic and keep your existing backup infrastructure configuration, It will save you about 10-12 hours of your backup window.
...

Understood

...
- Remove proxy roles from these three VMs and install proxy server on any other server that you don't need to protect. This will allow you to leverage VMware CBT (Changed Block Tracking) for AD, Files Server and GFI box which is disabled by default sine these are Veeam proxy servers. Currently the entire image of the VMs is read and then only changes are transferred to target repository. Given that you have only 1.4 GBs of changes for all VMs, I expect the incremental job pass to be accomplished within 1 hour.
...

I didn't know that CBT was disabled for VM containing proxy, now I see in the logs that reading these in hot-add mode has no [CBT] tag and that full content is read each day
Good advice not to put proxy on "production" VMs

...
Can you please tell me why did you install three proxy servers on each ESXi host? Did you want to use hotadd backup mode instead of network? How much time does it take you to backup all other VMs with these three proxy servers?

Yes, that was the point, to have a repartition between these 3 proxy/ESX.
These 3 proxy are defined as auto transport mode, and the jobs are always selecting hotadd mode (except for a very few not supported VM that use network mode, weekly jobs).

After this job (backup the 3 proxy starting at 7pm- which finishes around 11pm for less than 2GB transferred), it takes only 30 minutes to finish the 9 other VMs in CBT/hotadd mode (11.5GB transferred)... YOU ARE RIGHT, it takes nothing... I'm stupid not to have seen that

So to summarize :
- put proxy on non important VM (perhaps weekly backuped VMs ? ) or better, (re)install Veeam B&R in its own VM with let's say, 6 concurrent jobs (adjusting RAM/CPU) ?
- do weekly active full backup instead of synthetic
Anything else ?

[Vitaliy S.]

Yes, I would recommend installing Veeam proxy on any spare server, assuming that you have similar job performance rates for network and hotadd backup modes. Once you do this assign the max concurrent task limit according to available CPU resources (make sure your proxy server is not over-saturated during backup jobs runs).

As to moving Veeam backup server, then it also does make sense, cause currently your vCenter Server is also backed up with CBT option disabled. And yes, looks like active full should be a better option for your target NAS device.

Hope this helps!

[omerlin98000]

Thank you very much, Vitaliy
I'll keep you in touch after modifications...

PS: I have named this thread "Advice on best synthetic full schedule or changing my option", perhaps it will be more useful for readers to change this title with something more linked to our discussion ? Up to you...

[Vitaliy S.]

I have modified the original topic title, should be more relevant now.