backup times increasing, what am i doing wrong?

[33minutesinheaven]

So i read a few posts on here about decreasing backup times but no matter what i do, i just seem to make things worse

I have 3 esxi hosts in a cluster with about 40 VMs to backup.
The storage is located on a nimble array.
i am using veeam backup and rep 9.5 update 4.
the backup machine runs in its own VM on its own separate ESXI host.
All backups are set to "virtual appliance" mode on the backup proxy which is on the same veeam machine.
The backups are saved to a buffalo terrastation that seems to not be the bottleneck, but i am not sure.
All bottlenecks are 80-99% source.
an example job, file server backup. Processed 7.4 TB, read 48.7GB, transfered 13.1GB (3.7x), Processing Rate 41 MB/s, Duration 4:16:49, Speed 144.3 MB/S, Busy: Source 73% > Proxy 43% > Network 36% > Target 19% (Previously before i started changing things, this job was taking 2 hours, which i thought was bad before.). I seem to recall like 6 months ago, these jobs would all have no problem firing off in less than 30 minutes.


Things I have done this week to try and make times better.
- increased CPU available to veeam from 4 to 8 cores.
- increased ram available to veeam from 40gb to 50gb
- Adjusted backup proxy to be able to do 8 concurrent tasks
- adjusted backup repository concurrent tasks from 4 to 2
- played around with the timings of backups. They were all starting around the same time, staggered by an hour. I thought maybe they were stacking ontop of eachother and not being efficient, so i changed the timings so that different jobs would run every few hours all night.

When i look at the throughput graph, the backups seem to run fine, taking about 30 minutes for the example fileserver job above. But the runtime on the job is 4 hours... so what the heck is it doing for all that time? when i look at the details, under "Action" i see that its spending 1h 44m at the "required backup infrastructure resources have been assigned" step. and 47:19 at the "index guest file system" step. All other steps are not taking very long at all. But those numbers don't add up to 4h16m anyways...

its very confusing to me. Does anyone have any idea what i am doing wrong, or what directions i should be looking in? i originally thought it was my backup arrays performance, but now i am not so sure.

[PetrM]

Hello,

I think that the main issue is:

when i look at the details, under "Action" i see that its spending 1h 44m at the "required backup infrastructure resources have been assigned" step. and 47:19 at the "index guest file system" step

I would recommend to contact our support team, apparently a closer look at debug logs is required to better understand what's really happening under the hood and to define an exact operation which takes most of time.

adjusted backup repository concurrent tasks from 4 to 2

May be it would make sense to increase concurrent tasks for both proxy and repository components? It waits for task slot for almost 2 hours according to statistics provided.

By the way, indexing of guest file system should be used only if you need to perform one-click restore via Enterprise Manager, feel free to disable this option if you don't run EM in your environment.

Thanks!

[33minutesinheaven]

I was worried about overloading the storage array with too many concurrent jobs. I know with rotational disks on desktops in the olden days, performing more than one copy opperation at a time drastically reduced the copy times as they ran concurrently and had to share bandwidth.

That is very interesting about the indexing of guest file systems... i am not sure what you mean by enterprise manager, but i do restore backups like Restore -> VM -> Guest File Restore. For individual files, especially on the file server. I am not sure if you mean that this will or will not be affected if i turn off this setting. Can i still do this?

i will contact veeam support when i can, hopefully today and will post back with any finding they have.

[PetrM]

Yes, you can disable guest file indexing. Please look at this article to get more information about Enterprise Manager.

May be you could add more repositories if the current one does not allow running more than 2 tasks in parallel?
I assume that it waits 2 hours for repository but let's see what our support team finds in logs.

Thanks!

[33minutesinheaven]

but isnt a repository only considered a hardware array? so um adding more, well i just bought this one last year and it cost like 20k... unless i dont get your meaning.

i have no idea how many it "allows". I am not sure how i could figure that our except trial and error.

i dont think backup enterprise manager is nessecary for us as we only have one veeam instance and one physical site. Good to know i can disable! thats awesome and will surely help. will contact support soon.

[33minutesinheaven]

So here the recomendations.

First i am severley under cored still. I have assigned 8 cores but he thinks i need about 14 for my workload. No problem becuase this machine has 32 cores and isnt doing much else. So i will give it cores.
Also he recomended setting the concurrent tasks on my storage array to the same number as my backup proxy (in this case . We are going to try more cores and see if it can improve it. i will also turn off file guest indexing.

[TheBob]

I'm not a spezialist but and new at veeam but i had almost same problem and posting some issue here in forum.
my backup repository concurrent tasks is on 10, 2 lower at my CPU and that one has solved many problems. Before i had 4 Task.
mybe that one helps you a little bit.

[nitramd]

@33minutesinheaven,

Have a look at the Veeam Best Practices guide, slightly dated, in terms of software versions, but has good info: https://www.veeambp.com/

Two things should help right away, more cores and more memory.

[33minutesinheaven]

sadly it didnt help much. Still getting stuck on "required backup infrastructure resources have been assigned". i have 20 cores and 50gb ram currently. The interesting thing is that it has now changed so that my job is now Source 6% > Proxy 10% > Network 5% > Target 96%. But the storage array that i am saving to has 8 concurrent tasks now. Seems like its very much a balancing act, and that there are several causes of this message. Wish it was more descriptive.


Veeam support is now reviewing the logs. Will keep you posted.

[foggy]

How the NAS is added to Veeam B&R? If it is a CIFS share, its tasks are performed by the backups server itself (unless there's a dedicated gateway). What backup method is used by this job? Did both job runs for which you've provided bottleneck stats above perform full or incremental backup?

[33minutesinheaven]

The backup storage array is an iscsi target through esxi attached to the veeam VM and mounted as a drive letter. Its only mounted to this one vm.

The vms that are being backed up are on a nimble array which is mounted to the cluster again over 10gbe iscsi. same network and network switch. it is using the mehtod of virtual appliance to do the backups.

both are incremental backups.

veeam support is reviewing the logs. i hope that will be insightful. They said they would let me know today about it.

[foggy]

Ok, so the target data mover is also running on the backup server VM - keep that in mind when you think about the number of concurrent tasks for proxy and repository. By the backup method, I meant the backup chain type - is it forward/forever forward/reverse incremental? If it is the latter, that might explain the bottleneck target. But anyway, let's see what logs analysis gives us.

[PetrM]

Hello,

Still getting stuck on "required backup infrastructure resources have been assigned".

Please ask our support engineers to review logs and to define the cause of delay during assignment of resources to tasks, I believe that this detail is quite important.

I've just noticed that you're still on 9.5 u4: it's worth upgrading Veeam B&R to the version 10, there are significant performance optimizations including improvements in resource scheduler logic.

Thanks!

[33minutesinheaven]

Thanks. I had a bad time with the veeam one v10 upgrade, so i am going to hold off for now from moving b&r to v10.

The engineers were not able to determine a cause of the lag. We changed the backup repository and backup proxy tasks down to 4.
I have also re ordered the jobs. So the jobs that are pushing the least amount of data are now going first. That seems to have helped last night. Jobs that were taking 2 or 3 hours are now down to 20 minuites which is more reasonable to back up a few gigs.

Another thing he mentioned, is that we are doing forever forward backups and have no tasks to defrag the backup files. So i have put tasks to defrag on all of my jobs, staggered over saturdays and sundays during the month.

Its currently running a defrag task on my biggest backup job (7.4TB) that i set it to run last night and that is currently delaying everything. So i will have a better idea if changing the job order made a noticeable effect tomorrow. will keep the thread updated.