Veeam 7 - All proxies devoted to 1 backup job.

[BearHuntr]

I've started a POC test of Veeam 7 and I have moved 4 test jobs to the server and I'm seeing something weird that I think relates to Parellel processing. Typically, I have set up backup jobs to not start at the same time, so Job 1 starts at 6 pm, Job 2 at 6:05 pm, etc... With these staggered start times, it seems like all of the processing is done on Job 1, and once Job 1 gets nearly finished and frees up proxy resources, then Job 2 can begin. This was a problem for me last night as Job 1 was doing an active full and it ran until this morning and Job 2 didn't start doing anything until this morning as well. I have since beefed up the RAM on my proxy servers as they were a little low and changed the start times on the jobs to all be the same, but was just curious if this is normal or not? I never saw this happen on Veeam 6.5 with staggered backup job start times. Thanks!

[chrisdearden]

This is part of the new Parallel processing features - unless you restrict the number of proxies a given job can use , it will use as many as possible to back up multiple VM's and multiple discs on those VM's in parallel. If you have those jobs set up to use the new optimal compression settings , you can also run more parallel tasks with v7 than you could with v6. Its important that the repository side can keep up with this , so make sure your repositories have plenty of ram ( 4GB + 2GB per concurrent task is a good start ) or limit them to a fixed number of tasks.

Finally its posibble to run larger backup jobs , there is less call to split jobs up as much as you might with 6.5 - 8Tb of VM's per job shouldn't be a problem as long as you have sufficient resources to back them up.

[foggy]

Yes, what you're seeing is actually expected. With parallel processing turned on, all the tasks (disks) for the first VM in the first job are distributed among available slots on your proxy servers. If there are still available slots, the next VM's disks in the first job are processed. The tasks related to the next job start to be processed when the first job does not require any more free resources.

This differs from the algorithm used in v6.5, where the entire job was assigned to the proxy server and several jobs were processed simultaneously.

[BearHuntr]

How does this work if all my jobs are starting at the same time, does it balance between them? I really want to utilize the Parallel processing feature, but I'd rather not have the jobs run consecutively, especially if the first one is running an active full and takes over 12 hours to complete. It's not ideal to have the remaining jobs running in the morning. And, just so you know, we stopped using synthetic fulls as we back up to an Exagrid appliance, and the synthetic fulls were causing the Exagrid to grind almost to a halt and sometimes ran up to 36 or more hours.

Chris, you mention restricting the number of proxies that a given job can use, how can I do that? Thanks!

[foggy]

Chris, you mention restricting the number of proxies that a given job can use, how can I do that?

In the job settings, at the Storage step, you can explicitly select the proxy servers to be used by this job.

[chrisdearden]

In theory by using all of your available proxy slots and bandwidth for a backup you will process those backups faster then you would do with the v6 like processing - for example when you take a full backup of an exchange server with 6 volumes of it, we can process that Vm with 6 concurrent proxy threads , which will back them up considerably faster than if we had to process them serially.Is there any reason you want to controll your concurrent processing manually ?

for example if you had 10 available proxy slots ( 5x2vCPu proxies ) and you dedicated a pair of proxies to a given job ,it will only ever get processed by 4 threads , even if all of the other proxies are idle. Surely if you can process the job that much faster by using all 10 threads, you will complete your eventual backup window faster , have you split your jobs up due to different frequencies / destinations ?

[tsightler]

I'm going to guess he has them split so that he doesn't have to try to run a full backup of all of his VMs on the same schedule. Unfortunately the only solution I can come up with is to create multiple repositories and limit concurrency at the repository level.

[BearHuntr]

Yes, we have had an abundance of issues with writing our backups to the Exagrid appliance and when working with their support, they suggested that we only run no more than 1 or 2 active (or synthetic) fulls per day. So, we have had to space out the fulls in each job to run on a different day.

The situation I ran into today is what I'm trying to avoid, where a job running an active full takes all the proxies and no other jobs get processed until the next morning. I'd rather not limit jobs to specific proxies, but it would potentially be helpful to limit the number of parallel tasks that could be used per job. Right now, I'm running in a split situation as I test with some of my backup jobs running on the v7 server and the remaining jobs still on the v6.5 server, and half of my proxies are on one server and half are on the other. So, maybe I just don't have enough proxies. I've added some additional proxies to the v7 server for now and beefed them up with more RAM and I plan on migrating all backup jobs to the v7 server later this week, so then I should have all the proxies running for one server. I guess that I will just see how it pans out. If the active fulls cause too many delays, I may have to go back to synthetic fulls and deal with the Exagrid getting slow.

[tsightler]

Remember that you can always disable the parallel processing and get the same behavior as 6.5. Other options are to create a repository for each job, even if they point to the very same actual share. Then you can set some limit on parallelism for each repository. For example, if you have 2 proxies with 4vCPU each that would be 8 concurrent tasks. You could create 4 repositories pointing at the same share and limit each to 4 tasks. That way whichever job was running a full backup would be limited to 4 tasks, and proxy resources would be available for the incremental jobs.

[foggy]

How does this work if all my jobs are starting at the same time, does it balance between them?

On the "first come - first serve" basis (no balancing).

[trackstar]

Remember that you can always disable the parallel processing and get the same behavior as 6.5. Other options are to create a repository for each job, even if they point to the very same actual share. Then you can set some limit on parallelism for each repository. For example, if you have 2 proxies with 4vCPU each that would be 8 concurrent tasks. You could create 4 repositories pointing at the same share and limit each to 4 tasks. That way whichever job was running a full backup would be limited to 4 tasks, and proxy resources would be available for the incremental jobs.

How do I disable parallel processing? I have not yet determined what route to take. I just did a fresh new install of version 7 on a new server.

I have 2 VMs (Exchange 2010) in a DAG. Previously on v 6.5, VM2 would not run until VM1 finished (same job). I am concern with parallel processing due to the heartbeat connection between the VMs. In 6.5, we had to increase the heartbeat in the cluster and it worked wonderful after that.

[foggy]

Menu > Options > Advanced > Enable parallel processing.

[trackstar]

Menu > Options > Advanced > Enable parallel processing.

Thanks foggy.

[brupnick]

I've had similar experiences with parallel processing in my environment where it would be nice to have a bit more control over how the resources are distributed. I'm sure that if you asked 100 people how they would like to see this work, you would get about 50 different answers, but here are my suggestions:

1. Add an option to the parallel processing section that will guarantee resources to at least # VM(s) or disk(s) within a job. There would need to be some calculations done so as to take into consideration the limits of the proxy and storage target.
2. Provide the ability to assign priorities to jobs. Higher priority jobs jump the queue and are processed before lower priority jobs.

In my situation, I have three jobs: Daily - 45 Day (normal priority) with 48 VMs, Daily - 90 Day (highest priority) with 3 VMs, and Weekly - 45 Day (lowest priority) with 87 VMs. Similar to the OP, I start each job 5 minutes apart so that resources can be allocated to my highest priority job first. However, if there's a problem with any of the VMs in my highest priority job, the retry for any of those 3 VMs is now placed in the queue, which can be up to 135 VMs long. It would be nice if VBR realized that I had a higher priority VM to process and when the next batch of resources became available, allocated them a VM from a higher priority job (even if it was a retry) rather than the next VM in line.

I realize that everyone has their own configurations and ideals on how a certain part of a product would work best for them and it's up to Veeam to decide how to handle the many different environments of their customers. Overall, parallel processing is awesome and has cut my backup window significantly, but as with any new feature, could use a small amount of tweaking (or the ability to tweak) to be ideal (just my opinion).

[Gostev]

Provide the ability to assign priorities to jobs. Higher priority jobs jump the queue and are processed before lower priority jobs.

Job priorities are already in the product, just implicitly. Essentially, priority is based on the job start time. When backup infrastructure resources become available, they are offered to each running job in order according to the job start time. In other words, the job that started first will always be offered resources first.

[brupnick]

Provide the ability to assign priorities to jobs. Higher priority jobs jump the queue and are processed before lower priority jobs.

Thanks for the reply, Anton. This is why I space my jobs 5 minutes apart, but if I'm not mistaken, a retry of a job counts as a new start time? In my scenario, I have my most important job start first, but since it only has 3 VMs, if there is a failure, chances are that the retry for this job will be submitted before either of the other two jobs finishes. And since all of the VMs from the other two jobs are already in the queue, the fact that my most important job was initially started first doesn't affect its priority on the retry.

[dellock6]

If you look at the job progress windows, retry jobs have a totally different time than their original jobs, so I think yes retry job is considered as a new start of a job, with the only difference it only processed failed VMs.

Luca.

[brupnick]

Thanks for the confirmation, Luca. Because of this, it would be nice to have the ability to prioritize jobs based on something other than start time.

[dellock6]

It's not so easy I think. Surely you would say "use the original start time to prioritize the job, even during the retry", but you have to count also the different retry configuration of all the jobs. You can have a job set to retry 3 times with 1 hour interval, another with only one retry after 5 minutes, and so on.
At the end, this can become a nightmare, and almost impossible to guarantee the desired start times, so I think to use the restart time as e new start time is fair. After all, priority and queues are all mechanisms to distribute the load when backups are much more than the available proxies...

Luca.

[brupnick]

I don't disagree, which is why being able to assign a priority to jobs (High, Normal, Low, or something along those lines) might be the best option since you wouldn't have to track the initial start time through all the retries.

I have to assume that VBR already has some sort of queue mechanism with the introduction of parallel processing; there's something happening under the covers that decides which VM to process next when resources become available. I wouldn't think it would be too difficult to change that logic to say, "I have new resources available. Which VM in the queue is from a High priority job?" rather than, "I have new resources available. Which VM in the queue was added first?"

And now that I write that, using the original start time wouldn't be much different: "I have new resources available. Which VM in the queue had the earliest original start time?" but I still think setting a priority would be cleaner (see my first statement).

[tsightler]

I can see the argument for this, not everything can be dealt with as FIFO. For example cases where I'm running backups or replication of critical servers more often than other servers, like every hour. But if my nightly backups are running and using all resources then this critical job has to wait in line. I might want it to jump the line as resources become available.

[dellock6]

I agree with you both, each of us and all the guys at Veeam are right. There is a huge sea of ideas about to how to deal with parallel processing, it has really opened up a whole new way of managing backups, I see also other possibilities and future ideas, just to name few:
- reserve one or more queues specifically for a super-critical job
- release some processing slots from a low priority job as a high priority job is about to start, so it can start on time (for example, process the first 4 VM out of 10, suspend to process the high priority, thant resume the original job)

and I'm sure everyone can add his/her own idea. But all needs to come down to two aspects in order to succeed:
- keep the parallel processing as easy as possible, and not over-engineer the feature with the risk to become unusable and "hated" by users
- design a solution that works for the vast majority of customers. The more we move towards edge use cases, less the options are needed by most of us. Something like "activate and forget" by default, and if someone wants, an advanced configuration options. Think of it as vSphere DRS, default covers 99% of use cases, with rules you can tune it when needed.

Sure some additional flexiblity would be nice and welcome, but don't forget is the first version of this solution, and even now is offering a huge improvement in performances. Tuning is always possible

Luca.

[Gostev]

I am pretty sure we will be able to get away without adding complexity of managing job priorities (yet another dimension to think about). We've been discussing this a lot offline with Luca, and seem to found the way to improve parallel processing scheduling dramatically with just a few tweaks to the existing scheduler.

The only use case we did not discuss is "high priority" periodic job being stuck behind a bunch of daily jobs. Right now I am thinking that periodic jobs should simply be given priority over jobs on other schedules (e.g. daily) - because there is a reason why you make them periodic, and want them to be running much more often than other jobs (and that reason is the RPO).

[tsightler]

I don't think anyone is taking the approach that this was some huge, critical miss in this first implementation of parallel processing, and it's always possible to go back to the previous job behavior, I see this just as a discussion of the various issues that crop up with this new paradigm and kicking around ideas. Note that I only said I can see the argument as valid. I think there are likely other methods to address the issue as well. In a lot of ways I see it like the automatic proxy load balancing, the first version was pretty good but needed a few tweaks and now works in the vast majority of cases, however, users can still manually assign when absolutely required for a corner case. At this point I think it's a matter of understanding the issues and making those tweaks.

[dellock6]

The proxy management is a good comparison too, good catch Tom.
About the offline talks Anton quoted, yes the final idea sounds promising, and what a was stating before is exactly this: few and ongoing enhancements that can greatly improve the feature, without trying to add too many features at the same time... Can't wait to test it!

Luca.

[Gostev]

The only use case we did not discuss is "high priority" periodic job being stuck behind a bunch of daily jobs.

By the way, just to be clear, this issue is not specific to parallel processing, and it existed in all previous versions as well.

[tsightler]

True, but parallel processing makes it a more difficult issue to design for. Previously if I had enough proxy resources to run 4 jobs I could fairly easily create my job schedule to only run 3 of the longer jobs at once and even if somehow a fourth one would start as soon as any one would finish the remaining job would start. Now if there are three jobs started the periodic job ends up waiting in line behind all of them. I agree not a new issue, but previous simple methods to mitigate it don't work due to the new paradigm. There are other ways to work around it.

[brupnick]

Let me reiterate that parallel processing is a fantastic addition to v7 and I would rather have it in its current state than not at all. That being said, as with most things, the first iteration is usually great to show what can be done, but someone will always say that it's missing something. It's not until this feature is released to the masses that people (like me) weigh in on how they think its should evolve. In my case, I feel that parallel processing is far too powerful to be limited to FIFO only without a way for the user to customize it for their environment, even just a little.

[BearHuntr]

Since I started the topic, I should probably follow up with how I addressed the issue (especially since this showed up in the Sunday email from Gostev!) So, I shifted all of my job start times to be the same at 6 pm. I then calculated with my 7 proxies and the Veeam server itself all able to run 4 tasks each, that I had a total of 32 tasks for parallel processing. To avoid all 32 running on one job at a time, I created 3 repositories that each point to the same share on our Exagrid appliance for Veeam backups. I was able to split up the repositories to each use a different NIC on the Exagrid and set two repositories with 11 max tasks and the third with 10 max tasks (32 total.) I also set all of my daily jobs to use a weekly active full instead of synthetic fulls as synthetic was negatively impacting the Exagrid appliance and ran much longer than the Active fulls. Then I made sure that my jobs were split evenly among the 3 repositories and the active full day is different for each job. Now, most of my jobs are finished well before midnight and just the active full ones run til maybe 2 am. The only remaining issue that sometimes pops up is that if a VM within a job fails, it doesn't perform a retry until the entire rest of the job is done, so there can be a time when there are plenty of resources available, but it still waits to retry. It would be nice if the retry was per VM instead of per job as a whole, not sure if that is possible though. Either way, I'm very happy with version 7 now that I've got my schedule all figured out!

[Gostev]

Great to hear! Thank you very much for taking time to share your new v7 architecture.