Feature request - Queued exports

[TitaniumCoder477]

Greetings,

I have run into this before but especially now when I am trying to export points from over 30 assets across four jobs. Basically, it's super easy to kick off the exports... But Veeam does not queue them up. Instead, it starts them all at the same time, which can easily crush a backup server. This is abnormal behavior, in my opinion, since virtually everything else in Veeam is managed by proxy tasks, repository tasks, throttling settings, etc. I would like to see this same kind of scheduling/queuing logic in the export workflow.

Thanks!

[PetrM]

Hi James,

At first sight, it seems more like a technical problem than a feature request. Did I understand correctly that you have a problem with the backup server when you export 30 points from 4 different jobs? Do you have a support case for this issue?

Thanks!

[TitaniumCoder477]

What makes you think it's a technical problem? If I do a single export, there's low impact to the backup server. But if I want to export a point for 30 assets in a job, then that means Veeam spins up 30 exports at the same time. Since the export process is inherently CPU, RAM, and I/O intensive, this can easily bog down the backup server so that it's unresponsive. The problem, as I see it, is a design problem--Veeam needs to schedule/load balance exports, just like it does everything else. For example, if I have four jobs, backing up approximately 40 assets with an average of 1.5 disks each, and my proxies are configured to handle a total of 8 concurrent tasks, then clearly I won't be doing 60 tasks even if all four jobs run at the same time. Rather, Veeam will do 8 tasks, queuing up the rest as needed. The same design and implementation should be in place for exports, thus the feature request.

[PetrM]

Ok, I see your point. However, the export process might be I/O intensive from repository perspective, the backup server just manages the process, it does not interact directly with data. That's why the behavior seems to be unexpected from my point of view. Are you talking about backup server or repository?

Anyway, I'd prefer to treat this question as a technical issue as we're not talking about implementation of new functionality but about possible ways to fix the current behavior. Please let me clarify a couple of details with my colleagues from QA team, I suppose they tested this scenario. The first idea which comes to me is that the setting of concurrent tasks on repository must be respected by export, at least this setting is respected by other synthetic operations, for example synthetic full, merge, transform etc.

Thanks!

[TitaniumCoder477]

Well, interestingly I can speak to two different designs. The one that prompted the post initially was an "all-in-one" backup server (Dell hardware w/ Windows server and stacks of disks). So the resource consumption directly affected the backup server obviously. However, today I did an export of about 10 points at one time on a backup server with a Linux repo, and I verified the Linux resources were consumed to the extent that I had to cancel the operation and just copy the latest VBK + increments and VBM instead of doing an export, which I would have preferred. So you are correct in that resource consumption takes place in the realm of the repository role, but I still disagree that it's not a design flaw in Veeam. That is, Veeam appears to have no load balancing, scheduling, or otherwise logic built into the export process. So it's not fixing something that's broken. It's implementing something that's clearly missing.

[PetrM]

Hi James,

I've discussed this idea with my colleagues from QA and we decided to correct the behavior in one of our future releases: the maximal number of parallel exports will be limited by the setting of concurrent tasks on repository. Many thanks for pointing out this problem, your feedback is highly appreciated!

Thanks!