vCD copy jobs to StoreOnce hang at 0% - but not always

[DanielJ]

I would like to call some attention to a long running and serious problem which is being investigated in ticket 07324220. All involved systems are on their latest firmware/software versions.

The problem: a vCD backup copy job to a StoreOnce appliance hang at 0% and never starts processing any data. Sometimes it works directly by itself, often not. There is no error message, the job just hangs. If I stop it and start it again, it might work. Sometimes it requires 3-4 tries. If I want to make sure the backup copy is created successfully every day, I have to do this manual intervention every day. The job does not respect any backup window settings I have tried, it just drones on for any number of days until forced to stop. The job also blocks any other copy jobs to the StoreOnce. The job has been deleted and recreated a few times with the same results. The greatest mystery is how it can work fine one time and not the next, apparently random, with everything else being identical (all systems are idle when these copy jobs are running).

Support has mentioned a supposedly "known issue with copy jobs having many vApp objects and targeted to a StoreOnce repository". The problematic job has in its object list a vCD backup job, which in turn has one vCD organization in its list. This organization has 75 vApps with one vm each. That makes 150 objects in total and apparently this is too many. Support has suggested splitting up the job into several copy jobs with 10-15 objects per job. I understand this is all they can suggest as a workaround, but it is not a workable approach.

As I mentioned in the ticket: I can’t believe a problem like this has just sailed by QA. It continues to be my impression that vCD support is not tested properly. This is hardly the first time we have had vCD specific problems. We have a big vCD platform with thousands of vApps, and we have discussed starting to back it all up to the StoreOnce via copy jobs. This is just impossible if there is suddenly a practical limit of 10-15 objects per job.

I hope to be able to post a solution here if one is found.

[david.domask]

Hi Daniel,

Sorry to hear about the challenges here and thank you for the detailed explanation and the case number.

First, reviewing the case, I can see that the Support Team is working closely with RND on the issue; I'm afraid right now I also cannot explain the inconsistencies, but I can see you posed the same questions within the case, so let's allow Support and RND a little time to review and respond.

As for the issue itself, checking the history of the issue, it was only reported once earlier this year (a few weeks back) when the issue was first reported, and I cannot find any other cases excepting the original case and your case where it was hit, so at first blush it seems like an edge case, though I understand that doesn't change the impact you have to deal with. The previous case was closed without a resolution as there was no response within the case from the client.

Based on Support and RND's discussion and the answers within the case, I don't understand the issue to be about a specific limit of vApps, but rather that there is an issue with managing the sessions to Storeonce appliances which are limited; Veeam's resource scheduler should handle this correctly so you don't get such stalling like you're observing, so I think it's best we wait to see the result of Support and RND's discussion. Checking the case, I think the suggestion about splitting up the jobs was more to provide an potential immediate workaround while the research continued rather than as a long-term solution.

[DanielJ]

I'm happy to report that a hotfix was provided last week and that everything has worked fine since then. Anyone who has similar problems with vCD and StoreOnce, refer to ticket 07324220 and maybe the hotfix will apply to your setup too.

Just out of curiosity - what was the actual problem, and how could it appear so randomly?