Cloud connect issues and questions

[backupquestions]

I signed up with iLand a few months ago for cloud connect backups. We have 80 vms.

First question, does cloud connect scale well enough for larger customers? I have one copy job containing all 80 vms and I'm using per vm files. I have had nothing but problems for 2 months now.

The repository will stop responding after a few tasks run, and then resume again and repeat... Huge gaps of time between anything happening. I've had veeam tickets, iland tickets and now iland has a ticket with senior veeam people ongoing on our behalf. All anyone ever does is request logs but no answer on how to resolve.

I really want to know if cloud connect is working with people with hundred vms or close/more.

It seems to me like you almost need your own private cloud connect repo and even then I wonder how it would go.

Part of me thinks I will just wait until v10 and redo my setup and not use cloud connect, instead opting for copy mode to object storage.

Also thought of going to another provider, but then iland is the largest so it is scary on whether it would be any better. I've read on forums people having the same issues with cloud connect at larger sizes.

I can get whatever case number that iland opened on my behalf most recently for you, but I feel like they said they are already talking with higher level veeam guys.

I'm always staring at "cloud task not ready" or "initializing storage" for 20 or 30 min at a time or even more. Lots of times the job will never even process all vms in 24 hours.

We have 1 gb upload and the speed is fine when its running it is just the problem is that it is never consistently working away at the tasks... just constant unresponsive repo.

They have done all kinds of things like adjust the number of concurrent tasks... have us completely re seed to a new repository. Still the same bad results.

I'm going to end up losing my job due to inability to get backups off site and feel really let down. Veeam really shines on all local backup stuff but the cloud has been poor.

[anthonyspiteri79]

Hey there Alabaster.

There are a lot of variables to your post and the answers depend on specifics around both the SP and On-premises environments. Most issues with Cloud Connect Backup stem from the rate of change of those VM backups being copied up to a CCB repo vs physical variables like speed of connection, speed of disks, concurrency etc. I'm sure you have gone through all this with iLand and Veeam support? Sounds like iLand have done their part in trying to resolve the issue, but maybe a little more perseverance is required...

CCB is a solid product and has been very successful and reliable for us across all of our VCSPs who offer it as a service. The COPY mode you mention will certainly be an option, however there are a lot more pieces to get that setup... you will need a Scale Out Backup Repo locally to begin with, and then also understand what it means to offload/copy data into Object Storage. I know that iLand do offer Object Storage as part of their offering so it could be an option.

That said, I think it might be worth listing the support cases so that they can be followed up on.

[backupquestions]

Hi Anthony, thanks for your reply.

In our case, I don't believe it is the change rate. Here are the details. 350gb per day or less is what needs to be sent to CCB repo. We have 1 gb/sec upload rate, but this is of course limited by their ingest rate. We usually see anything from 30MB/sec to 45MB/sec or more (yes MB not mb) which is still decent and a third to half our upload ballpark area. Transferring 350GB can be done in only a few hours with basic math. Now I understand you have to add processing time for Veeam to calculate what to send and tasks to pick up and churn along etc, but even with this the normal times we had seen in the past were 6 hours or less and that is fine, but it just literally never works anymore no matter what they do for over 2 months now, and again they are supposed to be the very largest provider of this service for Veeam. So if I go elsewhere I don't know what to expect. I just need one job run every 24 hours is all I'm asking for.

I do use SOBR and offload already for my older archival GFS points via a backup copy job and that is working ok. But it seems like people will have to re design their jobs and take new active fulls to get everything simplified via one job with GFS rather than needing copy jobs too etc. I really don't want to have to re do everything to benefit from the simplification but I'll see how things turn out when that is available.

Anyway, back to the main topic... As soon as I get the ticket from them that I have asked for I will report it here. I really need this to be resolved soon or I don't know what I'm going to do. We are talking literally 2 months without being able to get recent backups off site no matter what I push on them or Veeam... It could be as simple as they have over provisioned their repo with too many customers, and the total concurrent tasks can not keep up, I don't know. But all we do is send logs to them and veeam and no one can seem to fix it. They told me to see how it goes this weekend, and well, as normal it got to 60 or so VMs out of 80 processed, and then its just stuck forever waiting on tasks to work.

"failed to start cloud repository agent" (or similar, going from memory here) "initializing storage", "cloud task not ready". etc...

[backupquestions]

Here is the ticket number on the Veeam side that they have been using. We had one good run last night for the first time in months, but I'm not holding my breath that it will continue stable, so we'll be monitoring. I'd like to have managers look over the ticket and ensure everything is being done that can be with iLand to resolve this. Veeam ticket 03909601 --- *note, this ticket itself is probably only a few days old, but iland has opened at least one other on my behalf, and ive opened some myself if you look at our support id which i can provide if needed. A lot of the effort has been between myself and iland so a lot of it is just ticket on their side.

[jgiardina]

Hi, all. We ran in to an issue with some RAID controller hardware via a documented vendor case some weeks ago, causing some issues on merge times. We are also working with Veeam engineering on another issue in one of our VCC stacks, DAL22, where VBR is causing delays to the back-end repositories. This is a new issue that we have come across, globally, and hope to have it fixed tomorrow.

@backupquestions - we will reach out directly to let you know the next steps.

[backupquestions]

For anyone following this thread, there is still no resolution on this. They received a "patch" from Veeam for their CCB environment and nothing is performing any different so far after that was done. They posted the same notice as last time that "jobs will experience delays in starting but should transfer at normal rates once started" (or similar to that statement) on their status page but that wasn't the case the first time around or this time. There are huge delays of several hours and then even when you get some traffic and processing, it still just drops out whenever and takes hours to resume. Last time they spent weeks saying there was "a backlog" of jobs on the environment so I wonder how long it will take for that to even clear, and even when it does will mine even work since it never did start working properly before.

iLand claims tier 3 is assisting them. What is going on and what are the details of the patch that was given to them? They asked me yet again for more logs which I provided today.

[rgouette]

My Case #04560663
Last post is > a year ago now, but I'm intensely curious how folk got on with the issue?
Did you resolve it? Did you move to another cloud provider?

I've been with iLand since March of 2020, and for a number of months I've had the dreaded "Error: Timeout to start agent"
I've been around the barn, with iLand & Veeam... logs are shipped, screenshots are taken, repositories are re-synced...
I'm seeing transfer sizes ranging from 38GB to 320GB
Our internet pipe is 50M/50M
VEEAM server is a VM
I do two backup jobs prior to my Cloud backup, and they run just fine.
At first I thought it was a bandwidth thing, because the WAN provider was seeing the pipe 'maxing out', but looking at
bandwidth reports from the vendor, I'm seeing times of failure that coincide with NON maxing bandwidth...
I don't know what else to do at this point, and it's pretty frustrating.