Cloud Connect continual issues with Merging incrmentals

[ckent]

Has anyone got this working properly ?
i seem to having continual and ongoing problems with the cloud connect.
Initial seeding and then daily backups work but as soon as the first attempt to merge the oldest incrmental into a full back up and the thing just falls over.

our cloud connect partner seem nice enough and are talking me they are working with veeam.
but its been several weeks now and we don't seem to be making any progress.
All the time i'm half concerned my off site backups cannot be trusted.

[dellock6]

As a first check, what amount of free space you have when the retention is reached and the first merge starts? The cloud repository needs to have enough free space to process the merge properly.

[veremin]

Can you provide with the exact error the job fails? Thanks.

[ckent]

My question was if anyone has gotten this to work properly.
Not a request for support as that avenue has been well and truly traversed with little progress.

[veremin]

Has anyone got this working properly?

Sure.

Since most of the jobs pointed to cloud repositories use forward forever incremental mode which implies regular increments merge, we would get tons of complains coming from both parties (SP, tenant) , if forward forever incremental mode didn't work for anyone at all.

Thanks.

[ckent]

i was hoping to hear from other customers who have got this working and have first hand experience.
I want to know if or how my set-up differs.
I like my SP but if they cannot get thier end working correctly then i will have to drop them.

[veremin]

The setup or particular settings do not matter, as merging process should not fail at first place. Thus, we asked questions regarding specific error and/or conditions under which job failed.

Anyway, you'd better investigate this issue with both our support team and your service provider, since only they can confirm both environments as well as analyze logs.

Thanks.

[ckent]

I already have a ticket which has been open for a month now and this isn't the first ticket i have logged on this matter.
all sorts of logs have gone back and forth, to be frank support is little more than scripted replys.
the first technician to deal with this ticket refused point blank to even speak with the service provider.

hence i was not asking for your support or blanket statements such as "it should not fail"
Neither of these help my situation or bring me any closer to a resolution.

what I was asking for was some users who have real world experience of this working for them.
Because a cloud based back up solution is no solution if it fails each time it wants to perform a merge.

[veremin]

Understood. Then, let's wait till other users chime in.

In the meantime, may I ask you to provide your support ticket number (either the latest one or the one where this matter has been discussed)? We'd be interested to follow the case and check what can be done in this regard.

Thank you for your understanding.

[ckent]

01716538

[csinetops]

We have this service with SingleHop and it works great. We did have a issue like you describe and the vendor had it resolved in one day. I think you need to lean on the vendor for a resolution! A month of failed roll ups is no good... Or, switch vendors, they are not all equal as I found out while demoing cloud connect vendors.

[ckent]

So after 2 months of working with the supplier and Veeam on this topic.
We have a resolution.

the vendor says and I quote
I spoke to Veeam yesterday at a meeting I had with a senior manager and they told me that the GFS part of cloud connect is not perfect at this stage and that many providers have similar problems.

Now I have been dealing with Vendors for 20 odd years and the number of people I trust I can count on one hand.
this is a vendor and specific individual who I do trust.

The resolution was to cancel the Veeam cloud connect service at a cost of £500 per month and go with an alternative product.

[veremin]

the vendor says and I quote

By vendor you mean the manufacturer of device that is used as a target repository by your Service Provider?

It seems from your case that CIFS share is leveraged as a repository on which tenants cloud repositories are created.

Generally speaking, NAS devices configured as CIFS shares are not the best choice for jobs involving synthetic activity, because they cannot cope with random I/O well. You can add Cloud Connect to this equation, and the situation would be even worst: multiple tenants' jobs, multiple synthetic transformation activities happening at the same moment - all of that would result in decreased performance for sure.

I may be mistaken here, but the device appears to be under-provisioned in terms of I/O.

So, with that in mind, I find the quoted statement a bit confusing, especially, a part about GFS functionality leaving much to be desired in Cloud Connect, as it works identically in both cases (local or cloud repository).

Thanks.

[martynuk]

As a Veeam customer we have been using Cloud Connect in production for a couple of weeks now (following a month-long trial), and looks good so far. Merges have been working fine for us.

[ckent]

By vendor you mean the manufacturer of device that is used as a target repository by your Service Provider?

By Vendor i mean the Veeam cloud connect partner who we use, selected from the list on this website.
And ones who have been working tirelessly with Dmitriy Mironov from Veeam to get this fixed.
And who have now been told as per my quote above that this is a known issue to Veeam.

Maybe Dmitriy Mironov would like to comment as he has been working on this ticket for well over a month now.
So Dmitry was it an issue with over provision of your partners storage ?
If so please let me know as i will walk away from them within a second.

[ckent]

Also v.Eremin the ticket number for this case is 01716538
Please feel free to assist Dmitriy as you see fit as ive been waiting over a month for a resolution.
If your cloud connect partner is being over provisioning as you say and that is the cause of my issues then i need to know.

[ckent]

As a Veeam customer we have been using Cloud Connect in production for a couple of weeks now (following a month-long trial), and looks good so far. Merges have been working fine for us.

Martin..
is it safe to assume the uk at the end of you nick is for UK ?
Is you cloud connect partner based in the UK and what sort of performance do you get.
Can you PM me some details of who they are.

Cheers

[martynuk]

ckent,

I've just sent you details of our provider (both we, and they, are in the UK).

They offer a free trial, so probably worth you talking to them, even if only to narrow down where your problems might be.

Good luck,
Martin

[ckent]

i see Veeam staff are very quick to defend the product and make general assumptions.
But when it comes to providing any actual help there no where to be seen.

Nice product when it works.
Consumer support when it dosnt.

[veremin]

But when it comes to providing any actual help there no where to be seen.

We've already contacted your support engineer to understand the details of your issue better before making any definite statements. Thanks.

[EugeneK]

The merge tasks are normally limited by the hardware capabilities: undersized Repositories (CPU/RAM), multiple Tenants hammering the same backend storage or overworked storage in terms of IO are the common factors.
The majority of NAS solutions are relying on RAID6 form factor, where the random IOPS are absorbed by the layer of SSD disks.
With Merge operations specifically, it requires not only write IO to be handled well (which is normally processed efficiently by the majority of the modern storage solutions), but also Reads, which are fairly random.
Random reads are typically at blame for the poor storage performance, since the penalty of not having efficient processing of this data and hitting raid6 array disks directly decreases the overall performance dramatically.

I'd recommend to ask your provider to run the tests on their end in terms of performance you can expect from the particular setup. Things like Tenants placement more evenly across the infrastructure or re-sizing repositories for CPU/RAM may go a long way.
While the SP is not necessarily over-provisioned, and the issue may be relevant to the corner case and how Veeam processes your particular set of data, chances are the problem is rather in the realm of bottlenecks introduced by the backend hardware in use to process your data, rather than Veeam software itself.

[Javezim]

One Query, Products like Shadowprotect/Imagemanager require nothing from the client side to process consolidation (ie. Merges)
Why is it that Veeam requires a constant connection from the client during the merge process?
Surely it would be easier if the File was just uploaded and on the Service Providers end Retention could be set to perform the merge.

[mongie]

Hi CKent,

We've been trialing VCC with a couple of providers (SingleHop, iland, VPLS) and I found with two of them, I couldn't restore guest windows files at all, the restore job would time out while mounting the backup files. I raised this with the first trial company, and they logged a ticket with Veeam - but I heard little back. I also logged a ticket myself, but the response I got from Veeam support was to confirm what version the provider was using, as they have to be identical (i'm fairly sure this is incorrect). I then went on leave and the ticket lapsed, but I found that there is obviously an importance on the underlying storage that the vendor is providing.

tldr; I feel your pain.