QNAP QuObject Setup

[walo]

To be honest, the hardware is anything but optimal.
The more powerful one has twelve HDDs (WD RED Pro, 12 TB) installed and only eight GB of RAM - hence the normal QTS firmware. There is also no cache.
But the performance of the object storage was surprisingly good, we achieved around 1 GBit/s with Veeam during backup copy.
The aim was for everything to run via an S2S, at approx. 300 MBit/s.

There are image level VM copies that run from a local repository (directly attached) of the backup server to the NAS.

Shouldn't slow hardware tend to make everything slow? Not that the function breaks down completely?

Regards

[RedVision81]

i just can recommend switching to QuTS hero if possible. you also won't have the problem with max files per LUN, cause there is a limitation in normal QTS when formatting the LUN depending on blocksize
And you need to add a way more ram, at least 32 or better 64 GB imho. with the latest update of QuObjects there are running quite more processes than before: - Increased the number of container server processes to improve availability under heavy loading.

first it slows down, and then it breaks down, in the beginning with less files of course it would have enough speed, but with more and more objects stored in the database it will get slower and slower.
do you have set storage optimziation in your main vm backup job to 4MB? a technician told me shortly it would be much better with 4MB when you store them in an object storage, cause it would create less objects.

how many objects / TBs do you already have in QuObjects? Are you using just one bucket or several ones?

[DE&C]

Thanks for your post and I can confirm a lot your points, many vendors don’t handle well veeam object storage backups and Wasabi is one them.

I would love to hear more about your experiences and testing

Sorry for my late reply. I haven't received any notification.

I'm happy to provide you more information. What do you want to know?

[RedVision81]

we are still testing with QuObjects and Veeam, but it drives me more and more crazy
we have a very powerful Qnap (QuTShero) imho and a brand new backup server.
Qnap is directly connected to the backup server with 2 x 10 gig as Team (802.3ad dynamic and jumbo frames 9000)
created a storage space(shared folder with worm functionality for immutability) on a 12 x 16TB Raid 6 storage pool with 2 x 2 TB SSD Cache Raid 1 enabled
created two buckets in this storage space and added both as a backup repository with direct connection type to Veeam B+R.

set up a backup copy job including 3 VM Backup Jobs from our ROBOs to copy the Backups from local Backup Server SAS Storage (12 x 16TB - Raid 6) to Qnap Bucket.

Initial Sync of 1st Job with 50 GB Backup took 2,5 hour
2nd Job with 180 GB took 3,5 hour
3rd Job with 194 GB took over 6 hours.
the Incremental Sync in the night was much faster and only took about 20-30 Minutes for each job.

trying to restore a VM from the object storage repository failed everytime: failed to retrieve certificate ....
switching from direct connection type of the repository to gateway with local backup server as gateway - restore is working but not very fast.

now added a 2nd backup copy job for our main VMs (62 VMs 24,5 TB) its much faster in Gateway Mode, but transfer speed is just round about 100 MB/s and it sill stucks everytime for a few minutes. i dont know why the job stucks every 10-15 Minutes because all systems are idle in this time.

is there also an option for the backup copy job, not starting all objects at once, instead it would be better doing one item/vm after another.

any ideas?

[DE&C]

@RedVision81: Have you limited the number of concurrent tasks per bucket? If not: Try setting this to "1". This helps to reduce the load on the endpoint.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Certificate: Are you using a self-signed certificate? You need to make sure that all your Veeam components (VBR, proxy) accept the certificate.
Have you checked that no firewall / security tool etc. is interfering with the connection? https://www.veeam.com/kb4328

Jumbo frames: I would not use this. You are much more likely to just cause problems than solve anything. I've never seen jumbo frames with object storage: For systems that are set up correctly and optimized for Veeam, jumbo frames are not necessary to achieve reasonable performance.

In general: Have you asked QNAP support what the maximum performance can be expected for your setup (when using object storage)?

Some other points

“a technician told me shortly it would be much better with 4MB when you store them in an object storage, cause it would create less objects.”
That is “only” true for “unoptimized” systems that can’t handle millions of small objects. Optimized systems have no problem with the standard size (1MB). For a QNAP system you should definitifely go for the 4MB. Don’t forget to do an active full backup after changing it.

“shared folder with worm functionality for immutability)”. Don’t do this. Go for object lock and let Veeam do the job. This will save you a lot of trouble in the future.

[RedVision81]

@DE&C

Have you limited the number of concurrent tasks per bucket? If not: Try setting this to "1". This helps to reduce the load on the endpoint.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

i think this will not help, because there is always only one backup copy task running. but it's worth a try

Certificate: Are you using a self-signed certificate? You need to make sure that all your Veeam components (VBR, proxy) accept the certificate.
Have you checked that no firewall / security tool etc. is interfering with the connection? https://www.veeam.com/kb4328

yes, we are using a self-signed, and there is no problem while doing the regular copy jobs, just when i try to restore.
Update: Problem solved: didnt recognize restore job is using the (auto)-proxy from robo location, and of course this proxy has no direct connection to object storage. switching to the local backup server as proxy works without any issues.

Jumbo frames: I would not use this. You are much more likely to just cause problems than solve anything. I've never seen jumbo frames with object storage: For systems that are set up correctly and optimized for Veeam, jumbo frames are not necessary to achieve reasonable performance.

ok, good point, alreaday tried with 1500 on both sides, but there was no difference so far. was still enabled, because we used the qnap as an iscsi target before.

In general: Have you asked QNAP support what the maximum performance can be expected for your setup (when using object storage)?

not yet, i can try to raise a ticket, but i would except not really a suitable answer.

“shared folder with worm functionality for immutability)”. Don’t do this. Go for object lock and let Veeam do the job. This will save you a lot of trouble in the future.

you have to enable worm feature when creating a storage space in quobjects (which is the same as creating a shared folder in storage/snapshots menu), otherwise you cannot enable object lock on the bucket.
this should be fine imho.

[RedVision81]

Finally, i think i found another issue:
Disabeld NIC Teaming on both ends, and just using a 1to1 10GB Connection by now.
Its much faster, more stable, and not stucking anymore.

[RedVision81]

ok after some more investigating its not the nic team, its definitely the Qnap or better QuObjects Software.
Used S3 Browser Software now, and its the same behavior as in veeam, transferring some Gigs of Data and then it gets stucked:

a lot of errors in the logs from s3 browser:
[2024-04-05 15:50:25] ServiceUnavailable: Please reduce your request rate TaskID: 2273
[2024-04-05 15:54:03] InternalServerError: The remote server returned an error: (500) Internal Server Error.Response: <html><h1>Internal Error</h1><p>The server has either erred or is incapable of performing the requested operation.</p></html> TaskID: 2274

now i raised a ticket at qnap, lets see what they are telling me.

[sfirmes]

We usually see this "ServiceUnavailable: Please reduce your request rate" or "(HTTP) 503 Service Unavailable server error response code" when the object storage platform is overwhelmed and can't handle the s3 api load being sent to it. Reducing the number of concurrent connections to the object storage is a good place to start. Here is the link: Configuring Object Storage Repositories

Hope this helps.

Steve

[RedVision81]

Hello Steve,
thanks i already set it to 1 and it still happens. imho there is a bug in quobjects software, because api requests are very low, and as already mentioned our qnap is very powerful (128GB RAM, SSD Cache, 12x16TB HDD) and this happens even if you copy a single file with 10-15 Gig.
Regards
Peter

[sfirmes]

Thanks for the update Peter.

If you PM me and provide me the QNAP ticket#, I will try to see what info I can get from QNAP as well.

Thanks again.

Steve

[DE&C]

Hello Steve,
thanks i already set it to 1 and it still happens. imho there is a bug in quobjects software, because api requests are very low, and as already mentioned our qnap is very powerful (128GB RAM, SSD Cache, 12x16TB HDD) and this happens even if you copy a single file with 10-15 Gig.
Regards
Peter

it just doesnt’ matter at all. You can have SSDS, CPUs etc. The problem is not the «power» but how vendors implement metadata handling.

As per my first post: Your solution won’t work by design - and if it works, it will be very slowly (metadata handling).
I suggest you to read carefully my in detail description about metadata here, so you understand the problem that’s behind.

post489837.html#p489837

If you have a car with 500 horsepower but no wheels, it won't move a bit.Now we give this car the thinnest and cheapest wheels possible from a tricycle for kids. Yes, now it might move a few meters before the wheels fall off. But: the whole structure is not made for this - so it's not worth investing time in this thing. And if they don't fall of: You will move very slowly (yes it moves and we can call it a "vehicle" but you won't win a F1 race).

And the other question is: How resilient is the whole solution against cybet attacks. And here I highly question the QNAP solution (as a whole thing, not just immutability). There is a reason people are paying money for solutions to get immutable object storage for Veeam and a whole certified solution for cyber resilience.

[gnalley]

I second DE&C's posts. While you may be able to get a QNAP platform working; the continual effort to keep the platform working (specifically through all the QNAP software 'upgrades') will not be worth your efforts. Your satisfaction with the platform (support, reliability, performance, troubleshooting tools, etc.) will be less than satisfactory in an enterprise use case. IMO the value of the platform is simply not there. QNAP IMO is a consumer/prosumer level product. I would not bet my career on it. Any yes...I personally own two and have owned QNAP since 2012.

[RedVision81]

true words of course, and after the first experiences now with QuObjects we wont use it for production anymore.
but if you dont try you will never now

[DE&C]

@RedVision81

If you have the right partner, you don't have to waste money or time. We can just tell our customer what solutions work out of the box with Veeam

[RedVision81]

@DE&C

its not about the right partner, its just about testing and developing solutions for our needs. i know of course there are several different solutions on the market, but we wanted to try with things first we already have.