Discussions related to using object storage as a backup target.
walo
Novice
Posts: 3
Liked: never
Joined: Feb 13, 2024 9:46 am
Contact:

Re: QNAP QuObject Setup

Post by walo »

To be honest, the hardware is anything but optimal.
The more powerful one has twelve HDDs (WD RED Pro, 12 TB) installed and only eight GB of RAM - hence the normal QTS firmware. There is also no cache.
But the performance of the object storage was surprisingly good, we achieved around 1 GBit/s with Veeam during backup copy.
The aim was for everything to run via an S2S, at approx. 300 MBit/s.

There are image level VM copies that run from a local repository (directly attached) of the backup server to the NAS.

Shouldn't slow hardware tend to make everything slow? Not that the function breaks down completely?

Regards
RedVision81
Enthusiast
Posts: 49
Liked: 3 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: QNAP QuObject Setup

Post by RedVision81 »

i just can recommend switching to QuTS hero if possible. you also won't have the problem with max files per LUN, cause there is a limitation in normal QTS when formatting the LUN depending on blocksize
And you need to add a way more ram, at least 32 or better 64 GB imho. with the latest update of QuObjects there are running quite more processes than before: - Increased the number of container server processes to improve availability under heavy loading.

first it slows down, and then it breaks down, in the beginning with less files of course it would have enough speed, but with more and more objects stored in the database it will get slower and slower.
do you have set storage optimziation in your main vm backup job to 4MB? a technician told me shortly it would be much better with 4MB when you store them in an object storage, cause it would create less objects.

how many objects / TBs do you already have in QuObjects? Are you using just one bucket or several ones?
DE&C
Service Provider
Posts: 42
Liked: 32 times
Joined: Aug 07, 2017 11:51 am
Full Name: William
Location: Zurich, Switzerland
Contact:

Re: QNAP QuObject Setup

Post by DE&C »

je1000 wrote: Feb 08, 2024 1:09 pm Thanks for your post and I can confirm a lot your points, many vendors don’t handle well veeam object storage backups and Wasabi is one them.

I would love to hear more about your experiences and testing
Sorry for my late reply. I haven't received any notification.

I'm happy to provide you more information. What do you want to know? :)
RedVision81
Enthusiast
Posts: 49
Liked: 3 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: QNAP QuObject Setup

Post by RedVision81 »

we are still testing with QuObjects and Veeam, but it drives me more and more crazy :)
we have a very powerful Qnap (QuTShero) imho and a brand new backup server.
Qnap is directly connected to the backup server with 2 x 10 gig as Team (802.3ad dynamic and jumbo frames 9000)
created a storage space(shared folder with worm functionality for immutability) on a 12 x 16TB Raid 6 storage pool with 2 x 2 TB SSD Cache Raid 1 enabled
created two buckets in this storage space and added both as a backup repository with direct connection type to Veeam B+R.

set up a backup copy job including 3 VM Backup Jobs from our ROBOs to copy the Backups from local Backup Server SAS Storage (12 x 16TB - Raid 6) to Qnap Bucket.

Initial Sync of 1st Job with 50 GB Backup took 2,5 hour
2nd Job with 180 GB took 3,5 hour
3rd Job with 194 GB took over 6 hours.
the Incremental Sync in the night was much faster and only took about 20-30 Minutes for each job.

trying to restore a VM from the object storage repository failed everytime: failed to retrieve certificate ....
switching from direct connection type of the repository to gateway with local backup server as gateway - restore is working but not very fast.

now added a 2nd backup copy job for our main VMs (62 VMs 24,5 TB) its much faster in Gateway Mode, but transfer speed is just round about 100 MB/s and it sill stucks everytime for a few minutes. i dont know why the job stucks every 10-15 Minutes because all systems are idle in this time.

is there also an option for the backup copy job, not starting all objects at once, instead it would be better doing one item/vm after another.

any ideas?
DE&C
Service Provider
Posts: 42
Liked: 32 times
Joined: Aug 07, 2017 11:51 am
Full Name: William
Location: Zurich, Switzerland
Contact:

Re: QNAP QuObject Setup

Post by DE&C »

@RedVision81: Have you limited the number of concurrent tasks per bucket? If not: Try setting this to "1". This helps to reduce the load on the endpoint.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Certificate: Are you using a self-signed certificate? You need to make sure that all your Veeam components (VBR, proxy) accept the certificate.
Have you checked that no firewall / security tool etc. is interfering with the connection? https://www.veeam.com/kb4328

Jumbo frames: I would not use this. You are much more likely to just cause problems than solve anything. I've never seen jumbo frames with object storage: For systems that are set up correctly and optimized for Veeam, jumbo frames are not necessary to achieve reasonable performance.

In general: Have you asked QNAP support what the maximum performance can be expected for your setup (when using object storage)?

Some other points

“a technician told me shortly it would be much better with 4MB when you store them in an object storage, cause it would create less objects.”
That is “only” true for “unoptimized” systems that can’t handle millions of small objects. Optimized systems have no problem with the standard size (1MB). For a QNAP system you should definitifely go for the 4MB. Don’t forget to do an active full backup after changing it.

“shared folder with worm functionality for immutability)”.
Don’t do this. Go for object lock and let Veeam do the job. This will save you a lot of trouble in the future.
RedVision81
Enthusiast
Posts: 49
Liked: 3 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: QNAP QuObject Setup

Post by RedVision81 »

@DE&C
Have you limited the number of concurrent tasks per bucket? If not: Try setting this to "1". This helps to reduce the load on the endpoint.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120
i think this will not help, because there is always only one backup copy task running. but it's worth a try
Certificate: Are you using a self-signed certificate? You need to make sure that all your Veeam components (VBR, proxy) accept the certificate.
Have you checked that no firewall / security tool etc. is interfering with the connection? https://www.veeam.com/kb4328
yes, we are using a self-signed, and there is no problem while doing the regular copy jobs, just when i try to restore.
Update: Problem solved: didnt recognize restore job is using the (auto)-proxy from robo location, and of course this proxy has no direct connection to object storage. switching to the local backup server as proxy works without any issues.
Jumbo frames: I would not use this. You are much more likely to just cause problems than solve anything. I've never seen jumbo frames with object storage: For systems that are set up correctly and optimized for Veeam, jumbo frames are not necessary to achieve reasonable performance.
ok, good point, alreaday tried with 1500 on both sides, but there was no difference so far. was still enabled, because we used the qnap as an iscsi target before.
In general: Have you asked QNAP support what the maximum performance can be expected for your setup (when using object storage)?
not yet, i can try to raise a ticket, but i would except not really a suitable answer.
“shared folder with worm functionality for immutability)”. Don’t do this. Go for object lock and let Veeam do the job. This will save you a lot of trouble in the future.
you have to enable worm feature when creating a storage space in quobjects (which is the same as creating a shared folder in storage/snapshots menu), otherwise you cannot enable object lock on the bucket.
this should be fine imho.
RedVision81
Enthusiast
Posts: 49
Liked: 3 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: QNAP QuObject Setup

Post by RedVision81 »

Finally, i think i found another issue:
Disabeld NIC Teaming on both ends, and just using a 1to1 10GB Connection by now.
Its much faster, more stable, and not stucking anymore.
RedVision81
Enthusiast
Posts: 49
Liked: 3 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: QNAP QuObject Setup

Post by RedVision81 »

ok after some more investigating its not the nic team, its definitely the Qnap or better QuObjects Software.
Used S3 Browser Software now, and its the same behavior as in veeam, transferring some Gigs of Data and then it gets stucked:

a lot of errors in the logs from s3 browser:
[2024-04-05 15:50:25] ServiceUnavailable: Please reduce your request rate TaskID: 2273
[2024-04-05 15:54:03] InternalServerError: The remote server returned an error: (500) Internal Server Error.Response: <html><h1>Internal Error</h1><p>The server has either erred or is incapable of performing the requested operation.</p></html> TaskID: 2274

now i raised a ticket at qnap, lets see what they are telling me.
sfirmes
Veeam Software
Posts: 287
Liked: 138 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: QNAP QuObject Setup

Post by sfirmes »

We usually see this "ServiceUnavailable: Please reduce your request rate" or "(HTTP) 503 Service Unavailable server error response code" when the object storage platform is overwhelmed and can't handle the s3 api load being sent to it. Reducing the number of concurrent connections to the object storage is a good place to start. Here is the link: Configuring Object Storage Repositories

Hope this helps.

Steve
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
RedVision81
Enthusiast
Posts: 49
Liked: 3 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: QNAP QuObject Setup

Post by RedVision81 »

Hello Steve,
thanks i already set it to 1 and it still happens. imho there is a bug in quobjects software, because api requests are very low, and as already mentioned our qnap is very powerful (128GB RAM, SSD Cache, 12x16TB HDD) and this happens even if you copy a single file with 10-15 Gig.
Regards
Peter
sfirmes
Veeam Software
Posts: 287
Liked: 138 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: QNAP QuObject Setup

Post by sfirmes » 1 person likes this post

Thanks for the update Peter.

If you PM me and provide me the QNAP ticket#, I will try to see what info I can get from QNAP as well.

Thanks again.

Steve
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
DE&C
Service Provider
Posts: 42
Liked: 32 times
Joined: Aug 07, 2017 11:51 am
Full Name: William
Location: Zurich, Switzerland
Contact:

Re: QNAP QuObject Setup

Post by DE&C » 3 people like this post

RedVision81 wrote: Apr 08, 2024 5:44 am Hello Steve,
thanks i already set it to 1 and it still happens. imho there is a bug in quobjects software, because api requests are very low, and as already mentioned our qnap is very powerful (128GB RAM, SSD Cache, 12x16TB HDD) and this happens even if you copy a single file with 10-15 Gig.
Regards
Peter
it just doesnt’ matter at all. You can have SSDS, CPUs etc. The problem is not the «power» but how vendors implement metadata handling.

As per my first post: Your solution won’t work by design - and if it works, it will be very slowly (metadata handling).
I suggest you to read carefully my in detail description about metadata here, so you understand the problem that’s behind.

post489837.html#p489837

If you have a car with 500 horsepower but no wheels, it won't move a bit.Now we give this car the thinnest and cheapest wheels possible from a tricycle for kids. Yes, now it might move a few meters before the wheels fall off. But: the whole structure is not made for this - so it's not worth investing time in this thing. And if they don't fall of: You will move very slowly (yes it moves and we can call it a "vehicle" but you won't win a F1 race).

And the other question is: How resilient is the whole solution against cybet attacks. And here I highly question the QNAP solution (as a whole thing, not just immutability). There is a reason people are paying money for solutions to get immutable object storage for Veeam and a whole certified solution for cyber resilience.
gnalley
Influencer
Posts: 13
Liked: 6 times
Joined: Oct 17, 2016 2:32 pm
Full Name: Gary Nalley
Contact:

Re: QNAP QuObject Setup

Post by gnalley » 1 person likes this post

I second DE&C's posts. While you may be able to get a QNAP platform working; the continual effort to keep the platform working (specifically through all the QNAP software 'upgrades') will not be worth your efforts. Your satisfaction with the platform (support, reliability, performance, troubleshooting tools, etc.) will be less than satisfactory in an enterprise use case. IMO the value of the platform is simply not there. QNAP IMO is a consumer/prosumer level product. I would not bet my career on it. Any yes...I personally own two and have owned QNAP since 2012.
RedVision81
Enthusiast
Posts: 49
Liked: 3 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: QNAP QuObject Setup

Post by RedVision81 »

true words of course, and after the first experiences now with QuObjects we wont use it for production anymore.
but if you dont try you will never now ;)
DE&C
Service Provider
Posts: 42
Liked: 32 times
Joined: Aug 07, 2017 11:51 am
Full Name: William
Location: Zurich, Switzerland
Contact:

Re: QNAP QuObject Setup

Post by DE&C »

@RedVision81

If you have the right partner, you don't have to waste money or time. We can just tell our customer what solutions work out of the box with Veeam :)
RedVision81
Enthusiast
Posts: 49
Liked: 3 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: QNAP QuObject Setup

Post by RedVision81 »

@DE&C

its not about the right partner, its just about testing and developing solutions for our needs. i know of course there are several different solutions on the market, but we wanted to try with things first we already have.
ryan.christensen
Service Provider
Posts: 31
Liked: 5 times
Joined: Apr 28, 2021 8:47 pm
Full Name: Ryan Christensen
Contact:

Re: QNAP QuObject Setup

Post by ryan.christensen »

QNAP is advertising on their front page "QuObjects achieves Veeam Ready - Object with Immutability".

QTS Certification
QuTS Hero Certification

Veeam has certified using QuObjects. So as long as there's enough resources and we're using the version of QuObjects specified or newer we should be fine, right?
RedVision81
Enthusiast
Posts: 49
Liked: 3 times
Joined: Jan 14, 2016 8:02 am
Full Name: Peter
Contact:

Re: QNAP QuObject Setup

Post by RedVision81 »

We decided to stop using QuObjects. We opened a Ticket at Qnap regarding Performance Issues. After almost 8 weeks of turning circles with no valid help, we decided to close the ticket and revert everything regarding QuObjects. Its just a nother Qnap App which is not usable for production use.

You are right, QuObjects is Veeam Ready, but its just a "Proof of Concept" if it is working. There are no Performance or Long-Time Tests.
DE&C
Service Provider
Posts: 42
Liked: 32 times
Joined: Aug 07, 2017 11:51 am
Full Name: William
Location: Zurich, Switzerland
Contact:

Re: QNAP QuObject Setup

Post by DE&C »

ryan.christensen wrote: Jun 09, 2024 11:04 pm QNAP is advertising on their front page "QuObjects achieves Veeam Ready - Object with Immutability".

QTS Certification
QuTS Hero Certification

Veeam has certified using QuObjects. So as long as there's enough resources and we're using the version of QuObjects specified or newer we should be fine, right?
most problems are only discovered when it's too late. So don't test something for your backups that you rely on in the worst case. Get a partner who knows what they are doing if you want to use immutability against ransomware. Examples from other object storage vendors: 

  • A customer lost a lot of data on an object storage system. As long as the Veeam server had all the metadata information, it worked. As soon as it was out of sync and the metadata had to be resynchronized, the whole setup collapsed (2-3 months until synchronization).
  • Another example: After 7 TB of data and some days later when the object lock (immutability) was renewed, the system collapsed due to too many API calls.


* I'm not allowed to name the vendors officially. This "customers" became customers of us after they had the problem, the setup was done by another partner.


A big misunderstanding is: There is NO real certification from Veeam - it is a list for compatibility, not more (even if it is often called "certified").
The Veeam-ready database is more or less….. let’s calI it “questionable”.

I say this officially as VASP on a quarterly basis as feedback to Veeam. The Veeam-ready DB only says: The vendor was able to perform some very simple tests. The API call worked and no errors occurred during the tests. There is no standard methodology, no performance goals etc. Nothing that matters in real life scenarios. This is the only reason 90% or more of the vendors make it to this list. It is therefore more or less useless for real life.

Would Veeam create a real “certification program” with different tiers / levels with goals to achieve, 90% or more of the vendors would be on a “use it at your own risk” level.
ryan.christensen
Service Provider
Posts: 31
Liked: 5 times
Joined: Apr 28, 2021 8:47 pm
Full Name: Ryan Christensen
Contact:

Re: QNAP QuObject Setup

Post by ryan.christensen »

Yikes. Thanks for the feedback both of you.
IT0818
Novice
Posts: 8
Liked: never
Joined: Feb 07, 2023 9:14 am
Contact:

Re: QNAP QuObject Setup

Post by IT0818 »

Has anything improved with the last QuObjects 2.4.1721 update?
DE&C
Service Provider
Posts: 42
Liked: 32 times
Joined: Aug 07, 2017 11:51 am
Full Name: William
Location: Zurich, Switzerland
Contact:

Re: QNAP QuObject Setup

Post by DE&C »

@IT0818: I don't know about this version. They would have to change the entire design for handling metadata (and I don't expect them to do that). If you need object storage, go with an object storage solution that is proven with Veeam.
Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests