-
- Novice
- Posts: 3
- Liked: never
- Joined: Feb 13, 2024 9:46 am
- Contact:
Re: QNAP QuObject Setup
To be honest, the hardware is anything but optimal.
The more powerful one has twelve HDDs (WD RED Pro, 12 TB) installed and only eight GB of RAM - hence the normal QTS firmware. There is also no cache.
But the performance of the object storage was surprisingly good, we achieved around 1 GBit/s with Veeam during backup copy.
The aim was for everything to run via an S2S, at approx. 300 MBit/s.
There are image level VM copies that run from a local repository (directly attached) of the backup server to the NAS.
Shouldn't slow hardware tend to make everything slow? Not that the function breaks down completely?
Regards
The more powerful one has twelve HDDs (WD RED Pro, 12 TB) installed and only eight GB of RAM - hence the normal QTS firmware. There is also no cache.
But the performance of the object storage was surprisingly good, we achieved around 1 GBit/s with Veeam during backup copy.
The aim was for everything to run via an S2S, at approx. 300 MBit/s.
There are image level VM copies that run from a local repository (directly attached) of the backup server to the NAS.
Shouldn't slow hardware tend to make everything slow? Not that the function breaks down completely?
Regards
-
- Enthusiast
- Posts: 49
- Liked: 3 times
- Joined: Jan 14, 2016 8:02 am
- Full Name: Peter
- Contact:
Re: QNAP QuObject Setup
i just can recommend switching to QuTS hero if possible. you also won't have the problem with max files per LUN, cause there is a limitation in normal QTS when formatting the LUN depending on blocksize
And you need to add a way more ram, at least 32 or better 64 GB imho. with the latest update of QuObjects there are running quite more processes than before: - Increased the number of container server processes to improve availability under heavy loading.
first it slows down, and then it breaks down, in the beginning with less files of course it would have enough speed, but with more and more objects stored in the database it will get slower and slower.
do you have set storage optimziation in your main vm backup job to 4MB? a technician told me shortly it would be much better with 4MB when you store them in an object storage, cause it would create less objects.
how many objects / TBs do you already have in QuObjects? Are you using just one bucket or several ones?
And you need to add a way more ram, at least 32 or better 64 GB imho. with the latest update of QuObjects there are running quite more processes than before: - Increased the number of container server processes to improve availability under heavy loading.
first it slows down, and then it breaks down, in the beginning with less files of course it would have enough speed, but with more and more objects stored in the database it will get slower and slower.
do you have set storage optimziation in your main vm backup job to 4MB? a technician told me shortly it would be much better with 4MB when you store them in an object storage, cause it would create less objects.
how many objects / TBs do you already have in QuObjects? Are you using just one bucket or several ones?
-
- Service Provider
- Posts: 39
- Liked: 32 times
- Joined: Aug 07, 2017 11:51 am
- Full Name: William
- Location: Zurich, Switzerland
- Contact:
Re: QNAP QuObject Setup
Sorry for my late reply. I haven't received any notification.
I'm happy to provide you more information. What do you want to know?
-
- Enthusiast
- Posts: 49
- Liked: 3 times
- Joined: Jan 14, 2016 8:02 am
- Full Name: Peter
- Contact:
Re: QNAP QuObject Setup
we are still testing with QuObjects and Veeam, but it drives me more and more crazy
we have a very powerful Qnap (QuTShero) imho and a brand new backup server.
Qnap is directly connected to the backup server with 2 x 10 gig as Team (802.3ad dynamic and jumbo frames 9000)
created a storage space(shared folder with worm functionality for immutability) on a 12 x 16TB Raid 6 storage pool with 2 x 2 TB SSD Cache Raid 1 enabled
created two buckets in this storage space and added both as a backup repository with direct connection type to Veeam B+R.
set up a backup copy job including 3 VM Backup Jobs from our ROBOs to copy the Backups from local Backup Server SAS Storage (12 x 16TB - Raid 6) to Qnap Bucket.
Initial Sync of 1st Job with 50 GB Backup took 2,5 hour
2nd Job with 180 GB took 3,5 hour
3rd Job with 194 GB took over 6 hours.
the Incremental Sync in the night was much faster and only took about 20-30 Minutes for each job.
trying to restore a VM from the object storage repository failed everytime: failed to retrieve certificate ....
switching from direct connection type of the repository to gateway with local backup server as gateway - restore is working but not very fast.
now added a 2nd backup copy job for our main VMs (62 VMs 24,5 TB) its much faster in Gateway Mode, but transfer speed is just round about 100 MB/s and it sill stucks everytime for a few minutes. i dont know why the job stucks every 10-15 Minutes because all systems are idle in this time.
is there also an option for the backup copy job, not starting all objects at once, instead it would be better doing one item/vm after another.
any ideas?
we have a very powerful Qnap (QuTShero) imho and a brand new backup server.
Qnap is directly connected to the backup server with 2 x 10 gig as Team (802.3ad dynamic and jumbo frames 9000)
created a storage space(shared folder with worm functionality for immutability) on a 12 x 16TB Raid 6 storage pool with 2 x 2 TB SSD Cache Raid 1 enabled
created two buckets in this storage space and added both as a backup repository with direct connection type to Veeam B+R.
set up a backup copy job including 3 VM Backup Jobs from our ROBOs to copy the Backups from local Backup Server SAS Storage (12 x 16TB - Raid 6) to Qnap Bucket.
Initial Sync of 1st Job with 50 GB Backup took 2,5 hour
2nd Job with 180 GB took 3,5 hour
3rd Job with 194 GB took over 6 hours.
the Incremental Sync in the night was much faster and only took about 20-30 Minutes for each job.
trying to restore a VM from the object storage repository failed everytime: failed to retrieve certificate ....
switching from direct connection type of the repository to gateway with local backup server as gateway - restore is working but not very fast.
now added a 2nd backup copy job for our main VMs (62 VMs 24,5 TB) its much faster in Gateway Mode, but transfer speed is just round about 100 MB/s and it sill stucks everytime for a few minutes. i dont know why the job stucks every 10-15 Minutes because all systems are idle in this time.
is there also an option for the backup copy job, not starting all objects at once, instead it would be better doing one item/vm after another.
any ideas?
-
- Service Provider
- Posts: 39
- Liked: 32 times
- Joined: Aug 07, 2017 11:51 am
- Full Name: William
- Location: Zurich, Switzerland
- Contact:
Re: QNAP QuObject Setup
@RedVision81: Have you limited the number of concurrent tasks per bucket? If not: Try setting this to "1". This helps to reduce the load on the endpoint.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120
Certificate: Are you using a self-signed certificate? You need to make sure that all your Veeam components (VBR, proxy) accept the certificate.
Have you checked that no firewall / security tool etc. is interfering with the connection? https://www.veeam.com/kb4328
Jumbo frames: I would not use this. You are much more likely to just cause problems than solve anything. I've never seen jumbo frames with object storage: For systems that are set up correctly and optimized for Veeam, jumbo frames are not necessary to achieve reasonable performance.
In general: Have you asked QNAP support what the maximum performance can be expected for your setup (when using object storage)?
Some other points
“a technician told me shortly it would be much better with 4MB when you store them in an object storage, cause it would create less objects.”
That is “only” true for “unoptimized” systems that can’t handle millions of small objects. Optimized systems have no problem with the standard size (1MB). For a QNAP system you should definitifely go for the 4MB. Don’t forget to do an active full backup after changing it.
“shared folder with worm functionality for immutability)”. Don’t do this. Go for object lock and let Veeam do the job. This will save you a lot of trouble in the future.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120
Certificate: Are you using a self-signed certificate? You need to make sure that all your Veeam components (VBR, proxy) accept the certificate.
Have you checked that no firewall / security tool etc. is interfering with the connection? https://www.veeam.com/kb4328
Jumbo frames: I would not use this. You are much more likely to just cause problems than solve anything. I've never seen jumbo frames with object storage: For systems that are set up correctly and optimized for Veeam, jumbo frames are not necessary to achieve reasonable performance.
In general: Have you asked QNAP support what the maximum performance can be expected for your setup (when using object storage)?
Some other points
“a technician told me shortly it would be much better with 4MB when you store them in an object storage, cause it would create less objects.”
That is “only” true for “unoptimized” systems that can’t handle millions of small objects. Optimized systems have no problem with the standard size (1MB). For a QNAP system you should definitifely go for the 4MB. Don’t forget to do an active full backup after changing it.
“shared folder with worm functionality for immutability)”. Don’t do this. Go for object lock and let Veeam do the job. This will save you a lot of trouble in the future.
-
- Enthusiast
- Posts: 49
- Liked: 3 times
- Joined: Jan 14, 2016 8:02 am
- Full Name: Peter
- Contact:
Re: QNAP QuObject Setup
@DE&C
Update: Problem solved: didnt recognize restore job is using the (auto)-proxy from robo location, and of course this proxy has no direct connection to object storage. switching to the local backup server as proxy works without any issues.
this should be fine imho.
i think this will not help, because there is always only one backup copy task running. but it's worth a tryHave you limited the number of concurrent tasks per bucket? If not: Try setting this to "1". This helps to reduce the load on the endpoint.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120
yes, we are using a self-signed, and there is no problem while doing the regular copy jobs, just when i try to restore.Certificate: Are you using a self-signed certificate? You need to make sure that all your Veeam components (VBR, proxy) accept the certificate.
Have you checked that no firewall / security tool etc. is interfering with the connection? https://www.veeam.com/kb4328
Update: Problem solved: didnt recognize restore job is using the (auto)-proxy from robo location, and of course this proxy has no direct connection to object storage. switching to the local backup server as proxy works without any issues.
ok, good point, alreaday tried with 1500 on both sides, but there was no difference so far. was still enabled, because we used the qnap as an iscsi target before.Jumbo frames: I would not use this. You are much more likely to just cause problems than solve anything. I've never seen jumbo frames with object storage: For systems that are set up correctly and optimized for Veeam, jumbo frames are not necessary to achieve reasonable performance.
not yet, i can try to raise a ticket, but i would except not really a suitable answer.In general: Have you asked QNAP support what the maximum performance can be expected for your setup (when using object storage)?
you have to enable worm feature when creating a storage space in quobjects (which is the same as creating a shared folder in storage/snapshots menu), otherwise you cannot enable object lock on the bucket.“shared folder with worm functionality for immutability)”. Don’t do this. Go for object lock and let Veeam do the job. This will save you a lot of trouble in the future.
this should be fine imho.
-
- Enthusiast
- Posts: 49
- Liked: 3 times
- Joined: Jan 14, 2016 8:02 am
- Full Name: Peter
- Contact:
Re: QNAP QuObject Setup
Finally, i think i found another issue:
Disabeld NIC Teaming on both ends, and just using a 1to1 10GB Connection by now.
Its much faster, more stable, and not stucking anymore.
Disabeld NIC Teaming on both ends, and just using a 1to1 10GB Connection by now.
Its much faster, more stable, and not stucking anymore.
-
- Enthusiast
- Posts: 49
- Liked: 3 times
- Joined: Jan 14, 2016 8:02 am
- Full Name: Peter
- Contact:
Re: QNAP QuObject Setup
ok after some more investigating its not the nic team, its definitely the Qnap or better QuObjects Software.
Used S3 Browser Software now, and its the same behavior as in veeam, transferring some Gigs of Data and then it gets stucked:
a lot of errors in the logs from s3 browser:
[2024-04-05 15:50:25] ServiceUnavailable: Please reduce your request rate TaskID: 2273
[2024-04-05 15:54:03] InternalServerError: The remote server returned an error: (500) Internal Server Error.Response: <html><h1>Internal Error</h1><p>The server has either erred or is incapable of performing the requested operation.</p></html> TaskID: 2274
now i raised a ticket at qnap, lets see what they are telling me.
Used S3 Browser Software now, and its the same behavior as in veeam, transferring some Gigs of Data and then it gets stucked:
a lot of errors in the logs from s3 browser:
[2024-04-05 15:50:25] ServiceUnavailable: Please reduce your request rate TaskID: 2273
[2024-04-05 15:54:03] InternalServerError: The remote server returned an error: (500) Internal Server Error.Response: <html><h1>Internal Error</h1><p>The server has either erred or is incapable of performing the requested operation.</p></html> TaskID: 2274
now i raised a ticket at qnap, lets see what they are telling me.
-
- Veeam Software
- Posts: 287
- Liked: 138 times
- Joined: Jul 24, 2018 8:38 pm
- Full Name: Stephen Firmes
- Contact:
Re: QNAP QuObject Setup
We usually see this "ServiceUnavailable: Please reduce your request rate" or "(HTTP) 503 Service Unavailable server error response code" when the object storage platform is overwhelmed and can't handle the s3 api load being sent to it. Reducing the number of concurrent connections to the object storage is a good place to start. Here is the link: Configuring Object Storage Repositories
Hope this helps.
Steve
Hope this helps.
Steve
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
-
- Enthusiast
- Posts: 49
- Liked: 3 times
- Joined: Jan 14, 2016 8:02 am
- Full Name: Peter
- Contact:
Re: QNAP QuObject Setup
Hello Steve,
thanks i already set it to 1 and it still happens. imho there is a bug in quobjects software, because api requests are very low, and as already mentioned our qnap is very powerful (128GB RAM, SSD Cache, 12x16TB HDD) and this happens even if you copy a single file with 10-15 Gig.
Regards
Peter
thanks i already set it to 1 and it still happens. imho there is a bug in quobjects software, because api requests are very low, and as already mentioned our qnap is very powerful (128GB RAM, SSD Cache, 12x16TB HDD) and this happens even if you copy a single file with 10-15 Gig.
Regards
Peter
-
- Veeam Software
- Posts: 287
- Liked: 138 times
- Joined: Jul 24, 2018 8:38 pm
- Full Name: Stephen Firmes
- Contact:
Re: QNAP QuObject Setup
Thanks for the update Peter.
If you PM me and provide me the QNAP ticket#, I will try to see what info I can get from QNAP as well.
Thanks again.
Steve
If you PM me and provide me the QNAP ticket#, I will try to see what info I can get from QNAP as well.
Thanks again.
Steve
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
-
- Service Provider
- Posts: 39
- Liked: 32 times
- Joined: Aug 07, 2017 11:51 am
- Full Name: William
- Location: Zurich, Switzerland
- Contact:
Re: QNAP QuObject Setup
it just doesnt’ matter at all. You can have SSDS, CPUs etc. The problem is not the «power» but how vendors implement metadata handling.RedVision81 wrote: ↑Apr 08, 2024 5:44 am Hello Steve,
thanks i already set it to 1 and it still happens. imho there is a bug in quobjects software, because api requests are very low, and as already mentioned our qnap is very powerful (128GB RAM, SSD Cache, 12x16TB HDD) and this happens even if you copy a single file with 10-15 Gig.
Regards
Peter
As per my first post: Your solution won’t work by design - and if it works, it will be very slowly (metadata handling).
I suggest you to read carefully my in detail description about metadata here, so you understand the problem that’s behind.
post489837.html#p489837
If you have a car with 500 horsepower but no wheels, it won't move a bit.Now we give this car the thinnest and cheapest wheels possible from a tricycle for kids. Yes, now it might move a few meters before the wheels fall off. But: the whole structure is not made for this - so it's not worth investing time in this thing. And if they don't fall of: You will move very slowly (yes it moves and we can call it a "vehicle" but you won't win a F1 race).
And the other question is: How resilient is the whole solution against cybet attacks. And here I highly question the QNAP solution (as a whole thing, not just immutability). There is a reason people are paying money for solutions to get immutable object storage for Veeam and a whole certified solution for cyber resilience.
-
- Influencer
- Posts: 13
- Liked: 6 times
- Joined: Oct 17, 2016 2:32 pm
- Full Name: Gary Nalley
- Contact:
Re: QNAP QuObject Setup
I second DE&C's posts. While you may be able to get a QNAP platform working; the continual effort to keep the platform working (specifically through all the QNAP software 'upgrades') will not be worth your efforts. Your satisfaction with the platform (support, reliability, performance, troubleshooting tools, etc.) will be less than satisfactory in an enterprise use case. IMO the value of the platform is simply not there. QNAP IMO is a consumer/prosumer level product. I would not bet my career on it. Any yes...I personally own two and have owned QNAP since 2012.
-
- Enthusiast
- Posts: 49
- Liked: 3 times
- Joined: Jan 14, 2016 8:02 am
- Full Name: Peter
- Contact:
Re: QNAP QuObject Setup
true words of course, and after the first experiences now with QuObjects we wont use it for production anymore.
but if you dont try you will never now
but if you dont try you will never now
-
- Service Provider
- Posts: 39
- Liked: 32 times
- Joined: Aug 07, 2017 11:51 am
- Full Name: William
- Location: Zurich, Switzerland
- Contact:
Re: QNAP QuObject Setup
@RedVision81
If you have the right partner, you don't have to waste money or time. We can just tell our customer what solutions work out of the box with Veeam
If you have the right partner, you don't have to waste money or time. We can just tell our customer what solutions work out of the box with Veeam
-
- Enthusiast
- Posts: 49
- Liked: 3 times
- Joined: Jan 14, 2016 8:02 am
- Full Name: Peter
- Contact:
Re: QNAP QuObject Setup
@DE&C
its not about the right partner, its just about testing and developing solutions for our needs. i know of course there are several different solutions on the market, but we wanted to try with things first we already have.
its not about the right partner, its just about testing and developing solutions for our needs. i know of course there are several different solutions on the market, but we wanted to try with things first we already have.
-
- Service Provider
- Posts: 31
- Liked: 5 times
- Joined: Apr 28, 2021 8:47 pm
- Full Name: Ryan Christensen
- Contact:
Re: QNAP QuObject Setup
QNAP is advertising on their front page "QuObjects achieves Veeam Ready - Object with Immutability".
QTS Certification
QuTS Hero Certification
Veeam has certified using QuObjects. So as long as there's enough resources and we're using the version of QuObjects specified or newer we should be fine, right?
QTS Certification
QuTS Hero Certification
Veeam has certified using QuObjects. So as long as there's enough resources and we're using the version of QuObjects specified or newer we should be fine, right?
-
- Enthusiast
- Posts: 49
- Liked: 3 times
- Joined: Jan 14, 2016 8:02 am
- Full Name: Peter
- Contact:
Re: QNAP QuObject Setup
We decided to stop using QuObjects. We opened a Ticket at Qnap regarding Performance Issues. After almost 8 weeks of turning circles with no valid help, we decided to close the ticket and revert everything regarding QuObjects. Its just a nother Qnap App which is not usable for production use.
You are right, QuObjects is Veeam Ready, but its just a "Proof of Concept" if it is working. There are no Performance or Long-Time Tests.
You are right, QuObjects is Veeam Ready, but its just a "Proof of Concept" if it is working. There are no Performance or Long-Time Tests.
-
- Service Provider
- Posts: 39
- Liked: 32 times
- Joined: Aug 07, 2017 11:51 am
- Full Name: William
- Location: Zurich, Switzerland
- Contact:
Re: QNAP QuObject Setup
most problems are only discovered when it's too late. So don't test something for your backups that you rely on in the worst case. Get a partner who knows what they are doing if you want to use immutability against ransomware. Examples from other object storage vendors:ryan.christensen wrote: ↑Jun 09, 2024 11:04 pm QNAP is advertising on their front page "QuObjects achieves Veeam Ready - Object with Immutability".
QTS Certification
QuTS Hero Certification
Veeam has certified using QuObjects. So as long as there's enough resources and we're using the version of QuObjects specified or newer we should be fine, right?
- A customer lost a lot of data on an object storage system. As long as the Veeam server had all the metadata information, it worked. As soon as it was out of sync and the metadata had to be resynchronized, the whole setup collapsed (2-3 months until synchronization).
- Another example: After 7 TB of data and some days later when the object lock (immutability) was renewed, the system collapsed due to too many API calls.
* I'm not allowed to name the vendors officially. This "customers" became customers of us after they had the problem, the setup was done by another partner.
A big misunderstanding is: There is NO real certification from Veeam - it is a list for compatibility, not more (even if it is often called "certified").
The Veeam-ready database is more or less….. let’s calI it “questionable”.
I say this officially as VASP on a quarterly basis as feedback to Veeam. The Veeam-ready DB only says: The vendor was able to perform some very simple tests. The API call worked and no errors occurred during the tests. There is no standard methodology, no performance goals etc. Nothing that matters in real life scenarios. This is the only reason 90% or more of the vendors make it to this list. It is therefore more or less useless for real life.
Would Veeam create a real “certification program” with different tiers / levels with goals to achieve, 90% or more of the vendors would be on a “use it at your own risk” level.
-
- Service Provider
- Posts: 31
- Liked: 5 times
- Joined: Apr 28, 2021 8:47 pm
- Full Name: Ryan Christensen
- Contact:
Re: QNAP QuObject Setup
Yikes. Thanks for the feedback both of you.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Feb 07, 2023 9:14 am
- Contact:
Re: QNAP QuObject Setup
Has anything improved with the last QuObjects 2.4.1721 update?
Who is online
Users browsing this forum: No registered users and 14 guests