QNAP QuObject Setup

[NorthGuard]

With v12 out, I am revisiting my configs to take advantage of direct backup to immutable s3 objects ( using QuObjects ). To do that, I have a quick question:

On the QNAP config side, do I enable Object Lock ( which auto-enables versioning ) and leave Default Retention at disabled?

It seems that if I enable retention, adding the bucket within Veeam throws an error that the compliance mode isn't supported. Is that because, in this setup, Veeam would actually be dictating the retention?

Thanks!

[sfirmes]

If you specify a retention period in the object storage gui, you will see that error. I wrote an article that might explain what you seeing. Check this out and I hope it helps.

https://community.veeam.com/blogs-and-p ... art-1-3670

[RedVision81]

@NorthGuard

We are thinking about using QuObjects as well for this purpose. Do you already have any experiences with QuObjects? Is it worth a try or maybe not reliable enough?
Any suggestions regarding sizing, OS (QuTS or QuTShero), Caching?

[NorthGuard]

I am slowly abandoning it because it isn't reliable.

It seems to work for some period of time ( weeks, months ) and then Veeam will randomly throw errors about not finding the API key or some other REST API error. Hours and days of troubleshooting and working with support didn't go anywhere.

I had about 6 separate implementations of QNAP QuObjects at various locations and the majority of them have eventually all done this.

So I would definitely not recommend it at this point.

If you really want onsite immutability, you could either spin up a linux hardened repo or spin up a Minio instance for s3 object storage.

[RedVision81]

ok, i expected that, cause qnap isnt very reliable with software in general
really a pity, cause this would have been an good solution for an onprem S3 Comp. Storage

[je1000]

I am slowly abandoning it because it isn't reliable.

It seems to work for some period of time ( weeks, months ) and then Veeam will randomly throw errors about not finding the API key or some other REST API error. Hours and days of troubleshooting and working with support didn't go anywhere.

I had about 6 separate implementations of QNAP QuObjects at various locations and the majority of them have eventually all done this.

So I would definitely not recommend it at this point.

If you really want onsite immutability, you could either spin up a linux hardened repo or spin up a Minio instance for s3 object storage.

Wondering how stable is Minio with s3 compared to a hardened repo?

[tyler.jurgens]

Stability for your Minio deployment will depend on your Minio deployment. We've been running large Minio clusters in production for years with great stability.

It really depends on your goal. Do you want a simple server running some RAID for a few disks? Use a HLR. If you want a high-performing erasure-coded cluster, Minio is great.

[RedVision81]

Any suggestions how to use a QNAP Storage System as S3 Object Storage without the QuObject Software?
Would it be possible putting a S3 Gateway in front offering the QNAP capacity via iscsi/nfs to it?
i often read Minio as a solution, but the former S3 Gateway is deprecated in the meantime and we are not very keen with Linux Systems.
Would Qnap Storage Gateway be a suitable solution for that?

[ober72]

I have never used a storage gateway but I do remember looking at the Minio one before they deprecated it, but in the end decided not to use. I could be wrong but I think the reason was that it did not fully implement all the S3 protocols, but it was a few years ago. If Linux expertise is an issue but you still want to have immutability and a hardened system then take a look at Object First. It is specifically made for Veeam and requires no Linux knowledge https://objectfirst.com/ . If you have to go with local non immutable storage then I would highly recommend also having an offsite option (follow the 3 2 1 rule) that offers immutability (Veeam service providers, Wassabi, Backblaze etc). Again I would highly recommend having immutability somewhere in your setup because of the ransomware threat.

[ober72]

Out of interest I looked up the reason why Minio deprecated the Gateway and this pretty much is why we did not implement it at my former workplace, "Second, the S3 API has evolved considerably since we started, and what began as inline translation morphed into something much more. Critical S3 capabilities like versioning, bucket replication, immutability/object locking, s3-select, encryption, and compression couldn’t be supported in the gateway mode without introducing a proprietary backend format."
So the lack of versioning immutability/object lock is critical.

[tyler.jurgens]

Minio supports versioning, bucket replication, immutability, and object locking - they just don't do that over the old S3 gateway. They support the S3 protocol natively in their Minio Server mode (whether you run it on bare metal or through K8s). The S3 gateway was essentially only there to support S3 API on whatever backend storage you wanted to use (NFS, etc).

Now, if you wanted to use the S3 gateway in front of your QNAP NFS volume or iSCSI LUN because you *really* need to use that QNAP iSCSI LUN or NFS volume, you'll be out of luck using Minio's old S3 gateway. Other S3 gateways may exist (or the old Minio S3 gateway may have been picked up by the open-source community and continued on), but I haven't looked.

If you want to use Minio natively to provide immutable S3-compatible object storage, you'll be just fine.

You could always deploy a Linux VM and do a single node Minio S3 deployment, which sits on your QNAP, but you will need linux skills to do that.

[ober72]

Good points made by Tyler. It was only the Minio gateway that did not support versioning etc. I think the issue for Peter was that he said they were not keen on Linux systems. While Minio does have a Windows Binary it won't support immutability of course.
I love Minio and have set it up a number of times (including in Docker Swarm cluster back when I could not do it on Kubernetes). However, it does require some learning to run on Linux, i.e. to run it and the OS correctly, keeping in mind security and so on. I will be honest if I were running production now I don't know if I would even trust myself despite having been using Linux since 1998. The problem in my case is keeping up with all the security issues while having to to do a million other things simultaneously.

[RedVision81]

Thank you guys for all that input.
We just want to keep things as easy as possible, because we hardly have any Linux skills and we dont want to over-archtitect our environment. Its also a question of support for now and in future cause someone will always have to deal with this system. I think an out-of-the box solution would be the best for us. QuObjects would have been a very easy and cost-efficient way but is not reliable. Hence i think we will have a look at more professional solutions like Scality Artesca or maybe Datacore Swarm e.g.

[ober72]

Hi Peter,

No problems at all. When I worked for service providers we had many clients who simply did not have administrators with Linux skills. Learning Windows is hard enough and time consuming and if you add in networking and so on. Linux is a great OS and personally my favorite (FreeBSD being a close second) but a badly administered Linux server is not much more secure or safer at all than windows at the end of the day. The other aspect that I want to stress is that even if an organization had such a person, if they left the company often there would be major issues if they could not find someone to replace them with the skill set needed. Again due today's security threats, ransomware and so on even experienced Linux admins might be excused for not wanting to take on the responsibility for guarantying the security of the backup files. After all the backups are the last life saver in the ransomware fight. So choosing a ready made solution like some of the solutions that you mentioned is a wise move in my opinion. In the case that you would like more information about Object First's solution please DM me.

cheers

[mwhite]

The issues we have encountered so far with it are;
- QuTS with Immutability, NAS requires 1 day retention at the filesystem level, which breaks VeeamBR but only after the block generation time (immutability + 10 days).
- QTS with VBM365, if the repo runs out of space the object store corrupts pretty bad. QuObjects creates objects with a really long file extension that kills VBM when it's rescanning.
- QNAP list on their page that the QTS has a 'file share limit' that you need to be aware of when using the app (without any further info on what this is referring to). After much time going back and forth with support, this turned out to be the inode limit of EXT4.
- QNAP support for the app is abysmal, they flat out refuse to do remote support or even talk on the phone, tickets are written with varying degrees of english comprehension making it difficult to convey complex issues. 90% of the time spent on tickets will be them asking for remote support to be re-enabled on the NAS.

Veeam might need to do some more validation with QuObjects or remove it from the compatibility list. I understand that they can't fully test every product for 6 months and I'm not sure what the validation process involves, so I'm not sure if there is much of a solution to prevent these issues in the future with new products.

[RedVision81]

have you also tried with QuTShero and ZFS Filesystem?
Would be great if you also can tell us something about your Qnap, (CPU, RAM, how many disks and may some additional ssd cache acceleration.)
We decided to give it a try in the beginning of our VBM365 journey, just for bridging the first year, having more time in selecting and getting a more professional solution.

[jcalvetm]

QuObjects + veeam to me is a fair solution but not perfect in my opinion,

Possible technical issues appart (existing or not), the thing with a local Qnap NAS is that is local. Surely the attacker is not going to be able to delete your inmutable backups in your repo via "S3 access", but if the attacker gets access to your QNAP management web interface, erasing-formatting-reconfiguring your disks is enough to deprive you of your backups.

(I am the owner of a QNAP NAS, I love it, but to me that is a concern).

[orb]

Hello,

We conducted a Proof of Concept (POC) with a customer using a configuration of 24 spindles in a RAID 60 array (2 sets of 10 + 2). From the outset, we did not anticipate an exceptional performance. We understood that utilizing SSDs for read/write caching would be essential. The test focused on a tiering scenario only, not direct backup.

We had to adjust the settings in Veeam because it was significantly straining the storage system.

• The performance met our expectations, although I do not have the exact figures at hand; it was between 20-40 MB/s.
  The system encountered failures when backing up VMs with disks of more than 2 TB.

I would not recommend it for production

My 2 cents

Oli

[NorthGuard]

Wondering how stable is Minio with s3 compared to a hardened repo?

I have MinIO on a couple of HP Proliant servers and so far it has been solid.

With that said, I also have an implementation with a Linux Hardened Repo - that has been in place for well over a year and never had any issues with it at all

[NorthGuard]

Any suggestions how to use a QNAP Storage System as S3 Object Storage without the QuObject Software?
Would it be possible putting a S3 Gateway in front offering the QNAP capacity via iscsi/nfs to it?
i often read Minio as a solution, but the former S3 Gateway is deprecated in the meantime and we are not very keen with Linux Systems.
Would Qnap Storage Gateway be a suitable solution for that?

The only way I found was to install a virtualized Linux instance and install MinIO or install a MinIO container.... BUT... unless you have a monster QNAP, you will have issues.

[gnalley]

Have you looked at Ootbi from ObjectFirst? https://objectfirst.com
We are currently evaluating it and have been very pleased with it so far. Purpose built object storage appliance designed with Veeam compatibility in mind. Take a look at "About Us"...you should recognize the founders names.

[BaptisteT]

If you look at Veeam ready program https://www.veeam.com/alliance-partner- ... ble&page=1 for QNAP, you will see they pass only with full SSD configuration.

[ober72]

Have you looked at Ootbi from ObjectFirst? https://objectfirst.com
We are currently evaluating it and have been very pleased with it so far. Purpose built object storage appliance designed with Veeam compatibility in mind. Take a look at "About Us"...you should recognize the founders names.

Thanks Gary, that is good to hear.
If anyone has any questions about Object First you are also welcome to DM me directly. I am the Community Manager at Object First.

cheers

Geoff

[DE&C]

About the QNAP and Minio part:

QNAP and any other small NAS vendors will probably not work for Veeam and their object storage integration on a long run with a huge backup set and immutability (even if certified). They have the same problem as most of the object storage solutions: They are not able to handle the metadata for the many small objects Veeam generates.

Take a look at my explanation on a Ceph dicussion: veeam-backup-for-microsoft-365-f47/s3-p ... 87127.html

Minio: MinIO is in my opinion way more complicated than a Linux Hardened Repository. Now you need Linux knowhow (even harden linux) and then the MinIO knowhow. And if you want support you have to pay for the license (which is ridiculous high, just compare it to other software defined object storage solutions). In my Opinion MinIO is not a solution I would use in SMB for Veeam (and also not for Enterprises for Veeam, if you calculate the TCO of this solution and do a real risk evaluation). There are many other things, for example: you highly depend on what ever linux and filesystem you are using. Because of the metadata design you probably need all flash.


Compatibility list
We had in the past many customers that used solutions from the compatibility list and still had huge problems. The compatibility list doesn't say anything about "real life" setup, how it is engineered or about product-readiness and security -> only that it worked for a specific test setup.

Conclusion: you need to ask the right questions and do an evaluation


I think for all discussions about object storage for Veeam (VBR) you need the following questions
- do you need professional support if something happens?
- do you have the knowhow for the setup (engineering) and all the security aspects? If not: does the vendor provide it?
- Do you have more then 10TB of data? (10TB with standard Veeam settings = minimum of 10mio objects)->can the solution handle xxx-Millions of small objects (<1MB)?
- Does the product have some history on the marked (proofed that it worked)?
- How is the scalability of the product? can you scale in the node (with more discs) and with more nodes?
- How is the whole solution aspect done from a security, engineering and operative perspective: different users / tenants / keys to separate your VBR servers, different users with different privileges, possibility to separate data and management traffic, can you start with one single node and expand it (and also expand it to a cluster or only to single nodes that are one logic appliance), is it all S3 Amazon "standard" or did the vendor implement something "it's own way" (like IAM, livecycle policies, special APIs that it works for dedicated application etc.)? https://www.veeam.com/kb3151 (IAM policy for S3 object storage)
- Does the system have dependencies that are not part of the solution (not from the same vendor, therefore not part of the support and design)?
- And all together it brings us the the question: is this product ready to be used?
->there are many more question we ask during an evaluation, this are just a few examples.

Here are more questions and explanations about the question: veeam-backup-for-microsoft-365-f47/s3-p ... 87127.html

My personal opinion
If we are putting here some solution that work and fulfil all above, check "Artesca" from Scality. They also have Appliances that are ready to use - and the pricing is very interesting (compared to other vendors mentioned). Of course you can use your own x86 hw for Artesca (starting point 50TB and one node, as long as the HW matches the requirements). I only post it, because some where asking about a solution that work and also mentioned other vendors

Disclaimer: I work for the VASP in Switzerland and we also sell Artesca (just to be transparent). We evaluated many solutions (all that were mentioned in this thread) in the last years to be used for Veeam. I can't make the whole evaluation public for NDA reasons.

If you have questions about object storage for Veeam, I'm happy to help.

[sfirmes]

@DE&amp;C, I would be very interested in talking with you with regards to your evaluations. In my role as an object storage SME here at Veeam working with our alliance partners, I work with most of our on-prem object storage partners and Scality is one of them.

[je1000]

Thanks for your post and I can confirm a lot your points, many vendors don’t handle well veeam object storage backups and Wasabi is one them.

I would love to hear more about your experiences and testing

About the QNAP and Minio part:

QNAP and any other small NAS vendors will probably not work for Veeam and their object storage integration on a long run with a huge backup set and immutability (even if certified). They have the same problem as most of the object storage solutions: They are not able to handle the metadata for the many small objects Veeam generates.

Take a look at my explanation on a Ceph dicussion: veeam-backup-for-microsoft-365-f47/s3-p ... 87127.html

Minio: MinIO is in my opinion way more complicated than a Linux Hardened Repository. Now you need Linux knowhow (even harden linux) and then the MinIO knowhow. And if you want support you have to pay for the license (which is ridiculous high, just compare it to other software defined object storage solutions). In my Opinion MinIO is not a solution I would use in SMB for Veeam (and also not for Enterprises for Veeam, if you calculate the TCO of this solution and do a real risk evaluation). There are many other things, for example: you highly depend on what ever linux and filesystem you are using. Because of the metadata design you probably need all flash.


Compatibility list
We had in the past many customers that used solutions from the compatibility list and still had huge problems. The compatibility list doesn't say anything about "real life" setup, how it is engineered or about product-readiness and security -> only that it worked for a specific test setup.

Conclusion: you need to ask the right questions and do an evaluation


I think for all discussions about object storage for Veeam (VBR) you need the following questions
- do you need professional support if something happens?
- do you have the knowhow for the setup (engineering) and all the security aspects? If not: does the vendor provide it?
- Do you have more then 10TB of data? (10TB with standard Veeam settings = minimum of 10mio objects)->can the solution handle xxx-Millions of small objects (<1MB)?
- Does the product have some history on the marked (proofed that it worked)?
- How is the scalability of the product? can you scale in the node (with more discs) and with more nodes?
- How is the whole solution aspect done from a security, engineering and operative perspective: different users / tenants / keys to separate your VBR servers, different users with different privileges, possibility to separate data and management traffic, can you start with one single node and expand it (and also expand it to a cluster or only to single nodes that are one logic appliance), is it all S3 Amazon "standard" or did the vendor implement something "it's own way" (like IAM, livecycle policies, special APIs that it works for dedicated application etc.)? https://www.veeam.com/kb3151 (IAM policy for S3 object storage)
- Does the system have dependencies that are not part of the solution (not from the same vendor, therefore not part of the support and design)?
- And all together it brings us the the question: is this product ready to be used?
->there are many more question we ask during an evaluation, this are just a few examples.

Here are more questions and explanations about the question: veeam-backup-for-microsoft-365-f47/s3-p ... 87127.html

My personal opinion
If we are putting here some solution that work and fulfil all above, check "Artesca" from Scality. They also have Appliances that are ready to use - and the pricing is very interesting (compared to other vendors mentioned). Of course you can use your own x86 hw for Artesca (starting point 50TB and one node, as long as the HW matches the requirements). I only post it, because some where asking about a solution that work and also mentioned other vendors

Disclaimer: I work for the VASP in Switzerland and we also sell Artesca (just to be transparent). We evaluated many solutions (all that were mentioned in this thread) in the last years to be used for Veeam. I can't make the whole evaluation public for NDA reasons.

If you have questions about object storage for Veeam, I'm happy to help.

[RedVision81]

We had a very good tech talk with a Veeam system engineer the past days, he told us QuObjects normally should work if the Qnap hardware is sized good enough.
Our Qnap is quite powerful i think, about the same as the listed Device in Veeam Ready Database, hence we will give it a try even for getting some first experiences in S3-Comp-Storage. Our Setup will look like this:
QuTShero - no Compression - no Deduplication - 2 x 10G SFP+ direct connected to veeam server
TS-h1283XU-RP with a Xeon E-2136 6Cores/12Threads - 128 GB RAM
12 x Seagate Exos X16 SATA 16TB (ST16000NM001G) Raid 6
2 x Samsung 980 Pro 1 TB Raid1 Apps/Data
2 x Samsung 990 Pro 1 TB Raid1 Caching

It would be great if QuObject Users/Testers can also post something about their hardware/configuration. Does anyone know where QuObjects stores the objects database?

Because we are still at the beginning of our M365 Migration, we don't have anything too loose, and if you don't try you never know. Let's see what happens when the first mailboxes are moved to M365. If it ends in a disaster with QuObjects, we even have tried, and can go ahead finding a better solution

We also had a closer look at the Ootbi Appliance from Object Fist. This seems to be a very interesting and reasonable solution especially in pricing compared to other solutions.
I really can recommend their Youtube Channel.

Regards
Peter

[ober72]

Thanks Peter,

By the way I just got asked about this again, what the heck is Ootbi It stands for out of the box immutability. If anyone has any questions, also about the community we are building just DM me so as not to spam the thread

[walo]

We are using QuObject on a TS-1273AU-RP and a TS-873AU-RP, but with QTS, not hero (5.1.5.2679).
We use them for backup copies and we were able to copy a few TB backups on it. But especially the smaller NAS runs in timeouts very often (Site-to-Site VPN, but stable, SMB works fine).

After the problem occurs, editing the properties of the repository in Veeam takes several minutes and at the point "Bucket" Veeam shows, immutability is not available/enabled for the repository. Sometimes, only rebooting the whole NAS or waiting a few hours fix this. Sometimes clicking next again, the problem is gone immediately. Have you ever had this problem?
So, the system is extremely unstable for us. Only one task per time is allowed in Veeam for these two repositories.

Thank you

[RedVision81]

how many and which disks, ram etc. have you configured in your qnaps? do you use additional caching?
which backups do you copy, vm data or m365 data? and how are they connected to your backup server?