Question on new V11 Linux repositories

[FrenchBlue]

Hello,

If we use a hardened Linux repository, can it be at the same time repository for local backups, and also target for backup copy jobs of another distant Veeam instance? (like it was possible with Windows repositories on separate disks).

Thanks.

[Gostev]

Hello! Yes, no issues there, it's no different from regular Windows or Linux repositories in that sense. As always, be sure not to overload the repository with too many concurrent tasks, as the "distant Veeam instance" will not be aware about tasks created by the "local Veeam instance" and vice versa. Thanks!

[FrenchBlue]

Thanks, perfectly clear.

[veremin]

Also, be sure to keep versions of the backups servers the same in order not to face the situation when one backup server updates the repository component, making it unreachable for yet to be updated backup server. Thanks!

[FrenchBlue]

Hello,

I tried to implement this exactly as described. Unfortunately, it seems this is causing an issue and that both VEEAM VBR instances can not share the same Linux VM as repository. As soon as I create the remote repository in the remote Veeam VBR instance (using a different filesystem as target of course), the local repository fails , the Linux server becomes "Unavailable" with a certificate error when trying to rescan it ("An unknown error occurred while trying to process the certificate"). Both Veeam VBR are in latest version 20210507.
Are you 100% sure this should work and has been tested?

Thanks.

[veremin]

We have thought about this scenario while developing and testing hardened repository, so the described behaviour does not look expected. We would appreciate, if you open a support case and share its number with us (so we can ask QA team to follow the investigation). Thanks!

[FrenchBlue]

Hello,

I'm going to do this (have to go through cloud support first) and will post the number here when it's done.

Thanks.

[FrenchBlue]

Note : we use the same Linux user for both local and remote repository connections, has the same pw on both repositories. But I don't think this could interfere with certificate issues?

[veremin]

No, it should not. So let's wait for the support ticket and see what engineer suggests after log review. Thanks!

[FrenchBlue]

Hello,

Case opened, #04842238

[veremin]

Thank you, will bring the case internally and update the topic with the more information. Thanks!

[FrenchBlue]

Hello,

Any news on the case?

Thanks.

[wishr]

Hi FrenchBlue,

I've pinged our support management to get your case processed with a little higher priority. We need it first to be fully analyzed by our tech. support team which will involve us (R&D) after that if required.

Thanks

[FrenchBlue]

OK thanks. We can do a remote session if needed of course.

[FrenchBlue]

Hello,

An udpate here. I tried to reproduce the issue on 2 new Linux repo machines. I couldn't. Then I checked again the existing ones, and noticed that for some reason, the local repo was defined with an uppercase hostname in 1 VBR instance, and with lowercase (as remote repo) on the other one. Could typically lead to the certificates issue we had. On this VBR VM, I then tried a nslookup on it, with lower case, and the DNS answered with upper case, weird! I rebooted the VBR VM (maybe a flushdns would have been enough), and then the nslookup returned lower case as expected. I then deleted and re-created the Linux server/repo, and the problem went away... why did the DNS return uppercase when queried with lowercase when the repo was created, the DNS entry being itself lowercase, no idea, but it seems it was the issue....

[mweissen13]

Well, since DNS queries should be case-insensitive I would say this is a bug on the Veeam side. Some DNS Server always return UPPERCASE, while other always return lowercase, while others return Mixed-Case (as typed in the zone file). You can never be sure which will happen, so this bug is bound to bite again in the future.

[FrenchBlue]

Ahh, interesting... but this will only cause an issue if the repository is used by 2 different Veeam VBR instances AND if one is registered with upper case and the other one with lower case, that's exactly the issue I hit. So maybe Veeam should force the repo hostname to lower case, whatever the DNS returns.

[Gostev]

Actually, we don't officially support and so don't test scenarios where backup infrastructure components are shared between different Veeam backup servers. Doing so may result in issues, because backup servers are not aware of each other when using a single resource.

[FrenchBlue]

Errrr, that was the original question of this post, and in March you answered that it was OK, then in June your colleague said it was planned/tested....

Of course, we were using 2 different filesystems/volume groups on the Linux VM. But anyway, given those issues, we finally have split the repositories in different VMs.

[Gostev]

Sure, it will work OK if you approach this correctly, such as: schedule the jobs not to overlap to avoid overload and/or conflicts, don't install an update on only one backup server etc. There are no technical reasons why this would not work in such an "isolated" setup, and in the past years I've ran into a number of customers who have been doing with little issues (except when they only upgrade one backup server, but not other).

My latest comment was specifically in regards to "Veeam needs to fix this or that bug with this scenario". This is not a scenario we test and officially support, so do expect possible bugs and don't expect fixes for them. We simply can't log any bugs for what is not officially a feature. Before we can do this, devs will require us to prioritize and then have them deliver the actual feature first: support for sharing backup infrastructure components between multiple backup servers.

[Gostev]

Sorry, somehow I missed we're talking about a repository registered with single-use credentials. Someone just reminded that because such repositories use certificate-based authentication, you won't be able to share the same Linux server between backup servers without some manual configuration changes, as if you just use UI than the backup server where you register the hardened repository last will "win" by configuring it to use its certificate.

[soncscy]

Hrm, does the forum FAQ need to be updated then?

post402811.html#p402811

It feels like point 4 is contrary to this restriction then; or does point 4 mean there is a supported workaround for the certificate issue you mentioned?

[HannesK]

sorry for the FAQ issue... actually I added that sentence on Monday morning around 8am after I saw this thread in the forum digest. Sorry for the confusion and I fixed the FAQ right now

[FrenchBlue]

Sorry, somehow I missed we're talking about a repository registered with single-use credentials. Someone just reminded that because such repositories use certificate-based authentication, you won't be able to share the same Linux server between backup servers without some manual configuration changes, as if you just use UI than the backup server where you register the hardened repository last will "win" by configuring it to use its certificate.

Hello,

As far as I remember, we didn't select single use credentials during the repository creation. Does it mean it should work without this?

[Gostev]

Correct. If you didn't use single-use credentials, then you're using classic Linux-based repository and not the new V11 Hardened Repository. Classic repositories use password-based authentication, so you should be able to share them between backup servers. But the same also means you won't have support for immutable backups and so protection against ransomware and hackers.

[mweissen13]

Sorry, somehow I missed we're talking about a repository registered with single-use credentials. Someone just reminded that because such repositories use certificate-based authentication, you won't be able to share the same Linux server between backup servers without some manual configuration changes, as if you just use UI than the backup server where you register the hardened repository last will "win" by configuring it to use its certificate.

Hello Gostev!
I am unsure if this is the right place here or if I should better create a new thread.

In our setup we are using a single Linux-based server as repository server formatted with XFS, leveraging the fast clone functionality.
This server has been added to who different and unrelated VBR installations (one of the a regular VBR installation and the other is Veeam Cloud Connect). We were using two different unprivileged users to connect the VBR servers to the repo server. This setup has worked fine since v10 and continues to work with v11. As long as I don't want to add the repository to fresh system. But then it will (temporarily) need sudo permissions and will install the new "hardened mode" Transport agent, which can not be shared between systems.

Will such a setup be supported with v12 or will we have to split our repository into two (or more) servers? Because if that's the case we will have to plan this carefully beforehand. Since it's also impossible to transport the backups which were using XFS fast clone to a new host without re-hydrating.

[Gostev]

Hello, I don't expect any changes with your scenario in the foreseeable future. Thanks!

[mweissen13]

Hello Gostev, thanks for your answer. Could you please be more specific? Support told me that since v11 all newly created Linux repositories would basically be created as "hardened", regardless of using single-use credentials or not. And the only way to use one Linux repo server for two or more unrelated VBR instances would be to use NFS or SMB (which would not allow to use fast clone).

At least I would expect that should be possible to install the Transport service multiple times on differing ports. And most importantly: without needing root privileges, not even temporarily.

[HannesK]

Hello,
I'm pretty sure that the support statement is just a wording thing ("hardened" vs. "immutable" vs. "hardened immutable"). there were some misunderstandings between departments. That was fixed just short time ago.

For installation of the persistent transport service (standard since V11), temporary sudo / root is required (same like for Windows repositories). Otherwise we could not create installation paths or for example make the service "autostart".

The "runtime" repository in V10 did not require root, that's correct. There we sent the binary again and again over the network and ran it temporarily. That overhead of re-deployment was removed in V11 by the persistent transport service

Best regards,
Hannes

[mweissen13]

Hello Hannes!
Thanks for the clarification. I have no problem giving temporary sudo / root permissions for the installation of the persistent transport service.

But with that config, will it still be possible to use one Linux repository for multiple otherwise unrelated VBR installations? At the moment this seems to be a problem because the VBR installation that comes second will replace the existing persistent transport service (and certificate?). And as result the first VBR instance will have problems after that. Also I think that with the centralized component it would be at least theoretically possible for an attacker to access/modify/delete the other repository's on the same server. With the "old" method and unprivileged user this is impossible (as long as there is no additional security problem on the Linux side).