[V13] Comments on initial deployment

[ashleyw]

Hi, Sorry late to the party on V13. I deployed the V13 appliance yesterday - looks great step forwards!

Got a couple of questions based on my initial deployment.

- Out the box installs can't connect to the update server. We don't use outbound web proxies in our environment, as we typically use Cisco Umbrella for DNS protection along with an array of other security tools. When I try and run the check update option, it gives an error of Failed to retrieve updates with the underlying error of;

Code: Select all

"Curl error (7): Couldn't connect to server for https://repository.veeam.com/rocky/9.2/vbr/13.0/mandatory/repodata/repomd.xml [Failed to connect to repository.veeam.com port 443: No route to host]".

I enabled SSH, and I can see form an ssh prompt that name resolution is working fine. I see there appears to be firewalld running. But I can't elevate privileges from veeamadmin so I can't check what the default rules are;

Code: Select all

sudo firewall-cmd --list-all-zones
[sudo] password for veeamadmin:
veeamadmin is not in the sudoers file.  This incident will be reported.

hopefully the security police won't come knocking at my door so my question is there is outbound URL whitelisting taking place - how do we allow outbound access to specific endpoints? Its probably not going to work just at an IP level as things like Cloudflare appears to be used for the update servers so they will be changing IPs.
- Out the box I can add a proxy. I deploy a fresh Rocky9 VM to act as a proxy and enable ssh password auth just as a test to keep things easy, but the appliance is unable to connect to the proxy so I'm assuming the same thing that is stopping the first thing from working applies to this. From an appliance SSH prompt I'm unable to ssh to the proxy VM.
- Our current system is running Veeam V12 and I need to keep this running for the time being (until we get the approval to go v13 and VUL - sadly bye bye sockets!). Previously I was able to add in a repository on the same Rocky9 OpenZFS based mass storage unit by adding the repository under a different directory. Is this going to be still possible with V13 or will the v13 agent attempt to push and incompatible proxy agent process on the VM which could break our V12 backups or will I need to test this? If there is a way of having both V12 and V13 configs at the same this would be ideal as I can benchmark the performance between Windows Veeam server verses the Linux Appliance under our full load - and I'd be hoping for a "linux for the win" scenario.
- Are there plans to allow for the deployment of virtualised hardened proxy environment that could be deployed directly from the Veeam console? ie. UI asks for which vcentre(or other hypervisor),network settings,proxy host name etc then everything else would be taken care of. this would further improve the security posture of the appliance.
- I'd really like to see the windows appliance client being unnecessary. On our corporate environment, every single executable has to be whitelisted in Threatlocker, so it would be nice if I didn't have to beg for corporate approval and I could instead get the full functionality from the web interface.
- See the appliance appears to be built on Rocky9.2. Are there any plans to track the latest versions of Rocky (currently 9.6 I believe, to help with security on the appliance itself?
- I'm using OpenZFS on V12 repositories with no issues whatsoever. I assume the connection to OpenZFS based repositories will still work (even if there is some tweaking allowed)?
- Many of our linux based based templates have been optimised to use virtualised "NVME controller 0" in VMware. These are backed up fine in V12, did I read that these VMs won't be backed up in V13 or perhaps I misread.

I want to see V13 rocking!
Big shout out to the team and all the hard work that's gone into this.

cheers
Ashley

[Gostev]

Hi, Ashley
1/ Yes, you will need work with your networking folks to ensure VSA can reach veeam.com
2/ Same, this seems like some firewall shenanigans in your environment. Do note however that we recommend using Veeam Infrastructure Appliance instead of BYO Linux for backup infrastructure components.
3/ This is not supported and therefore not tested, but most likely this will not work. But also no point to benchmark, we already did and V13 beat V12 hands down in terms of performance. And it's not Windows vs. Linux situation, as V13 is equally performant on both, as I explained in the sticky FAQ topic.
4/ For hardened backup infrastructure components, we offer Veeam Infrastructure Appliance (see the V13 What's New document for more info).
5/ Me too but understandably it will take us some more time to migrate almost 20 years worth of features to web UI.
6/ I'm not sure what you mean by the term "track". The special extended LTS version of Rocky we use (it's not vanilla 9.2 btw) will keep receiving security updates for 2+ more years, therefore we might end up skipping 9.6 altogether. However, this has not been decided yet simply because there's no pressure to make this decision.
7/ This has not been specifically tested with V13 and unfortunately we will not have QA resources for this testing until after 13.0.1 release.
8/ This does not ring any bells to me, please share what document or post you are referring to.
Have fun with V13!!

[ashleyw]

thanks, I had old duplicate entries for the DNS of the proxies on our DNS server configuration which was causing connectivity issues and causing the proxy deployment failures. I've fixed that but I'll use the appliance deployment method for the proxy OSs to keep things secure.
I'll give some feedback sometime next week assuming I can get V12 and V13 working at the same time (but using different proxies etc).
Sadly I don't have access to a complete parallel infrastructure setup so I'm going to need to be creative on my testing approach.

[ashleyw]

so I've got a basic test up and running and have the following comments;

1. JeOS seems to require 2x120GB disks even for a virtual server being used as a proxy server.
What is the reason for this where the document specifically states for the mount process;
Disk Space: 750 MB for Microsoft Windows-based proxies; 400 MB for Linux-based proxies.
We are currently running V12 with 4x Linux Proxies with 15GB OS disk which has been more than adequate for our use case.
But I have added a test as 1x8CPU,2x120GB disks and 24 GB ram to match our current V12 setup.

2. I've been able to dial in ZFS repository by adding it as a managed server and get the following warning when adding it:

Code: Select all

OpenZFS Fast Clone preview is enabled. this functionality is not supported for production use.

In our case I give it a different path folder on the same zvol (i.e. /VeeamBackup/v13backups instead of /VeeamBackup/v13backup)
Backups work fine - throughput was similar to using Rocky9 Linux repositories on V12 (which is great and expected).

3. "restore guest files" option doesn't give an option to specify the credentials for the source machine giving an error;

Code: Select all

The request does not have valid authentication credentials

This is for both windows and Linux single file restores. Is this feature working differently to v12?
I tried both by the webui and the Windows client.

4. When I added the IP of the network appliance, I made a typo on the gateway, and on DNS (I blame the keyboard ). I fixed that but when I tried to change the DNS settings via the webui, they were blanked afterwards. I was only able to make changes to the DNS configuration post install via the VMware console screen for the appliance. After making the change the appliance was able to update itself as expected.

5. Adding a Linux host
Adding a linux host using certficate-based authentication says "Requires a deployment kit pre-installed to the target machine."
Am I correct that to enable an existing Linux server to use certificate-based auth, I need to follow the process here?
https://helpcenter.veeam.com/docs/vbr/u ... tml?ver=13

6. Anton confirmed in a DM that JeOS is patched to make sure that its safe and secure and high performance, so it makes perfect sense to deploy it in the role of Proxies rather than to use generic Linux as it will help keep the security teams at ease! Currently we are making heavy use of OpenZFS for all our repositories and my expectation is that because this is still not officially supported, then I won't be able to deploy a ZFS repository using JeOS, so I'll need to keep using Rocky Linux for the repository?

7. SSO configuration. Ideally to suit the policies of our organisation, we'd prefer the WebUI of Veeam to be able to be registered as an Entra App so that we could keep a veeamadmin account as a break glass account, but to have normal backup admins using their Entra accounts (with MFA policy being defined on the Entra App config in O365). Is this a possibility going forwards or on the roadmap?

Overall, this is looking great and certainly keeps the patch management for the environment simple and secure.
Even better is JeOS (Just Enough OS) reminds me of that old Depeche Mode song "Just Can't Get Enough"!

[ashleyw]

just on point 7, a quick search showed this veeam-backup-replication-f2/users-with- ... 00124.html but it states the machine must be domain joined. This is not something we want to do due to organisational policy. All our other apps using web auth/SAML do not require devices like this to be domain joined so it would be great if Veeam could figure out a way to make this work without having to couple the server to AD.
thanks.

[Mildur]

SAML doesn't require a domain joined backup server.
If you have a IDP server in your active directory domain, you can login to your backup server through SSO/SAML. Or use Entra ID with an Entra ID application.

Domain joined is required if you want to use domain user accounts without SAML.

Best,
Fabian

[Gostev]

1. First disk: this is mostly due to the monstrous partitioning requirements of DISA STIG hardening, you end up with many partitions but we want each to have a reasonable size to reliable host the intended data. Second disk: to dump data from the first disk during future major OS upgrades.

2. Great to hear it was not accidentally broken

3. This is likely due to no Kerberos authentication available and no Veeam Deployment Kit on the target server. While NTLM support is discontinued for VSA.

4. Sounds like a bug.

5. Correct.

6. Correct.

[ashleyw]

thanks guys,
Just an update on the file level recovery issues.
There were two issues.
1. On the repository configuration, on the Linux mount server, I changed this to be the Appliance device itself. Prior to this I was using one of the deployed JeOS proxies, but I'm assuming the JeOS deployed proxy didn't have the data mover component (or some other component required by the file recovery process) on it so it wouldn't get to the screen that shows the tree view of the backed up OS file system. I didn't need to use a Windows mount server in my tests (which is great!).

2. For the file level recovery to work on a windows 2025 test VM, I had to enable the firewall inbound rule "File and Printer Sharing (SMB-in)", otherwise the connection hangs with "Checking credentials" and then eventually after about 10 minutes or so fails with "an unexpected server error occurred". It would be more useful if that timeout could be significantly reduced to say 10 seconds.
The default windows firewall configuration disables inbound SMB traffic, it might be worth highlighting that it may need to be temporarily enabled in the target machine for an FLR to succeed (this issue existed prior to v13 though).
In our case to lock inbound SMB traffic to a specific set of IPs (corresponding to where the traffic might originate) we run the following PowerShell command on the machine we need to restore individual files to allow SMB inbound traffic but only from our backup infrastructure (e.g. our 4xproxies and Veeam appliance);

Code: Select all

New-NetFirewallRule -DisplayName "Allow Veeam FLR inbound SMB" -Direction Inbound -Action Allow -protocol TCP -localport 445 -RemoteAddress @('1.2.3.4','1.2.3.5','1.2.3.6','1.2.3.7','1.2.3.8')

We don't often use the file level recovery, but the functionality of this in v13 is quick and efficient once its configured correctly.

So on my side everything is looking great, I'm going to tear everything down again, and recreate the appliance and proxies (on JeOS) and dial it in Entra and then recreate the V12 jobs in V13, then run some larger scale tests and compare the stats to V12.

cheers
Ashley

[ashleyw]

further testing of the appliance leads me to these points (which are most likely known by now I guess);
1. The Appliance WebUI provides doesn't currently provide access to VM exceptions on backup jobs, so this has to be done via the windows Veeam Backup and Replication client.
2. The Appliance WebUI provides doesn't currently provide access to the proxy affinity on backup repositories, so this has to be done via the windows Veeam Backup and Replication client.
3. I added our primary and replica Rocky9 OpenZFS based storage units as managed servers. For unknown reasons one server is being correctly flagged with update status showing as "N/A" in the managed servers view, but our other unit is showing a message of "Failed to check for updates". I think this was because during my testing I was trying to manually add the deployment kit at an ssh level, and something in the target is now flagging the machine incorrectly as a Veeam managed OS. I've tried uninstalling all Veeam components and removing and re-adding it back but I can't figure it out. Any ideas would be appreciated. I could reinstall Rocky Linux on the storage target and import the zpool again but this would take several hours so I'd prefer a hack if possible, for now.
4. Adding the deployment kit to a Rocky9 Linux host to enable managed server certificate-based authentication does not work, so I'll need to drop back to use ssh certificates for that.
5. I understand the JeOS storage requirements from what Anton mentioned, but it would still be beneficial if this could be lowered to make things easier - at least in the case of deploying just the proxy role on a VM. We are using thin provisioning for these though but the storage is still far too excessive for a lightweight minimal OS.
6. I'll try and get the auth dialed into SSO tomorrow, as it does concern me that the veeamadmin logon has MFA linked to an individual's authenticator app which isn't great from a business continuity perspective. We may have a way in our password vault to handle MFA authentication for shared teams, but is there any way of retrieving the code associated with the QR code after it has been added so we could attempt to register it into our vault?
7. I'm currently stress testing the appliance on a full backup run, using the same number of proxies (4) as we were before with same config (28GB,8CPU each proxy, but now on JeOS), with 25GbE networking to the storage targets. Storage targets are running Rocky9 Linux with large OpenZFS zpools (330TB of usable space on one, 126TB on the other) on Dell commodity tin. The Veeam appliance is 8CPU, 32GB ram for now.

Things are looking solid - nice work Veeam.

[ashleyw]

so I've run an active full backup on our workloads and then an incremental.
The incremental run through appeared to be about 10 minutes faster than the equivalent V12 setup which is great (see screen shot).

So only outstanding two issues for me are;
- I Still need to dial in SAML SSO into Entra . I'm just waiting for our internal teams to sort out the config on that and I don't see any blockers so expect to have this sorted tomorrow morning and don't expect any issues on the Veeam side of things.
- I still can't figure out why our primary repository server (Rocky9 Linux, just like the replica server) shows as "Failed to check for updates". See image below despite the DNS and other settings being the same on both servers. I think it was because I tried to install the deployment kit onto the "broken" server originally, so suspect the machineid has persisted into the Veeam database somehow, but I haven't been able to trace it.
- I noticed that I can't backup Veeam itself with Veeam on v13 resulting in a message; "The Veeam Software Appliance cannot be added to backup jobs to protect itself (<appliance name>)". This works fine for backing up a Windows based Veeam V12 console. I understand you can take a configuration backup but it might not be quite so convenient in some DR situations, so it would be great if it was technically possible to backup v13 appliance itself.
- I'm now going to start looking at the Enterprise Manager V13 and Veeam One V13 (but those two are generally not as critical to us).

The other quirks I can work around (i.e. the missing features in the WebUI - add SSO config, notification settings to the list of missing features), and hope that the webUI features can in future match the windows thick client. (Linux all the way )



cheers
Ashley

[Dima P.]

- I still can't figure out why our primary repository server (Rocky9 Linux, just like the replica server) shows as "Failed to check for updates". See image below despite the DNS and other settings being the same on both servers. I think it was because I tried to install the deployment kit onto the "broken" server originally, so suspect the machineid has persisted into the Veeam database somehow, but I haven't been able to trace it.

Please raise a support ticket and share the case ID with us. Since we dont manage the updates for Bring Your Own Linux installation likely it's caused by remote components update as you've suspected, but we would love to take a look at the logs.

- I noticed that I can't backup Veeam itself with Veeam on v13 resulting in a message; "The Veeam Software Appliance cannot be added to backup jobs to protect itself (<appliance name>)". This works fine for backing up a Windows based Veeam V12 console. I understand you can take a configuration backup but it might not be quite so convenient in some DR situations, so it would be great if it was technically possible to backup v13 appliance itself.

Thank you for the feedback, we will note it as a feature request for internal discussions!

[ashleyw]

Please raise a support ticket and share the case ID with us.

Thanks Dmitry for your comments, I've logged a job now. The case # 07826862.