[V13] Question regarding Veeam B&R 13.x

[Gostev]

Every time you come here to update on the progress and share that the environment keeps growing without any issues just makes my day!

Yeah you probably missed it because there's another thread complaining about wrong CPU/RAM values in VIA system requirements (erroneous copy/paste from VSA) therefore this documentation definitely exists. I know by heart that VIA is 2x 120GB, it's the legacy of managed hardened repository ISO and also the smallest flash drives still commonly available for servers these days.

Yes, VIA works for both deployment options because VBR V13 code is cross-platform. So one single codebase. The only difference is that with VSA this code is pre-installed on a Veeam-managed OS, while the alternative requires that you install this same code on a Windows OS yourself.

[Didi7]

Ok, holiday season begins very soon here for me, but I intend to extent this new environment still a little bit until Friday. I will possibly have a closer look at the full documentation during my flights, but I am very satisfied with what's running already.

Things will develop quickly I hope after my return and then the real party time begins with VSA, VIA Proxies and VIA Hardened Repositories, when they go into full production and soon with VDCV as well of course.

I already read about that erroneous copy/paste VSA requirements for VIA and cross-platform stuff (very well idea) as well.

As soon as VBR 13.x Windows-based gets released, I will start ditching all my Windows-based proxies here as well! Bye bye Microsoft Proxies.

Very nice development.

[Didi7]

Adding VIA-based Rocky 64-bit Linux VMs here takes a little bit more than 4 minutes (same VLAN). Is this expected behavior?

Ok, today I used the Windows Console instead of the VSA webpage. Compared to adding a VIA-based Rocky 64-bit Linux VM in the VSA, adding additional same kind of VIA-based machines in the Windows Console only takes 1 minute and 30 seconds.

Just a side-note.

Currently, that's one reason to use the Windows Console instead of the VSA for adding components, if you need to add a lot of machines, like I need to do now.

Linux deployment => I love those MSI-packages on Linux

[Didi7]

Addon to last post ...

During adding new VIA-based VMs I noticed at one of those machines an error or warning that updates cannot be checked. It worked for several machines before, so this must have something to do with this specific VM. So I decided to login via Host Management page and then I noticed that the DNS field was empty (I am absolutely sure I entered all 3 DNS servers there during installation), as it was double-checked as well. The same happened to the second machine of this location. I entered the 3 DNS servers and retried adding them to the VSA environment via the Windows console, same error and after checking again the DNS field was empty again, though I added the DNS servers and saved settings. Every time I enter them using the webpage on port 10443, the field is empty looking again. Then I decided to add those DNS servers via the console, that worked but as soon as I login to the website again and change the DNS servers and save them, the field is empty again. To me this means, this is a bug in the webpage, but there is a work-around.

Cheers

[Egor Yakovlev]

Linux deployment => I love those MSI-packages on Linux

Hi Didi.
Those .msi are needed for Windows machines guest processing using that Linux proxy, not for the proxy function itself. Just like there are some Linux components sitting on the Windows VBR server today.

[Didi7]

@Egor
I see.

[Didi7]

Addon to last post ...

During adding new VIA-based VMs I noticed at one of those machines an error or warning that updates cannot be checked. It worked for several machines before, so this must have something to do with this specific VM. So I decided to login via Host Management page and then I noticed that the DNS field was empty (I am absolutely sure I entered all 3 DNS servers there during installation), as it was double-checked as well. The same happened to the second machine of this location. I entered the 3 DNS servers and retried adding them to the VSA environment via the Windows console, same error and after checking again the DNS field was empty again, though I added the DNS servers and saved settings. Every time I enter them using the webpage on port 10443, the field is empty looking again. Then I decided to add those DNS servers via the console, that worked but as soon as I login to the website again and change the DNS servers and save them, the field is empty again. To me this means, this is a bug in the webpage, but there is a work-around.

Cheers

I think I now when DNS field is blank. I entered IP, Mask and Gateway, as well as 1 DNS server during initial installation and used the Advanced button to add 2 additional DNS servers, as I was unsure how to add more than 1 DNS server. So 3 DNS servers were added during installation. As soon as the webpage was available I double checked the DNS servers in the Host Management UI and probably because I saved this settings again, the entries for DNS server disappeared. I will check this again, as soon as I add the next VIA-based machines.

[Didi7]

Good morning,

today I finished installing and configuring a VIA-based Hardened Repository under Hyper-V (this time not VMware). I suppose the code-base is similar or the same.

One DNS server has been entered during initial installation, I added 2 additional DNS servers within the Advanced option, then I accessed the web page port 10443 the first time (didn't know that the Host Management UI had to be enabled first, but good => security). When I checked the network settings, I decided to add a fourth DNS server, saved the settings and re-entered the screen, the DNS field is blank now, so definitely a bug. Only way to get DNS servers back is by using the Console UI.

Cheers

[Gostev]

Yeah, that's specific to hardened repository deployment option. The UI will actually disable itself again after some timeout to reduce attack surface. @Dima P. now that I'm thinking about this, I think it's better to have it enabled by default but keep everything as is (so it shuts itself down normally N hours after activation). This way it will be immediately available for some hours after hardened repository deployment (which is the only time period when it's actually needed), so better experience.

[Didi7]

Ok, that's possibly a good idea to activate the WebUI in the beginning. I thought about some Firewall problems, as 443 (standard Rocky page was available).

Today, I re-checked the 'Software Update Summary' screen, which was sent by SMTP and noticed that machines are not in alphabetical order, maybe that would give a faster overview? Currently 22 machines, but the number will definitely rise beyond 70 or even more and then it's getting worser

[Gostev]

You mean you have 1 VSA and 21 VIA?

I agree it should be alphabetical. Perhaps start from VSA always as it's the most important, and then alphabetical VIA? VIAs are like cattle so some may be periodically offline or unreachable. And you probably want to see right away that VSA if fine, instead of looking it up within all the cattle.

[Didi7]

1 VSA, 1 vCenter, 20 VIAs (3 Proxies for an old Nutanix-cluster with ESXi using DirectNFS, 16 soon to become VMware Proxies, 1 Hardened Repository currently) and counting.

Yes, VIAs are periodically offline, e.g. during updates or when WAN-lines are not available, other than that they are mostly available.

Agreed => VSA first and the VIAs in alphabetical order => sounds good to me!

The 'Software Updates Summary' screen says 20 machines up-to-date (Hardened Repository was added today)

[Didi7]

Back on track after vacation and another virtualization project, I am know able to return to our new Veeam Infrastructure based on v13.

Yesterday I configured the BIOS on 9 physical HPE servers and I am now ready to deploy VIA Hardened Repositories on those systems. The first hurdle I encountered is the configuration of teamed LAN-Adapters.

If I decide to use only one LAN-adapter in the first initial installation, is it possible to enable teaming of LAN adapters later as well and is there any info available how to to this, as this is currently not part of the Veeam Infrastructure Appliance configuration and hidden in Advanced Options?

Regards,
Didi7

[HannesK]

Hi,
teaming / bonding is covered in the FAQ and user guide

Best regards
Hannes

[Didi7]

Hello Hannes,

thanks for pointing me in the right direction. A lot to learn regarding bonding in Rocky Linux

Let's get back to incorporating Veeam Hardened Repositories soon.

Regards,
Didi7

[Didi7]

OK, bonding now fully works on all repository server, though it took me some time to find out what settings are required and how things work. Unfortunately the status of both NICs are always UP in Veeam Host Management Console, allthough ports are shutdowned on the switch and are visibly DOWN in iLO environment as well. There is some improvement potential but I know that mmtui is planned to be replaced later somehow by incorporating this into Veeam GUI iirc.

[Didi7]

Btw, I noticed a bug (not sure if already known). Currently nearly half the amount of systems planned are integrated into the central Veeam B&R appliance, which means 30 systems now.

If e.g. we go to 'Managed Servers' or integrate new Backup Repositories and try to choose a system, which is not visible and we need to scroll down to select it and if you are not fast enough, the list jumps back to the top and you need to scroll down again. That's a little bit annoying.

Happens in Google Chrome, will try Edge and Firefox as well now.

[Egor Yakovlev]

I noticed a bug (not sure if already known)...
...if you are not fast enough, the list jumps back to the top...

I believe that has been fixed in the upcoming 13.0.1 release already.

[Didi7]

Ok, let's wait and see, if that is fixed in 13.0.1.

[Didi7]

Today I am forced to change all passwords for all accounts in the VBR appliance and at least some Veeam VIA proxies as well. Did not try all machines now. Other VIA proxies were added later.

Is this expected behavior after some time? Can it be deactivated (i do not remember any setting for this)?

If that is necessary every 2 months, it could get annoying having lots of infrastructure machines.

[Didi7]

I searched the User Guide for v13 but unfortunately I did not find any info about account password expiration regarding the central appliance or VIA-based machines. I tried some more VIA machines, some require password changes for all the accounts, on some I can still login without password change (became part of the infrastructure a couple of days later).

So my guess is, passwords for accounts in the VBR appliance or VIA-based machines expire after 60 days or something and at least in the VIA Host Management I did not find and option to deactivate password expiration.

Having 30 machines and soon more than 60 machines all in all, there is a lot of password changes necessary including adjustment in password database each 60 days for at least 2 or 3 accounts in every machine?

[Gostev]

As far as I know, it is a DISA STIG requirement to rotate passwords of local OS accounts every 2 months.

[Didi7]

Ok, I checked it and you are right ...




The VBR appliance and its Host Management UI and all VIA-based machines have local OS accounts. Let's assume a minimum of 2 accounts are configured (a minimum of one alternative account to 'veeamadmin') in each VIA-based machine.

Now, let's do the math for our environment ...

60 VIA-based machines (each minimum 2 accounts) => results in 120 local OS accounts
1 VBR appliance including Host Management UI (each minimum 2 accounts) => results in 4 local OS accounts

In total this would be 124 accounts (having only 2 local OS accounts) for which every 60 days you would have to change the password! I am not talking about the work that may need to be done in your password safe additionally.

That's a lot of work every 2 months IMO

Maybe something for temporary workers? Honestly, no!


Is it realistic or completely unrealistic, that Veeam can or will incorporate some kind of 'No password expiration' or 'Password expire each 180 days' or something like that, in order to avoid to change passwords for so many accounts depending on the size of each environment every 60 days?

For us, immutable backups in hardened repositories (all backup locations) and local accounts with MFA would be enough security, at least if password expiration could be extended beyond 60 days, otherwise that would really be hard.

[Didi7]

Ok, today I decided to start the nightmare and change passwords for 3 accounts on 28 different systems (VBR, VHR, VIA) including entering OTPs for some accounts and activating Web GUI on Hardened Repositories. This is by far not the amount of nodes that will belong to our new VBR infrastructure based on the Rocky appliance.

Now that took me 4 hours of work, including wrong entered passwords or expired OTPs. If systems will reach the amount of 60-70 nodes, this will get more than a nightmare (1 to 1,5 days of work each 2 months).

Besides sitting here and changing passwords like mad, I always think about my boss who says ... what the hell are you doing every 2 months on these specific days ... Now what would be the answer? Changing passwords like mad?

Honestly, this is so annoying. I really hope Veeam can find a way how to stop us changing passwords for all accounts belonging to the VBR infrastructure each 2 months *please*

Eating some food now!

Regards,
Didi7

[jorgedlcruz]

Hello Didi7,
Look, I must say your last two posts haunted me. I was convinced there should be another way, a better way. I am not saying this is the best way, (also this is 100% unsupported although it just mimics the WebUI flows same calls just programmatically) but without changing anything in the product, and entirely community based, it is the best I can offer. With this script you will be able to (using your secret TOTP) automate all in a breeze. In any case if adding the secrets is too much, you can use prompt. Prepare the scripts and just introduce the MFA per Appliance (veeamadmin, and SO), and call it a day.

• GitHub code: https://github.com/jorgedlcruz/veeam-appliance-password-rotate

The script has lots of options for you to automate this properly, moreover, I am saving the new passwords as CSV, but my idea is to include some password manager integration to directly POST it there, for the future. Maybe even get an email with the summary of the operations (you tell me which will you prefer)

I truly hope those 24 hours of work you are estimating turn into minutes if not down to zero seconds, if totally automated, your choice.