Posgresql - The SQL Server machine hosting the configuration database is currently unavailable

[sveloso]

Case# 05947391

Dear all,
I'm having some trouble with different jobs (only Tapejobs).
Specs: V12 with Postgres on the same machine (8CPU/32GBRam)
No heavy jobs running at that time when error is coming up.
Postgres database is around 32GB large.
Set-VBRPSQLDatabaseServerLimits was executed after migration to Postgres
At the moment it helps to restart the database service.

Job fails with this error:

The SQL Server machine hosting the configuration database is currently unavailable. Possible reasons are a network connectivity issue, server reboot, heavy load or hot backup.
Please try again later.
Error:
Timeout during reading attempt
The SQL Server machine hosting the configuration database is currently unavailable. Possible reasons are a network connectivity issue, server reboot, heavy load or hot backup.
Please try again later.
Error:
Timeout during reading attempt
Exception while reading from stream
Timeout during reading attempt

Set-VBRPSQLDatabaseServerLimits generates this:

# Do not edit this file manually!
# It will be overwritten by the ALTER SYSTEM command.
max_connections = '3000'
effective_cache_size = '22692MB'
checkpoint_completion_target = '0.9'
wal_buffers = '16MB'
default_statistics_target = '100'
random_page_cost = '1.1'
max_worker_processes = '16'
max_parallel_workers_per_gather = '4'
max_parallel_workers = '16'
max_parallel_maintenance_workers = '4'
min_wal_size = '2GB'
max_wal_size = '8GB'
hash_mem_multiplier = '2'
log_temp_files = '10MB'
log_lock_waits = 'on'
join_collapse_limit = '11'
geqo_threshold = '10'
log_line_prefix = '%m [%p] %q[user=%u,db=%d,app=%a] '
jit = 'off'
log_autovacuum_min_duration = '60s'
log_min_duration_statement = '1000'
log_statement_sample_rate = '0.01'
log_rotation_age = '10d'
shared_buffers = '512MB'
maintenance_work_mem = '1621MB'
work_mem = '9075kB'


Any suggestions?
Kind regards, Serafin

[HannesK]

Hello,
and welcome to the forums.

At the moment it helps to restart the database service.

that's not correct and support should investigate to find the root cause. Guessing for errors like that on a forum makes only little sense.

Best regards,
Hannes

[sveloso]

Hi Hannes,
sorry to say - but it helped this morning - tiny little backup job, no traffic on the veeam backup server.
Tried to restart that backup job, because it failed in the night with above mentioned error.
It failed 2 times again - after restarting the postgres service it completed as ususal.

Best regards, Serafin

[Richie Rogers]

Hi,
I'm getting this exact issue (had it for last couple of months - we only use tape backup once a month).
Is there a "proper" fix for this rather than just kicking it (restarting the PostGRESQL service)?
I have raised a support ticket for the issue today.

Thanks,
Richie

[Richie Rogers]

Sadly that "fix" doesn't work for us, so I suspect a reboot is required - still not an acceptable option to do every month, so hoping for a better solution.

Thanks,
Richie

[ddayton]

Running on the latest v12 release. Veeam B&R was installed on a physical server running Windows 2022 Standard that has a tape library with a LTO-7 tape drive. The install I selected it to use PostgreSQL and v15 was installed. Tape backups have been an issue. This last weekend the errror was:

Failed to save files info to the configuration database
Incremental backup: Failed to save files info to the configuration database
Processing finished with errors at 11/25/2023 4:46:10 AM

I have not rebooted or restarted any service and manually released the tape backup job, so far it is running for 43 minutes without error but it takes around 8 hours to complete.

[ddayton]

Update...

Tape job still running at 4:48pm and is 69% complete, 5.5TB of the 7.9TB processed with a rate of 288MB/s.

No errors so far.

[bct44]

Hello, with the release of v12.1 today, if was you i will ask to the support if there is knowns improvements about your issue.

[BSharp]

Case #07036949

I have been evaluating Veeam for a couple of months. I have found if the PostgreSQL server is the same box as the backup server, this "The SQL Server machine hosting the configuration database is currently unavailable...." error becomes more and more frequent to the point were no restarting of PostgreSQL or the box will allow the jobs to run or they run dreadfully slow.

In my latest attempt to get things working for us, I spun up a dedicated Linux PostgreSQL VM. So far, this has solved the issue. Veeam seems to put a heavy burden on the PostgreSQL database. I have no jobs currently running and the VM dedicated strictly to Veeam has a processor holding steady 99%.

Downside is that now that the jobs run flawlessly and use little to no resources on the actual server being backed up, with the LTO-8 drive attached, I can no longer do restores...I have no problem finding the files for the test restore. However once I build my restore files list and click next, I get "Failed to look up tapes" (Case #07036949 — Error: Failed to look up tapes.)

[Gostev]

FWIW 9 out of 10 PostgreSQL issues in support is due to user spinning out new PostgreSQL instances with the default settings, which are aimed at running PostgreSQL on Raspberry Pi or something.

P.S. You must be using a Community Edition, as our evaluation licenses are issued for 1 month only. But CE is limited to a handful workloads, 10 machines or 5TB of files to be exact - so there's should not be any "heavy burden" to start with. For example, in the above-mentioned 12.1, PostgreSQL is officially supported for environments 1000 times larger than CE limits.

[orange]

Hello, with the release of v12.1 today, if was you i will ask to the support if there is knowns improvements about your issue.

i am facing the same issues while opening backup to tape jobs with the msg: Timeout during reading attempt
was there any improvement on v12.1 or it was directly related to the PGSQL?

[sveloso]

Did you tried to set registry to

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\DatabaseConfigurations\PostgreSql

REG_DWORD
SqlStatementTimeout = 10000 (Decimal)

That might help - but is not the final solution

How big is your Postgres database?

[DotB]

We also had the same error.
The cause was our antivirus solution (Kaspersky Endpoint Security), which blocked data and processes of the PostgreSQL database.

As a result, the Veeam backup services kept losing the connection to the database. This led to various errors in the backup jobs and a Veeam Console that constantly lost the connection to the backup server or did not even establish it.

After changing the Kaspersky security policy by adding the path %ProgramFiles%\PostgreSQL\ to the trusted zone and allowing the execution of PostgreSQL processes, the errors were fixed.

Hope this helps

[HannesK]

Hello,
yes, security software causes a lot (Postgres) database problems... I have been involved in several cases. Sometimes things work to a certain point an then suddenly things break with the symptoms you mention.

Best regards,
Hannes