-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Aug 25, 2020 12:52 am
- Full Name: Mat Hanson
- Contact:
Backing up postgres db within Docker volume
I am designing a backup strategy for a Docker environment and am having difficulty finding information on how this will behave, so I was hoping from some input from anyone that understands these processes more than I!
The environment is as follows -
Veeam Backup and Replication v10
Linux Veeam Agent - 4.0.0.1961-1.el7
DockerCE 3:19.03.12-3.el7
Postgres 12.3
Here's a drawing of the solution -
The way that I am hoping that this will behave is that the Veeam Agent will use application-aware processing to communicate with postgres and get the database into a stable and consistent state to take a backup, and then it will take a volume-level backup of the "/db" volume.
Firstly, am I correct in my thinking of this process? And secondly, will this process freeze database usage while the backup is occurring?
The environment is as follows -
Veeam Backup and Replication v10
Linux Veeam Agent - 4.0.0.1961-1.el7
DockerCE 3:19.03.12-3.el7
Postgres 12.3
Here's a drawing of the solution -
The way that I am hoping that this will behave is that the Veeam Agent will use application-aware processing to communicate with postgres and get the database into a stable and consistent state to take a backup, and then it will take a volume-level backup of the "/db" volume.
Firstly, am I correct in my thinking of this process? And secondly, will this process freeze database usage while the backup is occurring?
-
- Product Manager
- Posts: 14287
- Liked: 2877 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Backing up postgres db within Docker volume
Hello,
and welcome to the forums.
The Veeam agent can detect the Postgres database on the "same platform". It cannot "look inside" hundreds or thousands of containers and check whether somewhere a Postgres database is running inside.
As a workaround, you could use pre / post-scripts that have knowledge about the containers.
Best regards,
Hannes
and welcome to the forums.
The Veeam agent can detect the Postgres database on the "same platform". It cannot "look inside" hundreds or thousands of containers and check whether somewhere a Postgres database is running inside.
As a workaround, you could use pre / post-scripts that have knowledge about the containers.
Best regards,
Hannes
-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Aug 25, 2020 12:52 am
- Full Name: Mat Hanson
- Contact:
Re: Backing up postgres db within Docker volume
Hi Hannes,
Thanks for the welcome and for the reply, appreciate it.
This is where I was unsure of the behaviour of the processing - the host only houses one postgres container and this instance is directly accessible via port 5432 of the host. I assumed that the Veeam agent would be communicating over this port and should be able to talk to the db as if it was directly installed on the host alongside the Veeam agent. Is my assumption incorrect?
Thanks for the welcome and for the reply, appreciate it.
This is where I was unsure of the behaviour of the processing - the host only houses one postgres container and this instance is directly accessible via port 5432 of the host. I assumed that the Veeam agent would be communicating over this port and should be able to talk to the db as if it was directly installed on the host alongside the Veeam agent. Is my assumption incorrect?
-
- Product Manager
- Posts: 6408
- Liked: 724 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: Backing up postgres db within Docker volume
Hi @MatHanson,
Actually, it might work (although we never tested it hence cannot be officially supported in the current version).
However, there is another snag - in order to back up pgsql, VAl relies on its own snapshot capabilities. Technically, a nested container will be included in the snapshot too, but that hasn't been extensively tested too.
Let me check with our dev team and I'll get back you shortly
Thanks!
Actually, it might work (although we never tested it hence cannot be officially supported in the current version).
However, there is another snag - in order to back up pgsql, VAl relies on its own snapshot capabilities. Technically, a nested container will be included in the snapshot too, but that hasn't been extensively tested too.
Let me check with our dev team and I'll get back you shortly
Thanks!
-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Aug 25, 2020 12:52 am
- Full Name: Mat Hanson
- Contact:
Re: Backing up postgres db within Docker volume
Hi @PTide,
Thanks for the information. I actually ran a test on this last night and the job did actually finish with "Require successful processing" checked. So I assume this means that the Veeam agent was able to communicate with the containerised postgres instance, as I now have a backup of the directory which holds the docker volume. I am going to do some further testing on this, but it looks promising! Would love to hear any feedback that the devs might be able to give. Thanks again!
Thanks for the information. I actually ran a test on this last night and the job did actually finish with "Require successful processing" checked. So I assume this means that the Veeam agent was able to communicate with the containerised postgres instance, as I now have a backup of the directory which holds the docker volume. I am going to do some further testing on this, but it looks promising! Would love to hear any feedback that the devs might be able to give. Thanks again!
-
- Product Manager
- Posts: 14287
- Liked: 2877 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Backing up postgres db within Docker volume
for successful processing, the logs should say something like "PostgeSQL checkpoint has been triggered successfully"
like on this picture
like on this picture
-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Aug 25, 2020 12:52 am
- Full Name: Mat Hanson
- Contact:
Re: Backing up postgres db within Docker volume
Hi Hannes,
So this looks like the processing didn't work then?
I have checked the job logs and can't find any mention of postgres, is there anything in particular that I can look for in the logs to figure out why it has failed the processing?
So this looks like the processing didn't work then?
I have checked the job logs and can't find any mention of postgres, is there anything in particular that I can look for in the logs to figure out why it has failed the processing?
-
- Product Manager
- Posts: 14287
- Liked: 2877 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Backing up postgres db within Docker volume
I did not talk to developers, but from my point of view, it cannot see it. That means, there is no "failure".
There is nothing in the logs, because from the agent perspective, it does not exist. Things that do not exist are not logged.
There is nothing in the logs, because from the agent perspective, it does not exist. Things that do not exist are not logged.
-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Aug 25, 2020 12:52 am
- Full Name: Mat Hanson
- Contact:
Re: Backing up postgres db within Docker volume
Hi Hannes,
I understand that, I am trying to understand why it can't see it - as I don't understand how Veeam is attempting to connect to the database. Does Veeam communicate with postgres via TCP/IP or via a unix-socket? If it's via unix-socket, then I can just use a bind-mount to expose the socket from within the container to /var/lib/postgresql on the host - I just can't find any information from Veeam explaining just how it is trying to look for this db instance?
I understand that, I am trying to understand why it can't see it - as I don't understand how Veeam is attempting to connect to the database. Does Veeam communicate with postgres via TCP/IP or via a unix-socket? If it's via unix-socket, then I can just use a bind-mount to expose the socket from within the container to /var/lib/postgresql on the host - I just can't find any information from Veeam explaining just how it is trying to look for this db instance?
-
- Product Manager
- Posts: 6408
- Liked: 724 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: Backing up postgres db within Docker volume
Veeam connects to the database via libpq library.
That is, if the library is present on the host (Docker) and is aware that it needs to contact a certain DB via TCP/IP on a certain port and IP address, and that port and address is reachable from the host, then it might work.
On the other hand, if you can tolerate a couple of minutes of transactions not getting into the backup, you could probably skip the AAIP completely. Given the nature of pgsql, you could just snapshot the whole things without even caring about creating a checkpoint (it is created automatically quite frequently anyway).
Thanks!
That is, if the library is present on the host (Docker) and is aware that it needs to contact a certain DB via TCP/IP on a certain port and IP address, and that port and address is reachable from the host, then it might work.
On the other hand, if you can tolerate a couple of minutes of transactions not getting into the backup, you could probably skip the AAIP completely. Given the nature of pgsql, you could just snapshot the whole things without even caring about creating a checkpoint (it is created automatically quite frequently anyway).
Thanks!
-
- Novice
- Posts: 6
- Liked: 1 time
- Joined: Aug 25, 2020 12:52 am
- Full Name: Mat Hanson
- Contact:
Re: Backing up postgres db within Docker volume
Hi PTide,
Thanks for that info... I didn't realise that it was safe to backup a running postgres database without AAIP. We can totally tolerate not having the current transactions in the backup, I just didn't realise this was the only drawback to not using AAIP. Thanks again, my job here is done!
Thanks for that info... I didn't realise that it was safe to backup a running postgres database without AAIP. We can totally tolerate not having the current transactions in the backup, I just didn't realise this was the only drawback to not using AAIP. Thanks again, my job here is done!
Who is online
Users browsing this forum: Google [Bot], Google Feedfetcher and 13 guests