Standalone backup agent for Linux servers and workstations on-premises or in the public cloud
Post Reply
PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide » Dec 20, 2016 3:10 pm

What do you have on /var/lib/centreon volume? Could you try to exclude the volume from backup and run the job again please?

Thanks

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o » Dec 20, 2016 3:29 pm

There are graphs from centreon but there's a lot of files.
I can run a test but at the end, I can't exclude that part from my backup

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide » Dec 20, 2016 4:22 pm

There are graphs from centreon but there's a lot of files.
Do you refer to RDD files? Is there anything else on the volume?
I can run a test but at the end, I can't exclude that part from my backup
Yes, please do so for testing.

Thank you

juiced
Novice
Posts: 6
Liked: never
Joined: Jan 14, 2016 8:40 am
Full Name: Joost Billiet
Contact:

Re: Snapshot overflow

Post by juiced » Dec 21, 2016 7:40 am

PTide wrote:Could you please describe job settings, target settings and the server that you are using? Has the job attempted retries?

Thanks
Hi,

it's an entire machine backup that is send to the B&R 9.5 repository.
The OS is a RedHat EL6

Yesterday it did 3 retry's before i manually stopped them.
The backup from today did just one try and halted with a different error

Backup 20/12/2016 23:00

Code: Select all

  23:00:02        Job Linuxdev started at 2016-12-20 23:00:02 CET
      23:00:04        Waiting for backup infrastructure resources availability                                                                                00:00:10
      23:00:14        Preparing to backup
      23:00:24        Creating volume snapshot                                                                                                                00:00:09
      23:00:33        Starting full backup to [BE1-PRD-BCK1]BE1-PRD-BCK1
      23:00:49        Backed up dm-1 199.9 MB at 74.4 MB/s                                                                                                    00:00:03
      23:01:01        Backed up dm-2 5.0 GB at 93.3 MB/s                                                                                                      00:00:55
      23:01:56        Backing up BIOS bootloader on /dev/sdb                                                                                                  00:00:00
      23:01:56        Backed up sdb 1.0 MB at 1.9 MB/s                                                                                                        00:00:01
      23:02:28        Backed up sas 396.2 GB at 139.3 MB/s                                                                                                    00:48:33
      23:51:25        Backed up VolGroup00 77.0 GB at 92.3 MB/s                                                                                               00:14:28
      00:10:23        [error] Backing up eva 317.2 GB at 12.7 MB/s (58%)                                                                                      07:06:25
      07:17:16 (+1)   [error] Failed to perform backup
      07:17:16 (+1)   [error] No space for snapshot
      07:17:16 (+1)   [error] No space for snapshot
Backup 19/12 23:00

Code: Select all

  23:00:01        Job Linuxdev started at 2016-12-19 23:00:01 CET
      23:00:05        Waiting for backup infrastructure resources availability                                                                                00:00:10
      23:00:15        Preparing to backup
      23:00:25        Creating volume snapshot                                                                                                                00:00:07
      23:00:32        Starting full backup to [BE1-PRD-BCK1]BE1-PRD-BCK1
      23:00:48        Backed up dm-1 199.9 MB at 72.9 MB/s                                                                                                    00:00:03
      23:00:58        Backed up dm-2 5.0 GB at 97.7 MB/s                                                                                                      00:00:52
      23:01:50        Backing up BIOS bootloader on /dev/sdb                                                                                                  00:00:00
      23:01:50        Backed up sdb 1.0 MB at 1.9 MB/s                                                                                                        00:00:01
      23:02:40        Backed up sas 396.2 GB at 140.9 MB/s                                                                                                    00:48:00
      23:51:16        Backed up VolGroup00 78.0 GB at 72.7 MB/s                                                                                               00:18:32
      00:13:05        [error] Backing up eva 93.6 GB at 12.5 MB/s (17%)                                                                                       02:07:44
      02:21:38 (+1)   [error] Failed to perform backup
      02:21:38 (+1)   [error] Snapshot overflow
      02:21:38 (+1)   [error] Snapshot overflow

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o » Dec 21, 2016 8:26 am

Hello,
PTide wrote:Do you refer to RDD files? Is there anything else on the volume?
Yes. There's Nothing else to me on that volume but as I told you, lots of files
PTide wrote:Yes, please do so for testing.
It works. So I got a partial backup but now, I would like to save the entire server.
I choose "Volume level backup >
[+] sda
---[+] sda1 /boot
[+] sdb
[+] sdc
[\] vg_gocentreon03
---[ ] lv_centreon /var/lib/centreon
---[+] lv_mysql /var/lib/mysql
---[+] lv_root /
---[+] lv_swap


I've just relaunch the job, with lv_centreon but ignore lv_mysql

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o » Dec 21, 2016 10:17 am

well, it works as well...

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide » Dec 21, 2016 11:05 am

well, it works as well...
Please run the entire machine backup again. Since it is going to be an incremental I expect it to finish successfully and to generate incremental file that will be almost as large as the total size of files kept on the centreon volume.

Thanks

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide » Dec 21, 2016 11:24 am

@juiced

Could you please collect logs, post df -h output, and open a case with support team and post your case ID?

Thanks

juiced
Novice
Posts: 6
Liked: never
Joined: Jan 14, 2016 8:40 am
Full Name: Joost Billiet
Contact:

Re: Snapshot overflow

Post by juiced » Dec 21, 2016 12:01 pm

PTide wrote:@juiced

Could you please collect logs, post df -h output, and open a case with support team and post your case ID?

Thanks

Case ID: 02016623
I've attached the log files to the support case.

df -h output:

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      4.0G  2.0G  1.9G  52% /
tmpfs                  32G   76K   32G   1% /dev/shm
/dev/mapper/mpathap3  194M   58M  127M  32% /boot
/dev/mapper/mpathap2  5.0G  4.6G  482M  91% /boot/efi
/dev/mapper/eva-home   97G   85G  6.9G  93% /home
/dev/mapper/VolGroup00-LogVol03
                      3.0G  208M  2.7G   8% /tmp
/dev/mapper/VolGroup00-LogVol02
                      6.0G  4.9G  796M  87% /usr
/dev/mapper/sas-var    15G  4.4G  9.5G  32% /var
/dev/mapper/sas-scratch
                       39G   24G   14G  64% /scratch
/dev/mapper/sas-dump   49G   38G  8.1G  83% /dump
/dev/mapper/sas-testing
                       49G   30G   17G  65% /testing
/dev/mapper/eva-pde   102G   65G   32G  67% /pde
/dev/mapper/eva-oracle
                       39G   31G  6.1G  84% /oracle
/dev/mapper/eva-progress
                       20G   18G  589M  97% /progress
/dev/mapper/eva-data   97G   91G  1.2G  99% /data
/dev/mapper/eva-build
                       58G   47G  8.6G  85% /build
/dev/mapper/eva-pdis  146G  132G  6.1G  96% /pdis
/dev/mapper/VolGroup00-ssd
                       65G   28G   34G  45% /ssd
/dev/mapper/sas-cyberlab
                      197G   79G  109G  42% /cyberlab

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o » Dec 21, 2016 12:02 pm

I was thinking the same thing but no... it failed :|

11:20:53 Starting incremental backup to [GOBACKUP01]Default Backup Repository
11:20:56 Backing up BIOS bootloader on /dev/sda 00:00:01
11:23:04 Backed up sda 8.0 MB at 628.7 kB/s 00:00:13
11:23:17 [error] Backing up vg_gocentreon03 142.1 GB at 25.3 MB/s (49%) 01:36:03
12:59:33 [error] Failed to perform backup
12:59:33 [error] No space for snapshot
12:59:33 [error] No space for snapshot

I suppose there are too many files but I am not sure it is the root cause.

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o » Dec 21, 2016 12:04 pm

@ juice
Hello,
Are you running the release or a beta ?
I use to have the same pb than you "snpshot overflow" with the beta. Now I am with the release, I got the message "no space for snapshot"...

juiced
Novice
Posts: 6
Liked: never
Joined: Jan 14, 2016 8:40 am
Full Name: Joost Billiet
Contact:

Re: Snapshot overflow

Post by juiced » Dec 21, 2016 12:06 pm

b1o wrote:@ juice
Hello,
Are you running the release or a beta ?
I use to have the same pb than you "snpshot overflow" with the beta. Now I am with the release, I got the message "no space for snapshot"...

it's the release version i'm running.
The snapshot overflow was the first time round, since then it's been the no space for snapshot error

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o » Dec 21, 2016 12:09 pm

ok
so we're in the same case.

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide » Dec 21, 2016 12:24 pm

I suppose there are too many files but I am not sure it is the root cause.
The root cause is that RRD files are being changed all the time thus making the snapshot file to grow huge. The very same graph historical data is also kept in the mysql database so basically when you backup both mysql and rdd you backup the same data twice. As far as I remember RRD files can be easily recreated, however you need to rebuild RRD in order to get historical data from mysql back into the graphs. In other words you should exclude the whole centreon volume that contains RRDs in it, as there is no point to backup the same data twice (not to mention that it requires enormous amount of space for a snapshot). The only drawback of such approach is that after bare-metal restore you'll have to recreate RRDs with centreon built-in tools.

P.S. I've sent you some case-specific instructions via PM.

Thanks

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o » Dec 21, 2016 3:45 pm

I was so hopefull this time... but it finally crashed with the same error.
gocentreon03 Error 14:10:42 16:42:29 217,1 GB 217,1 GB 38,5 GB 2:31:47 Backing up vg_gocentreon03 217.1 GB at 24.9 MB/s (76%)
Failed to perform backup
No space for snapshot
No space for snapshot

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests