Snapshot overflow

Backup agent for Linux servers and workstations on-premises or in the public cloud

Re: Snapshot overflow

Veeam Logoby PTide » Tue Dec 20, 2016 3:10 pm

What do you have on /var/lib/centreon volume? Could you try to exclude the volume from backup and run the job again please?

Thanks
PTide
Veeam Software
 
Posts: 3078
Liked: 251 times
Joined: Tue May 19, 2015 1:46 pm

Re: Snapshot overflow

Veeam Logoby b1o » Tue Dec 20, 2016 3:29 pm

There are graphs from centreon but there's a lot of files.
I can run a test but at the end, I can't exclude that part from my backup
b1o
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Fri Dec 09, 2016 10:49 am

Re: Snapshot overflow

Veeam Logoby PTide » Tue Dec 20, 2016 4:22 pm

There are graphs from centreon but there's a lot of files.

Do you refer to RDD files? Is there anything else on the volume?

I can run a test but at the end, I can't exclude that part from my backup

Yes, please do so for testing.

Thank you
PTide
Veeam Software
 
Posts: 3078
Liked: 251 times
Joined: Tue May 19, 2015 1:46 pm

Re: Snapshot overflow

Veeam Logoby juiced » Wed Dec 21, 2016 7:40 am

PTide wrote:Could you please describe job settings, target settings and the server that you are using? Has the job attempted retries?

Thanks


Hi,

it's an entire machine backup that is send to the B&R 9.5 repository.
The OS is a RedHat EL6

Yesterday it did 3 retry's before i manually stopped them.
The backup from today did just one try and halted with a different error

Backup 20/12/2016 23:00
Code: Select all
  23:00:02        Job Linuxdev started at 2016-12-20 23:00:02 CET
      23:00:04        Waiting for backup infrastructure resources availability                                                                                00:00:10
      23:00:14        Preparing to backup
      23:00:24        Creating volume snapshot                                                                                                                00:00:09
      23:00:33        Starting full backup to [BE1-PRD-BCK1]BE1-PRD-BCK1
      23:00:49        Backed up dm-1 199.9 MB at 74.4 MB/s                                                                                                    00:00:03
      23:01:01        Backed up dm-2 5.0 GB at 93.3 MB/s                                                                                                      00:00:55
      23:01:56        Backing up BIOS bootloader on /dev/sdb                                                                                                  00:00:00
      23:01:56        Backed up sdb 1.0 MB at 1.9 MB/s                                                                                                        00:00:01
      23:02:28        Backed up sas 396.2 GB at 139.3 MB/s                                                                                                    00:48:33
      23:51:25        Backed up VolGroup00 77.0 GB at 92.3 MB/s                                                                                               00:14:28
      00:10:23        [error] Backing up eva 317.2 GB at 12.7 MB/s (58%)                                                                                      07:06:25
      07:17:16 (+1)   [error] Failed to perform backup
      07:17:16 (+1)   [error] No space for snapshot
      07:17:16 (+1)   [error] No space for snapshot


Backup 19/12 23:00
Code: Select all
  23:00:01        Job Linuxdev started at 2016-12-19 23:00:01 CET
      23:00:05        Waiting for backup infrastructure resources availability                                                                                00:00:10
      23:00:15        Preparing to backup
      23:00:25        Creating volume snapshot                                                                                                                00:00:07
      23:00:32        Starting full backup to [BE1-PRD-BCK1]BE1-PRD-BCK1
      23:00:48        Backed up dm-1 199.9 MB at 72.9 MB/s                                                                                                    00:00:03
      23:00:58        Backed up dm-2 5.0 GB at 97.7 MB/s                                                                                                      00:00:52
      23:01:50        Backing up BIOS bootloader on /dev/sdb                                                                                                  00:00:00
      23:01:50        Backed up sdb 1.0 MB at 1.9 MB/s                                                                                                        00:00:01
      23:02:40        Backed up sas 396.2 GB at 140.9 MB/s                                                                                                    00:48:00
      23:51:16        Backed up VolGroup00 78.0 GB at 72.7 MB/s                                                                                               00:18:32
      00:13:05        [error] Backing up eva 93.6 GB at 12.5 MB/s (17%)                                                                                       02:07:44
      02:21:38 (+1)   [error] Failed to perform backup
      02:21:38 (+1)   [error] Snapshot overflow
      02:21:38 (+1)   [error] Snapshot overflow
juiced
Novice
 
Posts: 6
Liked: never
Joined: Thu Jan 14, 2016 8:40 am
Full Name: Joost Billiet

Re: Snapshot overflow

Veeam Logoby b1o » Wed Dec 21, 2016 8:26 am

Hello,

PTide wrote:Do you refer to RDD files? Is there anything else on the volume?


Yes. There's Nothing else to me on that volume but as I told you, lots of files

PTide wrote:Yes, please do so for testing.


It works. So I got a partial backup but now, I would like to save the entire server.
I choose "Volume level backup >
[+] sda
---[+] sda1 /boot
[+] sdb
[+] sdc
[\] vg_gocentreon03
---[ ] lv_centreon /var/lib/centreon
---[+] lv_mysql /var/lib/mysql
---[+] lv_root /
---[+] lv_swap


I've just relaunch the job, with lv_centreon but ignore lv_mysql
b1o
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Fri Dec 09, 2016 10:49 am

Re: Snapshot overflow

Veeam Logoby b1o » Wed Dec 21, 2016 10:17 am

well, it works as well...
b1o
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Fri Dec 09, 2016 10:49 am

Re: Snapshot overflow

Veeam Logoby PTide » Wed Dec 21, 2016 11:05 am

well, it works as well...

Please run the entire machine backup again. Since it is going to be an incremental I expect it to finish successfully and to generate incremental file that will be almost as large as the total size of files kept on the centreon volume.

Thanks
PTide
Veeam Software
 
Posts: 3078
Liked: 251 times
Joined: Tue May 19, 2015 1:46 pm

Re: Snapshot overflow

Veeam Logoby PTide » Wed Dec 21, 2016 11:24 am

@juiced

Could you please collect logs, post df -h output, and open a case with support team and post your case ID?

Thanks
PTide
Veeam Software
 
Posts: 3078
Liked: 251 times
Joined: Tue May 19, 2015 1:46 pm

Re: Snapshot overflow

Veeam Logoby juiced » Wed Dec 21, 2016 12:01 pm

PTide wrote:@juiced

Could you please collect logs, post df -h output, and open a case with support team and post your case ID?

Thanks



Case ID: 02016623
I've attached the log files to the support case.

df -h output:
Code: Select all
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      4.0G  2.0G  1.9G  52% /
tmpfs                  32G   76K   32G   1% /dev/shm
/dev/mapper/mpathap3  194M   58M  127M  32% /boot
/dev/mapper/mpathap2  5.0G  4.6G  482M  91% /boot/efi
/dev/mapper/eva-home   97G   85G  6.9G  93% /home
/dev/mapper/VolGroup00-LogVol03
                      3.0G  208M  2.7G   8% /tmp
/dev/mapper/VolGroup00-LogVol02
                      6.0G  4.9G  796M  87% /usr
/dev/mapper/sas-var    15G  4.4G  9.5G  32% /var
/dev/mapper/sas-scratch
                       39G   24G   14G  64% /scratch
/dev/mapper/sas-dump   49G   38G  8.1G  83% /dump
/dev/mapper/sas-testing
                       49G   30G   17G  65% /testing
/dev/mapper/eva-pde   102G   65G   32G  67% /pde
/dev/mapper/eva-oracle
                       39G   31G  6.1G  84% /oracle
/dev/mapper/eva-progress
                       20G   18G  589M  97% /progress
/dev/mapper/eva-data   97G   91G  1.2G  99% /data
/dev/mapper/eva-build
                       58G   47G  8.6G  85% /build
/dev/mapper/eva-pdis  146G  132G  6.1G  96% /pdis
/dev/mapper/VolGroup00-ssd
                       65G   28G   34G  45% /ssd
/dev/mapper/sas-cyberlab
                      197G   79G  109G  42% /cyberlab
juiced
Novice
 
Posts: 6
Liked: never
Joined: Thu Jan 14, 2016 8:40 am
Full Name: Joost Billiet

Re: Snapshot overflow

Veeam Logoby b1o » Wed Dec 21, 2016 12:02 pm

I was thinking the same thing but no... it failed :|

11:20:53 Starting incremental backup to [GOBACKUP01]Default Backup Repository
11:20:56 Backing up BIOS bootloader on /dev/sda 00:00:01
11:23:04 Backed up sda 8.0 MB at 628.7 kB/s 00:00:13
11:23:17 [error] Backing up vg_gocentreon03 142.1 GB at 25.3 MB/s (49%) 01:36:03
12:59:33 [error] Failed to perform backup
12:59:33 [error] No space for snapshot
12:59:33 [error] No space for snapshot

I suppose there are too many files but I am not sure it is the root cause.
b1o
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Fri Dec 09, 2016 10:49 am

Re: Snapshot overflow

Veeam Logoby b1o » Wed Dec 21, 2016 12:04 pm

@ juice
Hello,
Are you running the release or a beta ?
I use to have the same pb than you "snpshot overflow" with the beta. Now I am with the release, I got the message "no space for snapshot"...
b1o
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Fri Dec 09, 2016 10:49 am

Re: Snapshot overflow

Veeam Logoby juiced » Wed Dec 21, 2016 12:06 pm

b1o wrote:@ juice
Hello,
Are you running the release or a beta ?
I use to have the same pb than you "snpshot overflow" with the beta. Now I am with the release, I got the message "no space for snapshot"...



it's the release version i'm running.
The snapshot overflow was the first time round, since then it's been the no space for snapshot error
juiced
Novice
 
Posts: 6
Liked: never
Joined: Thu Jan 14, 2016 8:40 am
Full Name: Joost Billiet

Re: Snapshot overflow

Veeam Logoby b1o » Wed Dec 21, 2016 12:09 pm

ok
so we're in the same case.
b1o
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Fri Dec 09, 2016 10:49 am

Re: Snapshot overflow

Veeam Logoby PTide » Wed Dec 21, 2016 12:24 pm

I suppose there are too many files but I am not sure it is the root cause.

The root cause is that RRD files are being changed all the time thus making the snapshot file to grow huge. The very same graph historical data is also kept in the mysql database so basically when you backup both mysql and rdd you backup the same data twice. As far as I remember RRD files can be easily recreated, however you need to rebuild RRD in order to get historical data from mysql back into the graphs. In other words you should exclude the whole centreon volume that contains RRDs in it, as there is no point to backup the same data twice (not to mention that it requires enormous amount of space for a snapshot). The only drawback of such approach is that after bare-metal restore you'll have to recreate RRDs with centreon built-in tools.

P.S. I've sent you some case-specific instructions via PM.

Thanks
PTide
Veeam Software
 
Posts: 3078
Liked: 251 times
Joined: Tue May 19, 2015 1:46 pm

Re: Snapshot overflow

Veeam Logoby b1o » Wed Dec 21, 2016 3:45 pm

I was so hopefull this time... but it finally crashed with the same error.
gocentreon03Error14:10:4216:42:29217,1 GB217,1 GB38,5 GB2:31:47 Backing up vg_gocentreon03 217.1 GB at 24.9 MB/s (76%)
Failed to perform backup
No space for snapshot
No space for snapshot
b1o
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Fri Dec 09, 2016 10:49 am

PreviousNext

Return to Veeam Agent for Linux



Who is online

Users browsing this forum: No registered users and 3 guests