Standalone backup agents for Linux and Unix workloads on-premises or in the public cloud
PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

What do you have on /var/lib/centreon volume? Could you try to exclude the volume from backup and run the job again please?

Thanks

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o »

There are graphs from centreon but there's a lot of files.
I can run a test but at the end, I can't exclude that part from my backup

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

There are graphs from centreon but there's a lot of files.
Do you refer to RDD files? Is there anything else on the volume?
I can run a test but at the end, I can't exclude that part from my backup
Yes, please do so for testing.

Thank you

juiced
Novice
Posts: 6
Liked: never
Joined: Jan 14, 2016 8:40 am
Full Name: Joost Billiet
Contact:

Re: Snapshot overflow

Post by juiced »

PTide wrote:Could you please describe job settings, target settings and the server that you are using? Has the job attempted retries?

Thanks
Hi,

it's an entire machine backup that is send to the B&R 9.5 repository.
The OS is a RedHat EL6

Yesterday it did 3 retry's before i manually stopped them.
The backup from today did just one try and halted with a different error

Backup 20/12/2016 23:00

Code: Select all

  23:00:02        Job Linuxdev started at 2016-12-20 23:00:02 CET
      23:00:04        Waiting for backup infrastructure resources availability                                                                                00:00:10
      23:00:14        Preparing to backup
      23:00:24        Creating volume snapshot                                                                                                                00:00:09
      23:00:33        Starting full backup to [BE1-PRD-BCK1]BE1-PRD-BCK1
      23:00:49        Backed up dm-1 199.9 MB at 74.4 MB/s                                                                                                    00:00:03
      23:01:01        Backed up dm-2 5.0 GB at 93.3 MB/s                                                                                                      00:00:55
      23:01:56        Backing up BIOS bootloader on /dev/sdb                                                                                                  00:00:00
      23:01:56        Backed up sdb 1.0 MB at 1.9 MB/s                                                                                                        00:00:01
      23:02:28        Backed up sas 396.2 GB at 139.3 MB/s                                                                                                    00:48:33
      23:51:25        Backed up VolGroup00 77.0 GB at 92.3 MB/s                                                                                               00:14:28
      00:10:23        [error] Backing up eva 317.2 GB at 12.7 MB/s (58%)                                                                                      07:06:25
      07:17:16 (+1)   [error] Failed to perform backup
      07:17:16 (+1)   [error] No space for snapshot
      07:17:16 (+1)   [error] No space for snapshot
Backup 19/12 23:00

Code: Select all

  23:00:01        Job Linuxdev started at 2016-12-19 23:00:01 CET
      23:00:05        Waiting for backup infrastructure resources availability                                                                                00:00:10
      23:00:15        Preparing to backup
      23:00:25        Creating volume snapshot                                                                                                                00:00:07
      23:00:32        Starting full backup to [BE1-PRD-BCK1]BE1-PRD-BCK1
      23:00:48        Backed up dm-1 199.9 MB at 72.9 MB/s                                                                                                    00:00:03
      23:00:58        Backed up dm-2 5.0 GB at 97.7 MB/s                                                                                                      00:00:52
      23:01:50        Backing up BIOS bootloader on /dev/sdb                                                                                                  00:00:00
      23:01:50        Backed up sdb 1.0 MB at 1.9 MB/s                                                                                                        00:00:01
      23:02:40        Backed up sas 396.2 GB at 140.9 MB/s                                                                                                    00:48:00
      23:51:16        Backed up VolGroup00 78.0 GB at 72.7 MB/s                                                                                               00:18:32
      00:13:05        [error] Backing up eva 93.6 GB at 12.5 MB/s (17%)                                                                                       02:07:44
      02:21:38 (+1)   [error] Failed to perform backup
      02:21:38 (+1)   [error] Snapshot overflow
      02:21:38 (+1)   [error] Snapshot overflow

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o »

Hello,
PTide wrote:Do you refer to RDD files? Is there anything else on the volume?
Yes. There's Nothing else to me on that volume but as I told you, lots of files
PTide wrote:Yes, please do so for testing.
It works. So I got a partial backup but now, I would like to save the entire server.
I choose "Volume level backup >
[+] sda
---[+] sda1 /boot
[+] sdb
[+] sdc
[\] vg_gocentreon03
---[ ] lv_centreon /var/lib/centreon
---[+] lv_mysql /var/lib/mysql
---[+] lv_root /
---[+] lv_swap


I've just relaunch the job, with lv_centreon but ignore lv_mysql

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o »

well, it works as well...

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

well, it works as well...
Please run the entire machine backup again. Since it is going to be an incremental I expect it to finish successfully and to generate incremental file that will be almost as large as the total size of files kept on the centreon volume.

Thanks

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

@juiced

Could you please collect logs, post df -h output, and open a case with support team and post your case ID?

Thanks

juiced
Novice
Posts: 6
Liked: never
Joined: Jan 14, 2016 8:40 am
Full Name: Joost Billiet
Contact:

Re: Snapshot overflow

Post by juiced »

PTide wrote:@juiced

Could you please collect logs, post df -h output, and open a case with support team and post your case ID?

Thanks

Case ID: 02016623
I've attached the log files to the support case.

df -h output:

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      4.0G  2.0G  1.9G  52% /
tmpfs                  32G   76K   32G   1% /dev/shm
/dev/mapper/mpathap3  194M   58M  127M  32% /boot
/dev/mapper/mpathap2  5.0G  4.6G  482M  91% /boot/efi
/dev/mapper/eva-home   97G   85G  6.9G  93% /home
/dev/mapper/VolGroup00-LogVol03
                      3.0G  208M  2.7G   8% /tmp
/dev/mapper/VolGroup00-LogVol02
                      6.0G  4.9G  796M  87% /usr
/dev/mapper/sas-var    15G  4.4G  9.5G  32% /var
/dev/mapper/sas-scratch
                       39G   24G   14G  64% /scratch
/dev/mapper/sas-dump   49G   38G  8.1G  83% /dump
/dev/mapper/sas-testing
                       49G   30G   17G  65% /testing
/dev/mapper/eva-pde   102G   65G   32G  67% /pde
/dev/mapper/eva-oracle
                       39G   31G  6.1G  84% /oracle
/dev/mapper/eva-progress
                       20G   18G  589M  97% /progress
/dev/mapper/eva-data   97G   91G  1.2G  99% /data
/dev/mapper/eva-build
                       58G   47G  8.6G  85% /build
/dev/mapper/eva-pdis  146G  132G  6.1G  96% /pdis
/dev/mapper/VolGroup00-ssd
                       65G   28G   34G  45% /ssd
/dev/mapper/sas-cyberlab
                      197G   79G  109G  42% /cyberlab

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o »

I was thinking the same thing but no... it failed :|

11:20:53 Starting incremental backup to [GOBACKUP01]Default Backup Repository
11:20:56 Backing up BIOS bootloader on /dev/sda 00:00:01
11:23:04 Backed up sda 8.0 MB at 628.7 kB/s 00:00:13
11:23:17 [error] Backing up vg_gocentreon03 142.1 GB at 25.3 MB/s (49%) 01:36:03
12:59:33 [error] Failed to perform backup
12:59:33 [error] No space for snapshot
12:59:33 [error] No space for snapshot

I suppose there are too many files but I am not sure it is the root cause.

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o »

@ juice
Hello,
Are you running the release or a beta ?
I use to have the same pb than you "snpshot overflow" with the beta. Now I am with the release, I got the message "no space for snapshot"...

juiced
Novice
Posts: 6
Liked: never
Joined: Jan 14, 2016 8:40 am
Full Name: Joost Billiet
Contact:

Re: Snapshot overflow

Post by juiced »

b1o wrote:@ juice
Hello,
Are you running the release or a beta ?
I use to have the same pb than you "snpshot overflow" with the beta. Now I am with the release, I got the message "no space for snapshot"...

it's the release version i'm running.
The snapshot overflow was the first time round, since then it's been the no space for snapshot error

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o »

ok
so we're in the same case.

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

I suppose there are too many files but I am not sure it is the root cause.
The root cause is that RRD files are being changed all the time thus making the snapshot file to grow huge. The very same graph historical data is also kept in the mysql database so basically when you backup both mysql and rdd you backup the same data twice. As far as I remember RRD files can be easily recreated, however you need to rebuild RRD in order to get historical data from mysql back into the graphs. In other words you should exclude the whole centreon volume that contains RRDs in it, as there is no point to backup the same data twice (not to mention that it requires enormous amount of space for a snapshot). The only drawback of such approach is that after bare-metal restore you'll have to recreate RRDs with centreon built-in tools.

P.S. I've sent you some case-specific instructions via PM.

Thanks

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o »

I was so hopefull this time... but it finally crashed with the same error.
gocentreon03 Error 14:10:42 16:42:29 217,1 GB 217,1 GB 38,5 GB 2:31:47 Backing up vg_gocentreon03 217.1 GB at 24.9 MB/s (76%)
Failed to perform backup
No space for snapshot
No space for snapshot

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

Please upload new logs to the ftp the support has provided.

volganet
Lurker
Posts: 1
Liked: never
Joined: Dec 21, 2016 6:41 pm
Contact:

Re: Snapshot overflow

Post by volganet »

Same here. GA version, centos 7, lvm with 1.8 TB size, 1.1 free, entire machine backup to cifs w2016 share, after 2 or 3 8gb snapshots, job fails with snapshot overflow error, although there are lot of free space available.

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

@volganet

Hi,

I'm going to ask you the same questions - did any retries occur and what's the output of lsblk -af. Also please provide logs, and some info on what kind of software do you use on that server.

Thanks!

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o »

91% today and fails.
I've uploaded new logs to the support.

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

The only option is to set the parameter I've sent you even higher, but it's not a long term solution. If the data change rate gives you a peak some day, then the overflow will happen again. Should you decide to test it with a higher value please let me know about the result.

Thank you.

b1o
Enthusiast
Posts: 27
Liked: 1 time
Joined: Dec 09, 2016 10:49 am
Contact:

Re: Snapshot overflow

Post by b1o »

Hi,

As you told me, I am not pleased with that if you consider it is not a long time solution.
maybe the support will tell me more about it...

But otherwyse, it works, I only had a warning once " [warn] Some backup files are missing in repository" : Job BackupJob1 started at 2016-12-22 13:49:49 CET

thanks,

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

Glad to hear it works for you. We've already decided to optimize the snapshot algorithm for future releases - all because of your feedback, thank you for that!

nopyobe
Lurker
Posts: 2
Liked: never
Joined: Apr 16, 2012 7:25 pm
Full Name: Kevin Hayashi
Contact:

Re: Snapshot overflow

Post by nopyobe »

Can you also PM me the instructions for this fix? I am getting the Snapshot overflow error message when backing up using the release version of the Veeam agent for Linux. Linux OS is CentOS Linux release 7.3.1611 backing up to Veeam B&R 9.5.0.711.

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

Hi,

How many percents does the job get through before it fails? What kind of load do you have on that server? Please also provide lsblk -af output.

Thanks

nopyobe
Lurker
Posts: 2
Liked: never
Joined: Apr 16, 2012 7:25 pm
Full Name: Kevin Hayashi
Contact:

Re: Snapshot overflow

Post by nopyobe »

The percent complete varies. Initial backups of the server were fine but as soon as sdb was utilized, the backups started failing. The furthest that any one backup has completed since was 67%. sdb is used as a SAMBA share for multiple other Linux servers to write data to. I'll admit, that there is quite a bit of read/write going on.

NAME FSTYPE LABEL UUID MOUNTPOINT
sda
├─sda1 xfs 349a4411-3910-45a2-a6d6-776c0b15271e /boot
└─sda2 LVM2_member unTGgZ-BDoq-RNtN-phPt-hxJj-wycp-Y1f2KS
├─centos-root xfs 6591e244-9827-42ba-9ca6-7e0f3ad3f401 /
├─centos-swap swap 2afd9564-8c90-46af-ab22-93836924bad6 [SWAP]
└─centos-home xfs 3122fd8f-bd0b-467f-80af-b14ed2d5dfc7 /home
sdb
└─sdb1 ext3 e579fb5d-fa47-499f-af16-30237795ceea /backup
sr0

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

We need to review your logs. Please open a support case and post your case ID. Also please tell us what's the output of df -ah.

Thanks

EID-Servicios
Service Provider
Posts: 6
Liked: never
Joined: Jan 29, 2017 5:07 pm
Full Name: EID
Contact:

Re: Snapshot overflow

Post by EID-Servicios »

Hi PTide:

I also have this problem, can I get the instructions please in PM?

Thanks in advance,
Best regards.

PTide
Product Manager
Posts: 5626
Liked: 549 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshot overflow

Post by PTide »

@EID-Servicios

Hi,

Kindly contact our support team directly and provide them with logs. Also please post your case ID here.

Thank you

EID-Servicios
Service Provider
Posts: 6
Liked: never
Joined: Jan 29, 2017 5:07 pm
Full Name: EID
Contact:

Re: Snapshot overflow

Post by EID-Servicios »

@PTIde

Hi, I just open a case (# 02053551)

Best regards,

sdunfee
Lurker
Posts: 1
Liked: never
Joined: Feb 07, 2017 7:30 pm
Full Name: Stephen Dunfee
Contact:

Re: Snapshot overflow

Post by sdunfee »

PTide,
I have an Ubuntu 10.4 server with a 3TB drive that is getting this failure as well with the current production version. Can I get the instructions as well?

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests