[LOCKED] Snapshot overflow (see the last post)

[PTide]

What do you have on /var/lib/centreon volume? Could you try to exclude the volume from backup and run the job again please?

Thanks

[b1o]

There are graphs from centreon but there's a lot of files.
I can run a test but at the end, I can't exclude that part from my backup

[PTide]

There are graphs from centreon but there's a lot of files.

Do you refer to RDD files? Is there anything else on the volume?

I can run a test but at the end, I can't exclude that part from my backup

Yes, please do so for testing.

Thank you

[juiced]

Could you please describe job settings, target settings and the server that you are using? Has the job attempted retries?

Thanks

Hi,

it's an entire machine backup that is send to the B&R 9.5 repository.
The OS is a RedHat EL6

Yesterday it did 3 retry's before i manually stopped them.
The backup from today did just one try and halted with a different error

Backup 20/12/2016 23:00

Code: Select all

  23:00:02        Job Linuxdev started at 2016-12-20 23:00:02 CET
      23:00:04        Waiting for backup infrastructure resources availability                                                                                00:00:10
      23:00:14        Preparing to backup
      23:00:24        Creating volume snapshot                                                                                                                00:00:09
      23:00:33        Starting full backup to [BE1-PRD-BCK1]BE1-PRD-BCK1
      23:00:49        Backed up dm-1 199.9 MB at 74.4 MB/s                                                                                                    00:00:03
      23:01:01        Backed up dm-2 5.0 GB at 93.3 MB/s                                                                                                      00:00:55
      23:01:56        Backing up BIOS bootloader on /dev/sdb                                                                                                  00:00:00
      23:01:56        Backed up sdb 1.0 MB at 1.9 MB/s                                                                                                        00:00:01
      23:02:28        Backed up sas 396.2 GB at 139.3 MB/s                                                                                                    00:48:33
      23:51:25        Backed up VolGroup00 77.0 GB at 92.3 MB/s                                                                                               00:14:28
      00:10:23        [error] Backing up eva 317.2 GB at 12.7 MB/s (58%)                                                                                      07:06:25
      07:17:16 (+1)   [error] Failed to perform backup
      07:17:16 (+1)   [error] No space for snapshot
      07:17:16 (+1)   [error] No space for snapshot

Backup 19/12 23:00

Code: Select all

  23:00:01        Job Linuxdev started at 2016-12-19 23:00:01 CET
      23:00:05        Waiting for backup infrastructure resources availability                                                                                00:00:10
      23:00:15        Preparing to backup
      23:00:25        Creating volume snapshot                                                                                                                00:00:07
      23:00:32        Starting full backup to [BE1-PRD-BCK1]BE1-PRD-BCK1
      23:00:48        Backed up dm-1 199.9 MB at 72.9 MB/s                                                                                                    00:00:03
      23:00:58        Backed up dm-2 5.0 GB at 97.7 MB/s                                                                                                      00:00:52
      23:01:50        Backing up BIOS bootloader on /dev/sdb                                                                                                  00:00:00
      23:01:50        Backed up sdb 1.0 MB at 1.9 MB/s                                                                                                        00:00:01
      23:02:40        Backed up sas 396.2 GB at 140.9 MB/s                                                                                                    00:48:00
      23:51:16        Backed up VolGroup00 78.0 GB at 72.7 MB/s                                                                                               00:18:32
      00:13:05        [error] Backing up eva 93.6 GB at 12.5 MB/s (17%)                                                                                       02:07:44
      02:21:38 (+1)   [error] Failed to perform backup
      02:21:38 (+1)   [error] Snapshot overflow
      02:21:38 (+1)   [error] Snapshot overflow

[b1o]

Hello,

Do you refer to RDD files? Is there anything else on the volume?

Yes. There's Nothing else to me on that volume but as I told you, lots of files

Yes, please do so for testing.

It works. So I got a partial backup but now, I would like to save the entire server.
I choose "Volume level backup >
[+] sda
---[+] sda1 /boot
[+] sdb
[+] sdc
[\] vg_gocentreon03
---[ ] lv_centreon /var/lib/centreon
---[+] lv_mysql /var/lib/mysql
---[+] lv_root /
---[+] lv_swap


I've just relaunch the job, with lv_centreon but ignore lv_mysql

[b1o]

well, it works as well...

[PTide]

well, it works as well...

Please run the entire machine backup again. Since it is going to be an incremental I expect it to finish successfully and to generate incremental file that will be almost as large as the total size of files kept on the centreon volume.

Thanks

[PTide]

@juiced

Could you please collect logs, post df -h output, and open a case with support team and post your case ID?

Thanks

[juiced]

@juiced

Could you please collect logs, post df -h output, and open a case with support team and post your case ID?

Thanks

Case ID: 02016623
I've attached the log files to the support case.

df -h output:

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      4.0G  2.0G  1.9G  52% /
tmpfs                  32G   76K   32G   1% /dev/shm
/dev/mapper/mpathap3  194M   58M  127M  32% /boot
/dev/mapper/mpathap2  5.0G  4.6G  482M  91% /boot/efi
/dev/mapper/eva-home   97G   85G  6.9G  93% /home
/dev/mapper/VolGroup00-LogVol03
                      3.0G  208M  2.7G   8% /tmp
/dev/mapper/VolGroup00-LogVol02
                      6.0G  4.9G  796M  87% /usr
/dev/mapper/sas-var    15G  4.4G  9.5G  32% /var
/dev/mapper/sas-scratch
                       39G   24G   14G  64% /scratch
/dev/mapper/sas-dump   49G   38G  8.1G  83% /dump
/dev/mapper/sas-testing
                       49G   30G   17G  65% /testing
/dev/mapper/eva-pde   102G   65G   32G  67% /pde
/dev/mapper/eva-oracle
                       39G   31G  6.1G  84% /oracle
/dev/mapper/eva-progress
                       20G   18G  589M  97% /progress
/dev/mapper/eva-data   97G   91G  1.2G  99% /data
/dev/mapper/eva-build
                       58G   47G  8.6G  85% /build
/dev/mapper/eva-pdis  146G  132G  6.1G  96% /pdis
/dev/mapper/VolGroup00-ssd
                       65G   28G   34G  45% /ssd
/dev/mapper/sas-cyberlab
                      197G   79G  109G  42% /cyberlab

[b1o]

I was thinking the same thing but no... it failed

11:20:53 Starting incremental backup to [GOBACKUP01]Default Backup Repository
11:20:56 Backing up BIOS bootloader on /dev/sda 00:00:01
11:23:04 Backed up sda 8.0 MB at 628.7 kB/s 00:00:13
11:23:17 [error] Backing up vg_gocentreon03 142.1 GB at 25.3 MB/s (49%) 01:36:03
12:59:33 [error] Failed to perform backup
12:59:33 [error] No space for snapshot
12:59:33 [error] No space for snapshot

I suppose there are too many files but I am not sure it is the root cause.

[b1o]

@ juice
Hello,
Are you running the release or a beta ?
I use to have the same pb than you "snpshot overflow" with the beta. Now I am with the release, I got the message "no space for snapshot"...

[juiced]

@ juice
Hello,
Are you running the release or a beta ?
I use to have the same pb than you "snpshot overflow" with the beta. Now I am with the release, I got the message "no space for snapshot"...

it's the release version i'm running.
The snapshot overflow was the first time round, since then it's been the no space for snapshot error

[b1o]

ok
so we're in the same case.

[PTide]

I suppose there are too many files but I am not sure it is the root cause.

The root cause is that RRD files are being changed all the time thus making the snapshot file to grow huge. The very same graph historical data is also kept in the mysql database so basically when you backup both mysql and rdd you backup the same data twice. As far as I remember RRD files can be easily recreated, however you need to rebuild RRD in order to get historical data from mysql back into the graphs. In other words you should exclude the whole centreon volume that contains RRDs in it, as there is no point to backup the same data twice (not to mention that it requires enormous amount of space for a snapshot). The only drawback of such approach is that after bare-metal restore you'll have to recreate RRDs with centreon built-in tools.

P.S. I've sent you some case-specific instructions via PM.

Thanks

[b1o]

I was so hopefull this time... but it finally crashed with the same error.
gocentreon03 Error 14:10:42 16:42:29 217,1 GB 217,1 GB 38,5 GB 2:31:47 Backing up vg_gocentreon03 217.1 GB at 24.9 MB/s (76%)
Failed to perform backup
No space for snapshot
No space for snapshot

[PTide]

Please upload new logs to the ftp the support has provided.

[volganet]

Same here. GA version, centos 7, lvm with 1.8 TB size, 1.1 free, entire machine backup to cifs w2016 share, after 2 or 3 8gb snapshots, job fails with snapshot overflow error, although there are lot of free space available.

[PTide]

@volganet

Hi,

I'm going to ask you the same questions - did any retries occur and what's the output of lsblk -af. Also please provide logs, and some info on what kind of software do you use on that server.

Thanks!

[b1o]

91% today and fails.
I've uploaded new logs to the support.

[PTide]

The only option is to set the parameter I've sent you even higher, but it's not a long term solution. If the data change rate gives you a peak some day, then the overflow will happen again. Should you decide to test it with a higher value please let me know about the result.

Thank you.

[b1o]

Hi,

As you told me, I am not pleased with that if you consider it is not a long time solution.
maybe the support will tell me more about it...

But otherwyse, it works, I only had a warning once " [warn] Some backup files are missing in repository" : Job BackupJob1 started at 2016-12-22 13:49:49 CET

thanks,

[PTide]

Glad to hear it works for you. We've already decided to optimize the snapshot algorithm for future releases - all because of your feedback, thank you for that!

[nopyobe]

Can you also PM me the instructions for this fix? I am getting the Snapshot overflow error message when backing up using the release version of the Veeam agent for Linux. Linux OS is CentOS Linux release 7.3.1611 backing up to Veeam B&R 9.5.0.711.

[PTide]

Hi,

How many percents does the job get through before it fails? What kind of load do you have on that server? Please also provide lsblk -af output.

Thanks

[nopyobe]

The percent complete varies. Initial backups of the server were fine but as soon as sdb was utilized, the backups started failing. The furthest that any one backup has completed since was 67%. sdb is used as a SAMBA share for multiple other Linux servers to write data to. I'll admit, that there is quite a bit of read/write going on.

NAME FSTYPE LABEL UUID MOUNTPOINT
sda
├─sda1 xfs 349a4411-3910-45a2-a6d6-776c0b15271e /boot
└─sda2 LVM2_member unTGgZ-BDoq-RNtN-phPt-hxJj-wycp-Y1f2KS
├─centos-root xfs 6591e244-9827-42ba-9ca6-7e0f3ad3f401 /
├─centos-swap swap 2afd9564-8c90-46af-ab22-93836924bad6 [SWAP]
└─centos-home xfs 3122fd8f-bd0b-467f-80af-b14ed2d5dfc7 /home
sdb
└─sdb1 ext3 e579fb5d-fa47-499f-af16-30237795ceea /backup
sr0

[PTide]

We need to review your logs. Please open a support case and post your case ID. Also please tell us what's the output of df -ah.

Thanks

[EID-Servicios]

Hi PTide:

I also have this problem, can I get the instructions please in PM?

Thanks in advance,
Best regards.

[PTide]

@EID-Servicios

Hi,

Kindly contact our support team directly and provide them with logs. Also please post your case ID here.

Thank you

[EID-Servicios]

@PTIde

Hi, I just open a case (# 02053551)

Best regards,

[sdunfee]

PTide,
I have an Ubuntu 10.4 server with a 3TB drive that is getting this failure as well with the current production version. Can I get the instructions as well?