[LOCKED] Snapshot overflow (see the last post)

[PTide]

Hi,

You should contact our support team on that as it is difficult to tell what parameters should be adjusted without seeing the details.

Thanks

[RalfGoe]

There is no "silver bullet", every config require different adjustments. Kindly let our support team to review your setups and adjust them.

I was here three weeks ago (Veeam Support Case #02626516) and am amazed how many people are still asking for a solution in this issue:
The installation calculates an individual limit that is echoed in bytes (by a leading hashsign) in the portionSize-line of your local config file

/etc/veeam/veeam.ini

: mostly 1GB, maybe 4.
A Linux support engineer told me a good tuning rule is to start just by doubling this value.
I suppose vendor is still optimizing on this calculation or doing other usage statistics because they always ask you to upload your product logs bundle (/var/log/veeam/).

[PTide]

Generally speaking, yes, doubling usually works. On the other hand, if the agents has already increased portion size automatically and there is no more room on disk, then the system will fail to allocate the snapshot data file. We are considering to increase the initial default portion size, though.

Thanks

[Asasul]

Our backup process failed with the following errors,

Code: Select all

Backup                                                                 0%                                                                Status: Failed

     

     Summary                              Data

     Duration:           00:02:48         Processed:          260 MB (0%)
     Processing rate:    1.6 MB/s         Read:               260 MB
     Bottleneck:         Source           Transferred:        123.6 MB (2.1x)

      Time            Action                                                                                                                     Duration  

      00:06:37        Job bk-librenms-01 started at 2018-07-19 00:06:37 EDT
      00:06:37        Preparing to backup
      00:06:37        Creating volume snapshot                                                                                                   00:00:01
      00:06:40        Starting full backup to Repository_1
      00:06:40        Backing up BIOS bootloader on /dev/sda                                                                                     00:00:00
      00:06:42        Backed up sda 260 MB at 19.2 MB/s                                                                                          00:00:13
      00:09:21        [error] Backing up VolGroup                                                                                                00:00:00
      00:09:25        [error] Failed to perform backup
      00:09:25        [error] Snapshot overflow
      00:09:25        [error] Snapshot overflow
      00:09:25        [error] Processing finished with errors at 2018-07-19 00:09:25 EDT

[PTide]

Hi Asasul,

Please review this thread and try the proposed solutions. If that does not help then please contact our support team directly.

Thanks

[dprincipe]

Hi All, I have the same problem with snapshot overflow:

Code: Select all

[11.12.2018 12:00:52] <140001606088448> lpbcore|   Checking fail 'No space for snapshot'
[11.12.2018 12:00:52] <140001606088448> lpbcore|   Checking fail 'No space for snapshot' ok.
[11.12.2018 12:00:53] <140001606088448> lpbcore| ERR |Snapshot overflow

Can you kindly help me?

Regards,

[PTide]

Hi,

The reason for such error is that there is not enough space on the device that has been chosen by the agent for snapshot placement. In other words, the machine generates more data changes during backup procedure that the device allows to allocate.

You can try to adjust location and snapshot parameters on your own (please see some considerations here), or you can contact our support team for assistance.

Thanks!

[il_redcom]

Heloo.
I have the same error. But on the stage of backing up summary.xml
Can you help me?

Code: Select all

Time            Action                                                                                                                                               Duration
      01:00:02        Job Backup_AsteriskRedcom started at 2018-12-18 01:00:02 VLAT
      01:00:03        [warn] Backup is missing in repository
      01:00:03        Preparing to backup
      01:00:04        Creating volume snapshot                                                                                                                             00:00:03
      01:00:12        Starting full backup to Repository_3
      01:00:17        Backed up md0 242.9 MB at 8.5 MB/s                                                                                                                   00:00:28
      01:01:07        Backed up md1 37.8 GB at 39.6 MB/s                                                                                                                   00:16:18
      01:19:14        Backed up md3 220.6 GB at 11.9 MB/s                                                                                                                  05:15:55
      06:35:10        Backed up md2 4 GB at 250.8 MB/s                                                                                                                     00:00:16
      06:35:26        Backing up BIOS bootloader on /dev/sda                                                                                                               00:00:01
      06:35:27        Backed up sda 2.7 MB at 1015 KB/s                                                                                                                    00:00:02
      06:35:29        Backing up BIOS bootloader on /dev/sdb                                                                                                               00:00:01
      06:35:30        Backed up sdb 2.7 MB at 1 MB/s                                                                                                                       00:00:02
      06:35:33        Backing up summary.xml                                                                                                                               00:00:00
      06:36:07        Releasing snapshot                                                                                                                                   00:00:02
      06:36:07        [warn] Snapshot overflow
      06:36:09        [warn] Processing finished with warnings at 2018-12-18 06:36:09 VLAT

[Vitaliy S.]

Ilya, with any technical issue it is highly recommended to open a ticket with our technical team. Once you do this, please let us know your case ID.

[BR-Spiros Karampinis]

Hello folks,

I had the same issue, with the help of the Veeam support we have found a fitting solution.

The problem was that with the default settings, stretched snapshot,
the system wasn't able to create fast enough the next snapshot file, because of high IOPs,
leading to a failed job and the error snapshot overflow

The solution was to change few parameters in the veeam.ini file of the linux machine.

We have changed the parameter location, the type of the snapshot and the limits of it in the veeam.ini file.
Please create manually the folder for the snapshot file if it doesn't already exist.

You will find the veeam.ini in the folder /etc/veeam/

----

Before:
# Location folder for snapshot data, only for 'stretch' and 'common' snapshot
# location=

# Maximum possible snapshot data size, not for stretch snapshot
# maxSize= 21474836480

# Minimal possible snapshot data size, not for stretch snapshot
# minSize= 536870912

# Snapshot data type, can be 'stretch' (default) or 'common'
# type=

After:
# Location folder for snapshot data, only for 'stretch' and 'common' snapshot
location= /snapstore

# Maximum possible snapshot data size, not for stretch snapshot
maxSize= 21474836480

# Minimal possible snapshot data size, not for stretch snapshot
minSize= 5073741824

# Snapshot data type, can be 'stretch' (default) or 'common'
type= common

--------
After the change please restart the veeamservice

service veeamservice restart

With those settings, we change the snapshot type from stretched (default) to common,
we place the newly created snapshot to folder /snapstore and we limit its size to 20GB (maxSize= 21474836480 "byte")

Veeam Engineer :
The stretch snapshot is basically a set of files. When the first one is half-filled, the next one is created on-the-go and so on.
But in several cases under high I/O the next portion file fails to create on time, so the job fails.
Common snapshot is stable in such situations, the only point is that it should be configured according to your average changes size during the job.

Happy successfully backup jobs!

Kind regards
Spikar

[jcramerC3]

I am also having this issue. Can you send along those instructions please?

[PTide]

Hi,

Please check this and this posts.

Thanks!

[Ozgur Tezbasaran]

I would like to have one to, same problem.

thank you.

[PTide]

Ozgur,

Please check my previous answer,

Thanks!

[dovla091]

Veeam case number: Case #04343092

I was doing testing of Veeam product on my private laptop which is running lastest Fedora 32 kernel version 5.7.15 with LUKS full disk encryption on. I have installed from your repository the newest version of veeam 4.0.1.2365. I haven't provided any license, and I am currently using it in a "free mode". What I tried to do, is to make a full backup of machine (Fedora 32) to 512GB Sandisk SD card, approx 150GB free space, while host is around 60GB in total. When I configured everything and set that 1st.full backup, it starts to make a copy until it hits 1% then it exists with error: Failed to perform backup, Snapshot overflow, Processing finished with errors...

what can possibly go wrong. veeamsnap is there:
#lsmod | grep veeamsnap
veeamsnap 221184 0

#rpm -qa | grep veeam
veeam-release-fc24-1.0.7-1.x86_64
veeamsnap-4.0.1.2365-1.noarch
veeam-4.0.1.2365-1.fc24.x86_64

[Natalia Lupacheva]

Hi,

Thank you for sharing the support case id!
I've merged your post to the existing topic, so you can take a look at the discussion on the same issue while working with Support.
Please check Pavel's answer, worth it.

Thanks!

[jsousa08]

Hi all,

Its 2020 and get same message error. hehehe
Trying to backup 4.5TB from a SLES server. Using Veeam 10.
¿There is any KB that I can folow to solve this issue?
¿The Dynamic Snap its not available for this version?

Regards.

[PTide]

@jsousa08,

Well, it's 2020 and you might still run out of disk space/RAM/money/gas : )
Point is, that although we've improved our snapshot algorithms a lot since the first release, there are still some configurations and factors that might cause a snapshot to overflow. In such cases some not-very-complex tuning (either on your own or via support) helps to resolve the issue.

Please see my post above for explanation how snapshot algorithm works and what can be done.

If that doesn't help, please open a support case and post your case ID

Thank you!

[dmunoz]

Can I get that PM with the instructions as well ?

[Natalia Lupacheva]

Hi Daniel,

Did the instructions from Pavel's posts help?

Thanks!

[phiaramos]

Hello, we encountered the same error on our Physical server running on Suse OS. Can I get the instruction on how to solve this issue? Thank you!

[Vitaliy S.]

Hello Phia, have you tried following the instructions mentioned above? Did you face some issues with them?

[Alehandro]

Hello. How can I change size of snapshot. 1.5 Tb. Thanks.

[bluetony]

Hello, Can I get that PM with the instructions as well please?

[scoedia]

Hello,

Can I get that PM with the instructions as well please?

Thanks you.

[Gostev]

What PM are you all talking about guys?

[scoedia]

The instructions mentioned in the first messages about the solution to the "Snapshot overflow" error. Discusses how to change the settings in the veeam.ini file.

[PTide]

Hi,

Please check these posts:

- one
- two
- three

The issue with snapshot overflow is a well-known thing. Provided that your system does not impose technical limitations, the issue can be mitigated pretty easily even without opening a support case.

Therefore, I urge you to carefully read the linked posts above.

If none if those instructions work, feel free to send me a private message with a support case ID.

This topic is locked now.

Thanks!