Compact Error

[jamerson]

Hi Guys,
We are using forward forever job in VEEAM with the latest 9.5.0.1922,Incremental ( recommended ) without syntetic full backup peridodically during the comapct it always errors out with the below error. the job has enought storage 5.5TB free of 12TB.
on the Maintenance job we have enabled Perform backup files health check ( detects and auto-heals corruptions ) on each last friday of the month, also we do have full bacup file maintenance to defrag and compact full backup files every last saterday of the month

Code: Select all

Failed to compact full backup file Details: Agent: Failed to process method {Transform.CompileFIB}: Not enough storage is available to process this command. Failed to write data to the file [E:\Daily-OS\Daily-OSD2018-12-14T220049.temp]. 
Backup files health check has been completed
Failed to perform backup file verification Error: Insufficient system resources exist to complete the requested service. Failed to read data from the file [E:\Daily-OS\Daily-OSD2018-12-14T220049.vbk]. Agent failed to process method {Signature.FullRecheckBackup

[wishr]

Hi Julien,

Are you sure your schedule does not imply running Compact Active Full operation because I think it does, but you do not have enough space on the repository to handle the double size of the backup. What's the size of the backup?

Also, I would recommend taking a look at the limitations section here.

Thanks

[Alex Uryumtsev]

Hi Julian,

I want to specify previous question:

What's the size of the backup?

What is the size of full backup?
I am wondering because the limitations say that the backup repository must have enough space to store a file of the full backup size.

[jamerson]

the full backup is 5.25TB, i've checked the backup settings and even i've took a picture of the settings box,
its strange that why.https://ibb.co/pbzXTBY
https://ibb.co/BKnk752

[wishr]

Hi Julien,

I have a couple of additional questions because this amount of available space should be enough for processing the compact operation if there are no side factors such as other jobs reserving SOBR space, for example, etc.

Could you please let us know what is the target storage and is it SOBR or not? Also, maybe this storage serves as a target for Cloud Connect backups?

Thanks

[jamerson]

@wishr its a Synology Storage connecting using isci.
stroage is btrfs file system.
We are using RAID 10 Storage 4 Disks of 8TB.
The Synology is located in a different VLAN than the production VLAN. and using link aggregation ieee 802.3ad draft v1 with two 1GB NICS combine to 2000Mbps, Full Duplex . MTU 1500.

[jamerson]

I have been doing some research , is this a related issue to the PoolUsageMaximum on the registry ?

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

[wishr]

Hi Julien,

Thank you for the information, but from B&R perspective, do you use SOBR or not? SOBR has a space reservation mechanism that can potentially lead to similar symptoms.

The registry key you mentioned is so no so useful on x64 systems since their page pool is quite huge, and just in general, it's about controlling your RAM page pool housekeeping, not space management.

Have you been able to check what's happening on the storage when the error occurs - this may shed some light on the situation. You may request a Webex session with our support engineer to perform an investigation together.

Thanks

[jamerson]

Dear wishr
we are not using SOBR, i have read the SOBR page and i cannot seem to remebere us configuring this.
its just a backup job simple storing the backup files into the synology storage which is connect to the local VEEAM server as Drive E.
Drive D is added as a local Drive storage on the VEEAM Repo.
as i see its just a local Drive added to the VEEAM Repo see https://ibb.co/yffBfyD
and when i click on the SOBR https://ibb.co/Jvr83X2

the job does errors out and i see the file is created see screenshot https://ibb.co/p1TvSKZ

[wishr]

Hi Julien,

Now it's clear. Thank you for the inputs.

As you see in the last screenshot, the full backup file that is supposed to be compacted pushing the available storage to the limit (5.64 vs 5.5 TB) - I'm not even sure that is has been completely processed/generated. That's the root cause.

To work around that you may activate the "Remove deleted VMs data" option in the Maintenance settings section, decrease the retention value here and try to revert your initial job settings and restart the job. This might help only if you use Per-VM storage and there is some obsolete VMs data in the backup. If not, it will be required to add more storage/free some space/disable the compact operations. The last one is not recommended, of course, especially keeping in mind that the compact operation itself, once successfully completed, might decrease the size of a full backup file size significantly.

Should you have any questions, let us know. Thanks.

[jamerson]

@wishr a big thank you for the support. the job is running now.
The option defragement and compact full backup was already enabled, however this took like ages to finish and finally it errors out.
we are using forward forvever ( incremental ( recommended) without synitech full backup periodicaly, do you suggest to use the performace backup health check and Defragement and compact full backup file?

[wishr]

Hi Julien,

You are most welcome!

Yes, Defragment and Compact feature is useful, it is designed for backup modes that do no imply Active Full runs. Forever incremental is one of these modes. It's recommended to have this option enabled in order to remove blocks that are not necessary (obsolete VMs, etc), defragment the file, compact it that might result in disk space savings, however, this operation is resources consuming: it requires a quite a bit of time to complete and a lot of space on the storage (~+FullBackupSize space) is required for successful processing.

The purpose of the health check is to ensure there is no corruption of the backup files. More information can be found in our documentation by the links above.

Thanks