Host-based backup of VMware vSphere VMs.
Post Reply
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Compact Error

Post by jamerson »

Hi Guys,
We are using forward forever job in VEEAM with the latest 9.5.0.1922,Incremental ( recommended ) without syntetic full backup peridodically during the comapct it always errors out with the below error. the job has enought storage 5.5TB free of 12TB.
on the Maintenance job we have enabled Perform backup files health check ( detects and auto-heals corruptions ) on each last friday of the month, also we do have full bacup file maintenance to defrag and compact full backup files every last saterday of the month

Code: Select all

Failed to compact full backup file Details: Agent: Failed to process method {Transform.CompileFIB}: Not enough storage is available to process this command. Failed to write data to the file [E:\Daily-OS\Daily-OSD2018-12-14T220049.temp]. 
Backup files health check has been completed
Failed to perform backup file verification Error: Insufficient system resources exist to complete the requested service. Failed to read data from the file [E:\Daily-OS\Daily-OSD2018-12-14T220049.vbk]. Agent failed to process method {Signature.FullRecheckBackup
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Compact Error

Post by wishr »

Hi Julien,

Are you sure your schedule does not imply running Compact Active Full operation because I think it does, but you do not have enough space on the repository to handle the double size of the backup. What's the size of the backup?

Also, I would recommend taking a look at the limitations section here.

Thanks
Alex Uryumtsev
Enthusiast
Posts: 51
Liked: 7 times
Joined: Oct 29, 2018 9:56 am
Full Name: Alexander Uryumtsev
Contact:

Re: Compact Error

Post by Alex Uryumtsev »

Hi Julian,

I want to specify previous question:
What's the size of the backup?
What is the size of full backup?
I am wondering because the limitations say that the backup repository must have enough space to store a file of the full backup size.
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: Compact Error

Post by jamerson »

the full backup is 5.25TB, i've checked the backup settings and even i've took a picture of the settings box,
its strange that why.https://ibb.co/pbzXTBY
https://ibb.co/BKnk752
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Compact Error

Post by wishr »

Hi Julien,

I have a couple of additional questions because this amount of available space should be enough for processing the compact operation if there are no side factors such as other jobs reserving SOBR space, for example, etc.

Could you please let us know what is the target storage and is it SOBR or not? Also, maybe this storage serves as a target for Cloud Connect backups?

Thanks
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: Compact Error

Post by jamerson »

@wishr its a Synology Storage connecting using isci.
stroage is btrfs file system.
We are using RAID 10 Storage 4 Disks of 8TB.
The Synology is located in a different VLAN than the production VLAN. and using link aggregation ieee 802.3ad draft v1 with two 1GB NICS combine to 2000Mbps, Full Duplex . MTU 1500.
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: Compact Error

Post by jamerson »

I have been doing some research , is this a related issue to the PoolUsageMaximum on the registry ?

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Compact Error

Post by wishr »

Hi Julien,

Thank you for the information, but from B&R perspective, do you use SOBR or not? SOBR has a space reservation mechanism that can potentially lead to similar symptoms.

The registry key you mentioned is so no so useful on x64 systems since their page pool is quite huge, and just in general, it's about controlling your RAM page pool housekeeping, not space management.

Have you been able to check what's happening on the storage when the error occurs - this may shed some light on the situation. You may request a Webex session with our support engineer to perform an investigation together.

Thanks
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: Compact Error

Post by jamerson »

Dear wishr
we are not using SOBR, i have read the SOBR page and i cannot seem to remebere us configuring this.
its just a backup job simple storing the backup files into the synology storage which is connect to the local VEEAM server as Drive E.
Drive D is added as a local Drive storage on the VEEAM Repo.
as i see its just a local Drive added to the VEEAM Repo see https://ibb.co/yffBfyD
and when i click on the SOBR https://ibb.co/Jvr83X2

the job does errors out and i see the file is created see screenshot https://ibb.co/p1TvSKZ
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Compact Error

Post by wishr »

Hi Julien,

Now it's clear. Thank you for the inputs.

As you see in the last screenshot, the full backup file that is supposed to be compacted pushing the available storage to the limit (5.64 vs 5.5 TB) - I'm not even sure that is has been completely processed/generated. That's the root cause.

To work around that you may activate the "Remove deleted VMs data" option in the Maintenance settings section, decrease the retention value here and try to revert your initial job settings and restart the job. This might help only if you use Per-VM storage and there is some obsolete VMs data in the backup. If not, it will be required to add more storage/free some space/disable the compact operations. The last one is not recommended, of course, especially keeping in mind that the compact operation itself, once successfully completed, might decrease the size of a full backup file size significantly.

Should you have any questions, let us know. Thanks.
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: Compact Error

Post by jamerson »

@wishr a big thank you for the support. the job is running now.
The option defragement and compact full backup was already enabled, however this took like ages to finish and finally it errors out.
we are using forward forvever ( incremental ( recommended) without synitech full backup periodicaly, do you suggest to use the performace backup health check and Defragement and compact full backup file?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Compact Error

Post by wishr »

Hi Julien,

You are most welcome!

Yes, Defragment and Compact feature is useful, it is designed for backup modes that do no imply Active Full runs. Forever incremental is one of these modes. It's recommended to have this option enabled in order to remove blocks that are not necessary (obsolete VMs, etc), defragment the file, compact it that might result in disk space savings, however, this operation is resources consuming: it requires a quite a bit of time to complete and a lot of space on the storage (~+FullBackupSize space) is required for successful processing.

The purpose of the health check is to ensure there is no corruption of the backup files. More information can be found in our documentation by the links above.

Thanks
Post Reply

Who is online

Users browsing this forum: No registered users and 99 guests