Source backup file has different block size

[KimmoJ]

Ok, attempt 15 to write this question without sounding too whiny follows (I'm getting pretty frustrated):

I don't get it. The solutions suggested in this thread don't seem to work for me, probably because I don't understand how to apply them.

I have a number of daily backup jobs, that have admittedly had a mix of LAN target and direct target backup storage optimization. This happened due to some confusion as to which was more appropriate for locally mounted remote iSCSI storage when initially setting up the jobs.

What I'm trying to do is create one backup copy job by taking those existing backup jobs and selecting the jobs and adding them to it, in order to achieve a GFS situation where I get weekly, monthly etc restore points on a second data store.

Only a few machines actually back up, the rest remain pending and complain about block size. This happens even if I try to create an entirely new backup copy job with no stored backup copy data at all in the backup copy store.

Do I need to create one backup copy job per VM? Or one backup copy job per backup job? Is that hugely wasteful for some reason or just overly complex? Or will I have to just erase all my backups and recreate them with local storage optimization?

If the solution involves messily sacrificing a chicken on the keyboard while chanting an appeal to Satan, I can do that too.

[KimmoJ]

Doesn't seem to be a way to edit posts on here? Weird, I can edit this one but not the previous one... anyway:

To add to my post, I tried the four powershell commands by running them against the Backup Copy Job I created, as well as a separate backup copy job for just a single backup job, but this didn't change the error message: $Options.BackupStorageOptions.StgBlockSize = "KbBlockSize1024"

I even tried to change it to KbBlockSize512 in case that would somehow magically get them in sync but it had no effect whatsoever I could detect.

I can find no way to change anything about backup storage optimization in a backup copy job. There simply is no such choice even in the advanced settings. It may exist in a Replication job, but that's not what I'm trying to use.

[veremin]

Kimmo, since none of the proposed workarounds worked for you, have you opened a ticket with our support team? Thanks.

[haslund]

Hi,

I ran into this "warning" today:


The issue for me here is that this says Warning/Completed with warnings, when in fact nothing was Replicated at all.
I feel this warning should be changed to an error instead so the job fails.

Background:
Customer initially only implemented two replication jobs: One hourly for important VMs and one daily job for the remaining VMs.
After running only replications for some time customer finally installed new backup storage and backup jobs were setup.
The Replication Daily source was now changed from Production to Backup Repository.

I can see the Backup job uses "Local" storage optimizations, ie. 1 MB block size and the Replication Daily (initially from production) used the default storage optimizations "LAN", ie 512 KB block size.

The fastest way I can figure out to resolve this would be:
Change replica source from backup to production.
Change Storage Optimizations from LAN to Local.
Replicate.
Change replica source from production to backup.

Are there any problems handling the situation like this?

Thank you for your time.

Support case ID: 00804200

[veremin]

Are there any problems handling the situation like this?

Nope, as it's recommended approach.

[haslund]

Even though I made them match and ran replication again, it is still complaining about the mismatch.

Yes, the Backup Job has block size of 1024 KB, but I did also change the Replication job to Storage Optimization Local and performed a Replication run from Source production with success and then changed source back to Backup.

Here is logs of that Replication Job with source Production where the block size was 1024 KB as far as I can see: <StgBlockSize>KbBlockSize1024</StgBlockSize

Code: Select all

[16.02.2015 11:24:39] <01> Info     Starting job mode: 'Normal'
[16.02.2015 11:24:39] <01> Info     [Session] Id '6e4dfa6d-6c8a-4aa7-979e-574abfae19ca', State 'Working'
[16.02.2015 11:24:39] <01> Info     Custom command execution (pre script) disabled
[16.02.2015 11:24:40] <01> Info     [LeaseKeeper] Created, ttl 100sec
[16.02.2015 11:24:40] <01> Info     -------------------------------------------------------------------------------------
[16.02.2015 11:24:40] <01> Info     Job ID: [1f0ba0b1-f31a-46b6-92ad-861508d75226]
[16.02.2015 11:24:40] <01> Info     Job Name: [Replication Daily]
[16.02.2015 11:24:40] <01> Info     Job Mode: [Normal]
[16.02.2015 11:24:40] <01> Info     Job Type: [VDDK Replica]
[16.02.2015 11:24:40] <01> Info     Job Options: [<JobOptionsRoot><RunManually>False</RunManually><GroupSnapshotProcessing>True</GroupSnapshotProcessing><StgBlockSize>KbBlockSize1024</StgBlockSize><EnableDeduplication>True</EnableDeduplication><CompressionLevel>5</CompressionLevel><CheckRetention>True</CheckRetention><EnableDeletedVmDataRetention>False</EnableDeletedVmDataRetention><RetainDays>14</RetainDays><EnableIntegrityChecks>True</EnableIntegrityChecks><VMToolsQuiesce>False</VMToolsQuiesce><UseChangeTracking>True</UseChangeTracking><EnableChangeTracking>True</EnableChangeTracking><ExcludeSwapFile>True</ExcludeSwapFile><EmailNotification>False</EmailNotification><EmailNotificationAddresses /><UseCustomEmailNotificationOptions>False</UseCustomEmailNotificationOptions><EmailNotificationSubject>[%JobResult%] %JobName% (%VmCount% VMs) %Issues%</EmailNotificationSubject><EmailNotifyOnSuccess>True</EmailNotifyOnSuccess><EmailNotifyOnWarning>True</EmailNotifyOnWarning><EmailNotifyOnError>True</EmailNotifyOnError><EmailNotifyOnLastRetryOnly>True</EmailNotifyOnLastRetryOnly><SnmpNotification>False</SnmpNotification><UseSanSnapshots>True</UseSanSnapshots><FailoverFromSan>True</FailoverFromSan><PostJobCommand><Periodicity>Cycles</Periodicity><PreScriptEnabled>False</PreScriptEnabled><PreScriptCommandLine /><Frequency>1</Frequency><Enabled>False</Enabled><CommandLine /><Days><Day>Saturday</Day></Days></PostJobCommand><RetainCycles>2</RetainCycles><SourceProxyAutoDetect>False</SourceProxyAutoDetect><TargetProxyAutoDetect>False</TargetProxyAutoDetect><UseWan>False</UseWan><Backup2Vi>False</Backup2Vi><Templates>True</Templates><TemplatesOnce>True</TemplatesOnce><ClusterReference>domain-c7</ClusterReference><ClusterName>AET VM Backup</ClusterName><DatastoreName>EVA4400_Datastore_02</DatastoreName><DatastoreReference>datastore-85</DatastoreReference><HDTargetType>Datastore</HDTargetType><ReplicaTargetResourcePoolRef>resgroup-8</ReplicaTargetResourcePoolRef><ReplicaTargetVmFolderRef>group-v71</ReplicaTargetVmFolderRef><ReplicaNameSuffix>_replica</ReplicaNameSuffix><EnableInitialPass>False</EnableInitialPass><InitialSeeding>True</InitialSeeding><NetworkMapping>False</NetworkMapping><UseReIP>False</UseReIP><UseVmMapping>True</UseVmMapping><ReIPRules /><FullBackupDays /><FullBackupMonthlyScheduleOptions /><TransformToSyntethicDays /></JobOptionsRoot>]

I then changed the Replication source from Production back to Backup Repository and re-ran the job again: <StgBlockSize>KbBlockSize1024</StgBlockSize>

Code: Select all

[17.02.2015 03:00:08] <01> Info     Starting job mode: 'Normal'
[17.02.2015 03:00:08] <01> Info     [Session] Id 'a6d1a6fd-82eb-4b0c-a50d-ee25b7887b32', State 'Working'
[17.02.2015 03:00:08] <01> Info     Custom command execution (pre script) disabled
[17.02.2015 03:00:09] <01> Info     -------------------------------------------------------------------------------------
[17.02.2015 03:00:09] <01> Info     Job ID: [1f0ba0b1-f31a-46b6-92ad-861508d75226]
[17.02.2015 03:00:09] <01> Info     Job Name: [Replication Daily]
[17.02.2015 03:00:09] <01> Info     Job Mode: [Normal]
[17.02.2015 03:00:09] <01> Info     Job Type: [VDDK Replica]
[17.02.2015 03:00:09] <01> Info     Job Options: [<JobOptionsRoot><RunManually>False</RunManually><GroupSnapshotProcessing>True</GroupSnapshotProcessing><StgBlockSize>KbBlockSize1024</StgBlockSize><EnableDeduplication>True</EnableDeduplication><CompressionLevel>5</CompressionLevel><CheckRetention>True</CheckRetention><EnableDeletedVmDataRetention>False</EnableDeletedVmDataRetention><RetainDays>14</RetainDays><EnableIntegrityChecks>True</EnableIntegrityChecks><VMToolsQuiesce>False</VMToolsQuiesce><UseChangeTracking>True</UseChangeTracking><EnableChangeTracking>True</EnableChangeTracking><ExcludeSwapFile>True</ExcludeSwapFile><EmailNotification>True</EmailNotification><EmailNotificationAddresses>aetit@aet-biomass.com</EmailNotificationAddresses><UseCustomEmailNotificationOptions>True</UseCustomEmailNotificationOptions><EmailNotificationSubject>[%JobResult%] %JobName% (%VmCount% VMs) %Issues%</EmailNotificationSubject><EmailNotifyOnSuccess>True</EmailNotifyOnSuccess><EmailNotifyOnWarning>True</EmailNotifyOnWarning><EmailNotifyOnError>True</EmailNotifyOnError><EmailNotifyOnLastRetryOnly>True</EmailNotifyOnLastRetryOnly><SnmpNotification>False</SnmpNotification><UseSanSnapshots>True</UseSanSnapshots><FailoverFromSan>True</FailoverFromSan><PostJobCommand><Periodicity>Cycles</Periodicity><PreScriptEnabled>False</PreScriptEnabled><PreScriptCommandLine /><Frequency>1</Frequency><Enabled>False</Enabled><CommandLine /><Days><Day>Saturday</Day></Days></PostJobCommand><RetainCycles>2</RetainCycles><SourceProxyAutoDetect>False</SourceProxyAutoDetect><TargetProxyAutoDetect>False</TargetProxyAutoDetect><UseWan>False</UseWan><Backup2Vi>True</Backup2Vi><Templates>True</Templates><TemplatesOnce>True</TemplatesOnce><ClusterReference>domain-c7</ClusterReference><ClusterName>AET VM Backup</ClusterName><DatastoreName>EVA4400_Datastore_02</DatastoreName><DatastoreReference>datastore-85</DatastoreReference><HDTargetType>Datastore</HDTargetType><ReplicaTargetResourcePoolRef>resgroup-8</ReplicaTargetResourcePoolRef><ReplicaTargetVmFolderRef>group-v71</ReplicaTargetVmFolderRef><ReplicaNameSuffix>_replica</ReplicaNameSuffix><EnableInitialPass>False</EnableInitialPass><InitialSeeding>True</InitialSeeding><NetworkMapping>False</NetworkMapping><UseReIP>False</UseReIP><UseVmMapping>True</UseVmMapping><ReIPRules /><FullBackupDays /><FullBackupMonthlyScheduleOptions /><TransformToSyntethicDays /></JobOptionsRoot>]

However it still says VM is in storage with block size 'KbBlockSize1024'. Expecting storage with block size 'KbBlockSize512'

Code: Select all

[17.02.2015 03:01:26] <01> Info     [VbReplicaTaskBuilder] Trying to find source Oib(s) for object 'eb635cfd-a57c-458a-862f-e8b56b603e38' named 'vcenter-bck'
[17.02.2015 03:01:27] <01> Info     [OibFinder] Following Oibs are located in storages with different block sizes and excluded from search:
[17.02.2015 03:01:27] <01> Info         [OibFinder] Oib [vmname=vcenter-bck:creation_time=16-02-2015 19:35:33:alg=Increment:id=6c3c6006-2372-42cd-869f-7a49c83f2af1:point_id=1de07ed3-7c51-40fe-bba3-2738da37d956:storage_id=ab49d9ff-fe32-4043-97a8-e1151f37fd9a:parent_id=00000000-0000-0000-0000-000000000000:link_id=14289642-1a5f-4236-bcf5-c79132b7de28] is in storage with block size 'KbBlockSize1024'. Expecting storage with block size 'KbBlockSize512'
[17.02.2015 03:01:28] <01> Info     Task session 'ee071f67-99c5-4e27-8169-60bcefe45f52' has been completed, status: 'Warning', '0' of '0' bytes, '0' of '0' objects, details: ''
[17.02.2015 03:01:28] <01> Error    [VbReplicaTaskBuilder] Failed to find restore point for entry 'vcenter-bck'.  Restore point is located in backup file with a different block size (512KB expected)

Why is it saying it is expecting 512KB block size?
How do I resolve this?

Thank you.

[haslund]

FYI, just got response back from support it appears this is a bug in v8 that will be resolved in patch 2.
The current workaround, even though it makes no sense, is to change the replication block size to 256 KB.

I have made this change now and will post back when I know if it has resolved the issue.

I still feel that it is a big problem the GUI marks this as a Warning when in fact nothing is being replicated - it should be marked as Error.
Same point made I notice by another customer in http://forums.veeam.com/veeam-backup-re ... ml#p129939

[haslund]

It works now that the Backup Job is set to 1 MB block size and the Replication Job is set to 256 KB block size.
Looking forward to having this bug fixed in patch 2.

[veremin]

Speaking about notification, we do consider changing warning to error in one of the next product versions. So, thanks for the heads up.

[kjstech]

I had the same problem, because the block size autodetection for copy jobs is faulty. All my source backup jobs had blocksize 1024 (LAN target optimization), but some of them had older restorepoints that were made with blocksize 512 (Default/Local target), so the autodetection assumed blocksize 512 and copy jobs failed.

To fix it I disabled the copy job, removed the jobs backup files from disk, went into PowerShell and did the following:

Code: Select all

$Job = Get-VBRJob -name "USB Backup Copy"
$Options = $Job.GetOptions()
$Options.BackupStorageOptions.StgBlockSize = "KbBlockSize1024"
$job.SetOptions($Options)

After that I re-enabled the copy job and on the second run it started working.

This would be much easier to fix if the copy job options allowed to manually set the storage optimization.

I get this:
Get-VBRJob : The term 'GetVBRJob' is not recognized as the name of a cmdlet, function, script file, or operable program.

How do you run this, I need to get this completed.

[lightsout]

To use get-vbrjob, you mean launch PowerShell from the Veeam interface. Or in PowerShell, add the Veeam plugin:

Code: Select all

add-pssnapin VeeamPSSnapin

Here is the code I use for doing this. This change all Backup Copy jobs:

Code: Select all

$jobs=get-vbrjob | ? {$_.JobType -eq "BackupSync"}
foreach ($job in $jobs) {
  $job | Set-VBRJobAdvancedStorageOptions -EnableDeduplication $true -StorageBlockSize "KbBlockSize512" | out-null
}

This also turns deduplication on. This sets the block size to LAN target.

[kjstech]

To use get-vbrjob, you mean launch PowerShell from the Veeam interface. Or in PowerShell, add the Veeam plugin:

Code: Select all

add-pssnapin VeeamPSSnapin

Here is the code I use for doing this. This change all Backup Copy jobs:

Code: Select all

$jobs=get-vbrjob | ? {$_.JobType -eq "BackupSync"}
foreach ($job in $jobs) {
  $job | Set-VBRJobAdvancedStorageOptions -EnableDeduplication $true -StorageBlockSize "KbBlockSize512" | out-null
}

This also turns deduplication on. This sets the block size to LAN target.

No dice. I get this

Code: Select all

add-pssnapin : No snap-ins have been registered for Windows PowerShell version 4.
At line:1 char:1
+ add-pssnapin VeeamPSSnapin
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (VeeamPSSnapin:String) [Add-PSSnapin], PSArgumentException
    + FullyQualifiedErrorId : AddPSSnapInRead,Microsoft.PowerShell.Commands.AddPSSnapinCommand

We can take this discussion to my thread on Powershell here.
http://forums.veeam.com/powershell-f26/ ... 26429.html

[kjstech]

Ok got Powershell to work thanks to user in the other thread (Have to launch it FROM the Veeam B&R Application window the first time it is ever used).

Issue is, this 'fix' does not work. I used powershell to change it from 1024 to 512.

Look, I'm even verifying it prior to writing.

Code: Select all

> write $Options.BackupStorageOptions


CheckRetention               : True
RetainCycles                 : 14
KeepFirstFullBackup          : False
RetainDays                   : 14
EnableDeletedVmDataRetention : False
CompressionLevel             : -1
EnableDeduplication          : True
StgBlockSize                 : KbBlockSize512
EnableIntegrityChecks        : True
EnableFullBackup             : False
BackupIsAttached             : False
StorageEncryptionEnabled     : False

Still getting this at the end of the log while the job is "waiting for restore points"

Code: Select all

[23.02.2015 14:32:22] <01> Info     [OibActionCalc] Source Oib not found (NoOibAction)
[23.02.2015 14:32:22] <01> Info     Source backup file has different block size. Expected block size: 1024, actual: ([backup name: vnxfs2, block size: 512]). VM: 'MailArchive', ID: 'fc9eecca-cf62-4f19-b135-bcef289fc53d' Ref: 'vm-753'
[23.02.2015 14:32:22] <01> Info     [JobSession] totalObjects '16', totalSize '3.5 TB'

If the actual block size is 512, shouldn't the StgBlockSize value be equal to KbBlockSize512 to match?

[kjstech]

Got it to work. In the left hand side I had to go under Backups, Disk and right click on the backup copy job and select remove from disk.

[chimera]

Got it to work. In the left hand side I had to go under Backups, Disk and right click on the backup copy job and select remove from disk.

This did it for me too after other changes suggested earlier as well. So basically, the steps that worked for me (as taken Obroni and others above) and in layman's terms... as there are a few questions around where to configure this stuff...

1. Edit the replication job and change to the source VM (Edit job, Virtual Machine/Source button, "From production storage" as opposed to "From backup files")

2. Change block size (optimization) to match backup job (Edit job, Storage/Advanced button, Storage tab, Storage optimization setting)
(see here http://helpcenter.veeam.com/backup/80/v ... ge_vm.html for more information on the different block sizes)

3. Run replication job and wait for it to recalc digests (this can take a LONG time for large VMDK's)

4. Set Replication job back to source of backup job (Edit job, Virtual Machine/Source button, "From backup files")

5. If you get the "Restore point is located in backup file with a different block size.." warning again, then go to Replicas, right click and "Remove from disk"

6. Run the replication job again

In my case, I'm backing up to an EMC Datadomain which it recommends "Local target (16TB+ backup files)", therefore 8MB block size. Original job used defaults at 512KB hence why I was getting this issue. I'm quite impressed with the DataDomain to be honest, backup is insanely fast at 150MB/sec+ for synthetic fulls which is basically maxing out a 1Gbps connection. Going to LAG the NIC's soon see if I can crank this beast up to run faster. All dedup done on the appliance so no processing needed on Veeam server.

Would be good if the replication job could auto-detect the optimization settings if you're choosing to replicate from a backup source, as this should be obviously known to Veeam.

Cheers

[NexusNeil]

Hi All, I am experiencing an issue with 1 of our backup and replication jobs

Since moving the local backup destination and editing the off site replicaton job to suit

The replication job runs but with error's "Restore point is located in backup file with a different block size (1024KB expected)"

I am aware this is down to the compression level and storage optimization

My questions are:

Do I need to re-run a complete full backup then the replication job for the block sizes to sync again ?

Or is there a work round?

Keeping i mind I have changed the compression in the backup and replication jobs already, still got the error which points to the storage optimization which I can not alter in the replication job to suit

Cheers

[foggy]

Neil, the workaround to this issue is described on the first page of this thread, please review. Thanks.

[lmbarrett83]

Hi

I have been having the same issues since changing the replication source to the backup repository (Veeam 8 patch 1)

I have an ongoing case open with support (00870735) , but so far I have had to point the 3 agents that I have spoken to towards this forum post.
Unfortuantely I have also had to point out that the option to change the blocksize through the gui vanishes once you select backups as the source.

I have tried the following:
- Removing the replicas from the configuration database.
- Setting the blocksize in the backup job to LAN (512kb)
- Recreating the replication Job and setting the blocksize to (512kb) via the shell
- Running a active full on the backup job
- Running the replication job

After completing these steps i get a successful run.

On the next incremental I get the following warning: Restore point located in a backup file with a different block size (512kb expected).

The strange thing is that not all VMs in the job have the issue. it is currently 2 out of 3.

I've read that other customers with this error have selected "remove from disk" as opposed to "remove from replicas". Will this remove the seeded VMs on my target?

With this being WAN replication i'm trying to avoid removing the seeds as they account for Terabytes of data.

One last thing of note, there is a replication metadata folder in the backup repository could this have anything to do with my issue. does that need clearing manually?

Thanks

[foggy]

Have you tried the following steps to change the block size in the replica job?

1. Set the replication job back to source vm
2. Change Block size (optimization) to match backup job
3. Run replication job and wait for it to recalc digests
4. Finally set Replication job back to source of back up job

[lmbarrett83]

Hi foggy,

No I was using the following to avoid snapshotting the production CSV.

You can also try to change block size without changing the source type of replication job, using the following commandlet; might do the trick:

Code: Select all

$Job = Get-VBRJob -name "Backup Job 1"
$Options = $Job.GetOptions()
$Options.BackupStorageOptions.StgBlockSize = "New value"
$Job.SetOptions($Options)

Thanks.

I'll will report back once the job has completed. It is going to take a while.

Thanks

[lmbarrett83]

Hi Foggy

I tried the following

Have you tried the following steps to change the block size in the replica job?

After the source was changed to "actual VM state" from production the replication completed (after 13 hours). That said after changing the source back to "Backups" I got the same warning "Restore point is loacted in backup file with a different block size (512kb expected)"

Is there a config file that i cant edit to stop the check? This feature is not working as designed in our environments.

[foggy]

Lewis, this check cannot be disabled. I would recommend to ask your support engineer to take a closer look at what is happening. Thanks.

[lmbarrett83]

Foggy,

I did as you suggested and have contacted support. I spent a couple of hours with a support engineer and he was able to confirm issue. After some additional checks he was able to provide me with a private hotfix.

First impressions look good so far, but time we tell.

Thanks

[lmbarrett83]

Hi

Just a quick update.

Since the private fix provided by Veeam support was applied Replication (full) and Replication (incrementals) have been successful.

This has been across two jobs with multiple VMs.

The block size check is usually performed at the start of the job, so it's immediately apparent when it's not going to work.
I have even been able to use the powershell commands to change the block size of the replication job before a subsequent incremental run and this too has been successful.

The block size specified in the Replication job can now be set independently it seems and doesn't have to match, that of the backup.

Thanks

[lightsout]

That is an interesting hotfix, as I've got the same issue as you describe with the mismatched block sizes.

Is that hotfix part of Update 2? Or do I need to submit a case to get it?

[foggy]

Yep, right, this fix is integrated in Veeam B&R Update 2 and actually allows to change the block size in replication job independently. In case of block size mismatch, digests are re-calculated during the next job run and replication continues normally.

[ramnefors]

I have created a new Backup Copy Job for an existing full backup. The backup copy job always fails with:

2015-05-04 00:01:40 :: Source backup file has different block size. Expected block size: 512, actual: ([backup name: Full backup, block size: 1024])

B&R is v8 build 2021. I am surprised B&R does not pick up the block size from the existing job when a new copy job is created.

I have understood optimization can affect the block size. Full backup is set for storage > advanced > LAN optimization.

Copy job is based off "Full backup" job. There are no optimization available in the setup for the copy job (standard license).

Any suggestions?

[foggy]

Mats, is this a completely new backup copy job that has VMs from a single backup job as a source?

[ramnefors]

Correct.

The underlying "Full backup" is old, has been running under v7 previously.

The "Copy job" is new. I have also deleted and re-created it a couple of times with the same effect.

The "Copy job" has VM: "Full backup" (backup job) as objects to process.

[foggy]

Then I suggest to ask technical support to review your configuration closer. Chances are, some block size confusion takes place, for example due to storage optimization settings were changed on the original job. Btw, is it a single full or a backup chain with increments?