Differences in backup processing

[gingerdazza]

Hi.

I have a single B&R server attached to 4GB FC, and 10GbE
I have two VMware clusters:
VMware cluster 1 is traditional server + SAN
VMware cluster 2 is NFS based

When I backup VMs on cluster 1 I notice the jobs are only processing the used data - i.e. if a VM has 40GB HDD but only 16GB used, the job will state
"21/03/2016 09:18:34 :: Hard disk 1 (40.0 GB) 16.3 GB read at 40 MB/s [CBT]"

However, on the NFS cluster it processes ALL the data disk:
"06/04/2016 10:47:32 :: Hard disk 1 (40.0 GB) 38.0 GB read at 143 MB/s [CBT]"

Anyone got ideas about why the latter is processing white space on the drives?

[PTide]

Hi,

However, on the NFS cluster it processes ALL the data disk:
"06/04/2016 10:47:32 :: Hard disk 1 (40.0 GB) 38.0 GB read at 143 MB/s [CBT]"

Does that happen during Full backup? Also do those VMs run from thin disks?

Thank you.

[gingerdazza]

Yes, VMs are thin.
This is Active Full Backup.
I do have a few VMs on the NFS platform seemingly backing up 'thinly' as desired.
This suggests to me it's a CBT issue.
I will try re-running the CBT reset process and see what happens.

[gingerdazza]

Update...
When I do an incremental backup straight after a full it runs through very quickly suggesting CBT is at least mostly functional.
When I then afterwards re-run an Active Full it again processes/reads the entire disk instead of the thin data.
.......continuing........

[gingerdazza]

I'm at a loss. I ran the CBT Reset tool on a few VMs and tried again, but it's still not backing them up on a full backup 'thinly'. I see this error in the logs:

FYI
The new problematic cluster is running ESXi 6.0 Update 2 build 3620759

Code: Select all

Initializing vSphere disk changes tracker. Tracker parameters: [SOAP connection: our_vcenter_name;VM (mor): vm-60299;Disk key: 2000;Snapshot (mor): snapshot-60329;Change ID: *.].
[15.04.2016 08:16:07] < 14936> vim|     Initializing VI SOAP connection...
[15.04.2016 08:16:07] < 14936> vim|       Specification of VI connection
[15.04.2016 08:16:07] < 14936> vim|       {
[15.04.2016 08:16:07] < 14936> vim|         Connection ID: [our_vcenter_name].
[15.04.2016 08:16:07] < 14936> vim|         Host: [our_vcenter_name].
[15.04.2016 08:16:07] < 14936> vim|         Port: [443].
[15.04.2016 08:16:07] < 14936> vim|         Login: [domain\our_ad_account].
[15.04.2016 08:16:07] < 14936> vim|       }
[15.04.2016 08:16:07] < 14936> vim|       [SOAP] Successfully logged in ( server: [our_vcenter_name], user: [domain\our_ad_account], sessionKey: [520cf9bb-eb8b-783f-5cde-76f741e87bf3] )
[15.04.2016 08:16:07] < 14936> vim|     VI SOAP connection initialized.
[15.04.2016 08:16:07] < 14936> vim|     Operation 'RequestDiskChanges' failed: 'Soap fault. Error caused by file /vmfs/volumes/e84d2a36-03ef5a03/our_computer_name_NutBakTst/our_computer_name.vmdkDetail: '', endpoint: '''. Retrying...
[15.04.2016 08:16:10] < 14936> vim|     [SOAP] Logged out from the server [our_vcenter_name].
[15.04.2016 08:16:10] < 14936> vim|     [SOAP] Successfully logged in ( server: [our_vcenter_name], user: [domain\our_ad_account], sessionKey: [526addf9-b915-b135-0a8b-4a502584a76f] )
[15.04.2016 08:16:10] < 14936> vim|     Operation 'RequestDiskChanges' failed: 'Soap fault. Error caused by file /vmfs/volumes/e84d2a36-03ef5a03/our_computer_name_NutBakTst/our_computer_name.vmdkDetail: '', endpoint: '''. Retrying...
[15.04.2016 08:16:13] < 14936> vim|     [SOAP] Logged out from the server [our_vcenter_name].
[15.04.2016 08:16:13] < 14936> vim|     [SOAP] Successfully logged in ( server: [our_vcenter_name], user: [domain\our_ad_account], sessionKey: [5289c72b-e2f0-49ff-eb04-225b98f51738] )
[15.04.2016 08:16:13] < 14936> vim|     Failed to enumerate changed areas.
[15.04.2016 08:16:13] < 14936> dsk|   Initializing vSphere disk changes tracker. Tracker parameters: [SOAP connection: our_vcenter_name;VM (mor): vm-60299;Disk key: 2000;Snapshot (mor): snapshot-60329;Change ID: *.]. Failed.
[15.04.2016 08:16:13] < 14936> dsk| WARN|It is impossible to create disk changes tracker in accordance with the specified settings. Default tracker will be used.
[15.04.2016 08:16:13] < 14936> dsk| >>  |Soap fault. Error caused by file /vmfs/volumes/e84d2a36-03ef5a03/our_computer_name_NutBakTst/our_computer_name.vmdkDetail: '', endpoint: ''
[15.04.2016 08:16:13] < 14936> dsk| >>  |--tr:Failed to enumerate changed areas of the disk using CTK. Device key: [2000], size: [113816633344]. VM ref: [vm-60299]. Change ID: [*]
[15.04.2016 08:16:13] < 14936> dsk| >>  |--tr:Failed to initialize vSphere disk changes tracker. SOAP parameters: [Host: our_vcenter_name;Port: 443;Login: domain\our_ad_account;Thumbprint: BC:A3:DA:28:B2:D7:75:5C:CC:4F:77:54:3F:D5:99:50:18:6D:D1:7D.]. CTK parameters: [SOAP connection: our_vcenter_name;VM (mor): vm-60299;Disk key: 2000;Snapshot (mor): snapshot-60329;Change ID: *.].
[15.04.2016 08:16:13] < 14936> dsk|   Initializing swap CTK filter for disk 'VDDK:[our_datastore] our_computer_name_NutBakTst/our_computer_name.vmdk'. Std. block size: '1048576'.
[15.04.2016 08:16:13] < 14936> dsk| ERR |Failed to enumerate partition areas allocated for swap files. Partition ID: '0'. Disk: 'VDDK:[our_datastore] our_computer_name_NutBakTst/our_computer_name.vmdk'.
[15.04.2016 08:16:13] < 14936> dsk| >>  |Failed to find NTFS object [pagefile.sys].
[15.04.2016 08:16:13] < 14936> dsk| >>  |--tr:Failed to open NTFS object \pagefile.sys.

[PTide]

Yes, VMs are thin. This is Active Full Backup.

Then I suspect that's an expected behaviour due to VMware limitations that apply to CBT on thin disks on NFS storage thus resulting in full read during full backup.

Thank you.

[gingerdazza]

I suspected that, but I do have a few VMs that are backing up on the NFS fine .....................

[PTide]

Then please follow this KB in order to reset CBT for problematic VMs. If that does not help please contact our support team and post your case ID.

Thank you.

[gingerdazza]

Thanks. Have tried that KB, and the powershell script. Didn't seem to fix it. Have already logged a ticket. 1st level not sure what is going on. Escalated to QA. Call reference 01755478