Backing up AppStacks

[rhys.hammond]

Does anyone have any experience with backing up Appstacks using Veeam?
We are looking at utilising the Free edition of Veeam to copy the VMDKs from the vSAN datastore to Windows VM where the existing legacy backup solution can see the files and back them up.

I will need to work out a plan on how to handle the retention of the VMDKs, I am thinking some kind of powershell script to delete files in the folder older then n days.

Couple things I have noted;

File Copy jobs copy regular files between any servers managed by Veeam (Windows or Linux servers, or hypervisor hosts), and can be used for various administrative tasks.
File Copy jobs do not support processing of virtual disk files belonging to running VMs. (Gostev FAQ v7)
File Copy Jobs cannot copy to mapped drives because these are user mode settings, and file copy jobs do not provide way to type in the UNC path manually.

The Appstacks belong to multiple desktop view VMs attached as read-only.
Veeam will copy the thin-provisioned appstack vmdk and save it as a thick provisioned vmdk

VMware recommended backup method is to ensure that your third-party solution is targeting the storage where the VMDK files and supporting files exist.
Recently a VMware fling was released but it looks like this cannot be scheduled whereas the Veeam approach can be scheduled.
https://labs.vmware.com/flings/app-volu ... up-utility

Hoping to see some other feedback and possibly experiences when backing up Appstacks.

[rhys.hammond]

I've run up Veeam, configured a 'File Copy Job' using the appstack (appvolumes) VMDK, its stored on a vSAN datastore and thin provisioned.
The job completes with a successful status but the .VMDK files are 1KB in size. The VMDK.metadata is the correct size (varies from 20KB to 512KB).

Just tried to copy a VMDK from a powered off 'normal' VM and seeing the same issue, the .vmx , vmware.log ,etc file sizes are all normal but the .vmdk file is 1KB when it should be 10GB +.

Opened up a support case, 01789124.
Will post an update when I troubleshoot more.

[rhys.hammond]

Created a test VM with the following;

Test VM

SCSI 0:0 (Thick Provisioned Lazy Zero on vSAN)
SCSI 0:1 (Thin Provisioned on vSAN)
SCSI 0:2 (Thick Provisioned Lazy Zero on normal datastore)
SCSI 0:3 (Thin Provisioned on normal datastore)

Only the VMDKs stored on the normal datastore were entirely copied across, both VMDKs stored on the vSAN were 1KB regardless if thick or thin provisioned.
Looks like its a vSAN issue.

I have connected via winSCP, browsed to the vSAN datastore, examined the .VMDK files which all appear as 512bytes in size.
Created a new vhdd as thick provisioned eager zeroed. Refreshed winSCP and the new .vmdk appeared but still as 512 bytes in size.

All the .VMDK files appear as the correct file size when using the vsphere client datastore browser.

Very strange.

[rhys.hammond]

ok, so I believe this should definitely work but there is some kind of issue with the vSAN.
Looking at the below VMware thread, others have come across the same challenge around protecting Appstack/Appvolumes.
They have utilised Veeam as well, as expected the thin provisioned vmdks are copied as thick provisioned.

https://communities.vmware.com/thread/5 ... 0&tstart=0

I currently have a support ticket open with VMware to troubleshoot the issue with the vSAN, they believe SCP from vSAN will not work so that might rule out 'File Copy Jobs' with the vSAN as a source.

I am investigating the 'Storage Group' capability of the Appvolume manager which can automatically replication appstacks between datastores.
Hoping to replicate to a normal datastore which Veeam can then perform 'file copy' from.

Fingers Crossed.

[rhys.hammond]

Quick Update.

To copy Appstack VMDKs I was hoping to just perform a Veeam File Copy Job ( SCP ) of the header file and the flat file vmdk to a folder that could then be protected by the legacy backup solution. However as I have recently discovered, since I am using a vSAN datastore which does not use a flat-vmdk file this is proving a tad more difficult than I expected. VSAN storage employs object-level storage, which is different from traditional block-level storage. The VSAN objects are managed through a storage policy which, for example, can allow for greater redundancy for some virtual machines over others.

Because the reference in the VMDK file points to a VSAN DOM object, it cannot be copied through traditional means (SCP).

To work around this issue you will need traditional block-level storage which acts as a “middle man” to allow the SCP copy of VMDK files between environments.
Once on traditional storage you will be able to copy (SCP) the two VMDK files to wherever your need them to go. In this case, just to the local Veeam server is fine. Once its there the legacy backup software will protect the VMDK files.

source: https://blogs.vmware.com/consulting/201 ... orage.html

This leaves me with two options,

1) utilise vmkfstools to clone the VMDK to block level storage, I am not sure how easy this would be to script and automate. PowerCLI would be straight forward but I am dealing with vCLI here. Though I believe its possible in ESXi 6.0 to use vmkfstools with powercli now. I haven't had much time to research this option.

2) if dealing with Appstacks, utilise the Appvolume manager to replicate the VMDKs between other datastores (ensure one of the datastores is not a vSAN datastore)
Once the VMDKs are mirrored to the block level storage datastore (which I believe occurs every hour), Veeam should then be able to 'File Copy' these VMDKs / flat.VMDK files.
source: https://vdelboysview.com/2015/11/30/vmw ... with-2-10/


Now its time to test my theories.

[rhys.hammond]

The last challenge we encountered was how to perform SCP operations from a vSAN datastore, in the end, we used the ‘Storage Group’ capability of the AppVolume Manager to replicate AppStacks to another datastore that wasn’t a vSAN then Veeam performed a ‘File Copy’ which copied the VMDKs to a safe location protected by an existing backup solution. Now we only had the one datastore that was a vSAN so we configured an ‘OpenFiler’ VM to provide an iSCSI LUN to host another datastore.

Since our only datastore was a vSAN datastore, we configured an ‘OpenFiler’ VM to provide an iSCSI LUN to run up another datastore. Technically it’s a nested datastore but we only care about enabling access to the AppStacks VMDKs so we can copy them.

[rtheseeker]

I tried diong a file copy and we are using a trditional VMWARE infrastructure (Datastores, ESX hosts, VCENTER) not VSAN; however the file copy is still erroring out when copying these vmdks saying these are loked by VMS. Also, when a file copy runs; would it think provision all the vmdks and if the job fails(it showed around a TB of data that was copied) would it continue to utilize the space on the repository? This is a linux immutable storage

tg:datastore-112@appvolumes/packages/AppVol_Office_2016_R6-flat.vmdk' Error: DiskLib error: [13].The file is locked or in use -- File open failed: File not open

[rtheseeker]

At the moment, we managed to copy the vmdk files to Veeam Linux repository using file copy; however, the business is looking for a backup. would it be a good idea to create an SMB share; add it as a repo to Veeam; copy the files there and then run a backup of that share as well.