VMware VSAN 6.0

[omaresp]

Hello world!

One caveat of Veeam Backup and Replication is that every time a job runs, a VMware snapshot happens, even when using Direct SAN Access, only that snapshot lasts 1-5 minutes depending on the size of the VM and the amount of data it is processing.

We currently use Veeam Backup as a daily job, along with a backup copy, which has very minimal effect to the production environment as long as you plan for network failures to happen when the backup starts everyday.

However, we also replicate production servers to an Off-site location for DR purposes. Keeping in mind all servers replicated are of critical importance and require have a recovery timeframe of around 4 hours or less. Meaning if a production server is lost, we can recover the same server 4 hours back at the Off-site location.

As you are all aware, VMware Snapshots tend to stun the VM for a couple of seconds, causing a network drop several times while creating and removing the snapshot. A 2-3 seconds drops might not be an issue, but when you have replication of a server happening say every 4 hours, you will have several 2-3 seconds drops every 4 hours. These drops can have severe implications on production environments.

I understand this is not something Veeam can fix or workaround, since it is a VMware snapshot, the ball is on VMware.

Looking at the new release of VMware VSAN 6.0, they mention a new snapshot technology.

Has anyone tested or is currently working with this environment? Are VMware snapshots still prone to VM stuns while using VMware VSAN 6.0?

[Vitaliy S.]

Hi Omar,

Can't comment on VMware VSAN 6.0 usage, but if you can use backups from storage snapshot feature of Veeam B&R this VM stun should not be observed.

Thanks!

[omaresp]

you can use backups from storage snapshot feature of Veeam B&R this VM stun should not be observed.

I'm not sure what feature this is, how can I configure a job to replicate a VM to an off-site location only using SAN level snapshot?

[foggy]

Omar, what kind of primary storage do you have? Provided it is one of the supported storages, you can use storage snapshots for VM data processing, reducing impact on production VMs.

[omaresp]

Omar, what kind of primary storage do you have?

HP SAN

you can use storage snapshots for VM data processing, reducing impact on production VMs.

We currently use storage snapshots, but as I explained on my first post, there is still a VMware snapshot, which causes VM stuns.

[Vitaliy S.]

This stun usually happens when VM snapshot grows large. When using SAN snapshots VM snapshots do not grow as they would grow during normal backup job run. Can you confirm that SAN snapshots technology is used by your jobs?

[omaresp]

They are:

Code: Select all

***2015-06-19 04:02:06Creating VM snapshot***
2015-06-19 04:03:33Collecting disk files location data
***2015-06-19 04:06:05Removing VM snapshot***
2015-06-19 04:23:15Queued for processing at 6/19/2015 4:23:15 AM
2015-06-19 04:29:44Required backup infrastructure resources have been assigned
2015-06-19 04:29:47VM processing started at 6/19/2015 4:29:47 AM
2015-06-19 04:29:47VM size: 1.3 TB (760.1 GB used)
2015-06-19 04:29:49Discovering replica VM
2015-06-19 04:29:50Network traffic will be encrypted
2015-06-19 04:30:05Preparing replica VM
2015-06-19 04:30:21Processing configuration
2015-06-19 04:30:30Creating helper snapshot
2015-06-19 04:30:35Using backup proxy VMware Backup Proxy for retrieving Hard disk 1 data from storage snapshot
2015-06-19 04:30:36Using target proxy SERVER1 for disk Hard disk 1 [nbd]
2015-06-19 04:38:03Network traffic throttling is enabled
2015-06-19 04:54:58Hard disk 1 (60.0 GB)3.4 GB read at 2 MB/s [CBT]
2015-06-19 04:36:31Using backup proxy VMware Backup Proxy for retrieving Hard disk 2 data from storage snapshot
2015-06-19 04:36:47Using target proxy SERVER1 for disk Hard disk 2 [nbd]
2015-06-19 04:37:39Hard disk 2 (220.0 GB)34.8 MB read at 1 MB/s [CBT]
2015-06-19 04:37:52Using backup proxy VMware Backup Proxy for retrieving Hard disk 3 data from storage snapshot
2015-06-19 04:37:52Using target proxy SERVER1 for disk Hard disk 3 [nbd]
2015-06-19 04:39:28Hard disk 3 (1.0 TB)103.5 MB read at 2 MB/s [CBT]
2015-06-19 04:55:19Deleting helper snapshot
2015-06-19 04:55:29Finalizing
2015-06-19 04:55:33Busy: Source 20% > Proxy 20% > Network 97% > Target 2%
2015-06-19 04:55:33Primary bottleneck: Throttling
2015-06-19 04:55:33Network traffic verification detected no corrupted blocks
2015-06-19 04:55:33Processing finished at 6/19/2015 4:55:33 AM

6/19/2015 5:32:58 AMSuccessJob finished at 6/19/2015 5:32:58 AM
6/19/2015 5:32:58 AMSuccessPrimary bottleneck: Throttling
6/19/2015 5:32:58 AMSuccessLoad: Source 31% > Proxy 26% > Network 74% > Target 8%
***6/19/2015 5:32:58 AMSuccessDeleting storage snapshot***
6/19/2015 5:29:54 AMSuccess1 restore point removed by retention policy from VM SERVER2_replica
6/19/2015 5:10:58 AMSuccessProcessing SERVER2
6/19/2015 4:29:47 AMSuccessAll VMs have been queued for processing
6/19/2015 4:29:47 AMSuccessPreparing next VM for processing
6/19/2015 4:27:36 AMSuccessPreparing next VM for processing
6/19/2015 4:23:21 AMSuccessPreparing next VM for processing
6/19/2015 4:23:21 AMSuccessPreparing next VM for processing
***6/19/2015 4:04:25 AMSuccessCreating storage snapshot***
6/19/2015 4:01:16 AMSuccessPreparing next VM for processing
6/19/2015 4:01:16 AMSuccessRequired backup infrastructure resources have been assigned
6/19/2015 4:01:16 AMSuccessQueued for processing at 6/19/2015 4:01:16 AM
6/19/2015 4:01:14 AMSuccessChanged block tracking is enabled
6/19/2015 4:01:14 AMSuccessVM size: 3.1 TB (2.5 TB used)
6/19/2015 4:01:00 AMSuccessBuilding VM list
6/19/2015 4:00:08 AMSuccessJob started at 6/19/2015 4:00:05 AM

[Gostev]

I would say you need to take a closer look at potential production storage performance issues here. Because under normal circumstances, committing such a short lived VM snapshot should not have any noticeable impact on the respective VM. For example, your storage may simply be under-provisioned in terms of IOPS, while backup activity going in parallel with said snapshot removal will add even more heavy I/O load on already overloaded storage. By the way, if this is indeed the case, then enabling our Backup I/O Control feature may help here. Thanks!