Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

VMware specific discussions

Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby gfdos.sys » Wed Nov 01, 2017 3:59 pm

After the B&R 9.5U1 announcement https://www.veeam.com/kb2222
Virtual hardware version 13 support. vSphere 6.5 introduces the new VM hardware version which increases some configuration maximums and adds ability to add NVMe controllers to a VM. This update adds ability to process such VMs.

Using Veeam Backup 9.5U2 a windows vm with a virtual NVMe controler backups fine.
However if trying a Replication Job, the initial replication works, but subsequent Replication incrementals always fail.
I get virtual NVMe is a new feature in vSphere 6.5, but if it doesn't work with Backup *AND* Replication, why call attention to it as a feature that is supported in the 9.5U1 announcement?

Warning -- for now Backup for vms with virtual NVMe works but Replication does not.
(note: this is on vsphere 6.5U1 with the patch found here: https://kb.vmware.com/s/article/2151061?language=en_US
with Veeam Backup 9.5U2)
gfdos.sys
Novice
 
Posts: 6
Liked: 1 time
Joined: Wed Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby foggy » Wed Nov 01, 2017 4:28 pm

Hi Gabriel, do you have a support case ID for this issue?
foggy
Veeam Software
 
Posts: 15295
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby gfdos.sys » Wed Nov 01, 2017 5:24 pm 1 person likes this post

Case #02333485
Level 2 engineer looking at it. Apparently 2 other cases open right now.
In my case and one other case the "solution" was use SCSI not NVMe.
Waiting to hear back what they told the other case.

Figured I'd ask here and give the warning of what we are running into.

NVMe on top of HCI infrastructure is sweet, and gave us a nice performance boost. We are using DataCore VirtualSAN, so it is doing auto tiering of our SAS SSD and SAS SCSI, so why add extra "baggage" of scsi virtually?
Physical NVMe has a high cost, but if we already have the performance with HCI, it makes total sense. Only thing holding us back is the Replication needed for DR.

https://www.starwindsoftware.com/blog/i ... olutionary
gfdos.sys
Novice
 
Posts: 6
Liked: 1 time
Joined: Wed Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby foggy » Thu Nov 02, 2017 10:36 am 1 person likes this post

We were able to reproduce this internally, the issue is in failing Revert Snapshot operation. We will investigate and, if the issue is on Veeam B&R side, address it in one of the next updates (could be even the upcoming v9.5 Update 3).
foggy
Veeam Software
 
Posts: 15295
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby Gostev » Thu Nov 02, 2017 7:38 pm

Most likely some change in vSphere 6.5U1 or Veeam 9.5U2 is causing this... B&R 9.5U1 was indeed tested against VMs with NVMe controllers extensively, but it was done during plain vSphere 6.5 times. We will figure this out.
Gostev
Veeam Software
 
Posts: 21621
Liked: 2411 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby gfdos.sys » Fri Nov 03, 2017 2:30 pm

Veeam vmware on hpe hosts vmware on dell veeam host
7/20 9.5.0.823 6.5.0 5146846 6.5.0
9/22 NVMe Enabled
10/28 9.5.0.1038 6.5U1 5969303(Oct)+patch 6.5U1+patch

>Most likely some change in vSphere 6.5U1 or Veeam 9.5U2 is causing this...
Just for the record: We had the problem as soon as we enabled NVMe on vmware 6.5.0 5146846 on Veeam 9.5U1

>B&R 9.5U1 was indeed tested against VMs with NVMe controllers extensively
The "Full replica" works -- its any incremental replicas that fail.
I went back to my Veeam email logs and Replication failed on 9/22 --> at the first "replication incremental" after adding it.
"Creating helper snapshot Error: Detected an invalid snapshot configuration.
Error: Detected an invalid snapshot configuration."

the next day the error changes to:
"Preparing replica VM Error: Detected an invalid snapshot configuration.
Error: Detected an invalid snapshot configuration."

every attempt after that:
"Deleting helper snapshot Error: Unable to access file since it is locked
Error: Unable to access file since it is locked"

Backup works just fine all along.
gfdos.sys
Novice
 
Posts: 6
Liked: 1 time
Joined: Wed Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby gfdos.sys » Wed Nov 08, 2017 1:27 pm

I had 2 ideas for a workaround on this, What do you guys think?

Work around 1:
Rhetorical question -- Would this NVMe issue with replication happen with physical hardware?
In order to do that you would have to use the Veeam Agent to do the backup and replication....

Is that a possible workaround -- Installing the Veeam Agent on a vm with virtual NVMe?
After installing the Veeam Agent (I have never done backup that way) would I have to remove the vm from the backup/replica jobs and readd it?

Work around 2:
I presume this backup and replication job are using NBD mode.
All the VMs I'm backing up are stored on a Datacore "iSCSI" SAN. If I backed up the jobs in Direct SAN access mode would it avoid the issue of the "snapshots" being locked that the error message seems to suggest?
gfdos.sys
Novice
 
Posts: 6
Liked: 1 time
Joined: Wed Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby foggy » Wed Nov 08, 2017 4:01 pm

The issue is in reverting a snapshot of the replica VM, so the mentioned workarounds are not relevant.
foggy
Veeam Software
 
Posts: 15295
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby foggy » Thu Nov 09, 2017 10:18 am

Btw, as an update, it is currently being investigated with VMware and has just been escalated to T2 of their support.
foggy
Veeam Software
 
Posts: 15295
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby gfdos.sys » Mon Nov 13, 2017 3:44 pm

I was reading this artictle: https://static.rainfocus.com/vmware/vmw ... 01nYO0.pdf
Which is a presentation from vmWorld 2017 entitled "A Deep Dive into vSphere 6.5 Core Storage Features and Functionality [SER1143BU]"

I got to side 18 which was talking about differences between VMFS6 and VMFS5:
Resources for VMs (blocks, file descriptors, etc.) on earlier VMFS versions were
allocated on a per host basis (host-based block allocation affinity)
• Host contention issues arose when a VM/VMDK was created on one host, and then
vMotion was used to migrate the VM to another host
• If additional blocks were allocated to the VM/VMDK by the new host at the same time
as the original host tried to allocate blocks for a different VM in the same resource
group, the different hosts could contend for resource locks on the same resource
• This change introduces VM-based block allocation affinity, which will decrease
resource lock contention

I checked -- the datastores that the replicas are in a vmWare 6.5 VM on Server 2012 ReFS stored on *VMFS5*.
Any chance this is the issue? Ugly if it is because there is no easy conversion from VMFS5 to VMFS6 other than "get a new datastore (on additional storage you may not have), and then move the data from the old VMFS5 datastore to the new one".

I thought this because VMFS6 comes WITH 6.5..... So what about old VMFS5 datastores..... from text above they could be having resource lock contention --- that's what the error sounds like.

What do you think?
gfdos.sys
Novice
 
Posts: 6
Liked: 1 time
Joined: Wed Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby foggy » Tue Nov 14, 2017 5:07 pm

According to our QA, this doesn't look related, they were able to reproduce on VMFS6 as well.
foggy
Veeam Software
 
Posts: 15295
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Veeam Logoby gfdos.sys » Thu Nov 16, 2017 4:05 pm

Status update:
I heard back from the SE at Veeam, the case is now with the QA Department, and they will be creating a DCPN ticket to work with VMWare toward resolution.
gfdos.sys
Novice
 
Posts: 6
Liked: 1 time
Joined: Wed Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 1 guest