Discussions specific to the VMware vSphere hypervisor
gfdos.sys
Influencer
Posts: 10
Liked: 1 time
Joined: Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer
Contact:

Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by gfdos.sys » Nov 01, 2017 3:59 pm

After the B&R 9.5U1 announcement https://www.veeam.com/kb2222
Virtual hardware version 13 support. vSphere 6.5 introduces the new VM hardware version which increases some configuration maximums and adds ability to add NVMe controllers to a VM. This update adds ability to process such VMs.

Using Veeam Backup 9.5U2 a windows vm with a virtual NVMe controler backups fine.
However if trying a Replication Job, the initial replication works, but subsequent Replication incrementals always fail.
I get virtual NVMe is a new feature in vSphere 6.5, but if it doesn't work with Backup *AND* Replication, why call attention to it as a feature that is supported in the 9.5U1 announcement?

Warning -- for now Backup for vms with virtual NVMe works but Replication does not.
(note: this is on vsphere 6.5U1 with the patch found here: https://kb.vmware.com/s/article/2151061?language=en_US
with Veeam Backup 9.5U2)

foggy
Veeam Software
Posts: 16822
Liked: 1359 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by foggy » Nov 01, 2017 4:28 pm

Hi Gabriel, do you have a support case ID for this issue?

gfdos.sys
Influencer
Posts: 10
Liked: 1 time
Joined: Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by gfdos.sys » Nov 01, 2017 5:24 pm 1 person likes this post

Case #02333485
Level 2 engineer looking at it. Apparently 2 other cases open right now.
In my case and one other case the "solution" was use SCSI not NVMe.
Waiting to hear back what they told the other case.

Figured I'd ask here and give the warning of what we are running into.

NVMe on top of HCI infrastructure is sweet, and gave us a nice performance boost. We are using DataCore VirtualSAN, so it is doing auto tiering of our SAS SSD and SAS SCSI, so why add extra "baggage" of scsi virtually?
Physical NVMe has a high cost, but if we already have the performance with HCI, it makes total sense. Only thing holding us back is the Replication needed for DR.

https://www.starwindsoftware.com/blog/i ... olutionary

foggy
Veeam Software
Posts: 16822
Liked: 1359 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by foggy » Nov 02, 2017 10:36 am 1 person likes this post

We were able to reproduce this internally, the issue is in failing Revert Snapshot operation. We will investigate and, if the issue is on Veeam B&R side, address it in one of the next updates (could be even the upcoming v9.5 Update 3).

Gostev
Veeam Software
Posts: 22981
Liked: 2882 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by Gostev » Nov 02, 2017 7:38 pm

Most likely some change in vSphere 6.5U1 or Veeam 9.5U2 is causing this... B&R 9.5U1 was indeed tested against VMs with NVMe controllers extensively, but it was done during plain vSphere 6.5 times. We will figure this out.

gfdos.sys
Influencer
Posts: 10
Liked: 1 time
Joined: Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by gfdos.sys » Nov 03, 2017 2:30 pm

Veeam vmware on hpe hosts vmware on dell veeam host
7/20 9.5.0.823 6.5.0 5146846 6.5.0
9/22 NVMe Enabled
10/28 9.5.0.1038 6.5U1 5969303(Oct)+patch 6.5U1+patch

>Most likely some change in vSphere 6.5U1 or Veeam 9.5U2 is causing this...
Just for the record: We had the problem as soon as we enabled NVMe on vmware 6.5.0 5146846 on Veeam 9.5U1

>B&R 9.5U1 was indeed tested against VMs with NVMe controllers extensively
The "Full replica" works -- its any incremental replicas that fail.
I went back to my Veeam email logs and Replication failed on 9/22 --> at the first "replication incremental" after adding it.
"Creating helper snapshot Error: Detected an invalid snapshot configuration.
Error: Detected an invalid snapshot configuration."

the next day the error changes to:
"Preparing replica VM Error: Detected an invalid snapshot configuration.
Error: Detected an invalid snapshot configuration."

every attempt after that:
"Deleting helper snapshot Error: Unable to access file since it is locked
Error: Unable to access file since it is locked"

Backup works just fine all along.

gfdos.sys
Influencer
Posts: 10
Liked: 1 time
Joined: Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by gfdos.sys » Nov 08, 2017 1:27 pm

I had 2 ideas for a workaround on this, What do you guys think?

Work around 1:
Rhetorical question -- Would this NVMe issue with replication happen with physical hardware?
In order to do that you would have to use the Veeam Agent to do the backup and replication....

Is that a possible workaround -- Installing the Veeam Agent on a vm with virtual NVMe?
After installing the Veeam Agent (I have never done backup that way) would I have to remove the vm from the backup/replica jobs and readd it?

Work around 2:
I presume this backup and replication job are using NBD mode.
All the VMs I'm backing up are stored on a Datacore "iSCSI" SAN. If I backed up the jobs in Direct SAN access mode would it avoid the issue of the "snapshots" being locked that the error message seems to suggest?

foggy
Veeam Software
Posts: 16822
Liked: 1359 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by foggy » Nov 08, 2017 4:01 pm

The issue is in reverting a snapshot of the replica VM, so the mentioned workarounds are not relevant.

foggy
Veeam Software
Posts: 16822
Liked: 1359 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by foggy » Nov 09, 2017 10:18 am

Btw, as an update, it is currently being investigated with VMware and has just been escalated to T2 of their support.

gfdos.sys
Influencer
Posts: 10
Liked: 1 time
Joined: Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by gfdos.sys » Nov 13, 2017 3:44 pm

I was reading this artictle: https://static.rainfocus.com/vmware/vmw ... 01nYO0.pdf
Which is a presentation from vmWorld 2017 entitled "A Deep Dive into vSphere 6.5 Core Storage Features and Functionality [SER1143BU]"

I got to side 18 which was talking about differences between VMFS6 and VMFS5:
Resources for VMs (blocks, file descriptors, etc.) on earlier VMFS versions were
allocated on a per host basis (host-based block allocation affinity)
• Host contention issues arose when a VM/VMDK was created on one host, and then
vMotion was used to migrate the VM to another host
• If additional blocks were allocated to the VM/VMDK by the new host at the same time
as the original host tried to allocate blocks for a different VM in the same resource
group, the different hosts could contend for resource locks on the same resource
• This change introduces VM-based block allocation affinity, which will decrease
resource lock contention

I checked -- the datastores that the replicas are in a vmWare 6.5 VM on Server 2012 ReFS stored on *VMFS5*.
Any chance this is the issue? Ugly if it is because there is no easy conversion from VMFS5 to VMFS6 other than "get a new datastore (on additional storage you may not have), and then move the data from the old VMFS5 datastore to the new one".

I thought this because VMFS6 comes WITH 6.5..... So what about old VMFS5 datastores..... from text above they could be having resource lock contention --- that's what the error sounds like.

What do you think?

foggy
Veeam Software
Posts: 16822
Liked: 1359 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by foggy » Nov 14, 2017 5:07 pm

According to our QA, this doesn't look related, they were able to reproduce on VMFS6 as well.

gfdos.sys
Influencer
Posts: 10
Liked: 1 time
Joined: Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by gfdos.sys » Nov 16, 2017 4:05 pm

Status update:
I heard back from the SE at Veeam, the case is now with the QA Department, and they will be creating a DCPN ticket to work with VMWare toward resolution.

gfdos.sys
Influencer
Posts: 10
Liked: 1 time
Joined: Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by gfdos.sys » Nov 27, 2017 3:41 pm

Any chance NVMe issues are addressed in the RTM of Veeam Update 3?
Any other status update on this?

foggy
Veeam Software
Posts: 16822
Liked: 1359 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by foggy » Nov 28, 2017 4:09 pm

Definitely not in U3. The investigation is underway, VMware is involved, however, not much progress to share.

gfdos.sys
Influencer
Posts: 10
Liked: 1 time
Joined: Nov 01, 2017 3:31 pm
Full Name: Gabriel Fischer
Contact:

Re: Any Word on NVMe Replication (vSphere 6.5 + B&R 9.5U2)

Post by gfdos.sys » Jan 31, 2018 8:47 pm

Here is the latest from Veeam about the NVMe controllers......

I asked our QA team for the update for the issue with NVMe controllers and this issue is still under research from VMware side. The issue with NVMe controllers will be fixed as soon as VMware confirms a bug. So unfortunately, there is no ETA at this point.

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 23 guests