Tape Devices in a Virtual Machine Tape Server

[dcampregher]

Hello,

Is possible to configure the FC/SAS/SCSI/iSCSI Tape Devices ( Medium Changers and Drives ) in a Virtual Machine to do a Tape Server ? Or i need to have a physical server running Windows ?

If yes, how can i configure this environment in the ESXi ( Pass Trough, Direct Path etc ... ) ?

[Mike Resseler]

Hi Diogo,

Can you look at this KB to see if it fits your case? https://kb.vmware.com/selfservice/micro ... Id=1016407

[dcampregher]

So, the User Guide show this on the Tape Requirements:

LTO3 or later tape libraries (including VTL) and standalone drives are supported. Tape device must be directly attached to the backup server, to a tape server via SAS, FC or iSCSI interface. Note that VMware does not support connecting tape libraries to ESX(i) for VM pass-through.

What the VMware KB do is the "pass-through" configuration and this is not supported. I don't understand if is supported or not.

[Mike Resseler]

It might be something that needs to be fixed in the documentation which used to be not supported but changed. Let me check with the teams and probably Dima will come back to you asap with the final answer

[kmouline]

Hello Diogo,

Any update about that, I need to have confirmation if this conf is supported on a production environment or not, please also advise about VEEAM link regarding supported tape/library architectures.

In my case I need to know if a library attached to an ESXi 6 hosts by SAS can be used by a Virtual VEEAM backup Server.

Thank you,

[Dhammer]

We are on ESXi 5.5 and running two Dell TL1000 libraries via a virtual Veeam tape server and passthrough. It just worked. Hope it helps.

[veremin]

This configuration is not supported.

This does not mean the setup won't work. However, should you have issues with it, our support team or VMware one will be unlikely to help.

So, I'd recommend either attaching tape library to Windows-based physical server or exposing the device to VM via ISCSI.

Thanks.

[mike.r]

Just to understand it right:
Connecting a tape library via SAS to a VMware Host and installing a Veeam tape server, as a virtual machine on the same host is not allowed by VMware ?
Where did you read that ?

[veremin]

Tape drives. VMware does not support tape drives that are connected to ESX/ESXi hosts.

[Dhammer]

They are not saying it is not supported. They are saying that specifically they, VMWare, do not support it. See bullets1 & 2 and you may want to note the last one.

It's working for us using a Dell host and Dell 6Gbps SAS HBA adapter. We are also using Dell libraries.

Work with your hardware vendor and Veeam if possible.

From the KB:
•VMware does not provide support for backup tape drives and tape library devices or their functionality on ESXi/ESX hosts.
•The functionality of the tape library and the backup software in conjunction with VMware ESXi/ESX is partner-supported.
•VMware assumes that the third-party vendor has completed necessary validation to support this configuration in a production environment.
•VMware does not maintain a hardware compatibility guide that includes tested tape devices.
•Devices and configurations that worked on earlier releases of ESXi/ESX may not work with later ESXi releases.

[veremin]

They are saying that specifically they, VMWare, do not support it.

In other words, if tape device exposed via pass-through fails, it will be a software vendor that should address this problem guaranteeing stability.

That's why we've said that such scenario is not supported, as no additional code has been implemented from our side to guarantee tape pass-through reliability.

Nevertheless, it still doesn't mean that this configuration might work perfectly well in your case.

Thanks.

[Matt@Work]

Chatting to a colleague and throwing stupid idea's around. One of which was we have an ESXi host with an FC adapter that connects to our storage network and can also see a tape drive thats connected to a separate physical server.

If we mapped the tape drive to the ESXi host, is it possible to 'present' that to a new VM which would be a Veeam management server? Would it work, would performance be rubbish, am I barking up the wrong tree?

I put my back out yesterday, so I'm sat in the house just Reddit browsing and something another user said made me wonder the above.

Cheers

Matt

[PTide]

Hi,

It's not stupid at all and might even work fine, however it is not officially supported neither by Veeam nor by VMware, please check this thread for details.

Thank you.

[Matt@Work]

OK thanks. I'm not risking running in an unsupported environment when its mission critical for us.

[veremin]

If I were you, I'd rather connect a device to any Windows-based physical server existing in your environment and leverage it as tape server.

Should you encounter any issue, it would much easier to investigate it with support help, rather than on your own.

Thanks.

[Matt@Work]

Thats what we do now, I'l keep it that way.

[veremin]

Good to know. This way not only will you stay compliant with both VMware and Veeam, but also get a through support in case you need it. Thanks.

[VladV]

I'm sorry to dig up an old thread but would like to point out something. At least from my point of view...

Because I'm in the process of purchasing a tape library I've started planing for deployment. One thing I noticed was the KB mention in this topic which states that VMWare does not support Tape drives/libraries connected to ESXi 5 and up.

I am planning to add a SAS HBA card and dedicate it to an existing VM by using DirectPath I/O. This way, the card bypasses VMWare and is connected directly to the OS. After that, it shouldn't matter what you connect to the HBA from VMWare's perspective as it almost completely bypasses the hypervisor.

Is my thinking wrong? Are there other factors using this configuration that may affect stability or performance? I mean, I have been using this config to attach JBODs to Windows Storage Spaces for a long time now without any issues.

[francs]

I've learned this the hard way, after spending hours trying to make tape work through a VM.
Don't waste your time.
There's a reason it is not supported.
Even if you do get the OS and Veeam to see and use the library and tape devices you still end up with strange intermittent errors.
It's also not a Veeam/Windows only problem. I've had problems getting tape to work properly with TSM in a VM on ESX and even TSM on virtualized IBM Power servers.

[Dima P.]

Thanks francs. I completely agree - it's better to stick with supported configuration, as otherwise you may end up with an issue which we cannot fix as it’s related to third party components.

[infused]

Just use AWS VTL. It's so damm good. http://www.tecfused.com/2017/07/veeam-a ... ary-guide/

[Anders]

Just want to add that I've used both FC Attached Tape and SAS attached tape in virtual machines based on vmware 5.5 and 6.0 without any issues.

The trick is to attach the entire HBA Card to the virtual machine (thereby limiting the VM from vMotion etc) - But performance and stability is super. Filling up 2 LTO7 tapes at max speed (600 MB/s) is possible.

Its not optimal, but if you NEED the tape server to be virtual, its a working solution. The solutions I've been involved in, have been to provide easy faalback to the old backup solution to restore old files from tape, by moving the HBA to the old backup server (that was virtualized and put on the same host)

[fll]

Hello, can you recommend a Tape Library lto-8 iscsi to connect it to a virtualized Tape Server?

Thank you.

[Dima P.]

Hello fll.

Any LTO device will work with Veeam B&R and iSCSI connection should work without any issues. Just keep in mind that we do not officially support tape devices directly attached to ESXi and connected to a virtual machine via iSCSI passthrough.

[thomas.biesmans]

I'd like some clarification on Vlad's point as one of our sales has come across the "Note that VMware does not support connecting tape libraries to ESXi for VM pass-through." in your system requirements.

I keep noticing a lack of subtlety in definitions, especially about the difference between tape drive pass-through and using DirectPath I/O on an HBA level to connect to tape drives.

The 9 year old VMware KB, referenced in 7 year old release notes, talks about tape device pass-through when using an ancient SCSI controller. I think of this like an RDM, where you map your volumes to a VM instead of using it them as datastores, but in this case by adding a specific SCSI path to the VM. Adding tape drives in ESX 4 had been the cool new thing, as we previously installed Media Agent components (think the tape server package) inside the ESX 3.5 host itself through the CLI! Sure that this legacy functionality isn't supported.


Now, 2020, DirectPath I/O, the massively different ability to map a whole HBA to a VM is still a thing in vSphere 7.0 (configuring, not yet updated to 7.0, Requirements for Platforms and Devices, updated for vSphere 7.0). The HBA is solely dedicated to the single VM, install HBA drivers and you can use it. Sure there are trade-offs, aren't there always, doesn't matter for a single ESXi-based backup host for example.

You could do it for NICs if you want to, but you could also use FC HBAs. Using a dedicated FC HBA lets you access FC tape devices through a VM, or even allows you to use a VM for backups from storage snapshots, as the VM now has direct access to the FC storage array.


I cannot find any limitations imposed on DirectPath I/O HBAs in VMware's HCL or the last article link, for example not allowing them to be used to access FC tape devices.
In Veeam's Help Center I can only find a reference to 'tape device passthrough', which imho only concerns the first, older method I described.

So:

• Does Veeam support DirectPath I/O devices at all?
• Does Veeam support DirectPath I/O devices for storage snapshotting from FC-based arrays on a VM?
• Does Veeam support DirectPath I/O devices for tape connectivity from FC-based tape drives on a VM? If you do feel like the previous VMware KB is still related to this question, then I'm not sure we'll ever agree.

As an aside to further highlight the dangers of unclear definitions: you might also want to update the following line in the System Requirements "Tape device must be directly attached to the backup server, to a tape server via SAS, FC or iSCSI interface.", as this would not allow a physical server to access FC tape drives through a SAN switch, nor an iSCSI library through a network switch. Because that's the definition of 'direct' I feel we generally use in direct-attached vs fabric-attached and direct vs switched Ethernet connections on the physical layer and I cannot believe that this is what you meant by it.
Makes you wonder what VMware thought of when they included the following in their KB: "VMware no longer supports Tape Drives connected directly to ESXi 5.x and later."...

[RobTurk]

Tape is a sequential access device. What this means is that all SCSI commands to read/write/position depend on their exact order of arrival to maintain data integrity. Any 'meddling in the middle' by another process jeopardizes the order of commands, and with that, data integrity. When you attach a tape device to a VM, as long as everything else sits tight, things will likely work.

In a VMWare environment, the ESXi hypervisor sometimes needs to rescan or even reset storage adapters, for example when dynamically adding or removing data stores, or to recover from transient issues. A rescan sends out a number of commands to detect all attached devices, including attached tape drives. This may interfere with the stream of commands coming from a VM, upsetting the order of arrival.

Here's a (simplified) example of what might happen. Things may work for years without ever causing problems. Then one day during a backup, one of your disks starts to go iffy. A bad block develops. After a number of retries, VMWare decides to reset the storage adapters, to see if it can get the disk to behave again. Good news, the disk responds again. Bad news, this reset will cause the attached tape drive to rewind, and subsequent write commands will overwrite whatever is at the beginning of the tape. The VM may not have noticed.
A few minutes later that disk really gives up the ghost and you need to restore. From that tape that was just overwritten. Ouch..

To avoid scenario's like these, VMWare has declared tape attached to an ESXi host unsupported. It's to protect your data from the nature of the beast.

[tsightler]

Hi Thomas, thank you for bringing up this issue, it's definitely something that comes up from time-to-time and I largely agree with your post. For the most part the issues that Rob discusses above shouldn't really apply to DirectPath I/O as the host is passing the native PCI device through to the VM and the device driver within the VM controls the PCI device. ESXi is free to reset SCSI devices on the controllers in "owns" but won't touch devices using DirectPath I/O because those devices are owned by the VMs they've been assigned to, so I'm not too worried about that.

That being said, Veeam QA does not currently test DirectPath I/O configurations. While the overall expectation is that DirectPath I/O should work no differently than with physical hardware, the reality is that DirectPath I/O is deceptively simple in it's use while, underneath, is a very complex technology with many interactions between the hardware, hypervisor and guest VM. VMware has very specific recommendations for testing solutions that use DirectPath I/O for both the OEM system and device I/O vendor, you can read more about this here:

https://kb.vmware.com/s/article/2142307

Some example to the difference in behavior is that DirectPath I/O devices do not see PCI errors in the guest, these are trapped and handled by ESXi and will most likely cause an NMI, which will probably create a PSOD on the host (based on ESXi host settings). Probably unlikely to happen, but still something worth knowing since it could impact other VMs running on the host.

I've personally experienced some of the complication DirectPath I/O can introduce when I was working with a partner that was deploying a solution using DirectPath I/O and was fighting an issue where output to tape was failing intermittently with no discernible pattern. When the same controller was moved to a standalone server it worked fine. Working with VMware we eventually discovered that certain PCI interrupts were not being delivered correctly to the guest VM. We were able to implement a temporary workaround with a VM level setting to change the interrupt mode from MSI-X, and, eventually, VMware provided a patch for the issue which resolved the problem, but it was certainly more complex to troubleshoot due to the DirectPath I/O in the mix.

Because Veeam QA does not test configurations that use DirectPath I/O it's difficult to say that this configuration is specifically supported, but I do agree that the reason it's not supported isn't due to the reasons stated in those old VMware KB articles that reference SCSI passthrough. I will discuss internally with the team and see if we can work to improve the support statement.

[thomas.biesmans]

Hey Rob, Tom,

Thank you very much for both of your clarifications on regular and DirectPath I/O communication, and the time you took to reply! Being able to get such detailed feedback is one of the things that makes Veeam stand out on top. Very much appreciated!

About the use-case, as discussed with our presales: while large companies often have dedicated backup servers, some of our smaller to medium ones don't. Some have a Veeam VM on production ESXi hosts, but running on separate backup block storage hosting the Repos. Another recent use case is having a Veeam VM and a StoreOnce VSA VM as well (both on a dedicated backup server like HPE Apollo running ESXi) to add a bit of (not full) ransomware protection through Catalyst. Another use case I personally like very much is using two hosts for a small VMware HA backup cluster using dedicated block storage for backups.
Customers with synchronous replication (once again a European SMB thing) often like to combine such an ESXi-based backup host in a third site for a witness / quorum VM.

Sure they are not all ideal, but reality often has us looking for a best fit with limited means.

[alterschlagg]

Hello Tom and everyone,

any news on how you wanted to "see if [you] can work to improve the support statement"?

I have a use case where I could need Direct Path IO support for a virtual tape server as well....

Thanks for your great support,

Stefan

[Dima P.]

Hi folks,
Yes, we've agreed that such setup is supported, however if something goes wrong with vSphere part of the infrastructure we wont be able to investigate it. Here is the quote from the updated Supported Devices and Configuration

VMware does not support tape drives connected directly to ESXi 5.5 and later. For more information, see VMware vSphere Release Notes.
Veeam Backup & Replication will potentially work with such a configuration, but if there is a hardware-related issue, Veeam Support will not provide help to solve it. This functionality is vendor-supported.
For more details and recommendations on configuring vendor-supported tape drives and media changers on ESXi, refer to VMware documentation at this VMware KB article.