Virtual lab from storage snapshots - Less than par performance

[cfizz34]

Has anyone taken a production set of VM's (in my case a DC, fileshare, few sql servers, a/v servers, front end IIS boxes, be async server, and some Win10 guest VM's) and had good performance results when running all the VM's on the same host with the veeam lab feature (via snapshots)? The utilization on the host is not that bad but the end user experience from the Win10 guest side calling the front end boxes and clicking through the UI is slower than compared to that of the production nodes (prod takes 8 sec to call a page but lab can take 16-24 seconds or more). Just curious if any one has a Dynamics CRM lab where the end user results have been closer to prod so the testing is more consistent for my app/dev team.

[HannesK]

Hello,
of course, a virtual lab booted from a backup storage is slower than production.

Two questions
1) are you running on V10?
2) what kind of backup storage are you running and what type of production storage are you running?

Having half or one third of production speed sounds not to bad for me depending on the setup (SSDs for production and 7k disks for backup)

I have heard around 10 VMs in parallel some years ago from a customer that worked fine. I don't remember the application.

Best regards,
Hannes

[cfizz34]

They are running from snapshots off the primary array which is all flash.

I am running V10 P2 utilizing single host networking. Pure M50 v5.1.15 All-Flash array.

[cfizz34]

LAB HOST:
Hypervisor: VMware ESXi, 6.7.0, 15160138
Model: PowerEdge M620
Processor Type: Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz
Logical Processors: 32
319.96 GB MEMORY

PROD HOST:
Hypervisor: VMware ESXi, 6.7.0, 11675023
Model: PowerEdge M640
Processor Type: Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz
Logical Processors: 32
382.63 GB MEMORY

[Gostev]

They are running from snapshots off the primary array

In that case, at a first sight I can't see any reason why would VM performance in the virtual lab be so much different from the production environment.

For troubleshooting, I personally would try running some I/O test tool (like diskperf) in the production and lab VMs to ensure there's no performance overhead with Pure Storage when VM is running off of a snapshot volume. Doing this test will help to isolate any potential issues due to the storage architecture. Be sure to test both reads and writes.

[cfizz34]

LAB:
DDR-3

PROD:
DDR-4

[cfizz34]

So this is interesting....when the SQL server is running off a snapshot, the query takes 10 seconds. If I clone the server instead, the same query only takes 4 seconds as the same server that took 10 seconds. Guess I'll be calling vmware and pure on this...joy!!!! I really can't believe its such a big difference but that proves something is a miss.

[cfizz34]

Well I guess I have something off with the one esxi host since I replicated the lab on a diff host and its almost as fast as production now. Man, what a headache and time waster....all my hosts in the cluster are supposed to be identical but obviously something with the underlying hardware is not right.

[Gostev]

Sounds good, and thanks for coming back with the solution!

[cfizz34]

This one had me scratching my head for sure. So I rebuilt the entire ESXi host from scratch, starting with 6.5, then to 6.7, and after the rebuild it performs just as expected. Goes to show, run baselines on your hardware when both newly deployed and from time to time once in production.