Discussions specific to the VMware vSphere hypervisor
Post Reply
Delo123
Expert
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: [MERGED] VM unresponsive during removing /consolidating

Post by Delo123 » May 08, 2017 10:07 am

shlomia wrote:Hi,
I checked my ESXI's and they do not have this update installed.
I'm just scared to manually update, because it says that impact will be a reboot.
Although I have vsphere HA, do I need to scare to do any reboot to the esxi's?
Hi Shlomi,

Put your ESXi Host into maintenance mode before you apply updates, that way you are sure your production will not be harmed. After the ESXi Hosts has rebooted you can take it out of maintenance mode again.
When everything is ok you can proceed with the other hosts in your cluster...

petteripasanen
Lurker
Posts: 1
Liked: never
Joined: May 23, 2017 7:26 pm
Full Name: Petteri Pasanen
Contact:

[MERGED] File Share Witness fails because of backupping

Post by petteripasanen » May 23, 2017 7:43 pm

I've set up SQL Cluster which has File Share Witness pointing to FileServer where I store SQL Backups.
When Veeam is finishing backupping this FileServer the File Share Witness fails arbitrate the share because it can't connect to it which then leads Cluster Service shutting down because quorum was lost which then leads SQL Servers in Cluster to terminate unexpectedly.

How can I limit the affect Veeam has on the server it's backupping? I've tried enabling storage latency control but that didn't help.

foggy
Veeam Software
Posts: 18458
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy » May 25, 2017 3:09 pm

Hi Petteri, you can find some hints as well as explanation of why this happens in this thread.

analytix
Novice
Posts: 4
Liked: never
Joined: Aug 23, 2017 3:30 pm
Contact:

[MERGED] Connectivity issues while taking a snapshot

Post by analytix » Aug 23, 2017 4:04 pm

Hello,

We are using Veeam B&R Enterprise Plus 9.5 update 2. We have encountered a strange issue while taking a snapshot. We have 10 out of 30 VMs which requires to be backed up multiple times in a day and therefore, scheduled a backup job to run every 3 and 6 hours. After making this change, we anticipated that everything will go normal as we were previously taking a backup once a day.

Unfortunately, after scheduling a backup multiple times in a day, users started reporting error of connectivity errors and application errors. We have a virutalized SQL Server and quite a few critical file servers. While taking a snapshot, VM looses network connectivity for a few seconds ( 3 - 6 ping loss) and which is causing the issue. We manually performed the snapshot of VM using vSphere web client and discovered while continuously monitoring ping and result was same. Therefore, to narrow down the issue, it seems that it is VMWare which causes this.

Upon further investigation and research on the internet - we found that few ping loss is normal with VMWare while taking a snapshot. My question is is this something a normal behavior or not? If this is normal behavior then should we use Veeam Agent for Microsoft Windows / Linux instead of using VM Agent for taking a backup multiple times in a day.

Any help or advice is greatly appreciated.

foggy
Veeam Software
Posts: 18458
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy » Aug 23, 2017 4:21 pm

You're right, this behavior is typical for VMware, please review this thread for details and possible hints on how to eliminate this behavior.

analytix
Novice
Posts: 4
Liked: never
Joined: Aug 23, 2017 3:30 pm
Contact:

Re: Snapshot removal issues of a large VM

Post by analytix » Aug 23, 2017 9:31 pm

Thanks but not much hint from the thread. Any alternative solution would you recommend such as Veeam Agent for Windows / Linux?

DGrinev
Veeam Software
Posts: 1819
Liked: 230 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Snapshot removal issues of a large VM

Post by DGrinev » Aug 25, 2017 1:55 pm

Hi,

My two cents: You can use integration with storage systems, it should significantly reduce snapshot consolidation time.
Also, latest versions of VMware vSphere contain improvements for the subject. Thanks!

bramos
Lurker
Posts: 2
Liked: never
Joined: Mar 12, 2018 11:06 am
Full Name: Benja Dikar
Contact:

[MERGED] Full Backups caused cuts of communication in server

Post by bramos » Mar 13, 2018 7:54 am

I have Veeam Backup & Replication 9.5 and i when execute Virtual Machines full backups the production servers lost communication for a moments.
It´s expecialy problematic with Oracle DB contains Servers because the DB conections have been broken and lost comunication.

I'm sorry, because of my level of English it's a bit.

Regards.

foggy
Veeam Software
Posts: 18458
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy » Mar 13, 2018 10:04 am

Hi Benja, if this occurs at the time of snapshot commit, then this is expected, see above for details and some hints.

bramos
Lurker
Posts: 2
Liked: never
Joined: Mar 12, 2018 11:06 am
Full Name: Benja Dikar
Contact:

Re: Snapshot removal issues of a large VM

Post by bramos » Mar 13, 2018 11:20 am

Thanks, I'll review this post well

FrancWest
Expert
Posts: 118
Liked: 10 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: [MERGED] Connectivity issues while taking a snapshot

Post by FrancWest » Mar 17, 2018 8:38 pm

analytix wrote:Hello,

Upon further investigation and research on the internet - we found that few ping loss is normal with VMWare while taking a snapshot. My question is is this something a normal behavior or not? If this is normal behavior then should we use Veeam Agent for Microsoft Windows / Linux instead of using VM Agent for taking a backup multiple times in a day.

Any help or advice is greatly appreciated.
We have the same issue, underlying storage is an all flash SAN, but still the ping loss occurs. We ended up using the veeam agent for windows to backup our sql and exchange servers multiple times a day. No timeouts since then.

Franc.

unlucky
Novice
Posts: 3
Liked: 2 times
Joined: Mar 02, 2017 1:47 pm
Full Name: unlucky
Contact:

Re: Snapshot removal issues of a large VM

Post by unlucky » Jun 14, 2019 9:16 am

It´s quite an old topic, but we now hit the same issue. Our underlying storage is also an all flash SAN (iSCSI connection). All machines are affected, but as sql / exchange / file server are backed up multiple times a day it is noticable for the users. We´re running on Vmware 6.5U2.
I actually though we´re not doing rocket science, but obviously this is the case.

@Franc : Are you maybe willing to share some insights of your "fix" (using veeam agent for windows) ? Or have you faced timouts in the meantime as well ?

FrancWest
Expert
Posts: 118
Liked: 10 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: Snapshot removal issues of a large VM

Post by FrancWest » Jun 14, 2019 10:06 am

Hi,

ever since we use the Veeam Agent for Windows the backups run without issues during work-hours. No timeouts at all. The only thing I notice is that the VSS snapshot phase of SQL takes about 15 minutes to complete. Haven't found a solution yet, but it's not that disturbing since the backups are working fine.

unlucky
Novice
Posts: 3
Liked: 2 times
Joined: Mar 02, 2017 1:47 pm
Full Name: unlucky
Contact:

Re: Snapshot removal issues of a large VM

Post by unlucky » Jun 19, 2019 5:35 am

Hi,

ok, but as a result (afaik) you will lose direct storage integration as the agent communicates directly with the backupserver. I´ve tested it yesterday and it looks like the good old onhost backup on
Microsoft Hyper-V with 2K12. The bad thing is that you have to setup some kind of a new dedicated backup network, as we currently use storage snapshots and all backup traffic is handled in
the isolated iscsi network.

FrancWest
Expert
Posts: 118
Liked: 10 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: Snapshot removal issues of a large VM

Post by FrancWest » Jun 19, 2019 1:46 pm

Correct, but why not add a NIC to the VM and connect it to the iSCSI network? That way you can configure the backup server to connect to the SQL VM using the iSCSI network also.

mholman
Novice
Posts: 4
Liked: never
Joined: Jun 24, 2019 11:07 pm
Full Name: IT Support
Contact:

Re: Snapshot removal issues of a large VM

Post by mholman » Jun 24, 2019 11:21 pm

I see a lot of post stating that it is normal VMware behavior to take some time to snap or remove snaps; however, we have run several tests using a small vm guest (300MB), and we have found that snaps and snap removals, when doing directly from vCenter, is consistently 1s (maybe 2s). However, when doing the same from Veeam, we see it consistently takes 1+ minutes - sometimes a few minutes. So, I am not in agreeance that this is a VMware issue.

veremin
Product Manager
Posts: 17075
Liked: 1475 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » Jun 25, 2019 2:32 pm

In your test without Veeam, have you waited for the same time as it takes to perform backup before removing the snapshot - or you've removed it as soon as it was created, without giving it a chance to actually grow? Thanks!

mholman
Novice
Posts: 4
Liked: never
Joined: Jun 24, 2019 11:07 pm
Full Name: IT Support
Contact:

Re: Snapshot removal issues of a large VM

Post by mholman » Jun 25, 2019 4:11 pm

As mentioned, the vm guest is just a 300mb test vm. There were 151.4 KB worth of changes which should take no time to commit. The backup job within Veeam finishes about as quickly as it would take me to execute the commands manually within vCenter.

mholman
Novice
Posts: 4
Liked: never
Joined: Jun 24, 2019 11:07 pm
Full Name: IT Support
Contact:

Re: Snapshot removal issues of a large VM

Post by mholman » Jun 25, 2019 5:05 pm

Just to confirm, I just ran another manual test within vCenter with 40 min between snap and snap removal and the removal still just took 2s.

veremin
Product Manager
Posts: 17075
Liked: 1475 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » Jun 25, 2019 5:32 pm

In this case it might be worth reaching our support team with the results of your findings - assuming the tests were conducted properly, you should have seen the same behaviour in both cases.

Thanks!

Gostev
SVP, Product Management
Posts: 25145
Liked: 3698 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Snapshot removal issues of a large VM

Post by Gostev » Jun 25, 2019 8:47 pm

Both vCenter and Veeam use the same exact API calls to create and remove snapshots, so there's got to be some other difference in these experiments. Perhaps you can demonstrate the same live to our support engineer, who can then review the debug logs to find the difference.

For example, depending on your backup mode, there are other operations that will delay the beginning of snapshot removal, like hot remove of the backed up disk, and the debug logs will reflect these operations. Regardless though, the actual snapshot removal should take the same time in both cases with very small snapshots, because this operation is managed by ESXi and is performed by the same exact ESXi function (Veeam has no control over it whatsoever).

But that is assuming clean experiment, of course. For example, if that test backup job continues on to processing other VMs while the snapshot is being removed, this will put additional load on the datastore, thus slowing down the snapshot removal. Anyway, it should be super easy to get to the bottom of this difference. There has not been a month in the past 12 years of Veeam existence without similar concerns, and every time we were able to demonstrate all doubting customers the consistency of snapshot removal performance regardless of how it is initiated :D

mholman
Novice
Posts: 4
Liked: never
Joined: Jun 24, 2019 11:07 pm
Full Name: IT Support
Contact:

Re: Snapshot removal issues of a large VM

Post by mholman » Jul 01, 2019 4:08 pm

Working with Veeam support team it was identified this issue occurs when using vvols and hotadd with the proxy and vm guest on different hosts. https://vresolv.wordpress.com/2017/12/0 ... ed-on-vvol. They did provide a registry entry that will cause it to wait for a specified amount of time for the unbind. Initial testing so far seems to indicate this to be a valid workaround.

There are a few things I like about vvols, but this issue combined with not being able to take advantage of "backup from storage snapshots" may push us back to managing vmfs datastores.

Gostev
SVP, Product Management
Posts: 25145
Liked: 3698 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Snapshot removal issues of a large VM

Post by Gostev » Jul 02, 2019 9:36 am

Oh, I see where we had a disconnect. This entire thread is about VMFS actually... vVols is a totally different beast as it comes to VM snapshots handling, as it uses absolutely different architecture for those.

unlucky
Novice
Posts: 3
Liked: 2 times
Joined: Mar 02, 2017 1:47 pm
Full Name: unlucky
Contact:

Re: Snapshot removal issues of a large VM

Post by unlucky » Aug 20, 2019 2:19 pm 2 people like this post

Just as a follow up if someone else stumbles in since I still have an open case at Vmware and doing quite a lot of debugging with them.
Engineering Team is aware of that issue (Snapshot SEsparse disks) which should already be fixed in 6.7 latest build.
Unfortunately I cannot confirm as we are still running on 6.5. There should also be a fix released for 6.5 as well but no ETA yet.

preetam_de
Lurker
Posts: 1
Liked: never
Joined: May 17, 2019 8:40 am
Full Name: preetam_de
Contact:

Re: Snapshot removal issues of a large VM

Post by preetam_de » Oct 30, 2019 10:45 am

Have got any information from VMware support on this.
we are facing similiar issue.

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. » Nov 01, 2019 3:17 am

Since this is a pretty long standing issue, I suggest you also contacting the VMware team, so that they could see what version of vSphere you're at and assist if possible.

Post Reply

Who is online

Users browsing this forum: No registered users and 42 guests