Host-based backup of VMware vSphere VMs.
Post Reply
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: [MERGED] VM unresponsive during removing /consolidating

Post by Delo123 »

shlomia wrote:Hi,
I checked my ESXI's and they do not have this update installed.
I'm just scared to manually update, because it says that impact will be a reboot.
Although I have vsphere HA, do I need to scare to do any reboot to the esxi's?
Hi Shlomi,

Put your ESXi Host into maintenance mode before you apply updates, that way you are sure your production will not be harmed. After the ESXi Hosts has rebooted you can take it out of maintenance mode again.
When everything is ok you can proceed with the other hosts in your cluster...
petteripasanen
Lurker
Posts: 1
Liked: never
Joined: May 23, 2017 7:26 pm
Full Name: Petteri Pasanen
Contact:

[MERGED] File Share Witness fails because of backupping

Post by petteripasanen »

I've set up SQL Cluster which has File Share Witness pointing to FileServer where I store SQL Backups.
When Veeam is finishing backupping this FileServer the File Share Witness fails arbitrate the share because it can't connect to it which then leads Cluster Service shutting down because quorum was lost which then leads SQL Servers in Cluster to terminate unexpectedly.

How can I limit the affect Veeam has on the server it's backupping? I've tried enabling storage latency control but that didn't help.
foggy
Veeam Software
Posts: 21133
Liked: 2140 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Hi Petteri, you can find some hints as well as explanation of why this happens in this thread.
analytix
Novice
Posts: 4
Liked: never
Joined: Aug 23, 2017 3:30 pm
Contact:

[MERGED] Connectivity issues while taking a snapshot

Post by analytix »

Hello,

We are using Veeam B&R Enterprise Plus 9.5 update 2. We have encountered a strange issue while taking a snapshot. We have 10 out of 30 VMs which requires to be backed up multiple times in a day and therefore, scheduled a backup job to run every 3 and 6 hours. After making this change, we anticipated that everything will go normal as we were previously taking a backup once a day.

Unfortunately, after scheduling a backup multiple times in a day, users started reporting error of connectivity errors and application errors. We have a virutalized SQL Server and quite a few critical file servers. While taking a snapshot, VM looses network connectivity for a few seconds ( 3 - 6 ping loss) and which is causing the issue. We manually performed the snapshot of VM using vSphere web client and discovered while continuously monitoring ping and result was same. Therefore, to narrow down the issue, it seems that it is VMWare which causes this.

Upon further investigation and research on the internet - we found that few ping loss is normal with VMWare while taking a snapshot. My question is is this something a normal behavior or not? If this is normal behavior then should we use Veeam Agent for Microsoft Windows / Linux instead of using VM Agent for taking a backup multiple times in a day.

Any help or advice is greatly appreciated.
foggy
Veeam Software
Posts: 21133
Liked: 2140 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

You're right, this behavior is typical for VMware, please review this thread for details and possible hints on how to eliminate this behavior.
analytix
Novice
Posts: 4
Liked: never
Joined: Aug 23, 2017 3:30 pm
Contact:

Re: Snapshot removal issues of a large VM

Post by analytix »

Thanks but not much hint from the thread. Any alternative solution would you recommend such as Veeam Agent for Windows / Linux?
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Snapshot removal issues of a large VM

Post by DGrinev »

Hi,

My two cents: You can use integration with storage systems, it should significantly reduce snapshot consolidation time.
Also, latest versions of VMware vSphere contain improvements for the subject. Thanks!
bramos
Lurker
Posts: 2
Liked: never
Joined: Mar 12, 2018 11:06 am
Full Name: Benja Dikar
Contact:

[MERGED] Full Backups caused cuts of communication in server

Post by bramos »

I have Veeam Backup & Replication 9.5 and i when execute Virtual Machines full backups the production servers lost communication for a moments.
It´s expecialy problematic with Oracle DB contains Servers because the DB conections have been broken and lost comunication.

I'm sorry, because of my level of English it's a bit.

Regards.
foggy
Veeam Software
Posts: 21133
Liked: 2140 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Hi Benja, if this occurs at the time of snapshot commit, then this is expected, see above for details and some hints.
bramos
Lurker
Posts: 2
Liked: never
Joined: Mar 12, 2018 11:06 am
Full Name: Benja Dikar
Contact:

Re: Snapshot removal issues of a large VM

Post by bramos »

Thanks, I'll review this post well
FrancWest
Veteran
Posts: 522
Liked: 103 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: [MERGED] Connectivity issues while taking a snapshot

Post by FrancWest »

analytix wrote:Hello,

Upon further investigation and research on the internet - we found that few ping loss is normal with VMWare while taking a snapshot. My question is is this something a normal behavior or not? If this is normal behavior then should we use Veeam Agent for Microsoft Windows / Linux instead of using VM Agent for taking a backup multiple times in a day.

Any help or advice is greatly appreciated.
We have the same issue, underlying storage is an all flash SAN, but still the ping loss occurs. We ended up using the veeam agent for windows to backup our sql and exchange servers multiple times a day. No timeouts since then.

Franc.
unlucky
Novice
Posts: 3
Liked: 2 times
Joined: Mar 02, 2017 1:47 pm
Full Name: unlucky
Contact:

Re: Snapshot removal issues of a large VM

Post by unlucky »

It´s quite an old topic, but we now hit the same issue. Our underlying storage is also an all flash SAN (iSCSI connection). All machines are affected, but as sql / exchange / file server are backed up multiple times a day it is noticable for the users. We´re running on Vmware 6.5U2.
I actually though we´re not doing rocket science, but obviously this is the case.

@Franc : Are you maybe willing to share some insights of your "fix" (using veeam agent for windows) ? Or have you faced timouts in the meantime as well ?
FrancWest
Veteran
Posts: 522
Liked: 103 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: Snapshot removal issues of a large VM

Post by FrancWest »

Hi,

ever since we use the Veeam Agent for Windows the backups run without issues during work-hours. No timeouts at all. The only thing I notice is that the VSS snapshot phase of SQL takes about 15 minutes to complete. Haven't found a solution yet, but it's not that disturbing since the backups are working fine.
unlucky
Novice
Posts: 3
Liked: 2 times
Joined: Mar 02, 2017 1:47 pm
Full Name: unlucky
Contact:

Re: Snapshot removal issues of a large VM

Post by unlucky »

Hi,

ok, but as a result (afaik) you will lose direct storage integration as the agent communicates directly with the backupserver. I´ve tested it yesterday and it looks like the good old onhost backup on
Microsoft Hyper-V with 2K12. The bad thing is that you have to setup some kind of a new dedicated backup network, as we currently use storage snapshots and all backup traffic is handled in
the isolated iscsi network.
FrancWest
Veteran
Posts: 522
Liked: 103 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: Snapshot removal issues of a large VM

Post by FrancWest »

Correct, but why not add a NIC to the VM and connect it to the iSCSI network? That way you can configure the backup server to connect to the SQL VM using the iSCSI network also.
mholman
Novice
Posts: 4
Liked: never
Joined: Jun 24, 2019 11:07 pm
Full Name: IT Support
Contact:

Re: Snapshot removal issues of a large VM

Post by mholman »

I see a lot of post stating that it is normal VMware behavior to take some time to snap or remove snaps; however, we have run several tests using a small vm guest (300MB), and we have found that snaps and snap removals, when doing directly from vCenter, is consistently 1s (maybe 2s). However, when doing the same from Veeam, we see it consistently takes 1+ minutes - sometimes a few minutes. So, I am not in agreeance that this is a VMware issue.
veremin
Product Manager
Posts: 20376
Liked: 2290 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin »

In your test without Veeam, have you waited for the same time as it takes to perform backup before removing the snapshot - or you've removed it as soon as it was created, without giving it a chance to actually grow? Thanks!
mholman
Novice
Posts: 4
Liked: never
Joined: Jun 24, 2019 11:07 pm
Full Name: IT Support
Contact:

Re: Snapshot removal issues of a large VM

Post by mholman »

As mentioned, the vm guest is just a 300mb test vm. There were 151.4 KB worth of changes which should take no time to commit. The backup job within Veeam finishes about as quickly as it would take me to execute the commands manually within vCenter.
mholman
Novice
Posts: 4
Liked: never
Joined: Jun 24, 2019 11:07 pm
Full Name: IT Support
Contact:

Re: Snapshot removal issues of a large VM

Post by mholman »

Just to confirm, I just ran another manual test within vCenter with 40 min between snap and snap removal and the removal still just took 2s.
veremin
Product Manager
Posts: 20376
Liked: 2290 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin »

In this case it might be worth reaching our support team with the results of your findings - assuming the tests were conducted properly, you should have seen the same behaviour in both cases.

Thanks!
Gostev
Chief Product Officer
Posts: 31766
Liked: 7266 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Snapshot removal issues of a large VM

Post by Gostev »

Both vCenter and Veeam use the same exact API calls to create and remove snapshots, so there's got to be some other difference in these experiments. Perhaps you can demonstrate the same live to our support engineer, who can then review the debug logs to find the difference.

For example, depending on your backup mode, there are other operations that will delay the beginning of snapshot removal, like hot remove of the backed up disk, and the debug logs will reflect these operations. Regardless though, the actual snapshot removal should take the same time in both cases with very small snapshots, because this operation is managed by ESXi and is performed by the same exact ESXi function (Veeam has no control over it whatsoever).

But that is assuming clean experiment, of course. For example, if that test backup job continues on to processing other VMs while the snapshot is being removed, this will put additional load on the datastore, thus slowing down the snapshot removal. Anyway, it should be super easy to get to the bottom of this difference. There has not been a month in the past 12 years of Veeam existence without similar concerns, and every time we were able to demonstrate all doubting customers the consistency of snapshot removal performance regardless of how it is initiated :D
mholman
Novice
Posts: 4
Liked: never
Joined: Jun 24, 2019 11:07 pm
Full Name: IT Support
Contact:

Re: Snapshot removal issues of a large VM

Post by mholman »

Working with Veeam support team it was identified this issue occurs when using vvols and hotadd with the proxy and vm guest on different hosts. https://vresolv.wordpress.com/2017/12/0 ... ed-on-vvol. They did provide a registry entry that will cause it to wait for a specified amount of time for the unbind. Initial testing so far seems to indicate this to be a valid workaround.

There are a few things I like about vvols, but this issue combined with not being able to take advantage of "backup from storage snapshots" may push us back to managing vmfs datastores.
Gostev
Chief Product Officer
Posts: 31766
Liked: 7266 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Snapshot removal issues of a large VM

Post by Gostev »

Oh, I see where we had a disconnect. This entire thread is about VMFS actually... vVols is a totally different beast as it comes to VM snapshots handling, as it uses absolutely different architecture for those.
unlucky
Novice
Posts: 3
Liked: 2 times
Joined: Mar 02, 2017 1:47 pm
Full Name: unlucky
Contact:

Re: Snapshot removal issues of a large VM

Post by unlucky » 2 people like this post

Just as a follow up if someone else stumbles in since I still have an open case at Vmware and doing quite a lot of debugging with them.
Engineering Team is aware of that issue (Snapshot SEsparse disks) which should already be fixed in 6.7 latest build.
Unfortunately I cannot confirm as we are still running on 6.5. There should also be a fix released for 6.5 as well but no ETA yet.
preetam_de
Novice
Posts: 6
Liked: never
Joined: May 17, 2019 8:40 am
Full Name: preetam_de
Contact:

Re: Snapshot removal issues of a large VM

Post by preetam_de »

Have got any information from VMware support on this.
we are facing similiar issue.
Vitaliy S.
VP, Product Management
Posts: 27356
Liked: 2788 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Since this is a pretty long standing issue, I suggest you also contacting the VMware team, so that they could see what version of vSphere you're at and assist if possible.
isaez
Enthusiast
Posts: 29
Liked: 4 times
Joined: Aug 16, 2019 11:36 am
Full Name: Ivan Saez Scheihing
Contact:

[MERGED] Veeam 10 & VM backup

Post by isaez »

Hi,

We have a MSSQL 2016 VM server and when the VM is backuped the server becomes unreachable for a short period of time. It seems it happens when the snapshot is removed (Removing VM snapshot phase
which duration varies between 20 sec to 4 minutes. This happens during the night so most of the time no one notice it but some applications are busy during the night and generates errors where the MSSQL VM server is not reachable. Such an error is something like this:

020-10-23 00:05:53.723 +0200] [Error] Processing tasks failed: Liquit.DAL.DalUnexpectedException: Database error "Dal_Unexpected": A network-related or instance-specific error occurred while establishing a connection to SQL Server.

Is there a way (option) to avoid this unreachability?

regards,

Ivan
foggy
Veeam Software
Posts: 21133
Liked: 2140 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Hi Ivan, this behavior on snapshot commit is expected for highly transactional applications. Please review this thread for some typical recommendations for how to reduce this effect.
mdi02
Novice
Posts: 5
Liked: never
Joined: Feb 19, 2021 4:46 pm
Contact:

Re: Snapshot removal issues of a large VM

Post by mdi02 »

Hi,

We have an Oracle 19c Database on a Windows Server 2012 R2 VM Guest (with a Hyper-v server 2016) and we are having the same issue.
After the Veeam backup has started, we can find this event in the windows event viewer : Miniport NIC 'Microsoft Hyper-V Network Adapter' connected
Some critical processes are interrupted on our application (I guess during the snapshot removal).
I found a lot of tips on this topic but were not able to solve this issue yet.

Any help would be very appreciated, thanks!
foggy
Veeam Software
Posts: 21133
Liked: 2140 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

If none of the mentioned hints help, then faster storage is an option.
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 23 guests