Host-based backup of VMware vSphere VMs.
Post Reply
Niek
Novice
Posts: 8
Liked: never
Joined: Jun 26, 2014 10:53 am
Full Name: Niek Slenders
Contact:

VEEAM backup suddenly taking lots more time

Post by Niek »

Hello All,

Our veeam backup started posting a very weird result since our maintenance weekend at the 31st of May. It started with our Exchange suddenly taking way longer to backup (see data in Job Data) and since 2 days our small database server is starting to show the same symptoms. I have no clue as to why they suddenly do this. So i'm hoping posting here will help solve this issue.

what's changed
This phenomena is happening to 2 of our servers. The first one (Exchange 2010) is showing these symptoms since 1st of June.
At the 31st of may we did a major update round on all of our servers. The Exchange started showing these symptoms right after the patches.

Job Data
Exchange Server:
I cannot find the sizes for the Exchange server in the history that far back. So i can't provide those.
Previous Time: 7 Minutes | New Time: (02/06) 42 Minutes | Last time: (yesterday) : 1 Hour 15 Minutes.
Previous Processing Rate: 436MB/s | New Processing Rate (02/06): 101MB/s | New Processing Rate (Yesterday): 60MB/s

Server 2 (small database server):
Previous Size: 782MB | Previous Time: 7 Minutes
New Size: 1GB | New Time: 26 Minutes.
Previous Processing Rate: 352 MB/s | New Processing Rate: 173MB/s

The rate is slowed down by those machines only. Server 2 only started showing this issue 2 days ago.
I also noticed that our Quick-Ping monitor is losing pings during the end of the backup job. This also started happening when the backup time went up.
Before the 1st june we didn't have any problems with our veeam backups. But after the maintenance weekend suddenly this issue popped up and i have no clue as to what is causing it.

If i left out some information i will provide it

Thanks in advance.

Niek
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Gostev »

Niek wrote:If i left out some information i will provide it
Yes and the most important one, bottleneck stats ;)
Deon
Veeam Software
Posts: 25
Liked: 9 times
Joined: Jun 26, 2014 7:02 pm
Full Name: Denis Churaev
Location: Bucharest, Romania
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Deon »

Hi!

I also noticed that our Quick-Ping monitor is losing pings during the end of the backup job. This also started happening when the backup time went up.
This may happen because the job now runs much slower, so the snapshot grows bigger and the snapshot removal procedure suspends the VM for a noticeable amount of time.

As dear Gostev said, the first thing you need to check is the reported bottleneck.
By the way, 436MB/s sounds crazy, it's almost like it had a huge amount of unused disk space which was simply skipped during deduplication. So it would be cool if you could fire up a report for your job and check the amount of data it was processing before in comparison to the new one.

Also just an idea, after these updates the amount of disk changes per time could greatly increase due to some software changes, so maybe before you used to process very small increment, but now between job runs the exchange accumulates much more data to copy.

The important thing to compare here is - are full job runs (NB! Not synthetic fulls!) experiencing the same behavior? Or does the Active Full take the same amount of time and only increments take much longer? The latter would mean that you simply have a huge amount of changes on your disk between job runs, caused by something after the update (often security/antivirus software may be a root cause of it as well if they run scans towards the virtual drive).
Niek
Novice
Posts: 8
Liked: never
Joined: Jun 26, 2014 10:53 am
Full Name: Niek Slenders
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Niek »

Hi Guys,

Thanks for the replies. Here is the information :)

The bottleneck is reported as follows:
Before Updates:
Source: 97/98%
Proxy: 60/75/78/80%
Network:2/3/5%
Target: 2/3%

Yesterday:
Source: 98/99%
Proxy: 60/64/77/79%
Network: 0/1/6%
Target: 1%

The bottleneck hasn't changed very much. It always has been the source.
This may happen because the job now runs much slower, so the snapshot grows bigger and the snapshot removal procedure suspends the VM for a noticeable amount of time.
That must be it. Since it only happens to the machines which are processed very slowly.
Also just an idea, after these updates the amount of disk changes per time could greatly increase due to some software changes, so maybe before you used to process very small increment, but now between job runs the exchange accumulates much more data to copy.
Yes we do use incremental backups. But wouldn't the huge amount of data from updates only be a 1-time thing? it's not like the servers are restarted daily for updates. And why would this explain that the job slowly starts degrading even more?
The important thing to compare here is - are full job runs (NB! Not synthetic fulls!) experiencing the same behavior?
I cannot check this. Currently we only have incremental backups on veeam. The previous back-up administrator never used veeam for full backups. I could create a job to see what would happen, but would that have any impect on the issue i'm facing here? I mean, would a new job also suffer from these performance drops?

I checked the report but i didn't see any weird things that caught my eye. Like i said in my opening post the data being processed hasn't grown that much.
I uploaded the reports from before the maintenance and yesterday. Maybe i missed something. report.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by veremin »

Can you also check the what transport mode was used by the proxy server before and after slow down? As to full backup, there is no need to create additional job - you can right-click on existing job and select active full option. Thanks.
Niek
Novice
Posts: 8
Liked: never
Joined: Jun 26, 2014 10:53 am
Full Name: Niek Slenders
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Niek »

v.Eremin wrote:Can you also check the what transport mode was used by the proxy server before and after slow down? As to full backup, there is no need to create additional job - you can right-click on existing job and select active full option. Thanks.
Hi,

Here are the details for the proxy:
Server: Veeam
Transport: Automatic selection
Connected Datastore: Auto (Recommend)
Max concurend tasks: 2

This has never been changed.

I will run the active-full backup this weekend and post the results here on Monday.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by veremin »

Please, check the job session logs (previous and current one) and see what proxy mode was used? Select the particular VM to the left, and locate the transport mode ([san], [hotadd], or [nbd]). Thanks.
Niek
Novice
Posts: 8
Liked: never
Joined: Jun 26, 2014 10:53 am
Full Name: Niek Slenders
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Niek »

The proxy mode was hotadd.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by foggy »

Niek wrote: Yes we do use incremental backups. But wouldn't the huge amount of data from updates only be a 1-time thing? it's not like the servers are restarted daily for updates. And why would this explain that the job slowly starts degrading even more?
This is not the case, according to the reports you've provided, the Read and Transferred counters are in accordance with what they were prior the updates.

Could you please elaborate on the actual maintenance you've performed with this servers? What tasks did it include? I'm also interested in active full results...
Niek
Novice
Posts: 8
Liked: never
Joined: Jun 26, 2014 10:53 am
Full Name: Niek Slenders
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Niek »

Could you please elaborate on the actual maintenance you've performed with this servers? What tasks did it include?
During the maintenance the following has been performend:

Exchange:
increasing cpu cores to 4
Increasing memory to 8GB
Windows update
SP installation
Reboot(s)

Small Database Server (Filemaker):
Just a restart. We are slowly migrating to a new 2012 server with the newest version of the program.

In total we updateded and restarted: 15 servers
Old servers with just a reboot: 16 servers.

Other things (unrelated): Changing patches; cleaning up cable mess etc.
Niek
Novice
Posts: 8
Liked: never
Joined: Jun 26, 2014 10:53 am
Full Name: Niek Slenders
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Niek »

Good morning,

The full backups have been processed and here are the results.
Exchange:
msx2010 : 2:17:22
and in the details i find:
Removing VM snapshot - 1:06:20 (also a quick ping due to this)
(this job only runs on weekdays so no new normal data to compare)

Load: Source: 98% > Proxy 96% > Network 9% > Target 5%

Filemaker:
Sunday Night 3:00 : 25 minutes (19 minutes of removing snapshot)
Sunday Evening (Full backup) : 9:35 minutes
This night 3:00 : 4:46 minutes

Load (Full): Source: 92% > Proxy 90% > Network 19% > Target 12%

We didn't receive a quick-ping this time. I'm curious to see what it will do during the rest of the week, since it now looks alright.
Niek
Novice
Posts: 8
Liked: never
Joined: Jun 26, 2014 10:53 am
Full Name: Niek Slenders
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Niek »

The filemaker server hasn't reported any long backup times or time outs. I don't know what the full backup changed, but it works.
I noticed however that the job however took 3 minutes longer to complete while the amount of data remained the same. I'll keep an eye on this to be sure it doesn't go 'bad' again.

The Exchange is still an issue however, any thoughts on that? See my previous post for the Full backup info.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Vitaliy S. »

Niek wrote:Windows update
SP installation
This could have caused many block changes on the Exchange server and that is why the time required to back it up went up. Also I see that you have pretty similar bottleneck stats for both source and proxy, so I would suggest giving more CPU power to the proxy server or lower the compression level for the backup job in order to get better job performance.
Niek
Novice
Posts: 8
Liked: never
Joined: Jun 26, 2014 10:53 am
Full Name: Niek Slenders
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Niek »

Vitaliy S. wrote:This could have caused many block changes on the Exchange server and that is why the time required to back it up went up.
It is true that would have caused more data. And that backups times increase after updates is normal. But by this level?
If i take a look at the history:
30/5 data transferred: 12.5GB
Removing VM Snapshot: 2 minutes.

Yesterday:
Transferred: 13.8GB
Removing VM Snapshot: 1 hour 18 minutes 35 seconds.

See also the compairison by screenshot: (click on thumbnail for full image)
Image vs Image
Vitaliy S. wrote:Also I see that you have pretty similar bottleneck stats for both source and proxy, so I would suggest giving more CPU power to the proxy server or lower the compression level for the backup job in order to get better job performance.
That is true. But the previous backup times were very acceptable. And i don't see why all of sudden i should make a more powerful backup host. It is the removing VM snapshot that is taking all the time. A full backup somehow fixed it for filemaker. Not i'd like to see it fixed (and find the cause) for our Exchange host to.

Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VEEAM backup suddenly taking lots more time

Post by Vitaliy S. »

Yes, I agree that it is strange to see that difference between two similar snapshot commit operations, however Veeam does not commit the snapshot, it just sends the command to VMware to do that. Once snapshot is committed, VMware reports back to Veeam and we proceed to the next operation.

There is a couple of reasons that can lead to this behavior, so I would recommend starting your investigation from that list:

1. Snapshot grows large because of the changes written to the virtual disk
2. Large snapshot tree on the backed up VM
3. Heavy load on the datastore that hosts the VM in question

Thanks!
Post Reply

Who is online

Users browsing this forum: Google [Bot], Gostev, Semrush [Bot] and 86 guests