VEEAM backup suddenly taking lots more time

[Niek]

Hello All,

Our veeam backup started posting a very weird result since our maintenance weekend at the 31st of May. It started with our Exchange suddenly taking way longer to backup (see data in Job Data) and since 2 days our small database server is starting to show the same symptoms. I have no clue as to why they suddenly do this. So i'm hoping posting here will help solve this issue.

what's changed
This phenomena is happening to 2 of our servers. The first one (Exchange 2010) is showing these symptoms since 1st of June.
At the 31st of may we did a major update round on all of our servers. The Exchange started showing these symptoms right after the patches.

Job Data
Exchange Server:
I cannot find the sizes for the Exchange server in the history that far back. So i can't provide those.
Previous Time: 7 Minutes | New Time: (02/06) 42 Minutes | Last time: (yesterday) : 1 Hour 15 Minutes.
Previous Processing Rate: 436MB/s | New Processing Rate (02/06): 101MB/s | New Processing Rate (Yesterday): 60MB/s

Server 2 (small database server):
Previous Size: 782MB | Previous Time: 7 Minutes
New Size: 1GB | New Time: 26 Minutes.
Previous Processing Rate: 352 MB/s | New Processing Rate: 173MB/s

The rate is slowed down by those machines only. Server 2 only started showing this issue 2 days ago.
I also noticed that our Quick-Ping monitor is losing pings during the end of the backup job. This also started happening when the backup time went up.
Before the 1st june we didn't have any problems with our veeam backups. But after the maintenance weekend suddenly this issue popped up and i have no clue as to what is causing it.

If i left out some information i will provide it

Thanks in advance.

Niek

[Gostev]

If i left out some information i will provide it

Yes and the most important one, bottleneck stats

[Deon]

Hi!

I also noticed that our Quick-Ping monitor is losing pings during the end of the backup job. This also started happening when the backup time went up.

This may happen because the job now runs much slower, so the snapshot grows bigger and the snapshot removal procedure suspends the VM for a noticeable amount of time.

As dear Gostev said, the first thing you need to check is the reported bottleneck.
By the way, 436MB/s sounds crazy, it's almost like it had a huge amount of unused disk space which was simply skipped during deduplication. So it would be cool if you could fire up a report for your job and check the amount of data it was processing before in comparison to the new one.

Also just an idea, after these updates the amount of disk changes per time could greatly increase due to some software changes, so maybe before you used to process very small increment, but now between job runs the exchange accumulates much more data to copy.

The important thing to compare here is - are full job runs (NB! Not synthetic fulls!) experiencing the same behavior? Or does the Active Full take the same amount of time and only increments take much longer? The latter would mean that you simply have a huge amount of changes on your disk between job runs, caused by something after the update (often security/antivirus software may be a root cause of it as well if they run scans towards the virtual drive).

[Niek]

Hi Guys,

Thanks for the replies. Here is the information

The bottleneck is reported as follows:
Before Updates:
Source: 97/98%
Proxy: 60/75/78/80%
Network:2/3/5%
Target: 2/3%

Yesterday:
Source: 98/99%
Proxy: 60/64/77/79%
Network: 0/1/6%
Target: 1%

The bottleneck hasn't changed very much. It always has been the source.

This may happen because the job now runs much slower, so the snapshot grows bigger and the snapshot removal procedure suspends the VM for a noticeable amount of time.

That must be it. Since it only happens to the machines which are processed very slowly.

Also just an idea, after these updates the amount of disk changes per time could greatly increase due to some software changes, so maybe before you used to process very small increment, but now between job runs the exchange accumulates much more data to copy.

Yes we do use incremental backups. But wouldn't the huge amount of data from updates only be a 1-time thing? it's not like the servers are restarted daily for updates. And why would this explain that the job slowly starts degrading even more?

The important thing to compare here is - are full job runs (NB! Not synthetic fulls!) experiencing the same behavior?

I cannot check this. Currently we only have incremental backups on veeam. The previous back-up administrator never used veeam for full backups. I could create a job to see what would happen, but would that have any impect on the issue i'm facing here? I mean, would a new job also suffer from these performance drops?

I checked the report but i didn't see any weird things that caught my eye. Like i said in my opening post the data being processed hasn't grown that much.
I uploaded the reports from before the maintenance and yesterday. Maybe i missed something. report.

[veremin]

Can you also check the what transport mode was used by the proxy server before and after slow down? As to full backup, there is no need to create additional job - you can right-click on existing job and select active full option. Thanks.

[Niek]

Can you also check the what transport mode was used by the proxy server before and after slow down? As to full backup, there is no need to create additional job - you can right-click on existing job and select active full option. Thanks.

Hi,

Here are the details for the proxy:
Server: Veeam
Transport: Automatic selection
Connected Datastore: Auto (Recommend)
Max concurend tasks: 2

This has never been changed.

I will run the active-full backup this weekend and post the results here on Monday.

[veremin]

Please, check the job session logs (previous and current one) and see what proxy mode was used? Select the particular VM to the left, and locate the transport mode ([san], [hotadd], or [nbd]). Thanks.

[Niek]

The proxy mode was hotadd.

[foggy]

Yes we do use incremental backups. But wouldn't the huge amount of data from updates only be a 1-time thing? it's not like the servers are restarted daily for updates. And why would this explain that the job slowly starts degrading even more?

This is not the case, according to the reports you've provided, the Read and Transferred counters are in accordance with what they were prior the updates.

Could you please elaborate on the actual maintenance you've performed with this servers? What tasks did it include? I'm also interested in active full results...

[Niek]

Could you please elaborate on the actual maintenance you've performed with this servers? What tasks did it include?

During the maintenance the following has been performend:

Exchange:
increasing cpu cores to 4
Increasing memory to 8GB
Windows update
SP installation
Reboot(s)

Small Database Server (Filemaker):
Just a restart. We are slowly migrating to a new 2012 server with the newest version of the program.

In total we updateded and restarted: 15 servers
Old servers with just a reboot: 16 servers.

Other things (unrelated): Changing patches; cleaning up cable mess etc.

[Niek]

Good morning,

The full backups have been processed and here are the results.
Exchange:
msx2010 : 2:17:22
and in the details i find:
Removing VM snapshot - 1:06:20 (also a quick ping due to this)
(this job only runs on weekdays so no new normal data to compare)

Load: Source: 98% > Proxy 96% > Network 9% > Target 5%

Filemaker:
Sunday Night 3:00 : 25 minutes (19 minutes of removing snapshot)
Sunday Evening (Full backup) : 9:35 minutes
This night 3:00 : 4:46 minutes

Load (Full): Source: 92% > Proxy 90% > Network 19% > Target 12%

We didn't receive a quick-ping this time. I'm curious to see what it will do during the rest of the week, since it now looks alright.

[Niek]

The filemaker server hasn't reported any long backup times or time outs. I don't know what the full backup changed, but it works.
I noticed however that the job however took 3 minutes longer to complete while the amount of data remained the same. I'll keep an eye on this to be sure it doesn't go 'bad' again.

The Exchange is still an issue however, any thoughts on that? See my previous post for the Full backup info.

[Vitaliy S.]

Windows update
SP installation

This could have caused many block changes on the Exchange server and that is why the time required to back it up went up. Also I see that you have pretty similar bottleneck stats for both source and proxy, so I would suggest giving more CPU power to the proxy server or lower the compression level for the backup job in order to get better job performance.

[Niek]

This could have caused many block changes on the Exchange server and that is why the time required to back it up went up.

It is true that would have caused more data. And that backups times increase after updates is normal. But by this level?
If i take a look at the history:
30/5 data transferred: 12.5GB
Removing VM Snapshot: 2 minutes.

Yesterday:
Transferred: 13.8GB
Removing VM Snapshot: 1 hour 18 minutes 35 seconds.

See also the compairison by screenshot: (click on thumbnail for full image)
vs

Also I see that you have pretty similar bottleneck stats for both source and proxy, so I would suggest giving more CPU power to the proxy server or lower the compression level for the backup job in order to get better job performance.

That is true. But the previous backup times were very acceptable. And i don't see why all of sudden i should make a more powerful backup host. It is the removing VM snapshot that is taking all the time. A full backup somehow fixed it for filemaker. Not i'd like to see it fixed (and find the cause) for our Exchange host to.

Thanks.

[Vitaliy S.]

Yes, I agree that it is strange to see that difference between two similar snapshot commit operations, however Veeam does not commit the snapshot, it just sends the command to VMware to do that. Once snapshot is committed, VMware reports back to Veeam and we proceed to the next operation.

There is a couple of reasons that can lead to this behavior, so I would recommend starting your investigation from that list:

1. Snapshot grows large because of the changes written to the virtual disk
2. Large snapshot tree on the backed up VM
3. Heavy load on the datastore that hosts the VM in question

Thanks!