Veeam B&R jobs performing extremely slowly

[r.mckeon]

I have a small virtual setup which I'm backing up using Veeam B&R 7.0, utilizing both backup jobs and replication jobs. Here's a summary:

- 2 ESX 5.5 hosts (a file server and Exchange server on one, an SQL server and the Veeam VM on the other)

- Production storage is a Synology DS1513+ with four WD Red drives configured in RAID 10. iSCSI LUNs are used to mount datastores in the ESXs

- Backup storage for the Veeam Backup jobs are external USB hard disks outside the server room, targets are CIFS folders on the external hard disks

- Backup storage for the Veeam Replication jobs is another Synology DS1513+, targets are also iSCSI LUNs

- Switches are all gigabit per port

I know it's not a high-end set up, in fact the backup rate has never been higher than 20MB/s, and is usually between 7 - 17 MB/s. Very slow I know, but acceptable for this small setup as in all the backups only end up taking about 4 hours.

BUT, as of late the speed of the backups/replications has gone down dramatically to around 3MB/s, with Veeam showing the bottleneck as being the source (I assume it's referring to the production storage). In all other activities the Synology is performing well, but when Veeam performs backups and replication jobs in the past few days the read rate is extremely poor.

If it's of any help, I'm using a default backup proxy in Veeam with automatic selection. I didn't see the need for fine tuning the proxy as the scenario is so small and it's been working fine as is for the past two months. It's only in the last few days that the speed has dropped drastically. Also, the speed is especially poor when replicating the Exchange VM.

Anyone else experience something similair?

[Vitaliy S.]

Hi Rick,

What doe latency/read/write counters show for this datastore during normal and backup hours? Did you have source as the bottleneck during all runs even when the backup job performance was about 20 MB/s?

Thanks!

[r.mckeon]

Hi Rick,

What doe latency/read/write counters show for this datastore during normal and backup hours? Did you have source as the bottleneck during all runs even when the backup job performance was about 20 MB/s?

Thanks!

Source has always been the bottleneck, at 99%. The rest of the counters are very low percentages ranging from 0 - 10 % but never higher. It has always been this way but only recently the processing rate has halved

[foggy]

What transport mode was used by the proxy to read the data from the storage prior and after slow down? Were there probably any changes to the environment introduced at the moment you started experiencing slow backup performance?

[r.mckeon]

What transport mode was used by the proxy to read the data from the storage prior and after slow down? Were there probably any changes to the environment introduced at the moment you started experiencing slow backup performance?

No changes were made, and the Veeam has been configured the same way since day one. The transport mode is set to automatic, though I'm not sure how to confirm which mode is actually being used.

[foggy]

You can check this by opening the job session log, selecting the particular VM to the left, and locating the transport mode tag ([san], [hotadd], or [nbd]) next to the name of the proxy server assigned for the processing.

[r.mckeon]

You can check this by opening the job session log, selecting the particular VM to the left, and locating the transport mode tag ([san], [hotadd], or [nbd]) next to the name of the proxy server assigned for the processing.

NBD is being used in all cases, and always has been used.

[foggy]

And what about datastore counters mentioned by Vitaliy above?

[r.mckeon]

And what about datastore counters mentioned by Vitaliy above?

I'm sorry I don't understand the question. Are you asking how does the production storage latency/speeds compare during normal and backup hours? Much faster during normal hours, if you want I can try and give you specific data, but the point is that before the backups were reaching approx 20MB/s so it's proven that the production storage is capable of those speeds.

[Vitaliy S.]

Yes, I just wanted to know what is the datastore performance during the backup job run. If job performance has degraded, then you need to check what has changed when you first noticed this slow-down. Did you adjust your jobs schedule by any chance? What about network switches that your ESXi hosts are connected to? Could they probably fail over to 100Mb mode?

BTW, what operation takes most of the time according to the job log session?

[r.mckeon]

Again, nothing changed, jobs have been exactly the same since 2 months now. Storage has also been performing normally during normal hours. Network switches might be failing over to 100mb mode BUT Veeam shows Source as being the bottleneck, and also even if the switches fail over to 100mb, would that mean 3MB/s processing rate on jobs? It's an extremely slow speed.

The longest action during the job is the reading of the backup. There is a [CBT] tag next to the action. I can also see that the transport mode is failing over from hotadd to nbd...if I set this to nbd explicitly will it make a difference?

[Gostev]

even if the switches fail over to 100mb, would that mean 3MB/s processing rate on jobs?

That is absolutely correct, because management interface of ESXi (which is what NBD uses) is throttled by VMware. 3MB/s is exactly the processing rate I would expect for NBD over 100 Mbps.

On a separate note, if you'd like to keep this topic open, please include Veeam support case ID for this issue. This is mandatory, as explained when you click New Topic. Thank you for understanding!

[r.mckeon]

I don't have a support case ID...

FYI, I'm currently running an active full backup of one of the jobs. My jobs are reverse incremental, and I understand that every so often an active full backup is recommended. So I began running an active full backup to see if it makes any difference. Right now the processing rate is 6MB/s, Bottleneck is Source 99% and Proxy 11%, Network and Target are 0%. I'm backing up an Exchange VM from a Synology production storage with iSCSI LUNs, and the target is a USB hard drive on the same LAN. Compression is set to optimal and transport mode is ndb. In 1 hour it has only managed to process 21GB of data. I'm also checking my production storage and the load is minimal though Read rate is approx. 6MB/s.

[dellock6]

To be honest, production performances of a storage has little to do with backup performances.
Out of 100% size of the VM, during the day only 5-10% of it is read, the OS is almost always loaded in memory, so the performances are good.
When it comes to backups, especially full ones, Veeam retrieves "every" block of the disk, that is 100% of it. This is a completely different I/O pattern than what happens when the VM is executed.

Just to check, can you please go back in history and see one of the backup jobs of two months ago, when you say performances where good? I'd like to see the difference in performances between the two.

Also, you say no changes happened to the backup configuration, but what about the production storage? New VMs deployed, maybe with some scheduled activities during the same hours of the backup?

[r.mckeon]

On a separate note, if you'd like to keep this topic open, please include Veeam support case ID for this issue. This is mandatory, as explained when you click New Topic. Thank you for understanding!

I opened a support case...this is the ID: 00588225

[r.mckeon]

Just to check, can you please go back in history and see one of the backup jobs of two months ago, when you say performances where good? I'd like to see the difference in performances between the two.

Also, you say no changes happened to the backup configuration, but what about the production storage? New VMs deployed, maybe with some scheduled activities during the same hours of the backup?

I go back in history and check what exactly? The difference is that the processing rate used to be around 20MB/s and then it dropped to 3MB/s a few days ago. This week was just a normal business week, no changes at all, just normal production use.

Believe me when I say, no major changes were made before the drop in performance, no new VMs, no changes in configuration on ANYTHING, no increased workload.

[Vitaliy S.]

I can also see that the transport mode is failing over from hotadd to nbd...if I set this to nbd explicitly will it make a difference?

No, it won't, but I wonder if you had hotadd backup mode working previously or not...

I go back in history and check what exactly? The difference is that the processing rate used to be around 20MB/s and then it dropped to 3MB/s a few days ago. This week was just a normal business week, no changes at all, just normal production use.

I would recommend looking at the operation that takes most of the time for the jobs. If the timing is the same, then you should review other things such as change rate/transferred data/backup mode etc. You can ask our support team for assistance with logs review.

[r.mckeon]

No, it won't, but I wonder if you had hotadd backup mode working previously or not...

No I had never configured for hotadd.

I would recommend looking at the operation that takes most of the time for the jobs. If the timing is the same, then you should review other things such as change rate/transferred data/backup mode etc. You can ask our support team for assistance with logs review.

I submitted my logs to your support team on Friday. Still no answer as of yet.

[foggy]

Rick, please expect reply from support soon, I've asked them to look into your case.

[r.mckeon]

Ok I noticed something interesting. So the Veeam is running on a VM on ESX02. When backing up VMs which are on ESX01, the processing rate is around 3MB/s. When backing up VMs which are on ESX02, the processing rate is around 9 or 10 MB/s. Still very slow, and still a drop in performance from what it was, but my point is it's much faster when backing up VMs which are on the same ESX server.

This is what's interesting: Different ESX servers, different speeds...but SAME production storage. Could this be pointing to a network or VMware issue?

[Vitaliy S.]

When backing up VMs which are on ESX02, the processing rate is around 9 or 10 MB/s. Still very slow, and still a drop in performance from what it was, but my point is it's much faster when backing up VMs which are on the same ESX server.

I assume hotadd backup mode could be used for the VMs located locally to the proxy server, but you say you're using the same storage for both hosts...

This is what's interesting: Different ESX servers, different speeds...but SAME production storage. Could this be pointing to a network or VMware issue?

The easiest way to check that would be to migrate one VM that has low backup speeds to ESX02 and then run the job and compare the results. If the performance of the job goes up, then you should look at ESX01 configuration and especially it's network configuration (NICs, connection to switches etc.).

[r.mckeon]

The easiest way to check that would be to migrate one VM that has low backup speeds to ESX02 and then run the job and compare the results. If the performance of the job goes up, then you should look at ESX01 configuration and especially it's network configuration (NICs, connection to switches etc.).

Yes that makes sense. To be honest, it's easier said than done, but I will see what I can do. Definitely I will have to wait for a weekend to do it. In the meantime I will wait for the support's reply re the logs which I sent them.

[foggy]

I also wonder why does the job fail over to network and not uses hotadd, since everything that is required for that is proxy server access to the storage. Probably, some of the hotadd limitations take effect.

[r.mckeon]

I also wonder why does the job fail over to network and not uses hotadd, since everything that is required for that is proxy server access to the storage. Probably, some of the hotadd limitations take effect.

The requirements mention this: The host running backup proxy server VM must have all datastores where protected VMs' disks reside connected to it, otherwise hot add will not be possible

This isn't true for the VMs on the other host, as I didn't connect the iSCSI LUNs to both hosts to avoid problems. In fact, hotadd works for the VM which resides on the same host as the Veeam VM. I guess that explains the difference in performance between the different backup jobs, but it doesn't explain the overall drop in performance, which can be observed on ALL backup jobs.

[Vitaliy S.]

I guess that explains the difference in performance between the different backup jobs, but it doesn't explain the overall drop in performance, which can be observed on ALL backup jobs.

Yes, that's correct, and actually I assumed that you DO use shared storage for your hosts. Your second job most likely is using hotadd, and the first one is set to network mode. Given all that, there is no need to migrate VMs to the second host, as they will be backed up via hotadd and performance will be higher.

I would suggest waiting till our engineer finds out what is the difference between jobs with acceptable and low performance and then investigate network/ESXi configuration.

[foggy]

Also, comparing job sessions prior and after the slowdown would help (you can find the operation that started to take more time than it took previously). That is actually what Luca was asking about.

[r.mckeon]

Also, comparing job sessions prior and after the slowdown would help (you can find the operation that started to take more time than it took previously). That is actually what Luca was asking about.

I did compare but couldn't find any differences other than the actual backing up process of the VMDKs is much slower. I am communicating with support personnel now to arrange a remote session. Thanks for your assistance and I'll keep this thread updated.

[benthomas]

I'd be interested to hear what the outcome of all this was?
Also, was the production storage nearing capacity as the slowdown occurred?

[foggy]

According to the case (now closed), Rick was suggested to try using hotadd transport mode to get faster performance. Let's see whether he comes back to provide the results of his findings.