Copy Job processing rate, v7 against v8

[lando_uk]

Hi

We have recently upgraded from v7 to v8. I have a query about Copy Job speeds.

With v7 it was reporting our offsite jobs were getting about 186MB/s with the Bottleneck: Source. Since upgrading, the same jobs goes at 36MB/s with Bottleneck: Target. Is this just a reporting issue?

Thanks
Mark

[foggy]

Mark, what kind of target storage are we talking about?

[lando_uk]

R720xd, RAID6 12 x 4TB NLSAS

[PTide]

Wow, seems that your copy job has to last 5-6 times longer than before. Does it?

[foggy]

Were job/repository compression settings left intact after the upgrade? Do you have the latest Veeam B&R update installed (Update 2)?

[lando_uk]

Well its hard to say, I only have 1 or 2 v8 logs so far, and the day after the upgrade there was lots of failed copy reports spewed out until it sorted itself out.

We have active-active datacenters, (A and B) The offsite jobs from A to B seems to be slower than the offsites from B to A. (A is the B&R server)

After the upgrade, I didn't restart any of the other proxies or repositories, only the B&R server itself - would I need to restart the other servers?

Yes, I installed U2b.

[foggy]

No, automatic components upgrade should be enough. If none of the settings were changed and you feel the jobs started to run slower, I recommend to contact support to take a look.

[lando_uk]

Well i'll let it settle down, we only keep a small amount of restore points offsite, so once those have aged I'll investigate further.

Compression for these jobs has been set to high, they haven't changed - I'm just wondering if the v7 reporting didn't take into consideration the compression so reported higher speeds - Kind of like when you work a thin provisioned disk, the reported speed ramps up because its not measuring real network speed but something else.

[foggy]

As far as I know, nothing has changed in terms of reporting processing rates between v7 and v8

[lando_uk]

Out of interest - On my copy jobs, they are set to copy every 1 day, starting at 06:00 - My backups start at midnight and normally finish at 03:00 - I don't want copy jobs interfering with backup jobs as my storage only has so many spare IOPs - do I use the schedule to manage this or the job start time?

[PTide]

You can set your copy job to start in a couple of minutes after your backup job (0:15 for example). In such case your backup copy job will wait until your backup is done.

[lando_uk]

I have 8 backup jobs that piggy back off the 1st midnight one though, 300+ VM's in total, and I don't want Copy Job 1 kicking off when Backup Job 2 starts. Id rather they all start when all the backup jobs are complete.

[PTide]

A few questions to make sure that I got it right:

I don't want copy jobs interfering with backup jobs as my storage only has so many spare IOPs

Which storage did you mean - the one under backup jobs' repo, the one under backup copy jobs' repo, or production storage? Or maybe both backup and copy jobs use the same storage as a repo? Please give more details.

I have 8 backup jobs that piggy back off the 1st midnight one though

I have some doubts regarding the meaning of "piggybacking" in this context. Do you use job-chaining? Please clarify.

Thank you.

[lando_uk]

I mean, when my repository storage is busy backing up, I don't want the same storage getting hammered by copy jobs transferring the latest restore point offsite. That would slow down the backups and transforms. I want copies to get processed when all primary jobs are finished.

Yes, I mean job-chaining, all my jobs follow on from the first that starts a Middnight.

[PTide]

Ok, is that an assumption that backup copy reads will cause a significant overhead on storage, or you've tested that already?

Thank you.

[lando_uk]

Its an assumption that the storage will work better doing either reads or writes, but doing both at the same time will slow things down. This is a cheap RAID6 DAS, not a SAN that can do 10,000 random IOPs.

Well, things are definitely slower that they were on v7, so tomorrow I'm going to delete a copy job, change the Compression to Optimal and then re-seed it, see if goes back to 100MB/s.

[PTide]

I'd recommend to perform a test to see if it's really that bad for storage to run copy and backup jobs simultaneously. If the performance will be really poor then there is an option to setup a Backup Window so your copy job will transfer data only within allowed window.

Regarding your original post "v7 against v8" kindly open a ticket with support as it is an unexpected behaviour.

Thank you.

[lando_uk]

Hi, this morning I deleted the copy job from disk, changed the compression to Optimal (it was set to High) and then kicked off the job again. Its now running at 135MB/s (so that's line speed for our WAN) - So it seems that Compression has changed in v8, as I used to get 135MB/s with High in v7.

[PTide]

So it seems that Compression has changed in v8, as I used to get 135MB/s with High in v7.

That's strange. Compression has changed in v7.

I'm really curious about several things:

1. Could you please try setting compression back to High and run the job again to see if there will be a decrease in speed? Just to make sure.
2. In you original post you've mentioned 186 Mb/s speed. How did you get that speed if your WAN is 135MB/s ?

Thank you.

[lando_uk]

When speeds go greater than the 1Gbe WAN, I presume the reporting isn't measuring the real speed, its just measuring something like `vm size/time to complete`. So jobs with good compression/thin disks etc report speeds over 1Gbe.

I'll leave this job running, as it's production data and offsite copies we are working with, so I can't afford to have no copies for too long. I can only really delete and re-seed one job a day due to the size and copy window, but tomorrow when I do the next job, I'll do as you suggest and delete/re-seed using High compression to check speeds.

[lando_uk]

See what I mean here.

I have now reset jobs 1 and 2 with Optimal compression. From the GUI it says between them, they are transferring at 185MB/s - yet windows task manager telling a different story and is sending at roughly 1Gbe. (WAN line speed)

[lando_uk]

More info:



Here is the screenshot from Veeam One showing network throughput on the source repository.

Yesterdays figure is the speed of the 2 new jobs that I re-seeded with Optimal compression, they have a respectable speed (the WAN link also does other stuff) and finished at 9pm. Today, starting at 4am is the speed of the existing jobs that I haven't touched, they barely hit 25MB/s and are very bursty in nature compared to the more stable stream of yesterdays.

Also, from the CPU charts on the target repository, it looks like the compression takes place on the target after its sent, rather than on the source host - Surely it would save time and bandwidth compressing before being sent?

[PTide]

So, you mean that the jobs that were seeded yesterday with "Optimal" compression showed good speed of 180Mb/s, while other jobs that stayed with "High" compression showed a x7 times lower speed, showing target repository as a bottleneck, and CPU on target is loaded during those jobs, is that correct?

Do you use any WAN accelerators, by any chance?

[lando_uk]

Hi

Mostly correct, I wouldn't say x7 slower, as looking at real network charts its more like x3 slower.

If 3x slower is normal for High Compression, then I'll accept that and have to work around it, but v7 certainly wasn't x3 slower, so something has changed.

CPU wise, its not overloaded as the target has 24 cores, but it seems that the VeeamAgent is single threaded as just 1 of the cores is busy (about 75%) when 1 job is running with High compression. Going back in time on the CPU chart to before the v8 upgrade, CPU usage is generally much lower with the same settings.

So to me this says that High compression in v7 was much less resource intensive, is High compression in v8 much better than in v7 to warrant this impact?

[PTide]

Hi

CPU wise, its not overloaded as the target has 24 cores, but it seems that the VeeamAgent is single threaded as just 1 of the cores is busy (about 75%) when 1 job is running with High compression. Going back in time on the CPU chart to before the v8 upgrade, CPU usage is generally much lower with the same settings.

Compression should occur on the source side, therefore the situation you've explained is a valid reason to open a case with support team.

So to me this says that High compression in v7 was much less resource intensive, is High compression in v8 much better than in v7 to warrant this impact?

There were no changes in compression from v7 to v8 so the described behaviour is not normal.

With all said, I suggest you to open a ticket with our support team for a deeper investigation. Please don't forget to post you case ID here.

Thank you,

[lando_uk]

Case ID 01040163.

[lando_uk]

Well, still no further resolving this even though Support has seen the issue and agrees something is wrong. Tried turning off DisablePublicIPTrafficEncryption in the reg, but that made no difference.

Case still open...

[PTide]

Hi Mark,

Kindly keep working with support team. Lets see if there will be any outcome from tomorrow morning session. If nothing then the escalation might be needed.

Thank you.