Comprehensive data protection for all workloads
Post Reply
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Copy Job processing rate, v7 against v8

Post by lando_uk »

Hi

We have recently upgraded from v7 to v8. I have a query about Copy Job speeds.

With v7 it was reporting our offsite jobs were getting about 186MB/s with the Bottleneck: Source. Since upgrading, the same jobs goes at 36MB/s with Bottleneck: Target. Is this just a reporting issue?

Thanks
Mark
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Copy Job processing rate, v7 against v8

Post by foggy »

Mark, what kind of target storage are we talking about?
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

R720xd, RAID6 12 x 4TB NLSAS
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Copy Job processing rate, v7 against v8

Post by PTide »

Wow, seems that your copy job has to last 5-6 times longer than before. Does it?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Copy Job processing rate, v7 against v8

Post by foggy »

Were job/repository compression settings left intact after the upgrade? Do you have the latest Veeam B&R update installed (Update 2)?
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

Well its hard to say, I only have 1 or 2 v8 logs so far, and the day after the upgrade there was lots of failed copy reports spewed out until it sorted itself out.

We have active-active datacenters, (A and B) The offsite jobs from A to B seems to be slower than the offsites from B to A. (A is the B&R server)

After the upgrade, I didn't restart any of the other proxies or repositories, only the B&R server itself - would I need to restart the other servers?

Yes, I installed U2b.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Copy Job processing rate, v7 against v8

Post by foggy »

No, automatic components upgrade should be enough. If none of the settings were changed and you feel the jobs started to run slower, I recommend to contact support to take a look.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

Well i'll let it settle down, we only keep a small amount of restore points offsite, so once those have aged I'll investigate further.

Compression for these jobs has been set to high, they haven't changed - I'm just wondering if the v7 reporting didn't take into consideration the compression so reported higher speeds - Kind of like when you work a thin provisioned disk, the reported speed ramps up because its not measuring real network speed but something else.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Copy Job processing rate, v7 against v8

Post by foggy »

As far as I know, nothing has changed in terms of reporting processing rates between v7 and v8
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

Out of interest - On my copy jobs, they are set to copy every 1 day, starting at 06:00 - My backups start at midnight and normally finish at 03:00 - I don't want copy jobs interfering with backup jobs as my storage only has so many spare IOPs - do I use the schedule to manage this or the job start time?
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Copy Job processing rate, v7 against v8

Post by PTide »

You can set your copy job to start in a couple of minutes after your backup job (0:15 for example). In such case your backup copy job will wait until your backup is done.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

I have 8 backup jobs that piggy back off the 1st midnight one though, 300+ VM's in total, and I don't want Copy Job 1 kicking off when Backup Job 2 starts. Id rather they all start when all the backup jobs are complete.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Copy Job processing rate, v7 against v8

Post by PTide »

A few questions to make sure that I got it right:
I don't want copy jobs interfering with backup jobs as my storage only has so many spare IOPs
Which storage did you mean - the one under backup jobs' repo, the one under backup copy jobs' repo, or production storage? Or maybe both backup and copy jobs use the same storage as a repo? Please give more details.
I have 8 backup jobs that piggy back off the 1st midnight one though
I have some doubts regarding the meaning of "piggybacking" in this context. Do you use job-chaining? Please clarify.

Thank you.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

I mean, when my repository storage is busy backing up, I don't want the same storage getting hammered by copy jobs transferring the latest restore point offsite. That would slow down the backups and transforms. I want copies to get processed when all primary jobs are finished.

Yes, I mean job-chaining, all my jobs follow on from the first that starts a Middnight.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Copy Job processing rate, v7 against v8

Post by PTide »

Ok, is that an assumption that backup copy reads will cause a significant overhead on storage, or you've tested that already?

Thank you.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

Its an assumption that the storage will work better doing either reads or writes, but doing both at the same time will slow things down. This is a cheap RAID6 DAS, not a SAN that can do 10,000 random IOPs.

Well, things are definitely slower that they were on v7, so tomorrow I'm going to delete a copy job, change the Compression to Optimal and then re-seed it, see if goes back to 100MB/s.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Copy Job processing rate, v7 against v8

Post by PTide »

I'd recommend to perform a test to see if it's really that bad for storage to run copy and backup jobs simultaneously. If the performance will be really poor then there is an option to setup a Backup Window so your copy job will transfer data only within allowed window.

Regarding your original post "v7 against v8" kindly open a ticket with support as it is an unexpected behaviour.

Thank you.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

Hi, this morning I deleted the copy job from disk, changed the compression to Optimal (it was set to High) and then kicked off the job again. Its now running at 135MB/s (so that's line speed for our WAN) - So it seems that Compression has changed in v8, as I used to get 135MB/s with High in v7.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Copy Job processing rate, v7 against v8

Post by PTide »

So it seems that Compression has changed in v8, as I used to get 135MB/s with High in v7.
That's strange. Compression has changed in v7.

I'm really curious about several things:

1. Could you please try setting compression back to High and run the job again to see if there will be a decrease in speed? Just to make sure.
2. In you original post you've mentioned 186 Mb/s speed. How did you get that speed if your WAN is 135MB/s ?

Thank you.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

When speeds go greater than the 1Gbe WAN, I presume the reporting isn't measuring the real speed, its just measuring something like `vm size/time to complete`. So jobs with good compression/thin disks etc report speeds over 1Gbe.

I'll leave this job running, as it's production data and offsite copies we are working with, so I can't afford to have no copies for too long. I can only really delete and re-seed one job a day due to the size and copy window, but tomorrow when I do the next job, I'll do as you suggest and delete/re-seed using High compression to check speeds.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

Image

See what I mean here.

I have now reset jobs 1 and 2 with Optimal compression. From the GUI it says between them, they are transferring at 185MB/s - yet windows task manager telling a different story and is sending at roughly 1Gbe. (WAN line speed)
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

More info:

Image

Here is the screenshot from Veeam One showing network throughput on the source repository.

Yesterdays figure is the speed of the 2 new jobs that I re-seeded with Optimal compression, they have a respectable speed (the WAN link also does other stuff) and finished at 9pm. Today, starting at 4am is the speed of the existing jobs that I haven't touched, they barely hit 25MB/s and are very bursty in nature compared to the more stable stream of yesterdays.

Also, from the CPU charts on the target repository, it looks like the compression takes place on the target after its sent, rather than on the source host - Surely it would save time and bandwidth compressing before being sent?
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Copy Job processing rate, v7 against v8

Post by PTide »

So, you mean that the jobs that were seeded yesterday with "Optimal" compression showed good speed of 180Mb/s, while other jobs that stayed with "High" compression showed a x7 times lower speed, showing target repository as a bottleneck, and CPU on target is loaded during those jobs, is that correct?

Do you use any WAN accelerators, by any chance?
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

Hi

Mostly correct, I wouldn't say x7 slower, as looking at real network charts its more like x3 slower.

If 3x slower is normal for High Compression, then I'll accept that and have to work around it, but v7 certainly wasn't x3 slower, so something has changed.

CPU wise, its not overloaded as the target has 24 cores, but it seems that the VeeamAgent is single threaded as just 1 of the cores is busy (about 75%) when 1 job is running with High compression. Going back in time on the CPU chart to before the v8 upgrade, CPU usage is generally much lower with the same settings.

So to me this says that High compression in v7 was much less resource intensive, is High compression in v8 much better than in v7 to warrant this impact?
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Copy Job processing rate, v7 against v8

Post by PTide »

Hi
CPU wise, its not overloaded as the target has 24 cores, but it seems that the VeeamAgent is single threaded as just 1 of the cores is busy (about 75%) when 1 job is running with High compression. Going back in time on the CPU chart to before the v8 upgrade, CPU usage is generally much lower with the same settings.
Compression should occur on the source side, therefore the situation you've explained is a valid reason to open a case with support team.
So to me this says that High compression in v7 was much less resource intensive, is High compression in v8 much better than in v7 to warrant this impact?
There were no changes in compression from v7 to v8 so the described behaviour is not normal.

With all said, I suggest you to open a ticket with our support team for a deeper investigation. Please don't forget to post you case ID here.

Thank you,
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

Case ID 01040163.
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Copy Job processing rate, v7 against v8

Post by lando_uk »

Well, still no further resolving this even though Support has seen the issue and agrees something is wrong. Tried turning off DisablePublicIPTrafficEncryption in the reg, but that made no difference.

Case still open...
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Copy Job processing rate, v7 against v8

Post by PTide »

Hi Mark,

Kindly keep working with support team. Lets see if there will be any outcome from tomorrow morning session. If nothing then the escalation might be needed.

Thank you.
Post Reply

Who is online

Users browsing this forum: tinto1970 and 221 guests