Discussions specific to the VMware vSphere hypervisor
Post Reply
pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

How sequential are writes to repository?

Post by pkelly_sts » Apr 13, 2016 2:24 pm

I'm just curious as to how much load I'm putting on a repository if I increase the number of "Max concurrent tasks" to it, and am sending backups to it from 1 x San attached proxy & 3 x Proxy VMs. Does all the incoming data get written sequentially as a single "stream" of data to disk, or is there any kind of random disk thrashing taking place?

In this instance I'm talking about a single backup job of multiple VMs as opposed to multiple backup jobs.

foggy
Veeam Software
Posts: 18288
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How sequential are writes to repository?

Post by foggy » Apr 13, 2016 2:38 pm

I/O pattern depends on the backup method the job is using.

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » Apr 13, 2016 2:47 pm

Perfect, thanks!

(For this particular site an active full is similar speed to a synthetic full anyway (2.1Tb > 900Gb in 3hrs or so) so I've always just done active full weekly so for this particular site then, backups should be pretty much always sequential.)

foggy
Veeam Software
Posts: 18288
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How sequential are writes to repository?

Post by foggy » Apr 13, 2016 3:09 pm

Yes, active full is sequential, however, if touching production VMs each time is not desirable and you still have similar performance with synthetic fulls, you may consider switching to them, at least partially (e.g. weekly synthetic, monthly active).

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » Apr 13, 2016 3:15 pm

Yep, understand the consequences but active fulls really aren't an issue for us on this particular site anyway so as they easily fit into backup windows etc. for a long time I've just let run active ever since first deployment a couple of weeks ago.

Ah, just to clarify, this is active weekly only! Incremental daily of course.

Just for giggles I've increased the concurrent disk tasks from 10 to 12 to see if it makes much/any difference. Wouldn't bother me if it didn't, just interested in knowing the real performance ceiling for the storage we have.

foggy
Veeam Software
Posts: 18288
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How sequential are writes to repository?

Post by foggy » Apr 13, 2016 4:22 pm

If your storage is ok with multiple I/O streams, you might want to enable per-VM backup chains as well.

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » Apr 14, 2016 10:08 am

That's high on my list of areas to look into when we upgrade to v9 (which should hopefully be approved this afternoon :)

Now you mention it, I was thinking about per-vm chains this morning and wondered if they also actually help with seeding newly added VMs to off-site backup copy jobs? In other words, if I wanted to add a backup of a new 150Gb VM to a backup copy job (over a link slow enough that I wouldn't want to send that 150gb over the wire for the first full) can you "extract" the local copy of the per-vm backup of that VM, ship it over to remote site, and then use it to seed a copy job that is being sent to a scale out repository based at the remote site with per-vm's enabled?

Hopefully my question makes sense! (In this context it's based on my plan to migrate our last few physicals to VMs & how I might best seed a copy of their first backups to the remote site).

foggy
Veeam Software
Posts: 18288
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How sequential are writes to repository?

Post by foggy » Apr 14, 2016 10:20 am 1 person likes this post

This procedure should help. You cannot just copy the full backup for that VM from the per-VM backup chain and use it for seeding.

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » Apr 14, 2016 10:28 am

Ah, interesting-looking approach. I'll admit that at first reading I haven't 100% got my head around it but I'll file it away for when I've had more coffee - thanks for the pointer :)

foggy
Veeam Software
Posts: 18288
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How sequential are writes to repository?

Post by foggy » Apr 14, 2016 10:31 am 1 person likes this post

Yep, its pretty creative, but it works.

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » May 03, 2016 9:00 am

Hmm, well having switched to A) Scale-out repository & B) Synthetic weekly full for this specific site I'm really not seeing a whole heap of benefit # in this particular case #.

As I've mentioned above, weekly active fulls aren't much of an issue for us hitting the production environment anyway as they completed inside 2.5hrs or so. Having tried switching back to synthetic full, but to a scale-out repository with two simple repositories configured for performance but (perhaps crucially?) NOT yet enabling per-VM backup chains, the same weekly job took about 5:30hrs so over double the time.

The issue for us is having as wide a window as possible for off-site backup copies over a relatively slow link so losing those 3hrs of copy time is more of a problem, to me.

However, perhaps the per-vm chains would have helped? I doubt it as the first active full of those would still only have been going to the same storage anyway.

To clarify, the original repository is the same as I was previously using (so an active full of 2.2Tb compressed to 850Gb is completed in up to 2.5hrs) but added to the scale out, and I've introduced a second set of spindles of equal performance for the incremental repository part of the scale-out.

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » May 03, 2016 9:11 am

In fact, double-checking again the last active full took 2:19 in total whereas the synthetic full this weekend took 6:06 in total so nearer 3x longer (in my case, obviously everyone's site/setup is different)

foggy
Veeam Software
Posts: 18288
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How sequential are writes to repository?

Post by foggy » May 03, 2016 9:13 am

Looks like active fulls is the way to go in your case. Btw, you didn't mention what kind of repositories your extents are? Just Windows servers?

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » May 03, 2016 9:23 am

Agreed, useful to have tested though.

In this case the server is multi-role: Windows 2008 R2 Server, Dual 6-core, 24Gb mem, B&R Server, Proxy, 8Gb Direct-attach FC SAN (to VMFS) & 14 DAS disks, 12 dedicated to backup storage.

I do have some recently added SAN storage with more spindles per LUN (which is primarily off-site repositories for our main site) which I might be able to find some time to test against.

In fact, I did a similar test for those exact off-site copies in that I re-configured them as a remote scale-out repositoriy (similar setup, two sets of spindles) but I'm having trouble analysing the results to actually figure out how long the transforms are taking for that copy job :-/ (I was just scratching my head trying to figure out exactly what question to ask on the forums to figure out what I'm mis-understanding about transform operations! :) )

foggy
Veeam Software
Posts: 18288
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How sequential are writes to repository?

Post by foggy » May 03, 2016 10:30 am

This should be visible from the job stats (transformation is started right after all data transfer operations are completed).

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » May 03, 2016 10:48 am

That's exactly what I thought, and have been watching & seeing the daily merges happening right up until the 28th April which is the last mention I can see (in the job stats), even though I can see that the job is up to date with a .vbk file date-stamped early this morning.

However, looking through them again now I'm suddenly getting a regular "copy interval has expired" message but that confuses me even more! I don't yet fully understand that message (searching the forums in parallel to this post) but it's implying an issue, but still completed a .vbk last night, yet doesn't report a "merge completed" since the 28th??

Getting more & more confused! :-/

(FYI this backup copy job has been running fine for many weeks so it's not a new copy or anything along those lines)

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » May 03, 2016 11:23 am

(Sorry, to further clarify, the copy job is scheduled to start @ 19:40, which is 10 mins after its "parent" backup job is scheduled for, which generally only takes about 30 mins or so)

foggy
Veeam Software
Posts: 18288
Liked: 1568 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How sequential are writes to repository?

Post by foggy » May 03, 2016 11:56 am

It could be that some VMs were not processed during the interval or something else, you can contact support for assistance in logs review.

pkelly_sts
Expert
Posts: 571
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: How sequential are writes to repository?

Post by pkelly_sts » May 03, 2016 3:23 pm

Understood, thanks for the help so far. Have now logged as case 01788614.

Post Reply

Who is online

Users browsing this forum: pkelly_sts, wishr and 36 guests