number of vms per job

[dmay]

Just evaluating Veeam and wondering if someone can share the benefits of multiple vms per job?

As I understand this is a deduplication advantage. How much of an advantage? Has anyone performed backups using both the one vm per job and multiple per job mode and compared actually storage saved?

A single vm per job results in per vm file set. I foresee this having a significant benefit when you need to perform a restore from tape or when you need to send a backup file set to/from an offsite location. Smaller size. Less Internet GB transfer fee. Ability to send via portable HDD.

Looking for why multiple vms per job is recomended. What are the pros and cons of both methods?

Cheers,

Darrell

[foggy]

Darrel, you cannot have any absolute numbers here as the actual dedupe ratio depends on what kind of VMs do you combine in one job, the number of those VMs, and amount of changes in each of them. The best results (up to 10x and more of the original size) you will get if grouping VMs created from the same template (running the same OS, the same apps), so a good practice is to combine Windows boxes together and Linux boxes together and not both of them in one job.

You can review this white paper for some real-world numbers and I think find some related topics on this forum also.
Thanks.

[dmay]

Foggy, am I understanding the only benefit multiple vms per job provides is deduplication target space saving? Further this really works best if vms are identical builds of OS/apps. What if each vm is a fairly unique build with completely different apps, roles, functions?

[Gostev]

Then dedupe will be less effective of course. However, quite a few block will still be similar, if you are using the same OS across them, same files etc. Try to copy the same big ISO file to a few VMs and do an incremental backup, the size of VIB file will pleasantly surprise you

[dmay]

I'm still not sold on the overall dedupe benefit of one job. If the OS and programs are identical this is still the smallest portion of data size. Overall my data files are unique across my servers. In my envrionment I do not have the same files on multiple servers.

I see tremendous advantage in having small individual server file sets (.vbk/vib/vrb) versus a single massive size file set. Especially when performing restores from tape. If you have one massive file set that takes as an example 8 hours to backup to tape you would therefore need 8 hours to perform a restore from tape, before you have a chance to import and perform a Veeam restore. If I have small individual server file sets this same restore time period could become minutes to perform a tape restore of just a single smaller server file set.

In my opinion the speed of restore should be the most significant goal to achieve. The one job backup recommendation achieves this only when the files are current and on disk. If not on disk and on tape, the exact opposite takes place where the one job backup recommendation significantly increases restore time.

Seems to me the recommendation should be to keep backup file sets as small as possble. Maybe matching file set size to current removable hard drive capacities. Picking an example, if available removable hard drives are 2TB, then recommend to bundle multiple vms per job so they create a file set under 2TB and use removable disk for offsite storage.

Maybe this is just my experience. We typcially retore a file or folder. Requests are either to restore from a very recent point in time or way back when. We rarely have server failures. The speed to perform a typical file/folder restore is key for us.

[Gostev]

Darrell, you do list valid points specific to your environment. Every environment is different, just like backup targets are different. The flexibility to create jobs one way or another is there in the product, so that everyone could create the jobs in the way that suits his or her environment best!

[5150cd]

My environment:
2 ESXi 4.1 servers (Dell PE R710)
Each server has 5 VM's (8 Windows-based & 2 Linux-based)
Each VM is running on DAS
My Veeam server is running on a physical server (Quad core/8GB RAM/4disk RAID5)

Should I run all these VM's in one backup job or run them each in their own job? If I run them each in their own job should I schedule them one after another or a few at the same time, or all at the same time?

Just trying to get these to run as fast as possible and still make it simple for recovery. Thanks for the help in advance!

[Vitaliy S.]

Hi Chris,

Please look through our recommendations above. In addition to this, you may want to run 2-3 jobs at the same time in order to leverage the benefit of having Quad Core processor on the backup server. Be sure that your Veeam server's CPU usage is not maxed out during backup job runs, as it will reduce your overall jobs performance rates.

Thanks.

[5150cd]

Thanks for the reply Vitality. I think I still prefer to keep my jobs separate. I have a lot of disk space for my backups and I just like have them separate. However, I changed the scheduling around so that two jobs start on each server at the same time and then I daisy chain them together. Seems to be doing ok, never saw the CPU usage go above 70%. Thanks again!

[ChrisL]

Hi all,

On the subject of deduping with many VM's per job vs single VM per job, you make reference to big benefits when using dedupe while backing up simlar VM types in one job (similar OS, lots of simlar files, this makes a lot of sense)

My question is, is there any advantage to using dedupe at all when backing up a single VM per job? Presumably, if there is only one VM in the job, there won't actually be any duplication to be compared..?

[Vitaliy S.]

Even if you have just a single VM in the backup job, you might still have some deduplication, because deduplication is done on block level, but of course dedupe ratio will not be nearly as efficient as with having multiple VMs in the job.

[ChrisL]

Thanks for the reply, so, following on from that, I can see that the dedupe benefit drops with a single VM per job (but is still there to help a bit) but is this contradicted by a cost in terms of processing/RAM overhead to actually do the deduping. To be fair, I've not done the test, but would you expect having dedupe turned on for a particular job to have a significant effect on the time for the job to run, and an increase in CPU/RAM needed. Our 900GB fileserver is taking 20+ hours to run a (reverse incremental) rollback, with dedupe turned on, which seems quite exessive to us given the figure quoted elsewhere.

Is it worth trying changing the job to not dedupe (and actually, is this possible with an existing job?) and see if this changes?

[tsightler]

Our 900GB fileserver is taking 20+ hours to run a (reverse incremental) rollback, with dedupe turned on, which seems quite exessive to us given the figure quoted elsewhere.?

I think it is unlikely that dedupe is the cause of this level of performance. I would suggest troubleshooting to determine what is actually causing this to be slow. First thing I would look at is the size of the rollback files. 20+ hours is a very long time and there has to be some reason for it. Either no CBT being used, lots of changes on the server in question (either by users or by automated tools like background defrag), or the slowest target storage known to man. The size of the rollback files would give you some clue as to where to look.

If you are using V6, the bottleneck statistics will highlight the most likely culprit right away.

[ChrisL]

Thanks for the pointers, here's the summary from the latest job of the 'slow' job, with the duration column added for completion

12/12/2011 19:49:03 :: Job started at 12/12/2011 19:48:51
12/12/2011 19:49:04 :: Building VM list
12/12/2011 19:49:37 :: VM size: 1.2 TB
12/12/2011 19:49:37 :: Changed block tracking is enabled
12/12/2011 19:49:52 :: Preparing next VM for processing (Duration 1:48:24)
12/12/2011 21:38:16 :: Processing 'SMAUG (Staff data)' (Duration 18:31:31)
13/12/2011 16:09:50 :: All VMs have been processed
13/12/2011 16:19:55 :: Load: Source 3% > Proxy 51% > Network 99% > Target 99%
13/12/2011 16:19:55 :: Primary bottleneck: Network
13/12/2011 16:19:55 :: Job finished at 13/12/2011 16:19:55

..which clearly shows the network as the bottleneck. The SAN is connected via iSCSI, would this be what is referred to as 'network'? It's connected via a single 1GB ethernet which I assume is typically enough? I think, tom, you might be on to something with the slow local storage - I tried copying a 180GB file from a local folder to another local folder (4x2TB in RAID5) which reported 35MB/sec and took 1hour+ to complete, which seems a little slow.

[ChrisL]

Oh, and re: the rollback sizes..

VBK file - 930GB
typical VRB file - 75GB

and the report shows that CBT is being used

[tsightler]

13/12/2011 16:19:55 :: Load: Source 3% > Proxy 51% > Network 99% > Target 99%
13/12/2011 16:19:55 :: Primary bottleneck: Network

..which clearly shows the network as the bottleneck. The SAN is connected via iSCSI, would this be what is referred to as 'network'? It's connected via a single 1GB ethernet which I assume is typically enough? I think, tom, you might be on to something with the slow local storage - I tried copying a 180GB file from a local folder to another local folder (4x2TB in RAID5) which reported 35MB/sec and took 1hour+ to complete, which seems a little slow.

Actually I think this might be a small bug in the bottleneck detection logic. The target is actually the bottleneck here, the network is simply busy waiting because the target buffer is full. Obviously if the Target is 99% busy then the previous process is spending all of it's time waiting on the target.

So this problem is really all about target storage. Reverse incremental requires 3 I/O's for every changed block as the old block must be read from the .VBK file, written to the .VRB file, and the finally the new block is written to the .VBK. This typically makes the process very I/O bound, not really throughput bound. With 4x2TB disks you're only going to deliver about 225 IOPS at best (probably less), which will limit your throughput even more than the copy operation above.

I'm a little curious, would you expect your fileserver to be getting enough changes to generate 75GB of rollback a day? That feels pretty big for a 900GB server. I used to manage backups for fileservers of this size and much larger and change rates were generally 2-3% even with a few hundred users. Now if you have a very high user count, roaming profiles, etc, it could possibly be a much higher change rate, but it feels big. Online defrag or other tools will lead to very high change rates even with relatively little data change. Just some things to look for.

[ChrisL]

Hi Tom,

Our rollbacks have always been in about that region for this server, although proportionally of course, as the total size has increased. I might have a look and see if the server is doing something by itslef in the background, interesting idea about a defrag, but this is not something that we have consciously enabled. It's a relatively old Server 2003 VM that's been p2v'd when it was virtualised so it's been with us for quite a while now and is due to be migrated to a Server 2008R2 VM when we get the chance.

Interestingly, looking back, I think the performance dropped when we re-RAIDed the local storage from RAID0 (bigger capacity but much more vulnerable to failure) to RAID5 (less space but in theory more reliable) and I've since read around that there is a performance difference with different RAIDs, with 1 and ) being the faster ones. It's on a pretty good Adaptec hardware controller though so I would hope it shouldn't be too significant.

[ChrisL]

..oh, and we've got about 2000 users between the students and staff but no roaming profiles are on the servers, they are all mandatory profiles, so it's just file serving stuff that's changing, just apparently quite a lot of it!

[PaulFleetwood]

Thanks for the reply Vitality. I think I still prefer to keep my jobs separate. I have a lot of disk space for my backups and I just like have them separate. However, I changed the scheduling around so that two jobs start on each server at the same time and then I daisy chain them together. Seems to be doing ok, never saw the CPU usage go above 70%. Thanks again!

I'm also looking at the options between individual jobs per server, or grouping them together.....

I'm interested in the quote above - how do you 'daisy chain' the backups (I'm assuming this means there is some way of running them sequentially).

Currently I have all my jobs timed so that they (should) start after the previous one finishes. The problem comes as the data grows and some servers start taking longer (and the backups start overlapping).
It would be simpler to have one job with all the servers in, and tell Veeam to backup them up sequeintially (but I can't see anything to configure what order the servers will be backed up, or how mnay will run at once)
If I create a job for all my servers (and I'm only talking about 18 VMs) - will it try and back them all up at once? Surely this will grind the system to a halt??

If anyone has any suggestions on a simple way to get jobs runnign sequentially, I'd be very grateful.

Regards,
Paul

[Vitaliy S.]

Hello Paul, what version of Veeam B&R are you using? Please be aware that Veeam B&R processes all VMs sequentially and there is no way to change that. Moreover in v6 there is an opportunity to specify processing order for VMs being backed up.

Looks like this is exactly what you're looking for, right?

[PaulFleetwood]

Excellent - if it processes them sequentially, then this may be the answer to my problems. I'll look at testing a new backup job for all servers (in one job) and see how it goes....
We're on v 5.0.2.230
To be honest I don't think I'm particularly worried what order it processes the servers - does it do them in some logical order (i.e. alphabetical or similar), and does it do them in the same order each time?

[foggy]

This is a kind of internal logical order (the same each time) that does not generally match with the list displayed at the wizard step.

[Gostev]

In v6, you can set the processing order explicitly though.

[Kslawrence]

Hi,

Is it best practice in Veeam to create a single job with multiple vms in it or a job for each vm?

Example - We have a vsphere host with 6 vms running on it that have been added over time. I have created separate backup (to NAS) and replicas jobs for each server. So I have 12 jobs in total, all running at various times. This seems to work fine, albeit with the odd snapshot removal error, probably due to clashing jobs which seems to rectify itself fairly quickly.

Should I be instead running just two or three jobs - an hourly replica job for all critical servers, a four hourly replica job for less critical servers and a nightly backup to NAS job for all servers? I know I then wont be able to change the times for each server but this wouldn't be a big problem.

Hope to hear back

[Sethbartlett]

This is really up to you and what works best. If you are okay on space and how everything is running, leave it. If you move all the backup jobs into one, you will get better compression/de-duplication ratios but then you run a risk. The risk would be that all the VMs are under one set of backup files and if those files got deleted/destroyed/corrupted, you couldn't restore those VMs.

[Kslawrence]

Thanks Seth, that was what I was thinking too. I will give it a go with a combined job and see if it gives any big dedupe benefits. If not, ill probably revert to single jobs.

Cheers

[cowbsh]

Are there recommendations on when you should and should not backup multiple VMs in a single job? For instance, my Lync servers consist of 2 edge servers, 2 front end servers, and a database server. Additionally for SharePoint I have a front end server, central admin server, and a database server. Now, should I create a Lync job and a SharePoint job where each backs up it's respective server, should I create a separate job for the database servers, or each server have its own job?

[foggy]

Brent, please review the topic above for some considerations regarding grouping Vms. General recommendation is to group similar VMs (i.e. created from the same templates, for example) to get better dedupe rates. Additionally, this topic is also worth reviewing. Thanks.