Looking to Compare Performance

[brupnick]

Good morning-

I'm looking to compare the performance of my VBR backups with others in the community. Specifically, I have a 10 GB Linux VM that doesn't change much on a daily basis, but still takes 15 minutes to back up. To me, this seems unusually long for such a small VM, but I wanted to see if anyone else had a similar scenario and could tell me how long it takes on their system. I understand that there are a significant amount of variables around what I'm asking, but I can get into specifics of my environment if it comes to that.

I have checked the logs for this VM and it looks like a majority of the time is spent on tasks other than backing up the data, so I'm not sure if this is an issue with VBR or my vSphere server. Any help would be greatly appreciated.

[Sethbartlett]

If you look at your Backups pane and right click the job and go to properties, what size are your VRB/VIB files? Change block tracking may be picking up small changes of files being moved or being slightly modified and we will back that up as the bits have changed.

[brupnick]

If you look at your Backups pane and right click the job and go to properties, what size are your VRB/VIB files? Change block tracking may be picking up small changes of files being moved or being slightly modified and we will back that up as the bits have changed.

For last night's backup, I had 124.46 GB of data which resulted in a 57.03 GB file. And I agree with what you said about CBT, but considering the size of this VM and the fact that it hardly changes (it's a proxy server), I would think that this server would process much quicker.

I guess what I'm trying to figure out is why some of my VMs process extremely fast, but others take 10 times longer. Another example, I have two domain controllers, one 25 GB, the other 30 GB. In last night's backup, the 25 GB DC processed in 5:07. The 30 GB DC took 23:57. Both are DCs in domains that aren't really used, so there should be minimal changes to these VMs on a daily basis. So why does one take so much longer than the other to process?

[Gostev]

Amount of zero blocks in the image is different between the two?

[brupnick]

Amount of zero blocks in the image is different between the two?

I guess that's possible. Does VBR have any way of telling me that?

This was just a small sampling of 3 VMs. Below are all the entries from one of my jobs that processes DC, mail severs (x4) and some file servers. The file servers, which are the largest in the list, are also among the fastest servers processed. The DCs are the smallest and slowest. I know that processing rate is derived from size and duration, but as you can see below, my numbers are all over the place. Maybe more information is changing on the DCs than I realize, but my understanding was that these are usually among the quietest servers, especially from day to day.

Total size Processed size Processing rate Duration
170.01 GB 170.01 GB 112 MB/s 0:25:55  
775.00 GB 775.00 GB 375 MB/s 0:35:13  
45.00 GB 45.00 GB 179 MB/s 0:04:17  
25.00 GB 25.00 GB 92 MB/s 0:04:39  
30.00 GB 30.00 GB 22 MB/s 0:23:22  
30.00 GB 30.00 GB 98 MB/s 0:05:11  
25.00 GB 25.00 GB 25 MB/s 0:16:52  
25.00 GB 25.00 GB 170 MB/s 0:02:30  
15.00 GB 15.00 GB 93 MB/s 0:02:45  
15.00 GB 15.00 GB 93 MB/s 0:02:45  
80.00 GB 80.00 GB 89 MB/s 0:15:25  
35.00 GB 35.00 GB 25 MB/s 0:24:07  
117.00 GB 117.00 GB 101 MB/s 0:19:48  
15.00 GB 15.00 GB 16 MB/s 0:15:50  
15.00 GB 15.00 GB 16 MB/s 0:15:40  
615.00 GB 615.00 GB 761 MB/s 0:13:47  
35.00 GB 35.00 GB 25 MB/s 0:23:27  

[Gostev]

VBR cannot tell you this, but you can try to sdelete your slowest VM and see if that helps on 2nd incremental run after sdelete is done. Thanks.

[scott_mac]

Not sure how helpful or relevant it is... but the servers that we have differ significantly and the one obvious difference between Windows and Linux is the hardware version.... VM's that are version 7 and thus can use Change Block tracking in Veeam run orders of magnitude quicker.

For instance we have a 150Gb Server 2008 Version 7 VM that runs in around 5mins, compared to a 2Gb Linux server that takes about 2mins.

Veeam is installed on a Virtual Server itself but does have a good chunk of resource available....

[brupnick]

That's a good point, but all of the servers in the list from my previous post except for 2 are Windows servers with the most recent version of hardware and tools. And I have Linux servers in others jobs that are much larger and process in under 2 minutes.

Here's the job history for a Linux server for the past 2 weeks:

Code: Select all

Start Time			End Time			Performance Rate	Transferred Data	Status
9/20/2011 19:01	9/20/2011 19:17	11 MB/s			10.00 GB			Success
9/19/2011 19:01	9/19/2011 19:16	11 MB/s			10.00 GB			Success
9/15/2011 20:31	9/15/2011 20:46	12 MB/s			10.00 GB			Success
9/14/2011 20:50	9/14/2011 21:09	9 MB/s			10.00 GB			Success
9/13/2011 20:20	9/13/2011 20:34	12 MB/s			10.00 GB			Success
9/12/2011 20:00	9/12/2011 20:10	17 MB/s			10.00 GB			Success
9/9/2011 19:52		9/9/2011 20:01		20 MB/s			10.00 GB			Success
9/8/2011 19:36		9/8/2011 19:40		42 MB/s			10.00 GB			Success
9/7/2011 19:42		9/7/2011 19:46		47 MB/s			10.00 GB			Success
9/6/2011 19:30		9/6/2011 19:35		40 MB/s			10.00 GB			Success
9/5/2011 19:36		9/5/2011 19:40		39 MB/s			10.00 GB			Success
9/2/2011 19:43		9/2/2011 19:46		55 MB/s			10.00 GB			Success
9/1/2011 19:33		9/1/2011 19:37		46 MB/s			10.00 GB			Success
8/31/2011 19:32	8/31/2011 19:36	44 MB/s			10.00 GB			Success
8/29/2011 20:25	8/29/2011 20:29	40 MB/s			10.00 GB			Success
8/26/2011 19:59	8/26/2011 20:02	55 MB/s			10.00 GB			Success

Maybe I'm chasing something that I can never catch here, but if I have servers that are comparable in every way (OS, hardware, changes, etc), why am I seeing these large changes in processing times? As you can see from the above, for the same server, I'm seeing up to 6x slower performance. My VBR server has 4 processors and 8 GB of memory, so is vSphere my bottleneck?

[scott_mac]

I think you're right to want to look further into it.... my times are consistent across the board, but then again nothing changes on the Linux server I backup as it is just for iPad access to vSphere.

Have you tried using Direct SAN access rather than going through vSphere (assuming it's possible?) that might be a way of exonerating vSphere!?

[brupnick]

I think you're right to want to look further into it.... my times are consistent across the board, but then again nothing changes on the Linux server I backup as it is just for iPad access to vSphere.

Have you tried using Direct SAN access rather than going through vSphere (assuming it's possible?) that might be a way of exonerating vSphere!?

It's funny you mention that because I was just working on setting up a physical box to test this method and compare it to the current HOTADD system. Is this what you're using?

[scott_mac]

I use a mixture to be honest... but my times are consistent either way. I would be surprised if your vSphere box is the limiting factor as it's certainly pretty powerful! Direct SAN does definitely report faster times, but that may be the fact that one of my servers run in its own job as Direct, whereas others are mixed in with others and whilst individual times are listed I can't guarantee the accuracy - at face value Direct would suggest it's twice as quick, but there are so many variables such as number of changes on the servers that i doubt the accuracy of that.

Try the Direct access and rule it out for your piece of mind though.