Help Identifying Bottleneck

[pops106]

Hello, I have been trying to identify a bottleneck I seem to be seeing with our Veeam backups.

This is a brief overview of the current config.

2 x FC Attached AMS500 SANs with FC and SATA disks
2 x Veeam Backup Servers each FC attached
Backup Target is a AMS2100 SATA with 2 x Raid Groups one for each Backup Server
Using I/O Meter I can see we are able to get 350mb/s or there about on each backup server for 100% sequential reads, I need to test writes but would be good if I had a better understanding of the I/O load from Veeam.

I need some information on what to watch for and how I can prove where our bottleneck is, Running a full fresh backup of a 15GB VM which is almost full within the OS I get the following results;

Dedupe and Optimal Compression – 34MB/s and around 25% usage on one CPU
Dedupe and Low Compression – 39MB/s and around 40% usage on one CPU

I also tried deploying a VM and using Virtual Appliance mode and got around 31MB/s and no real difference in CPU usage, The CPU’s in the backup server are E5540’s and the VM running on X5450’s the VM had 4GB Ram and 4 vCPU’s.

I have also tested backing up to a FC LUN, a different SATA LUN and the internal backup servers SAS disks and results are very similar. I have also tried locating the VM I am testing on both SATA and FC disks with no real difference.

I am now very confused

[Gostev]

Hello, I just posted instructions couple of days ago here

[pops106]

That is perfect Gostev cheers.

Ok different VM 10GB;

Full - 31MB/s
Incremental - 51MB/s

I think with a bigger VM the incremental would be higher

I am going to use the suggested I/O meter settings in that post and see what results we get.

Cheers

[pops106]

Using the I/O Meter settings I am seeing almost exactly as above around 30.8MB/s

Where do I go from here Gostev, is there anything else I can look in to or is this a case of needing different hardware.

Cheers

[Gostev]

Full - 31MB/s
Incremental - 51MB/s

Low incremental speed in this test indicates that the actual issue here is with source VM data retrival speed, obviously 51MB/s is a joke for FC SAN, it should be at least a few times faster. Here is some numbers for Veeam Backup 3.0 on different FC SAN, essentially in the same test as you did (there was no changed block tracking at the time, and incremental pass had to read the whole source VM image to determine the changes during the incremental sync).

As a next steps, try to update HBA drivers, or replace HBA completely? Disable multipathing, update or uninstall multi-pathing software (at least VCB was very picky about multipathing, and vStorage has the same code base)? If nothing helps, probably the best bet would be to approach your SAN vendor with this?

Here is a very helpful thread from early days of Veeam Backup and VCB:
How to improve backup speed: VCB performance in SAN mode

Please keep us posted, it would be interesting to know how your troubleshooting goes.

[pops106]

I am afraid after going through all the hints and tips most of which I was already doing ie single path to all the LUNs etc I am still seeing the same performance issues.

The one thing I have found is that when running two jobs together it really shows up the issue, seems like a silly statement but these are different LUN's and possibly different SAN's that the VM's are sat on.

I have updated all drivers and firmwares and possibly gain 2MB but could be that there is slightly less workload on the SAN's at the moment.

I do by chance have a third site which has got the exact hardware, SAN, Brocades, Servers, HBA's everything so I am going to start working on that site to try and resolve the issue.

Cheers

[pops106]

OK...

I have assigned a LUN to the backup server from the same source SAN and tested using IOMeter;

100% Sequential reads - 350mb/s
Veeam type workload - 60mb/s

I then tried using the LUN as a target for the backup and go the exact same 30MB/s Veeam backup speed.

I have also made sure this new LUN is aligned wondering if that could be part of the issue but its a no go.

So my source SAN is good
The target SAN is good
CPU usage is not high

I am now completely confused, where do I go from here?

[pops106]

I have now a taken one of host out of the cluster and unassigned all disks to it. I have then created a whole new LUN and assigned it to just the one server, put a VM on it and now have a single ESX server to single backup server relationship with the same 30MB/s full backup, will try incremental in a minute.

I am going to try just as a test another vmware backup product in this same environment, I am not having a dig but need to find an answer.

[tsightler]

Using the I/O Meter settings I am seeing almost exactly as above around 30.8MB/s

Where do I go from here Gostev, is there anything else I can look in to or is this a case of needing different hardware.

Cheers

So don't you think this is the likely source of the issue? If you can only do 30.8MB/sec to your target storage then it's not going to support much faster backups than that. Why is this speed so poor? It has nothing to do with Veeam but will certainly limit Veeam's performance.

[pops106]

I completely understand what you are saying, but I have changed just about everything from the stats i was getting from my original post.

I now have a different target and source each have been tested to be able to push 350MB/s of sequential reads. From the advice given earlier it was highlighted it the source suffering from read performance.

Using IOMeter I have good performance from the source and target, CPU inst high I am at a loss.

Don't get me wrong, I have Veeam on many sites without issue and love the product but I am struggling to iron out the issue here.

As I mentioned I need to find a solution, this is for a client, I wont be offering any other products just need be seen to be using due diligence .

[pops106]

I switched the source and target SAN's and pretty much still got the same result.

I have got 2 backup servers with the same issues.

Possibly some kind of fabric problem... I am lost at this point.

[tsightler]

I completely understand what you are saying, but I have changed just about everything from the stats i was getting from my original post.

I now have a different target and source each have been tested to be able to push 350MB/s of sequential reads. From the advice given earlier it was highlighted it the source suffering from read performance.

Using IOMeter I have good performance from the source and target, CPU inst high I am at a loss.

Don't get me wrong, I have Veeam on many sites without issue and love the product but I am struggling to iron out the issue here.

As I mentioned I need to find a solution, this is for a client, I wont be offering any other products just need be seen to be using due diligence .

Sequential reads are not very important for the target. The target needs to be able to support small block writes and random small block reads and writes with very good performance. If it can't do this you're simply not going to get very good performance with Veeam.

[pops106]

I agree completely during a normal backup with plenty of changes but I am following the advice of backing up a VM which has no data change, it is switched off, hence very little writes are going to occur.

It is almost I have another issue, which I agree is outside of Veeam but my VM's perform well with SQL VM's pushing thousands of IO's and 200+MB/s when busy, I have also confirmed my backup server can get the same kind of performance.

The problem is, if I change the Source, Target or anything else I am still only getting 30MB/s

I am still inclined to say CPU but I do feel I may get different results with another product, weather good or bad.

[tsightler]

I agree completely during a normal backup with plenty of changes but I am following the advice of backing up a VM which has no data change, it is switched off, hence very little writes are going to occur.

I'm not following your logic here, are you saying that your incrementals are only running at 30MB/sec even with a powered off VM? Incrementals of a powered off VM should take no more than a minute or so since there would be no changes. Are your sure SAN mode is actually being used and Veeam is not failing over the NBD mode? Is CBT working?

What the latency of single block reads for the array? Can you tune the read-ahead cache of your LUN's? Veeam uses vStorage API which performs reads with a very shallow queue depth. Latency has a big impact on throughput. Just some other things to look at. If you run IOmeter with a single thread on a small queue depth what performance do you see?

[pops106]

It was recommended to help try identify the issue to switch off CBT and backup a switched off VM which means Veeam will have to do an entire read of all the blocks but not write any data to the target.

It is this performance it looks like I need to improve, I will have a play with IOMeter as suggested, much appreciated guys.

Cheers

[pops106]

It was recommended to help try identify the issue to switch off CBT and backup a switched off VM which means Veeam will have to do an entire read of all the blocks but not write any data to the target.

It is this performance it looks like I need to improve, I will have a play with IOMeter as suggested, much appreciated guys.

Cheers

[pops106]

Running a 100% Sequential Reads the latency is about 10ms and pushing 360MB/s

That is with 64 outstanding I/Os and 64K chunks

With the Veeam type test I am getting about 60MB/s and 70ms

Does that help?

Cheers

[tsightler]

Running a 100% Sequential Reads the latency is about 10ms and pushing 360MB/s
That is with 64 outstanding I/Os and 64K chunks

I'm not sure how much value there is in that test. How big is your dataset? Is it bigger that the array's cache size? Also, using 64 outstanding I/O's is a good way to stress and max out your storage to test for max throughput, but isn't a realistic number to expect from most single process reads. Veeam appears to perform only a single read at a time so a more realistic number would be the performance with 1 outstanding I/O, or maybe 2 or 4, but certainly not 64.

With the Veeam type test I am getting about 60MB/s and 70ms

That feels very high. If each and every read takes 70ms, then you could only read about 14 blocks per second. You won't get much throughput with that.

As an example, our Equallogic SAN typically has maximum latency in the 10-15ms range, and typical latency in the 4-5ms range, and that's iSCSI. FC should have better latency.

[pops106]

I will have a look at the latency again and see where we stand, I have changed a few settings tonight, things like prefetch and multistreams which to be honest I dont know a great deal about the multistreams option.

I have been using perfmon to try and see read/writes etc and the one thing I have noticed is that it is in noway steady throughput. It leaps up and down from 70 - 0 MB even when I know it is data that has changed and is been processed.

The prefecth and multistreams settings have helped though, before it wasnt peaking over 40MB, now we are seeing peaks of 120MB.

CHeers

[pops106]

Tested reading from the SAN with a single outstanding IO and a 8GB file size which is larger than the cache in the SAN. The Latency is 0.22ms reading from the source SAN so I havent got any latency problems there.

By any chance does anyone or Veeam have a IOMeter test file which can be used to try and identify the issue under a Veeam type workload?

Then hopefully a couple of us to test the same file and compare the results.

[pops106]

Found something new, one of my servers processes data much quicker than the other.

THIS IS ALL WITH CBT SWITCHED OFF

I have increased the queue depth to 32 and it has improved read performance so I am now according to PerfMon reading from the source around 150MB/s on each server.

What I have found is, I have added a new 10GB disk to my test VM which is powered off, it isn't formatted or anything else. When I back it up and it gets to the second empty 10GB disk perfmon doesn't read or write any data but the CPU / Server still seems to process the DATA seeing 50% ish CPU activity.

Server 1 doing an incremental only manages 46MB/s which is very similar to a Full of the same VM
Server 2 doing an incremental manages 87MB/s and the full 65MB/s

The second server processes data much quicker especially the "empty" data, I am guessing it still has to work the empty data into the compressed VBK files.

The source and target storage is the same for both server

[pops106]

One thing I have noticed is veeam.backup.manager.exe on the slower server1 is running at 25% almost constantly with veeamagent.exe running around 9>10% whilst on server 2 veeam.backup.manager.exe barely uses any CPU.

What role does the backup.manager.exe play?

Cheers

[pops106]

Does anyone know what role each of the services above play and why I am seeing different results on each server?

Cheers

[Gostev]

veeam.backup.manager.exe is the main job process (1 per job)
veeamagent.exe is the data mover process (created by the job process)

I believe results are different because first server is slower