Veeam 8 Performance Issue

VMware specific discussions

Veeam 8 Performance Issue

Veeam Logoby Jack1874 » Sat Oct 17, 2015 3:37 pm

Case # 01025881
Guys, we are looking for some advice here…

We believe we are having major performance issues when try to land our data to the HP StoreOnce.

Here is what our environment looks like …
Veeam 8 (Update 2)
Hypervisor: ESXi 5.5
Storage Array: HP 3PAR

Proxy Servers 1
Type: Physical
HP DL380p Gen8
RAM = 64 GB
Network = 10 GBe
O/S = Windows 2012 R2
Transport Mode = Direct SAN access
Max Concurrent Tasks = 12

Proxy Servers 2
HP BL60c Gen9
RAM = 64 GB
Network = 10 GBe
O/S = Windows 2012 R2
Transport Mode = Direct SAN access
Max Concurrent Tasks = 16

Repository
HP StoreOnce
Running NFS Shares

Jobs
Multiple jobs with small(er) amount of guests, dedicated proxy \ dedicated NFS share
Options
Parallel processing = enabled

Background
We have some very large VMs (1TB plus)
We are seeing transfer rates between 50 MB/s and 150 MB/s
The majority of the time it shows the network as the bottleneck.
We did try Jumbo Frames but found the connectivity to be too unstable.
Can anyone share their thoughts on out config, speeds, feeds etc? Does anyone have a similar configuration which they are running with success?
Feel free to ask as many questions as you like.
Jack1874
Enthusiast
 
Posts: 88
Liked: 4 times
Joined: Sat Oct 17, 2015 3:32 pm
Location: Canada
Full Name: Stuart Little

Re: Veeam 8 Performance Issue

Veeam Logoby Shestakov » Sat Oct 17, 2015 5:31 pm

Hello and welcome to the forums!

Your configuration looks good in general. 150 MB/s is not a bad performance by the way.

What is your full bottleneck statistics?
What transport mode is used? It can be recognized by [nbd], [hotadd] or [Direct SAN] tag in the jobs` actions.

Thanks!
Shestakov
Veeam Software
 
Posts: 4850
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Veeam 8 Performance Issue

Veeam Logoby tsightler » Sat Oct 17, 2015 5:37 pm

What happens if you disable one of the proxies, just as a test? Do the rates change? Does the bottleneck change?

Also, what is the actual data path. You say the "Network" is 10GbE, but is ingress and egress on the same interface? What about DirectSAN, is that iSCSI of FC over and HBA?

When you say that the performance is 150MB is that aggregate across all jobs together or are running mulitple jobs and seeing that performance on each?

Also, have you manually selected a gateway or did you leave it on "Automatic selection"?
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Veeam 8 Performance Issue

Veeam Logoby Jack1874 » Sat Oct 17, 2015 7:01 pm

Hi Shestakov, thanks for the welcome ...

The transport mode is [Direct SAN]

I'll post you the bottlneck statistics very soon... as I have some jobs just running. brb ...
Jack1874
Enthusiast
 
Posts: 88
Liked: 4 times
Joined: Sat Oct 17, 2015 3:32 pm
Location: Canada
Full Name: Stuart Little

Re: Veeam 8 Performance Issue

Veeam Logoby Jack1874 » Sat Oct 17, 2015 9:08 pm

Hi Shestakov, the 150 MB/s is the speed of some the individual copies inside the jobs.

On one of the proxies we just seen an average of 53 MB/s for the entire job. The job took 7 hours 20 mins and it tells is that the network is the bottleneck.

We processed 2 TB of data, we read 1.3 TB of data, and transferred 1.1TB of data.

There were other jobs running, but they were on the "other" proxy targeting different shares on the StoreOnce (using same 10 GBe uplink) on StoreOnce.


Thoughts?
Jack1874
Enthusiast
 
Posts: 88
Liked: 4 times
Joined: Sat Oct 17, 2015 3:32 pm
Location: Canada
Full Name: Stuart Little

Re: Veeam 8 Performance Issue

Veeam Logoby Jack1874 » Sat Oct 17, 2015 9:14 pm

Hi tsightler, we are using DirectSAN over F/C

My assumption is that we are using the 10GbE for both inbound and outbound.

I meant that we are seeing 150 MB/s that is on some individual tasks inside the job. The average is circa 53 MB/s. There are multiple jobs going to the HP StoreOnce repository but to different NFS shares.

We manually select the gateway.
Jack1874
Enthusiast
 
Posts: 88
Liked: 4 times
Joined: Sat Oct 17, 2015 3:32 pm
Location: Canada
Full Name: Stuart Little

Re: Veeam 8 Performance Issue

Veeam Logoby Shestakov » Sat Oct 17, 2015 10:07 pm

Thanks for the reply Jack,

I`m also curious, what is the percentage of "Network" in the bottleneck statistics versus "Source", "Proxy" and "Target"?
It helps understand how crucial the network bottleneck is.

Note that every infrastructure has a bottleneck, it doesn`t mean that something is wrong.

By the way 150 MB/s is 1.2 Gb/s what actually looks fast. Thanks!
Shestakov
Veeam Software
 
Posts: 4850
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Veeam 8 Performance Issue

Veeam Logoby tsightler » Sun Oct 18, 2015 1:15 am

Jack1874 wrote:I meant that we are seeing 150 MB/s that is on some individual tasks inside the job. The average is circa 53 MB/s. There are multiple jobs going to the HP StoreOnce repository but to different NFS shares.

I hate to keep asking questions, but the fact that you are seeing network bottleneck is telling me something, I just don't know what yet, so I'm trying to understand the dataflow. Network bottleneck refers to the traffic between the Veeam source and target data movers. Source data movers are the proxies, which are reading data via FC, and then sending that data to the target data mover, which is the process writing data to the StoreOnce.

But I'm confused by your answer above. You state that you are writing to NFS shares, so that would imply that there is a Linux system involved somewhere, but I don't see any mention of this anywhere. How are you writing via NFS to the StoreOnce?
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Veeam 8 Performance Issue

Veeam Logoby Jack1874 » Sun Oct 18, 2015 10:13 pm

We are using W2K12 (R2) as the proxies and have mounted the (SotreOnce) NFS shares to them. There is no Linux in our Veeam Architecture.

We decided to to use NFS as the thruput we were getting with CIFS was even worse than what we see with NFS shares
Jack1874
Enthusiast
 
Posts: 88
Liked: 4 times
Joined: Sat Oct 17, 2015 3:32 pm
Location: Canada
Full Name: Stuart Little

Re: Veeam 8 Performance Issue

Veeam Logoby chjones » Sun Oct 18, 2015 11:37 pm

We have a very similar setup to you (HP Blade Servers, 3PAR, StoreOnce, etc). My observation is:

What is the maximum concurrent tasks you have configured for the Veeam Repository? StoreOnce is great at ingesting data in a single stream, but not multiple streams at the same time on a NAS/CIFS Share. The best solution we found was to limit the number of concurrent streams to 12 in the backup console (we found this value specified in an old StoreOnce User Guide as the maximum supported NAS/CIFS streams supported).

We have two proxies, each with two eight-core CPUs (16 cores each) so can process 32 VMDKs with parallel processing enabled. We have the StoreOnce added as a CIFS Share. If we allow the proxies to flood the StoreOnce with 32 streams we see the performance nose-dive. Limiting to 12 seemed to give us the best of both worlds (we found this figure in an older StoreOnce User Guide).

Also important is the StoreOnce OS Version as older versions had performance issues specifically with Veeam. Anything higher than 3.11.4 I believe is best suited for Veeam backups. We are running 3.13.0 at present and have no issues (I believe 3.13.1 will be required for Veeam v9 and it's StoreOnce Catalyst integration).

If I am running a backup job of a VM with a single VMDK I can see upwards of 500MB/sec writing to the storeonce as it is a single stream of inbound data to the StoreOnce. As soon as other VMDKs are backed up at the same time the performance starts to drop, this is just the trade-off with using a dedupe appliance as the repository.

The average speed of my backup jobs (I have up to ten backup jobs all running at the same time but the concurrent operations set to 12, so usually about 4-7 VMs being processed at the same time as each VM usually has 2 or more VMDKs) and my jobs all average around 100-150MB/sec. Remember, that is up to 12 VMDKs being processed at the same time across a number of different jobs and each one processes at 100-150MB/sec.

To get the highest throughput possible you can disable parallel processing in your jobs or at the Veeam server level, however I wouldn't recommend doing this. I would rather process all my VMs at the same time than see my stat lines show higher throughput. As is with any file copy if you copy 1 file as opposed to 10 files at the same time, the performance overall will drop when doing multiple concurrent tasks. It's finding the sweet spot, which for us was 12 concurrent tasks (that is with a HP StoreOnce 4430 with 2 x 10GbE and 36 x 2TB 7200RPM SATA Disks).

As long as you are using the 3PAR Snapshot Integration then you have already offloaded the processing from your ESXi hosts so if the 3PAR Snapshots remains for, say, 20% longer whilst you process the data its not that big of a deal as there is no load on the ESXi hosts during the backup. That's how I look at it. My Active Full Backups on a weekend take about 24 hours to complete (some jobs finish in an hour or two whilst our Exchange 2013 VM which is 6.5TB takes almost a full day) but because the processing is away from the ESXi host I don't really mind.

When Veeam v9 comes with it's native StoreOnce Catalyst integration I am hoping to see about 3x improvement in backup performance. At the moment Veeam has to send every block to the StoreOnce and then the appliance decides whether it has to write it or can discard, in v9 that dedupe is offloaded to your proxy so only blocks required to be written are sent over to the StoreOnce. This dramatically increases the write performance as it allows the StoreOnce to simply act like any old NAS array and just concentrate on writing data, it doesn't have to do dedupe calculations.
chjones
Enthusiast
 
Posts: 83
Liked: 24 times
Joined: Tue Oct 30, 2012 7:53 pm
Full Name: Chris Jones

Re: Veeam 8 Performance Issue

Veeam Logoby Jack1874 » Mon Oct 19, 2015 3:23 pm

Hi CHjones,

I highlighted the amount of concurrent tasks on each proxy. Does that look ok to you? Should we lower that number ?

Proxy Servers 1
Type: Physical
HP DL380p Gen8
RAM = 64 GB
Network = 10 GBe
O/S = Windows 2012 R2
Transport Mode = Direct SAN access
Max Concurrent Tasks = 12

Proxy Servers 2
HP BL60c Gen9
RAM = 64 GB
Network = 10 GBe
O/S = Windows 2012 R2
Transport Mode = Direct SAN access
Max Concurrent Tasks = 16

StoreOnce Firmware Version
= Software Revision 3.13.0-1529.2
Jack1874
Enthusiast
 
Posts: 88
Liked: 4 times
Joined: Sat Oct 17, 2015 3:32 pm
Location: Canada
Full Name: Stuart Little

Re: Veeam 8 Performance Issue

Veeam Logoby Jack1874 » Mon Oct 19, 2015 3:29 pm

Hi , here are out statistics ...

Bottelneck: Network
Source = 11%
Proxy = 3 %
Network = 83 %
Target = 55 %

That is we are running multiple jobs and these are the stats for one of them
Jack1874
Enthusiast
 
Posts: 88
Liked: 4 times
Joined: Sat Oct 17, 2015 3:32 pm
Location: Canada
Full Name: Stuart Little

Re: Veeam 8 Performance Issue

Veeam Logoby Jack1874 » Mon Oct 19, 2015 3:35 pm

One other thing to add ...

We are configuring our jobs on a one-to-one relationship ...

Proxy Max Concurrent Tasks = 12
Repository Limit Max Concurrent Tasks = 4
Proxy1 --- >> NAS Share1
Proxy1--- >> NAS Share2
Proxy1 --- >> NAS Share3

Proxy Max Concurrent Tasks = 16
Repository Limit Max Concurrent Tasks = 4
Proxy2 --- >> NAS Share4
Proxy3--- >> NAS Share5
Proxy4 --- >> NAS Share6

Also, how about your NIC config on the StoreOnce?
On the StoreOnce we currently have tow 10GbE uplinks in fail-over mode. I'm thinking that having them as two separate uplinks is much better as we are looking for thruput and not redundancy. We would export one set of NFS shares on one uplink\IP and the other on another uplink\IP.

Thoughts?
Jack1874
Enthusiast
 
Posts: 88
Liked: 4 times
Joined: Sat Oct 17, 2015 3:32 pm
Location: Canada
Full Name: Stuart Little

Re: Veeam 8 Performance Issue

Veeam Logoby tsightler » Mon Oct 19, 2015 6:36 pm

None of the Veeam testing with HP Storeonce has been with Windows NFS client. I would expect that you should be able to get similar performance with CIFS by selecting the gateway to be the same box as the share is on as long as your StoreOnce has recent firmware.

However, we definitely need to figure out the network bottleneck, that part has nothing to do with the StoreOnce. Are you saying that you specifically select a proxy for each job, i.e. Job1 uses only Proxy 1 which uses NAS Share1? I wouldn't expect this to produce a network bottleneck. Are you definitely running v8 with the latest patch (at least update 2)? I'll try to pull your job logs and take a look.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Veeam 8 Performance Issue

Veeam Logoby Jack1874 » Mon Oct 19, 2015 7:35 pm

Hi tsightler, we have seen significant increases in performance when changing from CIFS to NFS. The StoreOnce is = Software Revision 3.13.0-1529.2

Yes. we are using 8.0.0.2030
Jack1874
Enthusiast
 
Posts: 88
Liked: 4 times
Joined: Sat Oct 17, 2015 3:32 pm
Location: Canada
Full Name: Stuart Little

Next

Return to VMware vSphere



Who is online

Users browsing this forum: Dima P., Exabot [Bot], stehei and 30 guests