Use of NIC when using Storage Infrastructure

[20100]

Hi

We are running 8 SP2 (was SP1 until yesterday) and are using HP 3PAR snaphot technology. This has been working really well since day 1.
We used to have 2 x 10Gb NIC teamed but had to remove the card in order to use a separate controller for the tape drive
We are now using the 4 x 1G build in 1Gb ports teamed.

We since noticed a huge drop in processing rate (not as significant as far as duration) from an average of 240Mb/s to 75Mb/s

We are a little perplexed as we did not think we were using the network to transfer data, and from we can see, the storage snapshots are used
(connectivity from Veeam to 3PAR is Fibre).

Is there something we overlooked or/and what do we need to ensure that all backup is done via the fibre card and not the Nic?

Thanks

[Vitaliy S.]

Hello,

In order to confirm that FC connection is used by the backup job, you can disable Microsoft iSCSI initiator service on the proxy server. Also make sure you have this setting is configured correctly > http://helpcenter.veeam.com/backup/80/v ... proxy.html

3. [Fibre Channel connection] The VMware backup proxy must have a Fibre channel adapter installed and must have access to WWN IDs used for Fibre Channel traffic on the storage system.
4. [Fibre channel connection] Fibre Channel devices must be properly installed and shown in Device Manager on the backup proxy. WWN IDs must be properly zoned on the Fibre Channel switch.
5. [Fibre channel connection] The VMware backup proxy must have access to LUNS with VM data and must be able to read data from these LUNs. To allow access for a VMware backup proxy, you must add the backup proxy the list of servers that have access to these LUNS.

Thank you!

[20100]

I checked everything and it looks fine. Note that the 3PAR has only Fibre not IscSi.

From Veeam, in the Storage Infrastructure, I can see the 3PAR and all its volume with WWN, reporting the size. I noticed the snapshot count is always 0 even during a backup and there is no snapshots in 'last snapshot' column. Perhaps it is normal.

I have noticed in Vcenter when a backup is performed, that Vmware does a snapshot, then release disk lease and then delete the snapshot, and this in a very short period of time. That indicates to me that the Vmware to 3PAR snapshot is taking place.

I am pretty sure Veeam also uses the 3PAR snapshot, as I can see "creating Storage snapshot" and "deleting storage snapshot"

What I do not understand is why the processing rate dropped dramatically when we changed from 10G to 1Gb NIC. I guess there must be some sort of dialog between VEEAM and Vcenter over the network at the beginning and end, but I would not expect that much traffic to make a difference.

[Vitaliy S.]

What do backup job bottleneck stats show after and before this change was made?

[20100]

Before on SP1, with 10G Nic the bottleneck was the Proxy with processing rate around 240Mb/s
when the 10Gb was replaced with 1Gb, the bottleneck was the Target with processing rate around 80Mb/s
With SP2, the bottleneck is now Network, with still the same 80Mb/s processing rate.

[Vitaliy S.]

Sorry for the late reply (was travelling), but can you also check the session details? What operation takes most of the time, data transfer or something else?

[20100]

Hi Vitaliy

I cannot really pinpoint the issue. As mentioned earlier in the post, the times do not seem that bad. Most of the 'longer times' are for 'processing' which I guess is the transfer.
The only difference between now with Gb cards instead of 10Gb is not really the duration of the backup but the way it reports the processing rate which never seem to go higher than 60Mb, when before it averaged around 200mb.

So, in terms of operational status, everything seems to be OK with backup times roughly similar to what it used to be. I was just curious why it started to report such slower processing rate.

I have tested a restore with the 1Gb cards and of course, it does make a huge difference compared with the 10gb cards, but this I would expect as we are using the LAN for restore.

[PTide]

Hi, 20100

You've mentioned "NIC teaming". How do you team them?

[Vitaliy S.]

I have tested a restore with the 1Gb cards and of course, it does make a huge difference compared with the 10gb cards, but this I would expect as we are using the LAN for restore.

When you're doing full VM restore it goes through the ESXi management interface which is throttled by default, I thought there should be pretty much the same performance in both configurations.

I cannot really pinpoint the issue. As mentioned earlier in the post, the times do not seem that bad. Most of the 'longer times' are for 'processing' which I guess is the transfer.
The only difference between now with Gb cards instead of 10Gb is not really the duration of the backup but the way it reports the processing rate which never seem to go higher than 60Mb, when before it averaged around 200mb.

If you need further assistance with this, our support team can take a look at the logs and find out how the traffic goes in your case.

[20100]

Hi Pavel

By NIC Teaming, I mean that in Windows 2012 servers, Server Manager, Local Server, NIC Teaming is enabled. All the NIC cards are active, with teaming mode set to 'switch independent, with load balancing mode set to Dynamic)

Cheers

[PTide]

20100,

So, you have 4 NICs teamed. Just to clarify - do you use them for iSCSI?

80Mb/s looks like a average speed for a single 1Gb link. You've mentioned that

So, in terms of operational status, everything seems to be OK with backup times roughly similar to what it used to be. I was just curious why it started to report such slower processing rate.

Looks like for some reason your 4x1Gb NICs are treated if there was one link instead of four, nevertheless they really work like 4x1Gb NICs. I'm not a pro in Windows, however I've had similar problems with NICs teaming in Linux, especially if those were used for things like iSCSI of NFS.

I think that you can do three things:

1) As Vitaliy has already suggested, please contact support team.
2) Play a little bit with teaming settings and see if anything changes. I'd try to use a "Standby Adapter" mode first in order to compare real processing time and see if it reports a different speed.
3) You may want to try using some kind of traffic analyzer to gather some statistics and see how and where the traffic flows. i.e. Wireshark can do the job.

Thank you.

[20100]

Hi

No the NIC are only for Ethernet. We use 3PAR snapshots for backup and I am 99% sure that all data are using the Fibre to the 3PAR.
That was really what triggered my curiosity: when I had to remove the 10Gb NIC to use the 1Gb one, the processing rate dropped dramatically and as far as I knew, the data transfer is performed via fibre between the Veeam server and the 3PAR.

I will be on leave soon for several weeks, and will log a case with support on my return

Cheers

[dellock6]

Re-reading the thread seems we overlooked another component: 3Par and Veeam proxy are connected via FC thus using DirectSAN, but then what is the final destination of the backups, the repository? Is the Veeam server also a repository or it sends data outside via network to another machine?

[20100]

Hi
Yes the Veeam server has all the roles and is also the repository.
It is an HP StoreEasy 1800 serie

Cheers

[dellock6]

We have some enhancements to improve the performances of communications between proxy and repository when they are on the same machine:
http://www.veeam.com/kb1982
I have a feeling the section "performance enhancements" could may be a solution for you. If you feel unsure, as suggested open a support ticket and talk with the support engineer about this feature we added.

Luca

[PTide]

Hi 20100,

No the NIC are only for Ethernet.

I'm very sorry, but I think I don't understand what you mean =)

as far as I knew, the data transfer is performed via fibre between the Veeam server and the 3PAR.

Well, I'd recommend you to double check that, because, as you've stated before:

"when I had to remove the 10Gb NIC to use the 1Gb one, the processing rate dropped dramatically"

on the other hand I feel some contradiction here:

"The only difference between now with Gb cards instead of 10Gb is not really the duration of the backup but the way it reports the processing rate"

Assuming that NICs replacement is the only thing that had happened to your infrastructure before the isuue ocurred, I'm 99% sure that such a strange behaviour has something to do with teaming.

However, it's hard to see all the details from here, and I think the better choice would be to wait for a support team's response.

Thank you.