bind IP to restore job

[joergr]

Hi there,
my veeam server is equiped with two 10gbe nics, one looking at the 172.16.x.x/16 network and one looking in the 10.10.0.0/19 network. my esxi and san machines are located in the 172.16.x.x/16 network. When backing up everything is great, veeam server uses the 172.16/16 network, thus has NOT to use a router. Good. BUT when doing a restore to an esxi machine it suddenly uses the 10.10.0.0/19 interface and thus the 1gig router is limiting the speed. How can i tell b+r 5 to use the 172?
Best regards,
Joerg

[Alexey D.]

Hello Joerg,

Please make a little test: open datastore browser from backup server and try to upload a file to that ESXi.
Is traffic also going through 10.10.?

Also, please post "route print" output, let's take a look.

[joergr]

Here you go

Strange: It uses the 172 interface when backing up (over san OR over network) but when trying to restore it uses 10 interface.

best regards,
Joerg


route print
===========================================================================
Schnittstellenliste
20...00 10 18 7e cb 58 ......Broadcom BCM57711 NetXtreme II 10 GigE (NDIS VBD C
lient) #2
19...00 10 18 7e cb 5a ......Broadcom BCM57711 NetXtreme II 10 GigE (NDIS VBD C
lient)
1...........................Software Loopback Interface 1
13...00 00 00 00 00 00 00 e0 Microsoft-ISATAP-Adapter
14...00 00 00 00 00 00 00 e0 Microsoft-ISATAP-Adapter #2
===========================================================================

IPv4-Routentabelle
===========================================================================
Aktive Routen:
Netzwerkziel Netzwerkmaske Gateway Schnittstelle Metrik
0.0.0.0 0.0.0.0 10.10.0.201 10.10.1.44 261
10.10.0.0 255.255.224.0 Auf Verbindung 10.10.1.44 261
10.10.1.44 255.255.255.255 Auf Verbindung 10.10.1.44 261
10.10.31.255 255.255.255.255 Auf Verbindung 10.10.1.44 261
127.0.0.0 255.0.0.0 Auf Verbindung 127.0.0.1 306
127.0.0.1 255.255.255.255 Auf Verbindung 127.0.0.1 306
127.255.255.255 255.255.255.255 Auf Verbindung 127.0.0.1 306
172.16.0.0 255.255.0.0 Auf Verbindung 172.16.150.95 261
172.16.150.95 255.255.255.255 Auf Verbindung 172.16.150.95 261
172.16.255.255 255.255.255.255 Auf Verbindung 172.16.150.95 261
224.0.0.0 240.0.0.0 Auf Verbindung 127.0.0.1 306
224.0.0.0 240.0.0.0 Auf Verbindung 10.10.1.44 261
224.0.0.0 240.0.0.0 Auf Verbindung 172.16.150.95 261
255.255.255.255 255.255.255.255 Auf Verbindung 127.0.0.1 306
255.255.255.255 255.255.255.255 Auf Verbindung 10.10.1.44 261
255.255.255.255 255.255.255.255 Auf Verbindung 172.16.150.95 261
===========================================================================
Ständige Routen:
Netzwerkadresse Netzmaske Gatewayadresse Metrik
0.0.0.0 0.0.0.0 10.10.0.201 Standard
===========================================================================

IPv6-Routentabelle
===========================================================================
Aktive Routen:
If Metrik Netzwerkziel Gateway
1 306 ::1/128 Auf Verbindung
20 261 fe80::/64 Auf Verbindung
20 261 fe80::2c88:c522:95f1:a28f/128
Auf Verbindung
1 306 ff00::/8 Auf Verbindung
20 261 ff00::/8 Auf Verbindung
===========================================================================
Ständige Routen:
Keine

[Alexey D.]

We need more information

Please provide "ipconfig /all" output and a screenshot with network configuration of your target ESXi.
Have you tried the test I had proposed in my first post?

[joergr]

okidoki (test follows):

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\administrator.ZSPHAINA>ipconfig /all

Windows-IP-Konfiguration

Hostname . . . . . . . . . . . . : vhesxbackup1
Primäres DNS-Suffix . . . . . . . : zsphaina.msft
Knotentyp . . . . . . . . . . . . : Hybrid
IP-Routing aktiviert . . . . . . : Nein
WINS-Proxy aktiviert . . . . . . : Nein
DNS-Suffixsuchliste . . . . . . . : zsphaina.msft

Ethernet-Adapter LAN-10GB:

Verbindungsspezifisches DNS-Suffix: zsphaina.msft
Beschreibung. . . . . . . . . . . : Broadcom BCM57711 NetXtreme II 10 GigE (N
DIS VBD Client) #2
Physikalische Adresse . . . . . . : 00-10-18-7E-CB-58
DHCP aktiviert. . . . . . . . . . : Nein
Autokonfiguration aktiviert . . . : Ja
Verbindungslokale IPv6-Adresse . : fe80::2c88:c522:95f1:a28f%20(Bevorzugt)
IPv4-Adresse . . . . . . . . . . : 10.10.1.44(Bevorzugt)
Subnetzmaske . . . . . . . . . . : 255.255.224.0
Standardgateway . . . . . . . . . : 10.10.0.201
DHCPv6-IAID . . . . . . . . . . . : 452988952
DHCPv6-Client-DUID. . . . . . . . : 00-01-00-01-14-4D-BF-D5-B8-AC-6F-87-D0-3C

DNS-Server . . . . . . . . . . . : 10.10.1.20
10.10.1.26
Primärer WINS-Server. . . . . . . : 10.10.1.20
Sekundärer WINS-Server. . . . . . : 10.10.1.26
NetBIOS über TCP/IP . . . . . . . : Aktiviert

Ethernet-Adapter iSCSI-10GB:

Verbindungsspezifisches DNS-Suffix:
Beschreibung. . . . . . . . . . . : Broadcom BCM57711 NetXtreme II 10 GigE (N
DIS VBD Client)
Physikalische Adresse . . . . . . : 00-10-18-7E-CB-5A
DHCP aktiviert. . . . . . . . . . : Nein
Autokonfiguration aktiviert . . . : Ja
IPv4-Adresse . . . . . . . . . . : 172.16.150.95(Bevorzugt)
Subnetzmaske . . . . . . . . . . : 255.255.0.0
Standardgateway . . . . . . . . . :
NetBIOS über TCP/IP . . . . . . . : Aktiviert

Tunneladapter isatap.zsphaina.msft:

Medienstatus. . . . . . . . . . . : Medium getrennt
Verbindungsspezifisches DNS-Suffix: zsphaina.msft
Beschreibung. . . . . . . . . . . : Microsoft-ISATAP-Adapter
Physikalische Adresse . . . . . . : 00-00-00-00-00-00-00-E0
DHCP aktiviert. . . . . . . . . . : Nein
Autokonfiguration aktiviert . . . : Ja

Tunneladapter isatap.{2222C0B9-C4C7-467F-94D2-5278FE3EAFE9}:

Medienstatus. . . . . . . . . . . : Medium getrennt
Verbindungsspezifisches DNS-Suffix:
Beschreibung. . . . . . . . . . . : Microsoft-ISATAP-Adapter #2
Physikalische Adresse . . . . . . : 00-00-00-00-00-00-00-E0
DHCP aktiviert. . . . . . . . . . : Nein
Autokonfiguration aktiviert . . . : Ja

C:\Users\administrator.ZSPHAINA>

[joergr]

me again. It is an ESXi host. Thus, i cannot upload files to the host.

[Gostev]

You can - just use Datastore Browser in vSphere Client (right-click datastore under your host).

[joergr]

of course, but your collegaue wrote "open datastore browser from backup server ". Thus i tried with the backup Server. I have no vsphere client on the backup server. And the backup server is the only one with this special constellation. If i use my workstation to upload it uses 172, because the ws has 172, too.

[Gostev]

Can you install vSphere Client on backup server for this test?

[Alexey D.]

More questions/suggestions.

Have you added this ESXi as standalone host or through vCenter, to backup console?

You may also try this as a workaround: add an entry to 'hosts' file at your backup server and explicilty specify IP of ESXi.
(We also think it could be a problem with DNS resolution, as one possibility).

[joergr]

Hi,

added through vcenter to backup console. It is no DNS problem. I will install vSphere client for the test.

best regards,
Joerg

[joergr]

Did the test. It is slow like hell with vsphere client - BUT it uses the correct (172) IP!!!

Any ideas how to get B+R to use this IP also? (The strange thing is when backing up it uses the correct one, only during restore it uses the wrong one).

I will do some research

Best regards,
Joerg

[Alexey D.]

What is the output for "nslookup <esxi>"?

[joergr]

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\administrator.ZSPHAINA>nslookup esx6
Server: vh-dc1.zsphaina.msft
Address: 10.10.1.20

Name: esx6.zsphaina.msft
Address: 172.16.150.90

C:\Users\administrator.ZSPHAINA>

[Alexey D.]

Joerg, please send us the logs of backup and restore jobs.
Currently we are running out of ideas.

[joergr]

Research completed:

Every time i configure systems in the 10 network and selecting dedicated nics only for iscsi traffic i deselect microsoft client and file and print share from these interfaces to get this little bit more speed out of it and to save overhead. So i thought well maybe b+r 5 is somehow looking into these settings when doing a restore and i selected them again and rebooted. Et voila: When doing restores NOW it uses the 172 interface

best regards,
Joerg

PS the trigger gave a test request from Trevor Bell concerning restore speed of huge vms, which i can run now, because my router won´t be the bottleneck in a 10 GB net

[Gostev]

Interesting... I wonder why this helped...

[Alexey D.]

Wow!

B&R does not look into these settings, but I guess it pushes Windows to make the decision, which route to choose.

Glad you solved the issue.

[TrevorBell]

Looking forward for some results now joerg

[joergr]

will do it tomorrow
no time left today ;( we had an equallogic issue where i had to focus 100% on.

[joergr]

Okidoki, Test is running, a full restore (no instant restore) of a 515GB VM completely back to an esxi server. Will report as soon as it finishes....runnig for an hour now and guessing it needs another hour. Let´s see....

[Gostev]

Restores to ESXi are limited by ESXi itself... having 100Mb is no different from having 10Gb LAN here

[joergr]

Anton is right!!! ESXi is soooooo slow when doing direct network transfers, unbelievable....i hope VMware will do something about that in the near future.....;-(

Anton - let´s say i would do an instant recovery test with svmotion - i guess that would be a huge ammount faster, right? What is your experience?

best regards,
Joerg

[Gostev]

We do not hold high hopes that VMware will resolve this any time soon as this looks to be architectural limitation. We are working on addressing this from our side... we do have some ideas how this can be improved to make full VM restores speed significantly faster.

Yes, restore is definitely huge amount faster with Instant VM Recovery - it is naturally instant then you just kick in SVMotion (preferably during off-hours), and this will still take significant time to move the data from vPower NFS datastore to production, but since it happens in background while VM is already running, you no longer care how much time it will take.

[joergr]

Wow, it just failed at 99% (quite at the end, after two hours transfering)

When backing up the huge vhd, about 300gig, at the end it said: Restore Job failes. Server Error: NFC Server is busy. Delivery of a FILE_PUT message has failed.

LOGS are on the way to you, Anton.

best regards,
Joerg

[Gostev]

This looks like some issue with host, the error is returned from ESXi host. May be connection limit exceeded. Definitely not an issue with backup file, or actual restore restore process from our side.

[joergr]

I guess i found the problem - the target store was formatted with block size of 1 mb thus the max vmdk file should only have 256GB - but i tried to copy a 300gig to it. That may be the glitch. And a suggestion for the next release - test if the Target Datastore has a block size which will fit the vm Anyhow - the speed test found a result - i have a time window of 2 hours for a 500 Gig VM - not bad. I now try the instant vm recovery speed.

best regards,
Joerg

[Gostev]

Yes, this makes sense.

Actually, result is not too shabby at all, that is 70MB/s if my math is write. I never seen faster than 10MB/s in our labs for restores to ESXi over 1Gb LAN. Do you have battery backed write cache and all that cool stuff, or is this is solely due to 10Gb ethernet? Would be really, really cool if you could retest same restore on 1Gb LAN.

[TrevorBell]

So is esxi really the way to go or stick with esx 4.1 ?

One question Anton if you run the Instant VM when are any data changes going to be committed ? e.g instant vm is a busy file server users make changes all day and it takes 2 days to copy the data to a new lun, what do you do if anything to commit the changes so you dont lose any data ?

[Gostev]

Hi Trevor, I just answered the same question a few days ago here, please take a look.