Veeam 9 Update 1 and ESX 6 Update 2

[mkretzer]

Hello,

is Veeam Update 1 supported with ESX 6 Update 2?

Markus

[Gostev]

Hello, we have not completed the testing it yet, as it just became available today. We will update once we hear back from our QC. Thanks!

[mkretzer]

Hello,

at least VC 6U2 seems to work with Veeam 9U1.

Markus

[Gostev]

Indeed, usually these updates are non-breaking, and until recently we did not even test them before declaring support, as they never ever broke anything. However, we had to change our practices recently due to all the major quality issues that have started plaguing VMware in 2015, so now we do require at least "smoke testing" to complete on our side before declaring the official support. Thanks!

[mkretzer]

That means if we have any issue we will not get help now?

Since we have to install new ESX today we thought it might be best to use the new version...

How long will Veeam need for initial tests? Should we install ESX 6 Update1b with all patches but update2 on the new hosts? That VC is already updated should not be an issue as data is transfered by ESX server correct?

What do you recommend?

Markus

[Gostev]

No, we certainly will not turn you in support just because you are using Update 2. Rather, we will investigate the issue with you and if it turns to be caused by Update 2, we will provide you the required hot fix (if the issue can be fixed on our side). In the end, it is in our best interest to become compatible with Update 2 ASAP.

I recommend going with Update 2 in your case, since the risk is quite low (it should not have too many differences with Update 1b, since it's been just a few weeks since the latter was released).

All initial tests should be completed by the end of the week.

[tinto1970]

I'm taking the risk too, in a little and not as critical cluster, because I'm worried about some issues u2 should resolve (i.e. multipath related).

Tomorrow morning i will post if something went wrong or not

[Gostev]

After the first day of testing we have not seen any issues so far.

[rreed]

Glad I stumbled onto this, we'll be rolling VMware u2 probably next week per an internal requirement - THAT SNMP ACTUALLY WORK W/ VCENTERS W/ EXTERNAL PSC'S. Anyways, likely by end of next week or early week after I can help report back if Veeam gets bent out of shape about it.

At that, I have not yet rolled Veeam u1 but need to look at the release notes to see if I should.

[tinto1970]

all went fine here with u2 (vcenter on windows, embedded, datastore iscsi)

[Nobody]

Are you also doing QC against Veeam 8 Update 3 and ESXi 6.0u2?

We are not on Veeam 9 yet and will be forced to update to vCenter 6.0u2 as soon as possible.
vCenter 6.0u1 was a ton of bugs and u2 makes it slightly better, but still some remaining.

I'm glad that we do not suffer the same pain with backups as we have with the vCenter...

[tinto1970]

We are not on Veeam 9.

why?

[Nobody]

why?

Still working on fixing VMware Bugs, no time to update working Veeam

[Gostev]

Are you also doing QC against Veeam 8 Update 3 and ESXi 6.0u2?

No, we are only testing the current version. However, I don't foresee v8 U3 having any issues if v9 does not have them, which is the case so far.

[mkretzer]

Ok no issues here as well with update 2.

Using Veeam 9U1 and direct SAN backup.

[mwvme]

Want to mention that I have been running v9 on U2 since the day it came out. No issues for me. All good.

Michael

[jamerson]

can we update to u2 now or have to wait for veeam ?
internally we have to update to u2 for security reasons but if the backup is not working than we can't .

[ashleyw]

unfortunately we've updated our development system - we are seeing terrible performance and blocking IOs.

We read down the support notes and see this bomb shell:

General Storage Issues

New Issue ESXi 6.0 Update 2 hosts connected to certain storage arrays with a particular version of the firmware might see I/O timeouts and subsequent aborts
When ESXi 6.0 Update 2 hosts connected to certain storage arrays with a particular version of the firmware send requests for SMART data to the storage array, and if the array responds with a PDL error, the PDL response behavior in 6.0 update 2 might result in a condition where these failed commands are continuously retried thereby blocking other commands. This error results in widespread I/O timeouts and subsequent aborts.

Also, the ESXi hosts might take a long time to reconnect to the vCenter Server after reboot or the hosts might go into a Not Responding state in the vCenter Server. Storage-related tasks such as HBA rescan might take a very long time to complete.

Workaround: To resolve this issue, see Knowledge Base article 2133286.

We are running Cisco UCS hooking up to HP MSA and IBM DS SANs (both via fibre).
We are seeing the same behavior after disabling the smartd daemon to see if that helps.

Code: Select all

2016-03-21T00:22:51.935Z cpu8:38771)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x12 (0x439dc0e690c0, 0) to dev "naa.600c0ff000da597fba0a954b01000000" on path vmhba1:C0:T1:L4" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

yep - VMware seem to have dropped another lemon of a release.
any suppor tnote that starts off with;

This is a known issue affecting VMware ESXi 6.0.
Currently, there is no resolution.

is a good reason to hang back....sigh.

[Gostev]

Hi, Ashley. Huge thanks for sharing your experience with the community! I bet this will save many people some sleepless night.

[Gostev]

can we update to u2 now or have to wait for veeam ?
internally we have to update to u2 for security reasons but if the backup is not working than we can't .

In regards to Veeam functionality, we're not seeing any issues in our testing so far.

[ashleyw]

We have ran further tests on our farm.
We disabled the smartd process as per the URL;
https://kb.vmware.com/selfservice/micro ... Id=2133286

We found the performance at the storage layer was still abysmal and we were seeing IO blocks still.
We then decided to reboot the host after disabling the smartd service, and now the servers are running fine after this.
So I can only guess there must be something in the start up sequence that changes behavior when the smartd service id disabled at boot time.

For a minute there we thought we'd have to roll back the hypervisors! phew!

[JoseKo]

I've updated VCenter and hosts to ESXi 6U2 -- all seems fine. My biggest question is whether I am able to update my storage from VSAN 6.1 to VSAN 6.2, also known as "disk format version 3".

ESXi6U2 corrects lots of problems with previous version, therefore it is a necessity for us to embrace U2 rather soon. We appreciate Veeam's collaboration to validate the new environment as son a possible.

[homerjnick]

Jeepers...we are on ESXi 5.1 with View 5.3.4 with Veeam 9 all on IBM HS22 Blades running off an IBM Storwize V5000...

Awaiting to install Cisco UCS Blades where we were planning to go to ESXi 6U2, View 6.2.2 with Veeam 9U1 but this project will stall as we keep waiting for a stable release of ESXi6!

The fact that the IO error is seen on IBM SAN's with Cisco UCS blades means at the moment NO CHANCE!

And VMWare's very on KB Support site is down at present!

[skrause]

If you are running Storwize, you won't have the same experiences as the other user who mentioned they were having issues with his IBM SAN as he said he is running a D series which is a totally different codebase than the Storwize/SVC code.

[homerjnick]

Yeah but I can't access the KB to see what storage/vendors are affected...a list would be great!

EDIT...link is back up with no mention as to which vendors affected...seems you either need to disable the SMART daemon or contact your storage vendor to see if they have a firmware to stop the PDL...

[lbenaglia]

Hi all,

Yesterday I applied the Update 2 on two different vSphere 6 clusters:

1) 12 Cisco B230 M2, UCS Manager 2.2(6d), Cisco UCS 6248UP and Nexus 5548UP connected to Pure Storage //m20 via FC (Purity 4.5.5)
2) 4 HP DL180 Gen9, switch HP FF 5700-40XG-2QSFP+, HP SN6000B 16Gb 48-port/48-port Active Fibre Channel Switch conneted to Pure Storage FA-405 (Purity 4.5.5)

In both cases I don't have any IO or backup issues using Veeam 8 Update 3. I'm waiting for Veeam 9 Update 1 to upgrade the backup servers and the proxies.

Ciao!

--
Lorenzo

[m.novelli]

Dell PowerVault MD3460 with SAS connections and December 2015 firmware seems to be NOT affected, vSphere upgraded 4 days ago, so far no issues.

Dell PowerVault MD3x owners should be on the safe side

Ciao, Marco

[Gostev]

Veeam 9 Update 1 will become generally available tomorrow, and we still have no issues to report on ESX 6 Update 2 based on our testing to date. Granted, we are testing only with Veeam 9 Update 1, but it should not make any difference comparing to Veeam 8 Update 3. Thanks!

[lbenaglia]

Today Cody Hosterman confirmed in this post that Pure Storage Flash Arrays are not susceptible to the PDL bug:
http://www.codyhosterman.com/2016/03/re ... lasharray/

Ciao!

--
Lorenzo

[danielvaknin]

Hi all,

We are considering an upgrade to our vCenter server from vCenter 5.5 U2 to vCenter 6.0 U1/2.
I saw that U2 just came out a few days ago and wanted to know if it is compatible with our Veeam version.
We are using Veam B&R v8 U3 (build 2084).

Thanks!