[SOLVED] Veeam and VMware 6.7

[Cloudmonkey]

Not quite an answer to the question, but the later version of VCSA 6.5 is based on Photon and the HTML5 UI is actually fairly identical to 6.7. Certainly more featured than the first 6.5 HTML UI. Obviously it works just fine with 9.5u3 so perhaps a handy stepping stone until 6.7 is supported.

[KKCChip]

I was able to get Veeam to work in VA mode after I disabled any Win 7/Server 2008 proxies. I haven't tested on Win 10, but Server 2012 and 2016 worked.

[nilsbeyer]

(...) For example, we have just found one critical bug with hot add leading to data corruption, and now need to build a code to work around this one.

Hmm, that's not good. Can you provide us 6.7 users at least with a hotfix for that bug so that we can have reliable hot-add based backups/replications until a new Veeam release has been published? Unfortunately, I am forced to use ESXi 6.7 because here, I have so fresh hardware where ESXi 6.5Ux crashes instantly; and 6.7 already has gone productively...

[Gostev]

Based on the complexity of this fix, most likely we will not be able to provide one as a hot fix.

In your situation, general recommendation is to use Veeam Agents for Windows and Linux until the official vSphere 6.7 support is released as a part of Veeam Backup & Replication Update 3a. Because even if we could provide the hot fix for this particular bug, there's no guarantee there are no other bugs leading to data corruption exist (yet to be found).

Thanks!

[nilsbeyer]

In your situation, general recommendation is to use Veeam Agents for Windows and Linux until the official vSphere 6.7 support is released as a part of Veeam Backup & Replication Update 3a.

So, you suggest that I should switch to Veeam Agents for Windows resp. for Linux instead of Veeam Proxies?

[Gostev]

Correct. By coincidence, there's a free 3 months agent subscription promo going right now.

[rstoner]

No, normally updates do not require any code changes from us, as they bring nothing but bug fixes.

Yes, usually that is correct but in this update (6.5 U2) VMware back ported some features from 6.7. Infact if you do install the update, you won't be able to upgrade to 6.7 GA. That's why I asked if we should wait before upgrading our infrastructure this time round to U2. Maybe VBR 9.5 3a will be able to handle all features.

[jamerson]

as I believen its should works on 6.5 U2 with the latest VEEAM.
i'v test it on a LAB and it does works i can backup and restore.
maybe someone else can correct me if i am wrong.

[foggy]

Hi Julien, there's a separate topic on 6.5 U2 support, please keep an eye on it.

[pnunn]

Thank you, without realizing it I find myself on the bleeding edge with the lab setup I've just spun up to test all of this out before deploying it to two data centers. Fixed the problem I was having,

Peter.

[pnunn]

Spoke too soon. Fixed for the linux servers, still broken for the windows ones...

Peter.

[jlhall1000]

Slightly off topic but in the beta testing did you see any major performance increases?

I've read multiple articles relating to improvements with the NBD

[Gostev]

Yes, but with certain caveats... see today's forum digest

[dairuin]

Hi all,
any news about release of this patch ?

thanks

[thalooka]

Yeah checking the news in forum daily for anything new since 6.7 was wanted by an pretty big customer against our advice and now they are using backup agents for just the important vm's

[Gostev]

Full regression testing cycle has now been completed, and the list of impacted transport modes (in addition to nbd) now also includes direct SAN and hot add... so yes, basically all of them got hit

The good news is that we should be able to work around most of the issues, possibly except one - but that will cause minor feature loss only. The bad news is that some may require VDDK patching, which is a fairly costly way to fix issues in terms of dev and QC time. Nevertheless, we should still be able to ship the update late next month.

The biggest unknown that may potentially impact the vSphere 6.7 support timelines right now is actually vSphere 6.5 U2 (support for which shares the same release vehicle), as the issue there is much more complex and looks to sit in ESXi itself - so not something we can easily fix or work around from our end. Of course, in worst case scenario we can ship Update 3a with 6.7 support only, postponing 6.5 U2 support until 6.5 U2a or something becomes is available... it is just that troubleshooting this vSphere 6.5 U2 issue takes a lot of our time (no material response from VMware Support for 10 days now, so we keep digging the issue on our end).

[EdFromOhio]

At the risk of sounding unappreciative of the hard work Veeam does, it seems ridiculous that it takes 2 full months after a public update to release an updated version of Veeam. And I am grateful for knowing when to expect an update. But what's the point in being a VMware partner if you wait until their production release to at least get a good head start on the next version?

Yes, 6.7 was a more significant update, but all my other vendors have already become 6.7 compliant. I made the mistake of trying 6.7 on my vCenter server and a couple of hosts, and now one of them can't be reverted. Due to my oversight in thinking that Veeam would be quick about an update, I'm dead in the water on 1 of my hosts and I'm not going through the pain of installing a bunch of agents to deal with it. To know that we're looking at another 5+ weeks is disappointing to say the least.

What makes that sting even more is Veeam is always pushing super hard on marketing like VeeamOn and lots of other marketing shows/parties, but it seems like the actual products aren't given nearly as much internal priority. Every person on the board of Veeam is in sales, marketing or finance, but not any significant technical level.

[Gostev]

But what's the point in being a VMware partner if you wait until their production release to at least get a good head start on the next version?

In fact, we don't wait. We did start almost half a year ago, as soon as vSphere 6.7 beta dropped. If we did not, we would not be able to ship vSphere 6.7 support until this fall due to the amount of work required to support new vSphere Web Client (which is a complete rewrite with all new API).

To know that we're looking at another 5+ weeks is disappointing to say the least.

I agree and we would love to change that as well. Please help us push VMware from your side to share the RTM code and final API documentation with its partners in advance, like Microsoft does - instead of on the same day with everyone else. This will solve this frustration for everyone.

it seems like the actual products aren't given nearly as much internal priority

I can assure you that it's a wrong impression. Veeam products are universally considered being one of the best on the market, both feature and reliability wise - which is simply impossible to achieve without giving them much internal priority and full focus.

Every person on the board of Veeam is in sales, marketing or finance, but not any significant technical level.

Now, this is simply not true. The reality is quite opposite: R&D is actually the only department at Veeam which never in the history of Veeam had a hiring cap. We literally NEVER stop hiring devs. The fact that you don't interact with our R&D as a part of pre- and post-sales process does not mean they are nearly non-existent

[paul]

OK so I don't post here much but I've been following these forums for many years now... and what blows my mind is the amount of patience Gostev has answering all the same low blows from people who jumped a gun of yet another vSphere release. They basically give him the exact same sh*t over, and over, and over again. And he still politely responds to all this nonsense - for years - unbelievable!! If I were him, I would lose it back a few vSphere releases ago

[Gostev]

1000 new customers every week... gotta be patient and welcoming

[Regnor]

I can assure you that it's a wrong impression. Veeam products are universally considered being one of the best on the market, both feature and reliability wise - which is simply impossible to achieve without giving them much internal priority and full focus.

+1

I also hate waiting for a new release/update, but as long as you maintain the stability I don't care if it takes 1 or 3 months

[EdFromOhio]

Gostev, thank you for your reply and clarification. We've been a Veeam customer for over 2 years, so we're not new. We just haven't been in this long of a delay before.

And Paul, while I appreciate your viewpoint, my beef isn't with Gostev. My frustration is with the lengthy delays between a VMware release and a Veeam update. There's no notice whatsoever on their site to let you know that they aren't yet functional on the 6.7 update. That's buried in the forums or your own testing. It should be more clear to all customers, not just long-time ones. The update prior to this latest one for B&R was almost a full year. It's not like we're all paying a few hundred bucks for support. Veeam is one of the most expensive software packages we use at my relatively small company, so the perception of cost vs. value definitely comes into discussion at every renewal.

And I will agree with you on one thing - this is a recurring subject. It is because it is a recurring issue which directly affects customers. It's easy to sit back and say that it's someone's fault for not doing all their homework, and you would have a point. However, with Veeam's resources, it should not take a full 2 months after a production release to update their product. Two weeks I could understand. At the very least, put up a notice to clearly say that Veeam isn't ready for 6.7 and an expected release time-frame so customers can plan accordingly.

[Andreas Neufert]

It is clearly stated in our documentation and release notes which VMware Versions we support and not.
https://helpcenter.veeam.com/docs/backu ... tml?ver=95
https://www.veeam.com/veeam_backup_9_5_ ... tes_rn.pdf Page 2

There is no way in the release model of VMware that development partners can be day 0 ready. As VMware do not handover the final code to dev partners upfront.
Usually VMware do some last minute changes that affect backup vendors heavily. For example at 6.5 release they changed in the release that NBD mode is now NDBSSL mandatory. We had to fix it in the last minute.

As Anton shared in the weekly digest we are waiting for some VMware Support cases to be addressed by VMware.

We always do full regression testing against the final version (which itself takes longer than 2 weeks to test all the thousands of different scenarios... otherwise we would end up in bad user experience with our 290.000 customers.

It doesn´t matter if it is upgrade to next windows version, Office Version or VMware version, it is always a process that need planning to not break anything in the environment.

[Andreas Neufert]

For those how have already updated to 6.7. You can maybe use a trial version or 3 month production license for free of our Agents:
https://www.veeam.com/windows-linux-ava ... gents.html

[Gostev]

We've been a Veeam customer for over 2 years, so we're not new. We just haven't been in this long of a delay before.

So if you were with Veeam for over 2 years now, then you definitely already saw us releasing vSphere 6.5 support on 18.01.2017, which was a little over 2 months of vSphere 6.5 GA that happened on 15.11.2016. Which I can tell you is about the same time frame as with all the previous vSphere releases, anyway.

Now, vSphere 6.7 was released on 17.04.18, so our late June release will fit the same support time frame as we had historically for many years. Given these facts, I am not sure what "this long of a delay" are you talking about?

[csydas]

There's no notice whatsoever on their site to let you know that they aren't yet functional on the 6.7 update.

Out of curiosity Ed, what method would you prefer?

I'm a very slow and cautious guy with my production shop - we check and check and check and double check before we make changes, because backups are the one thing that just have to work no matter what. In my mind, backups should be like a screwdriver or a hammer - you don't necessarily do anything fancy with them, but when you pick them up and turn/hit, they work, because that's all they do.

So when there's an update announced from VMware or Microsoft, I wait until there is published documentation from Veeam (and our few other side products) until we touch anything. Why mess with what isn't broken? For me, the platform support page is the authoritative document that tells me whether or not I should even be thinking about updating, not whether I will or not. But if that's not what works for you, what would be the best method to tell you "Hey, 6.x is released, we're not ready yet for it." As an IT provider, I know that's what I'd be asking if I were in Gostev's shoes.

As a customer, I see VMware updates (and for our small Hyper V cluster, Microsoft updates), as threats. Hearing that the partners for VMware don't get GA code until public release is absolutely absurd in my mind (directed @ VMware, not Veeam), and for me it's one of those things where VMware and Microsoft have to prove first in public, non-lab settings that their stuff works before I'm ready to commit. We're a medium sized shop, but we don't have the space for the necessary redundancy on some of our biggest servers, so we're heavily reliant on our backups in case "it" hits the fan. I would never dream of a Day1 update, much less pushing it to my production servers.

I know, hindsight is 20/20 here, and if I were in your position, I'd be a mixture of furious and beating myself up in between glasses of scotch. But really, I wouldn't know where else to put the blame but me.

[EdFromOhio]

Given these facts, I am not sure what "this long of a delay" are you talking about?

Two full months is a long time in software development. Just because that's the way it has always been doesn't mean it can't be improved, especially with Veeam's resources.

[EdFromOhio]

Out of curiosity Ed, what method would you prefer?

A simple notice on their web site would be helpful. And to Andreas Neufert, Veeam knows that their product doesn't work with the VMware update that came out a month ago. If you know your customers are going to be looking for that information, why not just have a simple one-sentence statement to the effect that "Veeam anticipates having an update to work with VMware 6.7 on June 30" or something like that? The site should be customer-centric, not employee-centric.

I know, hindsight is 20/20 here, and if I were in your position, I'd be a mixture of furious and beating myself up in between glasses of scotch. But really, I wouldn't know where else to put the blame but me.

And I'm not disagreeing that I jumped the gun. That's 100% my bad, and I'm living with that consequence every day until this update is available. Veeam support was an afterthought because my other vendors with more intertwined VMware wiring had their products updated weeks ago, so I made the very stupid assumption that Veeam would have no issues.

Just to clarify, I'm not trying to roast anyone. I'm simply venting frustration that 2 months is a long time, and just because that's the way it has historically been doesn't mean that improvements can't be made either internally and/or with the relationship with VMware. From a customer's perspective, it *appears* that Veeam is more interested in parties and VeeamOn rather than updating products since the last version was out about 10 months without an update. Thanks to Gostev for his insight, that perception has been adjusted.

[zimeon]

Any chance for a beta release to put on our test/dev/homelabs?

[Gostev]

We don't do betas for updates based on past experience, instead we do pilot deployments. These still have to wait until until QC signs off the RTM build, but for the following few weeks we only distribute the build through our support to ensure we did not accidentally overlook or introduce any major issues. These builds typically go to customers with support cases open on one of bugs that the given update fixes. I will let everyone here know when the RTM build ships, so that you could reach out to support to get one. Thanks!