[SOLVED] Veeam and VMware 6.7

[planetcoop]

@tsightler

I certainly understand as you mention code changes between RC/BETAs and GA. The RC worked with direct host connections from 9.5U3 but now does not. Clearly a change. The say way you describe testing is what my lab is for. We have started aggressively testing. Malware is important, vulnerabilities, you name it, you should test it. I also certainly understand the support ability of that.

I really do love veeam despite my frustrations here in the forums.

[kroll]

Did a test upgrade yesterday of 6.7 (ONLY VCSA not ESXi) and experienced the same errors as others with Veeam backups. Rolled the vCenter and PSC back to 6.5 and backup works normally.

[Gnaah]

Hi.

Actually I dont need the support part, just would like to see B&R working again

[DaStivi]

["brian.mathis"]I can confirm.....vCenter 6.7 (haven't upgraded my hosts yet) does NOT work with Veeam 9.5 Update 3. (Error: Object reference not set to an instance of an object)
Let's hope Update 3a becomes available soon."
Tested in our lab and same result here. Have not found any workaround yet other than adding 6.5 ESXi hosts directly.

i've updated an test vCenter and the hosts within… ah configured backupjob is still running without any issues… also Veeam B&R rescan of vCenter didn't quit with any error…

[DaStivi]

i did some research… i got one vm with the "object reference not set error"... this vm got back'd-up with NBD Mode... i need some time to realize that there was a snapshot open... looks like because of that the vm didn't got saved with my storage integration… but basically it looks like that with storage integration (backup from storage snapshot, or NFS Direct backup) the backup of vms on 6.7 does work, but not with NBD mode for example!

[Cragdoo]

direct quote from Gostev, in this weeks "The Word from Gostev" ...you do subscribe to this don't you??? if not ...shame on you

As far VMware vSphere 6.7 support in Veeam products, it is important to realize that as an ISV, we get the GA code at the same time with everyone else, so it will take us some time to test all of our products against it now that it's available. However, we already know that the most current Veeam Backup & Replication 9.5 Update 3 does NOT support vSphere 6.7 and as I mentioned here before, our release vehicle for adding this support will be Update 3a. Of course, we've already done a lot of work based on the pre-release code drops (for example, migrating our vSphere Web Client plug-in to HTML5 alone took us a few months). But it is only now that we can see the actual GA code (and notably, we're already finding some differences with the RC code) . Finally, since everyone will have the question about expected Update 3a timelines anyway: historically, we've been shipping new vSphere release support on average within 2 months from GA (with the official commitment for new platform support being 90 days from GA date). In this particular case however, there are additional unknowns in the equation: Windows Server 2016 RS4 and Windows 10 Spring Creators Update releases, support for which needs to catch the same release train. Although presumably, these should be shipped any day now – and testing the RC code has not revealed any major issues so far.

For those who did jump the gun and upgraded to vSphere 6.7 despite lack of official support, but unwilling to downgrade back to a supported vSphere version – remember that you can also use our agents to backup your VMs until Update 3a is shipped. We've actually just launched the 3 month production license program for Veeam Agents for Microsoft Windows and Linux, as well as Veeam Backup for Microst Office 365 - available to ALL customers and prospects without any restrictions. So if you have a need to protect your Windows and Linux workstation and servers whether they are physical or virtual, on-prem or in the cloud - or want a local backup of your corporate data hosted in Microsoft Office 365, you can now give our corresponding solutions a try. The license can be obtained directly from the download pages of Veeam Agents for Windows and Linux and Veeam Backup for Office 365.

[cgrossman]

i did some research… i got one vm with the "object reference not set error"... this vm got back'd-up with NBD Mode... i need some time to realize that there was a snapshot open... looks like because of that the vm didn't got saved with my storage integration… but basically it looks like that with storage integration (backup from storage snapshot, or NFS Direct backup) the backup of vms on 6.7 does work, but not with NBD mode for example!

DaStivi, you are awesome! Under Backup Infrastructure\Backup Proxies, I selected my proxy, went to Properties, and Transport mode was set to Automatic selection. I changed that to Virtual Appliance, and unchecked the box for Failover to network mode if primary mode fails. Now my backup WORKS! And, it's much faster. I was topping out around 120Mb throughput before, but now I'm seeing over 400Mb. Veeam Forums FTW!

[DaStivi]

DaStivi, you are awesome! Under Backup Infrastructure\Backup Proxies, I selected my proxy, went to Properties, and Transport mode was set to Automatic selection. I changed that to Virtual Appliance, and unchecked the box for Failover to network mode if primary mode fails. Now my backup WORKS! And, it's much faster. I was topping out around 120Mb throughput before, but now I'm seeing over 400Mb. Veeam Forums FTW!

nice workaround! btw. virtual appliance is always prefered to nbd… nbd uses the esx mgmt interface, and even there VMware trottle the traffic….

direct quote from Gostev, in this weeks "The Word from Gostev" ...you do subscribe to this don't you??? if not ...shame on you

thirst things first, vSphere was released even bevor Gostev sent his newsletter… and some users want's to update as soon as possible because knowing whats gooing is their daily business…

i didn't read someone to really flame about Veeam not support vSphere 6.7 from day-0, also everybody can see that when VMware changes things not vendor can instantly support such things, but this is exactly what these forums are for... to find people and share the knowledge and possibly, like we did, find workarounds!

i can guess that could even for Veeam be interessting to know that some combinations are working allready!
for me it looks like VMware changed a lot in the network transport mode, or mgmt interface of the esx/vCenter components….

[sideshowtob]

for those who have upgraded, are we talking your lab/test/dev environments? Surely no-one was stu... brave enough to upgrade to 6.7 on release day in production????

Lab, yes....of course that's what I meant...

Second post in 8 years!!

[Cragdoo]

thirst things first, vSphere was released even bevor Gostev sent his newsletter… and some users want's to update as soon as possible because knowing whats gooing is their daily business…

Given Gostev's newletter is a weekly one, and is sent out every Sunday Evening/Monday moring, this is the 1st newsletter since the 6.7 release, so I'd say that's a decent response from the Senior Vice President of Product Management. How many other vendors would do that?

i didn't read someone to really flame about Veeam not support vSphere 6.7 from day-0, also everybody can see that when VMware changes things not vendor can instantly support such things, but this is exactly what these forums are for... to find people and share the knowledge and possibly, like we did, find workarounds!

Not flame, pointing out facts. Also workarounds yes, supported no.

[leeg123]

i doubt this is useful info, but we upgraded our LAB to 6.7
We also can't backup, and this is totally fine. However as a test, we initiated an instant VM restore into the ESX 6.7 and it did successfully restore and boot
Again, not useful, but hopefully helpful to those out there who feel the need to test things that probably shouldn't even be bother with.
Eagerly awaiting update 3!!!

[nitramd]

So, do we always shoot first and ask questions later? I've done this a number of times in my career and have spent many a late night fixing the resulting problems.

If you've been in IT long enough and have had your arse chewed off often enough (I'm standing as I write this) then you learn to do research, to be skeptical, and to be patient.

As George Santayana said, "Those who do not learn history are doomed to repeat it."

[DaStivi]

i doubt this is useful info, but we upgraded our LAB to 6.7
We also can't backup, and this is totally fine. However as a test, we initiated an instant VM restore into the ESX 6.7 and it did successfully restore and boot
Again, not useful, but hopefully helpful to those out there who feel the need to test things that probably shouldn't even be bother with.
Eagerly awaiting update 3!!!

check a few posts above…
backup from storage snapshot seeams to work fine… also backup with appliance mode looks like to work fine…

[jozne]

Hi

So is there currently any way to get the backups working with vCenter 6.7? appliance mode does not work atleast for us, same errors.

2018-04-26 07:41:07 :: Preparing backup proxy xxxxxxx for disk Hard disk 1 [hotadd]
2018-04-26 07:41:27 :: Error: Object reference not set to an instance of an object.

It's great when your provider decides to go for 6.7 instead 6.5 in the last minute and your backups go to oblivion

[Gnaah]

Yes.
Is not Veeam that "Agile" anymore than what it was in previous years ?
Are you becoming too big perhaps, turning into another Elephant ?
I truly hope not.
A colleague of mine already said, why not bundle two things into one: "VeeaMWare"

[Stephenv]

I hope the update comes soon. as we are doing a Server refresh in a few weekes, and i'm building a new setup. going from ESX 5.01 to 6.7 the only thing come over from the old setup are the VM's no migration of vCenter.


Stephen

[Gostev]

Is not Veeam that "Agile" anymore than what it was in previous years ?
Are you becoming too big perhaps, turning into another Elephant ?

What makes you think Veeam is less agile than it was before? If you're talking about vSphere 6.7 support specifically - then as already mentioned earlier, it should be delivered within the exact same time frame as with all the previous vSphere releases (which was, on average, 2 months after vSphere GA).

[UAnton]

Veeam is slowpoke You can release update for support 6.7 before VMware release 6.7. You should not force client waiting.
P.S. I'm sure that you are cooperating with VMware and you knew about the changes for a long time!

[Gostev]

Actually, releasing support for 6.7 before VMware releases 6.7 is an extremely bad idea, because you can never guarantee that the code which worked with the RC build will continue to work reliably with the GA build. There're many examples of significant changes between RC and GA build with the previous vSphere releases. For example, in vSphere 6.5, those changes required us to pretty much scrap our vSphere 6.5 support code entirely, despite the fact it worked perfectly on the pre-release builds.

And in fact, just a few years ago one of the leading backup vendors (our competitor) did exactly what you suggested, announcing support for the new vSphere release on the day it went GA. Which made many of its customers upgrade and run into major backup and recovery reliability issues. It was a huge hit on their reputation - and in this light yes, we believe it will be better for us to take the "slowpoke" approach (as you put it), and test everything well on the final shipping code - even if this means we "force our clients waiting". We just value our customers' trust that we established in the past 10+ years too much to risk it.

Other than that, of course we were cooperating with VMware and have done everything that was possible using the pre-release code drops. For example, migrating Web Client plug-in to HTML5 was quite a major project that took us some months of playing with those pre-release bits earlier this year.

[dbhagen]

I came up in the ranks of system administration starting in backup admin. Look, I get it, wanting the features from the latest VMware updates. I keep checking this thread as well for when support is released.

But I can't fault VEEAM for taking this one at a "slowpoke" pace. If you're doing backups right, you're paranoid. Paranoid that the backups won't run, paranoid that the backups aren't valid, paranoid that the media might fail. You live and die by your backups, and as much as I would love the new UI fixes, I cannot risk my backups failing and my company being at risk. It's not fun to be pointed at as the reason we're not able to install the new software. It's even less fun to be pointed at when you can't backup or restore critical company data.

So just try to see it from both sides. They're doing their best, and it's up to us to do our best in waiting for the critical patches and vetting needed and communicating to our stakeholders why it's important to hold off.

[Regnor]

We all use Veeam because of its high stability so give them the necessary time for developing and testing new releases.

Like Anton, I also know a major backup vendor which supported 6.5 right on the release date, without even requiring an update for their software. Last time I've checked they were still using VDDK 6.0 and weren't able to backup VMs on VMFS6.

Also I wouldn't hurry to migrate to 6.7; you don't know if it's safe to use and stable enough at the moment.
If you need to upgrade then go to 6.5 and migrate to 6.7 later; upgrading vCenter and ESXi hosts doesn't take much time and isn't complicated at all.

[ArchanfelHUN]

Defending the Veeam team, vmware did a big changes in vsphere 6.7. They drop out the suse linux as base and using the vmware trademark mr. fancy PhotonOS. And do a LOT of changes during the services, using a completely different infrastructure in the whole vcenter server. Just see the changelog. This is not a bugfix update, its a huge update and the system improved a lot. Much faster, much more reliable. Please give the veeam team time to make the veeam backup compatible with the new system.

[phrynd]

I fully understand Veeam point of view, you cannot ensure a full support for a new vSphere version as soon as it becomes GA.
But, I'm quite sure, you were already working with VmWare at least for major changes and not with all details.
I think it could be a very good thing for you to deliver a beta version or a RC to the guys that want or need to upgrade their test environment.

Those guys could help you to detect some weird things or behavior.

Obviously, none of us will go to production without testing vSphere and Veeam together and with both on real versions no RC or beta.

Would be nice from veeam to post a link to the RC (or the mailbox to which ask for the RC) as soon as it will be available.

[ArchanfelHUN]

I try the workaround what you guys mentioned abobe. Set the proxy to virtual appliance and disable the fallback to network mode.
I have two VMware ESXi, 6.5.0, 7388607 ESXi server, both connected to the same veeam server. Both have vmfs6.
With one of them the backup working, the other however not.
I got this error:
Unable to allocate processing resources. Error: No backup proxy is able to process this VM due to proxy processing mode restrictions.

I dont get it, exactly the same, even the hw, two HP DL380 G9, same ESXi version, why not working both?

update: the veeam vm located on the esxi what is working. if i migrate to the other esxi that will work and the other not. so the veeam need to be the same esxi if the proxy set to the virtual appliance mode. this is normal behavior or i miss something?

[jasonph]

i did some research… i got one vm with the "object reference not set error"... this vm got back'd-up with NBD Mode... i need some time to realize that there was a snapshot open... looks like because of that the vm didn't got saved with my storage integration… but basically it looks like that with storage integration (backup from storage snapshot, or NFS Direct backup) the backup of vms on 6.7 does work, but not with NBD mode for example!

DaStivi, you are awesome! Under Backup Infrastructure\Backup Proxies, I selected my proxy, went to Properties, and Transport mode was set to Automatic selection. I changed that to Virtual Appliance, and unchecked the box for Failover to network mode if primary mode fails. Now my backup WORKS! And, it's much faster. I was topping out around 120Mb throughput before, but now I'm seeing over 400Mb. Veeam Forums FTW!

@DaStivi and @cgrossman - much respect. Veeam seems to work happily with VMware 6.7 in backup and replication using Virtual Applliance transport mode, and is faster than in Automatic mode.

[ArchanfelHUN]

Only solution what i found is to install a basic veeam backup proxy server into every esxi host and use as a remote proxy in the main backup server.
This is clearly not a good solution, since need to maintain unnecessary windows servers, need to licence that, and the veeam servers. So its no good.
The best way to not to update for 6.7 for now!!

I try the workaround what you guys mentioned abobe. Set the proxy to virtual appliance and disable the fallback to network mode.
I have two VMware ESXi, 6.5.0, 7388607 ESXi server, both connected to the same veeam server. Both have vmfs6.
With one of them the backup working, the other however not.
I got this error:
Unable to allocate processing resources. Error: No backup proxy is able to process this VM due to proxy processing mode restrictions.

I dont get it, exactly the same, even the hw, two HP DL380 G9, same ESXi version, why not working both?

update: the veeam vm located on the esxi what is working. if i migrate to the other esxi that will work and the other not. so the veeam need to be the same esxi if the proxy set to the virtual appliance mode. this is normal behavior or i miss something?

[justind]

Surely nobody would sign off of an upgrade to v6.7 in a production environment on release...

[rstoner]

Given that VMware also released 6.5 U2, I'm guessing we still have to wait for 9.5 U3a to be supported, correct?

[servicematica]

does anyone know how to tell when the veeam update will come out? or if you can download a beta?
thank you

[Gostev]

Given that VMware also released 6.5 U2, I'm guessing we still have to wait for 9.5 U3a to be supported, correct?

No, normally updates do not require any code changes from us, as they bring nothing but bug fixes.

does anyone know how to tell when the veeam update will come out? or if you can download a beta?

This will largely depend on the results of stress testing of the GA code, and the number of bugs we find. For example, we have just found one critical bug with hot add leading to data corruption, and now need to build a code to work around this one. However, this is nothing unusual - quite typical process of adding support for the new vSphere version - and as I noted before, normally it takes us 2 months on average, making late June a good estimate.