vSphere Power Management and Veeam problem?

[rhnb]

Hi
We run a 5 host VMware cluster and have power management enabled on 2 named hosts.
So hosts, 1,2 & 3 are ALWAYS up. Hosts 4 & 5 may be put in Standby mode if load is not high.

I'm seeing more and more of the following errors in Veeam...

Checking the license for the source host 'vmhost4.mycompany.uk'
Failed to validate license for the VM "MyVM".
Host with uuid "xxxxxx-xxxxxx-xxxxxx-xxxxxx" was not found

So, I'm guessing that when a backup job starts, Veeam collects the list of VM's to backup and also stores the host they're running on?
As the job progresses, if for example vmhost4 goes in to standby mode then when a VM which was on that host but has now been migrated to another host (lets say vmhost3) it seems it fails as it tries to validate the license for vmhost4 and can't contact it?
Funnily enough I never noticed any issue with this using v4.1.2 but maybe that was just luck?

I hope I'm wrong, as this means we won't be able to use Power Management and as we're an environmental organisation - that doesn't look good )

Cheeers... allan.

[joergr]

I would make sure, NO unplanned vmotion operations of any kind should occur during your backup window. Emergency/HA is something else, that would be necessary. But please always know this: The Backup Window is NOT a energy safe window. The opposite is true: The backup window is demanding most of power of the whole day in some cases, and it´s perfectly normal that way Sp please DO NOT allow any energy saving policies during your backup window.

This is also a good advice when looking at the special logic of how vmware snapshots are taken, handled and consolidated - especially when vmotion is in play - be careful with that.

And generally speaking, but this is my very personal opinion: I like the idea of energy saving. What i do NOT like is many many unplanned vmotion operations every day and every day. Vmotion is an extremely complex process and i personally don´t like the idea that huge tons of vm´s are vmotioned every day, i would not do that to my datacenter

best regards,
Joerg

[rhnb]

Thanks for your take on this. I take your points on board (but don't necessarily agree with them).

You sort of make it sound as if through the night hosts are stopping and starting willy nilly at an alarming rate. That just isn't the case - though I guess it could be if you have the power management set very aggressively.
A host will start usually not long after our backups start (we have 2 backup servers) and usually remain active until the backups have finished. Often when one of the backup servers has finished one host will go back into standby, and thats the issue here I think.

The problem we have (and I have a ticket open on this) is understanding when a server gets added into the list of licensed servers (Help | Licenses).
Our problem is that there are only 4 of our 5 servers listed in there and I don't know how to get the other one to appear. Could this be that when I installed the license (at upgrade time) that that particular server was in standby? What dictates which servers are listed in Help | Licenses?

[Bunce]

I disagree as well.

Some organisation's backup windows are pretty much continuous and overlap with 'production' windows.

Vmotion is a well accepted, safe method of managing resources - its just not for 'energy saving' but also managed via DRS for efficient and effective resource utilisation.

The whole point of Veeam allowing VM selection via VCenter / folders and not hosts is for this very reason - so VM's aren't tied to hosts during a backup window, so it simply doesn't make sense to say 'backup windows are not energy safe windows'.

I'm sure at worst, it's just a minor tweak that Veeam needs to make on the logic / stage that it performs its license check to support VM's changing hosts after a backup job has started (if this is in fact even the issue)

[rhnb]

Well at least I'm not in a minority of 1

Can anyone tell me what gets a host added to your list of licensed hosts?
I have a 10 cpu licence and 5 dual processor hosts. 4 of them appear in the list but the 5th (which as I say probably wasn't powered up when I upgraded and added the new license) doesn't appear.
I thought creating a job with that host as a target might do it, but it didn't. Has a job to actually run and use that target before it allocates a license to it?

[Gostev]

Host gets added to this list when you attempt to backup a VM located on it. This is dynamic process (does not happen just once when you add new license), as obviously any environment can grow with time and our licensing must handle the growth correctly, to report exceeded license usage as soon as it happens.

[rhnb]

So I guess the problem is as Bunce says when it does this license check.

I assumed that it wouldn't actually bother noting which host a VM was on until it came to do the actual backup, but it looks to me as though it stores this info when it builds its list of VMs, and then, when it finally comes around to backing up the VM checks theirs a licence for that host. If the VM has moved in between these stages from a host that wasn;t included in the licence list then it tries to conact the now closed down host for some reason, not the host the VM is currently running on. So, the logic does seem a little strange there.

I guess I can cobble a fix for this by putting a VM on that host and backing it up. I assume once a host has been 'used', it'll remain there in the licence list for good?

[Gostev]

Yes.

[rhnb]

I'm still having issues with this. Am I the only one on the planet using Power Management in VMware AND Veeam?

I did as suggested by tech support and made sure I'd done at least one backup of a VM on each host so that my 5 hosts were shown as licensed hosts. That worked fine. However, it is dynamic and (for example) when I looked at the logs for last night, a VM failed with this checking license error. The host in question is now no longer listed in the licensed hosts.

Gostev says "Host gets added to this list when you attempt to backup a VM located on it."

This is the problem, as it's just not using correct information is it.
When my 20 VM job starts, lets say VM20 is on host4. What is happening is that when VM20's turn comes to be backed up, if it's moved to host1, then for some obscure reason Veeam decides it still needs to check that host4 has a valid license. Why is that? The VM's not running on host4. As host4 is now in standby, it fails the job. On the retry, it works and backs up off host1. So, Veeam is cacheing which host each VM in that job is running on at the START of a job which seems odd.
The worst thing is - Veeam has now removed host4 from the list of licensed hosts.
The logic seems very poor here and needs changing.

I'll let you know what tech support say.

Cheeers... allan.

[Alexey D.]

Hello Allan,

Thanks for unveiling this case, we understand your concerns - R&D team will discuss what we can do with this.
But what is bothering you the most? Retry works and finally backs up that VM. And host4 will be licensed back when that VM returns to it.

[rhnb]

Hi Alexey
Thanks for the swift response.
Yes, retry works. Just not good for my blood pressure and I get additional logs mailed of course.
It's nice when you can just see 'Success' rather than 'Failed' followed by 'Success' so yes, I guess not a showstopper.
I also think that it would make your product look more professional in that it would be handling features of vSphere correctly rather than cludging it's way around them.

Personally I'd be happy with just adding my hosts manually to the licence manager and letting them stick there. If I replace a host I can remove the old/re-add the new or if I add a new extra host I need a new licence anyway. I'm not sure the 'dynamic' approach has any merit really.

Cheeers... allan.

[rhnb]

An update on this.
This got escalated to Tier3 support (but I have to say I got a much better response initially - I kept having to ask for updates). Here's the reply I got this morning...

"Sorry for the delayed response. I was completely sure that have sent you an update.
I've consulted with Development team. Guys told me that the problem appears due to we apply the license at beginning of the job.
Unfortunately, I cannot provide you with any information if this behavior should be changed in future. At the moment you may feel free to “retry” your jobs. With retry we will process only VMs which have not been processed during last run of the job. We will not try to backup or replicate your Vms once again. Just unprocessed VMs."


So, in a nutshell, Veeam doesn't support VMware Power Management. Doesn't look like they'll be busting a gut to fix it in the near future either (my interpretation of the above).
Ah well, the planet's doomed anyway eh? Why should we try saving power - I'm off to turn the air conditioning up.

[Gostev]

I do not know if this is the same support case (one of the reason I beg to always include support case ID when reporting any issues on forums), but we discussed something similar to this earlier today with devs, and decided this needs to be fixed (based on behavior description provided by support engineer), and asked our QC folks to file a bug. However, QC could not reproduce the issue internally. So, we requested that support engineer gets us the logs files so that devs could investigate this further.

[rhnb]

Sounds like it is the same support case (550892). I've been asked for the log files and my license file tonight, so will do that asap.
I purposely didn't include the support case number as the reply I got didn't indicate anything else was going to happen with this case and I thought we weren't supposed to discuss these sort of techy issues with the product on here, we had to raise a case with tech support which I did. I only posted the previous post to (as I thought) round off the thread as it was this morning. I had an earlier post removed as it was a tech support issue so I guess once bitten....
Anyhow, I'm glad it's going to be followed up. Seems to me it would be a very simple thing to fix if you simply let the customer license the hosts they wanted (bought licenses for) and left it at that.

[Gostev]

I had an earlier post removed as it was a tech support issue so I guess once bitten....

Funny... actually, the post was likely removed by moderator because it did NOT have support case number stated. There is sticky topic about this > If your post disappears.

Yes, I thought this should be simple fix as well based on the issue description support engineer provided. We just need to be able to reproduce it...

[Gostev]

Just to update, researching full logs showed there are no problems or bugs with Veeam licensing around vSphere Power Management. According to the log files, what happened is that after the backup job was created, one of the ESX hosts had its UUID (unique ID) changed for whatever reason (which is very strange by itself, I am not sure how this can happen). Since UUID is how hosts are tracked by jobs, so this caused lack of licenses.

If this happens to someone else, the workaround is either to re-create affected jobs, or clean up one table in configuration database with the help of our support.

Thanks.

[joergr]

Vmotion is a well accepted, safe method of managing resources - its just not for 'energy saving' but also managed via DRS for efficient and effective resource utilisation.

Just have to add something here, Bunce, of course you are right, vmotion IS a well accepted method. But it is a very complex method, which in my personal opinion require you to fully understand what is just happening under the hood. And at some rare problematic points, vmotion simply has to make a decision. Thus, i´d say i love vmotion, but i don´t use it when it is not absolutely necessary. And speaking of that, my personal opinion is: It is not necessary for energy saving purposes.

Please read this article by Kyle Gleed here, it is very new and already includes the 4.1 improvements of vmotion:
http://blogs.vmware.com/uptime/2011/02/ ... overs.html
And focus especially on the passage that starts with "When the preCopy cannot converge....".

Please understand: I don´t want you to fear vmotion, i just want you to fully understand what a complex operation a vmotion is and that complex operations should occur when they absolutely have to. But not a hundred times a day for some energy savers. But, then again, please: This is only my personal opinion.

Best regards,
Joerg

[rhnb]

Just to update, researching full logs showed there are no problems or bugs with Veeam licensing around vSphere Power Management. According to the log files, what happened is that after the backup job was created, one of the ESX hosts had its UUID (unique ID) changed for whatever reason (which is very strange by itself, I am not sure how this can happen). Since UUID is how hosts are tracked by jobs, so this caused lack of licenses.

If this happens to someone else, the workaround is either to re-create affected jobs, or clean up one table in configuration database with the help of our support.

Thanks.

Hmmm. Not so fast Gostev.
I got the email to test this supposed fix yesterday and applied it. I was given 2 options...
1) Recreate the jobs (no thanks)
2) issue a 'delete from [dbo].[HostsByJobs]'

I chose option 2 and then did a select on the table HostsByJobs - it was empty as expected - so far so good.
I fired up the Veeam console on one of the backup servers and 'strangely' when I looked at licences, it still showed one of my hosts in there. I expected it to be empty - no backups were running.
Did the same on my other backup server and that also showed the one host listed in the licences.

Backups ran last night and yes, you guessed it - 2 jobs failed with the 'Checking licence...' error.

So, the problem isn't fixed, and perhaps its not a good idea posting that it is BEFORE the customer has had a chance to test it and report back. After all, I've been waiting for a LONG time now, surely you can wait 24 hours to see if your fix works?

I've reported back to Tech support and am awaiting their next suggestion.

Cheeers... Allan.

[rhnb]

Joergr - I think we all hear what you're saying.
I've been using various versions of VMware for a long time now.
The truth is (and maybe we've been lucky), I've had no issues with vMotion at all - other than this Veeam licensing issue, which I'm not sure you can lay at vMotions door can you? When load increases and a host is brought out of hibernation, VMware has no problems whatsoever in migrating some VM's to the new host, they work flawlessly. The ONLY issue is that when we come to back them up, Veeam can't handle the fact the host has reappeared and doesn't seem to be able to allocate it a licence even though it has them available. So please, don't diss vMotion. It's working as designed, with no issues whatsoever.
If, at the end of the day, someone says 'Veeam doesn't handle Power Management correctly', I'd accept that, but don't start trying to make out we really shouldn't be doing this fancy Power Management stuff.
You say you don't like Power Management - each to their own, I'm not going to try and convince you otherwise. It's working great for us in our 'real world' environment with just 1 issue - Veeam. As an Ecological organisation we are committed to saving energy where possible, being as green as we can etc. As Tesco would say - every little helps I realise that may be a bizarre concept to some.
I suppose I could say that at the moment, I don't much like Veeam, but that may change. Maybe sometime down the line your views on PM will change too?

[Gostev]

perhaps its not a good idea posting that it is BEFORE the customer has had a chance to test it and report back. After all, I've been waiting for a LONG time now, surely you can wait 24 hours to see if your fix works?

If I don't post the update right away, the topic will drown and I will forget all about it. And since I already learnt that customers only post resolutions in 1 out of 100 technical issue topics that they create, I would rather post the update right away, than try to keep this in my head among 1000 things of other things to do and 100 of other topics to deal with, and eventually forget to update with high probability.

In the end, I did not lock this thread or anything, so we can continue this discussion for as long as required, and adjust workarounds accordingly. This won't really affect what I said about the root cause of the issue in your case, because this was clear from the log files. The issue is NOT connected with Power Management, as the name of this topic implies. We have also confirmed no DPM issues with our own lab testing. And I want the customers researching this forum to know about this, which is why I posted immediately instead of waiting.

I am sorry if this made you upset.

[rhnb]

If I don't post the update right away, the topic will drown and I will forget all about it. And since I already learnt that customers only post resolutions in 1 out of 100 technical issue topics that they create, I would rather post the update right away, than try to keep this in my head among 1000 things of other things to do and 100 of other topics to deal with, and eventually forget to update with high probability.

In the end, I did not lock this thread or anything, so we can continue this discussion for as long as required, and adjust workarounds accordingly. This won't really affect what I said about the root cause of the issue in your case, because this was clear from the log files. The issue is NOT connected with Power Management, as the name of this topic implies. We have also confirmed no DPM issues with our own lab testing. And I want the customers researching this forum to know about this, which is why I posted immediately instead of waiting.

I am sorry if this made you upset.

Upset? Moi?

I see what you're getting at re posting the 'update' and the 'root cause' but if I hadn't come back to this forum (as you say 99% don't update with the resolution) then your update is giving out wrong information isn't it? That hasn't fixed the problem I'm seeing and could give a false impression?
No one will be happier than me if it's not a PM issue. I look forward to a resolution to my 'licensing' problem.

[Gostev]

My update does not have "wrong information" on the root cause. The update is based on your logs research and lab testing. Thanks.

[rhnb]

OK - if you want to be pedantic, the root cause may be as you said (I wouldn't know - I wasn't party to the analysis of the logs etc).

I realise you seem to like to always have the last word, but you know fine well, I'm sure, that what I am getting at is that you said (and I quote)...

"If this happens to someone else, the workaround is either to re-create affected jobs, or clean up one table in configuration database with the help of our support."

It isn't a 'workaround'. It didn't work. In my book that's errr - wrong information?

This problem was reported in January (11th 'ish' I think). It's now the 18th March. I've reported back that the workaround didn't work, but still not heard anything from support (not even an acknowledgement). I think you should cut me a little slack for being a little miffed don't you?

[Gostev]

OK yes, you have all rights to be miffed

[Gostev]

Just to update, the work is still being done on researching your issue. I just got copied on one of the emails, this is how I know. Thank you for your patience.

[rhnb]

Just to close off this thread in case someone else hits it - it is a bug in 5.0.1 (see below) - and after not being able to replicate it in lab tests earlier - they can now.
I'm told 5.0.2 (with this bug fixed) should be available in 2,3, maybe 4 weeks.

"Hello Allan,

Thanks for the information provided.
Me and my colleagues spent extra time to confirm the issue in our Lab environment. And we've succeeded. Now we can confirm that this is a software bug. This should be fixed in Veeam Backup and Replication v.5.0.2. This release should be generally available soon.

Great thanks for all your cooperation and involvement on this matter Allan."

[Gostev]

Small correction, 5.0.2 will be available next week and will not include this fix (it is ready to go gold already). The hotfix will be available in 2-3 weeks, that will be installable on top of 5.0.2. Thanks.

[rhnb]

OK thanks for that.
Where's the best place to look to see when this hotfix comes out?
Will I get an email?
Cheeers... Allan.

[Gostev]

Yes, you will get an email from our support as soon as the hotfix is available. The fix should be fairly simple, but I would rather not post details at this time until you confirm that it fixes the issue - just do not want another epic fail in this thread from my side

Seriously though, in the last 3 years this is the first time when R&D came to wrong conclusion based on log/code review. And due to unlucky coincidence, QC managed to "prove" their findings because of what later appeared to be an issue with the test lab. We all down here are terribly sorry for this to have happened. Believe or not, we do not fail like this very often, this is definitely an exceptional case. We sincerely apologize and are humbly hoping this did not completely ruin your trust in us as a vendor.

[rhnb]

Thanks for the info - I'll look forward to the email re the hot-fix, and thanks for the apology. Much appreciated, and my trust is restored