Hot add not supported - and no warning about it

[bbricker]

This is a two part question- sorry if this has already been answered, I did some searching (but maybe not very well!)

I have a VM that has been saying that hot add is not supported and Veeam will failover to network mode. I actually think it's been doing this since I installed the vm (it was an OVF deployed VPN appliance). The vm has no snapshots, and is running on the same host and vmfs volume as many other vm's that backup just fine each night. Where would I start troubleshooting this?

Second part: Up until last week, I was on VB&R 6.0 so I hoping that upgrading to 6.5 might fix this issue. It did not. But what it did do is stop issuing a warning that the vm was failing back to network mode. I can't find a setting to change this, so I'm not sure if it's a new "feature" or a bug. Either way it's not very cool because I need to know when vm's stop working with hot add and fail to network mode. The nightly email report says success, and there is not even a note next to the vm in the details column. But if you go open the job details on the VB&R server and read through each message for that vm backup, you can see the message. There is a green check mark next to it, and no mention of a warning or exception. Not cool. I don't want to go through every job each day and read every detail to try and find if any of the vm's are failing to network mode, I need to know it is happening in the log results so it can be looked into. Can we please either go back to giving warnings on this behavior, or at least give the administrator the option? Maybe some people don't care about it and don't want to see a warning, but I certainly do

thanks!

[tsightler]

Does the OVF in question happen to deploy using IDE disk instead of SCSI? Hotadd is supported only for VMs that use SCSI virtual disks (it's SCSI Hotadd afterall). This is the most common case I've seen in the field for OVF appliances.

Regarding the warning, this was a change due to an overwhelming majority of request. There are a multitude of technical issues that can keep hotadd from working, and failing over to network mode is not really a problem in the vast majority of cases. In many environments there are some VMs that simply cannot be backed up with hotadd mode (similar to the issue you're seeing, but other setups also exist) and customers did not want to see the errors for these all the time. I'm sure you can find a long thread on the forum where you will see people making this request. You may very well be the first request I've seen that would like a return to the old behavior.

I know of many clients that ONLY use network mode (10Gb networks), because it requires less overhead for job startup/shutdown, and is less prone to issues with leaving phantom snapshots. Because of that, I am curious why you consider these warnings so important. I'm not aware of a way to get back the old "warning" behavior, but it's possible there is a way, I'd suggest opening a support case about that.

Barring a registry key or some other method to revert to the old behavior, it should be possible to write a Powershell script that could report on such behavior. Perhaps that would be an option for an interim solution.

[tsightler]

I know this doesn't address your request to have this behavior reverted to previous versions, but I thought I would offer this up as an interim option that might help. It's a simple powershell script that finds all backup sessions in the last 24 hours (easily changed), then loops through the logs of each task session looking for the "Failing over the network" and, if found, write the job name, VM name, and failover log message to the screen. It is lightly tested, but seems to work and I thought you might find it useful.

Code: Select all

$tasksessions = (Get-VBRBackupSession | Where-Object {$_.EndTime -ge (Get-Date).addhours(-24)}).GetTaskSessions() | Sort JobName
foreach ($tasksession in $tasksessions) {
    $failovermsg = $tasksession.Logger.GetLog().UpdatedRecords.Title | Select-String "failing over to network mode"
	if ($failovermsg) {
		write-host "Job Name:" $tasksession.JobName "  VM Name:" $tasksession.Name "  Message:" $failovermsg
	}
}

Output right now looks like the following but could easily be enhanced:

Code: Select all

Job Name: Lab Backup   VM Name: test_ide_vm   Message: Hot add is not supported for this disk, failing over to network mode...
Job Name: Test Backup 1   VM Name: pfsense_ovf   Message: Hot add is not supported for this disk, failing over to network mode...

[bbricker]

Does the OVF in question happen to deploy using IDE disk instead of SCSI? Hotadd is supported only for VMs that use SCSI virtual disks (it's SCSI Hotadd afterall). This is the most common case I've seen in the field for OVF appliances.

Good call, it is setup as IDE. And that sucks because I probably can't muck with that VM as it is a vendor supplied appliance.

Because of that, I am curious why you consider these warnings so important. I'm not aware of a way to get back the old "warning" behavior, but it's possible there is a way, I'd suggest opening a support case about that.

Because I have had instances, who knows why, where every single VM in a job spontaneously decides to throw a message about hotadd not being supported, and then every job after that did the same thing until I rebooted the VB&R server. Also a single VM that suddenly can't be hot-added might be indicative of another issue that you want to look into. I for one just like to know if something has changed so I can investigate.

I guess my thought is, even if several requests came in on a user forum to change the reporting status functionality, why would you implement that as a default without giving the entire user-base an option, when that expected result was always there? Now there's NOTHING to indicate something has happened with network failback unless you scroll through every single job result detail. Even then it's hard to catch because it still has a pretty little green check icon next to it.

Wouldn't it have made more sense for the devs to have just made a tiny little tick box in the options that says "do not generate warnings on network failback mode"? They could have even made it default to being a checked option. Then, anyone who DOES want to see those warnings just has to uncheck that box in their environment. Or EVEN BETTER yet- the backup email reports have a nice little "Details" column next to each VM, how about just put the warning message there? At least then it would catch my attention since it's in my inbox each day, and I wouldn't have to jump on the VB&R server or run a powershell script (which thanks, I will play with that).

PS: since I've ranted about a change I *don't* like in the new report status, I need to give kudos for one I do like. THANK YOU to the devs for making the list of VM's in the email report be in chronological order now. I never understood the seemingly random order that VM's were listed in the reports in 6.0. There was a time or two that I actually had to copy and paste the report into Excel and move things around so I could see start/stop times while trying to troubleshoot something

[tsightler]

Thank you very much to explain you reasoning, it's good feedback. I'm sure it wouldn't have been difficult to provide an option, but it's always a challenge to balance between adding more options and usability. Hopefully you will agree that part of Veeam's strength is that it's tools are generally simple and not loaded down with ton of options that are common in some other products. Keeping applications from turning into this requires careful evaluation of each possible option and their use and in this case I think it was determined that the vast majority of users simply didn't feel this warning was required. Of course, we're not always right, which is why we take feedback. It doesn't guarantee that this change will be made, but we do listen (it was listening to user feedback that caused us to remove the warning). Putting the warning in the details section on the email does seem like a good idea to me.

I feel like I should note that this change was actually made in 6.1 (released in June 2012), so it's actually been longer with the new behavior (no warning) than with the old 6.0 behavior. I honestly can't remember if 5.0 actually ended the job with warning on hotadd failover or not, but I don't think it did, so I believe the warning status was unique to 6.0. I believe that 6.1 was simply reverting to old, pre-6.0 behavior.

[bbricker]

Interesting about the versions, I have only ever been on 6.0 and 6.5.

[darryl.crossley]

Any ideas on how I get this to e-mail, seeing errors on the ''$msg.body' line.

$msg.body = "Job Name:" $tasksession.JobName " VM Name:" $tasksession.Name " Message:" $failovermsg ""

Code: Select all

if ( (Get-PSSnapin -Name "VeeamPSSnapIn" -ErrorAction SilentlyContinue) -eq $null )
{
    Add-PsSnapin "VeeamPSSnapIn"
}
$tasksessions = (Get-VBRBackupSession | Where-Object {$_.EndTime -ge (Get-Date).addhours(-24)}).GetTaskSessions() | Sort JobName
foreach ($tasksession in $tasksessions) {
    $failovermsg = $tasksession.Logger.GetLog().UpdatedRecords.Title | Select-String "failing over to network mode"
   if ($failovermsg) {
      write-host "Job Name:" $tasksession.JobName "  VM Name:" $tasksession.Name "  Message:" $failovermsg ""
   }
}
#email results
$smtpserver = ""
$msg = New-Object Net.Mail.MailMessage
$smtp = New-Object Net.Mail.SmtpClient($smtpServer)
$msg.From = ""
$msg.To.Add("d")
$msg.subject = "Failover Notification Report"
$msg.body = "Job Name:" $tasksession.JobName "  VM Name:" $tasksession.Name "  Message:" $failovermsg ""
$smtp.Send($msg)