Dev & Prod backup jobs vm confusion - Case 04102743

[Unreliably-Recurring]

B&R 9.5u4 running on its own server with connection to mgmt network & LAN.

-- Problem 1: VSS snapshot confusion --
Timeline:
26 Mar - long history of repeated successes on job backing up development vm originally created from storage snapshot and job backing up production vm.
27 Mar - B&R restore job replaces dev vm with backup of prod vm.
31 Mar morning - Manual run of dev backup job on new dev vm succeeds.
31 Mar evening - Scheduled run of dev backup job fails - "[vss snapshot in use by prod backup job]"
31 Mar evening - Scheduled run of prod backup job fails - "[vss snapshot in use by dev backup job]".

This failure repeats every time the jobs' respective schedules create concurrent runs.
When the jobs succeed, they show using the correct paths to their respective vms' files on the datastores.
The VMs, obviously, are named differently.

Question:
Why are the jobs taking over each others' vm snapshots?

-- Problem #2 : AAP test --
The dev vms are in a virtual network with no NICs.
They have the same FQDNs and IPs as the prod vms, which are on the LAN.

When trying to diagnose an AAP problem (before discovering that it's caused by a VSS snapshot conflict), the support tech tried a credentials test on the dev job's vms.
The B&R console runs the test against the FQDN/IP, which route to the prod, so the test succeeds.

Question:
Does B&R not know about the VMs' network differences?


$10k Question:
What is the expected method for backing up sandboxed dev environment vms?

[HannesK]

Hello,

Problem 1: the case comments say that it is about SQL backups. Just keep in mind that the Veeam log shipping service is running inside the VM. I guess the answer to your question is your problem 2

Problem 2: Veeam gets the IP address from VMware / Hyper-V tools. And if you have duplicate IP addresses, then it takes the shortest route. As you don't have any route to your dev machine, it cannot do application aware backup.

$10k Question:
a) as your dev VM has no NIC: not using application aware backup.
b) configure networking without duplicate IP addresses

I sent you a PM with my bank account

Best regards,
Hannes

PS: I still ask myself how you can have an IP address without a NIC. I assume that you simply did not connect the network to the outside world

[Unreliably-Recurring]

Thanks for the reply.

re Problem 1: but if job successes show that they used correct storage file paths for the vms' disks, then why do that same job's failures show that B&R has done a snapshot of a different vm?
Shouldn't the snapshots that B&R make be made based on the VM, and not the IP the VM?

The hypervisor management tool shows the IP addresses of the vms' virtual NICs; they don't need a connection to any network outside the hypervisor to have an IP address.

[HannesK]

hmm, I still do not know for sure whether your VMs have a NIC or not, but anyway... duplicate IP addresses do not work with application aware image processing (AAIP). that's expected. you could also try VMware-tools quiecsing. That does not allow log-shipping for SQL, but might work.

shows the IP addresses of the vms' virtual NICs

The dev vms are in a virtual network with no NICs.

[Unreliably-Recurring]

Virtual machines need virtual NICs in order to talk to other VMs within the same "Port Group"/"Private switch".
To do that, the virtual NICs have IP addresses. (<- your first quote from me)
The management tool shows these IP addresses.

"Port Group"/"Private switch" is/are inside a "Virtual"/"Internal" Switch.

A Virtual Switch doesn't need to have a NIC in a hypervisor. (<- your second quote from me)

This is one of the ways that VMs can be in a network sandbox.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Regardless of that, why can one B&R backup job take VSS snapshots of vms that aren't included in its job?

[HannesK]

a VM snapshot and a VSS snapshot are independent from each other.

a VM snapshot is triggered via VCenter: here it is easy to snapshot the correct VM, because of a unique ID
a VSS snapshot is triggered via IP protocol: how should the process deal with duplicate IP addresses? There is not only the "network cable way", but there is also the VIX way.