VM moved to new vCenter

[foggy]

As mentioned above, the script will not be published but rather used by support engineers (currently, at the higher tiers only). Please ask your case to be escalated.

[Shestakov]

The tool (script) was tested by our QA and now available for Veeam users through support team. However, keep in mind that the Support started to help customers with the tool just yesterday. Please treat possible roughness with understanding.

Christopher,
I`ve contacted our support team, your case will be escalated to the engineers helping with moref-ids.
Thanks!

[cjack03]

@Shestakov.

Many many thanks! I completely understand the stance of the support team, and I agree you can use something that is not completely validated, I was just hoping to make my life a little easier

I look forward to hearing from the engineers working on this in the future.

[Massimo Vitali]

We are migrating our environment from vSphere 5.1 to vSphere 6.0.
The first step was to upgrade Veeam Backup and Replication to the latest version and patch level, in order to have full support for vSphere 6.0. Our Veeam server is now at version 8.0.0.2030 and is ready to fully support the new vSphere environment.
Next step was to upgrade Virtual Center from 5.1 to 6.0. We decided to do a fresh installation of Virtual Center 6.0 on a new server instead of doing in-place migration, starting in a new, clean environment. We set up a new server running Windows 2012 R2 and vCenter Server 6.0.0 and replicated the same environment of the old server (clusters, folders, alarms, permissions). The new server obviously has a new IP and DNS name, and we plan to keep both servers active until the whole migration process is completed. We are now ready to move hosts and VMs from the old to the new vCenter server.
Which is the correct way to do this operation at Veeam Server side? We obviously want to keep existing jobs, repositories, restore points, CBT data, etc.
We have 3 clusters in our vSphere environment and, at Veeam side, we have a separate backup job and repository for each cluster. Each backup job includes all the VMs running on the corresponding cluster.
My idea is to move each cluster from the old to the new vCenter server and then modify the corresponding backup job to reflect this change. I plan to move the clusters in separate days, so during the migration period Veaam server must manage some jobs through the old vCenter Server and other jobs through the new vCenter server.

I plan to follow these steps:

• 1. On Veeam server, add the new vCenter server to Veeam inventory (Backup Infrastructure - Managed Servers)
  2. Unregister all hosts/VMs in Cluster 1 from the old vCenter server and register them on the new vCenter server (obviously when the corresponding Veeam backup job is not running)
  3. On Veeam server, update the registration of hosts/VMs in the backup job to point to the new vCenter server
  4. Wait for the next scheduled job to complete to ensure the backup works correctly
  5. Repeat steps 2-4 for Cluster 2 and Cluster 3
  6. Remove the old vCenter server from Veeam inventory
  7. Turn off old vCenter Server

My questions are:

• A. Is this the correct procedure to do the operation?
  B. Which are the detailed operations to execute step 3? (update the Veeam job to make it point to the new vCenter Server)

Regarding question B, there are some instructions in KB1299, but my VMs are not directly registered in the job (the whole cluster container is included in the job), so I think I cannot remove and add again each VM separately. I should remove and add again the whole cluster, but I think is not possible to have an empty job for some time. This step is quite obscure for me.
Can you help me giving the correct detailed sequence to succesfully execute step 3?

Thank you very much for your help and best regards

Massimo Vitali

[foggy]

Massimo, in general the procedure looks good, however it omits the important fact that after being registered in the new vCenter all VMs will get new MorefIDs, which will result in Veeam B&R treating them as completely new ones (and backing them up in a full mode). To avoid that, you can contact support for assistance, I believe they will be able to help, if your scenario meets the corresponding requirements.

[scott.edwards]

Foggy is on target here as I just did the same migration except to vCenter 5.5. To avoid a full backup you should be able to edit a job and map it to the existing chain on your repository.

[Massimo Vitali]

Foggy is on target here as I just did the same migration except to vCenter 5.5. To avoid a full backup you should be able to edit a job and map it to the existing chain on your repository.

Thank you to Foggy and Scott for the clarifications. I carefully read all posts in this thread and I understood that I can map the same jobs/repositories to the VMs through the new vCenter, but in any way the first job execution will produce a full backup of each VM because the ref-id of the VMs will change while mapping them through the new vCenter server, so they are seen in Veeam as completely new machines. The only way to avoid this is to instruct Veeam to map the new ref-id's to the old ones using a script under the supervision of Veeam support. Is this correct?

If so, I will open a support request as soon as I will be ready to start the migration, before doing any VM registration on the new server.

Thank you

[foggy]

Massimo, you've got everything correctly.

[jbsengineer]

We are looking at a similar upgrade path as Massimo. Does Veeam recommend we reset the backup job (i.e. Active Full, etc) at anypoint during a 6.0 migration? For example it sounds like we can remap ref-id's and morefs after the vCenter upgrade. But does the changes in CBT on ESXi 6.0 warrant a reset? Or can we seamless upgrade the hosts from 5.1->6.0 and Veeam wouldn't notice a difference. We have around 1 petabyte of Veeam data so resetting it, as you can imagine is difficult.

Thanks!

[veremin]

It will be a completely new vCenter introduced, right? Not just a in-place upgrade. In this case, once VMs are moved to a new vCenter, stop all existing VB&R jobs and reach our support team; they will apply the migration utility allowing you to bypass full backup cycle after migration. Thanks.

[foggy]

There's no need to trigger active full, however, since new vCenter has no CBT data for the migrated VMs, full VM data read will be required during the next job run after migration (only changes will be actually transferred, though).

[BoricuaTec]

Is there a migration tool in 8.0? I just called into support and they told me since I moved to 6 and did not reuse the DB that I had to manually remove and add all the VMs in their jobs.... Is this correct? There has to be a tool for this.....?

[Shestakov]

Hello Luis,
You are correct, there is an automation tool, but I`m not sure if it was tested for vSphere 6 as well. I`ll check it with our QA team and let you know shortly.
Thanks.

[BoricuaTec]

Is Veeam looking into making this easy in the future, this can turn out to be a lot of work.

[foggy]

The first step to make such migration scenarios easier was to create the support tool mentioned in this thread.

[veremin]

By the way, vSphere 6.0 has this issue addressed, as it supports cross vCenter vMotion. So, if you're on VB&R v8 U2 and using vSphere 6.0, you should not face the said problem. Thanks.

[Shestakov]

Got an answer from QA team, the tool works for vCenter v6 as well.
Thanks!

[rboynton]

Case #01118438:

Using Veeam 8. We have been Veeam users for quite a long time. When the VMWare environment is stable (no changes), then Veeam works very well. My issue comes into play when changes to the VMWare environment occur (using VMWare 5.5). Latest example: We had a scheduled power outage, so we shut down each VM, then all the physical hardware. After the power outage, we started everything back up. One host came back up, but VCenter showed no statistics under the Hosts tab. Also, we were unable to power on any VM's running under that host. Had to put that host in maintenance mode, then remove it from the cluster and add it back in. Seems easy enough, right? Everything under VMWare then worked as expected and we could VMotion to that host and power VM's on. Whew! It seems that all the backup jobs for VM's running on that host failed that night. We had to remove each VM from the job, and add it back in. Of course, the first backup was a full. If that was not bad enough, we run backup copy jobs to a DR site. Guess what? New job, new backup copy. Some of those VM's are huge (2+ TB), which causes issues over a very slow WAN so we must copy to an external drive and move to DR. Very frustrating how Veeam seems to be so sensitive! Instead of looking at how a VM is registered in VCenter, why can't Veeam simply match the .vmx file to a job? If this were the case, what happened to us would be a non-issue.

[foggy]

Rick, the behavior is expected in case you re-register host in virtual infrastructure. This resulted in VM unique identifiers (which are assigned by vCenter and used in Veeam B&R to track VMs) being changed and caused Veeam B&R to treat them as new VMs.

[rboynton]

Yes, that is what happened...

The issue for me is the recognition system is far too complex. There are too many ways for things to break. Simple is better in almost all cases. Why not have B&R track VM's by VMX name, then once found, find out what host they are running on, then process? If a host is pulled and put back into VCenter, nothing would break. I know this is a radical change from the way it is done now, but the man-hours to rebuild broken backup jobs AND the backup copy jobs is massive. Please have the development team revisit this. Going forward, this only makes sense.

[foggy]

Why not have B&R track VM's by VMX name, then once found, find out what host they are running on, then process?

Because VMX name is not unique.

[rboynton]

I am not trying to be difficult here. I have never seen a duplicate VMX file. If Veeam is backing up multiple clusters, then perhaps. The solution to that is to simply tie a backup job to a cluster. To create a job, Veeam would then search the datastores attached to that cluster for the desired VMX file. The current solution is prone to breaking, as I have found out more than once. I love Veeam, and have for many years now. I want the product to continue to be successful, but the current way of doing this is very fragile. In the real world, this is problematic to say the least. The development team is smart, so let them find a way to make this work. Please?

[Vitaliy S.]

To create a job, Veeam would then search the datastores attached to that cluster for the desired VMX file.

Based on my experience when working with large infrastructures (300+ hosts), datastore search tasks can take up to 24 hours. We are using these tasks in Veeam ONE to search through datastores for orphaned files and orphaned VMs, and I wouldn't recommend relying on this process here.

Querying moref ID is a quick API call and is more useful for tracking objects in vSphere infrastructures. Moreover every object in VMware has a moref ID, so using this property for object matching seems more reasonable.

While my explanation doesn't help you at all, and I understand your concern, I would probably recommend to contact our technical team next time you have the same issue and ask for assistance with moref ID changes in our database, so that you could continue running incremental job passes.

[dgomes]

Is there any way for Veeam partners to be able to do this themselves? We provide the service/support for the customers we sell Veeam to. We know what causes this problem and we know this script can fix it, but having to open a case on behalf of the customers every time this happens is getting pretty old.

[Vitaliy S.]

This script is not trivial and can easily mess everything up, that's why we strongly recommend to do that via our support team, but I see your point as well. How often does this situation happen for your clients?

[dgomes]

For SMB clients not often (2 - 4 hosts), but larger customers who often play around with their hosts/vcenter servers mess things up despite our warnings. Most of the time we tell them just restart their backup chain for affected jobs but it would be nice to have some easier method to fix this. Maybe a separate tool that asks for user input to gather the required variables to run the script correctly? On some partner-only download section, maybe even require the completion of a training video and a quiz to access it if you really need to cover all your bases.

Although I think the ultimate solution to this problem would be the ability to map VMs in a backup. The backup lists the VMs it contains and you match them to the VMs that are live in your environment. When the backup runs the first time after doing this it then checks if what you mapped makes sense.

[foggy]

I doubt they want to mess things up further by directly interacting with Veeam B&R database.

[veremin]

If they face this issue frequently, may be it's time to update vSphere 6.0 (once it stabilizes a bit) and put into use cross-vCenter migration functionality it has, forgetting about that problem once a for all. Thanks.

[JonSD]

This script is not trivial and can easily mess everything up, that's why we strongly recommend to do that via our support team, but I see your point as well. How often does this situation happen for your clients?

We migrated from the VCSA to vCenter Server this week and ran into the mismatched moref ID issue.

I called into support and opened a case, 01244927, and the support engineer I spoke to was not familiar with the script or any procedures related to migrating to a new vCenter.

[Vitaliy S.]

You can ask to escalate your ticket to a higher tier engineer and reference this thread for more details. Thanks!