Resize a disk deletes all your replica history?

[daniflexx]

Yesterday I resized a VM's disk. Pretty usual in my environment.

The replica job runned tonight and this mornig I found this 'warning' in my inbox (and logs):

- VM disk size was changed since last sync, deleting all restore points

And that's it. All my restorepoints are gone. I didn't ask to.

Really? Is that normal? Is there any way to disable this agressive behaviour?

I there is no way to disable it (I honestly don't know if it's possible) I have a feature request: instead of killing all the VM's history without asking, if you found a resized disk, please create anohter replica with another name. But please don't delete my backup history without asking or warning me it in uppercase.

Did I miss something in the user's manual?

thanks!

dani

[Gostev]

The replication restore points are stored as snapshots. And snapshots become obsolete as soon as you change parent disk size. I will think about the better handling of this situation (creating new replica is usually not an option because of lack of free disk space in most DR environments). Thanks!

[daniflexx]

Adding the an option "choose your resized disk handling strategy" to the job's definition would be nice to me. Maybe with the following options:

- Do nothing. Raise an error.
- Delete restore points.
- Create another replica.

I guess that the best strategy depends on your particular scenario. For instance, If my VM is also covered by a backup job I'd choose the delete restore points. But if in a particular case the replica is my only backup strategy I'd choose the "do nothing and raise an error" option.

thanks for taking this into consideration,

dani

[slidek9]

This:
VM disk size was changed since last sync, deleting all restore points
is Crazy!
If we add another 20GB to a fileserver disk, it wants to repopulate 1.4TB over a 47m/bit link (yes we have large disks as we are a large corp).
I spend every other day now, shipping a NAS box back and forth across the country, deleting and recreating jobs so I can reseed them; every time someone extends a disk.

It takes 3 days+ to repopulate some of our 2TB+ servers; so:
we lose our restore points
The re-seeding process (backup a 2+TB server to NAS = 3 days, transport = 1 day, actual seeding and catchup 3 to 4 days) takes over a week for large servers.
we also have no native site backups when the VM is locked for replication
We have no DR solution any longer
So now instead of being a solution, if we hit DR during a reseed peroid we would not have a backup to roll back too as we miss backups while the re-seeded process catches up - using Veeam for DR is potentially WORSE then not having a DR solution at all, given the frequency of disk extends, this a not a good place to be concerning dataloss caused by a backup/DR product.

As a fee paying customer I want to know what you are doing about this; and why would you let such a obvious flaw into a production build?

[Gostev]

So, I am confused by your post.

This topic talks about consequences of resizing source VM disks. This is actually caused by the platform limitation that we can do nothing about. VMware snapshots do not currently support changing the base disk size. As soon as VMware supports such scenario, we will not have to delete all snapshots upon base disk resizing. If enough people submit enhancement request to VMware, may be they will do this.

Now, your scenario does seem very different. Instead of resizing existing source disk, you are simply adding new one. This is not the scenario being discussed in this thread, and it should not cause similar consequences. Your replication should continue normally (without re-seeding the whole VM), and all existing restore points should remain intact. If you are not seeing this, please open a support case for further investigation, as this is not a known issue.

[tsightler]

As Anton has explained, changing the size of an existing disk simply isn't possible on the target because VMware doesn't support resizing a disk that currently has a snapshot. However, I fail to understand why this suddenly means you need to reseed. Basically, Veeam simply deletes all but the MOST RECENT, restore point, resizes the disk, recalculates the hashes (same processes as performing a reseed without having to ship drives back and forth) and then continues on. It's not catastrophic to the replication process.

I agree that it would be ideal to not loose the restore points, but unfortunately that's simply a limitation of the technology being used (VMware snapshots), however, there's no reason that you should be forced to reseed by "shipping" drives.

[slidek9]

So, I am confused by your post.

Now, your scenario does seem very different. Instead of resizing existing source disk, you are simply adding new one.

Sorry, I was talking about resizing a disk, not adding a new one. I understand this is a VMware limitation; and they need to fix this for the poeple who replicate many large disks.

[daniflexx]

I understand that VMWare has to solve it but we need Veeam to add a security mecanism to prevent accidental deletion of VM snapshoot's history.

In a medium company the people in charge of managing VMs and the team that handles backups are not the same. When resizing or adding disks to a VM it's a routine operation for the first team it causes lots of throuble and risk to the backup people.

A fix (well, design change) on this matter in the upcoming service pack would be great.

thanks,

dani

[Gostev]

Hi Dani - well, again - reading through my response above, adding disks to a VM will not cause any issues...

What are you proposing as the design change? Disabling the job completely in case source disk size change is detected? Not sure this is much better idea, because this means you become unprotected immediately, and your replication jobs will be requiring constant manual intervention to keep them going. And what would be your next steps anyway to make them work again? Controlled, manually triggered deletion of all restore points? I don't really follow how all of that is this better than how it works today.

[daniflexx]

I suggest that the job definition offers options on how to handle the situation of resized disks. Something like:

In the case of resized disks:

- Overwrite the existing replica (and delete all previous restore points). This is the current behavior.
- Fail the job leaving the replica intact. -> yes, this will leave the machine unprotected but in some scenarios is better than loosing 30 restore points.
- Create another replica with another name and raise a warning -> if the storage allows it, like in my case, this would be my default setting for all the jobs. Or maybe instead of creating a new replica it could rename the old replica to something like ServerABC_resized1_replica. The sysop will have to handle the situation later but is he knows what to do.

thanks!

dani

[AndyR]

On one of my replication jobs I keep getting this error:-

Code: Select all

VM disk size was changed since last sync, deleting all restore points

The VM disk size hasn't changed though, never has. The original VM was setup with a 20GB, 500GB and 1TB and the replica is the same (1TB excluded) but on every job run I get this error.

Any reason why?

Thanks.
Andy

[AndyR]

I just noticed on my replica VM, the 1TB disk that I have excluded from Source is show up as a 20Gb disk on the replica. I guess this is the problem, but why is it showing as a 20gb disk, I assumed it wouldn't show it all?

[Vitaliy S.]

Hi Andy,

Adding or removing VM disks from the replication job shouldn't cause this behavior. Please contact our support team to verify that source VM disks haven't been resized, this fact should be easy to verify via our replication job debug log files.

Thanks!

[AndyR]

Will do. All i can think is that the original backup job I used to seed the VM had the 1TB disk excluded as well. Not sure why its populating it with a 20GB disk on the DR end though, but will log a call.

Thanks.
Andy

[Gav@GH]

I just got burned by this too - I can't believe that deleting ALL restore points is the default behaviour and there is not even a warning about it before hand.

We have a file server which had it's hard drive filling up, so using the magic of VMware, was able to increase disk capacity. Came in the next day to review my Veeam backup report only to find in the the log:

VM disk size was changed since last sync, deleting all restore points

Naturally as Murphy's Law would have it, 2 days later I need to recover a file from a week ago and have nothing to fall back on.

I second the options above; either fail the job (which gives you an oportunity to at least manually make a copy of replica), or ideally just automatically make another copy.

[cag]

I have a VM where I would like to change (increase) a disk in size. The replication of the server is over a slow WAN connection and I would like to avoid re-seeding. What is the recommend procedure to prepare for the disk change. It would be nice if it just works but knowing that veeam depends on VMware snapshots and CBT I believe this will cause a error and cause a few problems.

Anyone with a successful painless procedure. ? Veeam version 7

[foggy]

Conrad, disk size change should be handled by Veeam B&R automatically, I just recommend to wait for the R2 update as there were a couple of issues regarding that and those are addressed in this update.