Discussions specific to the VMware vSphere hypervisor
daniflexx
Influencer
Posts: 19
Liked: 1 time
Joined: Feb 19, 2011 3:19 pm
Full Name: Dani Mora
Contact:

Resize a disk deletes all your replica history?

Post by daniflexx » Feb 20, 2012 10:40 pm 1 person likes this post

Yesterday I resized a VM's disk. Pretty usual in my environment.

The replica job runned tonight and this mornig I found this 'warning' in my inbox (and logs):

- VM disk size was changed since last sync, deleting all restore points

And that's it. All my restorepoints are gone. I didn't ask to.

Really? Is that normal? Is there any way to disable this agressive behaviour?

I there is no way to disable it (I honestly don't know if it's possible) I have a feature request: instead of killing all the VM's history without asking, if you found a resized disk, please create anohter replica with another name. But please don't delete my backup history without asking or warning me it in uppercase.

Did I miss something in the user's manual?

thanks!

dani

Gostev
SVP, Product Management
Posts: 24022
Liked: 3257 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Resize a disk deletes all your replica history?

Post by Gostev » Feb 20, 2012 11:09 pm

The replication restore points are stored as snapshots. And snapshots become obsolete as soon as you change parent disk size. I will think about the better handling of this situation (creating new replica is usually not an option because of lack of free disk space in most DR environments). Thanks!

daniflexx
Influencer
Posts: 19
Liked: 1 time
Joined: Feb 19, 2011 3:19 pm
Full Name: Dani Mora
Contact:

Re: Resize a disk deletes all your replica history?

Post by daniflexx » Feb 20, 2012 11:20 pm

Adding the an option "choose your resized disk handling strategy" to the job's definition would be nice to me. Maybe with the following options:

- Do nothing. Raise an error.
- Delete restore points.
- Create another replica.

I guess that the best strategy depends on your particular scenario. For instance, If my VM is also covered by a backup job I'd choose the delete restore points. But if in a particular case the replica is my only backup strategy I'd choose the "do nothing and raise an error" option.

thanks for taking this into consideration,

dani

slidek9
Enthusiast
Posts: 44
Liked: never
Joined: Apr 20, 2010 5:00 pm
Full Name: Leon
Contact:

Re: Resize a disk deletes all your replica history?

Post by slidek9 » Jul 31, 2012 10:42 am

This:
VM disk size was changed since last sync, deleting all restore points
is Crazy!
If we add another 20GB to a fileserver disk, it wants to repopulate 1.4TB over a 47m/bit link (yes we have large disks as we are a large corp).
I spend every other day now, shipping a NAS box back and forth across the country, deleting and recreating jobs so I can reseed them; every time someone extends a disk.

It takes 3 days+ to repopulate some of our 2TB+ servers; so:
we lose our restore points
The re-seeding process (backup a 2+TB server to NAS = 3 days, transport = 1 day, actual seeding and catchup 3 to 4 days) takes over a week for large servers.
we also have no native site backups when the VM is locked for replication
We have no DR solution any longer
So now instead of being a solution, if we hit DR during a reseed peroid we would not have a backup to roll back too as we miss backups while the re-seeded process catches up - using Veeam for DR is potentially WORSE then not having a DR solution at all, given the frequency of disk extends, this a not a good place to be concerning dataloss caused by a backup/DR product.

As a fee paying customer I want to know what you are doing about this; and why would you let such a obvious flaw into a production build?

Gostev
SVP, Product Management
Posts: 24022
Liked: 3257 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Resize a disk deletes all your replica history?

Post by Gostev » Jul 31, 2012 12:23 pm

So, I am confused by your post.

This topic talks about consequences of resizing source VM disks. This is actually caused by the platform limitation that we can do nothing about. VMware snapshots do not currently support changing the base disk size. As soon as VMware supports such scenario, we will not have to delete all snapshots upon base disk resizing. If enough people submit enhancement request to VMware, may be they will do this.

Now, your scenario does seem very different. Instead of resizing existing source disk, you are simply adding new one. This is not the scenario being discussed in this thread, and it should not cause similar consequences. Your replication should continue normally (without re-seeding the whole VM), and all existing restore points should remain intact. If you are not seeing this, please open a support case for further investigation, as this is not a known issue.

tsightler
VP, Product Management
Posts: 5298
Liked: 2153 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Resize a disk deletes all your replica history?

Post by tsightler » Jul 31, 2012 1:13 pm

As Anton has explained, changing the size of an existing disk simply isn't possible on the target because VMware doesn't support resizing a disk that currently has a snapshot. However, I fail to understand why this suddenly means you need to reseed. Basically, Veeam simply deletes all but the MOST RECENT, restore point, resizes the disk, recalculates the hashes (same processes as performing a reseed without having to ship drives back and forth) and then continues on. It's not catastrophic to the replication process.

I agree that it would be ideal to not loose the restore points, but unfortunately that's simply a limitation of the technology being used (VMware snapshots), however, there's no reason that you should be forced to reseed by "shipping" drives.

slidek9
Enthusiast
Posts: 44
Liked: never
Joined: Apr 20, 2010 5:00 pm
Full Name: Leon
Contact:

Re: Resize a disk deletes all your replica history?

Post by slidek9 » Aug 03, 2012 9:33 am

Gostev wrote:So, I am confused by your post.

Now, your scenario does seem very different. Instead of resizing existing source disk, you are simply adding new one.
Sorry, I was talking about resizing a disk, not adding a new one. I understand this is a VMware limitation; and they need to fix this for the poeple who replicate many large disks.

daniflexx
Influencer
Posts: 19
Liked: 1 time
Joined: Feb 19, 2011 3:19 pm
Full Name: Dani Mora
Contact:

Re: Resize a disk deletes all your replica history?

Post by daniflexx » Aug 06, 2012 8:21 am

I understand that VMWare has to solve it but we need Veeam to add a security mecanism to prevent accidental deletion of VM snapshoot's history.

In a medium company the people in charge of managing VMs and the team that handles backups are not the same. When resizing or adding disks to a VM it's a routine operation for the first team it causes lots of throuble and risk to the backup people.

A fix (well, design change) on this matter in the upcoming service pack would be great.

thanks,

dani

Gostev
SVP, Product Management
Posts: 24022
Liked: 3257 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Resize a disk deletes all your replica history?

Post by Gostev » Aug 06, 2012 10:04 am

Hi Dani - well, again - reading through my response above, adding disks to a VM will not cause any issues...

What are you proposing as the design change? Disabling the job completely in case source disk size change is detected? Not sure this is much better idea, because this means you become unprotected immediately, and your replication jobs will be requiring constant manual intervention to keep them going. And what would be your next steps anyway to make them work again? Controlled, manually triggered deletion of all restore points? I don't really follow how all of that is this better than how it works today.

daniflexx
Influencer
Posts: 19
Liked: 1 time
Joined: Feb 19, 2011 3:19 pm
Full Name: Dani Mora
Contact:

Re: Resize a disk deletes all your replica history?

Post by daniflexx » Aug 06, 2012 10:46 am

I suggest that the job definition offers options on how to handle the situation of resized disks. Something like:

In the case of resized disks:

- Overwrite the existing replica (and delete all previous restore points). This is the current behavior.
- Fail the job leaving the replica intact. -> yes, this will leave the machine unprotected but in some scenarios is better than loosing 30 restore points.
- Create another replica with another name and raise a warning -> if the storage allows it, like in my case, this would be my default setting for all the jobs. Or maybe instead of creating a new replica it could rename the old replica to something like ServerABC_resized1_replica. The sysop will have to handle the situation later but is he knows what to do.

thanks!

dani

AndyR
Enthusiast
Posts: 36
Liked: 1 time
Joined: Sep 10, 2010 4:07 am
Full Name: Andy Roberts
Contact:

[MERGED] VM disk size was changed since last sync

Post by AndyR » Oct 02, 2012 6:58 pm

On one of my replication jobs I keep getting this error:-

Code: Select all

VM disk size was changed since last sync, deleting all restore points
The VM disk size hasn't changed though, never has. The original VM was setup with a 20GB, 500GB and 1TB and the replica is the same (1TB excluded) but on every job run I get this error.

Any reason why?

Thanks.
Andy

AndyR
Enthusiast
Posts: 36
Liked: 1 time
Joined: Sep 10, 2010 4:07 am
Full Name: Andy Roberts
Contact:

Re: Resize a disk deletes all your replica history?

Post by AndyR » Oct 02, 2012 9:11 pm

I just noticed on my replica VM, the 1TB disk that I have excluded from Source is show up as a 20Gb disk on the replica. I guess this is the problem, but why is it showing as a 20gb disk, I assumed it wouldn't show it all?

Vitaliy S.
Product Manager
Posts: 22311
Liked: 1415 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Resize a disk deletes all your replica history?

Post by Vitaliy S. » Oct 02, 2012 9:40 pm

Hi Andy,

Adding or removing VM disks from the replication job shouldn't cause this behavior. Please contact our support team to verify that source VM disks haven't been resized, this fact should be easy to verify via our replication job debug log files.

Thanks!

AndyR
Enthusiast
Posts: 36
Liked: 1 time
Joined: Sep 10, 2010 4:07 am
Full Name: Andy Roberts
Contact:

Re: Resize a disk deletes all your replica history?

Post by AndyR » Oct 02, 2012 9:45 pm

Will do. All i can think is that the original backup job I used to seed the VM had the 1TB disk excluded as well. Not sure why its populating it with a 20GB disk on the DR end though, but will log a call.

Thanks.
Andy

Gav@GH
Influencer
Posts: 21
Liked: 15 times
Joined: Jul 20, 2012 12:27 am
Contact:

Re: Resize a disk deletes all your replica history?

Post by Gav@GH » Oct 12, 2012 7:04 am

I just got burned by this too - I can't believe that deleting ALL restore points is the default behaviour and there is not even a warning about it before hand.

We have a file server which had it's hard drive filling up, so using the magic of VMware, was able to increase disk capacity. Came in the next day to review my Veeam backup report only to find in the the log:
VM disk size was changed since last sync, deleting all restore points
Naturally as Murphy's Law would have it, 2 days later I need to recover a file from a week ago and have nothing to fall back on.

I second the options above; either fail the job (which gives you an oportunity to at least manually make a copy of replica), or ideally just automatically make another copy.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 17 guests