Host-based backup of VMware vSphere VMs.
Post Reply
daniflexx
Influencer
Posts: 19
Liked: 1 time
Joined: Feb 19, 2011 3:19 pm
Full Name: Dani Mora
Contact:

Resize a disk deletes all your replica history?

Post by daniflexx » 1 person likes this post

Yesterday I resized a VM's disk. Pretty usual in my environment.

The replica job runned tonight and this mornig I found this 'warning' in my inbox (and logs):

- VM disk size was changed since last sync, deleting all restore points

And that's it. All my restorepoints are gone. I didn't ask to.

Really? Is that normal? Is there any way to disable this agressive behaviour?

I there is no way to disable it (I honestly don't know if it's possible) I have a feature request: instead of killing all the VM's history without asking, if you found a resized disk, please create anohter replica with another name. But please don't delete my backup history without asking or warning me it in uppercase.

Did I miss something in the user's manual?

thanks!

dani
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Resize a disk deletes all your replica history?

Post by Gostev »

The replication restore points are stored as snapshots. And snapshots become obsolete as soon as you change parent disk size. I will think about the better handling of this situation (creating new replica is usually not an option because of lack of free disk space in most DR environments). Thanks!
daniflexx
Influencer
Posts: 19
Liked: 1 time
Joined: Feb 19, 2011 3:19 pm
Full Name: Dani Mora
Contact:

Re: Resize a disk deletes all your replica history?

Post by daniflexx »

Adding the an option "choose your resized disk handling strategy" to the job's definition would be nice to me. Maybe with the following options:

- Do nothing. Raise an error.
- Delete restore points.
- Create another replica.

I guess that the best strategy depends on your particular scenario. For instance, If my VM is also covered by a backup job I'd choose the delete restore points. But if in a particular case the replica is my only backup strategy I'd choose the "do nothing and raise an error" option.

thanks for taking this into consideration,

dani
slidek9
Enthusiast
Posts: 44
Liked: never
Joined: Apr 20, 2010 5:00 pm
Full Name: Leon
Contact:

Re: Resize a disk deletes all your replica history?

Post by slidek9 »

This:
VM disk size was changed since last sync, deleting all restore points
is Crazy!
If we add another 20GB to a fileserver disk, it wants to repopulate 1.4TB over a 47m/bit link (yes we have large disks as we are a large corp).
I spend every other day now, shipping a NAS box back and forth across the country, deleting and recreating jobs so I can reseed them; every time someone extends a disk.

It takes 3 days+ to repopulate some of our 2TB+ servers; so:
we lose our restore points
The re-seeding process (backup a 2+TB server to NAS = 3 days, transport = 1 day, actual seeding and catchup 3 to 4 days) takes over a week for large servers.
we also have no native site backups when the VM is locked for replication
We have no DR solution any longer
So now instead of being a solution, if we hit DR during a reseed peroid we would not have a backup to roll back too as we miss backups while the re-seeded process catches up - using Veeam for DR is potentially WORSE then not having a DR solution at all, given the frequency of disk extends, this a not a good place to be concerning dataloss caused by a backup/DR product.

As a fee paying customer I want to know what you are doing about this; and why would you let such a obvious flaw into a production build?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Resize a disk deletes all your replica history?

Post by Gostev »

So, I am confused by your post.

This topic talks about consequences of resizing source VM disks. This is actually caused by the platform limitation that we can do nothing about. VMware snapshots do not currently support changing the base disk size. As soon as VMware supports such scenario, we will not have to delete all snapshots upon base disk resizing. If enough people submit enhancement request to VMware, may be they will do this.

Now, your scenario does seem very different. Instead of resizing existing source disk, you are simply adding new one. This is not the scenario being discussed in this thread, and it should not cause similar consequences. Your replication should continue normally (without re-seeding the whole VM), and all existing restore points should remain intact. If you are not seeing this, please open a support case for further investigation, as this is not a known issue.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Resize a disk deletes all your replica history?

Post by tsightler »

As Anton has explained, changing the size of an existing disk simply isn't possible on the target because VMware doesn't support resizing a disk that currently has a snapshot. However, I fail to understand why this suddenly means you need to reseed. Basically, Veeam simply deletes all but the MOST RECENT, restore point, resizes the disk, recalculates the hashes (same processes as performing a reseed without having to ship drives back and forth) and then continues on. It's not catastrophic to the replication process.

I agree that it would be ideal to not loose the restore points, but unfortunately that's simply a limitation of the technology being used (VMware snapshots), however, there's no reason that you should be forced to reseed by "shipping" drives.
slidek9
Enthusiast
Posts: 44
Liked: never
Joined: Apr 20, 2010 5:00 pm
Full Name: Leon
Contact:

Re: Resize a disk deletes all your replica history?

Post by slidek9 »

Gostev wrote:So, I am confused by your post.

Now, your scenario does seem very different. Instead of resizing existing source disk, you are simply adding new one.
Sorry, I was talking about resizing a disk, not adding a new one. I understand this is a VMware limitation; and they need to fix this for the poeple who replicate many large disks.
daniflexx
Influencer
Posts: 19
Liked: 1 time
Joined: Feb 19, 2011 3:19 pm
Full Name: Dani Mora
Contact:

Re: Resize a disk deletes all your replica history?

Post by daniflexx »

I understand that VMWare has to solve it but we need Veeam to add a security mecanism to prevent accidental deletion of VM snapshoot's history.

In a medium company the people in charge of managing VMs and the team that handles backups are not the same. When resizing or adding disks to a VM it's a routine operation for the first team it causes lots of throuble and risk to the backup people.

A fix (well, design change) on this matter in the upcoming service pack would be great.

thanks,

dani
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Resize a disk deletes all your replica history?

Post by Gostev »

Hi Dani - well, again - reading through my response above, adding disks to a VM will not cause any issues...

What are you proposing as the design change? Disabling the job completely in case source disk size change is detected? Not sure this is much better idea, because this means you become unprotected immediately, and your replication jobs will be requiring constant manual intervention to keep them going. And what would be your next steps anyway to make them work again? Controlled, manually triggered deletion of all restore points? I don't really follow how all of that is this better than how it works today.
daniflexx
Influencer
Posts: 19
Liked: 1 time
Joined: Feb 19, 2011 3:19 pm
Full Name: Dani Mora
Contact:

Re: Resize a disk deletes all your replica history?

Post by daniflexx »

I suggest that the job definition offers options on how to handle the situation of resized disks. Something like:

In the case of resized disks:

- Overwrite the existing replica (and delete all previous restore points). This is the current behavior.
- Fail the job leaving the replica intact. -> yes, this will leave the machine unprotected but in some scenarios is better than loosing 30 restore points.
- Create another replica with another name and raise a warning -> if the storage allows it, like in my case, this would be my default setting for all the jobs. Or maybe instead of creating a new replica it could rename the old replica to something like ServerABC_resized1_replica. The sysop will have to handle the situation later but is he knows what to do.

thanks!

dani
AndyR
Enthusiast
Posts: 36
Liked: 1 time
Joined: Sep 10, 2010 4:07 am
Full Name: Andy Roberts
Contact:

[MERGED] VM disk size was changed since last sync

Post by AndyR »

On one of my replication jobs I keep getting this error:-

Code: Select all

VM disk size was changed since last sync, deleting all restore points
The VM disk size hasn't changed though, never has. The original VM was setup with a 20GB, 500GB and 1TB and the replica is the same (1TB excluded) but on every job run I get this error.

Any reason why?

Thanks.
Andy
AndyR
Enthusiast
Posts: 36
Liked: 1 time
Joined: Sep 10, 2010 4:07 am
Full Name: Andy Roberts
Contact:

Re: Resize a disk deletes all your replica history?

Post by AndyR »

I just noticed on my replica VM, the 1TB disk that I have excluded from Source is show up as a 20Gb disk on the replica. I guess this is the problem, but why is it showing as a 20gb disk, I assumed it wouldn't show it all?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Resize a disk deletes all your replica history?

Post by Vitaliy S. »

Hi Andy,

Adding or removing VM disks from the replication job shouldn't cause this behavior. Please contact our support team to verify that source VM disks haven't been resized, this fact should be easy to verify via our replication job debug log files.

Thanks!
AndyR
Enthusiast
Posts: 36
Liked: 1 time
Joined: Sep 10, 2010 4:07 am
Full Name: Andy Roberts
Contact:

Re: Resize a disk deletes all your replica history?

Post by AndyR »

Will do. All i can think is that the original backup job I used to seed the VM had the 1TB disk excluded as well. Not sure why its populating it with a 20GB disk on the DR end though, but will log a call.

Thanks.
Andy
Gav@GH
Influencer
Posts: 21
Liked: 15 times
Joined: Jul 20, 2012 12:27 am
Contact:

Re: Resize a disk deletes all your replica history?

Post by Gav@GH »

I just got burned by this too - I can't believe that deleting ALL restore points is the default behaviour and there is not even a warning about it before hand.

We have a file server which had it's hard drive filling up, so using the magic of VMware, was able to increase disk capacity. Came in the next day to review my Veeam backup report only to find in the the log:
VM disk size was changed since last sync, deleting all restore points
Naturally as Murphy's Law would have it, 2 days later I need to recover a file from a week ago and have nothing to fall back on.

I second the options above; either fail the job (which gives you an oportunity to at least manually make a copy of replica), or ideally just automatically make another copy.
cag
Enthusiast
Posts: 74
Liked: never
Joined: Mar 26, 2011 4:02 am
Full Name: Conrad Gotzmann
Contact:

[MERGED] How to plan for VM Disk size change and Replication

Post by cag »

I have a VM where I would like to change (increase) a disk in size. The replication of the server is over a slow WAN connection and I would like to avoid re-seeding. What is the recommend procedure to prepare for the disk change. It would be nice if it just works but knowing that veeam depends on VMware snapshots and CBT I believe this will cause a error and cause a few problems.

Anyone with a successful painless procedure. ? Veeam version 7
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Resize a disk deletes all your replica history?

Post by foggy »

Conrad, disk size change should be handled by Veeam B&R automatically, I just recommend to wait for the R2 update as there were a couple of issues regarding that and those are addressed in this update.
Post Reply

Who is online

Users browsing this forum: No registered users and 82 guests