Comprehensive data protection for all workloads
ortoscale
Service Provider
Posts: 224
Liked: 19 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

old backups are gone

Post by ortoscale »

Hi there. I've issues with Veeam and Vsphere migration to 5.1. We did downgrade to 5.0 because of that. One of backup jobs was doing weekly backup for 52 weeks. All jobs in Veeam were recreated. When I was checking (last week) Backup disks, i could see under job name new backup and old ones as separated. Today I've noticed that old ones are gone. I've also check physical location and it looks there are no backup for the past. Is it possible that job recreation with same name, overwrite physical backup on disk?

case#: 00175520

foggy
Veeam Software
Posts: 19523
Liked: 1780 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: old backups are gone

Post by foggy »

Most likely old backups were removed according to your retention policy - what are the retention settings for this job currently and how many restore points do you have on disk for it?

ortoscale
Service Provider
Posts: 224
Liked: 19 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: old backups are gone

Post by ortoscale »

60 now and before. So retention policy shouldn't do the job.

foggy
Veeam Software
Posts: 19523
Liked: 1780 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: old backups are gone

Post by foggy »

How about Deleted VMs retention period? My guess was that on re-registering VMs in vSphere they got new VM IDs, and after re-creating jobs and mapping them to the existing backups, the old backups were considered as containing deleted VMs and subjected to retention. Similar to this topic.

ortoscale
Service Provider
Posts: 224
Liked: 19 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: old backups are gone

Post by ortoscale »

foggy wrote:How about Deleted VMs retention period? My guess was that on re-registering VMs in vSphere they got new VM IDs, and after re-creating jobs and mapping them to the existing backups, the old backups were considered as containing deleted VMs and subjected to retention. Similar to this topic.
i didn't change that setting. It's 14 days by default. I do hope that's not the case. You should remove that default setting to do not delete by default.

ortoscale
Service Provider
Posts: 224
Liked: 19 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: old backups are gone

Post by ortoscale »

foggy wrote:How about Deleted VMs retention period? My guess was that on re-registering VMs in vSphere they got new VM IDs, and after re-creating jobs and mapping them to the existing backups, the old backups were considered as containing deleted VMs and subjected to retention. Similar to this topic.
one more thing. I've saw those backups in Veeam Console - Backups/Disk, are you sure that deleted backups are also shown there?

Vitaliy S.
Product Manager
Posts: 24330
Liked: 1880 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: old backups are gone

Post by Vitaliy S. »

bc2011 wrote:didn't change that setting. It's 14 days by default. I do hope that's not the case. You should remove that default setting to do not delete by default.
I don't believe that deleted VMs retention period is the main suspect, as you would probably had noticed the warning message discussed in this thread:
Job Report configuration
bc2011 wrote:one more thing. I've saw those backups in Veeam Console - Backups/Disk, are you sure that deleted backups are also shown there?
No, deleted VM backups should not be displayed in the backup console.

Let's see what our support team will say about this after reviewing the debug files.

foggy
Veeam Software
Posts: 19523
Liked: 1780 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: old backups are gone

Post by foggy »

Vitaliy S. wrote: No, deleted VM backups should not be displayed in the backup console.
They shouldn't be displayed there after being deleted from disk - and they are not, as you have initially stated:
bc2011 wrote:When I was checking (last week) Backup disks, i could see under job name new backup and old ones as separated. Today I've noticed that old ones are gone.
Anyway, if you do not see the message mentioned by Vitaliy in the job session log, this is not actually the case.

ortoscale
Service Provider
Posts: 224
Liked: 19 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: old backups are gone

Post by ortoscale »

We cleared things out, and I must admit that I'm not really happy with findings.

Fact1: I didn't delete any VMs, I've just delete and create particular job

Fact2: if you delete a job, Veeam will mark VMs that were part of that job as deleted VMs and VM deleted retention policy will start for them. That is not explained in your documentation at all. Also Veeam doesn't give away any warning at job deletion about that. That's insane. We are talking about customer data here, and it doesn't give any warning that data will be deleted ?!

Fact3: you cannot restore delete VMs back

http://veeampdf.s3.amazonaws.com/guide/ ... je%2FHQ%3D , page125

this should be backup & restore solution not backup & delete.

for more details you can check support case [ ref:_00D30RWR._50060OLjsz:ref ]

foggy
Veeam Software
Posts: 19523
Liked: 1780 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: old backups are gone

Post by foggy »

That is exactly what I was talking about.
bc2011 wrote:Fact1: I didn't delete any VMs, I've just delete and create particular job
Most likely the reason for that was re-installation of vCenter. Veeam B&R tracks VMs by their unique moref ID (assigned by vCenter). Every VM re-registeration in the virtual infrastructure changes this identifier causing the VM to be treated as a new VM.
http://www.virtuallyghetto.com/2011/11/ ... hange.html
bc2011 wrote:Fact2: if you delete a job, Veeam will mark VMs that were part of that job as deleted VMs and VM deleted retention policy will start for them.
Not if you delete a job but if you map the new job to the existing chain, then it will apply deleted VMs retention policy to the VMs with old VM IDs (which are not backed up by this new job). If you decide not to map the job, old restore points should be preserved (however, the first job run would trigger full backup).

ortoscale
Service Provider
Posts: 224
Liked: 19 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: old backups are gone

Post by ortoscale »

1)You should remove automatic deletion as default setting, customer should take care of disk usage not you
2) you should integrate warning message that will inform users that job deletion may also DELETE backed up data
3) you should explain this scenario in user manual, there is no docs about job deletion at all

This is feature request.

foggy
Veeam Software
Posts: 19523
Liked: 1780 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: old backups are gone

Post by foggy »

bc2011 wrote:2) you should integrate warning message that will inform users that job deletion may also DELETE backed up data
Job deletion does not delete any backed up data, please see my answer above.

Plus, there is a "<vm> is no longer processed by this job. Make sure this change is intentional." warning message logged in the job session on every job run during the whole deleted VMs retention period. Probably, the wording is not fully descriptive and does not explain the consequences, i agree.
bc2011 wrote:3) you should explain this scenario in user manual, there is no docs about job deletion at all
The user guide states that "When this retention period is over, data of the deleted VM will be removed from backup files."

rbrambley
Veeam Software
Posts: 470
Liked: 56 times
Joined: Jun 16, 2009 1:23 pm
Full Name: Rich Brambley
Contact:

Re: old backups are gone,

Post by rbrambley »

BC,

I think there is a major misunderstanding here. I know you provided a reference number earlier, but we can only find your case by ticket number. Can you provide that for us?

Some general points:

When you delete a job you are prompted to either delete the data on disk or keep it. If you keep it then that set of files will stay there forever unless you move it or delete it manually. Restore points and VM deletion settings have nothing to do with the files once the job is removed from Veeam. They are settings of an active Veeam job only.

If you import a stand alone set of Veeam backup files they will show up in the console in the Disk section. You can do all of your restores from these files regardless if a VM still exists in your environment or not.

If you manually remove the files of an active job from the backup storage first, and don't use the Veeam console, your database will be out of sync with the state of the backup files. This means the Veeam console will not be accurate. Veeam files should never be manually removed unless they are no longer tied to an active job.

Maybe we'll have a better understanding of what happened to you after looking at the support ticket.

ortoscale
Service Provider
Posts: 224
Liked: 19 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: old backups are gone,

Post by ortoscale »

00172140 - Veeam cannot connect to Vcenter after 5.1 upgrade
00175520 - old backups are gone

I really don't remember all things that I've tried during those two cases, I guess you can see all actions in support log file, that I've provided. I'm not sure whether I've reinstalled Veeam also (I've done that with Vcenter - downgrade to 5.0) but I'm 100% sure that i didn't receive any warning about VM deletion and I didn't remove anything manual from disk. So there is a chance that I didn't receive any warnings because jobs were recreated from scratch. Fact is that deleted retention policy erase those restore points.

best regards, Matjaz

rbrambley
Veeam Software
Posts: 470
Liked: 56 times
Joined: Jun 16, 2009 1:23 pm
Full Name: Rich Brambley
Contact:

Re: old backups are gone

Post by rbrambley »

BC,

I found your case 00175520 and read the notes. It seems the Support tech has had the same misunderstanding about this problem. The reference to page 125 of the User Guide is an explanation about VM retention in active jobs. In fact it states "when the job starts" in the first few sentences of that section. It does not apply to remaining Veeam backup files after you have deleted the job.

To test how Veeam actually does operate for yourself do the following.
- create a dummy VM (128GB ram, 1 vcpu, 1 gb hd, any os you want)
- don't power it on or install the os.
- create a Veeam backup job and set the VM retention period to 1
- back up the VM several times
- delete the job and choose to keep the files on disk
- delete the dummy VM from your environment
- watch the backup files for a few days or as long as you want

You will see that Veeam does not delete these files and all restore points remain.

If your Veeam repository settings auto import backup files you will see the backups in the disk section. Otherwise click the Import button and browse to these files and then they will be available for restores.

On the other hand, if after you delete the VM you keep the job the files will be deleted per the deleted VM retention period.

Is it possible that you kept your jobs long enough during your switch back and forth between vCenter versions that retention kicked in?

DennisRoos
Novice
Posts: 3
Liked: never
Joined: Apr 21, 2010 1:19 pm
Full Name: Dennis Roos
Contact:

Re: old backups are gone

Post by DennisRoos »

Would be wonderful if Veeam implemented some kind of re-mapping utility/possibility so when we move to a new vCenter and get new id's for our vm's, whe could re-match them by name again and be able to keep the backup history. Could maybe helped in this case to. I for sure would be better of if I had that possibility last time I switched out our vCenter environment. / Dennis.

Vitaliy S.
Product Manager
Posts: 24330
Liked: 1880 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: old backups are gone

Post by Vitaliy S. »

Thanks for the feedback.

ortoscale
Service Provider
Posts: 224
Liked: 19 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: old backups are gone

Post by ortoscale »

please check support case - you got reply to that. thanks. Matjaz
rbrambley wrote:BC,

I found your case 00175520 and read the notes. It seems the Support tech has had the same misunderstanding about this problem. The reference to page 125 of the User Guide is an explanation about VM retention in active jobs. In fact it states "when the job starts" in the first few sentences of that section. It does not apply to remaining Veeam backup files after you have deleted the job.

To test how Veeam actually does operate for yourself do the following.
- create a dummy VM (128GB ram, 1 vcpu, 1 gb hd, any os you want)
- don't power it on or install the os.
- create a Veeam backup job and set the VM retention period to 1
- back up the VM several times
- delete the job and choose to keep the files on disk
- delete the dummy VM from your environment
- watch the backup files for a few days or as long as you want

You will see that Veeam does not delete these files and all restore points remain.

If your Veeam repository settings auto import backup files you will see the backups in the disk section. Otherwise click the Import button and browse to these files and then they will be available for restores.

On the other hand, if after you delete the VM you keep the job the files will be deleted per the deleted VM retention period.

Is it possible that you kept your jobs long enough during your switch back and forth between vCenter versions that retention kicked in?

ortoscale
Service Provider
Posts: 224
Liked: 19 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: old backups are gone

Post by ortoscale »

case is closed now. this is from support.

--------------------------------------------

You have added the VMs into the job. Those VMs obtained an ID for each of them.
Job was recreated, VMs were Re-added into the job (which means they have been deleted and added back). VMs have obtained new ID.

After all in regards to VMs with "old" IDs, "Deleted VMs data retention period" parameter gets applied.
The default "Deleted VMs data retention period" amount defined as 14, no reconfiguration took place.
After that "old" backups with "old" IDs were considered to be deleted in regards to "Deleted VMs data retention period"


Here is output from the log file reflecting what happened. .

Line 24747: [06.01.2013 22:36:12] <01> Info [RetentionAlgorithm] Vm 'SRVSUR-SQL' contained 11 points and 0 from them have been removed.
Line 35049: [20.01.2013 21:26:59] <01> Info [RetentionAlgorithm] Vm 'SRVSUR-SQL' contained 11 points and 11 from them have been removed.

Line 24756: [06.01.2013 22:36:13] <01> Info [RetentionAlgorithm] Vm 'SRVSUR-APP' contained 13 points and 0 from them have been removed.
Line 35210: [20.01.2013 21:27:33] <01> Info [RetentionAlgorithm] Vm 'SRVSUR-APP' contained 13 points and 13 from them have been removed.

As we can see, right after 2 weeks have past, contained restore points have been deleted.

Regards,
Val Semakin
Veeam Support

------------------------------------------------------------
feature requests, user manual changes. Val has promised me to pass on those requests.

1)You should remove automatic deletion as default setting (from 5.x), disk usage is customer responsibility
2) you should integrate warning message that will inform users that job deletion may also DELETE backed up data, when deleting JOB
3) you should explain this scenario in user manual, there is no docs about job deletion at all

thanks to all for helping out. I do hope we won't need to restore data from past three months, because those backups are lost. Matjaz Antloga, BalkanCloud, information technologies.

k00laid
Veeam Vanguard
Posts: 200
Liked: 40 times
Joined: Jan 13, 2011 5:42 pm
Full Name: Jim Jones
Location: Hurricane, WV
Contact:

Re: old backups are gone

Post by k00laid » 1 person likes this post

I got nailed by this as well and have complained to anyone and everyone possible to get this BS setting removed. I wholeheartedly agree with BC that disk management should be on the customer, you don't need to have this setting there at all. Further I don't even have the option to disable it, I've had to set the setting to 999 for all of my jobs so that I *hopefully* won't run into it again.

If there is strength in numbers please add my vote for everything that BC asked for. In a perfect world this setting would die completely, but at a minimum if you are running into a point where backups are going to be deleted from the chain due to the deleted VM retention period then a user input should be required before the backup job will run.

TravisP
Influencer
Posts: 10
Liked: never
Joined: Nov 03, 2010 2:01 pm
Full Name: Travis Patenaude

[MERGED] Veeam Deleted VM Retention

Post by TravisP »

I ran into an interesting issue with Veeam and after going back and forth with Veeam support I understand what happened and wanted to just throw this out there to hopefully help other people from making this misconfiguration. I also believe that this is an issue with Veeam that like me can cause you to lose all your backup restore points.

Network is running VMware 4.1 with two hosts, one a production host and the second for Veeam replicas, they are using Veeam 6.5 latest patch. Two Veeam jobs are configured, VM backup to a Synology NAS that runs twice a day and set to keep 56 restore points running as reverse incremental. Second job is replication to off-site host that runs once a day and set to keep 4 restore points. I was tasked to upgrade vSphere to 5.1 and upgrade Veeam to v7, my plan of attack was to upgrade vCenter to 5.1 and upgrade the Replica host to 5.1u1. The production host and Veeam would be upgraded a week later due to scheduled maintenance window and waiting on Veeam licenses. After upgrading vCenter and the DR Host, I verified Veeam was able to communicate with vCenter and rescanned the vCenter server. I also verified the backup repository was available and Veeam could access it without issue. I allowed the backup schedule to run as normal and would follow up on the backups the next day to verify it was successful. Here is what I found. When vCenter was upgraded from 4.1 to 5.1 it changed the instance uuid of the VM’s, there was no indication of this during or after the vCenter upgrade and the only way to have seen this change was to check each VM’s UUID and compare it to the UUID after the upgrade of vCenter. Now the question is, how does this impact Veeam and your backup restore points? Veeam uses the UUID to identify the VM within vCenter, and if the UUID has changed for that VM, Veeam sees it has a new VM and that the VM with the UUID in the backup job has been deleted. Now within the Veeam backup job under storage > Advanced there is a setting to remove deleted vms from backup after set number of days. The default for this is 14 days, if you have your restore points set to less than 14 days then this option will never reach its limit and will not delete the vm from the backups. Now if you have your restore points higher than the Deleted VM retention policy, it will remove the VM from the backup and all restore points for them VM regardless of what your configured restore points are set too. So in my case I had the backup set to 56 restore points, but the Deleted VM retention was set at 14. So when Veeam ran the backup after vCenter was upgraded to 5.1 and the UUID’s changed on the vm’s, Veeam labeled these vm’s as deleted, and then deleted all restore points for all the VM’s.

After talking with Veeam on this I understand what happened, but I find it interesting that a secondary setting (Deleted VM retention policy) would be able to override the number of restore points set within the job and cause you to lose all restore points for a vm or even that job. Veeam mentioned that the deleted retention policy should be higher than the restore points, then the VM would never be deleted from the backup job. Since it will never remove the deleted vm from the normal backup because the job would remove the last backup that fell out of the restore point window and never get to the Deleted VM retention window. If this is the stance of Veeam for the Deleted VM Retention Policy, then it should be a check box and not an auto configured setting with in the job. This is an easy misconfiguration that can cause a lot of issues and data loss. If you need an old VM removed from the backup you should manually remove it, or manually enable the delete VM retention policy and select which vm’s to remove.
I just wanted to point this out so hopefully it will help someone else not make the same mistake and save backups. I also wanted some thoughts on the deleted VM retention and why that may override the restore point setting.

foggy
Veeam Software
Posts: 19523
Liked: 1780 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: old backups are gone

Post by foggy »

TravisP wrote:I allowed the backup schedule to run as normal and would follow up on the backups the next day to verify it was successful.
Travis, did you get the corresponding warning in the job session log after the VM IDs had changed? You should keep getting those warnings during the whole retention period (14 days).

Btw, some considerations behind this setting can be found here.

TravisP
Influencer
Posts: 10
Liked: never
Joined: Nov 03, 2010 2:01 pm
Full Name: Travis Patenaude

Re: old backups are gone

Post by TravisP »

After the vCenter upgrade the first backup completed on 8/23 then on 8/26 I received the message below.

8/26/2013 11:01:18 AM :: Removing 'E:\XXX-Production\VeeamBackups\XXX-Backup_to_NAS _iScsi_2013-07-25T090047.vrb' per retention policy

8/26/2013 11:01:40 AM :: Storage file 'E:\XXX-Production\VeeamBackups\XXX-Backup_to_XXX-NAS012013-08-23T110102.vbk' is missing from host 'This server'. If you are using initial replica seeding, please follow the instruction from the readme.txt file in the chosen seeding location. If you changed replica destination in the replication job settings, please make sure you have moved all replica files to the new destination correctly.

it does not show that the VM's were no longer part of the backup job or were going to be removed from the backup.

Veeam Case # 00447131 I uploaded the logs when I opened the case.

larry
Expert
Posts: 387
Liked: 94 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: old backups are gone

Post by larry »

thanks for the post, we are updating our VC next week and this would of got us.
I would think that Deleted VMs data retention period would be set to retention period. If I want 50 retention period then why would I not wat to keep 50. I know this is needed for when you reload a VC or delete a vm the backup file use to stay twice as large as it should.
But this would of deleted alot of my backups if I missed the daily warnings - which we wouldn't.

larry
Expert
Posts: 387
Liked: 94 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: old backups are gone

Post by larry »

I just fixed all mine.
If I understand this then if I bring up a clone as the new live VM then by default after 14 days all backups of the orginal VM will be deleted?

foggy
Veeam Software
Posts: 19523
Liked: 1780 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: old backups are gone

Post by foggy »

TravisP wrote:it does not show that the VM's were no longer part of the backup job or were going to be removed from the backup.
This should not be the case... When was vCenter upgrade perfromed?
larry wrote:If I understand this then if I bring up a clone as the new live VM then by default after 14 days all backups of the orginal VM will be deleted?
Only if the original VM was deleted or removed from the job.

TravisP
Influencer
Posts: 10
Liked: never
Joined: Nov 03, 2010 2:01 pm
Full Name: Travis Patenaude

Re: old backups are gone

Post by TravisP »

This should not be the case... When was vCenter upgrade performed?
vCenter and the DR Host was upgraded on 8/23. The production host was upgraded a week later on 8/30 along with Veeam to version 7.

foggy
Veeam Software
Posts: 19523
Liked: 1780 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: old backups are gone

Post by foggy »

And when restore points for the deleted VMs were removed? I can see the following lines in your log files:

Code: Select all

[23.08.2013 13:11:36] <01> Info     [RetentionAlgorithm] Vm 'XXX-EXCH01' contained 56 points and 0 from them have been removed.
however do see nothing like:

Code: Select all

...contained 56 points and 56 from them have been removed

TravisP
Influencer
Posts: 10
Liked: never
Joined: Nov 03, 2010 2:01 pm
Full Name: Travis Patenaude

Re: old backups are gone

Post by TravisP »

My last avalable restore point is from 8/28/2013 that I can actually restore files or vm from, But within Veeam it shows restore points past that date as good. If I attempt to restore files from anything past 8/28/2013 it fails stating it cannot find the vrb file.

foggy
Veeam Software
Posts: 19523
Liked: 1780 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: old backups are gone

Post by foggy »

I asked because if you have deleted VMs retention set to 14 days, then they should be removed on the 15th day after the vCenter upgrade, however there is no evidence of that in your log files, as far as I can tell. Anyway, my point is that Veeam B&R does not silently clean up deleted VMs data from the restore points, but rather displays warning in the job session log on each job run during the whole retention period before doing that. If you're positive that you didn't get those, it worth to be investigated.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 36 guests