Comprehensive data protection for all workloads
Post Reply
mvb89b
Influencer
Posts: 12
Liked: never
Joined: May 13, 2010 7:09 pm
Full Name: Mike Bommarito
Contact:

Backup Job Hung VEEAM 4.1.1

Post by mvb89b »

I have just noticed an issue that has been happening for about a week with a backup job of one of our SQL2005 VMs. Basically the backup was just hanging which would then cause the replication to fail as well at night. How can I force this job to stop and retry or fix this issue? I right click the job and hit stop but nothing happens and there is a snapshot on the VM it is trying to backup.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by Vitaliy S. »

Hello Mike, you may open Task Manager on the Veeam backup console and End all Veeam processes, then re-open the console, should help. Thanks!
mvb89b
Influencer
Posts: 12
Liked: never
Joined: May 13, 2010 7:09 pm
Full Name: Mike Bommarito
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by mvb89b »

So what about the snapshot that veeam backup has on the hanging VM?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by Gostev »

Remove snapshot manually using vSphere Client.
mvb89b
Influencer
Posts: 12
Liked: never
Joined: May 13, 2010 7:09 pm
Full Name: Mike Bommarito
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by mvb89b »

So stop processes on veeam backup console...

Then... Sorry I am new to this so just go to snapshot manager -> Delete VEEAM BACKUP TEMPORARY SNAPSHOT

Just for my sanity deleting the snapshot will keep the VM in it's current state even if the snapshot was created over a week ago and has been hung?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by Vitaliy S. »

Yes, that's right. Just delete the snapshot to commit all the changes to your source virtual disk.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by Gostev »

Deleting snapshot commits all snapshot data into virtual disks.
Reverting snapshot snapshots discards all changed data.
mvb89b
Influencer
Posts: 12
Liked: never
Joined: May 13, 2010 7:09 pm
Full Name: Mike Bommarito
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by mvb89b »

I actually performed the steps backwards and attempted to delete the snapshot before killing processes on veeam backup host. The error resulted in something along the lines of:

Cannot delete it is locked but the snapshot is gone from the manager. Is this an issue?

I then killed the backup processes and restarted veeam backup and it shows as stopped. Should I be concerned?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by Gostev »

You should check datastore, as because of the error you mention, it may still have snapshot files - despite Snapshot Manager not showing those.

If you find this to be the case, try the following:
1. Shutdown Veeam server.
2. Create new snapshot on this VM manually with vSphere Client, this should reveal hidden snapshot/
3. Delete all snapshots on this VM manually with vSphere Client. You should not get any errors now, and the process should complete successfully.

On a side note, you are not running the latest Veeam version, and I highly recommend upgrading.
mvb89b
Influencer
Posts: 12
Liked: never
Joined: May 13, 2010 7:09 pm
Full Name: Mike Bommarito
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by mvb89b »

Ok so I am going to perform these steps.

So when I attempted to delete the snapshot it wrote everything to disk but just failed at the last step of deleting the snapshot?

I just want to make sure I shouldn't backup the VM before doing this or mess anything up it is a production server.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by Gostev »

Only VMware can tell you what happened. But I do know VMware behaves strangely when some part of snapshot removal fails, which causes abandoned snapshots. I researched this sometime ago, and I found the logic they use does not account for possible failures on certain operations, or sequence of actions is not correct (e.g. snapshot record is removed from ESX database before the actual snapshot file is removed). This may already be fixed in 4.1, I did not check this since then.
mvb89b
Influencer
Posts: 12
Liked: never
Joined: May 13, 2010 7:09 pm
Full Name: Mike Bommarito
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by mvb89b »

So the previous steps corrected my issue for about a month now I am starting to see errors again. I believe they have to do with the amount of free space on the backup storage but I am unsure because there are a couple different errors that popup:

1. Backing up file "[datastore1] xxxx-flat.vmdk" BackupDisk failed Client error: The specified network name is no longer available Failed to read data from the file [xxxxx.vbk]. Server error: End of file

2. Backing up file "[datastore1] xxxx--flat.vmdk" BackupDisk failed Client error: The specified network name is no longer available Failed to write data to the file [xxxxx.vbk]. Server error: End of file

I got our SQL one to correctly backup yesterday but overnight it attempted to run and today it is still goingh 7 hours later and says 32% @ 1mb/s. I try to stop it and it tells me:
- Backup files for this job will be left in an inconsistent state are you sure?

I just want to stop the job / end the processes / delete the snapshot and try again.

Any help is appreciated... It seems to be this one VM that continues to give me trouble.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Hung VEEAM 4.1.1

Post by Vitaliy S. »

Well... judging by the error it seems like there was an intermittent connectivity issue to your source storage. Did you have any other VMs being backed up from the same datastore at the same time by the way? As for stopping a running backup job, than feel free to do that, the next job run should take care about snapshot removal etc.

If you need any assistance from our support team, feel free to contact us at support@veeam.com
Post Reply

Who is online

Users browsing this forum: No registered users and 282 guests