Snapshot removal WITH REPLICA vers 8.0.0.917

[royp]

We have Veeam BnR 8.0.0.917.
On one of the Vms a win 2008 file and print, we can complete a backup no problem what so ever with no issues at all.
HOWEVER
We cannot create a Replica.
For some bizarre reason it cannot remove the previous snapshot on the replica as all the options are greyed out on the replica VM.
The source machine is fine it has the one newly created snapshot with all options active IE create new snapshot and delete snapshots.
But the replica is just sat there with all its snapshot options greyed out and its power on off restart guest machine etc greyed out.
I cant even delete the replica or remove it from inventory because its options are all greyed out.
I then went into browsing the datastore to brutally delete the replica guest and start again, still no luck it says it is locked by an external application ie veeam Im assuming.
So I simply removed the datastore from within vcenter itself, deleted it completely.

Then recreated the store and let Veeam start a new replica job from scratch which it did as it had no snapshots on this newly created replica datastore.
Success.
New Replica created.
HOWEVER
On the creation of the next replica job, back to square 1 again, it was unable to remove the previous snapshot and is sat within vcenter again showing remove snapshot on 99% and all the replica machine options are greyed out with Veeam sat waiting

Code: Select all

10/03/2015 10:09:40 :: Queued for processing at 10/03/2015 10:09:40 
10/03/2015 10:09:41 :: Required backup infrastructure resources have been assigned 
10/03/2015 10:09:45 :: VM processing started at 10/03/2015 10:09:45 
10/03/2015 10:09:45 :: VM size: 2.0 TB (2.0 TB used) 
10/03/2015 10:09:50 :: Discovering replica VM 
10/03/2015 10:10:35 :: Getting VM info from vSphere 
10/03/2015 10:10:37 :: Network traffic will be encrypted 
10/03/2015 10:10:55 :: Inventorying guest system 
10/03/2015 10:11:05 :: Preparing guest for hot backup 
10/03/2015 10:11:12 :: Creating snapshot 
10/03/2015 10:11:19 :: Releasing guest 
10/03/2015 10:11:50 :: Preparing for transfer resume 
10/03/2015 10:14:36 :: Preparing replica VM 
10/03/2015 10:14:38 :: Processing configuration 

THIS IS AS FAR AS IT EVER GETS, ITS NOW ON 99% IN VCENTER WITH THE REPLICA SAT ALL GREYED OUT AGAIN.
SAT THERE NOW FOR 6 HOURS, LAST TIME IT WAS A FULL WEEKEND

[foggy]

On the creation of the next replica job, back to square 1 again, it was unable to remove the previous snapshot and is sat within vcenter again showing remove snapshot on 99% and all the replica machine options are greyed out with Veeam sat waiting

What kind of snapshot it is?

[royp]

Code: Select all

Veeam Replica Working Snapshot
<RPData PointTime="5247298517187871315" WorkingSnapshotTime="5247298517187871315" PointSize="0" PointType="EWorkingSnapshot" />
Veeam Replica Working Snapshot
<RPData PointTime="5247298667375095303" WorkingSnapshotTime="5247298667375095303" PointSize="0" PointType="EWorkingSnapshot" />

This is on the actual problematic replica vm.


ON THE SOURCE the one that is not greyed out we have the usual
VEEAM BACKUP TEMPORARY SNAPSHOT.
Please do not delete this snapshot. It is being used by Veeam Backup..
With all options still active.

[foggy]

Do you have any other VMs on the target host/datastore? What if you just try to create/remove the snapshot for it manually, via vSphere Client?

[royp]

Re other hosts and back end data store,
yes we have a mixture of live hosts and also replica's on the same host and data store all working fine.

I cannot create a snapshot manually as the options are all greyed out including the delete snapshot and delete from disk (ie delete the entire replica machine),
I cant even manually delete the snapshot in the data store as it is locked by an application etc.

I put a support call in with Veeam on Friday 6th and Im still chasing a response.
I put a support call in with VMware yesterday as I guess this could be an issue with either or both.

Its just this one VM guest that has this issue with the replica's, but works fine with backups.

[foggy]

yes we have a mixture of live hosts and also replica's on the same host and data store all working fine.

I was meaning creating/removing a snapshot of some other VM on the same host/datastore.

Its just this one VM guest that has this issue with the replica's, but works fine with backups.

So you can replicate other VMs fine, right?

[royp]

Had webex session with VMware who were exceptionally good.
Identified that the Veeam job had not hung, but ESX was still processing the Snapshot, albeit really slowly 7megs a sec.
I have since tried and tested this countless times now and always the same result, double checked the storage as well ie network side.

A Veeam Backup of the same VM Guest to the same HP Virtual San (VSA) Storage flies through really quickly (no snapshots pushed to the VSA or deleted using a backup job)
A replica of the same VM Guest, pushed to the same HP VSA will drag its feet each time as soon as it gets to pushing the snapshot over to the VSA Storage.

I have no idea why or how, but something about the creation and or deletion of the replica snapshots is causing replica processing to take forever.

We have HP Lefthand Storage shelves interestingly, and testing by pushing the replicas over to them works fine, extremely quick, but due to cost we cant push all our replicas across to the expensive HP Lefthand shelves as we would run out of space, hence the use of the cheaper HP Virtual San software. I wonder if anyone else has found this to be the case with the HP VSA and Veeam Snapshots etc.

[foggy]

Roy, have you tried to create/remove snapshot of some other VM residing on the same storage? Just to understand whether we can rule out Veeam B&R of the picture.

[royp]

Yes we have another replica on the same storage box etc which I have now realised that this isn't running quick ie 600mb of incremental changes processed at 3mb took 26 minutes.
Where as the Exchange server for instance, which pushes its veeam replicas to the expensive HP Lefthand storage as opposed to the cheap HP VSA storage did 5 gig of changes in 11minutes at 35 megs.

Looks to me as though its something to do with the way the HP VSA handles the replica jobs ie snapshots as the bog standard veeam backups fly through to the VSA, but Im guessing.

[foggy]

What do our support engineers say about that?

[royp]

Nothing.
I logged this a week ago today 6th March,
I had a missed call from Alex Lebedev on 11th March, He emailed to say he had tried to contact me.
I emailed back saying VMware support were looking into it for me.
I then emailed Alex Lebedev again yesterday at 14:40 putting down near enough what I have put here this morning to you asking if he had any idea's or advice, but as yet I have not had a reply.
Case # 00825570 [ ref:_00D30RWR._50060n3tJI:ref ]

[foggy]

I can see that you've been just contacted by engineer, so please continue working with him on this issue.

[royp]

Ok thanks Alexander,
Just had an email from Alex who is going to speak with your Senior Engineers to see if they have any clues as to what is happening or not happening.
Will update if there is a positive result incase others have the same issue in the future.
Thanks ever so much for your help, I do appreciate it.

[royp]

A quick update fyi
Just in case anyone else is using HP VSA to store some of their replicas on using Veeam V8.
Our replica's to the HP vsa were running at 3MB/s and when one of our file and print servers is 2tb in size it was obviously to slow.
Now thinking about down grading Veeam to v7 as it seems to have no issues with the replica's to our cheap storage, and it is also processing them quicker to our cheap storage than v8 is processing replicas to our expensive storage.

Ive just updated our support case with the below.

Note added: Morning Alexey. The initial Full replica using Veeam BnR v7 completed in a little over 10 hours at 58 MB/s. The next replica which I have just run which was obviously the incremental completed in 19 mins at 52MB/s. So this says to me quite clearly, that Veeam BnR V8.0.0.917 cannot do replicas to HP VSA storage as V7.0.0.871 has no issues what so ever