Unable to find scale-out repository extent...

[pkelly_sts]

[Case number 01788614]

Failed to merge full backup file error: Unable to find scale-out repository extent with previous backup files
Failed to generate points error: Unable to find scale-out repository extent with previous backup files

I thought I'd start sharing this one as it's getting a bit "interesting" and I've never seen the above errors posted anywhere on here so as it progresses it might help someone else but even though it's not yet resolved (currently awaiting a remote session with support) I wanted to check something whilst waiting...

This is related to a a backup copy job to a remote site.
1) Local backup to local scale-out repository consisting of two performance configured extents (originally created from two simple repositories, one of which contained the original backup chain before conversion to scale-out).
2) Copy job to a remote scale-out repository that was configured in the exact same way, i.e. an existing repository with backup chain, added into a new scale-out, then a second simple repository added to the scale-out to create a performance based scale-out repository.

The GFS backup copy basic process is working in that it's copying the .vib files, but it's the MERGE process that seems to be failing. The last successful merge was Saturday, all incrementals since then exist at the destination but the .VBK is dated saturday.

It turns out that we have an issue with our local scale-out repository in that there is somehow an "illegal character" in the path within the DB (and support is looking at a copy of the DB - I'm hoping I don't have to re-seed everything!) but, if you were to see the above errors, would that indicate to you a problem with the destination scale-out repository or the source one?

I'm thinking it must point more the the destination because all the files related to the merge exist there don't they? However, how dependent is the merge process (at the destination) on the source data/repository, given that it has already copied over the .vib file(s)?

[pkelly_sts]

Also for clarification, although this error is complaining about not finding extents, backups are continuing ok locally, to the extent that is affected by the "illegal character"...

[foggy]

Yes, this relates to the target repository, merge process is not dependent on the source data. So I suggest to look further into this with the help of your support engineer. Btw, is there a tape job that uses this backup copy job as a source?

[pkelly_sts]

Thanks, I suspected as much.

Yes, there is a tape job that relies on this copy job as a source, and there is also a replication job that relies on it. I'm assuming both of these would need to be started again from scratch if I do end up having to re-seed? (That's why I'm trying to avoid doing it!)

[foggy]

Yes, there is a tape job that relies on this copy job as a source

Then it is a known issue that will be addressed in the next update.

[pkelly_sts]

Ah, you mean the inability to find the extent? Is the tape job the cause of the problem? if so, if I disable the tape job then would it resolve my issue - which is the inability do the merge?

[foggy]

Not sure, but you can try.

[pkelly_sts]

Update: By golly I think we might have cracked it! At least for now anyway.

It's been a slow process with support but this morning I had the following set of instructions to try:

Please perform the following steps and update us with results:
1) Remove from Backups the Backup Copy Job chain.
2) Rescan the Scale-Out Repository after that.
3) You will be able to find the Backup Copy chain into the Disk (Imported) section.
4) After that please try to go to the properties of the Backup Copy chain itself.
5) If it is possible to browse, please map the Backup Copy chain into the Backup Copy Job and update with results about the merge operation.
6) Also try to rebuild the Scale-Out Repository and let me know is it possible for you or not.

I wasn't sure how best to carry out step 6 (I did reply a while ago but heard nothing yet) so I went ahead & completed 1 - 5 with no change, i.e. complaints about unable to find scale-out repository.

So, with nothing much to lose, I thought I'd try removing the copy job data selection from the remote site tape job (I guess that'll cause me to have to run a complete synthetic job again or will it still recognise that I have the data on tape already?) and retry steps 1 -5.

I'm pleased to say that it's currently in the process of creating a synthetic full!!!

So, at this point, properly-working backup copies are more important to me than remote tape copies so I guess I'll live without them for now in anticipation of the fix in the next release!

Thanks for the heads up, something last week made me think it was going to be more awkward removing the tape dependency side of things but I can't remember what that was now & just did it! :-/

(I might actually consider migrating the scale-out repository back to the original 2 simple repositories, is there an easy/safe way for such a migration?)

[pkelly_sts]

Actually related to this, now that it's finally generating a GFS restore point correctly, it'll likely conflict with the normal backup copy window tonight, e.g. it will probably, at best, be only 50% through the merge before the parent backup job has new restore points to copy. In such a conflict, which process wins: Getting another restore point copied (thus interrupting the GFS merge) or will the merge complete before another restore point is copied?

[foggy]

GFS transformation will be completed prior to creating a new restore point.

[pkelly_sts]

Thanks, makes sense but wanted to check.

[pkelly_sts]

Hmm, well although the merge process SEEMS to be working (and I'm getting new .vbk files) at the end of the process I'm still getting "Unable to find scale-out..." errors so now I'm not sure how to proceed and I'm not getting anywhere near reasonable response times from support unfortunately so I'm starting to get quite frustrated now :-/

[foggy]

Slow response times are probably caused by the fact that your engineer needs to check/sync with R&D regarding this, since it is an issue that is currently being investigated/addressed. You can always use the Talk to a Manager option to provide your feedback to support management.

[pkelly_sts]

Oh I entirely understand that there's not just one person behind the scenes working on my issue, and that I'm not the only customer that needs help, but I'm talking about things like where they send me a high-level list of things to try (at around 09:00), and I reply in 2hrs seeking clarification to one of the items in that list, I still haven't had a reply to that clarification request 26hrs later, nor to the email sent to them 24hrs previous to that one.

Oh well, I'll just have to work through it & wait patiently, reflecting my concerns in the post-closure questionnaire that normally comes through though I've never spotted the "Talk to a manager" option so I guess it won't hurt to give that a poke.

Thanks again.

[foggy]

I've also asked responsible person to take a closer look at your case. Sorry for inconvenience.

[pkelly_sts]

Thanks foggy, I sent an email to manager anyway explaining things and have had a chat so hopefully things will start moving again.

[pkelly_sts]

Final update on this, I did get the hotfix to fix the issue where (in my case at least) a remote tape job linked to a remote backup copy job caused the "unable to find" repository errors and things now seem to be running smoothly once again.

[wilkins44]

I'm curious what the final resolution was on this. I've had our three Backup Copy jobs running for about a week now, and over the weekend, I started getting the "Unable to find scale-out repository extent..." error on two of the jobs. Unfortunately, the two jobs that are generating the error are the ones that have running VMs on them.

The backup copy job is copying from one disk based scale-out repo to an offsite disk based scale-out.

[foggy]

Hi Jay, if these backup copy jobs are used by some tape job(s) as a source, you might request the hotfix from support (it might be included in v9 update 2 as well). Anyway, contacting support is recommended.

[wilkins44]

I'm happy to say that there is no tape in this environment at all! I got tired of tape drives eating tapes or having to troubleshoot tape drive issues.

I'll get in touch with Support. Thanks for the input!