Slow FLR Initialization

[crazzyeddie]

Hello everyone,

I'm hoping for some additional insight on an issue I'm having with Windows FLR jobs. I have a current case open (00846536).

When initiating the FLR job on any restore point (increment or full) I'm having to wait 10 minutes before the job actually starts to recover files. What is driving me crazy is that there is a flurry of disk activity at the beginning of the job, but then for over 9 minutes, there is zero disk activity. Around the 10 minute mark, the FLR wizard shows my files and/or restores what I've requested. Note that during a large portion of the 9 minutes, the FLR wizard is unresponsive.

The support rep in my case said this is normal because I'm on a RAID6, and I would completely understand if it was loading data during that period, but the 9 minutes of almost zero CPU/disk activity makes me believe there is something else going on. There was a suggestion to move my job to a "local disk," but this is technically a local disk (DAS) and, again, I see no disk activity that would lead me to believe that this would help.

Has anyone else experienced this issue?

Veeam B&R v8 Update 1
2.4ghz x 8 CPU
24GB RAM
RAID6 SAS DAS Storage

[nielsengelen]

Hi Will, I would advise you to open a support case and get this issue resolved asap.

[crazzyeddie]

Hi Niels,

I already have a case open, as indicated (case 00846536).

I'm just not receiving an answer that makes any sense to me ("This is expected behavior."), which is why I'm trying to make more sense of the situation.

[Vitaliy S.]

Hi Will,

It's not expected, but can you please tell me if you see the same issue with both native FLR wizard and other OS?

Thanks!

[crazzyeddie]

The Other OS wizard fails. The NFS datastore mounts, but the FLR Appliance is never added to the inventory... I will investigate this with my support rep.

[Vitaliy S.]

Also do you have any other job/VM to test the FLR restore wizard? Is it also slow?

[crazzyeddie]

I tested with another job, my second largest, also on the same DAS, and it was available within 1 minute. I was able to copy a large file off at 140-150MB/sec.

The job I was originally testing is 2.3TB on disk (after dedupe/compression)
The job I most recently tested is 1.1TB on disk (after dedupe/compression)

[Vitaliy S.]

Do you see the same performance issue for other VMs in the original job?

[crazzyeddie]

Yes, I'm seeing the same issue; no disk or CPU activity for 90% of the restore. It actually took a few minutes longer for the other VM.

[foggy]

Is guest OS file system indexing enabled for this job?

[Vitaliy S.]

Ok, in this case I would show both restore logs to your support engineer to find the reason for this difference in the performance rate.

[crazzyeddie]

Is guest OS file system indexing enabled for this job?

Yes, definitely. These are file servers- 3 of the 4 are over 1TB.

I tested the smallest VM in the job (140GB vs over 1TB) and it completed quickly.

I am understanding that the other VMs are large, and it would take much longer to load that data, but can't understand what is actually happening during the "dead time" where I'm seeing no resource usage. I also see the "progress wheel" in the FLR "Please wait..." dialog box freeze during this period.

[emachabert]

Are the "slow machines" windows 2k12 ones with dedupe enabled within the guest ?

[crazzyeddie]

No, most are 2008R2 guests with nothing complicated going on. The Veeam server is on 2012R2, but no dedupe going on.

[emachabert]

Ok. Because I once had a similar issue with a dedupe enabled 2k12 VM where veeam was taking hours to process ntfs reparse points, and it wasn't showing any disk activity.

Did you looked at the job logs ?

[crazzyeddie]

They've been submitted to my support rep, so I'm awaiting a reply on that.

[Vitaliy S.]

I see what Eric is referring to, nice catch, however in this thread there is another possible reason for slow initialization > Very slow initial loading of FLR, normal?

This should have been fixed by now, so I would appreciate if you could update this topic after logs review with your support engineer.

[crazzyeddie]

I definitely read that thread thoroughly before posting, but it obviously should (still) be fixed in v8.

I will definitely followup when I get more info.

Thank you all for being so helpful.