Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]"

[Jadwiseman]

Hi all,

I've got an issue at the moment where backup jobs are taking a longt time to begin processing - they get to the "Hard disk 1 (0.0 B) 0.0 B read at 0 KB/s [CBT]" and hang there for a long time before actually starting. I've had some jobs take 15 minutes to start wheras one job has took 40 minutes before starting. This is me running one or two jobs at a time, which the server has had no issue with in the past. Also 'Bottleneck' stays on "Detecting" until the job actually starts processing correctly.

It's not CBT related as i've tried disabling that on a job, but it made no difference. I've restarted the server, with no luck, and recreated a job from scratch again with no luck. Once the job begins, it correctly uses [san] and is a fast 400MB/s+ speed so the job completes quickly. I've also rescanned the backup infrastructure again with no issues.

After completing, the load is "Sourece 99% > Proxy 23% > Network 22% > Target 1%". This used to work without issue, and is a new HPDL380 G9 server comprising of 18x 1.2TB 10K disks in a RAID10 configuration. The server is not being overworked at all (1% CPU, and 12% memory)

Any ideas would be greatly appreciated.

[Delo123]

Did you have a look into the actual job logs? They are much more detailed and might give a clue what is happening in the "waiting time", i remember us having some of these issues to in some older release but i can't remember the resolution, but i believe it had something to do with the SQL Database size, but i really can't remember... Alternatively opening a ticket with Veeam support is of course the best way to have a look at your specific issue.

[Jadwiseman]

Thank you for getting back to me.

The SQL databases are tiny in size and currently have no limitations on how big they can grow.

I'm having a look at one of the detailed job logs, and can see that is gets stuck around the server thumbprint:

Code: Select all

[quote][22.02.2017 12:53:39] <10> Info     [NDLVEEAM-01.MoneyAdvice.net] [FC] Renaming remote file from [B:\Backup\NDLPOWERIQ\NDLPOWERIQ.vbm_4_tmp] to [B:\Backup\NDLPOWERIQ\NDLPOWERIQ.vbm]
[22.02.2017 12:53:39] <10> Info     [Backup] SetMetaUpdateTimeAndVersion, old value: [2/22/2017 12:53:04 PM,123] new value: [2/22/2017 12:53:39 PM,124]
[22.02.2017 12:53:39] <10> Info     Releasing mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 12:53:39] <10> Info     [BackupMetaUpdater] Backup meta generated
[22.02.2017 12:58:58] <47> Info     Removing NDLVCENTRE-01:443:MoneyAdvice\veeam:False::0:1 from cache. Reason: Expired
[22.02.2017 12:58:58] <47> Info     [Soap] Logout from "https://NDLVCENTRE-01:443/sdk"
[22.02.2017 12:58:58] <47> Info     Server thumbprint: C93AA154F7E8A087DEF25ABE33A2679819A0F198
[22.02.2017 13:07:57] <10> Info     [BackupMetaUpdater] Generating backup meta because it was requested by the following threads:
[22.02.2017 13:07:57] <10> Info         [BackupMetaUpdater] Thread <33> NDLPOWERIQ
[22.02.2017 13:07:57] <10> Info     Created mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 13:07:57] <10> Info     Acquired mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 13:07:57] <10> Info     [IncrementalMetaGenerator] Generating meta for backup [028782d5-c63d-4bc2-909a-ac57fc813c0c], usn [300571]
[22.02.2017 13:07:57] <10> Info     [BackupMeta] Saving backup meta to 'B:|Backup|NDLPOWERIQ|NDLPOWERIQ.vbm'[/quote]

Any ideas?

[foggy]

Jack, please let our engineers review the log files to define the reasons of this behavior.

[Jadwiseman]

Hi Foggy,

Thank you for the reply - I have already raised a support case with Veeam including the log files, I am just waiting for a reponse (Case #02079076).

I was just hoping to widen my support net by posting on the forums also just in-case someone has experienced this before.

[paulposition]

Hi Jadwiseman, did u solved? I'm experiencing the same issue
rgds

[foggy]

Hi Paolo, Jack's case was closed due to no response from him, please open your own case for investigation of the observed behavior.

[Einar]

I sometimes have the problem that one or more jobs hang like the one in the attached picture. This problem happens to random VM's, sometimes only to one VM at a time but sometimes several at the same time and it happened at random time.
Stopping the job never works so to stop this I must restart the Veeam services or simply reboot the Veeam server.
Not really a big problem but annoying. Is this a known issue? Is there some smart solution? And most important is there a way to stop the job without having to restart the hole system?



https://drive.google.com/file/d/0B80xR9 ... sp=sharing

[PTide]

Hi Einar,

Debug logs should help to identify the source for this delay, please open a support case and have our engineers review.

Thank you