Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]"

Availability for the Always-On Enterprise

Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]"

Veeam Logoby Jadwiseman » Wed Feb 22, 2017 11:45 am

Hi all,

I've got an issue at the moment where backup jobs are taking a longt time to begin processing - they get to the "Hard disk 1 (0.0 B) 0.0 B read at 0 KB/s [CBT]" and hang there for a long time before actually starting. I've had some jobs take 15 minutes to start wheras one job has took 40 minutes before starting. This is me running one or two jobs at a time, which the server has had no issue with in the past. Also 'Bottleneck' stays on "Detecting" until the job actually starts processing correctly.

It's not CBT related as i've tried disabling that on a job, but it made no difference. I've restarted the server, with no luck, and recreated a job from scratch again with no luck. Once the job begins, it correctly uses [san] and is a fast 400MB/s+ speed so the job completes quickly. I've also rescanned the backup infrastructure again with no issues.

After completing, the load is "Sourece 99% > Proxy 23% > Network 22% > Target 1%". This used to work without issue, and is a new HPDL380 G9 server comprising of 18x 1.2TB 10K disks in a RAID10 configuration. The server is not being overworked at all (1% CPU, and 12% memory)

Any ideas would be greatly appreciated.
Jadwiseman
Novice
 
Posts: 7
Liked: 1 time
Joined: Fri Feb 17, 2017 2:57 pm
Full Name: Jack Downing

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby Delo123 » Wed Feb 22, 2017 12:53 pm

Did you have a look into the actual job logs? They are much more detailed and might give a clue what is happening in the "waiting time", i remember us having some of these issues to in some older release but i can't remember the resolution, but i believe it had something to do with the SQL Database size, but i really can't remember... Alternatively opening a ticket with Veeam support is of course the best way to have a look at your specific issue.
Delo123
Expert
 
Posts: 299
Liked: 82 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby Jadwiseman » Wed Feb 22, 2017 1:13 pm

Thank you for getting back to me.

The SQL databases are tiny in size and currently have no limitations on how big they can grow.

I'm having a look at one of the detailed job logs, and can see that is gets stuck around the server thumbprint:

Code: Select all
[quote][22.02.2017 12:53:39] <10> Info     [NDLVEEAM-01.MoneyAdvice.net] [FC] Renaming remote file from [B:\Backup\NDLPOWERIQ\NDLPOWERIQ.vbm_4_tmp] to [B:\Backup\NDLPOWERIQ\NDLPOWERIQ.vbm]
[22.02.2017 12:53:39] <10> Info     [Backup] SetMetaUpdateTimeAndVersion, old value: [2/22/2017 12:53:04 PM,123] new value: [2/22/2017 12:53:39 PM,124]
[22.02.2017 12:53:39] <10> Info     Releasing mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 12:53:39] <10> Info     [BackupMetaUpdater] Backup meta generated
[22.02.2017 12:58:58] <47> Info     Removing NDLVCENTRE-01:443:MoneyAdvice\veeam:False::0:1 from cache. Reason: Expired
[22.02.2017 12:58:58] <47> Info     [Soap] Logout from "https://NDLVCENTRE-01:443/sdk"
[22.02.2017 12:58:58] <47> Info     Server thumbprint: C93AA154F7E8A087DEF25ABE33A2679819A0F198
[22.02.2017 13:07:57] <10> Info     [BackupMetaUpdater] Generating backup meta because it was requested by the following threads:
[22.02.2017 13:07:57] <10> Info         [BackupMetaUpdater] Thread <33> NDLPOWERIQ
[22.02.2017 13:07:57] <10> Info     Created mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 13:07:57] <10> Info     Acquired mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 13:07:57] <10> Info     [IncrementalMetaGenerator] Generating meta for backup [028782d5-c63d-4bc2-909a-ac57fc813c0c], usn [300571]
[22.02.2017 13:07:57] <10> Info     [BackupMeta] Saving backup meta to 'B:|Backup|NDLPOWERIQ|NDLPOWERIQ.vbm'[/quote]


Any ideas?
Jadwiseman
Novice
 
Posts: 7
Liked: 1 time
Joined: Fri Feb 17, 2017 2:57 pm
Full Name: Jack Downing

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby foggy » Wed Feb 22, 2017 1:17 pm 1 person likes this post

Jack, please let our engineers review the log files to define the reasons of this behavior.
foggy
Veeam Software
 
Posts: 13899
Liked: 1012 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby Jadwiseman » Wed Feb 22, 2017 2:25 pm 1 person likes this post

Hi Foggy,

Thank you for the reply - I have already raised a support case with Veeam including the log files, I am just waiting for a reponse (Case #02079076).

I was just hoping to widen my support net by posting on the forums also just in-case someone has experienced this before.
Jadwiseman
Novice
 
Posts: 7
Liked: 1 time
Joined: Fri Feb 17, 2017 2:57 pm
Full Name: Jack Downing


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], borismittelmann, Yahoo [Bot] and 11 guests