Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]"

Availability for the Always-On Enterprise

Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]"

Veeam Logoby Jadwiseman » Wed Feb 22, 2017 11:45 am

Hi all,

I've got an issue at the moment where backup jobs are taking a longt time to begin processing - they get to the "Hard disk 1 (0.0 B) 0.0 B read at 0 KB/s [CBT]" and hang there for a long time before actually starting. I've had some jobs take 15 minutes to start wheras one job has took 40 minutes before starting. This is me running one or two jobs at a time, which the server has had no issue with in the past. Also 'Bottleneck' stays on "Detecting" until the job actually starts processing correctly.

It's not CBT related as i've tried disabling that on a job, but it made no difference. I've restarted the server, with no luck, and recreated a job from scratch again with no luck. Once the job begins, it correctly uses [san] and is a fast 400MB/s+ speed so the job completes quickly. I've also rescanned the backup infrastructure again with no issues.

After completing, the load is "Sourece 99% > Proxy 23% > Network 22% > Target 1%". This used to work without issue, and is a new HPDL380 G9 server comprising of 18x 1.2TB 10K disks in a RAID10 configuration. The server is not being overworked at all (1% CPU, and 12% memory)

Any ideas would be greatly appreciated.
Jadwiseman
Novice
 
Posts: 7
Liked: 1 time
Joined: Fri Feb 17, 2017 2:57 pm
Full Name: Jack Downing

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby Delo123 » Wed Feb 22, 2017 12:53 pm

Did you have a look into the actual job logs? They are much more detailed and might give a clue what is happening in the "waiting time", i remember us having some of these issues to in some older release but i can't remember the resolution, but i believe it had something to do with the SQL Database size, but i really can't remember... Alternatively opening a ticket with Veeam support is of course the best way to have a look at your specific issue.
Delo123
Expert
 
Posts: 355
Liked: 102 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby Jadwiseman » Wed Feb 22, 2017 1:13 pm

Thank you for getting back to me.

The SQL databases are tiny in size and currently have no limitations on how big they can grow.

I'm having a look at one of the detailed job logs, and can see that is gets stuck around the server thumbprint:

Code: Select all
[quote][22.02.2017 12:53:39] <10> Info     [NDLVEEAM-01.MoneyAdvice.net] [FC] Renaming remote file from [B:\Backup\NDLPOWERIQ\NDLPOWERIQ.vbm_4_tmp] to [B:\Backup\NDLPOWERIQ\NDLPOWERIQ.vbm]
[22.02.2017 12:53:39] <10> Info     [Backup] SetMetaUpdateTimeAndVersion, old value: [2/22/2017 12:53:04 PM,123] new value: [2/22/2017 12:53:39 PM,124]
[22.02.2017 12:53:39] <10> Info     Releasing mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 12:53:39] <10> Info     [BackupMetaUpdater] Backup meta generated
[22.02.2017 12:58:58] <47> Info     Removing NDLVCENTRE-01:443:MoneyAdvice\veeam:False::0:1 from cache. Reason: Expired
[22.02.2017 12:58:58] <47> Info     [Soap] Logout from "https://NDLVCENTRE-01:443/sdk"
[22.02.2017 12:58:58] <47> Info     Server thumbprint: C93AA154F7E8A087DEF25ABE33A2679819A0F198
[22.02.2017 13:07:57] <10> Info     [BackupMetaUpdater] Generating backup meta because it was requested by the following threads:
[22.02.2017 13:07:57] <10> Info         [BackupMetaUpdater] Thread <33> NDLPOWERIQ
[22.02.2017 13:07:57] <10> Info     Created mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 13:07:57] <10> Info     Acquired mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 13:07:57] <10> Info     [IncrementalMetaGenerator] Generating meta for backup [028782d5-c63d-4bc2-909a-ac57fc813c0c], usn [300571]
[22.02.2017 13:07:57] <10> Info     [BackupMeta] Saving backup meta to 'B:|Backup|NDLPOWERIQ|NDLPOWERIQ.vbm'[/quote]


Any ideas?
Jadwiseman
Novice
 
Posts: 7
Liked: 1 time
Joined: Fri Feb 17, 2017 2:57 pm
Full Name: Jack Downing

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby foggy » Wed Feb 22, 2017 1:17 pm 1 person likes this post

Jack, please let our engineers review the log files to define the reasons of this behavior.
foggy
Veeam Software
 
Posts: 15272
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby Jadwiseman » Wed Feb 22, 2017 2:25 pm 1 person likes this post

Hi Foggy,

Thank you for the reply - I have already raised a support case with Veeam including the log files, I am just waiting for a reponse (Case #02079076).

I was just hoping to widen my support net by posting on the forums also just in-case someone has experienced this before.
Jadwiseman
Novice
 
Posts: 7
Liked: 1 time
Joined: Fri Feb 17, 2017 2:57 pm
Full Name: Jack Downing

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby paulposition » Mon Apr 24, 2017 7:22 pm

Hi Jadwiseman, did u solved? I'm experiencing the same issue
rgds
paulposition
Lurker
 
Posts: 1
Liked: never
Joined: Mon Apr 24, 2017 6:31 am
Full Name: PAOLO MENZAGHI

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby foggy » Tue Apr 25, 2017 11:08 am 2 people like this post

Hi Paolo, Jack's case was closed due to no response from him, please open your own case for investigation of the observed behavior.
foggy
Veeam Software
 
Posts: 15272
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

[MERGED] Job hangs

Veeam Logoby Einar » Thu Oct 05, 2017 11:10 am

I sometimes have the problem that one or more jobs hang like the one in the attached picture. This problem happens to random VM's, sometimes only to one VM at a time but sometimes several at the same time and it happened at random time.
Stopping the job never works so to stop this I must restart the Veeam services or simply reboot the Veeam server.
Not really a big problem but annoying. Is this a known issue? Is there some smart solution? And most important is there a way to stop the job without having to restart the hole system?

Image

https://drive.google.com/file/d/0B80xR9lJEUKOTXo4NE9rRGw1Ujg/view?usp=sharing
Einar
Novice
 
Posts: 9
Liked: never
Joined: Wed Sep 02, 2015 9:28 am
Full Name: Einar Steinsson

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Veeam Logoby PTide » Thu Oct 05, 2017 11:50 am

Hi Einar,

Debug logs should help to identify the source for this delay, please open a support case and have our engineers review.

Thank you
PTide
Veeam Software
 
Posts: 3230
Liked: 269 times
Joined: Tue May 19, 2015 1:46 pm


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Yahoo [Bot] and 5 guests