Comprehensive data protection for all workloads
Post Reply
Jadwiseman
Novice
Posts: 7
Liked: 1 time
Joined: Feb 17, 2017 2:57 pm
Full Name: Jack Downing
Contact:

Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]"

Post by Jadwiseman »

Hi all,

I've got an issue at the moment where backup jobs are taking a longt time to begin processing - they get to the "Hard disk 1 (0.0 B) 0.0 B read at 0 KB/s [CBT]" and hang there for a long time before actually starting. I've had some jobs take 15 minutes to start wheras one job has took 40 minutes before starting. This is me running one or two jobs at a time, which the server has had no issue with in the past. Also 'Bottleneck' stays on "Detecting" until the job actually starts processing correctly.

It's not CBT related as i've tried disabling that on a job, but it made no difference. I've restarted the server, with no luck, and recreated a job from scratch again with no luck. Once the job begins, it correctly uses [san] and is a fast 400MB/s+ speed so the job completes quickly. I've also rescanned the backup infrastructure again with no issues.

After completing, the load is "Sourece 99% > Proxy 23% > Network 22% > Target 1%". This used to work without issue, and is a new HPDL380 G9 server comprising of 18x 1.2TB 10K disks in a RAID10 configuration. The server is not being overworked at all (1% CPU, and 12% memory)

Any ideas would be greatly appreciated.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Post by Delo123 »

Did you have a look into the actual job logs? They are much more detailed and might give a clue what is happening in the "waiting time", i remember us having some of these issues to in some older release but i can't remember the resolution, but i believe it had something to do with the SQL Database size, but i really can't remember... Alternatively opening a ticket with Veeam support is of course the best way to have a look at your specific issue.
Jadwiseman
Novice
Posts: 7
Liked: 1 time
Joined: Feb 17, 2017 2:57 pm
Full Name: Jack Downing
Contact:

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Post by Jadwiseman »

Thank you for getting back to me.

The SQL databases are tiny in size and currently have no limitations on how big they can grow.

I'm having a look at one of the detailed job logs, and can see that is gets stuck around the server thumbprint:

Code: Select all

[quote][22.02.2017 12:53:39] <10> Info     [NDLVEEAM-01.MoneyAdvice.net] [FC] Renaming remote file from [B:\Backup\NDLPOWERIQ\NDLPOWERIQ.vbm_4_tmp] to [B:\Backup\NDLPOWERIQ\NDLPOWERIQ.vbm]
[22.02.2017 12:53:39] <10> Info     [Backup] SetMetaUpdateTimeAndVersion, old value: [2/22/2017 12:53:04 PM,123] new value: [2/22/2017 12:53:39 PM,124]
[22.02.2017 12:53:39] <10> Info     Releasing mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 12:53:39] <10> Info     [BackupMetaUpdater] Backup meta generated
[22.02.2017 12:58:58] <47> Info     Removing NDLVCENTRE-01:443:MoneyAdvice\veeam:False::0:1 from cache. Reason: Expired
[22.02.2017 12:58:58] <47> Info     [Soap] Logout from "https://NDLVCENTRE-01:443/sdk"
[22.02.2017 12:58:58] <47> Info     Server thumbprint: C93AA154F7E8A087DEF25ABE33A2679819A0F198
[22.02.2017 13:07:57] <10> Info     [BackupMetaUpdater] Generating backup meta because it was requested by the following threads:
[22.02.2017 13:07:57] <10> Info         [BackupMetaUpdater] Thread <33> NDLPOWERIQ
[22.02.2017 13:07:57] <10> Info     Created mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 13:07:57] <10> Info     Acquired mutex [Global\BackupMeta[811b870d-93e3-49ae-b2a9-ffe517c29546]]
[22.02.2017 13:07:57] <10> Info     [IncrementalMetaGenerator] Generating meta for backup [028782d5-c63d-4bc2-909a-ac57fc813c0c], usn [300571]
[22.02.2017 13:07:57] <10> Info     [BackupMeta] Saving backup meta to 'B:|Backup|NDLPOWERIQ|NDLPOWERIQ.vbm'[/quote]
Any ideas?
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Post by foggy » 1 person likes this post

Jack, please let our engineers review the log files to define the reasons of this behavior.
Jadwiseman
Novice
Posts: 7
Liked: 1 time
Joined: Feb 17, 2017 2:57 pm
Full Name: Jack Downing
Contact:

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Post by Jadwiseman » 1 person likes this post

Hi Foggy,

Thank you for the reply - I have already raised a support case with Veeam including the log files, I am just waiting for a reponse (Case #02079076).

I was just hoping to widen my support net by posting on the forums also just in-case someone has experienced this before.
paulposition
Lurker
Posts: 2
Liked: never
Joined: Apr 24, 2017 6:31 am
Full Name: PAOLO MENZAGHI
Contact:

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Post by paulposition »

Hi Jadwiseman, did u solved? I'm experiencing the same issue
rgds
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Post by foggy » 2 people like this post

Hi Paolo, Jack's case was closed due to no response from him, please open your own case for investigation of the observed behavior.
Einar
Enthusiast
Posts: 25
Liked: 1 time
Joined: Sep 02, 2015 9:28 am
Full Name: Einar Steinsson
Contact:

[MERGED] Job hangs

Post by Einar »

I sometimes have the problem that one or more jobs hang like the one in the attached picture. This problem happens to random VM's, sometimes only to one VM at a time but sometimes several at the same time and it happened at random time.
Stopping the job never works so to stop this I must restart the Veeam services or simply reboot the Veeam server.
Not really a big problem but annoying. Is this a known issue? Is there some smart solution? And most important is there a way to stop the job without having to restart the hole system?

Image

https://drive.google.com/file/d/0B80xR9 ... sp=sharing
PTide
Product Manager
Posts: 6551
Liked: 765 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Hanging at "Hard Disk 1 (0.0B) 0.0B read at 0 KB/s [CBT]

Post by PTide »

Hi Einar,

Debug logs should help to identify the source for this delay, please open a support case and have our engineers review.

Thank you
Post Reply

Who is online

Users browsing this forum: No registered users and 53 guests