Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
zx81
Service Provider
Posts: 13
Liked: 1 time
Joined: Nov 24, 2016 6:57 am
Location: Perth, Australia
Contact:

Failed to perform backup Snapshot deadlock detected

Post by zx81 »

We installed Veeam Agent for Linux on one of our Ubuntu Linux NASs. On attempting to backup the boot drive (/dev/sda) we got the following message, soon after which the kernel panicked and we had to power cycle the machine to recover. Luckily we were running DRDB (https://www.drbd.org/) and we failed over to the backup machine.

Code: Select all

Backing up sda
Failed to perform backup
Snapshot deadlock detected
Snapshot deadlock detected
We don't have support for this product, but I'm posting the details here in case it's of use to anyone.

The machine is running vanilla Ubuntu 14.04.5 LTS (3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) and we have DRDB 8.4.4 installed. I don't know if the following has anything to do with the crash, but we have configured DRDB to place it's meta-data onto a loop device. The device points to a file that lives in / on /dev/sda1, which is one of the partions we were trying to backup. So we do:

Code: Select all

 /sbin/losetup /dev/loop0 /drbd-metadata.img
And the DRDB config points to this device:

Code: Select all

resource r0 {
    net {
        shared-secret "xxxxx";
    }

    on drdb-active-01a {
        device    /dev/drbd0;
        disk      /dev/md2;
        address   10.206.4.151:7788;
        flexible-meta-disk /dev/loop0;
    }
Here’s the description of DRDB meta-data.

https://www.drbd.org/en/doc/users-guide ... s-metadata

Below is a snippet from the kernel logs showing the snapshot failure, followed by the first of a number of kernel panic Oops. I can provide the full logs until the machine restart if needed.

Code: Select all

Feb  9 13:52:54 drdb-active-01 kernel: [15976539.076181] veeamsnap: module verification failed: signature and/or  required key missing - tainting kernel
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090191]     veeamsnap:veeamsnap_main.c:veeamsnap_init Loading
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090194]     veeamsnap:veeamsnap_main.c:veeamsnap_init Version: 1.0.0.944
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090197]     veeamsnap:veeamsnap_main.c:veeamsnap_init Author: Veeam Software AG
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090199]     veeamsnap:veeamsnap_main.c:veeamsnap_init licence: GPL
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090201]     veeamsnap:veeamsnap_main.c:veeamsnap_init description: Veeam Snapshot Kernel Module
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090203]     veeamsnap:veeamsnap_main.c:veeamsnap_init zerosnapdata: 1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090205]     veeamsnap:veeamsnap_main.c:veeamsnap_init debuglogging: 0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090207]     veeamsnap:veeamsnap_main.c:veeamsnap_init start. container_alloc_counter=0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090209]     veeamsnap:veeamsnap_main.c:veeamsnap_init start. mem_cnt=0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090211]     veeamsnap:veeamsnap_main.c:veeamsnap_init start. vmem_cnt=0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090213]     veeamsnap:ctrl_pipe.c:ctrl_pipe_init .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090219]     veeamsnap:veeamsnap_main.c:veeamsnap_init Module major=248
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090240] bio: create slab <bio-1> at 1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090388]     veeamsnap:blk_dev_utile.c:blk_bioset_create Specific bio set created.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090396] bio: create slab <bio-2> at 2
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090502]     veeamsnap:dio_request.c:dio_bioset_create Specific bio set created.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090512]     veeamsnap:snapimage.c:snapimage_init .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090514]     veeamsnap:snapimage.c:snapimage_init Snapimage block device was registered. major=251
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090517]     veeamsnap:veeamsnap_main.c:veeamsnap_init end. container_alloc_counter=0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090519]     veeamsnap:veeamsnap_main.c:veeamsnap_init start. container_sl_alloc_counter=0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090521]     veeamsnap:veeamsnap_main.c:veeamsnap_init end. mem_cnt=1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.090523]     veeamsnap:veeamsnap_main.c:veeamsnap_init end. vmem_cnt=0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.097269]     veeamsnap:ctrl_fops.c:ctrl_open file=0xffff880bfbc3be00
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.097284]     veeamsnap:ctrl_pipe.c:ctrl_pipe_new .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.097700]     veeamsnap:ctrl_fops.c:ioctl_tracking_collect Collecting tracking device:
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.097719]     veeamsnap:tracking.c:tracking_collect Have not device under CBT.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.106988]     veeamsnap:tracking.c:tracking_add Adding. dev_id=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.106992]     veeamsnap:tracker.c:tracker_Create .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.107005]     veeamsnap:tracker.c:tracker_Create dev_id 8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.107008]     veeamsnap:tracker.c:tracker_Create bd_super=0xffff8805fc2e2000
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.107009]     veeamsnap:tracker.c:tracker_Create SectorStart    =0x800
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.107011]     veeamsnap:tracker.c:tracker_Create SectorsCapacity=0xba43000
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.107013]     veeamsnap:cbt_map.c:cbt_map_create CBT map create.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251231]     veeamsnap:tracker.c:tracker_Create freezed
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251246]     veeamsnap:tracker_queue.c:tracker_queue_Ref Creating tracker_queue for queue=0xffff8805fbac8000
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251254]     veeamsnap:tracker_queue.c:tracker_queue_Ref Tracker queue created. pTrQ=0xffff8805fcb89900
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251382]     veeamsnap:tracker.c:tracker_Create thawed.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251565]     veeamsnap:tracking.c:tracking_add Adding. dev_id=8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251571]     veeamsnap:tracker.c:tracker_Create .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251595]     veeamsnap:tracker.c:tracker_Create dev_id 8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251614]     veeamsnap:tracker.c:tracker_Create bd_super=0x          (null)
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251619]     veeamsnap:tracker.c:tracker_Create SectorStart    =0xd634800
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251625]     veeamsnap:tracker.c:tracker_Create SectorsCapacity=0x960000
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251630]     veeamsnap:cbt_map.c:cbt_map_create CBT map create.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251682]     veeamsnap:tracker.c:tracker_Create Device havn`t super block. It`s cannot be freeze.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.251688]     veeamsnap:tracker_queue.c:tracker_queue_Ref Queue already exist! queue=0x0xffff8805fbac8000
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576530]     veeamsnap:ctrl_pipe.c:ctrl_pipe_write length=56
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576545]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate unique_id=170092e8-7ba8-448d-9dfee0ec293f5c46
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576552]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate stretch_empty_limit=536870912
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576558]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate snapshotdata_dev_id=147:0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576563]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate dev_id_list_length=2
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576568]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate dev_id=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576573]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate dev_id=8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576582]     veeamsnap:snapshotdata.c:snapshotdata_stretch_create id=170092e8-7ba8-448d-9dfee0ec293f5c46
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576587]     veeamsnap:snapshotdata.c:snapshotdata_stretch_create dev_id=147:0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576602]     veeamsnap:snapshotdata.c:__snapshotdata_create_blk_info blocks count =0x0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576608]     veeamsnap:snapshotdata.c:snapshotdata_stretch_create New snapshot stretch added to container.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576613]     veeamsnap:snapshotdata.c:snapshotdata_stretch_add_dev device=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576640]     veeamsnap:snapshotdata.c:__snapshotdata_check_io_compatibility Compatible snapshot data device and 8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576720]     veeamsnap:snapshotdata.c:__snapshotdata_create_common Zero ranges bitmap create. sectors count=0xba43000
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576726]     veeamsnap:snapshotdata.c:snapshotdata_stretch_add_dev Haven`t zeroing ranges.8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576731]     veeamsnap:snapshotdata.c:snapshotdata_stretch_add_dev device=8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576745]     veeamsnap:snapshotdata.c:__snapshotdata_check_io_compatibility Compatible snapshot data device and 8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576810]     veeamsnap:snapshotdata.c:__snapshotdata_create_common Zero ranges bitmap create. sectors count=0x960000
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576815]     veeamsnap:snapshotdata.c:snapshotdata_stretch_add_dev Haven`t zeroing ranges.8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576820]     veeamsnap:ctrl_pipe.c:ctrl_pipe_request_acknowledge .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576825]     veeamsnap:ctrl_pipe.c:ctrl_pipe_push_request .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576838]     veeamsnap:ctrl_pipe.c:ctrl_pipe_poll POLLIN
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.576844]     veeamsnap:ctrl_pipe.c:ctrl_pipe_read .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.577001]     veeamsnap:ctrl_pipe.c:ctrl_pipe_write length=40
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.577007]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_next_portion .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.577015]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_next_portion unique_id=170092e8-7ba8-448d-9dfee0ec293f5c46
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.577031]     veeamsnap:snapshotdata.c:snapshotdata_FindByDevId Snapshot data for device not found. dev_id=147:0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.577036]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_next_portion Cannot set zero sectors for device=147:0
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.578450]     veeamsnap:snapshot.c:snapshot_Create device=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.578457]     veeamsnap:snapshot.c:snapshot_Create device=8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.578465]     veeamsnap:snapshot.c:_snapshot_add_tracker Adding. dev_id=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.578471]     veeamsnap:snapshot.c:_snapshot_add_tracker Device already under change tracking. Device=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.578477]     veeamsnap:snapshot.c:_snapshot_add_data Snapshot data exist for device=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.578482]     veeamsnap:snapshot.c:_snapshot_add_tracker Adding. dev_id=8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.578487]     veeamsnap:snapshot.c:_snapshot_add_tracker Device already under change tracking. Device=8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.578492]     veeamsnap:snapshot.c:_snapshot_add_data Snapshot data exist for device=8:3
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.578498]     veeamsnap:tracker.c:_tracker_freeze dev_id=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.652218]     veeamsnap:defer_io.c:defer_io_create .
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.652232]     veeamsnap:snapshotdata.c:snapshotdata_FindByDevId Snapshot data for device successfully found. dev_id=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.652237]     veeamsnap:defer_io.c:defer_io_create Snapshot data using for device=8:1
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.652392]     veeamsnap:defer_io.c:defer_io_create complete success.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.652403]     veeamsnap:defer_io.c:defer_io_work_thread started.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.652411]     veeamsnap:cbt_map.c:cbt_map_switch CBT map switch.
Feb  9 13:52:54 drdb-active-01 kernel: [15976539.652744]     veeamsnap:tracker.c:_tracker_freeze Snapshot created. New snap number=2
Feb  9 13:53:04 drdb-active-01 kernel: [15976549.645506] WRN veeamsnap:dio_request.h:dio_request_wait differed IO request timeout
Feb  9 13:53:14 drdb-active-01 kernel: [15976559.640894] WRN veeamsnap:dio_request.h:dio_request_wait differed IO request timeout
Feb  9 13:53:24 drdb-active-01 kernel: [15976569.636237] WRN veeamsnap:dio_request.h:dio_request_wait differed IO request timeout
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.631680] WRN veeamsnap:dio_request.h:dio_request_wait differed IO request timeout
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.631693] ERR veeamsnap:dio_request.h:dio_request_wait sect_processed=0x0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.631699] ERR veeamsnap:dio_request.h:dio_request_wait sect_len=0x8
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.631705] ERR veeamsnap:snapshotdata.c:snapshotdata_SetCorrupted Now snapshot corrupted.
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789136]     veeamsnap:blk_dev_utile.c:blk_thaw_bdev thaw_bdev for device=8:1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789151]     veeamsnap:tracker.c:_tracker_freeze dev_id=8:3
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789158]     veeamsnap:blk_dev_utile.c:blk_freeze_bdev Device havn`t super block. It`s cannot be freeze.
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789163]     veeamsnap:defer_io.c:defer_io_create .
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789173]     veeamsnap:snapshotdata.c:snapshotdata_FindByDevId Snapshot data for device successfully found. dev_id=8:3
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789178]     veeamsnap:defer_io.c:defer_io_create Snapshot data using for device=8:3
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789242] ERR veeamsnap:snapshotdata.c:snapshotdata_IsCorrupted Snapshot is corrupted for device 8:1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789352]     veeamsnap:defer_io.c:defer_io_create complete success.
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789359]     veeamsnap:defer_io.c:defer_io_work_thread started.
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789366]     veeamsnap:cbt_map.c:cbt_map_switch CBT map switch.
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789391]     veeamsnap:tracker.c:_tracker_freeze Snapshot created. New snap number=2
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789407]     veeamsnap:snapshot.c:snapshot_Create snapshot_id=0xffff8805fcb899c0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789414]     veeamsnap:snapimage.c:snapimage_create original_dev=8:1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789447]     veeamsnap:snapimage.c:snapimage_create image_dev=251:0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789605]     veeamsnap:snapimage.c:snapimage_create physical_block_size=512
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789611]     veeamsnap:snapimage.c:snapimage_create logical_block_size=512
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789663]     veeamsnap:snapimage.c:snapimage_create device name=veeamimage0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789668]     veeamsnap:snapimage.c:snapimage_create capacity=99998498816
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.789811]     veeamsnap:snapimage.c:snapimage_processor_thread started.
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.790092]     veeamsnap:snapimage.c:snapimage_create original_dev=8:3
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.790133]     veeamsnap:snapimage.c:snapimage_create image_dev=251:1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.790274]     veeamsnap:snapimage.c:snapimage_create physical_block_size=512
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.790280]     veeamsnap:snapimage.c:snapimage_create logical_block_size=512
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.790311]     veeamsnap:snapimage.c:snapimage_create device name=veeamimage1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.790316]     veeamsnap:snapimage.c:snapimage_create capacity=5033164800
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.790454]     veeamsnap:snapimage.c:snapimage_processor_thread started.
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.791049]     veeamsnap:ctrl_fops.c:ioctl_tracking_collect Collecting tracking device:
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.791073]     veeamsnap:tracking.c:tracking_collect dev_id=8:1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.791077]     veeamsnap:tracking.c:tracking_collect snap_number=1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.791082]     veeamsnap:tracking.c:tracking_collect cbt_map_size=381464
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.791087]     veeamsnap:tracking.c:tracking_collect dev_id=8:3
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.791092]     veeamsnap:tracking.c:tracking_collect snap_number=1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.791096]     veeamsnap:tracking.c:tracking_collect cbt_map_size=19200
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.792433]     veeamsnap:snapimage.c:__snapimage_open dev=251:0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.792530]     veeamsnap:snapimage.c:__snapimage_ioctl dev=251:0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.792536]     veeamsnap:snapimage.c:__snapimage_ioctl cmd=21297
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.792541]     veeamsnap:snapimage.c:__snapimage_ioctl arg=0x0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.792590] quiet_error: 36 callbacks suppressed
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.792596] Buffer I/O error on device veeamimage0, logical block 24413680
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.792940]     veeamsnap:snapimage.c:__snapimage_open dev=251:1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.793036]     veeamsnap:snapimage.c:__snapimage_ioctl dev=251:1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.793039]     veeamsnap:snapimage.c:__snapimage_ioctl cmd=21297
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.793042]     veeamsnap:snapimage.c:__snapimage_ioctl arg=0x0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.794741] Buffer I/O error on device veeamimage0, logical block 24413680
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.796884] Buffer I/O error on device veeamimage0, logical block 24413694
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.798947] Buffer I/O error on device veeamimage0, logical block 24413694
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.800949] Buffer I/O error on device veeamimage0, logical block 0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.802811] Buffer I/O error on device veeamimage0, logical block 0
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.804732] Buffer I/O error on device veeamimage0, logical block 1
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.805460] Buffer I/O error on device veeamimage0, logical block 24413695
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.806170] Buffer I/O error on device veeamimage0, logical block 24413695
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.806809] Buffer I/O error on device veeamimage0, logical block 24413695
Feb  9 13:53:34 drdb-active-01 kernel: [15976579.807756]     veeamsnap:snapimage.c:__snapimage_release dev id=251:0
Feb  9 13:53:35 drdb-active-01 kernel: [15976579.856470]     veeamsnap:snapimage.c:__snapimage_release dev id=251:1
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.904538]     veeamsnap:snapimage.c:__snapimage_open dev=251:0
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.904549]     veeamsnap:snapimage.c:__snapimage_ioctl dev=251:0
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.904551]     veeamsnap:snapimage.c:__snapimage_ioctl cmd=21297
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.904553]     veeamsnap:snapimage.c:__snapimage_ioctl arg=0x0
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.904852]     veeamsnap:snapimage.c:__snapimage_release dev id=251:0
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.904860]     veeamsnap:snapimage.c:__snapimage_open dev=251:1
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.904864]     veeamsnap:snapimage.c:__snapimage_ioctl dev=251:1
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.904866]     veeamsnap:snapimage.c:__snapimage_ioctl cmd=21297
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.904868]     veeamsnap:snapimage.c:__snapimage_ioctl arg=0x0
Feb  9 13:53:39 drdb-active-01 kernel: [15976583.916019]     veeamsnap:snapimage.c:__snapimage_release dev id=251:1
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786332]     veeamsnap:ctrl_fops.c:ctrl_open file=0xffff8805fcd59100
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786345]     veeamsnap:ctrl_pipe.c:ctrl_pipe_new .
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786365]     veeamsnap:snapimage.c:snapimage_collect_images count=0
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786371]     veeamsnap:snapimage.c:snapimage_collect_images real_count=2
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786429]     veeamsnap:snapimage.c:snapimage_collect_images count=2
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786434]     veeamsnap:snapimage.c:snapimage_collect_images real_count=2
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786677]     veeamsnap:ctrl_fops.c:ctrl_release file=0xffff8805fcd59100
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786684]     veeamsnap:ctrl_pipe.c:ctrl_pipe_release_cb .
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786792]     veeamsnap:snapimage.c:__snapimage_open dev=251:0
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786844]     veeamsnap:snapimage.c:__snapimage_ioctl dev=251:0
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786849]     veeamsnap:snapimage.c:__snapimage_ioctl cmd=21297
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.786854]     veeamsnap:snapimage.c:__snapimage_ioctl arg=0x0
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.787688]     veeamsnap:snapimage.c:__snapimage_release dev id=251:0
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861464]     veeamsnap:ctrl_fops.c:ctrl_open file=0xffff880144220000
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861472]     veeamsnap:ctrl_pipe.c:ctrl_pipe_new .
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861482]     veeamsnap:snapimage.c:snapimage_collect_images count=0
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861485]     veeamsnap:snapimage.c:snapimage_collect_images real_count=2
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861517]     veeamsnap:snapimage.c:snapimage_collect_images count=2
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861520]     veeamsnap:snapimage.c:snapimage_collect_images real_count=2
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861644]     veeamsnap:ctrl_fops.c:ctrl_release file=0xffff880144220000
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861647]     veeamsnap:ctrl_pipe.c:ctrl_pipe_release_cb .
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861687]     veeamsnap:snapimage.c:__snapimage_open dev=251:0
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861719]     veeamsnap:ctrl_fops.c:ctrl_open file=0xffff880144220100
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861722]     veeamsnap:ctrl_pipe.c:ctrl_pipe_new .
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861774]     veeamsnap:snapimage.c:snapimage_collect_images count=0
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861777]     veeamsnap:snapimage.c:snapimage_collect_images real_count=2
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861805]     veeamsnap:snapimage.c:snapimage_collect_images count=2
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861808]     veeamsnap:snapimage.c:snapimage_collect_images real_count=2
Feb  9 13:53:39 drdb-active-01 kernel: [15976584.861912]     veeamsnap:snapimage.c:__snapimage_open dev=251:1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.127736] quiet_error: 415 callbacks suppressed
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.127750] Buffer I/O error on device veeamimage0, logical block 0
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.129690] Buffer I/O error on device veeamimage0, logical block 1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.130522] Buffer I/O error on device veeamimage0, logical block 2
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.131237] Buffer I/O error on device veeamimage0, logical block 3
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.131946] Buffer I/O error on device veeamimage0, logical block 4
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.132609] Buffer I/O error on device veeamimage0, logical block 5
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.133248] Buffer I/O error on device veeamimage0, logical block 6
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.133860] Buffer I/O error on device veeamimage0, logical block 7
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.134465] Buffer I/O error on device veeamimage0, logical block 8
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.135059] Buffer I/O error on device veeamimage0, logical block 9
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.135939]     veeamsnap:ctrl_fops.c:ioctl_snapshot_errno Snapshot get errno for device
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.135944]     veeamsnap:snapshotdata.c:snapshotdata_FindByDevId Snapshot data for device successfully found. dev_id=8:1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.144599]     veeamsnap:snapimage.c:__snapimage_release dev id=251:1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.144853]     veeamsnap:snapimage.c:__snapimage_release dev id=251:0
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.144935]     veeamsnap:ctrl_fops.c:ctrl_release file=0xffff880144220100
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.144943]     veeamsnap:ctrl_pipe.c:ctrl_pipe_release_cb .
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.181131]     veeamsnap:ctrl_fops.c:ioctl_snapshot_errno Snapshot get errno for device
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.181146]     veeamsnap:snapshotdata.c:snapshotdata_FindByDevId Snapshot data for device successfully found. dev_id=8:1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.183821]     veeamsnap:ctrl_fops.c:ioctl_snapshot_destroy Snapshot destroy
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.183835]     veeamsnap:snapshot.c:_snapshot_destroy snapshot_id=0xffff8805fcb899c0
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.183842]     veeamsnap:snapimage.c:snapimage_stop original_dev=8:1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.183851]     veeamsnap:snapimage.c:snapimage_stop original_dev=8:3
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.183858]     veeamsnap:tracker.c:tracker_Unfreeze .
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.183864]     veeamsnap:tracker.c:_tracker_unfreeze dev_id=8:1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.183869]     veeamsnap:tracker.c:_tracker_unfreeze Snapshot freed. Active snap number=2
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.183875]     veeamsnap:defer_io.c:defer_io_close for device 8:1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.628917]     veeamsnap:defer_io.c:defer_io_work_thread complete.
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.628992]     veeamsnap:tracker.c:_tracker_unfreeze freezing device 8:1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.721546]     veeamsnap:tracker.c:_tracker_unfreeze freezed.
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.721677]     veeamsnap:tracker.c:_tracker_unfreeze thaw_bdev for device=8:1
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.721688]     veeamsnap:tracker.c:_tracker_unfreeze dev_id=8:3
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.721694]     veeamsnap:tracker.c:_tracker_unfreeze Snapshot freed. Active snap number=2
Feb  9 13:53:40 drdb-active-01 kernel: [15976585.721700]     veeamsnap:defer_io.c:defer_io_close for device 8:3
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.132694]     veeamsnap:defer_io.c:defer_io_work_thread complete.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.132733]     veeamsnap:tracker.c:_tracker_unfreeze Device havn`t super block. It`s cannot be freeze.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.132746]     veeamsnap:snapimage.c:snapimage_destroy original_dev=8:1
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.132751]     veeamsnap:snapimage.c:__snapimage_destroy delete disk.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133264]     veeamsnap:snapimage.c:__snapimage_destroy stop request processor.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133294]     veeamsnap:snapimage.c:snapimage_processor_thread stopped.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133323]     veeamsnap:snapimage.c:__snapimage_destroy cleanup queue.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133338]     veeamsnap:snapimage.c:__snapimage_destroy release disk structure.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133400]     veeamsnap:snapimage.c:__snapimage_free image owning counter =  0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133406]     veeamsnap:defer_io.c:__defer_io_destroy Processed MiB: 0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133411]     veeamsnap:defer_io.c:__defer_io_destroy Copied MiB: 0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133417]     veeamsnap:defer_io.c:__defer_io_destroy complete.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133425]     veeamsnap:snapimage.c:snapimage_destroy original_dev=8:3
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133429]     veeamsnap:snapimage.c:__snapimage_destroy delete disk.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133818]     veeamsnap:snapimage.c:__snapimage_destroy stop request processor.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133839]     veeamsnap:snapimage.c:snapimage_processor_thread stopped.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133861]     veeamsnap:snapimage.c:__snapimage_destroy cleanup queue.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133870]     veeamsnap:snapimage.c:__snapimage_destroy release disk structure.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133916]     veeamsnap:snapimage.c:__snapimage_free image owning counter =  0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133921]     veeamsnap:defer_io.c:__defer_io_destroy Processed MiB: 0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133926]     veeamsnap:defer_io.c:__defer_io_destroy Copied MiB: 0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133930]     veeamsnap:defer_io.c:__defer_io_destroy complete.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133938]     veeamsnap:snapshot.c:_snapshot_remove_device Removing. dev_id=8:1
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133945]     veeamsnap:snapshotdata.c:snapshotdata_DeleteByDevId Removing snapshot data for device=8:1
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.133952]     veeamsnap:snapshotdata.c:__snapshotdata_destroy Snapshot data filled MiB 0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134051]     veeamsnap:rangevector.c:rangevector_destroy blocks_cnt=0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134057]     veeamsnap:snapshotdata.c:__snapshotdata_destroy Zero sectors bitmap destroy.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134063]     veeamsnap:snapshot.c:_snapshot_remove_device Removing. dev_id=8:3
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134068]     veeamsnap:snapshotdata.c:snapshotdata_DeleteByDevId Removing snapshot data for device=8:3
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134073]     veeamsnap:snapshotdata.c:__snapshotdata_destroy Snapshot data filled MiB 0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134084]     veeamsnap:snapshotdata.c:snapshotdata_stretch_free id=170092e8-7ba8-448d-9dfee0ec293f5c46
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134089]     veeamsnap:ctrl_pipe.c:ctrl_pipe_request_terminate .
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134096]     veeamsnap:ctrl_pipe.c:ctrl_pipe_push_request .
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134102]     veeamsnap:rangevector.c:rangevector_destroy blocks_cnt=1
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134121]     veeamsnap:rangevector.c:rangevector_destroy blocks_cnt=0
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.134125]     veeamsnap:snapshotdata.c:__snapshotdata_destroy Zero sectors bitmap destroy.
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.137437]     veeamsnap:ctrl_pipe.c:ctrl_pipe_poll POLLIN
Feb  9 13:53:41 drdb-active-01 kernel: [15976586.137456]     veeamsnap:ctrl_pipe.c:ctrl_pipe_read .
Feb  9 13:53:42 drdb-active-01 kernel: [15976587.628882]     veeamsnap:ctrl_fops.c:ctrl_release file=0xffff880bfbc3be00
Feb  9 13:53:42 drdb-active-01 kernel: [15976587.628894]     veeamsnap:ctrl_pipe.c:ctrl_pipe_release_cb .
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.754838]     veeamsnap:ctrl_fops.c:ctrl_open file=0xffff8805fb145600
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.754851]     veeamsnap:ctrl_pipe.c:ctrl_pipe_new .
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.755237]     veeamsnap:ctrl_fops.c:ioctl_tracking_collect Collecting tracking device:
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.755262]     veeamsnap:tracking.c:tracking_collect dev_id=8:1
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.755267]     veeamsnap:tracking.c:tracking_collect snap_number=1
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.755272]     veeamsnap:tracking.c:tracking_collect cbt_map_size=381464
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.755277]     veeamsnap:tracking.c:tracking_collect dev_id=8:3
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.755282]     veeamsnap:tracking.c:tracking_collect snap_number=1
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.755287]     veeamsnap:tracking.c:tracking_collect cbt_map_size=19200
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.765690]     veeamsnap:tracking.c:tracking_add Adding. dev_id=8:1
Feb  9 15:43:35 drdb-active-01 kernel: [15983177.765700]     veeamsnap:tracking.c:tracking_add Device already tracking. Device=8:1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817136]     veeamsnap:ctrl_pipe.c:ctrl_pipe_write length=48
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817151]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate unique_id=24b102b6-3476-42a5-a4c5259322199693
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817157]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate stretch_empty_limit=536870912
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817164]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate snapshotdata_dev_id=147:0
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817169]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate dev_id_list_length=1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817174]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_initiate dev_id=8:1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817184]     veeamsnap:snapshotdata.c:snapshotdata_stretch_create id=24b102b6-3476-42a5-a4c5259322199693
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817189]     veeamsnap:snapshotdata.c:snapshotdata_stretch_create dev_id=147:0
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817205]     veeamsnap:snapshotdata.c:__snapshotdata_create_blk_info blocks count =0x0
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817211]     veeamsnap:snapshotdata.c:snapshotdata_stretch_create New snapshot stretch added to container.
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817216]     veeamsnap:snapshotdata.c:snapshotdata_stretch_add_dev device=8:1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817247]     veeamsnap:snapshotdata.c:__snapshotdata_check_io_compatibility Compatible snapshot data device and 8:1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817320]     veeamsnap:snapshotdata.c:__snapshotdata_create_common Zero ranges bitmap create. sectors count=0xba43000
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817326]     veeamsnap:snapshotdata.c:snapshotdata_stretch_add_dev Haven`t zeroing ranges.8:1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817331]     veeamsnap:ctrl_pipe.c:ctrl_pipe_request_acknowledge .
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817336]     veeamsnap:ctrl_pipe.c:ctrl_pipe_push_request .
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817350]     veeamsnap:ctrl_pipe.c:ctrl_pipe_poll POLLIN
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817357]     veeamsnap:ctrl_pipe.c:ctrl_pipe_read .
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817506]     veeamsnap:ctrl_pipe.c:ctrl_pipe_write length=40
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817511]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_next_portion .
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817520]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_next_portion unique_id=24b102b6-3476-42a5-a4c5259322199693
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817536]     veeamsnap:snapshotdata.c:snapshotdata_FindByDevId Snapshot data for device not found. dev_id=147:0
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.817542]     veeamsnap:ctrl_pipe.c:ctrl_pipe_command_next_portion Cannot set zero sectors for device=147:0
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.819014]     veeamsnap:snapshot.c:snapshot_Create device=8:1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.819026]     veeamsnap:snapshot.c:_snapshot_add_tracker Adding. dev_id=8:1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.819033]     veeamsnap:snapshot.c:_snapshot_add_tracker Device already under change tracking. Device=8:1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.819040]     veeamsnap:snapshot.c:_snapshot_add_data Snapshot data exist for device=8:1
Feb  9 15:43:36 drdb-active-01 kernel: [15983178.819046]     veeamsnap:tracker.c:_tracker_freeze dev_id=8:1
Feb  9 15:43:37 drdb-active-01 kernel: [15983178.892559]     veeamsnap:defer_io.c:defer_io_create .
Feb  9 15:43:37 drdb-active-01 kernel: [15983178.892574]     veeamsnap:snapshotdata.c:snapshotdata_FindByDevId Snapshot data for device successfully found. dev_id=8:1
Feb  9 15:43:37 drdb-active-01 kernel: [15983178.892580]     veeamsnap:defer_io.c:defer_io_create Snapshot data using for device=8:1
Feb  9 15:43:37 drdb-active-01 kernel: [15983178.892747]     veeamsnap:defer_io.c:defer_io_create complete success.
Feb  9 15:43:37 drdb-active-01 kernel: [15983178.892757]     veeamsnap:cbt_map.c:cbt_map_switch CBT map switch.
Feb  9 15:43:37 drdb-active-01 kernel: [15983178.892776]     veeamsnap:defer_io.c:defer_io_work_thread started.
Feb  9 15:43:37 drdb-active-01 kernel: [15983178.893096]     veeamsnap:tracker.c:_tracker_freeze Snapshot created. New snap number=3
Feb  9 15:43:47 drdb-active-01 kernel: [15983188.885285] WRN veeamsnap:dio_request.h:dio_request_wait differed IO request timeout
Feb  9 15:43:57 drdb-active-01 kernel: [15983198.880625] WRN veeamsnap:dio_request.h:dio_request_wait differed IO request timeout
Feb  9 15:44:07 drdb-active-01 kernel: [15983208.876054] WRN veeamsnap:dio_request.h:dio_request_wait differed IO request timeout
Feb  9 15:44:17 drdb-active-01 kernel: [15983218.871457] WRN veeamsnap:dio_request.h:dio_request_wait differed IO request timeout
Feb  9 15:44:17 drdb-active-01 kernel: [15983218.871469] ERR veeamsnap:dio_request.h:dio_request_wait sect_processed=0x0
Feb  9 15:44:17 drdb-active-01 kernel: [15983218.871475] ERR veeamsnap:dio_request.h:dio_request_wait sect_len=0x8
Feb  9 15:44:17 drdb-active-01 kernel: [15983218.871482] ERR veeamsnap:snapshotdata.c:snapshotdata_SetCorrupted Now snapshot corrupted.
Feb  9 15:44:17 drdb-active-01 kernel: [15983218.871542] ERR veeamsnap:snapshotdata.c:snapshotdata_IsCorrupted Snapshot is corrupted for device 8:1
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.028601]     veeamsnap:blk_dev_utile.c:blk_thaw_bdev thaw_bdev for device=8:1
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.028615]     veeamsnap:snapshot.c:snapshot_Create snapshot_id=0xffff880bfbc0e500
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.028622]     veeamsnap:snapimage.c:snapimage_create original_dev=8:1
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.028663]     veeamsnap:snapimage.c:snapimage_create image_dev=251:0
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.028816]     veeamsnap:snapimage.c:snapimage_create physical_block_size=512
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.028822]     veeamsnap:snapimage.c:snapimage_create logical_block_size=512
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.028854]     veeamsnap:snapimage.c:snapimage_create device name=veeamimage0
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.028860]     veeamsnap:snapimage.c:snapimage_create capacity=99998498816
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.029032]     veeamsnap:snapimage.c:snapimage_processor_thread started.
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.029736]     veeamsnap:ctrl_fops.c:ioctl_tracking_collect Collecting tracking device:
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.029761]     veeamsnap:tracking.c:tracking_collect dev_id=8:1
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.029766]     veeamsnap:tracking.c:tracking_collect snap_number=2
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.029771]     veeamsnap:tracking.c:tracking_collect cbt_map_size=381464
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.029775]     veeamsnap:tracking.c:tracking_collect dev_id=8:3
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.029780]     veeamsnap:tracking.c:tracking_collect snap_number=1
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.029784]     veeamsnap:tracking.c:tracking_collect cbt_map_size=19200
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.031740]     veeamsnap:snapimage.c:__snapimage_open dev=251:0
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.031847]     veeamsnap:snapimage.c:__snapimage_ioctl dev=251:0
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.031854]     veeamsnap:snapimage.c:__snapimage_ioctl cmd=21297
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.031859]     veeamsnap:snapimage.c:__snapimage_ioctl arg=0x0
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.031908] quiet_error: 55 callbacks suppressed
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.031913] Buffer I/O error on device veeamimage0, logical block 24413680
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.033458] Buffer I/O error on device veeamimage0, logical block 24413680
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.034949] Buffer I/O error on device veeamimage0, logical block 24413694
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.036444] Buffer I/O error on device veeamimage0, logical block 24413694
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.037893] Buffer I/O error on device veeamimage0, logical block 0
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.039349] Buffer I/O error on device veeamimage0, logical block 0
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.040787] Buffer I/O error on device veeamimage0, logical block 1
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.042214] Buffer I/O error on device veeamimage0, logical block 24413695
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.043907] Buffer I/O error on device veeamimage0, logical block 24413695
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.045377] Buffer I/O error on device veeamimage0, logical block 24413695
Feb  9 15:44:17 drdb-active-01 kernel: [15983219.047547]     veeamsnap:snapimage.c:__snapimage_release dev id=251:0
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.682007]     veeamsnap:snapimage.c:__snapimage_open dev=251:0
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.682018]     veeamsnap:snapimage.c:__snapimage_ioctl dev=251:0
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.682020]     veeamsnap:snapimage.c:__snapimage_ioctl cmd=21297
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.682022]     veeamsnap:snapimage.c:__snapimage_ioctl arg=0x0
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.682039] quiet_error: 136 callbacks suppressed
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.682041] Buffer I/O error on device veeamimage0, logical block 24413680
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.682551] Buffer I/O error on device veeamimage0, logical block 24413680
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.682985] Buffer I/O error on device veeamimage0, logical block 24413694
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.683406] Buffer I/O error on device veeamimage0, logical block 24413694
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.683820] Buffer I/O error on device veeamimage0, logical block 0
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.684230] Buffer I/O error on device veeamimage0, logical block 0
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.684632] Buffer I/O error on device veeamimage0, logical block 1
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.685055] Buffer I/O error on device veeamimage0, logical block 24413695
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.685478] Buffer I/O error on device veeamimage0, logical block 24413695
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.685880] Buffer I/O error on device veeamimage0, logical block 24413695
Feb  9 15:44:22 drdb-active-01 kernel: [15983224.686512]     veeamsnap:snapimage.c:__snapimage_release dev id=251:0
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129000]     veeamsnap:ctrl_fops.c:ctrl_open file=0xffff880106e4cc00
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129014]     veeamsnap:ctrl_pipe.c:ctrl_pipe_new .
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129034]     veeamsnap:snapimage.c:snapimage_collect_images count=0
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129040]     veeamsnap:snapimage.c:snapimage_collect_images real_count=1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129098]     veeamsnap:snapimage.c:snapimage_collect_images count=1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129103]     veeamsnap:snapimage.c:snapimage_collect_images real_count=1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129280]     veeamsnap:ctrl_fops.c:ctrl_release file=0xffff880106e4cc00
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129286]     veeamsnap:ctrl_pipe.c:ctrl_pipe_release_cb .
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129398]     veeamsnap:snapimage.c:__snapimage_open dev=251:0
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129452]     veeamsnap:snapimage.c:__snapimage_ioctl dev=251:0
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129458]     veeamsnap:snapimage.c:__snapimage_ioctl cmd=21297
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.129462]     veeamsnap:snapimage.c:__snapimage_ioctl arg=0x0
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.130293]     veeamsnap:snapimage.c:__snapimage_release dev id=251:0
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199729]     veeamsnap:ctrl_fops.c:ctrl_open file=0xffff880bfcce8600
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199737]     veeamsnap:ctrl_pipe.c:ctrl_pipe_new .
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199748]     veeamsnap:snapimage.c:snapimage_collect_images count=0
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199751]     veeamsnap:snapimage.c:snapimage_collect_images real_count=1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199783]     veeamsnap:snapimage.c:snapimage_collect_images count=1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199786]     veeamsnap:snapimage.c:snapimage_collect_images real_count=1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199880]     veeamsnap:ctrl_fops.c:ctrl_release file=0xffff880bfcce8600
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199883]     veeamsnap:ctrl_pipe.c:ctrl_pipe_release_cb .
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199931]     veeamsnap:snapimage.c:__snapimage_open dev=251:0
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199964]     veeamsnap:ctrl_fops.c:ctrl_open file=0xffff880bfcce8d00
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.199967]     veeamsnap:ctrl_pipe.c:ctrl_pipe_new .
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.560225]     veeamsnap:ctrl_fops.c:ioctl_snapshot_errno Snapshot get errno for device
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.560242]     veeamsnap:snapshotdata.c:snapshotdata_FindByDevId Snapshot data for device successfully found. dev_id=8:1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.572113]     veeamsnap:snapimage.c:__snapimage_release dev id=251:0
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.572206]     veeamsnap:ctrl_fops.c:ctrl_release file=0xffff880bfcce8d00
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.572213]     veeamsnap:ctrl_pipe.c:ctrl_pipe_release_cb .
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.612776]     veeamsnap:ctrl_fops.c:ioctl_snapshot_errno Snapshot get errno for device
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.612790]     veeamsnap:snapshotdata.c:snapshotdata_FindByDevId Snapshot data for device successfully found. dev_id=8:1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.615325]     veeamsnap:ctrl_fops.c:ioctl_snapshot_destroy Snapshot destroy
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.615338]     veeamsnap:snapshot.c:_snapshot_destroy snapshot_id=0xffff880bfbc0e500
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.615345]     veeamsnap:snapimage.c:snapimage_stop original_dev=8:1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.615355]     veeamsnap:tracker.c:tracker_Unfreeze .
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.615362]     veeamsnap:tracker.c:_tracker_unfreeze dev_id=8:1
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.615367]     veeamsnap:tracker.c:_tracker_unfreeze Snapshot freed. Active snap number=3
Feb  9 15:44:23 drdb-active-01 kernel: [15983225.615372]     veeamsnap:defer_io.c:defer_io_close for device 8:1
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.868239]     veeamsnap:defer_io.c:defer_io_work_thread complete.
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.868302]     veeamsnap:tracker.c:_tracker_unfreeze freezing device 8:1
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.977567]     veeamsnap:tracker.c:_tracker_unfreeze freezed.
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.977702]     veeamsnap:tracker.c:_tracker_unfreeze thaw_bdev for device=8:1
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.977712]     veeamsnap:snapimage.c:snapimage_destroy original_dev=8:1
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.977718]     veeamsnap:snapimage.c:__snapimage_destroy delete disk.
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978297]     veeamsnap:snapimage.c:__snapimage_destroy stop request processor.
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978332]     veeamsnap:snapimage.c:snapimage_processor_thread stopped.
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978363]     veeamsnap:snapimage.c:__snapimage_destroy cleanup queue.
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978381]     veeamsnap:snapimage.c:__snapimage_destroy release disk structure.
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978454]     veeamsnap:snapimage.c:__snapimage_free image owning counter =  0
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978461]     veeamsnap:defer_io.c:__defer_io_destroy Processed MiB: 0
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978465]     veeamsnap:defer_io.c:__defer_io_destroy Copied MiB: 0
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978471]     veeamsnap:defer_io.c:__defer_io_destroy complete.
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978481]     veeamsnap:snapshot.c:_snapshot_remove_device Removing. dev_id=8:1
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978488]     veeamsnap:snapshotdata.c:snapshotdata_DeleteByDevId Removing snapshot data for device=8:1
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978494]     veeamsnap:snapshotdata.c:__snapshotdata_destroy Snapshot data filled MiB 0
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978505]     veeamsnap:snapshotdata.c:snapshotdata_stretch_free id=24b102b6-3476-42a5-a4c5259322199693
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978511]     veeamsnap:ctrl_pipe.c:ctrl_pipe_request_terminate .
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978517]     veeamsnap:ctrl_pipe.c:ctrl_pipe_push_request .
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978525]     veeamsnap:rangevector.c:rangevector_destroy blocks_cnt=1
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978626]     veeamsnap:rangevector.c:rangevector_destroy blocks_cnt=0
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.978631]     veeamsnap:snapshotdata.c:__snapshotdata_destroy Zero sectors bitmap destroy.
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.980168]     veeamsnap:ctrl_pipe.c:ctrl_pipe_poll POLLIN
Feb  9 15:44:24 drdb-active-01 kernel: [15983225.980186]     veeamsnap:ctrl_pipe.c:ctrl_pipe_read .
Feb  9 15:44:25 drdb-active-01 kernel: [15983227.568255]     veeamsnap:ctrl_fops.c:ctrl_release file=0xffff8805fb145600
Feb  9 15:44:25 drdb-active-01 kernel: [15983227.568268]     veeamsnap:ctrl_pipe.c:ctrl_pipe_release_cb .
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.898522] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.899057] IP: [<ffffffff811a6285>] kmem_cache_alloc+0x75/0x1f0
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.899518] PGD 6246d4067 PUD bfbdff067 PMD 0
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.899972] Oops: 0000 [#1] SMP
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.900427] Modules linked in: veeamsnap(OX) drbd xfs ses enclosure lru_cache libcrc32c 8021q garp stp mrp llc nfsd auth_rpcgss nfs_acl nfs lockd ast sunrpc ttm kvm_amd drm_kms_helper fscache drm kvm joydev i2c_algo_bit syscopyarea amd64_edac_mod sysfillrect serio_raw edac_core sysimgblt edac_mce_amd k10temp i2c_piix4 ipmi_si mac_hid shpchp lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 ixgbe hid_generic pata_acpi raid0 e1000e mpt2sas usbhid dca multipath ptp raid_class psmouse ahci hid linear pata_atiixp libahci pps_core scsi_transport_sas mdio [last unloaded: drbd]
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.903620] CPU: 7 PID: 28665 Comm: ps Tainted: G           OX 3.13.0-92-generic #139-Ubuntu
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.904358] Hardware name: System manufacturer System Product Name/KCMA-D8, BIOS 3102    01/31/2013
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.905140] task: ffff880621196000 ti: ffff880b2d16c000 task.ti: ffff880b2d16c000
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.905946] RIP: 0010:[<ffffffff811a6285>]  [<ffffffff811a6285>] kmem_cache_alloc+0x75/0x1f0
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.906799] RSP: 0018:ffff880b2d16dd50  EFLAGS: 00010286
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.907664] RAX: 0000000000000000 RBX: 00007ffd73243100 RCX: 0000000019dd47f0
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.908565] RDX: 0000000019dd47ef RSI: 00000000000000d0 RDI: ffff880607803300
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.909487] RBP: ffff880b2d16dd80 R08: 0000000000016020 R09: ffffffff811d153f
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.910428] R10: 00007ffd73242ea0 R11: 0000000000000246 R12: 0000000000000001
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.911384] R13: 00000000000000d0 R14: ffff880607803300 R15: ffff880607803300
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.912366] FS:  00007f91cc35a740(0000) GS:ffff880c1fc40000(0000) knlGS:0000000000000000
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.913382] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.914416] CR2: 0000000000000001 CR3: 0000000ab3d14000 CR4: 00000000000007e0
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.915485] Stack:
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.916555]  ffffffff811d153f 00007ffd73243100 00007f91cbf2e65c 0000000000000000
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.917696]  0000000000000001 ffff880b2d16def8 ffff880b2d16ddc0 ffffffff811d153f
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.918867]  0000000000df8780 00007ffd73243100 0000000000000001 ffff880b2d16deb0
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.920063] Call Trace:
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.921264]  [<ffffffff811d153f>] ? getname_flags+0x4f/0x190
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.922493]  [<ffffffff811d153f>] getname_flags+0x4f/0x190
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.923734]  [<ffffffff811d21f2>] user_path_at_empty+0x32/0x90
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.924994]  [<ffffffff811e49e7>] ? seq_buf_alloc+0x17/0x40
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.926267]  [<ffffffff811d2261>] user_path_at+0x11/0x20
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.927550]  [<ffffffff811c6000>] vfs_fstatat+0x50/0xa0
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.928844]  [<ffffffff811c649f>] SYSC_newstat+0x1f/0x40
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.930153]  [<ffffffff811c114e>] ? vfs_read+0xee/0x160
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.931470]  [<ffffffff811c1c09>] ? SyS_read+0x49/0xa0
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.932792]  [<ffffffff811c66ee>] SyS_newstat+0xe/0x10
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.934131]  [<ffffffff8173a9dd>] system_call_fastpath+0x1a/0x1f
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.935482] Code: cd 00 00 49 8b 50 08 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 2f 01 00 00 48 85 c0 0f 84 26 01 00 00 49 63 46 20 48 8d 4a 01 4d 8b 06 <49> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.938477] RIP  [<ffffffff811a6285>] kmem_cache_alloc+0x75/0x1f0
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.939978]  RSP <ffff880b2d16dd50>
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.941481] CR2: 0000000000000001
Feb  9 15:48:41 drdb-active-01 kernel: [15983482.963209] ---[ end trace 9e8db092a89ac450 ]---
Feb  9 15:49:41 drdb-active-01 kernel: [15983542.723498] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
Feb  9 15:49:41 drdb-active-01 kernel: [15983542.728174] IP: [<ffffffff811a6285>] kmem_cache_alloc+0x75/0x1f0
Feb  9 15:49:41 drdb-active-01 kernel: [15983542.733063] PGD bfc470067 PUD 62735c067 PMD 0
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Failed to perform backup Snapshot deadlock detected

Post by nielsengelen »

I would advise to contact support in regards to this. DRBD is very sensitive and it might be that due to our snapshot being created it couldn't access the metadata and went into a failover state. Where the source machine still thought it was the main controller and decided to kernel panic in the end. Please post the support case number also for future references.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
FvdAa
Influencer
Posts: 10
Liked: never
Joined: Mar 13, 2017 9:11 am
Full Name: Frank van der Aa
Contact:

[MERGED] DRBD support

Post by FvdAa »

Running CentOS, but when configuring a DRBD resource, the backup fails every time.

Have been in contact with support, but DRBD isn't supported at this moment. Are there any plans to support DRBD resources?
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Failed to perform backup Snapshot deadlock detected

Post by PTide »

Hi,

We don't have plans for DRBD support yet, however this may change if we gather a significant amount of requests. Would you please describe the application of DRBD in your particular case?

Thank you
FvdAa
Influencer
Posts: 10
Liked: never
Joined: Mar 13, 2017 9:11 am
Full Name: Frank van der Aa
Contact:

Re: Failed to perform backup Snapshot deadlock detected

Post by FvdAa »

Hi,

we only use DRBD in active/passive setup with LVM underneath it. In those cases we have two servers; one running primary, the other secundairy. In most cases all DRBD resources are primary on one server, but we do have some cases in which we have two servers that have primary/secondary and secondary/primary resources.
Post Reply

Who is online

Users browsing this forum: Semrush [Bot] and 14 guests