Job failed only with message - [error] Job failed unexpectedly

[stevenfoo]

We have our job failed recently with message - [error] Job failed unexpectedly.

There is no other message. Any idea what is going on?

Code: Select all

  Summary                              Data

     Duration:           00:24:09         Processed:          58.5 GB (100%)
     Processing rate:    66.3 MB/s        Read:               58.5 GB
     Bottleneck:         Source           Transferred:        13.9 GB (4.2x)

      Time            Action                                                                                                                            Duration

      09:53:12        Job TESTINGDB started at 2018-12-13 09:53:12 MYT
      09:53:13        Preparing to backup
      09:53:18        Creating volume snapshot                                                                                                          00:00:26
      09:53:46        Starting incremental backup to Repository_2
      09:53:46        Backing up BIOS bootloader on /dev/sda                                                                                            00:00:01
      09:53:56        Backed up sda 58.5 GB at 66.3 MB/s                                                                                                00:15:04
      10:09:00        Backing up summary.xml                                                                                                            00:00:00
      10:09:07        Releasing snapshot                                                                                                                00:00:00
      10:17:21        [error] Job failed unexpectedly

[stevenfoo]

We have this server running kernel 2.6.32-754.6.3.el6.centos.plus.x86_64.

The other server running 2.6.32-754.3.5.el6.centos.plus.x86_64 do not have any issue.

Both server running the same Agent version: veeam-2.0.1.665-1.el6.x86_64 / veeamsnap-2.0.1.665-1.noarch / veeam-release-el6-1.0.5-1.x86_64

We notice after the releasing snapshot step log as below, then nothing happen for 6 mins. This time we notice the network cut off and the server sudden reboot by itself.

Code: Select all

Dec 13 11:04:42 penexsiodb kernel: ------------[ cut here ]------------
Dec 13 11:04:42 penexsiodb kernel: WARNING: at block/blk-throttle.c:1222 blk_throtl_drain+0xff/0x180() (Not tainted)
Dec 13 11:04:42 penexsiodb kernel: Hardware name: PowerEdge R730xd
Dec 13 11:04:42 penexsiodb kernel: Modules linked in: veeamsnap(U) des_generic ecb md4 ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge fuse nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs joydev autofs4 nls_utf8 cifs 8021q garp stp llc ipv6 vhost_net macvtap macvlan tun kvm_intel kvm uinput microcode iTCO_wdt iTCO_vendor_support dcdbas power_meter acpi_ipmi ipmi_si ipmi_msghandler i40e sb_edac edac_core lpc_ich mfd_core shpchp sg tg3 ptp pps_core ext4 jbd2 mbcache sd_mod crc_t10dif ahci megaraid_sas wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Dec 13 11:04:42 penexsiodb kernel: Pid: 7226, comm: Lpb Server thre Not tainted 2.6.32-754.6.3.el6.centos.plus.x86_64 #1
Dec 13 11:04:42 penexsiodb kernel: Call Trace:
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff810805c1>] ? warn_slowpath_common+0x91/0xe0
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff8108062a>] ? warn_slowpath_null+0x1a/0x20
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff812a088f>] ? blk_throtl_drain+0xff/0x180
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff8128fcb1>] ? __blk_drain_queue+0x91/0x140
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff8128ff8b>] ? blk_cleanup_queue+0xeb/0x1d0
Dec 13 11:04:42 penexsiodb kernel: [<ffffffffa0555899>] ? _snapimage_destroy+0x1d9/0x3a0 [veeamsnap]
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff81562524>] ? printk+0x55/0x59
Dec 13 11:04:42 penexsiodb kernel: [<ffffffffa05580ad>] ? snapimage_destroy+0xad/0x390 [veeamsnap]
Dec 13 11:04:42 penexsiodb kernel: [<ffffffffa05600da>] ? _snapshot_destroy+0x13a/0x3c0 [veeamsnap]
Dec 13 11:04:42 penexsiodb kernel: [<ffffffffa05603e2>] ? snapshot_Destroy+0x82/0x120 [veeamsnap]
Dec 13 11:04:42 penexsiodb kernel: [<ffffffffa0564549>] ? ioctl_snapshot_destroy+0xd9/0x240 [veeamsnap]
Dec 13 11:04:42 penexsiodb kernel: [<ffffffffa0562684>] ? ctrl_unlocked_ioctl+0x54/0x60 [veeamsnap]
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff811b95e9>] ? vfs_ioctl+0x29/0xc0
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff811b97a4>] ? do_vfs_ioctl+0x84/0x590
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff811a3934>] ? vfs_write+0x134/0x1a0
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff811b9d31>] ? sys_ioctl+0x81/0xa0
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff8156e272>] ? system_call_after_swapgs+0xa2/0x152
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff8156e27e>] ? system_call_after_swapgs+0xae/0x152
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff8156e272>] ? system_call_after_swapgs+0xa2/0x152
Dec 13 11:04:42 penexsiodb kernel: [<ffffffff8156e357>] ? system_call_fastpath+0x35/0x3a

[PTide]

Hi,

There is a bug in kernel 2.6.32-754.6.3.el6, please see this KB for details.

Thanks!

[stevenfoo]

Thank you. I managed to search for it as well at the Redhat KB.

Thank you very much again. I have apply the new kernel and pending reboot and test again.

[ortoscale]

Hi, do we have kernel issues again?

[error] Job failed unexpectedly

3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[ortoscale]

? it seems to me that veeam doesnt like redhat/centos/almalinux distros ?