I have a linux repository setup like Veeam's Tom Sightler Veeam-Docker vbrrepo and it went offline last night.
It looks like the zvol has no additional space while the xfs filesystem looks like it has plenty.
My guess is that the zvol is trying to grow, but has no space and puts the xfs filesystem offline. however there's 37T of freespace on the XFS filesystem.
I just don't know how to approach this situation.
On the host:
Code: Select all
bbuchanan@veeambrs03:~$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool01 118T 5.98T 96K /pool01
pool01/veeam 118T 2.79G 96K /pool01/veeam
pool01/veeam/repo_vol 118T 2.79G 78.3T -
Code: Select all
veeam@veeambrs03:~$ df -h
Filesystem Size Used Avail Use% Mounted on
overlay 439G 35G 383G 9% /
tmpfs 64M 0 64M 0% /dev
tmpfs 94G 0 94G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/sdb2 439G 35G 383G 9% /keys
tmpfs 94G 0 94G 0% /proc/acpi
tmpfs 94G 0 94G 0% /proc/scsi
tmpfs 94G 0 94G 0% /sys/firmware
/dev/zvol/pool01/veeam/repo_vol 77T 41T 37T 53% /repo
Code: Select all
bbuchanan@veeambrs03:~$ grep XFS /var/log/syslog
Nov 26 06:01:55 veeambrs03 kernel: [766748.837523] XFS (zd0): writeback error on sector 7190797832
Nov 26 06:01:55 veeambrs03 kernel: [766748.837730] XFS (zd0): writeback error on sector 7190795728
Nov 26 06:01:55 veeambrs03 kernel: [766748.838231] XFS (zd0): writeback error on sector 4295013544
Nov 26 06:01:55 veeambrs03 kernel: [766748.843401] XFS (zd0): log I/O error -5
Nov 26 06:01:55 veeambrs03 kernel: [766748.843468] XFS (zd0): xfs_do_force_shutdown(0x2) called from line 1250 of file fs/xfs/xfs_log.c. Return address = 00000000f77661f4
Nov 26 06:01:55 veeambrs03 kernel: [766748.843469] XFS (zd0): Log I/O Error Detected. Shutting down filesystem
Nov 26 06:01:55 veeambrs03 kernel: [766748.843496] XFS (zd0): Please unmount the filesystem and rectify the problem(s)
Nov 26 06:01:55 veeambrs03 kernel: [766748.858784] XFS (zd0): writeback error on sector 121017993416
Nov 26 06:01:55 veeambrs03 kernel: [766748.858845] XFS (zd0): writeback error on sector 119418807552
Nov 26 06:01:55 veeambrs03 kernel: [766748.861382] XFS (zd0): writeback error on sector 119626955952
Nov 26 06:01:55 veeambrs03 kernel: [766748.871254] XFS (zd0): writeback error on sector 121018460360
Nov 26 06:01:55 veeambrs03 kernel: [766748.875798] XFS (zd0): writeback error on sector 119419274496
Nov 26 06:01:55 veeambrs03 kernel: [766748.882613] XFS (zd0): writeback error on sector 118905590816
Nov 26 06:01:55 veeambrs03 kernel: [766748.887775] XFS (zd0): writeback error on sector 119419741440
Nov 26 10:18:04 veeambrs03 kernel: [782117.490233] XFS (zd0): Unmounting Filesystem
Nov 26 11:24:29 veeambrs03 kernel: [ 3743.632614] SGI XFS with ACLs, security attributes, realtime, no debug enabled
Nov 26 11:24:29 veeambrs03 kernel: [ 3743.639706] XFS (zd0): Mounting V5 Filesystem
Nov 26 11:24:29 veeambrs03 kernel: [ 3743.655595] XFS (zd0): Starting recovery (logdev: internal)
Nov 26 11:24:29 veeambrs03 kernel: [ 3743.727501] XFS (zd0): Ending recovery (logdev: internal)
Nov 26 11:24:43 veeambrs03 kernel: [ 3758.589815] XFS (zd0): Unmounting Filesystem
Nov 26 11:52:44 veeambrs03 kernel: [ 5439.146421] XFS (zd0): Mounting V5 Filesystem
Nov 26 11:52:44 veeambrs03 kernel: [ 5439.160595] XFS (zd0): Ending clean mount
Nov 26 13:04:48 veeambrs03 kernel: [ 9762.714854] XFS (zd0): log I/O error -5
Nov 26 13:04:48 veeambrs03 kernel: [ 9762.714950] XFS (zd0): xfs_do_force_shutdown(0x2) called from line 1250 of file fs/xfs/xfs_log.c. Return address = 0000000007ebdce4
Nov 26 13:04:48 veeambrs03 kernel: [ 9762.714952] XFS (zd0): Log I/O Error Detected. Shutting down filesystem
Nov 26 13:04:48 veeambrs03 kernel: [ 9762.714994] XFS (zd0): Please unmount the filesystem and rectify the problem(s)