Comprehensive data protection for all workloads
Post Reply
BrianBuchanan
Enthusiast
Posts: 52
Liked: 9 times
Joined: Nov 29, 2019 12:56 pm
Full Name: Brian Buchanan
Contact:

Problems with Linux Docker Repository on ZFS

Post by BrianBuchanan »

Case #04516245

I have a linux repository setup like Veeam's Tom Sightler Veeam-Docker vbrrepo and it went offline last night.

It looks like the zvol has no additional space while the xfs filesystem looks like it has plenty.

My guess is that the zvol is trying to grow, but has no space and puts the xfs filesystem offline. however there's 37T of freespace on the XFS filesystem.

I just don't know how to approach this situation.

On the host:

Code: Select all

bbuchanan@veeambrs03:~$ zfs list
NAME                    USED  AVAIL     REFER  MOUNTPOINT
pool01                  118T  5.98T       96K  /pool01
pool01/veeam            118T  2.79G       96K  /pool01/veeam
pool01/veeam/repo_vol   118T  2.79G     78.3T  -
Inside the docker container it appears to have sufficient space:

Code: Select all

veeam@veeambrs03:~$ df -h
Filesystem                       Size  Used Avail Use% Mounted on
overlay                          439G   35G  383G   9% /
tmpfs                             64M     0   64M   0% /dev
tmpfs                             94G     0   94G   0% /sys/fs/cgroup
shm                               64M     0   64M   0% /dev/shm
/dev/sdb2                        439G   35G  383G   9% /keys
tmpfs                             94G     0   94G   0% /proc/acpi
tmpfs                             94G     0   94G   0% /proc/scsi
tmpfs                             94G     0   94G   0% /sys/firmware
/dev/zvol/pool01/veeam/repo_vol   77T   41T   37T  53% /repo

Code: Select all

bbuchanan@veeambrs03:~$ grep XFS /var/log/syslog
Nov 26 06:01:55 veeambrs03 kernel: [766748.837523] XFS (zd0): writeback error on sector 7190797832
Nov 26 06:01:55 veeambrs03 kernel: [766748.837730] XFS (zd0): writeback error on sector 7190795728
Nov 26 06:01:55 veeambrs03 kernel: [766748.838231] XFS (zd0): writeback error on sector 4295013544
Nov 26 06:01:55 veeambrs03 kernel: [766748.843401] XFS (zd0): log I/O error -5
Nov 26 06:01:55 veeambrs03 kernel: [766748.843468] XFS (zd0): xfs_do_force_shutdown(0x2) called from line 1250 of file fs/xfs/xfs_log.c. Return address = 00000000f77661f4
Nov 26 06:01:55 veeambrs03 kernel: [766748.843469] XFS (zd0): Log I/O Error Detected. Shutting down filesystem
Nov 26 06:01:55 veeambrs03 kernel: [766748.843496] XFS (zd0): Please unmount the filesystem and rectify the problem(s)
Nov 26 06:01:55 veeambrs03 kernel: [766748.858784] XFS (zd0): writeback error on sector 121017993416
Nov 26 06:01:55 veeambrs03 kernel: [766748.858845] XFS (zd0): writeback error on sector 119418807552
Nov 26 06:01:55 veeambrs03 kernel: [766748.861382] XFS (zd0): writeback error on sector 119626955952
Nov 26 06:01:55 veeambrs03 kernel: [766748.871254] XFS (zd0): writeback error on sector 121018460360
Nov 26 06:01:55 veeambrs03 kernel: [766748.875798] XFS (zd0): writeback error on sector 119419274496
Nov 26 06:01:55 veeambrs03 kernel: [766748.882613] XFS (zd0): writeback error on sector 118905590816
Nov 26 06:01:55 veeambrs03 kernel: [766748.887775] XFS (zd0): writeback error on sector 119419741440
Nov 26 10:18:04 veeambrs03 kernel: [782117.490233] XFS (zd0): Unmounting Filesystem
Nov 26 11:24:29 veeambrs03 kernel: [ 3743.632614] SGI XFS with ACLs, security attributes, realtime, no debug enabled
Nov 26 11:24:29 veeambrs03 kernel: [ 3743.639706] XFS (zd0): Mounting V5 Filesystem
Nov 26 11:24:29 veeambrs03 kernel: [ 3743.655595] XFS (zd0): Starting recovery (logdev: internal)
Nov 26 11:24:29 veeambrs03 kernel: [ 3743.727501] XFS (zd0): Ending recovery (logdev: internal)
Nov 26 11:24:43 veeambrs03 kernel: [ 3758.589815] XFS (zd0): Unmounting Filesystem
Nov 26 11:52:44 veeambrs03 kernel: [ 5439.146421] XFS (zd0): Mounting V5 Filesystem
Nov 26 11:52:44 veeambrs03 kernel: [ 5439.160595] XFS (zd0): Ending clean mount
Nov 26 13:04:48 veeambrs03 kernel: [ 9762.714854] XFS (zd0): log I/O error -5
Nov 26 13:04:48 veeambrs03 kernel: [ 9762.714950] XFS (zd0): xfs_do_force_shutdown(0x2) called from line 1250 of file fs/xfs/xfs_log.c. Return address = 0000000007ebdce4
Nov 26 13:04:48 veeambrs03 kernel: [ 9762.714952] XFS (zd0): Log I/O Error Detected. Shutting down filesystem
Nov 26 13:04:48 veeambrs03 kernel: [ 9762.714994] XFS (zd0): Please unmount the filesystem and rectify the problem(s)
BrianBuchanan
Enthusiast
Posts: 52
Liked: 9 times
Joined: Nov 29, 2019 12:56 pm
Full Name: Brian Buchanan
Contact:

Re: Problems with Linux Docker Repository on ZFS

Post by BrianBuchanan »

Looks like the space is in snapshots

Code: Select all

bbuchanan@veeambrs03:~$ zfs list -o space -r pool01
NAME                   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
pool01                 5.98T   118T        0B     96K             0B       118T
pool01/veeam           2.79G   118T        0B     96K             0B       118T
pool01/veeam/repo_vol  2.79G   118T     39.7T   78.3T             0B         0B
BrianBuchanan
Enthusiast
Posts: 52
Liked: 9 times
Joined: Nov 29, 2019 12:56 pm
Full Name: Brian Buchanan
Contact:

Re: Problems with Linux Docker Repository on ZFS

Post by BrianBuchanan »

Might have it, but it could be a symptom of something else, maybe syncoid hasn't been running successfully.

Code: Select all

bbuchanan@veeambrs03:~$ zfs list -ro space -t all pool01
NAME                                                           AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
pool01                                                         5.98T   118T        0B     96K             0B       118T
pool01/veeam                                                   2.79G   118T        0B     96K             0B       118T
pool01/veeam/repo_vol                                          2.79G   118T     39.7T   78.3T             0B         0B
pool01/veeam/repo_vol@syncoid_veeambrs04_2020-10-16:00:30:15       -  9.74G         -       -              -          -
pool01/veeam/repo_vol@syncoid_veeambrs04_2020-10-16:00:45:18       -   365M         -       -              -          -
pool01/veeam/repo_vol@syncoid_veeambrs04_2020-10-16:09:10:46       -  38.1G         -       -              -          -
pool01/veeam/repo_vol@autosnap_2020-11-22_00:00:01_daily           -   102G         -       -              -          -
pool01/veeam/repo_vol@autosnap_2020-11-23_00:00:02_daily           -  1.04T         -       -              -          -
pool01/veeam/repo_vol@autosnap_2020-11-24_00:00:01_daily           -  5.28G         -       -              -          -
pool01/veeam/repo_vol@autosnap_2020-11-24_16:00:01_hourly          -     0B         -       -              -          -
pool01/veeam/repo_vol@autosnap_2020-11-24_17:00:01_hourly          -     0B         -       -              -          -
pool01/veeam/repo_vol@autosnap_2020-11-24_18:00:02_hourly          -   894M         -       -              -          -
... trimmed

bbuchanan@veeambrs03:~$ sudo zfs destroy pool01/veeam/repo_vol@syncoid_veeambrs04_2020-10-16:00:30:15
bbuchanan@veeambrs03:~$ zfs list -o space -r pool01
NAME                   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
pool01                 5.98T   118T        0B     96K             0B       118T
pool01/veeam           12.5G   118T        0B     96K             0B       118T
pool01/veeam/repo_vol  12.5G   118T     39.7T   78.3T             0B         0B
bbuchanan@veeambrs03:~$ sudo zfs destroy pool01/veeam/repo_vol@syncoid_veeambrs04_2020-10-16:00:45:18
bbuchanan@veeambrs03:~$ zfs list -o space -r pool01
NAME                   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
pool01                 5.91T   118T        0B     96K             0B       118T
pool01/veeam           89.7G   118T        0B     96K             0B       118T
pool01/veeam/repo_vol  89.7G   118T     39.6T   78.3T             0B         0B
bbuchanan@veeambrs03:~$ sudo zfs destroy pool01/veeam/repo_vol@syncoid_veeambrs04_2020-10-16:09:10:46
bbuchanan@veeambrs03:~$ zfs list -o space -r pool01
NAME                   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
pool01                    0B   139T        0B     96K             0B       139T
pool01/veeam           20.9T  97.1T        0B     96K             0B      97.1T
pool01/veeam/repo_vol  20.9T  97.1T     18.8T   78.3T             0B         0B
Post Reply

Who is online

Users browsing this forum: ante_704, mibrown9954, mrmccoy007, Semrush [Bot] and 246 guests