Issue with backup up large BTRFS volumes - kernel 5.0 and experimental veeamsnap

[gothaggis]

Case # 03539271

While we have Veeam Support, they won't provide support since I'm running CentOS with Kernel 5.0.9. They only support CentOS 7.6 with kernel 3.10 (since that is the official kernel of that version of CentOS), which doesn't support BTRFS in the Veeam Agent. File level backup isn't really an option when you are trying to back up nearly 50TB. I'm also using the experimental branch of veeamsnap that works with kernel 5.

I've run into issues attempting to backup all of my BTRFS subvolumes in one job - it seems to get to around 65% complete (after 3 days), then fails. The most recent error in the Veeam Agent is:

Code: Select all

 04:37:17 (+2)    [error] Failed to perform backup
 04:37:17 (+2)    [error] Input/output error

dmesg:

Code: Select all

[Mon May  6 04:32:53 2019] lo_write_bvec: 5472 callbacks suppressed
[Mon May  6 04:32:53 2019] loop: Write error at byte offset 11239965704192, length 4096.
[Mon May  6 04:32:53 2019] print_req_error: I/O error, dev loop1, sector 21953058016 flags 801
[Mon May  6 04:32:53 2019] loop: Write error at byte offset 11310809276416, length 4096.
[Mon May  6 04:32:53 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:53 2019] print_req_error: I/O error, dev loop1, sector 22091424368 flags 801
[Mon May  6 04:32:53 2019] loop: Write error at byte offset 11310817665024, length 4096.
[Mon May  6 04:32:53 2019] print_req_error: I/O error, dev loop1, sector 22091440752 flags 801
[Mon May  6 04:32:53 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:53 2019] loop: Write error at byte offset 11557782667264, length 4096.
[Mon May  6 04:32:53 2019] print_req_error: I/O error, dev loop1, sector 22573794272 flags 801
[Mon May  6 04:32:53 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:53 2019] loop: Write error at byte offset 11559936884736, length 4096.
[Mon May  6 04:32:53 2019] print_req_error: I/O error, dev loop1, sector 22578001728 flags 801
[Mon May  6 04:32:53 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:53 2019] loop: Write error at byte offset 11559937933312, length 4096.
[Mon May  6 04:32:53 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:53 2019] print_req_error: I/O error, dev loop1, sector 22578003776 flags 801
[Mon May  6 04:32:53 2019] loop: Write error at byte offset 11629733150720, length 4096.
[Mon May  6 04:32:53 2019] print_req_error: I/O error, dev loop1, sector 22714322560 flags 801
[Mon May  6 04:32:53 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:53 2019] loop: Write error at byte offset 11702749577216, length 4096.
[Mon May  6 04:32:53 2019] print_req_error: I/O error, dev loop1, sector 22856932768 flags 801
[Mon May  6 04:32:53 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:53 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:55 2019] loop: Write error at byte offset 12601470746624, length 4096.
[Mon May  6 04:32:55 2019] print_req_error: I/O error, dev loop1, sector 24612247336 flags 801
[Mon May  6 04:32:55 2019] loop: Write error at byte offset 12601471160320, length 4096.
[Mon May  6 04:32:55 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:55 2019] print_req_error: I/O error, dev loop1, sector 24612248360 flags 801
[Mon May  6 04:32:55 2019] BTRFS error (device loop1): bdev /dev/loop1 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
[Mon May  6 04:32:55 2019] BTRFS: error (device loop1) in btrfs_commit_transaction:2227: errno=-5 IO failure (Error while writing out transaction)
[Mon May  6 04:32:55 2019] BTRFS info (device loop1): forced readonly
[Mon May  6 04:32:55 2019] BTRFS warning (device loop1): Skipping commit of aborted transaction.
[Mon May  6 04:32:55 2019] BTRFS: error (device loop1) in cleanup_transaction:1831: errno=-5 IO failure
[Mon May  6 04:32:55 2019] BTRFS info (device loop1): delayed_refs has NO entry

I'm backing up to a remote Veeam repo that has over 100TB free (in fact, I'm the only job backing up to it at the moment) and both servers have 10GbE connections. Has anyone run into something like this?

In the meantime, I've decided to have different jobs for each subvolume (largest btrfs subvol is 16TB) and so far, jobs are successfully completing this way)

I should also note that this data is highly compressible - 26TB of data was compressing down to 2.7TB, which was great.

[PTide]

Hi,

Personally I haven't run into anything like that (maybe because I tend to stay away from BTRFS whenever possible ), however I can see this particular error:

Code: Select all

[Mon May  6 04:32:55 2019] BTRFS: error (device loop1) in cleanup_transaction:1831: errno=-5 IO failure

and there is an opinion that it might be purely BTRFS issue which can be fixed by certain btrfs tools.

P.S. In regards to

They only support CentOS 7.6 with kernel 3.10 (since that is the official kernel of that version of CentOS), which doesn't support BTRFS in the Veeam Agent.

I would like to clarify something - official BTRFS documentation says:

<...>Prerequisite is a fairly modern btrfs toolsuite, version 0.20rc1 seems to be good enough. Earlier versions may fail to list subvolumes when cleaner process is operating, and this could cause random failures during backup.

That said, many fixes have gone in btrfs send/receive and it is recommended to use at least a 3.14, or 3.15 kernel to have later patches. Similarly, you will want at least btrfs-tools 3.12, or later if available.<...>

Thanks!

[gothaggis]

ah, thanks for that link. I was wondering if some of those fixes might actually be in the centos kernel, since some things get backported (not sure about btrfs though, as it seems red hat are moving away from it)