Poor Performance on a 26tb zfs server

[Machariel]

Customer Case#: #04624819

Hello all, I'm a new customer and started implementation this week. Our main environment is VMWare with a few physical devices scattered about. Everything is going well with the exception of one of our physical Linux servers running Ubuntu 16.04. I've installed the agent and it's working but poor performance. The server itself has a total of 26tb of storage all on ZFS with 3 mirrors.

Since it had ZFS, I wasn't able to use snapshotting. As a result, the agent is running File Level backup with no snapshotting. The connection between this server and proxy is 1GB/s and I'm only seeing a processing rate of 12 MB/s. The bottleneck is showing Source: 5%, Proxy 60%, Network: 0%, and Target: 55%. Any clue where I can start to troubleshoot? Or am I just stuck with this performance because....well...ZFS? Thanks!.

Update: The job failed last night with errors...
20:15:16 (+2) [error] Failed to perform backup
20:15:16 (+2) [error] read: End of file
20:15:16 (+2) [error] Processing finished with errors at 2021-02-03 20:15:16 GMT

[HannesK]

Hello,
and welcome to the forums.

How fast is it if you copy files with scp to a different server (test with same folder)?

You could switch to NAS backup. Then you would get at least fast incremental backups.

ZFS is probably not the issue alone. It's probably more a combination of different things (hardware included). Support should be able to determine whether there is a way of tuning or not. I assume that you are using the latest version of the agent.

Best regards,
Hannes

[Machariel]

Thanks for the reply. I think I might have found the major issue. The CPU is just extremely pegged on this server. I have a user running some processes with a nice value of -20. Since I can't use snapshotting of any kind, I'm forced to use file level backup and scan the entire folders. And the poor VAL can't do much processing. I'm looking at 6 MB/s. At this rate, I'll have a full backup in 45 days.

I need to talk with the user tomorrow and ask why they have a processes running with such high priority that's going on 5 day runtime now.

[Mildur]

File copy for millions of small files will always be very slow. Veeam File Job or copy as a user in a file explorer, it‘s always slow and painful.
Only alternative is block level backups.

I hope, you can talk over your user and switch to volume level backup

[Machariel]

Thanks, and to make matters worst, the user is using all 72 threads on server. So my backups went down to 2MB/s (completely unusable). I emailed the user and let him know the situation and that the systems administrators need some CPU too once in awhile LOL.

I'd love to switch over to volume level backups, but to my understanding, I can't while using ZFS. Is that correct?

I can see all the devices (/dev/sda - /dev/sdf) when I create a volume job, but I haven't tried it yet as I figured the way ZFS worked, it would fail or be unrecoverable.

I'm really hoping Veeam develops a VAL that can utilize ZFS snapshotting in the future.

[HannesK]

Hello,
correct, snapshot-based backup is impossible today for ZFS

https://helpcenter.veeam.com/docs/agent ... tml?ver=40

For example, you can use the snapshot-less mode to back up data that resides in a file system that is not supported for snapshot-based backup with Veeam Agent, such as UFS, ZFS, GFS, GFS2 or OCFS2

Migrating to a virtual machine or switching to a file system with snapshot support can happen earlier then we add ZFS snapshot support. Well, or use NAS backup (that will also be slow for the first run, but probably fast enough for incrementals).

Best regards,
Hannes

[Machariel]

Thanks Hannes. The user finished his job and I'm able to breath again. The backup is running at 34 MB/s. That's not too bad for a first backup (considering i'm doing a full file level scan). At this rate, I can get the first initial full done in about 1.5 weeks. Then I'm assuming incrementals will be a lot quicker going forward.

Thanks for all your help.
-Mach