Transferred data 10x processed/read data

[DanielJ]

I am trying to replace a legacy backup product backing up a physical server running Red Hat and Oracle. As snapshotting caused problems for Oracle I switched over to file backup. I excluded /dev, /proc and /sys. Right now the job has been going for 11 hours and is showing Processed = 21.1 GB, Read = 21.1 GB and Transferred = 254 GB. What could be the problem here?

[DanielJ]

Looks like the process ran normally up to a certain point (started at 23:00):

[24.09.2019 23:12:17] <140538275673856> vmb | Session progress: 41%; processed: [22395323847/202834444288] read: [22395323847], transfer: [18788981763] speed: 31093086 bottleneck: 33/26/2/40
[24.09.2019 23:12:27] <140538275673856> vmb | pex: 22448318971/22448318971/0/22448318971/18792276044 / 33/26/15/2/40/18

From that point onwards, it read very little data and transferred a lot, until I stopped it after 11 hours:

[25.09.2019 10:11:58] <140538275673856> vmb | Session progress: 41%; processed: [22629546447/202834444288] read: [22629546447], transfer: [275681796682] speed: 561505 bottleneck: 1/24/3/32
[25.09.2019 10:12:09] <140538275673856> vmb | pex: 22629550982/22629550982/0/22629550982/275749115224 / 1/24/46/3/32/0

I will open a support ticket, I just wondered if anyone had seen this and knew what could cause it.

[DanielJ]

Opened ticket 03781512.

[DanielJ]

To clarify, the Oracle database directories are excluded too. There are only around 50 GB in total in the included directories.

[DanielJ]

I got a response from support along the lines that it takes a long time to read a lot of files. I know it does but as I interpret the logs, the agent sent 239 GB to the repository while reading 223 MB, this over the course of almost 11 hours. Maybe the problem is caused by having many small files in the same directory, there are three directories in the backup scope where each has over 150.000 files (xfs). I know snapshot would be the best but the customer doesn't want that (and it works fine with the other product, without snapshots).

[PTide]

Hi,

Thank you for getting back to us with the info. The problem with file-level backup performance (no matter if snapshots are used or not) is a known issue that will be fixed in the next update.

Do you mind telling what kind of problems did you encounter when you were using snapshots for Oracle? Also, have you tried using the built-in Oracle Application-Aware Processing functionality?

Thanks!

[DanielJ]

What happened was that Oracle crashed after its services stopped responding, and this happened as the snapshot was created. Now as Oracle is on its own volume group, we could try excluding that (we should have done that from the beginning, of course), but as the server is configured this would still include /orafra and neither I nor the DBA's have found any info on if snapshotting only that would cause problems too. I think we will have to find some other solution for this server.

[PTide]

I understand, but it's still not clear whether or not advanced settings for Oracle have been used. If yes, then the described behaviour is certainly a bug and must be addressed by our support team directly. If Oracle processing wasn't enabled for that backup job, then I'd really appreciate if yo u could tell why.

Thanks!

[DanielJ]

Oracle processing was not enabled because we did not want the job to do anything with Oracle. The DBA's want to control that themselves, so we plan on using the RMAN plugin for Oracle data. We just wanted to get the filesystem backup job moving first. Basically we are trying to replicate an older solution which too has an RMAN plugin emulating a tape drive, and a separate job for the filesystem. Now support has found that the server was probably running out of memory as the snapshot was taken, so we will see if we can get Oracle to restrict its memory usage to leave enough for the Veeam agent.

[PTide]

The DBA's want to control that themselves, so we plan on using the RMAN plugin for Oracle data.

Ok, I totally understand those concerns. Just so you know, there are also pre-freeze/post-thaw scripts that allow to perform custom actions right before and after a snapshot is taken. There are people who use that functionality with SAP HANA and it works fine. Thank you for the clarification

Cheers