Comprehensive data protection for all workloads
Post Reply
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

dedup in Rman backup in 9.5 update 4

Post by vanguard »

I just created my first Oracle backup using RMAN en Veeam 9.5 update 4. It works like a charm. The rman plugin of Veeam acts as a gateway from the Rman tape interface to my Veeam server. The backup and restores are done from the RMAN interface and after first configuration I don't need to know anything about Veeam.

However, I am not happy with the backup size and the CPU load during the backup. A lot of CPU cycles are used by 2 processes RmanPluginManager and VeeamAgent. The backup that is created is about half the size of the database. I assumed I needed to make another backup for the deduplicating to kick in. But additional backups all had the same size. This does not look like deduplicating, it must be compression. And not an efficient compression, it consumes a lot of CPU and compresses nu more than a factor 2

Anyone can shed a light on this?
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert »

Can you please tell me if you have enabled Oracle RMAN own compression?
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert » 1 person likes this post

The reasons why I am asking is that usually when someone report the high CPU usage and space issues, then it comes from double compression. RMAN used with "as compressed" and Veeam standard compression. Double compression usually lead into bigger than needed backup files. And processing is overall slow and CPU intensive.

Regarding Veeam own processing. As you concluded, Veeam space saving mainly comes from compression and not source side deduplication. Our deduplication work on fixed block length and not globally, so the space saving is minimal. Nevertheless we keep it enabled as it need near 0 ressources and save some space. The source side deduplication is much more affective at the VM backup where you have multiple VMs within a job as well it optimized data on source side that you read multiple times because of change block tracking usage of patterns sice.

Compression level is not aggressive and should use CPU acceleration.
In my tests I saw 1,2 Cores used per used RMAN device channel (Veeam Task slot) on the Oracle Server and arround 350MB of RAM. These numbers include RMAN specific tasks. RAM usage depends on speed of source, network and target. If you have a fast source and a slow network/target then you need aa bit more RAM than the 350MB.
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard »

No, RMAN compression is not active during this test.
I indeed can confirm that the CPU load of Veeam (Veeamagent +RmanpluginManager) is about 60% (with 2 cores and 1 Rman channel so 1.2 core per channel). And the compression is about a factor 2.

Then I tried an alternative: a compressed one-channel RMAN backup of the same database over the network to a fileserver, so no Veeam involved. As expected the RMAN compression leads to a high CPU load but it is a bit lower than with Veeam scenario. And i get a factor 6 compression, so 3x less storage usage and network load than with Veeam.
The elapsed time is about 15% shorter.

So this feels dissappointing to me.
Am i missing something?
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert »

Thanks for reporting this. Will share feedback with QA and we will have a look at this.
Report back here in some time.

Just to confirm, did you use the RTM version of Update 4? Or Beta2?
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert »

And please share the database size with us and what kind of data there is in the database?
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard »

It is a 31 GB database. About 9 GB of database are empty blocks, as the uncompressed RMAN backup size is 22 GB. RMAN always skips unused blocks ("Unused Block Compression").
So my Veeam backup is 10.5 Gb and the compressed RMAN backup is 3.7 GB.
The database contains just normal relational tables. No special stuff like binary or spatial data.

We are using the RTM version. The Plugin DLL has version 9.5.4.2399
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by tsightler » 1 person likes this post

Can I ask you to compare the speed between these two jobs? I would certainly expect RMAN native compression to produce backup files smaller than Veeam, but it should also need much more CPU to do so (or be significantly slower if running only a single channel). RMAN native compression is highly optimized for typical database workloads, and to maximize space savings, so it should be well expected that it's backups will be smaller.

Typically RMAN compression will use a single core per RMAN channel, allowing you to limit it's overall impact on the host CPU resources by allocating a proper number of channels, but this also limits it's overall throughput (perhaps not too noticable with such a small database).

Veeam uses a somewhat different approach as it leverages the LZ4 algorithm as the default/optimal algorithm and the goal of this is to strike a balance between CPU usage, compression and throughput while still achieving an average 2:1 compression ratio. CPU usage is largely defined by the throughput with a rough expectation of 100-150MBs for 100% of a single CPU core, although this obviously varies with the power of your processor (you can run lzbench again a similar source dataset to get some idea what a core is capable of). Higher throughput will lead to high CPU because there is more data that needs to be processed.

Regarding dedupe, it's really a non-factor in RMAN type backups with Veeam because Veeam does not dedupe across files, it only removes duplicate blocks within a file. Since RMAN always makes new files, there's no significant savings to be had with Veeam dedupe in concert with RMAN.
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard »

Thanks, I will retest both backup methods and give you more numbers.
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard »

"with a rough expectation of 100-150MBs..". I get only 15-20MBytes/sec. Or did you mean Mbits/sec?
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard »

Four tests with 31.6 GB database, 1 RMAN channel.

1) Backup to Veeam, Rman compression disabled:
elapsed time: 20min
backup storage: 10GB
CPU load (with 2 cores): 75%
CPU usage: 30min

2) Backup to Veeam, Rman compression enabled:
elapsed time: 23min
backup storage: 4.8 GB
CPU load (with 2 cores): 60%
CPU usage: 27.5 min

3) Backup to file server over network, Rman compression disabled:
elapsed time: 7min
backup storage: 22GB
CPU load (with 2 cores): 15%
CPU usage: 1min

4) Backup to file server over network, Rman compression enabled:
elapsed time: 17min
backup storage: 3.7 GB
CPU load (with 2 cores): 55%
CPU usage: 19min

The Veeam backup without RMan compression seems worst. Highest CPU usage and highest storage usage.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Gostev »

So looking at the first test, something is potentially off: there's just too little data over too much time for our light compression to keep the 2 core CPU as busy for 20 minutes. I am just comparing this to VM backup streams, where it will take no more than 2 minutes for a 2 core VMware backup proxy to create a 10 GB VM backup file file, with CPU staying about as busy. But 10 times longer with CPU staying as busy all the time just makes no sense to me?

Could there be an impact from some 3rd party software? Like some real-time antivirus protection validating all Veeam I/O and thus adding to CPU usage, while slowing down the processing performance? I suggest you investigate this more closely with support to see what introduces the bottleneck which makes the processing take whopping 20 minutes of continuous high CPU load.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Gostev »

On the other hand, compression numbers look good and your results are showing exactly the difference that I would expect to see from a purpose-built native compression algorithm vs. our general purpose one. It is just the CPU load that seems off to me, it almost looks like a wrong algorithm was selected in the plug-in code (we have a few selectable for VM backup, and for example using industry standard zlib increases CPU load like in a few times over Optimal).
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard »

You wrote: "it almost looks like a wrong algorithm was selected in the plug-in code"

Can we configure this ourself?

During the backup , all (extra) CPU of scenario 1 is used by the processes Veeamagent.exe and RmanPluginManager.exe. Not from the antivirus software. And this did not change after stopping the antivirus program.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Gostev »

I checked with the devs and they verified that the algorithm use is the correct one, so it must be something else.
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard »

Are there any plans for further investigation? With the current behaviour of the plugin I see no added value compared to writing compressed RMAN backups to a fileserver. Or am i missing something?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Gostev »

Did you open a support case? This has to happen before the investigation of your environment can be done.

If you don't need centralized management (visibility of RMAN jobs and RMAN backups in your Veeam console, along with the ability to use Veeam Explorer for Oracle), or the requirement to store RMAN backups in Veeam repository (particularly scale-out backup repository, using which increases the backup and restore throughput for large databases due to streaming to multiple extents at once) - then you can certainly continue using your current approach without changing anything.
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard »

OK, Thanks.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert » 1 person likes this post

I did some tests with a bigger database. As stated above the amount of data is too small to get reliable numbers. Compression depends as well on what you backup.

My tests with arround 150GB showed pretty similar results in case of backup time with a small overhead on the plug-in side.
Ressource usage was pretty the same at my VM. RMAN used arround 1 Core per channel and we used 1,2 Cores. This depends mainly on CPU clock speed.
RAM consumtion was 150MB-500MB depending on the speed of the disks (if one component is slower then the other, the plug-in will buffer more data in RAM.

The backup file size was slightly smaller with RMAN compression enabled but this effect is much smaller when you backup bigger databases.
joebranca
Enthusiast
Posts: 49
Liked: never
Joined: Oct 28, 2015 9:36 pm
Full Name: Joe Brancaleone
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by joebranca »

Is there a technical doc or KB for setting up Oracle RMAN plugin?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by foggy »

Hi Joe, here it is.
gene
Lurker
Posts: 2
Liked: 1 time
Joined: Jan 01, 2020 1:44 am
Full Name: Eugene
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by gene »

I'm having the same problem.
  • Installed VeeamBackup&Replication_9.5.4.2866.Update4b_20191210 and it updated its components on a Windows 2019 server
  • Installed VeeamPluginforOracleRMAN-9.5.4.2866-1.x86_64.rpm on the Oracle database server, which is running RHEL 7
  • Ran `OracleRMANConfigTool --wizard` and got it connected to the veeam server
  • Manually ran three RMAN backups
  • Checked disk utilisation after each backup, no apparent deduplication occuring
  • Ran a report against the job and it shows Dedupe 1x for each backup job that ran
Am I missing something?

Additional information:

RMAN config:

Code: Select all

ORACLE_SID=APTDB  ORACLE_HOME=/home/oracle/database

Oracle RMAN settings:
CONFIGURE DEFAULT DEVICE TYPE TO SBT_TAPE;
CONFIGURE CHANNEL DEVICE TYPE SBT_TAPE
PARMS 'SBT_LIBRARY=/opt/veeam/VeeamPluginforOracleRMAN/libOracleRMANPlugin.so'
FORMAT 'NnNnNnNnNn-NnNn-NnNn-NnNn-NnNnNnNnNn/RMAN_%I_%d_%T_%U.vab';
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1;
CONFIGURE DEVICE TYPE SBT_TAPE PARALLELISM 1;

Channel definition for RMAN scripts:
ALLOCATE CHANNEL VeeamAgentChannel1 DEVICE TYPE SBT_TAPE
PARMS 'SBT_LIBRARY=/opt/veeam/VeeamPluginforOracleRMAN/libOracleRMANPlugin.so'
FORMAT 'NnNnNnNnNn-NnNn-NnNn-NnNn-NnNnNnNnNn/RMAN_%I_%d_%T_%U.vab';
Veeam config on linux server:

Code: Select all

<Config>
    <VBRConnectionParams vbrHostName="86.75.30.9" vbrPort="10006" vbrUser="username" vbrDomain="" vbrPassword="SeCrEtSeCrEtSeCrEt==" />
    <AgentParams />
    <RepositoryParams parallelism="1">
        <Repository repositoryName="Default Backup Repository" repositoryID="NnNnNnNnNn-NnNn-NnNn-NnNn-NnNnNnNnNn" />
    </RepositoryParams>
    <PluginParameters />
</Config>
How the RMAN backup was kicked off:

Code: Select all

rman target /
run {
shutdown immediate;
startup mount;
backup database;
startup;
}
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by HannesK »

Hello Gene,
and welcome to the forum.
I'm having the same problem.
which one? there are two topics... deduplication and high CPU load.

As stated earlier, Veeam does not support deduplication for RMAN backups. So everything looks "as expected" for me.

Best regards,
Hannes
gene
Lurker
Posts: 2
Liked: 1 time
Joined: Jan 01, 2020 1:44 am
Full Name: Eugene
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by gene » 1 person likes this post

which one? there are two topics... deduplication and high CPU load.
Deduplication
As stated earlier, Veeam does not support deduplication for RMAN backups. So everything looks "as expected" for me.
I'm not seeing it plainly stated in this thread that it doesn't support it.

Additionally, the helpcenter documentation says inline deduplication is performed.
https://helpcenter.veeam.com/docs/backu ... l?ver=95u4
Veeam Plug-in uses built-in compression and inline deduplication during the backup. Do not use Oracle RMAN integrated compression as it can slow down the backup and restore processes. Prepare your Oracle RMAN scripts accordingly.

https://helpcenter.veeam.com/docs/backu ... l?ver=95u4
Veeam Plug-in uses built-in compression and inline deduplication functionality of Veeam Backup & Replication

https://helpcenter.veeam.com/docs/backu ... l?ver=95u4
During the backup, Veeam uses its own compression and inline deduplication
Based on that I would expect that deduplication should be occurring. If it really isn't supported I would expect a disclaimer that it isn't.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by PetrM » 1 person likes this post

Hi Eugene!

These statements are true for compression setting only.
Thanks for catching this, we will update our documentation accordingly.

Thanks!
Post Reply

Who is online

Users browsing this forum: bct44, Google [Bot] and 194 guests