Comprehensive data protection for all workloads
Post Reply
vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

dedup in Rman backup in 9.5 update 4

Post by vanguard » Jan 11, 2019 5:00 pm

I just created my first Oracle backup using RMAN en Veeam 9.5 update 4. It works like a charm. The rman plugin of Veeam acts as a gateway from the Rman tape interface to my Veeam server. The backup and restores are done from the RMAN interface and after first configuration I don't need to know anything about Veeam.

However, I am not happy with the backup size and the CPU load during the backup. A lot of CPU cycles are used by 2 processes RmanPluginManager and VeeamAgent. The backup that is created is about half the size of the database. I assumed I needed to make another backup for the deduplicating to kick in. But additional backups all had the same size. This does not look like deduplicating, it must be compression. And not an efficient compression, it consumes a lot of CPU and compresses nu more than a factor 2

Anyone can shed a light on this?

Andreas Neufert
Veeam Software
Posts: 3630
Liked: 638 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert » Jan 11, 2019 9:44 pm

Can you please tell me if you have enabled Oracle RMAN own compression?

Andreas Neufert
Veeam Software
Posts: 3630
Liked: 638 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert » Jan 14, 2019 7:22 am 1 person likes this post

The reasons why I am asking is that usually when someone report the high CPU usage and space issues, then it comes from double compression. RMAN used with "as compressed" and Veeam standard compression. Double compression usually lead into bigger than needed backup files. And processing is overall slow and CPU intensive.

Regarding Veeam own processing. As you concluded, Veeam space saving mainly comes from compression and not source side deduplication. Our deduplication work on fixed block length and not globally, so the space saving is minimal. Nevertheless we keep it enabled as it need near 0 ressources and save some space. The source side deduplication is much more affective at the VM backup where you have multiple VMs within a job as well it optimized data on source side that you read multiple times because of change block tracking usage of patterns sice.

Compression level is not aggressive and should use CPU acceleration.
In my tests I saw 1,2 Cores used per used RMAN device channel (Veeam Task slot) on the Oracle Server and arround 350MB of RAM. These numbers include RMAN specific tasks. RAM usage depends on speed of source, network and target. If you have a fast source and a slow network/target then you need aa bit more RAM than the 350MB.

vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard » Jan 14, 2019 11:30 am

No, RMAN compression is not active during this test.
I indeed can confirm that the CPU load of Veeam (Veeamagent +RmanpluginManager) is about 60% (with 2 cores and 1 Rman channel so 1.2 core per channel). And the compression is about a factor 2.

Then I tried an alternative: a compressed one-channel RMAN backup of the same database over the network to a fileserver, so no Veeam involved. As expected the RMAN compression leads to a high CPU load but it is a bit lower than with Veeam scenario. And i get a factor 6 compression, so 3x less storage usage and network load than with Veeam.
The elapsed time is about 15% shorter.

So this feels dissappointing to me.
Am i missing something?

Andreas Neufert
Veeam Software
Posts: 3630
Liked: 638 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert » Jan 14, 2019 11:34 am

Thanks for reporting this. Will share feedback with QA and we will have a look at this.
Report back here in some time.

Just to confirm, did you use the RTM version of Update 4? Or Beta2?

Andreas Neufert
Veeam Software
Posts: 3630
Liked: 638 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert » Jan 14, 2019 11:37 am

And please share the database size with us and what kind of data there is in the database?

vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard » Jan 14, 2019 12:17 pm

It is a 31 GB database. About 9 GB of database are empty blocks, as the uncompressed RMAN backup size is 22 GB. RMAN always skips unused blocks ("Unused Block Compression").
So my Veeam backup is 10.5 Gb and the compressed RMAN backup is 3.7 GB.
The database contains just normal relational tables. No special stuff like binary or spatial data.

We are using the RTM version. The Plugin DLL has version 9.5.4.2399

tsightler
VP, Product Management
Posts: 5351
Liked: 2190 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by tsightler » Jan 14, 2019 4:15 pm 1 person likes this post

Can I ask you to compare the speed between these two jobs? I would certainly expect RMAN native compression to produce backup files smaller than Veeam, but it should also need much more CPU to do so (or be significantly slower if running only a single channel). RMAN native compression is highly optimized for typical database workloads, and to maximize space savings, so it should be well expected that it's backups will be smaller.

Typically RMAN compression will use a single core per RMAN channel, allowing you to limit it's overall impact on the host CPU resources by allocating a proper number of channels, but this also limits it's overall throughput (perhaps not too noticable with such a small database).

Veeam uses a somewhat different approach as it leverages the LZ4 algorithm as the default/optimal algorithm and the goal of this is to strike a balance between CPU usage, compression and throughput while still achieving an average 2:1 compression ratio. CPU usage is largely defined by the throughput with a rough expectation of 100-150MBs for 100% of a single CPU core, although this obviously varies with the power of your processor (you can run lzbench again a similar source dataset to get some idea what a core is capable of). Higher throughput will lead to high CPU because there is more data that needs to be processed.

Regarding dedupe, it's really a non-factor in RMAN type backups with Veeam because Veeam does not dedupe across files, it only removes duplicate blocks within a file. Since RMAN always makes new files, there's no significant savings to be had with Veeam dedupe in concert with RMAN.

vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard » Jan 14, 2019 4:22 pm

Thanks, I will retest both backup methods and give you more numbers.

vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard » Jan 15, 2019 11:14 am

"with a rough expectation of 100-150MBs..". I get only 15-20MBytes/sec. Or did you mean Mbits/sec?

vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard » Jan 15, 2019 12:09 pm

Four tests with 31.6 GB database, 1 RMAN channel.

1) Backup to Veeam, Rman compression disabled:
elapsed time: 20min
backup storage: 10GB
CPU load (with 2 cores): 75%
CPU usage: 30min

2) Backup to Veeam, Rman compression enabled:
elapsed time: 23min
backup storage: 4.8 GB
CPU load (with 2 cores): 60%
CPU usage: 27.5 min

3) Backup to file server over network, Rman compression disabled:
elapsed time: 7min
backup storage: 22GB
CPU load (with 2 cores): 15%
CPU usage: 1min

4) Backup to file server over network, Rman compression enabled:
elapsed time: 17min
backup storage: 3.7 GB
CPU load (with 2 cores): 55%
CPU usage: 19min

The Veeam backup without RMan compression seems worst. Highest CPU usage and highest storage usage.

Gostev
SVP, Product Management
Posts: 24300
Liked: 3331 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Gostev » Jan 15, 2019 10:08 pm

So looking at the first test, something is potentially off: there's just too little data over too much time for our light compression to keep the 2 core CPU as busy for 20 minutes. I am just comparing this to VM backup streams, where it will take no more than 2 minutes for a 2 core VMware backup proxy to create a 10 GB VM backup file file, with CPU staying about as busy. But 10 times longer with CPU staying as busy all the time just makes no sense to me?

Could there be an impact from some 3rd party software? Like some real-time antivirus protection validating all Veeam I/O and thus adding to CPU usage, while slowing down the processing performance? I suggest you investigate this more closely with support to see what introduces the bottleneck which makes the processing take whopping 20 minutes of continuous high CPU load.

Gostev
SVP, Product Management
Posts: 24300
Liked: 3331 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Gostev » Jan 15, 2019 10:33 pm

On the other hand, compression numbers look good and your results are showing exactly the difference that I would expect to see from a purpose-built native compression algorithm vs. our general purpose one. It is just the CPU load that seems off to me, it almost looks like a wrong algorithm was selected in the plug-in code (we have a few selectable for VM backup, and for example using industry standard zlib increases CPU load like in a few times over Optimal).

vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard » Jan 17, 2019 9:19 am

You wrote: "it almost looks like a wrong algorithm was selected in the plug-in code"

Can we configure this ourself?

During the backup , all (extra) CPU of scenario 1 is used by the processes Veeamagent.exe and RmanPluginManager.exe. Not from the antivirus software. And this did not change after stopping the antivirus program.

Gostev
SVP, Product Management
Posts: 24300
Liked: 3331 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Gostev » Jan 17, 2019 6:06 pm

I checked with the devs and they verified that the algorithm use is the correct one, so it must be something else.

vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard » Jan 21, 2019 9:08 am

Are there any plans for further investigation? With the current behaviour of the plugin I see no added value compared to writing compressed RMAN backups to a fileserver. Or am i missing something?

Gostev
SVP, Product Management
Posts: 24300
Liked: 3331 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Gostev » Jan 21, 2019 8:34 pm

Did you open a support case? This has to happen before the investigation of your environment can be done.

If you don't need centralized management (visibility of RMAN jobs and RMAN backups in your Veeam console, along with the ability to use Veeam Explorer for Oracle), or the requirement to store RMAN backups in Veeam repository (particularly scale-out backup repository, using which increases the backup and restore throughput for large databases due to streaming to multiple extents at once) - then you can certainly continue using your current approach without changing anything.

vanguard
Influencer
Posts: 10
Liked: never
Joined: Jan 11, 2019 4:43 pm
Full Name: Veeamoracle
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by vanguard » Jan 22, 2019 12:38 pm

OK, Thanks.

Andreas Neufert
Veeam Software
Posts: 3630
Liked: 638 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by Andreas Neufert » Jan 30, 2019 2:03 pm 1 person likes this post

I did some tests with a bigger database. As stated above the amount of data is too small to get reliable numbers. Compression depends as well on what you backup.

My tests with arround 150GB showed pretty similar results in case of backup time with a small overhead on the plug-in side.
Ressource usage was pretty the same at my VM. RMAN used arround 1 Core per channel and we used 1,2 Cores. This depends mainly on CPU clock speed.
RAM consumtion was 150MB-500MB depending on the speed of the disks (if one component is slower then the other, the plug-in will buffer more data in RAM.

The backup file size was slightly smaller with RMAN compression enabled but this effect is much smaller when you backup bigger databases.

joebranca
Influencer
Posts: 23
Liked: never
Joined: Oct 28, 2015 9:36 pm
Full Name: Joe Brancaleone
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by joebranca » Feb 06, 2019 6:27 pm

Is there a technical doc or KB for setting up Oracle RMAN plugin?

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: dedup in Rman backup in 9.5 update 4

Post by foggy » Feb 08, 2019 12:47 pm

Hi Joe, here it is.

Post Reply

Who is online

Users browsing this forum: Google [Bot], gummett, wishr and 66 guests