v7 changes in processing rate calculations

[Fiskepudding]

Just did a upgrade from 6.5 to 7.

One of the first thing i tested was performance, without doing any changes on the jobs.
Previous backup of our SQL server showed stable process rate in the 70 MB/S range. (Full backups)

I ran this SQL job with 7 and showed a wooping 155MB/s processing rate... I thought WOW...

But looking at the data it made no sense. The compression/and or dedupe was worse (2,6x), so it transferred more a bit more data.
(in V6.5 the ratio was 3,2x).
But not enough to explain the drastic change...

So did the math..

V7 (Processed 33,7 GB)
34509 MB Processed/ 423 sec
= 81 MB/s (Statistics claims 155MB)


V6.5 (Processed 33,7 GB)
34509 MB Processed / 479 sec
= 72 MB/S (Witch is exactly what the log says)

Either the new processing rate is not calculated based on Processed data, or the calculations are wrong....

Job History V7:


Job History V6.5:

[Gostev]

The new processing rate is calculated off time when the data was moved, this was required due to introducing parallel processing and disk-level intelligent load balancing. Plus, I think it made the counter more useful overall, as it now represents real throughtput for virtual disk data processing, not affected by time to prepare VM for backup, and remove snapshot.

[Fiskepudding]

I see, its probably a more realistic number when it comes to the processing of data.

Is there a logic explanation to the rather big change in compression/dedupe?
FROM 3,2x to 2,6x (same VM, virtually same data).

Should I cahnge the jobs somehow to achieve the same ratio as in 6.5?
Compression is set to "high" and its optimized for "local target".
Inline deduplication is also turned on for the job.

+ 1 point for bottleneck detection. It seems much better.
With 6.5 it always claimed Source (but source is SSD and high spec, CPU and RAM).
Now v7 agrees with me, and states Proxy as the weak link

[tsightler]

One, question, is this an upgraded job or a brand new job? V7 uses a new compression level by default compared to previous versions. What was previously considered "optimal" compression is now called "High" compression so upgraded jobs will not use the new compression. The new compression algorithm in V7 which is now used when you select "Optimal" doesn't achieve quite the same compression level as the old one, but uses much less CPU and offers much higher throughput. The reason it's not clear to me which you are using is because you proxy is still showing very high CPU.

[Fiskepudding]

This was an upgrade from 6.5. I did not touch the jobs before starting an active full backup.
I am certain that optimal (recommended) settings were used before.
Now it set to High. (After update all my jobs are high.)

Yes, my proxy hit the roof on CPU, both 6.5 and 7. It shows 100% in windows and vSphere as well.

I ran a 1,1 TB job (file server), and the ratio was 1,2x and with 6.5 it was 1,3x.
Time to finish is about the same, actually a bit longer. Since the full 6.5 backup, the data on the file server might have grown a bit , but not by much.
Transferred with 6.5 = 861,8GB (Processed and read = 1,1TB)
Transferred with 7 = 901,5GB (Processed and read = 1,1TB)


Turned the compression back to Optimal (recommended) on my SQL server job and did a new active full backup.

It’s was a bit faster, and dedupe/compression is still “only” 2,6x.

[pfguglielmi]

Hi,

The new algorithm Tom is talking about is not used by default on upgraded jobs, that's why all your jobs were set to High compression (which is the former Optimal compression level) after the upgrade.

Now regarding your dedupe/compression ratios, you can try with Optimal compression and LAN Target storage optimization (smaller blocks, higher dedupe) on a test job.

Thanks!

[tsightler]

This was an upgrade from 6.5. I did not touch the jobs before starting an active full backup.
I am certain that optimal (recommended) settings were used before.
Now it set to High. (After update all my jobs are high.)

Right, this was my point, during an upgrade jobs that were previously set to "Optimal" in prior versions will now be considered "High" compression. To take advantage of the new compression algorithm, you have to change the jobs back to optimal and run an active full. The idea is that upgrades should not change the previous behavior. That's also why the new parallel processing is not enabled by default after an upgrade.

Regarding the slight loss in compression even when using High, I don't really have an answer for that, there may have been a slight tweak to this compression algorithm as well, or a small change in how blocks are handled. Did you compare the actual size of the resulting VBK.

I noticed you are using NBD mode and that network is showing as a potential bottleneck with your V7 backup, only just lower than the proxy. I don't know anything about your environment but it looks like your nearing the maximum.

[Gostev]

No changes in other compression algorithms. May be some bugs in actual reporting of compression ratio fixed (or added), but actual backup size should be the same between 6.5 "Optimal" and 7.0 "High" compression levels.

[Fiskepudding]

Thanx for your replyes

I noticed you are using NBD mode and that network is showing as a potential bottleneck with your V7 backup, only just lower than the proxy. I don't know anything about your environment but it looks like your nearing the maximum.

Network should not be the problem, it is 10Gbit. But I have started to test LAN Target, and it gives me better performance.

Right, this was my point, during an upgrade jobs that were previously set to "Optimal" in prior versions will now be considered "High" compression. To take advantage of the new compression algorithm, you have to change the jobs back to optimal and run an active full. The idea is that upgrades should not change the previous behavior. That's also why the new parallel processing is not enabled by default after an upgrade.

In my fist post, it shows SQL backup with 6.5 and 7.
V 6.5 had optimal compression.
V 7 Had High compression (chaned by upgrade, previous v 6.5 optimal)

There is a difference in compression showing in report, between these.

I did change it back to Optimal in V7 and ran a active full again just to see the results. (Last picture in second post)

Like expected, faster but at the cost of compression.



I have done some more testing and the actual VKB taken with V6.5 is the same size as the v7 backup. (In v7, the compresion is changed to high, by the upgrade).
Even tho the job statistics claims worse compression ratio for V7 (2,6x vs 3,2 for v6.5)

So all is good
The NEW optimal compression seems very fast, it does no longer max CPU on my proxy, and the VBK is not much larger, so i might considder to process more VM's at once, or start processing disk or VMS in parallel. In other words, great job with the new optimal compression.

However there seems some errors on the report, regarding compresion. At least this SQL job.

[tsightler]

Network should not be the problem, it is 10Gbit. But I have started to test LAN Target, and it gives me better performance.

While the physical network itself might not be the bottleneck, NBD mode is still limited by the fact that the transfer must be processed via the ESXi management stack which is quite resource limited. Also, the NFC protocol is not exactly known for it's high transfer speeds for a single stream. You could likely get additional throughput from running concurrent jobs if you have additional bandwidth, which is exactly where the new parallel processing should come in.

However there seems some errors on the report, regarding compresion. At least this SQL job.

I think the numbers in V7 actually look more accurate than the numbers in 6.5 and it sounds like things are working as expected. Let us know how your testing goes. V7 only reaches it's full potential when you leverage the new optimal compression and in-job parallel processing, especially for incremental runs.

[ejleipold]

Has anyone else upgraded from 6.5 to 7 and noticed a performance drop off?

My processing rate has dropped off yet the bottleneck %'s are all less than 10%.

[veremin]

Hi, Evan.

As mentioned above, there have been certain changes in how processing rate is calculated in version 7. So, it’s expected that the numbers you see with the up to date version don’t correspond with the ones that you used to see previously.

Also, you can recheck the processing rate, using the algorithm mentioned by Anton, and see whether the result you get coincides with the numbers shown in the backup console.

Thanks.

[ejleipold]

Ah looks like I had compression set to high (was a migrated job not a new job) so I've set it back to optimal, will see how that goes.

[tsightler]

Note that changes in compression do not take effect until a new active full is created. Also, you may want to consider enabling the parallel processing option at some point, this is where the bulk of the performance improvements come from, especially when running incremental backups.

[Fiskepudding]

As I understand it, you should not see any difference in the backup after the upgrade.
Because V6.5 Optimal = V7 High. (That’s why the upgrade process changed the job)
"Greenies", correct me here if i misunderstood this.

My test jobs showed difference in transferred size, but the actual VBK was the same size.

If you go back to Optimal (in V7) your backups wil be about 10% larger. But the good thing, or should i say GREAT thing, is that it draws much less resources of the proxy.

I did some testing. V6.5 = Optimal, V7= the NEW optimal. (IE, I changed the jobs back to optimal in V7)

Code: Select all

Version	No._Of_jobs	MB/s		Completion_time
6.5			1		     195	       7:03:39
7			3		       475	       4:28:01
7			7		       515	       2:41:35
7			8		       499	       2:49:22

On my system, running 7 jobs with the new parallel processing, cuts the backup window down more than half. Only cost med 10% more disk space.
(Running 8 jobs on a 8 core proxy, slows it down again)

While the physical network itself might not be the bottleneck, NBD mode is still limited by the fact that the transfer must be processed via the ESXi management stack which is quite resource limited. Also, the NFC protocol is not exactly known for it's high transfer speeds for a single stream. You could likely get additional throughput from running concurrent jobs if you have additional bandwidth, which is exactly where the new parallel processing should come in.

I guess I never bothered looking much into the transport mode, jobs are all set to auto. But that might be a topic for another day

[veremin]

"Greenies", correct me here if i misunderstood this.

No, you’ve got it rightly. The main idea was that the upgrade process shouldn’t change anything in previously-seen behavior. Therefore, new features, such as parallel processing, storage snapshots, etc. are disabled by default. Thanks.

[tsightler]

I guess I never bothered looking much into the transport mode, jobs are all set to auto. But that might be a topic for another day

Looks like your getting quite excellent performance now that you're running in parallel so it's probably not a major issue. I've pretty consistently seen NBD mode be able to deliver 350-400MB/s with 10GbE even with 6.5, and I anticipate it going higher with V7. Looks like your able to get close to 500MB/s, which is quite good.

[tullyal]

Hi,

Just wondering if there is any documentation out there than gives more detail on the job statistics - in particular on the bottleneck metrics and throughput calculations etc..

I can't see much about this in the user manual.

Al

[foggy]

Alan, there is a dedicated section regarding bottleneck analysis in the sticky FAQ, probably will be useful.

[Jon - MMIT]

I just upgraded to v7, yesterday afternoon, and it seems as though my throughput has fallen off a cliff. Prior to the upgrade I was on 6.5 patch 3, and my average throughput graph was showing >1 GB/s, with an average processing speed of 450 MB/s. Now that I'm running v7, Enterprise Manager is showing a throughput of <100 MB/s and a processing speed of 62 MB/s. Nothing has changed, aside from the version upgrade. My environment details are:

Primary Veeam Proxy: Dell R620 - dual quad core, 64GB RAM, Intel x520 NICs for both iSCSI and network
Primary Repository: Overland SnapServer DX2, 16GB RAM, Intel x520 NICs, using CIFS

Has anyone else noticed this? So far, I've not seen any problems, so I'm not sure if this is just a change in how the performance is reported.

[foggy]

Jon, there are some changes in how v7 calculates processing rate, please review this topic for more details.

[jim3cantos]

Note that changes in compression do not take effect until a new active full is created.

Is this also true for reversed incremental jobs?

[foggy]

Yes. Since with reversed incremental mode, changes are always written to the same backup file, it should be re-created for these compression/deduplication settings changes to take effect.

[leebtish]

Hi all,

We've recently upgraded to v7 which went smoothly. We're also running patch 1. We're seeing better dedupe stats now but backups are taking a bit longer which is disappointing. What I don't understand is the drop in processing rate performance in when I look at the jobs:

Here is the rate with v 6.5:

Duration 3:27:36
Processing Rate 278 MB/s
Processed 3.3 TB
Read 175.8 GB
Transferred 24.1 GB

Rate with v 7:

Duration 3:45:48
Processing rate: 39 MB/s
Processed 3.6 TB
Read 368.9 GB
Transferred 10.2 GB

I've read the other posts on the forums and I understand the processing rate calculation has changed, however, when I do the maths, the processing rate under version 7 doesn't add up (unless of course I have the calculation wrong). I'm not overly concerned as the backups are taking a similar amount of time but would like to know if I'm interpreting it incorrectly.

Just wanted to confirm if the figures above for v7 look correct. Going by my calculations in v7, it took 13,548 secs to process 3774873 MB = 279 Mb/s

Any ideas on this one ?

Thanks!

[foggy]

Lee, processing rate in v7 is calculated off the actually read data and the time it took to transfer this data, rather than the entire processed data and duration of the job you are currently using in your calculations. Thanks.

[Gostev]

We're seeing better dedupe stats now but backups are taking a bit longer which is disappointing.

Here is the rate with v 6.5:

Duration 3:27:36
Processing Rate 278 MB/s
Processed 3.3 TB
Read 175.8 GB
Transferred 24.1 GB

Rate with v 7:

Duration 3:45:48
Processing rate: 39 MB/s
Processed 3.6 TB
Read 368.9 GB
Transferred 10.2 GB

According to these stats, your backup job runs almost 2 times faster with v7. The job takes about same time to process twice as much data! Note the Read counters, it looks like your environment generates much more changes these days...

[leebtish]

Hi Gostev / Foggy,

Thanks for clearing that up. Wasn't sure if it was calculated on the read data or the entire processed data. Makes sense now, I can see from the read data we're getting better performance with v7.

We've just added more production servers so yep, data has increased a fair bit.

Thanks again.

[foggy]

You're welcome. Feel free to ask any additional questions if needed.

[ddockter]

I've noticed something a bit unusual since the upgrade to 7.0. The performance rates for my backups as shown in Enterprise Manager have decreased dramatically, but my backups are not taking significantly longer (I have yet to run full backups since the upgrade so don't know how much they will be impacted). As an example prior to the upgrade the rates for my incrementals were around 300 MB/s. After the upgrade it is about 72 MB/s. The odd thing is this is about the rate Enterprise Manager was showing for full backups (72 MB/s) prior to the 7.0 upgrade. The only thing I have changed on my backups is switching from no compression to dedup-friendly compression.

[foggy]

Doug, the processing rate calculation algorithm has changed in v7, please review this thread for details.