WAN-Accelerated Backup Copy Job > Bottleneck = Source WAN?

[itfnb]

I'm testing a Backup Copy Job utilizing WAN Acceleration. When running the initial full, I was getting a bottleneck at the Source WAN, but with low usage on all of the stages. I decided to redo the Source WAN Accelerator (even though the cache was less than 5GB of the allotted 100GB), moving it from 7K2 RPM drives to 15K. This had no impact on performance that I could tell when rerunning the job from scratch. After running for 1.5 hrs. this go-around, the WAN Source is still the bottleneck with 2% usage and all of the other stages showing 0%.

Is my Source WAN accelerator REALLY that slow? I put that SSD in the Target, based on what I read here…that the Target WAN Accelerator is usually the bottleneck (other than a slow network connection of course). Should I add an SSD to my Source, too?

FWIW, the target (currently sitting on the same 1Gbit LAN) has a fully-populated 100GB of WAN Acceleration cache applies to the VMs in this Backup Copy Job (I've recreated the job). The Backup Job it's copying is approximately 2TB in .vbk size. I already have one GFS Backup Copy Job that's running on this data and it runs super fast (no WAN Acceleration), but this one will take days to run.

One interesting behavior is that that Veeam is sending data across the network for about 5-10 seconds every 2.5 minutes. Watching Resource Monitor, the Target's SSD cache and internal 7k2 RPM HDD repository are largely waiting for something to do.

Any ideas? Thanks!

[Gostev]

FWIW, the target (currently sitting on the same 1Gbit LAN) ... one interesting behavior is that that Veeam is sending data across the network for about 5-10 seconds every 2.5 minutes.

This is exactly why we state that you can only saturate up to 100 Mbps WAN on average hardware. Remember that we provide WAN, not LAN acceleration as such, the acceleration engine is tuned to provide the best possible data reduction while keeping up with typical WAN links bandwidth.

Source WAN Accelerator does not use cache as heavy as Target WAN Accelerator (only for storing digests), so its cache speed does not matter as much. I think what you see is somewhat expected, as Source WAN Accelerator needs to read and process a lot of data during the full backup. Anyway, you can find out what the most time is spent for from the debug logs. We have a very comprehensive bottleneck analysis there. Just look for the following section in the debug log, the labels are pretty self-explanatory:

Code: Select all

[17.07.2013 16:07:16] <  1820> srct| Performance statistics:
[17.07.2013 16:07:16] <  1820> srct| SourceDiskRead           10 sec
[17.07.2013 16:07:16] <  1820> srct| SourceWriteCmdFile       2 sec
[17.07.2013 16:07:16] <  1820> srct| SourceCompressCmdFile    11 sec
[17.07.2013 16:07:16] <  1820> srct| SourceDeleteCmdFile      0 sec
[17.07.2013 16:07:16] <  1820> srct| SendCmdFile              121 sec
[17.07.2013 16:07:16] <  1820> srct| SendWAIT                 1224 sec
[17.07.2013 16:07:16] <  1820> srct| TargetDiskRead           83 sec
[17.07.2013 16:07:16] <  1820> srct| TargetDiskWrite          9 sec
[17.07.2013 16:07:16] <  1820> srct| TargetReadCmdFile        3 sec
[17.07.2013 16:07:16] <  1820> srct| TargetDecompressCmdFile  3 sec
[17.07.2013 16:07:16] <  1820> srct| TargetDeleteCmdFile      0 sec
[17.07.2013 16:07:16] <  1820> srct| TargetWAITCmdFile        298 sec
[17.07.2013 16:07:16] <  1820> srct| TargetTotalCmdFileProcessing1033 sec
[17.07.2013 16:07:16] <  1820> srct| TargetGlobalInit         0 sec
[17.07.2013 16:07:16] <  1820> srct| TargetGlobalWrite        917 sec
[17.07.2013 16:07:16] <  1820> srct| TargetGlobalRead         6 sec
[17.07.2013 16:07:16] <  1820> srct| TargetGlobalCommit       0 sec
[17.07.2013 16:07:16] <  1820> srct| AlgDiskProcessing        754 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitialization        32 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitNonZeroExtents    0 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitCBT               0 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitDigests           8 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitGlobalDedup       20 sec
[17.07.2013 16:07:16] <  1820> srct| AlgFinalization          11 sec
[17.07.2013 16:07:16] <  1820> srct| AlgCommitGlobalDedup     8 sec
[17.07.2013 16:07:16] <  1820> srct| AlgWAIT                  541 sec
[17.07.2013 16:07:16] <  1820> srct| AlgBlockProcessing       741 sec
[17.07.2013 16:07:16] <  1820> srct| TASK COMPLETED SUCCESSFULLY, elapsed: 1349.5460 sec.

[itfnb]

Thanks! Is it possible to edit the job to be direct and then turn the WAN Acceleration back on? I was just trying to seed the job prior to moving it off-site. Thanks again! -Jim

[Gostev]

Interesting question... this might actually work, based on what I know.

[tsightler]

You can switch easily between direct and WAN accelerated mode. On the first pass after the seed when you turn WAN acceleration back on the target side will build its global cache from the data in the seeded repo. This will still take time, but very little bandwidth.

To get maximum performance from the WAN accelerator you need an SSD on both sides. It appears that the majority of I/O time is spend reading/writing hashes. You can somewhat compensate for this on the source if you put enough memory in the box so that the hashes are cached by the OS. This isn't really possible on the target since you would need enough memory for not just the hashes but also the global cache data. In the end though, about the best I've been able to get is around 50-70Mbps of sustained traffic, but my SSDs are super fast.

[VladV]

In the end though, about the best I've been able to get is around 50-70Mbps of sustained traffic, but my SSDs are super fast.

If possible can you share what type and size of SSDs have you used? Thanks

[tsightler]

Ha, that should have said "aren't super fast". Sorry about that. They're just consumer grade Samsung 840 Pro drives.

[dellock6]

From several sources, I've understood the best choice, if budget allows, is Intel DCS3700. Another good choice would also be DCS3500, same read performances but slower on writes, it was specifically designed for read-intensive workloads like caching. Would be slower during cache warming (write into the SSD) but almost as fast once the cache is ready.
Samsung are ok for consumer (Tom is using them in his lab...), but heard bad stories in terms of reliability and performance predicatbility, too spikes up and down.

Luca.

[tsightler]

Indeed I would not recommend the Samsung devices for a production deployment. The point I was trying to make is that selecting fast SSDs designed specifically for consistent low latency operations would be a much better option. That typo made this much less clear.

[dellock6]

100% agree. In general, every time i read "caching" I translate it to "nand memory": SAN cache, server caches, hypervisor caches, everytime they are built with NAND memory for best performances. Veeam WAN accelration mades no difference

[itfnb]

I've been quite pleased with our Intel DCS3700s. I bought a couple as SSD-caches for our DR vSphere host servers' RAID controllers. They made for a nice performance increase since they're in front of 3.5" 7k2 SAS drives in RAID 10.

I guess I'll have to pick up another one for my Source WAN Acceleration cache.

[koenhelsen123]

hi Gostev, can you tell me where to find that log exactly? It doesn't seem to be in C:\ProgramData\Veeam\Backup
thx!

This is exactly why we state that you can only saturate up to 100 Mbps WAN on average hardware. Remember that we provide WAN, not LAN acceleration as such, the acceleration engine is tuned to provide the best possible data reduction while keeping up with typical WAN links bandwidth.

Source WAN Accelerator does not use cache as heavy as Target WAN Accelerator (only for storing digests), so its cache speed does not matter as much. I think what you see is somewhat expected, as Source WAN Accelerator needs to read and process a lot of data during the full backup. Anyway, you can find out what the most time is spent for from the debug logs. We have a very comprehensive bottleneck analysis there. Just look for the following section in the debug log, the labels are pretty self-explanatory:

Code: Select all

[17.07.2013 16:07:16] <  1820> srct| Performance statistics:
[17.07.2013 16:07:16] <  1820> srct| SourceDiskRead           10 sec
[17.07.2013 16:07:16] <  1820> srct| SourceWriteCmdFile       2 sec
[17.07.2013 16:07:16] <  1820> srct| SourceCompressCmdFile    11 sec
[17.07.2013 16:07:16] <  1820> srct| SourceDeleteCmdFile      0 sec
[17.07.2013 16:07:16] <  1820> srct| SendCmdFile              121 sec
[17.07.2013 16:07:16] <  1820> srct| SendWAIT                 1224 sec
[17.07.2013 16:07:16] <  1820> srct| TargetDiskRead           83 sec
[17.07.2013 16:07:16] <  1820> srct| TargetDiskWrite          9 sec
[17.07.2013 16:07:16] <  1820> srct| TargetReadCmdFile        3 sec
[17.07.2013 16:07:16] <  1820> srct| TargetDecompressCmdFile  3 sec
[17.07.2013 16:07:16] <  1820> srct| TargetDeleteCmdFile      0 sec
[17.07.2013 16:07:16] <  1820> srct| TargetWAITCmdFile        298 sec
[17.07.2013 16:07:16] <  1820> srct| TargetTotalCmdFileProcessing1033 sec
[17.07.2013 16:07:16] <  1820> srct| TargetGlobalInit         0 sec
[17.07.2013 16:07:16] <  1820> srct| TargetGlobalWrite        917 sec
[17.07.2013 16:07:16] <  1820> srct| TargetGlobalRead         6 sec
[17.07.2013 16:07:16] <  1820> srct| TargetGlobalCommit       0 sec
[17.07.2013 16:07:16] <  1820> srct| AlgDiskProcessing        754 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitialization        32 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitNonZeroExtents    0 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitCBT               0 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitDigests           8 sec
[17.07.2013 16:07:16] <  1820> srct| AlgInitGlobalDedup       20 sec
[17.07.2013 16:07:16] <  1820> srct| AlgFinalization          11 sec
[17.07.2013 16:07:16] <  1820> srct| AlgCommitGlobalDedup     8 sec
[17.07.2013 16:07:16] <  1820> srct| AlgWAIT                  541 sec
[17.07.2013 16:07:16] <  1820> srct| AlgBlockProcessing       741 sec
[17.07.2013 16:07:16] <  1820> srct| TASK COMPLETED SUCCESSFULLY, elapsed: 1349.5460 sec.

[Gostev]

Actually, this thread is 6 years old so most things have changed by now quite dramatically. So, it would be best to contact support for assistance.