Case #05025706
We are getting ready to make some major changes to our storage and encryption setup(non-existent right now) and I would like someone to review my settings real quick to tell me if there are any recommended changes. I will try and add as much detail here as possible in order to help your side get a head start or possibly not have to remote in and have the ability to answer from here.
We have the following repositories:
Repo1 --> 10.9TB Capacity --> 507GB Free
Repo2 --> 7.6TB Capacity --> 3.7TB Free
Repo3 --> 10.9TB Capacity --> 2.6TB Free
Repo4 --> 10.9TB Capacity --> 3.5TB Free (Location 2)
-On all 4 Repos, only the "Align backup file..." box is checked under "Storage Compatibility Settings".
-On all Backup Jobs, we have Compression level set to "Extreme" and Storage Optimization set to "WAN target" or "LAN target". Unknown at this time why some are set to WAN and some are LAN. Most are WAN.
-On all Backup Jobs, on the Storage Tab, all 3 boxes are checked at the top(Enable inline, Exclude Swap, Exclude deleted).
-On all Backup Copy Jobs the compression is set to Extreme, except for 1 which is set to Auto.
-No encryption is enabled except for backup config.
-We have a 200Mbps MLPS pipe between the two sites for a copy job. In our firewall we shape the bandwidth down to 50Mbps, and in the Network Config of Veeam, we also set it to 50Mbps non-business hours and 25Mbps during business hours.
-We have 3 Source WAN Accelerators and 2 Target WAN Accelerators(At Location 2).
-We have about 50 workloads, 45 of which are virtual, 5 physical.
-Backup job times are 5-20 minutes for 10 of the 14 jobs, then 32 mins, 45 mins and 2x at 90 mins.
-Backup Copy job times are 8,7,8,5,4,5,2 and 4 hours. Not really sure how to give more info on the make up of the Copy Jobs.
We are in year 5 with the main Veeam server right now(Repo 2), so we are looking to upgrade storage here shortly. Our main goals/ideas are:
-Enable encryption
-Improve backup time if we can. We know lowering the compression and Deduping will do so, but we don't quite know at what expense and how that will affect the Copy jobs and WAN Acceleration.
-Moving the Veeam Mgmt Console from the physical that it is on right now, to a virtual machine, isolating it from storage.
-Implement Hardened Repos, a Backup Appliance or a combo of both.
-And of course, add more backup space, whatever the end storage purchase may be
Can anyone who has knowledge of a similar setup/environment provide any info based on a recent change you made or your current setup that differs?
-
MCU_Networking
- Enthusiast
- Posts: 38
- Liked: 5 times
- Joined: May 02, 2016 10:21 pm
- Full Name: MCU Networking
- Contact:
-
soncscy
- Veteran
- Posts: 643
- Liked: 312 times
- Joined: Aug 04, 2019 2:57 pm
- Full Name: Harvey
- Contact:
Re: Encryption and Storage setup environment review
Hi Michael,
While the details are great, you're missing the two most important details to comment on your environment:
1. I/O performance from Diskspd for different Veeam operations as per https://www.veeam.com/kb2014. Test all these just to see what your storage can do; use a larger amount of data to get a good solid baseline
2. Filesystem on the repositories
Encryption for Veeam is transparent so you should see no impact turning it on/off except that you need to run an active full afterwards.
For your backup times, what is the bottleneck listed? Without knowing that you can't know where to focus!
Hardened Repositories are a good choice! I suggest Debian or Ubuntu, be sure to check that it's a supported release, and then just keep it simple! You should have a really good time with it.
While the details are great, you're missing the two most important details to comment on your environment:
1. I/O performance from Diskspd for different Veeam operations as per https://www.veeam.com/kb2014. Test all these just to see what your storage can do; use a larger amount of data to get a good solid baseline
2. Filesystem on the repositories
Encryption for Veeam is transparent so you should see no impact turning it on/off except that you need to run an active full afterwards.
For your backup times, what is the bottleneck listed? Without knowing that you can't know where to focus!
Hardened Repositories are a good choice! I suggest Debian or Ubuntu, be sure to check that it's a supported release, and then just keep it simple! You should have a really good time with it.
-
MCU_Networking
- Enthusiast
- Posts: 38
- Liked: 5 times
- Joined: May 02, 2016 10:21 pm
- Full Name: MCU Networking
- Contact:
Re: Encryption and Storage setup environment review
NTFS with Default 4k allocation, not sure if that is what you requested or not. For the diskspd results, should I post the entire results of each command or some specific information from each?
-
MCU_Networking
- Enthusiast
- Posts: 38
- Liked: 5 times
- Joined: May 02, 2016 10:21 pm
- Full Name: MCU Networking
- Contact:
Re: Encryption and Storage setup environment review
Ran these on the slowest Backup Server we have of the 3. Running 6 4TB 7.2K drivers in a RAID 10.
Here are the jobs that backup to it and their Bottlenecks:
Job 1 (1 physical about 1.6TB processed) > Target (65%)
Job 2 (1 virtual about 500GB processed) > Target (96%)
Job 3 (virtuals about 750GB processed) > Target (98%)
Job 4 (1 virtual about 800GB processed) > Target (96%)
Job 5 (virtuals about 500GB processed) > Target (96%)
Here are the jobs that backup to it and their Bottlenecks:
Job 1 (1 physical about 1.6TB processed) > Target (65%)
Job 2 (1 virtual about 500GB processed) > Target (96%)
Job 3 (virtuals about 750GB processed) > Target (98%)
Job 4 (1 virtual about 800GB processed) > Target (96%)
Job 5 (virtuals about 500GB processed) > Target (96%)
Code: Select all
##################################################################
Results of diskspd.exe -c25G -b512K -w100 -Sh -d600 testfile.dat
##################################################################
D:\Temp\DiskSpd\amd64>diskspd.exe -c25G -b512K -w100 -Sh -d600 testfile.dat
Command Line: diskspd.exe -c25G -b512K -w100 -Sh -d600 testfile.dat
Input parameters:
timespan: 1
-------------
duration: 600s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing write test
block size: 524288
using sequential I/O (stride: 524288)
number of outstanding I/O operations: 2
thread stride size: 0
threads per file: 1
using I/O Completion Ports
IO priority: normal
System information:
computer name: VeeamServer2
start time: 2021/09/30 18:13:08 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 600.00s
thread count: 1
proc count: 16
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 0.87%| 0.07%| 0.80%| 99.13%
1| 0.02%| 0.02%| 0.00%| 99.98%
2| 0.14%| 0.04%| 0.10%| 99.86%
3| 0.05%| 0.04%| 0.02%| 99.95%
4| 0.14%| 0.02%| 0.12%| 99.86%
5| 0.06%| 0.02%| 0.04%| 99.94%
6| 0.08%| 0.03%| 0.05%| 99.92%
7| 0.05%| 0.01%| 0.04%| 99.95%
8| 0.11%| 0.03%| 0.08%| 99.89%
9| 0.08%| 0.01%| 0.07%| 99.92%
10| 0.08%| 0.02%| 0.06%| 99.92%
11| 0.05%| 0.01%| 0.04%| 99.95%
12| 0.11%| 0.05%| 0.07%| 99.89%
13| 0.02%| 0.01%| 0.01%| 99.98%
14| 0.22%| 0.07%| 0.14%| 99.78%
15| 0.04%| 0.03%| 0.01%| 99.96%
-------------------------------------------
avg.| 0.13%| 0.03%| 0.10%| 99.87%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 69209686016 | 132007 | 110.01 | 220.01 | testfile.dat
(25GiB)
------------------------------------------------------------------------------
total: 69209686016 | 132007 | 110.01 | 220.01
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | testfile.dat
(25GiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 69209686016 | 132007 | 110.01 | 220.01 | testfile.dat
(25GiB)
------------------------------------------------------------------------------
total: 69209686016 | 132007 | 110.01 | 220.01
D:\Temp\DiskSpd\amd64>
##########################################################################
Results from: diskspd.exe -c100G -b512K -w50 -r4K -Sh -d600 testfile.dat
##########################################################################
D:\Temp\DiskSpd\amd64>diskspd.exe -c100G -b512K -w50 -r4K -Sh -d600 testfile.dat
Command Line: diskspd.exe -c100G -b512K -w50 -r4K -Sh -d600 testfile.dat
Input parameters:
timespan: 1
-------------
duration: 600s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 524288
using random I/O (alignment: 4096)
number of outstanding I/O operations: 2
thread stride size: 0
threads per file: 1
using I/O Completion Ports
IO priority: normal
System information:
computer name: VeeamServer2
start time: 2021/09/30 18:27:58 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 600.00s
thread count: 1
proc count: 16
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 0.46%| 0.02%| 0.43%| 99.54%
1| 0.01%| 0.01%| 0.00%| 99.99%
2| 0.06%| 0.01%| 0.04%| 99.94%
3| 0.02%| 0.01%| 0.01%| 99.98%
4| 0.03%| 0.00%| 0.03%| 99.97%
5| 0.00%| 0.00%| 0.00%| 100.00%
6| 0.02%| 0.00%| 0.02%| 99.98%
7| 0.01%| 0.00%| 0.01%| 99.99%
8| 0.06%| 0.00%| 0.05%| 99.94%
9| 0.02%| 0.01%| 0.01%| 99.98%
10| 0.04%| 0.00%| 0.03%| 99.96%
11| 0.02%| 0.01%| 0.01%| 99.98%
12| 0.04%| 0.01%| 0.03%| 99.96%
13| 0.08%| 0.02%| 0.06%| 99.92%
14| 0.02%| 0.01%| 0.02%| 99.98%
15| 0.01%| 0.01%| 0.00%| 99.99%
-------------------------------------------
avg.| 0.05%| 0.01%| 0.05%| 99.95%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 38910558208 | 74216 | 61.85 | 123.69 | testfile.dat
(100GiB)
------------------------------------------------------------------------------
total: 38910558208 | 74216 | 61.85 | 123.69
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 19441647616 | 37082 | 30.90 | 61.80 | testfile.dat
(100GiB)
------------------------------------------------------------------------------
total: 19441647616 | 37082 | 30.90 | 61.80
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 19468910592 | 37134 | 30.94 | 61.89 | testfile.dat
(100GiB)
------------------------------------------------------------------------------
total: 19468910592 | 37134 | 30.94 | 61.89
D:\Temp\DiskSpd\amd64>
#########################################################################
Results from: diskspd.exe -c100G -b512K -w67 -r4K -Sh -d600 testfile.dat
#########################################################################
D:\Temp\DiskSpd\amd64>diskspd.exe -c100G -b512K -w67 -r4K -Sh -d600 testfile.dat
Command Line: diskspd.exe -c100G -b512K -w67 -r4K -Sh -d600 testfile.dat
Input parameters:
timespan: 1
-------------
duration: 600s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 33/67)
block size: 524288
using random I/O (alignment: 4096)
number of outstanding I/O operations: 2
thread stride size: 0
threads per file: 1
using I/O Completion Ports
IO priority: normal
System information:
computer name: VeeamServer2
start time: 2021/09/30 18:39:23 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 600.00s
thread count: 1
proc count: 16
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 0.25%| 0.01%| 0.24%| 99.75%
1| 0.00%| 0.00%| 0.00%| 100.00%
2| 0.04%| 0.01%| 0.03%| 99.96%
3| 0.10%| 0.01%| 0.09%| 99.90%
4| 0.03%| 0.00%| 0.03%| 99.97%
5| 0.01%| 0.01%| 0.01%| 99.99%
6| 0.04%| 0.01%| 0.03%| 99.96%
7| 0.01%| 0.00%| 0.01%| 99.99%
8| 0.11%| 0.02%| 0.09%| 99.89%
9| 0.02%| 0.01%| 0.01%| 99.98%
10| 0.02%| 0.00%| 0.02%| 99.98%
11| 0.01%| 0.00%| 0.01%| 99.99%
12| 0.02%| 0.00%| 0.02%| 99.98%
13| 0.02%| 0.00%| 0.02%| 99.98%
14| 0.02%| 0.01%| 0.01%| 99.98%
15| 0.01%| 0.00%| 0.01%| 99.99%
-------------------------------------------
avg.| 0.04%| 0.00%| 0.04%| 99.96%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 24410849280 | 46560 | 38.80 | 77.60 | testfile.dat
(100GiB)
------------------------------------------------------------------------------
total: 24410849280 | 46560 | 38.80 | 77.60
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 8095006720 | 15440 | 12.87 | 25.73 | testfile.dat
(100GiB)
------------------------------------------------------------------------------
total: 8095006720 | 15440 | 12.87 | 25.73
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 16315842560 | 31120 | 25.93 | 51.87 | testfile.dat
(100GiB)
------------------------------------------------------------------------------
total: 16315842560 | 31120 | 25.93 | 51.87
D:\Temp\DiskSpd\amd64>
#######################################
Results from: diskspd.exe -Sh -d600 #1
#######################################
D:\Temp\DiskSpd\amd64>diskspd.exe -Sh -d600 #1
Command Line: diskspd.exe -Sh -d600 #1
Input parameters:
timespan: 1
-------------
duration: 600s
warm up time: 5s
cool down time: 0s
random seed: 0
path: '#1'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing read test
block size: 65536
using sequential I/O (stride: 65536)
number of outstanding I/O operations: 2
thread stride size: 0
threads per file: 1
using I/O Completion Ports
IO priority: normal
System information:
computer name: VeeamServer2
start time: 2021/09/30 18:55:51 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 600.01s
thread count: 1
proc count: 16
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 11.26%| 0.42%| 10.84%| 88.74%
1| 0.00%| 0.00%| 0.00%| 100.00%
2| 0.55%| 0.00%| 0.55%| 99.45%
3| 0.01%| 0.00%| 0.01%| 99.99%
4| 0.41%| 0.00%| 0.41%| 99.59%
5| 0.01%| 0.00%| 0.01%| 99.99%
6| 0.27%| 0.00%| 0.27%| 99.73%
7| 0.00%| 0.00%| 0.00%| 100.00%
8| 1.31%| 0.01%| 1.30%| 98.69%
9| 0.05%| 0.01%| 0.05%| 99.95%
10| 0.78%| 0.01%| 0.77%| 99.22%
11| 0.02%| 0.00%| 0.02%| 99.98%
12| 0.21%| 0.00%| 0.21%| 99.79%
13| 0.25%| 0.07%| 0.18%| 99.75%
14| 0.77%| 0.00%| 0.76%| 99.23%
15| 0.46%| 0.00%| 0.46%| 99.54%
-------------------------------------------
avg.| 1.02%| 0.03%| 0.99%| 98.98%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 387653632000 | 5915125 | 616.15 | 9858.36 | #1 (11176GiB)
------------------------------------------------------------------------------
total: 387653632000 | 5915125 | 616.15 | 9858.36
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 387653632000 | 5915125 | 616.15 | 9858.36 | #1 (11176GiB)
------------------------------------------------------------------------------
total: 387653632000 | 5915125 | 616.15 | 9858.36
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | #1 (11176GiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
D:\Temp\DiskSpd\amd64>
Who is online
Users browsing this forum: No registered users and 19 guests