Host-based backup of Microsoft Hyper-V VMs.
Post Reply
MCU_Networking
Enthusiast
Posts: 38
Liked: 5 times
Joined: May 02, 2016 10:21 pm
Full Name: MCU Networking
Contact:

Encryption and Storage setup environment review

Post by MCU_Networking »

Case #05025706

We are getting ready to make some major changes to our storage and encryption setup(non-existent right now) and I would like someone to review my settings real quick to tell me if there are any recommended changes. I will try and add as much detail here as possible in order to help your side get a head start or possibly not have to remote in and have the ability to answer from here.

We have the following repositories:
Repo1 --> 10.9TB Capacity --> 507GB Free
Repo2 --> 7.6TB Capacity --> 3.7TB Free
Repo3 --> 10.9TB Capacity --> 2.6TB Free
Repo4 --> 10.9TB Capacity --> 3.5TB Free (Location 2)

-On all 4 Repos, only the "Align backup file..." box is checked under "Storage Compatibility Settings".
-On all Backup Jobs, we have Compression level set to "Extreme" and Storage Optimization set to "WAN target" or "LAN target". Unknown at this time why some are set to WAN and some are LAN. Most are WAN.
-On all Backup Jobs, on the Storage Tab, all 3 boxes are checked at the top(Enable inline, Exclude Swap, Exclude deleted).
-On all Backup Copy Jobs the compression is set to Extreme, except for 1 which is set to Auto.
-No encryption is enabled except for backup config.
-We have a 200Mbps MLPS pipe between the two sites for a copy job. In our firewall we shape the bandwidth down to 50Mbps, and in the Network Config of Veeam, we also set it to 50Mbps non-business hours and 25Mbps during business hours.
-We have 3 Source WAN Accelerators and 2 Target WAN Accelerators(At Location 2).
-We have about 50 workloads, 45 of which are virtual, 5 physical.
-Backup job times are 5-20 minutes for 10 of the 14 jobs, then 32 mins, 45 mins and 2x at 90 mins.
-Backup Copy job times are 8,7,8,5,4,5,2 and 4 hours. Not really sure how to give more info on the make up of the Copy Jobs.


We are in year 5 with the main Veeam server right now(Repo 2), so we are looking to upgrade storage here shortly. Our main goals/ideas are:
-Enable encryption
-Improve backup time if we can. We know lowering the compression and Deduping will do so, but we don't quite know at what expense and how that will affect the Copy jobs and WAN Acceleration.
-Moving the Veeam Mgmt Console from the physical that it is on right now, to a virtual machine, isolating it from storage.
-Implement Hardened Repos, a Backup Appliance or a combo of both.
-And of course, add more backup space, whatever the end storage purchase may be



Can anyone who has knowledge of a similar setup/environment provide any info based on a recent change you made or your current setup that differs?
soncscy
Veteran
Posts: 643
Liked: 312 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey
Contact:

Re: Encryption and Storage setup environment review

Post by soncscy »

Hi Michael,

While the details are great, you're missing the two most important details to comment on your environment:

1. I/O performance from Diskspd for different Veeam operations as per https://www.veeam.com/kb2014. Test all these just to see what your storage can do; use a larger amount of data to get a good solid baseline
2. Filesystem on the repositories

Encryption for Veeam is transparent so you should see no impact turning it on/off except that you need to run an active full afterwards.

For your backup times, what is the bottleneck listed? Without knowing that you can't know where to focus! :)

Hardened Repositories are a good choice! I suggest Debian or Ubuntu, be sure to check that it's a supported release, and then just keep it simple! You should have a really good time with it.
MCU_Networking
Enthusiast
Posts: 38
Liked: 5 times
Joined: May 02, 2016 10:21 pm
Full Name: MCU Networking
Contact:

Re: Encryption and Storage setup environment review

Post by MCU_Networking »

NTFS with Default 4k allocation, not sure if that is what you requested or not. For the diskspd results, should I post the entire results of each command or some specific information from each?
MCU_Networking
Enthusiast
Posts: 38
Liked: 5 times
Joined: May 02, 2016 10:21 pm
Full Name: MCU Networking
Contact:

Re: Encryption and Storage setup environment review

Post by MCU_Networking »

Ran these on the slowest Backup Server we have of the 3. Running 6 4TB 7.2K drivers in a RAID 10.

Here are the jobs that backup to it and their Bottlenecks:
Job 1 (1 physical about 1.6TB processed) > Target (65%)
Job 2 (1 virtual about 500GB processed) > Target (96%)
Job 3 (virtuals about 750GB processed) > Target (98%)
Job 4 (1 virtual about 800GB processed) > Target (96%)
Job 5 (virtuals about 500GB processed) > Target (96%)

Code: Select all

##################################################################
Results of diskspd.exe -c25G -b512K -w100 -Sh -d600 testfile.dat 
##################################################################
D:\Temp\DiskSpd\amd64>diskspd.exe -c25G -b512K -w100 -Sh -d600 testfile.dat

Command Line: diskspd.exe -c25G -b512K -w100 -Sh -d600 testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 600s
        warm up time: 5s
        cool down time: 0s
        random seed: 0
        path: 'testfile.dat'
                think time: 0ms
                burst size: 0
                software cache disabled
                hardware write cache disabled, writethrough on
                performing write test
                block size: 524288
                using sequential I/O (stride: 524288)
                number of outstanding I/O operations: 2
                thread stride size: 0
                threads per file: 1
                using I/O Completion Ports
                IO priority: normal

System information:

        computer name: VeeamServer2
        start time: 2021/09/30 18:13:08 UTC

Results for timespan 1:
*******************************************************************************

actual test time:       600.00s
thread count:           1
proc count:             16

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   0.87%|   0.07%|    0.80%|  99.13%
   1|   0.02%|   0.02%|    0.00%|  99.98%
   2|   0.14%|   0.04%|    0.10%|  99.86%
   3|   0.05%|   0.04%|    0.02%|  99.95%
   4|   0.14%|   0.02%|    0.12%|  99.86%
   5|   0.06%|   0.02%|    0.04%|  99.94%
   6|   0.08%|   0.03%|    0.05%|  99.92%
   7|   0.05%|   0.01%|    0.04%|  99.95%
   8|   0.11%|   0.03%|    0.08%|  99.89%
   9|   0.08%|   0.01%|    0.07%|  99.92%
  10|   0.08%|   0.02%|    0.06%|  99.92%
  11|   0.05%|   0.01%|    0.04%|  99.95%
  12|   0.11%|   0.05%|    0.07%|  99.89%
  13|   0.02%|   0.01%|    0.01%|  99.98%
  14|   0.22%|   0.07%|    0.14%|  99.78%
  15|   0.04%|   0.03%|    0.01%|  99.96%
-------------------------------------------
avg.|   0.13%|   0.03%|    0.10%|  99.87%

Total IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     69209686016 |       132007 |     110.01 |     220.01 | testfile.dat
 (25GiB)
------------------------------------------------------------------------------
total:       69209686016 |       132007 |     110.01 |     220.01

Read IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 | testfile.dat
 (25GiB)
------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00

Write IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     69209686016 |       132007 |     110.01 |     220.01 | testfile.dat
 (25GiB)
------------------------------------------------------------------------------
total:       69209686016 |       132007 |     110.01 |     220.01

D:\Temp\DiskSpd\amd64>





##########################################################################
Results from: diskspd.exe -c100G -b512K -w50 -r4K -Sh -d600 testfile.dat
##########################################################################
D:\Temp\DiskSpd\amd64>diskspd.exe -c100G -b512K -w50 -r4K -Sh -d600 testfile.dat


Command Line: diskspd.exe -c100G -b512K -w50 -r4K -Sh -d600 testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 600s
        warm up time: 5s
        cool down time: 0s
        random seed: 0
        path: 'testfile.dat'
                think time: 0ms
                burst size: 0
                software cache disabled
                hardware write cache disabled, writethrough on
                performing mix test (read/write ratio: 50/50)
                block size: 524288
                using random I/O (alignment: 4096)
                number of outstanding I/O operations: 2
                thread stride size: 0
                threads per file: 1
                using I/O Completion Ports
                IO priority: normal

System information:

        computer name: VeeamServer2
        start time: 2021/09/30 18:27:58 UTC

Results for timespan 1:
*******************************************************************************

actual test time:       600.00s
thread count:           1
proc count:             16

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   0.46%|   0.02%|    0.43%|  99.54%
   1|   0.01%|   0.01%|    0.00%|  99.99%
   2|   0.06%|   0.01%|    0.04%|  99.94%
   3|   0.02%|   0.01%|    0.01%|  99.98%
   4|   0.03%|   0.00%|    0.03%|  99.97%
   5|   0.00%|   0.00%|    0.00%| 100.00%
   6|   0.02%|   0.00%|    0.02%|  99.98%
   7|   0.01%|   0.00%|    0.01%|  99.99%
   8|   0.06%|   0.00%|    0.05%|  99.94%
   9|   0.02%|   0.01%|    0.01%|  99.98%
  10|   0.04%|   0.00%|    0.03%|  99.96%
  11|   0.02%|   0.01%|    0.01%|  99.98%
  12|   0.04%|   0.01%|    0.03%|  99.96%
  13|   0.08%|   0.02%|    0.06%|  99.92%
  14|   0.02%|   0.01%|    0.02%|  99.98%
  15|   0.01%|   0.01%|    0.00%|  99.99%
-------------------------------------------
avg.|   0.05%|   0.01%|    0.05%|  99.95%

Total IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     38910558208 |        74216 |      61.85 |     123.69 | testfile.dat
 (100GiB)
------------------------------------------------------------------------------
total:       38910558208 |        74216 |      61.85 |     123.69

Read IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     19441647616 |        37082 |      30.90 |      61.80 | testfile.dat
 (100GiB)
------------------------------------------------------------------------------
total:       19441647616 |        37082 |      30.90 |      61.80

Write IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     19468910592 |        37134 |      30.94 |      61.89 | testfile.dat
 (100GiB)
------------------------------------------------------------------------------
total:       19468910592 |        37134 |      30.94 |      61.89

D:\Temp\DiskSpd\amd64>





#########################################################################
Results from: diskspd.exe -c100G -b512K -w67 -r4K -Sh -d600 testfile.dat
#########################################################################
D:\Temp\DiskSpd\amd64>diskspd.exe -c100G -b512K -w67 -r4K -Sh -d600 testfile.dat


Command Line: diskspd.exe -c100G -b512K -w67 -r4K -Sh -d600 testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 600s
        warm up time: 5s
        cool down time: 0s
        random seed: 0
        path: 'testfile.dat'
                think time: 0ms
                burst size: 0
                software cache disabled
                hardware write cache disabled, writethrough on
                performing mix test (read/write ratio: 33/67)
                block size: 524288
                using random I/O (alignment: 4096)
                number of outstanding I/O operations: 2
                thread stride size: 0
                threads per file: 1
                using I/O Completion Ports
                IO priority: normal

System information:

        computer name: VeeamServer2
        start time: 2021/09/30 18:39:23 UTC

Results for timespan 1:
*******************************************************************************

actual test time:       600.00s
thread count:           1
proc count:             16

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   0.25%|   0.01%|    0.24%|  99.75%
   1|   0.00%|   0.00%|    0.00%| 100.00%
   2|   0.04%|   0.01%|    0.03%|  99.96%
   3|   0.10%|   0.01%|    0.09%|  99.90%
   4|   0.03%|   0.00%|    0.03%|  99.97%
   5|   0.01%|   0.01%|    0.01%|  99.99%
   6|   0.04%|   0.01%|    0.03%|  99.96%
   7|   0.01%|   0.00%|    0.01%|  99.99%
   8|   0.11%|   0.02%|    0.09%|  99.89%
   9|   0.02%|   0.01%|    0.01%|  99.98%
  10|   0.02%|   0.00%|    0.02%|  99.98%
  11|   0.01%|   0.00%|    0.01%|  99.99%
  12|   0.02%|   0.00%|    0.02%|  99.98%
  13|   0.02%|   0.00%|    0.02%|  99.98%
  14|   0.02%|   0.01%|    0.01%|  99.98%
  15|   0.01%|   0.00%|    0.01%|  99.99%
-------------------------------------------
avg.|   0.04%|   0.00%|    0.04%|  99.96%

Total IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     24410849280 |        46560 |      38.80 |      77.60 | testfile.dat
 (100GiB)
------------------------------------------------------------------------------
total:       24410849280 |        46560 |      38.80 |      77.60

Read IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |      8095006720 |        15440 |      12.87 |      25.73 | testfile.dat
 (100GiB)
------------------------------------------------------------------------------
total:        8095006720 |        15440 |      12.87 |      25.73

Write IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     16315842560 |        31120 |      25.93 |      51.87 | testfile.dat
 (100GiB)
------------------------------------------------------------------------------
total:       16315842560 |        31120 |      25.93 |      51.87

D:\Temp\DiskSpd\amd64>





#######################################
Results from: diskspd.exe -Sh -d600 #1
#######################################
D:\Temp\DiskSpd\amd64>diskspd.exe -Sh -d600 #1

Command Line: diskspd.exe -Sh -d600 #1

Input parameters:

        timespan:   1
        -------------
        duration: 600s
        warm up time: 5s
        cool down time: 0s
        random seed: 0
        path: '#1'
                think time: 0ms
                burst size: 0
                software cache disabled
                hardware write cache disabled, writethrough on
                performing read test
                block size: 65536
                using sequential I/O (stride: 65536)
                number of outstanding I/O operations: 2
                thread stride size: 0
                threads per file: 1
                using I/O Completion Ports
                IO priority: normal

System information:

        computer name: VeeamServer2
        start time: 2021/09/30 18:55:51 UTC

Results for timespan 1:
*******************************************************************************

actual test time:       600.01s
thread count:           1
proc count:             16

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|  11.26%|   0.42%|   10.84%|  88.74%
   1|   0.00%|   0.00%|    0.00%| 100.00%
   2|   0.55%|   0.00%|    0.55%|  99.45%
   3|   0.01%|   0.00%|    0.01%|  99.99%
   4|   0.41%|   0.00%|    0.41%|  99.59%
   5|   0.01%|   0.00%|    0.01%|  99.99%
   6|   0.27%|   0.00%|    0.27%|  99.73%
   7|   0.00%|   0.00%|    0.00%| 100.00%
   8|   1.31%|   0.01%|    1.30%|  98.69%
   9|   0.05%|   0.01%|    0.05%|  99.95%
  10|   0.78%|   0.01%|    0.77%|  99.22%
  11|   0.02%|   0.00%|    0.02%|  99.98%
  12|   0.21%|   0.00%|    0.21%|  99.79%
  13|   0.25%|   0.07%|    0.18%|  99.75%
  14|   0.77%|   0.00%|    0.76%|  99.23%
  15|   0.46%|   0.00%|    0.46%|  99.54%
-------------------------------------------
avg.|   1.02%|   0.03%|    0.99%|  98.98%

Total IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |    387653632000 |      5915125 |     616.15 |    9858.36 | #1 (11176GiB)
------------------------------------------------------------------------------
total:      387653632000 |      5915125 |     616.15 |    9858.36

Read IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |    387653632000 |      5915125 |     616.15 |    9858.36 | #1 (11176GiB)
------------------------------------------------------------------------------
total:      387653632000 |      5915125 |     616.15 |    9858.36

Write IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 | #1 (11176GiB)
------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00

D:\Temp\DiskSpd\amd64>
Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests