Backup over iscsi

[TheBob]

Hello together,
we have plain installed V Backup & Replication Standard with support on Windows 10
Windows has also to iscsi connection to our Storage "Storage_SAN" and "Backup_SAN".
Storag_SAN are located all VM under VMware vCenter 6.7 and Veeam Plugin is installed
Backup_SAN for the Backups
Both Storages are Huawei OceanStor and there are integrated on vCenter also as iSCSI
Also both Storages and the VM are on same IP 10GB Subnet.
I can see both SAN's storages can make it online or offline on Win DiskManager but not assing a dirve letter
Both Storages are under Veeam - Storage Infrastructure and i can see the LUN's

So my Problem is how can i configure that my Backups going from "Storage_SAN" to "Backup_SAN" SAN over iscsi?
I have reed many Articles found on goolge but did't find a procedure what are the next steps.
What are the configuration under Backup Repositorys and Backup Proxies?

Pls. help about that
many thanks
TheBob

[Egor Yakovlev]

Scenario you are looking to achieve can be done with different approaches:
- Simple Direct SAN Access transport mode will read data directly from production LUNs(using iSCSI mounted LUNs) and transfer data to the target repository(iSCSI mounted target).
- Advanced Backup from SAN Snapshots - this will do the same but will use Huawei's LUN Snapshot as a source, reducing VM snapshot lifetime to mere seconds.
Check links for configuration guidance, and both options should work automatically if all required conditions are met. Repository will require no additional configurations, as for the Proxy - You can force Direct SAN Access in Backup Proxy settings.
/Thanks!

[foggy]

Here's the information on proxy server configuration for using backup from storage snapshots (this is the preferred method). As for the repository, then you need to add the same server as a backup repository pointing to the LUN on the second storage.

[TheBob]

Hello Egor and foggy,
thanks alot for your postings.
I have read all the information but there is no dokumentation how to do that like step by step.
Is there no documentation how to configure Direct SAN Access from Step by Step?

Some more infos from my Backup Server:
Nic 1: for Management 192.168.1.30
Nic 2: for SAN Connection 192.168.2.30
OS: Windnows 10
iSCSCI attached Storage "Storage-SAN = 192.168.2.10" and "Backup-SAN 192.168.2.20"

I have configured Backup Proxy:
The win10 over Nic1; Transport Mode = Direct storage Access; Connected Datastores = Auto
Traffic Rules = Internet; VM-SAN = 192.168.2.1 - 63
I have configured Backup Repositories:
1x win10/Nic2 to Backup-SAN over NFS3
1x win10/Nic2 to D:\Backups
I have no chane to select any iSCSI Luns or Volumes

I have configured Storage Infrastucture:
1x Huawei Storage-SAN with the LUN's over iSCSI
1x Huawei Backup-SAN with the LUN's over iSCSI

I have configured Backup Job:
Backup VM; Backup proxy = Auto; Backup repository = Backup-SAN

Is there any thing wrong in my configuration?

My Test-backup are running but the Speed is max. 332 MB/s which it looks quit low for 10GB and/or direct SAN Access or not?

thank you very much for helping!
best regards TheBob

[Egor Yakovlev]

There is no step-by-step guidance because it is hardware specific and shall be done according to your SAN storage vendor guide. What Veeam needs, is a production LUN visible under Proxy's Windows Disk Manager(offline, non-initialized) with Read Access on the LUN.
As for the performance, You can start by checking:
1. current used transport mode: in the job log, select a VM and find a line "Using Backup Proxy [Name] for Disk N [XXX]" where [XXX] will be currently used transport mode, [nbd] means network mode, [hotadd] means virtual appliance mode, [san] means direct san access mode.
2. job bottleneck statistics at the end. High percentage value(around 90%) usually points at lagging component that slows overall backup process.
/Cheers!

[TheBob]

Many thank's Egor,

This my Output from Enterprise Manager for a TestBackup:

Code: Select all

Queued for processing at 21.04.2020 15:54:08
Required backup infrastructure resources have been assigned
VM processing started at 21.04.2020 15:54:14
VM size: 20 GB (14,8 GB used)
Resetting CBT per job settings for active fulls
Getting VM info from vSphere
Creating VM snapshot
Saving [Storage_SAN] SUBU-TERM-R09-02/SUBU-TERM-R09-02.vmx
Saving [Storage_SAN] SUBU-TERM-R09-02/SUBU-TERM-R09-02.vmxf
Saving [Storage_SAN] SUBU-TERM-R09-02/SUBU-TERM-R09-02.nvram
Using backup proxy VMware Backup Proxy for disk Hard disk 1 [san]
Hard disk 1 (20 GB)
Removing VM snapshot
Finalizing
Busy: Source 30% > Proxy 66% > Network 33% > Target 74%
Primary bottleneck: Target
Network traffic verification detected no corrupted blocks
Processing finished at 21.04.2020 15:58:09

If it's possible i would like to send the Log for that backup to via Email to check if any thing is right and good configured?

Many Thank
TheBob

[Egor Yakovlev]

It is Direct SAN Access mode after all! I would check Proxy's CPU and RAM during backup - with Software iSCSI initiator that might be a bottleneck resource as you have source and target mounted on same server. Target is getting close to become a bottleneck(ideal case if it is!), so worth giving it a stress run with DiskSPD tool to see it's potential limit.
/Cheers!

[TheBob]

Hello Egor,

here is one output of DiskSPD

Storage Performance Report - W10-VEEAM-RV4
Generated on 23.Apr.2020 at 10:05:18

Code: Select all

Performance Tests
  
Name	         Path	  Block size	 Operation	 Read / Write	 Outstanding  IO	Threads	       IOPS	 MB/sec	 Latency	 CPU %
VM-Backup-SAN	X:\SpeedTest	8K	Random    	70% Read / 30% Write	8	             1	  56586.82	442.08	0.141	        35.52%
 

I need to use SMB for the test
The Backup Server has 16GB RAM and 4 CPU

[Egor Yakovlev]

- Default block size for iSCSI target would be "local" type 1M chunk, which after compression goes roughly to 512K. Your test was done on 8K block size.
- Make sure to select proper read\write balance, because on backups like Forward Incremental there are no reads from Repository, only Write IOs.
/Cheers!

[TheBob]

If i using 512 or 1024 the speed is much more:

Code: Select all

Performance Tests 512K
Name	Path	Block size	Operation	Read / Write	Outstanding IO	Threads	IOPS	MB/sec	Latency	CPU %
VM-Backup-SAN	X:\SpeedTest	512K	Random	70% Read / 30% Write	8	1	4946.74	2473.37	1.616	33.32%
   
Performance Tests 1024K
Name	Path	Block size	Operation	Read / Write	Outstanding IO	Threads	IOPS	MB/sec	Latency	CPU %
VM-Backup-SAN	X:\SpeedTest	1024K	Random	70% Read / 30% Write	8	1	2516.58	2516.58	3.181	29.69%
 

but how can i test the target / source the vm what i would like to backup?
That test is only for the speed from Backup server over smb to Backup-Storage.
And is there another problem why my speed is not high enough if i make a backup?

Many thank's for helping!

[Egor Yakovlev]

You can try grouping multiple VMs in a single Backup Job and raise Proxy CPU count up(say, 12?) to push more load on your infrastructure.
/Thanks!

[TheBob]

Thank you Egor!
i have already grouping my VM's. On next Weekend i run 2 Job's with 3 VM's in one Job and 7 VM's on other one.
The CPU count is no Problem, i will do that today.
Tell you on monday the Status!
Thank you very much for helping!

[TheBob]

GM again,

here some information about the 2 Job's

Code: Select all

Job Sunday Morning (Full) Project Systems:
Success	7	Start time	01:00:10	Total size	2,3 TB		Backup size	408,3 GB
Warning	0	End time	02:57:40	Data read	531,3 GB	Dedupe	4,5x
Error	0	Duration	1:57:30 	Transferred	409,1 GB	Compression	1,3x

Name		Status		Start time		End time		Size		Read		Transferred		Duration
S1		Success		01:00:29		01:35:04		64 GB		64 GB		29,6 GB			0:34:35 
S2		Success		01:00:29		01:09:09		16 GB		5,9 GB		2,7 GB			0:08:40 
S3		Success		01:00:29		02:21:32		750 GB		174,9 GB	149,8 GB		1:21:02 
S4		Success		01:00:29		01:20:38		64 GB		15,4 GB		10,2 GB			0:20:09 
S5		Success		01:09:11		01:59:13		200 GB		48 GB		34,5 GB			0:50:02 
S6		Success		01:20:39		02:02:27		200 GB		37,4 GB		29,3 GB			0:41:47 
S7		Success		01:35:07		02:57:32		1024 GB		185,6 GB		153 GB			1:22:24 

Processing rate: 79 MB/s
THROUGHPUT Speed: 333 MB/s

Code: Select all

Job Sunday Night (Full) Terminal Servers:
Success	3	Start time	22:00:04	Total size	1,1 TB		Backup size	118,9 GB
Warning	0	End time	22:38:40	Data read	353,8 GB	Dedupe	4,5x
Error	0	Duration	0:38:36 	Transferred	125 GB		Compression	2,0x

Name		Status		Start time		End time		Size		Read		Transferred		Duration
T1		Success		22:00:28		22:24:16		50 GB		48,4 GB			19 GB		0:23:47 
T2		Success		22:00:28		22:32:44		250 GB		183,6 GB		43,1 GB		0:32:15 
T3		Success		22:00:28		22:38:33		800 GB		121,8 GB		62,8 GB		0:38:04 

Processing rate: 164 MB/s
THROUGHPUT Speed: 442 MB/s

Is there any thing what can i do better?
Btw. how can i add images to my postings?

Thank's in advanced

[Egor Yakovlev]

Hi Bob!
Stats do look good, what were Bottlenecks for those? I can see servers S6 and S7 were waiting 20 and 35 minutes respectfully doing nothing, most likely because of Proxy CPU count (it can process 1 vmdk per core in parallel) or Backup Repository task limit (check Repo settings).
/Cheers!

[TheBob]

Hi Egor,
i have checked my Repo and the "Limit maximum concurrent task to" was 4.
I have changed it now to 10, maximum number of VM's in one Job is 8.
so mybe that one was the problem?

Thank's alot for helping!
BR Bob

[TheBob]

Hi Egor,
here are some information from last Weekend with 3 Full Backup Job's, from my site 2 are looking good but one is a little bit slow.

Code: Select all

Job Friday (Full) Terminal Servers:

Success	3	Start time	23:30:04	Total size	1,1 TB		Backup size	135,1 GB
Warning	0	End time	00:13:55	Data read	402,9 GB	Dedupe	3,7x
Error	0	Duration	0:43:51 	Transferred	141 GB		Compression	2,2x

Name		Status		Start time		End time		Size		Read		Transferred		Duration
Ubuntu		Success		23:30:27		23:56:12		50 GB		48,4 GB			19 GB		0:25:44 
Ubuntu		Success		23:30:27		00:00:23		250 GB		183,6 GB		43,0 GB		0:29:55 
W12		Success		23:30:27		00:13:49		800 GB		171,8 GB		79,1 GB		0:43:21 

Processing rate: 164 MB/s
THROUGHPUT Speed: 483 MB/s

Code: Select all

Job Saturday Night (Full) Project Systems:

Success	7	Start time	23:30:08	Total size	2,3 TB		Backup size	404,5 GB
Warning	0	End time	01:28:49	Data read	526,3 GB	Dedupe	4,6x
Error	0	Duration	1:58:40 	Transferred	405,5 GB	Compression	1,3x

Name		Status		Start time		End time		Size		Read		Transferred		Duration
CentOS		Success		23:30:32		00:09:46		64 GB		64 GB		29,6 GB			0:39:13 
ubuntu		Success		23:30:32		23:39:04		16 GB		6,6 GB		3,1 GB			0:08:31 
W10		Success		23:30:33		00:53:57		750 GB		174,3 GB	149,2 GB		1:23:24 
W10		Success		23:30:33		23:47:42		64 GB		14,6 GB		10,2 GB			0:17:09 
W10		Success		23:39:09		00:28:56		200 GB		47,4 GB		34,5 GB			0:49:47 
W10		Success		23:47:46		00:34:03		200 GB		37,4 GB		29,3 GB			0:46:16 
W16		Success		00:09:51		01:28:42		1024 GB		182,0 GB		149 GB			1:18:51 

Processing rate: 77 MB/s
THROUGHPUT Speed: 367 MB/s

Code: Select all

Job Sunday (Full) WebServer:

Success	8	Start time	23:30:08	Total size	256 GB		Backup size	46,0 GB
Warning	0	End time	23:47:29	Data read	204,5 GB	Dedupe	2,1x
Error	0	Duration	0:17:21 	Transferred	53,3 GB	Compression	2,7x

Name		Status		Start time		End time		Size		Read		Transferred		Duration
ubuntu	Success		23:30:33		23:37:15		16 GB		16 GB		4,9 GB			0:06:42 
ubuntu	Success		23:30:33		23:37:32		25 GB		23,9 GB		4,5 GB			0:06:59 
ubuntu	Success		23:30:33		23:37:27		50 GB		45,0 GB		5,0 GB			0:06:54 
ubuntu	Success		23:30:33		23:43:20		50 GB		26,4 GB		13,1 GB			0:12:47 
ubuntu	Success		23:37:19		23:43:30		25 GB		29,9 GB		3,8 GB			0:06:10 
ubuntu	Success		23:37:29		23:43:34		20 GB		14,6 GB		8,8 GB			0:06:04 
ubuntu	Success		23:37:35		23:44:20		30 GB		23,1 GB		3,2 GB			0:06:45 
ubuntu	Success		23:43:21		23:47:21		40 GB		31,7 GB		9,3 GB			0:04:00 

Processing rate: 224 MB/s
THROUGHPUT Speed: 626 MB/s

What do you thinking, is that OK or is there somthing else what i can do better?
many many thanks
Bob

[Egor Yakovlev]

I think they are great!
Note that your Proxy still handles only 4 VM in parallel, while others are waiting - for example W16 server was standing still for 39 minutes since job start. If you added more CPU cores already, make sure to raise Max Concurrent Tasks limit in Proxy settings(like you did for Repository tasks in the past) to bump the speed up.
/Cheers!

[TheBob]

Egor, thank you very much for all your help!!!
Proxy was limit to 4 current task! Now i have set this to 10 same like repo.

Thank you very much for helping and solving my issues.
Next Weekend I will see next result but for the moment every thing it's OK!

Many many thank!!!
BR Bob