Backup copy odd performance

[ckent]

Seeing some odd performance issues with a backup copy job.

Source is a Dell R710 with local storage.

Destination is a local HD connected over USB3.

The job contains 5vms and a sript is run to format the USB drive prior to the job.
These are the stats from the job.

VM1

Code: Select all

20/12/2015 23:31:48 :: Hard disk 1 (200.0 GB) 40.7 GB read at 82 MB/s
20/12/2015 23:40:30 :: Hard disk 2 (3.0 TB) 2.3 TB read at 72 MB/s
21/12/2015 08:58:02 :: Hard disk 3 (2.0 TB) 870.1 GB read at 63 MB/s
21/12/2015 12:56:59 :: Busy: Source 70% > Proxy 2% > Network 49% > Target 43% 
21/12/2015 12:56:59 :: Primary bottleneck: Source 

VM2

Code: Select all

21/12/2015 12:57:47 :: Hard disk 1 (200.0 GB) 41.8 GB read at 49 MB/s
21/12/2015 13:13:57 :: Busy: Source 54% > Proxy 1% > Network 49% > Target 46% 
21/12/2015 13:13:57 :: Primary bottleneck: Source 

VM3

Code: Select all

21/12/2015 13:14:49 :: Hard disk 1 (25.0 GB) 5.5 GB read at 82 MB/s
21/12/2015 13:17:55 :: Hard disk 2 (100.0 GB) 2.2 GB read at 113 MB/s
21/12/2015 13:19:32 :: Busy: Source 98% > Proxy 2% > Network 10% > Target 3% 
21/12/2015 13:19:32 :: Primary bottleneck: Source 

VM4

Code: Select all

21/12/2015 13:20:14 :: Hard disk 1 (40.0 GB) 15.9 GB read at 39 MB/s
21/12/2015 13:31:39 :: Hard disk 2 (1.0 TB) 202.6 GB read at 33 MB/s
21/12/2015 15:19:41 :: Busy: Source 35% > Proxy 1% > Network 32% > Target 70% 
21/12/2015 15:19:41 :: Primary bottleneck: Target 

VM5

Code: Select all

21/12/2015 15:20:30 :: Hard disk 1 (400.0 GB) 224.1 GB read at 36 MB/s
21/12/2015 17:11:04 :: Hard disk 2 (100.0 GB) 5.1 GB read at 98 MB/s
21/12/2015 17:14:28 :: Busy: Source 67% > Proxy 1% > Network 40% > Target 36% 
21/12/2015 17:14:28 :: Primary bottleneck: Source 

is the differences in performance due to a combination of both thin and thick provisioned disks ?
i suspect it would be far quicker just to drag the backup file form the local disk to the USB3 disk.

[PTide]

Hi,

The job contains 5vms and a sript is run to format the USB drive prior to the job.

May I ask you why do you need to format your USB drive every time?

is the differences in performance due to a combination of both thin and thick provisioned disks ?

Backup copy job transfers only changed blocks, so the type of disk gives no impact when you run backup copy job.

Seeing some odd performance issues with a backup copy job.

Are you talking about the differences in bottlenecks? Please elaborate?

i suspect it would be far quicker just to drag the backup file form the local disk to the USB3 disk.

Drag'n'drop would probably work better only in case of VM4 since the processing of all other VMs is slowed by source which is your primary repo.

Thank you.

[ckent]

May I ask you why do you need to format your USB drive every time?
I rotate 3 USB drives to keep 1 offsite at all times.
If i dont format first the job trys to create two copies of the backup files at the same time.

Backup copy job transfers only changed blocks, so the type of disk gives no impact when you run backup copy job.
See above.... the job has to create a whole new copy each time.

Are you talking about the differences in bottlenecks? Please elaborate?
Drag'n'drop would probably work better only in case of VM4 since the processing of all other VMs is slowed by source which is your primary repo.
The source for all VMs is a local Sata Raid with a read speed of 364 MB/s the target is a local disk (USB3) with a write speed of 154.8 MB/s
I can drag and drop the 3.4TB main .vbk and get a sustained copy speed of 140 MB/s estimated time of 7 hours.
As opposed to the 18 hours it takes Veeam to perform a backup copy job.

I would however much rather use the tools/functions built into Veeam to manage all backup jobs rather then rely on using external scripts or human intervention.

[PTide]

If i dont format first the job trys to create two copies of the backup files at the same time.

Is it trying to create two .vibs on the same USB drive? Please provide us some details.

See above.... the job has to create a whole new copy each time.

Still no impact - backup copy job synthesizes new restore points from primary backups. Virtual disks that are stored inside a backup, do not differ in terms of "thickness".

When backup copy job creates .vbk it copies data blocks that are necessary to build a full backup of a VM as of the most recent state. Data blocks can be copied from one or several backup files in the chain (except for reverse incremental chains), opposed to simple copying. Also backup Copy job performs some additional health checks. All these activities result in a slower processing speed. There are also other benefits of using a backup copy such as a different retention + GFS retention policy.

Please provide more details about "two copies" situation so we could find the way to improve your backup experience.

Thank you.

[ckent]

Is it trying to create two .vibs on the same USB drive? Please provide us some details.
i logged a ticket with the number ID# 01119586 becasue my backup copy to USB was failing due to a lack of space.
the reply i got was

Ho Colin,

The thing is that when you swap the drives, Veeam creates a new full backup:
http://helpcenter.veeam.com/backup/80/v ... igure.html

So, your drives have to have space enough to keep at least two full backups.

Also, you can remove everything from drive prior to writing a backup there using a pre-job script.

Thank you,
Eugene Morozov
Veeam Support

Nothing like a consistent message from support is there.

Long and short of it is that it is considerable quicker (7 hours compared to 18 hours) to copy the files in windows to copy the Veeam files to the USB disk than it is to use Veeam.
Which as i have already said feels like a dirty fix and not a solution, plus as you say we loose additional benefits.

[PTide]

As Helpcenter says:

If you use a Linux server or CIFS share as a backup repository with rotated drives, Veeam Backup & Replication employs a “cropped” mechanism of work with rotated drives. If Veeam Backup & Replication fails to detect previous restore points on the drive, it simply creates a full backup and start a new backup chain. Information about restore points that were previously created is removed from the Veeam Backup & Replication database. Actual backup files remain on drives.

So support engineer is right - you need either to have enough space to keep another full every time you rotate drives, or to use Windows-type repository instead of CIFS, or use pre-job script in order to delete old chain from USB drive.

In regards to backup copy speed - what have you specified as a gateway for your CIFS repository?

Thank you.

[ckent]

We are no longer using a windows share.
The USB drive is now directly attatched to the server with Veeam installed and is setup as a Windows Server not a CIFS share.
Can i now remove the script to wipe the drive each time ?

it still dosnt address the lack of performance.

[PTide]

The USB drive is now directly attatched to the server with Veeam installed and is setup as a Windows Server not a CIFS share.

Can i now remove the script to wipe the drive each time?

Yes, you can remove it - with local rotated usb drive backup copy job should create another increment.