Backup Copy Issues

[MarkusRadtke]

Hi,

following situation:
after upgrading to B&R v7 (from v6.5) we decided to create a new job which include all VMs. With 3 Proxies this job works fine. Then we want to have a secondary copy so we use the new feature Backup-Copy.

This new feature has brings 2 issues: slow transmit to the secondary location and errors with VM-templates.

Infrastructure Information:
- VMware vSphere 5.1
- 3 Hosts
- physical Backupserver Windows 2008 R2 Server with direct attached storage onsite
- physical Backupserver Windows 2008 R2 Server with direct attached storage offsite with 900 MBit-Connection (nearline 1 GBit)
- 3 veeam Proxy-Server (one of each host) running Windows 2008 R2 Server

- one job for all VMs incl. all templates (~180 VMs), running fine and successfull to onsite Backupserver daily with weekly synthetic fulls, dedupe and compression enabled
- one Backup-Copy-job, running continous with daily schedule, running very slow (~max 20-25 MBit/s) to offsite Backupserver, dedupe and conpression disabled


We have installed v7 R2 but the slow Backup-Copy is still present, also all VM-Templates fails with error "Incremental copy was not processed during the copy interval"

Thanks for your help,

regards, Markus

[foggy]

- one Backup-Copy-job, running continous with daily schedule, running very slow (~max 20-25 MBit/s) to offsite Backupserver, dedupe and conpression disabled

Markus, what are the bottleneck stats for this backup copy job? Do you use direct or WAN accelerated copy?

all VM-Templates fails with error "Incremental copy was not processed during the copy interval"

Please see this topic regarding that issue. The job fails since there're no restore points for templates to copy over (templates do not change).

[MarkusRadtke]

Hi foggy,

thanks for your answer.

The bottleneck is "Source". I don't find stats for the complete job, only for each processed VM? We don't use WAN accelerator cause we don't have the license. But is it needed with this fast "WAN" connection (near 1 GBit)?

Depending the "failed" VM-Template: Am I right, that this is only a "wrong" error-message because there are no restore-points so there is no backup needed? It's possible to change something to get better reports?

Tnaks a lot,

Markus

[foggy]

The bottleneck is "Source". I don't find stats for the complete job, only for each processed VM?

Stats for entire job can be found in the job statistics window, in the Summary section.

We don't use WAN accelerator cause we don't have the license. But is it needed with this fast "WAN" connection (near 1 GBit)?

No, I just wanted to clarify that.

Depending the "failed" VM-Template: Am I right, that this is only a "wrong" error-message because there are no restore-points so there is no backup needed? It's possible to change something to get better reports?

You can exclude templates from the backup copy job, if this is an acceptable workaround.

[MarkusRadtke]

Hi again,

here is the stat from the last job Load: Source 61% > Proxy 47% > Network 30% > Target 0%

Regarding the template-errors: I don't want to exclude the templates from the backup-job, but I want to have "clean" reports

regards, Markus

[foggy]

According to the bottleneck stats, your source storage cannot provide data any faster. What kind of storage it is?

Btw, why do you consider 20-25 MB/s to be slow?

[MarkusRadtke]

Hi foggy,

first of all sorry for my delay, I was ill last week.

Both backup-Servers are Zealbox Z20.15 - Servers with integrated Storage (Backup Storage = Raid6 on 22x 3 TB Nearline SAS, connected with Areca Raid Controller).

When I copy files from one to the other Backup server, i get ~70-80 MB/s so yes, 20-25 MB/s is slow. In my opinion the Backup copy job have to take the last restore-point and copy it to the other location.

When I set up a "normal" backup-Job to the other server i get normal performance, limited by the network bandwith.

In the past we first backup to internal Backup-server and after that a backup to secondary location. The new backup Copy Job is because of his design the better choice but this slow performance and the upcoming errors are showstoppers...

I want to test with a fresh installed Veeam Backup-server on a VM and use the two physical backup-server as target only to see if there is the same poor copy performance.

Again, thanks for your help!

Regards, Markus

[foggy]

When I copy files from one to the other Backup server, i get ~70-80 MB/s so yes, 20-25 MB/s is slow. In my opinion the Backup copy job have to take the last restore-point and copy it to the other location.

Backup copy job does not copy files but rather synthetically creates restore points in remote location from the changed blocks extracted from the source storage. Synthetic activity requires much more I/O comparing to simple sequential read/write, putting high load on the storage.

[Gostev]

File copy speed does not matter beyond copying videos at home... all other workloads require IOPS.
Why do you disable dedupe and compression? You are just killing your performance with these settings.

[MarkusRadtke]

Hi,

regarding dedupe and compression: I use both features in the backup-job and i was wondering if this is needed in the backup-copy-job too. In my opinion dedupe and compression produces more workload so i disabled it for the copy job only.

But its easy to re-enable it if this helps...

[MarkusRadtke]

Hi,

a few days later I will tell you the current status.

I've take a fresh install from Veeam B&R v7 R2 on a 2008R2-VM and it works much better than before! I see that the fresh install comes with MS SQL 2008 R2 express while the old server uses SQL 2005 express only. On the old server the CPU-Load from the SQL-Server was pretty high during the Backup Copy Job, on the new one it doesn't.

Current config:

- VMware vSphere 5.1
- 1x VM with 4 vCPU and 6 GB RAM running Windows Server 2008 R2 DC
- 3 proxies with the same specs (VMs)
- "old" Backup-Server with direct attached storage, using as internal LAN target
- secondary Backup-Server in offsite-location, using as backup-copy-target (900 MBit-WAN connection)


I have created 3 different Jobs:
- Backup all VMs in one job to internal Backup-Server (daily with sybnthetic full weekly, monthly active Full, dedupe and compression on auto)
- Backup Copy Job based on the upper job excluding template-folder (daily) to offsite Backup Server, dedupe and compression to auto
- Backup Template Folders to offsite Backup Server (weekly, dedupe and compression to auto)

I've also created new repositories on both servers to start completely clean. Now it works fine

Last active Full:
Success 184 Start time 21:30:09 Total size 7,2 TB Backup size 1,6 TB
Warning 0 End time 09:25:58 (+1) Data read 6,5 TB Dedupe 2,5x
Error 0 Duration 11:55:48 Transferred 2,5 TB Compression 1,8x

last increment:
Success 184 Start time 21:30:07 Total size 6,8 TB Backup size 30,2 GB
Warning 0 End time 01:13:44 (+1) Data read 2,1 TB Dedupe 1,0x
Error 0 Duration 3:43:36 Transferred 32,9 GB Compression 1,8x


The Backup Copy Jobs work much betther than before, with more than 40 MB/sec and good duration times (6-7 hours for 165 VMs)

So its on a good way. Sometimes a fresh install take the shot Hope that in the future the templates will be better integrated in the backup copy job (no error-messages during backup copy in case of no changed templates)


But for the while I'm ok with the current performance.