Copy job data transfer is not continuous

[Zach123]

Hi All

We have a copy job transferring backups from local in premises repository ( ReFS ) to cloud repository with bandwidth throttled during the business hours.

We have been facing issues with copy job not completing in time. What we have noticed is that the copy job data transfer is not continuous and sometimes it's just stalled for a couple of hours. Have a look at the screenshot attached.

https://imgur.com/a/CUMu223

Bottleneck does not seem to be the source as source (disk, RAM, CPU etc ) is enough. We also increased the number of tasks on the proxy ( which is also a veeam server )

a) Veeam server is the proxy server as well as the source backup repository
b) The number of concurrent task on the backup repository is increased from 2 to 8.
c) Backup repository is ReFS
d) Use multiple upload streams per job is set to 5 in the network traffic window.
e) There are no other backups running during that time.
f) Copy window is set to 24 hours

Also, we are trying to move backups from forward incremental backup with weekly synthetic to forever forward incremental with 60 restore points to save some space on the repository. Would there be any adverse impact on the copy job due to reading that long backup chain?

[HannesK]

Hello,
without logfiles it's not possible to see why there is no data transfer at specific times. So I recommend asking support for help. Please post the case number here for reference.

In general, your setup looks okay, but one two things are missing for me: are you using a WAN accelerator? And your "cloud repository" is a Veeam Cloud Connect repository?

Also, we are trying to move backups from forward incremental backup with weekly synthetic to forever forward incremental with 60 restore points to save some space on the repository. Would there be any adverse impact on the copy job due to reading that long backup chain?

that's no problem from a backup copy job perspective

Best regards,
Hannes

[Zach123]

Hi Hannes

Thanks for the response. I will open a support case soon and update here if I need more help.

Just to answer your query, "We are not using WAN Accelerator" and "Yes, backup repository" is a veeam cloud connect repository.

We have around 1.5 TB of data to transfer to the cloud repository each day and with a speed of 50 MB/s when it works( as shown in the job progress ), I was expecting the copy job to finish within 12 hours window.

[jmmarton]

In addition to opening a Veeam support ticket, you may wish to contact your service provider to see if they are aware of any issues on their end which could cause the BCJ to stall at times.

Joe

[Zach123]

Hi all

I have opened a support case ( 03636183 ) with Veeam.

It was recommended to increase the max concurrent task from the cloud provider and also put the larger VM's at the top of the copy job but that doesn't make a difference. It looks like the situation is even worse now.

Max concurrent task is changed from 4 to 6 by the cloud provider.

https://i.imgur.com/4wMcGBh.png

[mkretzer]

This looks *exactly* like our REFS issues. Are there any deletes going on on the source side a few hours before the copy? If yes REFS might be at fault!

[Zach123]

Hi

Thanks for the reply.

It's not a very huge setup. In total, we have 12 physical servers and 6 VM's getting backed up on this refs repository which is hosted on the Veeam server ( PowerEdge R740xd running windows server 2016 ). Total size of the repository is 87 TB configured with 64 k block size.

If by "Are there any deletes going on on the source side" you mean deletion of restore points, then there should not be many deletion happening other than what forward incremental backup does as part of managing backup chain daily.
The refs partition is dedicated for backups and not shared for any other data.

Can you give me more insight on the ReFS issue ( technote etc ) you are talking about.

[mkretzer]

You are right, with that size and forward incremental it should not be a big issue.

Your can check my other forum threads, lastest: veeam-backup-replication-f2/windows-201 ... 57726.html

[Zach123]

Hi

Thanks for looking into this. We are still working on the ticket ( 03636183 ) with Veeam support. They have suggested that the issue could be related to the repository hosted at iland looking at the pex values as below.

Code: Select all

[07.07.2019 05:35:37] <57> Info                   [AP] (9c0e) output: --size: 136364163072
[07.07.2019 05:35:37] <41> Info                   [AP] (9c0e) output: --pex:0;1048576;0;0;0;32;70;0;0;0;0;0;132069153378580000
[07.07.2019 05:41:37] <53> Info                   [AP] (9c0e) output: --pex:0;679477248;170917888;0;170917888;87239488;99;4;99;81;3;0;132069156972950000
[07.07.2019 05:44:42] <57> Info                   [AP] (9c0e) output: --pex:2;3018850304;442499072;0;442499072;227743792;2;4;99;99;2;98;132069158826320000
[07.07.2019 05:44:56] <53> Info                   [AP] (9c0e) output: --pex:7;9731833856;1101004800;0;1101004800;504679296;2;4;99;2;2;97;132069158963520000
[07.07.2019 05:45:07] <57> Info                   [AP] (9c0e) output: --pex:9;12694061056;1429209088;0;1429209088;662058512;3;4;99;2;2;97;132069159073380000
[07.07.2019 05:45:17] <57> Info                   [AP] (9c0e) output: --pex:11;15674114048;1747976192;0;1747976192;830155424;3;4;99;2;2;97;132069159176830000
[07.07.2019 05:45:27] <53> Info                   [AP] (9c0e) output: --pex:16;22048407552;2126512128;0;2126512128;1015647984;4;4;99;2;1;97;132069159276840000
[07.07.2019 05:45:37] <57> Info                   [AP] (9c0e) output: --pex:100;136364163072;2380267520;0;2380267520;1150411904;4;4;99;2;1;97;132069159378260000

I am not much concerned about the copy job transfer speed, but more concerned about the data transfer is not continuous.

Look how the data flow halts for few hours in between.

https://i.imgur.com/Wul73i0.png

The backup copy job source repository is 87.3 TB on a windows 2016 server.


C:\Windows\system32>fsutil fsinfo refsinfo v:\
REFS Volume Serial Number : 0x6c165b28165af296
REFS Version : 3.1
Number Sectors : 0x0000002ba93a0000
Total Clusters : 0x0000000057527400
Free Clusters : 0x000000000ab3e802
Total Reserved : 0x000000000059734b
Bytes Per Sector : 512
Bytes Per Physical Sector : 512
Bytes Per Cluster : 65536
Checksum Type: CHECKSUM_TYPE_NONE

VBR server is also the repository server and the proxy server.'
Max concurrent task on the proxy is set to 8 and for repository its set to 12.
There are hardly any backup jobs running when the copy job is run.

We are running at 9.5. update 4. Would there any benefit moving update 4a.
The source backup is performed by installing the veeam agent on Hyper-V VM's as they are using pass-through disks.

[Zach123]

Hi All

Just an update..We are still working on the same ticket.

The situation seems to have worsened since iland performed there infrastructure upgrade. We upgraded our Veeam infrastructure to 9.5 update 4b as well on the recommendation from Veeam and iland.Copy job have not started failing.

Have provided the logs to iland and Veeam and awaiting update. This is what I see in the last few lines of task logs

Code: Select all

[31.07.2019 04:47:11] <28> Error    bad allocation (Veeam.Backup.Common.CCppComponentException)
[31.07.2019 04:47:11] <28> Error       in c++: Failed to send EOF upload message.
[31.07.2019 04:47:11] <28> Error       in c++: Failed to join send retry thread. Transmission was not correctly finalized.
[31.07.2019 04:47:11] <28> Error       in c++: Failed to process conveyored task.
[31.07.2019 04:47:11] <28> Error    Failed to upload disk. (Veeam.Backup.Common.CCppComponentException)
[31.07.2019 04:47:11] <28> Error       in c++: Disk upload failed.
[31.07.2019 04:47:11] <28> Error    Agent failed to process method {DataTransfer.SyncDisk}. (Veeam.Backup.Common.CCppComponentException)
[31.07.2019 04:47:11] <28> Error    Exception from server: Thread not finished within [7200000] milliseconds. (Veeam.Backup.Common.CCppComponentException)
[31.07.2019 04:47:11] <28> Error       in c++: Failed to wait for thread [18128] to finish.
[31.07.2019 04:47:11] <28> Error       in c++: Unable to wait for finish of upload thread.
[31.07.2019 04:47:11] <28> Error    Unable to retrieve next block transmission command. Number of already processed blocks: [7124]. (Veeam.Backup.Common.CCppComponentException)
[31.07.2019 04:47:11] <28> Error       in c++: Next asynchronous read request cannot be processed.
[31.07.2019 04:47:11] <28> Error       in c++: Asynchronous data reader has failed.
[31.07.2019 04:47:11] <28> Error       in c++: Failed to process conveyored task.
[31.07.2019 04:47:11] <28> Error    Failed to download disk. (Veeam.Backup.Common.CCppComponentException)
[31.07.2019 04:47:11] <28> Error       in c++: Disk download failed.
[31.07.2019 04:47:11] <28> Error       in c++: Unable to run ProtoEx server session.
[31.07.2019 04:47:11] <28> Error       in c++: Failed to handle ProtoEx session.
[31.07.2019 04:47:11] <28> Error       in c++ event: ClientErrorEvt
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.AgentProvider.CClientAgentProtocol.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog, Boolean agentNoLog)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[31.07.2019 04:47:11] <28> Error    --- End of stack trace from previous location where exception was thrown ---
[31.07.2019 04:47:11] <28> Error       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Common.ExceptionFactory.ThrowNecessaryAggregateException(IEnumerable`1 exceptionsCollection)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.AgentProvider.CBackupClientImpl.ConstructException(Exception exception, String error)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.AgentProvider.CBackupClient.ConstructException(Exception ex, String error, Object[] args)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.AgentProvider.CBackupClient.OnInvokeError(Exception e, String command, CVcpCommandArgs inArgs)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.AgentProvider.CBackupClient.DataTransferSyncDisk(ISourceDiskSpec source, ITargetDiskSpec target, Boolean serverHandlesSource, String operationId, Boolean treatProcessedLikeReaded)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CAgentSyncDiskTransport.<>c__DisplayClass2.<SyncDisk>b__0()
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CAgentSyncDiskTransport.ProcessDisk(IDiskTaskSource diskTaskSource, IDiskTaskTarget diskTaskTarget, CDiskDigestHook sourceHook)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.HyperV.CHvDiskTransferPlatform.BackupDisk(IBackupClientInvoker sourceClientInvoker, IDiskTaskSource diskTaskSource, CBackupDiskTarget diskTaskTarget)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.HyperV.CHvDiskTransferPlatform.BackupDisk(IBackupClientInvoker client, IDiskTaskSource diskTaskSource, IDiskTaskTarget diskTaskTarget)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CBackupTransfer.ProcessDiskObject(IDiskTaskSource diskSource, IDiskTaskTarget diskTarget)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CBackupCopyDiskTransferAlg.ProcessDisk(IDiskTaskSource vmDiskSource, CBackupDiskTarget vmDiskTarget, IStopSessionSync stopSessionSync)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CBackupCopyDiskTransferAlg.ProcessDisk(CDiskTask diskTask, IStopSessionSync stopSessionSync)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CSerialDiskProcessor.StartDiskProcessing(CDiskTask diskTask, IDiskTransferAlg disksAlg)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CSerialDiskProcessor.ProcessDisks(IDiskResources diskResources, IEnumerable`1 diskBackupObjects, IDiskTransferAlg disksAlg)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CSerialDiskProcessor.ProcessDisks(IDiskTransferAlg disksAlg, IEnumerable`1 diskBackupObjects)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CTaskProcessor.ProcessTask(IVmBackupTask task, ITasksScheduler tasksScheduler, LogStorage logStorage, IStopSessionSync stopSessionControl, Boolean disableParallelProcessing)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CRemoteBackupTaskPerformer.ExecuteTask(IVmBackupTask task, ITaskSource vmSource, IBackupTaskTarget vmTarget, IStopSessionSync stopControl)
[31.07.2019 04:47:11] <28> Error       at Veeam.Backup.Core.CRemoteBackupTaskPerformer.ExecuteTask(IVmBackupTask task, CVbJobSource source, CBackupJobTarget jobTarget, IVcdVAppBackupTask parentTask, CMultiStopSessionSync stopControl)
[31.07.2019 04:47:11] <28> Info     [CloudGateSvc mel02-vccgw01-03.ilandcloud.com:6180]Request: [Db.WriteSessionLog]