Catalyst Copy ends in error when time restriction is in effect

[mkaaden]

Is it possible to suppress the error status on (catalyst) copy jobs when a time restriction is in effect ?
IMHO should it at least be a warning since remaining blocks eventually get copied when the time restriction is lifted.

[Dima P.]

Hello mkaaden,

Can you share a screenshot or text of the error you are referring to? Thank you in advance!

[mkaaden]

Do not how to upload an image and embedding does not seem to work, so here is a IMGUR link to a screenshot.
Names are obscured.

https://imgur.com/a/dkNGHHV

This a screenshot of the copy job next day where the failed object is processed again and finishes successful

https://imgur.com/VaQkYUP

[Andreas Neufert]

Can you please clarify what time restrictions you are talking about ?

[mkaaden]

This setup has 2 StoreOnce 5100 appliances
Backups are processed only to 1 StoreOnce and after that copied to next one.
This is to prevent having cross-copies jobs where both StoreOnces need to read and write at the same time, which they are bad at.

So in this case I do not want to run copies and backups in parallel, so the backup copy is restricted during backup.
Backups start at 20:00 and usually run up till 0:00 on weekdays.

In the weekend synthetic full takes considerably more time and varies a lot, so the copy has to start even later.
Ans apparently, due to the synthetic full, more blocks need to be copied, because a complete new object is being created.

So, in the weekends the Copy time limit is set from 20:00 to 10:00, on weekdays it is set from 20:00 - 6:00
The Houskeeping black out window on the StoreOnces is set accordingly, so it only runs during the time no jobs are being processed in Veeam.

So the backup copy window varies from 10 to 14 hours, which should be enough considering backups are run in less time.

I think one of the drawbacks of StoreOnce is that you can not pause a Copy job, only the system can do that if it runs out of resources.
So HPE should implement a pause function within the Catalyst API.

But in mean time is there a registry key to suppress the error when a Time Restriction kicks in.

[Andreas Neufert]

Thanks for sharing the details. It will take some time to check some things. As we are transporting whole files a stop and resume is not simple and would need a sync of the actual written data (lot´s of reads). So I think it is better to transfer the file just again. As the situation is expected in your case that the files stops syncing the message should be maybe a information type of message. Let me check some things here.

[Andreas Neufert]

Hello. I checked and this is the same as within our other jobs. The current thinking is that the job processing should complete in the officially processing time and the hard stop fro some time frames is an unusual case where for example production would be affected and therefore is OK to stop but warn you.

Alternatively you could work during the whole time and just reduce the transport traffic to some low value to not affect others?

[mkaaden]

It is not the issue that production gets affected by the copy, ergo, reducing transport traffic would only elongate the copy process.

The issue is that we do not want backup and copy to happen at same time because of the poor read/write IO capabilities of the StoreOnce appliances.
They really interfere with each other, making both processes nearly impossible to finish in a proper timeframe.

So in the mean time it would be nice to have a some sort control in whether the system produces an error or just a warning/informational message when a copy doesn't finish within a timeframe.

[Andreas Neufert]

OK maybe this is a workaround.
In general backups are higher prioritized than the backup copy job processing. A ideal backup windows is achieved if you start more job processings as the Repository task slot can handle to keep the backups running and make the repository task slots busy. There is no downside in this approach.

Only at the end of the backup processing, you will get available task slots and the copy process starts which slow down a bit the last backups maybe in your case.
We talked with HPE and as the Catalyst Copy process do not rehydrate/uncompress/undeduplicate data, the load should be distributed fine and impact should be OK that way.

This approach need some careful Proxy/Repo task slot planning but I think is a successfull solution for now.