Discussions related to exporting backups to tape and backing up directly to tape.
Post Reply
NelsonBA81
Novice
Posts: 5
Liked: never
Joined: Aug 12, 2020 9:33 am
Contact:

Backup from StoreOnce Catalyst to Tape Slow?

Post by NelsonBA81 »

Hello,

I've been doing some testing using some dedup appliances, and I've noticed that using the HPE StoreOnce VSA in Catalyst mode in backup to tape job, kind of slow.
What I mean, is that each copy of VBK file to tape, there is some sort of delay between them, creating the "tape shoe-shining or back-hitching" effect.
Anyway, I did created a NAS/CIFS repository in the same StoreOnce VSA using the same kind of backup files from the Catalyst repository, and the backup to tape from the CIFS repository, this symptom does not happen.

The log file that is located in:C:\ProgramData\Veeam\Backup\{BackupToTapeJob}\Agent.{BackupToTapeJob}.TapeVmBackup.log
Shows more clearly the symptom, where the copy between VBK files, there is some sort of delay, in this case, there is 12s pause to start to copy another VBK file to tape:

Code: Select all

[11.08.2020 22:37:00.585] <  3016> mt | File was backed up in elapsed: 197.2980, isIncomplete: false
[11.08.2020 22:37:00.585] <  3016> cli| 
[11.08.2020 22:37:00.585] <  3016> cli| _______________________________________________________________________________
[11.08.2020 22:37:00.585] <  3016> cli| Waiting for the next command.
[11.08.2020 22:37:12.550] <  3016> cli| Next client command: [backupSrvFileToTape].
This delays/pauses in the "Waiting for the next command." line for each copy of an VBK file, make the tape drive stop, reposition the tape, creating a bit more stress and slow down the total time to copy all backups to tape.

Using the CIFS repository, the delay/pause is about in less than a second, making the flow of the data to the tape drive more consistent and continuous, avoiding the stop/reposition/start of the tape.
Also, I did a bit further test, by installing the tape drive in another server and use the Tape Server service there, and the symptoms still happen in that server in the same way.

At the moment, I'm using an LTO5 SAS drive, and I do not notice problems with it, I can easily get 120MB/s in data transfers, so I don't see any sort of problems related to storage, cpu, drivers or even tape cartridges...

So, am I missing something? or maybe is there any sort for configuration in Veeam to reduce this "delay" between the "Waiting for the next command."

Best regards,
Nelson
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Backup from StoreOnce Catalyst to Tape Slow?

Post by Dima P. »

Hello and welcome to the community Nelson,

Mind me asking was that a vbk transfer or the full was synthesized from the increment on StoreOnce appliance? Thanks!
NelsonBA81
Novice
Posts: 5
Liked: never
Joined: Aug 12, 2020 9:33 am
Contact:

Re: Backup from StoreOnce Catalyst to Tape Slow?

Post by NelsonBA81 »

Hello Dima,

The Catalyst repository was created a few days ago from scratch, and the initial VBK's are only full backups, so no synthesized full backups, yet
I have VIB's of the daily increments too created after the fulls, but the chains are small, they have about 2-3 days, as this repo is too recent.

Thanks
CLDonohoe
Veeam Software
Posts: 36
Liked: 21 times
Joined: May 10, 2018 2:30 pm
Full Name: Christopher Donohoe
Contact:

Re: Backup from StoreOnce Catalyst to Tape Slow?

Post by CLDonohoe »

So as I understand your troubleshooting so far:

- StoreOnce Catalyst store to tape is slow and causes the shoe-shining effect on tape.
- StoreOnce CIFS share to tape maintains the minimum data rate for LTO5. 100 MB/s should keep the tape busy.

Have you tried a backup copy job from the StoreOnce Catalyst store to a different repository? I'd be interested in the throughput for that job.

Also, what's the StoreOnce VSA version you're using?
NelsonBA81
Novice
Posts: 5
Liked: never
Joined: Aug 12, 2020 9:33 am
Contact:

Re: Backup from StoreOnce Catalyst to Tape Slow?

Post by NelsonBA81 »

Hello CLDonohoe,

About copying from StoreOnce Catalyst to another repository, actually I didn't do that test before, as I don't see this as a transfer rate problem, but maybe something how the veeam services interact each other using the catalyst as source repo and tape as destination.
In any case, I've done a quick test doing a backup copy job to an empty drive, it isn't a fast drive but still I've around 80MB/s of constant write to it.

The version I'm using is the StoreOnce VSA v4.2.3-2011.40, is the free edition of 1TB.

Thanks
NelsonBA81
Novice
Posts: 5
Liked: never
Joined: Aug 12, 2020 9:33 am
Contact:

Re: Backup from StoreOnce Catalyst to Tape Slow?

Post by NelsonBA81 »

Hello,

I've done some testing with a trial version of Dell DataDomain Virtual Edition, and using the DDBoost feature, does not give the pauses in the tape drive.
So, I'm asking, has anyone here have noticed that kind of "problem" with tape drive pauses while copying backups from storeonce using catalyst to the drive?
Is there any way to change the "Waiting for the next command." time interval to a shorter one? maybe via registry key?

Thank you
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Backup from StoreOnce Catalyst to Tape Slow?

Post by Dima P. »

NelsonBA81,

Thank you for your time, very interesting results. We do not have any registry keys to adjust the tape communication logic, but I've passed the results of your findings to our RnD folks.
NelsonBA81
Novice
Posts: 5
Liked: never
Joined: Aug 12, 2020 9:33 am
Contact:

Re: Backup from StoreOnce Catalyst to Tape Slow?

Post by NelsonBA81 »

Hello Dima,

Thank you for the feedback.

I believe that it is possible to replicate easily this condition, by just:
- install StoreOnce VSA in ESXi server, the latest version I know is 4.2.3 that is available for free in HPE website
- add an SAS LTO Tape Drive to Veeam Server
- create a Catalyst repository in StoreOnce, and configure it in Veeam
- create a backup job pointing to Catalyst repo, and add a few VM's, 40-60Gb each if possible, as it is more "visible" and easy to analyse the condition I'm reporting
- create a full backup, and 1 or 2 increments, to have some smaller files to copy to tape
- create backup copy job to tape to copy all backup files created early

During the copy job to tape, you just have to watch/observe the activity of the tape drive, and also take a look at this log file:
C:\ProgramData\Veeam\Backup\{BackupToTapeJob}\Agent.{BackupToTapeJob}.TapeVmBackup.log

Which in my case, the tape drive stops constantly at this lines for each time it finishes copying a VBK file:

Code: Select all

[11.08.2020 22:37:00.585] <  3016> cli| _______________________________________________________________________________
[11.08.2020 22:37:00.585] <  3016> cli| Waiting for the next command.
[11.08.2020 22:37:12.550] <  3016> cli| Next client command: [backupSrvFileToTape].
The pause happens between the "Waiting for the next command" and "Next client command".
Sometimes have 10s delay, sometimes have more, seems to be random, and I don't understand why...
Other times, the delay is small, like a fraction of 1s, and I don't notice the tape drive stopping in those times.

Thank you for your attention
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Backup from StoreOnce Catalyst to Tape Slow?

Post by Dima P. »

NelsonBA81,

Per RnD folks the 'Waiting for the next command' pause may happen due to multiple factors, for example when we are receiving the response from the source device or when we are opening the file. Support team proposed to raise a case and they can provide the detailed explanation on the situation based on your debug logs. Cheers!
Post Reply

Who is online

Users browsing this forum: Semrush [Bot] and 2 guests