Discussions related to exporting backups to tape and backing up directly to tape.
Post Reply
seadave
Enthusiast
Posts: 36
Liked: 6 times
Joined: Oct 30, 2014 12:43 am
Location: Seattle
Contact:

Suggestings for improving tape UI and reporting

Post by seadave » 1 person likes this post

Based on the other posts I see here, I'm not the only one trying to survive the unpleasant experience of Veeam Backup to tape. Currently working through this with case 00728626. Before I get to my list of things that should be fixed to improve the product I will commend Veeam for the mostly flawless operation of the VM Backup portion. Despite the ability of it to take down your storage if not tuned properly when deployed (the default "Max concurrent tasks" setting in the Backup Proxies was too high and saturated our storage with concurrent snapshots at first), it works well and processes the jobs much quicker due to the use VADP and CBT, especially compared to our previous agent based solution.

Tapes are another story. We have been fighting to get reliable tape operation for two and a half months. Our initial problem was the increased amount of data that Veeam wanted to backup compared to our previous solution. It took way too long over the SAS to iSCSI bridge we were using. That topped out at 50MB/s. We decided to purchase a Dell SAS 6Gbs HBA and now we get ~120MB/s throughput (our drive's limit) which makes a huge difference. If you are connecting to your tape drive/library over 1Gbps ethernet and having problems, this may be your solution.

Our tape library is a Dell TL2000 with a single LTO4HH drive. We continue to have timeouts and failures (it worked without error for the last four years before we started using it with Veeam). The Synthesized VBK process is a nice concept but very poorly implemented. The UI to configure that is inane. Here is what I have noticed:

1. Returned tapes need to be moved to "free" and a short erase run on they will not be used for new jobs. I've noticed the sequence number changes from 0 to 1 when this is done. Not sure what this signifies, other than the obvious when it is part of a set and there is 3 tapes making up a sequence of 1,2 and 3.

2. Registry entry is required to prevent job from timing out (default is 60s which is too slow for our system apparently).
HKLM\SOFTWARE\Veeam\Veeam Backup and Replication:
REG_DWORD
TapeDeviceNotReadyTimeoutSec
Hex 78 / Dec 120

There needs to be some logic incorporated to recover the tape library if it times out or needs a tape ejected. This happens often enough, but not consistently. Manual intervention is required and then if we restart the job it succeeds. Maybe this is why so many vendors write their own tape drivers? Relying on Windows drivers can be easy but appears to be very problematic.

3. Tape backup jobs will copy ALL files that it thinks are not on a tape the first time, or any time it is run. (Tech Support tells me I can clone a job to prevent this from happening, but this is just nuts.)

EVERY system I have EVER used for the last 20 years allows you to do this OR make a copy of the MOST RECENT BACKUP ONLY! This is how Virtual Synthesized Backups (VSB) SHOULD work!!! There should be an option for VSB specifically and the user should be able to choose which day's backup will be used as the source for it. This has caused us weeks of frustration due to unpredictable operation, and the lack of specifics in the UI. No offense to the tech that has been assisting me, but it has been clear on several occasions that the VSB process is something he is not well versed in.

We run a standard backup schedule. M-F, we are using Forever Forward Incremental (FFI). We run tapes on Saturday. The original choice I made on the "Schedule" dialog box on the Media Pool page of the tape job was Friday. Made sense. "Which day do you want to use for the VSB source?" It never worked. After working with several techs and over a period of 45 days (we kept trying the "see if this works" method), it was determined due to "updated documentation" that I should choose Monday. That would "break the chain" of backups and create a full to tape for me. It is and it is not. It is now creating a VBK(s) and copying the VIBs (from I assume the prior week) also, in most cases. Don't get me started on the "Process incremental backup files" option. Trying to figure out if that should our should not be checked approached going on a holy quest. I'm currently under the impression that it SHOULD be checked for VSB to work.

We thought we finally had it working but now I get some jobs with just VIBs, some with what appear to be multiple VBKs (I assume these are the same large one spanning tapes), and errors about Timed out waiting for device ready status on MoveMedium operation. I'm definitely willing to try another tape library, but due to cost I absolutely need to know that is the problem before shelling out $5K to $10K for a new one. It really bothers me that we NEVER had a tape problem until we started using Veeam.

IMHO, if VSB worked as it should, the tapes would only contain a VBK created using the most recent data backed up. Maybe that just isn't possible doing FFI and it only works with Reverse Incremental (RI), but if that is the case, it needs to be spelled out more clearly or present UI feedback when making that type of configuration choice.

4. Tape Libraries Discovery jobs will start and hang if a multi-tape erase job is running. It shows a start time of two weeks prior??? Shouldn't there be logic where it polls the tape process and skips the discovery job if another process is accessing the tape? Not a big deal, but a symptom of some of the bigger issues I'm seeing I think.

5. Manually starting a VM backup or tape job that is part of a schedule will not run the others tied to it. This is really frustrating. I guess previously the opposite occurred which would be equally as bad. GIVE US THE OPTION please. Right click on job, "Run this job only", or "Run this job and all jobs scheduled to follow it" would be logical choices. We chain our jobs to follow each other which helps keep the backup process as short as possible.

6. I've posted about this before, but the reporting in Veeam related to backup jobs, especially tape jobs, is just terrible. Most of the reports are simple dumps of data and don't give the user control over the output.

The built in reports fired at the end of a job should show the following:


a. Remove the creation time/stamp date of the job. That is confusing and serves no purpose, at least make it an option or a end note.
b. Show the total elapsed time of the job if it was queued to run, but also the actual "run" time of the job. (Can't tell if it actually doing that currently.)
c. Show the total size of the VMs uncompressed, Data Read, Amount Transferred. (Looks to be basing Total size on amount transferred which isn't correct. Maybe a fourth category "RAW VMs size" is needed.)
d. List the VMs that were copied by name with time they were copied at.
e. Perhaps break down the amount of CBT data copied for each VM, showing delta from previous run. This can be important operational and forensic data.

The Veeam One tape job reports should do/show the following:

a. Allow the user to select the time frame of the report!!! (Last 7 days for example)
b. Allow the user to select the tape jobs in the report!
c. List each backup job and the VMs that make up that job.
d: List the tapes that are part of that backup job set.
e. List the files on each tape

Honestly, ANY report that doesn't allow the user to specify a time range and the objects down to the individual VMs that comprise it should be revamped. If you need a good model, look at how CommVault does it.

7. The option to put multiple jobs on the same tape or span all jobs over a set of tapes should be offered. Based on my most recent run of 11 jobs, Veeam is consuming 17 tapes with a total capacity of 12.7TB. The actual data on tape is only consuming 5.5TB or 8 tapes worth of data. That is costing me and extra $180 a week in tapes. That adds up to a lot of money over a fairly nominal retention period.

I have applied the latest Veeam One patch but the fixes I'm looking for do not seem to be present. I am pleased by the fairly rapid release cycle of updates, but I think implementing the above would make it much easier for the end users.
kmbelt
Influencer
Posts: 14
Liked: never
Joined: Sep 08, 2014 4:08 pm
Full Name: Kevin Belt
Contact:

Re: Suggestings for improving tape UI and reporting

Post by kmbelt »

I have had the same experience with Veeam and tape. If i had my choice, i would go right back to NetBackup that worked flawlessly. Instead, i constantly fight with this to make it fit our needs because it was forced on us by Corporated.

I just want a report that tells me what tapes were written to in the past "custom date range". Just to find out what tapes are written to so i can remove them daily is extra work because the reporting capability does nothing this simple. The reporting is actually a JOKE and i dont even use it for these reasons.
seadave
Enthusiast
Posts: 36
Liked: 6 times
Joined: Oct 30, 2014 12:43 am
Location: Seattle
Contact:

Re: Suggestings for improving tape UI and reporting

Post by seadave »

kmbelt wrote:I have had the same experience with Veeam and tape. If i had my choice, i would go right back to NetBackup that worked flawlessly. Instead, i constantly fight with this to make it fit our needs because it was forced on us by Corporated.

I just want a report that tells me what tapes were written to in the past "custom date range". Just to find out what tapes are written to so i can remove them daily is extra work because the reporting capability does nothing this simple. The reporting is actually a JOKE and i dont even use it for these reasons.
I agree it is in need of serious improvement. As I stated above I think the VM based backup of Veeam is great and it is working MUCH better than our old solution, but yes they need to address the reporting and tape issues ASAP or they will continue to lose customers that rely on these features. I don't think they appreciate the turmoil that is causes when you can't produce good reports to auditors. We are a public company so we need to comply with SarbOx among other regs. I have quarterly audits where they require me to prove that I'm shipping reliable tape jobs offsite. That was easy with CommVault, that has not been the case with Veeam. It sure seems that with a little attention and effort on their part, that will become the case.

I do give them credit for the openness of this forum. Many companies attempt to filter these type of critiques which I think is dishonest. Kudos to Veeam for allowing us to make note of our perceived deficiencies of the product.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Suggestings for improving tape UI and reporting

Post by Dima P. » 1 person likes this post

Hello seadave,
Thank you for sharing! I literary love all your posts because they indicate the exact place we need to pay attention and improve. Constructive criticism is always welcome here at the forums as it makes the product better.
Returned tapes need to be moved to "free" and a short erase run on they will not be used for new jobs. I've noticed the sequence number changes from 0 to 1 when this is done. Not sure what this signifies, other than the obvious when it is part of a set and there is 3 tapes making up a sequence of 1,2 and 3.
Correct me if I get this wrong – you have to erase tape media that was previously exported otherwise it is not being marked as free be retention?
Registry entry is required to prevent job from timing out (default is 60s which is too slow for our system apparently)
Would you like to see the time out value as a part of the UI? And I wonder about advanced logic you mentioned…
Tape backup jobs will copy ALL files that it thinks are not on a tape the first time, or any time it is run. (Tech Support tells me I can clone a job to prevent this from happening, but this is just nuts.)
I am with you on that and I can say that we have enhanced the synthesized full backup mechanism and by default, it would generate only the last missed full backup according to the schedule. I believe the main idea here is not to generate extra full backup on tape (so It would not spoil the tape media free space). The fixed or enhanced behavior will be available in the closest patch.

By the way, if you are using the incremental with periodic fulls, the tape job should prompt you on creation to push all existing chain or start from the last vbk file.
Tape Libraries Discovery jobs will start and hang if a multi-tape erase job is running. It shows a start time of two weeks prior???
Really interested why it’s happening, have you check with the support team? (I am sorry, right now I don’t have the ability to check the support case you posted)
The option to put multiple jobs on the same tape or span all jobs over a set of tapes should be offered.
Here I need to ask you about your tape job and media pool setup – you can use all existing tape space by setting the job to write into the media pool with the continue to use existing media set option enabled (or at least, create new media set on specific day). Or you are referring to multiple media pools / tape jobs?

Thank you once again. I look forward to your reply.
ccatlett1984
Enthusiast
Posts: 83
Liked: 9 times
Joined: Oct 31, 2013 5:11 pm
Full Name: Chris Catlett
Contact:

Re: Suggestings for improving tape UI and reporting

Post by ccatlett1984 »

Another report that would be nice, would be all VM restore points on a given "tape media set".
Our management wants hard copies to offsite with the tapes.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Suggestings for improving tape UI and reporting

Post by veremin »

Haven't you already seen Tape Report available in Veeam ONE; isn't that something you're looking for? Thanks.
ccatlett1984
Enthusiast
Posts: 83
Liked: 9 times
Joined: Oct 31, 2013 5:11 pm
Full Name: Chris Catlett
Contact:

Re: Suggestings for improving tape UI and reporting

Post by ccatlett1984 »

That report is not really what I'm looking for.

I want to be able to select a tape media set, and see the jobs / restore points on that set.

The Tape report isn't really what I'm looking for. The use case, is we print a hard copy that goes offsite with the tapes. I don't want that report to have other media sets on it.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Suggestings for improving tape UI and reporting

Post by Shestakov »

Hello Chris,
What if we add a column in the Tape Backups report showing which media set(s)/pool those tapes belong to? Will it be efficient?
Do you have any other preferences about Veeam One Tape-related reports?
Thanks!
ccatlett1984
Enthusiast
Posts: 83
Liked: 9 times
Joined: Oct 31, 2013 5:11 pm
Full Name: Chris Catlett
Contact:

Re: Suggestings for improving tape UI and reporting

Post by ccatlett1984 »

Not really. This report will be ran when media sets are sent offsite. I don't see it being useful to print a report that will just continue to grow with time, with more useless data each run.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Suggestings for improving tape UI and reporting

Post by veremin »

Not sure whether it helps, but there is a script previously created by me that shows you files (be it backup ones or just ordinary ones) and barcodes of media those reside on. Thanks.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Suggestings for improving tape UI and reporting

Post by Shestakov »

Chris,
So, probably we need another report showing latest runs of backup and file -to-tape jobs and tape IDs used.
Thanks for the feedback!
futureweb
Enthusiast
Posts: 82
Liked: 7 times
Joined: Sep 03, 2015 12:15 am
Full Name: Patrick
Contact:

Re: Suggestings for improving tape UI and reporting

Post by futureweb »

Hi,

How do you accomplish, that at the retention report you know.
- Which tapes to remove from the library?
Once a new Set is created, the old one needs to be removed. How do you know it's save to remove?

I see at "Other Media" all Tapes, where you have to look up, which has the Status "Online", but Online and Offline are mixed.

Is there any way to only show the Online-Tapes at "Other Media"?

Basically we need:
- Expired Tapes to know which has to be placed inside
- Online Media that is ready to go offsite.
- Good would also be the Vault, where it has to go. We handled that, as we named the Media Pool accordinly, but using a 2nd report for only Vault makes it harder.

Do I miss something how you can accomplish that, and can we modify the reports ourself other than with Custom Reports that is limited, or the limited settings on the report itself?

Thanks
Patrick
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Suggestings for improving tape UI and reporting

Post by Shestakov »

Hello Patrick,
futureweb wrote:Once a new Set is created, the old one needs to be removed. How do you know it's save to remove?
I guess we are talking about Tape Media Retention Period report.
Mediaset is consider as expired not after some job is done, but after preset number of days. We are notifying that those tapes protection period is expired.
We are also planning to add a report providing information about automatically exported tapes.
futureweb wrote:Is there any way to only show the Online-Tapes at "Other Media"?
The only way now is to save the report in .xlsx and filter offline tapes in the column. Would you like to have an option in the report parameters?
Thanks for the feedback!
futureweb
Enthusiast
Posts: 82
Liked: 7 times
Joined: Sep 03, 2015 12:15 am
Full Name: Patrick
Contact:

Re: Suggestings for improving tape UI and reporting

Post by futureweb »

The only way now is to save the report in .xlsx and filter offline tapes in the column. Would you like to have an option in the report parameters?
Thanks for the feedback!
This would be great, this way I could have make 2 reports.
1) with the other tapes, but only Online
2) with the expired tapes (as they are not offline).

Not sure if reports could be adjusted by us?

The Excel might be another solution when exporting it to a file, I thought about that to automate it this way, if there is no other solution. So I guess that would work too. Will have to check out the contents of the excel.

Mediaset is consider as expired not after some job is done, but after preset number of days. We are notifying that those tapes protection period is expired.
Yes, Retention Period Report.
Sure about expired tapes, with old ones, I didn't say it correct. I meant: Last night a new media set was created. This needs to be places offsite. Now we need to know, which Tapes need to go where.
Bringing expired back should work as is. But the other way - we just named the Media Set accordingly, so we know where it has to go (additionally placing it into a vault, which is not visible on this report)
but there is no sign that it could be removed savely. (We know usually it's save to remove, as we said, do a new media set once a day. But if there was any delay, it might be it's not offsite ready.
Hope you got what I mean?

I am coming from DPM where the report had those infos, I am just trying to explain to my people, how they know what goes where. With the excel I might be able to do what is desired anyway.

Thanks
Patrick
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Suggestings for improving tape UI and reporting

Post by Shestakov »

futureweb wrote:This would be great, this way I could have make 2 reports.
1) with the other tapes, but only Online
2) with the expired tapes (as they are not offline).

Not sure if reports could be adjusted by us?

The Excel might be another solution when exporting it to a file, I thought about that to automate it this way, if there is no other solution. So I guess that would work too. Will have to check out the contents of the excel.
For now there is no option other than "Excel" to filter the online/offline tapes. All Veeam Product Managers/Analysts review and answer topics in the Forums every day. So yes, you definitely can adjust reports sharing your feedback. That`s one of the main purposes of the Forums. However it doesn`t work like "Ask today, use tomorrow", since all requests are being moderated and discussed before implementation.

As to your request, it may make sense to add an option to the report to filter online/offline tapes, but let`s see if there is a more request such as yours.
futureweb wrote:Bringing expired back should work as is. But the other way - we just named the Media Set accordingly, so we know where it has to go (additionally placing it into a vault, which is not visible on this report)
but there is no sign that it could be removed savely. (We know usually it's save to remove, as we said, do a new media set once a day. But if there was any delay, it might be it's not offsite ready.
Hope you got what I mean?
I see the case. As I said, we are adding a new report with the information about tapes which were used by job, exported and need to be moved offsite. Will it help?
Thanks!
futureweb
Enthusiast
Posts: 82
Liked: 7 times
Joined: Sep 03, 2015 12:15 am
Full Name: Patrick
Contact:

Re: Suggestings for improving tape UI and reporting

Post by futureweb »

I see the case. As I said, we are adding a new report with the information about tapes which were used by job, exported and need to be moved offsite. Will it help?
Yes, I believe that would help too.

Thanks
Patrick
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Suggestings for improving tape UI and reporting

Post by Shestakov »

Great!
Thanks for the feedback once again!
Post Reply

Who is online

Users browsing this forum: No registered users and 14 guests