Comprehensive data protection for all workloads
Post Reply
TitaniumCoder477
Veteran
Posts: 315
Liked: 48 times
Joined: Apr 07, 2015 1:53 pm
Full Name: James Wilmoth
Location: Kannapolis, North Carolina, USA
Contact:

Wild logging and scary messages

Post by TitaniumCoder477 » 4 people like this post

Let me start with the good! V11 and V12 have included so many amazing features that I feel like a dirt bag for coming to this forum with my two beefs. The cool stuff I've seen and used already include moving/copying backups around, exporting to more flexible targets, deploying roles to more Linux servers instead of Windows servers, and moving away from SQL Express. I am very happy with all of these, the only downside of which is making sure the team looks for the best option in any situation vs. just go for the one that they are accustomed to using.

That said, V11 and V12 seemed to include two things that I really don't like. First, with V11, the logging in the UI itself has become a heck of a lot more cryptic. It's fine to include this kind of logging in the log files, but it's not OK to expect users to read it and know what the heck they should do. There needs to be a layer of abstraction or interpretation that takes place. Here is an example:
9/20/2023 5:10:27 AM :: Processing [REDACTED] Error: 'Failed to final AES Decrypt context.' (error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt)
AesAlg failed to decrypt, keySet: ID: 856c4c08598e5e7bbe11fa52e26fff28 (session), keys: 1, repair records: 1 (master keys: 9c13d731a1150b9378e800e3dcbfaf04)
Failed to upload disk '[REDACTED]_1-flat.vmdk'
Agent failed to process method {DataTransfer.SyncDisk}.

That message is entirely unhelpful to even a senior engineer with lots of experience*. It's my observation that Veeam's logging has taken a drastic turn for the worst in these two versions. And by worst, I mean "more complex" and "less intuitive/helpful." I get it that Veeam often just passes through error messages from the underlying infra, and that was OK for the occasional VMware or Hyper-V error. But this is not acceptable for every error message. If Veeam wants to cut down on calls to support (which is the goal of every software development company on the planet), they should make the error messages more helpful, not less helpful.

Secondly, stop with the super scary, paragraph-long dialog boxes/messages! These seemed to appear in V12 and usually involve the chains. But at this time, I feel like they are hiding around every corner, ready to jump out yelling like an Irish banshee! For example, trying to upgrade a backup job chain results in a massive warning that states a problem, warns of a consequence, practically begs for the job to be recreated, and then asks if I want to proceed?? Heck no. Anyone reading that paragraph would have to be insane to click the OK button. I'm fairly sure a bolt of lightning is included in the fine print somewhere too. My point here is that these need to be reworked. The larger question is what design changes have made it necessary to show these. It almost feels like Veeam got burned badly and had their lawyers write this verbiage. So instead of being helpful, providing guidance and recommendations, it's more like "say as much as you can to offload liability to the simpleton who clicks the OK button." :)

Image

There, I've said it. Now thank you again for an awesome product! :D

*I have over 9 years of supporting, remediating, architecting, and deploying Veeam solutions for all the clients of the MSP I work for. I have more experience and technical knowledge that Veeam's own T1 and T2 engineers. I usually "dumpster dive" the log files myself looking for technical clues before even thinking about opening a ticket because most of the issues I can figure out or resolve myself.
Mildur
Product Manager
Posts: 8735
Liked: 2294 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Wild logging and scary messages

Post by Mildur »

Hello James

Thank you for the feedback. All forms of feedback are valuable to us, and we genuinely appreciate your input.
Secondly, stop with the super scary, paragraph-long dialog boxes/messages! These seemed to appear in V12 and usually involve the chains.
I agree. The prompt in the screenshot provided contains an excessive amount of text. Let's see how we can optimize it.
It's fine to include this kind of logging in the log files, but it's not OK to expect users to read it and know what the heck they should do. There needs to be a layer of abstraction or interpretation that takes place. Here is an example:
The error message you've shared does not appear to be specific to Veeam products nor does it seem to originate from Veeam components. When these elements encounter an error, we capture the associated error message and log it in the job session.
Attempting to create wrappers for every conceivable error message to enhance readability is not easy. Numerous external factors come into play, including integrated open-source components, the operating system, VSS (Volume Shadow Copy Service), protected applications, and more. Hiding it completely behind a generic "Session failed" message would also be no help to solve the issue.

I will forward both feedbacks to our teams.

Best,
Fabian
Product Management Analyst @ Veeam Software
BackupBytesTim
Service Provider
Posts: 398
Liked: 57 times
Joined: Apr 29, 2022 2:41 pm
Full Name: Tim
Contact:

Re: Wild logging and scary messages

Post by BackupBytesTim » 1 person likes this post

I have to add that in my experience (and this seems to be the consensus of just about everyone I've discussed it with) the "error messages" Veeam gives have become less useful over time. I've learned that the error messages are typically always either incredibly long, vague, confusing, whatever, possibly passed through from a non-Veeam software component (in which case it's a little excusable on Veeam's part that it's not helpful) or they're overly simple and still useless like "The task failed unexpectedly" is one I get a lot, which means nothing useful, I've seen that particular error message caused by any ridiculous number of different problems and even if it only ever was caused by one single problem, I'd still question it, is a failure ever "expected", no. No it isn't. So saying the failure was "unexpected" is like... obviously it wasn't, and not helpful.

I too am used to rummaging through log files (which could also be better formatted in my opinion, but that's a totally different problem and I'd first start by having them at least use the same format across operating systems, sort of adds to the overall lack of cohesiveness Veeam has, like, the Veeam Platform as a whole seems to be a collection of different products developed by different teams, so there's not much similarity between them, even basic things like the format for log files is different). Back on the topic of my rummaging through log files, I can usually find something more useful looking at it myself, which is faster than support, but I've never had to actually look at log files for any other software I've ever used before (unless we count Windows Events, but that's a bit different) so I definitely think Veeam could improve there. I previously suggested having log files be visible via the VSPC directly, so we don't need to download the logs and look through them manually, and if that were a feature I definitely think something to isolate any "Error" or "Warning" lines and show them (plus the lines around them) would be useful.

Unfortunately I must also agree with the above opinion on at least tier 1 support. I find I often have to explain to them how Veeam works, particularly the support staff in the Windows Agent department (I'll admit that the other departments seem more knowledgeable even at the lowest levels of support, but that could be because tier 1 Agent support is expecting Joe Shmoe at his garage workshop to contact them and the VSPC department is only getting requests from service providers). I do understand "new staff still in training" and all that, but as a VCSP it would definitely be helpful if I could just skip low level support, if I'm going so far as to open a case, I can say with 99% certainty that I've already done everything they would ask me to do before they just escalate it anyways. I think as a service provider, who Veeam always refers to as a "partner" I should be getting some sort of higher level support staff automatically than the average individual who just got a license for their one desktop they do their taxes on each year.

Full disclosure, my primary comparison for pretty much everything with Veeam is Acronis, which for this specific comparison displays much more detailed error messages (than "Task failed unexpectedly") right in the web-based management console, but also is more clear and obvious what's wrong because they aren't typically enormously complex. And does also make the full logs available if desired. Also big thing Acronis does that Veeam doesn't (since it's related) is display status information and error or warning messages in real time. With Acronis if there's a problem it'll be displayed immediately, even if the job continues running doing other things (or it was a non-breaking message that wouldn't have stopped the backup anyways), so I can view messages and more detailed status information as the job runs. Also, with Veeam, when the job stops all I can see without downloading the logs is "Success" or "Failed" or "Warning" and whatever single error message Veeam decides to show, but I can't see anything until the job has stopped completely, so if it's just stuck, looking like it's doing nothing, I must download the logs, there is no alternative. Also if the computer goes offline, no option at all because it doesn't report status information back to the VSPC server until it finishes. Acronis would let me see messages along the way, so if a backup appeared "stuck" I could see what it was doing, things like the percentage completed, estimated time remaining, data transferred, data remaining on the disk to process, any messages generated so far, all that, which is much more useful than Veeam's "Running". Also because Acronis would transfer this information to the server in real time, if a computer goes offline at any point I could still see this information.

Just some of those overall things that made Acronis much more user-friendly to me as a service provider, I managed several times the number of systems as I have now with a far lower failure rate and far fewer staff relative to the number of customer devices being backed up, and I did not need to contact Acronis support every week as I do with Veeam, every single week. My overall impression of Veeam is that they're not interested in making support obsolete. In my opinion one of the primary goals of the development team should really be to render support obsolete, except for explaining basic usage to people who probably shouldn't be trying to administrate it themselves. The software should get less buggy, more user-friendly (intuitive, straightforward, easy to use), and generally require fewer support cases over time. However my actual experience with Veeam seems to be the opposite, it really seems like things have gotten buggier with the V12 release, and the interface is still incredibly confusing, and as a long-time system administrator and software developer (mainly for in-house administration tools) I still can't manage to understand from a technical perspective how Veeam's software works or why things were designed the way they were. I do debate on a regular basis with myself whether Veeam's development team intentionally makes software require the support staff to use successfully just so people need to pay for a license to use it, but that doesn't make sense to me as a business practice in this day and age, so again, I don't understand how the software is designed as confusingly as it seems to be.
Post Reply

Who is online

Users browsing this forum: maoty@actwill.com.cn, Semrush [Bot] and 122 guests