Performance degradation after applying fix (Case #07096373)

[mrjensen5147]

Hello,

After applying a private fix for v7.1.0.1401 (Fix_641296_7831ceb086) to fix the ‘Invalid serialized stream, unknown opcode found: 4' error.
Then we've seen a massive degradation in the performance. The ‘Invalid serialized stream, unknown opcode found: 4' issue is solved, however we saw on the servers, where we didn't apply the fix, that it got automaticly resolved aswell.

The issue then stands, the servers where we applied the fix, is having 200+ mail jobs, running for over 24 hours. Where the same jobs, usually ran for for around 1 hour (Alot less than an hour, and some for more)

We have a case running #07096373. But we need to know if anyone has seen similar behaviour or knows of a fix for this.

- Mathias

[Mildur]

Hi Matthias

Thanks for the case number.
I'm personally not aware of similar reports. Our support team will be aware if others have reported the same issue.
Therefore I suggest to continue with our support team on this case. You may always ask for a case escalation through our Talk to a manager button if you feel the case goes nowhere:
https://www.veeam.com/kb2320

Best,
Fabian

[MaartenA]

I'm not sure if this is the same problem but we also had problems with one proxy while the others with the same fix did not. Exchange jobs that were normally completed quickly also ran for +24 hours. We installed the latest update 1501 and had no more problems after that.

[mrjensen5147]

We have 2 VBOs, where we see the issue.
One of them is on version 1401, where we see this. But the other we also upgraded to 1501, but we still see the same issue. So i'm not sure if we're just really unlucky.

[Ben.online]

I'm pretty sure we have the same issue since 27 dec 23 case #07067854
So well before the xml fix..
i did an upgrade from 7.0.0.4388_P20231015 --> 7.1.0.1401_P20231218 and than the issue appeared

16-12-23 issue Exchange Backup Fails With: "There is an error in XML document" downloaded update files
22-12-23 7.0.0.2914 P20230302 upgraded to version --> 7.0.0.4388_P20231015 and Applied KB4527
27-12-23 3 warnings about a lib could not be found There are 2 new updates inc 7A and the exchange xml hotfix KB4527 is now part of the iso
installed both and ran test job 7.0.0.4388_P20231015 --> 7.1.0.1401_P20231218 version 7a
!! Issue originated here!!
24-1-24 Applied hotfix KB4540 and testing "Invalid serialized stream, unknown opcode found: 4" (veeam.com) Stream issue looks ok performance issue remains
30-1-24 7.1.0.1501_P20240123 geinstalleerd performance issue remains
30-1-24 New server version 7.1.0.1501_P20240123 Same issue after adding 3+ clients




case #07067854 Feel like the case is going nowhere after 4 weeks...
Already did an escalation issue lies with R&D ..


Meanwhile i setup a new server and migrated clients to that one.
Everything seemed fine at first.
Server reboots, service restarts, rescan proxy all worked fast and with no issue
Running (lardge 1400+ objects) jobs was fast even when the other server with same target was stalling for hours or days. Further ruling out the local S3 as an issue
I was unable to break/reporduce the issue. BUT after migrating more clients (current 6 or 7) the issue came back

It looks like the proxy service is unable to multitask after I think the upgrade to 7A
maybe only combined with object storage repositories and local cache.


When 2-3+ jobs are doing something running or retry it seems that the proxy just drowns in it's tasks.

No matter if it is a big workload just doing multiple tasks causes the service to drown.
With jobs that are able to finish in 5 minutes taking up 24+ hours and if the next job schedual starts it will end up with all jobs running and all in a perminent halt
I'ts better to run one big job 1000+ objects and 100GB than running 3 jobs 20 objects 100MB each.

A service restart/reboot also triggers this because the proxy service most likly is doing a lot of init tasks at that point, making the server useless for 3-6 hours.



Only workaround until now is: avoid reboots and proxy rescans, disable the jobs, run them manual sequentially.
Then the performance issue isn't there

Support has been most helpfull BUT has not been able to fiix or escalate enough to get this resolved

[Ben.online]

It would be realy interesting to know if anyone has this issue with any V7.0 build and even more interesting patched after 18-12-23
and if the issue occurs on V7.1.0.1301 or prior

[Ben.online]

Resolved it for now by downgrading a VBO server from 7.1.0.1501_P20240123 --> 7.0.0.4388 P20231015

RnD has been silent for since feb 1
case engineer is requesting multiple updates per day with no result

[Mildur]

Hi Ben

Our RnD team has a suspicion about the root cause.
I expect them to reach out soon through the support ticket.

Best,
Fabian

[Ben.online]

#07067854
installed a hotfix and the issue seems fix by time out cut off
case is under further investigation, but server is usable with the hotfix

[mrjensen5147]

To us, it also seems like the jobs are having a really hard time, just finishing the last object in the jobs. Which can go for 10+ hours.

It is kinda weird that 1 object, can halt it all.

However we haven't heard anything from R&D what so ever. Just that its under investigation. So i'm happy that you guys have recieved a Fix to atleast make the server usable.

[mrjensen5147]

Case #07096373

We ended up recieving an older Veeam.Exchange.Serialization.dll, to change on our Proxy.
After doing this, our jobs can finish in under 24 hours.

Before this all our jobs would hang for over 24 hours.

We split 200+ jobs up, and applied this fix on a proxy with 122 exchange jobs on it. These 122 jobs finished in under 24 hours, where the remaining jobs on the other proxy, that didn't recieve the .dll. Is still waiting to finish, and seems to be in a mode of not doing anything.