Multiple issues with V12

[masonit]

Hi!

After upgrade to v12 (12.0.0.1420 P20230412) we are seeing alot of issues that we did not see in v11. I have looked at the top issue tracker thread and also trying to search to see if there is other with same problem as we have. But to my surprise I can't find any useful information. But at the same time I don't think we are the only one with these issues. Below is some of the problems:

"All VMs have been prepared for storage snapshot"
- We have some jobs where we use storage snapshot. This step in v12 is taking a very longtime compared to v11. We are seeing this issue on a couple of jobs. But there is one job that is really bad. This step took ~3 - 5 min in v11. Now in v12 it takes ~6 h everyday. We can see that this problem started at the first run with v12.. Case: 06063316

"Error: Failed to check whether snapshot is in progress (network mode).RPC function call failed. Function name: [IsSnapshotInProgress]. Target machine:.RPC error:The RPC server is unavailable. Code: 1722
- We have started seeing many rpc errors during application aware processing. Also this is something that has started with the first run i v12. These are jobs that has worked fine with v11. But after upgrade not working anymore.. Case: 06066873

Veeam remote console crash / hangs
- 30 - 40% the remote console crash when we edit jobs. I happens only during edit of jobs. Never seen this before. Started with version 12. First we thought this happened beacuse of antivirus software. But we have the same issues also on servers without antivirus.

Error: Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown.
- VM backup fails with above error. Never seen this issue before v12.

Error: VSSControl: Failed to freeze guest over Hypervisor API, wait timeout
- VM backup fails with above error. These are vm backup that has worked fine in v11 but started getting this error in v12.

Troubleshooting steps we have taken is to restart all Windows Veeam servers in our environments and for some of the issues a support case is opened. Problem with the support cases is that they are threated like individual cases and troubleshooted like it is not a v12 issue. Even though the problem started with the first run in v12. Any one else seeing the same problems? We backup many vms and most of the still work fine in v12. But we are seeing alot more errors in v12 than with v11.

\Maso

[BackupBytesTim]

I can say we also see a lot more issues with V12, and associated agent versions, and we're working with Veeam support on several. Which is slow going and I'll agree that support usually doesn't want to troubleshoot more than surface issues. I imagine that's partially because they either don't want to fix issues that aren't experienced by everyone, or there's some reasoning that they're not explaining for how something like an unreliable connection results in things like thousands of times the data transfer as should be required, and other things like backup files that take up more space on the repository than the data did on the source computer. Hoping it's just that they're still overloaded with people needing support, but even if that's the case it would seem that they're more overloaded than usual after an update, which suggests more people having problems.

On to useful things, I can say we also see RPC issues semi-regularly, for us it's with Agent backups, not sure if that's the case for you or not, but restarting the agent services fixes the issue for a while.

For your AgentClosedException issue, it took us a while to figure it out too, but now that's actually a known issue affecting older Windows versions, Windows 7 and Server 2008.
The fix is here https://www.veeam.com/kb4419

Alternatively it should also be fixed in the latest agent version, if they're updated fully and still having the problem that might be a different issue.

As per usual there's no technical explanation of the problem, but my understanding is that even with the Agent installed on a computer it's still trying to access data by logging in remotely to the computer, normally it does this without actually going over a local network, using a "localhost" address, and the latest Agent update broke that and caused it to attempt a connection over the actual network, which was then blocked by the computer either at the firewall level or just because it wasn't configured for remote management. Not entirely sure why it only happened on old versions of Windows though. The patch can be installed on affected computers via the VSPC interface.

[masonit]

Thanks for reply. Our agents backup works fine. All our errors is on vsphere vm backup.

[BackupBytesTim]

Well, I'm interested in any answer you get from Veeam on those issues, but unfortunately I can't offer any help there. Our customer's are almost entirely using agents for backups and there's very few with VMs at all, and the ones with VMs mostly use Hyper-V.

[Kwa-GJ]

About this:

"Error: Failed to check whether snapshot is in progress (network mode).RPC function call failed. Function name: [IsSnapshotInProgress]. Target machine:.RPC error:The RPC server is unavailable. Code: 1722
- We have started seeing many rpc errors during application aware processing. Also this is something that has started with the first run i v12. These are jobs that has worked fine with v11. But after upgrade not working anymore.. Case: 06066873

We see exactly the same, but it seems to be only happening with vm's running old versions of vmware tools.

[lneuhoff]

Join the club
Support is not interested in doing any type of deep dive into them.
ONLY thing they do are to keep asking for logs.
We have raised multiple tickets and they have gone nowhere.
Working on going back to v11 since that worked a lot better.

[Gostev]

We see exactly the same, but it seems to be only happening with vm's running old versions of vmware tools.

Using outdated VMware Tools may cause various issues indeed, especially as you keep advancing vSphere version. I remember at least a few in the past years...

[Gostev]

We have raised multiple tickets and they have gone nowhere.

Something is off as I only see 2 tickets in the past 12 months on your email address. One of them is asking for a design/implementation advice, as opposed to a technical issue. And the latest is opened for a free product. Are you sure you're opening the support cases correctly, using your paid contract? I also let the support management know.

[sandsturm]

Error: Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown.
- VM backup fails with above error. Never seen this issue before v12.

We had the same issue (among many others) after upgrading to V12 and were able to solve above issue by decreasing the current amount of "Max concurrent tasks" on the proxy role itself and also on repository role (proxy and repository role are running on the same machine). Maybe this helps you as well?

sandsturm

[Gostev]

The amount of RAM is under the System Requirements for the number of concurrent tasks perhaps? It's a pretty common issue for users to oversubscribe backup infrastructure servers significantly, ignoring System Requirements.

[sandsturm]

I am aware of the system requirements and we know that the proxy/repo server is quite at the limit of these limits, but we haven't seen the error with Veeam v11: 'Veeam.Backup.AgentProvider.AgentClosedException' and with this message the problem that some of the VMs could not be backed up any more. This message appeared only after the upgrade to V12 and the message did not directly lead me to believe that this could be a resource problem.

[mcz]

Problem with the support cases is that they are threated like individual cases and troubleshooted like it is not a v12 issue. Even though the problem started with the first run in v12.

I do experience exactly the same and I'd ask veeam to probably change the approach for such cases as you do waste and loose a lot of time. I mean if it's that obvious that a job fails since the upgrade, someone should probably say "ok, let us check what has changed in this codepart to get an indication why it suddenly fails". If you went that road, I'm sure you would much faster find the cause of the issue most of the time...

[F.A.N.]

Support is pushing us towards V12, stating in most tickets that "this will only be fixed in V12".
Seeing I read more and more issue's in V12, I'm remaining at V11 for a while.

[Mildur]

Hi FAN

Version 11 is already in the End of Fix phase (February 2023):
https://www.veeam.com/product-lifecycle.html

Exceptions of this policy will be mainly made for security updates.

Best,
Fabian

[sandsturm]

Problem with the support cases is that they are threated like individual cases and troubleshooted like it is not a v12 issue. Even though the problem started with the first run in v12.

I do experience exactly the same and I'd ask veeam to probably change the approach for such cases as you do waste and loose a lot of time. I mean if it's that obvious that a job fails since the upgrade, someone should probably say "ok, let us check what has changed in this codepart to get an indication why it suddenly fails". If you went that road, I'm sure you would much faster find the cause of the issue most of the time...

exactly the same experience in our environment. It is a mess to have different Veeam cases which maybe all have the same reason (update to V12). We had to provide multiple time the same logs for different cases, Veeam was not able to use the same logs for another case. --> Nightmare

[masonit]

Hi!

We have managed to solve some of our problems.

We saw both errors below mainly for one virtual environment. We knew that there was a lot of portfiltering between networks in this environment and thats why we thought the errors could have something to do with firewalls blocking the traffic. The change we made in firewall was to increase the default dynamic rpc range 49152-65535. To 6005-65535 instead for backup traffic. After that below errors disapeared. As I said before these error started at the first run i v12. Either there is a change in v12 in what ports are used. Or there was a change in fw on the same day as we upgraded to v12, but I don't think so. Not sure what it is but can't see any updates about what ports used in v12. https://helpcenter.veeam.com/docs/backu ... ml?ver=120

- "All VMs have been prepared for storage snapshot"
- "Error: Failed to check whether snapshot is in progress (network mode).RPC function call failed. Function name: [IsSnapshotInProgress]. Target machine:.RPC error:The RPC server is unavailable. Code: 1722

We sometimes still see this error below but it is a lot better since restart of all Windows servers in the Veeam environment. I think best practice after Veeam upgrade is to restart everything. Even though the installation does not say so. I remember we had a similuar issue in v11 were after some days / weeks after upgrade many errors started to appear. Same then, a restart solved it.
-Error: Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown.

We have also seen many of this errors below on backup of vcloud vms. Mainly the vapp backup fails with this error. Same here, a restart of vbr seems have pretty much solved it.
Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.

\Maso

[hemexe]

Hello. I'm usually someone that loves changes or updates to software, but this version is so freaking buggy.
Moving tiles, which I do regularly is now a chore. When I select multiple processes by using shift or ctrl, they move "sometimes" or not at all.
When scrolling through the finished process tiles using the mouse wheel my part zooms in and out instead of the process list. Sometimes even both the part and process list scroll as well.

I've never had these issues on previous versions.
I like the look of V12, don't get me wrong.........but this version feels to me way more like a beta build than a finished product.

[RubinCompServ]

@hemexe,

Are you referring to VeeamONE or VBR?