Veeam VSS Guest Helper Issue - More data is available

[Lewpy]

One of my customer's Veeam server has developed an issue (the server has been in for years, over many Veeam versions and OS rebuilds).
Currently it is Windows Server 2019 with Veeam 11.0.1.1261 (20220302) installed.
It is off the Domain (in a Workgroup) and behind a firewall (it can get out, but very little can get in).
It backups of locally, replicates to a remote repository, and also send data to immutable S3 storage.
It backs up servers in an AD Forest, which has a Root Domain and two Child Domains. Plus some servers off the Domain too.
The last major reconfiguration was about 18 months ago.
As of a week ago, suddenly the Domain Account that Veeam uses to do VSS Guest Processing fails to connect via RPC, and falls back to VIX.
And this is for all servers in all 3 Domains, and they all started showing the issue at the same time.
I only noticed because the Domain Controllers were failing to do guest processing (as VIX and DCs have very stringent requirements).

When performing a Test in the Job's Guest Processing tab, the error comes back as

08/09/2022 16:30:15 :: Connecting to guest OS via RPC Error: Cannot upload file to admin share [\\<server FQDN>\ADMIN$].
Failed to upload file C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\VeeamVssTest_c0e4dc25-7292-486d-8bb9-33faea43f2f3.txt via administrative share
Cannot copy file. Source file: [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\VeeamVssTest_c0e4dc25-7292-486d-8bb9-33faea43f2f3.txt]. Target file: [\\<server FQDN>\ADMIN$\VeeamGuestHelpersTest\VeeamVssTest_c0e4dc25-7292-486d-8bb9-33faea43f2f3.txt].
CopyFile() failed.
Win32 error:More data is available.
Code: 234

Now "More data is available" makes me think that a buffer is too small to hold the data that RPC wants to return to Veeam itself?

If I manually use Windows Explorer, connect to the server using the same FQDN as Veeam, and authenticate using the same credentials as Veeam, then I can browse the ADMIN$ share and copy files to the server successfully.
So I know it isn't a firewall issue.

I've taken the following steps

• They were on Veeam Backup&Replication v11.0.1.1261 (20211005), so I upgrade them to the latest patch.
• The Veeam Backup server has been rebooted.
• Some of the backed up servers have been rebooted.
• The password for the Domain account Veeam uses has been changed.

None of these resolved the issue.

While the customer says nothing has changed, I can't rule that out.
Antivirus is rolling out a newer version, but not consistently upgraded on all machines at the moment. And nothing in the logs to suggest it is interfering.
They have some new EDR software running, but it has been in-place for a month or two and is only in "logging mode".

Anyone with any thoughts as to what could be the issue?
Or does this need a Support Ticket raised?

Thanks,
Lewis.

[PetrM]

Hi Lewis,

Certainly, you need to raise a support request because it's a technical issue and we cannot work with technical problems effectively on this forum. Please share a support case ID for our reference.

It's also difficult to comment on the source of the mentioned error but it seems your assumption about buffer size is close to the truth. Apparently, it does not come from our code as the error contains Win32 error code returned as a result of CopyFile() function. I'd recommend checking this KB while you're waiting for the results of analysis from our support team.

Thanks!

[Lewpy]

Hi Petr,

The customer has raised a Case (# 05623039).
Initial suggestions are around AV exclusions, but we've disabled the AV product and it hasn't resolved the issue.
And the Guest Processing Agent is the backup server itself (explicitly selected in the Backup jobs), so we know that a manual Windows Explorer file copy works fine between the same machines as Veeam is trying to do.

We'll see what the next steps are.

Thanks,
Lewis.

[PetrM]

Hi Lewis,

Many thanks for the update! Let's wait for the results of the research. I'm sure that our engineers will be able to find the root cause soon.

Thanks!