Applicationaware backup stopped working with Powershell direct

[JTT]

Veeam app aware backup stopped working with using PS direct.
The problem was, that some kind Microsoft folder file count limit was reached. After cleaning up the folder, backup is working, but every time a new or even a couple of .tmp files are created under C:\Users\veeamserviceaccount\AppData\Local\Temp.
I am now seeing the same kind of behaviour on other clients also.
Veeam support answered, but this does not help for the long rung. So should we deploy manual cleanup script for every server? Why cannot Veeam contact Microsoft and find a solution for this?

This is by Microsoft design as we see in their documentation, and is related to any application calling this .NET Method and as a result the .tmp files are created.
It appears that when initiating the psdirect session, this GetTempFileName Method is being called and such .tmp files gets generated by .NET itself.
As we see in the Microsoft article, this is limited to creating no more than 65535 files before throwing the IOException we have seen.
https://learn.microsoft.com/en-us/dotne ... ew=net-8.0


Now, unfortunately I could not gather more information about this as it is pertaining to 3'dr party applications (Microsoft, .NET) that we cannot troubleshoot.

Let me know if I could be of further assistance.


Veeam Support - Case # 07275337

[david.domask]

Hi JTT,

Thank you for sharing the case, it's an interesting situation I've not seen before.

Reading the case, I think the original Engineer's push to archive may have been a slight misunderstanding, I expect perhaps they did not connect that the files may be related to the backup process, and looks like the case was escalated to the Advanced Support Team. Please continue with Support on the investigation, as I think the last piece of the puzzle is "why are these files accumulating so fast and what is creating them/not deleting them?".

Please note if there are concerns on the processing of the case, feel free to use the Talk to a Manager button; this will connect you with Support Management and they will review the case and help with addressing the concerns.

For right now though, I think the case is with the appropriate resources, and Advanced Support is trying to isolate the issue a bit further.

[JTT]

Well the customer already contacted the manager:
Thank you for contacting us.
My name is Dmitry, I am a team leader for the Advanced Technical Support EMEA team.
The situation you experience is rather unique for us and judging by the nature of it it seems to be related to the specifics of you infrastructure setup rather than to Veeam code. I can see the engineer is trying to figure out what exactly is interfering with our normal business logic. We have a couple of ideas what else we can try, he will update you with the action plan in the next message.

It is not unique, as i wrote, we are seeing the same issue occuring in our shared Hyper-V clusters and VBR backup environment, were we use PS direct.
The difference is, that the case is related to a backup, that is run every hour. In our shared infra, backups does not run so often, that means also less .tmp files created.

[david.domask]

Hi JTT,

I think what was meant there was "it's a rather new situation without any previous cases showing the same behaviors and from the logs there is no deviation from normal Veeam operations in environments where the incorrect behavior is not happening"; that is to say, I don't think they mean to indicate that you don't have this in multiple locations, but more just informing that part of the longer processing is that this is new behavior for them.

So I think it's understood it's happening in multiple locations in your environment, the message is more to explain where the current research is going and why additional research is needed.

Thanks!

[tka]

Hi all!

Same here - did support come up with an explanation or solution?

Thanks!

t.

[david.domask]

Hi tka,

The investigation is still on-going. If you're seeing similar behavior or that PSDirect is no longer working, please open a Support Case and allow Support time to review the situation. Be sure to include logs from one of the affected jobs for Support to review. (Use the 1st radio button, select one of the affected jobs; it should be enough to start the investigation)

Thanks!