Host-based backup of VMware vSphere VMs.
Post Reply
SvenP
Influencer
Posts: 13
Liked: never
Joined: Mar 31, 2016 12:53 pm
Full Name: Sven Putze
Contact:

Version 12 and Application_Error Faulting application name: VeeamTransportSvc.exe

Post by SvenP »

Case # 07039359

Since we upgraded to Version 12, the VeeamTransportSvc.exe crashes approximately once a week.
of course we opened a ticket, we did some log file ping pong, we updated the components on the proxy, since the update routine did not do that correctly, we checked the antivrus settings, we confirmed that the backup storage is available all night albeit Veeam error messages claim otherwise.

A frustrating experience, but since most of the backups worked, it was inconvenient.

Last friday we updated to version 12.1.0.2131 (as requested be Veeam support) and now the shit really hits the fan.

- Somewhere around saturday morning the Veeam version 12.1.0.2131 decided it would not trust the certifcate of one of our HPE 3PARs.
It was not caused by an expired certificate. Nevertheless, going through the storage system dialog and confirming the certficate fixed that. But that should not have happened in the first place! Many backups failed because of that!
- Memory consuption of the Veeam backup server:
- With version 11 we had an average of 40% RAM usage on the 90GB machine.
- With version 12.0 this raised to an avergy of 70% RAM usage on the 90GB machine!
- With version 12.1.0.2131 it reaches an whopping average of 94% RAM usage on the 90GB machine with a tendency of swapping during the hot backup phase!
- Here is a selection of error from the last night:
> Error: Failed to prepare VM for processing: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
>
> Error: Failed to prepare VM for processing: No more threads can be created in the system. RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [IP:6162].
>
> Unable to allocate processing resources. Error: Some extents storing previous backup files are offline
>
> Error: Failed to connect to any of extents. Failed to connect to Veeam Data Mover Service on host 'NAME', port '6162'
>
> Resource not ready: Backup repository Unable to allocate processing resources. Error: No scale-out repository extents are available
>
> Failed to pre-process the job Error: [AP] (f5c0e8ae) Can't run command because agent is closed or has error
>
> Error: Not enough server memory resources are available to process this command. RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [IP:6162].
>
> Error: The RPC server is unavailable. RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [IP:6162].

As with version 12.0 the message "Error: Some extents storing previous backup files are offline" is still all lie, our monitoring shows that the backup storage is available.
- Version 12.1.0.2131 causes now frequent error messages like "Error: Cannot find LUN with SCSI ID 6000(...)"

Reminds strongly of an old KB: https://www.veeam.com/kb2365

We fixed this by cleaning the registry once in the morning since the yeas 2017! Now we do it every two hours during the night and it still does not help!
Retries work after a manual cleanup in the morning. Something has changed in the behavior but not for the better!
- The Veeam Data Mover Service crashes now every night! With version 12.0 it was once every week. What a disappointment!

Ah, did i mention that jobs that failed for whatever reason sit there and do nothing for almost an hour before they try again? So they are cycling and blocking each other from the resources.

Now we upgraded to 12.1.1.56. Let's see what happens next night.
SvenP
Influencer
Posts: 13
Liked: never
Joined: Mar 31, 2016 12:53 pm
Full Name: Sven Putze
Contact:

Re: Version 12 and Application_Error Faulting application name: VeeamTransportSvc.exe

Post by SvenP »

- Version 12.1.1.56 did not help, too.
- Funny observation: the storage snapshot are created but some of them are not exported from the 3PAR to the proxy server. This may be a random view in time, like e.g. snapshots that are being teared down again. Or this explains why the proxy cannot see the LUNs.
- All jobs are configured to fall back to network transport but they won't. Probably because the snapshot is created successfully. IMHO a logic bug.
- 128GB memory for the backup server is still not enough.
- Veeam support has announced to come back to us this evening. Let's see what they come up with.
- Meanwhile we created some proxy VMs and do most of the backups with pure network transport. Now the VM snapshots are kept a long time and removing them is a challenge for VMs with heavy I/O. But hey, the memory consumption is back to 40GB.
- And still some jobs block each other from backup storage access and run in circles.
- VeeamAgent.exe crashed once on one of the proxy servers that now plays the role of a pure storage server.
Mildur
Product Manager
Posts: 10100
Liked: 2696 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Version 12 and Application_Error Faulting application name: VeeamTransportSvc.exe

Post by Mildur »

Hello Sven

Thanks for the summary and case number.
I see that the case has escalated to the next tier. Please keep us updated.

Best,
Fabian
Product Management Analyst @ Veeam Software
SvenP
Influencer
Posts: 13
Liked: never
Joined: Mar 31, 2016 12:53 pm
Full Name: Sven Putze
Contact:

Re: Version 12 and Application_Error Faulting application name: VeeamTransportSvc.exe

Post by SvenP »

Well, well, well, first sign of the Veeam Support. The VeeamTransportSvc.exe is a *drumroll* 32-bit process and crashes because it hits the 4GB limit in our environment. A 64-bit version cannot be supplied as hotfix, have a good day.

And of course the other problems were not addressed at all.

Well, hello? The year 2003 is calling! Microsoft published Windows Server Enterprise/Datacenter Edition with 64 bit. And even if we would accept the fact that Veeam distributes 32 bit software to this day, wouldn't it be nice if the process manages its memory itself and enforces some kind of limit for parallel jobs?

Where are my smelling salts?
SvenP
Influencer
Posts: 13
Liked: never
Joined: Mar 31, 2016 12:53 pm
Full Name: Sven Putze
Contact:

Re: Version 12 and Application_Error Faulting application name: VeeamTransportSvc.exe

Post by SvenP »

Well, after weeks with almost no progress we and the support team have learned that the log file collection tool in the Veeam console does not collect all log files that the support wants or needs. Hopefully they communicate that to the dev team.
Now another support team member works on the case and guess what: "please provide some log files".
We are not happy to say the least. Licenses end this year...hm...
Mildur
Product Manager
Posts: 10100
Liked: 2696 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Version 12 and Application_Error Faulting application name: VeeamTransportSvc.exe

Post by Mildur »

Hello Sven

I apologize for the late answer. We were discussing your case today with our RnD team.

Can you please confirm again for me, there was absolutely no change between V11 and the update to V12?
- Number of SOBR extends?
- Number of jobs?
- Number of workloads?
- Any other configuration?

Can you also please confirm how many tasks you have set on your proxy and repository server?

Best,
Fabian
Product Management Analyst @ Veeam Software
djbo
Novice
Posts: 6
Liked: never
Joined: Jun 18, 2012 10:46 am
Full Name: David Valverde
Contact:

Re: Version 12 and Application_Error Faulting application name: VeeamTransportSvc.exe

Post by djbo »

Hello, a question, what version of vmware are you using and what version of HW on the machines that require an effort to delete the snapshot?

Regards
Mildur
Product Manager
Posts: 10100
Liked: 2696 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Version 12 and Application_Error Faulting application name: VeeamTransportSvc.exe

Post by Mildur »

Hi David

How is that question related to the transport service?
Do you see the same issue? If so, what's the case number?

Best,
Fabian
Product Management Analyst @ Veeam Software
Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot], Bing [Bot], JChen522 and 60 guests