Component Being Marked as Known Virus (RPCAssemblyServer.exe) During Install

[bg.ranken]

We were performing an upgrade from VeeamONE 12 to 12.1, and our AV solution (Carbon Black) blocked RPCAssemblyServer.exe and kill the install due to it being "Known Virus". In uploading it to VirusTotal, it appears it's flagged by two different vendors.

https://www.virustotal.com/gui/file/9af ... ?nocache=1

Can someone from Veeam confirm that this is a known file that should be included with VeeamONE? Looking at the EXE is it not signed by Veeam like the other EXE's typically are. If so you may need to submit something to these AV vendors to confirm the EXE is not a virus.

[jorgedlcruz]

Hello,
Thanks for reporting it. I am sending this right away to our RnD team.

Have you opened a support ticket as well? Just asking so I can put it together.

Thank you

[bg.ranken]

Yes I did, here's the case number: 07100412.

Currently our install was broken since it failed midway through, so opened a sev 2 ticket. Our security team isn't letting us proceed until we get some level of confirmation that this is a false positives, so at the moment our VeeamONE instance is down.

Normally we would have just bypassed this as it's most likely a false positive, but after SolarWinds our security team got a lot more strict on these types of things.

[jorgedlcruz]

Thanks for the case. And totally understand, I've sent it to the team right after your report, and we will come back to you in a few hours.

Thank you again

[bg.ranken]

Thank you, and no rush for us, at least as of now, we delayed our upgrade. I'd be more concerned for others that might have already upgraded to 12.1, if that component gets called then their AV might do the same thing and kill the task and/or delete it.

[jmacdone]

Same. Veeam upgrade + Carbon Black triggered an alert because of RPCAssemblyServer.exe - https://www.virustotal.com/gui/file/9af ... /detection

Any confirmation of false-positive available?

[jorgedlcruz]

Hello, we double checked this file – this file really belongs to our product (or more precise – it's used by Setup wizard only). Our Veeam Security Engineers reported false positives to both vendors – Carbon Black and SecureAge, but as they are third-party vendors, they might react slow, or never, let's see how it goes.

The file RPCAssemblyServer.exe doesn't contain Veeam digital signature indeed, but this is not a big problem – this file is embedded into a DLL which at the same time is embedded into an MSI, while both DLL and MSI are properly signed by us.

In future versions we are going to also sign the RPCAssemblyServer.exe.

Thanks for the reporting.

Bottom line: False positive.

[brkdncr]

After solarwinds, a lot of red flags go off when a vendor says “yeah it’s ours, don’t worry about it.”

Let’s start with a hash to confirm the engineers are working with the same file that’s published.

Next, let’s find out if this is a supply chain attack or really a false positive.

[NickKulkarni]

@jorgedlcruz

VoIP Software PBX vendor 3CX did exactly the same thing. It's ours its just a false positive.
https://www.cisa.gov/news-events/alerts ... desktopapp
https://www.wired.com/story/3cx-supply- ... times-two/

The Huge 3CX Breach Was Actually 2 Linked Supply Chain Attacks
The mass compromise of the VoIP firm’s customers is the first confirmed incident where one software-supply-chain attack enabled another, researchers say.

FYI I just had one of our AV solutions flag the latest Adobe Reader installer as malicious and put it in Quarantine. Reported it to the AV vendor as it is a waste of time talking to Adobe. Vendor has not come back to tell me why it got detected but have instead silently whitelisted it two days later. Again the lack of Transparency is worrying.

[NickKulkarni]

@brkdncr

Totally agree with you.

[jorgedlcruz]

Hello,
I am discussing with our Security team to provide full clarity on how this file is created/extracted from where, and to get MD5 and SHA from them.

As soon as I have much better understanding, we will share it here, perhaps even on an official KB so it gets officially published outside the forum.

I will let you know soon.

[jorgedlcruz]

I have confirmed now all the steps that lead to this unsigned .exe to be needed, here are the steps.

• When performing a v12.1 install, or update, more specifically the files called VeeamONE.Reporter.WebUI.x64.msi and VeeamONE.Agent.x64.msi are executed. Both signed by Veeam
• Those files will trigger a series of actions, install stuff, uncompress and execute stuff, etc. As part of this MSI, we have the RPCAssemblyCaller.dll signed by Veeam
• If the install, or upgrade are for the Web Client part, or the VeeamONE Agent part, then the dll extracts on a path like "C:\Users\*username*\local\temp\*temp folder*\RPCAssemblyServer.exe" a file called RPCAssemblyServer.exe, not-signed by Veeam currently but still our own file.

More information about the file:

• MD5: 5b95108ba4d6fa565d4402fc4fbd9d38
• SHA256: 9af4ff86a7383324150c563823a938cba50edcf45a8e5ec53d295e45a5cd8af8

Next steps we are taking:

• I am trying to see if we can sign the file and re-upload the ISO without disruption. To be confirmed
• If we can not do first step, discussing and planning how to create a KB for this specific use-case, specially having the hashes out there as well.

Will update you soon

[MPECSInc]

Begs the question since we don't have the environment at the moment to check: Who _is_ the executable signed by?!?

[mkaec]

It's probably unsigned. That's the default unless you go through the steps of doing the signing.

[jorgedlcruz]

Yes, the .exe is currently unsigned, but the file is inside a signed .dll and a signed .msi. And this file gets extracted at an specific point to be executed and it is closed right after, it doesn't live later anywhere in the Veeam ONE.

We have fixed this problem, signing the file inside already signed dll and already signed msi, and it will be available on the next cumulative release.

As mentioned, we reported it as false positive to Carbon Black and SecureAge, but that can take time if anything at all happens.

[NickKulkarni]

@jorgedlcruz

"Yes, the .exe is currently unsigned, but the file is inside a signed .dll and a signed .msi. And this file gets extracted at an specific point to be executed and it is closed right after, it doesn't live later anywhere in the Veeam ONE."

Sadly, whilst I see your logic in the above statement that is no longer how the EDR software treat this and for a very good reason. That behavoiur is very similar to the behaviour of the 3CX supply chain attack. As a result, most EDR software vendors are now actively scanning this type of behaviour. QED any unsigned files created by the signed process will most likely to be quarantined on sight if they cannot be marked safe.

[jorgedlcruz]

Hello,
Thank you Nicholas. I understand not having the .exe signed triggered this problem due the behaviour.

The file itself is legit, we shared the MD5 and SHA256 directly so you can check if required.
This problem is fixed on next releases, simply by signing digitally, the .EXE is the very same (no additional payloads or supply chain have happened here).

We reported it as false positive to Carbon Black and SecureAge.

Let us know if you have any further questions.

[NickKulkarni]

@jorgedlcruz

Hi Jorge,
Thanks for your reply and thank you for also sharing the MD5 and SHA256. A lot of us will sleep a little better knowing we can positively ID the file if we add it as an exception. I am also very glad to hear it has been fixed in the next release.

Please do not take offence at my comments, none is, or was, intended. I was just trying to point out that being inside a signed DLL and a signed .MSI can no longer be considered "safe" by any good quality EDR solution.

Here my blame is aimed mainly at the criminals in the cyberworld, not your product. Because of them deploying an unsigned executable or dll during an install, however briefly, from inside a signed dll or .msi can no longer be considered safe or to comply with the principles of "safe by design". That last part is a bit of buzzword now but, considering how valuable Veeam is in the fight against ransomware, it is something Veeam has to aim for.

I would also suggest that Carbon Black and SecureAge didn't really have a "false positive" if they were heuristically identifying the behavour, not the file itself, as suspicious. Everything is a "balance of probabilities" when analysing behvaiour. That there is no malicious payload is much harder, if not impossible, to ascertain with a lightweight scan during install. The old adage of "if it walks like a duck, talks like a duck, it is a duck" is the safest way to classify malicious behaviour. QED if it is a signed file but deploys an unsigned executable; shoot first, ask questions later.

Clearly the lesson has been learned and, and as you say, it has been fixed in the next release. Do we know when the next release is likely to be?

Thanks and kind regards.
Nick

[mamosorre84]

Hi Jorge,

sorry to open the conversation again.

I wanted to know if you have any other similar reported issues with other antivirus.

In particular we are having issues with a new installation of Veeam ONE 12.1 and Crowdstrike Falcon installed.
We are still analyzing the logs but suspect that the problem may be related to what was reported in this post.

Thanks

Marco S.

[jorgedlcruz]

Hello Marco,
We have not submitted it to Crowdstrike, we did it to Carbon Black and SecureAge but no answers so far. Do you mind creating a support ticket about the issue as well, so we got it controlled?

Thank you.