weird routing issue

[mcz]

Hi everybody,

I'm using version 10 on a Window 10 1909. Now everything was running quite well until this week when suddenly my surebackup jobs failed. I started an investigation and realized that it's all about the routing. In my virtual lab, I've specified a masquarade network 192.168.255.0/24 and when I run "route print", I always missed this target in the list. So I asked why veeam wouldn't setup this route via "route add". So I've checked the logs.

To my big surprise, I saw that veeam checks for the route and creates it - which mad me even more curious, why this route would finally miss in the list. Now what I did in the meanwhile was changing the masquarade network, rebooting, uninstalling antivirus and so on. Nothing helped. Now I've decided to add the route manually via:

route add 192.168.255.0 mask 255.255.255.0 192.168.0.243 -p

Now the route was in the list but only for a short moment!!! After 10 seconds or so, it simply disappeared!! I had to re-add it then again. No idea, what process or function is always killing this route. Now it gets even more weird: When I add this route and point it to a "wrong" gateway, it will be stable but when I use the the correct gateway again, the issue re-appears.

Anybody and idea what could go wrong? I know it's not really due to veeam (or at least I guess so) but maybe someone was having a similar situation...

Thanks!

[PetrM]

Hi Michael,

The routing table is updated automatically on the backup server in the beginning of SureBackup job when it starts its routing engine.

I would recommend to verify that causal relationship is determined correctly: maybe SureBackup fails due to another reason than a missing route?
It's normal that that the new route is removed once the job is completed, I've just reproduced this behavior in my lab.

I think that the best option is to contact our support team and ask them to investigate debug logs in order to find the cause of SureBackup job failure.
Please don't forget to share support case ID.

Thanks!

[mcz]

Hi Petr,

thanks for your response. I'm dead sure that it's due to the missing route. I mean veeam creates the route but during the job (after 10 seconds or so), the route is gone. That's why the ping-test fails. When I do a ping manually from the backup server via the masquarade network, it works but after some seconds I get a timeout due to the missing route. So weird!

If you can confirm that no other veeam component could accidentally remove the route during the surebackup job, then it must be an OS issue (I guess)...

[PetrM]

Hi Michael,

It's an interesting one but please let our support team to determine the root cause.
The troubleshooting has to be based on the precise analysis of debug logs and not on our suppositions, even if these suppositions seem to be 100 % true.

Thanks!

[mcz]

case #04282654

[PTide]

I am curious if the same weird behaviour can be observed if yo DON'T start the SureBackup job but still add the route manually.

Thanks!

[mcz]

Very interesting approach, haven't thought about it. Yes, it also disappears/gets removed when SureBackup isn't running. So it's a pure windows issue I'd say. But I can't imagine why that would ever be the case. And as mentioned, it's purely related to the ip. When I specify a different gateway, it stays in the list. No other IP on this VM has that gateway IP...

[roachg]

Hi Michael,

I'm experiencing the exact same thing and was wondering if you ever solved it and if so, how.

Thanks

[mcz]

Hi Glenn,

I'm somehow glad to hear that I'm not the only one. My approach was to use a different masquarade subnet and then it didn't trigger the bug. So it was a workaround but still I have no explanation why some subnets/ip's would be fine, while others get removed from the routing table...

Hope it helps!

[jtupeck]

Having a similar issue with a customer's environment. We discovered the route was being removed automatically after a few seconds, whether it was added manually before, or during the SureBackup job. Even before running a SureBackup job, the route was added, then disappeared from the routing table after about 5-10 seconds. Very strange. I will update if we find anything different for a solution and/or if using a different masquerade setting works.

[mcz]

Hi Jason,

yeah the thing is that it has nothing to with veeam, it must be a windows/driver issue. As you said, even when you manually add the route, it gets removed by whatever process. I guess the best way would be to open a ticket with Microsoft if that is possible.

[mdiver]

I just faced exactly the same issue. Interestingly with exactly the same masquerading IP range (192.168.255.0). So maybe some issue with the very address.
Switching to 192.168.254.0 solved the issue.
Trigger might have been Windows updates because the problem started right after patchday with this customer.

Thanks
Mike

[kojoman]

Hello together,

in my case it was surely a veeam issue.
The veeam server was crashing while a SureBackup-Job was running.
After the reboot of the veeam server a cleanup process ran automatically and deleted the objects from the SureBackup-Job in the VMware environment.
It looks like that this cleanup job is now running indefinitly and tries every 20 seconds to cleanup the route table entrys for the surebackup masquerade network.
We see this messages in the Svc.VeeamBackup.log every 20 seconds:

Code: Select all

[22.06.2022 15:54:25] <98> Info         [CTrackedActionManager] 5 action leases are expired.
[22.06.2022 15:54:25] <98> Info         [CTrackedActionManager] Rolling back action for lease ID: [0666834d-dbb6-4d0f-8c61-4969f54cfc26].
[22.06.2022 15:54:25] <98> Info         [StorageProfileTrackedAction] Finding action for lease: '0666834d-dbb6-4d0f-8c61-4969f54cfc26'.
[22.06.2022 15:54:25] <98> Info         [StorageProfileTrackedAction] Unable to find action for lease: '0666834d-dbb6-4d0f-8c61-4969f54cfc26'.
[22.06.2022 15:54:25] <98> Info         [CTrackedActionController] ActionController for lease id: [0666834d-dbb6-4d0f-8c61-4969f54cfc26]. Rollback all actions.
[22.06.2022 15:54:25] <98> Info         [CTrackedActionController] Running action [SureReplicaStateFinalizer], id: [0666834d-dbb6-4d0f-8c61-4969f54cfc26].
[22.06.2022 15:54:25] <98> Info         [SureReplicaStateTrackedAction] Finishing task session name: [SQL01], id: [4a8544fa-dd48-4ecb-b1bc-6907290d3883]
[22.06.2022 15:54:25] <98> Info         [SureReplicaStateTrackedAction] Oib is in nothing state: skip task session name: [SQL01], id: [4a8544fa-dd48-4ecb-b1bc-6907290d3883]
[22.06.2022 15:54:25] <98> Info         [CTrackedActionController] Running action [RemoteMount], id: [27b13688-6b77-49e8-8da5-da3f4ee10509].
[22.06.2022 15:54:25] <98> Info         [CRemoteMountTrackedAction] Unmounting remote mount tracked action. ActionId: [27b13688-6b77-49e8-8da5-da3f4ee10509], LeaseId: [0666834d-dbb6-4d0f-8c61-4969f54cfc26].
[22.06.2022 15:54:25] <98> Info         [CRemoteMountManager] Mount was not found. LeaseId: [0666834ddbb64d0f8c614969f54cfc26].
[22.06.2022 15:54:25] <98> Info         [CTrackedActionManager] Rolling back action for lease ID: [9cf11998-1fa3-40e4-8659-750ce38cde86].
[22.06.2022 15:54:25] <98> Info         [StorageProfileTrackedAction] Finding action for lease: '9cf11998-1fa3-40e4-8659-750ce38cde86'.
[22.06.2022 15:54:25] <98> Info         [StorageProfileTrackedAction] Unable to find action for lease: '9cf11998-1fa3-40e4-8659-750ce38cde86'.
[22.06.2022 15:54:25] <98> Info         [CTrackedActionController] ActionController for lease id: [9cf11998-1fa3-40e4-8659-750ce38cde86]. Rollback all actions.
[22.06.2022 15:54:25] <98> Info         [CTrackedActionController] Running action [SureBackup], id: [9cf11998-1fa3-40e4-8659-750ce38cde86].
[22.06.2022 15:54:25] <98> Info         [SureBackupTrackedAction] Finishing job session name: [SureBackup Job SQL], id: [7dbc1c9b-0bff-485b-833e-56cf89bf16e6]
[22.06.2022 15:54:25] <98> Info         [SureBackupTrackedAction] Removing route table destination: [192.168.0.0], mask: [255.255.255.0], gateway: [10.100.0.171]
[b][22.06.2022 15:54:25] <98> Error        [SureBackupTrackedAction] [Line 1] The route deletion failed: Element not found.
[22.06.2022 15:54:25] <98> Error        [SureBackupTrackedAction] [Line 2] 
[22.06.2022 15:54:25] <98> Error        [SureBackupTrackedAction] [Line 3] 
[22.06.2022 15:54:25] <98> Info         [SureBackupTrackedAction] Removing route table destination: [192.168.1.0], mask: [255.255.255.0], gateway: [10.100.0.171]
[22.06.2022 15:54:25] <98> Error        [SureBackupTrackedAction] [Line 1] The route deletion failed: Element not found.
[22.06.2022 15:54:25] <98> Error        [SureBackupTrackedAction] [Line 2] 
[22.06.2022 15:54:25] <98> Error        [SureBackupTrackedAction] [Line 3][/b]

A manual deletion of the Backup.TrackedActions.Leases in the database with the Veeam-Support solved the issue.

Regards
Philipp

[PetrM]

Hello Philipp and Welcome to Veeam R&D Forums!

Many thanks for sharing this information with us! However, I must note that manual updates at the level of configuration database can be performed by our support engineers only.

Thanks!

[david.domask]

Hi Philipp!

Thanks for sharing, and indeed, a crash like that would leave a mess likely in the database and when the process for cleaning the routes runs, it would get confused because of this stale entry.

I want to note though folks that this table contains a lot of important stuff during backup and restore operations, so you must not delete from it randomly! It may result in data loss if you delete the wrong thing!!!

If you have an issue, open a support case and review with Support; it should be fairly quick to see which TrackedAction needs to be cleared. If you're able to export a copy of the Veeam Database from your environment, do so when opening a case to expedite the resolution: https://www.veeam.com/kb1471 If you cannot due to security requirements/regulations, no worries, just let the Engineer know and they'll arrange a remote session to review.

[BrianBuchanan]

My case is #02306553 and I'm having this exact problem after a server crash. I can add the masquerade route and with 5-10 seconds it disappears. I have found rows in dbo.Backup.TrackedActions.Leases so I'm hoping support will be quick to resolve this.

Thanks!

[mcz]

Brian, this is exclusively a windows issue, it has nothing to do with the rows in the database...

[BrianBuchanan]

Support was able to examine my database export and found three rows in Backup.TrackedActions.Leases that were responsible for continuously removing the routes for the SureBackup Lab. They gave me a script to delete those rows and I was back in action.

[mcz]

Fantastic Brian, thanks for sharing the results. Looks like I was totally wrong, so the routes got really deleted by veeam... Thanks again!