All backup jobs failing instantly with “proxy offline time exceeded”

[MILJW002]

Hi All,

I just noticed that all the backup jobs have been failing for the last 2 days with a “proxy offline time exceeded” error. I didn’t notice earlier as the email job notifications aren’t errors, so I didn’t see it until checking just now.

I’m running 4.0.1.531, and nothing that I’m aware of has changed recently. I’ve Googled and checked these forums and can’t see anything that looks like this.

I’ve logged a case and the job number is 04265920.

Any help greatly appreciated.

James

[MILJW002]

I rolled the server back to a backup earlier in the week and it’s working again. So, I’ve still got the logs from when the fault was occurring and backups of the server with the fault, but at least this has restored services.

[Mike Resseler]

Hey James,

This is also a new one for me I'm afraid. Please keep us updated on what support will tell you

Thanks
Mike

[smedeDKsmeden]

I have the same problem. Server restarted and services are running.
This became a problem last friday.
Case number is 04265920

[bckp]

Same here, since July 03 jobs fail. I run v4.0.1.531
Case # 04264338

[nielsengelen]

@bckp can u let us know the support case ID as well?

[pieter_s]

Same issue here since July 4, all jobs fail with error 'Backup proxy offline time exceeded'.
VBO v4.0.1.531
Case #04267881

[nielsengelen]

Thanks for the case @pieter_s. It's being looked into so please keep working with support.

[ruben]

Hello,

Just to bump, I'm having the exact same problem.
Same version as you guys and it started 3:th of July.

Case # 04269431

[Polina]

RnD is now looking into this issue and we'll provide a fix as soon as possible.

Thanks to everyone for the heads up!

[jasonbyw]

Same issue here. Also started 3rd of July... between 8am and 12pm Australian time if that helps.
Nobody had an update yet?

[MILJW002]

If timelines help narrow down then I can share mine. I have a number of backup jobs running over the day (different data to different repositories) but I backup key users OneDrives and Mailbox every hour.

The last successful backup on the broken instance was 6:00am on Friday the 3rd July (South Australian local time).

The first failed backup is in the log at 7:30am (SA local time). No entries at 7:00am. So that narrows down when it went wrong for me. @jasonbyw, which state/Timezone are you in?

I’ve not heard from support, as after I logged the incident I rolled back the VM backup to earlier in the week, and it’s been backing up successfully since then. I don’t recall applying any patches in the window between that successful VM backup and when it broke on a Friday 3rd July morning.

I updated the case notes to say I’d restored services by rolling back the VM, so originally thought it was just me. After all of that this thread started to get other responses.

I’m somewhat reassured that it’s a wider problem, as I hate when there is a random break. It’s not good it broke, but at least it wasn’t just me.

As per my original post, I’ve still got the logs that I submitted, and daily backups of the broken server itself if that helps support.

J

[Polina]

A quick update:
The issue was caused by incorrect zone time converting. It is now resolved, the hotfix is ready and our support engineers will contact you shortly to provide it.

Thanks!

[jasonbyw]

I'm in QLD and the supplied patch from Veeam seems to work so far.

[smedeDKsmeden]

Yep, the fix looks fine.

[bckp]

When will we receive the patch? I'll be OOF for few days, would be nice to have the backup running again.

[Polina]

@bckp

Please contact support, if you need the hotfix ASAP.
The full patch should be available on the website and via product auto-updates next week.

[bckp]

When I opened the case, I expected to be notified that there is a patch or workaround available. 5 days without Backups might seem ok for veeam but I have a different expectation when it is about backup.
Anyway I requested the patch now.

[pieter_s]

Installed the fix and backups are working again.

[Polina]

@bckp
Our engineer is sending you the hotfix as we speak.

[bckp]

@Polina thanks
Job is running now, not yet finished - but looks ok so far

Another thing you want to check might be the notification mails, since it appeared in color green and shows Success
0 of 0 objects processed but under Details: Backup proxy offline time exceeded.

[Polina]

We'll check it, thanks! )

[kckong]

Same problem for me.

My support case id is 01673138.

Please send me the hotfix.

Thanks.
Danny

[Polina]

Hi Danny,

This case was opened on a different issue and is already closed. Kindly, open a new support call for VBO, share it's ID here and the hotfix will be instantly delivered to you.

Thanks!