Comprehensive data protection for all workloads
Post Reply
basildane
Influencer
Posts: 11
Liked: never
Joined: Aug 06, 2020 12:15 am
Full Name: Phil
Contact:

Upgraded to V10 today, now all VM backups fail

Post by basildane »

Case 04320318.

I've been running V9 and today upgraded to V10.0.1.4854 Community Edition.
Upgrade went smoothly, but now all VM backups fail. Windows agent backups still work.

The VM backups fail with errors like this:

Code: Select all

Error: Incorrect function. Asynchronous read operation failed Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: Incorrect function. Asynchronous read operation failed Unable to retrieve next block transmission command. Number of already processed blocks: [130010]. Failed to download disk 'morbo-flat.vmdk'. 
I have multiple esxi 6.5 hosts, and they all fail with similar errors. They were all working fine yesterday on Veeam V9. I've been looking at the logs but I haven't found anything useful.
Gostev
Chief Product Officer
Posts: 31561
Liked: 6725 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by Gostev »

There are no known issues with v10 that would match this description, so this needs to be investigated by support. It may take some time before they are able to review your case, as free products are supported on the best effort basis. Thanks!
basildane
Influencer
Posts: 11
Liked: never
Joined: Aug 06, 2020 12:15 am
Full Name: Phil
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by basildane »

After days of trial and error, I discovered that if I manually set the backup proxy to the WRONG server, the backup will work. It will take 5 times longer, but it completes without errors.

Reinstalling the proxy agents on the servers had no effect.
basildane
Influencer
Posts: 11
Liked: never
Joined: Aug 06, 2020 12:15 am
Full Name: Phil
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by basildane »

I have now completed rebuilding my backup server. I have fully wiped Veeam off and reinstalled it, and a new sql database.
Ran a backup and same error as before.
Egor Yakovlev
Veeam Software
Posts: 2537
Liked: 683 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by Egor Yakovlev »

Hi Phil,
- please, double check RAM consumption on Proxy servers during backup. One that is processing failed job has 10GB total physical RAM.
- logs attached to the case contain only last run - can you also confirm if job fails at the same data block processed or it is random? Logs attached to the case has "Unable to retrieve next block transmission command. Number of already processed blocks: [75375]", which is roughly 67% job progress.
- you can also try to force Network Transport mode on a proxy server used in the job and see if that is a hot-add problem
/Cheers!
basildane
Influencer
Posts: 11
Liked: never
Joined: Aug 06, 2020 12:15 am
Full Name: Phil
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by basildane »

I tried changing to Network Transport mode, the backup would have taken 3 days to finish so I aborted it. Checking other options now...
basildane
Influencer
Posts: 11
Liked: never
Joined: Aug 06, 2020 12:15 am
Full Name: Phil
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by basildane »

In desperation, I switched the backup to Windows Agent mode.
I installed the transport agent on the server.
I configured it to backup Entire Computer.

It comes back after 6 seconds and says: Nothing to process.

I've tried reinstalled the agents on the server, deleting and recreating the backup job, but I get exactly the same result each time.
Gostev
Chief Product Officer
Posts: 31561
Liked: 6725 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by Gostev »

If you did a clean install, then this is likely due to a misconfiguration, or perhaps some environmental problem, as no one else has reported similar issues in half a year that v10 has been generally available.

Please wait and see if our customer support is able to assign someone to assist you with determining the root cause.

Thanks!
basildane
Influencer
Posts: 11
Liked: never
Joined: Aug 06, 2020 12:15 am
Full Name: Phil
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by basildane »

Support closed the ticket without helping.
Gostev
Chief Product Officer
Posts: 31561
Liked: 6725 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by Gostev »

That means they don't have available engineers to work on support cases for free products right now. This is expected because we just released 10a, and many paying customers are upgrading. You can try opening the case again in a couple weeks, hopefully it gets quieter by then. Thanks!
neilvbell
Novice
Posts: 7
Liked: 3 times
Joined: Sep 22, 2014 7:51 am
Full Name: Neil V. Bell
Contact:

Re: Upgraded to V10 today, now all VM backups fail

Post by neilvbell »

Couple of things occur to me to be worth a check:
Are you using storage integration (such as for storage snapshots)? If so, go to the Storage section and check that it is all still set-up correctly.
Are your proxies physical or virtual? I have physical proxies connected to my storage but my management VM is virtual and it used to keep trying to use that to do backups. I manually removed it from the list of proxies for the jobs so it only tries to use the correct physical proxies.
Have you tried an Active Full?
Have you upgrade any other components, such as a proxy server, virtual hosts or management (such as vCenter)? Might need to reconnect.
Check that you can run a standard snapshot on your VMs.
Are you on a compatible version of virtual system?

Good luck.
Post Reply

Who is online

Users browsing this forum: Google [Bot], Semrush [Bot] and 119 guests