Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Beta2 - Job has failed - Input/output error

Post by matteos » 1 person likes this post

Hello,

I'm trying the Beta2 on a Ubuntu 16.04 LTS.
The destination of the job it's a NFS share. (exported by /data1/backup_tango_workstation_veeam/ tango(rw,sync,no_subtree_check)

The Job failed after about 10GB of backup..

Code: Select all

[13.12.2016 12:40:52] <139894003205888> lpbcore| ERR |Job has failed.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |Input/output error
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |POSIX: Failed to write data to the file [/tmp/veeam/192.168.1.1:data1backup_tango_workstation_veeam_1/tobler BackupJob1_1/BackupJob1_2016-12-13T123656.vbk].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to process de-duplicated block
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to write block to blocks store.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to append block. FIB index: [6891]. FIB: [582e2f1f].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to process area [Data block. Start offset: [7225737216], Length: [1048576], Area ID: [6891].].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to wait for recorder invoke.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to check queue and pending invoke status. Wait flag 'true'.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:FIB proxy was unable to process write operation, recorder '0x000x7f6c75062c70', area offset '7383023616', FIB 'Backup of the FIB 582e2f1f'.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Unable to asynchronously write data block. Block identity: [Data block. Start offset: [7383023616], Length: [1048576], Area ID: [7041].].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Processing of asynchronous write requests has failed. Output file: [Backup of the FIB 582e2f1f].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to process conveyored task.
Here the complete log: http://pastebin.com/ha3GXkdn

ideas?

Thanks

Matteo
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by nielsengelen »

Since today VAL is GA. Please try the final release and see if the issue still exists.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

vmniels wrote:Since today VAL is GA. Please try the final release and see if the issue still exists.
Ok, purged the old packages (beta) and installed the GA.

Created the new job, failed after a while...:

Code: Select all

      14:27:12        [error] Backing up sdb 13.8 GB at 57.7 MB/s (99%)                                                 00:05:06
      14:32:24        [error] Failed to perform backup
      14:32:24        [error] Input/output error
      14:32:24        [error] POSIX: Failed to write data to the file [/tmp/veeam/xxxx:data1backup_xxxx_wo...
      14:32:24        [error] Failed to download disk
      14:32:24        [error] read: End of file
      14:32:24        [error] Failed to upload disk
      14:32:24        [error] Agent failed to process method {DataTransfer.SyncDisk}
Ideas?

M.
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by nielsengelen »

Could you present us the logs and describe your job settings? Such as target and are you backing up a volume or entire computer?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

vmniels wrote:Could you present us the logs and describe your job settings? Such as target and are you backing up a volume or entire computer?
I linked in my opening post the log. the backup is configured:

Entire machine
Shared folder
NFS -> IPofmynfsserver / path
prejob empty postjob empty
schedule 3:00 every day


M.
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by nielsengelen »

Can you paste the output of dmesg -T as well? Looks like u get a timeout in the end towards the nfs server.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

vmniels wrote:Can you paste the output of dmesg -T as well? Looks like u get a timeout in the end towards the nfs server.
bingo:
[mar dic 13 18:54:39 2016] nfs: server 192.168.1.1 not responding, timed out
[mar dic 13 18:56:23 2016] nfs: server 192.168.1.1 not responding, timed out

here the whole dmesg -T output http://pastebin.com/tWaVgUbp

so seems a timeout of NFS on client side. On server side I don't see any logs/errors/timeout.

The NFS server is a Debian 8 with a ZFS (with 5 SATA disks) volume exported via NFS (with export rw,sync,no_subtree_check). The server has no load. It's only receiving this backup job...

M.
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by nielsengelen »

Is this a VM you are using as test? If so how much CPU/RAM resources do you have assigned to it?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

vmniels wrote:Is this a VM you are using as test? If so how much CPU/RAM resources do you have assigned to it?
source and target are physical pc.

I changed the NFS export from sync to async and now the backup has finished succesfully. It's very strange. What do you think?

now:
19:06:45 Job BackupJob1 started at 2016-12-13 19:06:45 CET
19:06:45 Preparing to backup
19:08:12 Creating volume snapshot 00:00:01
19:08:13 Starting full backup to Repository_1
19:08:14 Backing up BIOS bootloader on /dev/sdb 00:00:01
19:08:15 Backed up sdb 13.8 GB at 60.3 MB/s 00:03:54
19:12:20 Backed up data 85.2 GB at 79.9 MB/s 00:18:12
19:30:46 Backed up stefano 42.0 GB at 91.6 MB/s 00:07:49
19:38:35 Backing up summary.xml 00:00:00
19:38:43 Releasing snapshot 00:00:02
19:38:45 Backup completed


M.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by PTide »

If I'm not mistaken your opening message contains beta2 logs. Since you've upgraded to GA we need new logs from GA (even if the error is identical).

Regarding async options - it allows NFS server not to wait for pending changes to be written to a storage, instead it continues to process request coming from a client. It would be interesting to check io queue on the NFS server during the backup while in sync mode. Could you check that please?

Thanks
matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

PTide wrote:If I'm not mistaken your opening message contains beta2 logs. Since you've upgraded to GA we need new logs from GA (even if the error is identical).

Regarding async options - it allows NFS server not to wait for pending changes to be written to a storage, instead it continues to process request coming from a client. It would be interesting to check io queue on the NFS server during the backup while in sync mode. Could you check that please?

Thanks
I sent you via PM the log produced by GA with sync option when failed.

What about "check the IO queue on nfs server"? can you clarify what do you mean?

thanks

Matteo
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by PTide »

I suspect that sync mode does not work well because the IO load on the NFS storage is high (slow disks?) and data cannot get to the storage quickly enough. I asked if you could use some tools like iotop and iostat to check if my guess was correct.
agrosoft
Lurker
Posts: 1
Liked: 1 time
Joined: Jul 04, 2020 4:38 pm
Full Name: Tomáš Dvořák
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by agrosoft » 1 person likes this post

Hello, I can confirm that I had same problem and after switching to async mode, everything went well. When I started iotop on backup machine (also Linux) the IO usage was 100%.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by PTide »

@agrosoft,
Thank you for the confirmation,

Cheers
jrgallego
Lurker
Posts: 1
Liked: never
Joined: May 15, 2022 11:42 am
Full Name: JOSE GALLEGO
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by jrgallego »

Hello,
My environment is VMware 6.7, Veeam 10 and qnap via NFS
My veeam backup jobs are failing, so:

How can I change NFS export from sync to async
How can I configure the NFS exports and the jobs to async mode?

The error message is this:

Code: Select all

<<5/15/2022 5:11:30 PM :: Processing monitoring-grafana-1.<domain>.com Error: Times out waiting for NFS async IO block callback. Timeout: 180000 msec
Failed to open storage for read/write access. Storage: [10.161.244.200:/share/Veeam/LAT/LAT - Monitoring_qnap2/LAT - Monitoring_qnap2D2022-05-15T171130_3B81.vbk].>>
Thanks in advance.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by PTide »

Hi,

It looks like should have posted in the other sub-forum.

Are you trying to back up VMware VM with Veeam Agent for Linux, or you are trying to back up VMs using only VBR?

Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests