Beta2 - Job has failed - Input/output error

Backup agent for Linux servers and workstations on-premises or in the public cloud

Beta2 - Job has failed - Input/output error

Veeam Logoby matteos » Tue Dec 13, 2016 12:56 pm

Hello,

I'm trying the Beta2 on a Ubuntu 16.04 LTS.
The destination of the job it's a NFS share. (exported by /data1/backup_tango_workstation_veeam/ tango(rw,sync,no_subtree_check)

The Job failed after about 10GB of backup..

Code: Select all
[13.12.2016 12:40:52] <139894003205888> lpbcore| ERR |Job has failed.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |Input/output error
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |POSIX: Failed to write data to the file [/tmp/veeam/192.168.1.1:data1backup_tango_workstation_veeam_1/tobler BackupJob1_1/BackupJob1_2016-12-13T123656.vbk].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to process de-duplicated block
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to write block to blocks store.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to append block. FIB index: [6891]. FIB: [582e2f1f].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to process area [Data block. Start offset: [7225737216], Length: [1048576], Area ID: [6891].].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to wait for recorder invoke.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to check queue and pending invoke status. Wait flag 'true'.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:FIB proxy was unable to process write operation, recorder '0x000x7f6c75062c70', area offset '7383023616', FIB 'Backup of the FIB 582e2f1f'.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Unable to asynchronously write data block. Block identity: [Data block. Start offset: [7383023616], Length: [1048576], Area ID: [7041].].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Processing of asynchronous write requests has failed. Output file: [Backup of the FIB 582e2f1f].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to process conveyored task.


Here the complete log: http://pastebin.com/ha3GXkdn

ideas?

Thanks

Matteo
matteos
Novice
 
Posts: 7
Liked: never
Joined: Thu Apr 23, 2015 7:44 pm

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby vmniels » Tue Dec 13, 2016 12:58 pm

Since today VAL is GA. Please try the final release and see if the issue still exists.
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby matteos » Tue Dec 13, 2016 1:38 pm

vmniels wrote:Since today VAL is GA. Please try the final release and see if the issue still exists.


Ok, purged the old packages (beta) and installed the GA.

Created the new job, failed after a while...:

Code: Select all
      14:27:12        [error] Backing up sdb 13.8 GB at 57.7 MB/s (99%)                                                 00:05:06
      14:32:24        [error] Failed to perform backup
      14:32:24        [error] Input/output error
      14:32:24        [error] POSIX: Failed to write data to the file [/tmp/veeam/xxxx:data1backup_xxxx_wo...
      14:32:24        [error] Failed to download disk
      14:32:24        [error] read: End of file
      14:32:24        [error] Failed to upload disk
      14:32:24        [error] Agent failed to process method {DataTransfer.SyncDisk}

Ideas?

M.
matteos
Novice
 
Posts: 7
Liked: never
Joined: Thu Apr 23, 2015 7:44 pm

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby vmniels » Tue Dec 13, 2016 2:13 pm

Could you present us the logs and describe your job settings? Such as target and are you backing up a volume or entire computer?
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby matteos » Tue Dec 13, 2016 5:12 pm

vmniels wrote:Could you present us the logs and describe your job settings? Such as target and are you backing up a volume or entire computer?


I linked in my opening post the log. the backup is configured:

Entire machine
Shared folder
NFS -> IPofmynfsserver / path
prejob empty postjob empty
schedule 3:00 every day


M.
matteos
Novice
 
Posts: 7
Liked: never
Joined: Thu Apr 23, 2015 7:44 pm

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby vmniels » Tue Dec 13, 2016 5:21 pm

Can you paste the output of dmesg -T as well? Looks like u get a timeout in the end towards the nfs server.
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby matteos » Tue Dec 13, 2016 6:05 pm

vmniels wrote:Can you paste the output of dmesg -T as well? Looks like u get a timeout in the end towards the nfs server.


bingo:
[mar dic 13 18:54:39 2016] nfs: server 192.168.1.1 not responding, timed out
[mar dic 13 18:56:23 2016] nfs: server 192.168.1.1 not responding, timed out

here the whole dmesg -T output http://pastebin.com/tWaVgUbp

so seems a timeout of NFS on client side. On server side I don't see any logs/errors/timeout.

The NFS server is a Debian 8 with a ZFS (with 5 SATA disks) volume exported via NFS (with export rw,sync,no_subtree_check). The server has no load. It's only receiving this backup job...

M.
matteos
Novice
 
Posts: 7
Liked: never
Joined: Thu Apr 23, 2015 7:44 pm

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby vmniels » Tue Dec 13, 2016 6:06 pm

Is this a VM you are using as test? If so how much CPU/RAM resources do you have assigned to it?
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby matteos » Tue Dec 13, 2016 6:39 pm

vmniels wrote:Is this a VM you are using as test? If so how much CPU/RAM resources do you have assigned to it?


source and target are physical pc.

I changed the NFS export from sync to async and now the backup has finished succesfully. It's very strange. What do you think?

now:
19:06:45 Job BackupJob1 started at 2016-12-13 19:06:45 CET
19:06:45 Preparing to backup
19:08:12 Creating volume snapshot 00:00:01
19:08:13 Starting full backup to Repository_1
19:08:14 Backing up BIOS bootloader on /dev/sdb 00:00:01
19:08:15 Backed up sdb 13.8 GB at 60.3 MB/s 00:03:54
19:12:20 Backed up data 85.2 GB at 79.9 MB/s 00:18:12
19:30:46 Backed up stefano 42.0 GB at 91.6 MB/s 00:07:49
19:38:35 Backing up summary.xml 00:00:00
19:38:43 Releasing snapshot 00:00:02
19:38:45 Backup completed


M.
matteos
Novice
 
Posts: 7
Liked: never
Joined: Thu Apr 23, 2015 7:44 pm

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby PTide » Wed Dec 14, 2016 11:24 am

If I'm not mistaken your opening message contains beta2 logs. Since you've upgraded to GA we need new logs from GA (even if the error is identical).

Regarding async options - it allows NFS server not to wait for pending changes to be written to a storage, instead it continues to process request coming from a client. It would be interesting to check io queue on the NFS server during the backup while in sync mode. Could you check that please?

Thanks
PTide
Veeam Software
 
Posts: 2884
Liked: 236 times
Joined: Tue May 19, 2015 1:46 pm

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby matteos » Wed Dec 14, 2016 2:18 pm

PTide wrote:If I'm not mistaken your opening message contains beta2 logs. Since you've upgraded to GA we need new logs from GA (even if the error is identical).

Regarding async options - it allows NFS server not to wait for pending changes to be written to a storage, instead it continues to process request coming from a client. It would be interesting to check io queue on the NFS server during the backup while in sync mode. Could you check that please?

Thanks


I sent you via PM the log produced by GA with sync option when failed.

What about "check the IO queue on nfs server"? can you clarify what do you mean?

thanks

Matteo
matteos
Novice
 
Posts: 7
Liked: never
Joined: Thu Apr 23, 2015 7:44 pm

Re: Beta2 - Job has failed - Input/output error

Veeam Logoby PTide » Wed Dec 14, 2016 2:34 pm

I suspect that sync mode does not work well because the IO load on the NFS storage is high (slow disks?) and data cannot get to the storage quickly enough. I asked if you could use some tools like iotop and iostat to check if my guess was correct.
PTide
Veeam Software
 
Posts: 2884
Liked: 236 times
Joined: Tue May 19, 2015 1:46 pm


Return to Veeam Agent for Linux



Who is online

Users browsing this forum: No registered users and 2 guests