Standalone backup agents for Linux and Unix workloads on-premises or in the public cloud
Post Reply
matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Beta2 - Job has failed - Input/output error

Post by matteos » 1 person likes this post

Hello,

I'm trying the Beta2 on a Ubuntu 16.04 LTS.
The destination of the job it's a NFS share. (exported by /data1/backup_tango_workstation_veeam/ tango(rw,sync,no_subtree_check)

The Job failed after about 10GB of backup..

Code: Select all

[13.12.2016 12:40:52] <139894003205888> lpbcore| ERR |Job has failed.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |Input/output error
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |POSIX: Failed to write data to the file [/tmp/veeam/192.168.1.1:data1backup_tango_workstation_veeam_1/tobler BackupJob1_1/BackupJob1_2016-12-13T123656.vbk].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to process de-duplicated block
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to write block to blocks store.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to append block. FIB index: [6891]. FIB: [582e2f1f].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to process area [Data block. Start offset: [7225737216], Length: [1048576], Area ID: [6891].].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to wait for recorder invoke.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to check queue and pending invoke status. Wait flag 'true'.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:FIB proxy was unable to process write operation, recorder '0x000x7f6c75062c70', area offset '7383023616', FIB 'Backup of the FIB 582e2f1f'.
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Unable to asynchronously write data block. Block identity: [Data block. Start offset: [7383023616], Length: [1048576], Area ID: [7041].].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Processing of asynchronous write requests has failed. Output file: [Backup of the FIB 582e2f1f].
[13.12.2016 12:40:52] <139894003205888> lpbcore| >>  |--tr:Failed to process conveyored task.
Here the complete log: http://pastebin.com/ha3GXkdn

ideas?

Thanks

Matteo

nielsengelen
Veeam Software
Posts: 3740
Liked: 738 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by nielsengelen »

Since today VAL is GA. Please try the final release and see if the issue still exists.
https://foonet.be

matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

vmniels wrote:Since today VAL is GA. Please try the final release and see if the issue still exists.
Ok, purged the old packages (beta) and installed the GA.

Created the new job, failed after a while...:

Code: Select all

      14:27:12        [error] Backing up sdb 13.8 GB at 57.7 MB/s (99%)                                                 00:05:06
      14:32:24        [error] Failed to perform backup
      14:32:24        [error] Input/output error
      14:32:24        [error] POSIX: Failed to write data to the file [/tmp/veeam/xxxx:data1backup_xxxx_wo...
      14:32:24        [error] Failed to download disk
      14:32:24        [error] read: End of file
      14:32:24        [error] Failed to upload disk
      14:32:24        [error] Agent failed to process method {DataTransfer.SyncDisk}
Ideas?

M.

nielsengelen
Veeam Software
Posts: 3740
Liked: 738 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by nielsengelen »

Could you present us the logs and describe your job settings? Such as target and are you backing up a volume or entire computer?
https://foonet.be

matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

vmniels wrote:Could you present us the logs and describe your job settings? Such as target and are you backing up a volume or entire computer?
I linked in my opening post the log. the backup is configured:

Entire machine
Shared folder
NFS -> IPofmynfsserver / path
prejob empty postjob empty
schedule 3:00 every day


M.

nielsengelen
Veeam Software
Posts: 3740
Liked: 738 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by nielsengelen »

Can you paste the output of dmesg -T as well? Looks like u get a timeout in the end towards the nfs server.
https://foonet.be

matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

vmniels wrote:Can you paste the output of dmesg -T as well? Looks like u get a timeout in the end towards the nfs server.
bingo:
[mar dic 13 18:54:39 2016] nfs: server 192.168.1.1 not responding, timed out
[mar dic 13 18:56:23 2016] nfs: server 192.168.1.1 not responding, timed out

here the whole dmesg -T output http://pastebin.com/tWaVgUbp

so seems a timeout of NFS on client side. On server side I don't see any logs/errors/timeout.

The NFS server is a Debian 8 with a ZFS (with 5 SATA disks) volume exported via NFS (with export rw,sync,no_subtree_check). The server has no load. It's only receiving this backup job...

M.

nielsengelen
Veeam Software
Posts: 3740
Liked: 738 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by nielsengelen »

Is this a VM you are using as test? If so how much CPU/RAM resources do you have assigned to it?
https://foonet.be

matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

vmniels wrote:Is this a VM you are using as test? If so how much CPU/RAM resources do you have assigned to it?
source and target are physical pc.

I changed the NFS export from sync to async and now the backup has finished succesfully. It's very strange. What do you think?

now:
19:06:45 Job BackupJob1 started at 2016-12-13 19:06:45 CET
19:06:45 Preparing to backup
19:08:12 Creating volume snapshot 00:00:01
19:08:13 Starting full backup to Repository_1
19:08:14 Backing up BIOS bootloader on /dev/sdb 00:00:01
19:08:15 Backed up sdb 13.8 GB at 60.3 MB/s 00:03:54
19:12:20 Backed up data 85.2 GB at 79.9 MB/s 00:18:12
19:30:46 Backed up stefano 42.0 GB at 91.6 MB/s 00:07:49
19:38:35 Backing up summary.xml 00:00:00
19:38:43 Releasing snapshot 00:00:02
19:38:45 Backup completed


M.

PTide
Product Manager
Posts: 5712
Liked: 564 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by PTide »

If I'm not mistaken your opening message contains beta2 logs. Since you've upgraded to GA we need new logs from GA (even if the error is identical).

Regarding async options - it allows NFS server not to wait for pending changes to be written to a storage, instead it continues to process request coming from a client. It would be interesting to check io queue on the NFS server during the backup while in sync mode. Could you check that please?

Thanks

matteos
Novice
Posts: 7
Liked: 1 time
Joined: Apr 23, 2015 7:44 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by matteos »

PTide wrote:If I'm not mistaken your opening message contains beta2 logs. Since you've upgraded to GA we need new logs from GA (even if the error is identical).

Regarding async options - it allows NFS server not to wait for pending changes to be written to a storage, instead it continues to process request coming from a client. It would be interesting to check io queue on the NFS server during the backup while in sync mode. Could you check that please?

Thanks
I sent you via PM the log produced by GA with sync option when failed.

What about "check the IO queue on nfs server"? can you clarify what do you mean?

thanks

Matteo

PTide
Product Manager
Posts: 5712
Liked: 564 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by PTide »

I suspect that sync mode does not work well because the IO load on the NFS storage is high (slow disks?) and data cannot get to the storage quickly enough. I asked if you could use some tools like iotop and iostat to check if my guess was correct.

agrosoft
Lurker
Posts: 1
Liked: 1 time
Joined: Jul 04, 2020 4:38 pm
Full Name: Tomáš Dvořák
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by agrosoft » 1 person likes this post

Hello, I can confirm that I had same problem and after switching to async mode, everything went well. When I started iotop on backup machine (also Linux) the IO usage was 100%.

PTide
Product Manager
Posts: 5712
Liked: 564 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Beta2 - Job has failed - Input/output error

Post by PTide »

@agrosoft,
Thank you for the confirmation,

Cheers

Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests