First backup run failed: I/O error

Backup agent for Linux servers and workstations on-premises or in the public cloud

First backup run failed: I/O error

Veeam Logoby ovidiu » Sun Jul 10, 2016 10:18 am

Code: Select all
Backup session on jacob.ict-consult.co.za [75] %               Status: Failed




     Summary                                Data

     Duration:           00:09:21           Processed:          61.2 GB (75%)
     Processing rate:    38.2 MB/s          Read:               18.8 GB
     Bottleneck:         Source             Transferred:        3.9 GB (4.9x)


     Time                Action                                        Duration

     2016-07-10 11:47:45 Job started at 2016-07-10 11:47:45
     2016-07-10 11:47:45 Starting backup
     2016-07-10 11:47:45 Creating volume snapshot                      00:00:02
     2016-07-10 11:47:52 Backed up /dev/vda 501.0MB at 69.5MB/s        00:00:07
     2016-07-10 11:47:59 [error] Backing up vg_00 17.0GB at 35.9MB/s...00:08:55
     2016-07-10 11:57:06 [error] Input/output error
     2016-07-10 11:57:06 [error] Failed to perform backup


Here is the link to an upload of /var/log/veeam on dropbox. I couldn't spot anything sensitive in there but please let me know if I should obscure some parts? => https://www.dropbox.com/sh/za0ugrlkzez4 ... nVJ6a?dl=0
ovidiu
Influencer
 
Posts: 19
Liked: 1 time
Joined: Thu Jul 07, 2016 3:07 pm

Re: First backup run failed: I/O error

Veeam Logoby ovidiu » Sun Jul 10, 2016 10:33 am

tried again, got further but still I/O error. Could this be a problem with my NFS storage?
Code: Select all
Backup session on jacob.ict-consult.co.za [83] %               Status: Failed




     Summary                                Data

     Duration:           00:13:35           Processed:          67.3 GB (83%)
     Processing rate:    36.7 MB/s          Read:               24.9 GB
     Bottleneck:         Source             Transferred:        4.4 GB (5.7x)


     Time                Action                                        Duration

     2016-07-10 12:18:52 Job started at 2016-07-10 12:18:52
     2016-07-10 12:18:53 Starting backup
     2016-07-10 12:18:53 Creating volume snapshot                      00:00:05
     2016-07-10 12:19:04 Backed up /dev/vda 501.0MB at 70.8MB/s        00:00:07
     2016-07-10 12:19:11 [error] Backing up vg_00 22.7GB at 34.6MB/s...00:13:04
     2016-07-10 12:32:26 [error] Input/output error
     2016-07-10 12:32:26 [error] Failed to perform backup
ovidiu
Influencer
 
Posts: 19
Liked: 1 time
Joined: Thu Jul 07, 2016 3:07 pm

Re: First backup run failed: I/O error

Veeam Logoby ovidiu » Sun Jul 10, 2016 10:34 am

oh and on hte target I am left with a 4GB .vbk file, shouldn't VAL clean up after itself?
ovidiu
Influencer
 
Posts: 19
Liked: 1 time
Joined: Thu Jul 07, 2016 3:07 pm

Re: First backup run failed: I/O error

Veeam Logoby vmniels » Sun Jul 10, 2016 1:34 pm

Could you give some more info about the NFS share? How is it exported?
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: First backup run failed: I/O error

Veeam Logoby ovidiu » Sun Jul 10, 2016 2:05 pm

I cannot tell you anythign about it as it is being provided my my ISP: https://www.soyoustart.com/en/backup-storage.xml
ovidiu
Influencer
 
Posts: 19
Liked: 1 time
Joined: Thu Jul 07, 2016 3:07 pm

Re: First backup run failed: I/O error

Veeam Logoby vmniels » Sun Jul 10, 2016 2:08 pm

We'll have to wait for feedback from the devs on the logs but I/O error seems to be related due to overflow on the share (or a timeout). Do you see anything related nfs timeouts in dmesg ?
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: First backup run failed: I/O error

Veeam Logoby ovidiu » Sun Jul 10, 2016 3:38 pm

Here is the corresponding part of /var/log/messages starting with me installing nfs-utils before setting up VAL and running the first backup:

Code: Select all
Jul 10 11:45:42 jacob yum[5689]: Installed: nfs-utils-lib-1.1.5-11.el6.x86_64
Jul 10 11:45:42 jacob yum[5689]: Installed: 1:nfs-utils-1.2.3-70.el6.x86_64
Jul 10 11:47:22 jacob kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jul 10 11:47:22 jacob kernel: NFS: Registering the id_resolver key type
Jul 10 11:47:22 jacob kernel: FS-Cache: Netfs 'nfs' registered for caching
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out

there are plenty of further nfs timeouts after these...
ovidiu
Influencer
 
Posts: 19
Liked: 1 time
Joined: Thu Jul 07, 2016 3:07 pm

Re: First backup run failed: I/O error

Veeam Logoby vmniels » Sun Jul 10, 2016 4:06 pm

Can you try to make a backup to a local disk as a test and see if that works fine? It looks like the nfs share gets timeouts due latency or something and this ends up in a temporary death path.
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: First backup run failed: I/O error

Veeam Logoby ovidiu » Sun Jul 10, 2016 4:51 pm

a test backup to local storage completed just fine. anything we can do about the nfs probs from this side? I'm also checking with the storage provider to see if there is anything amss.
ovidiu
Influencer
 
Posts: 19
Liked: 1 time
Joined: Thu Jul 07, 2016 3:07 pm

Re: First backup run failed: I/O error

Veeam Logoby vmniels » Sun Jul 10, 2016 7:16 pm

How is your setup? As this is cloud storage via NFS, is the VM you are using for testing it in the same location or in another datacenter? If you manually mount it and try to write big files to it does it stay online?
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: First backup run failed: I/O error

Veeam Logoby ovidiu » Sun Jul 10, 2016 7:59 pm

I am renting a server from So You Start which runs Proxmox as hypervisor. There are a few LXCs and a few KVMs on this host.
The backup storage is in the same datacenter. I'll run a manual test:

- mounted nfs successfully
- fallocate -l 5G test.img
- cp test.img /mnt/test
cp test.img /mnt/test worked just fine.
ovidiu
Influencer
 
Posts: 19
Liked: 1 time
Joined: Thu Jul 07, 2016 3:07 pm

Re: First backup run failed: I/O error

Veeam Logoby ovidiu » Sun Jul 10, 2016 8:11 pm

It must be the provider as I just successfully ran a veeam backup. I'll take it up with the provider.
Code: Select all
Backup session on jacob.ict-consult.co.za [100] %             Status: Success




     Summary                                Data

     Duration:           00:10:00           Processed:          81.1 GB (100%)
     Processing rate:    67.3 MB/s          Read:               38.6 GB
     Bottleneck:         Source             Transferred:        5 GB (7.7x)


     Time                Action                                        Duration

     2016-07-10 21:59:08 Job started at 2016-07-10 21:59:08
     2016-07-10 21:59:08 Starting backup
     2016-07-10 21:59:09 Creating volume snapshot                      00:00:03
     2016-07-10 21:59:18 Backed up /dev/vda 501.0MB at 71.6MB/s        00:00:07
     2016-07-10 21:59:25 Backed up vg_00 35.5GB at 64.0MB/s            00:09:27
     2016-07-10 22:08:54 Backing up summary.xml                        00:00:00
     2016-07-10 22:09:08 Backup completed
ovidiu
Influencer
 
Posts: 19
Liked: 1 time
Joined: Thu Jul 07, 2016 3:07 pm

Re: First backup run failed: I/O error

Veeam Logoby vmniels » Sun Jul 10, 2016 8:47 pm

feels like a latency or specific setting issue. Keep us posted on feedback from them.
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: First backup run failed: I/O error

Veeam Logoby tdouble » Mon Aug 01, 2016 2:56 am

I am getting the same error. When I do an Entire Machine backup, it gets to 50% and then fails with the message I/O error. Looking at the Windows server NFS share, the file looks incomplete. When I do a Volume level back up and only select a few like home or swap, the job is successful, or if I just choose root, it fails. If I do a file level backup up and select a few folders, it is successful, but if I select several, it fails at about 53%. The Linux box is a CentOS 6.5 fully patched, and the Windows box is a Server 2012 R2 standard fully patched. Both boxes are in the same room and on the same network. Any assistance would be greatly appreciated! Thanks!
tdouble
Novice
 
Posts: 7
Liked: 1 time
Joined: Thu Jul 28, 2016 9:03 pm
Full Name: Todd Gorman

Re: First backup run failed: I/O error

Veeam Logoby vmniels » Mon Aug 01, 2016 9:53 am

How big is your server in total? If you mount the NFS share on the Linux server and write a big file to it (like using dd) does that work? By default the NFS driver by Windows isn't the best so tweaking may be needed.
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1440
Liked: 318 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Next

Return to Veeam Agent for Linux



Who is online

Users browsing this forum: No registered users and 2 guests