Backup agent for Linux servers and workstations on-premises or in the public cloud
ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

First backup run failed: I/O error

Post by ovidiu » Jul 10, 2016 10:18 am

Code: Select all

Backup session on jacob.ict-consult.co.za [75] %               Status: Failed




     Summary                                Data

     Duration:           00:09:21           Processed:          61.2 GB (75%)
     Processing rate:    38.2 MB/s          Read:               18.8 GB
     Bottleneck:         Source             Transferred:        3.9 GB (4.9x)


     Time                Action                                        Duration

     2016-07-10 11:47:45 Job started at 2016-07-10 11:47:45
     2016-07-10 11:47:45 Starting backup
     2016-07-10 11:47:45 Creating volume snapshot                      00:00:02
     2016-07-10 11:47:52 Backed up /dev/vda 501.0MB at 69.5MB/s        00:00:07
     2016-07-10 11:47:59 [error] Backing up vg_00 17.0GB at 35.9MB/s...00:08:55
     2016-07-10 11:57:06 [error] Input/output error
     2016-07-10 11:57:06 [error] Failed to perform backup
Here is the link to an upload of /var/log/veeam on dropbox. I couldn't spot anything sensitive in there but please let me know if I should obscure some parts? => https://www.dropbox.com/sh/za0ugrlkzez4 ... nVJ6a?dl=0

ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu » Jul 10, 2016 10:33 am

tried again, got further but still I/O error. Could this be a problem with my NFS storage?

Code: Select all

Backup session on jacob.ict-consult.co.za [83] %               Status: Failed




     Summary                                Data

     Duration:           00:13:35           Processed:          67.3 GB (83%)
     Processing rate:    36.7 MB/s          Read:               24.9 GB
     Bottleneck:         Source             Transferred:        4.4 GB (5.7x)


     Time                Action                                        Duration

     2016-07-10 12:18:52 Job started at 2016-07-10 12:18:52
     2016-07-10 12:18:53 Starting backup
     2016-07-10 12:18:53 Creating volume snapshot                      00:00:05
     2016-07-10 12:19:04 Backed up /dev/vda 501.0MB at 70.8MB/s        00:00:07
     2016-07-10 12:19:11 [error] Backing up vg_00 22.7GB at 34.6MB/s...00:13:04
     2016-07-10 12:32:26 [error] Input/output error
     2016-07-10 12:32:26 [error] Failed to perform backup

ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu » Jul 10, 2016 10:34 am

oh and on hte target I am left with a 4GB .vbk file, shouldn't VAL clean up after itself?

vmniels
Veeam Software
Posts: 2050
Liked: 453 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by vmniels » Jul 10, 2016 1:34 pm

Could you give some more info about the NFS share? How is it exported?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu » Jul 10, 2016 2:05 pm

I cannot tell you anythign about it as it is being provided my my ISP: https://www.soyoustart.com/en/backup-storage.xml

vmniels
Veeam Software
Posts: 2050
Liked: 453 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by vmniels » Jul 10, 2016 2:08 pm

We'll have to wait for feedback from the devs on the logs but I/O error seems to be related due to overflow on the share (or a timeout). Do you see anything related nfs timeouts in dmesg ?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu » Jul 10, 2016 3:38 pm

Here is the corresponding part of /var/log/messages starting with me installing nfs-utils before setting up VAL and running the first backup:

Code: Select all

Jul 10 11:45:42 jacob yum[5689]: Installed: nfs-utils-lib-1.1.5-11.el6.x86_64
Jul 10 11:45:42 jacob yum[5689]: Installed: 1:nfs-utils-1.2.3-70.el6.x86_64
Jul 10 11:47:22 jacob kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jul 10 11:47:22 jacob kernel: NFS: Registering the id_resolver key type
Jul 10 11:47:22 jacob kernel: FS-Cache: Netfs 'nfs' registered for caching
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
there are plenty of further nfs timeouts after these...

vmniels
Veeam Software
Posts: 2050
Liked: 453 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by vmniels » Jul 10, 2016 4:06 pm

Can you try to make a backup to a local disk as a test and see if that works fine? It looks like the nfs share gets timeouts due latency or something and this ends up in a temporary death path.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu » Jul 10, 2016 4:51 pm

a test backup to local storage completed just fine. anything we can do about the nfs probs from this side? I'm also checking with the storage provider to see if there is anything amss.

vmniels
Veeam Software
Posts: 2050
Liked: 453 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by vmniels » Jul 10, 2016 7:16 pm

How is your setup? As this is cloud storage via NFS, is the VM you are using for testing it in the same location or in another datacenter? If you manually mount it and try to write big files to it does it stay online?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu » Jul 10, 2016 7:59 pm

I am renting a server from So You Start which runs Proxmox as hypervisor. There are a few LXCs and a few KVMs on this host.
The backup storage is in the same datacenter. I'll run a manual test:

- mounted nfs successfully
- fallocate -l 5G test.img
- cp test.img /mnt/test
cp test.img /mnt/test worked just fine.

ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu » Jul 10, 2016 8:11 pm

It must be the provider as I just successfully ran a veeam backup. I'll take it up with the provider.

Code: Select all

Backup session on jacob.ict-consult.co.za [100] %             Status: Success




     Summary                                Data

     Duration:           00:10:00           Processed:          81.1 GB (100%)
     Processing rate:    67.3 MB/s          Read:               38.6 GB
     Bottleneck:         Source             Transferred:        5 GB (7.7x)


     Time                Action                                        Duration

     2016-07-10 21:59:08 Job started at 2016-07-10 21:59:08
     2016-07-10 21:59:08 Starting backup
     2016-07-10 21:59:09 Creating volume snapshot                      00:00:03
     2016-07-10 21:59:18 Backed up /dev/vda 501.0MB at 71.6MB/s        00:00:07
     2016-07-10 21:59:25 Backed up vg_00 35.5GB at 64.0MB/s            00:09:27
     2016-07-10 22:08:54 Backing up summary.xml                        00:00:00
     2016-07-10 22:09:08 Backup completed

vmniels
Veeam Software
Posts: 2050
Liked: 453 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by vmniels » Jul 10, 2016 8:47 pm

feels like a latency or specific setting issue. Keep us posted on feedback from them.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

tdouble
Novice
Posts: 7
Liked: 1 time
Joined: Jul 28, 2016 9:03 pm
Full Name: Todd Gorman
Contact:

Re: First backup run failed: I/O error

Post by tdouble » Aug 01, 2016 2:56 am

I am getting the same error. When I do an Entire Machine backup, it gets to 50% and then fails with the message I/O error. Looking at the Windows server NFS share, the file looks incomplete. When I do a Volume level back up and only select a few like home or swap, the job is successful, or if I just choose root, it fails. If I do a file level backup up and select a few folders, it is successful, but if I select several, it fails at about 53%. The Linux box is a CentOS 6.5 fully patched, and the Windows box is a Server 2012 R2 standard fully patched. Both boxes are in the same room and on the same network. Any assistance would be greatly appreciated! Thanks!

vmniels
Veeam Software
Posts: 2050
Liked: 453 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by vmniels » Aug 01, 2016 9:53 am

How big is your server in total? If you mount the NFS share on the Linux server and write a big file to it (like using dd) does that work? By default the NFS driver by Windows isn't the best so tweaking may be needed.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests