Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

First backup run failed: I/O error

Post by ovidiu »

Code: Select all

Backup session on jacob.ict-consult.co.za [75] %               Status: Failed




     Summary                                Data

     Duration:           00:09:21           Processed:          61.2 GB (75%)
     Processing rate:    38.2 MB/s          Read:               18.8 GB
     Bottleneck:         Source             Transferred:        3.9 GB (4.9x)


     Time                Action                                        Duration

     2016-07-10 11:47:45 Job started at 2016-07-10 11:47:45
     2016-07-10 11:47:45 Starting backup
     2016-07-10 11:47:45 Creating volume snapshot                      00:00:02
     2016-07-10 11:47:52 Backed up /dev/vda 501.0MB at 69.5MB/s        00:00:07
     2016-07-10 11:47:59 [error] Backing up vg_00 17.0GB at 35.9MB/s...00:08:55
     2016-07-10 11:57:06 [error] Input/output error
     2016-07-10 11:57:06 [error] Failed to perform backup
Here is the link to an upload of /var/log/veeam on dropbox. I couldn't spot anything sensitive in there but please let me know if I should obscure some parts? => https://www.dropbox.com/sh/za0ugrlkzez4 ... nVJ6a?dl=0
ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu »

tried again, got further but still I/O error. Could this be a problem with my NFS storage?

Code: Select all

Backup session on jacob.ict-consult.co.za [83] %               Status: Failed




     Summary                                Data

     Duration:           00:13:35           Processed:          67.3 GB (83%)
     Processing rate:    36.7 MB/s          Read:               24.9 GB
     Bottleneck:         Source             Transferred:        4.4 GB (5.7x)


     Time                Action                                        Duration

     2016-07-10 12:18:52 Job started at 2016-07-10 12:18:52
     2016-07-10 12:18:53 Starting backup
     2016-07-10 12:18:53 Creating volume snapshot                      00:00:05
     2016-07-10 12:19:04 Backed up /dev/vda 501.0MB at 70.8MB/s        00:00:07
     2016-07-10 12:19:11 [error] Backing up vg_00 22.7GB at 34.6MB/s...00:13:04
     2016-07-10 12:32:26 [error] Input/output error
     2016-07-10 12:32:26 [error] Failed to perform backup
ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu »

oh and on hte target I am left with a 4GB .vbk file, shouldn't VAL clean up after itself?
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by nielsengelen »

Could you give some more info about the NFS share? How is it exported?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu »

I cannot tell you anythign about it as it is being provided my my ISP: https://www.soyoustart.com/en/backup-storage.xml
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by nielsengelen »

We'll have to wait for feedback from the devs on the logs but I/O error seems to be related due to overflow on the share (or a timeout). Do you see anything related nfs timeouts in dmesg ?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu »

Here is the corresponding part of /var/log/messages starting with me installing nfs-utils before setting up VAL and running the first backup:

Code: Select all

Jul 10 11:45:42 jacob yum[5689]: Installed: nfs-utils-lib-1.1.5-11.el6.x86_64
Jul 10 11:45:42 jacob yum[5689]: Installed: 1:nfs-utils-1.2.3-70.el6.x86_64
Jul 10 11:47:22 jacob kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jul 10 11:47:22 jacob kernel: NFS: Registering the id_resolver key type
Jul 10 11:47:22 jacob kernel: FS-Cache: Netfs 'nfs' registered for caching
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
there are plenty of further nfs timeouts after these...
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by nielsengelen »

Can you try to make a backup to a local disk as a test and see if that works fine? It looks like the nfs share gets timeouts due latency or something and this ends up in a temporary death path.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu »

a test backup to local storage completed just fine. anything we can do about the nfs probs from this side? I'm also checking with the storage provider to see if there is anything amss.
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by nielsengelen »

How is your setup? As this is cloud storage via NFS, is the VM you are using for testing it in the same location or in another datacenter? If you manually mount it and try to write big files to it does it stay online?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu »

I am renting a server from So You Start which runs Proxmox as hypervisor. There are a few LXCs and a few KVMs on this host.
The backup storage is in the same datacenter. I'll run a manual test:

- mounted nfs successfully
- fallocate -l 5G test.img
- cp test.img /mnt/test
cp test.img /mnt/test worked just fine.
ovidiu
Influencer
Posts: 19
Liked: 1 time
Joined: Jul 07, 2016 3:07 pm
Contact:

Re: First backup run failed: I/O error

Post by ovidiu »

It must be the provider as I just successfully ran a veeam backup. I'll take it up with the provider.

Code: Select all

Backup session on jacob.ict-consult.co.za [100] %             Status: Success




     Summary                                Data

     Duration:           00:10:00           Processed:          81.1 GB (100%)
     Processing rate:    67.3 MB/s          Read:               38.6 GB
     Bottleneck:         Source             Transferred:        5 GB (7.7x)


     Time                Action                                        Duration

     2016-07-10 21:59:08 Job started at 2016-07-10 21:59:08
     2016-07-10 21:59:08 Starting backup
     2016-07-10 21:59:09 Creating volume snapshot                      00:00:03
     2016-07-10 21:59:18 Backed up /dev/vda 501.0MB at 71.6MB/s        00:00:07
     2016-07-10 21:59:25 Backed up vg_00 35.5GB at 64.0MB/s            00:09:27
     2016-07-10 22:08:54 Backing up summary.xml                        00:00:00
     2016-07-10 22:09:08 Backup completed
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by nielsengelen »

feels like a latency or specific setting issue. Keep us posted on feedback from them.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
tdouble
Novice
Posts: 7
Liked: 1 time
Joined: Jul 28, 2016 9:03 pm
Full Name: Todd Gorman
Contact:

Re: First backup run failed: I/O error

Post by tdouble »

I am getting the same error. When I do an Entire Machine backup, it gets to 50% and then fails with the message I/O error. Looking at the Windows server NFS share, the file looks incomplete. When I do a Volume level back up and only select a few like home or swap, the job is successful, or if I just choose root, it fails. If I do a file level backup up and select a few folders, it is successful, but if I select several, it fails at about 53%. The Linux box is a CentOS 6.5 fully patched, and the Windows box is a Server 2012 R2 standard fully patched. Both boxes are in the same room and on the same network. Any assistance would be greatly appreciated! Thanks!
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by nielsengelen »

How big is your server in total? If you mount the NFS share on the Linux server and write a big file to it (like using dd) does that work? By default the NFS driver by Windows isn't the best so tweaking may be needed.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
tdouble
Novice
Posts: 7
Liked: 1 time
Joined: Jul 28, 2016 9:03 pm
Full Name: Todd Gorman
Contact:

Re: First backup run failed: I/O error

Post by tdouble »

The CentOS 6.5 server is 500GB and the Windows server is 700GB. I was able to create another CentOS 6.5 server, set it up as the NFS server, and was able to run a full backup job from the original CentOS box to the new one using the Veeam Linux agent. Looks like there is an issue with the Windows driver.
tdouble
Novice
Posts: 7
Liked: 1 time
Joined: Jul 28, 2016 9:03 pm
Full Name: Todd Gorman
Contact:

Re: First backup run failed: I/O error

Post by tdouble »

I spoke too soon. I thought it was going to complete, but it did fail at 99% with an I/O error.
2016-08-04 07:51:27 Backed up /dev/sda 500.5MB at 36.2MB/s 00:00:14 │
2016-08-04 07:51:41 [error] Backing up vg_toddcentos 456.0...03:39:51 │
2016-08-04 11:31:42 [error] Input/output error ▒
2016-08-04 11:31:42 [error] Failed to perform backup

This was with a CentOS 6.5 500GB physical server to another CentOS 6.5 physical server 500GB on the same network.
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by nielsengelen »

Well it does seems that Windows isn't the best target. Did you by any chance did a retry to see if it would finish to 100%? Could you provide us the logs as well (dmesg -T, /var/log/veeam and syslog if possible).
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
tdouble
Novice
Posts: 7
Liked: 1 time
Joined: Jul 28, 2016 9:03 pm
Full Name: Todd Gorman
Contact:

Re: First backup run failed: I/O error

Post by tdouble »

The retry failed at 99% again.
When I try dmesg -T, /var/log/veeam, I get invalid option -T
With out the -T I cannot get the full logs which seems to way too much. I am no Linux guru, can you tell me the best way to get these logs for you?
Thanks!
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: First backup run failed: I/O error

Post by PTide »

Then please run the following commands:

dmesg > dmesg.log
tar -zcvf veeam_log.tar.gz /var/log/veeam.
tar -zcvf messages.tar.gz /var/log/messages
lsblk -af > lsblk.log

After you've executed the commands you can find dmesg.log, veeam_log.tar.gz, messages.tar.gz, lsblk.log files in your current directory, please send them to us via PM or share them.

Thanks
tdouble
Novice
Posts: 7
Liked: 1 time
Joined: Jul 28, 2016 9:03 pm
Full Name: Todd Gorman
Contact:

Re: First backup run failed: I/O error

Post by tdouble »

I have gotten the logs and put them in dropbox for you and have sent you the link in PM. Please let me know if there is anything else you need from me. Thanks!!
tdouble
Novice
Posts: 7
Liked: 1 time
Joined: Jul 28, 2016 9:03 pm
Full Name: Todd Gorman
Contact:

Re: First backup run failed: I/O error

Post by tdouble » 1 person likes this post

I just wanted to give an update on this. I have gotten Linux Beta V2 and had no issues going CentOS to CentOS. It was also very fast. It looks like there were major improvements in V2. I will try the CentOS to Windows server when I have time and will update this post. Thanks!
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: First backup run failed: I/O error

Post by nielsengelen »

This might be related to the default NFS service not being the best on Windows 2012 and before. There are alternatives such as FreeNFS however they hardly being updated.

A potention option would be to test this with Windows 2016 RTM edtion to see for improvements. In general NFS just works better on Linux ;-)
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests