First backup run failed: I/O error

[ovidiu]

Code: Select all

Backup session on jacob.ict-consult.co.za [75] %               Status: Failed




     Summary                                Data

     Duration:           00:09:21           Processed:          61.2 GB (75%)
     Processing rate:    38.2 MB/s          Read:               18.8 GB
     Bottleneck:         Source             Transferred:        3.9 GB (4.9x)


     Time                Action                                        Duration

     2016-07-10 11:47:45 Job started at 2016-07-10 11:47:45
     2016-07-10 11:47:45 Starting backup
     2016-07-10 11:47:45 Creating volume snapshot                      00:00:02
     2016-07-10 11:47:52 Backed up /dev/vda 501.0MB at 69.5MB/s        00:00:07
     2016-07-10 11:47:59 [error] Backing up vg_00 17.0GB at 35.9MB/s...00:08:55
     2016-07-10 11:57:06 [error] Input/output error
     2016-07-10 11:57:06 [error] Failed to perform backup

Here is the link to an upload of /var/log/veeam on dropbox. I couldn't spot anything sensitive in there but please let me know if I should obscure some parts? => https://www.dropbox.com/sh/za0ugrlkzez4 ... nVJ6a?dl=0

[ovidiu]

tried again, got further but still I/O error. Could this be a problem with my NFS storage?

Code: Select all

Backup session on jacob.ict-consult.co.za [83] %               Status: Failed




     Summary                                Data

     Duration:           00:13:35           Processed:          67.3 GB (83%)
     Processing rate:    36.7 MB/s          Read:               24.9 GB
     Bottleneck:         Source             Transferred:        4.4 GB (5.7x)


     Time                Action                                        Duration

     2016-07-10 12:18:52 Job started at 2016-07-10 12:18:52
     2016-07-10 12:18:53 Starting backup
     2016-07-10 12:18:53 Creating volume snapshot                      00:00:05
     2016-07-10 12:19:04 Backed up /dev/vda 501.0MB at 70.8MB/s        00:00:07
     2016-07-10 12:19:11 [error] Backing up vg_00 22.7GB at 34.6MB/s...00:13:04
     2016-07-10 12:32:26 [error] Input/output error
     2016-07-10 12:32:26 [error] Failed to perform backup

[ovidiu]

oh and on hte target I am left with a 4GB .vbk file, shouldn't VAL clean up after itself?

[nielsengelen]

Could you give some more info about the NFS share? How is it exported?

[ovidiu]

I cannot tell you anythign about it as it is being provided my my ISP: https://www.soyoustart.com/en/backup-storage.xml

[nielsengelen]

We'll have to wait for feedback from the devs on the logs but I/O error seems to be related due to overflow on the share (or a timeout). Do you see anything related nfs timeouts in dmesg ?

[ovidiu]

Here is the corresponding part of /var/log/messages starting with me installing nfs-utils before setting up VAL and running the first backup:

Code: Select all

Jul 10 11:45:42 jacob yum[5689]: Installed: nfs-utils-lib-1.1.5-11.el6.x86_64
Jul 10 11:45:42 jacob yum[5689]: Installed: 1:nfs-utils-1.2.3-70.el6.x86_64
Jul 10 11:47:22 jacob kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jul 10 11:47:22 jacob kernel: NFS: Registering the id_resolver key type
Jul 10 11:47:22 jacob kernel: FS-Cache: Netfs 'nfs' registered for caching
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out
Jul 10 11:55:50 jacob kernel: nfs: server ftpback-rbx3-408.ovh.net not responding, timed
out

there are plenty of further nfs timeouts after these...

[nielsengelen]

Can you try to make a backup to a local disk as a test and see if that works fine? It looks like the nfs share gets timeouts due latency or something and this ends up in a temporary death path.

[ovidiu]

a test backup to local storage completed just fine. anything we can do about the nfs probs from this side? I'm also checking with the storage provider to see if there is anything amss.

[nielsengelen]

How is your setup? As this is cloud storage via NFS, is the VM you are using for testing it in the same location or in another datacenter? If you manually mount it and try to write big files to it does it stay online?

[ovidiu]

I am renting a server from So You Start which runs Proxmox as hypervisor. There are a few LXCs and a few KVMs on this host.
The backup storage is in the same datacenter. I'll run a manual test:

- mounted nfs successfully
- fallocate -l 5G test.img
- cp test.img /mnt/test
cp test.img /mnt/test worked just fine.

[ovidiu]

It must be the provider as I just successfully ran a veeam backup. I'll take it up with the provider.

Code: Select all

Backup session on jacob.ict-consult.co.za [100] %             Status: Success




     Summary                                Data

     Duration:           00:10:00           Processed:          81.1 GB (100%)
     Processing rate:    67.3 MB/s          Read:               38.6 GB
     Bottleneck:         Source             Transferred:        5 GB (7.7x)


     Time                Action                                        Duration

     2016-07-10 21:59:08 Job started at 2016-07-10 21:59:08
     2016-07-10 21:59:08 Starting backup
     2016-07-10 21:59:09 Creating volume snapshot                      00:00:03
     2016-07-10 21:59:18 Backed up /dev/vda 501.0MB at 71.6MB/s        00:00:07
     2016-07-10 21:59:25 Backed up vg_00 35.5GB at 64.0MB/s            00:09:27
     2016-07-10 22:08:54 Backing up summary.xml                        00:00:00
     2016-07-10 22:09:08 Backup completed

[nielsengelen]

feels like a latency or specific setting issue. Keep us posted on feedback from them.

[tdouble]

I am getting the same error. When I do an Entire Machine backup, it gets to 50% and then fails with the message I/O error. Looking at the Windows server NFS share, the file looks incomplete. When I do a Volume level back up and only select a few like home or swap, the job is successful, or if I just choose root, it fails. If I do a file level backup up and select a few folders, it is successful, but if I select several, it fails at about 53%. The Linux box is a CentOS 6.5 fully patched, and the Windows box is a Server 2012 R2 standard fully patched. Both boxes are in the same room and on the same network. Any assistance would be greatly appreciated! Thanks!

[nielsengelen]

How big is your server in total? If you mount the NFS share on the Linux server and write a big file to it (like using dd) does that work? By default the NFS driver by Windows isn't the best so tweaking may be needed.

[tdouble]

The CentOS 6.5 server is 500GB and the Windows server is 700GB. I was able to create another CentOS 6.5 server, set it up as the NFS server, and was able to run a full backup job from the original CentOS box to the new one using the Veeam Linux agent. Looks like there is an issue with the Windows driver.

[tdouble]

I spoke too soon. I thought it was going to complete, but it did fail at 99% with an I/O error.
2016-08-04 07:51:27 Backed up /dev/sda 500.5MB at 36.2MB/s 00:00:14 │
2016-08-04 07:51:41 [error] Backing up vg_toddcentos 456.0...03:39:51 │
2016-08-04 11:31:42 [error] Input/output error ▒
2016-08-04 11:31:42 [error] Failed to perform backup

This was with a CentOS 6.5 500GB physical server to another CentOS 6.5 physical server 500GB on the same network.

[nielsengelen]

Well it does seems that Windows isn't the best target. Did you by any chance did a retry to see if it would finish to 100%? Could you provide us the logs as well (dmesg -T, /var/log/veeam and syslog if possible).

[tdouble]

The retry failed at 99% again.
When I try dmesg -T, /var/log/veeam, I get invalid option -T
With out the -T I cannot get the full logs which seems to way too much. I am no Linux guru, can you tell me the best way to get these logs for you?
Thanks!

[PTide]

Then please run the following commands:

dmesg > dmesg.log
tar -zcvf veeam_log.tar.gz /var/log/veeam.
tar -zcvf messages.tar.gz /var/log/messages
lsblk -af > lsblk.log

After you've executed the commands you can find dmesg.log, veeam_log.tar.gz, messages.tar.gz, lsblk.log files in your current directory, please send them to us via PM or share them.

Thanks

[tdouble]

I have gotten the logs and put them in dropbox for you and have sent you the link in PM. Please let me know if there is anything else you need from me. Thanks!!

[tdouble]

I just wanted to give an update on this. I have gotten Linux Beta V2 and had no issues going CentOS to CentOS. It was also very fast. It looks like there were major improvements in V2. I will try the CentOS to Windows server when I have time and will update this post. Thanks!

[nielsengelen]

This might be related to the default NFS service not being the best on Windows 2012 and before. There are alternatives such as FreeNFS however they hardly being updated.

A potention option would be to test this with Windows 2016 RTM edtion to see for improvements. In general NFS just works better on Linux