Sometimes I / O device error occurs in backup jobs

[wajika]

During scheduled scheduling, IO errors occasionally occur. I don't know the reason. I checked the disk and the windows event viewer and found no special exceptions.
Can anyone give me some advice? thanks

veeam is installed on vmware, and the storage of freenas is mounted to vmware.
veeam version: 10

Code: Select all

2021/5/21 17:34:20 :: Processing CRM-PRD-APP1 Error: The request could not be performed because of an I/O device error.
Failed to write data to the file [H:\Backups\CRMBackup_clone1\CRMBackup_clone1D2021-05-21T173419_75B1.vib].
Failed to download disk 'CRM-PRD-APP1-flat.vmdk'.
Shared memory connection was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

[Mildur]

NAS SMB Targets are not really stable Backup Targets for Veeam. I do not recommend to use it.

The reason why your backup is failing could be anything. Unstable network connection. Slow performance on your nas device. Slow raid controller in you nas. Is this a business environment or a homelab?
What is the hardware for the Freenas? Some old PC reused as a NAS or correct dimensioned hardware for Veeam Backup Repo Usage?

You can try to open a support case with Veeam.
But I don‘t see the issue on veeams site. They can help you to find out where the issue could be.

[wajika]

I use the new freenas (used for one year). Due to insufficient disk space on vsphere, I mounted a nas device (produced by ixsystems.com). There is no bottleneck in the network bandwidth, and the load on the NAS device is not very high.

freenas
CPU：Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
MEM: 40.0GiB

[Mildur]

I hope, i get your environment right:
- Some vSphere Servers
- Backup Server is on a VM?
- the FreeNAS Server is only for Backup Storage and is connected to the vsphere As a iSCSI (?) Datastore?

I have misread your configuration in the first place. I am sorry for that

Do you have disconnects on vsphere to the datastore? Or some suspicious events?

[wajika]

I'm sorry, but I didn't make it clear at the beginning, I will re-summarize it.

1. All systems are installed on the vsphere server (including veeam).
2. Freenas is only used for backup. It connects to vsphere as iSCSI storage.
3. I checked vsphere and found no special fault.

[Mildur]

I think it‘s best to open a support case by veeam. They will find more information to help you in the logs

[wajika]

I didn't buy veeam license, I use the community version.

[Mildur]

You can open support cases on best effort with the community edition.

[wajika]

okay, thank you

[wajika]

Can you share where to seek support?

[Mildur]

Of course.
Open your case here:

Https://support.veeam.com

[soncscy]

Just my input:

I'm sorry, but I didn't make it clear at the beginning, I will re-summarize it.

1. All systems are installed on the vsphere server (including veeam).
2. Freenas is only used for backup. It connects to vsphere as iSCSI storage.
3. I checked vsphere and found no special fault.

Based on your original post, I'm guessing that by 2 you mean that you have an iscsi drive attached to your Veeam Server as the H: drive, is that correct?

If that's the case, you need to check the Veeam Server System and Application logs on that server and look for iscsi events.
Also, start a wireshark dump on the Veeam Server during the backup and capture traffic on the NIC which connects to the iscsi target. Once the event happens, filter your traffic and just look for ip.src == Veeam server ip and ip.dst==iscsi target ip, and look for issues.

Almost certainly this is a network issue or the NAS itself not handling the IO; don't rely on just looking for peak load! This is not reliable at all, and faults typically happen very discretely.

[wajika]

H: drive is mounted by freenas as the iscsi drive.

Okay, I will try to use wireshark, there is a clue, I/O error problems often occur when backing up large files (for example, 1TB+ size).