Comprehensive data protection for all workloads
Post Reply
Ariel
Novice
Posts: 9
Liked: never
Joined: Oct 03, 2011 2:53 pm
Full Name: Ariel Boer
Contact:

Robocopy and corrupt files

Post by Ariel » Oct 03, 2011 3:06 pm

Hey, we are using Veeam to create reverse backups of servers, and we want to send the vbk's to an offsite drive. We have a virtual Veeam server, which backs up to an iSCSI drive, which is then also mapped to our physical backup server which contains our offsite drives, which then attempts to robocopy all the files (for now, we'll whittle it down to just the vbk's eventually) from the iSCSI drive to a local disk. However a large portion of the files (vbks, vrbs, some vibs) cause robocopy to throw the following error:

ERROR 1392 (0x00000570) Copying File H:\DENON\DENON-F2011-09-27T220047.vrb
The file or directory is corrupted and unreadable.

Then later when I tried to use grsync from the same location, and it threw the error "file has vanished"

So is Veeam pumping out broken files, or is robocopy somehow messing them up when it tries to copy them?

Gostev
SVP, Product Management
Posts: 24907
Liked: 3612 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Robocopy and corrupt files

Post by Gostev » Oct 03, 2011 4:17 pm

Sounds like some bad issues with the storage you are writing backups to. What is your "iSCSI drive", is that possibly some low-end NAS?

antspants77
Influencer
Posts: 20
Liked: 5 times
Joined: Mar 16, 2011 3:15 pm
Full Name: Tony Spencer
Contact:

Re: Robocopy and corrupt files

Post by antspants77 » Oct 04, 2011 7:43 am

If you have Surebackup running this will lock the files.
We had the Surebackups kept running, so that Exchange Restores would be quicker, but ran into this issue.
Also an issue with Synthetics when SureBackup is running.

Ariel
Novice
Posts: 9
Liked: never
Joined: Oct 03, 2011 2:53 pm
Full Name: Ariel Boer
Contact:

Re: Robocopy and corrupt files

Post by Ariel » Oct 12, 2011 12:54 pm

Gostev, we're using a Qnap box... you think this is causing issues?

Vitaliy S.
Product Manager
Posts: 23062
Liked: 1581 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Robocopy and corrupt files

Post by Vitaliy S. » Oct 12, 2011 1:50 pm

Yes, could be. I recall that our existing users complained about pretty bad iSCSI implementation on QNAP.
There are some existing topics about this > http://forums.veeam.com/search.php?keywords=QNAP+iscsi

Gostev
SVP, Product Management
Posts: 24907
Liked: 3612 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Robocopy and corrupt files

Post by Gostev » Oct 12, 2011 2:00 pm

I actually sent my QNAP back because of that. But the sad truth is, I am yet to find low-end NAS with reliable and performant iSCSI implementation.

Ariel
Novice
Posts: 9
Liked: never
Joined: Oct 03, 2011 2:53 pm
Full Name: Ariel Boer
Contact:

Re: Robocopy and corrupt files

Post by Ariel » Oct 12, 2011 3:36 pm

so what would be a proper option to replace this? should we use a network drive? can we use the qnap as a network drive instead of as an iscsi drive or do you recommend against it entirely? thank, and sorry if this is covered elsewhere...

Vitaliy S.
Product Manager
Posts: 23062
Liked: 1581 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Robocopy and corrupt files

Post by Vitaliy S. » Oct 12, 2011 4:04 pm

Given the issues you describe above, I would suggest going with CIFS/NFS.

For more details please look through this topic that highlights all pros and cons of using different backup targets: recommendations for veeam backup storage

Ariel
Novice
Posts: 9
Liked: never
Joined: Oct 03, 2011 2:53 pm
Full Name: Ariel Boer
Contact:

Re: Robocopy and corrupt files

Post by Ariel » Oct 12, 2011 6:13 pm

hmm, also just to throw into the mix which I had forgotten about, when using our Dell Equallogic SAN, the same corruption issues arose.

tsightler
VP, Product Management
Posts: 5437
Liked: 2257 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Robocopy and corrupt files

Post by tsightler » Oct 12, 2011 6:43 pm

Any chance there's some antivirus software running that is doing something behind the scenes. Veeam just creates regular files, there's nothing really special about them. Many users copy them off nightly or replicate them so if you're having issues just copying files around, that's pretty strange. Even if the file itself was somehow "corrupt" internally, you would still be able to copy it. Only if Veeam was actually corrupting the filesystem would it be able to cause such an error, and since Veeam simply reads/writes files via normal Windows API it would seem impossible to corrupt the filesystem.

Ariel
Novice
Posts: 9
Liked: never
Joined: Oct 03, 2011 2:53 pm
Full Name: Ariel Boer
Contact:

Re: Robocopy and corrupt files

Post by Ariel » Oct 12, 2011 6:48 pm

hmm, we have Trend Micro Worry Free Business A/V running, do you think we should try running after disabling it?

Vitaliy S.
Product Manager
Posts: 23062
Liked: 1581 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Robocopy and corrupt files

Post by Vitaliy S. » Oct 12, 2011 7:09 pm

Oh, definitely or you should at least add backup files into exception list.

Recently we had another problem reported where Trend Micro was involved: BSOD STOP: 0x0000003B SYSTEM_SERVICE_EXCEPTION

Good catch, Tom! :wink:

Gostev
SVP, Product Management
Posts: 24907
Liked: 3612 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Robocopy and corrupt files

Post by Gostev » Oct 12, 2011 9:23 pm

Yep, by all means, please exclude all backup files from the scope of antivirus.

tsightler
VP, Product Management
Posts: 5437
Liked: 2257 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Robocopy and corrupt files

Post by tsightler » Oct 12, 2011 10:43 pm

Yes, I suspect it may be tripping a virus definition or some other behavior and the A/V is blocking access to the file. Just a suspicion at this point but I would suggest disabling A/V to test for sure.

jsprinkleisg
Service Provider
Posts: 22
Liked: 4 times
Joined: Dec 09, 2009 9:59 pm
Full Name: James Sprinkle
Contact:

Re: Robocopy and corrupt files

Post by jsprinkleisg » Oct 17, 2011 7:36 pm

Ariel wrote:...We have a virtual Veeam server, which backs up to an iSCSI drive, which is then also mapped to our physical backup server...
Wait - kinda sounds like you have an iSCSI LUN mapped to two different servers simultaneously? If that's the case, that could definitely cause corruption problems.

tsightler
VP, Product Management
Posts: 5437
Liked: 2257 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Robocopy and corrupt files

Post by tsightler » Oct 17, 2011 8:07 pm

Certainly true. I guess I assumed he meant mapped as in "mapped drive". If your mapping an NTFS formatted iSCSI LUN to more than a single host, that's a path to certain corruption.

Gostev
SVP, Product Management
Posts: 24907
Liked: 3612 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Robocopy and corrupt files

Post by Gostev » Oct 21, 2011 11:10 am

Gostev wrote:I actually sent my QNAP back because of that. But the sad truth is, I am yet to find low-end NAS with reliable and performant iSCSI implementation.
Back from VMworld with brand new Drobo B800i. Planning to put it through some good testing in my lab with v6, and will report back. Folks from Drobo are claiming their focus is providing fast and reliable production and backup storage (without tons of fluff features), which sounds like exactly what I need. Will see what they've got and how well it works.

Ariel
Novice
Posts: 9
Liked: never
Joined: Oct 03, 2011 2:53 pm
Full Name: Ariel Boer
Contact:

Re: Robocopy and corrupt files

Post by Ariel » Nov 02, 2011 5:41 pm

jsprinkleisg wrote: Wait - kinda sounds like you have an iSCSI LUN mapped to two different servers simultaneously? If that's the case, that could definitely cause corruption problems.
you think so? I just jumped back on here to say that disabling AV did not help. How would you recommend we access that drive? Map to it like \\veeamServer\iscsiDrive?

Gostev
SVP, Product Management
Posts: 24907
Liked: 3612 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Robocopy and corrupt files

Post by Gostev » Nov 02, 2011 8:22 pm

He means you should not attach the same LUN via iSCSI initiator to two Windows servers at once, which is very valid suggestion. Attach it via iSCSI to backup server, and from other servers access it via a CIFS share.

Post Reply

Who is online

Users browsing this forum: MYantachka, yastt and 53 guests