Backup copy job for large VM failed after update to R2

[christiankelly]

Just in case anyone is thinking about updating to R2 if you have large copy jobs, you may want to hold off. I have a large 1TB busy (200GB/change rate per day) VM which was mostly working under 7.0 and after reading about the R2 update fixing some issues relating to copyjobs I updated and after that point never got backup copy to work again.

I opened a support case 00478591 but the support didn’t really have any clue other than going back to 7 which didn’t go very smooth, (Lost all my caching and after manual database edits it was hard to get the transports / WAN back to 7.0) I was without any offsite job for a week and now with the rebuild of the cache it may be a few more days before I have a backup offsite.

The odd thing was with R2 the copy would run for somewhere between 9m and 11m and then error with “Error: An existing connection was forcibly closed by the remote host --tr: Cannot write data to the socket. Data size: [1048576] -tr:Failed to send block ( offset: '3145656', data size: '1048552', index: '3')….” it would do that 5 times and then stop and wait for a new incremental and then do the same thing.

Very odd stuff. Anyway just posting in case anyone else is in a similar situation.

[veremin]

Hi, Christian,

I've asked QA team to take a look at your case and below there are few things they've noticed:

1) Your network seems to be very unstable with drops happening periodically. However, this is just FYI, since neither unstable network nor periodic drops should lead to the said error.

2) According to the corresponding logs, your target lacks free space, thus, the backup copy job can't write anything to it. The only thing that is still unclear is that you should have been notified about lack of free space by different message, not by the one you saw. Anyway, can you confirm that there is sufficient space in the target repository?

Thanks.

[christiankelly]

Hi Eremin,

Thanks for having QA take a look. As far as the WAN it's a 50mbit VPN tunnel but its east to west coast so about 80ms of latency.

The target has plenty of room, 900GB used of 1.5TB and Veeam 7.0 has been running all night without issue (about 21% done). So R2 was the culprit. I hope the logs are enough to show what was wrong. I asked support if there was a way engineering could take a look at the live server before we rolled back, but he said that would've taken many days.

Thanks,

[veremin]

I should have been more specific. According to your logs, there is not enough space for the target WAN accelerator cache. Can you confirm that target WAN accelerator's cache folder has sufficient amount of free disk space available?

Thanks.

[christiankelly]

I have a 600GB volume dedicated for the target WAN accelerator cache, and it has 200GB free. It's never been totally full as far as I can tell, but the size is = to the disk in any case.

[veremin]

Hmm, strange situation, indeed, since there is certainly an error in the provided logs, according to which, the task failed due to the lack of free space.

[17.11.2013 15:05:48] < 21920> ERR |There is not enough space on the disk.

So, can you confirm that there is sufficient space on each component that participates in backup copying process (be it source/target repository, source/target WAN accelerator, etc.)?

You may ask support team whether or not this is the root cause of the seen behavior.

Thanks.

[RMullis]

Hi Christian

I had a similar issue a while back. i used a VM for my wancache guest. I provisioned a 2nd disk for 225GB and set wancache at 200GB and ran into errors. I then added another 25GB and it worked great. Maybe you could try shrinking your cache setting just as a test.

Rick

[veremin]

Yesterday another user reported about the same issue. The root cause was the lack of free space. So, it's certainly worth double checking.

Thanks.