Pipeline timeout: it seems that pipeline hanged.

[louyo]

I have opened a support ticket because I think there is a bug in error handling, but I am posting here because I would like to know if I am expecting something the program won't do. I am in evaluation mode of version 4.1.2.
I made a full backup of our Sharepoint 2010 system, running on W2008R2. I copied it to an external drive and then brought it to our development system. This is running on ESXi (we have the essentials license but I am not sure the evaluation version cares). I am administering via a W7 VM (running on a Linux host) using vSphere client.
The backup is only about 6GB, the drive is 180GB. We are moving from WSS 2007 to 2010.
I fired up Veeam on the W7 VM, imported the backup file and started a restore, then went on to other things. When I checked on it, about 9 hours later, the progress indicator was sitting at 100%, the time remaining said 00:00:00, the time elapsed was over 9 hours. Files remaining was something like 1 of 8 (00 KB).
I looked in the RestoreAgent log and it had an entry every 30 minutes that said: "Pipeline timeout: it seems that pipeline hanged". It had been doing this for about 6 hours. No error messages or abort. (so much for unattended restores)
I suspended the vSphere Client VM system and then shut the ESXi server down. The next day, I tried to evaluate the results. The network switch was off, so there were some socket errors. It still had the dialog and it was now at 19 hours and counting (no, I don't think the backup fairy is coming to restore my VM, I just hadn't dismissed the dialog).
Now the log says the the agent session has finished successfully (even with the socket errors, but I am sure this is out of context).
So, I added the restored VM to the inventory and tried to power it on (it looked like all the files were there). I get "the file specified is not a virtual disk".
I will assume that the failure to report an error is a bug in the backup/restore program but will also assume that I am doing something wrong when trying to do the restore to another server.
Is this something people do all the time?

Lou

[Vitaliy S.]

Hello Lou,

Yes, that's exactly what most of our customers do, import the VBK file and perform a VM restore. But could you tell me if you're experiencing the same issues while uploading files to the ESXi Server using vSphere Client? Do you see any connectivity issues, possible timeouts?

I believe you saw a success status, because the files were successfully uploaded to the datastore. This procedure is similar to downloading a file from the FTP server or a Web Server, it only reports on a success, but not on the integrity.

If the restore process wasn't handled properly, I would recommend you to continue working with our technical team on this matter. Additionally, I would try to restore the same VM to the same datastore but to another folder, to see if the issue confirms. Along with that I would check the files (MD5sums) for the first restore and the second one.

Thank you!

[louyo]

Hello Lou,

Yes, that's exactly what most of our customers do, import the VBK file and perform a VM restore. But could you tell me if you're experiencing the same issues while uploading files to the ESXi Server using vSphere Client? Do you see any connectivity issues, possible timeouts?

I believe you saw a success status, because the files were successfully uploaded to the datastore. This procedure is similar to downloading a file from the FTP server or a Web Server, it only reports on a success, but not on the integrity.

If the restore process wasn't handled properly, I would recommend you to continue working with our technical team on this matter. Additionally, I would try to restore the same VM to the same datastore but to another folder, to see if the issue confirms. Along with that I would check the files (MD5sums) for the first restore and the second one.

Thank you!

Good to know. I have seen no failures using converter to move/copy VM's to the ESXi server.
I am working with support, just sent them the logs of another restore that gets almost done 178.60 GB of 180 (although the backup file itself is 6GB).
Again, it is logging Pipeline timeouts every 30 minutes. I cleaned out all folders and restore attempts and started over. I am in the process of installing the Veeam software on another system (64 bit so I have to download again).

It could very well be a glitch in my switch, I suppose, but there is only the one switch and 2 cables involved and I have not seen any other errors.
Who am I to judge, but it seems like your error reporting/error recovery software is not very robust. Both the restoragent and veeamshell logs show 4 of the pipeline timeout: pipeline hanged messages 30 minutes apart. That is over 2 hours without an error message or a log entry indicating that it is retrying.
Sorry if that sounds critical when I am a beginner with your program. I wonder if anyone else has seen this?

I have run several ping tests to make sure there are no fragmented packets, all looks OK over several hundred echoes.
I will press on, maybe do some packet sniffing.


Lou

[Vitaliy S.]

Lou,

I'm not sure that installing Veeam Backup server on another system would help, unless it is using different switch/connection to your ESXi Server. Restoring to ESXi was always slower compared to ESX Server with service console credentials specified.

Having intermittent pipeline hangs shouldn't affect the restore process as it keeps re-trying to restore the data, this was done to handle poor connections in order not to break on a single network glitch. But thank you for posting your feedback on this matter. I'm sure our support team will be able to tell you more after investigating all the log files.

Hope this makes sense!

[louyo]

Lou,

Hope this makes sense!

Yup. The whole thing came crashing down, finally. The ESXi server wound up with corrupted files and will have to be rebuilt. Although the only thing running was your software, I suspect a gremlin (or bad hard drive) was the culprit. I have sent those logs to the support team, but I would guess that the conclusion has to be "bad server".
Perhaps I will revisit your product when version 5 comes out. Over 2 hours of timeouts without any error messages or log entries seems extreme.
Thank you for your time.
Lou

[Gostev]

I believe that in this case, unexpected behavior might have been triggered due to host not performing as expected. Since we cannot model unexpected host behavior on our lab, error handling may not work as expected in situation with corrupted hosts.

[louyo]

I believe that in this case, unexpected behavior might have been triggered due to host not performing as expected. Since we cannot model unexpected host behavior on our lab, error handling may not work as expected in situation with corrupted hosts.

Agreed. Perhaps a developer with time on his hands will take a look and create a dialog that says something like "We haven't heard from the server for a while, do you want to keep waiting or abort?".

Lou
If a man says something in the forest and his wife is not there, is he still wrong?