Similar issue here:
Three servers, two with ESX 3.5 (Update 3), monitored smoothly with Veeam Monitor 188.8.131.52, one with ESXi 4.0, always showing "connection state, not responding".
Further investigation showed, that the system time of the machine differed more than two hours to the VeeAm Monitor PC. Correcting the system time on the ESX did the trick.
Question to the VeeAm-Support for further versions : Could you give a hint in the VeeAm Monitor, ex.: "Server not responding (Check System date on ESX host and VeeAm Monitor-Computer)"?