I would definitely go for physical server for environment of your size, and given you have ability to perform VCB SAN backup on FC4 (which is recommended by VMware as best possible backup method). This way you will get MUCH better performance, fully LAN free backup that does not touch production ESX hosts, and ability to grow your virtual environment without worrying about backup in future.
For larger environments, you are really not saving much by putting backup application in VM on ESX host. Backup processes are fairly resource-consuming, so Veeam Backup will load ESX host pretty heavily. Then, all backup I/O will go through ESX I/O stack (terabytes of data!), instead of being fully offloaded to storage layer as with VCB SAN mode. All this will definitely be affecting production VMs, and in the end you will find yourself having to add extra ESX host (which are WAY more expensive than cheap regular server, plus the cost of VMware license).