Should I use Veeam Agent for Windows Server and install it on each VM?
Correct. If you're after central agent management, either take a look at Veeam Availability Console or wait for the next release of Veeam Backup and Replication.
Using this licencing can I use a Veeam B&R on-prem or on Azure(?)
Correct, you can point all agents to a backup repository.
or should I also licence the server (how)?
Veeam Agent for Windows has its own licensing model. Basically, there are two scenarios:
1) Separate license distribution (install licenses one by one). In this case you will need to buy as many VAW licenses as there are VMs that need to be backed up
2) Central license distribution (one license is installed either on backup or VAC server; and each connected agent receives a license). In that case you will need to purchase one license containing certain number of workstations and servers (VAW editions)
Ask your sales representative for more information regarding that.
Is there any way i can replicate to VM to my on-prem infrastructure or Azure?
No. However, you can instantly restore physical server or workstation to a Hyper-V host (if you have any). Also, server or workstation can be restored to Azure thanks be for Direct Restore to Microsoft Azure functionality.