I also have a AltaVault and use it for backup copy jobs, running Veeam 9.5 U2 and AltaVault code 4.3.1, presented as CIFS share in Veeam. Proxies are Server 2016 boxes. I would add a backup repository big enough to have a few restore points in front of your AltaVault, then backup copy jobs to the AltaVault. This is the way I have it setup and it works well. My main jobs with a few days retention go to by main fast backup disk/repository ( a HP server with a direct attached shelf). Then I run copy jobs to get the data onto the AltaVault and streamed to Azure.
When we were looking at the AltaVault a year ago, the NetApp engineer recommended setting it up this way as you don't want to try and dump directly to a dedupe/compression device. Same with restores, if you have to do all your restores from it, it's going to be slower than having some plane-jane disk and a few normal restore points. The AltaVault really isn't meant to be the main landing site for backups. It's a long term/2nd tier/archival ( or whatever you want to call it) storage device. I think of it as a replacement for tape drives.
That being said. My biggest VM's are ~6TB, they take under 5 minutes to mount a FLR from local data on the AltaVault. Cloud data you have to factor in the time it takes to pre-populate the data on the AltaVault before kicking off the restore. What kind of restore times/speeds are you seeing?