We had the same issue of not using standard SQL port numbers. First we had to address the database restore as a restore to a new database and/or server. We would list the [server name],[port #] , this was good for single node SQL servers. Our next problem was 90% of our SQL servers were clustered in Availability Groups. Even though you could direct the restore over the custom port number through the SQL Explorer Wizard, when the wizard reached the Availability Group settings, it would show "not connected" to the other nodes in the cluster. This would cause the restore not to be written into the AG setting. We found that using SQL aliases would solve this issue. You can find the it here; [color=#0000FF]https://support.microsoft.com/en-us/help/936491/after-you-install-a-64-bit-version-of-sql-server-2005-on-a-64-bit-vers.
We created a .reg file with all the aliases instead of using cliconfg.exe which looked like this;
Windows Registry Editor Version 5.00
"[Server Name]"="DBMSSOCN,[Server Name],[Port#]"
This solved our issues and allows all restore communication to talk to the restore target servers via the custom port we have designated for SQL traffic