Migration of Veeam server

[siyanor]

Hi there,
We are currently having a Veeam server with a LTO 6 autoloader attached to it as SAS and disk library to save restore points on it.
Veeam is connected to our production servers via 2 ethernet cabled which is providing 1 GB of bandwidth, we are having 10 ESXI servers , with around 100 VM's and around 5 Agent based physical servers which Veeam is taking care of them but is really slow.

We have convinced our top management to upgrade of Veeam servers so we have purchased a brand new server with LTO 8 auto loader .new server will be connected to production and Auto loader via Fiber optic which I think will make our backup process much faster.
I have a few questions as below:
1- we are currently having around 20 TB data on old server , our support vendor is saying we will create new jobs on new server and there is no need to migrate configuration & data , as we will keep the old server until the retention period of restore points on old server is over and then we will demote the old server.Regarding the tape auto loader , we will attach the LTO 6 to new server in case we need to restore any data from LTO6 tapes .by the way,We did not encrypted the backup on tape,is this acceptable or we must migrate configuration file to new server?

2-we are having daily & monthly Jobs,Daily jobs needs to be backed up to tape as full backup,do you guys recommend ,reverse backup or creating virtual full backup on tape?is it recommended to keep all our VM which needs to be backed up to tape on daily bases in one job ? will this increase the restore time in case we need to restore one of the VM from tape ?

3 - we will take the old server to our DR site for replication , which I have never tried before,do we need to replicate all our jobs to off site or we can choose what to replicate ?


Thank you in advance.
Regards

[foggy]

Hi Ahmad, please find the answers below.

is this acceptable or we must migrate configuration file to new server?

It is totally acceptable and completely up to you whether you want to keep the backup server settings and jobs (and map the jobs to existing backups to continue backup chains) or create everything from scratch.

is it recommended to keep all our VM which needs to be backed up to tape on daily bases in one job ? will this increase the restore time in case we need to restore one of the VM from tape ?

No, you can use direct restore of a single VM to infrastructure.

3 - we will take the old server to our DR site for replication , which I have never tried before,do we need to replicate all our jobs to off site or we can choose what to replicate ?

You can choose what VMs to replicate (the most critical ones, for example).

[siyanor]

Hi foggy , thank you for your answer.

No, you can use direct restore of a single to infrastructure.

I have read the above link, It is stating that we should not use the direct restore if we are using parallel recording while recording backup, I think with Fiber connection we will go for parallel recording ( I assume parallel recording means running multiple backup jobs concurrently )as it will save time to take backup.

In case I want to keep all daily disk to tape in one job is there any option (or script )to make sure the tape job does not run until all backup to disk are completed?

[Adrian1980]

Hi siyanor,

to your last question:
In case your backup to disk is splitted to different jobs, you can start one of these jobs at a time, you can ensure it's end at last and link your Tape-Job to as after-run to this job.
As alternative you can analyse how long your backups run and plan the starttime of your Tape-Job with an enough reserve of time.

One hint for the "one-tape-job"-thing:
I remember, if you need to do a filelevel restore, you need to do a staged restore to disk for the full backup file before you can do file level restore.
I don't know yet, if this behavior changed with new 9.5 Update 4, but you have to be aware of this, before you are in the situation of need this.
You have to decide for yourself if a file-level-restore from tape can happen at any time and if it better, to split the jobs for smaller backupfiles.
Alternative, you can do a restore of the VM direct from tape an get the files you need direct from the VM, that's maybe much faster as a staged restore.

Why you write full to tape every day?

[siyanor]

Hi Adrian,
Thank you for your reply.
As I have mentioned in My first post, we are having around 100 VMs and Physical servers, which we take backup of them (monthly , Daily bases),currently we are facing issue when taking backup of them ,I have below configuration :
15 daily backup to disk >disk to tape (full on last Friday + incremental for whole month)
other VM + Physical (last Friday full backup to disk >disk to tape )
As you can see on above structure , we are facing issue on last Friday (as our bandwidth was only 1 GB and the tape writer which we had was connected trough SAS cable to our VEEAM so our backup was taking place really slow .

Why you write full to tape every day?

Due to audit rules which we have , we need to remove the tapes from auto loader on daily bases (and sent to a remote site),to comply to this rule we have to either have reverse backup on our disk or create full backup on tape(virtual full backup )on daily bases.

My Issue is that I am thinking one-tape-job will not be a good Idea as I think its better to go for parallel recording while taking backup to disk jobs.
I think I will need to create a script to run on each tape job (pre-job) to make sure that tapes job do not conflict with source job and other tape jobs.

Regards,

[Adrian1980]

Hi syanor,

looks like, you work in an difficult environment with some borders.

As I understand, your Veeam resides on physical hardware beside your ESXi environment.
Did you placed a veeam proxy inside the esxi environment?

[siyanor]

As I understand, your Veeam resides on physical hardware beside your ESXi environment.

Yes that is correct.

Did you placed a veeam proxy inside the esxi environment?

No we did not, our veeam proxy will be on the same machine as Veeam.

[foggy]

In case I want to keep all daily disk to tape in one job is there any option (or script )to make sure the tape job does not run until all backup to disk are completed?

Tape job is able to wait if some of the linked backups jobs are still running.

[Adrian1980]

Yes that is correct.


No we did not, our veeam proxy will be on the same machine as Veeam.

Hi siyanor,

that means there two possible ways, your Veeam get the Backups from vSphere.
1.) It takes backup via NBD (network block device)
2.) The Veeam Server have access to production storage and takes backup directly from them.

From your descrition of ypur environment, I am pretty sure it's uses the first way, right?

I suppose , hopefully, your 10 ESXi-Hosts are all member of a cluster with shared storage.
If this is the case, I recommend you to place a Veeam Proxy into your vSphere environment.
NBD is pretty unperformant, especially on your current small link to backup server.
I am pretty sure, placing a proxy on your vSphere, boosts up your backup to disk as minimum of 2-3 times.

[siyanor]

Hi Adrian,

hat means there two possible ways, your Veeam get the Backups from vSphere.
1.) It takes backup via NBD (network block device)
2.) The Veeam Server have access to production storage and takes backup directly from them.

From your descrition of ypur environment, I am pretty sure it's uses the first way, right?

I suppose , hopefully, your 10 ESXi-Hosts are all member of a cluster with shared storage.

Yes , that's correct , veeam taking backup via NBD.

I suppose , hopefully, your 10 ESXi-Hosts are all member of a cluster with shared storage.
If this is the case, I recommend you to place a Veeam Proxy into your vSphere environment.
NBD is pretty unperformed, especially on your current small link to backup server.

We have fiber link between storage & Veeam in new environment , will having a dedicated proxy boosts up our backup ?

My issue is mainly for tape backup , as I need full backup on daily bases,is there any script which we can use to make sure a tape backup runs after disk backup and if there is any other tape job ,doesn't conflict with it.

[Adrian1980]

Hi Siyanor,

at the moment, it's not cclearly to me, how the link between Veeam and the vSphere would be.
As I understand you Upgrade that link to 10gig link?

It's good to have a fast backup storage and Tape, but if you rely on NBD for the on-disk backup with 10gig link, it's probadly not as fast as it can with this link, because NDB ist slow.
Of cource with 10gig not so slow as with 1gig, but not as fast as it can be.

The faster you get your backups from vSphere to disk, the faster you can start with backup on-tape and backup finish faster overall.
Thats not the problem on Days with incremental to disk, but on days where the fulls run, you will notice the difference.

[siyanor]

Hi Adrian,
Thank you for your continued support.

at the moment, it's not clearly to me, how the link between Veeam and the vSphere would be.
As I understand you Upgrade that link to 10gig link?

we have bought a new set of hardware (repository server + Tape library ),our Veeam server in new environment is connected to a SAN switch , we are having enterprise license which we can not use the direct storage option.
for disk bakcup I have done a few test and the speed is between 500-700 Mbps , which is better than before but it did not boost the backup speed as I have expected.
I have just implemented a proxy in our virtual environment (as you suggest) but it dropping the backup speed to below 150 Mbps ( A vm with 4 GB RAM, 1CPU)

It's good to have a fast backup storage and Tape, but if you rely on NBD for the on-disk backup with 10gig link, it's probably not as fast as it can with this link, because NDB ist slow.
Of cource with 10gig not so slow as with 1gig, but not as fast as it can be.

The faster you get your backups from vSphere to disk, the faster you can start with backup on-tape and backup finish faster overall.
That is not the problem on Days with incremental to disk, but on days where the fulls run, you will notice the difference.

do you have any other suggestion to boost our backup speed?
Regarding the disk to tape backup , we need to create full backup to tape on daily bases, is there a good practice to do it , I am thinking of below options:
1-create a disk to tape backup , right after each disk backup.
2- create a disk backup and keep all daily backup disks in it .

is there any other option? what are the disadvantages of above options?

[Adrian1980]

Hello siyanor,

500-700 is OK for a 10gig link with NBD in use.
The proxy is really to slow, you have to check
- is the proxy respect the system requirements for proxy? (4 GB of RAM and 1 vCPU is far far to low, I think no System with this specs can handle 500-700 Mbit.
You need also a 1 vCPU per task, with represents 1 VMDK at a time.
If your Veeam without this new Proxy is in default, the proxy on the veeam side does 4 tasks at a time.
Give it a minimum of 4 vCPU und give more RAM and your results are better.)
- Is the VMs vSwitch connected with the 10gig Link?

At the moment I have no more ideas, because I did not have enough information about your infrastructure, etc...

For your disk to tape backup:
1.) That's a possible way but depends on your environment.
2.) I don't understand what you mean?

Greets,
Adrian

[oscaru]

Hi Siyanor,
1 vCPU is not enough for a proxy.
Proxy compress data and assigns a vCPU per each task.
Add more vCPUs to your proxy VM (at least 4 as Adrian1980 suggests), and performance will scale.

[siyanor]

Thank you to both of you guys answer,
I have to admit something strange, I have already added more vCPU ( 4 vCPU , 6 GB memorey) it increased a bit , so I did a testing and in statistic it was showing me that its a hotadd ( confirming I was using the appliance proxy) , after that I disabled the appliance proxy, and runned the task again , to my surprise it was showing [san] , awesome! , but it was strange.is this means same as SAN direct Access? .Is the speed of 500 - 700 Mbps is fine for what veeam is seeing as san proxy?

1.) That's a possible way but depends on your environment.
2.) I don't understand what you mean?

For second option , I mean like I will add all backup to disk jobs in one disk to tape job, instead of creating a tape backup for each disk backup.
But I think this option will not be a good idea as Veeam does not recommend it for parallel recording jobs(read something similar in Veeam university).

Regards,

[Adrian1980]

OK, you deactivated the Proxy completely and the Proxy on the Veea-Server does san-mode?
Did you Veeam Server have direct connection to Produktion Storage and have access to datastore?

[siyanor]

OK, you deactivated the Proxy completely and the Proxy on the Veea-Server does san-mode?

Veeam server has a default proxy which gets installed during installation.I have disabled the hotadd proxy (which I have created), and yes it was showing me [san]!

Did you Veeam Server have direct connection to Produktion Storage and have access to datastore?

We have connected our Veeam to same SAN switch as our production storage and hosts are connected to it.

[Adrian1980]

If your Veeam can now access the datastores directly, it's pretty good for Backup speed.
Was connecting to SAN-Switch enough to give Veeam-Server direct access or have you done configuration on production storage to get the LUN visible tp Veeam-Server?

[siyanor]

If your Veeam can now access the datastores directly, it's pretty good for Backup speed.
Was connecting to SAN-Switch enough to give Veeam-Server direct access or have you done configuration on production storage to get the LUN visible tp Veeam-Server?

I have to ask our Storage administrator for that ,As he was assigned for this project but then ha had to go aboard for an emergency leave.

So does showing [san] in statistics while taking backup indicates that there is direct access between Veeam and our storage?if it is , does the speed of 500-700 Mbps is good enough for such a method?

Regards,

[Adrian1980]

Hello Siyanar,

sounds pretty good.
If the speed is good enough for your envronment, I don't know.
It's depend on your Storage, Raid, Disks and other things.
And you have to respect, that production workload have priotity at every time, if all configured properly

For your second option, one disk to tape:
You can do that, but from my experience it's a bad idea.
You don't need a single Tape Job for every disk job, you can bundle in other job-packages, but don't one in all.

Greets,
Adrian

[siyanor]

Hi Adrian,
Thanks for your reply.

I have asked our Storage administrator regarding the San switch setting and he did not do anything on San switch side.I should tell that I am really impressed how Veeam could detect the best proxy option (even though its a default installation ).
Regarding Speed , I agree with you , even though we are using a raid 5 which is not the best option for Veeam but we have sacrificed performance over storage capacity.

For tape , I will take your advise and try to bundle similar disk jobs in one tape job to reduce the quantity of tape jobs.
Anyways due to company policy I need to create full backup on daily bases ,which I think the best option will be normal tape pool as I can create a virtual full backup on daily bases.

[Adrian1980]

Hello siyanor,

for conclusion I think, the output you get from the Storage is in green area, you have to remind that main taregt for the storage is to maintain the VMware and the VMs load and backup tasks only as a secondary workload.

I think, you are on the right way.

Greets,
Adrian