Best Practices/Advice Sought

[buck]

I have a small VMWare/Veem setup, basically the Essentials Package for both. I have 3 Dell 2950 hosts, and use some local storage for VM's, but mostly a Buffalo NAS for VM storage. I have a second Buffalo NAS device that Veeam uses to store backups. I have about 20 Virtual Machines, including a vCenter Server, Veeam Server, Domain Controller and SQL Server, along with various other VM's.

Until recently, I was happy with the setup. I haven't had issues with speed or anything else across the network. Then, while working on upgrading to ESXi version 5 and replacing an older PE2850 with a PE2950, I ran into a snag. The NAS device that houses my VM's died, and nearly became a catastrophic loss. I didn't have fresh backups because of the hardware upgrades, etc., so I was going to be in real trouble if I couldn't get the NAS back to life. It looks like a drive became corrupted, but it took 2 1/2 days of desperation and multiple attempts to get the NAS device to come back to life. As we speak, I am recovering data from the device, and it looks like I will be OK once I get the data pulled off the one unit onto other storage.

Meanwhile, I have rebuilt all the 2950s' local storage with RAID 10 instead of RAID 5, and the NAS devices will now be RAID 10 as well. Additionally, a new NAS will be added to replicate the one that is my primary storage unit, so if the primary unit fails, I have a secondary unit ready to go in no time.

Now that you have the background, here's the question. The older PE2850 has been wiped clean and is now running a new install of Windows Server 2008 R2 instead of ESXi. So, what's the best use for that server? I could install the Veeam software there and allow it to backup all the VMWare infrastructure from a physical machine rather than a virtual machine, so that if I DID suffer a datastore failure again, at least the Veeam info and backups would be readily accessible for restoration. (Since the Veeam Server was a VM before, it was going to take some work to restore THAT VM so I could begin to restore other VM's.) The backups will continue to go to the second NAS device, which will likely have some sort of redundancy added to it. I could also install vCenterServer on the PE2850 server, since it's such a pain in the butt to do VMWare upgrades to the host running vCenterServer, since the host has to be in Maintenance Mode, etc. and that makes running vCenter impossible. I also noticed that in version 5, the Host Update Utility is gone, which means it will be even harder to update that host when the time comes.

Can the PE2850 server run both? It's it advisable to run the Veeam software alongside the vCenterServer software? If so, that might be the best choice of all. That way, I think I would be covered in the case of any failure within the setup, except for a failure of the 2850! I might add some sort of external drive to that server and let Windows back itself up to that external drive all the time as well.

Right now I'm a bit paranoid, so I'm hoping to set my own mind at ease with this new setup.

[dellock6]

Hi, I will try to answer your several questions and doubts:
- vCenter is best kept in virtual since it can benefit from virtualization too. What if the physical server dies? It is also true that with the Essentials license, vCenter il slightly more than a ESXi aggregator. But on the other hand you can use Veeam to save it.
- if you have vCenter you could and should install the Update Manager to upgrade all the infrastructure, Host Update was meant for single hosts.
- Yes you can use the server to run Veeam Backup on it, and extract VMs from the VMware cluster and save them.
- Yes there is no issue in running Veeam and vCenter in the same server, also if they are physical you will also skip some problems of backing up it, since Veeam cannot save a physical machine

[rbrambley]

Buck,

Luca gave you some great advice, but I wanted to toss you an alternate idea.

Why not use the PE2850 as another stand alone hypervisor. ESXi or Hyper-V, Veeam supports both now. This allows you to:

• Run vCenter, Veeam, and other management vms isolated from each other
  gets vCenter off the host you want to put in maintenance mode
  have a 2nd host for vPower jobs like Instant Recovery, Surebackup, U-Air - BTW, this would have let you run your VMs for the 2.5 days you were rebuilding your NAS then migrate them from vPower back to production once ready. Users would have been connected the whole time
  ability to use Veeam replication of critical VMs to PE2850 for backup copy

You have some small details to work out:

• licensing - Veeam can't work with free ESXi, but we support every flavor of Hyper-V
  replication - have to go from ESX to ESX
  storage - run some VMs on PE2850
  ram and cpu - I assume you have plenty since the PE2850 used to be your prod host?
  etc

Build an extra VM and run our new Veeam ONE to Monitor and Manage your environment too. Download the 30 day trial
http://www.veeam.com/vmware-esx-managem ... ution.html
It defaults to a 7 day / 24 hour free version after 30 days.

[davidb1234]

We used to have our vcenter virtualized. Until you realize how awful of a situation you can be in any number of situations. Now I will never virtualize vcenter again. VMWARE support will strongly recommend you do not as well and I think it is good advice even though it can be done.

[rbrambley]

David,

I agree that virtual vCenter running in the environment it is managing leads to a lot of problems. Putting VC on a host outside of the cluster solves a lot of these issues and still allows you to take advantage of the convenience of virtualization. I can't argue with a physical vCenter though.

[buck]

Buck,

Luca gave you some great advice, but I wanted to toss you an alternate idea.

You have some small details to work out:

• licensing - Veeam can't work with free ESXi, but we support every flavor of Hyper-V
  replication - have to go from ESX to ESX
  storage - run some VMs on PE2850
  ram and cpu - I assume you have plenty since the PE2850 used to be your prod host?
  etc

How does the licensing come into play here? Do you mean Veeam can't run on the new ESXi host (assumed to be a free version of ESXi), or that it wouldn't be able to backup any of the VM's on that host?

Also, can you explain a little more about what you mean about the Replication going from ESX to ESX? I'm not sure what you mean there either.

One of the biggest drawbacks to using the PE2850 for vCenter that I can see is that the PE2850 can't be upgraded to ESXi version 5.0. That's the reason I ended up replacing it with another 2950, all of which got wiped clean & have fresh installs of ESXi ver 5.0, with the datastores formatted as VMFS5. I upgraded my licenses for the hosts & vSphere Essentials to version 5, so I'd like to keep all the hosts running the same version.

[rbrambley]

Veeam runs on windows. So, Veeam can run in a windows vm on free esxi. Veeam cannot back up any VMs running on free esxi, however. You need a minimum license for esxi for Veeam to run jobs on its guests.

Veeam can run on hyperv Vms and run jobs no matter what.

If you own Veeam you own our replication features too. Its included. Check the user guide, veeam.com, or search for our end user university to see click through demos.

[dellock6]

We used to have our vcenter virtualized. Until you realize how awful of a situation you can be in any number of situations. Now I will never virtualize vcenter again. VMWARE support will strongly recommend you do not as well and I think it is good advice even though it can be done.

David, do you have any doc about this statement? Because since vSphere 4.0 VMware suggests to place vCenter in a virtual machine. VMware was used to suggest running vCenter in physical in ESX 3.0 and 3.5 (and virtual vCenter was supported anyway). I know it's not Veeam related but I prefer to not let a thread with doubtful information open, for people coming here in the future...

[bgbGsy]

I to am interested by the comments not to virtualize the VCenter Server. Everything I've seen from VMware seens to recommend this. Even if the physical host with the vcenter guest goes down, HA can move it to another server; I have actually had this happen.

Backing up the VM can be a bit fiddley as you need to backup directly from the host and not through the vCenter. You need to edit the job if you vmotion the vcenter.

Last Sunday a utility power outage took out one of my hosts running the vCenter. Initially I though it had suffered file corruption becuase Vcenter wouldn't start. It actually turned out that my MS SQL trial license had run out, so when the vm rebooted SQL wasnt kicking off. (6 months trial period and I had forgotten to apply the product key)

So using Veeam 6 I instant restored the backed up vcenter VM to another host in a matter for a couple of minutes. I quickly realized it wasn't file corruption becuase of the same sypmtoms. Anyway, applied the key to SQL and VCenter booted right up.

So while I didn't actually need to keep the restored vcentre VM (applied the key to the live vm), it was really useful to diagnose the problem so quickly. With a physical 2008 Host running VCenter I wouldn't have necessarily been able to do this.

If you need to cold-migrate the vcenter, again this needs a bit of forward planning. But VMWare have notes for this. I have tested the process and found it worked as described. With standard vSwithces this is really straight forward, but if you are using vDS, then perhaps it is more complaciated.

Also, remeber that VMware themselves now supply a virtual appliance for vCenter if you don't want to use WIndows. As far as I know this is supported for production envirments (albeit with a smaller number of hosts)

So, what particular scenarios do you consider that a virtualized vCenter is more vunerable?

[rbrambley]

I'm not sure you are going to find an official document where VMware recommends running vCenter as either virtual or physical. I'll give you 2 current KB articles where they specifically tell you they support both, but they don't make a recommendation one way or another.

Installing vCenter Server 5 Best Practices
http://kb.vmware.com/selfservice/micros ... Id=2003790

Running vCenter in a Virtual Machine
http://kb.vmware.com/selfservice/docume ... alID=10087

Yes, there are many VMware engineers, vExperts, VCPs, VCDXs, etc that will tell you they always reccomend it. Some even will swear VMware told them to do so. Fact is it is a supported config only. BTW, I can understand where VMware Support, who has to help remind people all day long that vCenter is not going to start until their virtualized AD and SQL server is up and running, would probably recommend a physical box often! But for EVERY scenario and EVERY admin?

In my experience, virtualized vCenter is convenient and saves you hardware, headache, and time. It also has it's share of gotchas. Buck started this thread by explaining one. He had to put his host in maintenance mode and vCenter was a guest of that host. I'll sum up a lot of the challenges by saying it is not exactly ideal running the management server in the same environment it is managing.

I do not want to speak for Buck or anyone else, but I do not think this s a discussion about vCenter's vulnerability. The topic of "should I virtualize vCenter" is a logisitics and operational one. I brought it back up because Buck is a Veeam customer, and he can not take advantage of the convenience of backing up or replicating vCenter if he decides to go physical. I did not intend and I am in no way suggesting vCenter should always be virtual for everyone. I am also not advocating vCenter should be physical all the time. It's a choice we all can make ourselves after considering the pros and cons.

Buck has a server to repurpose now, and why not make it a hypervisor again so he can run more systems on the hardware platform. Continue to use Veeam to back up or replicate those guests. By keeping this second hypervisor outside of his production ESX Cluster he can avoid many of the gotchas others choose to avoid by using a physical vCenter.

[buck]

Thanks for the advice and discussions. I am going to move my vCenterServer VM and my Veeam B&R Server VM to a host running outside of my vSphere environment. That way, they will able to help me recover in the case of any failure like this in the future.

I do have a couple of points to add for clarification/follow-up in case someone comes across this thread later:
While the PowerEdge 2850 had the RAM & CPU's necessary (as it has been one of my 3 hosts for a couple of years now) it can't support 64-bit VM's (processors aren't VT-capable) so I will replace the 2850 with another 2950. I did get ESXi 5.0 to run on the 2850, even with a PERC 4e/Di card, but since my 2 VM's are both 64-bit, it didn't matter.

Also, just to be clear, I own the Essentials Package for both VMWare & Veeam, so some options (vMotion, HA, etc.) are not available to me. That has made it somewhat difficult to manage Host updates on the Host running the vCenter VM. In ESXi 5.0, the Host Update Utility is gone, so it will be even harder to do updates. By removing the vCenter VM from the vSphere cluster, I eliminate that problem, and bring an extra layer of protection to my install.

Thanks again for the help, I am much more comfortable with my setup now!

[rbrambley]

I own the Essentials Package for both VMWare & Veeam, so some options (vMotion, HA, etc.) are not available to me

Have you checked out the Migration button on the top of the v6 GUI?

[buck]

Not yet, I'm still running version 5. Once I get everything set up, I will install v6. Guess I have something to look forward to ?!

[Gostev]

Very much so

[buck]

I'm about to the last stages of my new setup, but have another question. My current Veeam B&R SQL databases are on my main SQL Server VM, which is a part of my vCenter infrastructure. To separate everything appropriately, should I move those databases from the SQL server to a local install on the Veeam Server's VM using SQL Express? My assumption is that if I didn't, and I lost vCenter access and my VM's, I would be unable to recover because the databases would be on a VM that's lost.

To move them, can I simply detach them from the main SQL Server, move the files, and then attach them on the Veeam Server VM? There must be a place somewhere in the Veeam B&R program to point it at the new DB location.

[Vitaliy S.]

You're right, you should also protect you Veeam B&R database.

On top of that, please be aware that we provide a standalone backup extraction tool (search this forum for extract.exe).This tool will allow you to restore VM files from VBK without having Veeam B&R console available.

To move them, can I simply detach them from the main SQL Server, move the files, and then attach them on the Veeam Server VM? There must be a place somewhere in the Veeam B&R program to point it at the new DB location.

Here you go an existing topic that covers this procedure in great details: Move Veeam database only to another sql server

[buck]

Thanks again for all the advice & help, I'm slowly getting there!

I now have a PE2950 running the free ESXi 5.0 Hypervisor. my vCenterServer VM & Veeam Server VM live on that host now, separate from my overall vSphere installation. 3 other hosts make up the vSphere Essentials infrastructure, and are all updated & running fine.

I have tested a couple of Backup jobs, and they seem to be working OK, I need to do a lot of tweaking to jobs & setups based on the type of VM's, etc., but that will come in time. Meanwhile, I've decided to try to set up a Replication Job to replicate a couple of my critical VM's to the stand-alone ESXi host as suggested. Unfortunately I'm not having any luck. Veeam B&R has my vSphere hosts all set up in it (I selected the vCenter Server IP and it found all the hosts & datastores associated with that when I set it up initially), but what I can't seem to get is the stand-alone Hypervisor to show up as a target for the Replication job. When I get to the "Select Host" section of the Destination dialog box, my only options are the vCenter Server & its 3 hosts. I go back to the main window and try to add the Stand-alone Hypervisor as a host, but I get the message saying it already exists. I don't see it anywhere listed, so I'm not sure where B&R thinks it is, but I don't seem to have access to it anywhere. I can't get past this point to set up the Replication job.

[Vitaliy S.]

Have you tried adding this host to Veeam B&R console before? Was this host managed by the existing vCenter Server? Try adding this host by IP address, should work.

[buck]

I've tried adding it that way (by IP address), but it says it already exists.

The IP address USED to be in the vCenter cluster, but no longer is. It's not shown as a part of the vCenter Server in the Veeam window or through the vSphere client, so I know it's not a part of the cluster anymore. Maybe Veeam thinks it is since it once was? If so, I have no idea how to make Veeam forget the original connection.

[Vitaliy S.]

Yes, looks like this might be the case. Please contact our support team to help you to remove this record from Veeam B&R configuration database.

[buck]

OK, I've opened ticket #5192597.

Thanks for the help.

[buck]

I was running into other issues as well with the older Host IP being listed in other places, so I uninstalled everything, deleted Registry keys & DB files, and started over. Now I can add the stand-alone Host properly, so that seems to have taken care of that problem.

Now, based on earlier advice, I decided to try a Replication Job, to replicate critical VM's to the stand-alone host. The test job failed, with the following errors:
"Preparing replica VM Error: Parameter count mismatch. Error: Parameter count mismatch." I found an old post about that error, which seems to indicate that a Replication Job will fail when trying to replicate to a stand-alone host running the free version of ESXi. Is that in fact the case? Do I have any other options, considering I need to stay within VMWare for my infrastructure?

[Gostev]

Could be - free ESXi is not supported as it has some APIs purposely disabled by VMware.