Volume size over 2TB

[egroeg]

Hi all.

I need to create a 3TB volume to a Win2008 R2 Virtual Machine.

It cannot be an RDM or direct iSCSI as I need it to be backed up with Veeam.

VMWare have a native ceiling cap of 2TB for volume sizes (I've extented my datastore to be 7TB, but it's the VOLUME level we're talking about here guys).

Does anyone have an recommendations for manners I can create a +2TB VMFS datastore?

Storage will be on an iSCSI EMC

[Gostev]

Hi, I suppose you could use spanned volume in Windows that is based on two 1.5TB VMDKs? The only gotcha is that file level restores will not work for this. Thanks.

[TaylorB]

Create two LUNs smaller than 2TB each. Initialize and format the first one as a VMFS datastore within Vmware. Then go into the properties for that datastore and click to "increase size". This will allow you to add what is called an extent, which extends the space across another LUN.

It's basically the VMware equivalent of the spanned disk, but your Windows VM won't know it isn't a single disk.

[Gostev]

He already did that, please read the first post carefully...

[rfn]

[merged]

Hi,

I'm going to migrate an Apple Xserve based file server onto a new VM with Windows Server 2008 R2. There's around 9 TB of data and that will keep growing at a steady pace.

How can I do this best so that I'm able to back up the VM with Veeam Backup? We're using vSphere 5 and VBR 6. The SAN is a HDS HUS 110 (new model) that we will receive in about a month.

I can't find a way to create a really big LUN (fx. 10 TB) and present it to the VM in a way that Veeam can backup. RDM's can be big now (with VMFS5) but vRDM's are still limited to 2 TB. LUN's connected with iSCSI inside the VM are also a no go.

I could create several 1 or 2 TB LUN's with big VMDK's and divide the data on several shares, but we would really like to present it as one big share.

Regards,
René Frej Nielsen

[dellock6]

Windows 2008 has mount points as linux has, you can map different vmdk as folder of the same base drive letter. Do not know however how Veeam will recognize this, but the underlying vmdk are there and are ntfs-formatted. Worth a try?

[Choodee]

Rene

I'm in the same boat as you (discontinue xServe with 9TB of data to a windows backend). These are the different scenario's I played with.

Hardware
x1 Poweredge R510 /w 12 900GB SAS drives configured as RAID5 (Production vSphere Server)
x1 Poweredge 2950 /w 6 2TB SATA drives configured as RAID5 (Veeam Replication Target)
no shared storage

Scenario 1
- x1 9TB LUN
- x4 2TB VMDK
- Windows 2008 VM

Setup: Setup all vmdks as separate drive in windows and create multiple shares. Present the shares as 1 logical hierarchy using Microsoft DFS.
Pro:
- Works well in windows environments.
- Don't have to worry about windows soft disk spanning/striping
- Full support from VEEAM including Instant FLR
Con:
- Inefficient use of space. eg some drives will have 50% free space while other would have 20%. This will prove problematic in the future as certain folders require more space.
- OSX doesn't like DFS very much. I get around it using by using an application such as ExtremeZIP but certain versions of OSX ie Lion poses issues with microsoft DFS.
- Had to breakup large folders to multiple subfolders creating folder sprawls and broken workflows.

Scenario 2
- x1 9TB LUN
- x4 2TB VMDK (Setup as a single 8TB windows spanned disk)
- Windows 2008 VM

Setup: Setup windows disk spanning and configure the x4 2TB drives as a single 8TB disk (Important: Use disk spanning and not disk striping - Spanning will allow you to add more disks in the future to expand the logical drives).
Pro:
- Single large datastore for storing all data.
- Preserve existing folder structure.
- Share a single root folder
- No need for DFS
- Works well for Mac and Windows environment
Con:
- Disk Spanning. A lot of people will lose sleep over dynamic disks and software raid but with a proper backup/replication the plan the risks are lower.
- Dynamic Disks. Veeam does not support Instant FLR from spanned/stripped dynamic disks.

In the end I chose Scenario 2 (this is after I spent two weeks on Scenario 1... live). The benefits outweighed the potential to lose the spanned disk due to a failed VMDK. However, I made sure that Veeam replication was working on my backup Poweredge 2950 and that I can failover easily if necessary. With Veeam replication, I can get fire up the replica in less than 10 minutes and be up and running in 15minutes. Furthermore, I am able to replicate only the delta changes from my backup replica to my live production after recovery (which is necessary since a full replication takes 30 hours on 8TB of data).

I've been running this setup for a month now without any problems. My MAC's and PC's don't have any issues accessing their data and without any noticeable speed issues. In the future, I will be able to scale up easily by adding more 2TB VMDK disks in the spanned volume. I'm planning the second part of this DR with offsite tape backup storage but so far i'm very happy with my decision.

PS I tested the resiliency of my spanned disk by taking one of the vmdk's offline. As expected, upon bootup Windows gave an error that the disk was broken which prevented any access to the disk. However, once I added the missing disk and restarted the server the logical spanned disk came back online without any issues. I think the probability of losing a single vmdk due to corruption is actually lower since I don't have the complexity of iSCSI/shared storage. My VMDK's are all local running on VMFS datastores.

My 2 cents
Sandee

[Cokovic]

You could setup a SureBackup job or use Other OS FLR to access the dynamic disk from your backup.

[Choodee]

You could setup a SureBackup job or use Other OS FLR to access the dynamic disk from your backup.

I've tried using Other OS FLR to restore a file in my spanned volume and it doesn't work. Veeam will mount the drives as individual vmdk's but won't span the disks to create the volume. The only option that works is to power up the replica and copy the data manually while the VM is running. This obviously has it's own issues with duplicate IP's and being in the same network but there are ways around that.

Sandee

[signal_lost]

Honestly I don't trust over 2TB to NTFS as chkdsks take FOREVER, and NTFS gets unhappy with the millions of tiny files that generally go with that type of workload.
Honestly if your talking that kind of workload and already buying a HUS, you should talk to your HDS partner about getting a HNAS to front he HUS to handle your file services. It can handle external tiering, backup, replication, NDMP and all the other goodies you need for keeping your file's protected.

If your using FC on the HUS you could always do pRDM's and use something with NDMP to move the data off to a VTL.

(Disclaimer I work for a HDS partner, and am glad the NDA is over for the HUS).

[Cokovic]

Hmm ok. And what about a SureBackup Job? There you should be able to access the spanned volume.

[rfn]

Honestly I don't trust over 2TB to NTFS as chkdsks take FOREVER, and NTFS gets unhappy with the millions of tiny files that generally go with that type of workload.
Honestly if your talking that kind of workload and already buying a HUS, you should talk to your HDS partner about getting a HNAS to front he HUS to handle your file services. It can handle external tiering, backup, replication, NDMP and all the other goodies you need for keeping your file's protected.

If your using FC on the HUS you could always do pRDM's and use something with NDMP to move the data off to a VTL.

(Disclaimer I work for a HDS partner, and am glad the NDA is over for the HUS).

Since we need to support our Mac users then we need to do the filesharing from Windows Server with ExtremeZ-IP (AppleShare server), so we're unable to use a NAS frontend. I'm also stretching our budget already by buying a new SAN and new servers so there's no money for a NAS box.

We're going to use iSCSI since we already do that for our EqualLogic SAN and buying FC equipment is just to expensive.

So if we're not going to use a NAS do you then suggest that we create many disks and share them one by one? That would be a disk per letter in the alphabet, and even when using thin provisioning it will take up a lot of space over time because of blocks that have been written once. I now that the HUS is supposed to do space reclamation but I don't know how effective it is.

[rfn]

Rene

I'm in the same boat as you (discontinue xServe with 9TB of data to a windows backend). These are the different scenario's I played with.

...

PS I tested the resiliency of my spanned disk by taking one of the vmdk's offline. As expected, upon bootup Windows gave an error that the disk was broken which prevented any access to the disk. However, once I added the missing disk and restarted the server the logical spanned disk came back online without any issues. I think the probability of losing a single vmdk due to corruption is actually lower since I don't have the complexity of iSCSI/shared storage. My VMDK's are all local running on VMFS datastores.

Hmm... I've been advised against this like you and would really like to avoid doing something that other wize guys is telling me not to... Our volume can easily grow to 15-20 TB in a couple of years so it's important that we're not down for a day if Windows needs to check the filesystem for some reason.

I'm almost considering doing a physical RDM and running another kind of backup solution on that server or even to create a physical server with a big LUN on the SAN. It's not what I really want but it might be the best solution?

[umenz]

Hi, I suppose you could use spanned volume in Windows that is based on two 1.5TB VMDKs? The only gotcha is that file level restores will not work for this. Thanks.

Is this right today ? - how we can provide 30 TB Volumes in ESXi 5.1 without splitting in 2 TB Drives and than span in Windows ?

Filelevel Restores are mandatory for me

[Gostev]

Today, you should simply upgrade to ESXi 5.5 that support 64TB VMDK.

[dellock6]

62 to be correct. But still plenty of space to create a 30TB volume.

[averylarry]

Isn't there still something you can't do in Veeam with a > 2Tb virtual disk? Like a snapshot or maybe it's hotadd . . .

[foggy]

Everything that is allowed by vSphere, is possible.

[averylarry]

Huh. Wonder when the 1.98Tb limitation for hotadd went away.

[tsightler]

With ESXi 5.5 when VMware finally broke the 2TB VMDK limit. Prior to that it was impossible to snapshot a VMDK that was larger than approximately 1.98TB. It wasn't really a hotadd limit (there were other limits around hotadd and block sizes mainly with older VMFS versions).

[Jahara]

Upgraded to VMFS5 but veeam is still seeing the 2TB cap on his VMDK which shouldn't be the case as it's no longer of VMFS3
but the vmdk has inheritied this limitation in the new file system (VMFS5)

[Vitaliy S.]

Are you able to create a disk larger than 2 TB manually via vSphere Client?