What size is the largest vm you backup ?

[iroche]

Just a quick Q out of interest more than anything . The biggest vm I am backing up is our corp file server coming in at just over 7TB spread across 5 VMDKS (breakdown below). Whats the biggest virtual machine you are backing up and have you ever tried to restore it??? I haven't tried the instant restore of this virtual machine as of yet. I keep 8 restore points on disk and run a incremental with a Synthetic job running every Saturday which then kicks off a tape backup of the full vbk. The backup with all the restore points consumes 5.2TB of space on the backup server.

System VMDK - 60GB
Disk 1 VMDK - 1.9TB
Disk 2 VMDK - 1.9TB
Disk 3 VMDK - 1.9TB
Disk 4 VMDK - 1.9TB
Disk 5 VMDK - 1.1TB

Cheers
Ian

[mvoinchet]

Hi iroche,

We are in the process of virtualizing our central project file server which has approx. 8TB of project data on it that is connected via a RDM and our storage is on Dell Equallogic arrays.. I apologize if this question is a bit off topic, but how did you accomplish the spanning between your vmdk files? Did you use Windows Disk spanning? Our plan is to migrate the project data to another equallogic array that hosts the vmdk files and are spanned in Windows to achieve the neccessary drive space to accomodate the data.

Overall we are pleased with Veeam as we currently backup our Exchange 2010 infrastructure (Only about 600GB) with it as well as many other smaller servers.

Thanks,
Mike

[iroche]

I created one LUN per VMDK . So as above I have 5 LUNS and each one I created a VMFS volume on each of these live each VMDK which are mapped to the file server.


Hope this helps.
Ian

[Patje]

Hi iroche,

We are in the process of virtualizing our central project file server which has approx. 8TB of project data on it that is connected via a RDM and our storage is on Dell Equallogic arrays.. I apologize if this question is a bit off topic, but how did you accomplish the spanning between your vmdk files? Did you use Windows Disk spanning? Our plan is to migrate the project data to another equallogic array that hosts the vmdk files and are spanned in Windows to achieve the neccessary drive space to accomodate the data.

Overall we are pleased with Veeam as we currently backup our Exchange 2010 infrastructure (Only about 600GB) with it as well as many other smaller servers.

Thanks,
Mike

Mike, I would go with Ian's setup and just use different drive letters instead of creating one big disk with disk spanning.
If I recall correctly, you need to create dynamic disks in order to stripe them, and dynamic disks and equallogic don't really get along.
Suppose it should work, and your volume gets corrupt for some reason, you'll be restoring quite a lot of data, no matter how good Veeam is.
So different disks, and use DFS for instance to present a unified share.

Hope it helps
Patrick

[mvoinchet]

Thanks Ian and Patrick for the prompt replies,

Our main project file server has a central drive letter mapped for everyone so creating separate drive letters for each VMFS (VMDK) volume would not work for us. I will do some further research regarding DFS however. Our technology vendor made the windows disk spanning feature seem like it wasn't a big deal to do, but I have to admit that I've seen several warnings about it in other places. We have run a data usage assessment in regards to archiving the older data that isn't being touched to reduce the size, but we still have move than 2TB of active project data, so we will still be exceeding the vmdk limts.

Thanks again guys.

Mike

[Reimold]

System VMDK - 60GB
Disk 1 VMDK - 1.9TB
Disk 2 VMDK - 1.9TB
Disk 3 VMDK - 1.9TB
Disk 4 VMDK - 1.9TB
Disk 5 VMDK - 1.1TB

Hi Ian,

i have nearly the same sized file server that i backup with veeam v6. I run the job in "reversed incremental" mode and just want to know how long does your job run when it creates a "synthetic full" ?

br

Dirk

[iroche]

It takes around 8 hours for the Synthetic full to complete, I then have a scripted BE job that runs and send the VBK to tape ( 10 LTO4 tapes in total).

[VirtualPlaya]

It takes around 8 hours for the Synthetic full to complete, I then have a scripted BE job that runs and send the VBK to tape ( 10 LTO4 tapes in total).

Only 8 hours for a full synthentic of around 9 TB???? Wow!!!

[Sebi!]

That´s really fast. So does someone know what the main points are for a fast synthetic full? When i check my stats i think its alot the CPU performance.

The duration at my system for a file server increased very much.

1.2TB -> 9h
1.6TB -> 21h
2.1TB -> 47h

And now i wanna do something that i get a "normal" time back.

Thanks

[foggy]

Target storage IOPS are much more important. Essentially, synthetic full is nothing but I/O load on the storage (as it produces random reads/writes versus sequential writes in case of full backup).

[Gostev]

Only 8 hours for a full synthentic of around 9 TB???? Wow!!!

That would require about 300MB/s throughput. Fast storage there, for sure!

[NightBird]

Humm... 300MB/S throughtput on random io patern !!! (as synthetic full do heavy random read/writes), I'm doubtful.

[Gostev]

Actually, synthetic full is about 90% sequential reads (aside of changed blocks from incremental backup files), so there is a lot of read ahead which most SANs are optimized for. And there are 100% sequential writes into the new full. That is, aside of randomness from the fact that both reads and writes are going in parallel.

It is transform option which adds heavy random I/O to synthetic fulls.

[vcharlie]

On the topic of Instant Recovery, we have a customer in Asia that tested Instant VM Recovery on a VM that was around 7TB in size. The VM came back in under 3 minutes. Nice!

[Gostev]

VM size does not affect its boot speed, of course. However, VM size does affect the speed of "normal" VM restore that you would have to do with other backup solution. It would probably take a few days with 7TB of data

[Sebi!]

Target storage IOPS are much more important. Essentially, synthetic full is nothing but I/O load on the storage (as it produces random reads/writes versus sequential writes in case of full backup).

Thanks for the reply. Now I checked my system and it´s not the processor but also not my storage. Right now I only have a 32bit Windows 2003 and it´s my RAM.
So I think i have to install an 64bit System.

[slam]

[merged]

To be more accurate, I should say VM to big that Veeam can backup but fail to restore.

We have a Windows 2008 R2 file server running on ESXi5 with eight luns of 2TB each, total 16TB of file storage. The server currently has over 8TB of data and it's growing fast. We use Veeam to backup the server, full backup every month and incremental every day. Ever since the .vbk file grew beyond 7TB in size, we started running into problem doing any kind of restore. Both file level restore and instant recovery failed. Even though the backup job is still running successfully every day. Of course we didn't notice the problem until we need to restore files from your backup.

We have a ticket open with Veeam support for over two months now and we are still running in circle with no real solution. They even sent us a custom build Veeam agent file but that didn't fix the problem. We have a Windows 2008 R2 server with 8 CPUs and 8GB RAM running the backup and repository server and backup proxy. A Windows 2008 R2 server with 8 CPU and 12GB RAM with iSCSI attached as the repository server.

Is anyone out there using Veeam to backup AND restore VM with this much data successfully?

[dellock6]

My personal record was a Zimbra Server (CentOS 5 based) with 9 vmdk for a total of 2 Tb, powered on with Instant Recovery and restored in production via Storage vMotion.

[slam]

Our vbk file is 8.4TB. Backup is still running but we cannot restore anything. Neither Instant Recovery or file level restore works.

[slam]

I've just got off the phone with tier-2 support on our case. I was told that Veeam does not recommend backing up VM's barger than 5TB. I wish they'd have told us that when we evaluate the product.

[jrickman]

8TB here, with a change rate of around 100GB over 24 hours. Full backups and snapshot removal are painful. Wish there was more fine grained job scheduling to control when a snapshot removal takes place for long running jobs like this.

[bobbyk18]

10TB allocated
8TB of it used

We separate it into two jobs:
one job for 6TB of it with a 4.3TB VBK run nightly
one job for 4TB of it with a 2.5TB VBK run once a month(archived data that doesn't change)
running ESXi 5 and Veeam 6 with an Equallogic PS6000.

We're now working on replicating it to a second site using Veeam. Our VRBs are around 45GB per day.

Backups are pretty good, replicas are a pain.

[sergeadam]

I've just got off the phone with tier-2 support on our case. I was told that Veeam does not recommend backing up VM's barger than 5TB. I wish they'd have told us that when we evaluate the product.

Then I'm in trouble because one of my file servers is at 8TB and growing.

I have it setup with Windows doing spanning of a bunch of 1.9TB VMDKs, all on the same LUN on an Eq PS4000 with no issue. The backups do seem to work.

[dellock6]

Can someone from Veeam confirm this? And most of all, if is only a "best practice" or a proper limit (I'm suspecting the former...)

[Vitaliy S.]

Hi Guys,

Not sure where this recommendation comes from, but I guess there was a misunderstanding between Stephen and our support engineer, I believe our support engineer might be talking about these 5 TBs (pretty old thread).

Thanks!

[Gostev]

Yes, most likely the engineer simply misunderstood my very old comment that Vitaly had referenced. I know for the fact they pick up lots of knowledge from these forums anyway, this 2 year old comment was made in v4 times, so it is not really applicable with all the engine improvements made since then, including new backup modes and especially new v6 architecture.

[slam]

Sergeadam, have you tested restoring any file using file level restore or instant recovery? I am also curious as how file level restore with Windows spanning volume.

As a best practice, we should test restoring the backups regularly. Especially for those of you who have .vbk file size over 7TB to make sure it works. I'd like to hear if other people have similar problem.

I finally tested restoring one of the eight 1.9TB vmdk file successfully last night. Though it took 8 hours to complete. It confirms that the backup works. Now only if Veeam can make the file level restore and Instant Recovery work also.

Gostev/Vitaliy, my support case is ID#5189322. If you can take a look, I'd greatly appreciate.

[slam]

I got the confirmation from the Veeam development team that we hit the architectural limitations of the VeeamAgent, which is still 32-bit. They are going to make a custom VeeamAgent that will allow us to specify the VD to mount for File Level Restore.

The problem here is not with backing up but restoring very large VMs >7TB using File Level Restore and Instant Recovery. For those of you who are using Veeam to backup large VMs, > 7TB, I'd suggest you guys to test if File Level Restore and Instant Recovery are working or not.

Gostev, Is that any plan to upgrade the VeeamAgent to 64-bit?

[Gostev]

Frankly, we did not even consider doing before your post above. Somehow, no one else had run into this limitation in the past 5 years, and we just did not see any reason to have 64 bit agent. Now, I need to discuss this with the devs and get their opinion on the effort required. Of course, we have no choice but to do this.

[tsightler]

I've pretty much always suggested that clients with servers this large to split the backup into multiple jobs, excluding specific disks to keep the total size down, but yes, more and more clients have servers that are this large or larger so it would be good to break this limit going forward.