Feature request : CBT support for non-windows SMB 3.0 shares

[b.lagace]

Hi all,

I was not sure to start the topic here or in Hyper-V specific section
Apologies if I made the wrong choice.

Change block tracking, the most beautifull invention imho that's required to backup large environments.
Unfortunatly it's only supported on SMB 3.0 shares presented by Window hosts.

But I can imagine that a some users have a Synology, Qnap, etc...NAS that is used to store the VM's.
Others entered a new area with real converged solutions like Nutanix, Simplivity, etc...

I myself have created a cluster with Nutanix nodes.
Nutanix uses local storage which is presented via SMB 3.0 to the Hyper-V nodes.
Backup and so work fine and specially parallel processing is awesome on this scale out solution.
The only backside of this is the lack of CBT support.
This implies that every night I need to do a full scan, which results in reading the FULL share!

It has such an impact, that I might be obliged to make major choices between two products I love so much.

As my cluster will grow fastly towards 12TB and I'm able to process currently around 1TB/hour, i'll never be able to backup my environment within the backup window.

I've already received words that I would be hard to do, as CBT driver has some limitations.
But somewhere i'm convinced that putting together the force of Veeam and Nutanix development team, a solution could be found.
Who knows, other non-windows SMB 3.0 providers are looking along and willing to help, cause afterall who wants to enter the market without
beeing able to be compatible with the n°1 backup software!

Do you find yourself in same situation or just agree that such effort would help the community, then please "like" or comment this thread to catch the developpers attention.
All input, including technical deep dive stuff, is welcome.
I can always forward it myself to the people I know within Nutanix.

Regards,

Bjorn Lagace
System Engineer
Ter Beke

[davy.neirynck]

Hi Bjorn,

I couldn't agree more, we are more and more looking into these types of converged solutions, it's only normal that we need to count on this to work.

Regards,
Davy Neirynck
Sr. IT Consultant
Xylos

[Gostev]

Hi all. Thank you for your feedback!

Back when we were designing our Hyper-V changed block tracking, we have been choosing between the two architectures, one of which allowed CBT on generic SMB shares. We had prototypes built for both, and found that one to be very unreliable in real world during situations like connection drops or I/O queue overfilling. And because reliability and data integrity of the overall solution was paramount, we have chosen not to implement that architecture, knowingly sacrificing support for CBT on generic SMB shares.

Unfortunately, with "legal" approaches available to ISVs today, there is simply no reliable way to track changes when it comes to generic SMB shares. Reliable tracking requires that we put the filter driver on the actual storage box, which most NAS don't support (except when NAS is backed by Windows Server, scenario in which we do support CBT today).

Now, the hypervisor itself would, in fact, be able to do the change tracking reliably, so I hope one day we will see this capability added to Hyper-V. Until than, there is little we can do except implementing storage-specific integrations, which in turn is not so good idea from business perspective for us (at least when it comes to newcomers with small market share).

Thanks!

[b.lagace]

Hi Gostev,

Thx for your response.
I received information that Nutanix has enough 'information' to deliver towards Veeam in order to get it working.
Problem is, I dont know which person I need to let talk to which person in order to make it happen.
My guess is partially 'politics' and the partnerships Veeam chooses.

I continue to work on all channels in my reach, cause reading out x tb's every night is just impossible.

Regards,
Bjorn

[b.lagace]

Guess nobody uses Hyper-V and non-windows SMB 3.0.
Makes me start to think i'd better move to VmWare ...

[Primetrack]

Some update on this?
Unfortunately we get this warning on our Nutanix SMBv3 share as well!
Kind a sad that there is no coorporation!

[Vitaliy S.]

Anton's response above is still up-to-date, so no changes in terms of vendor's specific CBT support.

[b.lagace]

I hope they did told you soo when you bought the box.
Cause you'll need to approach your backup in a scaled out thinking way...

Nutanix and Veeam wrote a best practices, but i'm not 100% agreeing with it .

I allow my backup to handle multiple vm's per repository but then limit the number of vm's / host that can be handled.

This way i'm sure it's taking a backup of vm's that are spreaded over hosts (read storage with Nutanix) and get some good speeds.

Are you running Hyper-V on the Nutanix or VmWare ?
I run Hyper-v, but advice VmWare.

drop me an email : bjorn.lagace@terbeke.com with your config on both nutanix/veeam/repository environment and what you wish to achieve.
I have almost a year experience now with debugging and optimising this

[b.lagace]

I'm searching for information about Windows Server 2015 (Server 10).
I hearded rumours that Microsoft would embed its own change block tracking (like VmWare).

This could provide a solution for non-windows SMB 3.0 shares.

Has Veeam been testing already with the technical preview?

Regards,
Bjorn

[Vitaliy S.]

Hi Bjorn,

Cannot comment on the rumors, but as a TAP partner, yes, we have early access to the bits of Microsoft Windows Server.

Thanks!

[Gostev]

We are also under NDA, so cannot really talk about any futures until Microsoft announces them officially.

[b.lagace]

No problem guys, I totally understand and respect that.

Afterall we need to have something in live to look forward to

[b.lagace]

For those interested, i'm referring to this information :

http://www.aidanfinn.com/?p=17505

[b.lagace]

From Microsoft :

One of our main goals going into this feature was to eliminate the need for backup vendors like Veeam to implement change block tracking filter drivers. As such we are working closely with a number of the backup vendors (including Veeam) to adopt our new model and thus not require a filter driver or any code to be installed on a file server.

I would encourage you to share your scenario with your backup vendor of choice (aka Veeam) as well – that will help make sure they prioritize these scenarios in order to support customers like yourself.


In short Microsoft will be ready for it in their next Windows Version.
Question is : Will Veeam adapt to the new method or continue to use their own developped filter driver ?

[Vitaliy S.]

Yes, Veeam has always been known as a vendor that fully supported all new features of Hyper-V server.

[b.lagace]

Something we always like to hear !

[dledoux]

Add another to the Nutanix camp. Really wish Veeam supported CBT on SMB 3.0 from Nutanix.

[b.lagace]

Hi,

Drop me an email!
Always good to have a fellow Nutanix user to talk things through.

bjorn.lagace@terbeke.be

Bjorn

[Primetrack]

Hi everyone.

Another wish for this feature from me

Happy datacenter with happy Nutanix people

Peter

[zaf]

Anything new to report here? Would love to see CBT on Nutanix

[Vitaliy S.]

No updates to this topic, answers on the previous page are still up-to-date.

[b.lagace]

An alternative is to coöperate with Nutanix and have the ability to read out their snapshots and export them to a library.

There's another backup vendor that has been working closely together with Nutanix in order to achieve this.

Kind Regards,
Bjorn.lagace@terbeke.Com

[Vitaliy S.]

I believe with Windows Server 2016 release support for non-windows SMB shares will not be needed?

[b.lagace]

According to my non-official information that's correct.
Microsoft will have its own CBT mechanisme like VmWare.

RCT -> Resilience Change Tracking I believe.

So basicly, if you wait long enough...problems get fixed ...

[zaf]

I hope they did told you soo when you bought the box.
Cause you'll need to approach your backup in a scaled out thinking way...

Nutanix and Veeam wrote a best practices, but i'm not 100% agreeing with it .

I allow my backup to handle multiple vm's per repository but then limit the number of vm's / host that can be handled.

This way i'm sure it's taking a backup of vm's that are spreaded over hosts (read storage with Nutanix) and get some good speeds.

Are you running Hyper-V on the Nutanix or VmWare ?
I run Hyper-v, but advice VmWare.

drop me an email : bjorn.lagace@terbeke.com with your config on both nutanix/veeam/repository environment and what you wish to achieve.
I have almost a year experience now with debugging and optimising this

How are you getting your jobs to limit the number of VMs per Nutanix node so that you can get enough streams going? I too am trying to squeeze everything into my backup windows since the lack of CBT has us reading all of the data nightly.

[b.lagace]

I agree with you as well.
The best practice they wrote (which I've seen before it was released) is not the best imho.

I have a readout peak of 1GB/s with an average processing rate of 250MB/s.
Backup proxy : on-host -> else you'll pump your network down !!!
Backup mode incremental so at end of backup it merges the files to purge out those outside retention date.

I currently only need to backup 5,5TB and it takes already 9 hours (3 hours merging on my EVA4400 repository with 36 FC disk @ 15k rpm)

So if you're planning to have more than 8TB data on a Nutanix you have to go for VmWare and Veeam.
With Hyper-V you'll have to wait for Server 2016 hopefully or have another backup method ...like snapshotting on the backup with retentions.
Alternativly (sorry Veeam) you can check with CommVault, as they can read out snapshot and export them incrementally...not tested myself.

[b.lagace]

PS : Dont forget :

Volume snapshots : Allow processing of multiple VM's with a single volume snapshot
Repository : Concurrent task : 4 or more
I/O control : enable parallel processing

If you handle 4 VM's at the time and those VM's have mulitple vhd's(for example 2), you'll be reading out 8 vhd's at average 50MB/s.
Nutanix limits the throughput per VHD, something a salesperson won't mention and support will describe as a protection future.

[b.lagace]

And completly offtopic and pure out of frustration : YES I have the satadom wearout (field advisory 38) ...It was discovered on my cluster

[zaf]

And completly offtopic and pure out of frustration : YES I have the satadom wearout (field advisory 38) ...It was discovered on my cluster

Here too. 200% wear. Fun times

[Dreadnought]

I think this is a similar/same issue your seeing on the Nutanix.

We run a number of Hyper-V clusters (have done for years), with iSCSI based storage currently one of our bigger clusters is 16 nodes running around 500VM's. Weve just finished putting in Veeam, all working ok no issues until we paused one of our nodes for maintenance purposes and all of sudden have warnings on all the backup jobs showing that CBT is no longer being used, backup times have gone from minutes to hours for customer VM's.

Unpaused the node so its back in production, CBT is now being used again. Bit of a ball ache when you have 500 VM's to backup/backup copy/replicate etc

The work around we have found that works is to evict the node from the cluster, perform any required maintenance than add the node back into the cluster.

This may well have been sorted with Server 2016 but Server 2016 still has to many major bugs in hyper-v for us to use as a production Hyper Visor at the moment, also a huge amount of work to upgrade all our clusters to 2016.

Something we will have to live with for now by the looks of things.