Hitachi Vantara Veeam Integration

[napoleon1769]

Is there new information regarding Hitachi Vantara Veeam integration. According to Hitachi, the plug-in is already developed and should be released by Veeam.

[Andreas Neufert]

Hi Kiril,
thank you for the request. While the above is correct, we can not comment about the progress based on contracts. Hitachi can maybe comment with more details. Thanks for understanding.

[Gostev]

"Developed" is just the first stage of the process, the easy one. Following that is at least a few months of bug fix by the storage vendor for the bugs our QC team finds.

The reason we cannot comment on particular integrations is that the release time depends heavily on the original quality of the code, and also on the speed of bug fix by the storage vendor. I can tell you that one of the vendors has been in this process for over a year now - albeit because their storage has many unique quirks so we keep finding bugs in various scenarios.

[mkretzer]

Any news about this?

[Gostev]

Last I heard, our QC was "winning" and the backlog of open bugs for Hitachi to fix was only growing... come back in a few months

[Gostev]

Update: internal communication I have just received suggests there has been a good progress made in the past few weeks since my previous post.

[mkretzer]

Nice. I have good connections with them (we only purchases Hitachi in the last few years) and i have let them know again and again that their management tools are just terrible and the situation with Veeam is unacceptable. I think every other customer should do the same, they need to understand where they need to change something to get better...
If their systems would not be superior in availability they would have already lost alot of customers...

[DonZoomik]

I agree - pretty good arrays with good feature set but management software sucks. I wonder if they'll ever do a proper VASA provider - first a stateless one and secondly a one that would live inside the array.

[soncscy]

@mkretzer,

Out of curiosity, what on the Hitachi boxes is attractive to you? I've never really considered then before to be honest.

And what's the deal with their management tools?

[mkretzer]

We have one special requirement: Absolutely ZERO downtime when doing maitenance like firmware updates. Not so much for HW failures (beacause these are much more uncommon). We are doing banking transactions for customers and our developers defined that 3 seconds is a great value for all sorts of timeouts...

Most arrays require some kind of MPIO failover which take somewhere between 15 and 60 seconds. Other systems that we tested (IBM for example, who were absolutely sure they do not have any downtime for updates) use NPIV to "simulate" a WWN which moves between controllers. In our tests even that caused failover times in the range of 5-10 seconds but the "failback" (controller starts up after the update and takes the volumes back) takes between 30 and 40 seconds. Then there are systems like DataCore which we used for nearly 10 years who do some smart things with ALUA to speed up the failover. Still, you "feel" the failover.

Such things do not exist with Hitachi:
We use Hitachi now for about 10 years and surely installed about 100 different firmware versions on 15 systems or so and we did not have even a second of measurable downtime. All paths stay fully active. We had firmware updates fail in the middle of the update - but not once IO got stalled by this. These firmware updates always include Hard Drive/SSD/FMD/NVME firmware which gets auto-updated with the controller firmware (which in itself is pretty cool).

On the other hand we had HP, Fujitsu and Dell systems for which we installed firmware "every now and then" and for everyone of those we had at least one situation where we at least lost redundancy for longer time because one controller crashed (sometimes stalling IO on both) or even the whole storage system crashed!

After 20 years of IT i am paranoid about nearly every kind of update. But Hitachi firmware? Give me a firmware ISO on a friday afternoon, i'll start the update without worrying that i won't get my weekend.

I have not found one other vendor which can deliver in that area like Hitachi can.

But their storage management - where should i start? Lets concentrate on the worst things (this is not all!)
- Years ago with HUS Systems they still had a nice, HTML based management - which lacked multi-storage management (we do sync mirror/GAD for everything, yes we are german) - but at least it was kind of fast.
- Then came the G-Series Systems. Storage management was mostly done via a Flash-Based interface which still lives on to this day (they found some kind of strange workaround via Adobe Air). It is SLOW und BUGGY... The storage itself did not provide much in the area of a web interface. Also, there were other management interfaces coming out which provided multi-storage management which all were equally bad and mostly provided similar features.
- The second generation of G-Series Systems got a HTML5 interface which was promised to get all the features needed - this looked GREAT and a step in the right direction. But since the first day we saw this interface about 3-4 years ago there was NOT ONE CHANGE! It gets crazier - from day one there was no button to map volumes to servers - but if you put /servers in the URL you get a whole management interface for that - and in 3-4 years nothing about this has changed!
- With the new E series the HTML5 is still there - without anything new. Also, the flash/air interface is no longer supported and you HAVE to use the even newer off-storage HTML5 management tool. This tool brings new kind of horrors:
-> The tool in fact are 4 different tools with different install routines (they might come from different parts of the company/companies they purchased?) and LICENSES
-> These tools itself can do nothing without a agent for each storage system
-> All this is avaiable as one virtual appliance image - if you use this the setup is at least possible without days of work - until you find out THAT USING THE PREINSTALLED AGENTS WHICH ARE ABLE TO MANAGE MULTIPLE STORAGE SYSTEMS FROM THIS ONE IMAGE ARE UNSUPPORTED WHEN USED WITHOUT INSTALLING SEPERATE VIRTUAL APPLIANCES FOR THESE AGENTS?!
-> It gets better, if you already ceated mirror pairs with this management tool/agents you cannot just transfer this into a "supported"/multi-agent environment - in theory, you have to tear the whole configuration down again or live with this kind of unsupported situation.
-> Every single tool/agent has to be updated individually - but has to be compatible with each other to some degree

We absolutely love to have these systems - if they fail, they fail without much of an impact. But we HATE situations where we have to do changes on them. This leads to strange strategies of using these systems - for example if we get a new Hitachi system we pre-create 100 thin-provisioned 10 TB volumes, mirror them and pray that we never have to touch this setup again...

[soncscy]

Wow, talk about delivery!

Thank you, this is incredibly insightful!

[DonZoomik]

Yeah... this mirrors my own experiences. Arguably Hitachi's active-active sync replication (Global Active Device) is the best (and oldest) on the market and this has been a decisionmaker a few times. But software is atrocious.
As a sidenote, HPE-s top-of-the-line storage isn't 3Par or Primera or Alterra - it's XP, and XP is... Hitachi.

[TaylorB]

Can confirm the above - I love the stability and performance of Hitachi, but it's a nightmare to work on. Takes me the better part of an hour to zone and map up all the datastores to a new VM host. Can do the same on Nimble in about 2 minutes, all without leaving Vcenter. But I've never had even a blip of downtime on the Hitachi. Just this past month our Nimble controller rebooted and failed over to the other node. HPE says it's a known bug and to download firmware update. I go to download firmware and it has been recalled and I have to wait. That's not something that would ever happen on Hitachi. Even so, the horrible management is probably going to drive us away from Hitachi long term.

[Andreas Neufert]

For those of you that have VSP experience. Did you consider VMAX/PowerMAX as based on feedback of some resellers they offer as well the same capabilities in case of firmware updates/failover scenarios not loosing IO or having additional latency. I am just asking to get some additional context for those systems.

[mkretzer]

Not yet, we use Hitachi for 9 years now, back then Dell did not have this (i believe). Until now i only knew of one EMC system that should have provided that functionality. I guess this is PowerMAX nox? Still, i would only believe it if i see it myself. As stated before, IBM told me the same thing (gladly i did not belive them and tested it before purchase).

The thing is that its not *only* the firmware update feature but the entire stability of the plattform. Really - not once we were disappointed. I cannot say that about alot of other vendors (Veeam comes in at a close second).

And for two generations now we were one of the first customers putting a newly released system into production. Normally this means trouble.

And we had failures - part of controllers, fibre boards, drives of any type, backplanes - thinking about it every part in a storage system that can fail has failed in all these years in one of our hitachi systems. Its how these failures are handled by firmware AND support. We have only NBD support on all our systems because our Hitachi contact told us "if there is something critical happenening YOU WILL GET HELP" (btw. the same as with Veeam basic support). And they delivered. The only problem is the definition of "critical". They belive in their systems as much as we do so anything failing and having redundancies is never critical (and with GAD that we have for everything they are right). But the one time something critical happened (which was not caused by the storage but some special, very high IO) they helped right away without thinking twice.

Thats how you keep at least a paranoid customer...

[DonZoomik]

IMHO PowerMAX is only high-end (no real experience). Hitachi starts out in midrange so there's probably a market that PowerMAX just doesn't cover.

Hitachi does not have a perfect track record for me. I had an uncommanded system shutdown in the middle of day that was suspected to be from power button design fault that was already supposed to have been fixed. A few SSD semi-failures that have effectively killed any array performance - both sides in case of GAD. Investigation went nowhere event though they were always the same SSD model.

Still good, I have far worse experience with other large vendors.

[mkretzer]

@Gostev: Any news about this?

[Andreas Neufert]

We are making good progress and have releases soon, however only the storage vendors can comment about the actual stages as it is their Software.

[Gostev]

@mkretzer all I can say I'm pretty good with my predictions our QC is too good. While Hitachi R&D is way more responsive and fast comparing to an average vendor, we're finding many issues for them to fix. At this point they want to postpone most of the open issues to the next plug-in release so that v1 could be shipped faster, which is also something to keep in mind for future early adopters like yourself.

[mkretzer]

What kind of issues?
Any eta for v1?

[Gostev]

Issues that Hitachi does not consider critical to fix immediately. I don't know details, I just see from weekly reports that a number of newly discovered bugs is getting postponed by Hitachi for v2 of the plug-in every week.

As for ETA, see the last post from Andreas. Veeam's involvement here is limited to testing their code drops and logging the bugs we find.

[foggy]

There will be a bunch of storage configuration-related restrictions. Everything will be documented by Hitachi in detail.

[Gostev]

It will be an interesting experiment indeed. We've never seen so many known limitations postponed to the next plug-in release by other vendors, not even close. On the other hand, we usually trust the vendor's judgement here, as presumably they know better from telemetry what is important for their customers and what is not.

[DonZoomik]

Now that the plugin has been out for a while - any experiences from users? I don't have any sites with Ent+ licensing and Hitachi combination unfortunately but according to release notes, it looks a bit raw...

Shame that only RAID850 series (G350+) and up is supported and RAID800 series (G200+) got left behind although they're quite similar internally.

[foggy]

We haven't heard of any escalations except for the GAD configurations so far. But it is just a couple of months since the release, I'd give Ent customers more time.

[mkretzer]

@foggy - as i understand it "modern" GAD (with VSM) is not supported at all?

[Andreas Neufert]

Hi all,
we support specific Software Version and later, while the vendors are responsible for the compatibility related to specific storage systems. Maybe reach out to Hitachi to check if they can enable G200+ support as well if the latest software can be run there?

The issue with the GAD volume is that the plug-in will not forward to Veeam both sides of the active-active volume. Hitachi is working on it.

[mkretzer]

G200 is Panama 1, this is a whole different generation as Panama 2 (G350 and up).

About GAD: why is it necessary to present both sides? Snapshots are always single-side even with GAD...

[foggy]

@foggy - as i understand it "modern" GAD (with VSM) is not supported at all?

GAD source volumes are returned by the plug-in and can be used for restores. GAD target volumes are not currently returned.

[pmorrissey]

Is there new information regarding Hitachi Vantara Veeam integration. According to Hitachi, the plug-in is already developed and should be released by Veeam.

Released July 2021: https://www.veeam.com/blog/hitachi-vant ... lugin.html