NOTE: Although this post references experiences I have had in work, the opinions expressed here are mine and mine alone.

If you follow me on Twitter you may have noticed my anti-RHEL (RedHat Enterprise Linux) outbursts today. I could keep twittering to try make my point, but sometimes 140 characters is just not enough, so I figured I’d blog about it instead and then tweet out the link to the blog post when I’m done.

In work we run two kinds of Linux servers, RedHat Enterprise Linux, and CentOS. We pay for RedHat, we don’t pay for CentOS (because it’s free). CentOS is based off the RedHat code base, but has some of the fancy stuff stripped out. Clearly, you would expect RHEL to give you the better experience since it has more features and you pay for support. Unfortunately, in my experience that’s just not how things are shaping up. CentOS has been completely problem and stress free (as well as financially free), while RHEL has not been such a smooth ride. Sure, most of the time it works just fine, but it definitely generates more stress for me than CentOS does, and that’s paid-for stress!

In theory you would expect THE biggest difference between the two to be the support. Unfortunately, for reasons I’ll explain in a moment, I don’t actually have any direct experiences of RedHat’s customer support. Before I arrived on the scene at work, people had simply given up on RedHat support. As I understand it, because we pay for the cheap version, we don’t get phone support, just email support (though I could be wrong about this). From what I can gather, some calls had been logged when the switch to RedHat was first made, but since these calls didn’t resolve the problems, people stopped opening tickets, choosing to google problems instead. When I ran into my first RHEL problem in work and asked one of the more senior people for the RedHat support details, I was told to just google the problem, apparently that would be quicker and better. Hence, all I can say is that in the past people I know well and trust became so despondent with low-end RHEL support that it’s simply not used anymore in our place. However, I have heard from other people that their high-end support is very good, but since we don’t have it I really can’t comment.

Rather annoyingly the story is quite similar with The RedHat Network, or RHN. This is a suite of tools designed to take all the pain out of admining a large number of servers. From what I can see, when you go for low-end RedHat support, you’re really paying for the RHN software tools. I have to say the theory seems sound. You get a web interface that allows you to see all your servers, what patches are available for them, and even an interface to push those patches out to the servers. However, just like the email support, we don’t use it anymore because apparently it wasn’t reliable. I’ve been told that patches would be pushed but never actually arrive, so we update our servers by SSHing into each one periodically and triggering an update from the command-line. On the newer versions of RHEL this is actually a nice experience because they use yum, it’s a much less pleasant experience on the older versions of RHEL that use up2date instead. Anyway, what I know for sure is that the RHN web interface was tried in work, but that it’s not used anymore.

So, we don’t use the email support because apparently we lost faith in it, and we don’t use RHN to automatically push updates for the same reason, but when we manually trigger updates, they do come from RHN, and this is where my personal experiences start to come in. Downloading updates from RHN is often shockingly slow. We have many gigabits of bandwidth into work (no exaggeration), and our servers get priority access to that juicy bandwidth, yet updates often trickle in. It’s infuriating to sit there watching updates leak in at home internet speeds when you have access to a blazingly fast educational internet connection!

Now I want to get on to what set me off today, server entitlements. The way you buy support for your servers is by paying for server entitlements. Each machine you update or manage through the RHN must have an entitlement or the updates simply won’t be applied. This makes sense for enhancement and bugfix updates, but I’m really not happy that it also applies for security updates. I see that as being bad internet citizenship. It just invites exploits which could infect countless people who interact with RHEL servers. Others may disagree with me on this, but I just don’t see it as acceptable to use security flaws in your OS as a stick to beat customers with.

However, what’s really annoying about entitlements is how you pay for them now. In the past you bought entitlements in bulk, and activated them as you needed them. An entitlement is not good for ever, it’s basically paid for per year, so it made sense that the clock on them would only start ticking when you started using them. This meant it was easy and sensible to have a stack of them in reserve. Buying in bulk meant less paperwork and less hassle. For some insane reason RedHat decided to change this, at least for us. Now the clock on entitlements starts ticking from the moment we BUY them, not from the moment we start USING them. This means it no-longer makes sense to order them in bulk and keep a stash in reserve. Instead it makes sense to buy in small quantities, hence lots more paperwork and effort. This also increases the chances of servers ending up without entitlements for short periods of time, and that makes the inability to apply security updates to un-entitled servers all the worse in my opinion.

Note: For clarity and fairness I do need to point out that I work for an educational institution, and that RedHat give us a cheaper price because of that. It’s possible that the way entitlements are now working for us is a side-effect of our educational pricing. Even so, it didn’t used to be this way, and this way is really annoying!

I hate paperwork, I hate purchase orders, and I hate hassle. And to me, that’s what RHEL is. Hassle. By contrast, I love managing our CentOS servers. No RHN slowdowns, no entitlements, no purchase orders, no hassle!

So, you’re probably wondering, why would anyone buy low-end RHEL support? If you really need support surely it would make sense to go for the high-end support that is, apparently, very good. And if you don’t need support, surely you should just use CentOS! That does make sense, except for one flaw, software vendor support matrixes! RHEL is very well supported, CentOS, not so much. It seems to me that it’s these support matrixes that to keep people on bargain-basement RHEL support levels, despite the appeal of CentOS.

I do also think that I need to point out that in the future the stress/benefit balance of RHEL may flip back. Right now we are not using fancy features RedHat clustering or the RedHat Global File system, but we expect to be rolling those out soon. If they work out well and live up to their promise, that gain may well completely out-balance the stress generated by RHN and the entitlements system.

Finally, A person claiming to be from RedHat support contacted me over twitter after reading my tweets. Assuming it really was a person from RedHat, that was great to see! And, to be honest, I think it would be perfectly reasonable for that person to say that if we want great support we should pay for great support, and not just take the cheapest option. However, I don’t think the way entitlements are managed (at least for us), the slowdowns in downloading updates from RHN, or the block on security updates for un-entitled systems, are good enough for any paid service, even the bargain-basement version!