Hi Clint, First of all let me say that I wholeheartedly appreciate the decision you announced earlier, I think it is well balanced and thought through. So what follows is just for the sake of discussion. How do 99.99% of software vendors support their general public? By publishing software updates in the form of releases. Not only Oracle. E.g. Monty Program publishes MariaDB 5.2.5, 5.2.6, ... 5.2.10. And they don't publish something like MariaDB 5.2.5.10 with cherry-picked security fixes. Apparently they believe that 5.2.10 is in any respect superior to 5.2.5 and should be used instead. Now I won't be surprised if Monty Program does cherry-picked custom builds for paying customers, but it is an exception made on individual basis, not a general support strategy. Now suppose (for the sake of argument) you switch from MySQL to MariaDB 5.2.10 in 12.04 LTS. When MariaDB 5.2.11 is released, you won't just upgrade to 5.2.11, you plan to cherry-pick the changes, because now you can, right? Implying that somehow you know better than the vendor what changes the user should be subjected to. By dismissing the rest of 5.2.11 changes you effectively dismiss Monty Program support and development efforts, imply doubt about their care for end-user and interfere in the vendor-user relationship, as in "we know better what's good for the user", while being neither the vendor nor the user. I know that you have nothing like this in mind, but that's how it seems to look. I understand the value in cherry-picking, but if we look at it once again, what we see? You want a linux distribution to act as a support contact for a very complex 3rd-party software. This alone is a tough call. In that you plan to adopt a strategy which software vendors use only in exceptional cases, for paying customers. You want to adopt this strategy for a _general_ user, without, actually, any fallbacks, as I understand. So getting back to concrete situation and numbers (10.04 LTS, MySQL 5.1.41): - what is the effort of cherry-picking the updates for 5.1.41 compared to just using the latest upstream release? - what fraction of users will be negatively affected by switching from 5.1.41 to 5.1.61? (I know there _may_ be some, but what _fraction_?) - what fraction of users won't notice the switch? - what fraction of users will be positively affected by the switch? - what fraction of users won't care because they were forced to use Oracle binary tarballs (or 3rd party deb builds) because Ubuntu failed to provide them with critical (for them) updates due to the policy of including only minimal changes? So in the end it turns out to be: a lot of extra work to protect insignificantly low fraction of non-paying (since you don't cherry-pick for money, you do it as a general policy) users, at the expense of everybody else (any effort spent on 0.01% of users is at the expense of the remaining 99.9%). And this is the idea of end-user support you're advocating.
This gets complicated though, because there are components of the distro that rely on libmysqlclient, and the client programs. These all start to get ugly when your new upstream version introduces incompatible syntax into /etc/mysql/my.cnf, or your shiny upstream version uses /tmp/mysql.sock while old stodgy client programs use /var/run/mysql.sock.
Once again, I understand the goals of cherry-picking, but the economy of this process is totally broken. It looks like a tremendous waste of time and effort. So, perhaps, instead of wasting time on it, you could focus on alleviating the problems mentioned above by improving on the packaging and dependency tracking mechanisms? E.g. by making it easy to rollback and stay on a fixed MySQL version for users who experience issues with new upstream release? Sincerely, Alex On 2012-02-17 21:33, Clint Byrum wrote:
Excerpts from Alex Yurchenko's message of Fri Feb 17 05:57:03 -0800 2012:
Finally! A post that makes all wrong points!
Hi Alex,
Am 16.02.2012 19:33, schrieb Alex Esterkin:
As an end user, I would most strongly dislike this. You clearly don't understand how corporate users think and operate, how they work with open source technologies, and how they plan and evolve their technical roadmaps.
I think I understand a bit of how corporate users think and operate.
When you are an enterprise user who has subscribed support from MySQL via Oracle you are enforced to use the Oracle binaries and cannot just use the distribution supplied binaries at all. This includes bugfixes and security fixes from your vendor, in
On 2012-02-17 11:53, Björn Boschman wrote: this
case Oracle (not Debian or any other distribution).
When you do not have such a subscription you rely on the support from your distribution.
Heaven, no! Otherwise I'd still be using MySQL 5.0.something on CentOS 5. And nobody wants that!
Indeed, there are times to diverge from the distro, and times not to.
This gets complicated though, because there are components of the distro that rely on libmysqlclient, and the client programs. These all start to get ugly when your new upstream version introduces incompatible syntax into /etc/mysql/my.cnf, or your shiny upstream version uses /tmp/mysql.sock while old stodgy client programs use /var/run/mysql.sock.
These aren't hard problems to solve, but they're annoying things that users have to deal with when they diverge from the distro.
Of course, I'm relying on Oracle to provide the latest security and bug fixes. Whoever wants/needs to run MySQL 5.5 has long given up on "their" distribution. And, really, why/how should I rely on somebody else but the vendor to fix his program?
Projects like Debian and Ubuntu give users full control over their software. While you may be in a good position to test and verify that a large changeset from MySQL's latest version doesn't break your app, many are not. By only introducing tiny, finite changes, the distros have provided a very stable (as in change, not uptime) version of MySQL that can be relied upon not to change (for better or worse) for the life of the release.
Your statement that anybody who needs mysql will just use the upstream mysql could use some data to support it. I don't have data that I can share on either use case, but I can say that I have encountered users who are grateful that they can trust 'apt-get upgrade' not to break their app.
That's the point this whole discussion is about. Neighter Debian nor Ubuntu can offer reliable bugfixes and security support.
Indeed, because, in general, they are not qualified to, and are not supposed to be. Their main task should be packaging and integration of software to allow painless installation and use. Key word here is 'painless'. And with respect to MySQL Linux distributions don't do that great of a job, because they are focusing on something they should not. And while Oracle fills in for RPM-based distros, Debian/Ubuntu users are on their own.
Perhaps distros serve another purpose, which is to give users a single point of contact for the support of all of the software on their system.
Debian and Ubuntu play this role, and it is in fact the more important role than convenience of installation. For a user who has time to find the upstream bug tracker, report bugs, and extract patches, sure, they do not need the distro version. In a previous life, I would not have used the distro mysql because it was a core competency. But I used the distro postgresql, because I didn't know anything about postgresql.
For a user who has a peripheral need for MySQL, the distro version makes a lot of sense. Perhaps they want to run Bacula and expand its capabilities beyond what sqlite gives them. They need an avenue for resolving their problem. The distribution serves this purpose by getting them in contact with Bacula and MySQL interested persons. For Ubuntu, they also have a clear path to commercial support by paying Canonical to resolve this problem for them.
So, for Debian and Ubuntu, being able to only introduce finite change post-release means that you can know whether or not you are faced with a problem that you created, or one that was already present in the upstream code.
Not because they don't want to.
And they better don't.
I may not understand much about corporate users thinking, but I do understand a thing or two about software development. And I can tell you that
1) it takes enormous effort to fix bugs in a 3rd party software and is a big waste of resources compared to the vendor doing it. Backporting bugfixes may lead to side effects and require extensive testing. Why duplicate efforts? Why don't you just let that 3rd party figure it out? After all it is between Oracle and MySQL users.
Indeed, I have wondered for a while if we are doing the wrong thing by focusing on the idea of only introducing finite change into the system. It has worked for a long time, but if its not possible, there are other ways to manage the effects of change.
Perhaps we should, instead, focus on automated regression testing and just accept the pain up front, adding regression tests as we go along.
2) it is _generally_ impossible to fix bugs without changing software behaviour and breaking compatibility, simply because many bugs define software behaviour and others may require fundamental architectural and/or protocol changes. So it is totally naive to hope to stay on ancient version, but keep it up to date at the same time. I really appreciate stability of CentOS as a development environment, but staying on the same MySQL version for 5 years is patently insane.
Is the phrase "If it isn't broken, don't fix it" insane?
3) the whole goal is generally pointless - say, I'm running Ubuntu 10.10 LTS, it has MySQL 5.1.41. If Ubuntu backports ALL fixes to it, it would simply become 5.1.61. So why not just upgrade to 5.1.61 and be done with it?
Since the release of Ubuntu 10.04 with MySQL 5.1.41, we have backported a few of the most critical fixes from MySQL. We would never backport all of them. We have a firm policy of only backporting things that lack simple workarounds or have a massive effect on the usability of a package.
Because of the opaque nature of the recent security announcements though, it is likely that we will end up just shipping 5.1.61 into lucid though. Its not something we really want to do, as it will challenge our ability to support the users, but we don't have much choice.
Their hands are bound because MySQL/Oracle somehow is not willing to provide important information such as detailed changelogs or security information.
Making it nice and easy for linux distributions takes effort too. And that means resources. And who's paying for that? Apparently Oracle sees little value there, and I can see why: making sure that 5.1.41 gets all fixes for 5 years instead of just upgrading to a newer version - nobody wants that! What's the point of releasing 5.1.42, etc, then?
Its not a resource issue. Oracle has a policy and they have their reasons for sticking to it, but as far as I know, none of those reasons are resource based.
This leads us to the following options: * Stay with MySQL but no security nor bugfixes * Search for an alternative which is even 100% compatible with MySQL + having full community support
Or, stop playing security police and focus on what distributions are supposed to do in the first place: package whatever is released and make it easy for the user to install and use?
See above. I do believe that distros have a much larger role to play than convenience.
From my personal as well as my business perspective I want a system where I can get bugfixes as well as security fixes. You should consider those questions in your roadmap as well.
And that's why, as it goes now, I'll be using RPM-based Linux distribution to run MySQL because there I can at least rely on Oracle to do the job. And that's quite ironic as I develop on Ubuntu.
You can run Oracle's static binary tarball releases on Ubuntu if you don't like the packaged versions.
-- Alexey Yurchenko, Codership Oy, www.codership.com Skype: alexey.yurchenko, Phone: +358-400-516-011