Colin Charles writes:

I was speaking to Otto (CEO, MariaDB Foundation) and discussing that we should move the lists from Launchpad to something else. We

The most important is that people can subscribe to the list without having to create an account. This is the problem with Launchpad. However, the problem with the mailing list is not where it is hosted. It is that the majority of development happens off-list! The developer mailing list is the core of a healthy open-source project that treats developers equally. It is where everyone, newbie or core developer, can first learn how the project runs, and later follow what is going on. But not so in MariaDB. So many people from SkySQL casually divide the world into first-class SkySQL employees, who are included in discussions. And second-class others, who are casually kept out, as a matter of course. Just look at the below random example from a closed mailing list open only to employees. Stuff like this happens all the time, on closed mailing lists, closed IRC channels, and latest I think a company-internal chat tool called "Slack". Basically, MariaDB is turning into a proprietary project to build a business for a struggling venture-funded startup. It just happens to be bound by the original GPL license to release source of the main product, though even that is desperately attempted being circumvented by stuff like "MariaDB enterprise". So Colin, and others: Why bother working to move the mailing list elsewhere? Spend first your efforts to coerce everyone to actually use the mailing list, and to remove non-public communication channels. Start with yourself. Cc all mails you send to the public mailing list. Stop using non-puclic IRC or other channels, and ask people to use the public channels if they want to contact you. Thanks, - Kristian. ======================================================================= From: Rasmus Johansson Subject: [All] Current and last 10.1.8 sprint To: "all@mariadb.org" Date: Wed, 30 Sep 2015 11:17:21 +0300 (2 weeks, 6 days, 21 hours ago) Hi all, I've now populated sprint 10.1.8-4 with the issues that we agreed upon yesterday on the maria call or over IRC after that. The sprint can be found here: https://mariadb.atlassian.net/secure/RapidBoard.jspa?rapidView=27 There are a few things missing still which I hope we can add today: - Bar will add some issues from the list of debian patches - Holyfoot might have one issue missing. I had documented that MDEV-7817 should be added to Holyfoot, but I think I got that number wrong. HF, could you give me the correct number? PLEASE NOTICE that this now is the last sprint of 10.1.8 after which the plan is to release it as the first stable GA release of 10.1 as discussed on the call. The release will happen after next week's Maria call. Please scream and shout if you think there are some issues in addition to the ones on the sprint that you think still must be fixed before going GA. All, please check that the sprint now includes what we discussed yesterday and that it does look realistic for yourself. BR, Rasmus ======================================================================= From: Elena Stepanova Subject: Re: [All] Current and last 10.1.8 sprint To: all@lists.askmonty.org Date: Thu, 1 Oct 2015 18:43:52 +0300 (2 weeks, 5 days, 13 hours ago) Hi Rasmus, Serg, all, On 30.09.2015 11:17, Rasmus Johansson wrote:

Hi all,

I've now populated sprint 10.1.8-4 with the issues that we agreed upon yesterday on the maria call or over IRC after that. The sprint can be found here:

https://mariadb.atlassian.net/secure/RapidBoard.jspa?rapidView=27

There are a few things missing still which I hope we can add today: - Bar will add some issues from the list of debian patches - Holyfoot might have one issue missing. I had documented that MDEV-7817 should be added to Holyfoot, but I think I got that number wrong. HF, could you give me the correct number?

PLEASE NOTICE that this now is the last sprint of 10.1.8 after which the plan is to release it as the first stable GA release of 10.1 as discussed on the call. The release will happen after next week's Maria call. Please scream and shout if you think there are some issues in addition to the ones on the sprint that you think still must be fixed before going GA.

/me screaming #1: https://mariadb.atlassian.net/browse/MDEV-8813 I don't see how we can go GA with binlog encryption (even disabled by default), but without any ability to use the logs for backup/SST/investigation/what-not. And it's a new feature, which we technically cannot add after GA. Regards, /E (to be continued) ======================================================================= From: Colin Charles Subject: Re: [All] Current and last 10.1.8 sprint To: Elena Stepanova Cc: all@lists.askmonty.org Date: Fri, 2 Oct 2015 00:00:15 +0800 (2 weeks, 5 days, 13 hours ago) Hi Elena, all,

On 1 Oct 2015, at 23:43, Elena Stepanova wrote:

...

PLEASE NOTICE that this now is the last sprint of 10.1.8 after which the plan is to release it as the first stable GA release of 10.1 as discussed on the call. The release will happen after next week's Maria call. Please scream and shout if you think there are some issues in addition to the ones on the sprint that you think still must be fixed before going GA.

/me screaming #1: https://mariadb.atlassian.net/browse/MDEV-8813

I don't see how we can go GA with binlog encryption (even disabled by default), but without any ability to use the logs for backup/SST/investigation/what-not. And it's a new feature, which we technically cannot add after GA.

mysqlbinlog is a client utility. It already has “differences”, i.e. no streaming backup, needing GTID support (https://mariadb.atlassian.net/browse/MDEV-4989), etc. I don’t see how this can hold back our GA, so that we can include this tool to work with things post-GA (maybe even for 10.2) But of course, I agree we should aim for everything to be “usable”. I don’t have a good solution to this problem, and I don’t see a time estimate for adding support to this for MDEV-8813. I also know that we definitely have to release 10.1 as GA *before* 5.7 is GA, so presumably all this happens before our company/dev meeting. Maybe we can just add this as a warning in the release notes and also when someone is turning on encryption? cheers, -colin ======================================================================= From: Elena Stepanova Subject: Re: [All] Current and last 10.1.8 sprint To: Colin Charles Cc: all@lists.askmonty.org Date: Thu, 1 Oct 2015 20:01:14 +0300 (2 weeks, 5 days, 12 hours ago) Hi Colin, all,

mysqlbinlog is a client utility. It already has “differences”, i.e. no streaming backup, needing GTID support (https://mariadb.atlassian.net/browse/MDEV-4989), etc. I don’t see how this can hold back our GA, so that we can include this tool to work with things post-GA (maybe even for 10.2)

But of course, I agree we should aim for everything to be “usable”. I don’t have a good solution to this problem, and I don’t see a time estimate for adding support to this for MDEV-8813. I also know that we definitely have to release 10.1 as GA *before* 5.7 is GA, so presumably all this happens before our company/dev meeting.

Maybe we can just add this as a warning in the release notes and also when someone is turning on encryption?

Okay, I understand -- at the end, business needs are above everything else (no sarcasm). Usability and quality are first to give way. But then, this call for "issues that must be fixed" is a pointless exercise -- there is nothing we can realistically add to the sprint without postponing the release, so lets just agree right now that 10.1 goes GA next week no matter what, and add a warning "if you use this server, you will get problems" (sarcasm... kind of). And please actually *remember* this decision, so that there is no "how come nobody said there were problems, we would have of course fixed everything before releasing" complaints later. I clearly remember them after our previous business-need-driven releases, and it was not fun. In fact, given that Daniel probably leaves for the meeting next Thursday (?), and knowing our buildbot and other infrastructural instability, we should aim to release the tree which we will have on Tuesday, the very latest. Which basically means it should be frozen on Saturday, to let it build fully, and leave Monday for fixing whatever gets broken by Friday pushes. Among other things, it most likely means *no* to systemd -- unless we are unbelievably lucky and the first systemd-enabled builds in buildbot (which will hopefully happen tonight) show great success, there will be no time to fix anything. Regards, /E ======================================================================= From: Colin Charles Subject: Re: [All] Current and last 10.1.8 sprint To: Elena Stepanova Cc: all@lists.askmonty.org Date: Fri, 2 Oct 2015 01:32:32 +0800 (2 weeks, 5 days, 11 hours ago) Hi Elena, all,

Okay, I understand -- at the end, business needs are above everything else (no sarcasm). Usability and quality are first to give way.

But then, this call for "issues that must be fixed" is a pointless exercise -- there is nothing we can realistically add to the sprint without postponing the release, so lets just agree right now that 10.1 goes GA next week no matter what, and add a warning "if you use this server, you will get problems" (sarcasm... kind of).

Overall, very good arguments and I cannot really claim to deviate from them. Rasmus might want to chime in, based on pressures from the products team?

And please actually *remember* this decision, so that there is no "how come nobody said there were problems, we would have of course fixed everything before releasing" complaints later. I clearly remember them after our previous business-need-driven releases, and it was not fun.

I wonder, what we do in situations like this. I mean there’s also the MariaDB Foundation whom are supposed to decide on things like this, but in theory we’re all captains… Do we query support @ MariaDB Corporation to figure out if this is something acceptable for them?

In fact, given that Daniel probably leaves for the meeting next Thursday (?), and knowing our buildbot and other infrastructural instability, we should aim to release the tree which we will have on Tuesday, the very latest. Which basically means it should be frozen on Saturday, to let it build fully, and leave Monday for fixing whatever gets broken by Friday pushes.

Among other things, it most likely means *no* to systemd -- unless we are unbelievably lucky and the first systemd-enabled builds in buildbot (which will hopefully happen tonight) show great success, there will be no time to fix anything.

Again more good points that I can’t argue with. Original email says, "The release will happen after next week's Maria call. Please scream and shout if you think there are some issues in addition to the ones on the sprint that you think still must be fixed before going GA.” You bring up a good point that nothing realistically gets fixed from now till then. So while I’m not Rasmus, maybe we do have to “delay” the release till we’re at the company meeting. Again, I have no idea what can get done from now till… whenever we freeze Frankly, as long as we release before 19 Oct, I think we’ll be before MySQL 5.7. However I think we need to also plan this as MariaDB Corporation will want to do marketing around this (and we now have new marketing lead(s)). I think the GA release is 5.7.9, seeing http://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-9.html, but also remember http://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-10.html is out, which means they’ve pretty much baked GA and are doing “internal smoke testing”, while ensuring that bug fixes go into 5.7.10. So to me, their GA is “Ready”, and I can only imagine when they’ll drop it to the world ======================================================================= From: Rasmus Johansson Subject: Re: [All] Current and last 10.1.8 sprint To: Colin Charles Cc: all@lists.askmonty.org Date: Thu, 1 Oct 2015 23:31:31 +0300 (2 weeks, 5 days, 9 hours ago) 1. (*) text/plain ( ) text/html Hi, I suggest we have a conversation about the critical items like systemd, encrypted binlogs and the timeline of the release tomorrow. I'll initiate a discussion on IRC and set up a call if needed. ======================================================================= From: Elena Stepanova Subject: Re: [All] Current and last 10.1.8 sprint To: Colin Charles Cc: all@lists.askmonty.org Date: Fri, 2 Oct 2015 03:30:07 +0300 (2 weeks, 5 days, 5 hours ago) Hi Colin, all, On 01.10.2015 20:32, Colin Charles wrote:

<skip>

Overall, very good arguments and I cannot really claim to deviate from them. Rasmus might want to chime in, based on pressures from the products team?

...

And please actually *remember* this decision, so that there is no "how come nobody said there were problems, we would have of course fixed everything before releasing" complaints later. I clearly remember them after our previous business-need-driven releases, and it was not fun.

I wonder, what we do in situations like this. I mean there’s also the MariaDB Foundation whom are supposed to decide on things like this, but in theory we’re all captains… Do we query support @ MariaDB Corporation to figure out if this is something acceptable for them?

It's easy to predict the feedback from support, as they are the ones who have to deal with customers' problems in GA. I suppose that's why they were kept out of the loop on previous occasions, at least they what they claimed. To make sure that at least members of this list are well-informed, I collected some quick statistics off the top of our JIRA. It's not very accurate, but it gives a good enough approximation. *5.5* (bugs said to affect 5.5 and possibly higher versions) ~100 bugs of Major Priority or higher; out of them, ~27 crashes. *10.0* (bugs said to affect 10.0 and possibly 10.1) ~ 178 bugs of Major Priority or higher; out of them, ~36 crashes. *10.1* (bugs said to affect 10.1 only) ~50 bugs of Major Priority or higher; out of them, ~14 crashes. Don't get excited yet. Most of 5.5 bugs affect 10.0 and 10.1. Probably nearly all 10.0 bugs affect 10.1. Remember, all of this is open statistics, everyone who wants can do the same search in JIRA. It does not evaluate the actual quality of the release, but it can tell people what *we know* about the quality of the release. Like, it makes it clear that we are about to release GA with ~70 crashing bugs, give or take. "Wrong result" bugs, which are actually even worse for many real-life scenarios, are more difficult to count, but there are plenty of them as well. I can admit right away that the release is by no means sufficiently tested, not even close. The problem is, even if it was tested more, the quality would not have become better -- as you can easily see from the numbers above, we don't have enough manpower for bugfixing.

...

In fact, given that Daniel probably leaves for the meeting next Thursday (?), and knowing our buildbot and other infrastructural instability, we should aim to release the tree which we will have on Tuesday, the very latest. Which basically means it should be frozen on Saturday, to let it build fully, and leave Monday for fixing whatever gets broken by Friday pushes.

Among other things, it most likely means *no* to systemd -- unless we are unbelievably lucky and the first systemd-enabled builds in buildbot (which will hopefully happen tonight) show great success, there will be no time to fix anything.

Again more good points that I can’t argue with.

Original email says, "The release will happen after next week's Maria call. Please scream and shout if you think there are some issues in addition to the ones on the sprint that you think still must be fixed before going GA.”

You bring up a good point that nothing realistically gets fixed from now till then. So while I’m not Rasmus, maybe we do have to “delay” the release till we’re at the company meeting. Again, I have no idea what can get done from now till… whenever we freeze

Frankly, as long as we release before 19 Oct, I think we’ll be before MySQL 5.7. However I think we need to also plan this as MariaDB Corporation will want to do marketing around this (and we now have new marketing lead(s)).

If we really must release before 19 Oct, I would not count on releasing during the meeting, it's too optimistic, we don't even know how well the connection will work. And bugfixing during the meeting should also be minimized, previous experience shows it does not do much good, probably due to all the distractions. Regards, /E ======================================================================= From: Sergei Golubchik Subject: Re: [All] Current and last 10.1.8 sprint To: Elena Stepanova Cc: all@lists.askmonty.org, Colin Charles Date: Fri, 2 Oct 2015 10:00:45 +0200 (2 weeks, 4 days, 21 hours ago) Hi, Elena! On Oct 02, Elena Stepanova wrote:

To make sure that at least members of this list are well-informed, I collected some quick statistics off the top of our JIRA. It's not very accurate, but it gives a good enough approximation.

*5.5* (bugs said to affect 5.5 and possibly higher versions)

I've also looked at bugs that we plan to fix in 5.5, that's about 2/3rd of affected.

~100 bugs of Major Priority or higher;

~67 bugs here, including, say, "MDEV-8557 [PATCH] Spelling mistakes".

out of them, ~27 crashes.

*10.0* (bugs said to affect 10.0 and possibly 10.1)

~ 178 bugs of Major Priority or higher; out of them, ~36 crashes.

*10.1* (bugs said to affect 10.1 only)

~50 bugs of Major Priority or higher; out of them, ~14 crashes.

Remember, all of this is open statistics, everyone who wants can do the same search in JIRA. It does not evaluate the actual quality of the release, but it can tell people what *we know* about the quality of the release.

Yes. In average (over last two years) we're getting ~20 major (or higher) bugs for 5.5 in a month. And fixing about as much.

If we really must release before 19 Oct, I would not count on releasing during the meeting, it's too optimistic, we don't even know how well the connection will work. And bugfixing during the meeting should also be minimized, previous experience shows it does not do much good, probably due to all the distractions.

It's not difficult to find good internet connection in Amsterdam. I woulnd't suggest to do bugfixing at the meeting, but there's still a week before it. And releasing on the meeting is possible, we've done it before. Regards, Sergei ======================================================================= From: Elena Stepanova Subject: Re: [All] Current and last 10.1.8 sprint To: Sergei Golubchik Cc: all@lists.askmonty.org, Colin Charles Date: Fri, 2 Oct 2015 13:37:44 +0300 (2 weeks, 4 days, 18 hours ago) Hi Sergei, On 02.10.2015 11:00, Sergei Golubchik wrote:

Hi, Elena!

On Oct 02, Elena Stepanova wrote:

...

To make sure that at least members of this list are well-informed, I collected some quick statistics off the top of our JIRA. It's not very accurate, but it gives a good enough approximation.

*5.5* (bugs said to affect 5.5 and possibly higher versions)

I've also looked at bugs that we plan to fix in 5.5, that's about 2/3rd of affected.

...

~100 bugs of Major Priority or higher;

~67 bugs here, including, say, "MDEV-8557 [PATCH] Spelling mistakes".

From the external perspective, I find 'Affects version' more useful

Right. than 'Fix version', because the former shows something real: version X.Y[.Z] is, or once was, affected by this bug. The meaning of our 'Fix version' field is vague, it's something like "We would like to fix it in version X.Y[.Z] and now we think there is a chance we will do it, but there is no guarantee when it will happen, or whether it will happen at all". Normally, users should not pay attention to it. My point was to show what an interested party (or an unfriendly blogger) can gather about any of our GA releases by a simple JIRA search. In fact, I'm surprised nobody out there blogged about our releases in this manner yet, they could make a good fun of us by matching these results with our release criteria published in the KB. Sooner or later it will happen, and I'd rather remind about it now than later listen to complaints from marketing or sales "It's bad for business, how could engineering allow it to happen?!".

It's not difficult to find good internet connection in Amsterdam. I woulnd't suggest to do bugfixing at the meeting, but there's still a week before it. And releasing on the meeting is possible, we've done it before.

Of course it's possible, it's just not reliable (even less reliable than doing it from home). What I meant was that if we really, really *must* release by 19th, I would not rely on doing it on the meeting. But even if we decide to do it from Amsterdam -- the current sprint ends on Tuesday, developers leave on Friday, so there is only Wednesday and Thursday to fix last-moment failures in buildbot, thus there is still no much room for fixing anything apart from the current sprint. Regards, /E =======================================================================