MySQL Distro Mtg in Brussels 2010/Feb/08
Schedule: At 17:45 Tomas and Kaj need to go on a company-wide conf call that all MySQLers need to join 18:00-19:00. Giuseppe has asked foa the bus to collect us 30 mins later at 19:00. He will keep us informed.
People present (clockwise)
Giuseppe Maxia, MySQL Community Team Lead, Italy (Sardinia)
Tomas Ulin, MySQL VP Engineering, Sweden (Stockholm)
Harmut Holzgraefe, MySQL Support, Germany (Bielefeld)
Lars Heill, MySQL Build, Norway (Trondheim)
Joro Kodinov, MySQL Engineering / 5.1, Bulgaria (Plovdiv)
Oden Eriksson, Mandriva, Sweden (Jokkmokk)
Mathias Gug, Canonical/Ubuntu, Canada (Montreal) [Monday only]
Robin H. Johnson, Gentoo Linux, Canada (Vancouver)
Michal Hrušecký, Novell/openSUSE, Czech Republic (Prague)
Geir Høydalsvik, MySQL QA, Norway (Trondheim)
Norbert Tretkowski, Debian Linux, Germany (Nürnberg)
Kaj Arnö, MySQL VP Community Relations, Germany (München) [Monday only]
Please enter your name in the area to the right.
Topics for day one (based on the goals identified by our distros):
1. Bugs process
2. QA practices
3. Packaging practices
4. Patches and Deltas (including configure options)
5. Security bugs
6. Overall workflow conclusions
Topics for day two:
7. Support
No issues from Hartmut.
Tomas asks which distributions provides MySQLsupport
* Mandriva doesn't
* Suse does it for advanced customers
* Gentoo does custom MySQL support for customers indirectly
* Debian doesn't
8. GUI tools / Workbench
* GUI tools not really usable on Linux, and not supported
* GUI patched by Suse but patches did not go upstream
* GUI tools provided by Debian with Suse patches
* Gentoo provides version 5.0_p12, unsupported
If we want to include WB 5.2 in new distributions, it should be GA before May (Ubuntu), or June (Suse) or July (Debian).
Norbert notes that DBAs don't like the idea of deploying a detabase design tool to do normal administration.
No third party GUI tools are shipped.
Debian doesn't ship WB because of version issues (Windows and Mac headers)
Suse to file a feature request about packaging issues for Workbench.
9. MySQL Cluster
Complaints from inside MySQL about distros shipping old versions.
Technical problem on MySQL and problem on distro side.
Till winter 2006, cluster+MySQL were together as one package.
Customer demand required forking Cluster from main MySQL.
Now distinct products. May remerge in future, but staying seperate for development.
Want 6.3/7.1 in main tree.
Gentoo has an issue about versioning and linkage to client libraries.
MySQL Cluster applies security patches with a delay. This could be an issue for the distributions. Tomas will look into the possibility of patching security bugs faster.
Anybody using non-cluster tarballs SHOULD remove NDB support. NDB should only be used with Cluster sources.
Note of Gentoo explicit correlation of 1 source tarball to 1 Gentoo package.
MySQL should fix the source code of the main server, to avoid the inclusion of the ndb binaries by chance.
The distributions will include both MySQL server AND Cluster, in a way that they are mutually exclusive.
ELF Linking is a major issue for distributions. Tomas is considering renaming libmysqlembedded ELF since it is not link-compatible between cluster and none.
mysql instance manager startup tool should not be used anymore, good design, bad implementation.
NDB may be dropped from main 5.1 tarball.
MySQL documentation should explicitly mention cross version up/downgrade procedures & problems when switching from 'regular' to 'cluster' versions and back.
10. Other tools / Connectors
Other goals identified during the presentation round
- Learning from others (MySQL from distros, distros from distros) -- jokingly named "industrial espionage" by Kaj, but of course this is a very valid goal for each party
1. Bugs process
Task: Review the process and systems for bugs (and Worklog for new tasks)
How to push bugs entered into launchpad from Ubuntu onwards to MySQL database
Presentation of the Bug process:
Joro
Anything to happen needs to be in 2 streams:
- BugDB: bugs.mysql.com
Need to file a bugs.
Review of feature bugs.
a. Need to have the bug in the Verified state to get the attention of the developers.
b. Scheduling session: between QA/Engineering team (triage comity) - prioritize Verified bugs:
1. Defect of the bug: 1 (worst) to 5 (wishlist). Documented in the help of the Bug DB.
2. Risk level: 1 to 5.
3. Workaround possibility: 1 (nothing) to 5 (easy workaround).
4. Impact: 1 (everyone) to 5 (reporter only, not always).
5. Effort: 1 (no effort) to 5. How much time/ressource. Cost of fixing.
c. Assign target.
- Worklog:
In GA:
1. no regression
2. fix crashers
2. QA practices
Task: Review the QA process. Understand the "daily build" goal of some distros.
Developer QA - MTR test suites:
Pushbuilds on every push - every time a developer pushes a change set
gtests now added to push builds, a little left to clean up
Server QA:
* Approve worklogs / QA
* Some maintenance of MTR testsuite
* random-query-generator (available on LP)
* (also found on http://forge.mysql.com/wiki/RQG presentation: http://forge.mysql.com/wiki/Random_Query_Generator)
System QA:
* non-functional tests
* performance
* upgrade/downgrade
* concurrency
* tests run for about a week for each release
* Some System QA tests are run in Pushbuild, too
Worklogs
* architectural review
* high level specs
* low level define
* available at http://forge.mysql.com/worklog/
* after approval, they will go in a stage branch
* every new feature has a test case/plan defined upfront and won't be commited to trunk if the QA fails.
* summary of tests before a release: http://blogs.sun.com/datacharmer/entry/how_mysql_tests_servers_before
MySQL Server QA would like to subscribe to all crash/error reporting facilities , that is, the "phone home" applications which record crashes and other failures that happen on the user's machines and database them for further analysis (http://en.wikipedia.org/wiki/Crash_reporter)
For example, Ubuntu have https://wiki.ubuntu.com/Apport, which files bugs directly into their LaunchPad bug tracker, but it is fairly difficult to separate the mysqld errors from all others, and I can not seem to find any crashes, only package installation issues.
That depends on bugs being filed for those crashed into the upstream bug system - Ubuntu intends to do that (has not been doing much of that so far, mostly pkg install). [Lars, from Mathias]
Ubuntu Stable release updates critieria:
https://wiki.ubuntu.com/StableReleaseUpdates
Tests run by:
Debian/Ubuntu: test-force during the build process, test-bt available by passing fulltest at package build time.
openSUSE runs --big-test manually randomly during development and before distribution release
The test suite is currently not run in mandriva cooker:
http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/mysql/current/SPECS/mysql.spec?revision=500359&view=markup
I think this is due to borkiness in cooker or added 3rd party storage engines. Usually the test suite are enabled and has to pass prior a release.
Upstream: test-bt and test-bt-debug for release, in various combinations.
Parallel testsuite runs:
export MTR_BUILD_THREAD="$((${RANDOM} % 100))"
3. Packaging practices (including configure options)
Task: Understanding how packaging is done in each distribution. Assess whether there are possibilities to share code.
At this point, each distro should also describe their goal and target audience (which determines packaging practices, and practices of accepting or declining patches)
Presentation about MySQL in SuSE
https://michal.hrusecky.net/download/suse-mysql.pdf
MySQL in Debian:
http://tretkowski.de/slides/MySQL_in_Debian.pdf
Planning to move from autotools to cmake.
Release process:
1. 7 days to build binaries (approx - from 1-2 hrs to 24 hrs per build (6 total for 5.1), depending on plf)
2. Run tests on binaries: make test-bt, test-bt-debug
5.1 bzr branches:
most of the patches are pushed in the bugtree: lp:~mysql/mysql-server/mysql-5.1-bugteam
There is a need amongst distros to ship libraries as shared, not statically linked. This is hard especially for GUI tools.
There is a stated interest (Robin from Gentoo) for MySQL to pass a wider range of configure options. Not all, but a wider range.
Testsuite failures due to SSL:
Testsuite call: test-ns, test-ps
Test names: openssl_1 rpl_openssl rpl.rpl_ssl rpl.rpl_ssl1 ssl ssl_8k_key ssl_compress ssl_connect rpl.rpl_heartbeat_ssl
SSL certificates should have NO explicit dates. There should be zero reliance on system clock (it CAN be 1970 start of epoch on some embedded systems).
Should be resolved by generating SSL certificates as part of the test setup process.
Fix from Red Hat: https://bugzilla.redhat.com/show_bug.cgi?id=e541233
Testsuite failures due to character sets:
./configure ... --with-charset=utf8 --with-collation=utf8_general_ci ...
Testsuite call: test-ns, test-ps
Breaks tests: main.mysql_client_test main.mysql_comments main.mysql_upgrade main.information_schema funcs_1.is_columns_mysql funcs_1.is_tables_mysql funcs_1.is_triggers
Should be resolved by reviewing the setup phase of the tests in question, to see that they explicitly set a character set. (Issue forwarded to the QA team)
A comprehensive presentation on how the build works is available in MySQL Forge
http://forge.mysql.com/wiki/Building_MySQL_Releases_on_Unix
Ubuntu/Debian asks if MySQL can add a set of specifications to build .deb packages, to make the upstream process of sharing patches easier.
4. Patches and Deltas
Task: Review the current deltas between the patches applied on MySQL a) between Debian
and Ubuntu, b) between MySQL and Debian. There are probably often just historical reasons (as opposed to good reasons) why there is a delta. We should understand the current deltas between the distros. Ideally, we should eliminate the underlying reasons for these deltas to exist.
Patches marked with "change in functionality" are currently not accepted by Ubuntu because they are believed to introduce new features. MySQL should probably change the wording to avoid this misunderstanding. The new functionality is introduced as a necessary side effect of a bug fix, or to make the functionality match the intended feature as described in the docs.
Ubuntu Stable release updates critieria:
https://wiki.ubuntu.com/StableReleaseUpdates
5. Security bugs
- Norbert: Some bugs are marked private. We cannot follow those discussions. We won't get notified when such bugs are fixed. There should be a private mailing list for sharing information about such bugs
- Kaj: There are mostly good reasons to keep some security bugs private. A good reason is to minimise the impact of security bugs on the user base, at least until there is a patch for the bug. A bad reason might be that MySQLers not exposed to the community and distros would too sloppily mark bugs as private. One result of today is that at least the MySQLers present in this mtg will understand the need for open communication towards the distributors of MySQL.
MySQL can implement a private mailing list, accessible to the Linux maintainers, where we publish the detail of the security bugs, minus the private comments.
6. Overall workflow conclusions
- who puts what into the bugs database
- is the MySQL Worklog on Forge productive to use
- what are the best practices for linking launchpad between upstream (MySQL), Debian, Ubuntu, others
- how should communication be tracked
No further items on this topic needed.
7. Support
Mandriva: Users report to distro bugzilla, and triage from there. Mandriva may have a contract w/ MySQL directly.
Suse: L3 in the new releases (Michal please explain here?)
Debian, Gentoo: No direct paid support, but some developers/external organizations may provide support.
8. GUI tools / Workbench
- "I want to complain about the GUI tools and Workbench"
- Kaj: Over the years, there have been many versions of GUI tools from MySQL. Some have been short lived. Initially, the current GUI team lead (Workbench team lead) Mike Zinner coded using the cross-platform Kylix tool, which provided the same functionality on each platform (which is good) but which was buggy (which is bad). Hence MySQL soon switched to native tools on each platform, often with Linux provided later or not yet / never. One reason for this was also the larger need for GUI tools on Windows than on Linux. <Kaj asks for an assessment by the distros of the importance of GUI tools and was met with some, but still rather limited, enthusiasm>.
- Workbench build scripts (half-hearted autotools setup, ...)
- "old" GUI tools End-Of-Life ...
Some GUI parts are explicitly part of the Certification process.
Workbench is to replace GUI tools (Query Browser and Administrator)
GA date unknown.
Giuseppe is interested in feedback from users, on state of GUI tools/WorkBench.
MySQL-gui-tools version bump request in Gentoo
http://bugs.gentoo.org/show_bug.cgi?id=272234
Notice no votes for it, and compile failures.
Norbert only hears complaints from DBAs that say they don't want a DB design tool mixed in with an Admin tool. One-task-one-tool philosophy.
Debian freeze in March or April.
SuSE freeze in June
http://www.suse.de/~coolo/opensuse_11.3/
The mandriva stuff for mysql-workbench-oss:
http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/mysql-workbench-oss/current/
Some other tools available:
- mtop
- mytop
- mysql sandbox
9. MySQL Cluster
- drop nbd from MySQL server (maybe error in official tarballs)
- only one should be installed (either Cluster or Server)
- ship libmysqld from cluster under different name
10. Other tools / Connectors
- Walkthrough of connectors provided by MySQL, which may be interesting to include in a distro
- Walkthrough of the most important connectors *not* provided directly by MySQL (example: MySQLdb for Python), also potentially good to have in a distro
Regarding the openssl/gnutls issue. It's a flaw in the RFC:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3555
The Mandriva presentation: http://nux.se/mysql/mandriva_sun.pdf
Unresolved build problems with mysql in Mandriva:
http://nux.se/mysql/buildproblems.html (Lars will follow up) - NDB-related
The only patch from Mandriva :)
http://nux.se/mysql/mysql-5.1.30-use_-avoid-version_for_plugins.diff
Release cycle
- GA releases every 12-18 months
- Max 2 releases in Active support
http://forge.mysql.com/wiki/Development_Cycle
Patching MySQL 5.0 for building a dynamic libmysqld
http://git.overlays.gentoo.org/gitweb/?p=proj/mysql-extras.git;a=blob;f=02040_all_embedded-library-shared-5.0.77.patch;h=4b34d12a70be96406http://datacharmer.blogspot.com/2010/02/linux-mysql-distros-meeting-in-brussels.html9755487ee9b6b9e72d99512;hb=ba9bdcffd5cf28edf51d7f0269a1027000557c27
Overall Gentoo patch set:
http://git.overlays.gentoo.org/gitweb/?p=proj/mysql-extras.git;a=summary
See the 000*index.txt file to figure out which patches get applied to which packages and versions, the name may not be enough.
Linux kernel "Developer Certificate of Origin"
http://www.mjmwired.net/kernel/Documentation/SubmittingPatches#286
As their equiv of the Sun Contributer agreement.
How to write ebuilds for Gentoo
http://devmanual.gentoo.org
Debian Developer's Reference
http://www.debian.org/doc/developers-reference/
openSUSE Packaging guidlines:
http://en.opensuse.org/openSUSE:Packaging_Guidelines
BuildService basic info: http://en.opensuse.org/Build_Service (can contact for advice on automating own builds)
Public build service that anybody can try and use http://build.opensuse.org
oden: about mandriva packaging: http://tinyurl.com/yg29lbt
Packages are built in the mdv build system using iurt and schedulers, etc. Info about iurt here: http://wiki.mandriva.com/en/Iurt
The mdv build system web output "gui" : http://kenobi.mandriva.com/bs/
Where to download cluster 6.3:
http://dev.mysql.com/downloads/cluster/6.3.html#downloads