Let’s start this 10th edition of Kernel Recipes !
This is the anniversary edition of Kernel Recipes, 12 years after the first edition.
David Miller is the godfather of this edition.
This is the live stream from those who prefer video over text.
A quarter century of Linux open source – Jens Axboe’s personal journey
In this non-technical talk, Jens explains how he got his start in Open Source, and Linux in particular.
In 1993, Jens discovered slackware in article, and downloaded it from a Swedish BBS, exploding the home Internet phone bill. He played with it Linux, and found the coolest thing was the kernel part, and started following LKML very closely.
After seeing the cdrom maintainer looking for a replacement, he offered help, and that’s how he became a maintainer. He learned a lot in the process, from optical media burning, to IDE and then the block layer.
The responsibilities of the Kernel Maintainer are numerous: from setting the general direction of the subsystem, reviewing changes, doing stable backports, and many others. One of those is dealing with bug reports.
Many bug reports include zero details, and that is very bad. But what is a good report ? A good bug report must include specifics on versions, what broke, relevant details, and the steps to reproduce it. Even better: include a fixing patch. A git bisect is also very helpful.
The kernel rate of change is one release every two months. That’s both fast and slow: while a patch can reach the upstream in 3 months, it might take a year or more to reach the user distributions.
As maintainer, it’s easy to have fatigue. There is no “off switch”: emails arrive every day, from all time zones; security issues need to be fixed during the weekend.
A solution to this is to find new people to help, and share the workload as group. But, Jens says, it’s very hard to find the right people.
A new contributor needs to understand how to use the email infrastructure the kernel relies on: tools like b4 and lei help; git send-email is good for sending patches.
When upstreaming, it’s good to do something good for the project as a preparation: not taking only your needs into account. If nobody pays attention to the patch, a trick is to “half-ass” the change, which attracts many reviewers. But it’s a difficult balance to strike.
Sometimes there might issues when submitting complex patchsets. A trick to work around that is to have multiple smaller series. When sending new versions of those patchsets, always wait until reviews on previous versions have calmed down.
Before submitting a patch, or even a new version of patch, always verify that it compiles, with no new warnings, and test it. A good commit message should explain the why; if it’s a fix explain what is broken, and what are the consequences of the patch. It should explain how the patch was tested. Many of those things should be doubled checked.
The Arm laptop project – Johan Hovold
This talk about Linaro’s Arm laptop project, on which Johan has been working for the past year and half. The goal is to run Linux on Windows On Arm Qualcomm laptops.
These laptop have a non-standard boot process, and previous work to get Linux to boot on them was already done.
Recently, the third-generation Lenovo ThinkPad X13s changed the game by providing a much more powerful platform to work on.
After boot EFI runtime services won’t be available. The custom ACPI implementation also made things more complicated.
The process make them work with mainline Linux included multiple phases, from reverse engineering, refactoring, adding features, reviewing patches, etc.
Johan worked with its own regularly rebased tree, and caught a few regressions upstream in the process.
For device firmware, the initial work used the Windows firmware, with work done in the background to have Lenovo and Qualcomm release redistributable firmware.
Over the past 5 kernel releases, many features were added. There are many that are still a work in progress: Camera, EFI variables, Fingerprint reader, Video acceleration, etc.
The power consumption situation is still being improved, but it’s already at 15h of idle time, and 29h of suspend time.
GPU support has been worked on by Rob Clark, the freedreno maintainer. The support is already in Mesa and the kernel; but it needs an extracted GMU firmware which is not yet in linux-firmware.
Bluetooth support was added, but the device address needs to be set manually, since it is stored externally, and this part still needs to be reverse engineered.
The touchpad on the laptop is dual-sourced, so there might be different controllers; the device tree should be updated by the boot firmware to tell which is present. Some other devices, like the touchscreen, are optionally present. Those devices can’t be all in the device tree, because of the way device probing works: they might be probed in parallel, which induces mapping races in the IRQ core (this was fixed in 6.4). Shared resources (like a reset GPIO) might be claimed temporarily; this caused an interrupt regression in 6.6-rc1. Pin configuration might be claimed before HID driver probes; so those might have to be moved to the parent bus node.
To fix all this, Johan thinks the devicetree specification should be extended to be able to mark devices as mutually exclusive, and even force sequential probing.
Other issues might arise when doing handover from the bootloader to the kernel, or from the early kernel, to late kernel if drivers are built as modules.
For future work, Johan says that support for the Camera ISP can only be done by Qualcomm. Same for virtualization, which needs early firmware fixes. Other areas planned include Bluetooth MAC address (BD_ADDR), Thermal throttling, special keyboard keys, TPM and USB PD.
It’s possible to get started with the X13s today: it needs a recent kernel (6.5+), up-to-date linux-firmware and alsa-ucm-conf, as well as two qualcomm userspace daemons. There are a few guides for available Fedora, Ubuntu or Debian. Support is good enough for everyday use of a few kernel maintainers, Johan says.
The maintainer’s POV – Borislav Petkov
Borislav wants to share his 10 year experience as maintainer in x86 land; he was once wondering why maintainers were so grumpy — now he knows.
The development process, he says is like a fence; you have stuff flying over the fence. Patches from submitters, most employed by businesses interested in getting code upstream. There are users and testers; unfortunately not enough testers.
Code submitters send code without necessarily helping with other tasks — that falls to the maintainer. Many people want to rush stuff, which leads to bad quality; things shouldn’t be rushed, but done properly: test properly, address feedback, etc.
On the other side of the fence, there are the maintainers. They don’t scale, so maintainer groups are a better solution, but scaling them is still hard. Maintainers need to handle bug fixes, code integration, reviews, their day job, and the new thing: embargoed hardware issues. Those are done behind closed doors, and often break things because they can’t benefit from the same testing. The last ones even broke clang builds for example.
New hardware features often need to be added in the kernel as well, and might not be well-thought from a software point of view.
Maintainers aren’t born grumpy, they become grumpy. The fix for that, according to Borislav, is to make everyone a review and maintainer. If everyone participates in reviews, and enhancement of the source code maintainability. The goal is to have everyone think about the big picture.
After sending patches, one should start reviewing other patches while waiting for their own review, Borislav says. Getting a code review is priceless. It requires thinking as maintainer: does the code make sense? would I accept it ? and would I maintain it for many years ?
Why are maintainers nitpicking about many small details ? There are many reasons. Good, concise and imperative commit titles save a loot of time when looking at a patch queue.
The commit messages should aim for clarity, and have all the necessary context for anyone to be able to understand what’s happening. Even yourself in a few years from now. This helps because maintainers are always doing git archeology. For example, when pondering if code can be removed, having the reason why it was added can help making a decision.
Structuring a good commit message is not rocket science, and Borislav shared a simple method that can be applied; when in doubt, read the Linux kernel documentation on submitting patches.
Adding tags like Fixes: <sha1-of-bug-introduction> is very helpful, because it helps when backporting. The Signed-off-By chain is mandatory. Link: is a pointer to the mailing list discussion on lore, also very helpful for code archeology.
Continue on the afternoon live blog.