Recent developments by Sartura include the integration of kernel version 4.19 into the mainline OpenWrt

OpenWrt is an open source firmware designed for embedded devices that is highly extensible and enables both users and customers to install a powerful, highly customizable variant of Linux on their routers. The OpenWrt community recently launched 18.06 stable release, which marked the first stable version after the OpenWrt-LEDE merge.

Throughout its releases, OpenWrt used LTS kernels to provide a stable distribution for embedded devices. Although LTS releases guarantee security support - six years since Linux 4.4 - they prove highly difficult to upgrade. The latest stable OpenWrt release uses kernel version 4.14 with most of the targets, and an updated 4.9 kernel for several older targets (see full changelog).

While the next OpenWrt stable version release (OpenWrt 19.01) targeted for January 2019 will likely use kernel version 4.14, the release after that, which is aimed for late summer 2019, should use kernel 4.19 only. As adding support for a new kernel is no simple task, it is required to start work on it as soon as possible in order to meet the project's versioning goals. In line with this, Sartura recently made first contributions for integrating this kernel version into the latest OpenWrt release candidate. The contribution is available here.

Porting Details

Generally, integrating a new kernel version is a challenging task, particularly in scenarios such as this one where there are several kernel versions in between which have to be taken into account. Similar workloads often require that more than several hundred patches are reviewed and applied on the new kernel. For this integration, we were fortunately able to remove some of the old patches as they are backported fixes from subsequent kernel versions. Some patches, however, add features that are already in the new kernel but its respective source code has gone through changes in the meantime. Such changes require that these kind of patches are detected and removed.

Other patches need to be manually applied and fixed to work with the latest kernel source, which proves especially tricky for large driver source files, even more so due to the fact that driver dependencies often change with the kernel. OpenWrt's flexibility in this regard is not up to standard of some other projects, and we were in some places forced to patch the OpenWrt source itself to make things work. These changes were mainly needed with some OpenWrt packages which depend on kernel headers and on source files.

Another peculiarity of OpenWrt is that it adds a dozen of custom files to the kernel source code. Some of these files are not compatible with the latest kernel version, and some have been added to the mainline kernel but on a different location. All this can lead to a time-consuming cycle of tracking and fixing, where numerous issues can easily arise if such custom files are not detected and resolved on time.

Lastly, after making all the changes, we need to thoroughly test that everything compiles as it should. This testing is done by compiling with every single kernel module enabled. During this procedure, we can detect patches that were not fixed properly and redo them. When everything compiles fine, we can run test the kernel itself, which we do by creating as many different kernel images as we can and booting them on various available hardware based on ARM and MIPS architectures.

Further steps

This blog post only skims through the complexity of integrating new kernel versions into the OpenWrt project. At this point, we hope for these changes to be approved and integrated into the project by the OpenWrt community. We will continue bringing necessary changes for delivering 4.19 stable release to OpenWrt, and we hope that the OpenWrt community will join us in these efforts.

We also invite other industry partners to join us in bringing the advancements of 4.19 kernel version to OpenWrt. If you want to collaborate or have any additional questions, feel free to reach us at info@sartura.hr.