Continuing the series with firmware integration of eBPF technologies using our Open Source Replica.one firmware build system.
In our previous post, we demonstrated how to write a simple BPF system call tracing application. This part examines how to build and run this application on a Raspberry Pi board using our Open Source Replica.one firmware build system. During this procedure, we will highlight some issues that occur when working on embedded Linux systems.
Introduction to embedded eBPF development
The Linux kernel supports various reduced instruction set (RISC) architectures (e.g., ARM, MIPS, and RISC-V) used extensively for embedded applications. Portability of the eBPF bytecode is not a concern for such embedded systems, given that the Linux kernel implements an eBPF virtual machine with no architecture-specific instructions. However, the portability of eBPF implementation concerning kernel data structures and interfaces cannot be guaranteed, which necessitates development practices specific to embedded devices.
Our previous post mentioned the two main approaches for developing eBPF programs: leveraging the BCC compiler or taking advantage of the BPF CO-RE infrastructure. Although embedded systems can support compilation, this is usually not the case. For this reason, we will focus primarily on embedded eBPF development using BPF CO-RE and the closely related libbpf library. As we have shown earlier, the general eBPF CO-RE development workflow consists of generating the appropriate vmlinux.h header file from the BTF information exposed in the running Linux kernel sysfs interface:
~ $ bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
However, accessing BTF information in such a way requires the target architecture hardware and - because the information would become a part of the firmware kernel image - a non-trivial amount of additional storage on the platform. The suggested alternative approach is to generate a special kernel image for the target platform based on a similar kernel configuration but with additional debugging flags enabled (e.g., CONFIG_DEBUG_INFO=y and others). Once we have a kernel binary image, we can investigate the location of this data in the image using command-line tools:
~ $ file vmlinux-arm64.elf
vmlinux-arm64.elf: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV),
statically linked, BuildID[sha1]=..., with debug_info, not stripped
~ $ objdump -h vmlinux-arm64.elf | grep BTF
14 .BTF 0035e12b ffff8000112654fc ffff8000112654fc 012754fc 2**0
15 .BTF_ids 000000a0 ffff8000115c3628 ffff8000115c3628 015d3628 2**0
The required BTF information can be extracted from this kernel image using bpftool(8):
~ $ bpftool btf dump file vmlinux-arm64.elf format c | tee vmlinux.h | tail
u64 romsize;
u32 romimage;
} mixed_mode;
};
#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute pop
#endif
#endif /* __VMLINUX_H__ */
It is important to note that the generated vmlinux.h is an authoritative information on the actual kernel structures for the particular target system, and particular kernel configuration. Compare for example the thread_info structure on x86-64 versus ARM64:
~ $ bpftool btf dump file vmlinux-arm64.elf format c | grep -m1 -A16 thread_info
struct thread_info {
long unsigned int flags;
u64 ttbr0;
union {
u64 preempt_count;
struct {
u32 count;
u32 need_resched;
} preempt;
};
};
~ $ bpftool btf dump file vmlinux-amd64.elf format c | grep -A4 thread_info
struct thread_info {
long unsigned int flags;
long unsigned int syscall_work;
u32 status;
};
For this reason, writing portable eBPF programs may require conditional implementations dependent on the architecture of the target platform and the kernel configuration. Thankfully, the libbpf library provides headers that alleviate some of the boilerplate code for special use-cases like BPF tracing. For example, the bpf_tracing.h header contains conditionals that define how BPF programs access data contained in the pt_regs kernel data structure depending on the target architecture:
/* Scan the ARCH passed in from ARCH env variable (see Makefile) */
#if defined(__TARGET_ARCH_x86)
#define bpf_target_x86
#define bpf_target_defined
[...]
#if defined(bpf_target_x86)
#if defined(__KERNEL__) || defined(__VMLINUX_H__)
#define PT_REGS_PARM1(x) ((x)->di)
#define PT_REGS_PARM2(x) ((x)->si)
#define PT_REGS_PARM3(x) ((x)->dx)
#define PT_REGS_PARM4(x) ((x)->cx)
#define PT_REGS_PARM5(x) ((x)->r8)
#define PT_REGS_RET(x) ((x)->sp)
#define PT_REGS_FP(x) ((x)->bp)
#define PT_REGS_RC(x) ((x)->ax)
#define PT_REGS_SP(x) ((x)->sp)
#define PT_REGS_IP(x) ((x)->ip)
[...]
Therefore, libbpf convenience macros are a highly suggested tool.
Regardless of the implementation details, eBPF programs are compiled into bytecode using either LLVM or the GCC compiler suite. Though the resulting bytecode will not be architecture-specific, it might significantly differ depending on the target architecture. The build process of eBPF programs for the target platform still falls under the term "cross-compilation" because we must observe special procedures for such systems. Most toolchains in the wild support cross-compilation; however, a large amount of work is needed to prepare the cross-compiled firmware manually. For this reason, it is much easier to integrate eBPF program packages into existing (firmware) build systems such as OpenWrt, Yocto, and Replica.one. This post focuses on integrating the eBPF example into the Replica.one firmware build system.
Packaging the eBPF example for the Gentoo distribution
The Replica.one build system is an Open Source firmware build system based on the Gentoo Linux distribution. One of the more powerful features of this build system is its tight integration with Gentoo’s package management system - Portage. Regarding eBPF development, we are interested in Portage's support for cross-compilation, which the Replica.one system provides transparently.
Packaging for the Gentoo distribution is based on ebuilds - plain text files written in a bash-compatible syntax similar to Arch Linux PKGBUILDs or *BSD ports collections. These text files contain all of the necessary information needed to create a binary package. They define the location and method for fetching source code, the procedures for preparing, configuring, building, and packaging this source code, and additional metadata such as dependency information and licenses. A powerful feature of the Portage package manager is its ability to define and select individual package "features" through USE flags. For example, many software packages use GNU Autoconf to prepare a configuration script, which generates a Makefile specific to a particular system configuration. The configure process allows arguments such as --with-
This section will explore basic ebuild and Portage features to successfully prepare and package our eBPF examples for the Gentoo distribution and consequently for the Replica.one build system firmware. The initial step is to prepare existing source code with relatively simple build infrastructure such as GNU autotools or CMake. The existing project file organization is as follows:
.
├── [4.0K] cmake/
│ └── [4.0K] Modules/
│ ├── [ 545] FindLIBBPF.cmake
│ └── [3.7K] UseBPF.cmake
├── [4.0K] include/
│ ├── [2.4M] vmlinux-arm64.h
│ ├── [3.7M] vmlinux-arm.h
│ └── [2.4M] vmlinux-x86.h
├── [4.0K] src/
│ ├── [ 259] hello.bpf.c
│ ├── [1.1K] hello.c
│ ├── [ 863] maps.bpf.c
│ ├── [2.6K] maps.c
│ └── [ 136] maps.h
└── [1.5K] CMakeLists.txt
To ease the setup, we will use a CMake-based infrastructure specially crafted to support cross-compilation of eBPF programs. The structure of CMake build system projects can vary significantly due to non-standard practices, although most CMake projects contain a top-level CMakeLists.txt recipe. We have the following header in our project recipe:
1 cmake_minimum_required(VERSION 3.5)
2 project(ebpf-core-sample)
3
4 list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/Modules")
5
6 find_package(LIBBPF REQUIRED)
7 include(UseBPF)
8
9 include_directories(${LIBBPF_INCLUDE_DIRS})
10 include_directories(${CMAKE_BINARY_DIR})
11
12 set(BPF_ARCH "" CACHE STRING "Architecture of eBPF programs")
13 option(INSTALL_VMLINUX "Enable installation of the vmlinux header" ON)
[...]
First, we initialize the CMake build system by declaring the minimum required CMake version and the project name. Afterward, we append our own cmake/Modules directory in the module search path; our project-scope CMake directory contains useful modules for dealing with the BPF infrastructure. For example, the FindLIBBPF.cmake module serves to find appropriate locations of the libbpf library and its header include directories via the find_package function. We mark LIBBPF_INCLUDE_DIRS content using the include_directories directive to be considered by the compiler preprocessor for the location of library headers. Additionally, we declare and set the BPF_ARCH variable, which specifies target architecture for cross-compilation, and the INSTALL_VMLINUX option. Using this information, we prepare the appropriate header file for the C preprocessor:
[...]
15 configure_file(${CMAKE_SOURCE_DIR}/include/vmlinux-${BPF_ARCH}.h
16 ${CMAKE_BINARY_DIR}/vmlinux.h
17 COPYONLY)
18 if(INSTALL_VMLINUX)
19 install(FILES "${CMAKE_BINARY_DIR}/vmlinux.h" DESTINATION include)
20 endif()
[...]
Here, we have copied the architecture-specific BTF-generated vmlinux header file into CMAKE_BINARY_DIR and conditionally marked it for installation if the INSTALL_VMLINUX option is enabled. Now we can define BPF targets:
[...]
25 BPF_TARGET(hello-bpf src/hello.bpf.c
26 ${CMAKE_BINARY_DIR}/hello.bpf.o
27 ${CMAKE_BINARY_DIR}/hello.skel.h
28 BPF_ARCH ${BPF_ARCH}
29 VMLINUX_FILE ${CMAKE_BINARY_DIR}/vmlinux.h)
30 install(FILES "${CMAKE_BINARY_DIR}/hello.bpf.o" DESTINATION share)
31
32 BPF_TARGET(maps-bpf src/maps.bpf.c
33 ${CMAKE_BINARY_DIR}/maps.bpf.o
34 ${CMAKE_BINARY_DIR}/maps.skel.h
35 BPF_ARCH ${BPF_ARCH}
36 VMLINUX_FILE ${CMAKE_BINARY_DIR}/vmlinux.h)
37 install(FILES "${CMAKE_BINARY_DIR}/maps.bpf.o" DESTINATION share)
[...]
The BPF_TARGET macro is defined in our cmake/Modules/UseBPF.cmake module, which specifies a generic process for building eBPF programs:
[...]
set(BPF_TARGET_usage "BPF_TARGET(<Name> <Input> <Output> <OutputSkel>
[COMPILE_FLAGS <string>] [VMLINUX_FILE <string>]
[BPF_ARCH <string>]")
[...]
add_custom_command(OUTPUT ${_bpf_OUTPUT}
COMMAND clang -target bpf -D__TARGET_ARCH_${_bpf_ARCH} ${_bpf_EXE_OPTS}
-I${LIBBPF_INCLUDE_DIRS} -I${CMAKE_BINARY_DIR}
-o ${_bpf_OUTPUT} -c ${_bpf_INPUT}
VERBATIM
DEPENDS ${_bpf_INPUT} ${_bpf_VMLINUX}
COMMENT "[BPF][${Name}] Building program ... "
WORKING_DIRECTORY ${_bpf_WORKING_DIR})
add_custom_command(OUTPUT ${_bpf_OUTPUT_SKEL}
COMMAND bpftool gen skeleton ${_bpf_OUTPUT} > ${_bpf_OUTPUT_SKEL}
VERBATIM
DEPENDS ${_bpf_OUTPUT}
COMMENT "[BPF][${Name}] Building program skeleton ... "
WORKING_DIRECTORY ${_bpf_WORKING_DIR})
[...]
In the above snippet, we can see that the macro first builds the eBPF bytecode using clang(1) and then the eBPF skeleton header file with bpftool(8). The macro allows passing additional options such as the BPF_ARCH necessary for defining the __TARGET_ARCH_
[...]
42 add_executable(hello src/hello.c ${BPF_hello-bpf_OUTPUT_SKEL})
43 target_link_libraries(hello ${LIBBPF_LIBRARIES})
44 install(TARGETS hello DESTINATION bin)
45
46 add_executable(maps src/maps.c ${BPF_maps-bpf_OUTPUT_SKEL})
47 target_link_libraries(maps ${LIBBPF_LIBRARIES})
48 install(TARGETS maps DESTINATION bin)
When defining the executable target with the add_executable function, we have specified BPF_<name>_OUTPUT_SKEL for both eBPF programs. This step is necessary to define correct dependencies between each target build step as represented by the following diagram:
As a consequence of the default CMake/system configuration, userspace C components are built using the GCC toolchain. The skeleton include file is simply a representation of the eBPF bytecode in pure C so any generic C compiler is appropriate for this step. As our last action, we need to modify existing *.bpf.c files to support the change in library include directives:
--- a/hello.bpf.c
+++ b/src/hello.bpf.c
@@ -1,5 +1,5 @@
#include "vmlinux.h"
-#include
+#include
[...]
--- a/maps.bpf.c
+++ b/src/maps.bpf.c
@@ -1,5 +1,5 @@
#include "vmlinux.h"
-#include
+#include
#include "maps.h"
[...]
The second portion of packaging the eBPF samples is to create an ebuild package definition for the Portage package manager. The Package Manager Specification (PMS) defines the ebuild structure and contents. The largest repository of such definitions is available in the official Gentoo package repository or overlay. Definitions provided in this repository serve as good examples of optimal packaging practices and how to write ebuilds. We start by defining some initial metadata regarding our package:
[...]
7 EAPI=7
8
9 inherit git-r3 cmake linux-info llvm toolchain-funcs
10
11 DESCRIPTION="Sample eBPF CO-RE applications"
12 HOMEPAGE="https://github.com/sartura/ebpf-core-sample/tree/devel-cmake"
13
14 EGIT_REPO_URI="https://github.com/sartura/ebpf-core-sample.git"
15 EGIT_BRANCH="devel-cmake"
16
17 SLOT="0"
18 KEYWORDS="amd64 arm arm64 x86"
[...]
We have defined an ebuild which follows the EAPI version 7 specification per the PMS. The Portage infrastructure provides the inherit command, which allows us to inherit useful tooling (eclasses) when dealing with different project build systems (e.g., GNU Autotools, CMake, Go, Rust, etc.). The git-r3 eclass will allow us to fetch package sources hosted as Git repositories, which we have utilized via the EGIT_* to specify the location and the appropriate Git branch. It is important to note that each subsequent rebuild of the package will pull the latest changes from the Git repository branch. Specifying a single commit, tag, or a versioned branch might be more appropriate, although the current approach is acceptable for Gentoo packaging when the package version is marked as “9999”. Portage defines package versions from ebuild filenames, so we will go ahead and create ebpf-core-sample-9999.ebuild.
The other important variable is KEYWORDS which defines machine architecture supported by the package. Since we have only generated vmlinux header files for ARM and x86 machines, we specify those architectures. This method is not exactly by the book because packages with version "9999" must be keyworded as unstable via the ~ key (e.g., ~arm, ~x86) and explicitly marked buildable, however, we will skip this in our example to keep it simple. Next, we define the features and dependencies of our package:
[...]
19 IUSE="+bpfobjs vmlinux"
20
21 DEPEND="dev-libs/libbpf:=
22 virtual/libelf"
23 RDEPEND="${DEPEND}"
24 BDEPEND="sys-devel/clang:=[llvm_targets_BPF(+)]
25 dev-util/bpftool"
26
27 RESTRICT="test strip"
28
29 CONFIG_CHECK="~BPF ~BPF_EVENTS ~BPF_JIT ~BPF_SYSCALL ~HAVE_EBPF_JIT
~DEBUG_INFO_BTF"
[...]
We have defined two USE flags to disable or enable additional package features at build time. Recall that we have a single CMake option named INSTALL_VMLINUX which controls whether we install the vmlinux header; the vmlinux USE flag will utilize this. The bpfobjs is enabled by default and controls whether we install eBPF bytecode objects on the system. Although CMake does not support this option, we can enable it through ebuild mechanisms. The DEPEND, RDEPEND, and BDEPEND specify "build-time", "run-time", and "host build-time" dependencies, respectively. We do require libbpf at run-time as it provides the libbpf.so shared object used by our userspace programs. However, we only need clang(1) and bpftool(8) on the host system, so we place these dependencies in BDEPEND. The CONFIG_CHECK flag is useful as it inspects the current kernel configuration for necessary kernel features required by the package. Now that we have specified our environment, we proceed to define some of the package build steps:
[...]
31 llvm_pkg_setup() {
32 # NOTE: Force BROOT-relative checks, a simplified approach based
33 # on the llvm.eclass file contents.
34 export PATH=$(get_llvm_prefix -b)/bin:$PATH
35 }
36
37 src_configure() {
38 local mycmakeargs=(
39 -DBPF_ARCH:STRING=$(tc-arch-kernel)
40 -DINSTALL_VMLINUX:BOOL=$(usex vmlinux)
41 )
42 cmake_src_configure
43 }
44
45 src_install() {
46 cmake_src_install
47
48 if ! use bpfobjs; then
49 rm -r "${ED}"/usr/share || die
50 fi
51 }
Out of around ten default steps, our code replaced only the src_configure and the src_install steps. We have modified the src_configure step to define the BPF_ARCH CMake variable for cross-compilation support, which is extracted from the environment by the tc-arch-kernel helper function provided by toolchain-funcs eclass. Additionally, we specify the INSTALL_VMLINUX boolean option through the usex boolean predicate function acting on the vmlinux USE flag.
Source: Ebuild Phase Functions - Gentoo Development Guide
The llvm_pkg_setup is a sub-step of pkg_setup that we need to override to resolve some cross-compilation quirks. In the src_install step, we use the bpfobjs USE flag to either remove or leave the eBPF bytecode objects installed. Had we not decided to support bpfobjs USE, we would not have needed to override this step. This feature is not supported (by design) by our CMake definitions, which is why we cannot include it as a part of the src_configure step.
With this, we conclude the packaging of the eBPF example: we created a simple CMake build system for the project and wrote a package definition for the Portage build system. The following section will show how to integrate this package recipe into the Replica.one build system to prepare a Raspberry Pi 4 firmware image with pre-installed eBPF samples.
Building the eBPF example using the Replica.one build system
As mentioned in the first section of this post, Replica.one is a firmware build system based on the Gentoo distribution and its package manager Portage. One of the devices supported in Replica.one is the Raspberry Pi 4 platform, which we will use to test eBPF functionality. After making sure we satisfy the requirements, we start per the quick start instructions:
~ $ git clone --depth=1 --shallow-submodules --recursive
https://github.com/sartura/replica.git && cd replica
Cloning into 'replica'...
remote: Enumerating objects: 220, done.
remote: Counting objects: 100% (220/220), done.
remote: Compressing objects: 100% (164/164), done.
[...]
The following tree diagram shows the directory structure relevant to the Raspberry Pi 4 target (rpi4):
.
├── [4.0K] config/
│ └── [4.0K] kernel/
│ └── [204K] rpi4_5.10.y_config
├── [4.0K] output/
├── [4.0K] overlay/
│ └── [4.0K] rpi4/
│ └── [4.0K] etc/
│ └── [4.0K] systemd/
│ └── [4.0K] network/
│ ├── [ 38] 25-wireless.network
│ └── [ 38] wan.network
│
├── [4.0K] repos/
│ └── [4.0K] replica/
│ └── [4.0K] profiles/
│ └── [4.0K] replica/
│ └── [4.0K] rpi4/
│ ├── [ 2] eapi
│ ├── [ 223] make.defaults
│ ├── [ 211] packages
│ ├── [ 94] package.use.mask
│ └── [ 144] parent
├── [4.0K] targets/
│ ├── [1.3K] rpi4.docker
│ └── [2.8K] rpi4.package*
├── [ 528] environment.in
├── [2.5K] environment.sh
└── [3.1K] Makefile
At this point, we must create the ebpf-core-sample-9999.ebuild file that we created in the previous section. Location-wise, we will create the ebuild file in the repos/replica "overlay" repository. We will choose the dev-util package category name as follows:
[...]
├── [4.0K] repos/
│ └── [4.0K] replica/
│ └── [4.0K] dev-util/
│ └── [4.0K] ebpf-core-sample/
│ └── [1.1K] ebpf-core-sample-9999.ebuild
[...]
Next, we include the package to be built for the Raspberry Pi 4 build target by adding "*dev-util/ebpf-core-sample" (note the asterisk preceding package name and category) in the repos/replica/profiles/replica/rpi4/packages file:
[...]
--- a/profiles/replica/rpi4/packages
+++ b/profiles/replica/rpi4/packages
@@ -10,3 +10,6 @@
# Raspberry Pi specific
*sys-boot/raspberrypi-firmware
+
+# eBPF
+*dev-util/ebpf-core-sample
Although we are now ready to build the firmware image, we should confirm that the kernel configuration supports BPF features:
~ $ cat config/kernel/rpi4_5.10.y_config | grep BPF
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_BPF_JIT_DEFAULT_ON=y
# CONFIG_BPF_PRELOAD is not set
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
# CONFIG_NET_CLS_BPF is not set
# CONFIG_NET_ACT_BPF is not set
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
# CONFIG_TEST_BPF is not set
~ $ cat config/kernel/rpi4_5.10.y_config | grep KPROBE
CONFIG_KPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_KPROBE_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
# CONFIG_KPROBE_EVENT_GEN_TEST is not set
# CONFIG_KPROBES_SANITY_TEST is not set
The output above confirms that the kernel supports the necessary features needed by our eBPF programs (the eBPF syscall and kprobe events). Finally, we invoke Replica.one to build firmware for the Raspberry Pi 4 target:
~ $ make CTARGET=aarch64-unknown-linux-gnu package_rpi4
docker pull gentoo/stage3:20210812
20210812: Pulling from gentoo/stage3
Digest: sha256:951e46036d1ee0e82f57112397658c274374a480ca1bb5dc54501ae09c019b34
Status: Image is up to date for gentoo/stage3:20210812
docker.io/gentoo/stage3:20210812
docker build . -f targets/rpi4.cache --build-arg CTARGET="aarch64-unknown-linux-gnu"
--build-arg BUILDKIT_INLINE_CACHE=1 --secret id=env,src=environment.cache
--tag replica/rpi4:latest
[+] Building 180.2s (20/30)
=> [internal] load build definition from rpi4.cache
=> => transferring dockerfile: 9.81kB
=> [internal] load .dockerignore
=> => transferring context: 201B
=> resolve image config for docker.io/docker/dockerfile:1.2
=> [auth] docker/dockerfile:pull token for registry-1.docker.io
=> CACHED docker-image://docker.io/docker/dockerfile:1.2@sha256:
e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc
=> [internal] load metadata for docker.io/gentoo/stage3:replica
=> [stage-0 1/23] FROM docker.io/gentoo/stage3:replica
=> [internal] load build context
=> => transferring context: 132.89MB
=> CACHED [stage-0 2/23] COPY ./repos/gentoo /var/db/repos/gentoo
[...]
=> => # >>> Jobs: 68 of 78 complete Load avg: 5.8, 12.5, 9.7
=> => # >>> Installing (69 of 78) dev-util/ebpf-core-sample-9999::
replica to /usr/aarch64-unknown-linux-gnu/
=> => # >>> Jobs: 68 of 78 complete Load avg: 5.8, 12.5, 9.7
=> => # >>> Jobs: 69 of 78 complete Load avg: 5.7, 12.3, 9.6
=> => # >>> Installing (70 of 78) sys-apps/systemd-248.6::gentoo
to /usr/aarch64-unknown-linux-gnu/
=> => # >>> Jobs: 69 of 78 complete Load avg: 5.7, 12.3, 9.6
[...]
=> exporting to image
=> => exporting layers
=> => writing image sha256:80616da6feef79de7649e8a038dce4913f07b5d
2761725f570a568f18f289b7a
=> => naming to docker.io/replica/rpi4:latest
=> exporting cache
=> => preparing build cache for export
docker run --privileged --rm --volume /home/jpetrina/replica/output:
/output -- replica/rpi4:latest
[...]
Writing superblocks and filesystem accounting information: done
1,031,264,780 99% 18.85MB/s 0:00:52 (xfr#10521, to-chk=0/13943)
mkfs.fat 4.2 (2021-01-31)
del devmap : loop2p2
del devmap : loop2p1
rm targets/rpi4.cache
The build should complete successfully, after which the firmware image should be ready for flashing. Consult the official Replica.one documentation for the Raspberry Pi 4 for installation instructions. The next section of the blog post will verify that the eBPF examples function as expected.
Running the eBPF example on Raspberry Pi 4
After packaging the eBPF samples for the Gentoo distribution and building the Raspberry Pi 4 firmware using the Replica.one system, we can move onto booting the firmware on live hardware. For this demonstration, we will use a simple USB-to-serial bridge to interact with the machine. The initial boot prints the following boot information:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] Linux version 5.10.52 (root@buildkitsandbox) (aarch64-unknown-
linux-gnu-gcc (Gentoo 10.3.0-r2 p3) 10.3.0, GNU ld (Gentoo 2.35.2 p1)
2.35.2) #1 SMP
[ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.4
[ 0.000000] Reserved memory: created CMA memory pool at 0x000000001ec00000,
size 256 MiB
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id
shared-dma-pool
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000000000-0x000000003fffffff]
[ 0.000000] DMA32 [mem 0x0000000040000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x00000001ffffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000003b3fffff]
[ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x00000001ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
[ 0.000000] percpu: Embedded 30 pages/cpu s84760 r8192 d29928 u122880
[ 0.000000] Detected PIPT I-cache on CPU0
[ 0.000000] CPU features: detected: Spectre-v2
[ 0.000000] CPU features: detected: Spectre-v4
[ 0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 2028544
[...]
[ 6.277357 ] Waiting for root device /dev/mmcblk0p2...
[ 6.286606 ] random: fast init done
[ 6.296350 ] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[ 6.325722 ] mmc1: new high speed SDIO card at address 0001
[ 6.344165 ] mmc0: new ultra high speed DDR50 SDHC card at address aaaa
[ 6.351127 ] mmcblk0: mmc0:aaaa SD32G 29.7 GiB
[ 6.357056 ] mmcblk0: p1 p2
[ 6.416103 ] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode.
Opts: (null)
[ 6.424326 ] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 6.431672 ] devtmpfs: mounted
[ 6.437257 ] Freeing unused kernel memory: 2944K
[ 6.446378 ] Run /sbin/init as init process
[ 6.478963 ] usb 1-1: New USB device found, idVendor=2109, idProduct=3431,
bcdDevice= 4.21
[ 6.487178 ] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 6.494319 ] usb 1-1: Product: USB2.0 Hub
[ 6.499689 ] hub 1-1:1.0: USB hub found
[ 6.503696 ] hub 1-1:1.0: 4 ports detected
[ 6.707586 ] systemd[1]: System time before build time, advancing clock.
[ 6.782364 ] NET: Registered protocol family 10
[ 6.807162 ] Segment Routing with IPv6
[ 6.824092 ] systemd[1]: systemd 248 running in system mode. (-PAM -AUDIT
-SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL -ACL +BLKID
-CURL -ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD -LIBCRYPTS
UP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE -BZIP2 +LZ4 -XZ -ZLIB
+ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 6.854412 ] systemd[1]: Detected architecture arm64.
[ 6.859453 ] systemd[1]: Detected first boot.
[ 6.864642 ] systemd[1]: No hostname configured, using default hostname.
[ 6.871371 ] systemd[1]: Hostname set to.
[...]
[ 10.354596 ] random: crng init done
[ 10.357992 ] random: 7 urandom warning(s) missed due to ratelimiting
[ 10.434220 ] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
This is localhost.unknown_domain (Linux aarch64 5.10.52) 14:58:38
localhost login:
The login username is "root", and the password is "replica". Once you log in, verify that the eBPF samples are installed onto the root file system and that they have been correctly cross-compiled for the aarch64/arm64 architecture using file(1):
This is localhost.unknown_domain (Linux aarch64 5.10.52) 15:37:26
localhost login: root
Password:
Last login: Fri Jul 23 15:37:13 +0000 2021 on /dev/ttyAMA0.
root@localhost ~ # file /usr/bin/{hello,maps} /usr/share/{hello,maps}.bpf.o
/usr/include/vmlinux.h
/usr/bin/hello: ELF 64-bit LSB pie executable, ARM aarch64, version 1
(SYSV),dynamically linked, interpreter /lib/ld-linux-aarch64.so.1,
for GNU/Linux 3.7.0, not stripped
/usr/bin/maps: ELF 64-bit LSB pie executable, ARM aarch64, version 1
(SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1,
for GNU/Linux 3.7.0, not stripped
/usr/share/hello.bpf.o: ELF 64-bit LSB relocatable, eBPF, version 1 (SYSV),
with debug_info, not stripped
/usr/share/maps.bpf.o: ELF 64-bit LSB relocatable, eBPF, version 1 (SYSV),
with debug_info, not stripped
/usr/include/vmlinux.h: cannot open `/usr/include/vmlinux.h' (No such file
or directory)
We see that the vmlinux header file has not been installed, which we expected given that the vmlinux USE flag in the ebuild was specified not to be enabled by default (even though CMake does enable it by default). To test the functionality of the eBPF samples, we invoke a timed action for each functionality:
root@localhost ~ # (sleep 3s && touch /tmp/test) & /usr/bin/hello
[1] 362
libbpf: elf: skipping unrecognized data section(4) .rodata.str1.1
bash-365 [002] d... 2818.921612: bpf_trace_printk: Hello world!
^C
[1]+ Done ( sleep 3s && touch /tmp/test )
The above output confirms that the hello eBPF program functions correctly; we have launched a background shell job that waits for 3 seconds before invoking the touch command. Since the shell uses execve(2) syscall when spawning each new process, this has resulted in output from the program. Similarly, we can confirm the functionality of the maps eBPF program:
root@localhost ~ # (sleep 3s && touch /tmp/test) & /usr/bin/maps
[1] 372
printing executed commands
Process Name = bash, uid = 0, pid = 375
^C
[1]+ Done ( sleep 3s && touch /tmp/test )
root@localhost ~ #
As we can see, the program correctly identified a new process spawned by the shell interpreter, which means that the eBPF portion of the program works correctly.
Conclusion
In this post, we have introduced the concept of embedded eBPF development and walked you through the process of packaging, building, and testing eBPF on an embedded platform using our Replica.one build system.
Sartura offers a range of eBPF development, integration and education services, including custom eBPF application development and eBPF training and support. To find out more about our eBPF services, contact us at info@sartura.hr.
Want to keep in touch with the rest of this blog series? Consider subscribing to our newsletter below.