Hello everyone. For a while now, my kernel development loop for BeagleBoard.org boards has looked
like this:

build the kernel on my PC
pull the SD card out of the board
copy the kernel and modules to it using an SD card adapter
put it back, and boot into a separate extlinux.conf entry (keeping the stock kernel as a backup
in case things go wrong).

For changes that only touched a driver, I could just scp the new module over and do the usual
rmmod/insmod.

This works, but it gets tiring quickly. I did try copying everything over the network instead, but
copying the modules with scp was so slow (thousands of small files) that physically moving the SD
card actually felt faster.

Since I have recently started working on some greybus kernel patches, I finally decided to set up
network booting. U-Boot now fetches the kernel, device tree and overlays from my PC over TFTP, and
the root filesystem is an NFS export from my PC. Installing modules is now just
make modules_install into a directory on my own disk. Nothing needs to be copied to the board at
all.

In this post, I will go over the setup and the various things that broke along the way.

Overview

The boot flow looks like this:

U-Boot brings up Ethernet and gets an address over DHCP.
U-Boot downloads a pxelinux.cfg file over TFTP. This uses the same format as extlinux.conf.
U-Boot downloads the kernel, device tree and overlays listed in it, and boots the kernel.
The kernel configures the network itself (ip=dhcp) and mounts its root filesystem over NFS.

My PC runs Fedora Atomic, so some of the paths below are specific to that, but the general setup
applies to any distro.

Host Setup

TFTP Server

I already had tftp-server layered on my system. It is socket activated, so it only runs while a
transfer is actually happening:

sudo systemctl enable --now tftp.socket

By default, it serves /var/lib/tftpboot. I just stuck with the default since well, I did not want
to maintain a separate system service and socket for custom folder. I do remember trying to get it
working as a user service in the past, but was unsuccessful.

NFS Server

The NFS server lives in the host kernel, so nfs-utils needs to be present on the host. The export
goes in /etc/exports:

/var/home/ayush/Public/FTP/beagleplay_debian  192.168.0.0/16(rw,no_root_squash,no_subtree_check,async)

no_root_squash is required. Without it, root on the board gets mapped to nobody, and anything
readable only by root (/etc/sudoers, SSH host keys, etc.) breaks.
async makes things noticeably faster. It can lose data if the host crashes, which is fine for
a development rootfs.

Then enable the server and open the firewall:

sudo systemctl enable --now nfs-server
sudo exportfs -ra
sudo firewall-cmd --permanent --add-service={nfs,rpc-bind,mountd,tftp}
sudo firewall-cmd --reload

sudo exportfs -v is useful to confirm that the options you wrote are actually the ones in effect.

Root Filesystem

I copied the root filesystem from the BeaglePlay Debian image. Ownership and permissions need to be
preserved exactly, so this needs to run as root, with --numeric-ids so that the board’s UIDs don’t
get remapped to users on the host:

sudo losetup -Pf --show beagleplay-debian.img      # prints e.g. /dev/loop0
lsblk /dev/loop0                                   # find the root partition
sudo mount /dev/loop0p3 /mnt
sudo rsync -aHAX --numeric-ids --filter='-x security.selinux' \
    /mnt/ ~/Public/FTP/beagleplay_debian/
sudo umount /mnt
sudo losetup -d /dev/loop0

The --filter bit is needed on SELinux hosts. Files created by rsync automatically get a host
SELinux label, and with -X, rsync tries to remove it to match the source, which SELinux denies.
I still want -X though, since Debian uses extended attributes for file capabilities.

The Debian image does need a few tweaks to boot from NFS:

/etc/fstab: Remove the root entry since root is already mounted by the kernel. I removed the
/boot/firmware entry as well.
bbbio-set-sysconf.service: Beagle’s first boot configuration service hard requires
/boot/firmware, so systemd refuses to boot (Failed to isolate default target) once that mount
is gone. Deleting or masking the service alone is not enough since a target requires it. Find what
pulls it in and remove that dependency:
```
sudo grep -rl 'bbbio-set-sysconf' etc/systemd usr/lib/systemd
```
zram: The image sets up swap on zram, which my kernel did not have. systemd then waits 90
seconds for a /dev/zram0 that never appears. Either enable CONFIG_ZRAM (along with one
compression backend) in the kernel, or disable it in the rootfs with an empty
/etc/systemd/zram-generator.conf.

Finally, to install modules without sudo, I changed the owner of just the modules directory. It
might not be all that useful if you can install as root, but since I am using toolbox containers, I
want to be able to write the modules from inside the rootless container.

sudo chown -R ayush: ~/Public/FTP/beagleplay_debian/lib/modules

Do not do this for the whole rootfs. The board sees your host UID as its own 1000 user, so
sudo stops working (and chown also clears setuid bits).

Kernel Configuration

Since there is no initramfs, everything needed to reach the NFS root must be built in. Anything
built as a module lives on the NFS root, so the kernel can’t load it until root is mounted. I
start from defconfig and make sure the following are =y:

./scripts/config --file "$O/.config" \
    -e NFS_FS -e NFS_V3 -e ROOT_NFS -e IP_PNP -e IP_PNP_DHCP \
    -e TI_K3_AM65_CPSW_NUSS -e TI_DAVINCI_MDIO
make O="$O" ARCH=arm64 olddefconfig

NOTE: olddefconfig quietly turns an option back into =m if one of its dependencies is still a
module, so it is worth grepping .config afterwards. If the boot hangs after IP-Config, one link
in the Ethernet chain is still a module.

Boot Configuration

The pxelinux.cfg/default file in /var/lib/tftpboot:

menu title BeaglePlay netboot
timeout 30
default dev

label dev
    kernel Image.gz
    fdt k3-am625-beagleplay.dtb
    fdtoverlays k3-am625-beagleplay-greybus-uart-node.dtbo
    append console=ttyS2,115200n8 root=/dev/nfs rw nfsroot=192.168.29.54:/var/home/ayush/Public/FTP/beagleplay_debian,vers=3,tcp ip=dhcp

A few things I learned the hard way here:

append must be a single line. U-Boot does not support continuation lines. When I split it,
U-Boot printed Ignoring unknown command for the nfsroot= line, and the kernel panicked with
No working init found.
The keyword is fdtoverlays, not fdtoverlay.
Overlays need an fdt entry. I initially left out fdt, hoping the overlay would be applied
on top of U-Boot’s own device tree. It is not. Without fdt U-Boot skips overlays completely and
boots with its built-in device tree. You can spot this in the boot log since the FDT address will
be near the top of RAM. U-Boot’s built-in device tree also usually lacks __symbols__, so
overlays would not apply to it anyway. Using the DTB from the same kernel tree is the right thing
to do regardless.
vers=3 avoids NFSv4 ID mapping issues for the root mount.

Then, from the U-Boot prompt over serial:

setenv autoload no
dhcp
setenv serverip 192.168.29.54
pxe get
pxe boot

The order is important. dhcp overwrites serverip with whatever the DHCP server sends, which in my
case is the router. So serverip needs to be set after dhcp. Otherwise, you will just see
TFTP timeouts.

NOTE: The missing environment variable: pxeuuid message is harmless. U-Boot just moves on to
the MAC and IP based config file names, and eventually default.

Sadly, the U-Boot build on my board does not have saveenv, so I can’t make this the default boot
command. If you can’t use saveenv either, an alternative is to keep loading the kernel from local
storage using a normal extlinux.conf entry, and only use NFS for root. The append line is the
same, so modules still don’t need to be copied.

The Development Loop

I build the kernel inside a toolbox container, so copying to /var/lib/tftpboot has to happen on
the host. A small shell function in ~/.bashrc takes care of that:

tftpcp() { flatpak-spawn --host cp -- "$@" /var/lib/tftpboot/; }

After that, deploying a new kernel looks like this:

make ARCH=arm64 Image.gz dtbs modules
tftpcp "arch/arm64/boot/Image.gz" "$O/arch/arm64/boot/dts/ti/k3-am625-beagleplay.dtb"
make ARCH=arm64 INSTALL_MOD_PATH=~/Public/FTP/beagleplay_debian modules_install

And then just reboot the board. For driver only changes, rmmod/insmod still works as before,
except now the new module is already on the board the moment modules_install finishes.

Setting a LOCALVERSION is a good idea, so the module directory of the development kernel doesn’t
collide with the stock kernel’s.

Future Plans

Try kexec for even faster iterations, skipping the firmware and U-Boot entirely.

Ending Thoughts

That is all for this post. The setup took an evening of debugging, but going from “pull the SD card
out” to “run make modules_install and reboot” was absolutely worth it. If you are doing kernel
development on any board with Ethernet and U-Boot, I would highly recommend setting this up.

Faster Kernel Development on BeaglePlay with TFTP and NFS Root