Linux kernel development
Introduction
This is following on from a talk I really enjoyed on how to create a linux kernel module using rust, but the presenter ran out of time. Please watch that video if you want more background on rust, why it's desirable in the kernel, and how a kernel module works differently from a normal binary.
We'll be working off jackos/linux which is a fork from Rust-for-Linux/linux, which itself forks from torvalds/linux
Raise a pull request or issue for any problems you have with this tutorial at: jackos/jackos.io.
Virtualization
You'll need to enable virtualization on your CPU in the bios, the steps to take are different depending on your motherboard and CPU, it may be called SVM, AMD-V, Intel Virtualization etc. Enable one of those options if you can find them, otherwise google something similar to virtualization amd asus or virtualization intel gigabyte
Dependencies
Choose an option below and follow the steps, the docker containers are over 6gb, so you may want to install everything natively if you have internet bandwidth limits. Alternatively you can create your own Dockerfile from the examples here
# Download and run an arch linux version of the docker container
docker run -it jackosio/rust-linux:arch
# Download and run an ubuntu version of the docker container
docker run -it jackosio/rust-linux:latest
# Install required packages
sudo pacman -Syuu --noconfirm bc bison curl clang diffutils flex git gcc llvm libelf lld ncurses make qemu-system-x86
# Save these to your ~/.bashrc or similar and start a new terminal session
export PATH="/root/.cargo/bin:${PATH}"
export MAKEFLAGS="-j16"
export LLVM="1"
# If you don't have rustup installed
curl https://sh.rustup.rs -sSf | bash -s -- -y
# Install the bindgen version required by the project
git clone https://github.com/rust-lang/rust-bindgen -b v0.56.0 --depth=1
cargo install --path rust-bindgen
# Clone the `Rust for Linux` repo
git clone https://github.com/jackos/linux -b tutorial-start --depth=1
cd linux
# Set your rustc version to the current version being used with Rust for Linux
rustup override set $(scripts/min-tool-version.sh rustc)
rustup component add rust-src
# Do an initial minimal build to make sure everything is working
make allnoconfig qemu-busybox-min.config rust.config
make
# Install required packages
sudo apt update
sudo apt install -y bc bison curl clang fish flex git gcc libclang-dev libelf-dev lld llvm-dev libncurses-dev make neovim qemu-system-x86
# Save these to your ~/.bashrc or similar and start a new terminal session
export PATH="/root/.cargo/bin:${PATH}"
export MAKEFLAGS="-j16"
export LLVM="1"
# If you don't have rustup installed
curl https://sh.rustup.rs -sSf | bash -s -- -y
# Install the bindgen version required by the project
git clone https://github.com/rust-lang/rust-bindgen -b v0.56.0 --depth=1
cargo install --path rust-bindgen
# Clone the `Rust for Linux` repo
git clone https://github.com/jackos/linux -b tutorial-start --depth=1
cd linux
# Set your rustc version to the current version being used with Rust for Linux
rustup override set $(scripts/min-tool-version.sh rustc)
rustup component add rust-src
# Do an initial minimal build to make sure everything is working
make allnoconfig qemu-busybox-min.config rust.config
make
IDE
If you're using vscode and docker you can connect into the docker container using the Remote Development extension, and install rust-analyzer after connecting to it. We'll add rust-analyzer support in a later step which will work with any editor supporting lsp such as neovim and helix.
Adding the Rust module
The module we'll be creating is called VDev short for Virtual Device, we'll add it to the Kconfig, so the Makefile configuration can find it:
linux/samples/rust/Kconfig
config SAMPLE_RUST_VDEV
tristate "Virtual Device"
help
This option builds the Rust virtual device module sample.
To compile this as a module, choose M here:
the module will be called rust_vdev.
If unsure, say N.
We also to specify where the Makefile can find the object file:
linux/samples/rust/Makefile
obj-$(CONFIG_SAMPLE_RUST_VDEV) += rust_vdev.o
Now let's create a new file and write a minimal module:
linux/samples/rust/rust_vdev.rs
//! Virtual Device Module
use kernel::prelude::*;
module! {
type: VDev,
name: b"vdev",
license: b"GPL",
}
struct VDev;
impl kernel::Module for VDev {
fn init(_name: &'static CStr, _module: &'static ThisModule) -> Result<Self> {
// Print a banner to make sure our module is working
pr_info!("------------------------\n");
pr_info!("starting virtual device!\n");
pr_info!("------------------------\n");
Ok(VDev)
}
}
The module! macro takes care of all the boilerplate, we'll build and run the VM next to make sure everything is working.
2: module working - file changes
Building and running the Kernel
The following command will bring up a TUI for setting the build configuration interactively, we need to enable our sample module:
make menuconfig
Follow the menu items, checking any boxes as you go with space:
- Kernel Hacking:
enter - Sample kernel code:
space+enter - Rust Samples:
space+enter - Virtual Device:
space+enter - Press exit three times and save config
Note: if you cloned the offical repo and you get an error about initrd.img you can either:
Run make and start the kernel in a VM:
make
qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23
If all went well you should see:
[0.623465] vdev: -----------------------
[0.623629] vdev: initialize vdev module!
[0.677356] vdev: -----------------------
Somewhere in the terminal
Restarting the kernel
If you want to reload on file changes you can initialize a "hello world" repo and run cargo watch with the -s flag:
cargo init .
cargo install cargo-watch
cargo watch -w ./samples/rust/rust_vdev.rs -cs 'make && qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23'
If you just want to run commands normally without a file watch, in the terminal running the qemu virtualization you can turn it off and start it again by running:
poweroff
make
qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23
We can add this to the Makefile to make it easier to run the command:
linux/Makefile
PHONY += rustwatch
rustwatch:
$(Q) cargo watch -w ./samples/rust/rust_vdev.rs -cs 'make && qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23'
PHONY += rustvm
rustvm:
$(Q) make && qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23
Now we can run the commands:
# Rebuild and start the VM
make rustvm
# Start a watch, which will rebuild and start the VM on file changes
make rustwatch
Fix Rust Analyzer
rust-analyzer is a Language Sever Protocol (lsp) implementation that provides features like completions and go to definition, to get it to work with our project run:
make rust-analyzer
This produces a rust-project.json allowing rust-anlyzer to parse a project without a Cargo.toml, we need to do this because rustc is being invoked directly by the Makefile.
Now that we have Rust Analyzer working I highly recommend you make use of Go to Definition to see how everything has been implemented. We're not using the std for our core functionality, we're using custom kernel implementations that are suited to the C bindings. E.g. a mutex lock will not return a poison Result because we don't want the whole kernel to panic if a single thread panics.
Register device
All the below changes are on linux/samples/rust/rust_vdev.rs
Add these imports:
use kernel::file::{File, Operations};
use kernel::{miscdev, Module};
Change the VDev struct to allow us to register a device into the /dev/ folder
struct VDev {
_dev: Pin<Box<miscdev::Registration<VDev>>>,
}
Change our Module implementation for VDev, you can see that miscdev::Registration is being called with an argument of vdev, so the device will be named /dev/vdev
impl Module for VDev {
fn init(_name: &'static CStr, _module: &'static ThisModule) -> Result<Self> {
pr_info!("-----------------------\n");
pr_info!("initialize vdev module!\n");
pr_info!("watching for changes...\n");
pr_info!("-----------------------\n");
let reg = miscdev::Registration::new_pinned(fmt!("vdev"), ())?;
Ok(Self { _dev: reg })
}
}
Add the minimal implementation for a device which will print "File was opened" when we perform a cat /dev/vdev
#[vtable]
impl Operations for VDev {
fn open(_context: &(), _file: &File) -> Result {
pr_info!("File was opened\n");
Ok(())
}
}
4: register device - file changes
Implement Read and Write
We're going to allow multiple threads to read and write from a place in memory, so we need a Mutex, we'll use smutext short for simple mutex, a custom kernel mutex that doesn't return a poison Result on lock().
Add the imports
use kernel::io_buffer::{IoBufferReader, IoBufferWriter};
use kernel::sync::smutex::Mutex;
use kernel::sync::{Ref, RefBorrow};
Add a struct representing a Device to hold onto data and track its own number
struct Device {
number: usize,
contents: Mutex<Vec<u8>>,
}
Change the Module implementation so that instead of passing a () to the miscdev::Registration we pass Ref<Device>
impl kernel::Module for VDev {
fn init(_name: &'static CStr, _module: &'static ThisModule) -> Result<Self> {
pr_info!("-----------------------\n");
pr_info!("initialize vdev module!\n");
pr_info!("watching for changes...\n");
pr_info!("-----------------------\n");
let reg = miscdev::Registration::new_pinned(
fmt!("vdev"),
// Add a Ref<Device>
Ref::try_new(Device {
number: 0,
contents: Mutex::new(Vec::<u8>::new()),
})
.unwrap(),
)?;
Ok(Self { _dev: reg })
}
}
Now let's add the correct associated types to the Operations implementation and add the read and write methods:
impl Operations for VDev {
// The data that is passed into the open method
type OpenData = Ref<Device>;
// The data that is returned by running an open method
type Data = Ref<Device>;
fn open(context: &Ref<Device>, _file: &File) -> Result<Ref<Device>> {
pr_info!("File for device {} was opened\n", context.number);
Ok(context.clone())
}
// Read the data contents and write them into the buffer provided
fn read(
data: RefBorrow<'_, Device>,
_file: &File,
writer: &mut impl IoBufferWriter,
offset: u64,
) -> Result<usize> {
pr_info!("File for device {} was read\n", data.number);
let offset = offset.try_into()?;
let vec = data.contents.lock();
let len = core::cmp::min(writer.len(), vec.len().saturating_sub(offset));
writer.write_slice(&vec[offset..][..len])?;
Ok(len)
}
// Read from the buffer and write the data in the contents after locking the mutex
fn write(
data: RefBorrow<'_, Device>,
_file: &File,
reader: &mut impl IoBufferReader,
_offset: u64,
) -> Result<usize> {
pr_info!("File for device {} was written\n", data.number);
let copy = reader.read_all()?;
let len = copy.len();
*data.contents.lock() = copy;
Ok(len)
}
}
5: read and write - file changes
Now this is all set up start the vm, if you set up the make command:
make rustvm
Or if you prefer to just run the commands:
make
qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23
In the terminal that has the VM runnning, run the commands:
echo "wow it works" > /dev/vdev
cat /dev/vdev
If everything is working you should see:
echo "wow it works" > /dev/vdev
[41.498265] vdev: File for device 1 was opened
[41.498564] vdev: File for device 1 was written
cat /dev/vdev
[65.435708] vdev: File for device 1 was opened
[65.436339] vdev: File for device 1 was read
wow it works
[65.436712] vdev: File for device 1 was read
Using kernel parameters
We're now going to set up a kernel parameter which we can change when we start the VM to modify behavior, in this case it'll start more devices
Add the flags import:
use kernel::file::{flags, File, Operations};
Modify the module! macro so that it now contains a parameter, devices will be the name of the parameter which can be accessed from vdev.devices:
module! {
type: VDev,
name: b"vdev",
license: b"GPL",
params: {
devices: u32 {
default: 1,
permissions: 0o644,
description: b"Number of virtual devices",
},
},
}
Let's change the structure of our devices so that it's a vec now:
struct VDev {
_devs: Vec<Pin<Box<miscdev::Registration<VDev>>>>,
}
Update the open method to clear the data if it's opened in write only mode
fn open(context: &Ref<Device>, file: &File) -> Result<Ref<Device>> {
pr_info!("File for device {} was opened\n", context.number);
if file.flags() & flags::O_ACCMODE == flags::O_WRONLY {
context.contents.lock().clear();
}
Ok(context.clone())
}
Update the write method to increase the size of the vec if required instead of allocating new memory
fn write(
data: RefBorrow<'_, Device>,
_file: &File,
reader: &mut impl IoBufferReader,
offset: u64,
) -> Result<usize> {
pr_info!("File for device {} was written\n", data.number);
let offset = offset.try_into()?;
let len = reader.len();
let new_len = len.checked_add(offset).ok_or(EINVAL)?;
let mut vec = data.contents.lock();
if new_len > vec.len() {
vec.try_resize(new_len, 0)?;
}
reader.read_slice(&mut vec[offset..][..len])?;
Ok(len)
}
Update the Module impl for VDev so that the same amount of devices are registered as specified by the kernel param.
impl Module for VDev {
fn init(_name: &'static CStr, module: &'static ThisModule) -> Result<Self> {
let count = {
let lock = module.kernel_param_lock();
(*devices.read(&lock)).try_into()?
};
pr_info!("-----------------------\n");
pr_info!("starting {} vdevices!\n", count);
pr_info!("watching for changes...\n");
pr_info!("-----------------------\n");
let mut devs = Vec::try_with_capacity(count)?;
for i in 0..count {
let dev = Ref::try_new(Device {
number: i,
contents: Mutex::new(Vec::new()),
})?;
let reg = miscdev::Registration::new_pinned(fmt!("vdev{i}"), dev)?;
devs.try_push(reg)?;
}
Ok(Self { _devs: devs })
}
}
Now we can change the Makefile adding the argument: -append "vdev.devices=4"
PHONY += rustwatch
rustwatch:
$(Q) cargo watch -w ./samples/rust/rust_vdev.rs -cs 'make && qemu-system-x86_64 -append "vdev.devices=4" -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23'
PHONY += rustvm
rustvm:
$(Q) make && qemu-system-x86_64 -append "vdev.devices=4" -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23
And then running rustvm or rustwatch
Or if you want to run the commands directly:
make
qemu-system-x86_64 -append "vdev.devices=4" -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23
Repo areas of interest
Now that you have a general idea of how to write your own module, take a look around in the repo, some areas of interest are:
linux/rustlinux/rust/kernellinux/Documentation/rust
And don't forget to have a look through all the samples and play around with them if you're interested:
linux/samples/rust
You can activate whichever ones you want with make menuconfig as before
That's it, thanks for reading, and please don't hesitate to raise an issue at: github:jackos/jackos.io if you have any suggestions or problems with this content.
I look forward to seeing your pull requests in the linux kernel!