Linux kernel development

Introduction

This is following on from a talk I really enjoyedopen in new window on how to create a linux kernel module using rust, but the presenter ran out of time. Please watch that video if you want more background on rust, why it's desirable in the kernel, and how a kernel module works differently from a normal binary.

We'll be working off jackos/linuxopen in new window which is a fork from Rust-for-Linux/linuxopen in new window, which itself forks from torvalds/linuxopen in new window

Raise a pull requestopen in new window or issueopen in new window for any problems you have with this tutorial at: jackos/jackos.ioopen in new window.

Virtualization

You'll need to enable virtualization on your CPU in the bios, the steps to take are different depending on your motherboard and CPU, it may be called SVM, AMD-V, Intel Virtualization etc. Enable one of those options if you can find them, otherwise google something similar to virtualization amd asus or virtualization intel gigabyte

Dependencies

Choose an option below and follow the steps, the docker containers are over 6gb, so you may want to install everything natively if you have internet bandwidth limits. Alternatively you can create your own Dockerfile from the examples hereopen in new window

# Download and run an arch linux version of the docker container
docker run -it jackosio/rust-linux:arch

# Download and run an ubuntu version of the docker container
docker run -it jackosio/rust-linux:latest

# Install required packages
sudo pacman -Syuu --noconfirm bc bison curl clang diffutils flex git gcc llvm libelf lld ncurses make qemu-system-x86

# Save these to your ~/.bashrc or similar and start a new terminal session 
export PATH="/root/.cargo/bin:${PATH}"
export MAKEFLAGS="-j16"
export LLVM="1"

# If you don't have rustup installed
curl https://sh.rustup.rs -sSf | bash -s -- -y

# Install the bindgen version required by the project
git clone https://github.com/rust-lang/rust-bindgen -b v0.56.0 --depth=1
cargo install --path rust-bindgen

# Clone the `Rust for Linux` repo
git clone https://github.com/jackos/linux -b tutorial-start --depth=1
cd linux

# Set your rustc version to the current version being used with Rust for Linux
rustup override set $(scripts/min-tool-version.sh rustc)
rustup component add rust-src

# Do an initial minimal build to make sure everything is working
make allnoconfig qemu-busybox-min.config rust.config
make

# Install required packages
sudo apt update
sudo apt install -y bc bison curl clang fish flex git gcc libclang-dev libelf-dev lld llvm-dev libncurses-dev make neovim qemu-system-x86

# Save these to your ~/.bashrc or similar and start a new terminal session 
export PATH="/root/.cargo/bin:${PATH}"
export MAKEFLAGS="-j16"
export LLVM="1"

# If you don't have rustup installed
curl https://sh.rustup.rs -sSf | bash -s -- -y

# Install the bindgen version required by the project
git clone https://github.com/rust-lang/rust-bindgen -b v0.56.0 --depth=1
cargo install --path rust-bindgen

# Clone the `Rust for Linux` repo
git clone https://github.com/jackos/linux -b tutorial-start --depth=1
cd linux

# Set your rustc version to the current version being used with Rust for Linux
rustup override set $(scripts/min-tool-version.sh rustc)
rustup component add rust-src

# Do an initial minimal build to make sure everything is working
make allnoconfig qemu-busybox-min.config rust.config
make

IDE

If you're using vscode and docker you can connect into the docker container using the Remote Developmentopen in new window extension, and install rust-analyzer after connecting to it. We'll add rust-analyzer support in a later step which will work with any editor supporting lsp such as neovim and helix.

Adding the Rust module

The module we'll be creating is called VDev short for Virtual Device, we'll add it to the Kconfig, so the Makefile configuration can find it:

linux/samples/rust/Kconfig

config SAMPLE_RUST_VDEV
	tristate "Virtual Device"
	help
	  This option builds the Rust virtual device module sample.

	  To compile this as a module, choose M here:
	  the module will be called rust_vdev.

	  If unsure, say N.

We also to specify where the Makefile can find the object file:

linux/samples/rust/Makefile

obj-$(CONFIG_SAMPLE_RUST_VDEV) 			+= rust_vdev.o

Now let's create a new file and write a minimal module:

linux/samples/rust/rust_vdev.rs

//! Virtual Device Module
use kernel::prelude::*;

module! {
    type: VDev,
    name: b"vdev",
    license: b"GPL",
}

struct VDev;

impl kernel::Module for VDev {
    fn init(_name: &'static CStr, _module: &'static ThisModule) -> Result<Self> {
        // Print a banner to make sure our module is working
        pr_info!("------------------------\n");
        pr_info!("starting virtual device!\n");
        pr_info!("------------------------\n");
        Ok(VDev)
    }
}

The module! macro takes care of all the boilerplate, we'll build and run the VM next to make sure everything is working.

2: module working - file changesopen in new window

Building and running the Kernel

The following command will bring up a TUI for setting the build configuration interactively, we need to enable our sample module:

make menuconfig

Follow the menu items, checking any boxes as you go with space:

  • Kernel Hacking: enter
  • Sample kernel code: space + enter
  • Rust Samples: space + enter
  • Virtual Device: space + enter
  • Press exit three times and save config

Note: if you cloned the offical repo and you get an error about initrd.img you can either:

Run make and start the kernel in a VM:

make
qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23

If all went well you should see:

[0.623465] vdev: -----------------------
[0.623629] vdev: initialize vdev module!
[0.677356] vdev: -----------------------

Somewhere in the terminal

Restarting the kernel

If you want to reload on file changes you can initialize a "hello world" repo and run cargo watch with the -s flag:

cargo init .
cargo install cargo-watch
cargo watch -w ./samples/rust/rust_vdev.rs -cs 'make && qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23'

If you just want to run commands normally without a file watch, in the terminal running the qemu virtualization you can turn it off and start it again by running:

poweroff
make
qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23

We can add this to the Makefile to make it easier to run the command:

linux/Makefile

PHONY += rustwatch
rustwatch:
	$(Q) cargo watch -w ./samples/rust/rust_vdev.rs -cs 'make && qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23'

PHONY += rustvm
rustvm:
	$(Q) make && qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23

Now we can run the commands:

# Rebuild and start the VM
make rustvm
# Start a watch, which will rebuild and start the VM on file changes
make rustwatch

3: add watch - file changesopen in new window

Fix Rust Analyzer

rust-analyzer is a Language Sever Protocol (lsp) implementation that provides features like completions and go to definition, to get it to work with our project run:

make rust-analyzer

This produces a rust-project.json allowing rust-anlyzer to parse a project without a Cargo.toml, we need to do this because rustc is being invoked directly by the Makefile.

Now that we have Rust Analyzer working I highly recommend you make use of Go to Definition to see how everything has been implemented. We're not using the std for our core functionality, we're using custom kernel implementations that are suited to the C bindings. E.g. a mutex lock will not return a poison Result because we don't want the whole kernel to panic if a single thread panics.

Register device

All the below changes are on linux/samples/rust/rust_vdev.rs

Add these imports:

use kernel::file::{File, Operations};
use kernel::{miscdev, Module};

Change the VDev struct to allow us to register a device into the /dev/ folder

struct VDev {
    _dev: Pin<Box<miscdev::Registration<VDev>>>,
}

Change our Module implementation for VDev, you can see that miscdev::Registration is being called with an argument of vdev, so the device will be named /dev/vdev

impl Module for VDev {
    fn init(_name: &'static CStr, _module: &'static ThisModule) -> Result<Self> {
        pr_info!("-----------------------\n");
        pr_info!("initialize vdev module!\n");
        pr_info!("watching for changes...\n");
        pr_info!("-----------------------\n");
        let reg = miscdev::Registration::new_pinned(fmt!("vdev"), ())?;
        Ok(Self { _dev: reg })
    }
}

Add the minimal implementation for a device which will print "File was opened" when we perform a cat /dev/vdev

#[vtable]
impl Operations for VDev {
    fn open(_context: &(), _file: &File) -> Result {
        pr_info!("File was opened\n");
        Ok(())
    }
}

4: register device - file changesopen in new window

Implement Read and Write

We're going to allow multiple threads to read and write from a place in memory, so we need a Mutex, we'll use smutext short for simple mutex, a custom kernel mutex that doesn't return a poison Result on lock().

Add the imports

use kernel::io_buffer::{IoBufferReader, IoBufferWriter};
use kernel::sync::smutex::Mutex;
use kernel::sync::{Ref, RefBorrow};

Add a struct representing a Device to hold onto data and track its own number

struct Device {
    number: usize,
    contents: Mutex<Vec<u8>>,
}

Now let's add the correct associated types to the Operations implementation and add the read and write methods:

impl Operations for VDev {
    // The data that is passed into the open method 
    type OpenData = Ref<Device>;
    // The data that is returned by running an open method
    type Data = Ref<Device>;

    fn open(context: &Ref<Device>, _file: &File) -> Result<Ref<Device>> {
        pr_info!("File for device {} was opened\n", context.number);
        Ok(context.clone())
    }

    // Read the data contents and write them into the buffer provided
    fn read(
        data: RefBorrow<'_, Device>,
        _file: &File,
        writer: &mut impl IoBufferWriter,
        offset: u64,
    ) -> Result<usize> {
        pr_info!("File for device {} was read\n", data.number);
        let offset = offset.try_into()?;
        let vec = data.contents.lock();
        let len = core::cmp::min(writer.len(), vec.len().saturating_sub(offset));
        writer.write_slice(&vec[offset..][..len])?;
        Ok(len)
    }

    // Read from the buffer and write the data in the contents after locking the mutex
    fn write(
        data: RefBorrow<'_, Device>,
        _file: &File,
        reader: &mut impl IoBufferReader,
        _offset: u64,
    ) -> Result<usize> {
        pr_info!("File for device {} was written\n", data.number);
        let copy = reader.read_all()?;
        let len = copy.len();
        *data.contents.lock() = copy;
        Ok(len)
    }
}

5: read and write - file changesopen in new window

Now this is all set up start the vm, if you set up the make command:

make rustvm

Or if you prefer to just run the commands:

make
qemu-system-x86_64 -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23

In the terminal that has the VM runnning, run the commands:

echo "wow it works" > /dev/vdev
cat /dev/vdev

If everything is working you should see:

echo "wow it works" > /dev/vdev
[41.498265] vdev: File for device 1 was opened
[41.498564] vdev: File for device 1 was written
cat /dev/vdev
[65.435708] vdev: File for device 1 was opened
[65.436339] vdev: File for device 1 was read
wow it works
[65.436712] vdev: File for device 1 was read

Using kernel parameters

We're now going to set up a kernel parameter which we can change when we start the VM to modify behavior, in this case it'll start more devices

Add the flags import:

use kernel::file::{flags, File, Operations};

Modify the module! macro so that it now contains a parameter, devices will be the name of the parameter which can be accessed from vdev.devices:

module! {
    type: VDev,
    name: b"vdev",
    license: b"GPL",
    params: {
        devices: u32 {
            default: 1,
            permissions: 0o644,
            description: b"Number of virtual devices",
        },
    },
}

Let's change the structure of our devices so that it's a vec now:

struct VDev {
    _devs: Vec<Pin<Box<miscdev::Registration<VDev>>>>,
}

Update the open method to clear the data if it's opened in write only mode

fn open(context: &Ref<Device>, file: &File) -> Result<Ref<Device>> {
    pr_info!("File for device {} was opened\n", context.number);
    if file.flags() & flags::O_ACCMODE == flags::O_WRONLY {
        context.contents.lock().clear();
    }
    Ok(context.clone())
}

Update the write method to increase the size of the vec if required instead of allocating new memory

fn write(
        data: RefBorrow<'_, Device>,
        _file: &File,
        reader: &mut impl IoBufferReader,
        offset: u64,
    ) -> Result<usize> {
        pr_info!("File for device {} was written\n", data.number);
        let offset = offset.try_into()?;
        let len = reader.len();
        let new_len = len.checked_add(offset).ok_or(EINVAL)?;
        let mut vec = data.contents.lock();
        if new_len > vec.len() {
            vec.try_resize(new_len, 0)?;
        }
        reader.read_slice(&mut vec[offset..][..len])?;
        Ok(len)
    }

Update the Module impl for VDev so that the same amount of devices are registered as specified by the kernel param.

impl Module for VDev {
  fn init(_name: &'static CStr, module: &'static ThisModule) -> Result<Self> {
      let count = {
          let lock = module.kernel_param_lock();
          (*devices.read(&lock)).try_into()?
      };
      pr_info!("-----------------------\n");
      pr_info!("starting {} vdevices!\n", count);
      pr_info!("watching for changes...\n");
      pr_info!("-----------------------\n");
      let mut devs = Vec::try_with_capacity(count)?;
      for i in 0..count {
          let dev = Ref::try_new(Device {
              number: i,
              contents: Mutex::new(Vec::new()),
          })?;
          let reg = miscdev::Registration::new_pinned(fmt!("vdev{i}"), dev)?;
          devs.try_push(reg)?;
      }
      Ok(Self { _devs: devs })
    }
}

Now we can change the Makefile adding the argument: -append "vdev.devices=4"

PHONY += rustwatch
rustwatch:
	$(Q) cargo watch -w ./samples/rust/rust_vdev.rs -cs 'make && qemu-system-x86_64 -append "vdev.devices=4" -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23'

PHONY += rustvm
rustvm:
	$(Q) make && qemu-system-x86_64 -append "vdev.devices=4" -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23

And then running rustvm or rustwatch

Or if you want to run the commands directly:

make
qemu-system-x86_64 -append "vdev.devices=4" -nographic -kernel vmlinux -initrd initrd.img -nic user,model=rtl8139,hostfwd=tcp::5555-:23

final fileopen in new window

Repo areas of interest

Now that you have a general idea of how to write your own module, take a look around in the repo, some areas of interest are:

  • linux/rust
  • linux/rust/kernel
  • linux/Documentation/rust

And don't forget to have a look through all the samples and play around with them if you're interested:

  • linux/samples/rust

You can activate whichever ones you want with make menuconfig as before

That's it, thanks for reading, and please don't hesitate to raise an issue at: github:jackos/jackos.ioopen in new window if you have any suggestions or problems with this content.

I look forward to seeing your pull requests in the linux kernel!