Overview

These are some notes for how to mount a RaspberryPi disk image, and use qemu-user-static to modify the image. For this example, RaspiOS but can be used/modified in general for any SBC using RaspberryPiImages.

We'll mount the disk image, chroot into it, and use QemuUserEmulation to update the image and execute the ARM code.

Setting up the host

Install the qemu-user-static and binfmt-support packages.

sudo apt-get install -y qemu qemu-user-static binfmt-support

The metapackage qemu-user-static provides user mode emulation binaries, built statically. In this mode QEMU can launch Linux processes compiled for one CPU on another CPU. If binfmt-support is installed, qemu-user-static package will register binary formats to run foreign binaries directly. After installed, you can check your ability to emulate the binary formats by checking for ARM support by with binfmt-support.

sudo update-binfmts --display

Note: If the host system is Debian Buster and the chroot will be Debian Bullseye, it is good to make sure the host has the version 5.2 of qemu-user-static, available from buster-backports. Otherwise some binaries may cause qemu-user-static to Seg Fault, see 987497. For example setting locale(s) in chroot fails without the upgrade of qemu-user-static.

Using an image file

For example, from the Raspberry Pi Foundation or directly from the downloads index page. You can choose from a few listed that will allow you to easily drop in the qemu-user-static binary after decompressing and adding a little extra space to work in.

mkdir ~/rpi
cd ~/rpi
wget https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2021-05-28/2021-05-07-raspios-buster-arm64.zip

Inflate the image (e.g., 2020-08-20-raspios-buster-arm64.img) with unzip.

sudo apt install -y unzip
unzip 2020-08-20-raspios-buster-arm64.zip
rm 2020-08-20-raspios-buster-arm64.zip

Resizing the operating system partition

After unzipping, you may want to increase the size of the disk image so it is more useful.

First, check out your disk image.

fdisk -lu 2020-08-20-raspios-buster-arm64.img
Disk 2020-08-20-raspios-buster-arm64.img: 3.52 GiB, 3779067904 bytes, 7380992 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xad09722e

Device                               Boot  Start     End Sectors  Size Id Type
2020-08-20-raspios-buster-arm64.img1        8192  532479  524288  256M  c W95 FA
2020-08-20-raspios-buster-arm64.img2      532480 7380991 6848512  3.3G 83 Linux

Add space with the dd command

The first partition is the boot partition (kernel and binary blobs), the second is the filesystem. We want to add space to the disk image, and expand the second partition. Add space to the image with the dd command. This example adds 1G.

sudo dd if=/dev/zero bs=1M count=1024 >> 2020-08-20-raspios-buster-arm64.img
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.7655 s, 91.3 MB/s

Use fdisk to display information

Now fdisk -lu will tell you it is 4.52 GiB, but the .img2 partition is 3.3G.

fdisk -lu 2020-08-20-raspios-buster-arm64.img
Disk 2020-08-20-raspios-buster-arm64.img: 4.52 GiB, 4852809728 bytes, 9478144 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xad09722e

Device                               Boot  Start     End Sectors  Size Id Type
2020-08-20-raspios-buster-arm64.img1        8192  532479  524288  256M  c W95 FA
2020-08-20-raspios-buster-arm64.img2      532480 7380991 6848512  3.3G 83 Linux

To fix this we will resize the partition to fill the rest of the image.

Start by creating a loopback device.

sudo losetup -f -P --show 2020-08-20-raspios-buster-arm64.img

This should set up /dev/loop0 as the whole image and /dev/loop0p2 as the partition we're expanding, but if you have other loop devices it will pick the next unused loop device drive assignment. The whole partition is displayed as /dev/loop0.

fdisk -lu /dev/loop0
Disk /dev/loop0: 4.52 GiB, 4852809728 bytes, 9478144 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xad09722e

Device       Boot  Start     End Sectors  Size Id Type
/dev/loop0p1        8192  532479  524288  256M  c W95 FAT32 (LBA)
/dev/loop0p2      532480 7380991 6848512  3.3G 83 Linux

The individual partition is /dev/loop0p1 the boot partition. The operating system partition is /dev/loop0p2.

fdisk -lu /dev/loop0p2
Disk /dev/loop0p2: 3.27 GiB, 3506438144 bytes, 6848512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 b

Use parted to resize the partition

In parted, resize the second partition to be the full size of /dev/loop0 as given on the line starting with Disk in the parted header output. NB: your numbers may be different than here.

sudo parted /dev/loop0
GNU Parted 3.3
Using /dev/loop0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: Loopback device (loopback)
Disk /dev/loop0: 4853MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  273MB   268MB   primary  fat32        lba
 2      273MB   3779MB  3506MB  primary  ext4

(parted) resizepart 4853MB
(parted) print
Model: Loopback device (loopback)
Disk /dev/loop0: 4853MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  273MB   268MB   primary  fat32        lba
 2      273MB   4853MB  4580MB  primary               lba

(parted) quit
Information: You may need to update /etc/fstab.

Unless this partition will have an fstab entry, ignore the message to update /etc/fstab.

Check the filesystem with e2fsck

Check the filesystem within the new partition:

e2fsck -f /dev/loop0p2
e2fsck 1.45.6 (20-Mar-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: 103288/214272 files (0.3% non-contiguous), 709633/856064 blocks

Expand the filesystem with resize2fs

We already resized the partition. Now resize (expand) the filesystem.

resize2fs /dev/loop0p2
resize2fs 1.45.6 (20-Mar-2020)
Resizing the filesystem on /dev/loop0p2 to 1118208 (4k) blocks.
The filesystem on /dev/loop0p2 is now 1118208 (4k) blocks long.

And you can check that it worked, it's 1 GB larger!

sudo parted /dev/loop0
GNU Parted 3.3
Using /dev/loop0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: Loopback device (loopback)
Disk /dev/loop0: 4853MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  273MB   268MB   primary  fat32        lba
 2      273MB   4853MB  4580MB  primary  ext4

(parted)quit

Mounting the image

Mount the image with root permissions:

sudo mount /dev/loop0p2 -o rw /mnt

Optional: If you want to mount the RaspiOS /boot

sudo mount /dev/loop0p1 -o rw /mnt/boot

Also mount necessary host directories.

cd /mnt
sudo mount --bind /dev dev/
sudo mount --bind /sys sys/
sudo mount --bind /proc proc/
sudo mount --bind /dev/pts dev/pts

(not necessary/advised if you use systemd-nspawn)

To get everything work (e.g., network) you need to comment out everything in /mnt/etc/ld.so.preload before you chroot in. Take care of that now!

Creating a Chroot

First, you need to make sure binfmt-support will be able to execute your code once you change your root filesystem. So you'll have to copy your QemuUserEmulation binary to the chroot.

cp /usr/bin/qemu-aarch64-static /mnt/usr/bin

(note, it's the same binary that you see when you do binfmt-support --display and look at the entry for aarch64 or armhf)

Use a different URL for the 32-bit image.

wget https://downloads.raspberrypi.org/raspios_full_armhf/images/raspios_full_armhf-2021-05-28/2021-05-07-raspios-buster-armhf-full.zip

Extract the .img file.

unzip raspios_full_armhf-2021-01-12/2021-01-11-raspios-buster-armhf-full.zip

After mounting the image file in the same manner as the ARM64 image, we use the armhf binary instead.

cp /usr/bin/qemu-arm-static /mnt/usr/bin

Now chroot in!

cd /mnt
sudo chroot . bin/bash

Systemd containers

To create a Systemd container use the systemd-nspawn command,

sudo apt install -y systemd-container
sudo systemd-nspawn -D /mnt bin/bash

because it does a better job of isolating the chroot environment from your host system.

If you have an issue with Internet connectivity using a Systemd container, exit and remove /mnt/etc/resolv.conf. Systemd will recreate the resolv.conf when your run systemd-nspawn again.

When you are done modifying your image, remember to clean up the loopback devices.

sudo losetup -d /dev/loop0

If you do not run raspi-config to configure locales and time zone data, you may find yourself on a fastly instance trying to upgrade with Apt.

Using other images

If you choose an image other than RaspiOS from that index, you should be advised you cannot create a Systemd container if either the host or guest distro release came before Systemd was implemented. For Debian this began with Jessie, and for Ubuntu 15.04 Vivid using Upstart.

OSMC must be installed before you can use the qemu-user-static binary.

  1. Take the OSMC microSD card out of the Pi
  2. Insert it into your computer
  3. Mount it and containerize it

For RaspiOS you can do the same thing after it is installed on the microSD card.

You do not have to create a loop device for a microSD card, just mount the partition.

sudo mount /dev/sdd2 /mnt

To mount the boot partition is:

sudo mount /dev/sdd1 /mnt/boot

Usually it is /dev/sdd ,/dev/sde, and after that /dev/sdf, etc.

Manjaro images for ARM you can run off the microSD card with the qemu-aarch64-static Qemu binary for ARM64 in this manner as well.

cp /usr/bin/qemu-aarch64-static /mnt/usr/bin/

Modifying the image

Play around!

Remove the desktop environment

For example, make it a slim server:

sudo apt-get remove -y --dry-run --auto-remove --purge libx11-.*

Make sure the package list is sane, then run again without --dry-run.

Upgrade to a new release

Upgrade to newer release (e.g., stretch, buster whatever is testing at the moment). You probably should have /boot mounted, as the kernel may be updated as well.

sudo sed -i 's/buster/bullseye/g' /etc/apt/sources.list
#ALSO CHECK ALL FILES IN /etc/apt/sources.list.d TO SEE IF THEY NEED TO BE UPDATED
sudo apt-get update
sudo apt-get dist-upgrade -y -o Dpkg::Options::="--force-confold"

Cleaning up

If you want to flash this to an SD card, remember to clean up!

  1. Uncomment /etc/ld.so.preload

  2. Exit the chroot (e.g., type exit)

  3. Unmount all that was mounted

umount /mnt/boot
umount /mnt/proc
umount /mnt/sys
umount /mnt/dev/pts
umount /mnt/dev/
umount /mnt

Your image is still in ~/rpi and ready to be flashed! (e.g., example instructions)

References

https://wiki.debian.org/QemuUserEmulation

http://hblok.net/blog/posts/2014/02/06/chroot-to-arm/

https://github.com/cleverca22/crosspiroot

http://linuxconfig.org/raspbian-gnu-linux-upgrade-from-wheezy-to-raspbian-jessie-8

http://sirlagz.net/2012/06/20/how-to-resize-partitions-on-an-image-file/

https://deb.debian.org/