Contents
Overview
These are some notes for how to mount a RaspberryPi disk image, and use qemu-user-static to modify the image. For this example, RaspiOS but can be used/modified in general for any SBC using RaspberryPiImages.
We'll mount the disk image, chroot into it, and use QemuUserEmulation to update the image and execute the ARM code.
Setting up the host
Install the qemu-user-static and binfmt-support packages.
sudo apt-get install -y qemu qemu-user-static binfmt-support
The metapackage qemu-user-static provides user mode emulation binaries, built statically. In this mode QEMU can launch Linux processes compiled for one CPU on another CPU. If binfmt-support is installed, qemu-user-static package will register binary formats to run foreign binaries directly. After installed, you can check your ability to emulate the binary formats by checking for ARM support by with binfmt-support.
sudo update-binfmts --display
Note: If the host system is Debian Buster and the chroot will be Debian Bullseye, it is good to make sure the host has the version 5.2 of qemu-user-static, available from buster-backports. Otherwise some binaries may cause qemu-user-static to Seg Fault, see 987497. For example setting locale(s) in chroot fails without the upgrade of qemu-user-static.
Using an image file
For example, from the Raspberry Pi Foundation or directly from the downloads index page. You can choose from a few listed that will allow you to easily drop in the qemu-user-static binary after decompressing and adding a little extra space to work in.
mkdir ~/rpi cd ~/rpi wget https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2021-05-28/2021-05-07-raspios-buster-arm64.zip
Inflate the image (e.g., 2020-08-20-raspios-buster-arm64.img) with unzip.
sudo apt install -y unzip unzip 2020-08-20-raspios-buster-arm64.zip rm 2020-08-20-raspios-buster-arm64.zip
Resizing the operating system partition
After unzipping, you may want to increase the size of the disk image so it is more useful.
First, check out your disk image.
fdisk -lu 2020-08-20-raspios-buster-arm64.img Disk 2020-08-20-raspios-buster-arm64.img: 3.52 GiB, 3779067904 bytes, 7380992 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xad09722e Device Boot Start End Sectors Size Id Type 2020-08-20-raspios-buster-arm64.img1 8192 532479 524288 256M c W95 FA 2020-08-20-raspios-buster-arm64.img2 532480 7380991 6848512 3.3G 83 Linux
Add space with the dd command
The first partition is the boot partition (kernel and binary blobs), the second is the filesystem. We want to add space to the disk image, and expand the second partition. Add space to the image with the dd command. This example adds 1G.
sudo dd if=/dev/zero bs=1M count=1024 >> 2020-08-20-raspios-buster-arm64.img 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.7655 s, 91.3 MB/s
Use fdisk to display information
Now fdisk -lu will tell you it is 4.52 GiB, but the .img2 partition is 3.3G.
fdisk -lu 2020-08-20-raspios-buster-arm64.img Disk 2020-08-20-raspios-buster-arm64.img: 4.52 GiB, 4852809728 bytes, 9478144 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xad09722e Device Boot Start End Sectors Size Id Type 2020-08-20-raspios-buster-arm64.img1 8192 532479 524288 256M c W95 FA 2020-08-20-raspios-buster-arm64.img2 532480 7380991 6848512 3.3G 83 Linux
To fix this we will resize the partition to fill the rest of the image.
Start by creating a loopback device.
sudo losetup -f -P --show 2020-08-20-raspios-buster-arm64.img
This should set up /dev/loop0 as the whole image and /dev/loop0p2 as the partition we're expanding, but if you have other loop devices it will pick the next unused loop device drive assignment. The whole partition is displayed as /dev/loop0.
fdisk -lu /dev/loop0 Disk /dev/loop0: 4.52 GiB, 4852809728 bytes, 9478144 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xad09722e Device Boot Start End Sectors Size Id Type /dev/loop0p1 8192 532479 524288 256M c W95 FAT32 (LBA) /dev/loop0p2 532480 7380991 6848512 3.3G 83 Linux
The individual partition is /dev/loop0p1 the boot partition. The operating system partition is /dev/loop0p2.
fdisk -lu /dev/loop0p2 Disk /dev/loop0p2: 3.27 GiB, 3506438144 bytes, 6848512 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 b
Use parted to resize the partition
In parted, resize the second partition to be the full size of /dev/loop0 as given on the line starting with Disk in the parted header output. NB: your numbers may be different than here.
sudo parted /dev/loop0 GNU Parted 3.3 Using /dev/loop0 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) print Model: Loopback device (loopback) Disk /dev/loop0: 4853MB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 4194kB 273MB 268MB primary fat32 lba 2 273MB 3779MB 3506MB primary ext4 (parted) resizepart 4853MB (parted) print Model: Loopback device (loopback) Disk /dev/loop0: 4853MB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 4194kB 273MB 268MB primary fat32 lba 2 273MB 4853MB 4580MB primary lba (parted) quit Information: You may need to update /etc/fstab.
Unless this partition will have an fstab entry, ignore the message to update /etc/fstab.
Check the filesystem with e2fsck
Check the filesystem within the new partition:
e2fsck -f /dev/loop0p2 e2fsck 1.45.6 (20-Mar-2020) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information rootfs: 103288/214272 files (0.3% non-contiguous), 709633/856064 blocks
Expand the filesystem with resize2fs
We already resized the partition. Now resize (expand) the filesystem.
resize2fs /dev/loop0p2 resize2fs 1.45.6 (20-Mar-2020) Resizing the filesystem on /dev/loop0p2 to 1118208 (4k) blocks. The filesystem on /dev/loop0p2 is now 1118208 (4k) blocks long.
And you can check that it worked, it's 1 GB larger!
sudo parted /dev/loop0 GNU Parted 3.3 Using /dev/loop0 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) print Model: Loopback device (loopback) Disk /dev/loop0: 4853MB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 4194kB 273MB 268MB primary fat32 lba 2 273MB 4853MB 4580MB primary ext4 (parted)quit
Mounting the image
Mount the image with root permissions:
sudo mount /dev/loop0p2 -o rw /mnt
Optional: If you want to mount the RaspiOS /boot
sudo mount /dev/loop0p1 -o rw /mnt/boot
Also mount necessary host directories.
cd /mnt sudo mount --bind /dev dev/ sudo mount --bind /sys sys/ sudo mount --bind /proc proc/ sudo mount --bind /dev/pts dev/pts
(not necessary/advised if you use systemd-nspawn)
To get everything work (e.g., network) you need to comment out everything in /mnt/etc/ld.so.preload before you chroot in. Take care of that now!
Creating a Chroot
First, you need to make sure binfmt-support will be able to execute your code once you change your root filesystem. So you'll have to copy your QemuUserEmulation binary to the chroot.
cp /usr/bin/qemu-aarch64-static /mnt/usr/bin
(note, it's the same binary that you see when you do binfmt-support --display and look at the entry for aarch64 or armhf)
Use a different URL for the 32-bit image.
wget https://downloads.raspberrypi.org/raspios_full_armhf/images/raspios_full_armhf-2021-05-28/2021-05-07-raspios-buster-armhf-full.zip
Extract the .img file.
unzip raspios_full_armhf-2021-01-12/2021-01-11-raspios-buster-armhf-full.zip
After mounting the image file in the same manner as the ARM64 image, we use the armhf binary instead.
cp /usr/bin/qemu-arm-static /mnt/usr/bin
Now chroot in!
cd /mnt sudo chroot . bin/bash
Systemd containers
To create a Systemd container use the systemd-nspawn command,
sudo apt install -y systemd-container sudo systemd-nspawn -D /mnt bin/bash
because it does a better job of isolating the chroot environment from your host system.
If you have an issue with Internet connectivity using a Systemd container, exit and remove /mnt/etc/resolv.conf. Systemd will recreate the resolv.conf when your run systemd-nspawn again.
When you are done modifying your image, remember to clean up the loopback devices.
sudo losetup -d /dev/loop0
If you do not run raspi-config to configure locales and time zone data, you may find yourself on a fastly instance trying to upgrade with Apt.
Using other images
If you choose an image other than RaspiOS from that index, you should be advised you cannot create a Systemd container if either the host or guest distro release came before Systemd was implemented. For Debian this began with Jessie, and for Ubuntu 15.04 Vivid using Upstart.
OSMC must be installed before you can use the qemu-user-static binary.
- Take the OSMC microSD card out of the Pi
- Insert it into your computer
- Mount it and containerize it
For RaspiOS you can do the same thing after it is installed on the microSD card.
You do not have to create a loop device for a microSD card, just mount the partition.
sudo mount /dev/sdd2 /mnt
To mount the boot partition is:
sudo mount /dev/sdd1 /mnt/boot
Usually it is /dev/sdd ,/dev/sde, and after that /dev/sdf, etc.
Manjaro images for ARM you can run off the microSD card with the qemu-aarch64-static Qemu binary for ARM64 in this manner as well.
cp /usr/bin/qemu-aarch64-static /mnt/usr/bin/
Modifying the image
Play around!
Remove the desktop environment
For example, make it a slim server:
sudo apt-get remove -y --dry-run --auto-remove --purge libx11-.*
Make sure the package list is sane, then run again without --dry-run.
Upgrade to a new release
Upgrade to newer release (e.g., stretch, buster whatever is testing at the moment). You probably should have /boot mounted, as the kernel may be updated as well.
sudo sed -i 's/buster/bullseye/g' /etc/apt/sources.list #ALSO CHECK ALL FILES IN /etc/apt/sources.list.d TO SEE IF THEY NEED TO BE UPDATED sudo apt-get update sudo apt-get dist-upgrade -y -o Dpkg::Options::="--force-confold"
Cleaning up
If you want to flash this to an SD card, remember to clean up!
Uncomment /etc/ld.so.preload
Exit the chroot (e.g., type exit)
- Unmount all that was mounted
umount /mnt/boot umount /mnt/proc umount /mnt/sys umount /mnt/dev/pts umount /mnt/dev/ umount /mnt
Your image is still in ~/rpi and ready to be flashed! (e.g., example instructions)
References
https://wiki.debian.org/QemuUserEmulation
http://hblok.net/blog/posts/2014/02/06/chroot-to-arm/
https://github.com/cleverca22/crosspiroot
http://linuxconfig.org/raspbian-gnu-linux-upgrade-from-wheezy-to-raspbian-jessie-8
http://sirlagz.net/2012/06/20/how-to-resize-partitions-on-an-image-file/