Raspberry Pi Emulation Using qemu-user-static

These are some notes for how to mount a Raspberry Pi disk image, and use qemu-user-static to modify the image. For this example, I will show Raspbian but can be used/modified in general for any SBC or Raspberry Pi image.

Overview

We'll mount the disk image, chroot in to it, then use QemuUserEmulation to update the image and execute the ARM code.

Set up Host

Install the qemu-user-static and binfmt-support packages.

apt-get install qemu qemu-user-static binfmt-support

"[qemu-user-static] provides the user mode emulation binaries, built statically. In this mode QEMU can launch Linux processes compiled for one CPU on another CPU... If binfmt-support package is installed, qemu-user-static package will register binary formats which the provided emulators can handle, so that it will be possible to run foreign binaries directly."

After installed, you can check your ability to emulate the binary formats by checking for ARM support by executing:

update-binfmts --display

Get and mount the image

Download the latest raspbian image

For example, from the Raspberry Pi Foundation

mkdir ~/rpi_image
cd ~/rpi_image
wget https://downloads.raspberrypi.org/raspbian/images/raspbian-2015-11-24/2015-11-21-raspbian-jessie.zip

Inflate the image (e.g., 2015-11-21-raspbian-jessie.img)

unzip 2015-11-21-raspbian-jessie.zip
rm 2015-11-21-raspbian-jessie.zip

Resize the image

After unzipping, you may want to increase the size of the disk image so it is more useful.

First check out your disk image:

$ fdisk -lu 2015-11-21-raspbian-jessie.img

Disk 2015-11-21-raspbian-jessie.img: 3.1 GiB, 3276800000 bytes, 6400000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa6202af7

Device                          Boot  Start     End Sectors Size Id Type
2015-11-21-raspbian-jessie.img1        8192  122879  114688  56M  c W95 FAT32 (L
2015-11-21-raspbian-jessie.img2      122880 6399999 6277120   3G 83 Linux

The first partition is boot (kernel and binary blobs), the second is the filesystem. So we want to add space to the disk image, then expand that second partition.

Add space to the image (this example adds 1G):

dd if=/dev/zero bs=1M count=1024 >> 2015-11-21-raspbian-jessie.img

Make a couple of loopback devices for the whole image and its partitions:

losetup -f -P --show 2015-11-21-raspbian-jessie.img

This should set up /dev/loop0 as the whole image and /dev/loop0p2 as the partition we're expanding

Now /dev/loop0 is the whole partition, /dev/loop0p2 is what we want to expand. In parted, remove the second partition, resize it to be the full size of /dev/loop0:

$ sudo parted /dev/loop0
GNU Parted 3.2
Using /dev/loop0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: Loopback device (loopback)
Disk /dev/loop0: 4351MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  62.9MB  58.7MB  primary  fat16        lba
 2      62.9MB  3277MB  3214MB  primary  ext4

(parted) rm 2                                                             
(parted) mkpart primary 62.9 4351                                         
(parted) print                                                            
Model: Loopback device (loopback)
Disk /dev/loop0: 4351MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  62.9MB  58.7MB  primary  fat16        lba
 2      62.9MB  4351MB  4288MB  primary               lba

Next, check and resize the new partition:

$ e2fsck -f /dev/loop0p2
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop2: 86233/261632 files (0.1% non-contiguous), 634636/1046784 blocks

$ resize2fs /dev/loop0p2
resize2fs 1.42.12 (29-Aug-2014)
Resizing the filesystem on /dev/loop2 to 1046784 (4k) blocks.
The filesystem on /dev/loop2 is now 1046784 (4k) blocks long.

And you can check that it worked, it's 1 GB larger!

parted /dev/loop0
GNU Parted 3.2
Using /dev/loop0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: Loopback device (loopback)
Disk /dev/loop0: 4351MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  62.9MB  58.7MB  primary  fat16        lba
 2      62.9MB  4351MB  4288MB  primary  ext4

Finally, you can clean up the loopback devices:

$ losetup -d /dev/loop0

Mount the image

With root permissions:

mkdir ~/rpi_mnt
losetup --show 2015-11-21-raspbian-jessie.img
mount /dev/loop0p2 -o rw ~/rpi_mnt

OPTIONAL: If you want to mount the raspbian /boot

mount /dev/loop0p1 -o rw ~/rpi_mnt/boot

and you can also:

cd ~/rpi_mnt
mount --bind /dev dev/
mount --bind /sys sys/
mount --bind /proc proc/
mount --bind /dev/pts dev/pts

(not necessary/advised if you use systemd-nspawn)

One thing to fix...

To get everything work (e.g., network) you need to comment out everything in ~/rpi_mnt/etc/ld.so.preload before chrooting in. Take care of that now!

chroot in to the image

First, you need to make sure binfmt-support will be able to execute your code once you change your root filesystem. So you'll have to copy your QemuUserEmulation binary to the chroot

cp /usr/bin/qemu-arm-static ~/rpi_mnt/usr/bin

(note, it's the same binary that you see when you do binfmt-support --display and look at the entry for ARM)

Now chroot in!

cd ~/rpi_mnt
chroot . bin/bash

If you use systemd, best practice to use systemd-nspawn:

systemd-nspawn -D ~/rpi_mnt bin/bash

because it does a better job of isolating the chroot environment from your host system.

Success!

$ uname -a
Linux HOSTNAME 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 18:31:11 UTC 2015 armv7l GNU/Linux

Use it!

Play around!

Remove Desktop Environment

For example, make it a slim server:

apt-get remove --dry-run --auto-remove --purge libx11-.*

Make sure the package list is sane, then run again without "--dry-run"

Upgrade to New Release

Upgrade to newer release (e.g., jessie, stretch, whatever is testing at the moment). You probably should have /boot mounted, as the kernel may be updated as well

sed -i 's/wheezy/jessie/g' /etc/apt/sources.list
#ALSO CHECK ALL FILES IN /etc/apt/sources.list.d TO SEE IF THEY NEED TO BE UPDATED
apt-get update
apt-get dist-upgrade -o Dpkg::Options::="--force-confold"

You most likely will want to keep the edited configuration files when given the option to keep the current or go with the maintainer's. I don't think Raspbian has gone through the effort to merge their changes yet.

Cleaning up

If you want to flash this to an SD card, remember to clean up

  1. uncomment /etc/ld.so.preload
  2. exit the chroot (e.g., type "exit")
  3. unmount all that was mounted
     umount ~/rpi_mnt/dev
     umount ~/rpi_mnt/sys
     umount ~/rpi_mnt/proc
     umount ~/rpi_mnt/dev/pts
     umount ~/rpi_mnt/boot
     umount ~/rpi_mnt
  4. your image is still in ~/rpi_image and ready to be flashed! (e.g., example instructions)

References

https://wiki.debian.org/QemuUserEmulation

http://hblok.net/blog/posts/2014/02/06/chroot-to-arm/

https://github.com/cleverca22/crosspiroot

http://linuxconfig.org/raspbian-gnu-linux-upgrade-from-wheezy-to-raspbian-jessie-8

http://sirlagz.net/2012/06/20/how-to-resize-partitions-on-an-image-file/