Merging several Debian Installation ISOs to a single one
Debian offers all its packages as sets of ISO 9660 installation images, of which the first ISO image is bootable and the others are storage for more packages.
From time to time the wish arises to combine all or a part of such an image set to a single bootable ISO. This article presents a sh script which emerged in the course of bug #1011343 and aims to fulfill that wish.
The script merge_debian_isos uses sudo for possibly mounting and unmounting the participating ISO images. sudo may also be needed for writing directly to a USB stick.
Contents
Obtain the needed free software
You will need at least the script and the program xorriso.
Get script merge_debian_isos
The script is currently part of the free software project which provides xorriso for packing up the ISO 9660 filesystems in the Debian installation images.
Download https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/test/merge_debian_isos
For verification download the .sig file and use gpg --verify like described for the GNU xorriso tarball.
Finally give x-permission to the downloaded file merge_debian_isos.
If you are curious about its help text, run
./merge_debian_isos -help 2>&1 | less
or peek ahead to the section with the text.
Make sure to have xorriso installed
The script relies on xorriso and its command -boot_image any replay. Therefore it needs xorriso version 1.4.2 or younger. The version of package "xorriso" in Debian 9 Stretch is young enough. The one of Debian 8 Jessie is not.
xorriso is available for most contemporary operating systems. Sometimes its package is named after the upstream source project "libisoburn".
Inquire the installed version by a run of:
xorriso -version
If needed get and compile GNU xorriso
If no version of xorriso is available which is young enough, then download the most modern GNU xorriso tarball from e.g. GNU xorriso home page.
For unpacking and compilation see section "Compilation, First Glimpse, Installation" in GNU xorriso's README file. The compilation environment has to provide the vanilla C development tools like: cc, libc plus its C headers, "make", ld and other binutils. Debian users may install meta package "build-essential".
Burning of optical media is supported only on GNU/Linux, Free|Net|OpenBSD, and Solaris.
After successful compilation there is no need for system-wide installation. You may keep the xorriso binary where it emerged and tell merge_debian_isos the path by
export XORRISO="$(pwd)"/xorriso/xorriso
Download the participating Debian installation ISOs
The complete image sets are offered as Jigdo files. For downloading the ISO images you need program jigdo-lite. In Debian it is part of package jigdo-file. See also the Debian CD Jigdo page, the Jigdo home page, and its Download area with source tarball and MS-Windows binaries.
Obtain .jigdo download URLs from Debian pages like the one for the amd64 DVD set.
Create a new directory, go to it, and use jigdo-lite like in this description how to do it on a Debian Live system.
Get image file number 1 of the desired set and as many of the others as you want. They are sorted by popularity of the packages in them. There is no need to get and merge the "debian-update" images which are offered along with the complete image sets.
Do not mix ISOs from different Debian releases or CPU architectures.
Merge the downloaded ISOs
For ease of the following examples, copy the downloaded script merge_debian_isos to the directory with the downloaded ISOs.
Merge the ISOs to a single image file
Let us assume that the three images debian-11.2.0-amd64-DVD-[123].iso have been downloaded and shall be merged. The name chosen for the resulting ISO image shall be "debian-11.2.0-amd64-DVD-1+2+3.iso". So run:
./merge_debian_isos debian-11.2.0-amd64-DVD-1+2+3.iso \
merge_mount/iso \
debian-11.2.0-amd64-DVD-[123].isoThe expectable messages are
merge_debian_isos starting with 3 ISO image files ... Arguments look acceptable. Mounting ISO images if not yet mounted ... Note: sudo mount debian-11.2.0-amd64-DVD-1.iso merge_mount/iso1 [sudo] password for thomas: mount: /dev/loop0 is write-protected, mounting read-only Note: sudo mount debian-11.2.0-amd64-DVD-2.iso merge_mount/iso2 ... Note: sudo mount debian-11.2.0-amd64-DVD-3.iso merge_mount/iso3 ... Copying dists directory and md5sum.txt from first ISO ... ... Determining Debian release in first ISO ... Will work along merge_mount/iso1/dists/bullseye/Release Composing new /README.txt ... ... Merging package description files ... ... Updating dists/bullseye/Release ... ... Merging md5sum.txt files ... ... Producing result ISO image ... Planned as imported package pool : merge_mount/iso1/pool Planned for merging into package pool: merge_mount/iso2/pool Planned for merging into package pool: merge_mount/iso3/pool Planned as imported /firmware : merge_mount/iso1/firmware Planned for merging into /firmware : merge_mount/iso3/firmware Running as xorriso program: ...some.path.../xorriso ... ... messages from xorriso run ... ... Writing to 'debian-11.2.0-amd64-DVD-1+2+3.iso' completed successfully. Run of xorriso program ended without error indication. Cleaning up temporary files and mount points ... Cleanup completed. Merge run ended with success indication.
If the involved disk is slow then it may be that sudo asks again for the password when cleaning up the mount points.
The resulting file "debian-11.2.0-amd64-DVD-1+2+3.iso" may be put onto optical medium or USB stick by the usual means for Debian installation ISOs.
There are several reasons why the script may refuse to work and rather report in the end:
--- Merge run aborted !
The messages before this final one hopefully give enough hints about what went wrong.
Merge the ISOs directly to an optical medium
Insert a writable optical medium into the drive and submit its device file path with prefix "mmc:" to indicate that xorriso shall burn the result_iso directly to it.
Assumed that the drive address is /dev/sr0 with a blank BD-R medium or a BD-RE with non-prescious content:
./merge_debian_isos mmc:/dev/sr0 \
merge_mount/iso \
debian-11.2.0-amd64-DVD-[123].isoThe main difference to above run will be an early message
merge_debian_isos starting with 3 ISO image files ... xorriso accepts 'mmc:/dev/sr0' as optical drive. Arguments look acceptable.
and probably a longer xorriso run than with a data file as result_iso. This increases the chance for a second password inquiry by sudo when the time for unmounting the participating ISOs has come.
Merge the ISOs directly to a USB stick on GNU/Linux
This usually demands superuser authority and thus can overwrite any disk device. One should take additional safety measures.
merge_debian_isos can cooperate with script xorriso-dd-target in order to restrict writing to disk-like devices which are USB attached or memory cards and bear nothing but FAT or ISO 9660 filesystems. xorriso-dd-target depends on lsblk and thus works only on GNU/Linux.
This shell script is available as Debian package since Debian 12 Bookworm. On older Debian or other GNU/Linux systems download it from https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/xorriso-dd-target/xorriso-dd-target
For gpg --verify download the .sig file.
Finally give x-permission to the downloaded file xorriso-dd-target and tell merge_debian_isos the directory path where it is stored. Like
export XORRISO_DD_TARGET_PATH="$(pwd)"
Run the script with pseudo result path "xorriso-dd-target-plug-test":
./merge_debian_isos xorriso-dd-target-plug-test \
merge_mount/iso \
debian-11.2.0-amd64-DVD-[123].isoThe path of the device file gets determined by a two-step plug dialog and a final confirmation input:
--------------------------------------------------------- Testing sudo to possibly get password prompting done now: [sudo] password for thomas: sudo /bin/lsblk seems ok. Caused by option -plug_test: Attempt to find the desired device by watching it appear after being plugged in. Step 1: Please make sure that the desired target device is plugged _out_ now. If it is currently plugged in, make sure to unmount all its fileystems and then unplug it. Press the Enter key when ready. Found and noted as _not_ desired: sda sdb sdc Step 2: Please plug in the desired target device and then press the Enter key. Waiting up to 10 seconds for a new device to be listed ... found: sdd Now waiting 5 seconds to let it settle ......... Found and noted as desired device: sdd sdd : YES : usb+ has_iso9660+ has_vfat+ : Intenso Ultra Line --------------------------------------------------------- xorriso-dd-target agrees that this is a suitable device. If you agree to xorriso running under sudo and overwriting the content of '/dev/sdd', then enter 'yes' : yes Will use sudo with xorriso and write to device. Actual target device path is '/dev/sdd' Arguments look acceptable.
The other messages are as in the example above.
If plugging the device with above dialog is not feasible, then you may give up some safety by naming the device directly. This still depends on xorriso-dd-target agreeing to the device connection and content. The only dialog is that the user has to confirm.
./merge_debian_isos /dev/sdd \
merge_mount/iso \
debian-11.2.0-amd64-DVD-[123].isoyields instead of above plug-test dialog:
merge_debian_isos starting with 3 ISO image files ... The result_iso path '/dev/sdd' is an existing File in /dev. Will only write to it if xorriso-dd-target agrees. xorriso-dd-target is /usr/bin/xorriso-dd-target Performing: xorriso-dd-target -with_sudo sdd --------------------------------------------------------- Testing sudo to possibly get password prompting done now: [sudo] password for thomas: sudo /bin/lsblk seems ok. sdd : YES : usb+ has_iso9660+ has_vfat+ --------------------------------------------------------- xorriso-dd-target agrees that this is a suitable device. If you agree to xorriso running under sudo and overwriting the content of '/dev/sdd', then enter 'yes' : yes Will use sudo with xorriso and write to device. Arguments look acceptable.
Dangerous last resort for merging directly to disk device
On operating systems other than GNU/Linux or with storage devices not acceptable to xorriso-dd-target it is possible to remove all safety precautions beyond those of xorriso, which can be overcome by "blanking" the device.
The user is responsible to name the correct device file and to have the necessary permissions to overwrite the device. If the device contains non-zero data in its first 64 KiB it is likely that it has to be blanked by xorriso before it is willing to write an ISO 9660 filesystem to it.
So after due evaluation of the device situation and on your very own risk you may use the device path prefix "stdio:", possibly with superuser powers:
xorriso -outdev stdio:/dev/sdd -blank as_needed
./merge_debian_isos stdio:/dev/sdd \
merge_mount/iso \
debian-11.2.0-amd64-DVD-[123].isoAfter the messages of the blanking run, merge_debian_isos will report:
merge_debian_isos starting with 3 ISO image files ...
WARNING: User insists in using 'stdio:/dev/sdd' without further preparations
or safety checks. The device might need blanking by xorriso
before it will be willing to write to it. Permissions might not
suffice. Be cautious when removing such obstacles. They might be
there for a good reason.
If you agree to xorriso trying to overwrite the content of 'stdio:/dev/sdd'
then enter 'yes' :
yes
Will try to overwrite the device content without sudo or pseudo-blanking.
Arguments look acceptable.Keep in mind that the xorriso run in merge_debian_isos ignores locally defined xorriso startup files (by command -no_rc).
For details about "stdio:" and blanking read man xorriso. In short: Files under /dev which are not optical drives are protected from being used by default setting -drive_class caution /dev. Prefix "stdio:" overrides this protection. Blanking non-optical media just means to write a few zero bytes to the first 64 KiB of the file or device, like is done for DVD+RW or BD-RE media too.
Write to bug-xorriso@gnu.org if questions remain.
Help text of merge_debian_isos
usage: merge_debian_isos result_iso mount_template iso1 iso2 [... isoN]
Mounts by sudo the ISO 9660 images iso1 to isoN at directories
mount_template1 to mount_templateN, if not already mounted that way.
Then the Debian pools and package lists get merged and a new
ISO 9660 image result_iso is produced. If iso1 is bootable then
then the new image will be bootable by the same means.
The file depicted by result_iso must not yet exist or has to be a
device which is acceptable for Linux-specific helper script
xorriso-dd-target. If xorriso-dd-target agrees and the user
confirms by input "yes" then xoriso will be run under sudo.
Exempted from this evaluation are addresses which begin by "mmc:"
for an optical drive on Linux, BSDs, Solaris, or by "stdio:/dev/"
for which the user takes full and dangerous responsibility.
Special result_iso path "xorriso-dd-target-plug-test" determines
on Linux the target USB stick by a dialog around plugging it in.
xorriso will be run under sudo, if xorriso-dd-target agrees and
the user confirms by input "yes".
At least the parent directory of mount_template must already exist
or has to be given by a plain name without "/" so that it can be
created in the current directory. (I.e. without even "./".)
All arguments must be single words without using quotation marks.
None of the isoN must be equal to another isoM.
This script creates and finally removes the following temporary tree
and files which must not yet exist in the current working directory:
./merged_dists , ./merged_md5sum.txt , ./merged_REAMDE.txt
./temp_file
Further it creates and finally removes directories mount_template*
if they are needed and do not exist when the script starts. If the
parent directory in the mount_template path does not exist and
its path contains no "/" then it gets created and finally removed.
The script depends on the following programs:
awk, basename, cat, chmod, cp, date, dirname, expr, fgrep, grep,
gunzip, gzip, head, ls, md5sum, mkdir, mount, mv, rm, rmdir,
sed, sh, sha256sum, sort, stat, sudo, umount, uniq, xorriso
With device writing involving helper script xorriso-dd-target:
dd, Linux kernel, lsblk, sleep, tr, uname, wc, whoami,
xorriso-dd-target
Recommended are: sha1sum, sha512sum
Exported non-empty variable MERGE_DATE enforces a particular
date string in the text which gets prepended to /README.txt .
Exported non-empty variable MERGE_FOR_DIST enforces the use of a
particular directory in /dists of iso1. Normally only one
such directory is found and thus no need to set MERGE_FOR_DIST.
Exported non-empty variable MERGE_KEEP_ISO prevents the removal
of result_iso after xorriso indicated failure of production.
Exported non-empty variable XORRISO overrides command xorriso.
This may be needed if installed xorriso is older than 1.4.2.
If XORRISO is set to "dummy" then no new ISO will emerge.
Exported non-empty variable XORRISO_DD_TARGET_PATH names the
directory where to find xorriso-dd-target, which evaluates the
suitability of result_iso devices or does the plug-test dialog.
Example using GNU xorriso-1.5.4 instead of /usr/bin/xorriso:
export XORRISO="$HOME"/xorriso-1.5.4/xorriso/xorriso
merge_debian_isos merged.iso merge_mount/iso \
debian-11.2.0-amd64-DVD-[12345].iso
Example writing to optical drive /dev/sr0 :
merge_debian_isos mmc:/dev/sr0 merge_mount/iso \
debian-11.2.0-amd64-DVD-[12345].iso
Example on Linux writing to USB stick with xorriso-dd-target-plug-test:
wget https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/xorriso-dd-target/xorriso-dd-target
chmod u+x xorriso-dd-target
export XORRISO_DD_TARGET_PATH="$(pwd)"
merge_debian_isos xorriso-dd-target-plug-test merge_mount/iso \
debian-11.2.0-amd64-DVD-[12345].iso
This leads to the first two steps of the xorriso-dd-target device
plug dialog. See: https://wiki.debian.org/XorrisoDdTarget
Finally input "yes" is required to authorize xorriso under sudo.