There are a number of different approaches to this, all them involve running the program and associated infrastructure under a virtual display servers.

For X11 there is a virtual server called Xvfb.

For Wayland sway with WLR_BACKEND=headless might be useful. Or the KDE KWayland test server might be useful too, libkf5wayland-dev has a program /usr/lib/x86_64-linux-gnu/libexec/org-kde-kf5-kwayland-testserver

Approaches

Accessibility infra

Selenium

selenium-webdriver-at-spi (not in Debian yet) can be used to write tests that use selenium to talk to the AT-SPI accessibility infrastructure in order to control and test programs. Tests that use Selenium can be written in several different languages.

Dogtail

dogtail talks to the AT-SPI accessibility infrastructure in order to control and test programs. Tests that use Dogtail are written in Python and can control GTK and KDE/Qt programs. Example tests are available.

Input/output

xdotool can be used send input (keyboard, mouse) and window manager actions to X11 windows. imagemagick convert can extract pixels and then the test can analyse them. Example tests are in src:bambam.

See also