There are a number of different approaches to this, all them involve running the program and associated infrastructure under a virtual display servers.
For X11 there is a virtual server called Xvfb.
For Wayland sway with WLR_BACKEND=headless might be useful. Or the KDE KWayland test server might be useful too, libkf5wayland-dev has a program /usr/lib/x86_64-linux-gnu/libexec/org-kde-kf5-kwayland-testserver
Approaches
Accessibility infra
Selenium
selenium-webdriver-at-spi (not in Debian yet) can be used to write tests that use selenium to talk to the AT-SPI accessibility infrastructure in order to control and test programs. Tests that use Selenium can be written in several different languages.
Dogtail
dogtail talks to the AT-SPI accessibility infrastructure in order to control and test programs. Tests that use Dogtail are written in Python and can control GTK and KDE/Qt programs. Example tests are available.
Input/output
xdotool can be used send input (keyboard, mouse) and window manager actions to X11 windows. imagemagick convert can extract pixels and then the test can analyse them. Example tests are in src:bambam.