There are a number of different approaches to this, all them involve running the program and associated infrastructure under a virtual display servers.
For X11 there is a virtual server called Xvfb.
For Wayland sway with WLR_BACKEND=headless might be useful. Or the KDE KWayland test server might be useful too, libkf5wayland-dev has a program /usr/lib/x86_64-linux-gnu/libexec/org-kde-kf5-kwayland-testserver
selenium-webdriver-at-spi (not in Debian yet) can be used to write tests that use selenium to talk to the AT-SPI accessibility infrastructure in order to control and test programs. Tests that use Selenium can be written in several different languages.
dogtail talks to the AT-SPI accessibility infrastructure in order to control and test programs. Tests that use Dogtail are written in Python and can control GTK and KDE/Qt programs. Example tests are available.