AppStream
Introduction
AppStream is a cross-distro XML format to provide metadata for software components and to assign unique identifiers to software.
In Debian, we parse all XML provided by upstream projects as well as other metadata (XDG Desktop Entry files, ...), and compile a single YAML metadata file from it, which is then shipped to users via APT.
The metadata can for example be used by software centers like GNOME Software or KDE Discover to display a user-friendly application-centric way on the package archive.
Additionally, AppStream is used by several other projects, like the Limba installer and LVFS (Linux Vendor Firmware Service) to provide metadata about provided software components. This data usually originates from external, non-Debian sources and is read by the AppStream client libraries libappstream and libappstream-glib as well.
Writing metainfo files
For information on how to write metainfo files, you can consult the Metadata Quickstart Guide at Freedesktop.
Current status
You can take a look at the metadata generator status at appstream.d.o, which includes hints in case data failed validation. It also allows taking a look at the raw metadata generated from the archive.
How to obtain the data?
Automatically
In order to get the latest data on Debian, just install the appstream package and run apt update. This will make APT download the necessary metadata, have it placed in the correct directory and ensure all caches are up to date.
Manually
It is recommended to have the data downloaded and kept up-to-date automatically. In case you want to add new data from non-APT repositories, which doesn't exist in a .deb package (which is a rare event), here are the necessary manual steps to make it available to AppStream-metadata-consuming applications:
Get up-to-date data (e.g. from appstream.d.o/data/: Navigate to the Debian suite you are running (e.g. "unstable"), and download the icon tarballs and *.yml.gz files for the archive components (main, contrib, non-free) you want data for.)
Copy the *.yml.gz files to /var/cache/app-info/yaml (you may rename them, e.g. to "unstable-main_amd64.yml.gz") and extract the 64x64 icon tarball to /var/cache/app-info/icons/%{suite}-%{acomponent}/64x64, where "suite" is the archive suite the icons are from (e.g. "unstable") and "acomponent" is the archive component (e.g. "main").
Run appstreamcli refresh-cache --force (needed for SCs which use the Xapian cache for fulltext search)
Format specification
You can find documentation about the AppStream XML format at freedesktop.org/software/appstream/docs. Debian's YAML-based metadata format, DEP-11, is documented there as well.
Guidelines for packagers
If you are maintaining a package containing AppStream metadata, or want to add metadata to a package, you might find the packaging guidelines section helpful, which also contains help and explanations for the most common mistakes.