Go Ecosystem Issues 2020

Go Ecosystem in Debian suffers from several issues, those were investigated and a summary posted on the Go mailing list:

Contents

  1. Go Ecosystem Issues 2020
    1. Trying to map the Go ecosystem issues in Debian
      1. unstable->testing migration
        1. How it can be a problem: example of the issue during Buster freeze
      2. stable updates (through security update or point release)
    2. potential solutions:
      1. Suggestion on how to move forward by ivodd
        1. References for dak

Trying to map the Go ecosystem issues in Debian

unstable->testing migration

Issue: we need to know, for a given package that is updated, what needs to be rebuilt (binNMU).

Currently, we know it by using Built-Using: field. This is generated by dh_golang at build time, using ${misc:Built-Using} substvar.

Note: this is a policy violation, we need to change this field. The field used by Rust is X-Cargo-Built-Using.

DISCUSSION: check with the Rust team? It would make sense to use the same, as we might then share tooling. Alternatively, just make our own and use X-Go-Built-Using (and tooling can check X-[^-]+-Built-Using

How it can be a problem: example of the issue during Buster freeze

During the freeze, people kept uploading new versions of packages to unstable, which meant that whenever a package in testing needed a fix, it would end up building with a more recent version of libraries than what was in testing… So we had to upload fake new releases to make the package go back to the previous version and respect the freeze.

OCAML & haskell also have the issue, but they also have very stricts deps, which means they have to coordinate a lot wrt. package migration.

In Go and Rust ecosystem, individual packages can migrate.

DISCUSSION: find a way to prevent this (migrating when a Build-depends is different than in testing). * Maybe using autopkgtest? * Britney should prevent somehow the migration of the rdepends until the build-depends has migrated? * just upload non-fixes to experimental during the freeze?

stable updates (through security update or point release)

Everything has to be rebuilt and manually uploaded (either by Security Team or Release team for a point release).

Same tool than for testing migration can be used in this case: we should be able to map easily what needs to be rebuilt.

Problem is, it gets particularly messy when it's a security update: dak database is public for the main archive, but not the security archive. Indeed, there might be embargoed issues, so this database needs to be kept private (and thus separate).

The issue is, in case of BinNMU, you still need the orig tarball, but if it's the first security update, you don't have it in the local database, and dak doesn't know how to check if it exists in another database, and even less how to get it.

It means that if you upload a new revision, you also need the orig tarball. If you do a binnmu, source isn't uploaded, and they don't have it in the security archive, which means it can't be uploaded (dak will scream at you)

potential solutions:

DISCUSSION:

1. make a full upload every time an upload is made on the SA for a package for the first time (so not a binnmu), and then you can make binnmu on this package afterwards? 1. (preferred) have some way in dak to detect that the source actually exists - even is it's a different db. → IOW: When I'm the security archive, if I don't have source, check the main archive and get the source from it if it's there.

Maybe Security and Release Teams would be OK with 1., if we provide tooling to help ? It sure seems the easiest solution.

Suggestion on how to move forward by ivodd

In the case we want to fix the issue on dak side - which would benefit not only Go, but other similar ecosystems in Debian - the suggested path forward is to try to create 2 archives using the interactive shell to replicate the problem. Note that we'll probably need help (info) from ftp team/security team because the databases will be empty initially.

Ivodd is working on a test that will send a binNMU, which is part of the failure case. ETA unknown, so please don't wait up on that.

ACTION: replicate the failure on a test dak instance.

References for dak