Differences between revisions 29 and 31 (spanning 2 versions)
Revision 29 as of 2017-02-13 19:02:43
Size: 8681
Editor: Infinity0
Comment: add caveat to account for autotools usage
Revision 31 as of 2017-02-14 18:40:25
Size: 1456
Editor: Infinity0
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
<<TableOfContents()>>
Line 5: Line 3:
{{{#!wiki comment
This section was sourced from [[ReproducibleBuilds/ProposalTemplate]].
 * If you need to edit text inside the `GetVal` parts, edit them in [[ReproducibleBuilds/BuildPathProposal/Dict]].
 * If you need to edit text outside the `GetVal` parts (but still in this section), please also make the equivalent edit back in the original [[ReproducibleBuilds/ProposalTemplate|template]]!
}}}
This is a standard that defines an environment variable `BUILD_PATH_PREFIX_MAP` that distributions can set centrally, and have build tools consume this in order to produce reproducible output.
Line 11: Line 5:
== Background == Before implementing this, you should scan through [[../StandardEnvironmentVariables#Checklist|our checklist]] to see if you can avoid implementing it.
Line 13: Line 7:
Sometimes build tools embed <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, buildtime_information)>> into the output. This causes the build to be unreproducible when <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, buildtime_action)>>. TODO: the actual proposal
Line 15: Line 9:
There are different reasons why this is done, most commonly <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, most_common_reason)>>. However, we have never seen nor do we foresee a reasonable use-case where the actual <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, buildtime_information_pedantic)>> is needed; all the use-cases we have seen, would equally be satisfied by using the <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, buildtime_information_better)>>. This has the additional benefit of giving reproducible build output.

In fact, it is already possible for upstream projects to do this. They could <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, potential_complex_strategy)>> - as opposed to <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, existing_unreproducible_strategy)>>.

However in many cases, upstream projects are not the ones directly responsible for embedding this information into the build output - it is some intermediate build tool that does it, e.g. <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, intermediate_irreproducibility_culprit)>>. So they (the upstream project) shouldn't have to implement this detection and then supply this information to that build tool, just to get reproducible build output. Therefore, we have chosen to define this standard.

== Description ==

This is a standard that defines an environment variable <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, envvar_name)>> that distributions can set centrally, and have build tools obey this in order to produce reproducible output even if projects that use these tools are not specifically aware of reproducible builds.

This minimises the work required across the whole software community, to achieve reproducible builds by default. However, it does add a tiny bit of complexity to specific build tools. So, before implementing this, you should go through the following checklist, to see if you can avoid implementing it:

Checklist: Do I really need to use this?

1. See if [[https://packages.debian.org/sid/strip-nondeterminism|strip-nondeterminism]] will get rid of the difference for you, or if it's easy to add this functionality to it. If yes, then you don't need this variable. For example, <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, example_stripnd)>>.

2. See if you can patch the tool that generates this information, to simply not generate this information in the first place. If this is a good idea, then you don't need this variable. For example, <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, example_deleteinfo)>>. See [[#hard-req-envvar]] for more discussion.

3. See if you can patch the tool that generates this information, to use this specification to generate a reproducible value for this information, based on a standardised environment variable set centrally by your distribution. '''This is what this document is about.'''

4. See if you can patch the package that uses the tool, to give it some tool-specific options like CLI arguments or its own environment variables, either to avoid generating this information or to generate a reproducible value. For example, <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, example_toolspecific)>>. In certain cases this may be a good solution, such as when the tool in question is the top-level buildsystem of the package and the options to be used are not too verbose. However, in most cases this should be '''avoided''' in favour of the previous options, since you will have to do this for every package that uses the tool. See [[#tool-specific-args]] for more discussion.

{{{#!wiki comment
End of the section that was sourced from [[ReproducibleBuilds/ProposalTemplate]].
}}}
<<TableOfContents()>>
Line 43: Line 13:
FIXME Please read our (TODO specification) for details.

See [[../StandardEnvironmentVariables|Standard Environment Variables]] for more detailed discussion of the rationales behind this mechanism.

Below we also have [[#More_detailed_discussion|more detailed discussion]] about this specific variable, as well as documentation on [[#history-and-alternatives|history and alternative proposals]].
Line 53: Line 27:
{{{#!wiki comment
This section was sourced from [[ReproducibleBuilds/ProposalTemplate]].
 * If you need to edit text inside the `GetVal` parts, edit them in [[ReproducibleBuilds/BuildPathProposal/Dict]].
 * If you need to edit text outside the `GetVal` parts (but still in this section), please also make the equivalent edit back in the original [[ReproducibleBuilds/ProposalTemplate|template]]!
}}}
Line 61: Line 29:
Sometimes developers of build tools do not want to support <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, envvar_name)>>, or they will tweak the suggestion to something related but different. We really do think the best approach is to use <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, envvar_name)>> exactly as-is described above in our proposal, without any variation. Here we explain our reasoning versus the arguments we have encountered: (See [[ReproducibleBuilds/StandardEnvironmentVariables#more-detailed-discussion|Standard Environment Variables]] for general arguments.)
Line 63: Line 31:
{{{#!wiki comment
End of the section that was sourced from [[ReproducibleBuilds/ProposalTemplate]].
}}}

FIXME (add other points here, before the "generic" points)

{{{#!wiki comment
This section was sourced from [[ReproducibleBuilds/ProposalTemplate]].
 * If you need to edit text inside the `GetVal` parts, edit them in [[ReproducibleBuilds/BuildPathProposal/Dict]].
 * If you need to edit text outside the `GetVal` parts (but still in this section), please also make the equivalent edit back in the original [[ReproducibleBuilds/ProposalTemplate|template]]!
}}}

<<Anchor(tool-specific-args)>>
=== "We'll add a command-line flag instead" ===

Build tool developers dislike the name <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, envvar_name)>> or the mechanism of environment variables, and prefer to use command line options, perhaps to be more consistent with the rest of their program.

We understand the desire to avoid inconsistency, or supporting seemingly arbitrary environment variables. However, <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, buildtime_information_short)>> are by far one of the largest issues that prevents reproducible builds from becoming reality, and every tool supporting a standardised mechanism would greatly lower the cost of that. If every tool supported different mechanisms to do the same thing, the overall cost is greatly higher. In other words, we ask that you consider global consistency across many different projects, over any smaller-scale inconsistency in your own project.

From another point of view: we agree that it is easy for users of your tool to add a command-line option, but then they have to specifically think about reproducible builds. This is not, and ought not to be, in the minds of most software developers. Reproducible builds should be the default situation for all software, and we shouldn't burden developers with these trivialities. Opt-in security is not security, because nobody will take the effort to opt-in even if in general they think it's a good idea. (This is not a paradox; people like getting things for free.)

<<Anchor(hard-req-envvar)>>
=== Hard-requirement on this variable for reproducibility ===

<<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, envvar_name)>> is meant to be a cost-reduction mechanism, to avoid having to globally patch many hundreds of projects in the same way.

However, if you are a project developer and you want to make ''your specific project'' reproducible, the best way is to do it without ''requiring'' the user to set <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, envvar_name)>> themselves. For example, you could avoid using things like <<GetVal(ReproducibleBuilds/BuildPathProposal/Dict, buildtime_information_example)>> (there might be better alternatives available), or you could detect a sensible default value in your top-level Makefile (or equivalent) and set it if unset.

{{{#!wiki comment
End of the section that was sourced from [[ReproducibleBuilds/ProposalTemplate]].
}}}

== History and alternate proposals ==
=== Comparison to SOURCE_DATE_EPOCH ===
Line 98: Line 34:

<<Anchor(history-and-alternatives)>>
== History and alternative proposals ==

FIXME: stuff about `--fdebug-prefix-map`, `DW_AT_producer`, etc.

----
'''Footnotes:'''

/!\ This is a draft/discussion for a spec in development. Some code and other notes are here: https://github.com/infinity0/rb-prefix-map/tree/master/consume

This is a standard that defines an environment variable BUILD_PATH_PREFIX_MAP that distributions can set centrally, and have build tools consume this in order to produce reproducible output.

Before implementing this, you should scan through our checklist to see if you can avoid implementing it.

TODO: the actual proposal

Proposal

Please read our (TODO specification) for details.

See Standard Environment Variables for more detailed discussion of the rationales behind this mechanism.

Below we also have more detailed discussion about this specific variable, as well as documentation on history and alternative proposals.

Setting the variable

FIXME

Reading the variable

FIXME

More detailed discussion

(See Standard Environment Variables for general arguments.)

Comparison to SOURCE_DATE_EPOCH

FIXME

History and alternative proposals

FIXME: stuff about --fdebug-prefix-map, DW_AT_producer, etc.


Footnotes: