Some ideas by madduck:
- support mergeWithUpstream like svn-buildpackage
- store location of upstream tarball in bzr
- configuration options per branch
- hooks, have to be authorised by person checking out (hash sum)
- goal: checkout/branch and build without much to configure
From starting work on it more has become apparent. Here are a few of the things I am trying to decide how to handle at the moment
- Differences between working tree and branch.
- svn- errors out if there are uncommited changes. This is irritating half the time, but I find it very useful, when for instance I have added a dpatch, and forget to svn add it. This saves a build cycle. However the local nature of bzr could remove some of the cases where you want to make a change, as you can commit, and not have the change propogate until you want it to. Also the ease of uncommit helps this. However it still doesn't help with the missing dpatch case.
madduck: good point, and I fully agree with your analysis. I'd say we might want to look into two modes, testing and release. Testing ignores, and release would warn. Release would also do stuff like call dch -r. Also, maybe there should be two testing modes, one that calls the build command, and one that simply calls fakeroot debian/rules binary?
JamesWestby: Good idea I think. The testing mode aliases to --ignore-changes and uses --quick-builder (which defaults to fakeroot debian/rules binary), The release mode doesn't use --ignore-changes nad uses --builder (Something like pdebuild). There should probably also be the relase command, which could do a build (but probably not), but tags (hopefully) the current revision with the appropriate version number etc.
< madduck> james_w: from my experience, i think it's *bad* to automatically tag. < madduck> i think what should happen is that following a release, bzr-bd should somehow mark the tree < madduck> and then warn the user until s/he tagged the release < madduck> just tagging automatically as part of the release build is not a good idea < madduck> or maybe add uploading facilities (via dput/dupload, then tag after a successful upload)?
JamesWestby: So I have implemented some of this. There is a quick-builder config option. This specifies the builder to use when --quick is used (it defaults to fakeroot debian/rules binary). There is also --reuse which is useful in merge mode as it saves on unpacking the upstream tarball each time. So there is a very fast way of hacking that allows you just to make modifications to the working tree, the can builddeb --quick --reuse --working-tree. Perhaps another command could be added that is an alias of all three, but maybe that is too much bloat, and should be left to the user to define an alias if they think it is useful. I am now going to try and implement a release command, which does exports everything, calls dch -r, does a full build, and then marks the tree as needing tagging.
- The config will be versioned under the branch, but this leaves the choice of which version to use. At the moment it uses the one on the filesystem (that isn't forced to exist), it would also be possible to use the one from the working tree (same as the filesystem, but by querying bzr for the file), or to use the one in the branch, which may not be up to date, and probably can't be if the user changes it for a local requirement. This is tied up with the point just above, and the one about the differences between the branch and the working tree.
madduck: Well, this is a difficult one, and I don't think either approach is any more "right". For hooks, it's definitely the "local" copy that should get run, because we need to authenticate hooks individually. But for the configuration... mh. Seems like it would make sense to update it first, print an informational message (and a diff in verbose mode), and use the branch version. After all, that's "current" at the time it was checked it.
JamesWestby: At the moment the build will error out if there are local changes, and so it will come to the users attention that something is wrong. However I need to decide where to get the files from. I'll try and come up with a sane scheme.
Thanks for your great work on this!
Hooks
JamesWestby: You have mentioned hooks a few times, and I wanted to try and get your thoughts on a few things (though I'm still a way off implementing them). You hae pointed out that running hooks by default is a bad idea. You have suggested md5sum $hook >> file might be a good way to allow the user to select which hooks to run. I think this approach is reasonale, and can certainly be wrapped to provide a more simple command. However the file storing the sums needs to be carefully handled. If it is at all possible that the file comes from the parent branch then the control is easily bypassed. I think there are a few options for this.
- Have file away from the branch (under ~ probably). Then it can't be under the control of the branch, but loses modularity.
madduck: this is what i had in mind. All the other means are inviting attackers, whereas this approach seems safe by design. I understand your modularity concern though.
- Add the file to the ignore list so that it is not commited by default. It's certainly possible to bypass this and commit it.
- Refuse to use any hooks if the branch contains the file. This can probably be beaten, especially if the branch is pushed to the user over a transport that doesn't update the working tree, and the user builds with --working-tree.
madduck: how about refuse to work if bzr knows anything about the file, other than an ignore entry? if someone were to come and change or install the file, s/he could not do this without bzr knowing about the file.
Have some extra information in the md5sum. It needs to be unknown to the attacker so things like username and hostame will not work. A password seems wrong for this, but maybe a nonce/password type thing would work. When I want to use a hook for the first time the plugin generates a random number and stores it somewhere under ~. Then to use a hook store md5(hook || number) in the file. That way even if this file makes it to your system, the plugin wont use the hook unless I can guess your number, or find a collision in md5. If we are really concerned here we could move to one of the sha family, and use a suitably long random number.
I have had a first stab at implementing hooks.
- You must store the sha1sum under ~. I chose sha1 as bzr provides that info easily.
- All the files go under ~/.bazaar/builddeb-hooks/ and are named by the file_id of the hook. This has a couple of drawbacks.
- Clutter will build up in this dir as it is hard to know what file corresponds to which branch.
- It requires files to be versioned. Doesn't allow quick local hooks that aren't added to the branch.
- Only one sha is recorded which means that if the user has two branches and wants to modify a hook they have to update the sha each time they switch a branch.
- For these reasons I might look at recording the hooks in ~/.bazaar/builddeb.conf (and only there).
I just had a package reviewed, and they suggested taking out the .bzr-builddeb dir, which appears in the .diff.gz. I have just implemented an rm -rf .bzr-builddeb after export, so that it is like the .bzr dir in effect. Two points then
Do we want the directory to be excluded? This means that you need the branch to get the information, not just the source package, but this seems reasonable anyway, as the file-ids would want to match if you wanted to contribute something back.
If we do want the directory moved is there a cleaner method? An exclude filter on the export would be nice (is this currently possible with the default-ignores thing?).
Some discussion with liw trying something I hadn't done before made me realise the following things are needed.
- tarball for non-merge
- Quite wasy to do as it just needs to grab the tarball from the configured dir and copy it to build-area. At the moment the orig-dir only has an effect for merge builds. So I need to make it work for a non-merge build. There are two ways to do this. Either have a config variable that states whether the package has an upstream tarball, or detect it by looking in the orig-dir for an appropriate named one. The first has the problem that it is an extra step and I am not sure what the default should be. Probably default to look for the tarball to avoid mistaken native packages. The second is probably the wrong approach as the user probably wants an error if they dont have the tarball.
- Done
- Quite wasy to do as it just needs to grab the tarball from the configured dir and copy it to build-area. At the moment the orig-dir only has an effect for merge builds. So I need to make it work for a non-merge build. There are two ways to do this. Either have a config variable that states whether the package has an upstream tarball, or detect it by looking in the orig-dir for an appropriate named one. The first has the problem that it is an extra step and I am not sure what the default should be. Probably default to look for the tarball to avoid mistaken native packages. The second is probably the wrong approach as the user probably wants an error if they dont have the tarball.
whole source -> tarball + diff (split on debian/)
- Should be quite easy to do. Just export the whole thing remove debian/ (or configurable path), and tar it up to the correct name. The remove the export and do it again. Adds another mode. This makes 3 now, merge and the last two. Should there be config variables for them all, or an enum type thing to choose between them. At the moment they are all mutually exclusive so that the enum would be better to avoid precedence problems. However if they don't stay mutually exclusive then there would be explosion of options for the field. I will go with variables for now as that is what exists at the moment and I don't know how many modes there are going to be.
- Done
- Should be quite easy to do. Just export the whole thing remove debian/ (or configurable path), and tar it up to the correct name. The remove the export and do it again. Adds another mode. This makes 3 now, merge and the last two. Should there be config variables for them all, or an enum type thing to choose between them. At the moment they are all mutually exclusive so that the enum would be better to avoid precedence problems. However if they don't stay mutually exclusive then there would be explosion of options for the field. I will go with variables for now as that is what exists at the moment and I don't know how many modes there are going to be.
- sub branch for debian/
- probably feasible, but I need to think about the ramifications. I also need to learn about nested branches so I can see if they would work for it better that anything I can currently do.
- Done it I think. I added an export-upstream option that will export the code for the .orig.tar.gz from another branch.
- With this is it a good idea to keep a upstream version to revision number mapping for all versions, rather than just the current? Or just leave that to the upstream branch?
- Done it I think. I added an export-upstream option that will export the code for the .orig.tar.gz from another branch.
- probably feasible, but I need to think about the ramifications. I also need to learn about nested branches so I can see if they would work for it better that anything I can currently do.
- Easy building of source packages.
- Add a source-builder option and use it when -S/--source is used. It defaults to dpkg-buildpackage -S.
- Done
- Add a source-builder option and use it when -S/--source is used. It defaults to dpkg-buildpackage -S.
- Don't read builders from the branch config files.
Other TODO items.
- Evaluate moving to bazaar's own config files.