The following discussion of Debian Python Policy took place between 07:46 and 09:02 UTC on 2005/11/20 on #debian-tech.


Summary and Log

A discussion about possible bugs in the current Debian Python Policy, in advance of the upcoming python2.4 transition.

   1 07:46 <vorlon> anyone interested in some brainstorming about Debian python policy?
   2 07:50 <aj> ?
   3 07:54 <vorlon> I have awkwardnesses like http://lists.debian.org/debian-python/2003/08/msg00133.html in mind :)
   4 07:56 <aj> what's the problem?
   5 07:59 <vorlon> well, Joe makes a valid point that bytecompiling does contribute to python transitions being more of a pain than they need to be, and his argument that they don't give us any performance gains stands unrefuted
   6 08:00 <aj> so suggest people don't bytecompile unless they've profiled it and seen a benefit?
   7 08:00 <aj> (and don't use pythonX.Y unless they've bytecompiled)
   8 08:00 <vorlon> is that a reasonable course of action?  We also have the issue that python will bytecompile them for you without asking, if it his access to do so. :)
   9 08:01 <aj> will it re-bytecompile when run with a different python version?
  10 08:01 <vorlon> hmm, let me test this
  11 08:02 -!- piman [piman@kai.vm.bytemark.co.uk] has joined #debian-tech
  12 08:02  * vorlon moos
  13 08:02 <piman> Heyas.
  14 08:02 <vorlon> hmm, guess I need to /install/ more than one python to test this. :)
  15 08:03 <piman> I was told the reason for the requirement was speed, not making sure modules got cleaned up.
  16 08:03 <aj> lucky that debian makes that so esay!
  17 08:03 <vorlon> piman: current question is, if a different python is used, and it has access, will it re-bytecompile even though the .pyc/.pyo technically aren't out-of-date?
  18 08:03 <piman> Yes.
  19 08:04 <vorlon> ok
  20 08:04 <piman> pyc is a version-dependent format.
  21 08:04 <piman> It's also in the past been arch-dependent, I don't think it is as of Python 2.3.
  22 08:04 <vorlon> right; .pyo is still arch-dependent, though?
  23 08:04 <piman> I'm unsure, I don't know much about the Python optimizer beyond the fact it doesn't optimize much.
  24 08:04 <vorlon> heh
  25 08:05 <piman> I think it just strips asserts and docstrings.
  26 08:05 <piman> So, it should still be arch-indep too.
  27 08:05 <vorlon> the other question is, are there possibly different optimization trade-offs at work here on other archs, that we need to think about?
  28 08:05 <piman> I'm also not sure there's a real guarantee that the pyc format is only minor-version-dependent.
  29 08:06 <piman> The only question is compile time, really. pyos are not faster than pycs.
  30 08:06 <piman> In the case of web stuff or programs called repeatedly and often (enemies-of-carlotta) the pyc can be a big win, assuming you properly keep it up to date.
  31 08:07 <vorlon> "only" minor-version-dependent?  Meaning what, it could be patch-level dependent?
  32 08:07 <piman> Yeah.
  33 08:07 <vorlon> hum.
  34 08:07  * piman is checking the Python docs now
  35 08:07 <piman> I recall strong words not to rely on it. :)
  36 08:08 <piman> http://docs.python.org/lib/module-marshal.html -- it doesn't say beyond "Therefore, the Python maintainers reserve the right to modify the marshal format in backward incompatible ways should the need arise."
  37 08:08 <vorlon> but do we still have the situation that invalid .pycs are recognized and regenerated if they can be, discarded if they can't be?
  38 08:08 <piman> Yeah.
  39 08:09 <vorlon> ok.  and .pyo's are just completely unnecessary?
  40 08:10 <piman> In theory they can be slightly faster, but I've also seen real-world python code where they would be dangerous, because hte author is using asserts for runtime checks.
  41 08:10 <piman> Such programs are buggy, and should be fixed, granted.
  42 08:10 <piman> But so are many progrms that fail at -O3, which we don't default to for similar reasons (well, unless it's Python, again...)
  43 08:11 <vorlon> hmmm. and .pyo's don't get auto-regenerated, apparently, they only get built at package install time by compile.py...
  44 08:11 <piman> They get built when you import with python -O.
  45 08:12 <vorlon> which is not a normal mode of operation, AFAIK?
  46 08:12 <piman> Right.
  47 08:13 <piman> piman@gotou:~$ diff /usr/lib/quodlibet/util.py[co]
  48 08:13 <piman> piman@gotou:~$
  49 08:14 <piman> I think that's a good example of how useless it is.
  50 08:14 <vorlon> <laugh>
  51 08:14 <vorlon> yeah, python2.3/2.4 seem to be standing up to all my efforts to confuse it with bogus .pycs
  52 08:15 <piman> I've never seen it fail. The only case for pyc/pyo is speed, which isn't a good basis for a 'must' directive.
  53 08:16 <aj> and to have it create the .pyc/.pyo you'd have to run it as root
  54 08:16  * vorlon nods
  55 08:16 <piman> Yeah, I mentioned that in 8/2003 :)
  56 08:16 <vorlon> which happens; mailman's list creation script is one example of this, I guess eoc has others
  57 08:16 <aj> eww
  58 08:17 <vorlon> but again, the .pyo isn't even created by default, and a .pyc that's created wrong (wrong perms, wrong version) will just get ignored
  59 08:17 <piman> Well, eoc just benefits from the speed because the compilation dominates its runtime.
  60 08:17 <aj> sure, but eoc can be compiled in that case
  61 08:17 <piman> Yeah.
  62 08:18 <piman> (I'm also going off what liw said here, and not actual benchmarks.)
  63 08:22 <vorlon> piman: so do you know why your comments weren't incorporated into the policy two years ago?  I don't see that your arguments were refuted on-list; I guess there's still an argument that having .pyc is a speed benefit in some cases, but at least for cleaning up on package removal, that can be done whether or not we're creating the files at install time...
  64 08:23 <piman> No, I have no idea. It came up again briefly when discussing why Debian Python migrations are so painful; I suggested the overly-tight dependencies, but didn't receive any responses.
  65 08:24 <piman> I think that was late 2004, but I really don't remember.
  66 08:24 <vorlon> Do you know offhand who's in charge of the python policy process today?  Matthias, Joss...?
  67 08:24 <vorlon> doing something saner with python policy is on the release team's todo list for etch, so this is the time for us to get cracking. :)
  68 08:24 <piman> I'm pretty sure it's Joss.
  69 08:24 <vorlon> ok
  70 08:25 <piman> Can you make -O3 disappear unless someone presents a benchmark, please? :)
  71 08:25 <piman> (That is, removed from the default compilation options.)
  72 08:26 <vorlon> heh... is that in the policy?  I can talk them out of including it in the policy, but I don't imagine taking it out of policy will magically fix the issue of it propagating from python to other python modules ;)
  73 08:26 <piman> No, sadly.
  74 08:27 <vorlon> The -O3 in the default compilation options has been discussed before, particularly right after gcc hit 4.0 and stuff started breaking everywhere with -O3... maybe you'd like to submit a patch? :)
  75 08:28 <piman> Does it actually make Python noticably faster? I asked and Matthias referred me to the archives for a benchmark, I didn't find out.
  76 08:28 <piman> s/out/one/
  77 08:28 <piman> Without that, my patch is probably just going to make python compile with -O2. ;)
  78 08:29 <vorlon> uh, I was assured that there are core python modules for which it does make a difference.  I'm not keen to try to argue that, I just want us not to be propagating souped-up options to other modules
  79 08:30 <piman> I would offer a patch but my python-related todo list is absolutely huge right now, and exams are coming up, it'd probably be over a month before I could look at it.
  80 08:30 <vorlon> k
  81 08:33 <vorlon> so, what other problems do we have that are making python transitions painful, that we might be able to solve? :)
  82 08:36 <piman> Well last time some moron had a circular conflict that prevented anything from transitioning. ;) But seriously, my only other one got dealt with, the addition of a /usr/lib/python search path for modules that will work with any version.
  83 08:36 <vorlon> the fact that we're currently supporting so many modules across so many python versions is an issue, I think; but maybe that can go away within the etch life cycle, to some extent?
  84 08:36 <piman> And yeah, the fact there are just too many modules.
  85 08:37 <vorlon> hmm.  How do we define the minimum version that a module must support to be included in /usr/lib/python?
  86 08:37 <vorlon> maybe that needs to be /usr/lib/python2 instead; similar to /usr/lib/perl5?
  87 08:38 <piman> Hrm. Python 1 is dead enough that I'm not sure it pays.
  88 08:38 <vorlon> ok; main question still, what's the minimum version something has to support to be there?
  89 08:39 <vorlon> otherwise, I think you get hung up on partial or out-of-order upgrades, where people think it's ok to package modules there because they support python2.5, but the previous stable version only had python2.4 and tries to use them, etc...
  90 08:39 <piman> Looking at what's in there now, probably 2.2...
  91 08:39 <piman> Assuming we want to document existing practice. :)
  92 08:40 <vorlon> hrm, I don't seem to have that directory here...
  93 08:40 <piman> linda and pychecker are in it for me.
  94 08:40 <piman> (/usr/lib/site-python, I made a mistake the first time)
  95 08:41 <piman> Sorry, I was hung up on the perl one.
  96 08:41 <vorlon> I still think /usr/lib/python2 would be a good idea, so that if/when python3 comes out there can be a clean cut for stuff which supports python3 but doesn't support python2.2
  97 08:41 <vorlon> ah, site-python, sure
  98 08:44 <piman> I basically agree site-python2 would've been better, though I'm not sure what we gain by changing it now.
  99 08:45 -!- pabs [~pabs@dsl-202-72-168-241.wa.westnet.com.au] has joined #debian-tech
 100 08:45 <vorlon> I know that having the python-foo and python2.3-foo packages together tends to lead to a fair number of packaging mistakes.  I think it also contributes to making transitions harder, doesn't it?  Since any application using /usr/bin/python means that any modules it uses have to be transitioned in lockstep with the interpreter?
 101 08:46 <piman> Right.
 102 08:46 <vorlon> maybe that's not avoidable, though...
 103 08:46 <vorlon> at least, not for anything that builds .so
 104 08:46 <vorlon> ..'s
 105 08:46 <piman> I think that's a minor of packages though.
 106 08:47 <piman> Most Python modules are written in Python.
 107 08:48 <piman> The problem is when a program depends on python2.x-foo when it should be python-foo. It's rare that a program actually needs that tight of a dependency.
 108 08:48 <vorlon> hmm... 202 packages in unstable for python2.3, according to Contents-i386.gz
 109 08:48 <piman> I think most of those have been fixed though.
 110 08:48 <vorlon> right, a program should only depend on python2.x-foo if it invokes /usr/bin/python2.x as interpreter
 111 08:48 <piman> Nevermind, the first program I picked to look at (hal-device-manager) has that problem.
 112 08:49 <piman> Executes /usr/bin/python, depends on python2.3-{dbus,gnome2,glade2}
 113 08:49 <piman> Installs no modules itself.
 114 08:49 <vorlon> (171 packages installing python2.3 .so's in stable)
 115 08:50 <piman> Really? Wow, nevermind. :)
 116 08:51 <vorlon> does it simplify anything if we don't *have* python2.x-foo packages, and just include those contents in python-foo when only one version of the module is needed?
 117 08:51 <vorlon> I'm thinking it doesn't, but I'm not sure
 118 08:52 <piman> Well, it means applications can't make mistakes like hal-device-manager did.
 119 08:52 <piman> There'd only be the "good" package to depend on.
 120 08:52 <vorlon> right; not sure it makes transitions easier, though, or makes partial upgrades any nicer on users
 121 08:53 <vorlon> of course, partial upgrades are only an issue there if you do have some packages depending on python2.x-foo directly
 122 08:53 <piman> I suspect it would make the life of NMUers easier. python-foo packages will usually upgrade on a rebuild. python2.x-foo packages require control/rules tweaking.
 123 08:53  * vorlon nods
 124 08:54 <vorlon> if no apps depend directly on your python2.x-foo, because they're all happily using /usr/bin/python, then having separate packages really doesn't make any sense...
 125 08:54  * piman nods
 126 08:55 <piman> 2.x-foo are only really good for developers.
 127 08:55 <piman> And usually that's to test on whatever version of Python Debian hasn't transitioned to yet (that's what I use them for)
 128 08:58 <piman> Though I'm reluctant to say that without input from more Python developers, and they don't seem to be awake.
 129 08:59 <vorlon> so, possible recommendations to improve python policy: don't mandate byte-compiling, but do mandate cleaning up byte-compiled files on removal, allowing packages to remove the upper bound of their python dep; add a common directory to the python search path, so that packages which don't require anything newer than python2.2 and don't build .so's can be shared across python versions; merge python-foo and python2.x-foo in cases ...
 130 08:59 <vorlon> ... where no other packages need to depend on python2.x-foo directly
 131 09:00 <piman> Get upstream to stop being grossly incompatible? :) But that probably won't happen.
 132 09:00 <vorlon> cool.  Yeah, I think the next step is to poke at the python maintainers some; I'll do that after I sleep for a few hours, I think. :)
 133 09:00 <piman> But yeah, those are the three I've always thought were holding it up.
 134 09:01 <piman> Thanks.
 135 09:01 <vorlon> Hah, any plan that depends on upstreams not changing things is doomed to failure. :)
 136 09:01 <vorlon> *trust* me on that one... ;)
 137 09:01 <piman> Hey, Python 2.5 changes the / operator ;)
 138 09:01 <piman> I expect a lot of breakage when that happens...
 139 09:01  * vorlon stares
 140 09:01 <piman> Well, they're saying 2.5, it might be pushed back. 1/2 = 0.5. 1//2 = 0.
 141 09:02 <vorlon> right, so we'll standardize /usr/lib/site-python just in time for a new upstream version that breaks anything that uses division. :)
 142 09:02 <vorlon> ohwell...
 143 09:02 <piman> I've been writing my code with //, it's ugly, oh well :/)