Differences between revisions 74 and 104 (spanning 30 versions)
Revision 74 as of 2015-11-14 18:43:58
Size: 5668
Editor: ?mintberrycrunch
Comment:
Revision 104 as of 2022-12-23 16:08:06
Size: 7668
Comment:
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
~-[[DebianWiki/EditorGuide#translation|Translation(s)]]: English - [[it/UltimateDebianDatabase|Italiano]] -~ ##For Translators - to have a constantly up to date translation header in you page, you can just add a line like the following (with the comment's character at the start of the line removed)
##<<Include(UltimateDebianDatabase, ,from="^##TAG:TRANSLATION-HEADER-START",to="^##TAG:TRANSLATION-HEADER-END")>>
##TAG:TRANSLATION-HEADER-START
~-[[DebianWiki/EditorGuide#translation|Translation(s)]]: [[UltimateDebianDatabase|English]] - [[es/UltimateDebianDatabase|Español]] - [[it/UltimateDebianDatabase|Italiano]] - [[uk/UltimateDebianDatabase|Українська]] -~
##TAG:TRANSLATION-HEADER-END
Line 5: Line 9:
'''Ultimate Debian Database (UDD) gathers a lot of data about various aspects of Debian in the same SQL database. It allows users to easily access and combine all these data.''' '''The Ultimate Debian Database (UDD) gathers a lot of data about various aspects of Debian in the same SQL database. It allows users to easily access and combine all these data.'''
Line 9: Line 13:
 * Some '''example queries''' are provided as CGI scripts, to make it easy for everyone to run them. You can [[https://udd.debian.org/cgi-bin/|browse them]] or [[https://anonscm.debian.org/gitweb/?p=collab-qa/udd.git;a=tree;f=web/cgi-bin|view the source code]].
 * '''Database schema''': https://udd.debian.org/schema/
 * '''Source code''': available in the git repository of the [[https://alioth.debian.org/projects/collab-qa/|collab-qa project]] at https://anonscm.debian.org/cgit/collab-qa/udd.git/
 * Some '''example queries''' are provided as CGI scripts, to make it easy for everyone to run them. You can [[https://udd.debian.org/cgi-bin/|browse them]] or [[https://salsa.debian.org/qa/udd/tree/master/web/cgi-bin|view the source code]].
 * '''Database schema''': https://udd.debian.org/schema/udd.html
 * '''Source code''': available in the git repository of the [[https://salsa.debian.org/qa|qa project]] at https://salsa.debian.org/qa/udd
Line 19: Line 23:
  * [[http://public-udd-mirror.xvm.mit.edu/|public UDD mirror]] allows anyone to query UDD using PostgreSQL command-line or GUI clients   * [[https://udd-mirror.debian.net|public UDD mirror]] allows anyone to query UDD using PostgreSQL command-line or GUI clients
Line 25: Line 29:
'''udd.debian.org''' is running on '''ullmann.debian.org'''. It accepts direct SSLed connections from ''quantz'' (''qa'') and ''moszumanska'' (''alioth''). '''udd.debian.org''' is running on '''ullmann.debian.org'''. It accepts direct SSLed connections from ''master'', ''coccia'', ''quantz'' (''qa'') and ''respighi'' (''release'') ([[https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/ferm/manifests/per_host.pp#L108|firewall config]]).
Line 35: Line 39:
print cursor.fetchall()[0][0] print(cursor.fetchall()[0][0])
Line 40: Line 44:
If you want to know precisely when a specific data source is updated, you can take a look at the [[https://udd.debian.org/crontabs.txt|crontab]] file. The ''timestamps'' table can tell you when a data source was last updated. If you want to know precisely when a specific data source is updated, you can take a look at this [[https://salsa.debian.org/qa/udd/-/blob/master/rudd.conf|config file]], or at the [[https://udd.debian.org/udd-status.cgi|current UDD status]].
Line 42: Line 46:
If you do not have access to the required machines, you can access the [[http://public-udd-mirror.xvm.mit.edu/|public UDD mirror]] directly from your own machines. If you do not have access to the required machines, you can access the [[https://udd-mirror.debian.net|public UDD mirror]] directly from your own machines:
 * command line:{{{
psql "postgresql://udd-mirror:udd-mirror@udd-mirror.debian.net/udd"
}}} For quick successes, inspect the database schema on https://udd.debian.org/schema/udd.html. psql shows tables with {{{\dt}}}. Have fun.

== Credits and citing UDD ==

UDD started as a Google Summer of Code project by Christian von Essen (Neronus), co-mentored by Lucas Nussbaum, Stefano Zacchiroli and Marc 'HE' Brockschmidt. It is now mainly maintained by Lucas Nussbaum, with help from others.

If you use UDD in research work, please cite [[https://ieeexplore.ieee.org/abstract/document/5463277|this paper]]:
 L. Nussbaum and S. Zacchiroli, "The Ultimate Debian Database: Consolidating bazaar metadata for Quality Assurance and data mining," 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), Cape Town, 2010, pp. 52-61, [[https://dx.doi.org/10.1109/MSR.2010.5463277|doi: 10.1109/MSR.2010.5463277]] ([[https://hal.inria.fr/inria-00502886/document|open access preprint]]).
Line 46: Line 60:
You can '''report bugs''' against the qa.debian.org pseudo-package, using the ''udd'' usertag and user ''qa.debian.org@packages.debian.org''. ([[https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=udd;users=qa.debian.org@packages.debian.org|list of bugs]]) You can '''report bugs''' against the qa.debian.org pseudo-package, using the ''udd'' usertag and user ''qa.debian.org@packages.debian.org''. ([[https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=udd;users=qa.debian.org@packages.debian.org|list of bugs]] ; or [[https://udd.debian.org/bugs/?release=na&merged=ign&done=ign&fnewerval=7&flastmodval=7&fusertag=only&fusertagtag=udd&fusertaguser=qa.debian.org%40packages.debian.org&allbugs=1&cseverity=1&ctags=1&sortby=id&sorto=asc&format=html#results|list of bugs using UDD]])
Line 48: Line 62:
== Credits ==

UDD started as a Google Summer of Code project by Christian von Essen (Neronus), co-mentored by Lucas Nussbaum, Stefano Zacchiroli and Marc 'HE' Brockschmidt. It is now mainly maintained by Lucas Nussbaum, with help from others.
Line 54: Line 65:
 * [[https://udd.debian.org/udd.sql.gz|full dump of the database]] generated every two days. (~1.1 GB and growing, to be restored using pg_restore. See [[http://anonscm.debian.org/cgit/collab-qa/udd.git/tree/scripts/recreate-db|this script]] for an example)  * [[https://udd.debian.org/dumps/udd.dump|full dump of the database]] generated every two days. (~1.9 GB and growing, to be restored using pg_restore. See [[https://salsa.debian.org/qa/udd/blob/master/scripts/recreate-db|this script]] for an example)
Line 61: Line 72:
Even if its main goal was not to serve as a scientific resarch tool, UDD was already featured in two publications:
Even if the UDD's main goal was not to serve as a scientific research tool, it has been featured in some publications and talks:

 * Norbert Preining. '''Analyzing Debian packages with Neo4j'''. [[https://www.meetup.com/Neo4j-Online-Meetup/events/243206424/|Neo4j Online Meetup]]. Article parts [[https://www.accelia.net/column/research/09/|1]], [[https://www.accelia.net/column/research/10/|2]], [[https://www.accelia.net/column/research/11/|3]], [[https://www.youtube.com/watch?v=lpqvv36SBQw|video]]

Translation(s): English - Español - Italiano - Українська


The Ultimate Debian Database (UDD) gathers a lot of data about various aspects of Debian in the same SQL database. It allows users to easily access and combine all these data.

Data currently being imported include: Packages and Sources files, from Debian and Ubuntu, Bugs from the Debian BTS, Popularity contest, History of uploads, History of migrations to testing, Lintian, Orphaned packages, Carnivore, Debtags, Ubuntu bugs (from Launchpad), Packages in NEW queue, DDTP translations.

For more information, please contact us on #debian-qa or debian-qa@lists.debian.org (mailing list subscription and archives).

Connecting to and using UDD

udd.debian.org is running on ullmann.debian.org. It accepts direct SSLed connections from master, coccia, quantz (qa) and respighi (release) (firewall config).

  • command-line:
    • psql service=udd

    • or: psql -U guest -h udd.debian.org -p 5452 udd

  • Python:

    import psycopg2
    conn = psycopg2.connect("service=udd")
    cursor = conn.cursor()
    cursor.execute("SELECT count(*) from sources where release='sid'")
    print(cursor.fetchall()[0][0])
  • Ruby (DBI): require 'dbi' ; dbh = DBI::connect('DBI:Pg:dbname=udd;port=5452;host=udd.debian.org', 'guest') 

  • Ruby (PG): require 'pg'; conn = PG.connect({:host => 'udd.debian.org', :port => 5452, :user => 'guest', :dbname => 'udd'}) 

If you want to know precisely when a specific data source is updated, you can take a look at this config file, or at the current UDD status.

If you do not have access to the required machines, you can access the public UDD mirror directly from your own machines:

  • command line:

    psql "postgresql://udd-mirror:udd-mirror@udd-mirror.debian.net/udd"

    For quick successes, inspect the database schema on https://udd.debian.org/schema/udd.html. psql shows tables with \dt. Have fun.

Credits and citing UDD

UDD started as a Google Summer of Code project by Christian von Essen (Neronus), co-mentored by Lucas Nussbaum, Stefano Zacchiroli and Marc 'HE' Brockschmidt. It is now mainly maintained by Lucas Nussbaum, with help from others.

If you use UDD in research work, please cite this paper:

  • L. Nussbaum and S. Zacchiroli, "The Ultimate Debian Database: Consolidating bazaar metadata for Quality Assurance and data mining," 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), Cape Town, 2010, pp. 52-61, doi: 10.1109/MSR.2010.5463277 (open access preprint).

Improving UDD

If you want to help improve UDD, you can set up your own instance as described on UltimateDebianDatabase/Hacking. You can report bugs against the qa.debian.org pseudo-package, using the udd usertag and user qa.debian.org@packages.debian.org. (list of bugs ; or list of bugs using UDD)

Other resources

See also :

References

Even if the UDD's main goal was not to serve as a scientific research tool, it has been featured in some publications and talks:

SubPages