Differences between revisions 23 and 24
Revision 23 as of 2015-03-18 10:57:21
Size: 6684
Comment:
Revision 24 as of 2016-07-02 18:13:45
Size: 1500
Comment:
Deletions are marked like this. Additions are marked like this.
Line 17: Line 17:
= Getting access to the DB = = Development environment with Vagrant =
Line 19: Line 19:
There are two ways to get access to a UDD DB:
 * access it remotely from alioth.debian.org or ullmann.debian.org
  * [++] very easy to setup
  * [+] data is from the live instance
  * [--] queries are slow: loading dmd.cgi takes ~ 30s
 * setup postgresql locally and import a dump
  * [--] harder to setup
  * [++] queries are much faster
  * [-] needs manual updates to keep data up-to-date
  * [+] You can work offline
The easiest way to hack on UDD is using the Vagrant development environment.
Line 30: Line 21:
== Access it remotely == Just run {{{vagrant up}}} in the UDD source code. See [[http://anonscm.debian.org/cgit/collab-qa/udd.git/tree/Vagrantfile|Vagrantfile]] and the [[http://anonscm.debian.org/cgit/collab-qa/udd.git/tree/vagrant|vagrant scripts]] for details.
Line 32: Line 23:
{{{
ssh -NL 5452:udd.debian.org:5452 aliothlogin@alioth.debian.org
}}}
Or directly on ullmann:
{{{
ssh -NL 5452:localhost:5452 debianlogin@ullmann.debian.org
}}}
This development environment supports setting up tunnels to access the main instance remotely, or dumping/importing data locally, depending on what you want to work on.
Line 40: Line 25:

== Create a Local Replica of UDD ==

As we can see from [[http://udd.debian.org/|UDD]] main page, there is a [[http://udd.debian.org/udd.sql.gz|dump]] available.

So if we want to do some test, maybe a local replica of UDD can be handy, and that's what this page will describe.

=== Install Needed Packages ===

UDD runs on PostgreSQL, so we need to install it plus the necessary extensions for UDD:

{{{
sudo apt-get install postgresql postgresql-plperl-9.4 postgresql-9.4-debversion
}}}

In case the latter two packages don’t work, you might need to upgrade the version number.

=== Setup PostgreSQL Server ===

Modify /etc/postgresql/9.4/main/pg_hba.conf to trust local connections:
{{{
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
}}}

Modify /etc/postgresql/9.4/main/postgresql.conf to listen on port 5452 (same as the real UDD instance on ullmann.debian.org):
{{{
port = 5452
}}}

Restart postgresql with this new configuration:
{{{
service postgresql restart
}}}

Now we got an "empty" PostgreSQL db server, let's import some bits.

First we create some users/roles by switching to the {{{postgres}}} user which is used for database administration, then running createuser:

{{{
$ sudo su postgres
$ createuser udd
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
}}}



This is the UDD user on the database, where the UDD objects will live.

{{{
$ createdb -T template0 -E SQL_ASCII udd
$ echo 'CREATE EXTENSION debversion' | psql udd
}}}

We create the database, named 'udd', forcing the encoding to SQL_ASCII, since that's the format of the export.
We base it off 'template0' because 'template1' (the default) might be set to UTF8 which prevents creation of new SQL_ASCII databases.

UDD is actually using a "guest" user:
{{{
createuser -lDRS guest
psql udd -c 'GRANT usage ON schema public TO PUBLIC;'
psql udd -c 'GRANT select ON all tables in schema public TO PUBLIC;'
}}}

=== Import UDD Dump ===

Now we only need to download the dump and import it, with:

{{{
$ zcat /path/to/udd.sql.gz | sudo -u postgres psql udd 2>&1 | tee udd-import.log
}}}

that will import the dump into udd db redirecting messages to udd-import.log.

In case of error messages, you may identify the problematic query by issueing again the psql command with the "-e" option :
{{{
$ zcat /path/to/udd.sql.gz | sudo -u postgres psql -e udd 2>&1 | tee udd-import.log
}}}

=== Reimport UDD ===

If we already have a UDD replicated on our local machine and want to reimport the UDD dump, we need to:

{{{
$ sudo su postgres
$ dropdb udd
$ createdb -T template0 -E SQL_ASCII udd
$ echo 'CREATE EXTENSION debversion' | psql udd
$ zcat /path/to/udd.sql.gz | psql udd
}}}

so simply drop the db, recreate it and import the dump again.

= Setup a local web server =

Install some Ruby packages, needed for dmd.cgi
{{{
sudo apt-get install ruby-dbi ruby-dbd-pg ruby-debian ruby-oj vpim
}}}

== Apache2 ==
Here is guidelines to provide UDD over localhost using '''apache2'''

{{{
sudo apt-get install apache2-mpm-worker
}}}

Minimal configuration that works, edit /etc/apache2/sites-enabled/000-default in the <!VirtualHost> declaration, add

{{{
Alias /udd /path/to/udd/web
 <Location /udd>
  AddHandler cgi-script .cgi
  Options +ExecCGI +Indexes
                # For wheezy / Apache 2.2:
  Order allow,deny
  Allow from all
                # For jessie / Apache 2.4:
                Require all granted
 </Location>
}}}

Restart apache with this new configuration
{{{
sudo service apache2 restart
}}}

 * Try: http://localhost/udd/dmd.cgi


== NGinx ==
If you want to use Nginx, setup is a bit more difficult because NGinx don't support CGI out-of-the-box.

{{{
sudo apt-get install nginx-full fcgiwrap
}}}

Minimal configuration that works, edit /etc/nginx/sites-available/default and replace with :

{{{
server {
        listen 80 default_server;

        root /path/to/udd/web;
        index index.html index.htm;
        server_name localhost;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ /index.html;
        }

        location ~ \.cgi$ {
          gzip off;
          fastcgi_pass unix:/var/run/fcgiwrap.socket;
          include /etc/nginx/fastcgi_params;
          # fastcgi_param SCRIPT_FILENAME /usr/lib$fastcgi_script_name;
        }
}
}}}

Restart nginx with this new configuration
{{{
sudo service nginx restart
}}}

 * Try: http://localhost/dmd.cgi

== Common problems ==
  * check file-system permissions (must be world-readable, or at least readable by www-data)
  * ensure you are running PostgreSQL on 5452 port
  * check that there is a "guest" account on PostgreSQL
If you don't want to use Vagrant, your best bet is to look at the [[http://anonscm.debian.org/cgit/collab-qa/udd.git/tree/vagrant/provision.sh|Vagrant provision script]] to understand how to setup your own instance manually.

Getting the code

Browse on gitweb

# clone over git://
git clone git://git.debian.org/collab-qa/udd.git

# or over ssh:
git clone ssh://git.debian.org/git/collab-qa/udd.git

Development environment with Vagrant

The easiest way to hack on UDD is using the Vagrant development environment.

Just run vagrant up in the UDD source code. See Vagrantfile and the vagrant scripts for details.

This development environment supports setting up tunnels to access the main instance remotely, or dumping/importing data locally, depending on what you want to work on.

If you don't want to use Vagrant, your best bet is to look at the Vagrant provision script to understand how to setup your own instance manually.

Hack, hack, hack

Please submit patches to debian-qa mailing list.

Bugs can be filed against the qa.debian.org pseudo-package, with user qa.debian.org@packages.debian.org and usertag udd. See this bug for an example.

(see this bug for the correct pseudo-headers).