Differences between revisions 302 and 390 (spanning 88 versions)
Revision 302 as of 2021-03-12 15:43:58
Size: 21846
Editor: Praveen A
Comment: add warning about fasttrack-staging repo
Revision 390 as of 2022-08-11 13:41:49
Size: 13169
Editor: Praveen A
Comment: gitlab 15.1.4 is now available
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
Gitlab is a git front end with repository management, graphs, links, goodies, commands to run, and review capabilities similar in feel to a self-hosted GitHub.  Gitlab is a git front end with repository management, graphs, links, goodies, commands to run, and review capabilities similar in feel to a self-hosted GitHub.

A new feature release of gitlab flows in the following direction: [[/devel|devel]] -> [[/staging|staging]] -> experimental -> unstable -> [[/fasttrack-staging|fasttrack-staging]] -> fasttrack (experimental and unstable may be skipped during freeze and transitions).
Line 8: Line 10:
New upstream releases (including security updates) are announced at https://about.gitlab.com/releases/categories/releases/. New upstream releases (including security updates) are announced at https://about.gitlab.com/releases/categories/releases/ and available as RSS feed https://about.gitlab.com/security-releases.xml.
Line 21: Line 23:
{{{#!wiki important
Make sure `contrib` section is enabled for official repos.
}}}

== Story of Gitlab packaging project/FAQ about Gitlab packaging ==

Gitlab Inc sponsored the packaging effort for 6+ years (2016-2022). Currently looking for donations at https://opencollective.com/debian-gitlab.

Though the dependencies are so many, the work benefits Debian immensely by maintaining many important build tools like webpack, rollup, babel, npm, yarn... And frameworks like ruby on rails.

The situation in packaging JavaScript modules is considerably improved over the years thanks to this packaging work. It took over 2 years for packaging handlebars_assets gem for diaspora because its embedded JavaScript library was using tools like gulp, webpack, jison etc and none of this was packaged for Debian. The whole JavaScript build tools were untouched for years in Debian and instead reverse engineering of the build tools for specific libraries was the norm (for example jquery), it was very hard and not scalable.

Out of 1600+ node modules gitlab needs, we have 1200+ modules already packaged. It is not impossible to have it in main (it was in main earlier before the nodejs modules exploded exponentially and if some more people join the team, currently it is mostly a single person work and at times some new people helping out with a handful of dependencies. I don't think it is much harder if you take whole gnome or KDE as comparison. We are able to pull it off because of team work. This project have also brought many new contributors to Debian.

While we might love long term supported releases, calling anything moving fast as insane and not able to adapt to change is a recipe for Debian becoming irrelevant over time. Many just want distros to be only a base os for shipping containers but that is not necessarily a good thing for users to have dependency on a single project for updates and lose choice and flexibility.

We created https://fasttrack.debian.net to serve gitlab as it did not fit into a debian stable release cycle and currently not just gitlab, but matrix synapse and virtual box is also shipped via FastTrack.
Line 22: Line 42:
 1. From gitlab 14.7.7: gitlab user needs to write to /var/lib/gitlab/.gem so if you installed gems manually as root user, you will need to update permissions or remove this directory. Gitlab package now installs required unpackaged gems in that directory automatically.
 1. From gitlab 14.2.6: postgresql database should be updated to 13 before installing gitlab 14.2. See [[gitlab/postgresql-update]] for steps to upgrading postgresql to 13.
 1. From gitlab 14.0.10: We no longer require a work around to install grpc and google-protobuf gems from rubygems.org, packaged versions now work.
 1. From 2021 July 23: We no longer require the personal repo of gitlab maintainer to install gitlab, all golang packages can now be added to fasttrack.debian.net directly. Thanks to Akshay S Dinesh for fixing this long pending bug (See https://salsa.debian.org/fasttrack-team/support/-/issues/8 for details).
 1. New workaround from gitlab 13.12.3: You need to use google-protobuf from rubygems.org ([[gitlab#gitlab-sidekiq_service_failure_work_around_.28install_google-protobuf_from_rubygems.org.29|see below for details]]).
 1. New from gitlab 13.10.4: Since this version includes gitlab-workhorse golang binary, it was moved to people.debian.org/~praveen/gitaly and should now include contrib section.
Line 25: Line 51:
 1. [[#From_gitlab_13.2.3|New in gitlab 13.2.3]]

== Buster Fast Track (Recommended) ==
Gitlab 13.8.5 is available in [[FastTrack|unofficial fasttrack repo]] targeting buster as base distribution. (no open security issues).

A video demo of the whole installation process with commentary on packaging challenges and troubleshooting steps is available [[https://peertube.debian.social/videos/watch/playlist/05c6b4d1-61b1-4751-bf61-6541307e9e51|on Debian Social Peertube instance]].

{{{#!wiki important
Some packages are still needed from personal repo of gitlab maintainer because golang packages cannot be uploaded in fasttrack.debian.net because of a [[https://salsa.debian.org/fasttrack-team/support/-/issues/8|bug in dak setup]]. Please note the new URL of this repo if you are upgrading from an older version (people.debian.org/~praveen/gitlab is now people.debian.org/~praveen/gitaly and gpg key used for signing is also changed).
}}}

{{{#!wiki important
Make sure `contrib` section is enabled for `buster` and `buster/updates`
}}}

Modify `/etc/apt/sources.list` to add official `buster-backports` and `contrib` section if missing:
{{{
deb http://deb.debian.org/debian buster main contrib
deb http://security.debian.org/ buster/updates main contrib
deb http://deb.debian.org/debian buster-backports main contrib
}}}

=== From gitlab 13.2.3 ===
{{{#!wiki important
Now gitaly is also built with ruby2.7 so it is now moved to buster-fasttrack from buster-backports. See the new repository link below.
}}}

Import fasttrack archive keyring:
{{{
# apt -t buster-backports install fasttrack-archive-keyring

== Bullseye FastTrack (Recommended) ==

gitlab 15.1.4 is available in Bullseye FastTrack (no open security issues).

Add fasttrack.debian.net as a trusted repo for apt,
{{{
# apt install fasttrack-archive-keyring ca-certificates
Line 59: Line 63:
deb https://fasttrack.debian.net/debian/ buster-fasttrack main contrib deb https://fasttrack.debian.net/debian/ bullseye-fasttrack main contrib
Line 61: Line 65:
deb https://fasttrack.debian.net/debian/ buster-backports main contrib
}}}

Eventually the packages in this repo will be moved to fasttrack or official backports after fixing https://salsa.debian.org/fasttrack-team/support/-/issues/8 (Need help)
{{{
deb https://people.debian.org/~praveen/gitaly buster-backports main
deb https://people.debian.org/~praveen/gitaly buster-fasttrack main
}}}

Optional: To avoid some hours of delay between upload and availability in archive
{{{
deb http://incoming.debian.org/debian-buildd buildd-buster-backports main contrib
}}}

Refresh list of available packages:
deb https://fasttrack.debian.net/debian/ bullseye-backports-staging main contrib
}}}

You will also need official bullseye-backports repo:
{{{
deb http://deb.debian.org/debian bullseye-backports main contrib
}}}

Note: You may also use a mirror of fasttrack repo like http://mirror.linux.pizza/debian-fasttrack/

and install gitlab
Line 78: Line 79:
}}}

You may encounter the following error message:
{{{
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0B76920762A6B785
}}}

If so, these commands can help (trust these repositories):
{{{
# wget https://people.debian.org/~praveen/gitaly/praveen.key.new.asc
# apt-key add praveen.key.new.asc
}}}

{{{#!wiki important
If you are upgrading ruby version to 2.7 (if your current ruby version is 2.5), you will need to use dist-upgrade first. Make sure /var/lib/gems/2.5.0/ is empty if you see any issues after the ruby version upgrade.

See https://git.fosscommunity.in/debian-ruby/TaskTracker/-/issues/166 for current status of ruby 2.7 in fasttrack.
}}}

=== sassc regression ===
{{{#!wiki important
Currently gitlab installation is broken due to libsass in buster-backports . See [[DebianBug:953415]] for more details.

A work around is to downgrade libsass and libsass-dev to versions in buster and hold it at the older version.

# apt install libsass1/buster libsass-dev/buster && apt-mark hold libsass1 libsass-dev
}}}

=== gitlab crash work around (install grpc from rubygems.org) ===

To avoid installation crashing with an error message like,

{{{
/usr/share/rubygems-integration/all/gems/gitaly-13.4.6/ruby/proto/gitaly/lint_pb.rb:17: [BUG] Segmentation fault at 0x0000000000000000
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux-gnu]
}}}

{{{#!wiki important
We have to use versions of grpc from rubygems.org (don't forget to remove this version when ruby-grpc is fixed in debian)

# apt -t buster-fasttrack install ruby2.7

# gem install -v 1.30.2 grpc

and follow [[gitlab#manually_installing_ruby_dependencies]] if you are seeing this crash again

See [[DebianBug:966653]] for details.
}}}

=== Use apt pinning for resolving dependencies ===

Since buster-backports and buster-fasttrack have lower apt priorities compared to buster, we have to manually give higher priority to packages we want installed from these suites.

{{{
Line 136: Line 83:
''Note: https://gitlab.debian.net is running on this version.''

== Buster Fast Track Staging ==

gitlab 13.8.5 is available in fasttrack-staging.

If you'd like to test gitlab packages before they are uploaded to buster-fasttrack, you can add the following lines to /etc/apt/sources.list.

{{{
deb https://people.debian.org/~praveen/fasttrack-staging buster-backports main contrib
deb https://people.debian.org/~praveen/fasttrack-staging buster-fasttrack main contrib
}}}

and install gitlab

{{{
# apt install gitlab-apt-pin-preferences
== Unstable (be careful when updating packages) ==
Gitlab 13.4.7 is available in unstable (many security release behind, see experimental for latest security updates).

We try to keep the version in unstable in a good shape with the latest security updates, but some times dependency updates break gitlab.

Now install gitlab

{{{
Line 156: Line 94:
{{{#!wiki important
This repo may have newer incompatible versions of dependencies when a new gitlab version is being prepared, so use this repo only when you want to install gitlab from this repo.
}}}

== Unstable (be careful when updating packages) ==
Gitlab 13.4.7 is available in unstable (many security release behind, see experimental).

We try to keep the version in unstable in a good shape with the latest security updates, but some times dependency updates break GitLab.

Now install gitlab

{{{
# apt install gitlab
}}}
Line 190: Line 113:
{{{#!wiki important
Downgrade ruby-autoprefixer-rails and node-autoprefixer packages to 10.3.1.0+dfsg1+~cs14.6.19-2 using https://snapshot.debian.org/package/node-autoprefixer/10.3.1.0%2Bdfsg1%2B%7Ecs14.6.19-2/ and hold them to this version.

# apt-mark hold node-autoprefixer ruby-autoprefixer-rails

See [[DebianBug:1009245]]
}}}

{{{#!wiki important
Downgrade ruby-github-linguist till gitaly is adapted to find languages.json from gem-install layout.

Get older version from https://snapshot.debian.org/package/ruby-github-linguist/7.12.2-1/

# apt-mark hold ruby-github-linguist
}}}
Line 192: Line 131:
gitlab 13.7.8 is available in experimental (no open security issues). But this version is not yet compatible with rugged/libgit2 1.x so not recommended (See [[DebianBug:979563]]) This issue is resolved in gitlab 13.9.3 (See [[gitlab/wip]] to install this version). The fix will be uploaded to experimental soon.

As a workaround, use snapshot.debian.org to install ruby-rugged 0.28 and its dependencies/reverse dependencies,

First install gitlab,
{{{
# apt install gitlab/experimental ruby-rack/experimental ruby-gitlab-labkit/experimental \
ruby-jaeger-client/experimental ruby-prometheus-client-mmap/experimental ruby-gitaly/experimental gitaly/experimental
}}}

Now downgrade ruby-rugged and its dependencies/reverse dependencies,
{{{
# wget http://snapshot.debian.org/archive/debian/20201103T210717Z/pool/main/r/ruby-rugged/ruby-rugged_0.28.4.1%2Bds-1%2Bb3_amd64.deb
# wget http://snapshot.debian.org/archive/debian/20200410T214949Z/pool/main/libg/libgit2/libgit2-28_0.28.5%2Bdfsg.1-1_amd64.deb
# dpkg -i ruby-rugged_0.28.4.1+ds-1+b3_amd64.deb libgit2-28_0.28.5+dfsg.1-1_amd64.deb
}}}

{{{
# wget http://snapshot.debian.org/archive/debian/20200113T205619Z/pool/main/r/ruby-gollum-rugged-adapter/ruby-gollum-rugged-adapter_0.4.4.2-2_all.deb
# dpkg -i ruby-gollum-rugged-adapter_0.4.4.2-2_all.deb
}}}

{{{
# wget http://snapshot.debian.org/archive/debian/20201201T205219Z/pool/main/r/ruby-gollum-lib/ruby-gollum-lib_4.2.7.9-3_all.deb
# dpkg -i ruby-gollum-lib_4.2.7.9-3_all.deb
}}}

{{{
# apt install ruby-licensee/unstable
}}}

{{{
# apt-mark hold ruby-licensee ruby-gollum-lib ruby-gollum-rugged-adapter ruby-rugged
}}}

Modify /usr/share/gitaly/ruby/Gemfile by applying this patch (save this as /tmp/gitaly-Gemfile.patch)

{{{
diff --git a/Gemfile.orig b/Gemfile
index ab7fcfe..b90a487 100644
--- a/Gemfile.orig
+++ b/Gemfile
@@ -1,12 +1,12 @@
 source 'https://rubygems.org'
 
-gem 'rugged', '~> 1.0'
+gem 'rugged', '~> 0.28'
 gem 'github-linguist', '~> 7.12', require: 'linguist'
 gem 'github-markup', '~> 1.7'
 gem 'activesupport', '~> 6.0.3', '>= 6.0.3.3'
 gem 'rdoc', '~> 6.0'
-gem 'gitlab-gollum-lib', '~> 4.2.7.10.gitlab.1', require: false
-gem 'gitlab-gollum-rugged_adapter', '~> 0.4.4.3.gitlab.1', require: false
+gem 'gitlab-gollum-lib', '~> 4.2.7.9', require: false
+gem 'gitlab-gollum-rugged_adapter', '~> 0.4.4.2', require: false
 gem 'grpc', '~> 1.30', '>= 1.30.2'
 gem 'sentry-raven', '~> 3.0', require: false
 gem 'faraday', '~> 1.0'
@@ -17,7 +17,7 @@ gem 'gitlab-labkit', '~> 0.13.2'
 
 # Detects the open source license the repository includes
 # This version needs to be in sync with GitLab CE/EE
-gem 'licensee', '~> 9.14'
+gem 'licensee', '~> 8.9'
 
 gem 'google-protobuf', '~> 3.12'
 }}}

{{{
# cd /usr/share/gitaly/ruby
# patch < /tmp/gitaly-Gemfile.patch
}}}

Regenerate Gemfile.lock
{{{
# cd /usr/share/gitaly/ruby
# truncate -s0 Gemfile.lock
# sudo -u gitlab bundle install --local
}}}

Now restart gitlab and gitaly services

{{{
# apt -f install
# systemctl restart gitlab gitaly
}}}
gitlab 15.1.4 is available in experimental (no open security issues).
Line 287: Line 141:
gitlab 13.8.5 is available in staging (when ruby-pg-query is accepted in the archive, this will be uploaded to experimental).

When gitlab is not ready for experimental/unstable (for example some dependencies need to clear NEW queue), it will be available from this repo.

Add this repo,

{{{
deb https://people.debian.org/~praveen/staging/ experimental main contrib
}}}

and install

{{{
# apt install gitlab ruby-grape-path-helpers/experimental ruby-marginalia/experimental \
ruby-invisible-captcha/experimental ruby-net-ldap/experimental ruby-grape/experimental \
ruby-html-pipeline/experimental ruby-puma-worker-killer/experimental \
ruby-state-machines-activerecord/experimental ruby-acts-as-taggable-on/experimental \
ruby-jira/experimental ruby-asana/experimental puma/experimental
}}}

== Gitlab on Stretch ==

This is moved to [[gitlab/stretch]] now.
When gitlab is not ready for official experimental/unstable (for example some dependencies need to clear NEW queue), it will be available from this repo.

{{{#!wiki important
You will have to follow the notes mentioned in unstable section above.
}}}

This section moved to [[gitlab/staging]].

== Gitlab on older releases ==

These versions no longer receive any secuity updates and it is recommended to upgrade to Debian 11 Bullseye to continue recieving security updates.

 * [[gitlab/buster|Debian 10 Buster]]
 * [[gitlab/stretch|Debian 9 Stretch]]
Line 314: Line 159:
To get a working configuration for your version the best thing to do is to follow the ([[https://gitlab.com/gitlab-org/gitlab-recipes/-/tree/master/web-server/apache|gitlab-recipes repository]]) instructions.

Basically you will have to :
Disable nginx
Set www-data as web server user.
Allow Modules dependencies
# mod_rewrite
# mod_ssl
# mod_proxy
# mod_proxy_http
# mod_headers

{{{
a2enmod rewrite proxy_http headers
}}}

and as says in recipe

Allow gitlab-workhorse to listen on port 8181, edit or create /etc/default/gitlab and change or add the following:
{{{
 gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp -listenAddr 127.0.0.1:8181 -authBackend http://127.0.0.1:8080"
}}}


== Gitlab Common Errors ==

If you have this kind of error
{{{
  Cloning into'public-test-project-ssh'...
  Received disconnect from YOUR_SERVER_IP port 22:2: Too many authentication failures
  Disconnected from YOUR_SERVER_IP port 22
  fatal: Can't read distant repository.
}}}
on an ssh clone
{{{
  git clone gitlab@ssh.your-gitlab-instance.org:public-test-group/public-test-project.git public-test-project-ssh
}}}
{{{#!wiki important
Note : In your command line host should match your ssh hostname for gitlab as group and project name.
}}}

To be sure you should find a line like the following in /var/log/auth.log

{{{
User gitlab from YOUR_CLIENT_IP not allowed because not listed in AllowUsers
}}}

You may have to allow gitlab user in /etc/ssh/sshd_config

{{{
AllowUsers user1 user2 gitlab
AllowGroups user1 group2 gitlab
}}}
{{{#!wiki important
Note : user1, user2, group2 are here for the example adapt to your setup.
}}}
Do not forget to restart the service

{{{
service ssh restart
}}}

== Debug installation and configuration ==

Gitlab is composed of several parts. As always logs (/var/log/gitlab), debian specific documentation (/usr/share/doc/gitlab/) and ([[https://docs.gitlab.com/ |official]]) documentation provide lots of information.

`gitlab-rake` is convenience script to easily run rake commands provided by gitlab in debian environment.

{{{
# gitlab-rake gitlab:check
# gitlab-rake gitlab:env:info
}}}

`gitlab-rails-console` is another convenience script to launch a rails console in gitlab debian environment.

{{{
# gitlab-rails-console
}}}

Just make sure you don't have any gems installed from rubygems.org conflicting with debian packaged versions of the gems required by gitlab. `/var/lib/gems/<ruby_version>` (`/var/lib/gems/2.5.0` or `/var/lib/gems/2.7.0`) should not have any gems.

== rake aborted errors during installation ==
If your update failed for some reason (usually when dependencies are not satisfied for `bundle install --local`), and you retry installation after fixing dependencies, you may see errors like
{{{
rake aborted!
NameError: uninitialized constant Gitlab::Patch::ActiveRecordQueryCache
/usr/share/gitlab/config/initializers/active_record_query_cache.rb:3:in `<top (required)>'
}}}

You will have to manually remove problem creating initializers from `/etc/gitlab/initializers`. A list of such initializers can be found from gitlab source package or from https://salsa.debian.org/ruby-team/gitlab in debian/maintscript` file.

All lines starting with `rm_conffile` and corresponding versions are what you need to check.

Additionally, if there is any extra files present in your system at /etc/gitlab/initializers that is not installed by the current package as shown in `dpkg -L gitlab | grep '/etc/gitlab/initializers'`, you need to remove that. We usually try to remove obsolete initializers automatically, but sometimes we miss some files.

The following command returns the list of obsolete initializer files:
{{{
for fname in /etc/gitlab/initializers/*; do dpkg -L gitlab | grep -qFx "$fname" || echo "$fname"; done
}}}

== manually installing ruby dependencies ==
When ruby dependencies are installed manually, we need to update gitlab and gitaly's Gemfile.lock to use the recently installed version. See [[DebianBug:944698]] for details.

{{{
cd /usr/share/gitlab
truncate -s0 Gemfile.lock
sudo -u gitlab bundle install --local # use the gitlab username instead of gitlab if you changed it
cd /usr/share/gitaly/ruby
truncate -s0 Gemfile.lock
sudo -u gitlab bundle install --local # use the gitlab username instead of gitlab if you changed it
}}}

== remote: GitLab: 401 Unauthorized ==
If you got this error after a purge and reinstall when trying git push, then it is because the old .gitlab_shell_secret was not cleaned up during purge. So you need to purge again and remove this file manually before you reinstall

{{{
# rm /usr/share/gitlab-shell/.gitlab_shell_secret
}}}

TODO: Clean this in purge

== your account is awaiting approval from your GitLab administrator ==
If you saw this message after creating an account for the first time,

`You have signed up successfully. However, we could not sign you in because your account is awaiting approval from your GitLab administrator.`

You can approve the user and grant Administration access by using `gitlab-rails-console` command. The steps are documented in `/usr/share/doc/gitlab/README.Debian.gz`.
The ([[https://gitlab.com/gitlab-org/gitlab-recipes/-/tree/master/web-server/apache|gitlab-recipes repository]]) instructions are wrong - apache supports proxying to UNIX sockets so there's no need to change any gitlab configuration to use TCP.

Basically you will have to:
 * disable nginx
 * enable apache modules:
  * mod_rewrite
  * mod_ssl (if needed)
  * mod_proxy
  * mod_proxy_http
  * mod_headers
 * add/modify apache configuration file

{{{
a2enmod rewrite ssl proxy_http headers
}}}

See below for Apache configuration file example (using Let's Encrypt SSL certificates and HTTP to HTTPS redirect). Replace YOUR_SERVER_FQDN string with your domain (e.g. git.example.org).

{{{
<VirtualHost *:80>
 ServerName YOUR_SERVER_FQDN
 Redirect / https://YOUR_SERVER_FQDN/
</VirtualHost>

<VirtualHost *:443>
 SSLCertificateFile /etc/letsencrypt/live/YOUR_SERVER_FQDN/fullchain.pem
 SSLCertificateKeyFile /etc/letsencrypt/live/YOUR_SERVER_FQDN/privkey.pem
 Include /etc/letsencrypt/options-ssl-apache.conf
 Header add Strict-Transport-Security: "max-age=15768000;includeSubdomains"
 ProxyPreserveHost On

 ServerName YOUR_SERVER_FQDN

 # Ensure that encoded slashes are not decoded but left in their encoded state.
 # http://doc.gitlab.com/ce/api/projects.html#get-single-project
 AllowEncodedSlashes NoDecode

 <Location />
  Require all granted
  ProxyPassReverse https://YOUR_SERVER_FQDN/
 </Location>

 RewriteEngine on
 #Forward all requests to gitlab-workhorse except existing files like error documents
 RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
 RewriteCond %{REQUEST_URI} ^/uploads/.*
 RewriteRule .* unix:/run/gitlab/gitlab-workhorse.socket|http://YOUR_SERVER_FQDN%{REQUEST_URI} [P,QSA,NE]

 RequestHeader set X_FORWARDED_PROTO 'https'
 RequestHeader set X-Forwarded-Ssl on

 # needed for downloading attachments
 DocumentRoot /var/lib/gitlab/public

 # Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.
 ErrorDocument 404 /404.html
 ErrorDocument 422 /422.html
 ErrorDocument 500 /500.html
 ErrorDocument 502 /502.html
 ErrorDocument 503 /503.html

 LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
 ErrorLog /var/log/apache2/YOUR_SERVER_FQDN_error.log
 CustomLog /var/log/apache2/YOUR_SERVER_FQDN_forwarded.log common_forwarded
 CustomLog /var/log/apache2/YOUR_SERVER_FQDN_access.log combined env=!dontlog
 CustomLog /var/log/apache2/YOUR_SERVER_FQDN.log combined
</VirtualHost>

}}}

== Troubleshooting common issues ==

See [[gitlab/troubleshooting]]

== Tips and tweaks ==

See [[/tweaks|Tweaks for GitLab]]
Line 443: Line 238:
You can reach the maintainers of the gitlab package via https://wiki.debian.org/gitlab/BackportChecklist

 1. Matrix at #debian-gitlab:poddery.com ([[https://chat.poddery.com/#/room/#debian-gitlab:poddery.com|join via browser]])
 1. IRC #debian-gitlab on OFTC network ([[https://webchat.oftc.net/?channels=debian-gitlab&uio=MT11bmRlZmluZWQb1|join via browser]])

== Maintainer's corner ==

 1. Installing gitlab on an lxc container to test - See [[gitlab/lxc]]
 1. Backport/Fasttrack checklist - See [[gitlab/BackportChecklist]]
 1. Dependencies Transitions checklist - See [[gitlab/DependenciesTransitions]]
 1. Managing Components - See [[gitlab/ManagingComponents]]
 1. Mixing apt and yarn installed node modules - See [[gitlab/yarn]]
 1. Manually running webpack to troubleshoot bundling issues - See [[gitlab/webpack]]
 1. Notes for next version of gitlab being prepared - See [[gitlab/wip]]

=== TODO ===
 1. Aim to get gitlab back in main by bullseye release by packaging all node dependencies. [[Javascript/Nodejs/Tasks/gitlab|current status]] This is now an Outreachy project (from December 2019 to March 2020).
 1. Get autopkgtest working so we can detect problems when someone updates dependencies without coordinating with us. [[https://git.fosscommunity.in/debian-ruby/TaskTracker/-/issues/154|Current Status]]

=== Prioritizing tasks ===

We should try to pick tasks according to the following priority list,

 1. Update to latest security release (in unstable/experimental and fasttrack). Remember to add the CVEs fixed in the release (listed in the release blog post) to `debian/changelog`. Subscribe to `Security Alerts: Notification alerts to critical security updates.` at https://about.gitlab.com/company/preference-center/ to get notified by email about new security releases. Usually we have security releases once or twice per month.
 1. Update to next feature release (in unstable/experimental and fasttrack) before support for current stable release ends (usually 3 months). See [[Teams/Ruby/Packaging/newUpstreamRailsApp]].
 1. We should try to keep the same upstream version in both unstable/experimental and fasttrack to avoid maintaining two versions. This means not starting a feature update close to a scheduled security release (usually at the end or starting of every month). Because a feature update may take more days to complete and involve complexities compared to a security update.
 1. Getting packages from experimental to unstable can take time as it involves transitions and coordinating with many others, so we should prioritize moving directly from experimental to fasttrack before we attempt moving from experimental to unstable.
 1. Package remaining node modules and switch to packaged versions. Update `0740-use-packaged-modules.patch` by removing the newly packaged module from `package.json` and adding it to `Depends` in `debian/control`. It is better to backport the packaged modules first to avoid updates getting blocked (sometimes you may need to backport a large number of build dependencies). See https://git.fosscommunity.in/debian-ruby/TaskTracker/-/issues/175
You can reach the maintainers of the gitlab package via

 1. Matrix at `#debian-gitlab:poddery.com` ([[https://chat.poddery.com/#/room/#debian-gitlab:poddery.com|join via browser]])
 1. IRC `#debian-gitlab` on OFTC network ([[https://webchat.oftc.net/?channels=debian-gitlab&uio=MT11bmRlZmluZWQb1|join via browser]])
 1. Via XMPP at `#debian-gitlab#poddery.com@matrix.org`

Documentation for [[gitlab/maintainers_corner|gitlab maintainers]]

Gitlab is a git front end with repository management, graphs, links, goodies, commands to run, and review capabilities similar in feel to a self-hosted ?GitHub.

A new feature release of gitlab flows in the following direction: devel -> staging -> experimental -> unstable -> fasttrack-staging -> fasttrack (experimental and unstable may be skipped during freeze and transitions).

See /omnibus if you are looking for instructions to install upstream provided packages.

New upstream releases (including security updates) are announced at https://about.gitlab.com/releases/categories/releases/ and available as RSS feed https://about.gitlab.com/security-releases.xml.

Debian is running its own Instance of GitLab under https://salsa.debian.org, which is not based on the packaged version.

Note 1: For a smooth upgrade experience, always stay on the latest major version of gitlab. For example, if latest version of gitlab is 12.0.0 and you are currently on 11.3.6, then update to 12.0.0 as soon as possible, certainly before the debian package is updated to 13.x. Ideally, you should update as soon as a new version with security updates is available.

Note 2: It is recommended to subscribe to changes in this wiki page or frequently check this page for updates. Alternatively, you can subscribe to https://tracker.debian.org/pkg/gitlab to get notified when new versions are uploaded.

Make sure contrib section is enabled for official repos.

Story of Gitlab packaging project/FAQ about Gitlab packaging

Gitlab Inc sponsored the packaging effort for 6+ years (2016-2022). Currently looking for donations at https://opencollective.com/debian-gitlab.

Though the dependencies are so many, the work benefits Debian immensely by maintaining many important build tools like webpack, rollup, babel, npm, yarn... And frameworks like ruby on rails.

The situation in packaging ?JavaScript modules is considerably improved over the years thanks to this packaging work. It took over 2 years for packaging handlebars_assets gem for diaspora because its embedded ?JavaScript library was using tools like gulp, webpack, jison etc and none of this was packaged for Debian. The whole ?JavaScript build tools were untouched for years in Debian and instead reverse engineering of the build tools for specific libraries was the norm (for example jquery), it was very hard and not scalable.

Out of 1600+ node modules gitlab needs, we have 1200+ modules already packaged. It is not impossible to have it in main (it was in main earlier before the nodejs modules exploded exponentially and if some more people join the team, currently it is mostly a single person work and at times some new people helping out with a handful of dependencies. I don't think it is much harder if you take whole gnome or KDE as comparison. We are able to pull it off because of team work. This project have also brought many new contributors to Debian.

While we might love long term supported releases, calling anything moving fast as insane and not able to adapt to change is a recipe for Debian becoming irrelevant over time. Many just want distros to be only a base os for shipping containers but that is not necessarily a good thing for users to have dependency on a single project for updates and lose choice and flexibility.

We created https://fasttrack.debian.net to serve gitlab as it did not fit into a debian stable release cycle and currently not just gitlab, but matrix synapse and virtual box is also shipped via FastTrack.

New changes

  1. From gitlab 14.7.7: gitlab user needs to write to /var/lib/gitlab/.gem so if you installed gems manually as root user, you will need to update permissions or remove this directory. Gitlab package now installs required unpackaged gems in that directory automatically.
  2. From gitlab 14.2.6: postgresql database should be updated to 13 before installing gitlab 14.2. See gitlab/postgresql-update for steps to upgrading postgresql to 13.

  3. From gitlab 14.0.10: We no longer require a work around to install grpc and google-protobuf gems from rubygems.org, packaged versions now work.
  4. From 2021 July 23: We no longer require the personal repo of gitlab maintainer to install gitlab, all golang packages can now be added to fasttrack.debian.net directly. Thanks to Akshay S Dinesh for fixing this long pending bug (See https://salsa.debian.org/fasttrack-team/support/-/issues/8 for details).

  5. New workaround from gitlab 13.12.3: You need to use google-protobuf from rubygems.org (see below for details).

  6. New from gitlab 13.10.4: Since this version includes gitlab-workhorse golang binary, it was moved to people.debian.org/~praveen/gitaly and should now include contrib section.
  7. New in gitlab 13.6.7-1: New user created after a fresh installation should be approved using gitlab-rails-console. See gitlab#your_account_is_awaiting_approval_from_your_GitLab_administrator for steps to approving users and creating a user with Administrative privileges.

  8. New in gitlab 13.3.8-1: You will need to install grpc gem from rubygems.org gem install -v 1.30.2 grpc (more details below). If you run gitaly on a different machine, you will need to do this on that machine as well.

  9. New in gitlab 13.2.6-3: We have switched to puma as application server replacing unicorn. Upstream already made the switch from 12.9 and unicorn support will be dropped in gitlab 14.0. They saw a memory reduction of 37% in gitlab.com after the switch. See more details about this switch at this upstream blog post https://about.gitlab.com/blog/2020/07/08/migrating-to-puma-on-gitlab/. puma defaults to listening only on unix sockets and if you are running gitaly on a different machine, then you will have to edit /etc/gitlab/puma.rb to bind to tcp:// url as well.

Bullseye FastTrack (Recommended)

gitlab 15.1.4 is available in Bullseye FastTrack (no open security issues).

Add fasttrack.debian.net as a trusted repo for apt,

# apt install fasttrack-archive-keyring ca-certificates

And add the following lines for fasttrack repo:

deb https://fasttrack.debian.net/debian/ bullseye-fasttrack main contrib
# For dependency packages not in testing only temporarily due to freeze, transitions or delayed by backports-new or NEW.
deb https://fasttrack.debian.net/debian/ bullseye-backports-staging main contrib

You will also need official bullseye-backports repo:

deb http://deb.debian.org/debian bullseye-backports main contrib

Note: You may also use a mirror of fasttrack repo like http://mirror.linux.pizza/debian-fasttrack/

and install gitlab

# apt update
# apt install gitlab-apt-pin-preferences
# apt install gitlab

Unstable (be careful when updating packages)

Gitlab 13.4.7 is available in unstable (many security release behind, see experimental for latest security updates).

We try to keep the version in unstable in a good shape with the latest security updates, but some times dependency updates break gitlab.

Now install gitlab

# apt install gitlab

Known Issues

Currently gitlab installation is broken due to a change in libsass. See 953415 for more details.

A work around is to downgrade libsass and libsass-dev to 3.6.1. Use http://snapshot.debian.org/package/libsass/3.6.1-1/ and hold it at the older version.

# wget http://snapshot.debian.org/archive/debian/20190705T210019Z/pool/main/libs/libsass/libsass-dev_3.6.1-1_amd64.deb

# wget http://snapshot.debian.org/archive/debian/20190705T210019Z/pool/main/libs/libsass/libsass1_3.6.1-1_amd64.deb

# dpkg -i libsass1_3.6.1-1_amd64.deb libsass-dev_3.6.1-1_amd64.deb

# apt-mark hold libsass1 libsass-dev

See gitlab#gitlab_crash_work_around_.28install_grpc_from_rubygems.org.29 for installing grpc gem from rubygems.org (just use apt install ruby2.7 for installing ruby)

Downgrade ruby-autoprefixer-rails and node-autoprefixer packages to 10.3.1.0+dfsg1+~cs14.6.19-2 using https://snapshot.debian.org/package/node-autoprefixer/10.3.1.0%2Bdfsg1%2B%7Ecs14.6.19-2/ and hold them to this version.

# apt-mark hold node-autoprefixer ruby-autoprefixer-rails

See 1009245

Downgrade ruby-github-linguist till gitaly is adapted to find languages.json from gem-install layout.

Get older version from https://snapshot.debian.org/package/ruby-github-linguist/7.12.2-1/

# apt-mark hold ruby-github-linguist

Experimental - During freeze and transitions

gitlab 15.1.4 is available in experimental (no open security issues).

If you are using experimental for the first time, check DebianExperimental.

You will have to follow the notes mentioned in unstable section above.

Experimental/unstable Staging

When gitlab is not ready for official experimental/unstable (for example some dependencies need to clear NEW queue), it will be available from this repo.

You will have to follow the notes mentioned in unstable section above.

This section moved to gitlab/staging.

Gitlab on older releases

These versions no longer receive any secuity updates and it is recommended to upgrade to Debian 11 Bullseye to continue recieving security updates.

Gitlab with apache2

Gitlab can use apache instead of nginx. The (gitlab-recipes repository) instructions are wrong - apache supports proxying to UNIX sockets so there's no need to change any gitlab configuration to use TCP.

Basically you will have to:

  • disable nginx
  • enable apache modules:
    • mod_rewrite
    • mod_ssl (if needed)
    • mod_proxy
    • mod_proxy_http
    • mod_headers
  • add/modify apache configuration file

a2enmod rewrite ssl proxy_http headers

See below for Apache configuration file example (using Let's Encrypt SSL certificates and HTTP to HTTPS redirect). Replace YOUR_SERVER_FQDN string with your domain (e.g. git.example.org).

<VirtualHost *:80>
        ServerName YOUR_SERVER_FQDN
        Redirect / https://YOUR_SERVER_FQDN/
</VirtualHost>

<VirtualHost *:443>
        SSLCertificateFile    /etc/letsencrypt/live/YOUR_SERVER_FQDN/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/YOUR_SERVER_FQDN/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
        Header add Strict-Transport-Security: "max-age=15768000;includeSubdomains"
        ProxyPreserveHost On

        ServerName YOUR_SERVER_FQDN

        # Ensure that encoded slashes are not decoded but left in their encoded state.
        # http://doc.gitlab.com/ce/api/projects.html#get-single-project
        AllowEncodedSlashes NoDecode

        <Location />
                Require all granted
                ProxyPassReverse https://YOUR_SERVER_FQDN/
        </Location>

        RewriteEngine on
        #Forward all requests to gitlab-workhorse except existing files like error documents
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
        RewriteCond %{REQUEST_URI} ^/uploads/.*
        RewriteRule .* unix:/run/gitlab/gitlab-workhorse.socket|http://YOUR_SERVER_FQDN%{REQUEST_URI} [P,QSA,NE]

        RequestHeader set X_FORWARDED_PROTO 'https'
        RequestHeader set X-Forwarded-Ssl on

        # needed for downloading attachments
        DocumentRoot /var/lib/gitlab/public

        # Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.
        ErrorDocument 404 /404.html
        ErrorDocument 422 /422.html
        ErrorDocument 500 /500.html
        ErrorDocument 502 /502.html
        ErrorDocument 503 /503.html

        LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
        ErrorLog /var/log/apache2/YOUR_SERVER_FQDN_error.log
        CustomLog /var/log/apache2/YOUR_SERVER_FQDN_forwarded.log common_forwarded
        CustomLog /var/log/apache2/YOUR_SERVER_FQDN_access.log combined env=!dontlog
        CustomLog /var/log/apache2/YOUR_SERVER_FQDN.log combined
</VirtualHost>

Troubleshooting common issues

See gitlab/troubleshooting

Tips and tweaks

See Tweaks for GitLab

Contact maintainers

You can reach the maintainers of the gitlab package via

  1. Matrix at #debian-gitlab:poddery.com (join via browser)

  2. IRC #debian-gitlab on OFTC network (join via browser)

  3. Via XMPP at #debian-gitlab#poddery.com@matrix.org

Documentation for gitlab maintainers