Differences between revisions 306 and 310 (spanning 4 versions)
Revision 306 as of 2021-03-14 15:52:25
Size: 22334
Editor: Praveen A
Revision 310 as of 2021-03-14 20:15:25
Size: 21851
Editor: Praveen A
Comment: ruby-doorkeeper-openid-connect should be downgraded as well
Deletions are marked like this. Additions are marked like this.
Line 131: Line 131:
# apt install ruby-doorkeeper=5.3.0-2~bpo10+1

# apt-mark hold ruby-doorkeeper
# apt install ruby-doorkeeper=5.3.0-2~bpo10+1  ruby-doorkeeper-openid-connect=1.7.4-2~bpo10+1

# apt-mark hold ruby-doorkeeper ruby-doorkeeper-openid-connect
Line 162: Line 162:
This repo may have newer incompatible versions of dependencies when a new gitlab version is being prepared, so use this repo only when you want to install gitlab from this repo. This repo may have newer incompatible versions of dependencies when a new gitlab version is being prepared, so use this repo only when you want to install gitlab from this repo. This repo should not be used till gitlab 13.9 is uploaded.
Line 304: Line 304:
gitlab 13.8.5 is available in staging (when ruby-pg-query is accepted in the archive, this will be uploaded to experimental).
Line 308: Line 306:
Add this repo,

deb https://people.debian.org/~praveen/staging/ experimental main contrib

and install

# apt install gitlab ruby-grape-path-helpers/experimental ruby-marginalia/experimental \
ruby-invisible-captcha/experimental ruby-net-ldap/experimental ruby-grape/experimental \
ruby-html-pipeline/experimental ruby-puma-worker-killer/experimental \
ruby-state-machines-activerecord/experimental ruby-acts-as-taggable-on/experimental \
ruby-jira/experimental ruby-asana/experimental puma/experimental
This section moved to [[gitlab/staging]].
Line 326: Line 310:
This is moved to [[gitlab/stretch]] now. This section moved to [[gitlab/stretch]].

Gitlab is a git front end with repository management, graphs, links, goodies, commands to run, and review capabilities similar in feel to a self-hosted ?GitHub.

See /omnibus if you are looking for instructions to install upstream provided packages.

New upstream releases (including security updates) are announced at https://about.gitlab.com/releases/categories/releases/.

Debian is running its own Instance of GitLab under https://salsa.debian.org, which is not based on the packaged version.

Note 1: For a smooth upgrade experience, always stay on the latest major version of gitlab. For example, if latest version of gitlab is 12.0.0 and you are currently on 11.3.6, then update to 12.0.0 as soon as possible, certainly before the debian package is updated to 13.x. Ideally, you should update as soon as a new version with security updates is available.

Note 2: It is recommended to subscribe to changes in this wiki page or frequently check this page for updates. Alternatively, you can subscribe to https://tracker.debian.org/pkg/gitlab to get notified when new versions are uploaded.

New changes

  1. New in gitlab 13.6.7-1: New user created after a fresh installation should be approved using gitlab-rails-console. See gitlab#your_account_is_awaiting_approval_from_your_GitLab_administrator for steps to approving users and creating a user with Administrative privileges.

  2. New in gitlab 13.3.8-1: You will need to install grpc gem from rubygems.org gem install -v 1.30.2 grpc (more details below). If you run gitaly on a different machine, you will need to do this on that machine as well.

  3. New in gitlab 13.2.6-3: We have switched to puma as application server replacing unicorn. Upstream already made the switch from 12.9 and unicorn support will be dropped in gitlab 14.0. They saw a memory reduction of 37% in gitlab.com after the switch. See more details about this switch at this upstream blog post https://about.gitlab.com/blog/2020/07/08/migrating-to-puma-on-gitlab/. puma defaults to listening only on unix sockets and if you are running gitaly on a different machine, then you will have to edit /etc/gitlab/puma.rb to bind to tcp:// url as well.

  4. New in gitlab 13.2.3

Buster Fast Track (Recommended)

Gitlab 13.8.5 is available in unofficial fasttrack repo targeting buster as base distribution. (no open security issues).

A video demo of the whole installation process with commentary on packaging challenges and troubleshooting steps is available on Debian Social Peertube instance.

Some packages are still needed from personal repo of gitlab maintainer because golang packages cannot be uploaded in fasttrack.debian.net because of a bug in dak setup. Please note the new URL of this repo if you are upgrading from an older version (people.debian.org/~praveen/gitlab is now people.debian.org/~praveen/gitaly and gpg key used for signing is also changed).

Make sure contrib section is enabled for buster and buster/updates

Modify /etc/apt/sources.list to add official buster-backports and contrib section if missing:

deb http://deb.debian.org/debian buster main contrib
deb http://security.debian.org/ buster/updates main contrib
deb http://deb.debian.org/debian buster-backports main contrib 

From gitlab 13.2.3

Now gitaly is also built with ruby2.7 so it is now moved to buster-fasttrack from buster-backports. See the new repository link below.

Import fasttrack archive keyring:

# apt -t buster-backports install fasttrack-archive-keyring

And add the following lines for fasttrack repo:

deb https://fasttrack.debian.net/debian/ buster-fasttrack main contrib
# For dependency packages not in testing only temporarily due to freeze, transitions or delayed by backports-new or NEW.
deb https://fasttrack.debian.net/debian/ buster-backports main contrib

Eventually the packages in this repo will be moved to fasttrack or official backports after fixing https://salsa.debian.org/fasttrack-team/support/-/issues/8 (Need help)

deb https://people.debian.org/~praveen/gitaly buster-backports main
deb https://people.debian.org/~praveen/gitaly buster-fasttrack main

Optional: To avoid some hours of delay between upload and availability in archive

deb http://incoming.debian.org/debian-buildd buildd-buster-backports main contrib

Refresh list of available packages:

# apt update

You may encounter the following error message:

The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0B76920762A6B785

If so, these commands can help (trust these repositories):

# wget https://people.debian.org/~praveen/gitaly/praveen.key.new.asc
# apt-key add praveen.key.new.asc

If you are upgrading ruby version to 2.7 (if your current ruby version is 2.5), you will need to use dist-upgrade first. Make sure /var/lib/gems/2.5.0/ is empty if you see any issues after the ruby version upgrade.

See https://git.fosscommunity.in/debian-ruby/TaskTracker/-/issues/166 for current status of ruby 2.7 in fasttrack.

sassc regression

Currently gitlab installation is broken due to libsass in buster-backports . See 953415 for more details.

A work around is to downgrade libsass and libsass-dev to versions in buster and hold it at the older version.

# apt install libsass1/buster libsass-dev/buster && apt-mark hold libsass1 libsass-dev

gitlab crash work around (install grpc from rubygems.org)

To avoid installation crashing with an error message like,

/usr/share/rubygems-integration/all/gems/gitaly-13.4.6/ruby/proto/gitaly/lint_pb.rb:17: [BUG] Segmentation fault at 0x0000000000000000
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux-gnu]

We have to use versions of grpc from rubygems.org (don't forget to remove this version when ruby-grpc is fixed in debian)

# apt -t buster-fasttrack install ruby2.7

# gem install -v 1.30.2 grpc

and follow gitlab#manually_installing_ruby_dependencies if you are seeing this crash again

ruby-doorkeeper 5.5 broke gitlab

ruby-doorkeeper update broke gitlab, you can hold the package at old version till gitlab 13.9 is available, which fixes this issue.

# apt install ruby-doorkeeper=5.3.0-2~bpo10+1 ruby-doorkeeper-openid-connect=1.7.4-2~bpo10+1

# apt-mark hold ruby-doorkeeper ruby-doorkeeper-openid-connect

See 966653 for details.

ruby-asciidoctor-kroki: version compatibility

For gitlab up to v13.8.x the version 0.2.2 of asciidoctor-kroki needs to be installed. For gitlab v13.9 or later, version 0.4.0 of asciidoctor-kroki is suitable.

Use apt pinning for resolving dependencies

Since buster-backports and buster-fasttrack have lower apt priorities compared to buster, we have to manually give higher priority to packages we want installed from these suites.

# apt install gitlab-apt-pin-preferences
# apt install gitlab


Note: https://gitlab.debian.net is running on this version.

Buster Fast Track Staging

If you'd like to test gitlab packages before they are uploaded to buster-fasttrack, you can add the following lines to /etc/apt/sources.list.

This repo may have newer incompatible versions of dependencies when a new gitlab version is being prepared, so use this repo only when you want to install gitlab from this repo. This repo should not be used till gitlab 13.9 is uploaded.

deb https://people.debian.org/~praveen/fasttrack-staging buster-backports main contrib
deb https://people.debian.org/~praveen/fasttrack-staging buster-fasttrack main contrib

and install gitlab

# apt install gitlab-apt-pin-preferences
# apt install gitlab

Unstable (be careful when updating packages)

Gitlab 13.4.7 is available in unstable (many security release behind, see experimental).

We try to keep the version in unstable in a good shape with the latest security updates, but some times dependency updates break GitLab.

Now install gitlab

# apt install gitlab

Known Issues

Currently gitlab installation is broken due to a change in libsass. See 953415 for more details.

A work around is to downgrade libsass and libsass-dev to 3.6.1. Use http://snapshot.debian.org/package/libsass/3.6.1-1/ and hold it at the older version.

# wget http://snapshot.debian.org/archive/debian/20190705T210019Z/pool/main/libs/libsass/libsass-dev_3.6.1-1_amd64.deb

# wget http://snapshot.debian.org/archive/debian/20190705T210019Z/pool/main/libs/libsass/libsass1_3.6.1-1_amd64.deb

# dpkg -i libsass1_3.6.1-1_amd64.deb libsass-dev_3.6.1-1_amd64.deb

# apt-mark hold libsass1 libsass-dev

See gitlab#gitlab_crash_work_around_.28install_grpc_from_rubygems.org.29 for installing grpc gem from rubygems.org (just use apt install ruby2.7 for installing ruby)

Experimental - During freeze and transitions

gitlab 13.7.8 is available in experimental (no open security issues). But this version is not yet compatible with rugged/libgit2 1.x so not recommended (See 979563) This issue is resolved in gitlab 13.9.3 (See ?gitlab/wip to install this version). The fix will be uploaded to experimental soon.

As a workaround, use snapshot.debian.org to install ruby-rugged 0.28 and its dependencies/reverse dependencies,

First install gitlab,

# apt install gitlab/experimental ruby-rack/experimental ruby-gitlab-labkit/experimental \
ruby-jaeger-client/experimental ruby-prometheus-client-mmap/experimental ruby-gitaly/experimental gitaly/experimental

Now downgrade ruby-rugged and its dependencies/reverse dependencies,

# wget http://snapshot.debian.org/archive/debian/20201103T210717Z/pool/main/r/ruby-rugged/ruby-rugged_0.28.4.1%2Bds-1%2Bb3_amd64.deb
# wget http://snapshot.debian.org/archive/debian/20200410T214949Z/pool/main/libg/libgit2/libgit2-28_0.28.5%2Bdfsg.1-1_amd64.deb
# dpkg -i ruby-rugged_0.28.4.1+ds-1+b3_amd64.deb libgit2-28_0.28.5+dfsg.1-1_amd64.deb

# wget http://snapshot.debian.org/archive/debian/20200113T205619Z/pool/main/r/ruby-gollum-rugged-adapter/ruby-gollum-rugged-adapter_0.4.4.2-2_all.deb
# dpkg -i ruby-gollum-rugged-adapter_0.4.4.2-2_all.deb

# wget http://snapshot.debian.org/archive/debian/20201201T205219Z/pool/main/r/ruby-gollum-lib/ruby-gollum-lib_4.2.7.9-3_all.deb
# dpkg -i ruby-gollum-lib_4.2.7.9-3_all.deb

# apt install ruby-licensee/unstable

# apt-mark hold ruby-licensee ruby-gollum-lib ruby-gollum-rugged-adapter ruby-rugged

Modify /usr/share/gitaly/ruby/Gemfile by applying this patch (save this as /tmp/gitaly-Gemfile.patch)

diff --git a/Gemfile.orig b/Gemfile
index ab7fcfe..b90a487 100644
--- a/Gemfile.orig
+++ b/Gemfile
@@ -1,12 +1,12 @@
 source 'https://rubygems.org'
-gem 'rugged', '~> 1.0'
+gem 'rugged', '~> 0.28'
 gem 'github-linguist', '~> 7.12', require: 'linguist'
 gem 'github-markup', '~> 1.7'
 gem 'activesupport', '~> 6.0.3', '>='
 gem 'rdoc', '~> 6.0'
-gem 'gitlab-gollum-lib', '~>', require: false
-gem 'gitlab-gollum-rugged_adapter', '~>', require: false
+gem 'gitlab-gollum-lib', '~>', require: false
+gem 'gitlab-gollum-rugged_adapter', '~>', require: false
 gem 'grpc', '~> 1.30', '>= 1.30.2'
 gem 'sentry-raven', '~> 3.0', require: false
 gem 'faraday', '~> 1.0'
@@ -17,7 +17,7 @@ gem 'gitlab-labkit', '~> 0.13.2'
 # Detects the open source license the repository includes
 # This version needs to be in sync with GitLab CE/EE
-gem 'licensee', '~> 9.14'
+gem 'licensee', '~> 8.9'
 gem 'google-protobuf', '~> 3.12'

# cd /usr/share/gitaly/ruby
# patch < /tmp/gitaly-Gemfile.patch

Regenerate Gemfile.lock

# cd /usr/share/gitaly/ruby
# truncate -s0 Gemfile.lock
# sudo -u gitlab bundle install --local

Now restart gitlab and gitaly services

# apt -f install
# systemctl restart gitlab gitaly

If you are using experimental for the first time, check DebianExperimental.

You will have to follow the notes mentioned in unstable section above.

Experimental/unstable Staging

When gitlab is not ready for experimental/unstable (for example some dependencies need to clear NEW queue), it will be available from this repo.

This section moved to gitlab/staging.

Gitlab on Stretch

This section moved to gitlab/stretch.

Gitlab with apache2

Gitlab can use apache instead of nginx. To get a working configuration for your version the best thing to do is to follow the (gitlab-recipes repository) instructions.

Basically you will have to : Disable nginx Set www-data as web server user. Allow Modules dependencies # mod_rewrite # mod_ssl # mod_proxy # mod_proxy_http # mod_headers

a2enmod rewrite proxy_http headers

and as says in recipe

Allow gitlab-workhorse to listen on port 8181, edit or create /etc/default/gitlab and change or add the following:

 gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp -listenAddr -authBackend"

Gitlab Common Errors

If you have this kind of error

  Cloning into'public-test-project-ssh'...
  Received disconnect from YOUR_SERVER_IP port 22:2: Too many authentication failures
  Disconnected from YOUR_SERVER_IP port 22
  fatal: Can't read distant repository.

on an ssh clone

  git clone gitlab@ssh.your-gitlab-instance.org:public-test-group/public-test-project.git public-test-project-ssh

Note : In your command line host should match your ssh hostname for gitlab as group and project name.

To be sure you should find a line like the following in /var/log/auth.log

User gitlab from YOUR_CLIENT_IP not allowed because not listed in AllowUsers

You may have to allow gitlab user in /etc/ssh/sshd_config

AllowUsers user1 user2 gitlab
AllowGroups user1 group2 gitlab

Note : user1, user2, group2 are here for the example adapt to your setup.

Do not forget to restart the service

service ssh restart

Debug installation and configuration

Gitlab is composed of several parts. As always logs (/var/log/gitlab), debian specific documentation (/usr/share/doc/gitlab/) and (official) documentation provide lots of information.

gitlab-rake is convenience script to easily run rake commands provided by gitlab in debian environment.

# gitlab-rake gitlab:check
# gitlab-rake gitlab:env:info

gitlab-rails-console is another convenience script to launch a rails console in gitlab debian environment.

# gitlab-rails-console

Just make sure you don't have any gems installed from rubygems.org conflicting with debian packaged versions of the gems required by gitlab. /var/lib/gems/<ruby_version> (/var/lib/gems/2.5.0 or /var/lib/gems/2.7.0) should not have any gems.

rake aborted errors during installation

If your update failed for some reason (usually when dependencies are not satisfied for bundle install --local), and you retry installation after fixing dependencies, you may see errors like

rake aborted!
NameError: uninitialized constant Gitlab::Patch::ActiveRecordQueryCache
/usr/share/gitlab/config/initializers/active_record_query_cache.rb:3:in `<top (required)>'

You will have to manually remove problem creating initializers from /etc/gitlab/initializers. A list of such initializers can be found from gitlab source package or from https://salsa.debian.org/ruby-team/gitlab in debian/maintscript` file.

All lines starting with rm_conffile and corresponding versions are what you need to check.

Additionally, if there is any extra files present in your system at /etc/gitlab/initializers that is not installed by the current package as shown in dpkg -L gitlab | grep '/etc/gitlab/initializers', you need to remove that. We usually try to remove obsolete initializers automatically, but sometimes we miss some files.

The following command returns the list of obsolete initializer files:

for fname in /etc/gitlab/initializers/*; do dpkg -L gitlab | grep -qFx "$fname" || echo "$fname"; done

manually installing ruby dependencies

When ruby dependencies are installed manually, we need to update gitlab and gitaly's Gemfile.lock to use the recently installed version. See 944698 for details.

cd /usr/share/gitlab
truncate -s0 Gemfile.lock
sudo -u gitlab bundle install --local # use the gitlab username instead of gitlab if you changed it
cd /usr/share/gitaly/ruby
truncate -s0 Gemfile.lock
sudo -u gitlab bundle install --local # use the gitlab username instead of gitlab if you changed it

remote: GitLab: 401 Unauthorized

If you got this error after a purge and reinstall when trying git push, then it is because the old .gitlab_shell_secret was not cleaned up during purge. So you need to purge again and remove this file manually before you reinstall

# rm /usr/share/gitlab-shell/.gitlab_shell_secret

TODO: Clean this in purge

your account is awaiting approval from your GitLab administrator

If you saw this message after creating an account for the first time,

You have signed up successfully. However, we could not sign you in because your account is awaiting approval from your GitLab administrator.

You can approve the user and grant Administration access by using gitlab-rails-console command. The steps are documented in /usr/share/doc/gitlab/README.Debian.gz.

Contact maintainers

You can reach the maintainers of the gitlab package via https://wiki.debian.org/gitlab/BackportChecklist

  1. Matrix at #debian-gitlab:poddery.com (join via browser)

  2. IRC #debian-gitlab on OFTC network (join via browser)

Maintainer's corner

  1. Installing gitlab on an lxc container to test - See gitlab/lxc

  2. Backport/Fasttrack checklist - See gitlab/BackportChecklist

  3. Dependencies Transitions checklist - See gitlab/DependenciesTransitions

  4. Managing Components - See gitlab/ManagingComponents

  5. Mixing apt and yarn installed node modules - See gitlab/yarn

  6. Manually running webpack to troubleshoot bundling issues - See gitlab/webpack

  7. Notes for next version of gitlab being prepared - See ?gitlab/wip


  1. Aim to get gitlab back in main by bullseye release by packaging all node dependencies. current status This is now an Outreachy project (from December 2019 to March 2020).

  2. Get autopkgtest working so we can detect problems when someone updates dependencies without coordinating with us. Current Status

Prioritizing tasks

We should try to pick tasks according to the following priority list,

  1. Update to latest security release (in unstable/experimental and fasttrack). Remember to add the CVEs fixed in the release (listed in the release blog post) to debian/changelog. Subscribe to Security Alerts: Notification alerts to critical security updates. at https://about.gitlab.com/company/preference-center/ to get notified by email about new security releases. Usually we have security releases once or twice per month.

  2. Update to next feature release (in unstable/experimental and fasttrack) before support for current stable release ends (usually 3 months). See Teams/Ruby/Packaging/newUpstreamRailsApp.

  3. We should try to keep the same upstream version in both unstable/experimental and fasttrack to avoid maintaining two versions. This means not starting a feature update close to a scheduled security release (usually at the end or starting of every month). Because a feature update may take more days to complete and involve complexities compared to a security update.
  4. Getting packages from experimental to unstable can take time as it involves transitions and coordinating with many others, so we should prioritize moving directly from experimental to fasttrack before we attempt moving from experimental to unstable.
  5. Package remaining node modules and switch to packaged versions. Update 0740-use-packaged-modules.patch by removing the newly packaged module from package.json and adding it to Depends in debian/control. It is better to backport the packaged modules first to avoid updates getting blocked (sometimes you may need to backport a large number of build dependencies). See https://git.fosscommunity.in/debian-ruby/TaskTracker/-/issues/175