# GitLab

Un des objectif de l'approche DevOps, est d'automatiser/industrialiser toutes actions qui sera amenée à être répétée plusieurs fois. Une approche est donc de supprimer la non valeur ajouté au travail.

# CI

# Example

Extracted from https://gitlab.com/baptiste-dauphin/doc/-/blob/master/.gitlab-ci.yml

image: node:10

before_script:
  - npm install gitbook-cli -g
  - gitbook fetch 3.2.3
  - gitbook install

test:
  stage: test
  script:
    - gitbook build . public
  rules:
    - if: $CI_COMMIT_BRANCH != "master"
      changes:
      - docs/**/*
      - .git*
      - book.json
    
pages:
  stage: deploy
  script:
    - gitbook build . public
  artifacts:
    paths:
      - public
    expire_in: 1 week
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      changes:
      - docs/**/*
      - .git*
      - book.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# Predefined Variables

https://docs.gitlab.com/ee/ci/variables/predefined_variables.html

# Runner

Get a secret from

From a GitLab Runner :

docker-compose --file docker-compose.yml run --rm vault vault kv get -field="$KEY" path/to/secret

# Self Hosted

image: docker:19.03.13

services:
  - docker:19.03.13-dind

stages:
  - upload

variables:
  DOCKER_HOST: tcp://docker:2375
  # This instructs Docker not to start over TLS. (i.e. not listen on 2376)
  DOCKER_TLS_CERTDIR: ""
  HTTPS_PROXY: "http://10.10.10.10:9999"
  HTTP_PROXY: "http://10.10.10.10:9999"
  NO_PROXY: "localhost,docker,127.0.0.1,.example.org,.example.org."

before_script:
  - echo "$CI_REGISTRY_PASSWORD" | docker login
      --username "$CI_REGISTRY_USER"
      --password-stdin "$CI_REGISTRY"

upload:
  stage: upload
  script:
    - docker build --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# Installation

# Packages

GitLab can be installed in most GNU/Linux distributions and in a number of cloud providers. To get the best experience from GitLab, you need to balance performance, reliability, ease of administration (backups, upgrades and troubleshooting), and cost of hosting.

There are many ways you can install GitLab depending on your platform:

Omnibus GitLab: The official deb/rpm packages that contain a bundle of GitLab and the various components it depends on like PostgreSQL, Redis, Sidekiq, etc. recommended Global config : /etc/gitlab/gitlab.rb
GitLab Helm chart: The cloud native Helm chart for installing GitLab and all its components on Kubernetes.
Docker: The Omnibus GitLab packages dockerized.
Source: Install GitLab and all its components from scratch.

If in doubt, choose Omnibus: The Omnibus GitLab packages are mature, scalable, support high availability and are used today on GitLab.com. The Helm charts are recommended for those who are familiar with Kubernetes.

# Via Omnibus package

Go to https://about.gitlab.com/install/#debian
Click on the distribution, it will dynamically change terminal steps lines.
Follow steps

# Manually configuring HTTPS

You can only changes gitlab global settings inside gitlab.rb and tell gitlab to restart gitlab-ctl reconfigure.
Gitlab Omnibus manage NGINX, Postgres, Redis BY ITSELF
That's why you can not manage theme directly as usual.

systemctl status nginx
● nginx.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

1
2
3
4

By default, Omnibus GitLab does not use HTTPS.
To enable HTTPS for the domain gitlab.example.com.

1 - Edit the external_url in /etc/gitlab/gitlab.rb.

external_url "https://gitlab.example.com"

nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.example.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.example.com.key"

registry['enable'] = true
registry_external_url 'https://registry.example.com'

registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/registry.example.com.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/registry.example.com.key"

1
2
3
4
5
6
7
8
9
10

2 - Create the /etc/gitlab/ssl directory and copy your key and certificate there

mkdir -p /etc/gitlab/ssl
chmod 755 /etc/gitlab/ssl
cp gitlab.example.com.key gitlab.example.com.crt /etc/gitlab/ssl/

1
2
3

Make sure you use the full certificate chain in order to prevent SSL errors when clients connect. The full certificate chain order should consist of the server certificate first, followed by all intermediate certificates.

3 - Restart Now, reconfigure GitLab:

gitlab-ctl reconfigure

If nothing happends, you can force reload nginx

gitlab-ctl hup nginx

# Update the SSL Certificates

If the content of your SSL certificates has been updated, but no configuration changes have been made to gitlab.rb, then gitlab-ctl reconfigure will not affect NGINX. Instead, run sudo gitlab-ctl hup nginx to cause NGINX to reload the existing configuration and new certificates gracefully.

gitlab-ctl hup nginx

sourceopen in new window

# Docker container registry (v2)

# Registry client - index.py (prefered)

One of my friend created a python script to request a registry original scriptopen in new window

# Registry client - reg

Docopen in new window

CI example : To use this example, change the IMAGE_TAG variable to match your needs:

stages:
  - build
  - clean

build_image:
  image: docker:19.03.12
  stage: build
  services:
    - docker:19.03.12-dind
  variables:
    IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $IMAGE_TAG .
    - docker push $IMAGE_TAG
  only:
    - branches
  except:
    - master

delete_image:
  image: docker:19.03.12
  stage: clean
  services:
    - docker:19.03.12-dind
  variables:
    IMAGE_TAG: $CI_PROJECT_PATH:$CI_COMMIT_REF_SLUG
    REG_SHA256: ade837fc5224acd8c34732bf54a94f579b47851cc6a7fd5899a98386b782e228
    REG_VERSION: 0.16.1
  before_script:
    - apk add --no-cache curl
    - curl --fail --show-error --location "https://github.com/genuinetools/reg/releases/download/v$REG_VERSION/reg-linux-amd64" --output /usr/local/bin/reg
    - echo "$REG_SHA256  /usr/local/bin/reg" | sha256sum -c -
    - chmod a+x /usr/local/bin/reg
  script:
    - /usr/local/bin/reg rm -d --auth-url $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $IMAGE_TAG
  only:
    - branches
  except:
    - master

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

# Installation of reg

Go to releasesopen in new window
Follow the instruction for your os arch For me amd64 - linux

# Export the sha256sum for verification.
export REG_SHA256="ade837fc5224acd8c34732bf54a94f579b47851cc6a7fd5899a98386b782e228"

# Download and check the sha256sum.
sudo curl -fSL "https://github.com/genuinetools/reg/releases/download/v0.16.1/reg-linux-amd64" -o "/usr/local/bin/reg" \
  && echo "${REG_SHA256}  /usr/local/bin/reg" | sha256sum -c - \
  && sudo chmod a+x "/usr/local/bin/reg"

# Run it!
reg -h

1
2
3
4
5
6
7
8
9
10

docker login --username baptiste registry.example.com

reg ls registry.example.com

1
2
3

# Garbage collection

Official docopen in new windowDanger: By running the built-in garbage collection command, it will cause downtime to the Container Registry. If you run this command on an instance in an environment where one of your other instances is still writing to the Registry storage, referenced manifests will be removed. To avoid that, make sure Registry is set to read-only mode before proceeding.

Container Registry can use considerable amounts of disk space. To clear up some unused layers, the registry includes a garbage collect command.

GitLab offers a set of APIs to manipulate the Container Registry and aid the process of removing unused tags. Currently, this is exposed using the API, ut in the future, these controls will be migrated to the GitLab interface.

Project maintainers can delete Container Registry tags in bulk periodically based on their own criteria, however, this alone does not recycle data, it only unlinks tags from manifests and image blobs. To recycle the Container Registry data in the whole GitLab instance, you can use the built-in command provided by gitlab-ctl.

# Understanding the content-addressable layers

Consider the following example, where you first build the image:

# This builds a image with content of sha256:111111
docker build -t my.registry.com/my.group/my.project:latest .
docker push my.registry.com/my.group/my.project:latest

1
2
3

Now, you do overwrite :latest with a new version:

# This builds a image with content of sha256:222222
docker build -t my.registry.com/my.group/my.project:latest .
docker push my.registry.com/my.group/my.project:latest

1
2
3

Now, the :latest tag points to manifest of sha256:222222. However, due to the architecture of registry, this data is still accessible when pulling the image my.registry.com/my.group/my.project@sha256:111111, even though it is no longer directly accessible via the :latest tag.

# Recycling unused tags

There are a couple of considerations you need to note before running the built-in command:

The built-in command will stop the registry before it starts the garbage collection.
The garbage collect command takes some time to complete, depending on the amount of data that exists.
If you changed the location of registry configuration file, you will need to specify its path.
After the garbage collection is done, the registry should start up automatically.

If you did not change the default location of the configuration file, run:

sudo gitlab-ctl registry-garbage-collect

This command will take some time to complete, depending on the amount of layers you have stored.

You may also remove all unreferenced manifests, although this is a way more destructive operation, and you should first understand the implications.

# Removing unused layers not referenced by manifests

WARNING : This tool option -m has a lot of bug on gitlab issues, so it's not recommended to run it without snapshot / backup.

The GitLab Container Registry follows the same default workflow as Docker Distribution: retain all layers, even ones that are unreferenced directly to allow all content to be accessed using context addressable identifiers.

However, in most workflows, you don't care about old layers if they are not directly referenced by the registry tag. The registry-garbage-collect command supports the -m switch to allow you to remove all unreferenced manifests and layers that are not directly accessible via tag:

sudo gitlab-ctl registry-garbage-collect -m

# Running the garbage collection on schedule

Ideally, you want to run the garbage collection of the registry regularly on a weekly basis at a time when the registry is not being in-use. The simplest way is to add a new crontab job that it will run periodically once a week.

Create a file under /etc/cron.d/registry-garbage-collect:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Run every Sunday at 04:05am
5 4 * * 0  root gitlab-ctl registry-garbage-collect

1
2
3
4
5

# Expiration policy / Cleanup policy

Renamed from "expiration policy" to "cleanup policy" in GitLab 13.2.
Enable automatically for project created after GitLab 12.8

Official docopen in new window

Field	Description
`Cleanup policy`	Turn the policy on or off.
`Expiration interval`	How long tags are exempt from being deleted.
`Expiration schedule`	How often the policy should run.
`Number of tags to retain`	How many tags to always keep for each image.
`Tags with names matching this regex pattern expire:`	The regex pattern that determines which tags to remove. For all tags, use .*. See other regex pattern examples.
`Tags with names matching this regex pattern are preserved:`	The regex pattern that determines which tags to preserve. The latest tag is always preserved. For all tags, use .*. See other regex pattern examples.

You can not change the default cleanup policy. The only thing you can change is after having enabled it to a project.

Here is the default :

# Upgrade

Snapshot your server (for vmware usage or similarly)
Make a system backup (Optional)
Warning : Can fully consumed your file system
saved here /var/opt/gitlab/backups

gitlab-rake gitlab:backup:create STRATEGY=copy

Update GitLab + all system packages

apt-get update
apt-get upgrade

1
2

Reboot if new linux version

systemctl reboot

If misbehaving after update (which happens often) you can force restart gitlab

gitlab-ctl restart

# Control gitlab

gitlab-ctl status

# Start all GitLab components
gitlab-ctl start

# Stop all GitLab components
gitlab-ctl stop

# Restart all GitLab components
gitlab-ctl restart



# Tail all logs; press Ctrl-C to exit
gitlab-ctl tail

# Drill down to a sub-directory of /var/log/gitlab
gitlab-ctl tail gitlab-rails

# Drill down to an individual file
gitlab-ctl tail nginx/gitlab_error.log

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

Sourcesopen in new window

# Runner

# Advanced config

# [runners.docker]

official docopen in new window
override hosts file (/etc/hosts) :

extra_hosts
/etc/gitlab-runner/config.toml
Override docker in docker default ca-certificate : Mount ca volume from host.

[[runners]]
  url = "http://git.example.org/ci"
  token = "TOKEN"
  name = "my_runner"
  executor = "docker"
  [runners.docker]
    host = "tcp://<DOCKER_DAEMON_IP>:2375"
    image = "..."
    ...
    extra_hosts = ["git.example.org:192.168.0.39"]
    volumes = ["/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro"]

1
2
3
4
5
6
7
8
9
10
11

# Object storage

# Migrate from local to remote S3

gitlab-rake gitlab:lfs:migrate
gitlab-rake gitlab:lfs:migrate_to_local

gitlab-rake gitlab:artifacts:migrate
gitlab-rake gitlab:artifacts:migrate_to_local

gitlab-rake "gitlab:uploads:migrate:all"
gitlab-rake "gitlab:uploads:migrate:all"

1
2
3
4
5
6
7
8

# Cleanup

Default : DRY_RUN=true

gitlab-rake gitlab:cleanup:orphan_lfs_file_references
gitlab-rake gitlab:cleanup:orphan_lfs_file_references DRY_RUN=false

gitlab-rake gitlab:cleanup:orphan_lfs_files
gitlab-rake gitlab:cleanup:orphan_lfs_files DRY_RUN=false

gitlab-rake gitlab:cleanup:project_uploads
gitlab-rake gitlab:cleanup:project_uploads DRY_RUN=false

gitlab-rake gitlab:cleanup:orphan_job_artifact_files
gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false

gitlab-rake gitlab:cleanup:orphan_job_artifact_files
gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Integrity check

gitlab-rake gitlab:check
gitlab-rake gitlab:git:fsck
gitlab-rake gitlab:artifacts:check
gitlab-rake gitlab:lfs:check
gitlab-rake gitlab:uploads:check
gitlab-rake gitlab:ldap:check
gitlab-rake gitlab:list_repos

1
2
3
4
5
6
7
8

# Pages

# New Pages website from a forked sample

To get started with GitLab Pages from a sample website, the easiest way to do it is by using one of the bundled templatesopen in new window. If you don’t find one that suits your needs, you can opt by forking (copying) a sample project from the most popular Static Site Generatorsopen in new window.

Fork a sample project from the GitLab Pages examples group.
From the left sidebar, navigate to your project’s CI/CD > Pipelines and click Run pipeline on master branch to trigger GitLab CI/CD to build and deploy your site to the server.
Once the pipeline has finished successfully, find the link to visit your website from your project’s Settings > Pages. It can take approximately 30 minutes to be deployed. Wait several minutes for your static files to be push and accessible to namespace.gitlab.io. At the start you'll just get 401 or 404 error.
(Bonus HTTPS) Once your site is correctly. You can request gitlab make a request to Let's encrypt on your behalf. It should take approximatively 30 minutes. You'll see a CN under your domain. and a link the access your website https. Because if you activate HTTPS just after pipeline finish, the let's encrypt will not work, and gitlab will not be validated by LE.

sourcesopen in new window

# Life Cycle

pages life cycleopen in new window

# Agile organization

GitLab for agile software develomentopen in new window

# Support version

Supported OSopen in new window

# Troubleshoot

Check the Registry logs (e.g. /var/log/gitlab/registry/current)
GitLab production logs for errors (e.g. /var/log/gitlab/gitlab-rails/production.log). You may be able to find clues there.
Default values (overrides by /etc/gitlab/gitlab.rb) (/opt/gitlab/etc/gitlab.rb.template)

# NGINX Logs

For Omnibus installations, NGINX logs reside in:

/var/log/gitlab/nginx/gitlab_access.log contains a log of requests made to GitLab.
/var/log/gitlab/nginx/gitlab_error.log contains a log of NGINX errors for GitLab.
/var/log/gitlab/nginx/gitlab_pages_access.log contains a log of requests made to Pages static sites.
/var/log/gitlab/nginx/gitlab_pages_error.log contains a log of NGINX errors for Pages static sites.
/var/log/gitlab/nginx/gitlab_registry_access.log contains a log of requests made to the Container Registry.
/var/log/gitlab/nginx/gitlab_registry_error.log contains a log of NGINX errors for the Container Registry.
/var/log/gitlab/nginx/gitlab_mattermost_access.log contains a log of requests made to Mattermost.
/var/log/gitlab/nginx/gitlab_mattermost_error.log contains a log of NGINX errors for Mattermost.

sourceopen in new window

# Pages logs

For Omnibus installations, Pages logs reside in /var/log/gitlab/gitlab-pages/current

# GitLab Omnibus official

Code : https://gitlab.com/gitlab-org/omnibus-gitlab
Changelog : https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/CHANGELOG.md
Architecture : https://docs.gitlab.com/ee/development/architecture.html#simplified-component-overview