LHCbDIRAC Releases

The following procedure applies fully to LHCbDIRAC production releases, like patches. For pre-releases (AKA certification releases, there are some minor changes to consider).

Prerequisites

The release manager needs to:

  • be aware of the LHCbDIRAC repository structure and branching as highlighted in the contribution guide.
  • have push access to the master branch of “upstream” (being part of the project “owners”)
  • have DIRAC installed
  • have been granted write access to <webService>
  • have “lhcb_admin” or “diracAdmin” role.
  • have a Proxy

The release manager of LHCbDIRAC has the triple role of:

  1. creating the release
  2. making basic verifications
  3. deploying it in production

1. Creating the release

Unless otherwise specified, (patch) releases of LHCbDIRAC are usually done “on top” of the latest production release of DIRAC. The following of this guide assumes the above is true.

Releases of LHCbDIRAC are stored on the Python Package Index (PyPI) and created in 3 steps:

  1. Merging “Merge Requests”
  2. Propagating to the devel branch (for patches)
  3. Creating a tag in Git and uploading it to the . This is typically handled by a manually triggered CI job.

Merging “Merge Requests”

Merge Requests (MR) that are targeted to the master branch and that have been approved by a reviewer are ready to be merged

Otherwise, simply click the “Accept merge request” button for each of them.

If you are making a Major release please merge devel to master follow the instruction: 1. Making a major releases.

Propagate to the devel branch

Automatic propagation

The LHCbDIRAC repo has a webhook installed to trigger a job in GitLab(lhcb-dirac/sweeper) for each MR action taken on the repo. The pipeline in this repo converts all the information from GitLab hook to environment variables that are accessible for usage. For each MR that it is determined that it targets master we call the script labelMR.py (in lhcb-dirac/sweeper) to add the label alsoTargeting:devel. This is a hint for the subsequent procedure to cherry-pick this MR into devel.

The automatic cherry-picking (also referred to as sweeping), is performed in the CI job MR_SWEEP, which tries to cheery-pick the merged MRs into a new branch and open a new MR against devel. This mechanism is only triggered on MRs that have the label alsoTargeting:devel. Once the mechanism finished the merged MR will receive the label sweep:done. In the case an automatic sweep is not possible (e.g. merge conflict) the MR will receive the label sweep:failed, in the comment of the MR you will see a link to the job where the sweep was attempted. In the log this job you will see a hint how to reproduce the failed merge, so the you can manually resolve the problem e.g.:

git checkout devel
git cherry-pick -m 1 9f142d8c1
git status

To not overlook failed sweeps, it is advisable that you subscribe to the label sweep:failed in GitLab Labels. If everything is successful, after merging something to master in a relatively short period of time a MR should appear that is the result of the sweep, it will have the label sweep:from master.

Manual propagation

Before you start doing any merging it’s good to setup the correct merge driver, you do this by adding to your .gitconfig

[merge “ours”]
driver = true

this lets git know which files you want to ignore when you merge master into devel.

Now, you need to make sure that what’s merged in master is propagated to the devel branch. From the local fork:

# get the updates (this never hurts!)
git fetch upstream
# create a "newDevel" branch which from the upstream/devel branch
git checkout -b newDevel upstream/devel
# merge in newDevel the content of upstream/master
git merge upstream/master

The last operation may result in potential conflicts. If happens, you’ll need to manually update the conflicting files (see e.g. this guide). As a general rule, prefer the master fixes to the “HEAD” (devel) fixes. Remember to add and commit once fixed.

Please fix the conflict if some files are conflicting. Do not forget to to execute the following:

git add -A && git commit -m " message"

Conflicts or not, you’ll need to push back to upstream:

# push "newDevel" to upstream/devel
git push upstream newDevel:devel
# delete your local newDevel
git checkout upstream/devel
git branch -d newDevel
# keep your repo up-to-date
git fetch upstream

Create/Trigger release

To create a release you go to the pipelines https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/pipelines and go the last pipeline of the branch you want to tag. At the end of the pipeline there is a manual trigger job with name make_tag, you click on it and you will get the following windows

trigger jobs

If you don’t specify any version (with VERSION) only the LHCbDIRAC patch/alpha/beta/rc version will be increased for +1. This will create the release notes and tag.

Automatic procedure

When a new git tag is pushed to the repository, a gitlab-ci job takes care of testing, creating the sdist/bdist, uploading it to PyPI, and to build the docker image. You can check it in the pipeline page of the repository (https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/pipelines).

It may happen that the pipeline fails. Often it is just a glitch on the runner side, if so just restart the job from the pipeline web interface. If any of the pipelines fails, don’t try to do a release manually but rather investigate why.

2. Make an LHCbWebDIRAC release (if required)

LHCbWebDIRAC releases are made by simply checking the last CI pipeline was successful and then creating a tag using the GitLab web interface. GitLab CI should then test and upload this tag to PyPI.

4. Deploying the release

Deploying a release means deploying it for the various installations:

* client
* server
* pilot

Release for client

Releases are automatically installed on to cvmfs with the deploy_on_cvmfs GitLab CI job. In addition, there is a manual GitLab CI job called set_cvmfs_prod_link, this sets the production version link to the current deployed version. If a roll-back is required to a previous version, this job can be re-tried from an older release pipeline to re-trigger the job. A overview of the current installation health can be found at here.

Note: It is normal for these jobs to show the RETRY status.

Server

Method 1 (preferred): inside the tag pipeline

The CI pipeline associate the tag pipeline has a manual job update_instance which you have to trigger. This will automatically apply the release to all machines that that constitute the respective instance. In the case of normal tags this is the production instance and in the case of alpha/beta/release candidate tag this is the certification instance. The update is based on the dirac command dirac-admin-update-instance. The same job will also update the pilot version via dirac-admin-update-pilot.

Method 2: web portal

Using the web portal:
  • You cannot do all the machines at once. Select a bunch of them (between 5 and 10). Fill in the version number and click update.
  • Repeat until you have them all.
  • Start again selecting them by block, but this time, click on “restart” to restart the components.

Method 3: interactive via sysadmin cli

To install it on the VOBOXes from lxplus:

lhcb-proxy-init -g lhcb_admin
dirac-admin-sysadmin-cli --host lbvoboxXYZ.cern.ch
> update LHCbDIRAC v10.4.2
> restart *

Pilot

Update the pilot version from the CS, keeping 2 pilot versions, for example:

/Operation/LHCb-Production/Pilot/Version = v10.4.2, v10.4.1

The newer version should be the first in the list

for checking and updating the pilot version. Note that you’ll need a proxy that can write in the CS (i.e. lhcb-admin). This script will make sure that the pilot version is update BOTH in the CS and in the json file used by pilots started in the vacuum.

1. Making a major releases

Making a major release requires manual intervention to update the package dependencies by editing the setup.cfg files of each package.

  • We typically constrain LHCb(Web)DIRAC to a specific minor release series
  • LHCbDIRAC has the dependency on DIRAC
  • LHCbWebDIRAC has the dependency on LHCbDIRAC
  • To update LHCbWebDIRAC this there will need to be a compatible (pre)release of LHCbDIRAC available first

The procedure is the same for LHCbWebDIRAC and LHCbDIRAC: you basically want devel to become the new master. In order to do so:

# Start from the master branch
git checkout upstream/master

# Create a commit that has two parents (master and devel) to preserve the history
# but only take the files of the parents (i.e. no changes to the files)
# and do not create the commit just yet
# (https://www.atlassian.com/git/tutorials/using-branches/merge-strategy)
git merge -s ours --no-commit upstream/devel

# Remove all the files present
git rm -r .

# Checkout all the files that are in the devel branch (notice the dot at the end)
git checkout upstream/devel .

# finish the merging
git merge --continue

# push back
git push upstream/master

You can then trigger the release pipeline normally, specifying a version such as v10.4.0.

Note about the git procedure

Ultimately, the aim is that devel becomes our new master. However, bluntly doing this would resoled in conflicts for everybody that has a clone of the LHCbDIRAC repo. Thus, this idea of doing a merge with the -s ours strategy. This allows to create a commit that has as parent both master and devel, and avoid any forced push. The rest of the procedure (rm/checkout) is to make sure that we do not keep files that were meant to be in master only.