The following procedure applies fully to LHCbDIRAC production releases, like patches. For pre-releases (AKA certification releases, there are some minor changes to consider).
The release manager needs to:
- be aware of the LHCbDIRAC repository structure and branching as highlighted in the contribution guide.
- have push access to the master branch of “upstream” (being part of the project “owners”)
- have DIRAC installed
- have been granted write access to <webService>
- have “lhcb_admin” or “diracAdmin” role.
- have a Proxy
The release manager of LHCbDIRAC has the triple role of:
- creating the release
- making basic verifications
- deploying it in production
1. Creating the release¶
Unless otherwise specified, (patch) releases of LHCbDIRAC are usually done “on top” of the latest production release of DIRAC. The following of this guide assumes the above is true.
Releases of LHCbDIRAC are stored on the Python Package Index (PyPI) and created in 3 steps:
- Merging “Merge Requests”
- Propagating to the devel branch (for patches)
- Creating a tag in Git and uploading it to the . This is typically handled by a manually triggered CI job.
Merging “Merge Requests”¶
Merge Requests (MR) that are targeted to the master branch and that have been approved by a reviewer are ready to be merged
Otherwise, simply click the “Accept merge request” button for each of them.
If you are making a Major release please merge devel to master follow the instruction: 1. Making a major releases.
Propagate to the devel branch¶
The LHCbDIRAC repo has a webhook installed to trigger a job in GitLab(lhcb-dirac/sweeper) for each MR action taken on the repo.
The pipeline in this repo converts all the information from GitLab hook to environment variables that are accessible for usage.
For each MR that it is determined that it targets
master we call the script
labelMR.py (in lhcb-dirac/sweeper) to add the label
This is a hint for the subsequent procedure to cherry-pick this MR into
The automatic cherry-picking (also referred to as sweeping), is performed in the CI job
MR_SWEEP, which tries to cheery-pick the merged MRs into a new branch and open a new MR against
This mechanism is only triggered on MRs that have the label
alsoTargeting:devel. Once the mechanism finished the merged MR will receive the label
In the case an automatic sweep is not possible (e.g. merge conflict) the MR will receive the label
sweep:failed, in the comment of the MR you will see a link to the job where the sweep was attempted.
In the log this job you will see a hint how to reproduce the failed merge, so the you can manually resolve the problem e.g.:
git checkout devel git cherry-pick -m 1 9f142d8c1 git status
To not overlook failed sweeps, it is advisable that you subscribe to the label sweep:failed in GitLab Labels.
If everything is successful, after merging something to
master in a relatively short period of time a MR should appear that is the result of the sweep, it will have the label
Before you start doing any merging it’s good to setup the correct merge driver, you do this by adding to your .gitconfig
- [merge “ours”]
- driver = true
this lets git know which files you want to ignore when you merge master into devel.
Now, you need to make sure that what’s merged in master is propagated to the devel branch. From the local fork:
# get the updates (this never hurts!) git fetch upstream # create a "newDevel" branch which from the upstream/devel branch git checkout -b newDevel upstream/devel # merge in newDevel the content of upstream/master git merge upstream/master
The last operation may result in potential conflicts. If happens, you’ll need to manually update the conflicting files (see e.g. this guide). As a general rule, prefer the master fixes to the “HEAD” (devel) fixes. Remember to add and commit once fixed.
Please fix the conflict if some files are conflicting. Do not forget to to execute the following:
git add -A && git commit -m " message"
Conflicts or not, you’ll need to push back to upstream:
# push "newDevel" to upstream/devel git push upstream newDevel:devel # delete your local newDevel git checkout upstream/devel git branch -d newDevel # keep your repo up-to-date git fetch upstream
To create a release you go to the pipelines https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/pipelines and go the last pipeline of the branch you want to tag. At the end of the pipeline there is a manual trigger job with name make_tag, you click on it and you will get the following windows
If you don’t specify any version (with VERSION) only the LHCbDIRAC patch/alpha/beta/rc version will be increased for +1. This will create the release notes and tag.
When a new git tag is pushed to the repository, a gitlab-ci job takes care of testing, creating the sdist/bdist, uploading it to PyPI, and to build the docker image. You can check it in the pipeline page of the repository (https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/pipelines).
It may happen that the pipeline fails. Often it is just a glitch on the runner side, if so just restart the job from the pipeline web interface. If any of the pipelines fails, don’t try to do a release manually but rather investigate why.
2. Make an LHCbWebDIRAC release (if required)¶
LHCbWebDIRAC releases are made by simply checking the last CI pipeline was successful and then creating a tag using the GitLab web interface. GitLab CI should then test and upload this tag to PyPI.
3. Advertise the new release¶
Before you start the release you must write an Elog entry 1 hour before you start the deployment. You have to select Production and Release tick boxes.
When the intervention is over you must notify the users (reply to the Elog message).
4. Deploying the release¶
Deploying a release means deploying it for the various installations:
* client * server * pilot
Release for client¶
Releases are automatically installed on to cvmfs with the
deploy_on_cvmfs GitLab CI job.
In addition, there is a manual GitLab CI job called
set_cvmfs_prod_link, this sets the production version link to the current deployed version.
If a roll-back is required to a previous version, this job can be re-tried from an older release pipeline to re-trigger the job.
A overview of the current installation health can be found at here.
Note: It is normal for these jobs to show the
Method 1 (preferred): inside the tag pipeline¶
The CI pipeline associate the tag pipeline has a manual job
update_instance which you have to trigger. This will automatically apply the release to all machines that that constitute the respective instance.
In the case of normal tags this is the production instance and in the case of alpha/beta/release candidate tag this is the certification instance.
The update is based on the dirac command
dirac-admin-update-instance. The same job will also update the pilot version via
Method 2: web portal¶
- Using the web portal:
- You cannot do all the machines at once. Select a bunch of them (between 5 and 10). Fill in the version number and click update.
- Repeat until you have them all.
- Start again selecting them by block, but this time, click on “restart” to restart the components.
Method 3: interactive via sysadmin cli¶
To install it on the VOBOXes from lxplus:
lhcb-proxy-init -g lhcb_admin dirac-admin-sysadmin-cli --host lbvoboxXYZ.cern.ch > update LHCbDIRAC v10.4.2 > restart *
Update the pilot version from the CS, keeping 2 pilot versions, for example:
/Operation/LHCb-Production/Pilot/Version = v10.4.2, v10.4.1
The newer version should be the first in the list
for checking and updating the pilot version. Note that you’ll need a proxy that can write in the CS (i.e. lhcb-admin). This script will make sure that the pilot version is update BOTH in the CS and in the json file used by pilots started in the vacuum.
1. Making a major releases¶
Making a major release requires manual intervention to update the package dependencies by editing the setup.cfg files of each package.
- We typically constrain LHCb(Web)DIRAC to a specific minor release series
- LHCbDIRAC has the dependency on DIRAC
- LHCbWebDIRAC has the dependency on LHCbDIRAC
- To update LHCbWebDIRAC this there will need to be a compatible (pre)release of LHCbDIRAC available first
The procedure is the same for LHCbWebDIRAC and LHCbDIRAC: you basically want
devel to become the new
master. In order to do so:
# Start from the master branch git checkout upstream/master # Create a commit that has two parents (master and devel) to preserve the history # but only take the files of the parents (i.e. no changes to the files) # and do not create the commit just yet # (https://www.atlassian.com/git/tutorials/using-branches/merge-strategy) git merge -s ours --no-commit upstream/devel # Remove all the files present git rm -r . # Checkout all the files that are in the devel branch (notice the dot at the end) git checkout upstream/devel . # finish the merging git merge --continue # push back git push upstream/master
You can then trigger the release pipeline normally, specifying a version such as
Note about the git procedure¶
Ultimately, the aim is that
devel becomes our new
However, bluntly doing this would resoled in conflicts for everybody that has a clone of the LHCbDIRAC repo.
Thus, this idea of doing a merge with the
-s ours strategy. This allows to create a commit that has as parent both
devel, and avoid any forced push.
The rest of the procedure (
rm/checkout) is to make sure that we do not keep files that were meant to be in