================== LHCbDIRAC Releases ================== The following procedure applies fully to LHCbDIRAC production releases, like patches. For pre-releases (AKA certification releases, there are some minor changes to consider). Prerequisites ============= The release manager needs to: - be aware of the LHCbDIRAC repository structure and branching as highlighted in the `contribution guide `_. - have forked LHCbDIRAC on GitLab as a "personal project" (called "origin" from now on) - have cloned origin locally - have added ``_ as "upstream" repository to the local clone - have push access to the master branch of "upstream" (being part of the project "owners") - have DIRAC installed - have been grated write access to - have "lhcb_admin" or "diracAdmin" role. - have a Proxy The release manager of LHCbDIRAC has the triple role of: 1. creating the release 2. making basic verifications 3. deploying it in production 1. Creating the release ======================= Unless otherwise specified, (patch) releases of LHCbDIRAC are usually done "on top" of the latest production release of DIRAC. The following of this guide assumes the above is true. Creating a release of LHCbDIRAC means creating a tarball that contains the release code. This is done in 3 steps: 1. Merging "Merge Requests" 2. Propagating to the devel branch 3. Creating the release tarball, add uploading it to the LHCb web service But before: Pre ``` Verify what is the last tag of DIRAC:: # it should be in this list: git describe --tags $(git rev-list --tags --max-count=10) A tarball containing it is should be already uploaded `here `_ You may also look inside the .cfg file for the DIRAC release you're looking for: it will contain an "Externals" version number, that should also be a tarball uploaded in the same location as above. If all the above is ok, we can start creating the LHCbDIRAC release. Merging "Merge Requests" ```````````````````````` `Merge Requests (MR) `_ that are targeted to the master branch and that have been approved by a reviewer are ready to be merged If there are no MRs, or none ready: please skip to the "update the CHANGELOG" subsection. Otherwise, simply click the "Accept merge request" button for each of them. Then, from the LHCbDIRAC local fork you need to update some files:: # if you start from scratch otherwise skip the first 2 commands mkdir $(date +20%y%m%d) && cd $(date +20%y%m%d) git clone https://:@gitlab.cern.ch:8443/lhcb-dirac/LHCbDIRAC.git git remote add upstream https://:@gitlab.cern.ch:8443/lhcb-dirac/LHCbDIRAC.git # update your "local" upstream/master branch git fetch upstream # create a "newMaster" branch which from the upstream/master branch git checkout -b newMaster upstream/master # determine the tag you're going to create by checking what was the last one from the following list (add 1 to the "p"): git describe --tags $(git rev-list --tags --max-count=5) # Update the version in the __init__ file: vim LHCbDIRAC/__init__.py # Update the version in the releases.cfg file: vim LHCbDIRAC/releases.cfg # For updating the CHANGELOG, get what's changed since the last tag t=$(git describe --abbrev=0 --tags); git --no-pager log ${t}..HEAD --no-merges --pretty=format:'* %s'; # copy the output, add it to the CHANGELOG (please also add the DIRAC version) vim CHANGELOG # please, remove comments like "fix" or "pylint" or "typo"... #If needed, change the versions of the packages vim dist-tools/projectConfig.json git add -A && git commit -av -m "" Time to tag and push:: # make the tag git tag -a -m # push "newMaster" to upstream/master git push --tags upstream newMaster:master # delete your local newMaster # before change your branch use git checkout "existing branch name" git branch -d newMaster Remember: you can use "git status" at any point in time to make sure what's the current status. Propagate to the devel branch ````````````````````````````` Now, you need to make sure that what's merged in master is propagated to the devel branch. From the local fork:: # get the updates (this never hurts!) git fetch upstream # create a "newDevel" branch which from the upstream/devel branch git checkout -b newDevel upstream/devel # merge in newDevel the content of upstream/master git merge upstream/master The last operation may result in potential conflicts. If happens, you'll need to manually update the conflicting files (see e.g. this `guide `_). As a general rule, prefer the master fixes to the "HEAD" (devel) fixes. Remember to add and commit once fixed. Note: For porting the LHCbDIRAC.init.py from master to devel, we prefer the HEAD version (only for this file!!!) Plase fix the conflict if some files are conflicting. Do not forget to to execute the following:: git add -A && git commit -m " message" Conflicts or not, you'll need to push back to upstream:: # push "newDevel" to upstream/devel git push upstream newDevel:devel # delete your local newDevel git branch -d newDevel # keep your repo up-to-date git fetch upstream Creating the release tarball, add uploading it to the LHCb web service `````````````````````````````````````````````````````````````````````` Login on lxplus, run :: lb-run LHCbDirac/prod bash -norc git archive --remote ssh://git@gitlab.cern.ch:7999/lhcb-dirac/LHCbDIRAC.git devel LHCbDIRAC/releases.cfg | tar -x -v -f - --transform 's|^LHCbDIRAC/||' LHCbDIRAC/releases.cfg dirac-distribution -r v8r3p1 -l LHCb -C file:///`pwd`/releases.cfg (this may take some time) Don't forget to read the last line of the previous command to copy the generated files at the right place. The format is something like:: ( cd /tmp/joel/tmpxg8UuvDiracDist ; tar -cf - *.tar.gz *.md5 *.cfg ) | ssh lhcbprod@lxplus.cern.ch 'cd /afs/cern.ch/lhcb/distribution/DIRAC3/tars && tar -xvf - && ls *.tar.gz > tars.list' And just copy/paste/execute it. If you do not have access to lhcbprod, you can use your user name. 2. Making basic verifications ============================= Once the tarball is done and uploaded, the release manager is asked to make basic verifications, via Jenkins, if the release has been correctly created. At this `link `_ you'll find some Jenkins Jobs ready to be started. Please start the following Jenkins jobs and come back in about an hour to see the results for all of them. 1. https://lhcb-jenkins.cern.ch/jenkins/view/LHCbDIRAC/job/!RELEASE!__pylint_unit/ the !RELEASE! is the actual relase for example: https://lhcb-jenkins.cern.ch/jenkins/view/LHCbDIRAC/job/v8r5__pylint_unit/ This job will: run pylint (errors only), run all the unit tests found in the system, assess the coverage. The job should be considered successful if: - the pylint error report didn't increase from the previous job run - the test results didn't get worse from the previous job run - the coverage didn't drop from the previous job run 2. https://lhcb-jenkins.cern.ch/jenkins/view/LHCbDIRAC/job/!RELEASE!__pilot/ This job will simply install the pilot. Please just check if the result does not show in an "unstable" status 3. https://lhcb-jenkins.cern.ch/jenkins/view/LHCbDIRAC/job/!RELEASE!__/ TODO 3. Advertise the new release ============================ Before you start the release you must write an Elog entry 1 hour before you start the deployment. You have to select Production and Release tick boxes. When the intervention is over you must notify the users (reply to the Elog message). 4. Deploying the release ======================== Deploying a release means deploying it for the various installations:: * client * server * pilot release for client `````````````````` Please refer to this `TWIKI page `_ a quick test to validate the installation is to run the SHELL script $LHCBRELEASE/LHCBDIRAC/LHCBDIRAC_vXrY/LHCbDiracSys/test/client_test.csh go to this `web page `_ for asking to install the client release in AFS and CVMFS: * in the field "Project list" put : "Dirac vNrMpK LHCbDirac vArBpC" (NOTE: If LHCbGRID is already released, please use only DIRAC and LHCbDIRAC: DIRAC vNrMpK LHCbDirac vArBpC) * in the field "platforms" put : "x86_64-slc6-gcc48-opt x86_64-slc6-gcc49-opt" Then click on the "BUILD" button * within 10-15 min the build should start to appear in the nightlies page https://lhcb-nightlies.cern.ch/release/ * if there is a problem in the build, it can be re-started via the dedicated button (it will not restart by itself after a retag) If it is the production release, and only in this case, once satisfied by the build, take note of the build id (you can use the direct link icon) and make the request via https://sft.its.cern.ch/jira/browse/LHCBDEP. * NOTE: If some package is already released, please do not indicate in the Jira task. For example: a Jira task when: * DIRAC is not released, then the message in the JIRA task: Summary:Dirac v6r14p37 and LHCbDirac v8r2p50; Description: Please release Dirac and LHCbDirac in this order based on build 1526 * DIRAC is released, then the message in the JIRA task: Summary:LHCbDirac v8r2p50; Description: Please release LHCbDirac based on build 1526 Server `````` To install it on the VOBOXes from lxplus:: lhcb-proxy-init -g diracAdmin dirac-admin-sysadmin-cli --host volhcbXX.cern.ch >update LHCbDIRAC-v8r3p32 >restart * The (better) alternative is using the web portal or using the following script: LHCbDIRAC/LHCbDiracPolicy/scripts/create_vobox_update. The recommended way is the following:: ssh lxplus mkdir DiracInstall; cd DiracInstall cp LHCbDIRAC/LHCbDiracPolicy/scripts/create_vobox_update . cp LHCbDIRAC/LHCbDiracPolicy/scripts/skel_vobox_update . python create_vobox_update v8r2p30 This command will create 6 files called "vobox_update_MyLetter" then you can run in 6 windows the recipe for one single machine like that:: ssh lxplus cd DiracInstall ; lb-run LHCbDIRAC/prod bash -norc ; lhcb-proxy-init -g lhcb_admin; dirac-admin-sysadmin-cli and from the prompt :: [host] : execfile vobox_update_MyLetter [host] : quit Note:: It is normal if you see the following errors: --> Executing restart Framework SystemAdministrator [ERROR] Exception while reading from peer: (-1, 'Unexpected EOF') In case of failure you have to update the machine by hand. Example of a typical failure:: --> Executing update v8r2p42 Software update can take a while, please wait ... [ERROR] Failed to update the software Timeout (240 seconds) for '['dirac-install', '-r', 'v8r2p42', '-t', 'server', '-e', 'LHCb', '-e', 'LHCb', '/opt/dirac/etc/dirac.cfg']' call Login to the failing machine, become dirac, execute manually the update, and restart everything. For example:: ssh lbvobox11 sudo su - dirac dirac-install -r v8r2p42 -t server -e LHCb -e LHCb /opt/dirac/etc/dirac.cfg lhcb-restart-agent-service runsvctrl t startup/Framework_SystemAdministrator/ Specify that this error can be ignored (but should be fixed ! ):: 2016-05-17 12:00:00 UTC dirac-install [ERROR] Requirements installation script /opt/dirac/versions/v8r2p42_1463486162/scripts/dirac-externals-requirements failed. Check /opt/dirac/versions/v8r2p42_1463486162/scripts/dirac-externals-requirements.err WebPortal ````````` When the web portal machine is updated then you have to compile the WebApp:: ssh lhcb-portal-dirac.cern.ch sudo su - dirac dirac-install -r VERSIONTOBEINSTALLED -t server -l LHCb -e LHCb,LHCbWeb,WebAppDIRAC /opt/dirac/etc/dirac.cfg (for example: dirac-install -r v8r4p2 -t server -l LHCb -e LHCb,LHCbWeb,WebAppDIRAC /opt/dirac/etc/dirac.cfg) dirac-webapp-compile When the compilation is finished:: lhcb-restart-agent-service runsvctrl t startup/Framework_SystemAdministrator/ ```` TODO ```` When the machines are updated, then you have to go through all the components and check the errors. There are two possibilities: 1. Use the Web portal (SystemAdministrator) 2. Command line:: for h in $(grep 'set host' vobox_update_* | awk {'print $NF'}); do echo "show errors" | dirac-admin-sysadmin-cli -H $h; done | less Pilot ````` Use the following script (from, e.g., lxplus after having run `lb-run LHCbDIRAC tcsh`):: dirac-pilot-version -S v8r2p42 for checking and updating the pilot version. Note that you'll need a proxy that can write in the CS (i.e. lhcb-admin). This script will make sure that the pilot version is update BOTH in the CS and in the json file used by pilots started in the vacuum. Basic instruction how to merging the devel branch into master (NOT for PATCH release) ``````````````````````````````````````````````````````````````````````````````````````` Our developer model is to keep only two branches: master and devel. When we made a major release we have to merge devel to master. Before the merging please create a new branch based on master using the web interface of GitLab. This is for safety. After you can merege devel to master:: mkdir $(date +20%y%m%d) && cd $(date +20%y%m%d) git clone ssh://git@gitlab.cern.ch:7999/lhcb-dirac/LHCbDIRAC.git cd LHCbDIRAC git remote rename origin upstream git fetch upstream git checkout -b newMaster upstream/master git merge upstream/devel git push upstream newMaster:master 5. Mesos cluster ======================== Mesos is currently only used for the certification. In order to push a new version on the Mesos cluster, 3 steps are needed: - Build the new image - Push it the lhcbdirac gitlab repository - Update the version of the running containers All these functionalities have been wrapped up in a script (dirac-docker-mgmt), available on all the lbmesosadm* machines (01, 02) The next steps are the following:: # build the new image # this will download the necessary files, and build # the image localy dirac-docker-mgmt.py -v v8r5 --build # Push it to the remote lhcbdirac registry # Your credentials for gitlab will be asked dirac-docker-mgmt.py -v v8r5 --release # Update the version of the running containers # The services and number of instances running # will be preserved dirac-docker-mgmt.py -v v8r5 --deploy