============== LHCbDIRAC Logs ============== Job logs aka LogSE ================== The LogSE is used to store the log files from the production jobs. it is defined in `/Operations/defaults/LogStorage/LogSE = LogSE-EOS`. The content is exposed via the normal EOS protocols, but also via a `CERN web service `_ . Any member of the lhcb-geoc group can `manage this web service `_. The reason for having this web service is to be able to use htaccess and php magic. The aim of the php and htaccess magic is to transparently move from a log directory per job to one zip archive per job, and still expose the content the same way, allowing for easy browsing. The `.htaccess`, `listzip.php` and `extract.php` are stored at the root of the logSE. The php scripts are shared with the Core soft team and available in `this repo `_. The `.htaccess` content is pasted bellow:: # Option mandatory for CERN website exploration to work Options +Indexes # Allows rewrite ruls RewriteEngine On # this is a clever trick to avoid RewriteBase # see http://stackoverflow.com/a/21063276 # basically, {ENV:BASE} eval to the location of the htaccess RewriteCond "%{REQUEST_URI}::$1" "^(.*?/)(.*)::\2$" RewriteRule "^(.*)$" "-" [E=BASE:%1] # These rules expect path that follows the convention # /LOG/// # In practice, what we have is # MC/2018/LOG/// # What we want to be able to do is to expose the exact same way the log files of a job: # * stored in a directory # * stored in a zip file , containing itself a directory # Aim: list the zip file as if it was a directory # If the URL targets is a non existing directory RewriteCond "%{DOCUMENT_ROOT}/%{REQUEST_URI}" !-d # if the url is if the form "/LOG//// (Note the "/: at the end) # we redirect to "{ENV:BASE}/listzip.php?zip=/LOG///.zip RewriteRule "^(.*/LOG/[0-9]+/[0-9]+/([0-9]+))/$" "%{ENV:BASE}listzip.php?zip=$1.zip" [PT,B,L] # Aim: extract artifacts from specific zip files # If we have a URL targetting a non existing file RewriteCond "%{DOCUMENT_ROOT}/%{REQUEST_URI}" !-f # if the url is if the form "/LOG//// # we redirect to "{ENV:BASE}/extract.php?zip=/LOG///.zip&path=/ RewriteRule "^(.*/LOG/[0-9]+/[0-9]+/([0-9]+))/(.+)" "%{ENV:BASE}extract.php?zip=$1.zip&path=$2/$3" [PT,B,L] Centralized logging =================== ----- TL;DR ----- All the logs (up to the VERBOSE level) from the Agents and services are visible on `this kibana instance `_. ------------ More details ------------ Each and every component send their logs at the VERBOSE level in a message queue. This is configured using the `message queue backend `_ , and the queue is described in the `MQServices resources `_ The logs are then consumed by a logstash server (`lbdiraclogstash{01,02}.cern.ch`), and forwarded to ElasticSearch. This is configured in the `ai-puppet-hostgroup-volhcb repository `_. ------------ Data storage ------------ We are using Elasticsearch (ES) to store the data, which is provided by IT `CERN centralized ES service `_. The ES configuration can be found in `it-elasticsearch-project repositiry `_. ------------------ Data Visualization ------------------ Kibana is used to visualize the data, which is accessible `in this link `_. To access the Kibana you have to be a member of lhcb-geoc egroup. There are predefined dash boards which you can access under Dash boards menu.