Components¶
The popularity analysis relies on a lot of components
StorageUsageDB¶
Despite its name, that is where both the StorageUsageAgent and the PopularityAgent stores their data. It is exposed via the StorageUsageHandler and the DataUsageHandler
StorageUsageAgent¶
This agent scans the DFC and stores the size and number of files per directory and per StorageElement in the StorageUsageDB.
StorageHistoryAgent¶
- This agent crawls the StorageUsageDB, convert each directory into a bookkeeping path and fill in the following accounting:
- Storage: space used/free per storage and/or directory
- Data storage: spaced used per bookkeeping path
- user storage: like Storage, but for user directories
DataUsageHandler¶
This service is called by the jobs to declare their use of a given directory. It is stored per directory and per day.
PopularityAgent¶
This agent goes through the StorageUsageDB and creates accounting entries for the popularity. It also caches the BK dictionary for each directory in the StorageUSageDB.
DataPop server¶
Yandex provided service that consumes our popularity CSV and make prediction on which dataset to remove. It is ran on our mesos cluster: https://lbmesosms02.cern.ch/marathon/ui/#/apps/%2Fdatapopserv
PopularityAnalysisAgent¶
- This agents creates two files:
- one CSV containing a summary of the popularity (see popularity.csv file ).
- one CSV, generated from the first one through the DataPop server