What to do if some files have been processed multiply within a production ?¶
Here the example is for stripping <STRIPPING> and merging <MERGING>
0. Quickly stop the merging productions as well as the stripping productions!¶
Good to also stop the removal productions if any
1. Check whether their output was merged¶
grep ProcMultDesc CheckDescendantsResults_<STRIPPING>.txt | dirac-bookkeeping-get-file-descendants —Prod <STRIPPING> —All | dirac-transformation-debug <MERGING>
Look at whether the multiple descendants have been merged, in which case they are set Processed in the final list. You can also see all files that are eventually already part of merging jobs (status Assigned).
If the number of RDST files is small and only 1 or 2 streams have been merged, one may afford just removing the merged files (i.e. the output of the merging jobs that ran): select the Processed files above and do:
dirac-bookkeeping-get-file-descendants —Term
<paste list of Processed files>
Else, one needs to remove the whole runs that are affected :(
2. Get the list of runs¶
grep ProcMultDesc CheckDescendantsResults_<STRIPPING>.txt | dirac-bookkeeping-file-path —Summary —GroupBy RunNumber —List
This prints out how many files per run and at the end the list of affected runs… Now starts the fun!
3. Get the list of files to be removed¶
3.0 Check whether some files are being merged¶
dirac-transformation-debug <MERGING> —Status Assigned —Run <list of runs>
As long as you have files Assigned, better do nothing. if you are in a hurry, you may kill the corresponding jobs and proceed, but this may cause troubles…
3.1 Merged files¶
dirac-bookkeeping-get-files —Prod <MERGING> —Run <list of runs> —Output toRemove<STRIPPING>-merged.txt
3.2 Non-merged files¶
dirac-bookkeeping-get-files —Prod <STRIPPING> —Run <list of runs> —Output toRemove<STRIPPING>-nonMerged.txt
if this fails (there is a bug in the BK, but could be fixed ;-)
dirac-loop —Item <list of runs> ‘dirac-bookkeeping-get-files —Prod <STRIPPING> —Run @arg@:@arg@ —Visi No’ > toRemove<STRIPPING>-nonMerged.txt
3.3 Remove these files¶
Firstly enable the xxx-ARCHIVE SEs for removal:
dirac-dms-replica-stats —File toRemove<STRIPPING>-merged.txt
will give you the list of ARCHIVE SEs concerned…
dirac-admin-enable-se —AllowRemove <list of ARCHIVE SEs >
dirac-transformation-add-files —File toRemove<STRIPPING>-merged.txt,toRemove<STRIPPING>-nonMerged.txt Remove-all-replicas-CleanTrans
This is unfortunately not enough as you should set Removed the files that were Processed already, since the transformation above would not change status of Processed files…:
dirac-bookkeeping-get-file-ancestors —File toRemove<STRIPPING>-merged.txt | dirac-transformation-reset-files —NewStatus Removed <MERGING>
4. Get the list of RDST files to reprocess and reset them¶
4.0 Check whether some files are being processed¶
dirac-transformation-debug <STRIPPING> —Status Assigned —Run <list of runs>
As long as you have files Assigned, better do nothing If you are in a hurry, you may kill the corresponding jobs and proceed, to your own risk!
4.1 Get list of RDST files¶
dirac-bookkeeping-get-files —BK <BK path> —Run <list of runs> —Output rdstToReprocess-<STRIPPING>.txt
4.2 Re-stage these files and their ancestors¶
dirac-transformation-add-files —File rdstToReprocess-<STRIPPING>.txt PreStage-with-ancestors
or if this transformation doesn’t exist:
dirac-dms-add-transformation —Plugin ReplicateWithAncestors —File rdstToReprocess-<STRIPPING>.txt —Name PreStage-with-ancestors —Start
4.3 Reset RDST files Unused in the stripping production¶
Get the status of those files in the stripping production, in case they are already set Problematic or NotProcessed:
dirac-transformation-debug —File rdstToReprocess-<STRIPPING>.txt —Status Problematic,NotProcessed <STRIPPING>
Save the list of files that are in either of these statuses in order to reset them later
When you are sure no file is Assigned:
dirac-transformation-debug —File rdstToReprocess-<STRIPPING>.txt —Status Assigned <STRIPPING>
returns no files!
dirac-transformation-reset-files —File rdstToReprocess-<STRIPPING>.txt <STRIPPING>
and then reset Problematic files you have saved as such
dirac-transformation-reset-files —NewStatus Problematic —Term <STRIPPING>
<paste the list of Problematic >
dirac-transformation-reset-files —NewStatus NotProcessed —Term <STRIPPING>
<paste the list of NotProcessed>
5. Make a few more checks¶
From time to time better do a further check that all is OK as you may have also other errors like files in FC without BK flag, in which case you should removed them:
dirac-production-check-descendants —File rdstToReprocess-<STRIPPING>.txt <STRIPPING>
6. Restart productions¶
Restart the stripping production first, do a few more checks as in 5. and when confident, restart the merging production
7. Not to be forgotten¶
At the end of the productions, one should remove from Tier1-Buffer the RDST files that have been re-processed as well as their RAW ancestors. This is an additional duty for the DM cleaning that anyway must take place. If all is OK it is enough to just remove all files and ancestors from Buffer…