diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 13f374bd8741b..8590178ca82eb 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -32,7 +32,6 @@ interpreter/cling/tools/packaging @vgvassilev /misc/ @dpiparo /montecarlo/ @dpiparo /net/ @gganis -/proof/ @gganis /roofit/ @lmoneta @guitargeek /rootx/ @dpiparo /sql/ @pcanal @linev diff --git a/CMakeLists.txt b/CMakeLists.txt index f2c75af416fef..3834caab36f2a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -409,9 +409,6 @@ add_subdirectory (misc) add_subdirectory (main) add_subdirectory (bindings) add_subdirectory (sql) -if(proof) - add_subdirectory(proof) -endif() if(tmva) add_subdirectory(tmva) endif() diff --git a/README/README.AUTH b/README/README.AUTH index 261df0c89a9c8..ead2831036b7c 100644 --- a/README/README.AUTH +++ b/README/README.AUTH @@ -26,12 +26,7 @@ table (column Sfx) to the specified protocol: for example TFile *f = TFile::Open("roots://host.doma.in/~fserv/TheFile.root","read") -requires the use of the SRP method, and - - gROOT->Proof("proofk://lxplus079.cern.ch") - -asks for Kerberos authentication in starting a proof session on node -lxplus079.cern.ch. +requires the use of the SRP method. Defaults can be changed on {host, user} base via the file $HOME/.rootauthrc; the header of the file $ROOTSYS/etc/system.rootauthrc, automatically generated @@ -77,52 +72,6 @@ the server sends back the list of the remaining methods accepted (if any); the client compares the server list with its own list of remaining methods and makes a new attempt if the overlap of the two lists is not empty; and so on. - -Slave/Data servers authentication during a PROOF session -======================================================== - -During a PROOF session there is the potential problem of Master/Slave or -Slave/Data_Server authentication. For slaves, the list of methods to be tried -is specified in the proof.conf file as a list of methods short names. However, -before build the corresponding entry in THostAuth (see below) TProofServ checks -that the method can be applied, i.e. that there exist valid credentials. -The way the latter are transmitted depends on the method and on the -Client/Master authentication method. - - * UsrPwd: to authenticate 'usrpwd' to slaves, the master needs the relevant - entry in the .netrc or .rootnetrc files; however, if the - client/master authentication was also 'usrpwd', the password is - already present on the master process and is used for later clear - authentications. - - * SRP: to authenticate 'SRP' to slaves, the master needs the relevant entry - in the .netrc or .rootnetrc files; the syntax is the same as for - 'usrpwd' authentication with the keyword 'secure' at the place of - 'machine'. However, if the client/master authentication was also - 'SRP', the master can receive the password from the client; the - password is sent encrypted with the internal RSA key generated for - the session. To use this option, set 'Proofd.SendSRPPwd 1' in your - .rootrc (default is 0). - - * Krb5/Globus to authenticate Krb5/Globus to slaves, the master needs globus - credentials; this is possible (and automatic) only if the - client/master authentication was also Krb5/Globus. - - * UidGid to authenticate 'uidgid' to slaves, the user must have the same - (uid,gid) on master and slaves. - -Negotiation is active also between master and slaves, so asking for 'uidgid' -first may accelerate the authentication process if the server accepts it. - -The method actually used is listed by gProof->Print("a"). - -If the slaves need to access data servers which are not part of the proof -cluster, the login info vis-a-vis of these may be specified with the proofserv -card in the .rootauthrc files (see below and etc/system.rootauthrc for -details); the collected information is then transmitted to all the active -slaves upon creation. - - Entries in .rootrc ================== @@ -243,8 +192,6 @@ classes to display/modify/create THostAuth interactively. = "s" list of active security context (default) "h" the content of the instantiated THostAuth objects in standard list - "p" the content of the instantiated THostAuth objects in - the proof list Example: @@ -308,13 +255,13 @@ Info in : + Method: 1 (SRP) Ok:0 Ko:0 Dets:pt:no ru:1 us:gani Info in : +------------------------------------------------------------------+ Info in <::Print>: +---------------------------- END ---------------------------------+ - This is the list build following the 'proofserv' directives in .rootauthrc + This is the list build following the directives in .rootauthrc * THostAuth *TAuthenticate::GetHostAuth(,,,); Returns a pointer to the THostAuth object pertaining to (host,user) if it exist, 0 otherwise. If ="R" (default) the search is performed in the - standard list, if ="P" in the proof list. The last argument + standard list. The last argument is a pointer to an integer: if defined (.ne. 0) the pointed location is filled with 1 if the match is exact, with 0 if an matching entry with wild cards was found. diff --git a/README/README.SELECTOR b/README/README.SELECTOR index 9d6faaa71ffd4..05d6dfa33e523 100644 --- a/README/README.SELECTOR +++ b/README/README.SELECTOR @@ -10,32 +10,17 @@ derives from the TSelector class and implements the member functions with specific analysis algorithms. When running the analysis, ROOT calls the member functions in a well defined sequence and with well defined arguments. By following the model this analysis class can be -used to process data sequentialy on a local workstation and in batch -or in parallel using PROOF. +used to process data sequentially. ROOT can generate a skeleton class for a given TTree. This skeleton class is a good a starting point for the analysis class. It is recommended that users follow this method. -When running with PROOF a number of "slave" processes are used to -analyze the events. The user creates a PROOF session from the client -workstation which allocates a number of slaves. The slaves instantiate -an object of the users analysis class. Each slave processes a fraction -of the events as determined by the relative performance of the servers -on which the slaves are running. The PROOF system takes care of distributing -the work. It calls the TSelector functions in each slave. It also -distributes the input list to the slaves. This is a TList with streamable -objects provided in the client. After processing the events PROOF combines -the partial results of the slaves and returns the consolidated objects -(e.g. histograms) to the client session. - The two sequences below show the order in which the TSelector member functions are called when either processing a tree or chain on a single -workstation or when using PROOF to process trees or collections of keyed -objects on a distributed system. When running on a sequential query -the user calls TTree::Process() and TChain::Process(), when using PROOF -the user calls TDSet::Process() (a few other entry points are available -as well). Each of the member functions is described in detail after the +workstation. When running on a sequential query +the user calls TTree::Process() and TChain::Process(). +Each of the member functions is described in detail after the call sequences. @@ -57,29 +42,6 @@ Init() SlaveTerminate() Terminate() - -Distributed, parallel query, using PROOF: - -++ CLIENT Session ++ ++ (n) SLAVES ++ -Begin() - SlaveBegin() - Init() - Notify() - Process() - ... - Process() - ... - Init() - Notify() - Process() - ... - Process() - ... - SlaveTerminate() -Terminate() - - - ============================================================================== Main Framework Functions ============================================================================== @@ -89,18 +51,12 @@ The Begin() and SlaveBegin() member functions The Begin() function is called at the start of the query. It always runs in the client ROOT session. The SlaveBegin() function is either called -in the client or when running with PROOF, on each of the slaves. +in the client. All initialization that is needed for Process() (see below) must therefore be put in SlaveBegin(). Code which needs to access the local client environment, e.g. graphics or the filesystem must be put in Begin(). -When running with PROOF the input list (fInput) is distributed to the -slaves after Begin() returns and before SlaveBegin() is called. -This way objects on the client can be made available to the -TSelector instances in the slaves. -The tree argument is deprecated. (In the case of PROOF the tree is not -available on the client and 0 will be passed. The Init() function should -be used to implement operations depending on the tree) +The tree argument is deprecated. Signature: @@ -115,8 +71,7 @@ The Init() member function The Init() function is called when the selector needs to initialize a new tree or chain. Typically here the branch addresses of the tree will be set. It is normally not necessary to make changes to the generated -code, but the routine can be extended by the user if needed. Init() will -be called many times when running with PROOF. +code, but the routine can be extended by the user if needed. Signature: @@ -127,8 +82,8 @@ The Notify() member function ---------------------------- The Notify() function is called when a new file is opened. This can be either -for a new TTree in a TChain or when when a new TTree is started when using -PROOF. Typically here the branch pointers will be retrieved. It is normaly +for a new TTree in a TChain. +Typically here the branch pointers will be retrieved. It is normally not necessary to make changes to the generated code, but the routine can be extended by the user if needed. @@ -140,13 +95,11 @@ Signature: The Process() member function ----------------------------- -The Process() function is called for each entry in the tree (or possibly -keyed object in the case of PROOF) to be processed. The entry argument +The Process() function is called for each entry in the tree +to be processed. The entry argument specifies which entry in the currently loaded tree is to be processed. It can be passed to either TTree::GetEntry() or TBranch::GetEntry() -to read either all or the required parts of the data. When processing -keyed objects with PROOF, the object is already loaded and is available -via the fObject pointer. +to read either all or the required parts of the data. This function should contain the "body" of the analysis. It can contain simple or elaborate selection criteria, run algorithms on the data @@ -161,11 +114,7 @@ The SlaveTerminate() and Terminate() member functions ----------------------------------------------------- The SlaveTerminate() function is called after all entries or objects -have been processed. When running with PROOF it is executed by -each of the slaves. It can be used to do post processing before the -partial results of the slaves are merged. After SlaveTerminate() -the objects in the fOutput lists in the slaves are combined by the -PROOF system and returned to the client ROOT session. +have been processed. The Terminate() function is the last function to be called during a query. It always runs on the client, it can be used to present the results graphically or save the results to file. @@ -198,8 +147,7 @@ Signature: The SetInputList() member function ---------------------------------- -Setter for the input list of objects to be transfered to the -remote PROOF servers. The input list is transfered after the execution +The input list is transfered after the execution of the Begin() function, so objects can still be added in Begin() to this list. These objects are then available during the selection process (e.g. predefined histograms, etc.). Does not transfer ownership. @@ -214,10 +162,8 @@ The GetOutputList() member function Getter for the output list of objects to be transfered back to the client. The output list on each slave is transfered back to the client -session after the execution of the SlaveTerminate() function. The PROOF -master server merges the objects from the slave output lists in a single -output list (merging partial objects into a single one). Ownership remains -with the selector. Each query will clear this list. +session after the execution of the SlaveTerminate() function. +Ownership remains with the selector. Each query will clear this list. Signature: diff --git a/README/ReleaseNotes/empty.md b/README/ReleaseNotes/empty.md index 1e2e17edb9059..306d36ec8ce51 100644 --- a/README/ReleaseNotes/empty.md +++ b/README/ReleaseNotes/empty.md @@ -69,62 +69,24 @@ The following people have contributed to this new version: ## Deprecation and Removal +## Python Interface -## Core Libraries +## I/O +## Core -## I/O Libraries +## Histograms +## Math -## TTree Libraries +## RooFit +## Graphics and GUI -## Histogram Libraries +## Geometry +## Documentation and Examples -## Math Libraries - - -## RooFit Libraries - -## Graphics Backends - -## 2D Graphics Libraries - - -## 3D Graphics Libraries - - -## Geometry Libraries - - -## Database Libraries - - -## Networking Libraries - - -## GUI Libraries - - -## Montecarlo Libraries - - -## PROOF Libraries - - -## Language Bindings - - -## JavaScript ROOT - - -## Tutorials - - -## Class Reference Guide - - -## Build, Configuration and Testing Infrastructure +## Build, Configuration and Testing diff --git a/README/ReleaseNotes/v638/index.md b/README/ReleaseNotes/v638/index.md index f82528dea1d6e..519740250905c 100644 --- a/README/ReleaseNotes/v638/index.md +++ b/README/ReleaseNotes/v638/index.md @@ -5,6 +5,7 @@ * The build options `mysql`, `odbc` and `pgsql`, that were deprecated in ROOT 6.36, are now removed. * The `TGLWSIncludes.h` header is deprecated and will be removed in ROOT 6.40 * The `ROOT::Math::TDataPointN` class that can be used with the `ROOT::Math::KDETree` was removed. Use the templated `TDataPoint` instead. +* The Parallel ROOT Facility, `PROOF`, has been removed from the repository. ## Core Libraries * Behavior change: when selecting a template instantiation for a dictionary, all the template arguments have to be fully defined - the forward declarations are not enough any more. The error prompted by the dictionary generator will be `Warning: Unused class rule: MyTemplate`. diff --git a/bindings/jupyroot/python/JupyROOT/helpers/cppcompleter.py b/bindings/jupyroot/python/JupyROOT/helpers/cppcompleter.py index 3c669dd9d17cd..ef0e88d0529d4 100644 --- a/bindings/jupyroot/python/JupyROOT/helpers/cppcompleter.py +++ b/bindings/jupyroot/python/JupyROOT/helpers/cppcompleter.py @@ -86,7 +86,6 @@ class CppCompleter(object): TROOT::IsLineProcessing TROOT::IsModified TROOT::IsOnHeap - TROOT::IsProofServ TROOT::IsRootFile TROOT::IsSortable TROOT::IsWebDisplay diff --git a/bindings/pyroot/pythonizations/inc/TPyDispatcher.h b/bindings/pyroot/pythonizations/inc/TPyDispatcher.h index d1deb23df76be..4ed15dda7ed13 100644 --- a/bindings/pyroot/pythonizations/inc/TPyDispatcher.h +++ b/bindings/pyroot/pythonizations/inc/TPyDispatcher.h @@ -31,9 +31,7 @@ class TGVFileSplitter; class TList; class TObject; class TPad; -class TProofProgressInfo; class TQCommand; -class TSlave; class TSocket; class TVirtualPad; @@ -131,36 +129,6 @@ class TPyDispatcher : public TObject { PyObject *Dispatch(TGListTreeItem *item, TDNDData *data); PyObject *Dispatch(const char *name, const TList *attr); - // for PROOF - PyObject *Dispatch(const char *msg, Bool_t all) { return DispatchVA("si", msg, all); } - PyObject *Dispatch(Long64_t total, Long64_t processed) { return DispatchVA("LL", total, processed); } - PyObject *Dispatch(Long64_t total, Long64_t processed, Long64_t bytesread, Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) - { - return DispatchVA("LLLffff", total, processed, bytesread, initTime, procTime, evtrti, mbrti); - } - PyObject *Dispatch(Long64_t total, Long64_t processed, Long64_t bytesread, Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti, Int_t actw, Int_t tses, Float_t eses) - { - return DispatchVA("LLLffffiif", total, processed, bytesread, initTime, procTime, evtrti, mbrti, actw, tses, eses); - } - PyObject *Dispatch(const char *sel, Int_t sz, Long64_t fst, Long64_t ent) - { - return DispatchVA("siLL", sel, sz, fst, ent); - } - PyObject *Dispatch(const char *msg, Bool_t status, Int_t done, Int_t total) - { - return DispatchVA("siii", msg, status, done, total); - } - - PyObject *Dispatch(TSlave *slave, Long64_t total, Long64_t processed) - { - return DispatchVA1("TSlave", slave, "LL", total, processed); - } - PyObject *Dispatch(TProofProgressInfo *pi) { return DispatchVA1("TProofProgressInfo", pi, 0); } - PyObject *Dispatch(TSlave *slave) { return DispatchVA("TSlave", slave, 0); } - PyObject *Dispatch(TSlave *slave, TProofProgressInfo *pi); - private: PyObject *fCallable; //! callable object to be dispatched }; diff --git a/bindings/pyroot/pythonizations/src/TPyDispatcher.cxx b/bindings/pyroot/pythonizations/src/TPyDispatcher.cxx index c5786cb388451..5c3cbd33c45a0 100644 --- a/bindings/pyroot/pythonizations/src/TPyDispatcher.cxx +++ b/bindings/pyroot/pythonizations/src/TPyDispatcher.cxx @@ -264,22 +264,3 @@ PyObject *TPyDispatcher::Dispatch(const char *name, const TList *attr) return result; } - -//////////////////////////////////////////////////////////////////////////////// - -PyObject *TPyDispatcher::Dispatch(TSlave *slave, TProofProgressInfo *pi) -{ - PyObject *args = PyTuple_New(2); - PyTuple_SET_ITEM(args, 0, CPyCppyy::Instance_FromVoidPtr(slave, "TSlave")); - PyTuple_SET_ITEM(args, 1, CPyCppyy::Instance_FromVoidPtr(pi, "TProofProgressInfo")); - - PyObject *result = PyObject_CallObject(fCallable, args); - Py_XDECREF(args); - - if (!result) { - PyErr_Print(); - return 0; - } - - return result; -} diff --git a/cmake/modules/RootBuildOptions.cmake b/cmake/modules/RootBuildOptions.cmake index 3bf686aadbc95..ae3310e0cab74 100644 --- a/cmake/modules/RootBuildOptions.cmake +++ b/cmake/modules/RootBuildOptions.cmake @@ -147,7 +147,6 @@ ROOT_BUILD_OPTION(minuit2_mpi OFF "Enable support for MPI in Minuit2") ROOT_BUILD_OPTION(minuit2_omp OFF "Enable support for OpenMP in Minuit2") ROOT_BUILD_OPTION(mpi OFF "Enable support for Message Passing Interface (MPI)") ROOT_BUILD_OPTION(opengl ON "Enable support for OpenGL (requires libGL and libGLU)") -ROOT_BUILD_OPTION(proof OFF "Enable support for PROOF") ROOT_BUILD_OPTION(pyroot ON "Enable support for automatic Python bindings (PyROOT)") ROOT_BUILD_OPTION(pythia8 OFF "Enable support for Pythia 8.x [GPL]") ROOT_BUILD_OPTION(qt6web OFF "Enable support for Qt6 web-based display (requires Qt6::WebEngineCore and Qt6::WebEngineWidgets)") @@ -236,7 +235,6 @@ if(all) set(imt_defvalue ON) set(mathmore_defvalue ON) set(opengl_defvalue ON) - set(proof_defvalue OFF) set(pythia8_defvalue ON) set(pyroot_defvalue ON) set(qt6web_defvalue ON) @@ -393,7 +391,7 @@ endif() # Please notify SPI when adding to this list foreach(opt afdsmgrd afs alien bonjour builtin_afterimage castor chirp cxx11 cxx14 cxx17 cxxmodules exceptions geocad gfal glite globus gsl_shared hdfs html ios jemalloc krb5 - ldap memstat minuit2 monalisa oracle pyroot-python2 pyroot_legacy + ldap memstat minuit2 monalisa oracle proof pyroot-python2 pyroot_legacy pythia6 pythia6_nolink python qt qtgsi qt5web rfio ruby sapdb srp table tcmalloc vmc xproofd mysql odbc pgsql) if(${opt}) diff --git a/cmake/modules/RootConfiguration.cmake b/cmake/modules/RootConfiguration.cmake index 3ea25fd7fd76e..bd82fdf6021fe 100644 --- a/cmake/modules/RootConfiguration.cmake +++ b/cmake/modules/RootConfiguration.cmake @@ -745,7 +745,6 @@ if(thisroot_scripts) endif() configure_file(${CMAKE_SOURCE_DIR}/config/setxrd.csh ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/setxrd.csh COPYONLY) configure_file(${CMAKE_SOURCE_DIR}/config/setxrd.sh ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/setxrd.sh COPYONLY) -configure_file(${CMAKE_SOURCE_DIR}/config/proofserv.in ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/proofserv @ONLY NEWLINE_STYLE UNIX) configure_file(${CMAKE_SOURCE_DIR}/config/roots.in ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/roots @ONLY NEWLINE_STYLE UNIX) configure_file(${CMAKE_SOURCE_DIR}/config/rootssh ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/rootssh @ONLY NEWLINE_STYLE UNIX) if(WIN32) @@ -794,7 +793,6 @@ endif() install(FILES ${CMAKE_BINARY_DIR}/installtree/root-config ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/roots - ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/proofserv ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/rootssh PERMISSIONS OWNER_EXECUTE OWNER_WRITE OWNER_READ GROUP_EXECUTE GROUP_READ diff --git a/config/proofserv.in b/config/proofserv.in deleted file mode 100755 index dc4f10db30a40..0000000000000 --- a/config/proofserv.in +++ /dev/null @@ -1,106 +0,0 @@ -#!/bin/sh - -# -# $Id$ -# -# Template for the proofserv wrapper script. -# The proofserv wrapper script can be used to initialize the -# environment for proofserv as needed. It could be extended -# to select a specific ROOT version. It also allows debug -# tools like valgrind etc to be used. -# -# This example script should be sufficient for most installations -# but can be modified as needed. -# - -# -# Setup notification -# -if [ -n "$ROOTPROOFLOGFILE" ]; then -# Use the standard log file - LOGFILE="$ROOTPROOFLOGFILE" -else -# Use a temp file - if [ -n "$TMPDIR" ]; then - LOGFILE="$TMPDIR/proofserv.log" - else - LOGFILE="/tmp/proofserv.log" - fi -fi - -# -# If requested, initialize the environment. -# The PROOF_INITCMD variable contains the command to be executed to initialize -# the environment. -# For a simple variable setting just use 'echo export VAR=value', e.g. -# -# root [] TProof::AddEnvVar("PROOF_INITCMD", -# "echo export LD_LIBRARY_PATH=/some/new/libpath:$LDLIBRARY_PATH") -# -# If the setup is defined by a script, e.g. /some/path/setup-env.sh, then the -# script should be sourced: -# -# root [] TProof::AddEnvVar("PROOF_INITCMD","source /some/path/setup-env.sh") -# -# If the script outputs the command to be executed, e.g. /some/path/getscram.sh, -# then just put the script path: -# -# root [] TProof::AddEnvVar("PROOF_INITCMD","/some/path/getscram.sh") -# - -if [ -n "$PROOF_INITCMD" ]; then - NOW=`date +%H:%M:%S` - echo "$NOW: $1: initializing environment with: $PROOF_INITCMD" >> "$LOGFILE" - eval `$PROOF_INITCMD` -fi - -# -# Run master, workers or all with a debug command. -# E.g in the client do: -# root [] TProof::AddEnvVar("PROOF_WRAPPERCMD","valgrind --log-file=/tmp/vg") -# -if [ -n "$PROOF_WRAPPERCMD" ]; then - WRAPPER="$PROOF_WRAPPERCMD " -else - if [ -n "$PROOF_TOPMASTER_WRAPPERCMD" -a "$1" = "proofserv" -a "$5" = "2" ]; then - WRAPPER="$PROOF_TOPMASTER_WRAPPERCMD " - fi - if [ -n "$PROOF_MASTER_WRAPPERCMD" -a "$1" = "proofserv" ]; then - WRAPPER="$PROOF_MASTER_WRAPPERCMD " - fi - if [ -n "$PROOF_SUBMASTER_WRAPPERCMD" -a "$1" = "proofserv" -a "$5" = "1" ]; then - WRAPPER="$PROOF_SUBMASTER_WRAPPERCMD " - fi - if [ -n "$PROOF_SLAVE_WRAPPERCMD" -a "$1" = "proofslave" ]; then - WRAPPER="$PROOF_SLAVE_WRAPPERCMD " - fi -fi - -# Apply limit on the resident memory, if requested -ECHOMEM="" -if [ "x$PROOF_RESMEMMAX" != "x" ]; then - resmemmax=`expr $PROOF_RESMEMMAX \* 1024` - ulimit -m $resmemmax - ECHOMEM="yes" -fi -# Apply limit on the virtual memory, if requested -if [ "x$PROOF_VIRTMEMMAX" != "x" -o "x$ROOTPROOFASHARD" != "x" ]; then - if [ "x$PROOF_VIRTMEMMAX" != "x" ]; then - virmemmax=`expr $PROOF_VIRTMEMMAX \* 1024` - else - virmemmax=`expr $ROOTPROOFASHARD \* 1024` - fi - ulimit -v $virmemmax - ECHOMEM="yes" -fi -if [ "x$ECHOMEM" != "x" ]; then - NOW=`date +%H:%M:%S` - echo "$NOW: $1: limiting {resident, virtual} memory to {$resmemmax,$virmemmax} kBytes" >> $LOGFILE -fi - -if [ "x$WRAPPER" != "x" ]; then - NOW=`date +%H:%M:%S` - echo "$NOW: $1: executing $WRAPPER @bindir@/proofserv.exe $@" >> $LOGFILE -fi - -exec $WRAPPER @bindir@/proofserv.exe "$@" diff --git a/config/rootauthrc.in b/config/rootauthrc.in index 09f2dc741e34e..a7aef25af3374 100644 --- a/config/rootauthrc.in +++ b/config/rootauthrc.in @@ -132,20 +132,4 @@ # The fourth directive includes the content of the system # defaults. # -# - Finally, also supported are lines of the form: -# -# proofserv [:][:[:...[:]]] \ -# [:][:[:...[:]]] \ -# ... [:][:[:...[:]]] -# -# which are active only for PROOF sessions and specify the list of hosts -# for which the authentication info should be transmitted to the slaves -# of the PROOF cluster; these directives are useful, for example, in -# the case of data servers external to the PROOF cluster that you may -# want to access via a given 'user' and a given authentication 'method'; -# 'user' and 'method' are not mandatory; for each (an user, method) -# specified with 'proofserv' all the information that can be collected -# from the rest of the .rootauthrc file is sent to slaves via the master -# -# default list usrpwd diff --git a/config/rootrc.in b/config/rootrc.in index c437b829f68f2..8dfbc64481ca4 100644 --- a/config/rootrc.in +++ b/config/rootrc.in @@ -218,7 +218,7 @@ Gui.MimeTypeFile: $(HOME)/.root.mimes Browser.Name: @root_browser_class@ # Browser Options (plugins) # F: File browser E: Text Editor H: HTML browser -# C: Canvas I: I/O redirection P: Proof G: GL viewer +# C: Canvas I: I/O redirection G: GL viewer Browser.Options: FCEI # Can be either small, big, list, details Browser.IconStyle: small @@ -383,35 +383,6 @@ ACLiC.Linkdef: _linkdef # Add extra options to rootcling invocation by ACLiC #ACLiC.ExtraRootclingFlags: [-optA ... -optZ] -# PROOF related variables -# -# PROOF debug options. -# Proof.DebugLevel: 0 -# Proof.DebugMask: -1 -# -# PROOF GDB hooks -# allows a debugger to be attached early in the startup phase of proofserv -# 0 - don't wait -# 1 - master proofserv enters wait loop -# 2 - slave proofserv enters wait loop -# 3 - any proofserv enters wait loop -# -# Proof.GdbHook: 0 -# -# To control the number of processes in PROOF-Lite (0 disables PROOF-Lite). -# This setting cannot be overwritten in the user rootrc files. -# ProofLite.MaxWorkers: -1 -# -# On the master enable parallel startup of workers using threads -# Proof.ParallelStartup: no -# -# Proof.StatsHist: no -# Proof.StatsTrace: no -# Proof.SlaveStatsTrace: no -# -# Proof.CondorHome: /opt/condor -# Proof.CondorConfig: /opt/condor/etc/condor_config - # Connection is shutdown at timeout expiration. Timeout is in seconds. # Negotiation cannot be attempted at low level (i.e. inside # TAuthenticate::Authenticate()) because of synchronization @@ -508,12 +479,6 @@ TS3WebFile.Root.MultiRangeServer: Huawei OBS Url.Special: file: hpss: dcache: +Url.Special: /alien/- -# PROOF XRD client variables -# Debug level (<0 : errors, 0 : minimal, 1 : low, 2 : medium, 3 : high) [-1] -# XProof.Debug: 0 -# Socket read timeout [in secs: default 10 secs] -# XProof.ReadTimeout: 10 - # XNet.Debug - log verbosity level (default 0) # (0 = nothing, # 1 = Info (messages of interest to the user) diff --git a/core/base/inc/MessageTypes.h b/core/base/inc/MessageTypes.h index cd93ea0dccdbc..b17e626660418 100644 --- a/core/base/inc/MessageTypes.h +++ b/core/base/inc/MessageTypes.h @@ -37,67 +37,6 @@ enum EMessageTypes { kMESS_STREAMERINFO = 6, //TStreamerInfo object follows kMESS_PROCESSID = 7, //TProcessID object follows - //---- PROOF message opcodes (1000 - 1999) - kPROOF_GROUPVIEW = 1000, //groupview follows - kPROOF_STOP = 1001, //stop proof server - kPROOF_FATAL = 1002, //server got fatal error and died - kPROOF_LOGLEVEL = 1003, //loglevel follows - kPROOF_LOGFILE = 1004, //log file length and content follows - kPROOF_LOGDONE = 1005, //log file received, status follows - kPROOF_STATUS = 1006, //print status of worker - (OBSOLETE Message) - kPROOF_PING = 1007, //ping worker - kPROOF_PRINT = 1008, //ask master to print config - kPROOF_RESET = 1009, //reset worker - kPROOF_GETOBJECT = 1010, //ask for object with given name - kPROOF_GETPACKET = 1011, //ask for next packet - kPROOF_CHECKFILE = 1012, //filename and md5 follows - kPROOF_SENDFILE = 1013, //filename, length and file follows - kPROOF_PARALLEL = 1014, //number of parallel workers follows - kPROOF_PROCESS = 1015, //process events, DSet and input list follow - kPROOF_OUTPUTLIST = 1016, //return the output list from Process() - kPROOF_AUTOBIN = 1017, //callback for auto binning - kPROOF_CACHE = 1018, //cache and package handling messages - kPROOF_GETENTRIES = 1019, //report back number of entries to master - kPROOF_PROGRESS = 1020, //event loop progress - kPROOF_FEEDBACK = 1021, //intermediate version of objects - kPROOF_STOPPROCESS = 1022, //stop or abort the current process call - kPROOF_HOSTAUTH = 1023, //HostAuth info follows - kPROOF_GETSLAVEINFO = 1024, //get worker info from master - kPROOF_GETTREEHEADER = 1025, //get tree object - kPROOF_GETOUTPUTLIST = 1026, //get the output list names - kPROOF_GETSTATS = 1027, //get statistics of workers - kPROOF_GETPARALLEL = 1028, //get number of parallel workers - kPROOF_VALIDATE_DSET = 1029, //validate a TDSet - kPROOF_DATA_READY = 1030, //ask if the data is ready on nodes - kPROOF_QUERYLIST = 1031, //ask/send the list of queries - kPROOF_RETRIEVE = 1032, //asynchronous retrieve of query results - kPROOF_ARCHIVE = 1033, //archive query results - kPROOF_REMOVE = 1034, //remove query results from the lists - kPROOF_STARTPROCESS = 1035, //signals the start of query processing - kPROOF_SETIDLE = 1036, //signals idle state of session - kPROOF_QUERYSUBMITTED = 1037, //signals querysubmission - kPROOF_SESSIONTAG = 1038, //message with unique session tag - kPROOF_MAXQUERIES = 1039, //message with max number of queries - kPROOF_CLEANUPSESSION = 1040, //cleanup session query area - kPROOF_SERVERSTARTED = 1041, //signal completion of a server startup - kPROOF_DATASETS = 1042, //dataset management - kPROOF_PACKAGE_LIST = 1043, //a list of package names (TObjString's) follows - kPROOF_MESSAGE = 1044, //a message for the client follows - kPROOF_LIB_INC_PATH = 1045, //a list of lib/inc paths follows - kPROOF_WORKERLISTS = 1046, //an action on any of the worker list follows - kPROOF_DATASET_STATUS = 1047, //status of data set preparation before processing - kPROOF_OUTPUTOBJECT = 1048, //output object follows - kPROOF_SETENV = 1049, //buffer with env vars to set - kPROOF_REALTIMELOG = 1050, //switch on/off real-time retrieval of log messages - kPROOF_VERSARCHCOMP = 1051, //String with worker version/architecture/compiler follows - kPROOF_ENDINIT = 1052, //signals end of initialization on worker - kPROOF_TOUCH = 1053, //touch the client admin file - kPROOF_FORK = 1054, //ask the worker to clone itself - kPROOF_GOASYNC = 1055, //switch to asynchronous mode - kPROOF_SUBMERGER = 1056, //sub-merger based approach in finalization - kPROOF_ECHO = 1057, //object echo request from client - kPROOF_SENDOUTPUT = 1058, //control output sending - //---- ROOTD message opcodes (2000 - 2099) kROOTD_USER = 2000, //user id follows kROOTD_PASS = 2001, //passwd follows diff --git a/core/base/inc/TFileCollection.h b/core/base/inc/TFileCollection.h index eba5331f7b50f..47f6298aacd31 100644 --- a/core/base/inc/TFileCollection.h +++ b/core/base/inc/TFileCollection.h @@ -18,7 +18,7 @@ // // // Class that contains a list of TFileInfo's and accumulated meta // // data information about its entries. This class is used to describe // -// file sets as stored by Grid file catalogs, by PROOF or any other // +// file sets as stored by Grid file catalogs, by any // // collection of TFile names. // // // ////////////////////////////////////////////////////////////////////////// diff --git a/core/base/inc/TROOT.h b/core/base/inc/TROOT.h index 6ddff151842e8..1cebafc5a30dd 100644 --- a/core/base/inc/TROOT.h +++ b/core/base/inc/TROOT.h @@ -167,7 +167,6 @@ friend TROOT *ROOT::Internal::GetROOT2(); TSeqCollection *fStreamerInfo = nullptr; ///< List of active StreamerInfo classes TCollection *fClassGenerators = nullptr; ///< List of user defined class generators; TSeqCollection *fSecContexts = nullptr; ///< List of security contexts (TSecContext) - TSeqCollection *fProofs = nullptr; ///< List of proof sessions TSeqCollection *fClipboard = nullptr; ///< List of clipboard objects TSeqCollection *fDataSets = nullptr; ///< List of data sets (TDSet or TChain) AListOfEnums_t fEnums = nullptr; ///< List of enum types @@ -258,7 +257,6 @@ friend TROOT *ROOT::Internal::GetROOT2(); TSeqCollection *GetListOfMessageHandlers() const { return fMessageHandlers; } TCollection *GetListOfClassGenerators() const { return fClassGenerators; } TSeqCollection *GetListOfSecContexts() const { return fSecContexts; } - TSeqCollection *GetListOfProofs() const { return fProofs; } TSeqCollection *GetClipboard() const { return fClipboard; } TSeqCollection *GetListOfDataSets() const { return fDataSets; } TCollection *GetListOfEnums(Bool_t load = kFALSE); @@ -290,7 +288,6 @@ friend TROOT *ROOT::Internal::GetROOT2(); Bool_t IsInterrupted() const { return fInterrupt; } Bool_t IsEscaped() const { return fEscape; } Bool_t IsLineProcessing() const { return fLineIsProcessing ? kTRUE : kFALSE; } - Bool_t IsProofServ() const { return fName == "proofserv" ? kTRUE : kFALSE; } Bool_t IsRootFile(const char *filename) const; Bool_t IsWebDisplay() const { return fIsWebDisplay; } Bool_t IsWebDisplayBatch() const { return fIsWebDisplayBatch; } diff --git a/core/base/inc/TUrl.h b/core/base/inc/TUrl.h index e40a066ceac4e..0c44d85c4134a 100644 --- a/core/base/inc/TUrl.h +++ b/core/base/inc/TUrl.h @@ -34,7 +34,7 @@ class TUrl : public TObject { private: mutable TString fUrl; // full URL - TString fProtocol; // protocol: http, ftp, news, root, proof, ... + TString fProtocol; // protocol: http, ftp, news, root, ... TString fUser; // user name TString fPasswd; // password TString fHost; // remote host diff --git a/core/base/inc/TVirtualPerfStats.h b/core/base/inc/TVirtualPerfStats.h index 07bef1db3cb16..75e65b9898c1a 100644 --- a/core/base/inc/TVirtualPerfStats.h +++ b/core/base/inc/TVirtualPerfStats.h @@ -16,7 +16,7 @@ // // // TVirtualPerfStats // // // -// Provides the interface for the PROOF internal performance measurment // +// Provides the interface for an internal performance measurment // // and event tracing. // // // ////////////////////////////////////////////////////////////////////////// @@ -90,7 +90,7 @@ class TVirtualPerfStats : public TObject { static const char *EventType(EEventType type); - ClassDefOverride(TVirtualPerfStats,0) // ABC for collecting PROOF statistics + ClassDefOverride(TVirtualPerfStats,0) }; #define gPerfStats (TVirtualPerfStats::CurrentPerfStats()) diff --git a/core/base/src/TApplication.cxx b/core/base/src/TApplication.cxx index b00a9b3522a27..b52b8af8b9881 100644 --- a/core/base/src/TApplication.cxx +++ b/core/base/src/TApplication.cxx @@ -111,8 +111,7 @@ TApplication::TApplication() : /// line options recognized by TApplication are described in the GetOptions() /// method. The recognized options are removed from the argument array. /// The original list of argument options can be retrieved via the Argc() -/// and Argv() methods. The appClassName "proofserv" is reserved for the -/// PROOF system. The "options" and "numOptions" arguments are not used, +/// and Argv() methods. The "options" and "numOptions" arguments are not used, /// except if you want to by-pass the argv processing by GetOptions() /// in which case you should specify numOptions<0. All options will /// still be available via the Argv() method for later use. diff --git a/core/base/src/TFileCollection.cxx b/core/base/src/TFileCollection.cxx index 7b205edb57efc..d0c25c2597c64 100644 --- a/core/base/src/TFileCollection.cxx +++ b/core/base/src/TFileCollection.cxx @@ -14,7 +14,7 @@ Class that contains a list of TFileInfo's and accumulated meta data information about its entries. This class is used to describe -file sets as stored by Grid file catalogs, by PROOF or any other +file sets as stored by Grid file catalogs, by any collection of TFile names. */ diff --git a/core/base/src/TROOT.cxx b/core/base/src/TROOT.cxx index 64222d36bdd22..966bb404867d5 100644 --- a/core/base/src/TROOT.cxx +++ b/core/base/src/TROOT.cxx @@ -752,7 +752,6 @@ TROOT::TROOT(const char *name, const char *title, VoidFuncPtr_t *initfunc) : TDi fCleanups = setNameLocked(new THashList, "Cleanups"); fMessageHandlers = setNameLocked(new TList, "MessageHandlers"); fSecContexts = setNameLocked(new TList, "SecContexts"); - fProofs = setNameLocked(new TList, "Proofs"); fClipboard = setNameLocked(new TList, "Clipboard"); fDataSets = setNameLocked(new TList, "DataSets"); fTypes = new TListOfTypes; fTypes->UseRWLock(); @@ -778,7 +777,6 @@ TROOT::TROOT(const char *name, const char *title, VoidFuncPtr_t *initfunc) : TDi fRootFolder->AddFolder("Cleanups", "List of RecursiveRemove Collections",fCleanups); fRootFolder->AddFolder("StreamerInfo","List of Active StreamerInfo Classes",fStreamerInfo); fRootFolder->AddFolder("SecContexts","List of Security Contexts",fSecContexts); - fRootFolder->AddFolder("PROOF Sessions", "List of PROOF sessions",fProofs); fRootFolder->AddFolder("ROOT Memory","List of Objects in the gROOT Directory",fList); fRootFolder->AddFolder("ROOT Files","List of Connected ROOT Files",fFiles); @@ -851,7 +849,6 @@ TROOT::TROOT(const char *name, const char *title, VoidFuncPtr_t *initfunc) : TDi // Set initial/default list of browsable objects fBrowsables->Add(fRootFolder, "root"); - fBrowsables->Add(fProofs, "PROOF Sessions"); fBrowsables->Add(workdir, gSystem->WorkingDirectory()); fBrowsables->Add(fFiles, "ROOT Files"); @@ -942,7 +939,6 @@ TROOT::~TROOT() #ifdef R__COMPLETE_MEM_TERMINATION SafeDelete(fCanvases); SafeDelete(fTasks); - SafeDelete(fProofs); SafeDelete(fDataSets); SafeDelete(fClipboard); @@ -1205,7 +1201,7 @@ void TROOT::CloseFiles() socket->SetBit(kMustCleanup); fClosedObjects->AddLast(socket); } else { - // Crap ... this is not a socket, likely Proof or something, let's try to find a Close + // Crap ... this is not a socket, let's try to find a Close Longptr_t other_offset; CallFunc_t *otherCloser = gInterpreter->CallFunc_Factory(); gInterpreter->CallFunc_SetFuncProto(otherCloser, socket->IsA()->GetClassInfo(), "Close", "", &other_offset); diff --git a/core/base/src/TUrl.cxx b/core/base/src/TUrl.cxx index dac1c75c8b9c5..0de3ddc3c02c0 100644 --- a/core/base/src/TUrl.cxx +++ b/core/base/src/TUrl.cxx @@ -62,15 +62,14 @@ ClassImp(TUrl); /// ~~~ {.cpp} /// url: [proto://][user[:passwd]@]host[:port]/file.ext[?options][#anchor] /// ~~~ -/// Known protocols: http, root, proof, ftp, news and any special protocols +/// Known protocols: http, root, ftp, news and any special protocols /// defined in the rootrc Url.Special key. /// The default protocol is "http", unless defaultIsFile is true in which /// case the url is assumed to be of type "file". /// If a passwd contains a @ it must be escaped by a \\, e.g. /// "pip@" becomes "pip\\@". /// -/// Default ports: http=80, root=1094, proof=1093, ftp=20, news=119. -/// Port #1093 has been assigned by IANA (www.iana.org) to proofd. +/// Default ports: http=80, root=1094, ftp=20, news=119. /// Port #1094 has been assigned by IANA (www.iana.org) to rootd. TUrl::TUrl(const char *url, Bool_t defaultIsFile) @@ -96,15 +95,14 @@ TUrl::~TUrl() ///~~~ {.cpp} /// url: [proto://][user[:passwd]@]host[:port]/file.ext[?options][#anchor] ///~~~ -/// Known protocols: http, root, proof, ftp, news and any special protocols +/// Known protocols: http, root, ftp, news and any special protocols /// defined in the rootrc Url.Special key. /// The default protocol is "http", unless defaultIsFile is true in which /// case the url is assumed to be of type "file". /// If a passwd contains a @ it must be escaped by a \\, e.g. /// "pip@" becomes "pip\\@". /// -/// Default ports: http=80, root=1094, proof=1093, ftp=20, news=119. -/// Port #1093 has been assigned by IANA (www.iana.org) to proofd. +/// Default ports: http=80, root=1094, ftp=20, news=119. /// Port #1094 has been assigned by IANA (www.iana.org) to rootd. void TUrl::SetUrl(const char *url, Bool_t defaultIsFile) @@ -427,7 +425,6 @@ const char *TUrl::GetUrl(Bool_t withDeflt) const Bool_t deflt = kFALSE; if ((!fProtocol.CompareTo("http") && fPort == 80) || - (fProtocol.BeginsWith("proof") && fPort == 1093) || (fProtocol.BeginsWith("root") && fPort == 1094) || (!fProtocol.CompareTo("ftp") && fPort == 20) || (!fProtocol.CompareTo("news") && fPort == 119) || @@ -536,8 +533,6 @@ void TUrl::SetProtocol(const char *proto, Bool_t setDefaultPort) fPort = 80; else if (!fProtocol.CompareTo("https")) fPort = 443; - else if (fProtocol.BeginsWith("proof")) // can also be proofs or proofk - fPort = 1093; else if (fProtocol.BeginsWith("root")) // can also be roots or rootk fPort = 1094; else if (!fProtocol.CompareTo("ftp")) diff --git a/core/base/src/TVirtualPerfStats.cxx b/core/base/src/TVirtualPerfStats.cxx index aa38b4f674cc9..f331db25086e9 100644 --- a/core/base/src/TVirtualPerfStats.cxx +++ b/core/base/src/TVirtualPerfStats.cxx @@ -12,7 +12,7 @@ /** \class TVirtualPerfStats \ingroup Base -Provides the interface for the PROOF internal performance measurement +Provides the interface for the an internal performance measurement and event tracing. */ diff --git a/core/meta/inc/TFileMergeInfo.h b/core/meta/inc/TFileMergeInfo.h index 32575f09b2006..5404fc8d1b6d0 100644 --- a/core/meta/inc/TFileMergeInfo.h +++ b/core/meta/inc/TFileMergeInfo.h @@ -1,4 +1,3 @@ -// @(#)root/proofplayer:$Id$ // Author: Philippe Canal May, 2011 /************************************************************************* diff --git a/documentation/doxygen/makeinput.sh b/documentation/doxygen/makeinput.sh index 1f590ac945986..a980f6877b1eb 100755 --- a/documentation/doxygen/makeinput.sh +++ b/documentation/doxygen/makeinput.sh @@ -66,7 +66,6 @@ echo " ../../net/http/ \\" >> Doxyfile_INPUT echo " ../../net/net/ \\" >> Doxyfile_INPUT echo " ../../net/netxng/ \\" >> Doxyfile_INPUT echo " ../../net/httpsniff/ \\" >> Doxyfile_INPUT -echo " ../../proof/ \\" >> Doxyfile_INPUT echo " ../../tmva/ \\" >> Doxyfile_INPUT echo " ../../roofit/ \\" >> Doxyfile_INPUT echo " ../../tree/ \\" >> Doxyfile_INPUT diff --git a/documentation/primer/filio.md b/documentation/primer/filio.md index be95bfb800c50..03ca2612d12a9 100644 --- a/documentation/primer/filio.md +++ b/documentation/primer/filio.md @@ -202,19 +202,15 @@ to insert your own analysis code, `MySelector.C`. @ROOT_INCLUDE_FILE macros/makeMySelector.C ``` -The template contains the entry points `Begin()` and `SlaveBegin()` -called before processing of the `TChain` starts, `Process()` called for -every entry of the chain, and `SlaveTerminate()` and `Terminate()` +The template contains the entry points `Begin()` before processing +of the `TChain` starts, `Process()` called for +every entry of the chain, and `Terminate()` called after the last entry has been processed. Typically, initialization like booking of histograms is performed in -`SlaveBegin()`, the analysis, i.e. the selection of entries, +`Begin()`, the analysis, i.e. the selection of entries, calculations and filling of histograms, is done in `Process()`, and final operations like plotting and storing of results happen in -`SlaveTerminate()` or `Terminate()`. - -The entry points `SlaveBegin()` and `SlaveTerminate()` are called on -so-called slave nodes only if parallel processing via `PROOF` or -`PROOF lite` is enabled, as will be explained below. +`Terminate()`. A simple example of a selector class is shown in the macro `MySelector.C`. The example is executed with the following sequence of @@ -231,8 +227,8 @@ initiates the compilation of the `MySelector.C` with the system compiler in order to improve performance. The code in `MySelector.C`, shown in the listing below, books some -histograms in `SlaveBegin()` and adds them to the instance `fOutput`, -which is of the class `TList` [^6]. The final processing in +histograms in `Begin()` and adds them to the instance `fOutput`, +which is of the class `TList`. The final processing in `Terminate()` allows to access histograms and store, display or save them as pictures. This is shown in the example via the `TList` `fOutput`. See the commented listing below for more details; most of the @@ -243,98 +239,6 @@ text is actually comments generated automatically by @ROOT_INCLUDE_FILE macros/MySelector.C ``` -### *For power-users:* Multi-core processing with `PROOF lite` ### - - -The processing of n-tuples via a selector function of type `TSelector` -through `TChain::Process()`, as described at the end of the previous -section, offers an additional advantage in particular for very large -data sets: on distributed systems or multi-core architectures, portions -of data can be processed in parallel, thus significantly reducing the -execution time. On modern computers with multi-core CPUs or -hardware-threading enabled, this allows a much faster turnaround of -analyses, since all the available CPU power is used. - -On distributed systems, a PROOF server and worker nodes have to be set -up, as described in detail in the ROOT documentation. On a single -computer with multiple cores, `PROOF lite` can be used instead. Try the -following little macro, `RunMySelector.C`, which contains two extra -lines compared to the example above (adjust the number of workers -according to the number of CPU cores): - -``` {.cpp} -{// set up a TChain -TChain *ch=new TChain("cond_data", "My Chain for Example N-Tuple"); - ch->Add("conductivity_experiment*.root"); -// eventually, start Proof Lite on cores -TProof::Open("workers=4"); -ch->SetProof(); -ch->Process("MySelector.C+");} -``` - -The first command, `TProof::Open(const char*)` starts a local PROOF -server (if no arguments are specified, all cores will be used), and the -command `ch->SetProof();` enables processing of the chain using PROOF. -Now, when issuing the command `ch->Process("MySelector.C+);`, the code -in `MySelector.C` is compiled and executed on each slave node. The -methods `Begin()` and `Terminate()` are executed on the master only. The -list of n-tuple files is analysed, and portions of the data are assigned -to the available slave processes. Histograms booked in `SlaveBegin()` -exist in the processes on the slave nodes, and are filled accordingly. -Upon termination, the PROOF master collects the histograms from the -slaves and merges them. In `Terminate()` all merged histograms are -available and can be inspected, analysed or stored. The histograms are -handled via the instances `fOutput` of class `TList` in each slave -process, and can be retrieved from this list after merging in -`Terminate`. - -To explore the power of this mechanism, generate some very large -n-tuples using the script from the section -[Storing Arbitrary N-tuples](#storing-arbitrary-n-tuples) - -you could try 10 000 000 events (this -results in a large n-tuple of about 160 MByte in size). You could also -generate a large number of files and use wildcards to add the to the -`TChain`. Now execute: `> root -l RunMySelector.C` and watch what -happens: - -``` {.cpp} -Processing RunMySelector.C... - +++ Starting PROOF-Lite with 4 workers +++ -Opening connections to workers: OK (4 workers) -Setting up worker servers: OK (4 workers) -PROOF set to parallel mode (4 workers) - -Info in : starting query: 1 -Info in : nwrks: 4 -Info in : creating shared library - ~/DivingROOT/macros/MySelector_C.so -*==* ----- Begin of Job ----- Date/Time = Wed Feb 15 23:00:04 2012 -Looking up for exact location of files: OK (4 files) -Looking up for exact location of files: OK (4 files) -Info in : - Setting max number of workers per node to 4 -Validating files: OK (4 files) -Info in : - fraction of remote files 1.000000 -Info in : - file ResistanceDistribution.png has been created -*==* ----- End of Job ----- Date/Time = Wed Feb 15 23:00:08 2012 -Lite-0: all output objects have been merged -``` - -Log files of the whole processing chain are kept in the directory -`~.proof` for each worker node. This is very helpful for debugging or if -something goes wrong. As the method described here also works without -using PROOF, the development work on an analysis script can be done in -the standard way on a small subset of the data, and only for the full -processing one would use parallelism via PROOF. - -It is worth to remind the reader that the speed of typical data analysis -programs limited by the I/O speed (for example the latencies implied by -reading data from a hard drive). It is therefore expected that this -limitation cannot be eliminated with the usage of any parallel analysis -toolkit. - ### Optimisation Regarding N-tuples ### ROOT automatically applies compression algorithms on n-tuples to reduce @@ -370,5 +274,3 @@ minutes. beneficial to store meta data and payload data separately, i.e. write the meta data tree in a bulk to a file at the end of your job instead of writing both trees interleaved. - -[^6]: The usage of `fOutput` is not really needed for this simple example, but it allows re-usage of the exact code in parallel processing with `PROOF` (see next section). diff --git a/documentation/users-guide/InstallandBuild.md b/documentation/users-guide/InstallandBuild.md index 8175e169eaac9..7e7c9e14a376a 100644 --- a/documentation/users-guide/InstallandBuild.md +++ b/documentation/users-guide/InstallandBuild.md @@ -579,106 +579,6 @@ The additional include directives for ACLiC compilations are set by: ACLiC.IncludePaths: -I/where/the/includes/are ``` -### PROOF Related Variables - - -PROOF debug options. - -``` {.cpp} -Proof.DebugLevel: 0 -Proof.DebugMask:-1 -``` - -PROOF GDB hooks allows a debugger to be attached early in the startup -phase of `proofserv: `0 - don't wait; 1 - master proofserv enters wait -loop; 2 - slave proofserv enters wait loop; 3 - any proofserv enters -wait loop - -``` {.cpp} -Proof.GdbHook: 0 -``` - -On the master to enable the parallel startup of workers using threads -set next to "`yes`" (default is "`no`"): - -``` {.cpp} -Proof.ParallelStartup: no -``` - -``` {.cpp} -Proof.StatsHist: no -Proof.StatsTrace: no -Proof.SlaveStatsTrace: no -``` - -``` {.cpp} -Proof.CondorHome: /opt/condor -Proof.CondorConfig: /opt/condor/etc/condor_config -``` - -``` {.cpp} -PEAC.GmUrl: http://somewhere:8080/clarens/ -PEAC.LmUrl: http://elsewhere:8080/clarens/ -``` - -#### Server Authentication in TServerSocket - -General setting: file with server access rules - -``` {.cpp} -SrvAuth.DaemonRc: /etc/root/system.daemonrc -``` - -Check of host equivalence via `/etc/hosts`.`equiv` or `$HOME/.rhosts`. - -``` {.cpp} -SrvAuth.CheckHostsEquivalence: 1 -``` - -Force file opening via **`TNetFile`** (**`TNetXNGFile`**) if a hostname is -specified in the Url. By default, for local files `TFile::Open()` -invokes directly **`TFile.`** - -``` {.cpp} -TFile.ForceRemote: yes -``` - -Special cases for the **`TUrl`** parser, where the special cases are -parsed in a protocol + file part, like rfio:host:/path/file.root, -castor:/path/file.root or /alien/path/file.root. In case the file -namespace descriptor ends with - the namespace is not a part of the -filename. Extend in private .rootrc with a +Url.Special line. - -``` {.cpp} -Url.Special: file: rfio: hpss: castor: dcache: -+Url.Special: /alien/- /castor/ -``` - -#### PROOF XRD Client Variables - -Debug level (if \<=0 : none, 1 : low, 2 : medium, 3 : high) - -``` {.cpp} -XProof.Debug: 0 -``` - -Socket read timeout [in secs: default 10 secs] - -``` {.cpp} -XProof.ReadTimeout: 10 -``` - -`XNet.PrintTAG` - Print a particular string the developers can choose to -quickly recognize the version at run time [default - 0] - -Example of custom setting for the Rint application (root.exe). This -overrides the default specified above for a generic application. Color 5 -is yellow. - -``` {.cpp} -Rint.Canvas.HighLightColor: 5 -``` - ## Documentation to Download diff --git a/documentation/users-guide/PROOF.md b/documentation/users-guide/PROOF.md deleted file mode 100644 index ae42c371bd45a..0000000000000 --- a/documentation/users-guide/PROOF.md +++ /dev/null @@ -1,49 +0,0 @@ -# PROOF: Parallel Processing -\index{PROOF} -\index{parallel processing} - -The Parallel ROOT Facility, PROOF, is an extension of ROOT allowing -transparent analysis of large sets of ROOT files in parallel on remote -computer clusters or multi-core computers. The main design goals for the -PROOF system are: - -*Transparency* : there should be as little difference as possible -between a local ROOT based analysis session and a remote parallel PROOF -session, both being interactive and giving the same results. - -*Scalability* : the basic architecture should not put any implicit -limitations on the number of computers that can be used in parallel. - -*Adaptability* : the system should be able to adapt itself to variations -in the remote environment (changing load on the cluster nodes, network -interruptions, etc.). - -Being an extension of the ROOT system, PROOF is designed to work on -objects in ROOT data stores, though, for the time being, it mainly -addresses the case of **`TTree`** based object collections. - -PROOF is primarily meant as an interactive alternative to batch systems -for Central Analysis Facilities and departmental workgroups (Tier-2's). -However, thanks to a multi-tier architecture allowing multiple levels of -masters, it can be easily adapted to wide range virtual clusters -distributed over geographically separated domains and heterogeneous -machines (GRIDs). - -While pure interactivity might not always be possible when performing a -complicated analysis on a very large data set, PROOF still tries to give -the user the interactive experience with something we call "interactive -batch". With "interactive batch" the user can start very long running -queries, disconnect the client and at any time, any location and from -any computer reconnect to the query to monitor its progress or retrieve -the results. This feature gives it a distinct advantage over purely -batch based solutions, that only provide an answer once all sub-jobs -have been finished. - -![The Multi-tier structure of a PROOF cluster](pictures/03000200.png) - -Details about the PROOF system and the way to use it can be found at - [^1] - -The PROOF development is a joint effort between CERN and MIT. - -[^1]: http://root.cern.ch/twiki/bin/view/ROOT/PROOF diff --git a/etc/plugins/TChain/P010_TProofChain.C b/etc/plugins/TChain/P010_TProofChain.C deleted file mode 100644 index 6bf9135656295..0000000000000 --- a/etc/plugins/TChain/P010_TProofChain.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TProofChain() -{ - gPluginMgr->AddHandler("TChain", "^proof", "TProofChain", - "Proof", "TProofChain(TChain *, Bool_t)"); -} diff --git a/etc/plugins/TDataSetManager/P010_TDataSetManagerFile.C b/etc/plugins/TDataSetManager/P010_TDataSetManagerFile.C deleted file mode 100644 index c88f16c90f8c7..0000000000000 --- a/etc/plugins/TDataSetManager/P010_TDataSetManagerFile.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TDataSetManagerFile() -{ - gPluginMgr->AddHandler("TDataSetManager", "^file", "TDataSetManagerFile", "Proof", - "TDataSetManagerFile(const char *,const char *,const char *)"); -} diff --git a/etc/plugins/TProof/P010_TProofCondor.C b/etc/plugins/TProof/P010_TProofCondor.C deleted file mode 100644 index f72534f4f1c5a..0000000000000 --- a/etc/plugins/TProof/P010_TProofCondor.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TProofCondor() -{ - gPluginMgr->AddHandler("TProof", "^condor:", "TProofCondor", - "Proof", "TProofCondor(const char*,const char*,const char*,Int_t,const char*)"); -} diff --git a/etc/plugins/TProof/P020_TProofSuperMaster.C b/etc/plugins/TProof/P020_TProofSuperMaster.C deleted file mode 100644 index aff61197fe6ff..0000000000000 --- a/etc/plugins/TProof/P020_TProofSuperMaster.C +++ /dev/null @@ -1,5 +0,0 @@ -void P020_TProofSuperMaster() -{ - gPluginMgr->AddHandler("TProof", "^sm:", "TProofSuperMaster", - "Proof", "TProofSuperMaster(const char*,const char*,const char*,Int_t,const char*)"); -} diff --git a/etc/plugins/TProof/P030_TProofLite.C b/etc/plugins/TProof/P030_TProofLite.C deleted file mode 100644 index fbf72aee21442..0000000000000 --- a/etc/plugins/TProof/P030_TProofLite.C +++ /dev/null @@ -1,5 +0,0 @@ -void P030_TProofLite() -{ - gPluginMgr->AddHandler("TProof", "^lite:", "TProofLite", - "Proof", "TProofLite(const char*,const char*,const char*,Int_t,const char*,TProofMgr *)"); -} diff --git a/etc/plugins/TProof/P040_TProof.C b/etc/plugins/TProof/P040_TProof.C deleted file mode 100644 index 8c0009612ad31..0000000000000 --- a/etc/plugins/TProof/P040_TProof.C +++ /dev/null @@ -1,5 +0,0 @@ -void P040_TProof() -{ - gPluginMgr->AddHandler("TProof", "*", "TProof", - "Proof", "TProof(const char*,const char*,const char*,Int_t,const char *)"); -} diff --git a/etc/plugins/TProofMgr/P010_TXProofMgr.C b/etc/plugins/TProofMgr/P010_TXProofMgr.C deleted file mode 100644 index 44b87de07d7ec..0000000000000 --- a/etc/plugins/TProofMgr/P010_TXProofMgr.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TXProofMgr() -{ - gPluginMgr->AddHandler("TProofMgr", "^xpd", "TXProofMgr", - "Proofx", "TXProofMgr(const char *,Int_t,const char *)"); -} diff --git a/etc/plugins/TProofMonSender/P020_TProofMonSenderSQL.C b/etc/plugins/TProofMonSender/P020_TProofMonSenderSQL.C deleted file mode 100644 index b86fd6afb4a0c..0000000000000 --- a/etc/plugins/TProofMonSender/P020_TProofMonSenderSQL.C +++ /dev/null @@ -1,5 +0,0 @@ -void P020_TProofMonSenderSQL() -{ - gPluginMgr->AddHandler("TProofMonSender", "SQL", "TProofMonSenderSQL", - "ProofPlayer", "TProofMonSenderSQL(const char *,const char *,const char *,const char *,const char *,const char *)"); -} diff --git a/etc/plugins/TProofProgressDialog/P010_TProofProgressDialog.C b/etc/plugins/TProofProgressDialog/P010_TProofProgressDialog.C deleted file mode 100644 index 42f4d43b7cdf9..0000000000000 --- a/etc/plugins/TProofProgressDialog/P010_TProofProgressDialog.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TProofProgressDialog() -{ - gPluginMgr->AddHandler("TProofProgressDialog", "*", "TProofProgressDialog", - "SessionViewer", "TProofProgressDialog(TProof*,const char*,Int_t,Long64_t,Long64_t)"); -} diff --git a/etc/plugins/TProofProgressLog/P010_TProofProgressLog.C b/etc/plugins/TProofProgressLog/P010_TProofProgressLog.C deleted file mode 100644 index dbe2a42124a2c..0000000000000 --- a/etc/plugins/TProofProgressLog/P010_TProofProgressLog.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TProofProgressLog() -{ - gPluginMgr->AddHandler("TProofProgressLog", "*", "TProofProgressLog", - "SessionViewer", "TProofProgressLog(const char*,Int_t,Long64_t,Long64_t)"); -} diff --git a/etc/plugins/TProofServ/P010_TXProofServ.C b/etc/plugins/TProofServ/P010_TXProofServ.C deleted file mode 100644 index a67149d47a99d..0000000000000 --- a/etc/plugins/TProofServ/P010_TXProofServ.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TXProofServ() -{ - gPluginMgr->AddHandler("TProofServ", "^xpd", "TXProofServ", - "Proofx", "TXProofServ(Int_t *,char **)"); -} diff --git a/etc/plugins/TSessionViewer/P010_TSessionViewer.C b/etc/plugins/TSessionViewer/P010_TSessionViewer.C deleted file mode 100644 index 72d74a48347f8..0000000000000 --- a/etc/plugins/TSessionViewer/P010_TSessionViewer.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TSessionViewer() -{ - gPluginMgr->AddHandler("TSessionViewer", "*", "TSessionViewer", - "TreeViewer", "TSessionViewer()"); -} diff --git a/etc/plugins/TSlave/P010_TXSlave.C b/etc/plugins/TSlave/P010_TXSlave.C deleted file mode 100644 index 20652a9d9d0e8..0000000000000 --- a/etc/plugins/TSlave/P010_TXSlave.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TXSlave() -{ - gPluginMgr->AddHandler("TSlave", "^xpd", "TXSlave", - "Proofx", "TXSlave(const char *,const char *,Int_t,const char *, TProof *,Int_t,const char *,const char *)"); -} diff --git a/etc/plugins/TVirtualProofPlayer/P010_TProofPlayer.C b/etc/plugins/TVirtualProofPlayer/P010_TProofPlayer.C deleted file mode 100644 index 7dae503ab4e6d..0000000000000 --- a/etc/plugins/TVirtualProofPlayer/P010_TProofPlayer.C +++ /dev/null @@ -1,5 +0,0 @@ -void P010_TProofPlayer() -{ - gPluginMgr->AddHandler("TVirtualProofPlayer", "base", "TProofPlayer", - "ProofPlayer", "TProofPlayer(TProof*)"); -} diff --git a/etc/plugins/TVirtualProofPlayer/P020_TProofPlayerRemote.C b/etc/plugins/TVirtualProofPlayer/P020_TProofPlayerRemote.C deleted file mode 100644 index 79cc72219a740..0000000000000 --- a/etc/plugins/TVirtualProofPlayer/P020_TProofPlayerRemote.C +++ /dev/null @@ -1,5 +0,0 @@ -void P020_TProofPlayerRemote() -{ - gPluginMgr->AddHandler("TVirtualProofPlayer", "remote", "TProofPlayerRemote", - "ProofPlayer", "TProofPlayerRemote(TProof*)"); -} diff --git a/etc/plugins/TVirtualProofPlayer/P030_TProofPlayerLocal.C b/etc/plugins/TVirtualProofPlayer/P030_TProofPlayerLocal.C deleted file mode 100644 index 39717726c91f0..0000000000000 --- a/etc/plugins/TVirtualProofPlayer/P030_TProofPlayerLocal.C +++ /dev/null @@ -1,5 +0,0 @@ -void P030_TProofPlayerLocal() -{ - gPluginMgr->AddHandler("TVirtualProofPlayer", "local", "TProofPlayerLocal", - "ProofPlayer", "TProofPlayerLocal(TProof*)"); -} diff --git a/etc/plugins/TVirtualProofPlayer/P040_TProofPlayerSlave.C b/etc/plugins/TVirtualProofPlayer/P040_TProofPlayerSlave.C deleted file mode 100644 index c7146b6e217c8..0000000000000 --- a/etc/plugins/TVirtualProofPlayer/P040_TProofPlayerSlave.C +++ /dev/null @@ -1,5 +0,0 @@ -void P040_TProofPlayerSlave() -{ - gPluginMgr->AddHandler("TVirtualProofPlayer", "slave", "TProofPlayerSlave", - "ProofPlayer", "TProofPlayerSlave(TSocket*)"); -} diff --git a/etc/plugins/TVirtualProofPlayer/P050_TProofPlayerSuperMaster.C b/etc/plugins/TVirtualProofPlayer/P050_TProofPlayerSuperMaster.C deleted file mode 100644 index bfaf571933d90..0000000000000 --- a/etc/plugins/TVirtualProofPlayer/P050_TProofPlayerSuperMaster.C +++ /dev/null @@ -1,5 +0,0 @@ -void P050_TProofPlayerSuperMaster() -{ - gPluginMgr->AddHandler("TVirtualProofPlayer", "sm", "TProofPlayerSuperMaster", - "ProofPlayer", "TProofPlayerSuperMaster(TProof*)"); -} diff --git a/etc/plugins/TVirtualProofPlayer/P060_TProofPlayerLite.C b/etc/plugins/TVirtualProofPlayer/P060_TProofPlayerLite.C deleted file mode 100644 index f4410d9c07f05..0000000000000 --- a/etc/plugins/TVirtualProofPlayer/P060_TProofPlayerLite.C +++ /dev/null @@ -1,5 +0,0 @@ -void P060_TProofPlayerLite() -{ - gPluginMgr->AddHandler("TVirtualProofPlayer", "lite", "TProofPlayerLite", - "ProofPlayer", "TProofPlayerLite(TProof*)"); -} diff --git a/etc/proof/cluster.conf.sample b/etc/proof/cluster.conf.sample deleted file mode 100644 index 0012dd6946d52..0000000000000 --- a/etc/proof/cluster.conf.sample +++ /dev/null @@ -1,4 +0,0 @@ -pcna49a -pcna49b -pcna49c -pcna49d diff --git a/etc/proof/motd.sample b/etc/proof/motd.sample deleted file mode 100644 index 837136d51c571..0000000000000 --- a/etc/proof/motd.sample +++ /dev/null @@ -1,6 +0,0 @@ -*** Welcome to the ALICE PROOF cluster. -*** -*** Disk space is low, please clean up. -*** Tomorrow no service, due to maintenance. -*** -*** -- the Fonz diff --git a/etc/proof/noproof.sample b/etc/proof/noproof.sample deleted file mode 100644 index 3dcf640b24bb7..0000000000000 --- a/etc/proof/noproof.sample +++ /dev/null @@ -1,3 +0,0 @@ -+++ Sorry no PROOF service today +++ - --- The Fonz diff --git a/etc/proof/proof.conf.sample b/etc/proof/proof.conf.sample deleted file mode 100644 index db281f3736d69..0000000000000 --- a/etc/proof/proof.conf.sample +++ /dev/null @@ -1,57 +0,0 @@ -# PROOF config file. It has a very simple format: -# -# master [image=] [workdir=] -# worker [perf=] [image=] [port=] -# [workdir=] -# submaster [image=] [port=] -# [config=] -# condorworker [image=] [perf=] -# [workdir=] -# user on -# -# For each node in the cluster that can run a master add a "master" line. -# For each node in the cluster that can run a worker add a "worker" line. -# A single node can run multiple workers (e.g. an SMP machine). In that case -# repeat the worker line N times, where N is the number of CPU's in the machine. -# Perfindex is a number specifying the relative speed of each CPU in the -# cluster. Say, the fastest CPU is 100, a CPU 30% slower has perf=70. The -# perfindex is used for load balancing. By default perf=100. The perfindex -# is mostly an indication since the parallel algorithm is adaptive and will -# adjust itself to the current load on each machine. -# Image is used to tell the system that system images are shared between -# nodes/CPU's in the cluster. For example if all nodes share a common user -# directory, like with AFS, set image=afs ( is arbitrary but should -# be unique for each different image). By default imagename is the hostname. -# Port is used to tell PROOF to connect to the workers on the specified -# port (in case proofd was started by the user instead of via inetd). -# -# For condorworkers the special image IGNORE can be used to exclude -# a worker from the set of active workers. The worker (Condor VM) will -# be suspended and resumed. This supports situations where there a more -# VMs then CPUs. -# -# Users can be redirected to have their master server run on a pre-determined -# node. For that use the "user" line. -# -# For backward compatibility it is still possible to specify the methods -# used for worker authentication as a list following the 'worker ' -# (e.g. worker usrpwd), this is however deprecated. -# Authentication directives are specified via $ROOTSYS/etc/system.rootauthrc -# and/or $HOME/.rootauthrc files. See $ROOTSYS/etc/system.rootauthrc for details. -# - -#master pcna49a image=nfs - -#worker pcna49a perf=100 image=nfs -#worker pcna49b perf=100 image=nfs -#worker pcna49c perf=100 image=nfs -#worker pcna49d perf=100 image=nfs -#worker pcna49e perf=75 image=nfs - -#user rdm on pcna49d - - -master localhost - -worker localhost -worker localhost diff --git a/etc/proof/proofbench/README b/etc/proof/proofbench/README deleted file mode 100644 index 7473bcb663eb1..0000000000000 --- a/etc/proof/proofbench/README +++ /dev/null @@ -1,3 +0,0 @@ - - After building this directory contains the default PAR files needed to run the PROOF-Bench suite - (see $ROOTSYS/proof/proofbench). diff --git a/etc/proof/rootnetrc.sample b/etc/proof/rootnetrc.sample deleted file mode 100644 index b8cf89db0439d..0000000000000 --- a/etc/proof/rootnetrc.sample +++ /dev/null @@ -1,9 +0,0 @@ -# Sample ~/.rootnetrc file. This file must be available on all nodes -# in the PROOF cluster so that the PROOF servers can open files via TNetFile -# without having to prompt for passwords. The file must be mode 0600 -# otherwise it will be considered compromised. If you are security conscious -# you install this file only via a secure ssh link. - -machine node1.cern.ch login rdm password secretpasswd -machine node2.cern.ch login rdm password secretpasswd -machine fileserv1.cern.ch login rdm password filesrvpasswd diff --git a/etc/proof/utils/circle.sh b/etc/proof/utils/circle.sh deleted file mode 100755 index 5d8ce690be9b9..0000000000000 --- a/etc/proof/utils/circle.sh +++ /dev/null @@ -1,241 +0,0 @@ -#!/bin/bash -# -# Circle files and commands over a cluster using scp and ssh. -# -# Author: Fons Rademakers - -# On which platform do we run -ostype=`uname -s` - -cwhich="type -path" -lsopt="--color=never" -ptimeout="-W" -if [ "$ostype" = "Darwin" ]; then - lsopt="" - ptimeout="-t" -fi - -# Where to store values from a previous run -saverc="$HOME/.circlerc" - -#--- stored in $saverc -nodes= -base= -domain= -lower=0 -upper=0 -width=1 -ping=1 -puser= -sshopt="-o StrictHostKeyChecking=no -o PasswordAuthentication=no" -#--- stored in $saverc - -prog=`basename $0` -progdir=`dirname $0` -runningdir=`pwd` -if echo $progdir | grep -q -s ^/ || echo $progdir | grep -q -s ^~ ; then - # absolute path - fullpath=$progdir -else - # relative path - fullpath=$runningdir/$progdir -fi - -# Set default arguments values -oper= -user= -nodelist= -ignore="no" -cluster= -ssho= -sshcmd= -scp1= -scp2= -pkey="$HOME/.ssh/id_dsa.pub" - -# Process command line arguments -while [ $# -gt 0 ]; do - case "$1" in - -i|--ignore) - ignore="yes" - shift - ;; - -u|--user) - shift - user="$1" - shift - ;; - -c|--cluster) - shift - cluster="$1" - saverc="${saverc}-${cluster}" - shift - ;; - -s|-sshopt) - shift - ssho="$1" - shift - ;; - -n|--nodes) - shift - nodelist="$1" - shift - ;; - -h|--help) - echo "Usage: $prog ssh|scp|key [-u user] [-n \"nodes\"] [-c cluster] [-s \"sshopt\"] [-i] [-h] [ARGS...] " - echo "Circle files and commands over a cluster using scp and ssh." - echo "" - echo " -u, --user specify the user on cluster" - echo " -n, --nodes nodes on which to run or copy" - echo " -c, --cluster cluster name, used to identify config file $saverc" - echo " -s, --sshopt ssh options string" - echo " -i, --ignore ignore an existing $saverc file" - echo " -h, --help display this help and exit" - echo "" - echo " ssh use ssh to execute commands on all cluster nades," - echo " commands are specified as a single string in \"ARGS\"." - echo " scp use scp to copy files on all cluster nodes," - echo " the \"what\" to \"where\" are specified as two ARGS." - echo " key copy the users $HOME/.ssh/id_dsa.pub to all cluster" - echo " nodes, no ARGS." - echo "ARGS, depending on the command to execute (ssh, scp, key)." - exit 0 - ;; - -*) - echo "$prog: invalid option -- $1" - echo "Try \`$prog --help' for more information." - exit 1 - ;; - ssh) - oper="$1" - shift - ;; - scp) - oper="$1" - shift - ;; - key) - oper="$1" - if [ ! -r $pkey ]; then - echo "$prog: no public key $pkey to copy around" - exit 1 - fi - shift - ;; - *) - if [ "x$oper" = "xssh" ] && [ "x$sshcmd" = "x" ]; then - sshcmd="$1" - shift - elif [ "x$oper" = "xscp" ] && [ "x$scp1" = "x"] && [ "x$scp2" = "x" ]; then - scp1="$1" - shift - scp2="$1" - shift - else - echo "$prog: unexpected argument -- $1" - echo "Try \`$prog --help' for more information." - exit 1 - fi - ;; - esac -done - -if [ -r $saverc ] && [ "$ignore" = "no" ]; then - source $saverc -fi - -if [ "x$nodelist" != "x" ]; then - nodes="$nodelist" -fi - -if [ "x$user" != "x" ]; then - puser="$user" -fi - -if [ "x$ssho" != "x" ]; then - sshopt="$ssho" -fi - -# Store settings in $saverc -echo "nodes=\"$nodes\"" > $saverc -echo "base=\"$base\"" >> $saverc -echo "domain=\"$domain\"" >> $saverc -echo "lower=\"$lower\"" >> $saverc -echo "upper=\"$upper\"" >> $saverc -echo "width=\"$width\"" >> $saverc -echo "ping=\"$ping\"" >> $saverc -echo "puser=\"$puser\"" >> $saverc -echo "sshopt=\"$sshopt\"" >> $saverc - - -nodelist() { - # This function will create a list of node names using a base - # node name [$1] and a starting [$2] and ending [$3] sequence number. - # The zero padded width is specified in [$4]. - # To exclude nodes that don't report to ping set [$5] to 1. - # The result will be stored in nodes variable. - - if test $# -lt 6; then - echo "$prog nodes: Too few arguments" - return 1 - fi - - # Save arguments in local names - nbase=$1; shift - ndomain=$1; shift - nfirst=$1; shift - nlast=$1; shift - nwidth=$1; shift - nping=$1; shift - - form="%0${nwidth}d" - - i=$nfirst - nodes= - while test $i -le $nlast; do - ii=`printf $form $i` - if [ "$ndomain" = "-" ]; then - node=$nbase$ii - else - node=$nbase$ii.$ndomain - fi - if [ "x$nping" = "x1" ]; then - p=`ping -c 1 $ptimeout 1 $node > /dev/null 2>&1` - if [ $? -ne 0 ]; then - node= - fi - fi - if [ "x$node" != "x" ]; then - if [ "x$nodes" != "x" ]; then - nodes="$nodes " - fi - nodes=$nodes$node - fi - i=$((i+1)) - done -} - -if [ "x$nodes" = "x" ]; then - nodelist $base $domain $lower $upper $width $ping && exit 1 -fi - -case $oper in -ssh) - for i in $nodes; do - echo "ssh $sshopt $puser@$i $sshcmd" - ssh $sshopt $puser@$i "$sshcmd" - done - ;; -scp) - for i in $nodes; do - echo "scp $sshopt $scp1 $puser@$i:$scp2" - scp $sshopt $scp1 $puser@$i:$scp2 - done - ;; -key) - for i in $nodes; do - echo "add $pkey to authorized_keys on $i" - cat $pkey | ssh $sshopt $puser@$i "cat - >> ~/.ssh/authorized_keys" - done - ;; -esac diff --git a/etc/proof/utils/crypt b/etc/proof/utils/crypt deleted file mode 100755 index b78cab1bae78f..0000000000000 --- a/etc/proof/utils/crypt +++ /dev/null @@ -1,14 +0,0 @@ -#! /usr/bin/perl - -# -# Generate a passwd which can be used for the UsrPwd authentication method. -# Store the passwd in ~/.rootdpass. -# - -srand(time()); -my $randletter = "(int (rand(26)) + (int (rand(1) + .5) % 2 ? 65 : 97))"; -my $salt = sprintf("%c%c", eval $randletter, eval $randletter); -my $plaintext = shift; -my $crypttext = crypt($plaintext, $salt); - -print "${crypttext}\n"; diff --git a/etc/proof/utils/makepbenchpars.sh b/etc/proof/utils/makepbenchpars.sh deleted file mode 100755 index 750d34bc4af60..0000000000000 --- a/etc/proof/utils/makepbenchpars.sh +++ /dev/null @@ -1,88 +0,0 @@ -#!/bin/sh - -# This scrypt generates the PAR files required by ProofBench. It takes three arguments, all mandatory: -# -# ./makepbenchpars.sh {ProofBenchDataSel,ProofBenchCPUSel} -# -# The scrypt is invoked by the CMake-based build system (see proof/proofbench/CMakeLists.txt) - - -PARNAME=$1 -if test "x$PARNAME" = "x" ; then - echo "Entering a PAR name is mandatory" - exit 1 -fi -if test ! "x$PARNAME" = "xProofBenchDataSel" && test ! "x$PARNAME" = "xProofBenchCPUSel" ; then - echo "PAR name must be either xProofBenchDataSel or xProofBenchCPUSel" - exit 1 -fi -ROOT_SRCDIR=$2 -if test "x$ROOT_SRCDIR" = "x" ; then - echo "Entering the ROOT source dir is mandatory" - exit 1 -fi -ROOT_BUILDDIR=$3 -if test "x$ROOT_BUILDDIR" = "x" ; then - echo "Entering the ROOT build dir is mandatory" - exit 1 -fi - - -MODDIR=$ROOT_SRCDIR/proof/proofbench -MODDIRS=$MODDIR/src -MODDIRI=$MODDIR/inc -PBPARDIR=$ROOT_BUILDDIR/etc/proof/proofbench -PARBLDDIR=$ROOT_BUILDDIR/proof/proofbench - -##### ProofBenchDataSel PAR file ##### -if test "x$PARNAME" = "xProofBenchDataSel"; then - PARDIR=$PARBLDDIR/ProofBenchDataSel - PARH="$ROOT_SRCDIR/test/Event.h $MODDIRI/TProofBenchTypes.h \ - $MODDIRI/TSelEventGen.h $MODDIRI/TSelEvent.h $MODDIRI/TSelHandleDataSet.h" - PARS="$ROOT_SRCDIR/test/Event.cxx \ - $MODDIRS/TSelEventGen.cxx $MODDIRS/TSelEvent.cxx $MODDIRS/TSelHandleDataSet.cxx" - PARF=$PBPARDIR/ProofBenchDataSel.par - -##### ProofBenchCPUSel PAR file ##### -elif test "x$PARNAME" = "xProofBenchCPUSel"; then - - PARDIR=$PARBLDDIR/ProofBenchCPUSel - PARH="$MODDIRI/TProofBenchTypes.h $MODDIRI/TSelHist.h" - PARS=$MODDIRS/TSelHist.cxx - PARF=$PBPARDIR/ProofBenchCPUSel.par -fi -PARINF=$PARDIR/PROOF-INF - -# The PAR top directory ... -if test -d $PARDIR; then - rm -fr $PARDIR -fi -# ... its PROOF-INF -mkdir -p $PARINF -for f in $PARH $PARS; do - cp -rp $f $PARDIR -done -# Its SETUP.C ... -echo "#include \"TClass.h\"" > $PARINF/SETUP.C -echo "#include \"TROOT.h\"" >> $PARINF/SETUP.C -echo "Int_t SETUP() {" >> $PARINF/SETUP.C -echo " if (!TClass::GetClass(\"TPBReadType\")) {" >> $PARINF/SETUP.C -echo " gROOT->ProcessLine(\".L TProofBenchTypes.h+\");" >> $PARINF/SETUP.C -echo " }" >> $PARINF/SETUP.C -for f in $PARS; do - b=`basename $f`; - echo " gROOT->ProcessLine(\".L $b+\");" >> $PARINF/SETUP.C -done -echo " return 0;" >> $PARINF/SETUP.C -echo "}" >> $PARINF/SETUP.C - -builddir=`pwd` -cd $PARBLDDIR -par=`basename $PARF` -pard=`basename $PARDIR` -tar cf - $pard | gzip > $par || exit 1 -mv $par $PBPARDIR || exit 1; -cd $builddir; -rm -fr $PARDIR -exit 0 - diff --git a/etc/proof/utils/pcmd b/etc/proof/utils/pcmd deleted file mode 100755 index 6dd9c9d4a7539..0000000000000 --- a/etc/proof/utils/pcmd +++ /dev/null @@ -1,22 +0,0 @@ -#! /bin/sh -# -# Usage: pcmd -# -# Execute a command on all machines listed in the $MACHINES file. -# -# Author: rdm -# - -MACHINES=/usr/local/root/proof/etc/cluster.conf - -cmd=$* - -if [ -f $MACHINES ] -then - machines=`cat $MACHINES` - for i in $machines - do - echo "$i: $cmd" - rsh $i $cmd - done -fi diff --git a/etc/proof/utils/pload b/etc/proof/utils/pload deleted file mode 100755 index 2ddf74a50c06f..0000000000000 --- a/etc/proof/utils/pload +++ /dev/null @@ -1,31 +0,0 @@ -#! /bin/sh -# -# Usage: pload -# -# Display load statistics of all proof machines (using xload). -# -# Author: rdm -# - -if [ $# = 0 ] -then - display=$DISPLAY -else - display=$1 -fi - -MACHINES=/usr/local/root/proof/etc/cluster.conf -if test -f $MACHINES && test "$display" -then - machines=`cat $MACHINES` - typeset -i offset=772 - - for i in $machines - do - rsh $i /usr/bin/X11/xload -bg blue -fg yellow -disp $display \ - -geometry 64x64-0+$offset & - offset=offset-77 - done - - unset offset -fi diff --git a/etc/proof/utils/pps b/etc/proof/utils/pps deleted file mode 100755 index 32c8227355f58..0000000000000 --- a/etc/proof/utils/pps +++ /dev/null @@ -1,48 +0,0 @@ -#! /bin/bash -# -# Usage: pps [user] -# -# Show the root and rootserv processes running on all machines. -# -# Author: rdm -# -AWK=/usr/bin/awk -CAT=/bin/cat -REMSH=/usr/bin/rsh -PS=/bin/ps - -USAGE='Usage: pps [user]' -MACHINES=/usr/local/root/proof/etc/cluster.conf - -case $# in - 2|3|4|5|6) echo $USAGE >&2; exit 1 -esac - -if [ -n "$1" ] -then - user=$1 -else - user=p -fi - -if [ -f $MACHINES ] -then - machines=`$CAT $MACHINES` - for i in $machines - do - echo "========== $i ==========" - $REMSH $i $PS -uaxw | $AWK 'BEGIN { ms = ss = 0 } - {if (/'$user'/ && /proofs/ && !/awk/) - if (substr($9,1,1) ~ /[0-9]/) { - printf("%-10s %-7s %s %6s %s %s\n", $1,$2,$9,$10,$11,$12) - if ($12 == "proofserv") ms++; - if ($12 == "proofslave") ss++; - } else { - printf("%-10s %-7s %s %-6s %6s %s %s\n",$1,$2,$9,$10,$11,$12,$13) - if ($13 == "proofserv") ms++; - if ($13 == "proofslave") ss++;} - } - END { printf("\n%d Master, %d Slave servers\n", ms, ss) }' - echo ' ' - done -fi diff --git a/etc/proof/utils/proof-facility b/etc/proof/utils/proof-facility deleted file mode 100755 index cfee37920e4dc..0000000000000 --- a/etc/proof/utils/proof-facility +++ /dev/null @@ -1,81 +0,0 @@ -#!/bin/sh - -if [ $# -lt 1 ] ; then - echo "I need some options" - echo "$0 [option]" - echo " -d This is the master node, so we need a file name" - echo " E.g. /usr/share/root/proof. This option will" - echo " create the directory /log, and/or" - echo " the file /log/proof.log if they " - echo " don't exist" - echo " -m This is a slave node, se we need the master" - echo " hostname. E.g. pmaster.cern.ch" - echo " -c Clean entries in /etc/syslog.conf" - echo " " -fi - -case $1 in - "-d") - if [ "x$2" = "x" ] ; then - echo "I need the PROOF config directory" - exit 1 - fi - file="$2/log" - echo "Master node configuration - logfile: $file/proof.log" - # - # Create log directory, if it doesn't exist already - # - if [ ! -d $file ] ; then mkdir -p $file ; fi - # - # Create empty log file if it doen't exist already - # - file="$file/proof.log" - if [ ! -f $file ] ; then echo "" > $file ; fi - ;; - "-m") - if [ "x$2" = "x" ] ; then - echo "I need the PROOF master hostname" - exit 1 - fi - file="@$2" - echo "Slave node configuration - master: $file" - ;; - "-c") - echo "Cleaning configuration" - clean="yes" - ;; - "*") - echo "I need one of the options -d, -m, -c" - ;; -esac - -if [ "x$clean" = "xyes" ] ; then - syslog-facility remove local5 - syslog-facility remove local6 - exit 0 -fi - -while [ true ] ; do - localnum=`syslog-facility set 'debug' $file 'none' /var/log/messages` - if [ "x$localnum" = "xlocal5" ] ; then - localnum=`syslog-facility set 'debug' $file 'none' /var/log/messages` - if [ "x$localnum" != "xlocal6" ] ; then - echo "Couldn't get local6 facility, trying to revert" - syslog-facility remove local5 - if [ "x$localnum" != "none" ] ; then - syslog-facility remove $localnum - fi - fi - break - elif [ "x$localnum" = "xnone" ] ; then - echo "Seems like all local facilities are tied up at the moment" - echo "You need to manually edit the /etc/syslog.conf file" - break - fi - numbers="$numbers $localnum" -done - -for num in $numbers ; do - syslog-facility remove $num -done - diff --git a/etc/proof/utils/proofctl.sh b/etc/proof/utils/proofctl.sh deleted file mode 100755 index c1a3a229ec2af..0000000000000 --- a/etc/proof/utils/proofctl.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash -# -# PROOF control script. -# This script controls the daemons on a RROOF cluster. -# -# Author: Fons Rademakers. - -# Commands to support: -# --start -# --stop -# --restart -# --status -# --version diff --git a/etc/proof/utils/proofinstall.sh b/etc/proof/utils/proofinstall.sh deleted file mode 100755 index f7fd7e6b7abdd..0000000000000 --- a/etc/proof/utils/proofinstall.sh +++ /dev/null @@ -1,851 +0,0 @@ -#!/bin/bash -# -# PROOF setup script. -# This script distributes the ROOT binary on a cluster. -# Unpacks the binary, and start the necessary xrootd daemon. -# -# Author: Fons Rademakers - -# On which platform do we run -ostype=`uname -s` - -cwhich="type -path" -lsopt="--color=never" -ptimeout="-W" -if [ "$ostype" = "Darwin" ]; then - lsopt="" - ptimeout="-t" -fi - -# Where to store values from a previous run -saverc="$HOME/.proofinstallrc" - -#--- stored in $saverc -nodes= -domain=`hostname | awk -F '.' '{ print $2 "." $3 }'` -base= -lower=0 -upper=0 -width=1 -rdomain="$domain" -rootbinary= -remoterootbinary= -autoremoterootbinary= -puser="$LOGNAME" -pmaster= -pport=1093 -pdatadir="proof/data" -pworkers=0 -pconfig= -#--- stored in $saverc - -prompt_rootrepository="ftp://root.cern/root/" - -prog=`basename $0` -progdir=`dirname $0` -runningdir=`pwd` -if echo $progdir | grep "^/" > /dev/null 2>& 1 || \ - echo $progdir | grep "^~" > /dev/null 2>& 1; then - # absolute path - fullpath=$progdir -else - # relative path - fullpath=$runningdir/$progdir -fi - -# Helper programs -circle=$fullpath/circle.sh - -# Set default argument values -batch="no" -ignore="no" -cluster= - -# Process command line arguments -while [ $# -gt 0 ]; do - case "$1" in - -b|--batch) - batch="yes" - shift - ;; - -i|--ignore) - ignore="yes" - shift - ;; - -h|--help) - echo "Usage: $prog [-b] [-i] [-h] [CLUSTER]" - echo "Install and start PROOF on a cluster." - echo "" - echo " -b, --batch run in batch mode, requires a" - echo " $saverc file, created" - echo " by an interactive run" - echo " mutual exclusive with the --ignore option" - echo " -i, --ignore ignore an existing $saverc file" - echo " mutual exclusive with the --batch option" - echo " -h, --help display this help and exit" - echo "" - echo "CLUSTER, store and use the config for a specific cluster." - exit 0 - ;; - -*) - echo "$prog: invalid option -- $1" - echo "Try \`$prog --help' for more information." - exit 1 - ;; - *) - cluster="$1" - saverc="${saverc}-${cluster}" - shift - ;; - esac -done - -if [ "$ignore" = "yes" ] && [ "$batch" = "yes" ]; then - echo "$prog: the options --ignore and --batch are mutual exclusive." - exit 1 -fi - -if [ ! -r "$saverc" ] && [ "$batch" = "yes" ]; then - echo "$prog: need a $saverc file with" - echo "parameters to be able to run in batch mode." - echo "First run interactively to create one, like" - echo "\"$prog $cluster\"." - exit 1 -fi - -if [ -r "$saverc" ] && [ "$ignore" = "no" ]; then - source $saverc -fi - - -# Setup ssh-agent and load key using ssh-add subroutine. -### To be added. - -if [ "$batch" = "no" ]; then - -# Print welcome and instructions. -clear -echo "" -echo " Welcome to the PROOF Installer" -echo "" -echo "To install PROOF on a cluster the following is needed:" -echo "" -echo " - list of host names" -echo " - ssh login capability on these machines" -echo " - a binary ROOT distribution compatible with the cluster," -echo " or a download location (used by curl or wget)" -echo "" -echo "If any of the above is not true, please interrupt the installer" -echo "and re-run later." -echo "" -echo "If more than one answer is prompted, the one in upper case is the default." -echo "" -echo -n "Are you ready to continue [Y/n]: " -read answ &> /dev/null -if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting the install..." - exit 0 -fi - -# Ask user for list of nodes to be used. -REPEAT_LOOP=1 -while [ "$REPEAT_LOOP" = 1 ]; do - clear - echo "" - echo "Give list of host names." - echo "" - echo "Please make your selection by entering an option:" - echo "" - echo " 1. Individual host names." - echo " 2. A range of host names." - echo " h. Help." - echo " x. Exit." - echo "" - echo -n "Please type a selection: " - read answ &> /dev/null - case $answ in - 1) - clear - echo "" - echo -n "Give list of host names [$nodes]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - nodes="$answ" - fi - if [ "x$nodes" = "x" ]; then - echo -n "No hosts specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - - echo "" - echo -n "Give domain name, if not already specified (- = no domain) [$domain]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - domain=`echo $answ | sed "s/^\.//"` - fi - if [ "$domain" != "-" ]; then - n= - for i in $nodes; do - if [ "x$n" != "x" ]; then - n="$n " - fi - n="$n$i.$domain" - done - nodes="$n" - fi - - echo "" - echo "The list of hosts is:" - echo "" - echo $nodes - echo "" - echo -n "Is this correct [Y/n]: " - read answ &> /dev/null - if [ "x$answ" = "x" ] || [ "$answ" = "Y" ] || [ "$answ" = "y" ]; then - REPEAT_LOOP=0 - fi - ;; - 2) - nodes= - clear - echo "" - echo -n "Give host base name [$base]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - base="$answ" - fi - if [ "x$base" = "x" ]; then - echo -n "No base specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - - echo "" - echo -n "Give lower bound [$lower]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - lower="$answ" - fi - - echo "" - echo -n "Give upper bound [$upper]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - upper="$answ" - fi - if [ $lower -gt $upper ]; then - echo -n "lower ($lower) > upper ($upper), try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - - echo "" - echo -n "Give width [$width]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - width="$answ" - fi - - echo "" - echo -n "Give domain name, if not already specified (- = no domain) [$rdomain]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - rdomain=`echo $answ | sed "s/^\.//"` - fi - - form="%0${width}d" - - i=$lower - while [ $i -le $upper ]; do - ii=`printf $form $i` - n="$base$ii" - if [ "$rdomain" != "-" ]; then - n="$n.$rdomain" - fi - if [ "x$nodes" != "x" ]; then - nodes="$nodes " - fi - nodes="$nodes$n" - i=$((i+1)) - done - - echo "" - echo "The list of hosts is:" - echo "" - echo $nodes - echo "" - echo -n "Is this correct [Y/n]: " - read answ &> /dev/null - if [ "x$answ" = "x" ] || [ "$answ" = "Y" ] || [ "$answ" = "y" ]; then - REPEAT_LOOP=0 - fi - ;; - h) - echo "---------------------------------------------------------------------------" - echo "Node selection options:" - echo "" - echo "-- Option 1 --" - echo "Give a list of space separated host names. If they are in the same domain" - echo "don't specify the domains. You will be asked later for the domain." - echo "Example: thisnode1 thatnode2 ..." - echo "" - echo "-- Option 2 --" - echo "Give the host base name, the begin and end of the range and the width." - echo "Example:" - echo " node- 1 20 3" - echo "Defines:" - echo " node-001 node-002 node-003 ... node-020" - echo "" - echo "-- Option h --" - echo "Selecting option h displays this help message." - echo " " - echo "-- Option x --" - echo "Selecting option x exits the PROOF installation." - echo "---------------------------------------------------------------------------" - echo -n "Press to continue..." - read answ &> /dev/null - ;; - x) - echo "Exiting the install..." - exit 0 - ;; - *) - echo "Invalid Choice. Please try again." - REPEAT_LOOP=1 - ;; - esac - - if [ "$REPEAT_LOOP" = 0 ]; then - # Ask if we can ping the nodes - echo "" - echo -n "Check if hosts are alive using ping [Y/n]: " - read answ &> /dev/null - if [ "x$answ" = "x" ] || [ "$answ" = "Y" ] || [ "$answ" = "y" ]; then - n= - for i in $nodes; do - ping -c 1 $ptimeout 1 $i > /dev/null 2>&1 - if [ $? -eq 0 ]; then - if [ "x$n" != "x" ]; then - n="$n " - fi - n=$n$i - else - echo "Host $i not alive" - fi - done - nodes=$n - fi - - if [ "x$nodes" = "x" ]; then - echo -n "No hosts alive, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - REPEAT_LOOP=1 - continue - fi - - echo "" - echo "The list of available nodes is:" - echo "" - echo $nodes - echo "" - echo -n "Is this correct [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - REPEAT_LOOP=1 - fi - fi -done - -# Ask user which ROOT binary distribution to use. -# Either a local file, a remote file or a remote repository can be specified. -REPEAT_LOOP=1 -while [ "$REPEAT_LOOP" = 1 ]; do - bintype=0 - clear - echo "" - echo "Give ROOT binary distribution to be installed." - echo "" - echo "Please make your selection by entering an option:" - echo "" - echo " 1. Local ROOT binary distribution file." - echo " 2. Remote ROOT binary distribution file." - echo " 3. Remote ROOT binary distribution automatically" - echo " matching cluster node architecture." - echo " h. Help." - echo " x. Exit." - echo "" - echo -n "Please type a selection: " - read answ &> /dev/null - case $answ in - 1) - clear - echo "" - echo -n "Give local ROOT binary distribution [$rootbinary]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - rootbinary=`eval echo $answ` - fi - if [ "x$rootbinary" = "x" ]; then - echo -n "No local ROOT binary distribution file specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - REPEAT_LOOP=0 - bintype=1 - ;; - 2) - clear - echo "" - echo -n "Give remote ROOT binary distribution [$remoterootbinary]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - remoterootbinary=`eval echo $answ` - fi - if [ "x$remoterootbinary" = "x" ]; then - echo -n "No remote ROOT binary distribution file specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - REPEAT_LOOP=0 - bintype=2 - ;; - 3) - clear - echo "" - echo -n "Give URL to ROOT binary repository [$prompt_rootrepository]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - autoremoterootbinary="$answ" - else - autoremoterootbinary="$prompt_rootrepository" - fi - if [ "x$autoremoterootbinary" = "x" ]; then - echo -n "No URL to ROOT binary repository specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - REPEAT_LOOP=0 - bintype=3 - ;; - h) - echo "---------------------------------------------------------------------------" - echo "ROOT binary distribution selection options:" - echo "" - echo "-- Option 1 --" - echo "Give path to local ROOT binary distribution tar file." - echo "This only makes sense if all the cluster nodes have the same architecture." - echo "Example: ~/root_v5.17.04.Linux.slc4_amd64.gcc3.4.tar.gz" - echo "" - echo "-- Option 2 --" - echo "Give URL to remote ROOT binary distribution tar file." - echo "This only makes sense if all the cluster nodes have the same architecture." - echo "Requires curl or wget on this machine." - echo "Example: ftp://root.cern/root/root_v5.17.04.Linux.slc4_amd64.gcc3.4.tar.gz" - echo "" - echo "-- Option 3 --" - echo "Give URL to remote ROOT binary distribution repository." - echo "The remote nodes will try to download the binary version matching" - echo "their architecture. Requires curl or wget on the remote nodes." - echo "Example: ftp://root.cern/root/" - echo "" - echo "-- Option h --" - echo "Selecting option h displays this help message." - echo " " - echo "-- Option x --" - echo "Selecting option x exits the PROOF installation." - echo "---------------------------------------------------------------------------" - echo -n "Press to continue..." - read answ &> /dev/null - ;; - x) - echo "Exiting the install..." - exit 0 - ;; - *) - echo "Invalid Choice. Please try again." - REPEAT_LOOP=1 - ;; - esac - - if [ "$REPEAT_LOOP" = 0 ]; then - echo "" - case $bintype in - 1) - echo "The local ROOT binary distribution to be used is:" - echo "" - echo $rootbinary - ;; - 2) - echo "The remote ROOT binary distribution to be used is:" - echo "" - echo $remoterootbinary - ;; - 3) - echo "The remote ROOT binary distribution repository to be used is:" - echo "" - echo $autoremoterootbinary - ;; - esac - echo "" - echo -n "Is this correct [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - REPEAT_LOOP=1 - continue - fi - - case $bintype in - 1) - if [ ! -r "$rootbinary" ]; then - echo "" - echo "File $rootbinary not found." - echo -n "Try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - REPEAT_LOOP=1 - fi - ;; - 2) - if `$cwhich curl > /dev/null 2>&1`; then - echo "" - echo "Downloading $remoterootbinary..." - echo "" - rootbinary=`basename $remoterootbinary`.$$ - curl -o $rootbinary $remoterootbinary - if [ "$?" -ne 0 ]; then - echo "" - echo "Remote file $remoterootbinary not found." - echo -n "Try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - REPEAT_LOOP=1 - fi - elif `$cwhich wget > /dev/null 2>&1`; then - echo "" - echo "Downloading $remoterootbinary..." - echo "" - rootbinary=`basename $remoterootbinary`.$$ - wget -O $rootbinary $remoterootbinary - if [ "$?" -ne 0 ]; then - echo "" - echo "Remote file $remoterootbinary not found." - echo -n "Try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - REPEAT_LOOP=1 - fi - else - echo "" - echo "Neither curl nor wget available. Try getting the ROOT binary" - echo "distribution file in some other way." - echo "" - echo "Exiting! Press to terminate install." - read - exit 0 - fi - esac - fi -done - -# Ask the user PROOF specific configuration parameters. -REPEAT_LOOP=1 -while [ "$REPEAT_LOOP" = 1 ]; do - conftype=0 - clear - echo "" - echo "Give PROOF specific configuration parameters." - echo "" - echo "Please make your selection by entering an option:" - echo "" - echo " 1. Basic PROOF configuration parameters." - echo " 2. Advanced PROOF configuration parameters." - echo " h. Help." - echo " x. Exit." - echo "" - echo -n "Please type a selection: " - read answ &> /dev/null - case $answ in - 1) - clear - echo "" - echo "Give basic PROOF specific configuration parameters." - echo "" - echo -n "Give name of user under which to run PROOF [$puser]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - puser="$answ" - fi - if [ "x$puser" = "x" ]; then - echo -n "No user specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - - echo "" - nodearr=($nodes) - pmaster=${nodearr[0]} - echo -n "Give host on which to run the PROOF master [$pmaster]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - pmaster="$answ" - fi - if [ "x$pmaster" = "x" ]; then - echo -n "No PROOF master host specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - - echo "" - echo -n "Give port on which the PROOF xrootd daemons listen [$pport]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - pport="$answ" - fi - if [ "x$pport" = "x" ]; then - echo -n "No port specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - - echo "" - echo -n "Give the data file upload directory [$pdatadir]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - pdatadir=`eval echo $answ` - fi - if [ "x$pdatadir" = "x" ]; then - echo -n "No data upload directory specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - - echo "" - echo -n "Give number of workers per host (0 = 1 worker per core) [$pworkers]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - pworkers="$answ" - fi - if [ "x$pworkers" = "x" ]; then - echo -n "No workers per host specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - - REPEAT_LOOP=0 - conftype=1 - ;; - 2) - clear - echo "" - echo "Give advanced PROOF specific configuration parameters." - echo "" - echo -n "Give name of custom PROOF configuration file [$pconfig]: " - read answ &> /dev/null - if [ "x$answ" != "x" ]; then - pconfig=`eval echo $answ` - fi - if [ "x$pconfig" = "x" ]; then - echo -n "No config file specified, try again [Y/n]: " - read answ &> /dev/null - if [ "x$answ" != "x" ] && [ "$answ" != "Y" ] && [ "$answ" != "y" ]; then - echo "Exiting! Press to terminate install." - read - exit 0 - fi - continue - fi - - REPEAT_LOOP=0 - conftype=2 - ;; - h) - echo "---------------------------------------------------------------------------" - echo "PROOF specific configuration parameters:" - echo "" - echo "-- Option 1 --" - echo "Give the following basic configuration parameters:" - echo " - the user name under which to run PROOF, in general this is the same" - echo " user name as used for ssh access." - echo " - host on which to run the PROOF master, you will be prompted with" - echo " the first host in the list of hosts you specified." - echo " - port on which the PROOF xrootd daemons will listen. Give some high" - echo " port number like 210210 or 9191. You will be prompted with a" - echo " port that is not being used on any of the cluster nodes." - echo " - data file upload directory, by default this is \"proof/data\"," - echo " relative to your remote home directory." - echo " - number of workers per host, the default is as many workers" - echo " as there are cores (0 = 1 worker per core)." - echo "" - echo "-- Option 2 --" - echo "Give a custom PROOF configuration file containing your own advanced" - echo "settings, e.g. \"myxpd.conf\"." - echo "" - echo "-- Option h --" - echo "Selecting option h displays this help message." - echo " " - echo "-- Option x --" - echo "Selecting option x exits the PROOF installation." - echo "---------------------------------------------------------------------------" - echo -n "Press to continue..." - read answ &> /dev/null - ;; - x) - echo "Exiting the install..." - exit 0 - ;; - *) - echo "Invalid Choice. Please try again." - REPEAT_LOOP=1 - ;; - esac - - if [ "$REPEAT_LOOP" = 0 ]; then - echo "" - echo "The following PROOF configuration parameters have been specified:" - echo "" - case $conftype in - 1) - echo " - Name of user under which to run PROOF: $puser" - echo " - Node on which to run the PROOF master: $pmaster" - echo " - Port on which the PROOF daemons listen: $pport" - echo " - Data file upload directory: $pdatadir" - echo " - Number of workers per host (0 = 1 per core): $pworkers" - ;; - 2) - echo " - Custom PROOF configuration file: $pconfig" - ;; - esac - echo "" - echo -n "Is this correct [Y/n]: " - read answ &> /dev/null - if [ "x$answ" = "x" ] || [ "$answ" = "Y" ] || [ "$answ" = "y" ]; then - REPEAT_LOOP=0 - fi - fi -done - -# End of info gathering - - -# Store settings in $saverc -echo "nodes=\"$nodes\"" > $saverc -echo "domain=\"-\"" >> $saverc -echo "base=\"$base\"" >> $saverc -echo "lower=\"$lower\"" >> $saverc -echo "upper=\"$upper\"" >> $saverc -echo "width=\"$width\"" >> $saverc -echo "rdomain=\"$rdomain\"" >> $saverc -echo "rootbinary=\"$rootbinary\"" >> $saverc -echo "remoterootbinary=\"$remoterootbinary\"" >> $saverc -echo "autoremoterootbinary=\"$autoremoterootbinary\"" >> $saverc -echo "puser=\"$puser\"" >> $saverc -echo "pmaster=\"$pmaster\"" >> $saverc -echo "pport=\"$pport\"" >> $saverc -echo "pdatadir=\"$pdatadir\"" >> $saverc -echo "pworkers=\"$pworkers\"" >> $saverc -echo "pconfig=\"$pconfig\"" >> $saverc - -clear -echo "" -echo "Starting installation of PROOF on the cluster." -echo "" -echo "You might be asked to type your ssh password for user \"$puser\"" -echo "several times. We try to use ssh-agent to store your ssh keys," -echo "but if it does not work, you will have to type your password" -echo "up to three times per node. If you decide to quit and setup" -echo "automatic ssh access, re-running of the installer is simple as" -echo "all parameters have been saved. You can either run the installer" -echo "in batch (./proofinstall.sh -b) or you will be prompted with" -echo "your current choices." -echo "" -echo "Press to continue installation." -read - -# end of "$batch" = "no" -fi - -# Check if we can login to the nodes without passwd -# If we cannot copy public key to node - - -# Execute setup: -if [ -r "$rootbinary" ]; then - echo $circle scp -u $puser -n \"$nodes\" "$rootbinary" -fi -echo $circle scp -u $puser -n \"$nodes\" remoteproofsetup.sh -echo $circle ssh -u $puser -n \"$nodes\" "remoteproofsetup.sh $pmaster $pport $autoremoterootbinary" - -exit 0 diff --git a/etc/proof/utils/push b/etc/proof/utils/push deleted file mode 100755 index f27f7889fa725..0000000000000 --- a/etc/proof/utils/push +++ /dev/null @@ -1,41 +0,0 @@ -#! /bin/sh -# -# Usage: push [file] -# -# Push a file from MASTER to all other machines -# -# ATTENTION specify full pathname, e.g. /etc/passwd -# -# Author: rdm -# -USAGE='Usage: push file' -MACHINES=/usr/local/root/proof/etc/cluster.conf - -MASTER=pcna49a -DOMAIN=cern.ch - -case $# in - 0|2|3|4|5|6) echo $USAGE >&2; exit 1 -esac - -if [ `hostname` != $MASTER.$DOMAIN ] -then - echo "Must run on $MASTER" >&2; exit 1 -fi - -file=$1 - -if [ -f $MACHINES ] -then - machines=`cat $MACHINES` - echo -n "File $file pushed to" - for i in $machines - do - if [ $i != $MASTER ] - then - echo -n " $i" - rcp $file $i:$file - fi - done - echo -fi diff --git a/etc/proof/xpd.cf.sample b/etc/proof/xpd.cf.sample deleted file mode 100644 index fedfcff69855c..0000000000000 --- a/etc/proof/xpd.cf.sample +++ /dev/null @@ -1,244 +0,0 @@ -### -### Example of simple xrootd config file. -### -### The first part enables a simple data server exposing to clients -### the root paths /tmp and /data1. -### The second part shows how to enable password-based strong -### authentication using the ROOT special password in $HOME/.rootdpass -### The third part shows how to concurrently enable the PROOF serving -### features -### -### To load this configuration file: -### -### ~> xrootd -c $ROOTSYS/etc/proof/xpd.cf -### -### See http://www.slac.stanford.edu/xrootd for more details on the -### data serving part. -### - -### -### Part one: data serving -### -xrd.protocol xrootd * -xrootd.fslib libXrdOfs.so - -### Specify a non-default port here: -### - overwritten by -p on the command line -# xrd.port 5151 - -### Export path directives, i.e. the root paths which can accessed -### by clients ('xrootd.export /' exports the whole file system). -### An arbitrary number of these can be defined. The default is -### to export /tmp. -### NB: specifying any of these directives removes the default /tmp -### from the internal list; in such a case a directive needs to -### be given explicitly if /tmp needs to be exposed. -xrootd.export /tmp -xrootd.export /pool/data - -### -### Part two: security directives: -### - -### Example: enable password-based strong authentication checking -### also the special ROOT password in $HOME/.rootdpass. -### NB: if the application complains about a missing password file in -### $HOME/.xrd/ just create an empty one running 'xrdpwdadmin' and -### replying to the questions -### ( with the path to the ROOT distribution) -# xrootd.seclib /lib/libXrdSec.so -# sec.protocol pwd -d:0 -a:1 -vc:1 -upwd:2 -cryptfile:.rootdpass - -### -### Part three: enable PROOF serving -### - -### -### Directives governing the behaviour of the XrdProofd plug-in. -### Except when explicitly indicated, all the following directives support -### an optional 'if ' condition at the end of the line, e.g. -### xpd.rootsys /opt/root if lxb*.cern.ch -### xpd.rootsys /usr/local if lxp*.cern.ch -### Patterns may contain any number of wild cards; the best match is retained -### (max number of matching chars; if two are equal, the last specified wins). -### -### Available ROOT versions: the first 'rootsys' defines the default one; -### specifying a tag is optional: if missing, the ROOT version tag is taken -### (however, the tag must be unique, the first occurence is retained). -### If no 'xpd.rootsys' valid directives are specified, $ROOTSYS is used as -### default ROOT version. -# xpd.rootsys /opt/root [tag_for_default_version] -# xpd.rootsys /opt/root-dev [tag_for_an_alternative_version] -### -### Location of the temporary directory [/tmp] -# xpd.tmp /usr/tmp -### -### Internal wait timeout in secs [5] -# xpd.intwait 500 -### -### Max number of PROOF sessions [-1, i.e. no limit] -# xpd.maxsessions 10 -### -### Number of workers for local sessions [number of CPUs] -# xpd.localwrks 2 -### -### Multiuser option -### Default 1 (==ON) when running as superuser, 0 (==OFF) when running as normal. -### user. In the case the daemon has normal privileges, all users run under the -### effective user starting the daemon and privacy of sandboxes is not ensured -# xpd.multiuser 1 -### -### Defines what to do when no client sessions are attached to a client area. -# Format: -# xpd.shutdown -# where: -# is the type of action to be taken when a client completely -# disconnets; the options are: -# 0 remain connected -# 1 terminate when idle -# 2 terminate no matter the processing state -# -# is the delay after which the action for option 1 or 2 -# is performed; in seconds; to indicate minutes or hours use -# the suffix 'm' or 'h', respectively; e.g. 5m for 5 minutes. -# default: -# xpd.shutdown 1 0 -# xpd.shutdown 1 1m -# xpd.shutdown 1 1s -### -### Image name of this server [node name] -# xpd.image -### -### Working directory for sessions [/proof] -### If this directive is given, the user working directories will be in the -### form / -# xpd.workdir -### -### Dataset root directory [/dataset] -### If this directive is given, the user dataset directories will be in the -### form //, with ="default" if -### does not belongs to any of the defined groups. -# xpd.datasetdir -### -### Max number of old PROOF sessions for which the working directory -### is kept with all the relevant files in (logs, env, ...); non-positive -### values mean no limit [10] -# xpd.maxoldlogs 10 -### -### Modify priority of sessions belonging to by -### If is missing, apply the change to all sessions. -### This directive requires special privileges, so it may be ineffective -### if these are missing -# xpd.priority [if ] -# xpd.priority 4 -# xpd.priority 6 if thatuser -### -### Resource finder -### NB: 'if ' not supported for this directive. -# "static", i.e. using a config file -# path alternative config file -# [$ROOTSYS/proof/etc/proof.conf] -# if "yes": enable user private config files at -# $HOME/.proof.conf or $HOME/., where -# is the second argument to -# TProof::Open("","") ["no"] -# Maximum number of workers to be assigned to user -# session [-1, i.e. all] -# If != -1, specify the way workers -# are chosen: -# "roundrobin" round-robin selection in bunches -# of n(=) workers. -# Example: -# N = 10 (available workers), n = 4: -# 1st (session): 1-4, 2nd: 5-8, -# 3rd: 9,10,1,2, 4th: 3-6, ... -# "random" random choice (a worker is not -# assigned twice) -# xpd.resource static [] [ucfg:] [wmx:] [selopt:] -# xpd.resource static ~/.proof.test.conf wmx:2 selopt:random -# xpd.resource static $ROOTSYS/etc/proof.conf - -### -### Master(s) allowed to connect. Directive active only for Worker or -### Submaster session requests. Multiple 'allow' directives can -### be specified. By default all connections are allowed. -# xpd.allow lxb6041.cern.ch -### -### Server role (master, submaster, worker) [default: any] -### Allows to control the cluster structure. -### The following (commented) example will set lxb6041 as master, and all -### the others lxb* as workers -# xpd.role worker if lxb*.cern.ch -# xpd.role master if lxb6041.cern.ch -### -### URL and namespace for the local storage if different from defaults. -### By the default it is assumed that the pool space on the cluster is -### accessed via a redirector running at the top master under the common -### namespace /proofpool. -# xpd.poolurl lxb0105.cern.ch -# xpd.namespace /store - -### -### Specifies tracing options. Valid keywords are: -### req trace protocol requests [on]* -### login trace details about login requests [on]* -### act trace internal actions [off] -### rsp trace server replies [off] -### fork trace proofserv forks [on]* -### dbg trace details about actions [off] -### hdbg trace more details about actions [off] -### err trace errors [on] -### inflt trace details about inflate factors [off] -### all trace everything -### -### Defaults are shown in brackets; '*' shows the default when the '-d' -### option is passed on the command line. Each option may be -### optionally prefixed by a minus sign to turn off the setting. -### Order matters: 'all' in last position enables everything; in first -### position is corrected by subsequent settings -### -# xpd.trace fork -err rsp - -### Super-users directive: specify a comma-serarated list of users with -### special privileges; the effective user under which the daemon is run -### (-R option on the command line) is always privileged. -# xpd.superusers usr1,usr2 - -### User access control directive: specifies a comma-separated list -### of users allowed to connect to the cluster. -# xpd.allowedusers usr1,usr2,usr3 - -### -### Group information file -### Defines the file containing the information about the composition -### of the group and their properties. See example in xpd.groups.sample . -# xpd.groupfile $ROOTSYS/etc/proof/xpd.groups - -### -### This directive may be used to set additional environment variables -### for 'proofserv'. This is useful, for instance, to set client-side -### security options. It is possible to set some context depending -### keyworks which will be expanded before launching 'proofserv'; the -### syntax is ; keywords currently recognized are: -### --> expanded to workdir (see above) -### --> expanded to the user's username -### Example: -### -### xpd.putenv MYENV=//.creds -### -### with 'xpd.workdir /tmp/proof' will set MYENV to "/tmp/proof/minni/.creds" -### for user 'minni' and to "/tmp/proof/pippo/.creds" for user 'pippo'. -### There can be as 'putenv' directives as needed. -# xpd.putenv XrdSecPWDSRVPUK=//.creds/pwdsrvpuk - -### -### This directive may be used to set additional rootrc-like variables -### for 'proofserv'. This allows to control everything from this configuration -### file. At start-up, 'proofserv' will read the additional directives -### from the file "session.rootrc" created in the session working dir -### by XProofd; "session.rootrc" is actually a symlink to the real file -### whose name is in the form -### --.rootrc -### Example: -### -### xpd.putrc AName.AVar: AValue diff --git a/etc/proof/xpd.groups.sample b/etc/proof/xpd.groups.sample deleted file mode 100644 index 2998dcdf90da7..0000000000000 --- a/etc/proof/xpd.groups.sample +++ /dev/null @@ -1,30 +0,0 @@ -### -### Example of file defining groups in PROOF. -### The availability and location of this file is communicated to XProofd -### by the directive 'xpd.groupfile '. -### - -### -### Definition of groups. -### A user can be part of many groups; the default group is defined by the -### first occurence, so the order is important. -# group tpc ernst,pippo,mst -# group ecal abd,karl,ernst - -### -### Define group properties. -### A property is defined by a name, a nominal value and an effective value -### (values must be integers); the syntax is -### property [] -### The defaul effective value is the nominal value. -### It is up to the application to give a meaning to the values. -### Example: properties of group 'tpc' -# property tpc priority 200 -# property tpc diskquota 100 - -### Example: properties of group 'ecal' -# property ecal priority 100 -# property ecal diskquota 30 - -### Include a file with other directives (5.27/04 and higher only) -# include /path/sub-group.file diff --git a/gui/CMakeLists.txt b/gui/CMakeLists.txt index 43cbd8cbdb7cc..bb127b3e475e9 100644 --- a/gui/CMakeLists.txt +++ b/gui/CMakeLists.txt @@ -10,9 +10,6 @@ add_subdirectory(fitpanel) add_subdirectory(guibuilder) add_subdirectory(guihtml) add_subdirectory(recorder) -if(proof) - add_subdirectory(sessionviewer) -endif() if(webgui) if(cefweb) diff --git a/gui/gui/src/HelpText.cxx b/gui/gui/src/HelpText.cxx index 4740696a72feb..3f42528e05d76 100644 --- a/gui/gui/src/HelpText.cxx +++ b/gui/gui/src/HelpText.cxx @@ -14,12 +14,7 @@ histogramming, fitting and minimization algorithms) and visualization\n\ tools. The user interacts with ROOT via a graphical user interface, the\n\ command line or batch scripts. The command and scripting language\n\ is C++ (using the interpreter) and large scripts can be compiled and\n\ -dynamically linked in. Using the PROOF (Parallel ROOT Facility)\n\ -extension large databases can be analysed in parallel on MPP's,\n\ -SMP's or loosely coupled workstation/PC clusters. ROOT also\n\ -contains a C++ to HTML documentation generation system using\n\ -the interpreter's dictionaries (the reference manual on\n\ -the web is generated that way) and a rich set of interprocess\n\ +dynamically linked in. ROOT also provides a rich set of interprocess\n\ communication classes allowing the transfer of complete objects\n\ from one process to another.\n\ "; diff --git a/gui/gui/src/TGApplication.cxx b/gui/gui/src/TGApplication.cxx index 72d7dbbf95f4e..d5ace716386e6 100644 --- a/gui/gui/src/TGApplication.cxx +++ b/gui/gui/src/TGApplication.cxx @@ -80,23 +80,6 @@ TGApplication::TGApplication(const char *appClassName, // a GUI application is never run in batch mode gROOT->SetBatch(kFALSE); - if (strcmp(appClassName, "proofserv")) { - const char *ttpath = gEnv->GetValue("Root.TTFontPath", - TROOT::GetTTFFontDir()); - char *ttfont = gSystem->Which(ttpath, "arialbd.ttf", kReadPermission); - // Added by cholm for use of DFSG - fonts - based on fix by Kevin - if (!ttfont) - ttfont = gSystem->Which(ttpath, "FreeSansBold.ttf", kReadPermission); - if (ttfont && gEnv->GetValue("Root.UseTTFonts", 1)) { - TPluginHandler *h; - if ((h = gROOT->GetPluginManager()->FindHandler("TVirtualX", "x11ttf"))) - if (h->LoadPlugin() == -1) - Info("TGApplication", "no TTF support"); - } - - delete [] ttfont; - } - // Create the canvas colors early so they are allocated before // any color table expensive bitmaps get allocated in GUI routines (like // creation of XPM bitmaps). diff --git a/gui/gui/src/TRootBrowser.cxx b/gui/gui/src/TRootBrowser.cxx index 5ab9c6ef18bb7..68a07b1adea73 100644 --- a/gui/gui/src/TRootBrowser.cxx +++ b/gui/gui/src/TRootBrowser.cxx @@ -55,7 +55,6 @@ Here is the list of available options: - F: File browser - E: Text Editor - H: HTML browser C: Canvas I: I/O redirection - - P: Proof - G: GL viewer */ @@ -904,12 +903,6 @@ void TRootBrowser::InitPlugins(Option_t *opt) ++fNbInitPlugins; } - // PROOF plugin... - if (opt[i] == 'P') { - cmd.Form("new TSessionViewer();"); - ExecPlugin("PROOF", 0, cmd.Data(), 1); - ++fNbInitPlugins; - } } // --- Right bottom area diff --git a/gui/sessionviewer/CMakeLists.txt b/gui/sessionviewer/CMakeLists.txt deleted file mode 100644 index 5c29533405ac9..0000000000000 --- a/gui/sessionviewer/CMakeLists.txt +++ /dev/null @@ -1,34 +0,0 @@ -# Copyright (C) 1995-2019, Rene Brun and Fons Rademakers. -# All rights reserved. -# -# For the licensing terms see $ROOTSYS/LICENSE. -# For the list of contributors see $ROOTSYS/README/CREDITS. - -############################################################################ -# CMakeLists.txt file for building ROOT gui/sessionviewer package -############################################################################ - -ROOT_STANDARD_LIBRARY_PACKAGE(SessionViewer - HEADERS - TProofProgressDialog.h - TProofProgressLog.h - TProofProgressMemoryPlot.h - TSessionDialogs.h - TSessionLogView.h - TSessionViewer.h - SOURCES - src/TProofProgressDialog.cxx - src/TProofProgressLog.cxx - src/TProofProgressMemoryPlot.cxx - src/TSessionDialogs.cxx - src/TSessionLogView.cxx - src/TSessionViewer.cxx - DEPENDENCIES - Proof - Gui - Hist - Gpad - Graf - Tree - MathCore -) diff --git a/gui/sessionviewer/doc/index.md b/gui/sessionviewer/doc/index.md deleted file mode 100644 index a7fb17fc9c9c6..0000000000000 --- a/gui/sessionviewer/doc/index.md +++ /dev/null @@ -1,3 +0,0 @@ -\defgroup sessionviewer Session Viewer -\ingroup gui -\brief Session Viewer as used in Proof. diff --git a/gui/sessionviewer/inc/LinkDef.h b/gui/sessionviewer/inc/LinkDef.h deleted file mode 100644 index 7896028bca636..0000000000000 --- a/gui/sessionviewer/inc/LinkDef.h +++ /dev/null @@ -1,37 +0,0 @@ -/* @(#)root/sessionviewer:$Id$ */ - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifdef __CLING__ - -#pragma link off all globals; -#pragma link off all classes; -#pragma link off all functions; - -#pragma link C++ global gSessionViewer; - -#pragma link C++ class TProofProgressDialog+; -#pragma link C++ class TProofProgressLog+; -#pragma link C++ class TProofProgressMemoryPlot+; -#pragma link C++ class TSessionLogView+; -#pragma link C++ class TSessionServerFrame+; -#pragma link C++ class TSessionFrame+; -#pragma link C++ class TEditQueryFrame+; -#pragma link C++ class TSessionQueryFrame+; -#pragma link C++ class TSessionOutputFrame+; -#pragma link C++ class TSessionInputFrame+; -#pragma link C++ class TSessionViewer+; -#pragma link C++ class TQueryDescription+; -#pragma link C++ class TSessionDescription+; -#pragma link C++ class TPackageDescription+; -#pragma link C++ class TNewQueryDlg+; -#pragma link C++ class TNewChainDlg+; -#pragma link C++ class TUploadDataSetDlg+; - -#endif diff --git a/gui/sessionviewer/inc/TProofProgressDialog.h b/gui/sessionviewer/inc/TProofProgressDialog.h deleted file mode 100644 index a51728405830d..0000000000000 --- a/gui/sessionviewer/inc/TProofProgressDialog.h +++ /dev/null @@ -1,136 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Fons Rademakers 21/03/03 - -/************************************************************************* - * Copyright (C) 1995-2003, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofProgressDialog -#define ROOT_TProofProgressDialog - - -#include "TTime.h" -#include "TString.h" - -class TGTransientFrame; -class TGProgressBar; -class TGTextButton; -class TGCheckButton; -class TGLabel; -class TGTextBuffer; -class TGTextEntry; -class TProof; -class TProofProgressLog; -class TProofProgressMemoryPlot; -class TNtuple; -class TGraph; -class TGSpeedo; - -class TProofProgressDialog { - - friend class TProofProgressLog; - friend class TProofProgressMemoryPlot; - -private: - enum EQueryStatus { kRunning = 0, kDone, kStopped, kAborted, kIncomplete }; - - TGTransientFrame *fDialog; // transient frame, main dialog window - TGProgressBar *fBar; // progress bar - TGTextButton *fClose; - TGTextButton *fStop; - TGTextButton *fAbort; - TGTextButton *fAsyn; - TGTextButton *fLog; - TGTextButton *fRatePlot; - TGTextButton *fMemPlot; - TGTextButton *fUpdtSpeedo; - TGCheckButton *fKeepToggle; - TGCheckButton *fLogQueryToggle; - TGTextBuffer *fTextQuery; - TGTextEntry *fEntry; - TGLabel *fTitleLab; - TGLabel *fFilesEvents; - TGLabel *fTimeLab; - TGLabel *fProcessed; - TGLabel *fEstim; - TGLabel *fTotal; - TGLabel *fRate; - TGLabel *fInit; - TGLabel *fSelector; - Bool_t fSpeedoEnabled; // whether to enable the speedometer - TGSpeedo *fSpeedo; // speedometer - TGCheckButton *fSmoothSpeedo; // use smooth speedometer update - TProofProgressLog *fLogWindow; // transient frame for logs - TProofProgressMemoryPlot *fMemWindow; // transient frame for memory plots - TProof *fProof; - TTime fStartTime; - TTime fEndTime; - Long64_t fPrevProcessed; - Long64_t fPrevTotal; - Long64_t fFirst; - Long64_t fEntries; - Int_t fFiles; - EQueryStatus fStatus; - Bool_t fKeep; - Bool_t fLogQuery; - TNtuple *fRatePoints; - TGraph *fRateGraph; - TGraph *fMBRtGraph; - TGraph *fActWGraph; - TGraph *fTotSGraph; - TGraph *fEffSGraph; - Float_t fInitTime; - Float_t fProcTime; - Double_t fAvgRate; - Double_t fAvgMBRate; - Int_t fRightInfo; - - TString fSessionUrl; - - Float_t AdjustBytes(Float_t mbs, TString &sf); - - static Bool_t fgKeepDefault; - static Bool_t fgLogQueryDefault; - static TString fgTextQueryDefault; - -public: - TProofProgressDialog(TProof *proof, const char *selector, - Int_t files, Long64_t first, Long64_t entries); - virtual ~TProofProgressDialog(); - - void ResetProgressDialog(const char *sel, Int_t sz, Long64_t fst, Long64_t ent); - void Progress(Long64_t total, Long64_t processed); - void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) { - Progress(total, processed, bytesread, initTime, procTime, - evtrti, mbrti, -1, -1, -1.); } - void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti, Int_t actw, Int_t tses, Float_t eses); - void DisableAsyn(); - void IndicateStop(Bool_t aborted); - void LogMessage(const char *msg, Bool_t all); - - void CloseWindow(); - void DoClose(); - void DoLog(); - void DoKeep(Bool_t on); - void DoSetLogQuery(Bool_t on); - void DoStop(); - void DoAbort(); - void DoAsyn(); - void DoPlotRateGraph(); - void DoMemoryPlot(); - void DoEnableSpeedo(); - void ToggleOdometerInfos(); - void ToggleThreshold(); - - ClassDef(TProofProgressDialog,0) //PROOF progress dialog -}; - -#endif diff --git a/gui/sessionviewer/inc/TProofProgressLog.h b/gui/sessionviewer/inc/TProofProgressLog.h deleted file mode 100644 index 43494d7cbdef1..0000000000000 --- a/gui/sessionviewer/inc/TProofProgressLog.h +++ /dev/null @@ -1,96 +0,0 @@ -// @(#)root/sessionviewer:$Id: c52dc6cb225c25e34d388329d41570a484d98ce0 $ -// Author: G Ganis, Aug 2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofProgressLog -#define ROOT_TProofProgressLog - - -#include "TGFrame.h" -#include "TString.h" - -class TGTextView; -class TGTextButton; -class TProofProgressDialog; -class TProofLog; -class TGTextEntry; -class TGNumberEntry; -class TGListBox; -class TGSplitButton; -class TGVerticalFrame; -class TGCheckButton; -class TGLabel; - -class TProofProgressLog : public TGTransientFrame { - -private: - enum ETextType { kRaw = 0, kStd = 1, kGrep = 2 }; - - TString fSessionUrl; - Int_t fSessionIdx; - - TGTextView *fText; // text widget - TGTextButton *fClose; // close button - TGListBox *fLogList; // list of workers - TGTextButton *fLogNew; // display logs button - TProofProgressDialog *fDialog; // owner dialog - TProofLog *fProofLog; // the log - TGNumberEntry *fLinesFrom; // starting line - TGNumberEntry *fLinesTo; // ending line - TGTextEntry *fGrepText; // text to grep for in the logs - TGTextEntry *fUrlText; // url to connect to - TGNumberEntry *fSessNum; // relative index of the session to get - TGTextEntry *fFileName; // file to save to - TGTextButton *fSave; // save button - TGTextButton *fGrepButton; //grep button - TGTextButton *fUrlButton; //rebuild button - TGCheckButton *fAllLines; // display all lines button - TGCheckButton *fRawLines; // display raw lines button - TGSplitButton *fAllWorkers; // display all workers button - TGVerticalFrame *fVworkers; // Vertical frame - TGLabel *fGrepLabel; // label indicating if grepping or piping - TGCheckButton *fGrepCheckCmd; // checkbox active if piping - TGCheckButton *fGrepCheckInv; // checkbox active if inverting - - Bool_t fFullText; // 0 - when grep was called - Int_t fTextType; // Type of retrieval - - void Init(Int_t w = 700, Int_t h = 600); - -public: - TProofProgressLog(TProofProgressDialog *d, Int_t w = 700, Int_t h = 600); - TProofProgressLog(const char *url = nullptr, Int_t sessionidx = 0, Int_t w = 700, Int_t h = 600); - ~TProofProgressLog() override; - - void BuildLogList(Bool_t create = kFALSE); - void DoLog(Bool_t grep=kFALSE); - void LogMessage(const char *msg, Bool_t all); - void Rebuild(); - - void LoadBuffer(const char *buffer); - void AddBuffer(const char *buffer); - - void LoadFile(const char *file); - - void Clear(Option_t * = nullptr) override; - void Popup(); - void SaveToFile(); - void NoLineEntry(); - void SetGrepView(); - void Select(Int_t id, Bool_t all = kTRUE); - - void SetUrl(const char *url) { fSessionUrl = url; } - // slots - void CloseWindow() override; - - ClassDefOverride(TProofProgressLog,0) //Class implementing a log graphic box -}; - -#endif diff --git a/gui/sessionviewer/inc/TProofProgressMemoryPlot.h b/gui/sessionviewer/inc/TProofProgressMemoryPlot.h deleted file mode 100644 index 5f5a08b442ced..0000000000000 --- a/gui/sessionviewer/inc/TProofProgressMemoryPlot.h +++ /dev/null @@ -1,62 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Anna Kreshuk 18/07/2008 - -/************************************************************************* - * Copyright (C) 1995-2003, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofProgressMemoryPlot -#define ROOT_TProofProgressMemoryPlot - - -#include "TGFrame.h" - -class TGListBox; -class TGTextButton; -class TRootEmbeddedCanvas; -class TProofProgressDialog; -class TGSplitButton; -class TProofLog; -class TMultiGraph; -class TGraph; -class TProofLogElem; - -class TProofProgressMemoryPlot : public TGTransientFrame { - - protected: - TProofProgressDialog *fDialog; - TGListBox *fWorkers; - TGTextButton *fPlot; - TGSplitButton *fAllWorkers; // display all workers button - TRootEmbeddedCanvas *fWorkersPlot; - TRootEmbeddedCanvas *fMasterPlot; - TProofLog *fProofLog; - TMultiGraph *fWPlot; - TMultiGraph *fMPlot; - TMultiGraph *fAPlot; - Bool_t fFullLogs; - - TGListBox* BuildLogList(TGFrame *parent); - TGraph* DoWorkerPlot(TProofLogElem *ple); - TGraph* DoMasterPlot(TProofLogElem *ple); - TGraph* DoAveragePlot(Int_t &max_el, Int_t &min_el); - TGraph* DoAveragePlotOld(Int_t &max_el, Int_t &min_el); - - Int_t ParseLine(TString l, Long64_t &v, Long64_t &r, Long64_t &e); - - public: - TProofProgressMemoryPlot(TProofProgressDialog *d, Int_t w = 700, Int_t h = 300); - ~TProofProgressMemoryPlot() override; - - void Clear(Option_t * = nullptr) override; - void DoPlot(); - void Select(Int_t id); - - ClassDefOverride(TProofProgressMemoryPlot,0) //PROOF progress memory plots -}; - -#endif diff --git a/gui/sessionviewer/inc/TSessionDialogs.h b/gui/sessionviewer/inc/TSessionDialogs.h deleted file mode 100644 index e66ee569800be..0000000000000 --- a/gui/sessionviewer/inc/TSessionDialogs.h +++ /dev/null @@ -1,157 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Marek Biskup, Jakub Madejczyk, Bertrand Bellenot 10/08/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TSessionDialogs -#define ROOT_TSessionDialogs - - -#include "TSessionViewer.h" - -class TList; -class TSeqCollection; -class TChain; -class TDSet; -class TGTextEntry; -class TGTextButton; -class TGTextBuffer; -class TGCheckButton; -class TGLabel; -class TGListView; -class TGPicture; -class TGFileContainer; - -////////////////////////////////////////////////////////////////////////// -// New Chain Dialog -////////////////////////////////////////////////////////////////////////// - -class TNewChainDlg : public TGTransientFrame { - -private: - TGFileContainer *fContents; // macro files container - TGListView *fListView; // memory objects list view - TGLVContainer *fLVContainer; // and its container - TGTextBuffer *fNameBuf; // buffer for dataset name - TGTextEntry *fName; // dataset name text entry - TGTextButton *fOkButton; // ok button - TGTextButton *fCancelButton; // cancel button - TSeqCollection *fChains; // collection of datasets - TObject *fChain; // actual TDSet or TChain - -public: - TNewChainDlg(const TGWindow *p=nullptr, const TGWindow *main=nullptr); - ~TNewChainDlg() override; - - void UpdateList(); - virtual void OnDoubleClick(TGLVEntry*,Int_t); - virtual void DisplayDirectory(const TString &fname); - void OnElementClicked(TGLVEntry* entry, Int_t btn); - void OnElementSelected(TObject *obj); //*SIGNAL* - - Bool_t ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t parm2) override; - void CloseWindow() override; - - ClassDefOverride(TNewChainDlg, 0) // New chain dialog -}; - -////////////////////////////////////////////////////////////////////////// -// New Query Dialog -////////////////////////////////////////////////////////////////////////// - -class TNewQueryDlg : public TGTransientFrame { - -private: - Bool_t fEditMode; // kTRUE if used to edit existing query - Bool_t fModified; // kTRUE if settings have changed - TGCompositeFrame *fFrmNewQuery; // top (main) frame - TGCompositeFrame *fFrmMore; // options frame - TGTextButton *fBtnMore; // "more >>" / "less <<" button - TGTextButton *fBtnClose; // close button - TGTextButton *fBtnSave; // save button - TGTextButton *fBtnSubmit; // save & submit button - - TGTextEntry *fTxtQueryName; // query name text entry - TGTextEntry *fTxtChain; // chain name text entry - TGTextEntry *fTxtSelector; // selector name text entry - TGTextEntry *fTxtOptions; // options text entry - TGNumberEntry *fNumEntries; // number of entries selector - TGNumberEntry *fNumFirstEntry; // first entry selector - TGTextEntry *fTxtEventList; // event list text entry - TSessionViewer *fViewer; // pointer on main viewer - TQueryDescription *fQuery; // query description class - TObject *fChain; // actual TChain - -public: - TNewQueryDlg(TSessionViewer *gui, Int_t Width, Int_t Height, - TQueryDescription *query = nullptr, Bool_t editmode = kFALSE); - ~TNewQueryDlg() override; - void Build(TSessionViewer *gui); - void OnNewQueryMore(); - void OnBrowseChain(); - void OnBrowseSelector(); - void OnBrowseEventList(); - void OnBtnSaveClicked(); - void OnBtnCloseClicked(); - void OnBtnSubmitClicked(); - void OnElementSelected(TObject *obj); - void CloseWindow() override; - void Popup(); - void SettingsChanged(); - void UpdateFields(TQueryDescription *desc); - Bool_t ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t parm2) override; - - ClassDefOverride(TNewQueryDlg, 0) // New query dialog -}; - -////////////////////////////////////////////////////////////////////////// -// Upload DataSet Dialog -////////////////////////////////////////////////////////////////////////// - -class TUploadDataSetDlg : public TGTransientFrame { - -private: - Bool_t fUploading; - TList *fSkippedFiles; // List of skipped files - TGTextEntry *fDSetName; // dataset name text entry - TGTextEntry *fDestinationURL; // destination URL text entry - TGTextEntry *fLocationURL; // location URL text entry - TGListView *fListView; // dataset files list view - TGLVContainer *fLVContainer; // and its container - TGTextButton *fAddButton; // Add >> button - TGTextButton *fBrowseButton; // Browse... button - TGTextButton *fRemoveButton; // Remove button - TGTextButton *fClearButton; // Clear button - TGCheckButton *fOverwriteDSet; // overwrite DataSet - TGCheckButton *fOverwriteFiles; // overwrite All Files - TGCheckButton *fAppendFiles; // append files - TGTextButton *fUploadButton; // Upload button - TGTextButton *fCloseDlgButton; // Close Dialog button - TSessionViewer *fViewer; // pointer on main viewer - -public: - TUploadDataSetDlg(TSessionViewer *gui, Int_t w, Int_t h); - ~TUploadDataSetDlg() override; - - void CloseWindow() override; - Bool_t ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t parm2) override; - void AddFiles(const char *fileName); - void AddFiles(TList *fileList); - void BrowseFiles(); - void ClearFiles(); - void RemoveFile(); - void UploadDataSet(); - void OnOverwriteDataset(Bool_t on); - void OnOverwriteFiles(Bool_t on); - void OnAppendFiles(Bool_t on); - - ClassDefOverride(TUploadDataSetDlg, 0) // New query dialog -}; - -#endif diff --git a/gui/sessionviewer/inc/TSessionLogView.h b/gui/sessionviewer/inc/TSessionLogView.h deleted file mode 100644 index d0a0ecf68b9dc..0000000000000 --- a/gui/sessionviewer/inc/TSessionLogView.h +++ /dev/null @@ -1,50 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Bertrand Bellenot, Gerri Ganis 15/09/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TSessionLogView -#define ROOT_TSessionLogView - -#include "TGFrame.h" - -#include "TGButton.h" - -#include "TGTextView.h" - -class TSessionViewer; - -class TSessionLogView : public TGTransientFrame { - -private: - TSessionViewer *fViewer; // pointer on main viewer - TGTextView *fTextView; // Text view widget - TGTextButton *fClose; // OK button - TGLayoutHints *fL1; // layout of TGTextEdit - TGLayoutHints *fL2; // layout of OK button - -public: - TSessionLogView(TSessionViewer *viewer, UInt_t w, UInt_t h); - ~TSessionLogView() override; - - void AddBuffer(const char *buffer); - void LoadBuffer(const char *buffer); - void LoadFile(const char *file); - - Bool_t ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t parm2) override; - - void CloseWindow() override; - void ClearLogView(); - void Popup(); - void SetTitle(); - - ClassDefOverride(TSessionLogView, 0) // PROOF progress dialog -}; - -#endif diff --git a/gui/sessionviewer/inc/TSessionViewer.h b/gui/sessionviewer/inc/TSessionViewer.h deleted file mode 100644 index 62f21dd23ec2d..0000000000000 --- a/gui/sessionviewer/inc/TSessionViewer.h +++ /dev/null @@ -1,636 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Marek Biskup, Jakub Madejczyk, Bertrand Bellenot 10/08/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -#ifndef ROOT_TSessionViewer -#define ROOT_TSessionViewer - -#include "TGFrame.h" - -#include "TString.h" - -#include "TGTextEntry.h" - -#include "TGNumberEntry.h" - -#include "TGTab.h" - -#include "TGListView.h" - -#include "TTime.h" - -#include -#include - -class TList; -class TChain; -class TDSet; -class TGNumberEntry; -class TGTextEntry; -class TGTextButton; -class TGCheckButton; -class TGTextBuffer; -class TGTableLayout; -class TGIcon; -class TGLabel; -class TGHProgressBar; -class TGPopupMenu; -class TGLVContainer; -class TGListView; -class TGLVEntry; -class TGCanvas; -class TGListTree; -class TGListTreeItem; -class TGStatusBar; -class TGPicture; -class TGMenuBar; -class TGPopupMenu; -class TGToolBar; -class TGTextView; -class TGTab; -class TRootEmbeddedCanvas; -class TGListBox; -class TCanvas; -class TEnv; -struct UserGroup_t; - -class TProofMgr; -class TProof; -class TSessionViewer; -class TSessionLogView; -class TQueryResult; -class TContextMenu; - -////////////////////////////////////////////////////////////////////////// -// // -// TSessionViewer - A GUI for ROOT / PROOF Sessions // -// // -////////////////////////////////////////////////////////////////////////// - -////////////////////////////////////////////////////////////////////////// -// TQueryDescription class : Description of queries -////////////////////////////////////////////////////////////////////////// - -class TQueryDescription : public TObject { - -public: - enum ESessionQueryStatus { - kSessionQueryAborted = 0, - kSessionQuerySubmitted, - kSessionQueryRunning, - kSessionQueryStopped, - kSessionQueryCompleted, - kSessionQueryFinalized, - kSessionQueryCreated, - kSessionQueryFromProof - }; - - ESessionQueryStatus fStatus; // query status - TString fReference; // query reference string (unique identifier) - TString fQueryName; // query name - TString fSelectorString; // selector name - TString fTDSetString; // dataset name - TString fOptions; // query processing options - TString fEventList; // event list - Int_t fNbFiles; // number of files to process - Long64_t fNoEntries; // number of events/entries to process - Long64_t fFirstEntry; // first event/entry to process - TTime fStartTime; // start time of the query - TTime fEndTime; // end time of the query - TObject *fChain; // dataset on which to process selector - TQueryResult *fResult; // query result received back - - const char *GetName() const override { return fQueryName; } - - ClassDefOverride(TQueryDescription, 1) // Query description -}; - - -enum EMenuIdentification { - kMenuAddToFeedback, - kMenuShow, - kMenuRemoveFromFeedback -}; - -////////////////////////////////////////////////////////////////////////// -// TSessionDescription class : Description of Session -////////////////////////////////////////////////////////////////////////// - -class TSessionDescription : public TObject { - -public: - TString fTag; // session unique identifier - TString fName; // session name - TString fAddress; // server address - Int_t fPort; // communication port - TString fConfigFile; // configuration file name - Int_t fLogLevel; // log (debug) level - TString fUserName; // user name (on server) - Bool_t fConnected; // kTRUE if connected - Bool_t fAttached; // kTRUE if attached - Bool_t fLocal; // kTRUE if session is local - Bool_t fSync; // kTRUE if in sync mode - Bool_t fAutoEnable; // enable packages at session startup time - TList *fQueries; // list of queries in this session - TList *fPackages; // list of packages - TQueryDescription *fActQuery; // current (actual) query - TProof *fProof; // pointer on TProof used by this session - TProofMgr *fProofMgr; // Proof sessions manager - Int_t fNbHistos; // number of feedback histos - - const char *GetName() const override { return fName; } - - ClassDefOverride(TSessionDescription, 1) // Session description -}; - -////////////////////////////////////////////////////////////////////////// -// TPackageDescription class : Description of Package -////////////////////////////////////////////////////////////////////////// - -class TPackageDescription : public TObject { - -public: - TString fName; // package name - TString fPathName; // full path name of package - Int_t fId; // package id - Bool_t fUploaded; // package has been uploaded - Bool_t fEnabled; // package has been enabled - - const char *GetName() const override { return fName; } - - ClassDefOverride(TPackageDescription, 1) // Package description -}; - -////////////////////////////////////////////////////////////////////////// -// // -// TSessionServerFrame // -// A composite Frame used in the right part of the Session Viewer GUI // -// for any information relative to server side : address, port, user... // -// // -////////////////////////////////////////////////////////////////////////// - -class TSessionServerFrame : public TGCompositeFrame { - -private: - TGCompositeFrame *fFrmNewServer; // main group frame - TGTextEntry *fTxtName; // connection name text entry - TGTextEntry *fTxtAddress; // server address text entry - TGNumberEntry *fNumPort; // port number selector - TGNumberEntry *fLogLevel; // log (debug) level selector - TGTextEntry *fTxtConfig; // configuration file text entry - TGTextEntry *fTxtUsrName; // user name text entry - TGCheckButton *fSync; // sync / async flag selector - TSessionViewer *fViewer; // pointer on the main viewer - TGTextButton *fBtnAdd; // "Add" button - TGTextButton *fBtnConnect; // "Connect" button - -public: - TSessionServerFrame(TGWindow *parent, Int_t w, Int_t h); - ~TSessionServerFrame() override; - - void Build(TSessionViewer *gui); - - const char *GetName() const override { return fTxtName->GetText(); } - const char *GetAddress() const { return fTxtAddress->GetText(); } - Int_t GetPortNumber() const { return fNumPort->GetIntNumber(); } - Int_t GetLogLevel() const { return fLogLevel->GetIntNumber(); } - const char *GetConfigText() const { return fTxtConfig->GetText(); } - const char *GetUserName() const { return fTxtUsrName->GetText(); } - Bool_t IsSync() const { return (Bool_t)(fSync->GetState() == kButtonDown); } - - void SetAddEnabled(Bool_t on = kTRUE) { - on == kTRUE ? ShowFrame(fBtnAdd) : HideFrame(fBtnAdd); } - void SetConnectEnabled(Bool_t on = kTRUE) { - on == kTRUE ? ShowFrame(fBtnConnect) : HideFrame(fBtnConnect); } - void SetName(const char *str) override { fTxtName->SetText(str); } - void SetAddress(const char *str) { fTxtAddress->SetText(str); } - void SetPortNumber(Int_t port) { fNumPort->SetIntNumber(port); } - void SetLogLevel(Int_t log) { fLogLevel->SetIntNumber(log); } - void SetConfigText(const char *str) { fTxtConfig->SetText(str); } - void SetUserName(const char *str) { fTxtUsrName->SetText(str); } - void SetSync(Bool_t sync) { - fSync->SetState(sync ? kButtonDown : kButtonUp); } - - void SettingsChanged(); - - void OnBtnConnectClicked(); - void OnBtnNewServerClicked(); - void OnBtnDeleteClicked(); - void OnBtnAddClicked(); - void OnConfigFileClicked(); - void Update(TSessionDescription* desc); - Bool_t HandleExpose(Event_t *event) override; - Bool_t ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t parm2) override; - - ClassDefOverride(TSessionServerFrame, 0) // Server frame -}; - - -////////////////////////////////////////////////////////////////////////// -// // -// TSessionFrame // -// A composite Frame used in the right part of the Session Viewer GUI // -// for any information, settings or controls relative to the current // -// session. // -// // -////////////////////////////////////////////////////////////////////////// - -class TSessionFrame : public TGCompositeFrame { - -private: - - TGTab *fTab; // main tab frame - TGCompositeFrame *fFA, *fFB, *fFC; - TGCompositeFrame *fFD, *fFE; // five tabs element - TGTextEntry *fCommandTxt; // Command line text entry - TGTextBuffer *fCommandBuf; // Command line text buffer - TGTextView *fInfoTextView; // summary on current query - TGCheckButton *fClearCheck; // clear text view after each command - TGTextButton *fBtnShowLog; // show log button - TGTextButton *fBtnNewQuery; // new query button - TGTextButton *fBtnGetQueries; // get entries button - // Packages tab related items - TGListBox *fLBPackages; // packages listbox - TGTextButton *fBtnAdd; // add package button - TGTextButton *fBtnRemove; // remove package button - TGTextButton *fBtnUp; // move package up button - TGTextButton *fBtnDown; // move package down button - TGTextButton *fBtnShow; // show packages button - TGTextButton *fBtnShowEnabled; // show enabled packages button - TGCheckButton *fChkMulti; // multiple selection check - TGCheckButton *fChkEnable; // enable at session startup check - TGTextButton *fBtnUpload; // upload packages button - TGTextButton *fBtnEnable; // enable packages button - TGTextButton *fBtnClear; // clear all packages button - TGTextButton *fBtnDisable; // disable packages button - // Datasets tab related items - TGCanvas *fDSetView; // dataset tree view - TGListTree *fDataSetTree; // dataset list tree - TGTextButton *fBtnUploadDSet; // upload dataset button - TGTextButton *fBtnRemoveDSet; // remove dataset button - TGTextButton *fBtnVerifyDSet; // verify dataset button - TGTextButton *fBtnRefresh; // refresh list button - // Options tab related items - TGTextEntry *fTxtParallel; // parallel nodes text entry - TGNumberEntry *fLogLevel; // log level number entry - TGTextButton *fApplyLogLevel; // apply log level button - TGTextButton *fApplyParallel; // apply parallel nodes button - - TSessionViewer *fViewer; // pointer on main viewer - TGLabel *fInfoLine[19]; // infos on session - -public: - TSessionFrame(TGWindow* parent, Int_t w, Int_t h); - ~TSessionFrame() override; - - void Build(TSessionViewer *gui); - void CheckAutoEnPack(Bool_t checked = kTRUE) { - fChkEnable->SetState(checked ? kButtonDown : kButtonUp); } - Int_t GetLogLevel() const { return fLogLevel->GetIntNumber(); } - void SetLogLevel(Int_t log) { fLogLevel->SetIntNumber(log); } - TGTab *GetTab() const { return fTab; } - - //Function that handle input from user: - void OnApplyLogLevel(); - void OnApplyParallel(); - void OnBtnAddClicked(); - void OnBtnRemoveClicked(); - void OnBtnUpClicked(); - void OnBtnDownClicked(); - void OnBtnShowLogClicked(); - void OnBtnNewQueryClicked(); - void OnBtnGetQueriesClicked(); - void OnBtnDisconnectClicked(); - void OnCommandLine(); - void OnUploadPackages(); - void OnEnablePackages(); - void OnDisablePackages(); - void OnClearPackages(); - void OnMultipleSelection(Bool_t on); - void OnStartupEnable(Bool_t on); - void ProofInfos(); - void SetLocal(Bool_t local = kTRUE); - void ShutdownSession(); - void UpdatePackages(); - void OnBtnUploadDSet(); - void OnBtnRemoveDSet(); - void OnBtnVerifyDSet(); - void UpdateListOfDataSets(); - - ClassDefOverride(TSessionFrame, 0) // Session frame -}; - -////////////////////////////////////////////////////////////////////////// -// New Query Dialog -////////////////////////////////////////////////////////////////////////// - -class TEditQueryFrame : public TGCompositeFrame { - -private: - TGCompositeFrame *fFrmMore; // options frame - TGTextButton *fBtnMore; // "more >>" / "less <<" button - - TGTextEntry *fTxtQueryName; // query name text entry - TGTextEntry *fTxtChain; // chain name text entry - TGTextEntry *fTxtSelector; // selector name text entry - TGTextEntry *fTxtOptions; // options text entry - TGNumberEntry *fNumEntries; // number of entries selector - TGNumberEntry *fNumFirstEntry; // first entry selector - TGTextEntry *fTxtParFile; // parameter file name text entry - TGTextEntry *fTxtEventList; // event list text entry - TSessionViewer *fViewer; // pointer on main viewer - TQueryDescription *fQuery; // query description class - TObject *fChain; // actual TChain - -public: - TEditQueryFrame(TGWindow* p, Int_t w, Int_t h); - ~TEditQueryFrame() override; - void Build(TSessionViewer *gui); - void OnNewQueryMore(); - void OnBrowseChain(); - void OnBrowseSelector(); - void OnBrowseEventList(); - void OnBtnSave(); - void OnElementSelected(TObject *obj); - void SettingsChanged(); - void UpdateFields(TQueryDescription *desc); - - ClassDefOverride(TEditQueryFrame, 0) // Edit query frame -}; - -////////////////////////////////////////////////////////////////////////// -// // -// TSessionQueryFrame // -// A composite Frame used in the right part of the Session Viewer GUI // -// for any information, settings or controls relative to queries. // -// // -////////////////////////////////////////////////////////////////////////// - -class TSessionQueryFrame : public TGCompositeFrame { - -private: - - enum EQueryStatus { kRunning = 0, kDone, kStopped, kAborted }; - - TGTextButton *fBtnSubmit; // submit query button - TGTextButton *fBtnFinalize; // finalize query button - TGTextButton *fBtnStop; // stop process button - TGTextButton *fBtnAbort; // abort process button - TGTextButton *fBtnShowLog; // show log button - TGTextButton *fBtnRetrieve; // retrieve query button - TGTextButton *fBtnSave; // save query button - TGTextView *fInfoTextView; // summary on current query - - Bool_t fModified; // kTRUE if settings have changed - Int_t fFiles; // number of files processed - Long64_t fFirst; // first event/entry to process - Long64_t fEntries; // number of events/entries to process - Long64_t fPrevTotal; // used for progress bar - Long64_t fPrevProcessed; // used for progress bar - TGLabel *fLabInfos; // infos on current process - TGLabel *fLabStatus; // actual process status - TGLabel *fTotal; // total progress info - TGLabel *fRate; // rate of process in events/sec - EQueryStatus fStatus; // status of actual query - TGTab *fTab; // main tab frame - TGCompositeFrame *fFA, *fFB, *fFC; // three tabs element - TEditQueryFrame *fFD; // fourth tab element (edit query frame) - TGHProgressBar *frmProg; // current process progress bar - TRootEmbeddedCanvas *fECanvas; // node statistics embeded canvas - TCanvas *fStatsCanvas; // node statistics canvas - TSessionViewer *fViewer; // pointer on main viewer - TQueryDescription *fDesc; // query description - -public: - TSessionQueryFrame(TGWindow* parent, Int_t w, Int_t h); - ~TSessionQueryFrame() override; - - void Build(TSessionViewer *gui); - - TCanvas *GetStatsCanvas() const { return fStatsCanvas; } - TEditQueryFrame *GetQueryEditFrame() const { return fFD; } - TGTab *GetTab() const { return fTab; } - - void Feedback(TList *objs); - void Modified(Bool_t mod = kTRUE); - void Progress(Long64_t total, Long64_t processed); - void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti, Int_t actw, Int_t tses, Float_t eses); - void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) { - Progress(total, processed, bytesread, initTime, procTime, - evtrti, mbrti, -1, -1, -1.); } - void ProgressLocal(Long64_t total, Long64_t processed); - void IndicateStop(Bool_t aborted); - void ResetProgressDialog(const char *selec, Int_t files, Long64_t first, Long64_t entries); - - //Function that handle input from user: - void OnBtnSubmit(); - void OnBtnFinalize(); - void OnBtnStop(); - void OnBtnAbort(); - void OnBtnShowLog(); - void OnBtnRetrieve(); - void UpdateInfos(); - void UpdateButtons(TQueryDescription *desc); - void UpdateHistos(TList *objs); - - ClassDefOverride(TSessionQueryFrame, 0) // Query frame -}; - -////////////////////////////////////////////////////////////////////////// -// // -// TSessionOutputFrame // -// A composite Frame used in the right part of the Session Viewer GUI // -// displaying output list objects coming from query result. // -// // -////////////////////////////////////////////////////////////////////////// - -class TSessionOutputFrame : public TGCompositeFrame { - -private: - TGLVEntry *fEntryTmp; // used to transfer to feedback - TGLVContainer *fLVContainer; // output list view - TSessionViewer *fViewer; // pointer on the main viewer - -public: - TSessionOutputFrame(TGWindow* parent, Int_t w, Int_t h); - ~TSessionOutputFrame() override; - - void AddObject(TObject *obj); - void Build(TSessionViewer *gui); - TGLVContainer *GetLVContainer() { return fLVContainer; } - void OnElementClicked(TGLVEntry* entry, Int_t btn, Int_t x, Int_t y); - void OnElementDblClicked(TGLVEntry *entry ,Int_t btn, Int_t x, Int_t y); - void RemoveAll() override { fLVContainer->RemoveAll(); } - - ClassDefOverride(TSessionOutputFrame, 0) // Output frame -}; - -////////////////////////////////////////////////////////////////////////// -// // -// TSessionInputFrame // -// A composite Frame used in the right part of the Session Viewer GUI // -// displaying input list objects coming from query result. // -// // -////////////////////////////////////////////////////////////////////////// - -class TSessionInputFrame : public TGCompositeFrame { - -private: - TSessionViewer *fViewer; // pointer on the main viewer - TGLVContainer *fLVContainer; // container for the input list view - -public: - TSessionInputFrame(TGWindow* parent, Int_t w, Int_t h); - ~TSessionInputFrame() override; - - void AddObject(TObject *obj); - void Build(TSessionViewer *gui); - void RemoveAll() override { fLVContainer->RemoveAll(); } - TGLVContainer *GetLVContainer() { return fLVContainer; } - - ClassDefOverride(TSessionInputFrame, 0) // Input frame -}; - - -////////////////////////////////////////////////////////////////////////// -// // -// TSessionViewer // -// This is the main widget, mother of all the previous classes // -// Used to manage sessions, servers, queries... // -// // -////////////////////////////////////////////////////////////////////////// - -class TSessionViewer : public TGMainFrame { - -private: - time_t fStart, fElapsed; // time of connection - Bool_t fChangePic; // KTRUE if animation active - Bool_t fBusy; // KTRUE if busy i.e : connecting - TGHorizontalFrame *fHf; // - TGVerticalFrame *fV1; // - TGVerticalFrame *fV2; // - TSessionServerFrame *fServerFrame; // right side server frame - TSessionFrame *fSessionFrame; // right side session frame - TSessionQueryFrame *fQueryFrame; // right side query frame - TSessionOutputFrame *fOutputFrame; // output frame - TSessionInputFrame *fInputFrame; // input frame - TSessionLogView *fLogWindow; // external log window - TSessionDescription *fActDesc; // actual session description - TList *fSessions; // list of sessions - const TGPicture *fLocal; // local session icon picture - const TGPicture *fProofCon; // connected server icon picture - const TGPicture *fProofDiscon; // disconnected server icon picture - const TGPicture *fQueryCon; // connected(?) query icon picture - const TGPicture *fQueryDiscon; // disconnected(?) query icon picture - const TGPicture *fBaseIcon; // base list tree icon picture - - TGFrame *fActFrame; // actual (displayed) frame - TGToolBar *fToolBar; // application tool bar - TGMenuBar *fMenuBar; // application main menu bar - TGPopupMenu *fFileMenu; // file menu entry - TGPopupMenu *fSessionMenu; // session menu entry - TGPopupMenu *fQueryMenu; // query menu entry - TGPopupMenu *fOptionsMenu; // options menu entry - TGPopupMenu *fCascadeMenu; // options menu entry - TGPopupMenu *fHelpMenu; // help menu entry - - TGPopupMenu *fPopupSrv; // server related popup menu - TGPopupMenu *fPopupQry; // query related popup menu - TContextMenu *fContextMenu; // input/output objects context menu - - TGHProgressBar *fConnectProg; // connection progress bar - TGCanvas *fTreeView; // main right sessions/queries tree view - TGListTree *fSessionHierarchy; // main sessions/queries hierarchy list tree - TGListTreeItem *fSessionItem; // base (main) session list tree item - TGStatusBar *fStatusBar; // bottom status bar - TGPicture *fRightIconPicture; // lower bottom left icon used to show connection status - TGIcon *fRightIcon; // associated picture - TTimer *fTimer; // timer used to change icon picture - UserGroup_t *fUserGroup; // user connected to session - Bool_t fAutoSave; // kTRUE if config is to be saved on exit - TString fConfigFile; // configuration file name - TEnv *fViewerEnv; // viewer's configuration - -public: - - TSessionViewer(const char *title = "ROOT Session Viewer", UInt_t w = 550, UInt_t h = 320); - TSessionViewer(const char *title, Int_t x, Int_t y, UInt_t w, UInt_t h); - ~TSessionViewer() override; - virtual void Build(); - Bool_t ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t) override; - - TSessionServerFrame *GetServerFrame() const { return fServerFrame; } - TSessionFrame *GetSessionFrame() const { return fSessionFrame; } - TSessionQueryFrame *GetQueryFrame() const { return fQueryFrame; } - TSessionOutputFrame *GetOutputFrame() const { return fOutputFrame; } - TSessionInputFrame *GetInputFrame() const { return fInputFrame; } - TSessionDescription *GetActDesc() const { return fActDesc; } - TList *GetSessions() const { return fSessions; } - TGListTree *GetSessionHierarchy() const { return fSessionHierarchy; } - TGListTreeItem *GetSessionItem() const { return fSessionItem; } - const TGPicture *GetLocalPict() const { return fLocal; } - const TGPicture *GetProofConPict() const { return fProofCon; } - const TGPicture *GetProofDisconPict() const { return fProofDiscon; } - const TGPicture *GetQueryConPict() const { return fQueryCon; } - const TGPicture *GetQueryDisconPict() const { return fQueryDiscon; } - const TGPicture *GetBasePict() const { return fBaseIcon; } - TGPopupMenu *GetPopupSrv() const { return fPopupSrv; } - TGPopupMenu *GetPopupQry() const { return fPopupQry; } - TContextMenu *GetContextMenu() const { return fContextMenu; } - TGStatusBar *GetStatusBar() const { return fStatusBar; } - TGHProgressBar *GetConnectProg() const { return fConnectProg; } - TGPopupMenu *GetCascadeMenu() const { return fCascadeMenu; } - TGPopupMenu *GetOptionsMenu() const { return fOptionsMenu; } - - void ChangeRightLogo(const char *name); - void CleanupSession(); - void CloseWindow() override; - void DisableTimer(); - void EditQuery(); - void EnableTimer(); - Bool_t HandleTimer(TTimer *) override; - Bool_t IsBusy() const { return fBusy; } - Bool_t IsAutoSave() const { return fAutoSave; } - void LogMessage(const char *msg, Bool_t all); - void MyHandleMenu(Int_t); - void OnCascadeMenu(); - void OnListTreeClicked(TGListTreeItem *entry, Int_t btn, Int_t x, Int_t y); - void OnListTreeDoubleClicked(TGListTreeItem *entry, Int_t btn); - void QueryResultReady(char *query); - void DeleteQuery(); - void ReadConfiguration(const char *filename = nullptr); - void ResetSession(); - void UpdateListOfProofs(); - void UpdateListOfSessions(); - void UpdateListOfPackages(); - void WriteConfiguration(const char *filename = nullptr); - void SetBusy(Bool_t busy = kTRUE) { fBusy = busy; } - void SetChangePic(Bool_t change) { fChangePic = change;} - void SetLogWindow(TSessionLogView *log) { fLogWindow = log; } - void ShowEnabledPackages(); - void ShowPackages(); - void ShowInfo(const char *txt); - void ShowLog(const char *queryref); - void ShowStatus(); - void StartupMessage(char *msg, Bool_t stat, Int_t curr, Int_t total); - void StartViewer(); - void Terminate(); - - ClassDefOverride(TSessionViewer, 0) // Session Viewer -}; - -R__EXTERN TSessionViewer *gSessionViewer; - -#endif diff --git a/gui/sessionviewer/src/TProofProgressDialog.cxx b/gui/sessionviewer/src/TProofProgressDialog.cxx deleted file mode 100644 index 0395186c4ee80..0000000000000 --- a/gui/sessionviewer/src/TProofProgressDialog.cxx +++ /dev/null @@ -1,1342 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Fons Rademakers 21/03/03 - -/************************************************************************* - * Copyright (C) 1995-2003, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -/** \class TProofProgressDialog - \ingroup sessionviewer - -This class provides a query progress bar. - -*/ - - -#include "TProofProgressDialog.h" -#include "TProofProgressLog.h" -#include "TProofProgressMemoryPlot.h" -#include "TEnv.h" -#include "TError.h" -#include "TGLabel.h" -#include "TGButton.h" -#include "TGTextEntry.h" -#include "TGProgressBar.h" -#include "TGSpeedo.h" -#include "TProof.h" -#include "TSlave.h" -#include "TSystem.h" -#include "TTimer.h" -#include "TGraph.h" -#include "TNtuple.h" -#include "TCanvas.h" -#include "TLine.h" -#include "TAxis.h" -#include "TPaveText.h" -#include "TMath.h" -#include "TH1F.h" -#include "THLimitsFinder.h" -#include "TVirtualX.h" - - -#ifdef PPD_SRV_NEWER -#undef PPD_SRV_NEWER -#endif -#define PPD_SRV_NEWER(v) (fProof && fProof->GetRemoteProtocol() > v) - -Bool_t TProofProgressDialog::fgKeepDefault = kTRUE; -Bool_t TProofProgressDialog::fgLogQueryDefault = kFALSE; -TString TProofProgressDialog::fgTextQueryDefault = "last"; - -//static const Int_t gSVNMemPlot = 25090; - -ClassImp(TProofProgressDialog); - -//////////////////////////////////////////////////////////////////////////////// -/// Create PROOF processing progress dialog. - -TProofProgressDialog::TProofProgressDialog(TProof *proof, const char *selector, - Int_t files, Long64_t first, - Long64_t entries) : fDialog(0), - fBar(0), fClose(0), fStop(0), fAbort(0), fAsyn(0), fLog(0), fRatePlot(0), - fMemPlot(0), fKeepToggle(0), fLogQueryToggle(0), fTextQuery(0), fEntry(0), - fTitleLab(0), fFilesEvents(0), fTimeLab(0), fProcessed(0), fEstim(0), - fTotal(0), fRate(0), fInit(0), fSelector(0), fSpeedo(0), fSmoothSpeedo(0) -{ - fProof = proof; - fFiles = files; - fFirst = first; - fEntries = entries; - fPrevProcessed = 0; - fPrevTotal = 0; - fLogWindow = 0; - fMemWindow = 0; - fStatus = kRunning; - fKeep = fgKeepDefault; - fLogQuery = fgLogQueryDefault; - fRatePoints = 0; - fRateGraph = 0; - fMBRtGraph = 0; - fActWGraph = 0; - fTotSGraph = 0; - fEffSGraph = 0; - fProcTime = 0.; - fInitTime = 0.; - fAvgRate = 0.; - fAvgMBRate = 0.; - fRightInfo = 0; - fSpeedoEnabled = kFALSE; - fSpeedo = 0; - fUpdtSpeedo = 0; - fSmoothSpeedo = 0; - - // Make sure we are attached to a good instance - if (!proof || !(proof->IsValid())) { - Error("TProofProgressDialog", "proof instance is invalid (%p, %s): protocol error?", - proof, (proof && !(proof->IsValid())) ? "invalid" : "undef"); - return; - } - - // Have to save this information here, in case gProof is dead when - // the logs are requested - fSessionUrl = (proof && proof->GetManager()) ? proof->GetManager()->GetUrl() : ""; - - if (PPD_SRV_NEWER(25)) { - fRatePoints = new TNtuple("RateNtuple","Rate progress info","tm:evr:mbr:act:tos:efs"); - } else if (PPD_SRV_NEWER(11)) { - fRatePoints = new TNtuple("RateNtuple","Rate progress info","tm:evr:mbr"); - } - - fDialog = new TGTransientFrame(0, 0, 10, 10); - fDialog->Connect("CloseWindow()", "TProofProgressDialog", this, "DoClose()"); - fDialog->DontCallClose(); - fDialog->SetCleanup(kDeepCleanup); - -//======================================================================================= - - TGHorizontalFrame *hf4 = new TGHorizontalFrame(fDialog, 100, 100); - - TGVerticalFrame *vf4 = new TGVerticalFrame(hf4, 100, 100); - - // Title label - TString buf; - buf.Form("Executing on PROOF cluster \"%s\" with %d parallel workers:", - fProof ? fProof->GetMaster() : "", - fProof ? fProof->GetParallel() : 0); - fTitleLab = new TGLabel(vf4, buf); - fTitleLab->SetTextJustify(kTextTop | kTextLeft); - vf4->AddFrame(fTitleLab, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 5, 0)); - buf.Form("Selector: %s", selector); - fSelector = new TGLabel(vf4, buf); - fSelector->SetTextJustify(kTextTop | kTextLeft); - vf4->AddFrame(fSelector, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 5, 0)); - buf.Form("%d files, number of events %lld, starting event %lld", - fFiles, fEntries, fFirst); - fFilesEvents = new TGLabel(vf4, buf); - fFilesEvents->SetTextJustify(kTextTop | kTextLeft); - vf4->AddFrame(fFilesEvents, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 5, 0)); - - // Progress bar - fBar = new TGHProgressBar(vf4, TGProgressBar::kFancy, 200); - fBar->SetBarColor("green"); - fBar->Percent(kTRUE); - fBar->ShowPos(kTRUE); - vf4->AddFrame(fBar, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 10, 10, 5, 5)); - - // Status labels - if (PPD_SRV_NEWER(11)) { - TGHorizontalFrame *hf0 = new TGHorizontalFrame(vf4, 0, 0); - TGCompositeFrame *cf0 = new TGCompositeFrame(hf0, 110, 0, kFixedWidth); - cf0->AddFrame(new TGLabel(cf0, "Initialization time:")); - hf0->AddFrame(cf0); - fInit = new TGLabel(hf0, "- secs"); - fInit->SetTextJustify(kTextTop | kTextLeft); - hf0->AddFrame(fInit, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 0, 0)); - vf4->AddFrame(hf0, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 5, 0)); - } - - TGHorizontalFrame *hf1 = new TGHorizontalFrame(vf4, 0, 0); - TGCompositeFrame *cf1 = new TGCompositeFrame(hf1, 110, 0, kFixedWidth); - fTimeLab = new TGLabel(cf1, "Estimated time left:"); - fTimeLab->SetTextJustify(kTextTop | kTextLeft); - cf1->AddFrame(fTimeLab, new TGLayoutHints(kLHintsLeft)); - hf1->AddFrame(cf1); - fEstim = new TGLabel(hf1, "- sec"); - fEstim->SetTextJustify(kTextTop | kTextLeft); - hf1->AddFrame(fEstim, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 0, 0)); - vf4->AddFrame(hf1, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 5, 0)); - - hf1 = new TGHorizontalFrame(vf4, 0, 0); - cf1 = new TGCompositeFrame(hf1, 110, 0, kFixedWidth); - fProcessed = new TGLabel(cf1, "Processing status:"); - fProcessed->SetTextJustify(kTextTop | kTextLeft); - cf1->AddFrame(fProcessed, new TGLayoutHints(kLHintsLeft)); - hf1->AddFrame(cf1); - fTotal= new TGLabel(hf1, "- / - events"); - fTotal->SetTextJustify(kTextTop | kTextLeft); - hf1->AddFrame(fTotal, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 0, 0)); - - vf4->AddFrame(hf1, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 5, 0)); - - TGHorizontalFrame *hf2 = new TGHorizontalFrame(vf4, 0, 0); - TGCompositeFrame *cf2 = new TGCompositeFrame(hf2, 110, 0, kFixedWidth); - cf2->AddFrame(new TGLabel(cf2, "Processing rate:")); - hf2->AddFrame(cf2); - fRate = new TGLabel(hf2, "- events/sec \n"); - fRate->SetTextJustify(kTextTop | kTextLeft); - hf2->AddFrame(fRate, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 0, 0)); - vf4->AddFrame(hf2, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 10, 10, 5, 0)); - - // Keep toggle button - fKeepToggle = new TGCheckButton(vf4, - new TGHotString("Close dialog when processing is complete")); - if (!fKeep) fKeepToggle->SetState(kButtonDown); - fKeepToggle->Connect("Toggled(Bool_t)", - "TProofProgressDialog", this, "DoKeep(Bool_t)"); - vf4->AddFrame(fKeepToggle, new TGLayoutHints(kLHintsBottom, 10, 10, 10, 5)); - - hf4->AddFrame(vf4, new TGLayoutHints(kLHintsExpandY | kLHintsExpandX)); - - TGVerticalFrame *vf51 = new TGVerticalFrame(hf4, 20, 20); - - Int_t enablespeedo = gEnv->GetValue("Proof.EnableSpeedo", 0); - if (enablespeedo) fSpeedoEnabled = kTRUE; - - fSpeedo = new TGSpeedo(vf51, 0.0, 1.0, "", " Ev/s"); - if (fSpeedoEnabled) { - fSpeedo->Connect("OdoClicked()", "TProofProgressDialog", this, "ToggleOdometerInfos()"); - fSpeedo->Connect("LedClicked()", "TProofProgressDialog", this, "ToggleThreshold()"); - } - vf51->AddFrame(fSpeedo); - fSpeedo->SetDisplayText("Init Time", "[ms]"); - fSpeedo->EnablePeakMark(); - fSpeedo->SetThresholds(0.0, 25.0, 50.0); - fSpeedo->SetThresholdColors(TGSpeedo::kRed, TGSpeedo::kOrange, TGSpeedo::kGreen); - fSpeedo->SetOdoValue(0); - fSpeedo->EnableMeanMark(); - - fSmoothSpeedo = new TGCheckButton(vf51, new TGHotString("Smooth speedometer update")); - if (fSpeedoEnabled) { - fSmoothSpeedo->SetState(kButtonDown); - fSmoothSpeedo->SetToolTipText("Control smoothness in refreshing the speedo"); - } else { - fSmoothSpeedo->SetToolTipText("Speedo refreshing is disabled"); - fSmoothSpeedo->SetState(kButtonDisabled); - } - vf51->AddFrame(fSmoothSpeedo, new TGLayoutHints(kLHintsBottom | kLHintsCenterX, 0, 0, 5, 0)); - - hf4->AddFrame(vf51, new TGLayoutHints(kLHintsBottom, 5, 5, 5, 5)); - - fDialog->AddFrame(hf4, new TGLayoutHints(kLHintsTop | kLHintsExpandX, 5, 5, 5, 5)); - -//========================================================================================== - - // Stop, cancel and close buttons - TGHorizontalFrame *hf3 = new TGHorizontalFrame(fDialog, 60, 20); - - fAsyn = new TGTextButton(hf3, "&Run in background"); - if (fProof->GetRemoteProtocol() >= 22 && fProof->IsSync()) { - fAsyn->SetToolTipText("Continue running in the background (asynchronous mode), releasing the ROOT prompt"); - } else { - fAsyn->SetToolTipText("Switch to asynchronous mode disabled: functionality not supported by the server"); - fAsyn->SetState(kButtonDisabled); - } - fAsyn->Connect("Clicked()", "TProofProgressDialog", this, "DoAsyn()"); - hf3->AddFrame(fAsyn, new TGLayoutHints(kLHintsCenterY | kLHintsExpandX, 7, 7, 0, 0)); - - fStop = new TGTextButton(hf3, "&Stop"); - fStop->SetToolTipText("Stop processing, Terminate() will be executed"); - fStop->Connect("Clicked()", "TProofProgressDialog", this, "DoStop()"); - hf3->AddFrame(fStop, new TGLayoutHints(kLHintsCenterY | kLHintsExpandX, 7, 7, 0, 0)); - - fAbort = new TGTextButton(hf3, "&Cancel"); - fAbort->SetToolTipText("Cancel processing, Terminate() will NOT be executed"); - fAbort->Connect("Clicked()", "TProofProgressDialog", this, "DoAbort()"); - hf3->AddFrame(fAbort, new TGLayoutHints(kLHintsCenterY | kLHintsExpandX, 7, 7, 0, 0)); - - fClose = new TGTextButton(hf3, "&Close"); - fClose->SetToolTipText("Close this dialog"); - fClose->SetState(kButtonDisabled); - fClose->Connect("Clicked()", "TProofProgressDialog", this, "DoClose()"); - hf3->AddFrame(fClose, new TGLayoutHints(kLHintsCenterY | kLHintsExpandX, 7, 7, 0, 0)); - - fDialog->AddFrame(hf3, new TGLayoutHints(kLHintsBottom | kLHintsCenterX | kLHintsExpandX, 5, 5, 5, 5)); - - TGHorizontalFrame *hf5 = new TGHorizontalFrame(fDialog, 60, 20); - - fLog = new TGTextButton(hf5, "&Show Logs"); - fLog->SetToolTipText("Show query log messages"); - fLog->Connect("Clicked()", "TProofProgressDialog", this, "DoLog()"); - hf5->AddFrame(fLog, new TGLayoutHints(kLHintsCenterY | kLHintsExpandX, 7, 7, 0, 0)); - - if (PPD_SRV_NEWER(11)) { - fRatePlot = new TGTextButton(hf5, "&Performance plot"); - fRatePlot->SetToolTipText("Show rates, chunck sizes, cluster activities ... vs time"); - fRatePlot->SetState(kButtonDisabled); - fRatePlot->Connect("Clicked()", "TProofProgressDialog", this, "DoPlotRateGraph()"); - hf5->AddFrame(fRatePlot, new TGLayoutHints(kLHintsCenterY | kLHintsExpandX, 7, 7, 0, 0)); - } - - fMemPlot = new TGTextButton(hf5, "&Memory Plot"); - fMemPlot->Connect("Clicked()", "TProofProgressDialog", this, "DoMemoryPlot()"); - fMemPlot->SetToolTipText("Show memory consumption vs entry / merging phase"); - hf5->AddFrame(fMemPlot, new TGLayoutHints(kLHintsCenterY | kLHintsExpandX, 7, 7, 0, 0)); - - fUpdtSpeedo = new TGTextButton(hf5, "&Enable speedometer"); - fUpdtSpeedo->Connect("Clicked()", "TProofProgressDialog", this, "DoEnableSpeedo()"); - if (fSpeedoEnabled) { - fUpdtSpeedo->ChangeText("&Disable speedometer"); - fUpdtSpeedo->SetToolTipText("Disable speedometer"); - } else { - fUpdtSpeedo->ChangeText("&Enable speedometer"); - fUpdtSpeedo->SetToolTipText("Enable speedometer (may have an impact on performance)"); - } - hf5->AddFrame(fUpdtSpeedo, new TGLayoutHints(kLHintsCenterY | kLHintsExpandX, 7, 7, 0, 0)); - - fDialog->AddFrame(hf5, new TGLayoutHints(kLHintsBottom | kLHintsCenterX | kLHintsExpandX, 5, 5, 5, 5)); - - // Only enable if master supports it - if (!PPD_SRV_NEWER(18)) { - fMemPlot->SetState(kButtonDisabled); - TString tip = TString::Format("Not supported by the master: required protocol 19 > %d", - (fProof ? fProof->GetRemoteProtocol() : -1)); - fMemPlot->SetToolTipText(tip.Data()); - } else { - fMemPlot->SetToolTipText("Show memory consumption"); - } - - // Connect slot to proof progress signal - if (fProof) { - fProof->Connect("Progress(Long64_t,Long64_t)", "TProofProgressDialog", - this, "Progress(Long64_t,Long64_t)"); - fProof->Connect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - "TProofProgressDialog", this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - fProof->Connect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)", - "TProofProgressDialog", this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)"); - fProof->Connect("StopProcess(Bool_t)", "TProofProgressDialog", this, - "IndicateStop(Bool_t)"); - fProof->Connect("ResetProgressDialog(const char*,Int_t,Long64_t,Long64_t)", - "TProofProgressDialog", this, - "ResetProgressDialog(const char*,Int_t,Long64_t,Long64_t)"); - fProof->Connect("CloseProgressDialog()", "TProofProgressDialog", this, "DoClose()"); - fProof->Connect("DisableGoAsyn()", "TProofProgressDialog", this, "DisableAsyn()"); - } - - // Set dialog title - if (fProof) { - if (strlen(fProof->GetUser()) > 0) - fDialog->SetWindowName(Form("PROOF Query Progress: %s@%s", - fProof->GetUser(), fProof->GetMaster())); - else - fDialog->SetWindowName(Form("PROOF Query Progress: %s", fProof->GetMaster())); - } else - fDialog->SetWindowName("PROOF Query Progress: "); - - // Map all widgets and calculate size of dialog - fDialog->MapSubwindows(); - - fDialog->Resize(fDialog->GetDefaultSize()); - - const TGWindow *main = gClient->GetRoot(); - // Position relative to the parent window (which is the root window) - Window_t wdum; - int ax, ay; - Int_t mw = ((TGFrame *) main)->GetWidth(); - Int_t mh = ((TGFrame *) main)->GetHeight(); - Int_t width = fDialog->GetDefaultWidth(); - Int_t height = fDialog->GetDefaultHeight(); - - gVirtualX->TranslateCoordinates(main->GetId(), main->GetId(), - (mw - width), (mh - height) >> 1, ax, ay, wdum); - - // Make the message box non-resizable - fDialog->SetWMSize(width, height); - fDialog->SetWMSizeHints(width, height, width, height, 0, 0); - - fDialog->SetMWMHints(kMWMDecorAll | kMWMDecorResizeH | kMWMDecorMaximize | - kMWMDecorMinimize | kMWMDecorMenu, - kMWMFuncAll | kMWMFuncResize | kMWMFuncMaximize | - kMWMFuncMinimize, - kMWMInputModeless); - - fDialog->Move(ax-10, ay - mh/4); - fDialog->SetWMPosition(ax-10, ay - mh/4); - // Popup dialog and wait till user replies - fDialog->MapWindow(); - - fStartTime = gSystem->Now(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Toggle information displayed in Analog Meter - -void TProofProgressDialog::ToggleOdometerInfos() -{ - if (fRightInfo < 1) - fRightInfo++; - else - fRightInfo = 0; - if (fRightInfo == 0) { - fSpeedo->SetDisplayText("Init Time", "[ms]"); - fSpeedo->SetOdoValue((Int_t)(fInitTime * 1000.0)); - } - else if (fRightInfo == 1) { - fSpeedo->SetDisplayText("Proc Time", "[ms]"); - fSpeedo->SetOdoValue((Int_t)(fProcTime * 1000.0)); - } -} - -//////////////////////////////////////////////////////////////////////////////// - -void TProofProgressDialog::ToggleThreshold() -{ - if (fSpeedo->IsThresholdActive()) { - fSpeedo->DisableThreshold(); - fSpeedo->Glow(TGSpeedo::kNoglow); - } - else - fSpeedo->EnableThreshold(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset dialog box preparing for new query - -void TProofProgressDialog::ResetProgressDialog(const char *selec, - Int_t files, Long64_t first, - Long64_t entries) -{ - TString buf; - - // Update title - buf.Form("Executing on PROOF cluster \"%s\" with %d parallel workers:", - fProof ? fProof->GetMaster() : "", - fProof ? fProof->GetParallel() : 0); - fTitleLab->SetText(buf); - - // Reset members - fFiles = files; - fFirst = first; - fEntries = entries; - fPrevProcessed = 0; - fPrevTotal = 0; - fStatus = kRunning; - - // Update selector name - buf.Form("Selector: %s", selec); - fSelector->SetText(buf); - - // Reset 'estim' and 'processed' text - fTimeLab->SetText("Estimated time left:"); - fProcessed->SetText("Processing status:"); - - // Update numbers - buf.Form("%d files, number of events %lld, starting event %lld", - fFiles, fEntries, fFirst); - fFilesEvents->SetText(buf); - - // Reset progress bar - fBar->SetBarColor("green"); - fBar->Reset(); - - // Reset speedo - fSpeedo->SetMinMaxScale(0.0, 1.0); - fSpeedo->SetMeanValue(0.0); - fSpeedo->ResetPeakVal(); - - // Reset buttons - fStop->SetState(kButtonUp); - fAbort->SetState(kButtonUp); - fClose->SetState(kButtonDisabled); - if (fProof && fProof->IsSync() && fProof->GetRemoteProtocol() >= 22) { - fAsyn->SetState(kButtonUp); - } else { - fAsyn->SetState(kButtonDisabled); - } - - // Reconnect the slots - if (fProof) { - fProof->Connect("Progress(Long64_t,Long64_t)", "TProofProgressDialog", - this, "Progress(Long64_t,Long64_t)"); - fProof->Connect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - "TProofProgressDialog", this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - fProof->Connect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)", - "TProofProgressDialog", this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)"); - fProof->Connect("StopProcess(Bool_t)", "TProofProgressDialog", this, - "IndicateStop(Bool_t)"); - fProof->Connect("DisableGoAsyn()", "TProofProgressDialog", this, "DisableAsyn()"); - } - - // Reset start time - fStartTime = gSystem->Now(); - - // Clear the list of performances points - if (PPD_SRV_NEWER(11)) - fRatePoints->Reset(); - SafeDelete(fRateGraph); - SafeDelete(fMBRtGraph); - SafeDelete(fActWGraph); - SafeDelete(fTotSGraph); - SafeDelete(fEffSGraph); - fAvgRate = 0.; - fAvgMBRate = 0.; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update progress bar and status labels. -/// Use "processed == total" or "processed < 0" to indicate end of processing. - -void TProofProgressDialog::Progress(Long64_t total, Long64_t processed) -{ - Long_t tt; - UInt_t hh=0, mm=0, ss=0; - TString buf; - TString stm; - static const char *cproc[] = { "running", "done", - "STOPPED", "ABORTED", "***EVENTS SKIPPED***"}; - - // Update title - buf.Form("Executing on PROOF cluster \"%s\" with %d parallel workers:", - fProof ? fProof->GetMaster() : "", - fProof ? fProof->GetParallel() : 0); - fTitleLab->SetText(buf); - - if (total < 0) - total = fPrevTotal; - else - fPrevTotal = total; - - // Nothing to update - if (fPrevProcessed == processed) - return; - - // Number of processed events - Long64_t evproc = (processed >= 0) ? processed : fPrevProcessed; - - if (fEntries != total) { - fEntries = total; - buf.Form("%d files, number of events %lld, starting event %lld", - fFiles, fEntries, fFirst); - fFilesEvents->SetText(buf); - } - - // Update position - Float_t pos = Float_t(Double_t(evproc * 100)/Double_t(total)); - fBar->SetPosition(pos); - - // get current time - fEndTime = gSystem->Now(); - TTime tdiff = fEndTime - fStartTime; - Float_t eta = 0; - if (evproc > 0) - eta = ((Float_t)((Long64_t)tdiff)*total/Float_t(evproc) - Long64_t(tdiff))/1000.; - - if (processed >= 0 && processed >= total) { - tt = (Long_t)Long64_t(tdiff)/1000; - if (tt > 0) { - hh = (UInt_t)(tt / 3600); - mm = (UInt_t)((tt % 3600) / 60); - ss = (UInt_t)((tt % 3600) % 60); - } - if (hh) - stm.Form("%d h %d min %d sec", hh, mm, ss); - else if (mm) - stm.Form("%d min %d sec", mm, ss); - else - stm.Form("%d sec", ss); - fProcessed->SetText("Processed:"); - buf.Form("%lld events in %s\n", total, stm.Data()); - fTotal->SetText(buf); - - fEstim->SetText("0 sec"); - - if (fProof) { - fProof->Disconnect("Progress(Long64_t,Long64_t)", this, - "Progress(Long64_t,Long64_t)"); - fProof->Disconnect("StopProcess(Bool_t)", this, - "IndicateStop(Bool_t)"); - fProof->Disconnect("DisableGoAsyn()", this, "DisableAsyn()"); - } - - // Set button state - fAsyn->SetState(kButtonDisabled); - fStop->SetState(kButtonDisabled); - fAbort->SetState(kButtonDisabled); - fClose->SetState(kButtonUp); - if (!fKeep) DoClose(); - - // Set the status to done - fStatus = kDone; - - } else { - // A negative value for process indicates that we are finished, - // no matter whether the processing was complete - Bool_t incomplete = (processed < 0 && - (fPrevProcessed < total || fPrevProcessed == 0)) - ? kTRUE : kFALSE; - if (incomplete) { - fStatus = kIncomplete; - // We use a different color to highlight incompletion - fBar->SetBarColor("magenta"); - } - tt = (Long_t)eta; - if (tt > 0) { - hh = (UInt_t)(tt / 3600); - mm = (UInt_t)((tt % 3600) / 60); - ss = (UInt_t)((tt % 3600) % 60); - } - if (hh) - stm.Form("%d h %d min %d sec", hh, mm, ss); - else if (mm) - stm.Form("%d min %d sec", mm, ss); - else - stm.Form("%d sec", ss); - - fEstim->SetText(stm.Data()); - buf.Form("%lld / %lld events", evproc, total); - if (fStatus > kDone) { - buf += TString::Format(" - %s", cproc[fStatus]); - } - fTotal->SetText(buf); - - buf.Form("%.1f events/sec\n", Float_t(evproc)/Long64_t(tdiff)*1000.); - fRate->SetText(buf); - - if (processed < 0) { - // And we disable the buttons - fAsyn->SetState(kButtonDisabled); - fStop->SetState(kButtonDisabled); - fAbort->SetState(kButtonDisabled); - fClose->SetState(kButtonUp); - - // Set the status to done - fStatus = kDone; - } - } - fPrevProcessed = evproc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update progress bar and status labels. -/// Use "processed == total" or "processed < 0" to indicate end of processing. - -void TProofProgressDialog::Progress(Long64_t total, Long64_t processed, - Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti, - Int_t actw, Int_t tses, Float_t eses) -{ - Double_t BinLow, BinHigh; - Int_t nbins; - Long_t tt; - UInt_t hh=0, mm=0, ss=0; - TString buf; - TString stm; - static const char *cproc[] = { "running", "done", - "STOPPED", "ABORTED", "***EVENTS SKIPPED***"}; - - // Update title - buf.Form("Executing on PROOF cluster \"%s\" with %d parallel workers:", - fProof ? fProof->GetMaster() : "", - fProof ? fProof->GetParallel() : 0); - fTitleLab->SetText(buf); - - if (gDebug > 1) - Info("Progress","t: %lld, p: %lld, itm: %f, ptm: %f", total, processed, initTime, procTime); - - if (initTime >= 0.) { - // Set init time - fInitTime = initTime; - buf.Form("%.1f secs", initTime); - fInit->SetText(buf); - if (fSpeedoEnabled && fRightInfo == 0) - fSpeedo->SetOdoValue((Int_t)(fInitTime * 1000.0)); - } - - Bool_t over = kFALSE; - if (total < 0) { - total = fPrevTotal; - over = kTRUE; - } else { - fPrevTotal = total; - } - - // Show proc time by default when switching from init to proc - if (processed > 0 && fPrevProcessed <= 0) - while (fRightInfo != 1) - ToggleOdometerInfos(); - - // Nothing to update - if (fPrevProcessed == processed) - return; - - // Number of processed events - Long64_t evproc = (processed >= 0) ? processed : fPrevProcessed; - Float_t mbsproc = bytesread / TMath::Power(2.,20.); - - if (fEntries != total) { - fEntries = total; - buf.Form("%d files, number of events %lld, starting event %lld", - fFiles, fEntries, fFirst); - fFilesEvents->SetText(buf); - } - - // Update position - Float_t pos = Float_t(Double_t(evproc * 100)/Double_t(total)); - fBar->SetPosition(pos); - - Float_t eta = 0; - if (evproc > 0 && procTime > 0.) - eta = (Float_t) (total - evproc) / (Double_t)evproc * procTime; - - // Update average rates - if (procTime > 0.) { - fProcTime = procTime; - fAvgRate = Float_t(evproc) / procTime; - fAvgMBRate = mbsproc / procTime; - } - - if (fSpeedoEnabled) { - if (fRightInfo == 0) - fSpeedo->SetOdoValue((Int_t)(fInitTime * 1000.0)); - else if (fRightInfo == 1) - fSpeedo->SetOdoValue((Int_t)(fProcTime * 1000.0)); - } - - if (over || (processed >= 0 && processed >= total)) { - - // A negative value for process indicates that we are finished, - // no matter whether the processing was complete - Bool_t incomplete = (processed < 0 && - (fPrevProcessed < total || fPrevProcessed == 0)) - ? kTRUE : kFALSE; - TString st = ""; - if (incomplete) { - fStatus = kIncomplete; - // We use a different color to highlight incompletion - fBar->SetBarColor("magenta"); - st = TString::Format(" %s", cproc[fStatus]); - } - - tt = (Long_t)fProcTime; - if (tt > 0) { - hh = (UInt_t)(tt / 3600); - mm = (UInt_t)((tt % 3600) / 60); - ss = (UInt_t)((tt % 3600) % 60); - } - if (hh) - stm.Form("%d h %d min %d sec", hh, mm, ss); - else if (mm) - stm.Form("%d min %d sec", mm, ss); - else - stm.Form("%d sec", ss); - fProcessed->SetText("Processed:"); - TString sf("MB"); - Float_t xb = fAvgMBRate*fProcTime; - xb = AdjustBytes(xb, sf); - buf.Form("%lld events (%.2f %s)\n", - std::max(fPrevProcessed, processed), xb, sf.Data()); - fTotal->SetText(buf); - buf.Form("%s %s\n", stm.Data(), st.Data()); - fTimeLab->SetText("Processing time:"); - fEstim->SetText(buf); - buf.Form("%.1f evts/sec (%.1f MB/sec)\n", fAvgRate, fAvgMBRate); - fRate->SetText(buf); - // Fill rate graph - Bool_t useAvg = gEnv->GetValue("Proof.RatePlotUseAvg", 0); - if (useAvg) { - if (fAvgRate > 0.) { - fRatePoints->Fill(procTime, fAvgRate, fAvgMBRate); - fRatePlot->SetState(kButtonUp); - } - } else { - if (evtrti > 0.) { - fRatePoints->Fill(procTime, evtrti, mbrti, (Float_t)actw, (Float_t)tses, eses); - fRatePlot->SetState(kButtonUp); - } - } - - if (fProof) { - fProof->Disconnect("Progress(Long64_t,Long64_t)", this, - "Progress(Long64_t,Long64_t)"); - fProof->Disconnect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - fProof->Disconnect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)", - this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)"); - fProof->Disconnect("StopProcess(Bool_t)", this, "IndicateStop(Bool_t)"); - fProof->Disconnect("DisableGoAsyn()", this, "DisableAsyn()"); - } - - // Set button state - fAsyn->SetState(kButtonDisabled); - fStop->SetState(kButtonDisabled); - fAbort->SetState(kButtonDisabled); - fClose->SetState(kButtonUp); - - if (fSmoothSpeedo->GetState() == kButtonDown) - fSpeedo->SetScaleValue(0.0, 0); - else - fSpeedo->SetScaleValue(0.0); - fSpeedo->Glow(TGSpeedo::kNoglow); - - if (!fKeep) DoClose(); - - // Set the status to done - fStatus = kDone; - - } else { - // A negative value for process indicates that we are finished, - // no matter whether the processing was complete - Bool_t incomplete = (processed < 0 && - (fPrevProcessed < total || fPrevProcessed == 0)) - ? kTRUE : kFALSE; - if (incomplete) { - fStatus = kIncomplete; - // We use a different color to highlight incompletion - fBar->SetBarColor("magenta"); - } - tt = (Long_t)eta; - if (tt > 0) { - hh = (UInt_t)(tt / 3600); - mm = (UInt_t)((tt % 3600) / 60); - ss = (UInt_t)((tt % 3600) % 60); - } - if (hh) - stm.Form("%d h %d min %d sec", hh, mm, ss); - else if (mm) - stm.Form("%d min %d sec", mm, ss); - else - stm.Form("%d sec", ss); - - fEstim->SetText(stm.Data()); - TString sf("MB"); - Float_t xb = AdjustBytes(mbsproc, sf); - buf.Form("%lld / %lld events - %.2f %s", evproc, total, xb, sf.Data()); - if (fStatus > kDone) { - buf += TString::Format(" - %s", cproc[fStatus]); - } - fTotal->SetText(buf); - - // Post - if (evtrti > 0.) { - buf.Form("%.1f evts/sec \navg: %.1f evts/sec (%.1f MB/sec)", - evtrti, fAvgRate, fAvgMBRate); - fRatePoints->Fill(procTime, evtrti, mbrti, (Float_t)actw, (Float_t)tses, eses); - fRatePlot->SetState(kButtonUp); - if (fSpeedoEnabled) { - if (evtrti > fSpeedo->GetScaleMax()) { - nbins = 4; - BinLow = fSpeedo->GetScaleMin(); - BinHigh = 1.5 * evtrti; - THLimitsFinder::OptimizeLimits(4, nbins, BinLow, BinHigh, kFALSE); - fSpeedo->SetMinMaxScale(fSpeedo->GetScaleMin(), BinHigh); - } - if (fSmoothSpeedo->GetState() == kButtonDown) - fSpeedo->SetScaleValue(evtrti, 0); - else - fSpeedo->SetScaleValue(evtrti); - fSpeedo->SetMeanValue(fAvgRate); - } - } else { - buf.Form("avg: %.1f evts/sec (%.1f MB/sec)", fAvgRate, fAvgMBRate); - } - fRate->SetText(buf); - - if (processed < 0) { - // And we disable the buttons - fAsyn->SetState(kButtonDisabled); - fStop->SetState(kButtonDisabled); - fAbort->SetState(kButtonDisabled); - fClose->SetState(kButtonUp); - - if (fSpeedoEnabled) { - if (fSmoothSpeedo->GetState() == kButtonDown) - fSpeedo->SetScaleValue(0.0, 0); - else - fSpeedo->SetScaleValue(0.0); - fSpeedo->Glow(TGSpeedo::kNoglow); - } - - // Set the status to done - fStatus = kDone; - } - } - fPrevProcessed = evproc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Transform MBs to GBs ot TBs and get the correct suffix - -Float_t TProofProgressDialog::AdjustBytes(Float_t mbs, TString &sf) -{ - Float_t xb = mbs; - sf = "MB"; - if (xb > 1024.) { - xb = xb / 1024.; - sf = "GB"; - } - if (xb > 1024.) { - xb = xb / 1024.; - sf = "TB"; - } - // Done - return xb; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup dialog. - -TProofProgressDialog::~TProofProgressDialog() -{ - if (fProof) { - fProof->Disconnect("Progress(Long64_t,Long64_t)", this, - "Progress(Long64_t,Long64_t)"); - fProof->Disconnect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - fProof->Disconnect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)", - this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)"); - fProof->Disconnect("StopProcess(Bool_t)", this, "IndicateStop(Bool_t)"); - fProof->Disconnect("DisableGoAsyn()", this, "DisableAsyn()"); - fProof->Disconnect("ResetProgressDialog(const char*,Int_t,Long64_t,Long64_t)", - this, - "ResetProgressDialog(const char*,Int_t,Long64_t,Long64_t)"); - fProof->Disconnect("CloseProgressDialog()", this, "CloseProgressDialog()"); - fProof->ResetProgressDialogStatus(); - // We are called after a TProofDetach: we delete the instance - if (!fProof->IsValid()) - SafeDelete(fProof); - } - if (fLogWindow) - delete fLogWindow; - if (fMemWindow) - delete fMemWindow; - fDialog->Cleanup(); - delete fDialog; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Called when dialog is closed. - -void TProofProgressDialog::CloseWindow() -{ - delete this; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Disable the asyn switch when an external request for going asynchronous is issued - -void TProofProgressDialog::DisableAsyn() -{ - fProof->Disconnect("DisableGoAsyn()", this, "DisableAsyn()"); - fAsyn->SetState(kButtonDisabled); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Indicate that Cancel or Stop was clicked. - -void TProofProgressDialog::IndicateStop(Bool_t aborted) -{ - if (aborted == kTRUE) - fBar->SetBarColor("red"); - else - fBar->SetBarColor("yellow"); - - if (fProof) { - fProof->Disconnect("Progress(Long64_t,Long64_t)", this, - "Progress(Long64_t,Long64_t)"); - fProof->Disconnect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - fProof->Disconnect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)", - this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)"); - fProof->Disconnect("StopProcess(Bool_t)", this, "IndicateStop(Bool_t)"); - fProof->Disconnect("DisableGoAsyn()", this, "DisableAsyn()"); - // These buttons are meaningless at this point - fAsyn->SetState(kButtonDisabled); - fStop->SetState(kButtonDisabled); - fAbort->SetState(kButtonDisabled); - } - - fClose->SetState(kButtonUp); - if (!fKeep) - DoClose(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load/append a log msg in the log frame, if open - -void TProofProgressDialog::LogMessage(const char *msg, Bool_t all) -{ - if (fLogWindow) { - if (all) { - // load buffer - fLogWindow->LoadBuffer(msg); - } else { - // append - fLogWindow->AddBuffer(msg); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close dialog. - -void TProofProgressDialog::DoClose() -{ - fClose->SetState(kButtonDisabled); - TTimer::SingleShot(50, "TProofProgressDialog", this, "CloseWindow()"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ask proof session for logs - -void TProofProgressDialog::DoLog() -{ - if (fProof) { - if (!fLogWindow) { - fLogWindow = new TProofProgressLog(this); - if (!fLogWindow->TestBit(TObject::kInvalidObject)) - fLogWindow->DoLog(); - } else { - // Clear window - if (!fLogWindow->TestBit(TObject::kInvalidObject)) { - fLogWindow->Clear(); - fLogWindow->DoLog(); - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle keep toggle button. - -void TProofProgressDialog::DoKeep(Bool_t) -{ - fKeep = !fKeep; - - // Last choice will be the default for the future - fgKeepDefault = fKeep; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle log-current-query-only toggle button. - -void TProofProgressDialog::DoSetLogQuery(Bool_t) -{ - fLogQuery = !fLogQuery; - fEntry->SetEnabled(fLogQuery); - if (fLogQuery) - fEntry->SetToolTipText("Enter the query number ('last' for the last query)",50); - else - fEntry->SetToolTipText(0); - - // Last choice will be the default for the future - fgLogQueryDefault = fLogQuery; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle Stop button. - -void TProofProgressDialog::DoStop() -{ - // Do not wait for ever, but al least 10 seconds - Long_t timeout = gEnv->GetValue("Proof.ShutdownTimeout", 60) / 2; - timeout = (timeout > 10) ? timeout : 10; - fProof->StopProcess(kFALSE, timeout); - fStatus = kStopped; - - // Set buttons states - fAsyn->SetState(kButtonDisabled); - fStop->SetState(kButtonDisabled); - fAbort->SetState(kButtonDisabled); - fClose->SetState(kButtonUp); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle Cancel button. - -void TProofProgressDialog::DoAbort() -{ - fProof->StopProcess(kTRUE); - fStatus = kAborted; - - // Set buttons states - fAsyn->SetState(kButtonDisabled); - fStop->SetState(kButtonDisabled); - fAbort->SetState(kButtonDisabled); - fClose->SetState(kButtonUp); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle Asyn button. - -void TProofProgressDialog::DoAsyn() -{ - fProof->GoAsynchronous(); - - // Set buttons states - fAsyn->SetState(kButtonDisabled); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle Plot Rate Graph. - -void TProofProgressDialog::DoPlotRateGraph() -{ - // We must have some point to plot - if (!fRatePoints || fRatePoints->GetEntries() <= 0) { - Info("DoPlotRateGraph","list is empty!"); - return; - } - - // Fill the graphs - Int_t np = (Int_t)fRatePoints->GetEntries(); - Double_t eymx = -1., bymx = -1., wymx = -1., tymx=-1., symx = -1.; - SafeDelete(fRateGraph); - SafeDelete(fMBRtGraph); - SafeDelete(fActWGraph); - SafeDelete(fTotSGraph); - SafeDelete(fEffSGraph); - fRateGraph = new TGraph(np); - fMBRtGraph = new TGraph(np); - if (PPD_SRV_NEWER(25)) { - fActWGraph = new TGraph(np); - fTotSGraph = new TGraph(np); - fEffSGraph = new TGraph(np); - } - Float_t *nar = fRatePoints->GetArgs(); - Int_t ii = 0; - for ( ; ii < np; ++ii) { - fRatePoints->GetEntry(ii); - if (!(nar[1] > 0.)) continue; - // Evts/s - fRateGraph->SetPoint(ii, (Double_t) nar[0], (Double_t) nar[1]); - eymx = (nar[1] > eymx) ? nar[1] : eymx; - // MBs/s - fMBRtGraph->SetPoint(ii, (Double_t) nar[0], (Double_t) nar[2]); - bymx = (nar[2] > bymx) ? nar[2] : bymx; - // Active workers - if (PPD_SRV_NEWER(25)) { - fActWGraph->SetPoint(ii, (Double_t) nar[0], (Double_t) nar[3]); - wymx = (nar[3] > wymx) ? nar[3] : wymx; - } - // Sessions info - if (PPD_SRV_NEWER(25)) { - fTotSGraph->SetPoint(ii, (Double_t) nar[0], (Double_t) nar[4]); - tymx = (nar[4] > tymx) ? nar[4] : tymx; - fEffSGraph->SetPoint(ii, (Double_t) nar[0], (Double_t) nar[5]); - symx = (nar[5] > symx) ? nar[5] : symx; - } - } - - // Pad numbering - Int_t npads = 4; - Int_t kEvrt = 1; - Int_t kMBrt = 2; - Int_t kActW = 3; - Int_t kSess = 4; - if (bymx <= 0.) { - SafeDelete(fMBRtGraph); - npads--; - kActW--; - kSess--; - } - if (wymx <= 0.) { - SafeDelete(fActWGraph); - npads--; - kSess--; - } - // Plot only if more than one active session during the query - if (tymx <= 1.) { - SafeDelete(fTotSGraph); - SafeDelete(fEffSGraph); - npads--; - kSess--; - } - if (tymx <= 0.) SafeDelete(fTotSGraph); - if (symx <= 0.) SafeDelete(fEffSGraph); - - // Create a canvas - Int_t jsz = 200*npads; - TCanvas *c1 = new TCanvas("c1","Rate vs Time",200,10,700,jsz); - c1->SetFillColor(0); - c1->SetGrid(); - c1->SetBorderMode(0); - c1->SetFrameBorderMode(0); - - // Padding - c1->Divide(1, npads); - - // Event Rate plot - TPad *cpad = (TPad *) c1->GetPad(kEvrt); - if (cpad) { - cpad->cd(); - cpad->SetFillColor(0); - cpad->SetBorderMode(20); - cpad->SetFrameBorderMode(0); - } - fRateGraph->SetMinimum(0.); - fRateGraph->SetMaximum(eymx*1.1); - fRateGraph->SetLineColor(50); - fRateGraph->SetLineWidth(2); - fRateGraph->SetMarkerColor(38); - fRateGraph->SetMarkerStyle(25); - fRateGraph->SetMarkerSize(0.8); - fRateGraph->SetTitle("Processing rate (evts/sec)"); - fRateGraph->GetXaxis()->SetTitle("elapsed time (sec)"); - fRateGraph->Draw("ALP"); - - // Line with average - TLine *line = new TLine(fRateGraph->GetXaxis()->GetXmin(),fAvgRate, - fRateGraph->GetXaxis()->GetXmax(),fAvgRate); - line->SetLineColor(8); - line->SetLineStyle(2); - line->SetLineWidth(2); - line->Draw(); - - // Label - Double_t xax0 = fRateGraph->GetXaxis()->GetXmin(); - Double_t xax1 = fRateGraph->GetXaxis()->GetXmax(); - Double_t yax0 = 0.; - Double_t yax1 = eymx*1.1; - Double_t x0 = xax0 + 0.05 * (xax1 - xax0); - Double_t x1 = xax0 + 0.60 * (xax1 - xax0); - Double_t y0 = yax0 + 0.10 * (yax1 - yax0); - Double_t y1 = yax0 + 0.20 * (yax1 - yax0); - TPaveText *pt = new TPaveText(x0, y0, x1, y1, "br"); - pt->SetFillColor(0); - pt->AddText(Form("Global average: %.2f evts/sec", fAvgRate)); - pt->Draw(); - - // MB Rate plot - if (fMBRtGraph) { - cpad = (TPad *) c1->GetPad(kMBrt); - if (cpad) { - cpad->cd(); - cpad->SetFillColor(0); - cpad->SetBorderMode(0); - cpad->SetFrameBorderMode(0); - } - fMBRtGraph->SetFillColor(38); - TH1F *graph2 = new TH1F("graph2","Average read chunck size (MBs/request)",100, - fRateGraph->GetXaxis()->GetXmin(),fRateGraph->GetXaxis()->GetXmax()); - graph2->SetMinimum(0); - graph2->SetMaximum(1.1*bymx); - graph2->SetDirectory(0); - graph2->SetStats(0); - graph2->GetXaxis()->SetTitle("elapsed time (sec)"); - fMBRtGraph->SetHistogram(graph2); - fMBRtGraph->Draw("AB"); - } - - // MB Rate plot - if (fActWGraph) { - cpad = (TPad *) c1->GetPad(kActW); - if (cpad) { - cpad->cd(); - cpad->SetFillColor(0); - cpad->SetBorderMode(0); - cpad->SetFrameBorderMode(0); - } - fActWGraph->SetMinimum(0.); - fActWGraph->SetMaximum(wymx*1.1); - fActWGraph->SetLineColor(50); - fActWGraph->SetLineWidth(2); - fActWGraph->SetMarkerColor(38); - fActWGraph->SetMarkerStyle(25); - fActWGraph->SetMarkerSize(0.8); - fActWGraph->SetTitle("Active workers"); - fActWGraph->GetXaxis()->SetTitle("elapsed time (sec)"); - fActWGraph->Draw("ALP"); - } - - // MB Rate plot - if (fTotSGraph) { - cpad = (TPad *) c1->GetPad(kSess); - if (cpad) { - cpad->cd(); - cpad->SetFillColor(0); - cpad->SetBorderMode(0); - cpad->SetFrameBorderMode(0); - } - fTotSGraph->SetMinimum(0.); - fTotSGraph->SetMaximum(tymx*1.1); - fTotSGraph->SetLineColor(50); - fTotSGraph->SetLineWidth(2); - fTotSGraph->SetMarkerColor(38); - fTotSGraph->SetMarkerStyle(25); - fTotSGraph->SetMarkerSize(0.8); - fTotSGraph->SetTitle("Active, Effective sessions"); - fTotSGraph->GetXaxis()->SetTitle("elapsed time (sec)"); - fTotSGraph->Draw("ALP"); - - // Effective sessions - if (fEffSGraph) { - fEffSGraph->SetMinimum(0.); - fEffSGraph->SetMaximum(tymx*1.1); - fEffSGraph->SetLineColor(38); - fEffSGraph->SetLineWidth(2); - fEffSGraph->SetMarkerColor(50); - fEffSGraph->SetMarkerStyle(21); - fEffSGraph->SetMarkerSize(0.6); - fEffSGraph->Draw("SLP"); - } - } - - c1->Modified(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Do a memory plot - -void TProofProgressDialog::DoMemoryPlot() -{ - if (!fMemWindow) { - fMemWindow = new TProofProgressMemoryPlot(this, 500, 300); - fMemWindow->DoPlot(); - } else { - // Clear window - fMemWindow->Clear(); - fMemWindow->DoPlot(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable/Disable speedometer - -void TProofProgressDialog::DoEnableSpeedo() -{ - if (!fSpeedoEnabled) { - // Enable and connect - fSpeedoEnabled = kTRUE; - fSpeedo->Connect("OdoClicked()", "TProofProgressDialog", this, "ToggleOdometerInfos()"); - fSpeedo->Connect("LedClicked()", "TProofProgressDialog", this, "ToggleThreshold()"); - fUpdtSpeedo->ChangeText("&Disable speedometer"); - fUpdtSpeedo->SetToolTipText("Disable speedometer"); - fSmoothSpeedo->SetState(kButtonDown); - fSmoothSpeedo->SetToolTipText("Control smoothness in refreshing the speedo"); - } else { - // Disable and disconnect - fSpeedoEnabled = kFALSE; - // Reset speedo - fSpeedo->SetScaleValue(0); - fUpdtSpeedo->ChangeText("&Enable speedometer"); - fUpdtSpeedo->SetToolTipText("Enable speedometer (may have an impact on performance)"); - fSmoothSpeedo->SetToolTipText("Speedo refreshing is disabled"); - fSmoothSpeedo->SetState(kButtonDisabled); - } -} diff --git a/gui/sessionviewer/src/TProofProgressLog.cxx b/gui/sessionviewer/src/TProofProgressLog.cxx deleted file mode 100644 index b43755c8808ff..0000000000000 --- a/gui/sessionviewer/src/TProofProgressLog.cxx +++ /dev/null @@ -1,677 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: G Ganis, Jul 2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#include "TError.h" -#include "TPRegexp.h" -#include "TGFrame.h" -#include "TGTextView.h" -#include "TGLabel.h" -#include "TProof.h" -#include "TProofProgressDialog.h" -#include "TProofProgressLog.h" -#include "TProofLog.h" -#include "TGNumberEntry.h" -#include "TGListBox.h" -#include "TGButton.h" - -const UInt_t kLogElemFilled = BIT(17); // If the log element has been retrieved at least once -const UInt_t kDefaultActive = BIT(18); // If the log element is active by default - - -/** \class TProofProgressLog - \ingroup sessionviewer - -Dialog used to display Proof session logs from the Proof progress -dialog. -It uses TProofMgr::GetSessionLogs() mechanism internally - -*/ - - -ClassImp(TProofProgressLog); - -//////////////////////////////////////////////////////////////////////////////// -/// Create a window frame for log messages. - -TProofProgressLog::TProofProgressLog(TProofProgressDialog *d, Int_t w, Int_t h) : - TGTransientFrame(gClient->GetRoot(), gClient->GetRoot(), w, h) -{ - fDialog = d; - if (fDialog) fSessionUrl = fDialog->fSessionUrl; - fSessionIdx = 0; - - Init(w, h); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a window frame for log messages. - -TProofProgressLog::TProofProgressLog(const char *url, Int_t idx, Int_t w, Int_t h) : - TGTransientFrame(gClient->GetRoot(), gClient->GetRoot(), w, h) -{ - fDialog = 0; - fSessionUrl = url; - fSessionIdx = (idx > 0) ? -idx : idx; - - Init(w, h); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Init window frame for log messages. - -void TProofProgressLog::Init(Int_t w, Int_t h) -{ - fProofLog = 0; - fFullText = kFALSE; - fTextType = kStd; - // use hierarchical cleaning - SetCleanup(kDeepCleanup); - - //The text window - TGHorizontalFrame *htotal = new TGHorizontalFrame(this, w, h); - TGVerticalFrame *vtextbox = new TGVerticalFrame(htotal, w, h); - //fText = new TGTextView(this, w, h); - fText = new TGTextView(vtextbox, w, h); - vtextbox->AddFrame(fText, new TGLayoutHints(kLHintsTop | kLHintsExpandX | kLHintsExpandY, 3, 3, 3, 3)); - - //The frame for choosing workers - fVworkers = new TGVerticalFrame(htotal); - // URL choice - TGLabel *laburl = new TGLabel(fVworkers, "Enter cluster URL:"); - fVworkers->AddFrame(laburl, new TGLayoutHints(kLHintsTop | kLHintsLeft, 5, 2, 2, 2)); - fUrlText = new TGTextEntry(fVworkers); - fUrlText->SetText(fSessionUrl.Data()); - fVworkers->AddFrame(fUrlText, new TGLayoutHints(kLHintsTop | kLHintsExpandX, 5, 0, 0, 0)); - //The lower row of number entries and buttons - TGHorizontalFrame *hfurlbox = new TGHorizontalFrame(fVworkers, 20, 20); - TGLabel *labsess = new TGLabel(hfurlbox, "Enter session:"); - hfurlbox->AddFrame(labsess, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 5, 2, 2, 2)); - fSessNum = new TGNumberEntry(hfurlbox, 0, 5, -1, TGNumberFormat::kNESInteger); - fSessNum->SetLimits(TGNumberFormat::kNELLimitMax, 0., 0.); - fSessNum->SetIntNumber(0); - fSessNum->GetNumberEntry()->SetToolTipText("Use 0 for the last known one," - " negative numbers for the previous ones, e.g. -1 for the last-but-one"); - hfurlbox->AddFrame(fSessNum, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 0, 0, 0)); - fUrlButton = new TGTextButton(hfurlbox, "Get logs info"); - fUrlButton->Connect("Clicked()", "TProofProgressLog", this, "Rebuild()"); - hfurlbox->AddFrame(fUrlButton, new TGLayoutHints(kLHintsCenterY | kLHintsRight, 4, 0, 0, 0)); - fVworkers->AddFrame(hfurlbox, new TGLayoutHints(kLHintsTop | kLHintsLeft | kLHintsExpandX, 2, 2, 2, 2)); - - TGNumberEntry *nent = new TGNumberEntry(hfurlbox); - fVworkers->AddFrame(nent, new TGLayoutHints(kLHintsTop | kLHintsLeft, 4, 0, 0, 0)); - - //The list of workers - fLogList = 0; - BuildLogList(kTRUE); - fLogList->Resize(102,52); - fLogList->SetMultipleSelections(kTRUE); - - //The SelectAll/ClearAll buttons - TGHorizontalFrame *hfselbox = new TGHorizontalFrame(fVworkers, 20, 20); - TGLabel *label1 = new TGLabel(hfselbox,"Choose workers:"); - hfselbox->AddFrame(label1, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 0, 0, 0, 0)); - TGTextButton *selall = new TGTextButton(hfselbox, " &All "); - selall->Connect("Clicked()", "TProofProgressLog", this, "Select(=0)"); - hfselbox->AddFrame(selall, new TGLayoutHints(kLHintsCenterY | kLHintsRight, 10, 0, 0, 0)); - TGTextButton *clearall = new TGTextButton(hfselbox, " &Clear "); - clearall->Connect("Clicked()", "TProofProgressLog", this, "Select(=1)"); - hfselbox->AddFrame(clearall, new TGLayoutHints(kLHintsCenterY | kLHintsRight, 10, 0, 0, 0)); - - //select the default actives to start with - Select(0, kFALSE); - - //Display button - fLogNew = new TGTextButton(fVworkers, "&Display"); - fLogNew->Connect("Clicked()", "TProofProgressLog", this, "DoLog(=kFALSE)"); - //fLogNew->Resize(102, 20); - // fLogNew->SetMargins(1, 1, 0, 1); - fLogNew->SetTextColor(0xffffff, kFALSE); - fLogNew->SetBackgroundColor(0x000044); - fVworkers->AddFrame(hfselbox, new TGLayoutHints(kLHintsExpandX | kLHintsTop, 5, 2, 2, 2)); - fVworkers->AddFrame(fLogList, new TGLayoutHints(kLHintsExpandX | kLHintsTop | kLHintsExpandY, 2, 2, 5, 2)); - fVworkers->AddFrame(fLogNew, new TGLayoutHints(kLHintsExpandX | kLHintsTop , 2, 2, 1, 5)); - - htotal->AddFrame(fVworkers, new TGLayoutHints(kLHintsCenterY | kLHintsLeft | kLHintsExpandY, 2, 2, 2, 2)); - - //The lower row of number entries and buttons - TGHorizontalFrame *hflogbox = new TGHorizontalFrame(vtextbox, 550, 20); - fClose = new TGTextButton(hflogbox, " &Close "); - fClose->Connect("Clicked()", "TProofProgressLog", this, "CloseWindow()"); - hflogbox->AddFrame(fClose, new TGLayoutHints(kLHintsCenterY | - kLHintsRight, 10, 2, 2, 2)); - - //Saving to a file controls - fSave = new TGTextButton(hflogbox, "&Save"); - fSave->Connect("Clicked()", "TProofProgressLog", this, "SaveToFile()"); - hflogbox->AddFrame(fSave, new TGLayoutHints(kLHintsCenterY | kLHintsRight, 4, 0, 0, 0)); - fFileName = new TGTextEntry(hflogbox); - fFileName->SetText(".log"); - hflogbox->AddFrame(fFileName, new TGLayoutHints(kLHintsCenterY | kLHintsRight | kLHintsExpandX)); - TGLabel *label10 = new TGLabel(hflogbox, "Save to a file:"); - hflogbox->AddFrame(label10, new TGLayoutHints(kLHintsCenterY | kLHintsRight, 50, 2, 2, 2)); - - //Choose the number of lines to display - TGVerticalFrame *vlines = new TGVerticalFrame(hflogbox); - TGHorizontalFrame *vlines_buttons = new TGHorizontalFrame(vlines); - TGLabel *label2 = new TGLabel(vlines_buttons, "Lines:"); - vlines_buttons->AddFrame(label2, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - - fAllLines = new TGCheckButton(vlines_buttons, "all"); - fAllLines->SetToolTipText("Retrieve all lines (service messages excluded)"); - fAllLines->SetState(kButtonUp); - fAllLines->Connect("Clicked()", "TProofProgressLog", this, "NoLineEntry()"); - vlines_buttons->AddFrame(fAllLines, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - - fRawLines = new TGCheckButton(vlines_buttons, "svcmsg"); - fRawLines->SetToolTipText("Retrieve all type of lines, service messages included"); - fRawLines->SetState(kButtonUp); - vlines_buttons->AddFrame(fRawLines, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - - TGLabel *label11 = new TGLabel(vlines_buttons, "From"); - vlines_buttons->AddFrame(label11, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - - fLinesFrom = new TGNumberEntry(vlines_buttons, 0, 5, -1, TGNumberFormat::kNESInteger); - // coverity[negative_returns]: no problem with -100, the format is kNESInteger - fLinesFrom->SetIntNumber(-100); - fLinesFrom->GetNumberEntry()->SetToolTipText("Negative values indicate \"tail\" action"); - - - vlines_buttons->AddFrame(fLinesFrom, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - - TGLabel *label3 = new TGLabel(vlines_buttons, "to"); - vlines_buttons->AddFrame(label3, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - fLinesTo = new TGNumberEntry(vlines_buttons, 0, 5, -1, TGNumberFormat::kNESInteger); - vlines_buttons->AddFrame(fLinesTo, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - vlines->AddFrame(vlines_buttons, new TGLayoutHints(kLHintsCenterY)); - hflogbox->AddFrame(vlines, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - - // - // Lowest line, with filter (grep or cmd pipe) controls - // - - TGHorizontalFrame *hfgrepbox = new TGHorizontalFrame(vtextbox, 550, 20); - - // Grep/pipe label, textbox and button - fGrepLabel = new TGLabel(hfgrepbox, ""); - hfgrepbox->AddFrame(fGrepLabel, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - fGrepText = new TGTextEntry(hfgrepbox); - { - TGDimension dim = fGrepText->GetDefaultSize(); - fGrepText->SetDefaultSize(400, dim.fHeight); - } - hfgrepbox->AddFrame(fGrepText, new TGLayoutHints( kLHintsCenterY | kLHintsLeft | kLHintsExpandX)); - fGrepButton = new TGTextButton(hfgrepbox, "Filter"); - fGrepButton->Connect("Clicked()", "TProofProgressLog", this, "DoLog(=kTRUE)"); - hfgrepbox->AddFrame(fGrepButton, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 4, 10, 0, 0)); // l r t b - - // Checkbox for inverting selection or giving a pipe command - fGrepCheckInv = new TGCheckButton(hfgrepbox, "invert match"); - fGrepCheckInv->Connect("Clicked()", "TProofProgressLog", this, "SetGrepView()"); - hfgrepbox->AddFrame(fGrepCheckInv, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - - fGrepCheckCmd = new TGCheckButton(hfgrepbox, "is a pipe command"); - fGrepCheckCmd->Connect("Clicked()", "TProofProgressLog", this, "SetGrepView()"); - hfgrepbox->AddFrame(fGrepCheckCmd, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - - // fRawLines->SetToolTipText("Retrieve all type of lines, service messages included"); - // fRawLines->SetState(kButtonUp); - // vlines_buttons->AddFrame(fRawLines, new TGLayoutHints(kLHintsCenterY | kLHintsLeft, 2, 2, 2, 2)); - - // - // Add frames to the global picture - // - - vtextbox->AddFrame(hflogbox, new TGLayoutHints(kLHintsTop | kLHintsLeft | kLHintsExpandX, 2, 2, 2, 2)); - vtextbox->AddFrame(hfgrepbox, new TGLayoutHints(kLHintsBottom | kLHintsLeft | kLHintsExpandX, 2, 2, 2, 2)); - htotal->AddFrame(vtextbox, new TGLayoutHints(kLHintsExpandX | kLHintsExpandY | kLHintsRight, 3, 3, 3, 3)); - AddFrame(htotal, new TGLayoutHints(kLHintsExpandX | - kLHintsExpandY, 3, 3, 3, 3)); - SetGrepView(); - MapSubwindows(); - Resize(); - CenterOnParent(); - Popup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TProofProgressLog::~TProofProgressLog() -{ - // Cleanup the log object - SafeDelete(fProofLog); - - // Detach from owner dialog - if (fDialog) { - fDialog->fLogWindow = 0; - fDialog->fProof->Disconnect("LogMessage(const char*,Bool_t)", this, - "LogMessage(const char*,Bool_t)"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show log window. - -void TProofProgressLog::Popup() -{ - MapWindow(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear log window. - -void TProofProgressLog::Clear(Option_t *) -{ - if (fText) - fText->Clear(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load a text buffer in the window. - -void TProofProgressLog::LoadBuffer(const char *buffer) -{ - if (fText) - fText->LoadBuffer(buffer); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load a file in the window. - -void TProofProgressLog::LoadFile(const char *file) -{ - if (fText) - fText->LoadFile(file); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add text to the window. - -void TProofProgressLog::AddBuffer(const char *buffer) -{ - if (fText) { - TGText txt; - txt.LoadBuffer(buffer); - fText->AddText(&txt); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle close button or when closed via window manager action. - -void TProofProgressLog::CloseWindow() -{ - DeleteWindow(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build the list of workers. For this, extract the logs and take the names -/// of TProofLogElements - -void TProofProgressLog::BuildLogList(Bool_t create) -{ - // Set title - TString title; - title.Form("PROOF - Processing logs for session 'undefined'"); - SetWindowName(title.Data()); - SetIconName(title.Data()); - - // Create the list-box now - if (create) { - if (fLogList) delete fLogList; - fLogList = new TGListBox(fVworkers); - } else { - // Reset - Int_t nent = fLogList->GetNumberOfEntries(); - fLogList->RemoveEntries(0,nent); - fLogList->Layout(); - } - - if (fSessionUrl.IsNull()) { - if (gDebug > 0) - Info("BuildLogList", "session URL undefined - do nothing"); - return; - } - TProofMgr *mgr = TProof::Mgr(fSessionUrl.Data()); - if (!mgr || !mgr->IsValid()) { - Warning("BuildLogList", "unable open a manager connection to %s", - fSessionUrl.Data()); - return; - } - if (!(fProofLog = mgr->GetSessionLogs(fSessionIdx,"NR"))) { - Warning("BuildLogList", "unable to get logs from %s", - fSessionUrl.Data()); - return; - } - // Set title - title.Form("PROOF - Processing logs for session '%s', started on %s at %s", - fProofLog->GetName(), fProofLog->StartTime().AsString(), - fProofLog->GetTitle()); - SetWindowName(title.Data()); - SetIconName(title.Data()); - - TList *elem = fProofLog->GetListOfLogs(); - TIter next(elem); - TProofLogElem *pe = 0; - - Int_t is = 0; - TGLBEntry *ent = 0; - TString buf; - while ((pe=(TProofLogElem*)next())){ - TUrl url(pe->GetTitle()); - buf.Form("%s %s", pe->GetName(), url.GetHost()); - fLogList->AddEntry(buf.Data(), is); - if ((ent = fLogList->FindEntry(buf.Data()))) { - ent->ResetBit(kLogElemFilled); - ent->ResetBit(kDefaultActive); - if (!(pe->IsWorker())) ent->SetBit(kDefaultActive); - } - is++; - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display logs. 'grep' is set to kTRUE if it is invoked by pressing the -/// 'Filter' button. - -void TProofProgressLog::DoLog(Bool_t grep) -{ - Clear(); - - if (!fGrepText) { - Warning("DoLog", "no text: do nothing!"); - return; - } - - TString greptext = fGrepText->GetText(); - greptext.Remove(TString::kBoth, ' '); - if (greptext.IsNull()) { - grep = kFALSE; - } - else if (!fGrepCheckCmd->IsOn()) { - // Not a command: sanitize string - TPMERegexp san("(^|[^\\\\])([^a-zA-Z0-9_=\\\\/.-])"); - while ( san.Substitute(greptext, "$1\\$2") > 0 ); - } - - Int_t from, to; - if (fAllLines->IsOn()){ - from = 0; - to = -1; - } else { - from = fLinesFrom->GetIntNumber(); - to = fLinesTo->GetIntNumber(); - } - - // Create the TProofLog instance - if (!fProofLog) { - TProofMgr *mgr = 0; - if ((mgr = TProof::Mgr(fSessionUrl.Data()))) { - if (!(fProofLog = mgr->GetSessionLogs(fSessionIdx, "NR"))) { - Warning("DoLog", "unable to instantiate TProofLog for %s", - fSessionUrl.Data()); - } - } else { - Warning("DoLog", "unable to instantiate a TProofMgr for %s", - fSessionUrl.Data()); - } - } - - // Pipe command for filtering - TString pipeCommand; - - // Filter out SvcMsg - if (!fRawLines->IsOn()) { - pipeCommand = "grep -v \"| SvcMsg\""; - } - - // Default is not retrieving - Bool_t retrieve = kFALSE; - if (!grep) { - // Not invoked via 'Filter' button - if (!fFullText || - ((fTextType != kRaw && fRawLines->IsOn()) || - (fTextType != kStd && !fRawLines->IsOn())) || - (fDialog && fDialog->fStatus==TProofProgressDialog::kRunning)) { - retrieve = kTRUE; - if (fRawLines->IsOn()) { - fTextType = kRaw; - } else { - fTextType = kStd; - } - if (fDialog && fDialog->fStatus != TProofProgressDialog::kRunning) - fFullText = kTRUE; - } - } else { - retrieve = kTRUE; - fTextType = kGrep; - - if (!pipeCommand.IsNull()) - pipeCommand.Append('|'); - - if (fGrepCheckCmd->IsOn()) { - pipeCommand.Append(greptext); - } - else { - pipeCommand.Append("grep "); - if (fGrepCheckInv->IsOn()) - pipeCommand.Append("-v "); - pipeCommand.Append("-- "); - pipeCommand.Append(greptext); // sanitized - } - - if (fDialog && fDialog->fStatus != TProofProgressDialog::kRunning) - fFullText = kTRUE; - } - - // Display now - if (fProofLog) { - TList *selected = new TList; - fLogList->GetSelectedEntries(selected); - TIter next(selected); - TGTextLBEntry *selentry; - Bool_t logonly = fProofLog->LogToBox(); - fProofLog->SetLogToBox(kTRUE); - - fProofLog->Connect("Prt(const char*)", "TProofProgressLog", - this, "LogMessage(const char*, Bool_t)"); - while ((selentry=(TGTextLBEntry*)next())){ - TString ord = selentry->GetText()->GetString(); - Int_t is = ord.Index(" "); - if (is != kNPOS) ord.Remove(is); - if (retrieve || !selentry->TestBit(kLogElemFilled)) { - pipeCommand.Prepend('|'); - if (fTextType == kRaw) { - if (gDebug >= 2) - Info("DoLog", "Retrieving unfiltered log for %s", ord.Data()); - fProofLog->Retrieve(ord.Data(), TProofLog::kTrailing, 0, 0); - } - else { - if (gDebug >= 2) - Info("DoLog", "Retrieving log for %s filtered with %s", - ord.Data(), pipeCommand.Data()); - fProofLog->Retrieve(ord.Data(), TProofLog::kGrep, 0, pipeCommand.Data()); - } - selentry->SetBit(kLogElemFilled); - } - fProofLog->Display(ord.Data(), from, to); - } - fProofLog->SetLogToBox(logonly); - fProofLog->Disconnect("Prt(const char*)", this, "LogMessage(const char*, Bool_t)"); - delete selected; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load/append a log msg in the log frame, if open - -void TProofProgressLog::LogMessage(const char *msg, Bool_t all) -{ - if (all) { - // load buffer - LoadBuffer(msg); - } else { - // append - AddBuffer(msg); - } -} - -//////////////////////////////////////////////////////////////////////////////// -///Save the logs to a file -///Only the name of the file is taken, no expansion - -void TProofProgressLog::SaveToFile() -{ - if (!fProofLog) DoLog(); - - // File name: the default is .log - TString filename = fFileName->GetText(); - if (filename.IsNull() || filename == ".log") { - filename = (fDialog && fDialog->fProof) ? - TString::Format("%s.log", fDialog->fProof->GetName()) : - TString("proof.log"); - } - - TList *selected = new TList; - fLogList->GetSelectedEntries(selected); - TIter next(selected); - TGTextLBEntry *selentry; - Bool_t writemode=kTRUE; - const char *option; - TString ord; - while ((selentry=(TGTextLBEntry*)next())){ - ord = selentry->GetText()->GetString(); - Int_t isp = ord.Index(' '); - if (isp != kNPOS) ord.Remove(isp); - //open the file in "w" mode for the first time - option = writemode ? "w" : "a"; - fProofLog->Save(ord.Data(), filename.Data(), option); - writemode=kFALSE; - } - delete selected; - - Info("SaveToFile", "logs saved to file %s", filename.Data()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Sets the view of grep filters according to the value of checkboxes - -void TProofProgressLog::SetGrepView() -{ - if (fGrepCheckCmd->IsOn()) { - fGrepLabel->SetText("Pipe log through command:"); - fGrepCheckInv->SetDisabledAndSelected(kFALSE); - } - else { - fGrepLabel->SetText("Grep:"); - Bool_t u = fGrepCheckInv->IsOn(); - fGrepCheckInv->SetEnabled(kTRUE); - if (u) { - fGrepLabel->SetText("Show lines not matching:"); - fGrepCheckInv->SetState(kButtonDown); - } - else { - fGrepLabel->SetText("Show lines matching:"); - fGrepCheckInv->SetState(kButtonUp); - } - } - - // Ugly but necessary const_cast - TGFrame *frame = dynamic_cast( const_cast(fGrepLabel->GetParent()) ); - if (frame) frame->Layout(); - -} - -//////////////////////////////////////////////////////////////////////////////// -///Enable/disable the line number entry - -void TProofProgressLog::NoLineEntry() -{ - if (fAllLines->IsOn()){ - //disable the line number entry - fLinesFrom->SetState(kFALSE); - fLinesTo->SetState(kFALSE); - } else { - fLinesFrom->SetState(kTRUE); - fLinesTo->SetState(kTRUE); - } -} - -//////////////////////////////////////////////////////////////////////////////// -///actions of select all/clear all button - -void TProofProgressLog::Select(Int_t id, Bool_t all) -{ - Int_t nen = fLogList->GetNumberOfEntries(); - Bool_t sel = id ? 0 : 1; - - TGLBEntry *ent = 0; - for (Int_t ie=0; ieSelect(ie, sel); - } else { - if ((ent = fLogList->GetEntry(ie))) { - if (ent->TestBit(kDefaultActive)) fLogList->Select(ie, sel); - } - } - } -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Rebuild the log info for a new entered session - -void TProofProgressLog::Rebuild() -{ - // Check if we need to remake the TProofLog object - Bool_t sameurl = kFALSE; - TUrl url(fUrlText->GetText()); - TUrl urlref(fSessionUrl.Data()); - if (!strcmp(url.GetHostFQDN(), urlref.GetHostFQDN())) { - if (url.GetPort() == urlref.GetPort()) { - if (!strcmp(url.GetUser(), urlref.GetUser())) { - sameurl = kTRUE; - } - } - } - Int_t idx = 0; - if (sameurl) { - idx = fSessNum->GetIntNumber(); - if (idx == fSessionIdx) { - Info("Rebuild", "same parameters {%s, %s}, {%d, %d}: no need to rebuild TProofLog", - url.GetUrl(), urlref.GetUrl(), idx, fSessionIdx); - return; - } - } - // Cleanup current TProofLog - if (fProofLog) { - delete fProofLog; - fProofLog = nullptr; - } - - // Set new parameters - fSessionUrl = fUrlText->GetText(); - fSessionIdx = idx; - - // Rebuild the list now - BuildLogList(kFALSE); - - // Select the default actives to start with - Select(0, kFALSE); - // Redraw - fLogList->Layout(); - - // Done - return; -} diff --git a/gui/sessionviewer/src/TProofProgressMemoryPlot.cxx b/gui/sessionviewer/src/TProofProgressMemoryPlot.cxx deleted file mode 100644 index 701e51d4e1e60..0000000000000 --- a/gui/sessionviewer/src/TProofProgressMemoryPlot.cxx +++ /dev/null @@ -1,652 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Anna Kreshuk 18/07/2008 - -/************************************************************************* - * Copyright (C) 1995-2008, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -/** \class TProofProgressMemoryPlot - \ingroup sessionviewer - -This class implements a dialog, used to display the memory footprint -on the proof workers and master. For the workers, memory is plotted -as a function of number of events processed. For the master, it is -plotted as a function of number of objects merged - -*/ - - -#include "TProofProgressMemoryPlot.h" -#include "TProofProgressDialog.h" -#include "TRootEmbeddedCanvas.h" -#include "TCanvas.h" -#include "TGListBox.h" -#include "TGButton.h" -#include "TGLabel.h" -#include "TGMenu.h" -#include "TProofLog.h" -#include "TUrl.h" -#include "TProof.h" -#include "TError.h" -#include "TGFrame.h" -#include "TMacro.h" -#include "TObjString.h" -#include "TMultiGraph.h" -#include "TGraph.h" -#include "TLegend.h" -#include "TAxis.h" -#include "snprintf.h" - -#define kMemValuePos 8 -#define kMemValuePosMaster 8 -#define kEventNumberPos 13 - -ClassImp(TProofProgressMemoryPlot); - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor - -TProofProgressMemoryPlot::TProofProgressMemoryPlot(TProofProgressDialog *d, - Int_t w, Int_t h) - : TGTransientFrame(gClient->GetRoot(), - gClient->GetRoot(), w, h) -{ - fDialog = d; - fProofLog = 0; - fWPlot = 0; - fMPlot = 0; - fAPlot = 0; - fFullLogs = kFALSE; - - // use hierarchical cleaning - SetCleanup(kDeepCleanup); - TGHorizontalFrame *htotal = new TGHorizontalFrame(this, w, h); - //The frame for choosing workers - TGVerticalFrame *vworkers = new TGVerticalFrame(htotal); - TGLabel *label1 = new TGLabel(vworkers,"Choose workers:"); - - //The list of workers - fWorkers = BuildLogList(vworkers); - fWorkers->Resize(102,52); - fWorkers->SetMultipleSelections(kTRUE); - - //The SelectAll/ClearAll button - TGPopupMenu *pm = new TGPopupMenu(gClient->GetRoot()); - pm->AddEntry("Select All", 0); - pm->AddEntry("Clear All", 1); - - fAllWorkers = new TGSplitButton(vworkers, new TGHotString("Select ... "), pm); - fAllWorkers->Connect("ItemClicked(Int_t)", "TProofProgressMemoryPlot", this, - "Select(Int_t)"); - fAllWorkers->SetSplit(kFALSE); - //select all for the first display - Select(1); - fWorkers->Select(0, kTRUE); - fWorkers->Select(1, kTRUE); - - fPlot = new TGTextButton(vworkers, "Plot"); - fPlot->Connect("Clicked()", "TProofProgressMemoryPlot", this, "DoPlot()"); - vworkers->AddFrame(label1, new TGLayoutHints(kLHintsLeft | kLHintsTop, 7, 2, 5, 2)); - vworkers->AddFrame(fAllWorkers, new TGLayoutHints(kLHintsExpandX | kLHintsTop, 5, 2, 2, 2)); - vworkers->AddFrame(fWorkers, new TGLayoutHints(kLHintsExpandX | kLHintsTop | kLHintsExpandY, 2, 2, 5, 2)); - vworkers->AddFrame(fPlot, new TGLayoutHints(kLHintsExpandX | kLHintsBottom, 2, 2, 2, 2)); - htotal->AddFrame(vworkers, new TGLayoutHints(kLHintsCenterY | kLHintsLeft | kLHintsExpandY, 2, 2, 2, 2)); - - fWorkersPlot = new TRootEmbeddedCanvas("WorkersCanvas", htotal, 300, 300); - htotal->AddFrame(fWorkersPlot, new TGLayoutHints(kLHintsCenterY | kLHintsLeft | kLHintsExpandX | kLHintsExpandY, 2, 2, 2, 2)); - fMasterPlot = new TRootEmbeddedCanvas("MasterCanvas", htotal, 300, 300); - htotal->AddFrame(fMasterPlot, new TGLayoutHints(kLHintsCenterY | kLHintsLeft | kLHintsExpandX | kLHintsExpandY, 2, 2, 2 ,2)); - - AddFrame(htotal, new TGLayoutHints(kLHintsExpandX | kLHintsExpandY, 2, 2, 2, 2)); - TString title; - title.Form("PROOF Memory Consumption: %s", (fDialog->fProof ? - fDialog->fProof->GetMaster() : "")); - SetWindowName(title); - SetIconName(title); - - MapSubwindows(); - Resize(); - CenterOnParent(); - MapWindow(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TProofProgressMemoryPlot::~TProofProgressMemoryPlot() -{ - if (fProofLog){ - delete fProofLog; - fProofLog = 0; - } - if (fMPlot){ - delete fMPlot; - fMPlot = 0; - } - if (fWPlot){ - delete fWPlot; - fWPlot = 0; - } - - fProofLog = 0; - fDialog->fMemWindow = 0; - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build the list of workers. For this, extract the logs and take the names -/// of TProofLogElements - -TGListBox* TProofProgressMemoryPlot::BuildLogList(TGFrame *parent) -{ - TGListBox *c = new TGListBox(parent); - c->AddEntry("average", 0); - - SafeDelete(fProofLog); - fProofLog = 0; - - TProofMgr *mgr = TProof::Mgr(fDialog->fSessionUrl.Data()); - if (mgr) fProofLog = mgr->GetSessionLogs(0, 0, "Svc.*Memory"); - if (fDialog->fStatus==TProofProgressDialog::kRunning) { - fFullLogs = kFALSE; - } else { - fFullLogs = kTRUE; - } - - if (fProofLog) { - TList *elem = fProofLog->GetListOfLogs(); - TIter next(elem); - TProofLogElem *pe = 0; - - TString buf; - Int_t is = 1; - while ((pe=(TProofLogElem*)next())){ - TUrl url(pe->GetTitle()); - buf = TString::Format("%s %s", pe->GetName(), url.GetHost()); - c->AddEntry(buf.Data(), is); - is++; - } - } - return c; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear the canvases - -void TProofProgressMemoryPlot::Clear(Option_t *) -{ - if (fWorkersPlot) - fWorkersPlot->GetCanvas()->Clear(); - if (fMasterPlot) - fMasterPlot->GetCanvas()->Clear(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw the plot from the logs - -void TProofProgressMemoryPlot::DoPlot() -{ - Clear(); - - if (!fProofLog || !fFullLogs || - (fDialog && fDialog->fStatus == TProofProgressDialog::kRunning)){ - - SafeDelete(fProofLog); - if (fDialog) { - TProofMgr *mgr = TProof::Mgr(fDialog->fSessionUrl.Data()); - if (mgr) fProofLog = mgr->GetSessionLogs(0, 0, "Svc.*Memory"); - if (fDialog->fStatus==TProofProgressDialog::kRunning) { - fFullLogs = kFALSE; - } else { - fFullLogs = kTRUE; - } - } else { - Error("DoPlot", "TProofProgessDialog instance undefined - protocol error?"); - return; - } - } - - // Make sure we have something to parse - if (!fProofLog) { - Error("DoPlot", "could not get a valid instance of TProofLog"); - return; - } - - char name[512]; //should be long enough - - TList *elem = fProofLog->GetListOfLogs(); - if (!elem) {Error("DoPlot", "No log elements\n"); return;} - TIter next(elem); - TProofLogElem *ple=0; - - Int_t iwelem = 0; - Int_t imelem = 0; - TGraph *gr=0; - - TList *selected = new TList; - fWorkers->GetSelectedEntries(selected); - TIter nextworker(selected); - TGTextLBEntry *selworker; - TLegend *legw = 0; - TLegend *legm = 0; - - //delete the current multigraphs - if (fWPlot){ - delete fWPlot; - fWPlot = 0; - } - if (fMPlot) { - delete fMPlot; - fMPlot = 0; - } - - //loop over the selected workers in the list - Int_t max = -1; - Int_t min = -1; - while ((selworker=(TGTextLBEntry*)nextworker())){ - - snprintf(name, sizeof(name)-1, "%s", selworker->GetText()->GetString()); - char *token; - token = strtok(name, " "); - if (token && !strcmp(token, "average")) { //change that to id comparison later - gr = DoAveragePlot(max, min); - if (gr && gr->GetN()>0){ - if (!fWPlot) { - fWPlot = new TMultiGraph(); - } - if (!legw) { - legw = new TLegend(0.1, 0.7, 0.4, 0.9); - legw->SetHeader("Workers"); - } - gr->SetMarkerColor(1); - gr->SetMarkerStyle(2); - gr->SetMarkerSize(1); - gr->SetLineWidth(2); - gr->SetLineColor(1); - fWPlot->Add(gr, "l"); - legw->AddEntry(gr, token, "l"); - } - TProofLogElem *pltemp = (TProofLogElem*)elem->At(min+1); - gr = DoWorkerPlot(pltemp); - if (gr && gr->GetN()>0){ - if (!fWPlot) { - fWPlot = new TMultiGraph(); - } - if (!legw) { - legw = new TLegend(0.1, 0.7, 0.4, 0.9); - legw->SetHeader("Workers"); - } - gr->SetLineWidth(2); - gr->SetLineColor(2); - gr->SetLineStyle(3); - fWPlot->Add(gr, "l"); - legw->AddEntry(gr, TString::Format("%s - min", pltemp->GetName()) , "l"); - } - pltemp = (TProofLogElem*)elem->At(max+1); - gr = DoWorkerPlot(pltemp); - if (gr && gr->GetN()>0){ - if (!fWPlot) { - fWPlot = new TMultiGraph(); - } - if (!legw) { - legw = new TLegend(0.1, 0.7, 0.4, 0.9); - legw->SetHeader("Workers"); - } - gr->SetLineWidth(2); - gr->SetLineColor(2); - gr->SetLineStyle(2); - fWPlot->Add(gr, "l"); - legw->AddEntry(gr, TString::Format("%s - max", pltemp->GetName()), "l"); - } - - continue; - } - - - ple = (TProofLogElem*)elem->FindObject(token); - const char *role = ple->GetRole(); - if (role[0]=='w'){ - //role should be equal to "worker", only check the 1st char - - gr = DoWorkerPlot(ple); - if (gr && gr->GetN()>0) { - if (!fWPlot) { - fWPlot = new TMultiGraph(); - } - if (!legw) { - legw = new TLegend(0.1, 0.7, 0.4, 0.9); - legw->SetHeader("Workers"); - } - gr->SetLineWidth(2); - gr->SetLineColor(iwelem+3); - fWPlot->Add(gr, "l"); - legw->AddEntry(gr, token, "l"); - iwelem++; - } - } else { - //a master or submaster log - //display without meaningful labels for now - gr = DoMasterPlot(ple); - if (gr && gr->GetN()>0){ - if (!fMPlot){ - fMPlot = new TMultiGraph(); - } - if (!legm) { - legm = new TLegend(0.1, 0.7, 0.4, 0.9); - legm->SetHeader("Master"); - } - gr->SetLineWidth(2); - gr->SetLineColor(imelem+1); - fMPlot->Add(gr, "l"); - legm->AddEntry(gr, token, "l"); - imelem++; - } - } - - } - - if (fWPlot){ - fWorkersPlot->GetCanvas()->cd(); - fWPlot->Draw("a"); - if (fWPlot->GetXaxis()) - fWPlot->GetXaxis()->SetTitle("Events Processed"); - if (fWPlot->GetYaxis()) - fWPlot->GetYaxis()->SetTitle("MBytes"); - if (legw) legw->Draw(); - - } - if (fMPlot) { - fMasterPlot->GetCanvas()->cd(); - fMPlot->Draw("a"); - if (fMPlot->GetXaxis()) - fMPlot->GetXaxis()->SetTitle("Objects Merged"); - if (fMPlot->GetYaxis()) - fMPlot->GetYaxis()->SetTitle("MBytes"); - if (legm) legm->Draw(); - } - fWorkersPlot->GetCanvas()->Update(); - fMasterPlot->GetCanvas()->Update(); - - delete selected; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create the average plots - -TGraph *TProofProgressMemoryPlot::DoAveragePlot(Int_t &max_el, Int_t &min_el) -{ - TList *elem = fProofLog->GetListOfLogs(); - if (!elem) { - Error("DoAveragePlot", "Empty log"); - return 0; - } - TIter next(elem); - - TProofLogElem *ple=0; - Double_t max_av = 0; - Double_t min_av = 10E9; - - Long64_t maxevent = 0; - Long64_t step = -1; - TObjString *curline = 0; - TObjString *prevline = 0; - Long64_t curevent_value; - Long64_t prevevent_value; - Long64_t *last = new Long64_t[elem->GetEntries()]; - Long64_t vmem = -1, rmem = -1, nevt = -1; - TString token; - Int_t ielem=0; - for (Int_t i=0; iGetEntries(); i++) { - last[i] = 0; - } - while ((ple = (TProofLogElem *)next())){ - //find the maximal entry processed in the last query - const char *role = ple->GetRole(); - if (role[0] != 'w') continue; //skip the master log - TList *lines = ple->GetMacro()->GetListOfLines(); - if (!lines || lines->GetSize() <= 0) continue; - curline = (TObjString *) lines->Last(); - if (!curline) continue; - curevent_value = 0; - if (ParseLine(curline->String(), vmem, rmem, curevent_value) != 0) { - Warning("DoAveragePlot", "error parsing line: '%s'", curline->String().Data()); - continue; - } - if (maxevent < curevent_value) maxevent = curevent_value; - last[ielem] = curevent_value; - if (step < 0) { - // Find the step - prevline = (TObjString *)lines->Before(curline); - if (prevline) { - prevevent_value = 0; - if (ParseLine(prevline->String(), vmem, rmem, prevevent_value) != 0) { - Warning("DoAveragePlot", "error parsing line: '%s'", curline->String().Data()); - } else { - step = curevent_value - prevevent_value; - } - } - } - ielem++; - } - Int_t maxlines = Int_t(maxevent/(1.*step)); - //transform the array of last event numbers to an array of numbers of lines - for (Int_t i=0; iGetRole(); - if (role[0]!='w') continue; - TList *lines = ple->GetMacro()->GetListOfLines(); - if (!lines || lines->GetSize() <= 0) continue; - TIter prev(lines, kIterBackward); - iline=0; - cur_av = 0; - while ((curline = (TObjString*)prev()) && ilineString(), vmem, rmem, nevt) != 0) { - Warning("DoWorkerPlot", "error parsing line: '%s'", curline->String().Data()); - continue; - } - av_mem[last[ielem] -1 - iline] += vmem; //last[ielem] is the number of lines for - nw[last[ielem] -1 - iline]++; //this query and this element - if (last[ielem] > 0) cur_av += (Double_t)vmem / last[ielem]; - iline++; - } - if (cur_av > max_av){ - max_av = cur_av; - max_el = ielem; - } - if (cur_av < min_av){ - min_av = cur_av; - min_el = ielem; - } - ielem++; - } - - TGraph *gr = new TGraph(maxlines); - //make an average - for (Int_t i=0; iSetPoint(i, (i+1)*step, av_mem[i]/(nw[i]*1024.)); - } - delete [] av_mem; - av_mem = 0; - delete [] nw; - nw = 0; - delete [] last; - last = 0; - return gr; - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Extract from line 'l' the virtual memory 'v', the resident memory 'r' and the -/// number of events 'e'. -/// The line is assumed to be in the form -/// "... Memory 130868 virtual 31540 ... event 5550" -/// The fields are only filled if >= 0 . -/// Return 0 on success, -1 if any of the values could not be filled (the output -/// fields are not touched in such a case). - -Int_t TProofProgressMemoryPlot::ParseLine(TString l, - Long64_t &v, Long64_t &r, Long64_t &e) -{ - // Something to parse is mandatory - if (l.IsNull()) return -1; - - // At least one field needs to be filled - if (v < 0 && r < 0 && e < 0) return 0; - - // Position at the start of the relevant info - Int_t from = kNPOS; - if ((from = l.Index("Memory")) == kNPOS) return -1; - - // Prepare extraction - from += 7; - TString tok; - - // The virtual memory - if (v >= 0) { - if (!l.Tokenize(tok, from, " ")) return -1; - v = tok.Atoll(); - } - - // The resident memory - if (r >= 0) { - if ((from = l.Index("virtual", from)) == kNPOS) return -1; - from += 8; - if (!l.Tokenize(tok, from, " ")) return -1; - r = tok.Atoll(); - } - - // The number of events - if (e >= 0) { - if ((from = l.Index("event", from)) == kNPOS) return -1; - from += 6; - if (!l.Tokenize(tok, from, " ")) return -1; - e = tok.Atoll(); - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make a memory consumption graph for a worker - -TGraph *TProofProgressMemoryPlot::DoWorkerPlot(TProofLogElem *ple) -{ - TObjString *curline; - TList *lines = ple->GetMacro()->GetListOfLines(); - if (!lines) { - //the log is empty so far - return 0; - } - - Long64_t vmem = -1, rmem = -1, nevt = -1; - - //find the last event value - curline = (TObjString*)lines->Last(); - Long64_t lastevent_value = 0; - if (ParseLine(curline->String(), vmem, rmem, lastevent_value) != 0) { - Error("DoWorkerPlot", "error parsing line: '%s'", curline->String().Data()); - return 0; - } - - //find the step - TObjString *prevline = (TObjString*)lines->Before(curline); - Long64_t prevevent_value = 0; - if (prevline && ParseLine(prevline->String(), vmem, rmem, prevevent_value) != 0) { - Error("DoWorkerPlot", "error parsing line: '%s'", prevline->String().Data()); - return 0; - } - Long64_t step = lastevent_value - prevevent_value; - if (step <= 0) { - Error("DoWorkerPlot", "null or negative step (%lld) - cannot continue", step); - return 0; - } - - Int_t nlines = lastevent_value/step; - TGraph *gr = new TGraph(nlines); - - TIter prevl(lines, kIterBackward); - Int_t iline = 0; - TString token; - while ((curline = (TObjString*)prevl()) && ilineString(), vmem, rmem, nevt) != 0) { - Warning("DoWorkerPlot", "error parsing line: '%s'", curline->String().Data()); - continue; - } - gr->SetPoint(nlines-1-iline, lastevent_value-iline*step, vmem/1024.); - iline++; - } - - return gr; -} - -//////////////////////////////////////////////////////////////////////////////// -/// a master or submaster log -/// display without meaningful labels for now - -TGraph *TProofProgressMemoryPlot::DoMasterPlot(TProofLogElem *ple) -{ - TList *lines = ple->GetMacro()->GetListOfLines(); - TIter prevline(lines, kIterBackward); - Int_t iline=0; - TObjString *curline; - //count the number of lines that belong to the last query - while ((curline = (TObjString*)prevline())) { - if (curline->String().Contains("Start")) break; - iline++; - } - - Long64_t vmem = -1, rmem = -1, nevt = -1; - - Int_t nlines = iline; - TString token; - TGraph *gr = new TGraph(nlines); - prevline.Reset(); - iline = 0; - while ((curline = (TObjString*)prevline()) && ilineString(), vmem, rmem, nevt) != 0) { - Warning("DoWorkerPlot", "error parsing line: '%s'", curline->String().Data()); - continue; - } - gr->SetPoint(nlines-iline, nlines-iline, vmem/1024.); - iline++; - } - return gr; -} - -//////////////////////////////////////////////////////////////////////////////// -///actions of select all/clear all button - -void TProofProgressMemoryPlot::Select(Int_t id) -{ - Int_t nen = fWorkers->GetNumberOfEntries(); - Bool_t sel = id ? 0 : 1; - - for (Int_t ie=0; ieSelect(ie, sel); - } -} diff --git a/gui/sessionviewer/src/TSessionDialogs.cxx b/gui/sessionviewer/src/TSessionDialogs.cxx deleted file mode 100644 index 95e8f65f2ca11..0000000000000 --- a/gui/sessionviewer/src/TSessionDialogs.cxx +++ /dev/null @@ -1,1374 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Marek Biskup, Jakub Madejczyk, Bertrand Bellenot 10/08/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -/** \class TSessionDialogs - \ingroup sessionviewer - -This file defines several dialogs that are used by TSessionViewer. -The following dialogs are available: TNewChainDlg and TNewQueryDlg. - -*/ - - -#include "TSessionDialogs.h" -#include "TSessionViewer.h" -#include "TROOT.h" -#include "TSystem.h" -#include "TGButton.h" -#include "TList.h" -#include "TChain.h" -#include "TDSet.h" -#include "TGTextEntry.h" -#include "TGTextBuffer.h" -#include "TGNumberEntry.h" -#include "TGLabel.h" -#include "TGListView.h" -#include "TGFSContainer.h" -#include "TGFileDialog.h" -#include "TGListTree.h" -#include "TInterpreter.h" -#include "TApplication.h" -#include "TObjString.h" -#include "TGTableLayout.h" -#include "TProof.h" -#include "TFileInfo.h" -#include "TGMsgBox.h" -#include "TRegexp.h" -#include "TVirtualX.h" - -ClassImp(TNewChainDlg); -ClassImp(TNewQueryDlg); - -/* not yet used -static const char *gParTypes[] = { - "Par files", "*.par", - "All files", "*", - 0, 0 -}; -*/ - -static const char *gDatasetTypes[] = { - "ROOT files", "*.root", - "All files", "*", - 0, 0 -}; - -static const char *gFileTypes[] = { - "C files", "*.[C|c]*", - "ROOT files", "*.root", - "All files", "*", - 0, 0 -}; - -////////////////////////////////////////////////////////////////////////// -// New Chain Dialog -////////////////////////////////////////////////////////////////////////// - -//////////////////////////////////////////////////////////////////////////////// -/// Create a new chain dialog box. Used to list chains present in memory -/// and offers the possibility to create new ones by executing macros -/// directly from the associate file container. - -TNewChainDlg::TNewChainDlg(const TGWindow *p, const TGWindow *main) : - TGTransientFrame(p, main, 350, 300, kVerticalFrame) -{ - Pixel_t backgnd; - if (!p || !main) return; - SetCleanup(kDeepCleanup); - fClient->GetColorByName("#F0FFF0", backgnd); - AddFrame(new TGLabel(this, new TGHotString("List of Chains in Memory :")), - new TGLayoutHints(kLHintsLeft, 5, 5, 7, 2) ); - - // Add TGListView used to show objects in memory - fListView = new TGListView(this, 300, 100); - fLVContainer = new TGLVContainer(fListView, kSunkenFrame, GetWhitePixel()); - fLVContainer->Associate(fListView); - fLVContainer->SetViewMode(kLVSmallIcons); - fLVContainer->SetCleanup(kDeepCleanup); - AddFrame(fListView, new TGLayoutHints(kLHintsExpandX | kLHintsExpandY, 4, 4, 4, 4)); - - fListView->Connect("Clicked(TGLVEntry*, Int_t)", "TNewChainDlg", - this, "OnElementClicked(TGLVEntry* ,Int_t)"); - - // Add text entry showing type and name of user's selection - TGCompositeFrame* frmSel = new TGHorizontalFrame(this, 300, 100); - frmSel->SetCleanup(kDeepCleanup); - frmSel->AddFrame(new TGLabel(frmSel, new TGHotString("Selected chain :")), - new TGLayoutHints(kLHintsLeft | kLHintsCenterY, 5, 5, 5, 5) ); - fNameBuf = new TGTextBuffer(100); - fName = new TGTextEntry(frmSel, fNameBuf); - fName->Resize(200, fName->GetDefaultHeight()); - fName->Associate(this); - fName->SetEnabled(kFALSE); - fName->ChangeBackground(backgnd); - frmSel->AddFrame(fName, new TGLayoutHints(kLHintsLeft | kLHintsCenterY | kLHintsExpandX, 5, 5, 5, 5)); - AddFrame(frmSel, new TGLayoutHints(kLHintsExpandX, 5, 5, 5, 5)); - - AddFrame(new TGLabel(this, "Double-click on the macro to be executed to create a new Chain:"), - new TGLayoutHints(kLHintsCenterX, 5, 5, 5, 2)); - - // Add TGListview / TGFileContainer to allow user to execute Macros - // for the creation of new TChains / TDSets - TGListView* lv = new TGListView(this, 300, 100); - AddFrame(lv,new TGLayoutHints(kLHintsExpandX | kLHintsExpandY, 5, 5, 2, 5)); - - Pixel_t white; - gClient->GetColorByName("white",white); - fContents = new TGFileContainer(lv, kSunkenFrame, white); - fContents->SetCleanup(kDeepCleanup); - fContents->SetFilter("*.[C|c]*"); - fContents->SetViewMode(kLVSmallIcons); - fContents->Associate(this); - fContents->SetDefaultHeaders(); - fContents->DisplayDirectory(); - fContents->AddFile(".."); // up level directory - fContents->Resize(); - fContents->StopRefreshTimer(); // stop refreshing - - // position relative to the parent's window - Window_t wdummy; - Int_t ax, ay; - gVirtualX->TranslateCoordinates( main->GetId(), - fClient->GetDefaultRoot()->GetId(), - 0, 0, ax, ay, wdummy); - Move(ax + 200, ay + 35); - - TGCompositeFrame *tmp; - AddFrame(tmp = new TGCompositeFrame(this, 140, 20, kHorizontalFrame), - new TGLayoutHints(kLHintsLeft | kLHintsExpandX)); - tmp->SetCleanup(kDeepCleanup); - // Apply and Close buttons - tmp->AddFrame(fOkButton = new TGTextButton(tmp, "&Ok", 0), - new TGLayoutHints(kLHintsExpandX, 5, 5, 5, 5)); - tmp->AddFrame(fCancelButton = new TGTextButton(tmp, "&Cancel", 1), - new TGLayoutHints(kLHintsExpandX, 5, 5, 5, 5)); - fOkButton->Associate(this); - fCancelButton->Associate(this); - fOkButton->SetEnabled(kFALSE); - - SetWindowName("Chain Selection Dialog"); - MapSubwindows(); - Layout(); - Resize(GetDefaultSize()); - MapWindow(); - UpdateList(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete chain dialog. - -TNewChainDlg::~TNewChainDlg() -{ - if (IsZombie()) return; - delete fLVContainer; - delete fContents; - Cleanup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Emits OnElementSelected signal if dset is not zero. - -void TNewChainDlg::OnElementSelected(TObject *obj) -{ - if (obj && (obj->IsA() == TChain::Class() || - obj->IsA() == TDSet::Class())) { - Emit("OnElementSelected(TObject *)", (Longptr_t)obj); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle click in the Memory list view and put the type -/// and name of selected object in the text entry. - -void TNewChainDlg::OnElementClicked(TGLVEntry *entry, Int_t) -{ - fChain = (TObject *)entry->GetUserData(); - if (fChain->IsA() == TChain::Class()) { - TString s = TString::Format("%s : %s" , ((TChain *)fChain)->GetTitle(), - ((TChain *)fChain)->GetName()); - fName->SetText(s); - } - else if (fChain->IsA() == TDSet::Class()) { - TString s = TString::Format("%s : %s" , ((TDSet *)fChain)->GetName(), - ((TDSet *)fChain)->GetObjName()); - fName->SetText(s); - } - fOkButton->SetEnabled(kTRUE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update Memory list view. - -void TNewChainDlg::UpdateList() -{ - TGLVEntry *item=0; - TObject *obj = 0; - fChains = gROOT->GetListOfDataSets(); - fLVContainer->RemoveAll(); - if (!fChains) return; - TIter next(fChains); - // loop on the list of chains/datasets in memory, - // and fill the associated listview - while ((obj = (TObject *)next())) { - item = 0; - if (obj->IsA() == TChain::Class()) { - const char *title = ((TChain *)obj)->GetTitle(); - if (!title[0]) - ((TChain *)obj)->SetTitle("TChain"); - item = new TGLVEntry(fLVContainer, ((TChain *)obj)->GetName(), - ((TChain *)obj)->GetTitle()); - } - else if (obj->IsA() == TDSet::Class()) { - item = new TGLVEntry(fLVContainer, ((TDSet *)obj)->GetObjName(), - ((TDSet *)obj)->GetName()); - } - if (item) { - item->SetUserData(obj); - fLVContainer->AddItem(item); - } - } - fClient->NeedRedraw(fLVContainer); - Resize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display content of directory. - -void TNewChainDlg::DisplayDirectory(const TString &fname) -{ - fContents->SetDefaultHeaders(); - gSystem->ChangeDirectory(fname); - fContents->ChangeDirectory(fname); - fContents->DisplayDirectory(); - fContents->AddFile(".."); // up level directory - Resize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle double click in the File container. - -void TNewChainDlg::OnDoubleClick(TGLVEntry* f, Int_t btn) -{ - if (btn!=kButton1) return; - gVirtualX->SetCursor(fContents->GetId(),gVirtualX->CreateCursor(kWatch)); - - TString name(f->GetTitle()); - - // Check if the file is a root macro file type - if (name.Contains(".C")) { - // form the command - TString command = TString::Format(".x %s/%s", - gSystem->UnixPathName(fContents->GetDirectory()), - name.Data()); - // and process - gApplication->ProcessLine(command.Data()); - UpdateList(); - } else { - // if double clicked on a directory, then display it - DisplayDirectory(name); - } - gVirtualX->SetCursor(fContents->GetId(),gVirtualX->CreateCursor(kPointer)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process messages for new chain dialog. - -Bool_t TNewChainDlg::ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t) -{ - switch (GET_MSG(msg)) { - case kC_COMMAND: - switch (GET_SUBMSG(msg)) { - case kCM_BUTTON: - switch (parm1) { - - case 0: - // Apply button - fOkButton->SetEnabled(kFALSE); - OnElementSelected(fChain); - DeleteWindow(); - break; - - case 1: - // Close button - fChain = 0; - DeleteWindow(); - break; - } - break; - default: - break; - } - break; - - case kC_CONTAINER: - switch (GET_SUBMSG(msg)) { - case kCT_ITEMDBLCLICK: - if (parm1==kButton1) { - TGLVEntry *lv_entry = (TGLVEntry *)fContents->GetLastActive(); - if (lv_entry) OnDoubleClick(lv_entry, parm1); - } - break; - } - break; - default: - break; - } - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close file dialog. - -void TNewChainDlg::CloseWindow() -{ - DeleteWindow(); -} - - -////////////////////////////////////////////////////////////////////////// -// New Query Dialog -////////////////////////////////////////////////////////////////////////// - -//////////////////////////////////////////////////////////////////////////////// -/// Create a new Query dialog, used by the Session Viewer, to Edit a Query if -/// the editmode flag is set, or to create a new one if not set. - -TNewQueryDlg::TNewQueryDlg(TSessionViewer *gui, Int_t Width, Int_t Height, - TQueryDescription *query, Bool_t editmode) : - TGTransientFrame(gClient->GetRoot(), gui, Width, Height) -{ - Window_t wdummy; - Int_t ax, ay; - fEditMode = editmode; - fModified = kFALSE; - fChain = 0; - fQuery = query; - if (fQuery && fQuery->fChain) { - fChain = fQuery->fChain; - } - Build(gui); - // if in edit mode, update fields with query description data - if (editmode && query) - UpdateFields(query); - else if (!editmode) { - TQueryDescription *fquery; - fquery = (TQueryDescription *)fViewer->GetActDesc()->fQueries->Last(); - if(fquery) - fTxtQueryName->SetText(fquery->fQueryName); - else - fTxtQueryName->SetText("Query 1"); - } - MapSubwindows(); - Resize(Width, Height); - // hide options frame - fFrmNewQuery->HideFrame(fFrmMore); - fBtnMore->SetText(" More >> "); - SetWMSizeHints(Width+5, Height+25, Width+5, Height+25, 1, 1); - ChangeOptions(GetOptions() | kFixedSize); - Layout(); - SetWindowName("Query Dialog"); - // Position relative to parent - gVirtualX->TranslateCoordinates( fViewer->GetId(), - fClient->GetDefaultRoot()->GetId(), - 0, 0, ax, ay, wdummy); - Move(ax + fViewer->GetWidth()/2, ay + 35); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete query dialog. - -TNewQueryDlg::~TNewQueryDlg() -{ - if (IsZombie()) return; - Cleanup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build the "new query" dialog. - -void TNewQueryDlg::Build(TSessionViewer *gui) -{ - TGButton* btnTmp; - fViewer = gui; - SetLayoutManager(new TGVerticalLayout(this)); - SetCleanup(kDeepCleanup); - SetMinWidth(500); - fFrmNewQuery = new TGGroupFrame(this, "New Query"); - fFrmNewQuery->SetCleanup(kDeepCleanup); - - AddFrame(fFrmNewQuery, new TGLayoutHints(kLHintsExpandX | - kLHintsExpandY, 2, 2, 2, 2)); - fFrmNewQuery->SetLayoutManager(new TGTableLayout(fFrmNewQuery, 6, 5)); - - // add "Query Name" label and text entry - fFrmNewQuery->AddFrame(new TGLabel(fFrmNewQuery, "Query Name :"), - new TGTableLayoutHints(0, 1, 0, 1, kLHintsCenterY, 0, 5, 4, 0)); - fFrmNewQuery->AddFrame(fTxtQueryName = new TGTextEntry(fFrmNewQuery, - (const char *)0, 1), new TGTableLayoutHints(1, 2, 0, 1, - kLHintsCenterY, 5, 5, 4, 0)); - - // add "TChain" label and text entry - fFrmNewQuery->AddFrame(new TGLabel(fFrmNewQuery, "TChain :"), - new TGTableLayoutHints(0, 1, 1, 2, kLHintsCenterY, 0, 5, 4, 0)); - fFrmNewQuery->AddFrame(fTxtChain = new TGTextEntry(fFrmNewQuery, - (const char *)0, 2), new TGTableLayoutHints(1, 2, 1, 2, - kLHintsCenterY, 5, 5, 4, 0)); - fTxtChain->SetToolTipText("Specify TChain or TDSet from memory or file"); - fTxtChain->SetEnabled(kFALSE); - // add "Browse" button - fFrmNewQuery->AddFrame(btnTmp = new TGTextButton(fFrmNewQuery, "Browse..."), - new TGTableLayoutHints(2, 3, 1, 2, kLHintsCenterY, 5, 0, 4, 8)); - btnTmp->Connect("Clicked()", "TNewQueryDlg", this, "OnBrowseChain()"); - - // add "Selector" label and text entry - fFrmNewQuery->AddFrame(new TGLabel(fFrmNewQuery, "Selector :"), - new TGTableLayoutHints(0, 1, 2, 3, kLHintsCenterY, 0, 5, 0, 0)); - fFrmNewQuery->AddFrame(fTxtSelector = new TGTextEntry(fFrmNewQuery, - (const char *)0, 3), new TGTableLayoutHints(1, 2, 2, 3, - kLHintsCenterY, 5, 5, 0, 0)); - // add "Browse" button - fFrmNewQuery->AddFrame(btnTmp = new TGTextButton(fFrmNewQuery, "Browse..."), - new TGTableLayoutHints(2, 3, 2, 3, kLHintsCenterY, 5, 0, 0, 8)); - btnTmp->Connect("Clicked()", "TNewQueryDlg", this, "OnBrowseSelector()"); - - // add "Less <<" ("More >>") button - fFrmNewQuery->AddFrame(fBtnMore = new TGTextButton(fFrmNewQuery, " Less << "), - new TGTableLayoutHints(2, 3, 4, 5, kLHintsCenterY, 5, 5, 4, 0)); - fBtnMore->Connect("Clicked()", "TNewQueryDlg", this, "OnNewQueryMore()"); - - // add (initially hidden) options frame - fFrmMore = new TGCompositeFrame(fFrmNewQuery, 200, 200); - fFrmMore->SetCleanup(kDeepCleanup); - - fFrmNewQuery->AddFrame(fFrmMore, new TGTableLayoutHints(0, 3, 5, 6, - kLHintsExpandX | kLHintsExpandY)); - fFrmMore->SetLayoutManager(new TGTableLayout(fFrmMore, 4, 3)); - - // add "Options" label and text entry - fFrmMore->AddFrame(new TGLabel(fFrmMore, "Options :"), - new TGTableLayoutHints(0, 1, 0, 1, kLHintsCenterY, 0, 5, 0, 0)); - fFrmMore->AddFrame(fTxtOptions = new TGTextEntry(fFrmMore, - (const char *)0, 4), new TGTableLayoutHints(1, 2, 0, 1, 0, 22, - 0, 0, 8)); - fTxtOptions->SetText("ASYN"); - - // add "Nb Entries" label and number entry - fFrmMore->AddFrame(new TGLabel(fFrmMore, "Nb Entries :"), - new TGTableLayoutHints(0, 1, 1, 2, kLHintsCenterY, 0, 5, 0, 0)); - fFrmMore->AddFrame(fNumEntries = new TGNumberEntry(fFrmMore, 0, 5, -1, - TGNumberFormat::kNESInteger, TGNumberFormat::kNEAAnyNumber, - TGNumberFormat::kNELNoLimits), new TGTableLayoutHints(1, 2, 1, 2, - 0, 22, 0, 0, 8)); - // coverity[negative_returns]: no problem with -1, the format is kNESInteger - fNumEntries->SetIntNumber(-1); - // add "First Entry" label and number entry - fFrmMore->AddFrame(new TGLabel(fFrmMore, "First entry :"), - new TGTableLayoutHints(0, 1, 2, 3, kLHintsCenterY, 0, 5, 0, 0)); - fFrmMore->AddFrame(fNumFirstEntry = new TGNumberEntry(fFrmMore, 0, 5, -1, - TGNumberFormat::kNESInteger, TGNumberFormat::kNEANonNegative, - TGNumberFormat::kNELNoLimits), new TGTableLayoutHints(1, 2, 2, 3, 0, - 22, 0, 0, 8)); - - // add "Event list" label and text entry - fFrmMore->AddFrame(new TGLabel(fFrmMore, "Event list :"), - new TGTableLayoutHints(0, 1, 3, 4, kLHintsCenterY, 0, 5, 0, 0)); - fFrmMore->AddFrame(fTxtEventList = new TGTextEntry(fFrmMore, - (const char *)0, 6), new TGTableLayoutHints(1, 2, 3, 4, 0, 22, - 5, 0, 0)); - // add "Browse" button - fFrmMore->AddFrame(btnTmp = new TGTextButton(fFrmMore, "Browse..."), - new TGTableLayoutHints(2, 3, 3, 4, 0, 6, 0, 0, 8)); - btnTmp->Connect("Clicked()", "TNewQueryDlg", this, "OnBrowseEventList()"); - - fTxtQueryName->Associate(this); - fTxtChain->Associate(this); - fTxtSelector->Associate(this); - fTxtOptions->Associate(this); - fNumEntries->Associate(this); - fNumFirstEntry->Associate(this); - fTxtEventList->Associate(this); - - fTxtQueryName->Connect("TextChanged(char*)", "TNewQueryDlg", this, - "SettingsChanged()"); - fTxtChain->Connect("TextChanged(char*)", "TNewQueryDlg", this, - "SettingsChanged()"); - fTxtSelector->Connect("TextChanged(char*)", "TNewQueryDlg", this, - "SettingsChanged()"); - fTxtOptions->Connect("TextChanged(char*)", "TNewQueryDlg", this, - "SettingsChanged()"); - fNumEntries->Connect("ValueChanged(Long_t)", "TNewQueryDlg", this, - "SettingsChanged()"); - fNumFirstEntry->Connect("ValueChanged(Long_t)", "TNewQueryDlg", this, - "SettingsChanged()"); - fTxtEventList->Connect("TextChanged(char*)", "TNewQueryDlg", this, - "SettingsChanged()"); - - TGCompositeFrame *tmp; - AddFrame(tmp = new TGCompositeFrame(this, 140, 20, kHorizontalFrame), - new TGLayoutHints(kLHintsLeft | kLHintsExpandX)); - tmp->SetCleanup(kDeepCleanup); - // Add "Save" and "Save & Submit" buttons if we are in edition mode - // or "Add" and "Add & Submit" if we are not in edition mode. - if (fEditMode) { - fBtnSave = new TGTextButton(tmp, "Save"); - fBtnSubmit = new TGTextButton(tmp, "Save && Submit"); - } - else { - fBtnSave = new TGTextButton(tmp, "Add"); - fBtnSubmit = new TGTextButton(tmp, "Add && Submit"); - } - tmp->AddFrame(fBtnSave, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, - 3, 3, 3, 3)); - tmp->AddFrame(fBtnSubmit, new TGLayoutHints(kLHintsLeft | kLHintsExpandX, - 3, 3, 3, 3)); - fBtnSave->Connect("Clicked()", "TNewQueryDlg", this, "OnBtnSaveClicked()"); - fBtnSubmit->Connect("Clicked()", "TNewQueryDlg", this, "OnBtnSubmitClicked()"); - tmp->AddFrame(fBtnClose = new TGTextButton(tmp, "Close"), - new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 3, 3, 3, 3)); - fBtnClose->Connect("Clicked()", "TNewQueryDlg", this, "OnBtnCloseClicked()"); - fBtnSave->SetState(kButtonDisabled); - fBtnSubmit->SetState(kButtonDisabled); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Called when window is closed via the window manager. - -void TNewQueryDlg::CloseWindow() -{ - DeleteWindow(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show/hide options frame and update button text accordingly. - -void TNewQueryDlg::OnNewQueryMore() -{ - if (fFrmNewQuery->IsVisible(fFrmMore)) { - fFrmNewQuery->HideFrame(fFrmMore); - fBtnMore->SetText(" More >> "); - } - else { - fFrmNewQuery->ShowFrame(fFrmMore); - fBtnMore->SetText(" Less << "); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Call new chain dialog. - -void TNewQueryDlg::OnBrowseChain() -{ - TNewChainDlg *dlg = new TNewChainDlg(fClient->GetRoot(), this); - dlg->Connect("OnElementSelected(TObject *)", "TNewQueryDlg", - this, "OnElementSelected(TObject *)"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle OnElementSelected signal coming from new chain dialog. - -void TNewQueryDlg::OnElementSelected(TObject *obj) -{ - if (obj) { - fChain = obj; - if (obj->IsA() == TChain::Class()) - fTxtChain->SetText(((TChain *)fChain)->GetName()); - else if (obj->IsA() == TDSet::Class()) - fTxtChain->SetText(((TDSet *)fChain)->GetObjName()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Open file browser to choose selector macro. - -void TNewQueryDlg::OnBrowseSelector() -{ - TGFileInfo fi; - fi.fFileTypes = gFileTypes; - new TGFileDialog(fClient->GetRoot(), this, kFDOpen, &fi); - if (!fi.fFilename) return; - fTxtSelector->SetText(gSystem->UnixPathName(fi.fFilename)); -} - -//////////////////////////////////////////////////////////////////////////////// -///Browse event list - -void TNewQueryDlg::OnBrowseEventList() -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save current settings in main session viewer. - -void TNewQueryDlg::OnBtnSaveClicked() -{ - // if we are in edition mode and query description is valid, - // use it, otherwise create a new one - TQueryDescription *newquery; - if (fEditMode && fQuery) - newquery = fQuery; - else - newquery = new TQueryDescription(); - - // update query description fields - newquery->fSelectorString = fTxtSelector->GetText(); - if (fChain) { - newquery->fTDSetString = fChain->GetName(); - newquery->fChain = fChain; - } - else { - newquery->fTDSetString = ""; - newquery->fChain = 0; - } - newquery->fQueryName = fTxtQueryName->GetText(); - newquery->fOptions.Form("%s",fTxtOptions->GetText()); - newquery->fNoEntries = fNumEntries->GetIntNumber(); - newquery->fFirstEntry = fNumFirstEntry->GetIntNumber(); - newquery->fNbFiles = 0; - newquery->fResult = 0; - - if (newquery->fChain) { - if (newquery->fChain->IsA() == TChain::Class()) - newquery->fNbFiles = ((TChain *)newquery->fChain)->GetListOfFiles()->GetEntriesFast(); - else if (newquery->fChain->IsA() == TDSet::Class()) - newquery->fNbFiles = ((TDSet *)newquery->fChain)->GetListOfElements()->GetSize(); - } - if (!fEditMode) { - // if not in editor mode, create a new list tree item - // and set user data to the newly created query description - newquery->fResult = 0; - newquery->fStatus = TQueryDescription::kSessionQueryCreated; - - TQueryDescription *fquery; - fquery = (TQueryDescription *)fViewer->GetActDesc()->fQueries->FindObject(newquery->fQueryName); - while (fquery) { - int e = 1, j = 0, idx = 0; - const char *name = fquery->fQueryName; - for (int i=strlen(name)-1;i>0;i--) { - if (isdigit(name[i])) { - idx += (name[i]-'0') * e; - e *= 10; - j++; - } - else - break; - } - if (idx > 0) { - idx++; - newquery->fQueryName.Remove(strlen(name)-j,j); - newquery->fQueryName.Append(Form("%d",idx)); - } - else - newquery->fQueryName.Append(" 1"); - fquery = (TQueryDescription *)fViewer->GetActDesc()->fQueries->FindObject(newquery->fQueryName); - } - fTxtQueryName->SetText(newquery->fQueryName); - fViewer->GetActDesc()->fQueries->Add((TObject *)newquery); - TGListTreeItem *item = fViewer->GetSessionHierarchy()->FindChildByData( - fViewer->GetSessionItem(), fViewer->GetActDesc()); - TGListTreeItem *item2 = fViewer->GetSessionHierarchy()->AddItem(item, - newquery->fQueryName, fViewer->GetQueryConPict(), fViewer->GetQueryConPict()); - item2->SetUserData(newquery); - fViewer->GetSessionHierarchy()->OpenItem(item); - fViewer->GetSessionHierarchy()->ClearHighlighted(); - fViewer->GetSessionHierarchy()->HighlightItem(item2); - fViewer->GetSessionHierarchy()->SetSelected(item2); - fViewer->OnListTreeClicked(item2, 1, 0, 0); - } - else { - // else if in editor mode, just update user data with modified - // query description - TGListTreeItem *item = fViewer->GetSessionHierarchy()->GetSelected(); - fViewer->GetSessionHierarchy()->RenameItem(item, newquery->fQueryName); - item->SetUserData(newquery); - } - // update list tree - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); - fTxtQueryName->SelectAll(); - fTxtQueryName->SetFocus(); - fViewer->WriteConfiguration(); - fModified = kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save and submit query description. - -void TNewQueryDlg::OnBtnSubmitClicked() -{ - OnBtnSaveClicked(); - fViewer->GetQueryFrame()->OnBtnSubmit(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close dialog. - -void TNewQueryDlg::OnBtnCloseClicked() -{ - Int_t result = kMBNo; - if (fModified) { - new TGMsgBox(fClient->GetRoot(), this, "Modified Settings", - "Do you wish to SAVE changes ?", 0, - kMBYes | kMBNo | kMBCancel, &result); - if (result == kMBYes) { - OnBtnSaveClicked(); - } - } - if (result == kMBNo) { - DeleteWindow(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display dialog and set focus to query name text entry. - -void TNewQueryDlg::Popup() -{ - MapWindow(); - fTxtQueryName->SetFocus(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Settings have changed, update GUI accordingly. - -void TNewQueryDlg::SettingsChanged() -{ - if (fEditMode && fQuery) { - if ((strcmp(fQuery->fSelectorString.Data(), fTxtSelector->GetText())) || - (strcmp(fQuery->fQueryName.Data(), fTxtQueryName->GetText())) || - (strcmp(fQuery->fOptions.Data(), fTxtOptions->GetText())) || - (fQuery->fNoEntries != fNumEntries->GetIntNumber()) || - (fQuery->fFirstEntry != fNumFirstEntry->GetIntNumber()) || - (fQuery->fChain != fChain)) { - fModified = kTRUE; - } - else { - fModified = kFALSE; - } - } - else { - if ((fTxtQueryName->GetText()) && - ((fTxtQueryName->GetText()) || - (fTxtChain->GetText()))) - fModified = kTRUE; - else - fModified = kFALSE; - } - if (fModified) { - fBtnSave->SetState(kButtonUp); - fBtnSubmit->SetState(kButtonUp); - } - else { - fBtnSave->SetState(kButtonDisabled); - fBtnSubmit->SetState(kButtonDisabled); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update entry fields with query description values. - -void TNewQueryDlg::UpdateFields(TQueryDescription *desc) -{ - fQuery = desc; - fTxtQueryName->SetText(desc->fQueryName); - fTxtChain->SetText(""); - if (desc->fChain) - fTxtChain->SetText(desc->fTDSetString); - fTxtSelector->SetText(desc->fSelectorString); - fTxtOptions->SetText(desc->fOptions); - fNumEntries->SetIntNumber(desc->fNoEntries); - fNumFirstEntry->SetIntNumber(desc->fFirstEntry); - fTxtEventList->SetText(desc->fEventList); -} -//////////////////////////////////////////////////////////////////////////////// -/// Process messages for new query dialog. -/// Essentially used to navigate between text entry fields. - -Bool_t TNewQueryDlg::ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t) -{ - switch (GET_MSG(msg)) { - case kC_TEXTENTRY: - switch (GET_SUBMSG(msg)) { - case kTE_ENTER: - case kTE_TAB: - switch (parm1) { - case 1: // Query Name - fTxtChain->SelectAll(); - fTxtChain->SetFocus(); - break; - case 2: // Chain Name - fTxtSelector->SelectAll(); - fTxtSelector->SetFocus(); - break; - case 3: // Selector Name - fTxtOptions->SelectAll(); - fTxtOptions->SetFocus(); - break; - case 4: // Options - fTxtEventList->SelectAll(); - fTxtEventList->SetFocus(); - break; - case 6: // Event List - fTxtQueryName->SelectAll(); - fTxtQueryName->SetFocus(); - break; - } - break; - - default: - break; - } - break; - - default: - break; - } - return kTRUE; -} - -////////////////////////////////////////////////////////////////////////// -// Upload DataSet Dialog -////////////////////////////////////////////////////////////////////////// - -//////////////////////////////////////////////////////////////////////////////// -/// Create a Upload DataSet dialog box. Used to create and upload a dataset - -TUploadDataSetDlg::TUploadDataSetDlg(TSessionViewer *gui, Int_t w, Int_t h) : - TGTransientFrame(gClient->GetRoot(), gui, w, h) -{ - fUploading = kFALSE; - if (!gui) return; - fViewer = gui; - - SetCleanup(kDeepCleanup); - TGHorizontalFrame *hFrame1 = new TGHorizontalFrame(this); - hFrame1->SetCleanup(kDeepCleanup); - hFrame1->AddFrame(new TGLabel(hFrame1,"Name of DataSet :"), - new TGLayoutHints(kLHintsLeft | kLHintsCenterY, - 10, 10, 5, 5)); - fDSetName = new TGTextEntry(hFrame1, new TGTextBuffer(50)); - fDSetName->SetText("DataSet1"); - fDSetName->Resize(150, fDSetName->GetDefaultHeight()); - hFrame1->AddFrame(fDSetName, new TGLayoutHints(kLHintsLeft | kLHintsCenterY, - 10, 10, 5, 5)); - AddFrame(hFrame1, new TGLayoutHints(kLHintsLeft | kLHintsTop | kLHintsExpandX, - 2, 2, 2, 2)); - - // "DataSet Files" group frame - TGGroupFrame *groupFrame1 = new TGGroupFrame(this, "DataSet Files"); - groupFrame1->SetCleanup(kDeepCleanup); - - // horizontal frame for files location URL - TGHorizontalFrame *hFrame11 = new TGHorizontalFrame(groupFrame1); - hFrame11->SetCleanup(kDeepCleanup); - hFrame11->AddFrame(new TGLabel(hFrame11,"Location URL :"), - new TGLayoutHints(kLHintsLeft | kLHintsCenterY, - 10, 10, 5, 5)); - fLocationURL = new TGTextEntry(hFrame11, new TGTextBuffer(150)); - fLocationURL->SetToolTipText("Enter location URL (i.e \"root://host//path/to/file.root\")"); - fLocationURL->Resize(210, fLocationURL->GetDefaultHeight()); - hFrame11->AddFrame(fLocationURL, new TGLayoutHints(kLHintsLeft | - kLHintsCenterY, 10, 10, 5, 5)); - fAddButton = new TGTextButton(hFrame11, " Add >> ", 0); - fAddButton->SetToolTipText("Add file(s) to the list"); - fAddButton->Associate(this); - hFrame11->AddFrame(fAddButton, new TGLayoutHints(kLHintsLeft | kLHintsCenterY | - kLHintsExpandX, 5, 10, 5, 5)); - groupFrame1->AddFrame(hFrame11, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX, 2, 2, 2, 2)); - // horizontal frame for the list view displaying list of files - // and for a vertical frame with control buttons - TGHorizontalFrame *hFrame2 = new TGHorizontalFrame(groupFrame1); - hFrame2->SetCleanup(kDeepCleanup); - - // list view - // Add TGListView used to show list of files - fListView = new TGListView(hFrame2, 300, 100); - fLVContainer = new TGLVContainer(fListView, kSunkenFrame, GetWhitePixel()); - fLVContainer->Associate(fListView); - fLVContainer->SetViewMode(kLVDetails); - fLVContainer->SetCleanup(kDeepCleanup); - fLVContainer->SetHeaders(1); - fLVContainer->SetHeader("File Name", kTextLeft, kTextLeft , 0); - hFrame2->AddFrame(fListView, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX | kLHintsExpandY, 2, 2, 10, 10)); - - // vertical frame for control buttons - TGVerticalFrame *vFrame1 = new TGVerticalFrame(hFrame2); - vFrame1->SetCleanup(kDeepCleanup); - - fBrowseButton = new TGTextButton(vFrame1, " Browse... ", 1); - fBrowseButton->SetToolTipText("Add file(s) to the list"); - fBrowseButton->Associate(this); - vFrame1->AddFrame(fBrowseButton, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX, 15, 5, 5, 5)); - fRemoveButton = new TGTextButton(vFrame1, " Remove ", 2); - fRemoveButton->SetToolTipText("Remove selected file from the list"); - fRemoveButton->Associate(this); - vFrame1->AddFrame(fRemoveButton, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX, 15, 5, 5, 5)); - fClearButton = new TGTextButton(vFrame1, " Clear ", 3); - fClearButton->SetToolTipText("Clear list of files"); - fClearButton->Associate(this); - vFrame1->AddFrame(fClearButton, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX, 15, 5, 5, 5)); - - fOverwriteDSet = new TGCheckButton(vFrame1, "Overwrite DataSet"); - fOverwriteDSet->SetToolTipText("Overwrite DataSet"); - vFrame1->AddFrame(fOverwriteDSet, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX, 15, 5, 5, 5)); - fOverwriteFiles = new TGCheckButton(vFrame1, "Overwrite Files"); - fOverwriteFiles->SetToolTipText("Overwrite files in DataSet"); - vFrame1->AddFrame(fOverwriteFiles, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX, 15, 5, 5, 5)); - fAppendFiles = new TGCheckButton(vFrame1, "Append Files"); - fAppendFiles->SetToolTipText("Append files in DataSet"); - vFrame1->AddFrame(fAppendFiles, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX, 15, 5, 5, 5)); - - fOverwriteDSet->Connect("Toggled(Bool_t)", "TUploadDataSetDlg", this, - "OnOverwriteDataset(Bool_t)"); - fOverwriteFiles->Connect("Toggled(Bool_t)", "TUploadDataSetDlg", this, - "OnOverwriteFiles(Bool_t)"); - fAppendFiles->Connect("Toggled(Bool_t)", "TUploadDataSetDlg", this, - "OnAppendFiles(Bool_t)"); - - hFrame2->AddFrame(vFrame1, new TGLayoutHints(kLHintsRight | kLHintsTop | - kLHintsExpandY, 2, 2, 2, 2)); - groupFrame1->AddFrame(hFrame2, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX | kLHintsExpandY, 2, 2, 2, 2)); - - AddFrame(groupFrame1, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX, 5, 5, 2, 2)); - - // horizontal frame for destination URL - TGHorizontalFrame *hFrame3 = new TGHorizontalFrame(this); - hFrame3->SetCleanup(kDeepCleanup); - hFrame3->AddFrame(new TGLabel(hFrame3,"Destination URL :"), - new TGLayoutHints(kLHintsLeft | kLHintsCenterY, - 15, 10, 5, 5)); - fDestinationURL = new TGTextEntry(hFrame3, new TGTextBuffer(150)); - if (fViewer->GetActDesc()->fConnected && - fViewer->GetActDesc()->fAttached && - fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - // const char *dest = fViewer->GetActDesc()->fProof->GetDataPoolUrl(); - // fDestinationURL->SetText(dest); - } - fDestinationURL->SetToolTipText("Enter destination URL ( relative to \" root://host//proofpool/user/ \" )"); - fDestinationURL->Resize(305, fDestinationURL->GetDefaultHeight()); - hFrame3->AddFrame(fDestinationURL, new TGLayoutHints(kLHintsLeft | - kLHintsCenterY, 10, 15, 5, 5)); - AddFrame(hFrame3, new TGLayoutHints(kLHintsLeft | kLHintsTop | kLHintsExpandX, - 2, 2, 2, 2)); - - // horizontal frame for upload and close buttons - TGHorizontalFrame *hFrame4 = new TGHorizontalFrame(this); - hFrame4->SetCleanup(kDeepCleanup); - fUploadButton = new TGTextButton(hFrame4, "Upload DataSet", 10); - fUploadButton->SetToolTipText("Upload the dataset to the cluster"); - fUploadButton->Associate(this); - hFrame4->AddFrame(fUploadButton, new TGLayoutHints(kLHintsLeft | kLHintsCenterY | - kLHintsExpandX, 15, 15, 2, 2)); - fCloseDlgButton = new TGTextButton(hFrame4, "Close Dialog", 11); - fCloseDlgButton->SetToolTipText("Close the dialog"); - fCloseDlgButton->Associate(this); - hFrame4->AddFrame(fCloseDlgButton, new TGLayoutHints(kLHintsLeft | kLHintsCenterY | - kLHintsExpandX, 15, 15, 2, 2)); - AddFrame(hFrame4, new TGLayoutHints(kLHintsLeft | kLHintsTop | kLHintsExpandX, - 2, 2, 2, 2)); - - // position relative to the parent's window - Window_t wdummy; - Int_t ax, ay; - gVirtualX->TranslateCoordinates( gui->GetId(), - fClient->GetDefaultRoot()->GetId(), - 0, 0, ax, ay, wdummy); - Move(ax + 250, ay + 200); - - SetWindowName("Upload DataSet Dialog"); - MapSubwindows(); - MapWindow(); - - Resize(w, h); - SetWMSizeHints(w+5, h+5, w+5, h+5, 1, 1); - ChangeOptions(GetOptions() | kFixedSize); - Layout(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete chain dialog. - -TUploadDataSetDlg::~TUploadDataSetDlg() -{ - if (IsZombie()) return; - Cleanup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close upload dataset dialog. - -void TUploadDataSetDlg::CloseWindow() -{ - if (!fUploading) - DeleteWindow(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process messages for upload dataset dialog. - -Bool_t TUploadDataSetDlg::ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t) -{ - switch (GET_MSG(msg)) { - case kC_COMMAND: - switch (GET_SUBMSG(msg)) { - case kCM_BUTTON: - switch (parm1) { - case 0: - // Add button - if (fLocationURL->GetText()) - AddFiles(fLocationURL->GetText()); - break; - case 1: - // Add button - BrowseFiles(); - break; - case 2: - // Remove button - RemoveFile(); - break; - case 3: - // Clear button - ClearFiles(); - break; - case 10: - // Upload button - UploadDataSet(); - break; - case 11: - // Close button - CloseWindow(); - break; - } - break; - default: - break; - } - break; - - default: - break; - } - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add File name(s) from the file location URL to the list view. - -void TUploadDataSetDlg::AddFiles(const char *fileName) -{ - if (strlen(fileName) < 5) - return; - if (strstr(fileName,"*.")) { - // wildcarding case - void *filesDir = gSystem->OpenDirectory(gSystem->GetDirName(fileName)); - const char* ent; - TString filesExp(gSystem->BaseName(fileName)); - filesExp.ReplaceAll("*",".*"); - TRegexp rg(filesExp); - while ((ent = gSystem->GetDirEntry(filesDir))) { - TString entryString(ent); - if (entryString.Index(rg) != kNPOS && - gSystem->AccessPathName(Form("%s/%s", gSystem->GetDirName(fileName).Data(), - ent), kReadPermission) == kFALSE) { - TString text = TString::Format("%s/%s", - gSystem->UnixPathName(gSystem->GetDirName(fileName)), ent); - if (!fLVContainer->FindItem(text.Data())) { - TGLVEntry *entry = new TGLVEntry(fLVContainer, text.Data(), text.Data()); - entry->SetPictures(gClient->GetPicture("rootdb_t.xpm"), - gClient->GetPicture("rootdb_t.xpm")); - fLVContainer->AddItem(entry); - } - } - } - gSystem->FreeDirectory(filesDir); - } - else { - // single file - if (!fLVContainer->FindItem(fileName)) { - TGLVEntry *entry = new TGLVEntry(fLVContainer, fileName, fileName); - entry->SetPictures(gClient->GetPicture("rootdb_t.xpm"), - gClient->GetPicture("rootdb_t.xpm")); - fLVContainer->AddItem(entry); - } - } - // update list view - fListView->AdjustHeaders(); - fListView->Layout(); - fClient->NeedRedraw(fLVContainer); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add File name(s) from the file location URL to the list view. - -void TUploadDataSetDlg::AddFiles(TList *fileList) -{ - TObjString *el; - TIter next(fileList); - while ((el = (TObjString *) next())) { - TString fileName = TString::Format("%s/%s", - gSystem->UnixPathName(gSystem->GetDirName(el->GetString())), - gSystem->BaseName(el->GetString())); - // single file - if (!fLVContainer->FindItem(fileName.Data())) { - TGLVEntry *entry = new TGLVEntry(fLVContainer, fileName.Data(), fileName.Data()); - entry->SetPictures(gClient->GetPicture("rootdb_t.xpm"), - gClient->GetPicture("rootdb_t.xpm")); - fLVContainer->AddItem(entry); - } - } - // update list view - fListView->AdjustHeaders(); - fListView->Layout(); - fClient->NeedRedraw(fLVContainer); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Opens the TGFileDialog to allow user to select local file(s) to be added -/// in the list view of dataset files. - -void TUploadDataSetDlg::BrowseFiles() -{ - TGFileInfo fi; - fi.fFileTypes = gDatasetTypes; - fi.SetFilename("*.root"); - new TGFileDialog(fClient->GetRoot(), this, kFDOpen, &fi); - if (fi.fMultipleSelection && fi.fFileNamesList) { - AddFiles(fi.fFileNamesList); - } - else if (fi.fFilename) { - AddFiles(fi.fFilename); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear content of the list view. - -void TUploadDataSetDlg::ClearFiles() -{ - fLVContainer->RemoveAll(); - fListView->Layout(); - // update list view - fClient->NeedRedraw(fLVContainer); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Notification of Overwrite Dataset check button. - -void TUploadDataSetDlg::OnOverwriteDataset(Bool_t on) -{ - if (on && fAppendFiles->IsOn()) - fAppendFiles->SetState(kButtonUp); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Notification of Overwrite Files check button. - -void TUploadDataSetDlg::OnOverwriteFiles(Bool_t) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Notification of Append Files check button. - -void TUploadDataSetDlg::OnAppendFiles(Bool_t on) -{ - if (on && fOverwriteDSet->IsOn()) - fOverwriteDSet->SetState(kButtonUp); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove the selected entry from the list view. - -void TUploadDataSetDlg::RemoveFile() -{ - TGFrame *item = (TGFrame *)fLVContainer->GetLastActive(); - fLVContainer->RemoveItem(item); - // update list view - fListView->AdjustHeaders(); - fListView->Layout(); - fClient->NeedRedraw(fLVContainer); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Upload the dataset to the server. - -void TUploadDataSetDlg::UploadDataSet() -{ - Int_t retval; - TString fileList; - const char *dsetName = fDSetName->GetText(); - const char *destination = fDestinationURL->GetText(); - UInt_t flags = 0; - - if (fUploading) - return; - - if (!fViewer->GetActDesc()->fConnected || - !fViewer->GetActDesc()->fAttached || - !fViewer->GetActDesc()->fProof || - !fViewer->GetActDesc()->fProof->IsValid()) { - return; - } - - TList *skippedFiles = new TList(); - TList *datasetFiles = new TList(); - - // Format upload flags with user selection - if (fOverwriteDSet->IsOn()) - flags |= TProof::kOverwriteDataSet; - else - flags |= TProof::kNoOverwriteDataSet; - if (fOverwriteFiles->IsOn()) - flags |= TProof::kOverwriteAllFiles; - else - flags |= TProof::kOverwriteNoFiles; - if (fAppendFiles->IsOn()) { - flags |= TProof::kAppend; - if (flags & TProof::kNoOverwriteDataSet) - flags &= ~TProof::kNoOverwriteDataSet; - } - - Int_t ret = 0; - TIter next(fLVContainer->GetList()); - TGFrameElement *el; - TGLVEntry *entry; - - while ((el = (TGFrameElement *)next())) { - entry = (TGLVEntry *) el->fFrame; - const char *fname = gSystem->UnixPathName(entry->GetTitle()); - datasetFiles->Add(new TFileInfo(fname)); - } - fUploading = kTRUE; - fUploadButton->SetState(kButtonDisabled); - fCloseDlgButton->SetState(kButtonDisabled); - - if (strlen(destination) < 2) destination = 0; - - // GG 17/8/2012 -- BEGIN - // NB: UploadDataSet is obsolete; these changes are the minimal ones to make - // the build after the removal of an obsolete structure in TProof.h; - // but all this needs to be reconsidered. - ret = fViewer->GetActDesc()->fProof->UploadDataSet(dsetName, - datasetFiles, destination, flags, skippedFiles); -#if 0 - if (ret == TProof::kDataSetExists) { - // ask user what to do : - // cancel/overwrite and change option - new TGMsgBox(fClient->GetRoot(), this, "Upload DataSet", - TString::Format("The dataset \"%s\" already exists on the cluster ! Overwrite ?", - dsetName), kMBIconQuestion, kMBYes | kMBNo | kMBCancel | kMBAppend, - &retval); - if (retval == kMBYes) { - ret = fViewer->GetActDesc()->fProof->UploadDataSet(dsetName, - datasetFiles, destination, - TProof::kOverwriteDataSet | - TProof::kOverwriteNoFiles, - skippedFiles); - } - if (retval == kMBAppend) { - ret = fViewer->GetActDesc()->fProof->UploadDataSet(dsetName, - datasetFiles, destination, - TProof::kAppend | - TProof::kOverwriteNoFiles, - skippedFiles); - } - } -#endif - if (ret != 0) { - // Inform user - new TGMsgBox(fClient->GetRoot(), this, "Upload DataSet", - "Failed uploading dataset/files to the cluster", - kMBIconExclamation, kMBOk, &retval); - fUploading = kFALSE; - fUploadButton->SetState(kButtonUp); - fCloseDlgButton->SetState(kButtonUp); - return; - } - // Here we cope with files that existed on the cluster and were skipped. - if (skippedFiles->GetSize()) { - TIter nexts(skippedFiles); - while (TFileInfo *obj = (TFileInfo*)nexts()) { - // Notify user that file: obj->GetFirstUrl()->GetUrl() exists on - // the cluster and ask user what to do - new TGMsgBox(fClient->GetRoot(), this, "Upload DataSet", - TString::Format("The file \"%s\" already exists on the cluster ! Overwrite ?", - obj->GetFirstUrl()->GetUrl()), kMBIconQuestion, - kMBYes | kMBNo | kMBYesAll | kMBNoAll | kMBDismiss, &retval); - if (retval == kMBYesAll) { - ret = fViewer->GetActDesc()->fProof->UploadDataSet(dsetName, - skippedFiles, destination, - TProof::kAppend | - TProof::kOverwriteAllFiles); - if (ret != 0) { - // Inform user - new TGMsgBox(fClient->GetRoot(), this, "Upload DataSet", - TString::Format("Failed uploading \"%s\" to the cluster", - obj->GetFirstUrl()->GetUrl()), kMBIconExclamation, - kMBOk, &retval); - } - else { - new TGMsgBox(fClient->GetRoot(), this, "Upload DataSet", - "Files have been successfully uploaded to the cluster", - kMBIconAsterisk, kMBOk, &retval); - } - fUploading = kFALSE; - fUploadButton->SetState(kButtonUp); - fCloseDlgButton->SetState(kButtonUp); - return; - } - if ((retval == kMBNoAll) || (retval == kMBDismiss)) { - break; - } - if (retval == kMBYes) { - // Append one file to the dataSet - ret = fViewer->GetActDesc()->fProof->UploadDataSet(dsetName, - obj->GetFirstUrl()->GetUrl(), destination, - TProof::kAppend | TProof::kOverwriteAllFiles); - if (ret != 0) { - // Inform user - new TGMsgBox(fClient->GetRoot(), this, "Upload DataSet", - TString::Format("Failed uploading \"%s\" to the cluster", - obj->GetFirstUrl()->GetUrl()), kMBIconExclamation, - kMBOk, &retval); - } - else { - new TGMsgBox(fClient->GetRoot(), this, "Upload DataSet", - "Files have been successfully uploaded to the cluster", - kMBIconAsterisk, kMBOk, &retval); - } - } - } - skippedFiles->Clear(); - } - else { - new TGMsgBox(fClient->GetRoot(), this, "Upload DataSet", - "Files have been successfully uploaded to the cluster", - kMBIconAsterisk, kMBOk, &retval); - } - // GG 17/8/2012 -- END - - // finally, update list of datasets in session viewer - fViewer->GetSessionFrame()->UpdateListOfDataSets(); - fUploading = kFALSE; - fUploadButton->SetState(kButtonUp); - fCloseDlgButton->SetState(kButtonUp); -} - diff --git a/gui/sessionviewer/src/TSessionLogView.cxx b/gui/sessionviewer/src/TSessionLogView.cxx deleted file mode 100644 index 1542103485a0a..0000000000000 --- a/gui/sessionviewer/src/TSessionLogView.cxx +++ /dev/null @@ -1,153 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Bertrand Bellenot, Gerri Ganis 15/09/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#include "TSessionLogView.h" -#include "TSessionViewer.h" -#include "TProof.h" -#include "KeySymbols.h" - - -/** \class TSessionLogView - \ingroup sessionviewer - -Dialog used to display session logs from the session viewer - -*/ - - -ClassImp(TSessionLogView); - -//////////////////////////////////////////////////////////////////////////////// -/// Create an editor in a dialog. - -TSessionLogView::TSessionLogView(TSessionViewer *viewer, UInt_t w, UInt_t h) : - TGTransientFrame(gClient->GetRoot(), viewer, w, h) -{ - fViewer = viewer; - fTextView = new TGTextView(this, w, h, kSunkenFrame | kDoubleBorder); - fL1 = new TGLayoutHints(kLHintsExpandX | kLHintsExpandY, 3, 3, 3, 3); - AddFrame(fTextView, fL1); - - fClose = new TGTextButton(this, " &Close "); - fL2 = new TGLayoutHints(kLHintsBottom | kLHintsCenterX, 0, 0, 5, 5); - AddFrame(fClose, fL2); - - SetTitle(); - fViewer->SetLogWindow(this); - - MapSubwindows(); - - Resize(GetDefaultSize()); -} - -//////////////////////////////////////////////////////////////////////////////// - -TSessionLogView::~TSessionLogView() -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set title in editor window. - -void TSessionLogView::SetTitle() -{ - TString title; - title.Form("PROOF Processing Logs: %s", (fViewer->GetActDesc()->fProof ? - fViewer->GetActDesc()->fProof->GetMaster() : "")); - SetWindowName(title); - SetIconName(title); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show editor. - -void TSessionLogView::Popup() -{ - MapWindow(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load a text buffer in the editor. - -void TSessionLogView::AddBuffer(const char *buffer) -{ - TGText txt; - txt.LoadBuffer(buffer); - fTextView->AddText(&txt); - fTextView->ShowBottom(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear log window. - -void TSessionLogView::ClearLogView() -{ - fTextView->Clear(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load a text buffer in the editor. - -void TSessionLogView::LoadBuffer(const char *buffer) -{ - fTextView->LoadBuffer(buffer); - fTextView->ShowBottom(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load a file in the editor. - -void TSessionLogView::LoadFile(const char *file) -{ - fTextView->LoadFile(file); - fTextView->ShowBottom(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Called when closed via window manager action. - -void TSessionLogView::CloseWindow() -{ - if (fViewer->GetActDesc()->fProof) { - fViewer->GetActDesc()->fProof->Disconnect( - "LogMessage(const char*,Bool_t)", fViewer, - "LogMessage(const char*,Bool_t)"); - } - fViewer->SetLogWindow(0); - delete fTextView; - delete fClose; - delete fL1; - delete fL2; - DestroyWindow(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process OK button. - -Bool_t TSessionLogView::ProcessMessage(Longptr_t msg, Longptr_t, Longptr_t) -{ - switch (GET_MSG(msg)) { - case kC_COMMAND: - switch (GET_SUBMSG(msg)) { - case kCM_BUTTON: - // Only one button and one action... - CloseWindow(); - break; - default: - break; - } - break; - default: - break; - } - return kTRUE; -} - diff --git a/gui/sessionviewer/src/TSessionViewer.cxx b/gui/sessionviewer/src/TSessionViewer.cxx deleted file mode 100644 index e5b5e38d741c6..0000000000000 --- a/gui/sessionviewer/src/TSessionViewer.cxx +++ /dev/null @@ -1,5844 +0,0 @@ -// @(#)root/sessionviewer:$Id$ -// Author: Marek Biskup, Jakub Madejczyk, Bertrand Bellenot 10/08/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -/** \class TSessionViewer - \ingroup sessionviewer - -Widget used to manage PROOF or local sessions, PROOF connections, -queries construction and results handling. - -*/ - - -#include "TApplication.h" -#include "TROOT.h" -#include "THashList.h" -#include "TClass.h" -#include "TSystem.h" -#include "TGFileDialog.h" -#include "TBrowser.h" -#include "TGButton.h" -#include "TGLayout.h" -#include "TGListTree.h" -#include "TGCanvas.h" -#include "TGLabel.h" -#include "TGTextEntry.h" -#include "TGNumberEntry.h" -#include "TGTableLayout.h" -#include "TGListBox.h" -#include "TGSplitter.h" -#include "TGProgressBar.h" -#include "TGListView.h" -#include "TGMsgBox.h" -#include "TGMenu.h" -#include "TGStatusBar.h" -#include "TGIcon.h" -#include "TChain.h" -#include "TDSet.h" -#include "TFileInfo.h" -#include "TObjString.h" -#include "TProof.h" -#include "TRandom.h" -#include "TSessionViewer.h" -#include "TSessionLogView.h" -#include "TQueryResult.h" -#include "TGTextView.h" -#include "TGToolBar.h" -#include "TGTab.h" -#include "TRootEmbeddedCanvas.h" -#include "TCanvas.h" -#include "TGMimeTypes.h" -#include "TInterpreter.h" -#include "TContextMenu.h" -#include "TG3DLine.h" -#include "TSessionDialogs.h" -#include "TEnv.h" -#include "TH2.h" -#include "TVirtualTreePlayer.h" -#include "TSelector.h" -#include "TFileCollection.h" -#include "TVirtualX.h" -#include "HelpText.h" -#include "TRootHelpDialog.h" -#include "snprintf.h" -#ifdef WIN32 -#include "TWin32SplashThread.h" -#endif - -#include - -TSessionViewer *gSessionViewer = nullptr; - -const char *kConfigFile = ".proofgui.conf"; - -ClassImp(TQueryDescription); -ClassImp(TSessionDescription); -ClassImp(TSessionServerFrame); -ClassImp(TSessionFrame); -ClassImp(TSessionQueryFrame); -ClassImp(TSessionOutputFrame); -ClassImp(TSessionInputFrame); -ClassImp(TSessionViewer); - -const char *xpm_names[] = { - "monitor01.xpm", - "monitor02.xpm", - "monitor03.xpm", - "monitor04.xpm", - 0 -}; - -const char *conftypes[] = { - "Config files", "*.conf", - "All files", "*.*", - 0, 0 -}; - -const char *pkgtypes[] = { - "Package files", "*.par", - "All files", "*.*", - 0, 0 -}; - -const char *macrotypes[] = { - "C files", "*.[C|c]*", - "All files", "*", - 0, 0 -}; - - -const char *kFeedbackHistos[] = { - "PROOF_PacketsHist", - "PROOF_EventsHist", - "PROOF_NodeHist", - "PROOF_LatencyHist", - "PROOF_ProcTimeHist", - "PROOF_CpuTimeHist", - 0 -}; - -const char* const kSession_RedirectFile = ".templog"; -const char* const kSession_RedirectCmd = ".tempcmd"; - -// Menu command id's -enum ESessionViewerCommands { - kFileLoadConfig, - kFileSaveConfig, - kFileCloseViewer, - kFileQuit, - - kSessionNew, - kSessionAdd, - kSessionDelete, - kSessionGetQueries, - - kSessionConnect, - kSessionDisconnect, - kSessionShutdown, - kSessionCleanup, - kSessionBrowse, - kSessionShowStatus, - kSessionReset, - - kQueryNew, - kQueryEdit, - kQueryDelete, - kQuerySubmit, - kQueryStartViewer, - - kOptionsAutoSave, - kOptionsStatsHist, - kOptionsStatsTrace, - kOptionsSlaveStatsTrace, - kOptionsFeedback, - - kHelpAbout -}; - -const char *xpm_toolbar[] = { - "fileopen.xpm", - "filesaveas.xpm", - "", - "connect.xpm", - "disconnect.xpm", - "", - "query_new.xpm", - "query_submit.xpm", - "", - "about.xpm", - "", - "quit.xpm", - 0 -}; - -ToolBarData_t tb_data[] = { - { "", "Open Config File", kFALSE, kFileLoadConfig, 0 }, - { "", "Save Config File", kFALSE, kFileSaveConfig, 0 }, - { "", 0, 0, -1, 0 }, - { "", "Connect", kFALSE, kSessionConnect, 0 }, - { "", "Disconnect", kFALSE, kSessionDisconnect, 0 }, - { "", 0, 0, -1, 0 }, - { "", "New Query", kFALSE, kQueryNew, 0 }, - { "", "Submit Query", kFALSE, kQuerySubmit, 0 }, - { "", 0, 0, -1, 0 }, - { "", "About Root", kFALSE, kHelpAbout, 0 }, - { "", 0, 0, -1, 0 }, - { "", "Exit Root", kFALSE, kFileQuit, 0 }, - { 0, 0, 0, 0, 0 } -}; - - -//////////////////////////////////////////////////////////////////////////////// -// Server Frame - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor. - -TSessionServerFrame::TSessionServerFrame(TGWindow* p, Int_t w, Int_t h) : - TGCompositeFrame(p, w, h), fFrmNewServer(0), fTxtName(0), fTxtAddress(0), - fNumPort(0), fLogLevel(0), fTxtConfig(0), fTxtUsrName(0), fSync(0), - fViewer(0), fBtnAdd(0), fBtnConnect(0) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TSessionServerFrame::~TSessionServerFrame() -{ - Cleanup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build server configuration frame. - -void TSessionServerFrame::Build(TSessionViewer *gui) -{ - SetLayoutManager(new TGVerticalLayout(this)); - - SetCleanup(kDeepCleanup); - - fViewer = gui; - fFrmNewServer = new TGGroupFrame(this, "New Session"); - fFrmNewServer->SetCleanup(kDeepCleanup); - - AddFrame(fFrmNewServer, new TGLayoutHints(kLHintsExpandX, 2, 2, 2, 2)); - - fFrmNewServer->SetLayoutManager(new TGMatrixLayout(fFrmNewServer, 0, 2, 8)); - - fFrmNewServer->AddFrame(new TGLabel(fFrmNewServer, "Session Name:"), - new TGLayoutHints(kLHintsLeft, 3, 3, 3, 3)); - fFrmNewServer->AddFrame(fTxtName = new TGTextEntry(fFrmNewServer, - (const char *)0, 1), new TGLayoutHints()); - fTxtName->Resize(156, fTxtName->GetDefaultHeight()); - fTxtName->Associate(this); - fFrmNewServer->AddFrame(new TGLabel(fFrmNewServer, "Server name:"), - new TGLayoutHints(kLHintsLeft, 3, 3, 3, 3)); - fFrmNewServer->AddFrame(fTxtAddress = new TGTextEntry(fFrmNewServer, - (const char *)0, 2), new TGLayoutHints()); - fTxtAddress->Resize(156, fTxtAddress->GetDefaultHeight()); - fTxtAddress->Associate(this); - fFrmNewServer->AddFrame(new TGLabel(fFrmNewServer, "Port (default: 1093):"), - new TGLayoutHints(kLHintsLeft, 3, 3, 3, 3)); - fFrmNewServer->AddFrame(fNumPort = new TGNumberEntry(fFrmNewServer, 1093, 5, - 3, TGNumberFormat::kNESInteger,TGNumberFormat::kNEANonNegative, - TGNumberFormat::kNELLimitMinMax, 0, 65535),new TGLayoutHints()); - fNumPort->Associate(this); - fFrmNewServer->AddFrame(new TGLabel(fFrmNewServer, "Configuration File:"), - new TGLayoutHints(kLHintsLeft, 3, 3, 3, 3)); - fFrmNewServer->AddFrame(fTxtConfig = new TGTextEntry(fFrmNewServer, - (const char *)0, 4), new TGLayoutHints()); - fTxtConfig->Resize(156, fTxtConfig->GetDefaultHeight()); - fTxtConfig->Associate(this); - fFrmNewServer->AddFrame(new TGLabel(fFrmNewServer, "Log Level:"), - new TGLayoutHints(kLHintsLeft, 3, 3, 3, 3)); - - fFrmNewServer->AddFrame(fLogLevel = new TGNumberEntry(fFrmNewServer, 0, 5, 5, - TGNumberFormat::kNESInteger, - TGNumberFormat::kNEANonNegative, - TGNumberFormat::kNELLimitMinMax, 0, 5), - new TGLayoutHints(kLHintsLeft, 3, 3, 3, 3)); - fLogLevel->Associate(this); - - fFrmNewServer->AddFrame(new TGLabel(fFrmNewServer, "User Name:"), - new TGLayoutHints(kLHintsLeft, 3, 3, 3, 3)); - fFrmNewServer->AddFrame(fTxtUsrName = new TGTextEntry(fFrmNewServer, - (const char *)0, 6), new TGLayoutHints()); - fTxtUsrName->Resize(156, fTxtUsrName->GetDefaultHeight()); - fTxtUsrName->Associate(this); - - fFrmNewServer->AddFrame(new TGLabel(fFrmNewServer, "Process mode :"), - new TGLayoutHints(kLHintsLeft | kLHintsBottom | kLHintsExpandX, - 3, 3, 3, 3)); - fFrmNewServer->AddFrame(fSync = new TGCheckButton(fFrmNewServer, - "&Synchronous"), new TGLayoutHints(kLHintsLeft | kLHintsBottom | - kLHintsExpandX, 3, 3, 3, 3)); - fSync->SetToolTipText("Default Process Mode"); - fSync->SetState(kButtonDown); - - AddFrame(fBtnAdd = new TGTextButton(this, " Save "), - new TGLayoutHints(kLHintsTop | kLHintsCenterX, 5, 5, 15, 5)); - fBtnAdd->SetToolTipText("Add server to the list"); - fBtnAdd->Connect("Clicked()", "TSessionServerFrame", this, - "OnBtnAddClicked()"); - AddFrame(fBtnConnect = new TGTextButton(this, " Connect "), - new TGLayoutHints(kLHintsTop | kLHintsCenterX, 5, 5, 15, 5)); - fBtnConnect->Connect("Clicked()", "TSessionServerFrame", this, - "OnBtnConnectClicked()"); - fBtnConnect->SetToolTipText("Connect to the selected server"); - - fTxtConfig->Connect("DoubleClicked()", "TSessionServerFrame", this, - "OnConfigFileClicked()"); - - fTxtName->Connect("TextChanged(char*)", "TSessionServerFrame", this, - "SettingsChanged()"); - fTxtAddress->Connect("TextChanged(char*)", "TSessionServerFrame", this, - "SettingsChanged()"); - fTxtConfig->Connect("TextChanged(char*)", "TSessionServerFrame", this, - "SettingsChanged()"); - fTxtUsrName->Connect("TextChanged(char*)", "TSessionServerFrame", this, - "SettingsChanged()"); - fSync->Connect("Clicked()", "TSessionServerFrame", this, - "SettingsChanged()"); - fLogLevel->Connect("ValueChanged(Long_t)", "TSessionServerFrame", this, - "SettingsChanged()"); - fLogLevel->Connect("ValueSet(Long_t)", "TSessionServerFrame", this, - "SettingsChanged()"); - fNumPort->Connect("ValueChanged(Long_t)", "TSessionServerFrame", this, - "SettingsChanged()"); - fNumPort->Connect("ValueSet(Long_t)", "TSessionServerFrame", this, - "SettingsChanged()"); - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Settings have changed, update GUI accordingly. - -void TSessionServerFrame::SettingsChanged() -{ - TGTextEntry *sender = dynamic_cast((TQObject*)gTQSender); - Bool_t issync = (fSync->GetState() == kButtonDown); - if ((fViewer->GetActDesc()->fLocal) || - (strcmp(fViewer->GetActDesc()->GetName(), fTxtName->GetText())) || - (strcmp(fViewer->GetActDesc()->fAddress.Data(), fTxtAddress->GetText())) || - (strcmp(fViewer->GetActDesc()->fConfigFile.Data(), fTxtConfig->GetText())) || - (strcmp(fViewer->GetActDesc()->fUserName.Data(), fTxtUsrName->GetText())) || - (fViewer->GetActDesc()->fLogLevel != fLogLevel->GetIntNumber()) || - (fViewer->GetActDesc()->fPort != fNumPort->GetIntNumber()) || - (fViewer->GetActDesc()->fSync != issync)) { - ShowFrame(fBtnAdd); - HideFrame(fBtnConnect); - } - else { - HideFrame(fBtnAdd); - ShowFrame(fBtnConnect); - } - if (sender) { - sender->SetFocus(); - } -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Handle expose event in server frame. - -Bool_t TSessionServerFrame::HandleExpose(Event_t * /*event*/) -{ - //fTxtName->SelectAll(); - //fTxtName->SetFocus(); - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Browse configuration files. - -void TSessionServerFrame::OnConfigFileClicked() -{ - // do nothing if connection in progress - if (fViewer->IsBusy()) - return; - TGFileInfo fi; - fi.fFileTypes = conftypes; - new TGFileDialog(fClient->GetRoot(), fViewer, kFDOpen, &fi); - if (!fi.fFilename) return; - fTxtConfig->SetText(gSystem->BaseName(fi.fFilename)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete selected session configuration (remove it from the list). - -void TSessionServerFrame::OnBtnDeleteClicked() -{ - // do nothing if connection in progress - if (fViewer->IsBusy()) - return; - TString name(fTxtName->GetText()); - TIter next(fViewer->GetSessions()); - TSessionDescription *desc = fViewer->GetActDesc(); - - if (desc->fLocal) { - Int_t retval; - new TGMsgBox(fClient->GetRoot(), this, "Error Deleting Session", - "Deleting Local Sessions is not allowed !", - kMBIconExclamation,kMBOk,&retval); - return; - } - // ask for confirmation - TString m; - m.Form("Are you sure to delete the server \"%s\"", - desc->fName.Data()); - Int_t result; - new TGMsgBox(fClient->GetRoot(), this, "", m.Data(), 0, - kMBOk | kMBCancel, &result); - // if confirmed, delete it - if (result == kMBOk) { - // remove the Proof session from gROOT list of Proofs - if (desc->fConnected && desc->fAttached && desc->fProof) { - desc->fProof->Detach("S"); - } - // remove it from our sessions list - fViewer->GetSessions()->Remove((TObject *)desc); - // update configuration file - TGListTreeItem *item = fViewer->GetSessionHierarchy()->GetSelected(); - fViewer->GetSessionHierarchy()->DeleteItem(item); - - TObject *obj = fViewer->GetSessions()->Last(); - item = fViewer->GetSessionHierarchy()->FindChildByData( - fViewer->GetSessionItem(), (void *)obj); - if (item) { - fViewer->GetSessionHierarchy()->ClearHighlighted(); - fViewer->GetSessionHierarchy()->OpenItem(item); - fViewer->GetSessionHierarchy()->HighlightItem(item); - fViewer->GetSessionHierarchy()->SetSelected(item); - fViewer->GetSessionHierarchy()->ClearViewPort(); - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); - fViewer->OnListTreeClicked(item, 1, 0, 0); - } - } - if (fViewer->IsAutoSave()) - fViewer->WriteConfiguration(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Connect to selected server. - -void TSessionServerFrame::OnBtnConnectClicked() -{ - // do nothing if connection in progress - if (fViewer->IsBusy()) - return; - - if (!fViewer->GetSessions()->FindObject(fTxtName->GetText())) { - OnBtnAddClicked(); - } - else { - fViewer->GetActDesc()->fAddress = fTxtAddress->GetText(); - fViewer->GetActDesc()->fPort = fNumPort->GetIntNumber(); - if (strlen(fTxtConfig->GetText()) > 1) - fViewer->GetActDesc()->fConfigFile = TString(fTxtConfig->GetText()); - else - fViewer->GetActDesc()->fConfigFile = ""; - fViewer->GetActDesc()->fLogLevel = fLogLevel->GetIntNumber(); - fViewer->GetActDesc()->fUserName = fTxtUsrName->GetText(); - fViewer->GetActDesc()->fSync = (fSync->GetState() == kButtonDown); - if (fViewer->IsAutoSave()) - fViewer->WriteConfiguration(); - } - // set flag busy - fViewer->SetBusy(); - // avoid input events in list tree while connecting - fViewer->GetSessionHierarchy()->RemoveInput(kPointerMotionMask | - kEnterWindowMask | kLeaveWindowMask | kKeyPressMask); - gVirtualX->GrabButton(fViewer->GetSessionHierarchy()->GetId(), kAnyButton, - kAnyModifier, kButtonPressMask | kButtonReleaseMask, kNone, kNone, kFALSE); - // set watch cursor to indicate connection in progress - gVirtualX->SetCursor(fViewer->GetSessionHierarchy()->GetId(), - gVirtualX->CreateCursor(kWatch)); - gVirtualX->SetCursor(GetId(),gVirtualX->CreateCursor(kWatch)); - // display connection progress bar in first part of status bar - fViewer->GetStatusBar()->GetBarPart(0)->ShowFrame(fViewer->GetConnectProg()); - // connect to proof startup message (to update progress bar) - TQObject::Connect("TProof", "StartupMessage(char *,Bool_t,Int_t,Int_t)", - "TSessionViewer", fViewer, "StartupMessage(char *,Bool_t,Int_t,Int_t)"); - // collect and set-up configuration - TString address = fTxtAddress->GetText(); - TString url; - if (address == "lite://") { - url = address; - } - else { - // collect and set-up configuration - url = fTxtUsrName->GetText(); - url += "@"; url += address.Data(); - if (fNumPort->GetIntNumber() > 0) { - url += ":"; - url += fNumPort->GetIntNumber(); - } - } - - TProofDesc *desc; - fViewer->GetActDesc()->fProofMgr = TProofMgr::Create(url); - if (!fViewer->GetActDesc()->fProofMgr || - !fViewer->GetActDesc()->fProofMgr->IsValid()) { - // hide connection progress bar from status bar - fViewer->GetStatusBar()->GetBarPart(0)->HideFrame(fViewer->GetConnectProg()); - // release busy flag - fViewer->SetBusy(kFALSE); - // restore cursors and input - gVirtualX->SetCursor(GetId(), 0); - gVirtualX->GrabButton(fViewer->GetSessionHierarchy()->GetId(), kAnyButton, - kAnyModifier, kButtonPressMask | kButtonReleaseMask, kNone, kNone); - fViewer->GetSessionHierarchy()->AddInput(kPointerMotionMask | - kEnterWindowMask | kLeaveWindowMask | kKeyPressMask); - gVirtualX->SetCursor(fViewer->GetSessionHierarchy()->GetId(), 0); - return; - } - fViewer->UpdateListOfSessions(); - // check if the session already exist before to recreate it - TList *sessions = fViewer->GetActDesc()->fProofMgr->QuerySessions(""); - if (sessions) { - TIter nextp(sessions); - // loop over existing Proof sessions - while ((desc = (TProofDesc *)nextp())) { - if ((desc->GetName() == fViewer->GetActDesc()->fTag) || - (desc->GetTitle() == fViewer->GetActDesc()->fName)) { - fViewer->GetActDesc()->fProof = - fViewer->GetActDesc()->fProofMgr->AttachSession(desc->GetLocalId(), kTRUE); - fViewer->GetActDesc()->fTag = desc->GetName(); - fViewer->GetActDesc()->fProof->SetAlias(fViewer->GetActDesc()->fName); - fViewer->GetActDesc()->fConnected = kTRUE; - fViewer->GetActDesc()->fAttached = kTRUE; - - if (fViewer->GetOptionsMenu()->IsEntryChecked(kOptionsFeedback)) { - Int_t i = 0; - // browse list of feedback histos and check user's selected ones - while (kFeedbackHistos[i]) { - if (fViewer->GetCascadeMenu()->IsEntryChecked(41+i)) { - fViewer->GetActDesc()->fProof->AddFeedback(kFeedbackHistos[i]); - fViewer->GetActDesc()->fNbHistos++; - } - i++; - } - // connect feedback signal - fViewer->GetActDesc()->fProof->Connect("Feedback(TList *objs)", - "TSessionQueryFrame", fViewer->GetQueryFrame(), - "Feedback(TList *objs)"); - gROOT->Time(); - } - else { - // if feedback option not selected, clear Proof's feedback option - fViewer->GetActDesc()->fProof->ClearFeedback(); - } - - break; - } - } - } - if (fViewer->GetActDesc()->fProof == 0) { - if (fViewer->GetActDesc()->fProofMgr->IsValid()) { - fViewer->GetActDesc()->fProof = fViewer->GetActDesc()->fProofMgr->CreateSession( - fViewer->GetActDesc()->fConfigFile); - desc = 0; - sessions = fViewer->GetActDesc()->fProofMgr->QuerySessions(""); - if (sessions) - desc = (TProofDesc *)sessions->Last(); - if (desc) { - fViewer->GetActDesc()->fProof->SetAlias(fViewer->GetActDesc()->fName); - fViewer->GetActDesc()->fTag = desc->GetName(); - fViewer->GetActDesc()->fConnected = kTRUE; - fViewer->GetActDesc()->fAttached = kTRUE; - } - } - } - if (fViewer->GetActDesc()->fProof) { - fViewer->GetActDesc()->fConfigFile = fViewer->GetActDesc()->fProof->GetConfFile(); - fViewer->GetActDesc()->fUserName = fViewer->GetActDesc()->fProof->GetUser(); - fViewer->GetActDesc()->fPort = fViewer->GetActDesc()->fProof->GetPort(); - fViewer->GetActDesc()->fLogLevel = fViewer->GetActDesc()->fProof->GetLogLevel(); - if (fViewer->GetActDesc()->fLogLevel < 0) - fViewer->GetActDesc()->fLogLevel = 0; - if (fViewer->GetActDesc()->fProof->IsLite()) - fViewer->GetActDesc()->fAddress = "lite://"; - else - fViewer->GetActDesc()->fAddress = fViewer->GetActDesc()->fProof->GetMaster(); - fViewer->GetActDesc()->fConnected = kTRUE; - fViewer->GetActDesc()->fProof->SetBit(TProof::kUsingSessionGui); - } - fViewer->UpdateListOfSessions(); - - // check if connected and valid - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - // set log level - fViewer->GetActDesc()->fProof->SetLogLevel(fViewer->GetActDesc()->fLogLevel); - // set query type (synch / asynch) - fViewer->GetActDesc()->fProof->SetQueryMode(fViewer->GetActDesc()->fSync ? - TProof::kSync : TProof::kAsync); - // set connected flag - fViewer->GetActDesc()->fConnected = kTRUE; - // change list tree item picture to connected pixmap - TGListTreeItem *item = fViewer->GetSessionHierarchy()->FindChildByData( - fViewer->GetSessionItem(),fViewer->GetActDesc()); - if (item) { - item->SetPictures(fViewer->GetProofConPict(), fViewer->GetProofConPict()); - // update viewer - fViewer->OnListTreeClicked(item, 1, 0, 0); - fViewer->GetSessionHierarchy()->ClearViewPort(); - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); - } - // connect to progress related signals - fViewer->GetActDesc()->fProof->Connect("Progress(Long64_t,Long64_t)", - "TSessionQueryFrame", fViewer->GetQueryFrame(), - "Progress(Long64_t,Long64_t)"); - fViewer->GetActDesc()->fProof->Connect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - "TSessionQueryFrame", fViewer->GetQueryFrame(), - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - fViewer->GetActDesc()->fProof->Connect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)", - "TSessionQueryFrame", fViewer->GetQueryFrame(), - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)"); - fViewer->GetActDesc()->fProof->Connect("StopProcess(Bool_t)", - "TSessionQueryFrame", fViewer->GetQueryFrame(), - "IndicateStop(Bool_t)"); - fViewer->GetActDesc()->fProof->Connect( - "ResetProgressDialog(const char*,Int_t,Long64_t,Long64_t)", - "TSessionQueryFrame", fViewer->GetQueryFrame(), - "ResetProgressDialog(const char*,Int_t,Long64_t,Long64_t)"); - // enable timer used for status bar icon's animation - fViewer->EnableTimer(); - // change status bar right icon to connected pixmap - fViewer->ChangeRightLogo("monitor01.xpm"); - // do not animate yet - fViewer->SetChangePic(kFALSE); - // connect to signal "query result ready" - fViewer->GetActDesc()->fProof->Connect("QueryResultReady(char *)", - "TSessionViewer", fViewer, "QueryResultReady(char *)"); - // display connection information on status bar - TString msg; - msg.Form("PROOF Cluster %s ready", fViewer->GetActDesc()->fName.Data()); - fViewer->GetStatusBar()->SetText(msg.Data(), 1); - fViewer->GetSessionFrame()->ProofInfos(); - fViewer->UpdateListOfPackages(); - fViewer->GetSessionFrame()->UpdateListOfDataSets(); - // Enable previously uploaded packages if in auto-enable mode - if (fViewer->GetActDesc()->fAutoEnable) { - TPackageDescription *package; - TIter next(fViewer->GetActDesc()->fPackages); - while ((package = (TPackageDescription *)next())) { - if (!package->fEnabled) { - if (fViewer->GetActDesc()->fProof->EnablePackage(package->fName) != 0) - Error("Submit", "Enable package failed"); - else { - package->fEnabled = kTRUE; - fViewer->GetSessionFrame()->UpdatePackages(); - } - } - } - } - } - // hide connection progress bar from status bar - fViewer->GetStatusBar()->GetBarPart(0)->HideFrame(fViewer->GetConnectProg()); - // release busy flag - fViewer->SetBusy(kFALSE); - // restore cursors and input - gVirtualX->SetCursor(GetId(), 0); - gVirtualX->GrabButton(fViewer->GetSessionHierarchy()->GetId(), kAnyButton, - kAnyModifier, kButtonPressMask | kButtonReleaseMask, kNone, kNone); - fViewer->GetSessionHierarchy()->AddInput(kPointerMotionMask | - kEnterWindowMask | kLeaveWindowMask | kKeyPressMask); - gVirtualX->SetCursor(fViewer->GetSessionHierarchy()->GetId(), 0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset server configuration fields. - -void TSessionServerFrame::OnBtnNewServerClicked() -{ - // do nothing if connection in progress - if (fViewer->IsBusy()) - return; - fViewer->GetSessionHierarchy()->ClearHighlighted(); - fViewer->GetSessionHierarchy()->OpenItem(fViewer->GetSessionItem()); - fViewer->GetSessionHierarchy()->HighlightItem(fViewer->GetSessionItem()); - fViewer->GetSessionHierarchy()->SetSelected(fViewer->GetSessionItem()); - fViewer->OnListTreeClicked(fViewer->GetSessionItem(), 1, 0, 0); - fViewer->GetSessionHierarchy()->ClearViewPort(); - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); - fTxtName->SetText(""); - fTxtAddress->SetText(""); - fTxtConfig->SetText(""); - fNumPort->SetIntNumber(1093); - fLogLevel->SetIntNumber(0); - fTxtUsrName->SetText(""); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add newly created session configuration in the list of sessions. - -void TSessionServerFrame::OnBtnAddClicked() -{ - Int_t retval; - Bool_t newSession = kTRUE; - TSessionDescription* desc = 0; - // do nothing if connection in progress - if (fViewer->IsBusy()) - return; - - if ((!fTxtName->GetBuffer()->GetTextLength()) || - (!fTxtAddress->GetBuffer()->GetTextLength()) || - (!fTxtUsrName->GetBuffer()->GetTextLength())) { - new TGMsgBox(fClient->GetRoot(), fViewer, "Error Adding Session", - "At least one required field is empty !", - kMBIconExclamation, kMBOk, &retval); - return; - } - TObject *obj = fViewer->GetSessions()->FindObject(fTxtName->GetText()); - if (obj) - desc = dynamic_cast(obj); - if (desc) { - new TGMsgBox(fClient->GetRoot(), fViewer, "Adding Session", - TString::Format("The session \"%s\" already exists ! Overwrite ?", - fTxtName->GetText()), kMBIconQuestion, kMBYes | kMBNo | - kMBCancel, &retval); - if (retval != kMBYes) - return; - newSession = kFALSE; - } - if (newSession) { - desc = new TSessionDescription(); - desc->fName = fTxtName->GetText(); - desc->fTag = ""; - desc->fQueries = new TList(); - desc->fPackages = new TList(); - desc->fActQuery = 0; - desc->fProof = 0; - desc->fProofMgr = 0; - desc->fAutoEnable = kFALSE; - desc->fAddress = fTxtAddress->GetText(); - desc->fPort = fNumPort->GetIntNumber(); - desc->fConnected = kFALSE; - desc->fAttached = kFALSE; - desc->fLocal = kFALSE; - if (strlen(fTxtConfig->GetText()) > 1) - desc->fConfigFile = TString(fTxtConfig->GetText()); - else - desc->fConfigFile = ""; - desc->fLogLevel = fLogLevel->GetIntNumber(); - desc->fUserName = fTxtUsrName->GetText(); - desc->fSync = (fSync->GetState() == kButtonDown); - // add newly created session config to our session list - fViewer->GetSessions()->Add((TObject *)desc); - // save into configuration file - TGListTreeItem *item = fViewer->GetSessionHierarchy()->AddItem( - fViewer->GetSessionItem(), desc->fName.Data(), - fViewer->GetProofDisconPict(), fViewer->GetProofDisconPict()); - fViewer->GetSessionHierarchy()->SetToolTipItem(item, "Proof Session"); - item->SetUserData(desc); - fViewer->GetSessionHierarchy()->ClearHighlighted(); - fViewer->GetSessionHierarchy()->OpenItem(fViewer->GetSessionItem()); - fViewer->GetSessionHierarchy()->OpenItem(item); - fViewer->GetSessionHierarchy()->HighlightItem(item); - fViewer->GetSessionHierarchy()->SetSelected(item); - fViewer->GetSessionHierarchy()->ClearViewPort(); - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); - fViewer->OnListTreeClicked(item, 1, 0, 0); - } - else { - fViewer->GetActDesc()->fName = fTxtName->GetText(); - fViewer->GetActDesc()->fAddress = fTxtAddress->GetText(); - fViewer->GetActDesc()->fPort = fNumPort->GetIntNumber(); - if (strlen(fTxtConfig->GetText()) > 1) - fViewer->GetActDesc()->fConfigFile = TString(fTxtConfig->GetText()); - fViewer->GetActDesc()->fLogLevel = fLogLevel->GetIntNumber(); - fViewer->GetActDesc()->fUserName = fTxtUsrName->GetText(); - fViewer->GetActDesc()->fSync = (fSync->GetState() == kButtonDown); - TGListTreeItem *item2 = fViewer->GetSessionHierarchy()->GetSelected(); - item2->SetUserData(fViewer->GetActDesc()); - fViewer->OnListTreeClicked(fViewer->GetSessionHierarchy()->GetSelected(), - 1, 0, 0); - } - HideFrame(fBtnAdd); - ShowFrame(fBtnConnect); - if (fViewer->IsAutoSave()) - fViewer->WriteConfiguration(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update fields with values from session description desc. - -void TSessionServerFrame::Update(TSessionDescription* desc) -{ - if (desc->fLocal) { - fTxtName->SetText(""); - fTxtAddress->SetText(""); - fNumPort->SetIntNumber(1093); - fTxtConfig->SetText(""); - fTxtUsrName->SetText(""); - fLogLevel->SetIntNumber(0); - return; - } - - fTxtName->SetText(desc->fName); - fTxtAddress->SetText(desc->fAddress); - fNumPort->SetIntNumber(desc->fPort); - fLogLevel->SetIntNumber(desc->fLogLevel); - - if (desc->fConfigFile.Length() > 1) { - fTxtConfig->SetText(desc->fConfigFile); - } - else { - fTxtConfig->SetText(""); - } - fTxtUsrName->SetText(desc->fUserName); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process messages for session server frame. -/// Used to navigate between text entry fields. - -Bool_t TSessionServerFrame::ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t) -{ - switch (GET_MSG(msg)) { - case kC_TEXTENTRY: - switch (GET_SUBMSG(msg)) { - case kTE_ENTER: - case kTE_TAB: - switch (parm1) { - case 1: // session name - fTxtAddress->SelectAll(); - fTxtAddress->SetFocus(); - break; - case 2: // server address - fNumPort->GetNumberEntry()->SelectAll(); - fNumPort->GetNumberEntry()->SetFocus(); - break; - case 3: // port number - fTxtConfig->SelectAll(); - fTxtConfig->SetFocus(); - break; - case 4: // configuration file - fLogLevel->GetNumberEntry()->SelectAll(); - fLogLevel->GetNumberEntry()->SetFocus(); - break; - case 5: // log level - fTxtUsrName->SelectAll(); - fTxtUsrName->SetFocus(); - break; - case 6: // user name - fTxtName->SelectAll(); - fTxtName->SetFocus(); - break; - } - break; - - default: - break; - } - break; - - default: - break; - } - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -// Session Frame - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor. - -TSessionFrame::TSessionFrame(TGWindow* p, Int_t w, Int_t h) : - TGCompositeFrame(p, w, h), fTab(0), fFA(0), fFB(0), fFC(0), fFD(0), fFE(0), - fCommandTxt(0), fCommandBuf(0), fInfoTextView(0), fClearCheck(0), - fBtnShowLog(0), fBtnNewQuery(0), fBtnGetQueries(0), fLBPackages(0), - fBtnAdd(0), fBtnRemove(0), fBtnUp(0), fBtnDown(0), fBtnShow(0), - fBtnShowEnabled(0), fChkMulti(0), fChkEnable(0), fBtnUpload(0), - fBtnEnable(0), fBtnClear(0), fBtnDisable(0), fDSetView(0), fDataSetTree(0), - fBtnUploadDSet(0), fBtnRemoveDSet(0), fBtnVerifyDSet(0), fBtnRefresh(0), - fTxtParallel(0), fLogLevel(0), fApplyLogLevel(0), fApplyParallel(0), - fViewer(0) -{ - for (int i=0;i<19;++i) fInfoLine[i] = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TSessionFrame::~TSessionFrame() -{ - Cleanup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build session frame. - -void TSessionFrame::Build(TSessionViewer *gui) -{ - SetLayoutManager(new TGVerticalLayout(this)); - SetCleanup(kDeepCleanup); - fViewer = gui; - Int_t i,j; - - // main session tab - fTab = new TGTab(this, 200, 200); - AddFrame(fTab, new TGLayoutHints(kLHintsTop | kLHintsExpandX | - kLHintsExpandY, 2, 2, 2, 2)); - - // add "Status" tab element - TGCompositeFrame *tf = fTab->AddTab("Status"); - fFA = new TGCompositeFrame(tf, 100, 100, kVerticalFrame); - tf->AddFrame(fFA, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX | kLHintsExpandY)); - - // add first session information line - fInfoLine[0] = new TGLabel(fFA, " "); - fFA->AddFrame(fInfoLine[0], new TGLayoutHints(kLHintsCenterX | - kLHintsExpandX, 5, 5, 15, 5)); - - TGCompositeFrame* frmInfos = new TGHorizontalFrame(fFA, 350, 100); - frmInfos->SetLayoutManager(new TGTableLayout(frmInfos, 9, 2)); - - // add session information lines - j = 0; - for (i=0;i<17;i+=2) { - fInfoLine[i+1] = new TGLabel(frmInfos, " "); - frmInfos->AddFrame(fInfoLine[i+1], new TGTableLayoutHints(0, 1, j, j+1, - kLHintsLeft | kLHintsCenterY, 5, 5, 2, 2)); - fInfoLine[i+2] = new TGLabel(frmInfos, " "); - frmInfos->AddFrame(fInfoLine[i+2], new TGTableLayoutHints(1, 2, j, j+1, - kLHintsLeft | kLHintsCenterY, 5, 5, 2, 2)); - j++; - } - fFA->AddFrame(frmInfos, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX | kLHintsExpandY, 5, 5, 5, 5)); - - // add "new query" and "get queries" buttons - TGCompositeFrame* frmBut1 = new TGHorizontalFrame(fFA, 350, 100); - frmBut1->SetCleanup(kDeepCleanup); - frmBut1->AddFrame(fBtnNewQuery = new TGTextButton(frmBut1, "New Query..."), - new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 5, 5, 5, 5)); - fBtnNewQuery->SetToolTipText("Open New Query Dialog"); - frmBut1->AddFrame(fBtnGetQueries = new TGTextButton(frmBut1, " Get Queries "), - new TGLayoutHints(kLHintsLeft | kLHintsExpandX, 5, 5, 5, 5)); - fBtnGetQueries->SetToolTipText("Get List of Queries from the server"); - fBtnShowLog = new TGTextButton(frmBut1, "Show log..."); - fBtnShowLog->SetToolTipText("Show Session log (opens log window)"); - frmBut1->AddFrame(fBtnShowLog, new TGLayoutHints(kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fFA->AddFrame(frmBut1, new TGLayoutHints(kLHintsLeft | kLHintsBottom | - kLHintsExpandX)); - - // add "Commands" tab element - tf = fTab->AddTab("Commands"); - fFC = new TGCompositeFrame(tf, 100, 100, kVerticalFrame); - tf->AddFrame(fFC, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX | kLHintsExpandY)); - - // add command line label and text entry - TGCompositeFrame* frmCmd = new TGHorizontalFrame(fFC, 350, 100); - frmCmd->SetCleanup(kDeepCleanup); - frmCmd->AddFrame(new TGLabel(frmCmd, "Command Line :"), - new TGLayoutHints(kLHintsLeft | kLHintsCenterY, 5, 5, 15, 5)); - fCommandBuf = new TGTextBuffer(120); - frmCmd->AddFrame(fCommandTxt = new TGTextEntry(frmCmd, - fCommandBuf ),new TGLayoutHints(kLHintsLeft | kLHintsCenterY | - kLHintsExpandX, 5, 5, 15, 5)); - fFC->AddFrame(frmCmd, new TGLayoutHints(kLHintsExpandX, 5, 5, 10, 5)); - // connect command line text entry to "return pressed" signal - fCommandTxt->Connect("ReturnPressed()", "TSessionFrame", this, - "OnCommandLine()"); - - // check box for option "clear view" - fClearCheck = new TGCheckButton(fFC, "Clear view after each command"); - fFC->AddFrame(fClearCheck,new TGLayoutHints(kLHintsLeft | kLHintsTop, - 10, 5, 5, 5)); - fClearCheck->SetState(kButtonUp); - // add text view for redirected output - fFC->AddFrame(new TGLabel(fFC, "Output :"), - new TGLayoutHints(kLHintsLeft | kLHintsTop, 10, 5, 5, 5)); - fInfoTextView = new TGTextView(fFC, 330, 150, "", kSunkenFrame | - kDoubleBorder); - fFC->AddFrame(fInfoTextView, new TGLayoutHints(kLHintsLeft | - kLHintsTop | kLHintsExpandX | kLHintsExpandY, 10, 10, 5, 5)); - - // add "Packages" tab element - tf = fTab->AddTab("Packages"); - fFB = new TGCompositeFrame(tf, 100, 100, kVerticalFrame); - tf->AddFrame(fFB, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX | kLHintsExpandY)); - - // new frame containing packages listbox and control buttons - TGCompositeFrame* frmcanvas = new TGHorizontalFrame(fFB, 350, 100); - - // packages listbox - fLBPackages = new TGListBox(frmcanvas); - fLBPackages->Resize(80,150); - fLBPackages->SetMultipleSelections(kFALSE); - frmcanvas->AddFrame(fLBPackages, new TGLayoutHints(kLHintsExpandX | - kLHintsExpandY, 5, 5, 5, 5)); - // control buttons frame - TGCompositeFrame* frmBut2 = new TGVerticalFrame(frmcanvas, 150, 100); - - fChkMulti = new TGCheckButton(frmBut2, "Multiple Selection"); - fChkMulti->SetToolTipText("Enable multiple selection in the package list"); - frmBut2->AddFrame(fChkMulti, new TGLayoutHints(kLHintsLeft, 5, 5, 5, 5)); - - fBtnAdd = new TGTextButton(frmBut2, " Add... "); - fBtnAdd->SetToolTipText("Add a package to the list"); - frmBut2->AddFrame(fBtnAdd,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fBtnRemove = new TGTextButton(frmBut2, "Remove"); - fBtnRemove->SetToolTipText("Remove package from the list"); - frmBut2->AddFrame(fBtnRemove,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fBtnUp = new TGTextButton(frmBut2, "Move Up"); - fBtnUp->SetToolTipText("Move package one step upward in the list"); - frmBut2->AddFrame(fBtnUp,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fBtnDown = new TGTextButton(frmBut2, "Move Down"); - fBtnDown->SetToolTipText("Move package one step downward in the list"); - frmBut2->AddFrame(fBtnDown,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - frmcanvas->AddFrame(frmBut2, new TGLayoutHints(kLHintsLeft | kLHintsCenterY | - kLHintsExpandY)); - fFB->AddFrame(frmcanvas, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX | kLHintsExpandY)); - - TGCompositeFrame* frmLeg = new TGHorizontalFrame(fFB, 300, 100); - frmLeg->SetCleanup(kDeepCleanup); - TGPicture *pic1 = (TGPicture *)fClient->GetPicture("package.xpm"); - TGIcon *icn1 = new TGIcon(frmLeg, pic1, pic1->GetWidth(), pic1->GetHeight()); - frmLeg->AddFrame(icn1, new TGLayoutHints(kLHintsLeft | kLHintsTop, - 5, 5, 0, 5)); - frmLeg->AddFrame(new TGLabel(frmLeg, ": Local"), - new TGLayoutHints(kLHintsLeft | kLHintsTop, 0, 10, 0, 5)); - - TGPicture *pic2 = (TGPicture *)fClient->GetPicture("package_delete.xpm"); - TGIcon *icn2 = new TGIcon(frmLeg, pic2, pic2->GetWidth(), pic2->GetHeight()); - frmLeg->AddFrame(icn2, new TGLayoutHints(kLHintsLeft | kLHintsTop, - 5, 5, 0, 5)); - frmLeg->AddFrame(new TGLabel(frmLeg, ": Uploaded"), - new TGLayoutHints(kLHintsLeft | kLHintsTop, 0, 10, 0, 5)); - - TGPicture *pic3 = (TGPicture *)fClient->GetPicture("package_add.xpm"); - TGIcon *icn3 = new TGIcon(frmLeg, pic3, pic3->GetWidth(), pic3->GetHeight()); - frmLeg->AddFrame(icn3, new TGLayoutHints(kLHintsLeft | kLHintsTop, - 5, 5, 0, 5)); - frmLeg->AddFrame(new TGLabel(frmLeg, ": Enabled"), - new TGLayoutHints(kLHintsLeft | kLHintsTop, 0, 10, 0, 5)); - fFB->AddFrame(frmLeg, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX, 0, 0, 0, 0)); - - TGCompositeFrame* frmBtn = new TGHorizontalFrame(fFB, 300, 100); - frmBtn->SetCleanup(kDeepCleanup); - frmBtn->AddFrame(fBtnUpload = new TGTextButton(frmBtn, - " Upload "), new TGLayoutHints(kLHintsLeft | kLHintsExpandX | - kLHintsCenterY, 5, 5, 5, 5)); - fBtnUpload->SetToolTipText("Upload selected package(s) to the server"); - frmBtn->AddFrame(fBtnEnable = new TGTextButton(frmBtn, - " Enable "), new TGLayoutHints(kLHintsLeft | kLHintsExpandX | - kLHintsCenterY, 5, 5, 5, 5)); - fBtnEnable->SetToolTipText("Enable selected package(s) on the server"); - frmBtn->AddFrame(fBtnDisable = new TGTextButton(frmBtn, - " Disable "), new TGLayoutHints(kLHintsLeft | kLHintsExpandX | - kLHintsCenterY, 5, 5, 5, 5)); - fBtnDisable->SetToolTipText("Disable selected package(s) on the server"); - frmBtn->AddFrame(fBtnClear = new TGTextButton(frmBtn, - " Clear "), new TGLayoutHints(kLHintsLeft | kLHintsExpandX | - kLHintsCenterY, 5, 5, 5, 5)); - fBtnClear->SetToolTipText("Clear all packages on the server"); - fFB->AddFrame(frmBtn, new TGLayoutHints(kLHintsExpandX, 0, 0, 0, 0)); - - fBtnClear->SetEnabled(kFALSE); - - TGCompositeFrame* frmBtn3 = new TGHorizontalFrame(fFB, 300, 100); - frmBtn3->SetCleanup(kDeepCleanup); - fBtnShow = new TGTextButton(frmBtn3, "Show packages"); - fBtnShow->SetToolTipText("Show (list) available packages on the server"); - frmBtn3->AddFrame(fBtnShow,new TGLayoutHints(kLHintsCenterY | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fBtnShowEnabled = new TGTextButton(frmBtn3, "Show Enabled"); - fBtnShowEnabled->SetToolTipText("Show (list) enabled packages on the server"); - frmBtn3->AddFrame(fBtnShowEnabled,new TGLayoutHints(kLHintsCenterY | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fFB->AddFrame(frmBtn3, new TGLayoutHints(kLHintsExpandX, 0, 0, 0, 0)); - - fChkEnable = new TGCheckButton(fFB, "Enable at session startup"); - fChkEnable->SetToolTipText("Enable packages on the server at startup time"); - fFB->AddFrame(fChkEnable, new TGLayoutHints(kLHintsLeft, 5, 5, 5, 5)); - - // add "DataSets" tab element - tf = fTab->AddTab("DataSets"); - fFE = new TGCompositeFrame(tf, 100, 100, kVerticalFrame); - tf->AddFrame(fFE, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX | kLHintsExpandY)); - - // new frame containing datasets treeview and control buttons - TGCompositeFrame* frmdataset = new TGHorizontalFrame(fFE, 350, 100); - - // datasets list tree - fDSetView = new TGCanvas(frmdataset, 200, 200, kSunkenFrame | kDoubleBorder); - frmdataset->AddFrame(fDSetView, new TGLayoutHints(kLHintsExpandX | kLHintsExpandY, - 5, 5, 5, 5)); - fDataSetTree = new TGListTree(fDSetView, kHorizontalFrame); - fDataSetTree->AddItem(0, "DataSets"); - - // control buttons frame - TGCompositeFrame* frmBut3 = new TGVerticalFrame(frmdataset, 150, 100); - - fBtnUploadDSet = new TGTextButton(frmBut3, " Upload... "); - fBtnUploadDSet->SetToolTipText("Upload a dataset to the cluster"); - frmBut3->AddFrame(fBtnUploadDSet, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fBtnRemoveDSet = new TGTextButton(frmBut3, "Remove"); - fBtnRemoveDSet->SetToolTipText("Remove dataset from the cluster"); - frmBut3->AddFrame(fBtnRemoveDSet,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fBtnVerifyDSet = new TGTextButton(frmBut3, "Verify"); - fBtnVerifyDSet->SetToolTipText("Verify dataset on the cluster"); - frmBut3->AddFrame(fBtnVerifyDSet,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fBtnRefresh = new TGTextButton(frmBut3, "Refresh List"); - fBtnRefresh->SetToolTipText("Refresh List of DataSet/Files present on the cluster"); - frmBut3->AddFrame(fBtnRefresh,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 15, 5)); - - frmdataset->AddFrame(frmBut3, new TGLayoutHints(kLHintsLeft | kLHintsCenterY | - kLHintsExpandY, 5, 5, 5, 0)); - - fFE->AddFrame(frmdataset, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX | kLHintsExpandY)); - - // add "Options" tab element - tf = fTab->AddTab("Options"); - fFD = new TGCompositeFrame(tf, 100, 100, kVerticalFrame); - tf->AddFrame(fFD, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX | kLHintsExpandY)); - - // add Log Level label and text entry - TGCompositeFrame* frmLog = new TGHorizontalFrame(fFD, 310, 100, kFixedWidth); - frmLog->SetCleanup(kDeepCleanup); - frmLog->AddFrame(fApplyLogLevel = new TGTextButton(frmLog, - " Apply "), new TGLayoutHints(kLHintsRight | - kLHintsCenterY, 10, 5, 5, 5)); - fApplyLogLevel->SetToolTipText("Apply currently selected log level"); - fLogLevel = new TGNumberEntry(frmLog, 0, 5, 5, TGNumberFormat::kNESInteger, - TGNumberFormat::kNEANonNegative, TGNumberFormat::kNELLimitMinMax, 0, 5); - frmLog->AddFrame(fLogLevel, new TGLayoutHints(kLHintsRight | - kLHintsCenterY, 5, 5, 5, 5)); - frmLog->AddFrame(new TGLabel(frmLog, "Log Level :"), - new TGLayoutHints(kLHintsRight | kLHintsCenterY, 5, 5, 5, 5)); - fFD->AddFrame(frmLog, new TGLayoutHints(kLHintsLeft, 5, 5, 15, 5)); - - // add Parallel Nodes label and text entry - TGCompositeFrame* frmPar = new TGHorizontalFrame(fFD, 310, 100, kFixedWidth); - frmPar->SetCleanup(kDeepCleanup); - frmPar->AddFrame(fApplyParallel = new TGTextButton(frmPar, - " Apply "), new TGLayoutHints(kLHintsRight | - kLHintsCenterY, 10, 5, 5, 5)); - fApplyParallel->SetToolTipText("Apply currently selected parallel nodes"); - fTxtParallel = new TGTextEntry(frmPar); - fTxtParallel->SetAlignment(kTextRight); - fTxtParallel->SetText("99999"); - fTxtParallel->Resize(fLogLevel->GetDefaultWidth(), fTxtParallel->GetDefaultHeight()); - frmPar->AddFrame(fTxtParallel, new TGLayoutHints(kLHintsRight | - kLHintsCenterY, 5, 5, 5, 5)); - frmPar->AddFrame(new TGLabel(frmPar, "Set Parallel Nodes :"), - new TGLayoutHints(kLHintsRight | kLHintsCenterY, 5, 5, 5, 5)); - fFD->AddFrame(frmPar, new TGLayoutHints(kLHintsLeft, 5, 5, 5, 5)); - - // connect button actions to functions - fBtnShowLog->Connect("Clicked()", "TSessionFrame", this, - "OnBtnShowLogClicked()"); - fBtnNewQuery->Connect("Clicked()", "TSessionFrame", this, - "OnBtnNewQueryClicked()"); - fBtnGetQueries->Connect("Clicked()", "TSessionFrame", this, - "OnBtnGetQueriesClicked()"); - - fChkEnable->Connect("Toggled(Bool_t)", "TSessionFrame", this, - "OnStartupEnable(Bool_t)"); - fChkMulti->Connect("Toggled(Bool_t)", "TSessionFrame", this, - "OnMultipleSelection(Bool_t)"); - fBtnAdd->Connect("Clicked()", "TSessionFrame", this, - "OnBtnAddClicked()"); - fBtnRemove->Connect("Clicked()", "TSessionFrame", this, - "OnBtnRemoveClicked()"); - fBtnUp->Connect("Clicked()", "TSessionFrame", this, - "OnBtnUpClicked()"); - fBtnDown->Connect("Clicked()", "TSessionFrame", this, - "OnBtnDownClicked()"); - fApplyLogLevel->Connect("Clicked()", "TSessionFrame", this, - "OnApplyLogLevel()"); - fApplyParallel->Connect("Clicked()", "TSessionFrame", this, - "OnApplyParallel()"); - fBtnUpload->Connect("Clicked()", "TSessionFrame", this, - "OnUploadPackages()"); - fBtnEnable->Connect("Clicked()", "TSessionFrame", this, - "OnEnablePackages()"); - fBtnDisable->Connect("Clicked()", "TSessionFrame", this, - "OnDisablePackages()"); - fBtnClear->Connect("Clicked()", "TSessionFrame", this, - "OnClearPackages()"); - fBtnShowEnabled->Connect("Clicked()", "TSessionViewer", fViewer, - "ShowEnabledPackages()"); - fBtnShow->Connect("Clicked()", "TSessionViewer", fViewer, - "ShowPackages()"); - - fBtnUploadDSet->Connect("Clicked()", "TSessionFrame", this, - "OnBtnUploadDSet()"); - fBtnRemoveDSet->Connect("Clicked()", "TSessionFrame", this, - "OnBtnRemoveDSet()"); - fBtnVerifyDSet->Connect("Clicked()", "TSessionFrame", this, - "OnBtnVerifyDSet()"); - fBtnRefresh->Connect("Clicked()", "TSessionFrame", this, - "UpdateListOfDataSets()"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display information on current session. - -void TSessionFrame::ProofInfos() -{ - TString buf; - - // if local session - if (fViewer->GetActDesc()->fLocal) { - buf.Form("*** Local Session on %s ***", gSystem->HostName()); - fInfoLine[0]->SetText(buf.Data()); - UserGroup_t *userGroup = gSystem->GetUserInfo(); - fInfoLine[1]->SetText("User :"); - if (userGroup) buf.Form("%s", userGroup->fRealName.Data()); - fInfoLine[2]->SetText(buf.Data()); - fInfoLine[3]->SetText("Working directory :"); - buf.Form("%s", gSystem->WorkingDirectory()); - fInfoLine[4]->SetText(buf.Data()); - fInfoLine[5]->SetText(" "); - fInfoLine[6]->SetText(" "); - fInfoLine[7]->SetText(" "); - fInfoLine[8]->SetText(" "); - fInfoLine[9]->SetText(" "); - fInfoLine[10]->SetText(" "); - fInfoLine[11]->SetText(" "); - fInfoLine[12]->SetText(" "); - fInfoLine[13]->SetText(" "); - fInfoLine[14]->SetText(" "); - fInfoLine[15]->SetText(" "); - fInfoLine[16]->SetText(" "); - fInfoLine[17]->SetText(" "); - fInfoLine[18]->SetText(" "); - if (userGroup) delete userGroup; - Layout(); - Resize(GetDefaultSize()); - return; - } - // return if not a valid Proof session - if (!fViewer->GetActDesc()->fConnected || - !fViewer->GetActDesc()->fAttached || - !fViewer->GetActDesc()->fProof || - !fViewer->GetActDesc()->fProof->IsValid()) - return; - - if (!fViewer->GetActDesc()->fProof->IsMaster()) { - if (fViewer->GetActDesc()->fProof->IsParallel()) - buf.Form("*** Connected to %s (parallel mode, %d workers) ***", - fViewer->GetActDesc()->fProof->GetMaster(), - fViewer->GetActDesc()->fProof->GetParallel()); - else - buf.Form("*** Connected to %s (sequential mode) ***", - fViewer->GetActDesc()->fProof->GetMaster()); - fInfoLine[0]->SetText(buf.Data()); - fInfoLine[1]->SetText("Port number : "); - buf.Form("%d", fViewer->GetActDesc()->fProof->GetPort()); - fInfoLine[2]->SetText(buf.Data()); - fInfoLine[3]->SetText("User : "); - buf.Form("%s", fViewer->GetActDesc()->fProof->GetUser()); - fInfoLine[4]->SetText(buf.Data()); - fInfoLine[5]->SetText("Client protocol version : "); - buf.Form("%d", fViewer->GetActDesc()->fProof->GetClientProtocol()); - fInfoLine[6]->SetText(buf.Data()); - fInfoLine[7]->SetText("Remote protocol version : "); - buf.Form("%d", fViewer->GetActDesc()->fProof->GetRemoteProtocol()); - fInfoLine[8]->SetText(buf.Data()); - fInfoLine[9]->SetText("Log level : "); - buf.Form("%d", fViewer->GetActDesc()->fProof->GetLogLevel()); - fInfoLine[10]->SetText(buf.Data()); - fInfoLine[11]->SetText("Session unique tag : "); - buf.Form("%s", fViewer->GetActDesc()->fProof->IsValid() ? - fViewer->GetActDesc()->fProof->GetSessionTag() : " "); - fInfoLine[12]->SetText(buf.Data()); - fInfoLine[13]->SetText("Total MB's processed :"); - buf.Form("%.2f", float(fViewer->GetActDesc()->fProof->GetBytesRead())/(1024*1024)); - fInfoLine[14]->SetText(buf.Data()); - fInfoLine[15]->SetText("Total real time used (s) :"); - buf.Form("%.3f", fViewer->GetActDesc()->fProof->GetRealTime()); - fInfoLine[16]->SetText(buf.Data()); - fInfoLine[17]->SetText("Total CPU time used (s) :"); - buf.Form("%.3f", fViewer->GetActDesc()->fProof->GetCpuTime()); - fInfoLine[18]->SetText(buf.Data()); - } - else { - if (fViewer->GetActDesc()->fProof->IsParallel()) - buf.Form("*** Master server %s (parallel mode, %d workers) ***", - fViewer->GetActDesc()->fProof->GetMaster(), - fViewer->GetActDesc()->fProof->GetParallel()); - else - buf.Form("*** Master server %s (sequential mode) ***", - fViewer->GetActDesc()->fProof->GetMaster()); - fInfoLine[0]->SetText(buf.Data()); - fInfoLine[1]->SetText("Port number : "); - buf.Form("%d", fViewer->GetActDesc()->fProof->GetPort()); - fInfoLine[2]->SetText(buf.Data()); - fInfoLine[3]->SetText("User : "); - buf.Form("%s", fViewer->GetActDesc()->fProof->GetUser()); - fInfoLine[4]->SetText(buf.Data()); - fInfoLine[5]->SetText("Protocol version : "); - buf.Form("%d", fViewer->GetActDesc()->fProof->GetClientProtocol()); - fInfoLine[6]->SetText(buf.Data()); - fInfoLine[7]->SetText("Image name : "); - buf.Form("%s",fViewer->GetActDesc()->fProof->GetImage()); - fInfoLine[8]->SetText(buf.Data()); - fInfoLine[9]->SetText("Config directory : "); - buf.Form("%s", fViewer->GetActDesc()->fProof->GetConfDir()); - fInfoLine[10]->SetText(buf.Data()); - fInfoLine[11]->SetText("Config file : "); - buf.Form("%s", fViewer->GetActDesc()->fProof->GetConfFile()); - fInfoLine[12]->SetText(buf.Data()); - fInfoLine[13]->SetText("Total MB's processed :"); - buf.Form("%.2f", float(fViewer->GetActDesc()->fProof->GetBytesRead())/(1024*1024)); - fInfoLine[14]->SetText(buf.Data()); - fInfoLine[15]->SetText("Total real time used (s) :"); - buf.Form("%.3f", fViewer->GetActDesc()->fProof->GetRealTime()); - fInfoLine[16]->SetText(buf.Data()); - fInfoLine[17]->SetText("Total CPU time used (s) :"); - buf.Form("%.3f", fViewer->GetActDesc()->fProof->GetCpuTime()); - fInfoLine[18]->SetText(buf.Data()); - } - Layout(); - Resize(GetDefaultSize()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Open Upload Dataset dialog. - -void TSessionFrame::OnBtnUploadDSet() -{ - if (fViewer->IsBusy()) - return; - if (fViewer->GetActDesc()->fLocal) return; - new TUploadDataSetDlg(fViewer, 450, 360); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update list of dataset present on the cluster. - -void TSessionFrame::UpdateListOfDataSets() -{ - TObjString *dsetname; - TFileInfo *dsetfilename; - // cleanup the list - fDataSetTree->DeleteChildren(fDataSetTree->GetFirstItem()); - if (fViewer->GetActDesc()->fConnected && fViewer->GetActDesc()->fAttached && - fViewer->GetActDesc()->fProof && fViewer->GetActDesc()->fProof->IsValid() && - fViewer->GetActDesc()->fProof->IsParallel()) { - - const TGPicture *dseticon = fClient->GetPicture("rootdb_t.xpm"); - // ask for the list of datasets - - // TODO: is now returning a TMap; viewer has to be adapted - TList *dsetlist = 0; //fViewer->GetActDesc()->fProof->GetDataSets(); - // coverity[dead_error_condition]: to be changed for TMap usage - if (dsetlist) { - TGListTreeItem *dsetitem; - fDataSetTree->OpenItem(fDataSetTree->GetFirstItem()); - TIter nextdset(dsetlist); - while ((dsetname = (TObjString *)nextdset())) { - if (!fDataSetTree->FindItemByObj(fDataSetTree->GetFirstItem(), dsetname)) { - // add the dataset in the tree - dsetitem = fDataSetTree->AddItem(fDataSetTree->GetFirstItem(), - dsetname->GetName(), dsetname); - // ask for the list of files in the dataset - TList *dsetfilelist = fViewer->GetActDesc()->fProof->GetDataSet( - dsetname->GetName())->GetList(); - if(dsetfilelist) { - TIter nextdsetfile(dsetfilelist); - while ((dsetfilename = (TFileInfo *)nextdsetfile())) { - if (! fDataSetTree->FindItemByObj(dsetitem, dsetfilename)) { - // if not already in, add the file name in the tree - fDataSetTree->AddItem(dsetitem, - dsetfilename->GetFirstUrl()->GetUrl(), - dsetfilename, dseticon, dseticon); - } - } - // open the dataset item in order to show the files - fDataSetTree->OpenItem(dsetitem); - } - } - } - } - } - // refresh list tree - fClient->NeedRedraw(fDataSetTree); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove dataset from the list and from the cluster. - -void TSessionFrame::OnBtnRemoveDSet() -{ - TGListTreeItem *item; - TObjString *obj = 0; - if (fViewer->GetActDesc()->fLocal) return; - - item = fDataSetTree->GetSelected(); - if (!item) return; - if (item->GetParent() == 0) return; - if (item->GetParent() == fDataSetTree->GetFirstItem()) { - // Dataset itself - obj = (TObjString *)item->GetUserData(); - } - else if (item->GetParent()->GetParent() == fDataSetTree->GetFirstItem()) { - // One file of the dataset - obj = (TObjString *)item->GetParent()->GetUserData(); - } - - // if valid Proof session, set parallel slaves - if (obj && fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - fViewer->GetActDesc()->fProof->RemoveDataSet(obj->GetName()); - UpdateListOfDataSets(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Verify that the files in the selected dataset are present on the cluster. - -void TSessionFrame::OnBtnVerifyDSet() -{ - TGListTreeItem *item; - TObjString *obj = 0; - if (fViewer->GetActDesc()->fLocal) return; - - item = fDataSetTree->GetSelected(); - if (!item) return; - if (item->GetParent() == 0) return; - if (item->GetParent() == fDataSetTree->GetFirstItem()) { - // Dataset itself - obj = (TObjString *)item->GetUserData(); - } - else if (item->GetParent()->GetParent() == fDataSetTree->GetFirstItem()) { - // One file of the dataset - obj = (TObjString *)item->GetParent()->GetUserData(); - } - - // if valid Proof session, set parallel slaves - if (obj && fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - fViewer->GetActDesc()->fProof->VerifyDataSet(obj->GetName()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Apply selected log level on current session. - -void TSessionFrame::OnApplyLogLevel() -{ - // if local session, do nothing - if (fViewer->GetActDesc()->fLocal) return; - // if valid Proof session, set log level - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - fViewer->GetActDesc()->fLogLevel = fLogLevel->GetIntNumber(); - fViewer->GetActDesc()->fProof->SetLogLevel(fViewer->GetActDesc()->fLogLevel); - } - fViewer->GetSessionFrame()->ProofInfos(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Apply selected number of workers on current Proof session. - -void TSessionFrame::OnApplyParallel() -{ - // if local session, do nothing - if (fViewer->GetActDesc()->fLocal) return; - // if valid Proof session, set parallel slaves - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - Int_t nodes = atoi(fTxtParallel->GetText()); - fViewer->GetActDesc()->fProof->SetParallel(nodes); - } - fViewer->GetSessionFrame()->ProofInfos(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle multiple selection check button. - -void TSessionFrame::OnMultipleSelection(Bool_t on) -{ - fLBPackages->SetMultipleSelections(on); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle multiple selection check button. - -void TSessionFrame::OnStartupEnable(Bool_t on) -{ - if (fViewer->GetActDesc()) - fViewer->GetActDesc()->fAutoEnable = on; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update list of packages. - -void TSessionFrame::UpdatePackages() -{ - TPackageDescription *package; - const TGPicture *pict; - fLBPackages->RemoveEntries(0, fLBPackages->GetNumberOfEntries()); - TIter next(fViewer->GetActDesc()->fPackages); - while ((package = (TPackageDescription *)next())) { - if (package->fEnabled) - pict = fClient->GetPicture("package_add.xpm"); - else if (package->fUploaded) - pict = fClient->GetPicture("package_delete.xpm"); - else - pict = fClient->GetPicture("package.xpm"); - TGIconLBEntry *entry = new TGIconLBEntry(fLBPackages->GetContainer(), - package->fId, package->fPathName, pict); - fLBPackages->AddEntry(entry, new TGLayoutHints(kLHintsExpandX | kLHintsTop)); - } - fLBPackages->Layout(); - fClient->NeedRedraw(fLBPackages->GetContainer()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Upload selected package(s) to the current session. - -void TSessionFrame::OnUploadPackages() -{ - // if local session, do nothing - if (fViewer->GetActDesc()->fLocal) return; - // if valid Proof session, upload packages - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - TObject *obj; - TList selected; - fLBPackages->GetSelectedEntries(&selected); - TIter next(&selected); - while ((obj = next())) { - TString name = obj->GetTitle(); - if (fViewer->GetActDesc()->fProof->UploadPackage(name.Data()) != 0) - Error("Submit", "Upload package failed"); - else { - TObject *o = fViewer->GetActDesc()->fPackages->FindObject(gSystem->BaseName(name)); - if (!o) continue; - TPackageDescription *package = - dynamic_cast(o); - if (package) { - package->fUploaded = kTRUE; - ((TGIconLBEntry *)obj)->SetPicture( - fClient->GetPicture("package_delete.xpm")); - } - } - } - UpdatePackages(); - } - fLBPackages->Layout(); - fClient->NeedRedraw(fLBPackages->GetContainer()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable selected package(s) in the current session. - -void TSessionFrame::OnEnablePackages() -{ - // if local session, do nothing - if (fViewer->GetActDesc()->fLocal) return; - // if valid Proof session, enable packages - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - TObject *obj; - TList selected; - fBtnEnable->SetState(kButtonDisabled); - fLBPackages->GetSelectedEntries(&selected); - TIter next(&selected); - while ((obj = next())) { - TString name = obj->GetTitle(); - TObject *o = fViewer->GetActDesc()->fPackages->FindObject(gSystem->BaseName(name)); - if (!o) continue; - TPackageDescription *package = - dynamic_cast(o); - if (package) { - if (!package->fUploaded) { - if (fViewer->GetActDesc()->fProof->UploadPackage(name.Data()) != 0) - Error("Submit", "Upload package failed"); - else { - package->fUploaded = kTRUE; - ((TGIconLBEntry *)obj)->SetPicture( - fClient->GetPicture("package_delete.xpm")); - } - } - } - if (fViewer->GetActDesc()->fProof->EnablePackage(name) != 0) - Error("Submit", "Enable package failed"); - else { - package->fEnabled = kTRUE; - ((TGIconLBEntry *)obj)->SetPicture(fClient->GetPicture("package_add.xpm")); - } - } - UpdatePackages(); - fBtnEnable->SetState(kButtonUp); - } - fLBPackages->Layout(); - fClient->NeedRedraw(fLBPackages->GetContainer()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Disable selected package(s) in the current session. - -void TSessionFrame::OnDisablePackages() -{ - // if local session, do nothing - if (fViewer->GetActDesc()->fLocal) return; - // if valid Proof session, disable (clear) packages - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - TObject *obj; - TList selected; - fLBPackages->GetSelectedEntries(&selected); - TIter next(&selected); - while ((obj = next())) { - TString name = obj->GetTitle(); - if (fViewer->GetActDesc()->fProof->ClearPackage(name) != 0) - Error("Submit", "Clear package failed"); - else { - TObject *o = fViewer->GetActDesc()->fPackages->FindObject(gSystem->BaseName(name)); - if (!o) continue; - TPackageDescription *package = - dynamic_cast(o); - if (package) { - package->fEnabled = kFALSE; - package->fUploaded = kFALSE; - ((TGIconLBEntry *)obj)->SetPicture(fClient->GetPicture("package.xpm")); - } - } - } - UpdatePackages(); - } - fLBPackages->Layout(); - fClient->NeedRedraw(fLBPackages->GetContainer()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear (disable) all packages in the current session. - -void TSessionFrame::OnClearPackages() -{ - TPackageDescription *package; - // if local session, do nothing - if (fViewer->GetActDesc()->fLocal) return; - // if valid Proof session, clear packages - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - if (fViewer->GetActDesc()->fProof->ClearPackages() != 0) - Error("Submit", "Clear packages failed"); - else { - TIter next(fViewer->GetActDesc()->fPackages); - while ((package = (TPackageDescription *)next())) { - package->fEnabled = kFALSE; - } - } - } - fLBPackages->Layout(); - fClient->NeedRedraw(fLBPackages->GetContainer()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Open file dialog and add selected package file to the list. - -void TSessionFrame::OnBtnAddClicked() -{ - if (fViewer->IsBusy()) - return; - TGFileInfo fi; - TPackageDescription *package; - TGIconLBEntry *entry; - fi.fFileTypes = pkgtypes; - new TGFileDialog(fClient->GetRoot(), fViewer, kFDOpen, &fi); - if (fi.fMultipleSelection && fi.fFileNamesList) { - TObjString *el; - TIter next(fi.fFileNamesList); - while ((el = (TObjString *) next())) { - package = new TPackageDescription; - package->fName = gSystem->BaseName(gSystem->UnixPathName(el->GetString())); - package->fPathName = gSystem->UnixPathName(el->GetString()); - package->fId = fViewer->GetActDesc()->fPackages->GetEntries(); - package->fUploaded = kFALSE; - package->fEnabled = kFALSE; - fViewer->GetActDesc()->fPackages->Add((TObject *)package); - entry = new TGIconLBEntry(fLBPackages->GetContainer(), - package->fId, package->fPathName, - fClient->GetPicture("package.xpm")); - fLBPackages->AddEntry(entry, new TGLayoutHints(kLHintsExpandX | kLHintsTop)); - } - } - else if (fi.fFilename) { - package = new TPackageDescription; - package->fName = gSystem->BaseName(gSystem->UnixPathName(fi.fFilename)); - package->fPathName = gSystem->UnixPathName(fi.fFilename); - package->fId = fViewer->GetActDesc()->fPackages->GetEntries(); - package->fUploaded = kFALSE; - package->fEnabled = kFALSE; - fViewer->GetActDesc()->fPackages->Add((TObject *)package); - entry = new TGIconLBEntry(fLBPackages->GetContainer(), - package->fId, package->fPathName, - fClient->GetPicture("package.xpm")); - fLBPackages->AddEntry(entry, new TGLayoutHints(kLHintsExpandX | kLHintsTop)); - } - fLBPackages->Layout(); - fClient->NeedRedraw(fLBPackages->GetContainer()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove selected package from the list. - -void TSessionFrame::OnBtnRemoveClicked() -{ - TPackageDescription *package; - const TGPicture *pict; - Int_t pos = fLBPackages->GetSelected(); - fLBPackages->RemoveEntries(0, fLBPackages->GetNumberOfEntries()); - fViewer->GetActDesc()->fPackages->Remove( - fViewer->GetActDesc()->fPackages->At(pos)); - Int_t id = 0; - TIter next(fViewer->GetActDesc()->fPackages); - while ((package = (TPackageDescription *)next())) { - package->fId = id; - id++; - if (package->fEnabled) - pict = fClient->GetPicture("package_add.xpm"); - else if (package->fUploaded) - pict = fClient->GetPicture("package_delete.xpm"); - else - pict = fClient->GetPicture("package.xpm"); - TGIconLBEntry *entry = new TGIconLBEntry(fLBPackages->GetContainer(), - package->fId, package->fPathName, pict); - fLBPackages->AddEntry(entry, new TGLayoutHints(kLHintsExpandX | kLHintsTop)); - } - fLBPackages->Layout(); - fClient->NeedRedraw(fLBPackages->GetContainer()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Move selected package entry one position up in the list. - -void TSessionFrame::OnBtnUpClicked() -{ - TPackageDescription *package; - const TGPicture *pict; - Int_t pos = fLBPackages->GetSelected(); - if (pos <= 0) return; - fLBPackages->RemoveEntries(0, fLBPackages->GetNumberOfEntries()); - package = (TPackageDescription *)fViewer->GetActDesc()->fPackages->At(pos); - fViewer->GetActDesc()->fPackages->Remove( - fViewer->GetActDesc()->fPackages->At(pos)); - package->fId -= 1; - fViewer->GetActDesc()->fPackages->AddAt(package, package->fId); - Int_t id = 0; - TIter next(fViewer->GetActDesc()->fPackages); - while ((package = (TPackageDescription *)next())) { - package->fId = id; - id++; - if (package->fEnabled) - pict = fClient->GetPicture("package_add.xpm"); - else if (package->fUploaded) - pict = fClient->GetPicture("package_delete.xpm"); - else - pict = fClient->GetPicture("package.xpm"); - TGIconLBEntry *entry = new TGIconLBEntry(fLBPackages->GetContainer(), - package->fId, package->fPathName, pict); - fLBPackages->AddEntry(entry, new TGLayoutHints(kLHintsExpandX | kLHintsTop)); - } - fLBPackages->Select(pos-1); - fLBPackages->Layout(); - fClient->NeedRedraw(fLBPackages->GetContainer()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Move selected package entry one position down in the list. - -void TSessionFrame::OnBtnDownClicked() -{ - TPackageDescription *package; - const TGPicture *pict; - Int_t pos = fLBPackages->GetSelected(); - if (pos == -1 || pos == fViewer->GetActDesc()->fPackages->GetEntries()-1) - return; - fLBPackages->RemoveEntries(0, fLBPackages->GetNumberOfEntries()); - package = (TPackageDescription *)fViewer->GetActDesc()->fPackages->At(pos); - fViewer->GetActDesc()->fPackages->Remove( - fViewer->GetActDesc()->fPackages->At(pos)); - package->fId += 1; - fViewer->GetActDesc()->fPackages->AddAt(package, package->fId); - Int_t id = 0; - TIter next(fViewer->GetActDesc()->fPackages); - while ((package = (TPackageDescription *)next())) { - package->fId = id; - id++; - if (package->fEnabled) - pict = fClient->GetPicture("package_add.xpm"); - else if (package->fUploaded) - pict = fClient->GetPicture("package_delete.xpm"); - else - pict = fClient->GetPicture("package.xpm"); - TGIconLBEntry *entry = new TGIconLBEntry(fLBPackages->GetContainer(), - package->fId, package->fPathName, pict); - fLBPackages->AddEntry(entry, new TGLayoutHints(kLHintsExpandX | kLHintsTop)); - } - fLBPackages->Select(pos+1); - fLBPackages->Layout(); - fClient->NeedRedraw(fLBPackages->GetContainer()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Disconnect from current Proof session. - -void TSessionFrame::OnBtnDisconnectClicked() -{ - // if local session, do nothing - if (fViewer->GetActDesc()->fLocal) return; - // if valid Proof session, disconnect (close) - if (fViewer->GetActDesc()->fAttached && - fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - fViewer->GetActDesc()->fProof->Detach(); - } - // reset connected flag - fViewer->GetActDesc()->fAttached = kFALSE; - fViewer->GetActDesc()->fProof = 0; - // disable animation timer - fViewer->DisableTimer(); - // change list tree item picture to disconnected pixmap - TGListTreeItem *item = fViewer->GetSessionHierarchy()->FindChildByData( - fViewer->GetSessionItem(), fViewer->GetActDesc()); - if (item) { - item->SetPictures(fViewer->GetProofDisconPict(), - fViewer->GetProofDisconPict()); - } - // update viewer - fViewer->OnListTreeClicked(fViewer->GetSessionHierarchy()->GetSelected(), - 1, 0, 0); - fViewer->GetSessionHierarchy()->ClearViewPort(); - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); - fViewer->GetStatusBar()->SetText("", 1); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show session log. - -void TSessionFrame::OnBtnShowLogClicked() -{ - fViewer->ShowLog(0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Call "New Query" Dialog. - -void TSessionFrame::OnBtnNewQueryClicked() -{ - TNewQueryDlg *dlg = new TNewQueryDlg(fViewer, 350, 310); - dlg->Popup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get list of queries from current Proof server and populate the list tree. - -void TSessionFrame::OnBtnGetQueriesClicked() -{ - TList *lqueries = 0; - TQueryResult *query = 0; - TQueryDescription *newquery = 0, *lquery = 0; - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - lqueries = fViewer->GetActDesc()->fProof->GetListOfQueries(); - } - if (lqueries) { - TIter nextp(lqueries); - // loop over list of queries received from Proof server - while ((query = (TQueryResult *)nextp())) { - // create new query description - newquery = new TQueryDescription(); - newquery->fReference = TString::Format("%s:%s", query->GetTitle(), - query->GetName()); - // check in our tree if it is already there - TGListTreeItem *item = - fViewer->GetSessionHierarchy()->FindChildByData( - fViewer->GetSessionItem(), fViewer->GetActDesc()); - // if already there, skip - if (fViewer->GetSessionHierarchy()->FindChildByName(item, - newquery->fReference.Data())) - continue; - // check also in our query description list - Bool_t found = kFALSE; - TIter nextp2(fViewer->GetActDesc()->fQueries); - while ((lquery = (TQueryDescription *)nextp2())) { - if (lquery->fReference.CompareTo(newquery->fReference) == 0) { - found = kTRUE; - break; - } - } - if (found) continue; - // build new query description with infos from Proof - newquery->fStatus = query->IsFinalized() ? - TQueryDescription::kSessionQueryFinalized : - (TQueryDescription::ESessionQueryStatus)query->GetStatus(); - newquery->fSelectorString = query->GetSelecImp()->GetName(); - newquery->fQueryName = TString::Format("%s:%s", query->GetTitle(), - query->GetName()); - newquery->fOptions = query->GetOptions(); - newquery->fEventList = ""; - newquery->fNbFiles = 0; - newquery->fNoEntries = query->GetEntries(); - newquery->fFirstEntry = query->GetFirst(); - newquery->fResult = query; - newquery->fChain = 0; - fViewer->GetActDesc()->fQueries->Add((TObject *)newquery); - TGListTreeItem *item2 = fViewer->GetSessionHierarchy()->AddItem(item, - newquery->fQueryName, fViewer->GetQueryConPict(), - fViewer->GetQueryConPict()); - item2->SetUserData(newquery); - if (query->GetInputList()) - fViewer->GetSessionHierarchy()->AddItem(item2, "InputList"); - if (query->GetOutputList()) - fViewer->GetSessionHierarchy()->AddItem(item2, "OutputList"); - } - } - // at the end, update list tree - fViewer->GetSessionHierarchy()->ClearViewPort(); - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Command line handling. - -void TSessionFrame::OnCommandLine() -{ - // get command string - const char *cmd = fCommandTxt->GetText(); - char opt[2]; - // form temporary file path - TString pathtmp = TString::Format("%s/%s", gSystem->TempDirectory(), - kSession_RedirectCmd); - // if check box "clear view" is checked, open temp file in write mode - // (overwrite), in append mode otherwise. - if (fClearCheck->IsOn()) - snprintf(opt, 2, "w"); - else - snprintf(opt, 2, "a"); - - // if valid Proof session, pass the command to Proof - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - // redirect stdout/stderr to temp file - if (gSystem->RedirectOutput(pathtmp.Data(), opt) != 0) { - Error("ShowStatus", "stdout/stderr redirection failed; skipping"); - return; - } - // execute command line - fViewer->GetActDesc()->fProof->Exec(cmd); - // restore back stdout/stderr - if (gSystem->RedirectOutput(0) != 0) { - Error("ShowStatus", "stdout/stderr retore failed; skipping"); - return; - } - // if check box "clear view" is checked, clear text view - if (fClearCheck->IsOn()) - fInfoTextView->Clear(); - // load (display) temp file in text view - fInfoTextView->LoadFile(pathtmp.Data()); - // set focus to "command line" text entry - fCommandTxt->SetFocus(); - } - else { - // if no Proof session, or Proof session not valid, - // lets execute command line by TApplication - - // redirect stdout/stderr to temp file - if (gSystem->RedirectOutput(pathtmp.Data(), opt) != 0) { - Error("ShowStatus", "stdout/stderr redirection failed; skipping"); - } - // execute command line - gApplication->ProcessLine(cmd); - // restore back stdout/stderr - if (gSystem->RedirectOutput(0) != 0) { - Error("ShowStatus", "stdout/stderr retore failed; skipping"); - } - // if check box "clear view" is checked, clear text view - if (fClearCheck->IsOn()) - fInfoTextView->Clear(); - // load (display) temp file in text view - fInfoTextView->LoadFile(pathtmp.Data()); - // set focus to "command line" text entry - fCommandTxt->SetFocus(); - } - // display bottom of text view - fInfoTextView->ShowBottom(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Switch widgets status/visibility for local/remote sessions. - -void TSessionFrame::SetLocal(Bool_t local) -{ - if (local) { - fBtnGetQueries->SetState(kButtonDisabled); - fBtnShowLog->SetState(kButtonDisabled); - fTab->HideFrame(fTab->GetTabTab("Options")); - fTab->HideFrame(fTab->GetTabTab("Packages")); - fTab->HideFrame(fTab->GetTabTab("DataSets")); - } - else { - fBtnGetQueries->SetState(kButtonUp); - fBtnShowLog->SetState(kButtonUp); - fTab->ShowFrame(fTab->GetTabTab("Options")); - fTab->ShowFrame(fTab->GetTabTab("Packages")); - fTab->ShowFrame(fTab->GetTabTab("DataSets")); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Shutdown current session. - -void TSessionFrame::ShutdownSession() -{ - // do nothing if connection in progress - if (fViewer->IsBusy()) - return; - - if (fViewer->GetActDesc()->fLocal) { - Int_t retval; - new TGMsgBox(fClient->GetRoot(), this, "Error Shutting down Session", - "Shutting down Local Sessions is not allowed !", - kMBIconExclamation,kMBOk,&retval); - return; - } - if (!fViewer->GetActDesc()->fAttached || - !fViewer->GetActDesc()->fProof || - !fViewer->GetActDesc()->fProof->IsValid()) - return; - // ask for confirmation - TString m; - m.Form("Are you sure to shutdown the session \"%s\"", - fViewer->GetActDesc()->fName.Data()); - Int_t result; - new TGMsgBox(fClient->GetRoot(), this, "", m.Data(), 0, - kMBOk | kMBCancel, &result); - // if confirmed, delete it - if (result != kMBOk) - return; - // remove the Proof session from gROOT list of Proofs - fViewer->GetActDesc()->fProof->Detach("S"); - // reset connected flag - fViewer->GetActDesc()->fAttached = kFALSE; - fViewer->GetActDesc()->fProof = 0; - // disable animation timer - fViewer->DisableTimer(); - // change list tree item picture to disconnected pixmap - TGListTreeItem *item = fViewer->GetSessionHierarchy()->FindChildByData( - fViewer->GetSessionItem(), fViewer->GetActDesc()); - if (item) { - item->SetPictures(fViewer->GetProofDisconPict(), - fViewer->GetProofDisconPict()); - } - // update viewer - fViewer->OnListTreeClicked(fViewer->GetSessionHierarchy()->GetSelected(), - 1, 0, 0); - fViewer->GetSessionHierarchy()->ClearViewPort(); - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); - fViewer->GetStatusBar()->SetText("", 1); -} - -////////////////////////////////////////////////////////////////////////// -// Edit Query Frame -////////////////////////////////////////////////////////////////////////// - -//////////////////////////////////////////////////////////////////////////////// -/// Create a new Query dialog, used by the Session Viewer, to Edit a Query if -/// the editmode flag is set, or to create a new one if not set. - -TEditQueryFrame::TEditQueryFrame(TGWindow* p, Int_t w, Int_t h) : - TGCompositeFrame(p, w, h, kVerticalFrame), fFrmMore(0), fBtnMore(0), - fTxtQueryName(0), fTxtChain(0), fTxtSelector(0), fTxtOptions(0), - fNumEntries(0), fNumFirstEntry(0), fTxtParFile(0), fTxtEventList(0), - fViewer(0), fQuery(0), fChain(0) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete query dialog. - -TEditQueryFrame::~TEditQueryFrame() -{ - Cleanup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build the "new query" dialog. - -void TEditQueryFrame::Build(TSessionViewer *gui) -{ - TGButton *btnTmp; - fViewer = gui; - SetCleanup(kDeepCleanup); - SetLayoutManager(new TGTableLayout(this, 6, 5)); - - // add "Query Name" label and text entry - AddFrame(new TGLabel(this, "Query Name :"), - new TGTableLayoutHints(0, 1, 0, 1, kLHintsCenterY, 5, 5, 4, 0)); - AddFrame(fTxtQueryName = new TGTextEntry(this, - (const char *)0, 1), new TGTableLayoutHints(1, 2, 0, 1, - kLHintsCenterY, 5, 5, 4, 0)); - - // add "TChain" label and text entry - AddFrame(new TGLabel(this, "TChain :"), - new TGTableLayoutHints(0, 1, 1, 2, kLHintsCenterY, 5, 5, 4, 0)); - AddFrame(fTxtChain = new TGTextEntry(this, - (const char *)0, 2), new TGTableLayoutHints(1, 2, 1, 2, - kLHintsCenterY, 5, 5, 4, 0)); - fTxtChain->SetToolTipText("Specify TChain or TDSet from memory or file"); - fTxtChain->SetEnabled(kFALSE); - // add "Browse" button - AddFrame(btnTmp = new TGTextButton(this, "Browse..."), - new TGTableLayoutHints(2, 3, 1, 2, kLHintsCenterY, 5, 0, 4, 8)); - btnTmp->Connect("Clicked()", "TEditQueryFrame", this, "OnBrowseChain()"); - - // add "Selector" label and text entry - AddFrame(new TGLabel(this, "Selector :"), - new TGTableLayoutHints(0, 1, 2, 3, kLHintsCenterY, 5, 5, 0, 0)); - AddFrame(fTxtSelector = new TGTextEntry(this, - (const char *)0, 3), new TGTableLayoutHints(1, 2, 2, 3, - kLHintsCenterY, 5, 5, 0, 0)); - // add "Browse" button - AddFrame(btnTmp = new TGTextButton(this, "Browse..."), - new TGTableLayoutHints(2, 3, 2, 3, kLHintsCenterY, 5, 0, 0, 8)); - btnTmp->Connect("Clicked()", "TEditQueryFrame", this, "OnBrowseSelector()"); - - // add "Less <<" ("More >>") button - AddFrame(fBtnMore = new TGTextButton(this, " Less << "), - new TGTableLayoutHints(2, 3, 4, 5, kLHintsCenterY, 5, 5, 4, 0)); - fBtnMore->Connect("Clicked()", "TEditQueryFrame", this, "OnNewQueryMore()"); - - // add (initially hidden) options frame - fFrmMore = new TGCompositeFrame(this, 200, 200); - fFrmMore->SetCleanup(kDeepCleanup); - - AddFrame(fFrmMore, new TGTableLayoutHints(0, 3, 5, 6, - kLHintsExpandX | kLHintsExpandY)); - fFrmMore->SetLayoutManager(new TGTableLayout(fFrmMore, 4, 3)); - - // add "Options" label and text entry - fFrmMore->AddFrame(new TGLabel(fFrmMore, "Options :"), - new TGTableLayoutHints(0, 1, 0, 1, kLHintsCenterY, 5, 5, 0, 0)); - fFrmMore->AddFrame(fTxtOptions = new TGTextEntry(fFrmMore, - (const char *)0, 4), new TGTableLayoutHints(1, 2, 0, 1, 0, 17, - 0, 0, 8)); - //fTxtOptions->SetText("ASYN"); - fTxtOptions->SetText(""); - - // add "Nb Entries" label and number entry - fFrmMore->AddFrame(new TGLabel(fFrmMore, "Nb Entries :"), - new TGTableLayoutHints(0, 1, 1, 2, kLHintsCenterY, 5, 5, 0, 0)); - fFrmMore->AddFrame(fNumEntries = new TGNumberEntry(fFrmMore, 0, 5, -1, - TGNumberFormat::kNESInteger, TGNumberFormat::kNEAAnyNumber, - TGNumberFormat::kNELNoLimits), new TGTableLayoutHints(1, 2, 1, 2, - 0, 17, 0, 0, 8)); - // coverity[negative_returns]: no problem with -1, the format is kNESInteger - fNumEntries->SetIntNumber(-1); - // add "First Entry" label and number entry - fFrmMore->AddFrame(new TGLabel(fFrmMore, "First entry :"), - new TGTableLayoutHints(0, 1, 2, 3, kLHintsCenterY, 5, 5, 0, 0)); - fFrmMore->AddFrame(fNumFirstEntry = new TGNumberEntry(fFrmMore, 0, 5, -1, - TGNumberFormat::kNESInteger, TGNumberFormat::kNEANonNegative, - TGNumberFormat::kNELNoLimits), new TGTableLayoutHints(1, 2, 2, 3, 0, - 17, 0, 0, 8)); - - // add "Event list" label and text entry - fFrmMore->AddFrame(new TGLabel(fFrmMore, "Event list :"), - new TGTableLayoutHints(0, 1, 3, 4, kLHintsCenterY, 5, 5, 0, 0)); - fFrmMore->AddFrame(fTxtEventList = new TGTextEntry(fFrmMore, - (const char *)0, 6), new TGTableLayoutHints(1, 2, 3, 4, 0, 17, - 5, 0, 0)); - // add "Browse" button - fFrmMore->AddFrame(btnTmp = new TGTextButton(fFrmMore, "Browse..."), - new TGTableLayoutHints(2, 3, 3, 4, 0, 6, 0, 0, 8)); - btnTmp->Connect("Clicked()", "TEditQueryFrame", this, "OnBrowseEventList()"); - - fTxtQueryName->Associate(this); - fTxtChain->Associate(this); - fTxtSelector->Associate(this); - fTxtOptions->Associate(this); - fNumEntries->Associate(this); - fNumFirstEntry->Associate(this); - fTxtEventList->Associate(this); - - fTxtQueryName->Connect("TextChanged(char*)", "TEditQueryFrame", this, - "SettingsChanged()"); - fTxtChain->Connect("TextChanged(char*)", "TEditQueryFrame", this, - "SettingsChanged()"); - fTxtSelector->Connect("TextChanged(char*)", "TEditQueryFrame", this, - "SettingsChanged()"); - fTxtOptions->Connect("TextChanged(char*)", "TEditQueryFrame", this, - "SettingsChanged()"); - fNumEntries->Connect("ValueChanged(Long_t)", "TEditQueryFrame", this, - "SettingsChanged()"); - fNumFirstEntry->Connect("ValueChanged(Long_t)", "TEditQueryFrame", this, - "SettingsChanged()"); - fTxtEventList->Connect("TextChanged(char*)", "TEditQueryFrame", this, - "SettingsChanged()"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show/hide options frame and update button text accordingly. - -void TEditQueryFrame::OnNewQueryMore() -{ - if (IsVisible(fFrmMore)) { - HideFrame(fFrmMore); - fBtnMore->SetText(" More >> "); - } - else { - ShowFrame(fFrmMore); - fBtnMore->SetText(" Less << "); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Call new chain dialog. - -void TEditQueryFrame::OnBrowseChain() -{ - TNewChainDlg *dlg = new TNewChainDlg(fClient->GetRoot(), this); - dlg->Connect("OnElementSelected(TObject *)", "TEditQueryFrame", - this, "OnElementSelected(TObject *)"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle OnElementSelected signal coming from new chain dialog. - -void TEditQueryFrame::OnElementSelected(TObject *obj) -{ - if (obj) { - fChain = obj; - if (obj->IsA() == TChain::Class()) - fTxtChain->SetText(((TChain *)fChain)->GetName()); - else if (obj->IsA() == TDSet::Class()) - fTxtChain->SetText(((TDSet *)fChain)->GetObjName()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Open file browser to choose selector macro. - -void TEditQueryFrame::OnBrowseSelector() -{ - TGFileInfo fi; - fi.fFileTypes = macrotypes; - new TGFileDialog(fClient->GetRoot(), this, kFDOpen, &fi); - if (!fi.fFilename) return; - fTxtSelector->SetText(gSystem->UnixPathName(fi.fFilename)); -} - -//////////////////////////////////////////////////////////////////////////////// -///Browse event list - -void TEditQueryFrame::OnBrowseEventList() -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save current settings in main session viewer. - -void TEditQueryFrame::OnBtnSave() -{ - // if we are in edition mode and query description is valid, - // use it, otherwise create a new one - TQueryDescription *newquery; - if (fQuery) - newquery = fQuery; - else - newquery = new TQueryDescription(); - - // update query description fields - newquery->fSelectorString = fTxtSelector->GetText(); - if (fChain) { - newquery->fTDSetString = fChain->GetName(); - newquery->fChain = fChain; - } - else { - newquery->fTDSetString = ""; - newquery->fChain = 0; - } - newquery->fQueryName = fTxtQueryName->GetText(); - newquery->fOptions = fTxtOptions->GetText(); - newquery->fNoEntries = fNumEntries->GetIntNumber(); - newquery->fFirstEntry = fNumFirstEntry->GetIntNumber(); - newquery->fNbFiles = 0; - newquery->fResult = 0; - - if (newquery->fChain) { - if (newquery->fChain->IsA() == TChain::Class()) - newquery->fNbFiles = ((TChain *)newquery->fChain)->GetListOfFiles()->GetEntriesFast(); - else if (newquery->fChain->IsA() == TDSet::Class()) - newquery->fNbFiles = ((TDSet *)newquery->fChain)->GetListOfElements()->GetSize(); - } - // update user data with modified query description - TGListTreeItem *item = fViewer->GetSessionHierarchy()->GetSelected(); - fViewer->GetSessionHierarchy()->RenameItem(item, newquery->fQueryName); - item->SetUserData(newquery); - // update list tree - fViewer->GetSessionHierarchy()->ClearViewPort(); - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); - fTxtQueryName->SelectAll(); - fTxtQueryName->SetFocus(); - fViewer->WriteConfiguration(); - fViewer->GetQueryFrame()->Modified(kFALSE); - if (fViewer->GetActDesc()->fLocal || - (fViewer->GetActDesc()->fConnected && - fViewer->GetActDesc()->fAttached && - fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid())) { - fViewer->GetQueryFrame()->GetTab()->SetTab("Status"); - fViewer->GetQueryFrame()->OnBtnSubmit(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Settings have changed, update GUI accordingly. - -void TEditQueryFrame::SettingsChanged() -{ - if (fQuery) { - if ((strcmp(fQuery->fSelectorString.Data(), fTxtSelector->GetText())) || - (strcmp(fQuery->fQueryName.Data(), fTxtQueryName->GetText())) || - (strcmp(fQuery->fOptions.Data(), fTxtOptions->GetText())) || - (fQuery->fNoEntries != fNumEntries->GetIntNumber()) || - (fQuery->fFirstEntry != fNumFirstEntry->GetIntNumber()) || - (fQuery->fChain != fChain)) { - fViewer->GetQueryFrame()->Modified(kTRUE); - } - else { - fViewer->GetQueryFrame()->Modified(kFALSE); - } - } - else { - if ((fTxtQueryName->GetText()) && - ((fTxtQueryName->GetText()) || - (fTxtChain->GetText()))) - fViewer->GetQueryFrame()->Modified(kTRUE); - else - fViewer->GetQueryFrame()->Modified(kFALSE); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update entry fields with query description values. - -void TEditQueryFrame::UpdateFields(TQueryDescription *desc) -{ - fChain = 0; - fQuery = desc; - fTxtChain->SetText(""); - if (desc->fChain) { - fChain = desc->fChain; - fTxtChain->SetText(desc->fTDSetString); - } - fTxtQueryName->SetText(desc->fQueryName); - fTxtSelector->SetText(desc->fSelectorString); - fTxtOptions->SetText(desc->fOptions); - fNumEntries->SetIntNumber(desc->fNoEntries); - fNumFirstEntry->SetIntNumber(desc->fFirstEntry); - fTxtEventList->SetText(desc->fEventList); -} - -//////////////////////////////////////////////////////////////////////////////// -// Query Frame - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TSessionQueryFrame::TSessionQueryFrame(TGWindow* p, Int_t w, Int_t h) : - TGCompositeFrame(p, w, h), fBtnSubmit(0), fBtnFinalize(0), fBtnStop(0), - fBtnAbort(0), fBtnShowLog(0), fBtnRetrieve(0), fBtnSave(0), fInfoTextView(0), - fModified(0), fFiles(0), fFirst(0), fEntries(0), fPrevTotal(0), - fPrevProcessed(0), fLabInfos(0), fLabStatus(0), fTotal(0), fRate(0), - fStatus(kStopped), fTab(0), fFA(0), fFB(0), fFC(0), fFD(0), frmProg(0), - fECanvas(0), fStatsCanvas(0), fViewer(0), fDesc(0) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TSessionQueryFrame::~TSessionQueryFrame() -{ - Cleanup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build query information frame. - -void TSessionQueryFrame::Build(TSessionViewer *gui) -{ - SetLayoutManager(new TGVerticalLayout(this)); - SetCleanup(kDeepCleanup); - fFirst = fEntries = fPrevTotal = 0; - fPrevProcessed = 0; - fViewer = gui; - fModified = kFALSE; - - // main query tab - fTab = new TGTab(this, 200, 200); - AddFrame(fTab, new TGLayoutHints(kLHintsTop | kLHintsExpandX | - kLHintsExpandY, 2, 2, 2, 2)); - - // add "Status" tab element - TGCompositeFrame *tf = fTab->AddTab("Status"); - fFB = new TGCompositeFrame(tf, 100, 100, kVerticalFrame); - tf->AddFrame(fFB, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX | kLHintsExpandY)); - - // new frame containing control buttons and feedback histos canvas - TGCompositeFrame* frmcanvas = new TGHorizontalFrame(fFB, 350, 100); - // control buttons frame - TGCompositeFrame* frmBut2 = new TGVerticalFrame(frmcanvas, 150, 100); - fBtnSubmit = new TGTextButton(frmBut2, " Submit "); - fBtnSubmit->SetToolTipText("Submit (process) selected query"); - frmBut2->AddFrame(fBtnSubmit,new TGLayoutHints(kLHintsCenterY | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fBtnStop = new TGTextButton(frmBut2, "Stop"); - fBtnStop->SetToolTipText("Stop processing query"); - frmBut2->AddFrame(fBtnStop,new TGLayoutHints(kLHintsCenterY | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - fBtnAbort = new TGTextButton(frmBut2, "Abort"); - fBtnAbort->SetToolTipText("Abort processing query"); - frmBut2->AddFrame(fBtnAbort,new TGLayoutHints(kLHintsCenterY | kLHintsLeft | - kLHintsExpandX, 5, 5, 5, 5)); - frmcanvas->AddFrame(frmBut2, new TGLayoutHints(kLHintsLeft | kLHintsCenterY | - kLHintsExpandY)); - // feedback histos embedded canvas - fECanvas = new TRootEmbeddedCanvas("fECanvas", frmcanvas, 400, 150); - fStatsCanvas = fECanvas->GetCanvas(); - fStatsCanvas->SetFillColor(10); - fStatsCanvas->SetBorderMode(0); - frmcanvas->AddFrame(fECanvas, new TGLayoutHints(kLHintsExpandX | kLHintsExpandY, - 4, 4, 4, 4)); - fFB->AddFrame(frmcanvas, new TGLayoutHints(kLHintsLeft | kLHintsTop | - kLHintsExpandX | kLHintsExpandY)); - - // progress infos label - fLabInfos = new TGLabel(fFB, " "); - fFB->AddFrame(fLabInfos, new TGLayoutHints(kLHintsLeft, 5, 5, 5, 5)); - // progress status label - fLabStatus = new TGLabel(fFB, " "); - fFB->AddFrame(fLabStatus, new TGLayoutHints(kLHintsLeft, 5, 5, 5, 5)); - - //progress bar - frmProg = new TGHProgressBar(fFB, TGProgressBar::kFancy, 350 - 20); - frmProg->ShowPosition(); - frmProg->SetBarColor("green"); - fFB->AddFrame(frmProg, new TGLayoutHints(kLHintsExpandX, 5, 5, 5, 5)); - // total progress infos - fFB->AddFrame(fTotal = new TGLabel(fFB, - " Estimated time left : 0 sec (--- events of --- processed) "), - new TGLayoutHints(kLHintsLeft, 5, 5, 5, 5)); - // progress rate infos - fFB->AddFrame(fRate = new TGLabel(fFB, - " Processing Rate : -- events/sec "), - new TGLayoutHints(kLHintsLeft, 5, 5, 5, 5)); - - // add "Results" tab element - tf = fTab->AddTab("Results"); - fFC = new TGCompositeFrame(tf, 100, 100, kVerticalFrame); - tf->AddFrame(fFC, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX | kLHintsExpandY)); - // query result (header) information text view - fInfoTextView = new TGTextView(fFC, 330, 185, "", kSunkenFrame | - kDoubleBorder); - fFC->AddFrame(fInfoTextView, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandY | kLHintsExpandX, 5, 5, 10, 10)); - - // add "Retrieve", "Finalize" and "Show Log" buttons - TGCompositeFrame* frmBut3 = new TGHorizontalFrame(fFC, 350, 100); - fBtnRetrieve = new TGTextButton(frmBut3, "Retrieve"); - fBtnRetrieve->SetToolTipText("Retrieve query results"); - frmBut3->AddFrame(fBtnRetrieve,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 10, 10)); - fBtnFinalize = new TGTextButton(frmBut3, "Finalize"); - fBtnFinalize->SetToolTipText("Finalize query"); - frmBut3->AddFrame(fBtnFinalize,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 10, 10)); - fBtnShowLog = new TGTextButton(frmBut3, "Show Log"); - fBtnShowLog->SetToolTipText("Show query log (open log window)"); - frmBut3->AddFrame(fBtnShowLog,new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 5, 5, 10, 10)); - fFC->AddFrame(frmBut3, new TGLayoutHints(kLHintsLeft | kLHintsTop | kLHintsExpandX)); - - // add "Results" tab element - tf = fTab->AddTab("Edit Query"); - fFD = new TEditQueryFrame(tf, 100, 100); - fFD->Build(fViewer); - tf->AddFrame(fFD, new TGLayoutHints(kLHintsTop | kLHintsLeft, 5, 5, 10, 0)); - TString btntxt; - if (fViewer->GetActDesc()->fLocal || - (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid())) { - btntxt = " Submit "; - } - else { - btntxt = " Apply changes "; - } - tf->AddFrame(fBtnSave = new TGTextButton(tf, btntxt), - new TGLayoutHints(kLHintsTop | kLHintsLeft, 10, 5, 25, 5)); - - // connect button actions to functions - fBtnSave->Connect("Clicked()", "TEditQueryFrame", fFD, - "OnBtnSave()"); - fBtnSubmit->Connect("Clicked()", "TSessionQueryFrame", this, - "OnBtnSubmit()"); - fBtnFinalize->Connect("Clicked()", "TSessionQueryFrame", this, - "OnBtnFinalize()"); - fBtnStop->Connect("Clicked()", "TSessionQueryFrame", this, - "OnBtnStop()"); - fBtnAbort->Connect("Clicked()", "TSessionQueryFrame", this, - "OnBtnAbort()"); - fBtnShowLog->Connect("Clicked()", "TSessionQueryFrame", this, - "OnBtnShowLog()"); - fBtnRetrieve->Connect("Clicked()", "TSessionQueryFrame", this, - "OnBtnRetrieve()"); -// fBtnSave->SetState(kButtonDisabled); - Resize(350, 310); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Notify changes in query editor settings. - -void TSessionQueryFrame::Modified(Bool_t mod) -{ - fModified = mod; - if (fModified) { - fBtnSave->SetState(kButtonUp); - } - else { - fBtnSave->SetState(kButtonDisabled); - } - if (fViewer->GetActDesc()->fLocal || - (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid())) - fBtnSave->SetState(kButtonUp); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Feedback function connected to Feedback signal. -/// Used to update feedback histograms. - -void TSessionQueryFrame::Feedback(TList *objs) -{ - // if no actual session, just return - if (!fViewer->GetActDesc()->fAttached) - return; - if (!fViewer->GetActDesc()->fProof) - return; - if ((fViewer->GetActDesc()->fActQuery) && - (fViewer->GetActDesc()->fActQuery->fStatus != - TQueryDescription::kSessionQuerySubmitted) && - (fViewer->GetActDesc()->fActQuery->fStatus != - TQueryDescription::kSessionQueryRunning) ) - return; - TProof *sender = dynamic_cast((TQObject*)gTQSender); - // if Proof sender match actual session one, update feedback histos - if (sender && (sender == fViewer->GetActDesc()->fProof)) - UpdateHistos(objs); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update feedback histograms. - -void TSessionQueryFrame::UpdateHistos(TList *objs) -{ - TVirtualPad *save = gPad; - TObject *o; - Int_t pos = 1; - Int_t i = 0; - while (kFeedbackHistos[i]) { - // check if user has selected this histogram in the option menu - if (fViewer->GetCascadeMenu()->IsEntryChecked(41+i)) { - if ( (o = objs->FindObject(kFeedbackHistos[i]))) { - fStatsCanvas->cd(pos); - gPad->SetEditable(kTRUE); - if (TH1 *h = dynamic_cast(o)) { - h->SetStats(0); - h->SetBarWidth(0.75); - h->SetBarOffset(0.125); - h->SetFillColor(9); - h->Draw("bar"); - pos++; - } - else if (TH2 *h2 = dynamic_cast(o)) { - h2->Draw(); - pos++; - } - gPad->Modified(); - } - } - i++; - } - // update canvas - fStatsCanvas->cd(); - fStatsCanvas->Modified(); - fStatsCanvas->Update(); - if (save != 0) { - save->cd(); - } else { - gPad = 0; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update progress bar and status labels. - -void TSessionQueryFrame::Progress(Long64_t total, Long64_t processed) -{ - Long_t tt; - UInt_t hh=0, mm=0, ss=0; - TString stm; - // if no actual session, just return - if (!fViewer->GetActDesc()->fProof) - return; - // if Proof sender does't match actual session one, return - TProof *sender = dynamic_cast((TQObject*)gTQSender); - if (!sender || (sender != fViewer->GetActDesc()->fProof)) - return; - - if ((fViewer->GetActDesc()->fActQuery) && - (fViewer->GetActDesc()->fActQuery->fStatus != - TQueryDescription::kSessionQuerySubmitted) && - (fViewer->GetActDesc()->fActQuery->fStatus != - TQueryDescription::kSessionQueryRunning) ) { - fTotal->SetText(" Estimated time left : 0 sec (0 events of 0 processed) "); - fRate->SetText(" Processing Rate : 0.0f events/sec "); - frmProg->Reset(); - fFB->Layout(); - return; - } - - if (total < 0) - total = fPrevTotal; - else - fPrevTotal = total; - - // if no change since last call, just return - if (fPrevProcessed == processed) - return; - TString buf; - - // Update information at first call - if (fEntries != total) { - buf.Form("PROOF cluster : \"%s\" - %d worker nodes", - fViewer->GetActDesc()->fProof->GetMaster(), - fViewer->GetActDesc()->fProof->GetParallel()); - fLabInfos->SetText(buf); - - fEntries = total; - buf.Form(" %d files, %lld events, starting event %lld", - fFiles, fEntries, fFirst); - fLabStatus->SetText(buf); - } - - // compute progress bar position and update - Float_t pos = (Float_t)((Double_t)(processed * 100)/(Double_t)total); - frmProg->SetPosition(pos); - // if 100%, stop animation and set icon to "connected" - if (pos >= 100.0) { - fViewer->SetChangePic(kFALSE); - fViewer->ChangeRightLogo("monitor01.xpm"); - } - - // get current time - if ((fViewer->GetActDesc()->fActQuery->fStatus == - TQueryDescription::kSessionQueryRunning) || - (fViewer->GetActDesc()->fActQuery->fStatus == - TQueryDescription::kSessionQuerySubmitted)) - fViewer->GetActDesc()->fActQuery->fEndTime = gSystem->Now(); - TTime tdiff = fViewer->GetActDesc()->fActQuery->fEndTime - - fViewer->GetActDesc()->fActQuery->fStartTime; - Float_t eta = 0; - if (processed) - eta = ((Float_t)((Long64_t)tdiff)*total/Float_t(processed) - - Long64_t(tdiff))/1000.; - - tt = (Long_t)eta; - if (tt > 0) { - hh = (UInt_t)(tt / 3600); - mm = (UInt_t)((tt % 3600) / 60); - ss = (UInt_t)((tt % 3600) % 60); - } - if (hh) - stm.Form("%d h %d min %d sec", hh, mm, ss); - else if (mm) - stm.Form("%d min %d sec", mm, ss); - else - stm.Form("%d sec", ss); - if (processed == total) { - // finished - tt = (Long_t) Long64_t(tdiff)/1000; - if (tt > 0) { - hh = (UInt_t)(tt / 3600); - mm = (UInt_t)((tt % 3600) / 60); - ss = (UInt_t)((tt % 3600) % 60); - } - if (hh) - stm.Form("%d h %d min %d sec", hh, mm, ss); - else if (mm) - stm.Form("%d min %d sec", mm, ss); - else - stm.Form("%d sec", ss); - buf.Form(" Processed : %lld events in %s", total, stm.Data()); - fTotal->SetText(buf.Data()); - } else { - // update status infos - buf.Form(" Estimated time left : %s (%lld events of %lld processed) ", - stm.Data(), processed, total); - fTotal->SetText(buf.Data()); - } - if (processed > 0 && (Long64_t)tdiff > 0) { - buf.Form(" Processing Rate : %.1f events/sec ", - (Float_t)processed/(Long64_t)tdiff*1000.); - fRate->SetText(buf); - } - fPrevProcessed = processed; - - fFB->Layout(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// New version of Progress (just forward to the old version -/// for the time being). - -void TSessionQueryFrame::Progress(Long64_t total, Long64_t processed, - Long64_t /*bytesread*/, Float_t /*initTime*/, - Float_t /*procTime*/, Float_t /*evtrti*/, - Float_t /*mbrti*/, Int_t /*actw*/, - Int_t /*tses*/, Float_t /*eses*/) -{ - Progress(total, processed); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update progress bar and status labels. - -void TSessionQueryFrame::ProgressLocal(Long64_t total, Long64_t processed) -{ - Long_t tt; - UInt_t hh=0, mm=0, ss=0; - TString stm; - TString cproc; - Int_t status; - - switch (fViewer->GetActDesc()->fActQuery->fStatus) { - - case TQueryDescription::kSessionQueryAborted: - cproc = " - ABORTED"; - status = kAborted; - break; - case TQueryDescription::kSessionQueryStopped: - cproc = " - STOPPED"; - status = kStopped; - break; - case TQueryDescription::kSessionQueryRunning: - cproc = " "; - status = kRunning; - break; - case TQueryDescription::kSessionQueryCompleted: - case TQueryDescription::kSessionQueryFinalized: - cproc = " "; - status = kDone; - break; - default: - status = -1; - break; - } - if (processed < 0) processed = 0; - - frmProg->SetBarColor("green"); - if (status == kAborted) - frmProg->SetBarColor("red"); - else if (status == kStopped) - frmProg->SetBarColor("yellow"); - else if (status == -1 ) { - fTotal->SetText(" Estimated time left : 0 sec (0 events of 0 processed) "); - fRate->SetText(" Processing Rate : 0.0f events/sec "); - frmProg->Reset(); - fFB->Layout(); - return; - } - - if (total < 0) - total = fPrevTotal; - else - fPrevTotal = total; - - // if no change since last call, just return - TString buf; - - // Update information at first call - if (fEntries != total) { - fLabInfos->SetText("Local Session"); - - fEntries = total; - buf.Form(" %d files, %lld events, starting event %lld", - fFiles, fEntries, fFirst); - fLabStatus->SetText(buf.Data()); - } - - // compute progress bar position and update - Float_t pos = 0.0; - if (processed > 0 && total > 0) - pos = (Float_t)((Double_t)(processed * 100)/(Double_t)total); - frmProg->SetPosition(pos); - // if 100%, stop animation and set icon to "connected" - if (pos >= 100.0) { - fViewer->SetChangePic(kFALSE); - fViewer->ChangeRightLogo("monitor01.xpm"); - } - - // get current time - if (status == kRunning) - fViewer->GetActDesc()->fActQuery->fEndTime = gSystem->Now(); - TTime tdiff = fViewer->GetActDesc()->fActQuery->fEndTime - - fViewer->GetActDesc()->fActQuery->fStartTime; - Float_t eta = 0; - if (processed) - eta = ((Float_t)((Long64_t)tdiff)*total/(Float_t)(processed) - - (Long64_t)(tdiff))/1000.; - - tt = (Long_t)eta; - if (tt > 0) { - hh = (UInt_t)(tt / 3600); - mm = (UInt_t)((tt % 3600) / 60); - ss = (UInt_t)((tt % 3600) % 60); - } - if (hh) - stm = TString::Format("%d h %d min %d sec", hh, mm, ss); - else if (mm) - stm = TString::Format("%d min %d sec", mm, ss); - else - stm = TString::Format("%d sec", ss); - if ((processed != total) && (status == kRunning)) { - // update status infos - buf.Form(" Estimated time left : %s (%lld events of %lld processed) ", - stm.Data(), processed, total); - fTotal->SetText(buf); - } else { - tt = (Long_t) Long64_t(tdiff)/1000; - if (tt > 0) { - hh = (UInt_t)(tt / 3600); - mm = (UInt_t)((tt % 3600) / 60); - ss = (UInt_t)((tt % 3600) % 60); - } - if (hh) - stm = TString::Format("%d h %d min %d sec", hh, mm, ss); - else if (mm) - stm = TString::Format("%d min %d sec", mm, ss); - else - stm = TString::Format("%d sec", ss); - buf.Form(" Processed : %ld events in %s", - (Long_t)processed, stm.Data()); - buf += cproc; - fTotal->SetText(buf.Data()); - } - if (processed > 0 && (Long64_t)tdiff > 0) { - buf.Form(" Processing Rate : %.1f events/sec ", - (Float_t)processed/(Long64_t)tdiff*1000.); - fRate->SetText(buf.Data()); - } - fPrevProcessed = processed; - - fFB->Layout(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Indicate that Cancel or Stop was clicked. - -void TSessionQueryFrame::IndicateStop(Bool_t aborted) -{ - if (aborted == kTRUE) { - // Aborted - frmProg->SetBarColor("red"); - } - else { - // Stopped - frmProg->SetBarColor("yellow"); - } - // disconnect progress related signals - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - fViewer->GetActDesc()->fProof->Disconnect("Progress(Long64_t,Long64_t)", - this, "Progress(Long64_t,Long64_t)"); - fViewer->GetActDesc()->fProof->Disconnect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - this, "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - fViewer->GetActDesc()->fProof->Disconnect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)", - this, "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)"); - fViewer->GetActDesc()->fProof->Disconnect("StopProcess(Bool_t)", this, - "IndicateStop(Bool_t)"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset progress frame information fields. - -void TSessionQueryFrame::ResetProgressDialog(const char * /*selector*/, Int_t files, - Long64_t first, Long64_t entries) -{ - TString buf; - fFiles = files > 0 ? files : 0; - fFirst = first; - fEntries = entries; - fPrevProcessed = 0; - fPrevTotal = 0; - - if (!fViewer->GetActDesc()->fLocal) { - frmProg->SetBarColor("green"); - frmProg->Reset(); - } - - buf.Form("%0d files, %0lld events, starting event %0lld", - fFiles > 0 ? fFiles : 0, fEntries > 0 ? fEntries : 0, - fFirst >= 0 ? fFirst : 0); - fLabStatus->SetText(buf.Data()); - // Reconnect the slots - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - fViewer->GetActDesc()->fProof->Connect("Progress(Long64_t,Long64_t)", - "TSessionQueryFrame", this, "Progress(Long64_t,Long64_t)"); - fViewer->GetActDesc()->fProof->Connect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - "TSessionQueryFrame", this, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - fViewer->GetActDesc()->fProof->Connect("StopProcess(Bool_t)", - "TSessionQueryFrame", this, "IndicateStop(Bool_t)"); - buf.Form("PROOF cluster : \"%s\" - %d worker nodes", - fViewer->GetActDesc()->fProof->GetMaster(), - fViewer->GetActDesc()->fProof->GetParallel()); - fLabInfos->SetText(buf.Data()); - } - else if (fViewer->GetActDesc()->fLocal) { - fStatsCanvas->Clear(); - fLabInfos->SetText("Local Session"); - fLabStatus->SetText(" "); - } - else { - fLabInfos->SetText(" "); - fLabStatus->SetText(" "); - } - fFB->Layout(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize query. - -void TSessionQueryFrame::OnBtnFinalize() -{ - // check if Proof is valid - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - gPad->SetEditable(kFALSE); - TGListTreeItem *item = fViewer->GetSessionHierarchy()->GetSelected(); - if (!item) return; - TObject *obj = (TObject *)item->GetUserData(); - if ((obj) && (obj->IsA() == TQueryDescription::Class())) { - // as it can take time, set watch cursor - gVirtualX->SetCursor(GetId(),gVirtualX->CreateCursor(kWatch)); - TQueryDescription *query = (TQueryDescription *)obj; - fViewer->GetActDesc()->fProof->Finalize(query->fReference); - UpdateButtons(query); - // restore cursor - gVirtualX->SetCursor(GetId(), 0); - } - } - if (fViewer->GetActDesc()->fLocal) { - gPad->SetEditable(kFALSE); - TChain *chain = (TChain *)fViewer->GetActDesc()->fActQuery->fChain; - if (chain && chain->GetPlayer()) - chain->GetPlayer()->GetSelectorFromFile()->Terminate(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Stop processing query. - -void TSessionQueryFrame::OnBtnStop() -{ - // check for proof validity - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - fViewer->GetActDesc()->fProof->StopProcess(kFALSE); - } - if (fViewer->GetActDesc()->fLocal) { - gROOT->SetInterrupt(); - fViewer->GetActDesc()->fActQuery->fStatus = - TQueryDescription::kSessionQueryStopped; - } - // stop icon animation and set connected icon - fViewer->ChangeRightLogo("monitor01.xpm"); - fViewer->SetChangePic(kFALSE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show query log. - -void TSessionQueryFrame::OnBtnShowLog() -{ - TGListTreeItem *item = fViewer->GetSessionHierarchy()->GetSelected(); - if (!item) return; - TObject *obj = (TObject *)item->GetUserData(); - if ((!obj) || (obj->IsA() != TQueryDescription::Class())) - return; - TQueryDescription *query = (TQueryDescription *)obj; - fViewer->ShowLog(query->fReference.Data()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Retrieve query. - -void TSessionQueryFrame::OnBtnRetrieve() -{ - // check for proof validity - if (fViewer->GetActDesc()->fAttached && - fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - TGListTreeItem *item = fViewer->GetSessionHierarchy()->GetSelected(); - if (!item) return; - TObject *obj = (TObject *)item->GetUserData(); - if (obj && obj->IsA() == TQueryDescription::Class()) { - // as it can take time, set watch cursor - gVirtualX->SetCursor(GetId(), gVirtualX->CreateCursor(kWatch)); - TQueryDescription *query = (TQueryDescription *)obj; - Int_t rc = fViewer->GetActDesc()->fProof->Retrieve(query->fReference); - if (rc == 0) - fViewer->OnCascadeMenu(); - // restore cursor - gVirtualX->SetCursor(GetId(), 0); - } - } - if (fViewer->GetActDesc()->fLocal) { - TGListTreeItem *item=0, *item2=0; - item = fViewer->GetSessionHierarchy()->FindItemByObj(fViewer->GetSessionItem(), - fViewer->GetActDesc()); - if (item) { - item2 = fViewer->GetSessionHierarchy()->FindItemByObj(item, - fViewer->GetActDesc()->fActQuery); - } - if (item2) { - // add input and output list entries - TChain *chain = (TChain *)fViewer->GetActDesc()->fActQuery->fChain; - if (chain && chain->GetPlayer()) { - TSelector *selector = chain->GetPlayer()->GetSelectorFromFile(); - if (selector) { - TList *objlist = selector->GetOutputList(); - if (objlist) - if (!fViewer->GetSessionHierarchy()->FindChildByName(item2, "OutputList")) - fViewer->GetSessionHierarchy()->AddItem(item2, "OutputList"); - } - } - } - // update list tree, query frame information, and buttons state - fViewer->GetSessionHierarchy()->ClearViewPort(); - fClient->NeedRedraw(fViewer->GetSessionHierarchy()); - UpdateInfos(); - UpdateButtons(fViewer->GetActDesc()->fActQuery); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Abort processing query. - -void TSessionQueryFrame::OnBtnAbort() -{ - // check for proof validity - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - fViewer->GetActDesc()->fProof->StopProcess(kTRUE); - } - if (fViewer->GetActDesc()->fLocal) { - gROOT->SetInterrupt(); - fViewer->GetActDesc()->fActQuery->fStatus = - TQueryDescription::kSessionQueryAborted; - } - // stop icon animation and set connected icon - fViewer->ChangeRightLogo("monitor01.xpm"); - fViewer->SetChangePic(kFALSE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Submit query. - -void TSessionQueryFrame::OnBtnSubmit() -{ - Int_t retval; - Long64_t id = 0; - TGListTreeItem *item = fViewer->GetSessionHierarchy()->GetSelected(); - if (!item) return; - // retrieve query description attached to list tree item - TObject *obj = (TObject *)item->GetUserData(); - if (!obj || obj->IsA() != TQueryDescription::Class()) - return; - TQueryDescription *newquery = (TQueryDescription *)obj; - // reset progress information - ResetProgressDialog(newquery->fSelectorString, - newquery->fNbFiles, newquery->fFirstEntry, newquery->fNoEntries); - // set query start time - newquery->fStartTime = gSystem->Now(); - fViewer->GetActDesc()->fNbHistos = 0; - // check for proof validity - if (fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) { - fViewer->GetActDesc()->fProof->SetBit(TProof::kUsingSessionGui); - // set query description status to submitted - newquery->fStatus = TQueryDescription::kSessionQuerySubmitted; - // if feedback option selected - if (fViewer->GetOptionsMenu()->IsEntryChecked(kOptionsFeedback)) { - Int_t i = 0; - // browse list of feedback histos and check user's selected ones - while (kFeedbackHistos[i]) { - if (fViewer->GetCascadeMenu()->IsEntryChecked(41+i)) { - fViewer->GetActDesc()->fProof->AddFeedback(kFeedbackHistos[i]); - fViewer->GetActDesc()->fNbHistos++; - } - i++; - } - // connect feedback signal - fViewer->GetActDesc()->fProof->Connect("Feedback(TList *objs)", - "TSessionQueryFrame", fViewer->GetQueryFrame(), - "Feedback(TList *objs)"); - gROOT->Time(); - } - else { - // if feedback option not selected, clear Proof's feedback option - fViewer->GetActDesc()->fProof->ClearFeedback(); - } - // set current proof session - fViewer->GetActDesc()->fProof->cd(); - // check if parameter file has been specified - if (newquery->fChain) { - if (fViewer->GetActDesc()->fProof->IsLite()) { - newquery->fOptions = ""; - } - // set query reference id - newquery->fReference= TString::Format("session-%s:q%d", - fViewer->GetActDesc()->fProof->GetSessionTag(), - fViewer->GetActDesc()->fProof->GetSeqNum()+1); - if (newquery->fChain->IsA() == TChain::Class()) { - // TChain case - newquery->fStatus = TQueryDescription::kSessionQuerySubmitted; - ((TChain *)newquery->fChain)->SetProof(fViewer->GetActDesc()->fProof); - id = ((TChain *)newquery->fChain)->Process(newquery->fSelectorString, - newquery->fOptions, - newquery->fNoEntries > 0 ? newquery->fNoEntries : 1234567890, - newquery->fFirstEntry); - } - else if (newquery->fChain->IsA() == TDSet::Class()) { - // TDSet case - newquery->fStatus = TQueryDescription::kSessionQuerySubmitted; - id = ((TDSet *)newquery->fChain)->Process(newquery->fSelectorString, - newquery->fOptions, - newquery->fNoEntries, - newquery->fFirstEntry); - } - } - else { - Error("Submit", "No TChain defined; skipping"); - newquery->fStatus = TQueryDescription::kSessionQueryCreated; - return; - } - // set query reference id to unique identifier - newquery->fReference= TString::Format("session-%s:q%lld", - fViewer->GetActDesc()->fProof->GetSessionTag(), id); - // start icon animation - fViewer->SetChangePic(kTRUE); - } - else if (fViewer->GetActDesc()->fLocal) { // local session case - // if feedback option selected - if (fViewer->GetOptionsMenu()->IsEntryChecked(kOptionsFeedback)) { - Int_t i = 0; - // browse list of feedback histos and check user's selected ones - while (kFeedbackHistos[i]) { - if (fViewer->GetCascadeMenu()->IsEntryChecked(41+i)) { - fViewer->GetActDesc()->fNbHistos++; - } - i++; - } - } - if (newquery->fChain) { - if (newquery->fChain->IsA() == TChain::Class()) { - // TChain case - newquery->fStatus = TQueryDescription::kSessionQueryRunning; - fViewer->EnableTimer(); - UpdateButtons(newquery); - gPad->SetEditable(kFALSE); - ((TChain *)newquery->fChain)->SetTimerInterval(100); - id = ((TChain *)newquery->fChain)->Process(newquery->fSelectorString, - newquery->fOptions, - newquery->fNoEntries > 0 ? newquery->fNoEntries : 1234567890, - newquery->fFirstEntry); - ((TChain *)newquery->fChain)->SetTimerInterval(0); - OnBtnRetrieve(); - TChain *chain = (TChain *)newquery->fChain; - ProgressLocal(chain->GetEntries(), - chain->GetReadEntry()+1); - if ((newquery->fStatus != TQueryDescription::kSessionQueryAborted) && - (newquery->fStatus != TQueryDescription::kSessionQueryStopped)) - newquery->fStatus = TQueryDescription::kSessionQueryCompleted; - UpdateButtons(newquery); - } - else { - new TGMsgBox(fClient->GetRoot(), this, "Error Submitting Query", - "Only TChains are allowed in Local Session (no TDSet) !", - kMBIconExclamation,kMBOk,&retval); - } - } - else { - Error("Submit", "No TChain defined; skipping"); - newquery->fStatus = TQueryDescription::kSessionQueryCreated; - return; - } - // set query reference id to unique identifier - newquery->fReference = TString::Format("local-session-%s:q%lld", newquery->fQueryName.Data(), id); - } - // update buttons state - UpdateButtons(newquery); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update buttons state for the current query status. - -void TSessionQueryFrame::UpdateButtons(TQueryDescription *desc) -{ - TGListTreeItem *item = fViewer->GetSessionHierarchy()->GetSelected(); - if (!item) return; - // retrieve query description attached to list tree item - TObject *obj = (TObject *)item->GetUserData(); - if (!obj || obj->IsA() != TQueryDescription::Class()) - return; - TQueryDescription *query = (TQueryDescription *)obj; - if (desc != query) return; - - Bool_t submit_en = kFALSE; - if ((fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid()) || - fViewer->GetActDesc()->fLocal) - submit_en = kTRUE; - - switch (desc->fStatus) { - case TQueryDescription::kSessionQueryFromProof: - fBtnSubmit->SetEnabled(submit_en); - fBtnFinalize->SetEnabled(kTRUE); - fBtnStop->SetEnabled(kFALSE); - fBtnAbort->SetEnabled(kFALSE); - fBtnShowLog->SetEnabled(kTRUE); - fBtnRetrieve->SetEnabled(kTRUE); - break; - - case TQueryDescription::kSessionQueryCompleted: - fBtnSubmit->SetEnabled(submit_en); - fBtnFinalize->SetEnabled(kTRUE); - if (((desc->fResult == 0) || (desc->fResult && - (desc->fResult->IsFinalized() || - (desc->fResult->GetInputObject("TDSet") == 0)))) && - !(fViewer->GetActDesc()->fLocal)) - fBtnFinalize->SetEnabled(kFALSE); - fBtnStop->SetEnabled(kFALSE); - fBtnAbort->SetEnabled(kFALSE); - fBtnShowLog->SetEnabled(kTRUE); - fBtnRetrieve->SetEnabled(kTRUE); - break; - - case TQueryDescription::kSessionQueryCreated: - fBtnSubmit->SetEnabled(submit_en); - fBtnFinalize->SetEnabled(kFALSE); - fBtnStop->SetEnabled(kFALSE); - fBtnAbort->SetEnabled(kFALSE); - fBtnShowLog->SetEnabled(kTRUE); - fBtnRetrieve->SetEnabled(kFALSE); - break; - - case TQueryDescription::kSessionQuerySubmitted: - fBtnSubmit->SetEnabled(kFALSE); - fBtnFinalize->SetEnabled(kFALSE); - fBtnStop->SetEnabled(kTRUE); - fBtnAbort->SetEnabled(kTRUE); - fBtnShowLog->SetEnabled(kTRUE); - fBtnRetrieve->SetEnabled(kFALSE); - break; - - case TQueryDescription::kSessionQueryRunning: - fBtnSubmit->SetEnabled(kFALSE); - fBtnFinalize->SetEnabled(kFALSE); - fBtnStop->SetEnabled(kTRUE); - fBtnAbort->SetEnabled(kTRUE); - fBtnShowLog->SetEnabled(kTRUE); - fBtnRetrieve->SetEnabled(kFALSE); - break; - - case TQueryDescription::kSessionQueryStopped: - fBtnSubmit->SetEnabled(submit_en); - fBtnFinalize->SetEnabled(kTRUE); - fBtnStop->SetEnabled(kFALSE); - fBtnAbort->SetEnabled(kFALSE); - fBtnShowLog->SetEnabled(kTRUE); - fBtnRetrieve->SetEnabled(kTRUE); - break; - - case TQueryDescription::kSessionQueryAborted: - fBtnSubmit->SetEnabled(submit_en); - fBtnFinalize->SetEnabled(kFALSE); - fBtnStop->SetEnabled(kFALSE); - fBtnAbort->SetEnabled(kFALSE); - fBtnShowLog->SetEnabled(kTRUE); - fBtnRetrieve->SetEnabled(kFALSE); - break; - - case TQueryDescription::kSessionQueryFinalized: - fBtnSubmit->SetEnabled(submit_en); - fBtnFinalize->SetEnabled(kFALSE); - fBtnStop->SetEnabled(kFALSE); - fBtnAbort->SetEnabled(kFALSE); - fBtnShowLog->SetEnabled(kTRUE); - fBtnRetrieve->SetEnabled(kFALSE); - break; - - default: - break; - } - if (fViewer->GetActDesc()->fLocal && - !(fViewer->GetActDesc()->fActQuery->fChain)) { - fBtnFinalize->SetEnabled(kFALSE); - fBtnRetrieve->SetEnabled(kFALSE); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update query information (header) text view. - -void TSessionQueryFrame::UpdateInfos() -{ - TString buffer; - const char *qst[] = {"aborted ", "submitted", "running ", - "stopped ", "completed"}; - - if (fViewer->GetActDesc()->fActQuery) - fFD->UpdateFields(fViewer->GetActDesc()->fActQuery); - - if (fViewer->GetActDesc()->fLocal || - (fViewer->GetActDesc()->fConnected && - fViewer->GetActDesc()->fAttached && - fViewer->GetActDesc()->fProof && - fViewer->GetActDesc()->fProof->IsValid())) { - fBtnSave->SetText(" Submit "); - } - else { - fBtnSave->SetText(" Apply changes "); - } - fClient->NeedRedraw(fBtnSave); - fInfoTextView->Clear(); - if (!fViewer->GetActDesc()->fActQuery || - !fViewer->GetActDesc()->fActQuery->fResult) { - ResetProgressDialog("", 0, 0, 0); - if (fViewer->GetActDesc()->fLocal) { - if (fViewer->GetActDesc()->fActQuery) { - TChain *chain = (TChain *)fViewer->GetActDesc()->fActQuery->fChain; - if (chain) { - ProgressLocal(chain->GetEntries(), - chain->GetReadEntry()+1); - } - else { - ProgressLocal(0, 0); - } - UpdateButtons(fViewer->GetActDesc()->fActQuery); - } - } - else { - fTotal->SetText(" Estimated time left : 0 sec (0 events of 0 processed) "); - fRate->SetText(" Processing Rate : 0.0f events/sec "); - frmProg->Reset(); - fFB->Layout(); - } - return; - } - TQueryResult *result = fViewer->GetActDesc()->fActQuery->fResult; - - // Status label - Int_t st = (result->GetStatus() > 0 && result->GetStatus() <= - TQueryResult::kCompleted) ? result->GetStatus() : 0; - - Int_t qry = result->GetSeqNum(); - - buffer = TString::Format("------------------------------------------------------\n"); - // Print header - if (!result->IsDraw()) { - const char *fin = result->IsFinalized() ? "finalized" : qst[st]; - const char *arc = result->IsArchived() ? "(A)" : ""; - buffer = TString::Format(" Query No : %d\n", qry); - buffer += TString::Format(" Ref : \"%s:%s\"\n", result->GetTitle(), - result->GetName()); - buffer += TString::Format(" Selector : %s\n", - result->GetSelecImp()->GetTitle()); - buffer += TString::Format("Status : %9s%s\n", fin, arc); - buffer += TString::Format("------------------------------------------------------\n"); - } else { - buffer += TString::Format(" Query No : %d\n", qry); - buffer += TString::Format(" Ref : \"%s:%s\"\n", result->GetTitle(), - result->GetName()); - buffer += TString::Format(" Selector : %s\n", - result->GetSelecImp()->GetTitle()); - buffer += TString::Format("------------------------------------------------------\n"); - } - - // Time information - Int_t elapsed = (Int_t)(result->GetEndTime().Convert() - - result->GetStartTime().Convert()); - buffer += TString::Format(" Started : %s\n", - result->GetStartTime().AsString()); - buffer += TString::Format(" Real time : %d sec (CPU time: %.1f sec)\n", - elapsed, result->GetUsedCPU()); - - // Number of events processed, rate, size - Double_t rate = 0.0; - if (result->GetEntries() > -1 && elapsed > 0) - rate = result->GetEntries() / (Double_t)elapsed ; - Float_t size = ((Float_t)result->GetBytes())/(1024*1024); - buffer += TString::Format(" Processed : %lld events (size: %.3f MBs)\n", - result->GetEntries(), size); - buffer += TString::Format(" Rate : %.1f evts/sec\n", rate); - - // Package information - if (strlen(result->GetParList()) > 1) { - buffer += TString::Format(" Packages : %s\n", result->GetParList()); - } - - // Result information - TString res = result->GetResultFile(); - if (!result->IsArchived()) { - Int_t dq = res.Index("queries"); - if (dq > -1) { - res.Remove(0,res.Index("queries")); - res.Insert(0,"/"); - } - if (res.BeginsWith("-")) { - res = (result->GetStatus() == TQueryResult::kAborted) ? - "not available" : "sent to client"; - } - } - if (res.Length() > 1) { - buffer += TString::Format("------------------------------------------------------\n"); - buffer += TString::Format(" Results : %s\n", res.Data()); - } - - if (result->GetOutputList() && result->GetOutputList()->GetSize() > 0) { - buffer += TString::Format(" Outlist : %d objects\n", - result->GetOutputList()->GetSize()); - buffer += TString::Format("------------------------------------------------------\n"); - } - fInfoTextView->LoadBuffer(buffer.Data()); - - //Float_t pos = Float_t((Double_t)(result->GetEntries() * 100)/(Double_t)total); - if (result->GetStatus() == TQueryResult::kAborted) - frmProg->SetBarColor("red"); - else if (result->GetStatus() == TQueryResult::kStopped) - frmProg->SetBarColor("yellow"); - else - frmProg->SetBarColor("green"); - - frmProg->SetPosition(100.0); - - buffer = TString::Format(" Processed : %lld events in %.1f sec", result->GetEntries(), - (Float_t)elapsed); - fTotal->SetText(buffer.Data()); - buffer = TString::Format(" Processing Rate : %.1f events/sec ", rate); - fRate->SetText(buffer.Data()); - fFB->Layout(); -} - -////////////////////////////////////////////////////////////////////////////////////////// -// Output frame - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor. - -TSessionOutputFrame::TSessionOutputFrame(TGWindow* p, Int_t w, Int_t h) : - TGCompositeFrame(p, w, h), fEntryTmp(0), fLVContainer(0), fViewer(0) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TSessionOutputFrame::~TSessionOutputFrame() -{ - delete fLVContainer; // this container is inside the TGListView and is not - // deleted automatically - Cleanup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build query output information frame. - -void TSessionOutputFrame::Build(TSessionViewer *gui) -{ - fViewer = gui; - SetLayoutManager(new TGVerticalLayout(this)); - SetCleanup(kDeepCleanup); - - // Container of object TGListView - TGListView *frmListView = new TGListView(this, 340, 190); - fLVContainer = new TGLVContainer(frmListView, kSunkenFrame, GetWhitePixel()); - fLVContainer->Associate(frmListView); - fLVContainer->SetCleanup(kDeepCleanup); - AddFrame(frmListView, new TGLayoutHints(kLHintsExpandX | kLHintsExpandY, - 4, 4, 4, 4)); - - frmListView->Connect("Clicked(TGLVEntry*, Int_t, Int_t, Int_t)", - "TSessionOutputFrame", this, - "OnElementClicked(TGLVEntry* ,Int_t, Int_t, Int_t)"); - frmListView->Connect("DoubleClicked(TGLVEntry*, Int_t, Int_t, Int_t)", - "TSessionOutputFrame", this, - "OnElementDblClicked(TGLVEntry* ,Int_t, Int_t, Int_t)"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle mouse clicks on list view items. - -void TSessionOutputFrame::OnElementClicked(TGLVEntry* entry, Int_t btn, Int_t x, - Int_t y) -{ - TObject *obj = (TObject *)entry->GetUserData(); - if ((obj) && (btn ==3)) { - // if right button, popup context menu - fViewer->GetContextMenu()->Popup(x, y, obj, (TBrowser *)0); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle double-clicks on list view items. - -void TSessionOutputFrame::OnElementDblClicked(TGLVEntry* entry, Int_t , Int_t, Int_t) -{ - char action[512]; - TString act; - TObject *obj = (TObject *)entry->GetUserData(); - if (!obj) return; - TString ext = obj->GetName(); - gPad->SetEditable(kFALSE); - // check default action from root.mimes - if (fClient->GetMimeTypeList()->GetAction(obj->IsA()->GetName(), action)) { - act = TString::Format("((%s*)0x%zx)%s", obj->IsA()->GetName(), (size_t)obj, action); - if (act[0] == '!') { - act.Remove(0, 1); - gSystem->Exec(act.Data()); - } else { - // do not allow browse - if (!act.Contains("Browse")) - gROOT->ProcessLine(act.Data()); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add object to output list view. - -void TSessionOutputFrame::AddObject(TObject *obj) -{ - TGLVEntry *item; - if (obj) { - item = new TGLVEntry(fLVContainer, obj->GetName(), obj->IsA()->GetName()); - item->SetUserData(obj); - fLVContainer->AddItem(item); - } -} - -////////////////////////////////////////////////////////////////////////////////////////// -// Input Frame - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor. - -TSessionInputFrame::TSessionInputFrame(TGWindow* p, Int_t w, Int_t h) : - TGCompositeFrame(p, w, h), fViewer(0), fLVContainer(0) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TSessionInputFrame::~TSessionInputFrame() -{ - delete fLVContainer; // this container is inside the TGListView and is not - // deleted automatically - Cleanup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build query input information frame. - -void TSessionInputFrame::Build(TSessionViewer *gui) -{ - fViewer = gui; - SetLayoutManager(new TGVerticalLayout(this)); - SetCleanup(kDeepCleanup); - - // Container of object TGListView - TGListView *frmListView = new TGListView(this, 340, 190); - fLVContainer = new TGLVContainer(frmListView, kSunkenFrame, GetWhitePixel()); - fLVContainer->Associate(frmListView); - fLVContainer->SetCleanup(kDeepCleanup); - AddFrame(frmListView, new TGLayoutHints(kLHintsExpandX | kLHintsExpandY, - 4, 4, 4, 4)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add object to input list view. - -void TSessionInputFrame::AddObject(TObject *obj) -{ - TGLVEntry *item; - if (obj) { - item = new TGLVEntry(fLVContainer, obj->GetName(), obj->IsA()->GetName()); - item->SetUserData(obj); - fLVContainer->AddItem(item); - } -} - -////////////////////////////////////////////////////////////////////////////////////////// -// Session Viewer Main Frame - -//////////////////////////////////////////////////////////////////////////////// -/// Main Session viewer constructor. - -TSessionViewer::TSessionViewer(const char *name, UInt_t w, UInt_t h) : - TGMainFrame(gClient->GetRoot(), w, h), fSessionHierarchy(0), fSessionItem(0) -{ - // only one session viewer allowed - if (gSessionViewer) - // coverity[uninit_member]: already done - return; - Build(); - SetWindowName(name); - Resize(w, h); - gSessionViewer = this; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Main Session viewer constructor. - -TSessionViewer::TSessionViewer(const char *name, Int_t x, Int_t y, UInt_t w, - UInt_t h) : TGMainFrame(gClient->GetRoot(), w, h), - fSessionHierarchy(0), fSessionItem(0) -{ - // only one session viewer allowed - if (gSessionViewer) - // coverity[uninit_member]: already done - return; - Build(); - SetWindowName(name); - Move(x, y); - Resize(w, h); - gSessionViewer = this; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Read configuration file and populate list of sessions -/// list of queries and list of packages. -/// Read and set also global options as feedback histos. - -void TSessionViewer::ReadConfiguration(const char *filename) -{ - if (fViewerEnv) - delete fViewerEnv; - fViewerEnv = new TEnv(); - - const char *fn = (filename && strlen(filename)) ? filename : fConfigFile.Data(); - - fViewerEnv->ReadFile(fn, kEnvUser); - - Bool_t bval = (Bool_t)fViewerEnv->GetValue("Option.Feedback", 1); - if (bval) - fOptionsMenu->CheckEntry(kOptionsFeedback); - else - fOptionsMenu->UnCheckEntry(kOptionsFeedback); - - bval = (Bool_t)fViewerEnv->GetValue("Option.MasterHistos", 1); - if (bval) { - fOptionsMenu->CheckEntry(kOptionsStatsHist); - gEnv->SetValue("Proof.StatsHist", 1); - } - else { - fOptionsMenu->UnCheckEntry(kOptionsStatsHist); - gEnv->SetValue("Proof.StatsHist", 0); - } - bval = (Bool_t)fViewerEnv->GetValue("Option.MasterEvents", 0); - if (bval) - fOptionsMenu->CheckEntry(kOptionsStatsTrace); - else - fOptionsMenu->UnCheckEntry(kOptionsStatsTrace); - bval = (Bool_t)fViewerEnv->GetValue("Option.WorkerEvents", 0); - if (bval) - fOptionsMenu->CheckEntry(kOptionsSlaveStatsTrace); - else - fOptionsMenu->UnCheckEntry(kOptionsSlaveStatsTrace); - - Int_t i = 0; - while (kFeedbackHistos[i]) { - bval = (Bool_t)fViewerEnv->GetValue(Form("Option.%s",kFeedbackHistos[i]), - i == 1 ? 1 : 0); - if (bval) - fCascadeMenu->CheckEntry(41+i); - else - fCascadeMenu->UnCheckEntry(41+i); - i++; - } - TSessionDescription *proofDesc; - fSessions->Delete(); - if (fSessionItem) - fSessionHierarchy->DeleteChildren(fSessionItem); - else - fSessionItem = fSessionHierarchy->AddItem(0, "Sessions", fBaseIcon, - fBaseIcon); - // add local session description - TGListTreeItem *item = fSessionHierarchy->AddItem(fSessionItem, "Local", - fLocal, fLocal); - fSessionHierarchy->SetToolTipItem(item, "Local Session"); - TSessionDescription *localdesc = new TSessionDescription(); - localdesc->fTag = ""; - localdesc->fName = "Local"; - localdesc->fAddress = "Local"; - localdesc->fPort = 0; - localdesc->fConfigFile = ""; - localdesc->fLogLevel = 0; - localdesc->fUserName = ""; - localdesc->fQueries = new TList(); - localdesc->fPackages = new TList(); - localdesc->fActQuery = 0; - localdesc->fProof = 0; - localdesc->fProofMgr = 0; - localdesc->fAttached = kFALSE; - localdesc->fConnected = kFALSE; - localdesc->fLocal = kTRUE; - localdesc->fSync = kTRUE; - localdesc->fAutoEnable = kFALSE; - localdesc->fNbHistos = 0; - item->SetUserData(localdesc); - fSessions->Add((TObject *)localdesc); - fActDesc = localdesc; - - SysInfo_t info; - gSystem->GetSysInfo(&info); - // if the machine has more than one CPU, add one PROOF lite session - // (not supported on Windows yet) - if (!info.fOS.Contains("Microsoft") && info.fCpus > 1) { - // add proof lite session description - item = fSessionHierarchy->AddItem(fSessionItem, "Lite", - fProofDiscon, fProofDiscon); - fSessionHierarchy->SetToolTipItem(item, "PROOF Lite"); - TSessionDescription *litedesc = new TSessionDescription(); - litedesc->fTag = ""; - litedesc->fName = "PROOF Lite"; - litedesc->fAddress = "lite://"; - litedesc->fPort = 0; - litedesc->fConfigFile = ""; - litedesc->fLogLevel = 0; - litedesc->fUserName = ""; - litedesc->fQueries = new TList(); - litedesc->fPackages = new TList(); - litedesc->fActQuery = 0; - litedesc->fProof = 0; - litedesc->fProofMgr = 0; - litedesc->fAttached = kFALSE; - litedesc->fConnected = kFALSE; - litedesc->fLocal = kFALSE; - litedesc->fSync = kTRUE; - litedesc->fAutoEnable = kFALSE; - litedesc->fNbHistos = 0; - item->SetUserData(litedesc); - fSessions->Add((TObject *)litedesc); - fActDesc = litedesc; - } - TIter next(fViewerEnv->GetTable()); - TEnvRec *er; - while ((er = (TEnvRec*) next())) { - const char *s; - if ((s = strstr(er->GetName(), "SessionDescription."))) { - const char *val = fViewerEnv->GetValue(s, (const char*)0); - if (val) { - Int_t cnt = 0; - char *v = StrDup(val); - s += 7; - while (1) { - TString name = strtok(!cnt ? v : 0, ";"); - if (name.IsNull()) break; - TString sessiontag = strtok(0, ";"); - TString address = strtok(0, ";"); - if (address.IsNull()) break; - TString port = strtok(0, ";"); - if (port.IsNull()) break; - TString loglevel = strtok(0, ";"); - if (loglevel.IsNull()) break; - TString configfile = strtok(0, ";"); - TString user = strtok(0, ";"); - if (user.IsNull()) break; - TString sync = strtok(0, ";"); - TString autoen = strtok(0, ";"); - - // build session description - proofDesc = new TSessionDescription(); - proofDesc->fTag = sessiontag.Length() > 2 ? sessiontag.Data() : ""; - proofDesc->fName = name; - proofDesc->fAddress = address; - proofDesc->fPort = atoi(port); - proofDesc->fConfigFile = configfile.Length() > 2 ? configfile.Data() : ""; - proofDesc->fLogLevel = atoi(loglevel); - proofDesc->fConnected = kFALSE; - proofDesc->fAttached = kFALSE; - proofDesc->fLocal = kFALSE; - proofDesc->fQueries = new TList(); - proofDesc->fPackages = new TList(); - proofDesc->fActQuery = 0; - proofDesc->fProof = 0; - proofDesc->fProofMgr = 0; - proofDesc->fSync = (Bool_t)(atoi(sync)); - proofDesc->fAutoEnable = (Bool_t)(atoi(autoen)); - proofDesc->fUserName = user; - fSessions->Add((TObject *)proofDesc); - item = fSessionHierarchy->AddItem( - fSessionItem, proofDesc->fName.Data(), - fProofDiscon, fProofDiscon); - fSessionHierarchy->SetToolTipItem(item, "Proof Session"); - item->SetUserData(proofDesc); - fActDesc = proofDesc; - cnt++; - } - delete [] v; - } - } - if ((s = strstr(er->GetName(), "QueryDescription."))) { - const char *val = fViewerEnv->GetValue(s, (const char*)0); - if (val) { - Int_t cnt = 0; - char *v = StrDup(val); - s += 7; - while (1) { - - TString status = strtok(!cnt ? v : 0, ";"); - if (status.IsNull()) break; - - TString reference = strtok(0, ";"); - if (reference.IsNull()) break; - TString queryname = strtok(0, ";"); - if (queryname.IsNull()) break; - TString selector = strtok(0, ";"); - if (selector.IsNull()) break; - TString dset = strtok(0, ";"); - TString options = strtok(0, ";"); - TString eventlist = strtok(0, ";"); - TString nbfiles = strtok(0, ";"); - TString nbentries = strtok(0, ";"); - TString firstentry = strtok(0, ";"); - - TQueryDescription *newquery = new TQueryDescription(); - newquery->fStatus = - (TQueryDescription::ESessionQueryStatus)(atoi(status)); - newquery->fSelectorString = selector.Length() > 2 ? selector.Data() : ""; - newquery->fReference = reference.Length() > 2 ? reference.Data() : ""; - newquery->fTDSetString = dset.Length() > 2 ? dset.Data() : ""; - newquery->fQueryName = queryname.Length() > 2 ? queryname.Data() : ""; - newquery->fOptions = options.Length() > 2 ? options.Data() : ""; - newquery->fEventList = eventlist.Length() > 2 ? eventlist.Data() : ""; - newquery->fNbFiles = atoi(nbfiles); - newquery->fNoEntries = atoi(nbentries); - newquery->fFirstEntry = atoi(firstentry); - newquery->fResult = 0; - newquery->fChain = 0; - fActDesc->fQueries->Add((TObject *)newquery); - cnt++; - TGListTreeItem *item1 = fSessionHierarchy->FindChildByData( - fSessionItem, fActDesc); - TGListTreeItem *item2 = fSessionHierarchy->AddItem( - item1, newquery->fQueryName, fQueryCon, fQueryCon); - item2->SetUserData(newquery); - } - delete [] v; - } - } - } - fSessionHierarchy->ClearHighlighted(); - fSessionHierarchy->OpenItem(fSessionItem); - if (fActDesc == localdesc) { - fSessionHierarchy->HighlightItem(fSessionItem); - fSessionHierarchy->SetSelected(fSessionItem); - } - else { - fSessionHierarchy->OpenItem(item); - fSessionHierarchy->HighlightItem(item); - fSessionHierarchy->SetSelected(item); - } - fSessionHierarchy->ClearViewPort(); - fClient->NeedRedraw(fSessionHierarchy); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update list of existing Proof sessions. - -void TSessionViewer::UpdateListOfProofs() -{ - // get list of proof sessions - Bool_t found = kFALSE; - Bool_t exists = kFALSE; - TGListTreeItem *item = 0; - TSeqCollection *proofs = gROOT->GetListOfProofs(); - TSessionDescription *desc = 0; - TSessionDescription *newdesc; - if (proofs) { - TObject *o = proofs->First(); - if (o && dynamic_cast(o)) { - TProofMgr *mgr = dynamic_cast(o); - if (mgr && mgr->QuerySessions("L")) { - TIter nxd(mgr->QuerySessions("L")); - TProofDesc *d = 0; - TProof *p = 0; - while ((d = (TProofDesc *)nxd())) { - TIter nextfs(fSessions); - // check if session exists in the list - exists = kFALSE; - while ((desc = (TSessionDescription *)nextfs())) { - if ((desc->fTag == d->GetName()) || - (desc->fName == d->GetTitle())) { - exists = kTRUE; - break; - } - } - TIter nexts(fSessions); - found = kFALSE; - p = d->GetProof(); - while ((desc = (TSessionDescription *)nexts())) { - if (desc->fConnected && desc->fAttached) - continue; - if (p && ((exists && ((desc->fTag == d->GetName()) || - (desc->fName == d->GetTitle()))) || - (!exists && (desc->fAddress == p->GetMaster())))) { - desc->fConnected = kTRUE; - desc->fAttached = kTRUE; - desc->fProof = p; - desc->fProofMgr = mgr; - desc->fTag = d->GetName(); - item = fSessionHierarchy->FindChildByData(fSessionItem, - desc); - if (item) { - item->SetPictures(fProofCon, fProofCon); - if (item == fSessionHierarchy->GetSelected()) { - fActDesc->fProof->Connect("Progress(Long64_t,Long64_t)", - "TSessionQueryFrame", fQueryFrame, - "Progress(Long64_t,Long64_t)"); - fActDesc->fProof->Connect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - "TSessionQueryFrame", fQueryFrame, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - fActDesc->fProof->Connect("StopProcess(Bool_t)", - "TSessionQueryFrame", fQueryFrame, - "IndicateStop(Bool_t)"); - fActDesc->fProof->Connect( - "ResetProgressDialog(const char*, Int_t,Long64_t,Long64_t)", - "TSessionQueryFrame", fQueryFrame, - "ResetProgressDialog(const char*,Int_t,Long64_t,Long64_t)"); - // enable timer used for status bar icon's animation - EnableTimer(); - // change status bar right icon to connected pixmap - ChangeRightLogo("monitor01.xpm"); - // do not animate yet - SetChangePic(kFALSE); - // connect to signal "query result ready" - fActDesc->fProof->Connect("QueryResultReady(char *)", - "TSessionViewer", this, "QueryResultReady(char *)"); - // display connection information on status bar - TString msg; - msg.Form("PROOF Cluster %s ready", fActDesc->fName.Data()); - fStatusBar->SetText(msg.Data(), 1); - UpdateListOfPackages(); - fSessionFrame->UpdatePackages(); - fSessionFrame->UpdateListOfDataSets(); - fPopupSrv->DisableEntry(kSessionConnect); - fSessionMenu->DisableEntry(kSessionConnect); - fPopupSrv->EnableEntry(kSessionDisconnect); - fSessionMenu->EnableEntry(kSessionDisconnect); - if (fToolBar->GetButton(kSessionDisconnect)) - fToolBar->GetButton(kSessionDisconnect)->SetState(kButtonUp); - if (fToolBar->GetButton(kSessionConnect)) - fToolBar->GetButton(kSessionConnect)->SetState(kButtonDisabled); - fSessionFrame->SetLogLevel(fActDesc->fLogLevel); - // update session information frame - fSessionFrame->ProofInfos(); - fSessionFrame->SetLocal(kFALSE); - if (fActFrame != fSessionFrame) { - fV2->HideFrame(fActFrame); - fV2->ShowFrame(fSessionFrame); - fActFrame = fSessionFrame; - } - } - } - if (desc->fLogLevel < 0) - desc->fLogLevel = 0; - found = kTRUE; - break; - } - } - if (found) continue; - newdesc = new TSessionDescription(); - // and fill information from Proof session - newdesc->fTag = d->GetName(); - newdesc->fName = d->GetTitle(); - newdesc->fAddress = d->GetTitle(); - newdesc->fConnected = kFALSE; - newdesc->fAttached = kFALSE; - newdesc->fProofMgr = mgr; - p = d->GetProof(); - if (p) { - newdesc->fConnected = kTRUE; - newdesc->fAttached = kTRUE; - if (p->IsLite()) - newdesc->fAddress = "lite://"; - else - newdesc->fAddress = p->GetMaster(); - newdesc->fConfigFile = p->GetConfFile(); - newdesc->fUserName = p->GetUser(); - newdesc->fPort = p->GetPort(); - newdesc->fLogLevel = p->GetLogLevel(); - newdesc->fProof = p; - newdesc->fProof->Connect("Progress(Long64_t,Long64_t)", - "TSessionQueryFrame", fQueryFrame, - "Progress(Long64_t,Long64_t)"); - newdesc->fProof->Connect("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - "TSessionQueryFrame", fQueryFrame, - "Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)"); - newdesc->fProof->Connect("StopProcess(Bool_t)", - "TSessionQueryFrame", fQueryFrame, - "IndicateStop(Bool_t)"); - newdesc->fProof->Connect( - "ResetProgressDialog(const char*, Int_t,Long64_t,Long64_t)", - "TSessionQueryFrame", fQueryFrame, - "ResetProgressDialog(const char*,Int_t,Long64_t,Long64_t)"); - // enable timer used for status bar icon's animation - EnableTimer(); - // change status bar right icon to connected pixmap - ChangeRightLogo("monitor01.xpm"); - // do not animate yet - SetChangePic(kFALSE); - // connect to signal "query result ready" - newdesc->fProof->Connect("QueryResultReady(char *)", - "TSessionViewer", this, "QueryResultReady(char *)"); - } - newdesc->fQueries = new TList(); - newdesc->fPackages = new TList(); - if (newdesc->fLogLevel < 0) - newdesc->fLogLevel = 0; - newdesc->fActQuery = 0; - newdesc->fLocal = kFALSE; - newdesc->fSync = kFALSE; - newdesc->fAutoEnable = kFALSE; - newdesc->fNbHistos = 0; - // add new session description in list tree - if (p) - item = fSessionHierarchy->AddItem(fSessionItem, newdesc->fName.Data(), - fProofCon, fProofCon); - else - item = fSessionHierarchy->AddItem(fSessionItem, newdesc->fName.Data(), - fProofDiscon, fProofDiscon); - fSessionHierarchy->SetToolTipItem(item, "Proof Session"); - item ->SetUserData(newdesc); - // and in our session description list - fSessions->Add(newdesc); - } - } - return; - } - TIter nextp(proofs); - TProof *proof; - // loop over existing Proof sessions - while ((proof = (TProof *)nextp())) { - TIter nexts(fSessions); - found = kFALSE; - // check if session is already in the list - while ((desc = (TSessionDescription *)nexts())) { - if (desc->fProof == proof) { - desc->fConnected = kTRUE; - desc->fAttached = kTRUE; - found = kTRUE; - break; - } - } - if (found) continue; - // create new session description - newdesc = new TSessionDescription(); - // and fill information from Proof session - newdesc->fName = proof->GetMaster(); - newdesc->fConfigFile = proof->GetConfFile(); - newdesc->fUserName = proof->GetUser(); - newdesc->fPort = proof->GetPort(); - newdesc->fLogLevel = proof->GetLogLevel(); - if (newdesc->fLogLevel < 0) - newdesc->fLogLevel = 0; - if (proof->IsLite()) - newdesc->fAddress = "lite://"; - else - newdesc->fAddress = proof->GetMaster(); - newdesc->fQueries = new TList(); - newdesc->fPackages = new TList(); - newdesc->fProof = proof; - newdesc->fActQuery = 0; - newdesc->fConnected = kTRUE; - newdesc->fAttached = kTRUE; - newdesc->fLocal = kFALSE; - newdesc->fSync = kFALSE; - newdesc->fAutoEnable = kFALSE; - newdesc->fNbHistos = 0; - // add new session description in list tree - item = fSessionHierarchy->AddItem(fSessionItem, newdesc->fName.Data(), - fProofCon, fProofCon); - fSessionHierarchy->SetToolTipItem(item, "Proof Session"); - item ->SetUserData(newdesc); - // and in our session description list - fSessions->Add(newdesc); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update list of existing Proof sessions. - -void TSessionViewer::UpdateListOfSessions() -{ - // get list of proof sessions - TGListTreeItem *item; - TList *sessions = fActDesc->fProofMgr->QuerySessions(""); - if (sessions) { - TIter nextp(sessions); - TProofDesc *pdesc; - TProof *proof; - TSessionDescription *newdesc; - // loop over existing Proof sessions - while ((pdesc = (TProofDesc *)nextp())) { - TIter nexts(fSessions); - TSessionDescription *desc = 0; - Bool_t found = kFALSE; - // check if session is already in the list - while ((desc = (TSessionDescription *)nexts())) { - if ((desc->fTag == pdesc->GetName()) || - (desc->fName == pdesc->GetTitle())) { - desc->fConnected = kTRUE; - found = kTRUE; - break; - } - } - if (found) continue; - // create new session description - newdesc = new TSessionDescription(); - // and fill information from Proof session - newdesc->fTag = pdesc->GetName(); - newdesc->fName = pdesc->GetTitle(); - proof = pdesc->GetProof(); - if (proof) { - newdesc->fConfigFile = proof->GetConfFile(); - newdesc->fUserName = proof->GetUser(); - newdesc->fPort = proof->GetPort(); - newdesc->fLogLevel = proof->GetLogLevel(); - if (newdesc->fLogLevel < 0) - newdesc->fLogLevel = 0; - if (proof->IsLite()) - newdesc->fAddress = "lite://"; - else - newdesc->fAddress = proof->GetMaster(); - newdesc->fProof = proof; - } - else { - newdesc->fProof = 0; - newdesc->fConfigFile = ""; - newdesc->fUserName = fActDesc->fUserName; - newdesc->fPort = fActDesc->fPort; - newdesc->fLogLevel = 0; - newdesc->fAddress = fActDesc->fAddress; - } - newdesc->fQueries = new TList(); - newdesc->fPackages = new TList(); - newdesc->fProofMgr = fActDesc->fProofMgr; - newdesc->fActQuery = 0; - newdesc->fConnected = kTRUE; - newdesc->fAttached = kFALSE; - newdesc->fLocal = kFALSE; - newdesc->fSync = kFALSE; - newdesc->fAutoEnable = kFALSE; - newdesc->fNbHistos = 0; - // add new session description in list tree - item = fSessionHierarchy->AddItem(fSessionItem, newdesc->fName.Data(), - fProofDiscon, fProofDiscon); - fSessionHierarchy->SetToolTipItem(item, "Proof Session"); - item ->SetUserData(newdesc); - // and in our session description list - fSessions->Add(newdesc); - // set actual description to the last one - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save actual configuration in config file "filename". - -void TSessionViewer::WriteConfiguration(const char *filename) -{ - TSessionDescription *session; - TQueryDescription *query; - Int_t scnt = 0, qcnt = 1; - const char *fname = filename ? filename : fConfigFile.Data(); - - delete fViewerEnv; - gSystem->Unlink(fname); - fViewerEnv = new TEnv(); - - fViewerEnv->SetValue("Option.Feedback", - (Int_t)fOptionsMenu->IsEntryChecked(kOptionsFeedback)); - fViewerEnv->SetValue("Option.MasterHistos", - (Int_t)fOptionsMenu->IsEntryChecked(kOptionsStatsHist)); - fViewerEnv->SetValue("Option.MasterEvents", - (Int_t)fOptionsMenu->IsEntryChecked(kOptionsStatsTrace)); - fViewerEnv->SetValue("Option.WorkerEvents", - (Int_t)fOptionsMenu->IsEntryChecked(kOptionsSlaveStatsTrace)); - - Int_t i = 0; - // browse list of feedback histos and check user's selected ones - while (kFeedbackHistos[i]) { - fViewerEnv->SetValue(Form("Option.%s",kFeedbackHistos[i]), - (Int_t)fCascadeMenu->IsEntryChecked(41+i)); - i++; - } - - TIter snext(fSessions); - while ((session = (TSessionDescription *) snext())) { - if ((scnt > 0) && ((session->fAddress.Length() < 3) || - session->fUserName.Length() < 2)) { - // skip gROOT's list of sessions - continue; - } - if ((scnt > 0) && (session->fName == session->fAddress)) { - // skip gROOT's list of proofs - continue; - } - TString sessionstring; - sessionstring += session->fName; - sessionstring += ";"; - sessionstring += session->fTag.Length() > 1 ? session->fTag.Data() : " "; - sessionstring += ";"; - sessionstring += session->fAddress; - sessionstring += ";"; - sessionstring += TString::Format("%d", session->fPort); - sessionstring += ";"; - sessionstring += TString::Format("%d", session->fLogLevel); - sessionstring += ";"; - sessionstring += session->fConfigFile.Length() > 1 ? session->fConfigFile.Data() : " "; - sessionstring += ";"; - sessionstring += session->fUserName; - sessionstring += ";"; - sessionstring += TString::Format("%d", session->fSync); - sessionstring += ";"; - sessionstring += TString::Format("%d", session->fAutoEnable); - if (scnt > 0) // skip local session - fViewerEnv->SetValue(Form("SessionDescription.%d",scnt), sessionstring); - scnt++; - - TIter qnext(session->fQueries); - while ((query = (TQueryDescription *) qnext())) { - TString querystring; - querystring += TString::Format("%d", query->fStatus); - querystring += ";"; - querystring += query->fReference.Length() > 1 ? query->fReference.Data() : " "; - querystring += ";"; - querystring += query->fQueryName; - querystring += ";"; - querystring += query->fSelectorString.Length() > 1 ? query->fSelectorString.Data() : " "; - querystring += ";"; - querystring += query->fTDSetString.Length() > 1 ? query->fTDSetString.Data() : " "; - querystring += ";"; - querystring += query->fOptions.Length() > 1 ? query->fOptions.Data() : " "; - querystring += ";"; - querystring += query->fEventList.Length() > 1 ? query->fEventList.Data() : " "; - querystring += ";"; - querystring += TString::Format("%d",query->fNbFiles); - querystring += ";"; - querystring += TString::Format("%lld",query->fNoEntries); - querystring += ";"; - querystring += TString::Format("%lld",query->fFirstEntry); - fViewerEnv->SetValue(Form("QueryDescription.%d",qcnt), querystring); - qcnt++; - } - } - - fViewerEnv->WriteFile(fname); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build main session viewer frame and subframes. - -void TSessionViewer::Build() -{ - TString buf; - fActDesc = 0; - fActFrame = 0; - fLogWindow = 0; - fBusy = kFALSE; - fAutoSave = kTRUE; - fChangePic = kFALSE; - fStart = fElapsed = 0; - - SetCleanup(kDeepCleanup); - // set minimun size - SetWMSizeHints(400 + 200, 370+50, 2000, 1000, 1, 1); - - // collect icons - fLocal = fClient->GetPicture("local_session.xpm"); - fProofCon = fClient->GetPicture("proof_connected.xpm"); - fProofDiscon = fClient->GetPicture("proof_disconnected.xpm"); - fQueryCon = fClient->GetPicture("query_connected.xpm"); - fQueryDiscon = fClient->GetPicture("query_disconnected.xpm"); - fBaseIcon = fClient->GetPicture("proof_base.xpm"); - - //--- File menu - fFileMenu = new TGPopupMenu(fClient->GetDefaultRoot()); - fFileMenu->AddEntry("&Load Config...", kFileLoadConfig); - fFileMenu->AddEntry("&Save Config...", kFileSaveConfig); - fFileMenu->AddSeparator(); - fFileMenu->AddEntry("&Close Viewer", kFileCloseViewer); - fFileMenu->AddSeparator(); - fFileMenu->AddEntry("&Quit ROOT", kFileQuit); - - //--- Session menu - fSessionMenu = new TGPopupMenu(gClient->GetDefaultRoot()); - fSessionMenu->AddLabel("Session Management"); - fSessionMenu->AddSeparator(); - fSessionMenu->AddEntry("&New Session", kSessionNew); - fSessionMenu->AddEntry("&Add to the list", kSessionAdd); - fSessionMenu->AddEntry("De&lete", kSessionDelete); - fSessionMenu->AddSeparator(); - fSessionMenu->AddEntry("&Connect...", kSessionConnect); - fSessionMenu->AddEntry("&Disconnect", kSessionDisconnect); - fSessionMenu->AddEntry("Shutdo&wn", kSessionShutdown); - fSessionMenu->AddEntry("&Show status",kSessionShowStatus); - fSessionMenu->AddEntry("&Get Queries",kSessionGetQueries); - fSessionMenu->AddSeparator(); - fSessionMenu->AddEntry("&Cleanup", kSessionCleanup); - fSessionMenu->AddEntry("&Reset",kSessionReset); - fSessionMenu->DisableEntry(kSessionAdd); - - //--- Query menu - fQueryMenu = new TGPopupMenu(gClient->GetDefaultRoot()); - fQueryMenu->AddLabel("Query Management"); - fQueryMenu->AddSeparator(); - fQueryMenu->AddEntry("&New...", kQueryNew); - fQueryMenu->AddEntry("&Edit", kQueryEdit); - fQueryMenu->AddEntry("&Submit", kQuerySubmit); - fQueryMenu->AddSeparator(); - fQueryMenu->AddEntry("Start &Viewer", kQueryStartViewer); - fQueryMenu->AddSeparator(); - fQueryMenu->AddEntry("&Delete", kQueryDelete); - - fViewerEnv = 0; -#ifdef WIN32 - fConfigFile = TString::Format("%s\\%s", gSystem->HomeDirectory(), kConfigFile); -#else - fConfigFile = TString::Format("%s/%s", gSystem->HomeDirectory(), kConfigFile); -#endif - - fCascadeMenu = new TGPopupMenu(fClient->GetDefaultRoot()); - Int_t i = 0; - while (kFeedbackHistos[i]) { - fCascadeMenu->AddEntry(kFeedbackHistos[i], 41+i); - i++; - } - fCascadeMenu->AddEntry("User defined...", 50); - // disable it for now (until implemented) - fCascadeMenu->DisableEntry(50); - - //--- Options menu - fOptionsMenu = new TGPopupMenu(fClient->GetDefaultRoot()); - fOptionsMenu->AddLabel("Global Options"); - fOptionsMenu->AddSeparator(); - fOptionsMenu->AddEntry("&Autosave Config", kOptionsAutoSave); - fOptionsMenu->AddSeparator(); - fOptionsMenu->AddEntry("Master &Histos", kOptionsStatsHist); - fOptionsMenu->AddEntry("&Master Events", kOptionsStatsTrace); - fOptionsMenu->AddEntry("&Worker Events", kOptionsSlaveStatsTrace); - fOptionsMenu->AddSeparator(); - fOptionsMenu->AddEntry("Feedback &Active", kOptionsFeedback); - fOptionsMenu->AddSeparator(); - fOptionsMenu->AddPopup("&Feedback Histos", fCascadeMenu); - fOptionsMenu->CheckEntry(kOptionsAutoSave); - - //--- Help menu - fHelpMenu = new TGPopupMenu(gClient->GetDefaultRoot()); - fHelpMenu->AddEntry("&About ROOT...", kHelpAbout); - - fFileMenu->Associate(this); - fSessionMenu->Associate(this); - fQueryMenu->Associate(this); - fOptionsMenu->Associate(this); - fCascadeMenu->Associate(this); - fHelpMenu->Associate(this); - - //--- create menubar and add popup menus - fMenuBar = new TGMenuBar(this, 1, 1, kHorizontalFrame); - - fMenuBar->AddPopup("&File", fFileMenu, new TGLayoutHints(kLHintsTop | - kLHintsLeft, 0, 4, 0, 0)); - fMenuBar->AddPopup("&Session", fSessionMenu, new TGLayoutHints(kLHintsTop | - kLHintsLeft, 0, 4, 0, 0)); - fMenuBar->AddPopup("&Query", fQueryMenu, new TGLayoutHints(kLHintsTop | - kLHintsLeft, 0, 4, 0, 0)); - fMenuBar->AddPopup("&Options", fOptionsMenu, new TGLayoutHints(kLHintsTop | - kLHintsLeft, 0, 4, 0, 0)); - fMenuBar->AddPopup("&Help", fHelpMenu, new TGLayoutHints(kLHintsTop | - kLHintsRight)); - - TGHorizontal3DLine *toolBarSep = new TGHorizontal3DLine(this); - AddFrame(toolBarSep, new TGLayoutHints(kLHintsTop | kLHintsExpandX)); - - AddFrame(fMenuBar, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 0, 0, 1, 1)); - - toolBarSep = new TGHorizontal3DLine(this); - AddFrame(toolBarSep, new TGLayoutHints(kLHintsTop | kLHintsExpandX)); - - //---- toolbar - - int spacing = 8; - fToolBar = new TGToolBar(this, 60, 20, kHorizontalFrame); - for (int ii = 0; xpm_toolbar[ii]; ii++) { - tb_data[ii].fPixmap = xpm_toolbar[ii]; - if (strlen(xpm_toolbar[ii]) == 0) { - spacing = 8; - continue; - } - fToolBar->AddButton(this, &tb_data[ii], spacing); - spacing = 0; - } - AddFrame(fToolBar, new TGLayoutHints(kLHintsTop | kLHintsExpandX, 0, 0, 0, 0)); - toolBarSep = new TGHorizontal3DLine(this); - AddFrame(toolBarSep, new TGLayoutHints(kLHintsTop | kLHintsExpandX)); - if (fToolBar->GetButton(kQuerySubmit)) - fToolBar->GetButton(kQuerySubmit)->SetState(kButtonDisabled); - - fPopupSrv = new TGPopupMenu(fClient->GetDefaultRoot()); - fPopupSrv->AddEntry("Connect",kSessionConnect); - fPopupSrv->AddEntry("Disconnect",kSessionDisconnect); - fPopupSrv->AddEntry("Shutdown",kSessionShutdown); - fPopupSrv->AddEntry("Browse",kSessionBrowse); - fPopupSrv->AddEntry("Show status",kSessionShowStatus); - fPopupSrv->AddEntry("Delete", kSessionDelete); - fPopupSrv->AddEntry("Get Queries",kSessionGetQueries); - fPopupSrv->AddSeparator(); - fPopupSrv->AddEntry("Cleanup", kSessionCleanup); - fPopupSrv->AddEntry("Reset",kSessionReset); - fPopupSrv->Connect("Activated(Int_t)","TSessionViewer", this, - "MyHandleMenu(Int_t)"); - - fPopupQry = new TGPopupMenu(fClient->GetDefaultRoot()); - fPopupQry->AddEntry("Edit",kQueryEdit); - fPopupQry->AddEntry("Submit",kQuerySubmit); - fPopupQry->AddSeparator(); - fPopupQry->AddEntry("Start &Viewer", kQueryStartViewer); - fPopupQry->AddSeparator(); - fPopupQry->AddEntry("Delete",kQueryDelete); - fPopupQry->Connect("Activated(Int_t)","TSessionViewer", this, - "MyHandleMenu(Int_t)"); - - - fSessionMenu->DisableEntry(kSessionGetQueries); - fSessionMenu->DisableEntry(kSessionShowStatus); - fPopupSrv->DisableEntry(kSessionGetQueries); - fPopupSrv->DisableEntry(kSessionShowStatus); - fPopupSrv->DisableEntry(kSessionDisconnect); - fPopupSrv->DisableEntry(kSessionShutdown); - fPopupSrv->DisableEntry(kSessionCleanup); - fPopupSrv->DisableEntry(kSessionReset); - fSessionMenu->DisableEntry(kSessionDisconnect); - fSessionMenu->DisableEntry(kSessionShutdown); - fSessionMenu->DisableEntry(kSessionCleanup); - fSessionMenu->DisableEntry(kSessionReset); - if (fToolBar->GetButton(kSessionDisconnect)) - fToolBar->GetButton(kSessionDisconnect)->SetState(kButtonDisabled); - - //--- Horizontal mother frame ----------------------------------------------- - fHf = new TGHorizontalFrame(this, 10, 10); - fHf->SetCleanup(kDeepCleanup); - - //--- fV1 ------------------------------------------------------------------- - fV1 = new TGVerticalFrame(fHf, 100, 100, kFixedWidth); - fV1->SetCleanup(kDeepCleanup); - - fTreeView = new TGCanvas(fV1, 100, 200, kSunkenFrame | kDoubleBorder); - fV1->AddFrame(fTreeView, new TGLayoutHints(kLHintsExpandX | kLHintsExpandY, - 2, 0, 0, 0)); - fSessionHierarchy = new TGListTree(fTreeView, kHorizontalFrame); - fSessionHierarchy->DisableOpen(); - fSessionHierarchy->Connect("Clicked(TGListTreeItem*,Int_t,Int_t,Int_t)", - "TSessionViewer", this, - "OnListTreeClicked(TGListTreeItem*, Int_t, Int_t, Int_t)"); - fSessionHierarchy->Connect("DoubleClicked(TGListTreeItem*,Int_t)", - "TSessionViewer", this, - "OnListTreeDoubleClicked(TGListTreeItem*, Int_t)"); - fV1->Resize(fTreeView->GetDefaultWidth()+100, fV1->GetDefaultHeight()); - - //--- fV2 ------------------------------------------------------------------- - fV2 = new TGVerticalFrame(fHf, 350, 310); - fV2->SetCleanup(kDeepCleanup); - - //--- Server Frame ---------------------------------------------------------- - fServerFrame = new TSessionServerFrame(fV2, 350, 310); - fSessions = new TList; - ReadConfiguration(); - fServerFrame->Build(this); - fV2->AddFrame(fServerFrame, new TGLayoutHints(kLHintsTop | kLHintsExpandX | - kLHintsExpandY, 2, 0, 1, 2)); - - //--- Session Frame --------------------------------------------------------- - fSessionFrame = new TSessionFrame(fV2, 350, 310); - fSessionFrame->Build(this); - fV2->AddFrame(fSessionFrame, new TGLayoutHints(kLHintsTop | kLHintsExpandX | - kLHintsExpandY, 2, 0, 1, 2)); - - //--- Query Frame ----------------------------------------------------------- - fQueryFrame = new TSessionQueryFrame(fV2, 350, 310); - fQueryFrame->Build(this); - fV2->AddFrame(fQueryFrame, new TGLayoutHints(kLHintsTop | kLHintsExpandX | - kLHintsExpandY, 2, 0, 1, 2)); - - //--- Output Frame ---------------------------------------------------------- - fOutputFrame = new TSessionOutputFrame(fV2, 350, 310); - fOutputFrame->Build(this); - fV2->AddFrame(fOutputFrame, new TGLayoutHints(kLHintsTop | kLHintsExpandX | - kLHintsExpandY, 2, 0, 1, 2)); - - //--- Input Frame ----------------------------------------------------------- - fInputFrame = new TSessionInputFrame(fV2, 350, 310); - fInputFrame->Build(this); - fV2->AddFrame(fInputFrame, new TGLayoutHints(kLHintsTop | kLHintsExpandX | - kLHintsExpandY, 2, 0, 1, 2)); - - fHf->AddFrame(fV1, new TGLayoutHints(kLHintsLeft | kLHintsExpandY)); - - // add vertical splitter between list tree and frames - TGVSplitter *splitter = new TGVSplitter(fHf, 4); - splitter->SetFrame(fV1, kTRUE); - fHf->AddFrame(splitter,new TGLayoutHints(kLHintsLeft | kLHintsExpandY)); - fHf->AddFrame(new TGVertical3DLine(fHf), new TGLayoutHints(kLHintsLeft | - kLHintsExpandY)); - - fHf->AddFrame(fV2, new TGLayoutHints(kLHintsRight | kLHintsExpandX | - kLHintsExpandY)); - - AddFrame(fHf, new TGLayoutHints(kLHintsRight | kLHintsExpandX | - kLHintsExpandY)); - - // if description available, update server infos frame - if (fActDesc) { - if (!fActDesc->fLocal) { - fServerFrame->Update(fActDesc); - } - else { - fServerFrame->SetAddEnabled(); - fServerFrame->SetConnectEnabled(kFALSE); - } - } - - //--- Status Bar ------------------------------------------------------------ - int parts[] = { 36, 49, 15 }; - fStatusBar = new TGStatusBar(this, 10, 10); - fStatusBar->SetCleanup(kDeepCleanup); - fStatusBar->SetParts(parts, 3); - for (int p = 0; p < 3; ++p) - fStatusBar->GetBarPart(p)->SetCleanup(kDeepCleanup); - AddFrame(fStatusBar, new TGLayoutHints(kLHintsTop | kLHintsLeft | - kLHintsExpandX, 0, 0, 1, 1)); - - // connection icon (animation) and time info - fStatusBar->SetText(" 00:00:00", 2); - TGCompositeFrame *leftpart = fStatusBar->GetBarPart(2); - fRightIconPicture = (TGPicture *)fClient->GetPicture("proof_disconnected.xpm"); - fRightIcon = new TGIcon(leftpart, fRightIconPicture, - fRightIconPicture->GetWidth(),fRightIconPicture->GetHeight()); - leftpart->AddFrame(fRightIcon, new TGLayoutHints(kLHintsLeft, 2, 0, 0, 0)); - - // connection progress bar - TGCompositeFrame *rightpart = fStatusBar->GetBarPart(0); - fConnectProg = new TGHProgressBar(rightpart, TGProgressBar::kStandard, 100); - fConnectProg->ShowPosition(); - fConnectProg->SetBarColor("green"); - rightpart->AddFrame(fConnectProg, new TGLayoutHints(kLHintsExpandX, 1, 1, 1, 1)); - - // add user info - fUserGroup = gSystem->GetUserInfo(); - buf.Form("User : %s - %s", fUserGroup->fRealName.Data(), - fUserGroup->fGroup.Data()); - fStatusBar->SetText(buf.Data(), 1); - - fTimer = 0; - - // create context menu - fContextMenu = new TContextMenu("SessionViewerContextMenu") ; - - SetWindowName("ROOT Session Viewer"); - MapSubwindows(); - MapWindow(); - - // hide frames - fServerFrame->SetAddEnabled(kFALSE); - fStatusBar->GetBarPart(0)->HideFrame(fConnectProg); - fV2->HideFrame(fSessionFrame); - fV2->HideFrame(fQueryFrame); - fV2->HideFrame(fOutputFrame); - fV2->HideFrame(fInputFrame); - fQueryFrame->GetQueryEditFrame()->OnNewQueryMore(); - fActFrame = fServerFrame; - UpdateListOfProofs(); - Resize(610, 420); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TSessionViewer::~TSessionViewer() -{ - delete fUserGroup; - if (gSessionViewer == this) - gSessionViewer = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle mouse clicks in list tree. - -void TSessionViewer::OnListTreeClicked(TGListTreeItem *entry, Int_t btn, - Int_t x, Int_t y) -{ - TList *objlist; - TObject *obj; - TString msg; - - fSessionMenu->DisableEntry(kSessionAdd); - if (fToolBar->GetButton(kQuerySubmit)) - fToolBar->GetButton(kQuerySubmit)->SetState(kButtonDisabled); - if (entry->GetParent() == 0) { // PROOF - // switch frames only if actual one doesn't match - if (fActFrame != fServerFrame) { - fV2->HideFrame(fActFrame); - fV2->ShowFrame(fServerFrame); - fActFrame = fServerFrame; - } - fSessionMenu->DisableEntry(kSessionDelete); - fSessionMenu->EnableEntry(kSessionAdd); - fServerFrame->SetAddEnabled(); - fServerFrame->SetConnectEnabled(kFALSE); - fPopupSrv->DisableEntry(kSessionConnect); - fSessionMenu->DisableEntry(kSessionConnect); - if (fToolBar->GetButton(kSessionConnect)) - fToolBar->GetButton(kSessionConnect)->SetState(kButtonDisabled); - } - else if (entry->GetParent()->GetParent() == 0) { // Server - if (entry->GetUserData()) { - obj = (TObject *)entry->GetUserData(); - if (!obj || obj->IsA() != TSessionDescription::Class()) - return; - // update server frame information - fServerFrame->Update((TSessionDescription *)obj); - fActDesc = (TSessionDescription*)obj; - // if Proof valid, update connection infos - if (fActDesc->fConnected && fActDesc->fAttached && - fActDesc->fProof && fActDesc->fProof->IsValid()) { - fActDesc->fProof->cd(); - msg.Form("PROOF Cluster %s ready", fActDesc->fName.Data()); - } - else { - msg.Form("PROOF Cluster %s not connected", fActDesc->fName.Data()); - } - fStatusBar->SetText(msg.Data(), 1); - } - if ((fActDesc->fConnected) && (fActDesc->fAttached)) { - fPopupSrv->DisableEntry(kSessionConnect); - fSessionMenu->DisableEntry(kSessionConnect); - if (fToolBar->GetButton(kSessionConnect)) - fToolBar->GetButton(kSessionConnect)->SetState(kButtonDisabled); - UpdateListOfPackages(); - fSessionFrame->UpdateListOfDataSets(); - } - else { - fPopupSrv->EnableEntry(kSessionConnect); - fSessionMenu->EnableEntry(kSessionConnect); - if (fToolBar->GetButton(kSessionConnect)) - fToolBar->GetButton(kSessionConnect)->SetState(kButtonUp); - } - // local session - if (fActDesc->fLocal) { - if (fActFrame != fSessionFrame) { - fV2->HideFrame(fActFrame); - fV2->ShowFrame(fSessionFrame); - fActFrame = fSessionFrame; - UpdateListOfPackages(); - fSessionFrame->UpdateListOfDataSets(); - } - fSessionFrame->SetLocal(); - fServerFrame->SetAddEnabled(); - fServerFrame->SetConnectEnabled(kFALSE); - } - // proof session not connected - if ((!fActDesc->fLocal) && (!fActDesc->fAttached) && - (fActFrame != fServerFrame)) { - fV2->HideFrame(fActFrame); - fV2->ShowFrame(fServerFrame); - fActFrame = fServerFrame; - } - // proof session connected - if ((!fActDesc->fLocal) && (fActDesc->fConnected) && - (fActDesc->fAttached)) { - if (fActFrame != fSessionFrame) { - fV2->HideFrame(fActFrame); - fV2->ShowFrame(fSessionFrame); - fActFrame = fSessionFrame; - } - fSessionFrame->SetLocal(kFALSE); - } - fSessionFrame->SetLogLevel(fActDesc->fLogLevel); - fServerFrame->SetLogLevel(fActDesc->fLogLevel); - if (fActDesc->fAutoEnable) - fSessionFrame->CheckAutoEnPack(kTRUE); - else - fSessionFrame->CheckAutoEnPack(kFALSE); - // update session information frame - fSessionFrame->ProofInfos(); - fSessionFrame->UpdatePackages(); - fServerFrame->SetAddEnabled(kFALSE); - fServerFrame->SetConnectEnabled(); - } - else if (entry->GetParent()->GetParent()->GetParent() == 0) { // query - obj = (TObject *)entry->GetParent()->GetUserData(); - if (obj && obj->IsA() == TSessionDescription::Class()) { - fActDesc = (TSessionDescription *)obj; - } - obj = (TObject *)entry->GetUserData(); - if (obj && obj->IsA() == TQueryDescription::Class()) { - fActDesc->fActQuery = (TQueryDescription *)obj; - } - // update query information and buttons state - fQueryFrame->UpdateInfos(); - fQueryFrame->UpdateButtons(fActDesc->fActQuery); - if (fActFrame != fQueryFrame) { - fV2->HideFrame(fActFrame); - fV2->ShowFrame(fQueryFrame); - fActFrame = fQueryFrame; - } - if ((fActDesc->fConnected) && (fActDesc->fAttached) && - (fActDesc->fActQuery->fStatus != TQueryDescription::kSessionQueryRunning) && - (fActDesc->fActQuery->fStatus != TQueryDescription::kSessionQuerySubmitted) && - (fToolBar->GetButton(kQuerySubmit)) ) - fToolBar->GetButton(kQuerySubmit)->SetState(kButtonUp); - // trick to update feedback histos - OnCascadeMenu(); - } - else { // a list (input, output) - obj = (TObject *)entry->GetParent()->GetParent()->GetUserData(); - if (obj && obj->IsA() == TSessionDescription::Class()) { - fActDesc = (TSessionDescription *)obj; - } - obj = (TObject *)entry->GetParent()->GetUserData(); - if (obj && obj->IsA() == TQueryDescription::Class()) { - fActDesc->fActQuery = (TQueryDescription *)obj; - } - if (fActDesc->fActQuery) { - // update input/output list views - fInputFrame->RemoveAll(); - fOutputFrame->RemoveAll(); - if (fActDesc->fActQuery->fResult) { - objlist = fActDesc->fActQuery->fResult->GetOutputList(); - if (objlist) { - TIter nexto(objlist); - while ((obj = (TObject *) nexto())) { - fOutputFrame->AddObject(obj); - } - } - objlist = fActDesc->fActQuery->fResult->GetInputList(); - if (objlist) { - TIter nexti(objlist); - while ((obj = (TObject *) nexti())) { - fInputFrame->AddObject(obj); - } - } - } - else { - TChain *chain = (TChain *)fActDesc->fActQuery->fChain; - if (chain && chain->GetPlayer()) { - objlist = chain->GetPlayer()->GetSelectorFromFile()->GetOutputList(); - if (objlist) { - TIter nexto(objlist); - while ((obj = (TObject *) nexto())) { - fOutputFrame->AddObject(obj); - } - } - } - } - fInputFrame->Resize(); - fOutputFrame->Resize(); - fClient->NeedRedraw(fOutputFrame->GetLVContainer()); - fClient->NeedRedraw(fInputFrame->GetLVContainer()); - } - // switch frames - if (strstr(entry->GetText(),"Output")) { - if (fActFrame != fOutputFrame) { - fV2->HideFrame(fActFrame); - fV2->ShowFrame(fOutputFrame); - fActFrame = fOutputFrame; - } - } - else if (strstr(entry->GetText(),"Input")) { - if (fActFrame != fInputFrame) { - fV2->HideFrame(fActFrame); - fV2->ShowFrame(fInputFrame); - fActFrame = fInputFrame; - } - } - } - if (btn == 3) { // right button - // place popup menus - TGListTreeItem *item = fSessionHierarchy->GetSelected(); - if (!item) return; - obj = (TObject *)item->GetUserData(); - if (obj && obj->IsA() == TQueryDescription::Class()) { - fPopupQry->PlaceMenu(x, y, 1, 1); - } - else if (obj && obj->IsA() == TSessionDescription::Class()) { - if (!fActDesc->fLocal) - fPopupSrv->PlaceMenu(x, y, 1, 1); - } - } - // enable / disable menu entries - if (fActDesc->fConnected && fActDesc->fAttached) { - fSessionMenu->EnableEntry(kSessionGetQueries); - fSessionMenu->EnableEntry(kSessionShowStatus); - fPopupSrv->EnableEntry(kSessionGetQueries); - fPopupSrv->EnableEntry(kSessionShowStatus); - fPopupSrv->EnableEntry(kSessionDisconnect); - fPopupSrv->EnableEntry(kSessionShutdown); - fPopupSrv->EnableEntry(kSessionCleanup); - fPopupSrv->EnableEntry(kSessionReset); - fSessionMenu->EnableEntry(kSessionDisconnect); - fSessionMenu->EnableEntry(kSessionShutdown); - fSessionMenu->EnableEntry(kSessionCleanup); - fSessionMenu->EnableEntry(kSessionReset); - if (fToolBar->GetButton(kSessionDisconnect)) - fToolBar->GetButton(kSessionDisconnect)->SetState(kButtonUp); - fQueryMenu->EnableEntry(kQuerySubmit); - fPopupQry->EnableEntry(kQuerySubmit); - } - else { - fSessionMenu->DisableEntry(kSessionGetQueries); - fSessionMenu->DisableEntry(kSessionShowStatus); - fPopupSrv->DisableEntry(kSessionGetQueries); - fPopupSrv->DisableEntry(kSessionShowStatus); - if (entry->GetParent() != 0) - fSessionMenu->EnableEntry(kSessionDelete); - fPopupSrv->EnableEntry(kSessionDelete); - fPopupSrv->DisableEntry(kSessionDisconnect); - fPopupSrv->DisableEntry(kSessionShutdown); - fPopupSrv->DisableEntry(kSessionCleanup); - fPopupSrv->DisableEntry(kSessionReset); - fSessionMenu->DisableEntry(kSessionDisconnect); - fSessionMenu->DisableEntry(kSessionShutdown); - fSessionMenu->DisableEntry(kSessionCleanup); - fSessionMenu->DisableEntry(kSessionReset); - if (fToolBar->GetButton(kSessionDisconnect)) - fToolBar->GetButton(kSessionDisconnect)->SetState(kButtonDisabled); - fQueryMenu->DisableEntry(kQuerySubmit); - fPopupQry->DisableEntry(kQuerySubmit); - } - if (fActDesc->fLocal) { - fSessionMenu->DisableEntry(kSessionDelete); - fSessionMenu->DisableEntry(kSessionConnect); - fSessionMenu->DisableEntry(kSessionDisconnect); - fSessionMenu->DisableEntry(kSessionShutdown); - fSessionMenu->DisableEntry(kSessionCleanup); - fSessionMenu->DisableEntry(kSessionReset); - if (fToolBar->GetButton(kSessionDisconnect)) - fToolBar->GetButton(kSessionDisconnect)->SetState(kButtonDisabled); - if (fToolBar->GetButton(kSessionConnect)) - fToolBar->GetButton(kSessionConnect)->SetState(kButtonDisabled); - fQueryMenu->EnableEntry(kQuerySubmit); - fPopupQry->EnableEntry(kQuerySubmit); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle mouse double clicks in list tree (connect to server). - -void TSessionViewer::OnListTreeDoubleClicked(TGListTreeItem *entry, Int_t /*btn*/) -{ - if (entry == fSessionItem) - return; - if (entry->GetParent()->GetParent() == 0) { // Server - if (entry->GetUserData()) { - TObject *obj = (TObject *)entry->GetUserData(); - if ((!obj) || (obj->IsA() != TSessionDescription::Class())) - return; - fActDesc = (TSessionDescription*)obj; - // if Proof valid, update connection infos - } - if ((!fActDesc->fLocal) && ((!fActDesc->fConnected) || - (!fActDesc->fAttached))) { - fServerFrame->OnBtnConnectClicked(); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Terminate Session : save configuration, clean temporary files and close -/// Proof connections. - -void TSessionViewer::Terminate() -{ - // clean-up temporary files - TString pathtmp; - pathtmp = TString::Format("%s/%s", gSystem->TempDirectory(), kSession_RedirectFile); - if (!gSystem->AccessPathName(pathtmp)) { - gSystem->Unlink(pathtmp); - } - pathtmp = TString::Format("%s/%s", gSystem->TempDirectory(), kSession_RedirectCmd); - if (!gSystem->AccessPathName(pathtmp)) { - gSystem->Unlink(pathtmp); - } - // close opened Proof sessions (if any) - TIter next(fSessions); - TSessionDescription *desc = 0; - while ((desc = (TSessionDescription *)next())) { - if (desc->fAttached && desc->fProof && - desc->fProof->IsValid()) - desc->fProof->Detach(); - } - // Save configuration - if (fAutoSave) - WriteConfiguration(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close main Session Viewer window. - -void TSessionViewer::CloseWindow() -{ - // clean-up temporary files - TString pathtmp; - pathtmp = TString::Format("%s/%s", gSystem->TempDirectory(), kSession_RedirectFile); - if (!gSystem->AccessPathName(pathtmp)) { - gSystem->Unlink(pathtmp); - } - pathtmp = TString::Format("%s/%s", gSystem->TempDirectory(), kSession_RedirectCmd); - if (!gSystem->AccessPathName(pathtmp)) { - gSystem->Unlink(pathtmp); - } - // Save configuration - if (fAutoSave) - WriteConfiguration(); - Cleanup(); - fSessions->Delete(); - if (fSessionItem) - fSessionHierarchy->DeleteChildren(fSessionItem); - delete fSessionHierarchy; // this has been put int TGCanvas which isn't a - // TGComposite frame and doesn't do cleanups. - fClient->FreePicture(fLocal); - fClient->FreePicture(fProofCon); - fClient->FreePicture(fProofDiscon); - fClient->FreePicture(fQueryCon); - fClient->FreePicture(fQueryDiscon); - fClient->FreePicture(fBaseIcon); - delete fTimer; - DeleteWindow(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Change the right logo (used for animation). - -void TSessionViewer::ChangeRightLogo(const char *name) -{ - fClient->FreePicture(fRightIconPicture); - fRightIconPicture = (TGPicture *)fClient->GetPicture(name); - fRightIcon->SetPicture(fRightIconPicture); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable animation timer. - -void TSessionViewer::EnableTimer() -{ - if (!fTimer) fTimer = new TTimer(this, 500); - fTimer->Reset(); - fTimer->TurnOn(); - time( &fStart ); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Disable animation timer. - -void TSessionViewer::DisableTimer() -{ - if (fTimer) - fTimer->TurnOff(); - ChangeRightLogo("proof_disconnected.xpm"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle animation timer. - -Bool_t TSessionViewer::HandleTimer(TTimer *) -{ - TString buf; - struct tm *connected; - Int_t count = gRandom->Integer(4); - if (count > 3) { - count = 0; - } - if (fChangePic) - ChangeRightLogo(xpm_names[count]); - time( &fElapsed ); - time_t elapsed_time = (time_t)difftime( fElapsed, fStart ); - connected = gmtime( &elapsed_time ); - if (connected) { - buf.Form(" %02d:%02d:%02d", connected->tm_hour, - connected->tm_min, connected->tm_sec); - fStatusBar->SetText(buf.Data(), 2); - } - else { - fStatusBar->SetText(" 00:00:00", 2); - } - - if (fActDesc->fLocal) { - if ((fActDesc->fActQuery) && - (fActDesc->fActQuery->fStatus == - TQueryDescription::kSessionQueryRunning)) { - TChain *chain = (TChain *)fActDesc->fActQuery->fChain; - if (chain) - fQueryFrame->ProgressLocal(chain->GetEntries(), - chain->GetReadEntry()+1); - } - } - - fTimer->Reset(); - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load/append a log msg in the log frame. - -void TSessionViewer::LogMessage(const char *msg, Bool_t all) -{ - if (fLogWindow) { - if (all) { - // load buffer - fLogWindow->LoadBuffer(msg); - } else { - // append - fLogWindow->AddBuffer(msg); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle signal "query result ready" coming from Proof session. - -void TSessionViewer::QueryResultReady(char *query) -{ - TString strtmp; - strtmp.Form("Query Result Ready for %s", query); - // show information on status bar - ShowInfo(strtmp.Data()); - TGListTreeItem *item=0, *item2=0; - TQueryDescription *lquery = 0; - // loop over actual queries to find which one is ready - - TIter nexts(fSessions); - TSessionDescription *desc = 0; - // check if session is already in the list - while ((desc = (TSessionDescription *)nexts())) { - if (desc && !desc->fAttached) - continue; - TIter nextp(desc->fQueries); - while ((lquery = (TQueryDescription *)nextp())) { - if (lquery->fReference.Contains(query)) { - // results are ready for this query - lquery->fResult = desc->fProof->GetQueryResult(query); - lquery->fStatus = TQueryDescription::kSessionQueryFromProof; - if (!lquery->fResult) - break; - // get query status - lquery->fStatus = lquery->fResult->IsFinalized() ? - TQueryDescription::kSessionQueryFinalized : - (TQueryDescription::ESessionQueryStatus)lquery->fResult->GetStatus(); - // get data set - TObject *o = lquery->fResult->GetInputObject("TDSet"); - if (o) - lquery->fChain = (TDSet *) o; - item = fSessionHierarchy->FindItemByObj(fSessionItem, desc); - if (item) { - item2 = fSessionHierarchy->FindItemByObj(item, lquery); - } - if (item2) { - // add input and output list entries - if (lquery->fResult->GetInputList()) - if (!fSessionHierarchy->FindChildByName(item2, "InputList")) - fSessionHierarchy->AddItem(item2, "InputList"); - if (lquery->fResult->GetOutputList()) - if (!fSessionHierarchy->FindChildByName(item2, "OutputList")) - fSessionHierarchy->AddItem(item2, "OutputList"); - } - // update list tree, query frame information, and buttons state - fSessionHierarchy->ClearViewPort(); - fClient->NeedRedraw(fSessionHierarchy); - fQueryFrame->UpdateInfos(); - fQueryFrame->UpdateButtons(lquery); - break; - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clean-up Proof session. - -void TSessionViewer::CleanupSession() -{ - TGListTreeItem *item = fSessionHierarchy->GetSelected(); - if (!item) return; - TObject *obj = (TObject *)item->GetUserData(); - if (!obj || obj->IsA() != TSessionDescription::Class()) return; - if (!fActDesc->fProof || !fActDesc->fProof->IsValid()) return; - TString m; - m.Form("Are you sure to cleanup the session \"%s::%s\"", - fActDesc->fName.Data(), fActDesc->fTag.Data()); - Int_t result; - new TGMsgBox(fClient->GetRoot(), this, "", m.Data(), 0, - kMBYes | kMBNo | kMBCancel, &result); - if (result == kMBYes) { - // send cleanup request for the session specified by the tag reference - TString sessiontag; - sessiontag.Form("session-%s",fActDesc->fTag.Data()); - fActDesc->fProof->CleanupSession(sessiontag.Data()); - // clear the list of queries - fActDesc->fQueries->Clear(); - fSessionHierarchy->DeleteChildren(item); - fSessionFrame->OnBtnGetQueriesClicked(); - if (fAutoSave) - WriteConfiguration(); - } - // update list tree - fSessionHierarchy->ClearViewPort(); - fClient->NeedRedraw(fSessionHierarchy); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset Proof session. - -void TSessionViewer::ResetSession() -{ - TGListTreeItem *item = fSessionHierarchy->GetSelected(); - if (!item) return; - TObject *obj = (TObject *)item->GetUserData(); - if (!obj || obj->IsA() != TSessionDescription::Class()) return; - if (!fActDesc->fProof || !fActDesc->fProof->IsValid()) return; - TString m; - m.Form("Do you really want to reset the session \"%s::%s\"", - fActDesc->fName.Data(), fActDesc->fAddress.Data()); - Int_t result; - new TGMsgBox(fClient->GetRoot(), this, "", m.Data(), 0, - kMBYes | kMBNo | kMBCancel, &result); - if (result == kMBYes) { - // reset the session - TProofMgr *mgr = TProof::Mgr(fActDesc->fAddress); - if (mgr && mgr->IsValid()) { - mgr->Reset(fActDesc->fUserName); - } - // reset connected flag - fActDesc->fAttached = kFALSE; - fActDesc->fProof = 0; - // disable animation timer - DisableTimer(); - // change list tree item picture to disconnected pixmap - TGListTreeItem *item2 = fSessionHierarchy->FindChildByData( - fSessionItem, fActDesc); - if (item2) item2->SetPictures(fProofDiscon, fProofDiscon); - - OnListTreeClicked(fSessionHierarchy->GetSelected(), 1, 0, 0); - fSessionHierarchy->ClearViewPort(); - fClient->NeedRedraw(fSessionHierarchy); - fStatusBar->SetText("", 1); - } - // update list tree - fSessionHierarchy->ClearViewPort(); - fClient->NeedRedraw(fSessionHierarchy); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete query from list tree and ask user if they want to delete it also -/// from server. - -void TSessionViewer::DeleteQuery() -{ - TGListTreeItem *item = fSessionHierarchy->GetSelected(); - if (!item) return; - TObject *obj = (TObject *)item->GetUserData(); - if (!obj || obj->IsA() != TQueryDescription::Class()) return; - TQueryDescription *query = (TQueryDescription *)obj; - TString m; - Int_t result = 0; - - if (fActDesc->fAttached && fActDesc->fProof && fActDesc->fProof->IsValid()) { - if ((fActDesc->fActQuery->fStatus == TQueryDescription::kSessionQuerySubmitted) || - (fActDesc->fActQuery->fStatus == TQueryDescription::kSessionQueryRunning) ) { - new TGMsgBox(fClient->GetRoot(), this, "Delete Query", - "Deleting running queries is not allowed", kMBIconExclamation, - kMBOk, &result); - return; - } - m.Form("Do you want to delete query \"%s\" from server too ?", - query->fQueryName.Data()); - new TGMsgBox(fClient->GetRoot(), this, "", m.Data(), kMBIconQuestion, - kMBYes | kMBNo | kMBCancel, &result); - } - else { - m.Form("Dou you really want to delete query \"%s\" ?", - query->fQueryName.Data()); - new TGMsgBox(fClient->GetRoot(), this, "", m.Data(), kMBIconQuestion, - kMBOk | kMBCancel, &result); - } - if (result == kMBYes) { - fActDesc->fProof->Remove(query->fReference.Data()); - fActDesc->fQueries->Remove((TObject *)query); - fSessionHierarchy->DeleteItem(item); - delete query; - } - else if (result == kMBNo || result == kMBOk) { - fActDesc->fQueries->Remove((TObject *)query); - fSessionHierarchy->DeleteItem(item); - delete query; - } - fSessionHierarchy->ClearViewPort(); - fClient->NeedRedraw(fSessionHierarchy); - if (fAutoSave) - WriteConfiguration(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Edit currently selected query. - -void TSessionViewer::EditQuery() -{ - TGListTreeItem *item = fSessionHierarchy->GetSelected(); - if (!item) return; - TObject *obj = (TObject *)item->GetUserData(); - if (!obj || obj->IsA() != TQueryDescription::Class()) return; - TQueryDescription *query = (TQueryDescription *)obj; - TNewQueryDlg *dlg = new TNewQueryDlg(this, 350, 310, query, kTRUE); - dlg->Popup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Start TreeViewer from selected TChain. - -void TSessionViewer::StartViewer() -{ - TGListTreeItem *item = fSessionHierarchy->GetSelected(); - if (!item) return; - TObject *obj = (TObject *)item->GetUserData(); - if (!obj || obj->IsA() != TQueryDescription::Class()) return; - TQueryDescription *query = (TQueryDescription *)obj; - if (!query->fChain && query->fResult && - (obj = query->fResult->GetInputObject("TDSet"))) { - query->fChain = (TDSet *) obj; - } - if (!query->fChain) return; - if (query->fChain->IsA() == TChain::Class()) - ((TChain *)query->fChain)->StartViewer(); - else if (query->fChain->IsA() == TDSet::Class()) - ((TDSet *)query->fChain)->StartViewer(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Query the list of uploaded packages from proof and display it -/// into a new text window. - -void TSessionViewer::ShowPackages() -{ - Window_t wdummy; - Int_t ax, ay; - - if (fActDesc->fLocal) return; - if (!fActDesc->fProof || !fActDesc->fProof->IsValid()) - return; - TString pathtmp = TString::Format("%s/%s", gSystem->TempDirectory(), - kSession_RedirectFile); - // redirect stdout/stderr to temp file - if (gSystem->RedirectOutput(pathtmp.Data(), "w") != 0) { - Error("ShowStatus", "stdout/stderr redirection failed; skipping"); - return; - } - fActDesc->fProof->ShowPackages(kTRUE); - // restore stdout/stderr - if (gSystem->RedirectOutput(0) != 0) { - Error("ShowStatus", "stdout/stderr retore failed; skipping"); - return; - } - if (!fLogWindow) { - fLogWindow = new TSessionLogView(this, 700, 100); - } else { - // Clear window - fLogWindow->Clear(); - } - fLogWindow->LoadFile(pathtmp.Data()); - gVirtualX->TranslateCoordinates(GetId(), fClient->GetDefaultRoot()->GetId(), - 0, 0, ax, ay, wdummy); - fLogWindow->Move(ax, ay + GetHeight() + 35); - fLogWindow->Popup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update the list of packages. - -void TSessionViewer::UpdateListOfPackages() -{ - TObjString *packname; - TPackageDescription *package; - if (fActDesc->fConnected && fActDesc->fAttached && - fActDesc->fProof && fActDesc->fProof->IsValid() && - fActDesc->fProof->IsParallel()) { - //fActDesc->fPackages->Clear(); - TList *packlist = fActDesc->fProof->GetListOfEnabledPackages(); - if(packlist) { - TIter nextenabled(packlist); - while ((packname = (TObjString *)nextenabled())) { - package = new TPackageDescription; - package->fName = packname->GetName(); - package->fName += ".par"; - package->fPathName = package->fName; - package->fId = fActDesc->fPackages->GetEntries(); - package->fUploaded = kTRUE; - package->fEnabled = kTRUE; - if (!fActDesc->fPackages->FindObject(package->fName)) { - fActDesc->fPackages->Add((TObject *)package); - } - } - } - packlist = fActDesc->fProof->GetListOfPackages(); - if(packlist) { - TIter nextpack(packlist); - while ((packname = (TObjString *)nextpack())) { - package = new TPackageDescription; - package->fName = packname->GetName(); - package->fName += ".par"; - package->fPathName = package->fName; - package->fId = fActDesc->fPackages->GetEntries(); - package->fUploaded = kTRUE; - package->fEnabled = kFALSE; - if (!fActDesc->fPackages->FindObject(package->fName)) { - fActDesc->fPackages->Add((TObject *)package); - } - } - } - } -// fSessionFrame->UpdatePackages(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Query list of enabled packages from proof and display it -/// into a new text window. - -void TSessionViewer::ShowEnabledPackages() -{ - Window_t wdummy; - Int_t ax, ay; - - if (fActDesc->fLocal) return; - if (!fActDesc->fProof || !fActDesc->fProof->IsValid()) - return; - TString pathtmp = TString::Format("%s/%s", gSystem->TempDirectory(), - kSession_RedirectFile); - // redirect stdout/stderr to temp file - if (gSystem->RedirectOutput(pathtmp.Data(), "w") != 0) { - Error("ShowStatus", "stdout/stderr redirection failed; skipping"); - return; - } - fActDesc->fProof->ShowEnabledPackages(kTRUE); - // restore stdout/stderr - if (gSystem->RedirectOutput(0) != 0) { - Error("ShowStatus", "stdout/stderr retore failed; skipping"); - return; - } - if (!fLogWindow) { - fLogWindow = new TSessionLogView(this, 700, 100); - } else { - // Clear window - fLogWindow->Clear(); - } - fLogWindow->LoadFile(pathtmp.Data()); - gVirtualX->TranslateCoordinates(GetId(), fClient->GetDefaultRoot()->GetId(), - 0, 0, ax, ay, wdummy); - fLogWindow->Move(ax, ay + GetHeight() + 35); - fLogWindow->Popup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display the content of the temporary log file for queryref -/// into a new text window. - -void TSessionViewer::ShowLog(const char *queryref) -{ - Window_t wdummy; - Int_t ax, ay; - - if (fActDesc->fProof) { - gVirtualX->SetCursor(GetId(),gVirtualX->CreateCursor(kWatch)); - if (!fLogWindow) { - fLogWindow = new TSessionLogView(this, 700, 100); - } else { - // Clear window - fLogWindow->Clear(); - } - fActDesc->fProof->Connect("LogMessage(const char*,Bool_t)", - "TSessionViewer", this, "LogMessage(const char*,Bool_t)"); - Bool_t logonly = fActDesc->fProof->SendingLogToWindow(); - fActDesc->fProof->SendLogToWindow(kTRUE); - if (queryref) - fActDesc->fProof->ShowLog(queryref); - else - fActDesc->fProof->ShowLog(0); - fActDesc->fProof->SendLogToWindow(logonly); - // set log window position at the bottom of Session Viewer - gVirtualX->TranslateCoordinates(GetId(), - fClient->GetDefaultRoot()->GetId(), 0, 0, ax, ay, wdummy); - fLogWindow->Move(ax, ay + GetHeight() + 35); - fLogWindow->Popup(); - gVirtualX->SetCursor(GetId(), 0); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display text in status bar. - -void TSessionViewer::ShowInfo(const char *txt) -{ - fStatusBar->SetText(txt,0); - fClient->NeedRedraw(fStatusBar); - gSystem->ProcessEvents(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Retrieve and display Proof status. - -void TSessionViewer::ShowStatus() -{ - Window_t wdummy; - Int_t ax, ay; - - if (!fActDesc->fProof || !fActDesc->fProof->IsValid()) - return; - TString pathtmp = TString::Format("%s/%s", gSystem->TempDirectory(), - kSession_RedirectFile); - // redirect stdout/stderr to temp file - if (gSystem->RedirectOutput(pathtmp.Data(), "w") != 0) { - Error("ShowStatus", "stdout/stderr redirection failed; skipping"); - return; - } - fActDesc->fProof->GetStatus(); - // restore stdout/stderr - if (gSystem->RedirectOutput(0) != 0) { - Error("ShowStatus", "stdout/stderr retore failed; skipping"); - return; - } - if (!fLogWindow) { - fLogWindow = new TSessionLogView(this, 700, 100); - } else { - // Clear window - fLogWindow->Clear(); - } - fLogWindow->LoadFile(pathtmp.Data()); - gVirtualX->TranslateCoordinates(GetId(), fClient->GetDefaultRoot()->GetId(), - 0, 0, ax, ay, wdummy); - fLogWindow->Move(ax, ay + GetHeight() + 35); - fLogWindow->Popup(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle startup message (connection progress) coming from Proof session. - -void TSessionViewer::StartupMessage(char *msg, Bool_t, Int_t done, Int_t total) -{ - Float_t pos = Float_t(Double_t(done * 100)/Double_t(total)); - fConnectProg->SetPosition(pos); - fStatusBar->SetText(msg, 1); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle session viewer custom popup menus. - -void TSessionViewer::MyHandleMenu(Int_t id) -{ - switch (id) { - - case kSessionDelete: - fServerFrame->OnBtnDeleteClicked(); - break; - - case kSessionConnect: - fServerFrame->OnBtnConnectClicked(); - break; - - case kSessionDisconnect: - fSessionFrame->OnBtnDisconnectClicked(); - break; - - case kSessionShutdown: - fSessionFrame->ShutdownSession(); - break; - - case kSessionCleanup: - CleanupSession(); - break; - - case kSessionReset: - ResetSession(); - break; - - case kSessionBrowse: - if (fActDesc->fProof && fActDesc->fProof->IsValid()) { - TBrowser *b = new TBrowser(); - fActDesc->fProof->Browse(b); - } - break; - - case kSessionShowStatus: - ShowStatus(); - break; - - case kSessionGetQueries: - fSessionFrame->OnBtnGetQueriesClicked(); - break; - - case kQueryEdit: - EditQuery(); - break; - - case kQueryDelete: - DeleteQuery(); - break; - - case kQueryStartViewer: - StartViewer(); - break; - - case kQuerySubmit: - fQueryFrame->OnBtnSubmit(); - break; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle feedback histograms configuration menu. - -void TSessionViewer::OnCascadeMenu() -{ - // divide stats canvas by number of selected feedback histos - fQueryFrame->GetStatsCanvas()->cd(); - fQueryFrame->GetStatsCanvas()->Clear(); - fQueryFrame->GetStatsCanvas()->Modified(); - fQueryFrame->GetStatsCanvas()->Update(); - if (!fActDesc || !fActDesc->fActQuery) return; - fActDesc->fNbHistos = 0; - Int_t i = 0; - - if (fActDesc->fAttached && fActDesc->fProof && - fActDesc->fProof->IsValid()) { - if (fOptionsMenu->IsEntryChecked(kOptionsFeedback)) { - // browse list of feedback histos and check user's selected ones - while (kFeedbackHistos[i]) { - if (fCascadeMenu->IsEntryChecked(41+i)) { - fActDesc->fProof->AddFeedback(kFeedbackHistos[i]); - } - i++; - } - } - else { - // if feedback option not selected, clear Proof's feedback option - fActDesc->fProof->ClearFeedback(); - } - } - - i = 0; - // loop over feedback histo list - while (kFeedbackHistos[i]) { - // check if user has selected this histogram in the option menu - if (fCascadeMenu->IsEntryChecked(41+i)) - fActDesc->fNbHistos++; - i++; - } - fQueryFrame->GetStatsCanvas()->SetEditable(kTRUE); - fQueryFrame->GetStatsCanvas()->Clear(); - if (fActDesc->fNbHistos == 4) - fQueryFrame->GetStatsCanvas()->Divide(2, 2); - else if (fActDesc->fNbHistos > 4) - fQueryFrame->GetStatsCanvas()->Divide(3, 2); - else - fQueryFrame->GetStatsCanvas()->Divide(fActDesc->fNbHistos, 1); - - // if actual query has results, update feedback histos - if (fActDesc->fActQuery && fActDesc->fActQuery->fResult && - fActDesc->fActQuery->fResult->GetOutputList()) { - fQueryFrame->UpdateHistos(fActDesc->fActQuery->fResult->GetOutputList()); - fQueryFrame->ResetProgressDialog("", 0, 0, 0); - } - else if (fActDesc->fActQuery) { - fQueryFrame->ResetProgressDialog(fActDesc->fActQuery->fSelectorString, - fActDesc->fActQuery->fNbFiles, - fActDesc->fActQuery->fFirstEntry, - fActDesc->fActQuery->fNoEntries); - } - fQueryFrame->UpdateInfos(); -} -//////////////////////////////////////////////////////////////////////////////// -/// Handle messages send to the TSessionViewer object. E.g. all menu entries -/// messages. - -Bool_t TSessionViewer::ProcessMessage(Longptr_t msg, Longptr_t parm1, Longptr_t) -{ - TNewQueryDlg *dlg; - - switch (GET_MSG(msg)) { - case kC_COMMAND: - switch (GET_SUBMSG(msg)) { - case kCM_BUTTON: - case kCM_MENU: - switch (parm1) { - - case kFileCloseViewer: - CloseWindow(); - break; - - case kFileLoadConfig: - { - TGFileInfo fi; - fi.SetFilename(gSystem->BaseName(fConfigFile)); - fi.SetIniDir(gSystem->HomeDirectory()); - fi.fFileTypes = conftypes; - new TGFileDialog(fClient->GetRoot(), this, kFDOpen, &fi); - if (fi.fFilename) { - fConfigFile = fi.fFilename; - ReadConfiguration(fConfigFile); - OnListTreeClicked(fSessionHierarchy->GetSelected(), 1, 0, 0); - } - } - break; - - case kFileSaveConfig: - { - TGFileInfo fi; - fi.SetFilename(gSystem->BaseName(fConfigFile)); - fi.SetIniDir(gSystem->HomeDirectory()); - fi.fFileTypes = conftypes; - new TGFileDialog(fClient->GetRoot(), this, kFDSave, &fi); - if (fi.fFilename) { - fConfigFile = fi.fFilename; - WriteConfiguration(fConfigFile); - } - } - break; - - case kFileQuit: - Terminate(); - if (!gApplication->ReturnFromRun()) - delete this; - gApplication->Terminate(0); - break; - - case kSessionNew: - fServerFrame->OnBtnNewServerClicked(); - break; - - case kSessionAdd: - fServerFrame->OnBtnAddClicked(); - break; - - case kSessionDelete: - fServerFrame->OnBtnDeleteClicked(); - break; - - case kSessionCleanup: - CleanupSession(); - break; - - case kSessionReset: - ResetSession(); - break; - - case kSessionConnect: - fServerFrame->OnBtnConnectClicked(); - break; - - case kSessionDisconnect: - fSessionFrame->OnBtnDisconnectClicked(); - break; - - case kSessionShutdown: - fSessionFrame->ShutdownSession(); - break; - - case kSessionShowStatus: - ShowStatus(); - break; - - case kSessionGetQueries: - fSessionFrame->OnBtnGetQueriesClicked(); - break; - - case kQueryNew: - dlg = new TNewQueryDlg(this, 350, 310); - dlg->Popup(); - break; - - case kQueryEdit: - EditQuery(); - break; - - case kQueryDelete: - DeleteQuery(); - break; - - case kQueryStartViewer: - StartViewer(); - break; - - case kQuerySubmit: - fQueryFrame->OnBtnSubmit(); - break; - - case kOptionsAutoSave: - if(fOptionsMenu->IsEntryChecked(kOptionsAutoSave)) { - fOptionsMenu->UnCheckEntry(kOptionsAutoSave); - fAutoSave = kFALSE; - } - else { - fOptionsMenu->CheckEntry(kOptionsAutoSave); - fAutoSave = kTRUE; - } - break; - - case kOptionsStatsHist: - if(fOptionsMenu->IsEntryChecked(kOptionsStatsHist)) { - fOptionsMenu->UnCheckEntry(kOptionsStatsHist); - gEnv->SetValue("Proof.StatsHist", 0); - } - else { - fOptionsMenu->CheckEntry(kOptionsStatsHist); - gEnv->SetValue("Proof.StatsHist", 1); - } - break; - - case kOptionsStatsTrace: - if(fOptionsMenu->IsEntryChecked(kOptionsStatsTrace)) { - fOptionsMenu->UnCheckEntry(kOptionsStatsTrace); - gEnv->SetValue("Proof.StatsTrace", 0); - } - else { - fOptionsMenu->CheckEntry(kOptionsStatsTrace); - gEnv->SetValue("Proof.StatsTrace", 1); - } - break; - - case kOptionsSlaveStatsTrace: - if(fOptionsMenu->IsEntryChecked(kOptionsSlaveStatsTrace)) { - fOptionsMenu->UnCheckEntry(kOptionsSlaveStatsTrace); - gEnv->SetValue("Proof.SlaveStatsTrace", 0); - } - else { - fOptionsMenu->CheckEntry(kOptionsSlaveStatsTrace); - gEnv->SetValue("Proof.SlaveStatsTrace", 1); - } - break; - - case kOptionsFeedback: - if(fOptionsMenu->IsEntryChecked(kOptionsFeedback)) { - fOptionsMenu->UnCheckEntry(kOptionsFeedback); - } - else { - fOptionsMenu->CheckEntry(kOptionsFeedback); - } - break; - - case 41: - case 42: - case 43: - case 44: - case 45: - case 46: - if (fCascadeMenu->IsEntryChecked(parm1)) { - fCascadeMenu->UnCheckEntry(parm1); - } - else { - fCascadeMenu->CheckEntry(parm1); - } - OnCascadeMenu(); - break; - - case 50: - if (fCascadeMenu->IsEntryChecked(parm1)) { - fCascadeMenu->UnCheckEntry(parm1); - } - else { - fCascadeMenu->CheckEntry(parm1); - } - OnCascadeMenu(); - break; - - case kHelpAbout: - { -#ifdef WIN32 - new TWin32SplashThread(kTRUE); -#else - char str[32]; - snprintf(str, 32, "About ROOT %s...", gROOT->GetVersion()); - TRootHelpDialog *hd = new TRootHelpDialog(this, str, 600, 400); - hd->SetText(gHelpAbout); - hd->Popup(); -#endif - } - break; - - default: - break; - } - default: - break; - } - default: - break; - } - - return kTRUE; -} diff --git a/icons/proof_base.xpm b/icons/proof_base.xpm deleted file mode 100644 index 59d639e69d363..0000000000000 --- a/icons/proof_base.xpm +++ /dev/null @@ -1,279 +0,0 @@ -/* XPM */ -static char *proof_base_xpm[] = { -/* width height num_colors chars_per_pixel */ -" 16 16 256 2", -/* colors */ -"`` c #000000", -"`. c #ffffff", -"`# c #5151b5", -"`a c #6a6ac7", -"`b c #595983", -"`c c #9494cd", -"`d c #8d8dc4", -"`e c #9c9cd5", -"`f c #9d9dcb", -"`g c #9898c4", -"`h c #7d7da1", -"`i c #a4a4ce", -"`j c #8787a9", -"`k c #626277", -"`l c #b1b1d1", -"`m c #6e6e82", -"`n c #b6b6d6", -"`o c #aeaecc", -"`p c #aaaac7", -"`q c #808091", -"`r c #8a8a9c", -"`s c #bdbdd0", -"`t c #a5a5b6", -"`u c #c7c7d8", -"`v c #535359", -"`w c #c2c2cd", -"`x c #cfcfd9", -"`y c #c7c7d0", -"`z c #e1e1eb", -"`A c #cdcdd6", -"`B c #4c4c4f", -"`C c #d7d7de", -"`D c #808084", -"`E c #d2d2d6", -"`F c #6a6a6c", -"`G c #757576", -"`H c #5c5c5d", -"`I c #d5d5d6", -"`J c #acacad", -"`K c #666796", -"`L c #8284b8", -"`M c #9899ba", -"`N c #b6b7d2", -"`O c #8e90b9", -"`P c #a9abd3", -"`Q c #7479b8", -"`R c #696da2", -"`S c #a1a3c1", -"`T c #9497aa", -"`U c #d2d4df", -"`V c #6472b5", -"`W c #99a3d2", -"`X c #8d9dd9", -"`Y c #e5e7ee", -"`Z c #6781d3", -"`0 c #b8bcc9", -"`1 c #555965", -"`2 c #809bce", -"`3 c #b2bfd6", -"`4 c #5987cf", -"`5 c #749bdd", -"`6 c #83a2d6", -"`7 c #9ca3ad", -"`8 c #87acde", -".` c #8ca7c8", -".. c #99bfe9", -".# c #a8c8ed", -".a c #a4bdd8", -".b c #e1f0ff", -".c c #9d9e9f", -".d c #dceeff", -".e c #45a1ee", -".f c #6db0e7", -".g c #58b6fa", -".h c #77c5fc", -".i c #73bcef", -".j c #21a8fe", -".k c #30b1ff", -".l c #43b4fc", -".m c #84cbfa", -".n c #92bdd9", -".o c #42baff", -".p c #4cbeff", -".q c #56c4ff", -".r c #5dc8ff", -".s c #65cbff", -".t c #8fb4c8", -".u c #6ed0ff", -".v c #74d4ff", -".w c #94c5dc", -".x c #82dbff", -".y c #88ddff", -".z c #93e2ff", -".A c #9ae6ff", -".B c #9fe8ff", -".C c #a0dff3", -".D c #b1e4f5", -".E c #abcbd6", -".F c #b8bbbc", -".G c #a4ecff", -".H c #abefff", -".I c #b3f2ff", -".J c #c0f4ff", -".K c #b4d8de", -".L c #c6fcff", -".M c #e2ffff", -".N c None s None", -".O c #f9eed5", -".P c #e8b145", -".Q c #fbd4a2", -".R c #b4752e", -".S c #efd7c6", -".T c #f9f9f9", -".U c #efefef", -".V c #ececec", -".W c #e9e9e9", -".X c #e6e6e6", -".Y c #e4e4e4", -".Z c #e1e1e1", -".0 c #d0d0d0", -".1 c #c9c9c9", -".2 c #999999", -".3 c #949494", -".4 c #8f8f8f", -".5 c #888888", -".6 c #7c7c7c", -".7 c #737373", -".8 c #707070", -"#` c #646464", -"#. c #262626", -"## c #000000", -"#a c #000000", -"#b c #000000", -"#c c #000000", -"#d c #000000", -"#e c #000000", -"#f c #000000", -"#g c #000000", -"#h c #000000", -"#i c #000000", -"#j c #000000", -"#k c #000000", -"#l c #000000", -"#m c #000000", -"#n c #000000", -"#o c #000000", -"#p c #000000", -"#q c #000000", -"#r c #000000", -"#s c #000000", -"#t c #000000", -"#u c #000000", -"#v c #000000", -"#w c #000000", -"#x c #000000", -"#y c #000000", -"#z c #000000", -"#A c #000000", -"#B c #000000", -"#C c #000000", -"#D c #000000", -"#E c #000000", -"#F c #000000", -"#G c #000000", -"#H c #000000", -"#I c #000000", -"#J c #000000", -"#K c #000000", -"#L c #000000", -"#M c #000000", -"#N c #000000", -"#O c #000000", -"#P c #000000", -"#Q c #000000", -"#R c #000000", -"#S c #000000", -"#T c #000000", -"#U c #000000", -"#V c #000000", -"#W c #000000", -"#X c #000000", -"#Y c #000000", -"#Z c #000000", -"#0 c #000000", -"#1 c #000000", -"#2 c #000000", -"#3 c #000000", -"#4 c #000000", -"#5 c #000000", -"#6 c #000000", -"#7 c #000000", -"#8 c #000000", -"a` c #000000", -"a. c #000000", -"a# c #000000", -"aa c #000000", -"ab c #000000", -"ac c #000000", -"ad c #000000", -"ae c #000000", -"af c #000000", -"ag c #000000", -"ah c #000000", -"ai c #000000", -"aj c #000000", -"ak c #000000", -"al c #000000", -"am c #000000", -"an c #000000", -"ao c #000000", -"ap c #000000", -"aq c #000000", -"ar c #000000", -"as c #000000", -"at c #000000", -"au c #000000", -"av c #000000", -"aw c #000000", -"ax c #000000", -"ay c #000000", -"az c #000000", -"aA c #000000", -"aB c #000000", -"aC c #000000", -"aD c #000000", -"aE c #000000", -"aF c #000000", -"aG c #000000", -"aH c #000000", -"aI c #000000", -"aJ c #000000", -"aK c #000000", -"aL c #000000", -"aM c #000000", -"aN c #000000", -"aO c #000000", -"aP c #000000", -"aQ c #000000", -"aR c #000000", -"aS c #000000", -"aT c #000000", -"aU c #000000", -"aV c #000000", -"aW c #000000", -"aX c #000000", -"aY c #000000", -"aZ c #000000", -"a0 c #000000", -"a1 c #000000", -"a2 c #000000", -"a3 c #000000", -"a4 c #000000", -"a5 c #000000", -"a6 c #000000", -"a7 c #000000", -"a8 c #000000", -/* pixels */ -".N.N.N.N.N`i`i`i`i`i`6.N.N.N.N.N", -".N.N.N.N.N`i.a.C.H.G.f.3.N.N.N.N", -".N.N.N.N.N`i.a.x.z.y`5.8.N.N.N.N", -".N.N.N.N.N`i.n.p.r.r`Z.8.N.N.N.N", -".N.N.N.N.N`N`5.e.f`5`b.6.N.N.N.N", -".N.N.N.N.N.N.8.F#.`B.8.8.N.N.N.N", -"`i`i`i`i`i`6.N`0#.`i`i`i`i`i`6.N", -"`i.a.C.H.G.f.N`w#.`i.a.C.H.G.f.3", -"`i.a.x.z.y`5.4.P.P`D.a.x.z.y`5.8", -"`i.n.p.r.r`Z#..R.R#..n.p.r.r`Z.8", -"`N`5.e.f`5`K`H`H`H`H`5.e.f`5`b.6", -".N`D`P`c`Q`v.N.N.N.N`W`P`c`Q`H.N", -"`D.V`z`s`g`j.N.N.N`D.V`z`s`g`j.N", -"`D`N`S`S`M`m`D.N.N`D`N`S`S`M`m`D", -".N`D`t`t`k.8`D.N.N.N`D`t`t`k.8`D", -".N.N`D`D`D`D.N.N.N.N.N`D`D`D`D.N" -}; diff --git a/icons/proof_connected.xpm b/icons/proof_connected.xpm deleted file mode 100644 index ec03a21b01c00..0000000000000 --- a/icons/proof_connected.xpm +++ /dev/null @@ -1,279 +0,0 @@ -/* XPM */ -static char *proof_connected_xpm[] = { -/* width height num_colors chars_per_pixel */ -" 16 16 256 2", -/* colors */ -"`` c #000000", -"`. c #ffffff", -"`# c None s None", -"`a c #6060c6", -"`b c #6d6dcc", -"`c c #7878cb", -"`d c #7a7acc", -"`e c #8484cd", -"`f c #8989d4", -"`g c #4d4d71", -"`h c #9393cc", -"`i c #8a8abf", -"`j c #9797d0", -"`k c #a0a0cc", -"`l c #9797c0", -"`m c #b0b0d6", -"`n c #c0c0e0", -"`o c #9696ae", -"`p c #c0c0db", -"`q c #adadc5", -"`r c #c7c7db", -"`s c #c5c5d9", -"`t c #b9b9cc", -"`u c #babacb", -"`v c #cdcddd", -"`w c #d2d2db", -"`x c #d9d9dd", -"`y c #e0e0e1", -"`z c #dededf", -"`A c #89a6cf", -"`B c #53b3f6", -"`C c #43b1f9", -"`D c #91caed", -"`E c #33b3ff", -"`F c #46bcff", -"`G c #49beff", -"`H c #4dc0ff", -"`I c #5cc7ff", -"`J c #63caff", -"`K c #66ccff", -"`L c #85cff7", -"`M c #70d1ff", -"`N c #79d5ff", -"`O c #93b6c6", -"`P c #80d9ff", -"`Q c #8edafb", -"`R c #9ddcf5", -"`S c #8cdfff", -"`T c #8fe0ff", -"`U c #99e5ff", -"`V c #9ce7ff", -"`W c #9fe8ff", -"`X c #a3eaff", -"`Y c #b1e7f4", -"`Z c #c6fcff", -"`0 c #f3f3f3", -"`1 c #f2f2f2", -"`2 c #737373", -"`3 c #707070", -"`4 c #000000", -"`5 c #000000", -"`6 c #000000", -"`7 c #000000", -"`8 c #000000", -".` c #000000", -".. c #000000", -".# c #000000", -".a c #000000", -".b c #000000", -".c c #000000", -".d c #000000", -".e c #000000", -".f c #000000", -".g c #000000", -".h c #000000", -".i c #000000", -".j c #000000", -".k c #000000", -".l c #000000", -".m c #000000", -".n c #000000", -".o c #000000", -".p c #000000", -".q c #000000", -".r c #000000", -".s c #000000", -".t c #000000", -".u c #000000", -".v c #000000", -".w c #000000", -".x c #000000", -".y c #000000", -".z c #000000", -".A c #000000", -".B c #000000", -".C c #000000", -".D c #000000", -".E c #000000", -".F c #000000", -".G c #000000", -".H c #000000", -".I c #000000", -".J c #000000", -".K c #000000", -".L c #000000", -".M c #000000", -".N c #000000", -".O c #000000", -".P c #000000", -".Q c #000000", -".R c #000000", -".S c #000000", -".T c #000000", -".U c #000000", -".V c #000000", -".W c #000000", -".X c #000000", -".Y c #000000", -".Z c #000000", -".0 c #000000", -".1 c #000000", -".2 c #000000", -".3 c #000000", -".4 c #000000", -".5 c #000000", -".6 c #000000", -".7 c #000000", -".8 c #000000", -"#` c #000000", -"#. c #000000", -"## c #000000", -"#a c #000000", -"#b c #000000", -"#c c #000000", -"#d c #000000", -"#e c #000000", -"#f c #000000", -"#g c #000000", -"#h c #000000", -"#i c #000000", -"#j c #000000", -"#k c #000000", -"#l c #000000", -"#m c #000000", -"#n c #000000", -"#o c #000000", -"#p c #000000", -"#q c #000000", -"#r c #000000", -"#s c #000000", -"#t c #000000", -"#u c #000000", -"#v c #000000", -"#w c #000000", -"#x c #000000", -"#y c #000000", -"#z c #000000", -"#A c #000000", -"#B c #000000", -"#C c #000000", -"#D c #000000", -"#E c #000000", -"#F c #000000", -"#G c #000000", -"#H c #000000", -"#I c #000000", -"#J c #000000", -"#K c #000000", -"#L c #000000", -"#M c #000000", -"#N c #000000", -"#O c #000000", -"#P c #000000", -"#Q c #000000", -"#R c #000000", -"#S c #000000", -"#T c #000000", -"#U c #000000", -"#V c #000000", -"#W c #000000", -"#X c #000000", -"#Y c #000000", -"#Z c #000000", -"#0 c #000000", -"#1 c #000000", -"#2 c #000000", -"#3 c #000000", -"#4 c #000000", -"#5 c #000000", -"#6 c #000000", -"#7 c #000000", -"#8 c #000000", -"a` c #000000", -"a. c #000000", -"a# c #000000", -"aa c #000000", -"ab c #000000", -"ac c #000000", -"ad c #000000", -"ae c #000000", -"af c #000000", -"ag c #000000", -"ah c #000000", -"ai c #000000", -"aj c #000000", -"ak c #000000", -"al c #000000", -"am c #000000", -"an c #000000", -"ao c #000000", -"ap c #000000", -"aq c #000000", -"ar c #000000", -"as c #000000", -"at c #000000", -"au c #000000", -"av c #000000", -"aw c #000000", -"ax c #000000", -"ay c #000000", -"az c #000000", -"aA c #000000", -"aB c #000000", -"aC c #000000", -"aD c #000000", -"aE c #000000", -"aF c #000000", -"aG c #000000", -"aH c #000000", -"aI c #000000", -"aJ c #000000", -"aK c #000000", -"aL c #000000", -"aM c #000000", -"aN c #000000", -"aO c #000000", -"aP c #000000", -"aQ c #000000", -"aR c #000000", -"aS c #000000", -"aT c #000000", -"aU c #000000", -"aV c #000000", -"aW c #000000", -"aX c #000000", -"aY c #000000", -"aZ c #000000", -"a0 c #000000", -"a1 c #000000", -"a2 c #000000", -"a3 c #000000", -"a4 c #000000", -"a5 c #000000", -"a6 c #000000", -"a7 c #000000", -"a8 c #000000", -/* pixels */ -"`#`#`#`#`#`#`u`u`u`u`A`A`A`c`#`#", -"`#`#`#`#`#`#`O`L`T`Y`Z`R`D`c`3`#", -"`#`#`#`#`#`#`O`L`T`W`X`U`Q`c`3`#", -"`#`u`u`u`u`A`A`A`c`U`U`S`M`c`3`#", -"`#`O`L`T`Y`Z`V`D`c`K`K`K`K`c`3`#", -"`#`O`L`T`U`U`U`L`c`K`K`I`I`c`3`#", -"`#`O`L`P`P`P`N`K`c`E`E`B`B`c`3`#", -"`#`O`K`K`K`K`K`J`c`e`e`c`g`3`3`#", -"`#`O`F`F`H`H`G`C`c`s`j`b`b`3`#`#", -"`#`O`B`B`B`B`B`B`c`z`m`h`b`3`#`#", -"`#`#`o`f`f`f`a`r`y`x`v`t`b`3`#`#", -"`#`#`l`n`s`j`d`i`b`b`b`b`3`3`#`#", -"`#`j`0`1`z`p`k`b`3`3`3`3`2`#`#`#", -"`#`j`x`y`w`v`q`b`3`#`#`#`#`#`#`#", -"`#`#`b`b`b`b`b`3`3`#`#`#`#`#`#`#", -"`#`#`#`3`3`3`3`3`#`#`#`#`#`#`#`#" -}; diff --git a/icons/proof_disconnected.xpm b/icons/proof_disconnected.xpm deleted file mode 100644 index c3ce3572dd3ac..0000000000000 --- a/icons/proof_disconnected.xpm +++ /dev/null @@ -1,279 +0,0 @@ -/* XPM */ -static char *proof_disconnected_xpm[] = { -/* width height num_colors chars_per_pixel */ -" 16 16 256 2", -/* colors */ -"`` c #ca1718", -"`. c #ca1719", -"`# c #c71c1e", -"`a c #d12425", -"`b c #d32629", -"`c c #ef2d30", -"`d c #e33132", -"`e c #fb3839", -"`f c #ff4142", -"`g c #db4f51", -"`h c #f96f70", -"`i c #ff8889", -"`j c #f83339", -"`k c None s None", -"`l c #6060c6", -"`m c #5c5c9c", -"`n c #7878cb", -"`o c #37375d", -"`p c #7a7acc", -"`q c #8484cd", -"`r c #8989d4", -"`s c #8a8abf", -"`t c #9797d0", -"`u c #9292c0", -"`v c #6e6e8f", -"`w c #a0a0cc", -"`x c #9797c0", -"`y c #c0c0e0", -"`z c #a0a0b7", -"`A c #c0c0db", -"`B c #adadc5", -"`C c #c7c7db", -"`D c #c5c5d9", -"`E c #babacb", -"`F c #cdcddd", -"`G c #d2d2db", -"`H c #d9d9dd", -"`I c #e0e0e1", -"`J c #dededf", -"`K c #89a6cf", -"`L c #93b6c6", -"`M c #73392a", -"`N c #74392b", -"`O c #723b2d", -"`P c #7c4537", -"`Q c #7b3629", -"`R c #8a4a3f", -"`S c #9a4236", -"`T c #92544a", -"`U c #943a30", -"`V c #9e534b", -"`W c #8c2e26", -"`X c #992720", -"`Y c #952820", -"`Z c #962921", -"`0 c #9d2d26", -"`1 c #ba2d29", -"`2 c #972f2a", -"`3 c #c9504d", -"`4 c #d06f6b", -"`5 c #b01715", -"`6 c #d11f1f", -"`7 c #c51d1d", -"`8 c #d22121", -".` c #dc2525", -".. c #de2828", -".# c #f32f2d", -".a c #e82c2b", -".b c #ca2626", -".c c #df312f", -".d c #ff3636", -".e c #d83232", -".f c #fa4040", -".g c #d23636", -".h c #e93d3d", -".i c #fc4544", -".j c #da3e3d", -".k c #cf403d", -".l c #e74746", -".m c #ff4e4e", -".n c #db4b4a", -".o c #fe5a5a", -".p c #ff5b5b", -".q c #fe625f", -".r c #f75d5d", -".s c #eb5f5f", -".t c #ff6a6a", -".u c #ff6c6c", -".v c #ff6e6e", -".w c #e26262", -".x c #d85f5f", -".y c #f0716f", -".z c #ff7878", -".A c #fe7a7a", -".B c #f97878", -".C c #e57574", -".D c #f39999", -".E c #f3f3f3", -".F c #f2f2f2", -".G c #000000", -".H c #000000", -".I c #000000", -".J c #000000", -".K c #000000", -".L c #000000", -".M c #000000", -".N c #000000", -".O c #000000", -".P c #000000", -".Q c #000000", -".R c #000000", -".S c #000000", -".T c #000000", -".U c #000000", -".V c #000000", -".W c #000000", -".X c #000000", -".Y c #000000", -".Z c #000000", -".0 c #000000", -".1 c #000000", -".2 c #000000", -".3 c #000000", -".4 c #000000", -".5 c #000000", -".6 c #000000", -".7 c #000000", -".8 c #000000", -"#` c #000000", -"#. c #000000", -"## c #000000", -"#a c #000000", -"#b c #000000", -"#c c #000000", -"#d c #000000", -"#e c #000000", -"#f c #000000", -"#g c #000000", -"#h c #000000", -"#i c #000000", -"#j c #000000", -"#k c #000000", -"#l c #000000", -"#m c #000000", -"#n c #000000", -"#o c #000000", -"#p c #000000", -"#q c #000000", -"#r c #000000", -"#s c #000000", -"#t c #000000", -"#u c #000000", -"#v c #000000", -"#w c #000000", -"#x c #000000", -"#y c #000000", -"#z c #000000", -"#A c #000000", -"#B c #000000", -"#C c #000000", -"#D c #000000", -"#E c #000000", -"#F c #000000", -"#G c #000000", -"#H c #000000", -"#I c #000000", -"#J c #000000", -"#K c #000000", -"#L c #000000", -"#M c #000000", -"#N c #000000", -"#O c #000000", -"#P c #000000", -"#Q c #000000", -"#R c #000000", -"#S c #000000", -"#T c #000000", -"#U c #000000", -"#V c #000000", -"#W c #000000", -"#X c #000000", -"#Y c #000000", -"#Z c #000000", -"#0 c #000000", -"#1 c #000000", -"#2 c #000000", -"#3 c #000000", -"#4 c #000000", -"#5 c #000000", -"#6 c #000000", -"#7 c #000000", -"#8 c #000000", -"a` c #000000", -"a. c #000000", -"a# c #000000", -"aa c #000000", -"ab c #000000", -"ac c #000000", -"ad c #000000", -"ae c #000000", -"af c #000000", -"ag c #000000", -"ah c #000000", -"ai c #000000", -"aj c #000000", -"ak c #000000", -"al c #000000", -"am c #000000", -"an c #000000", -"ao c #000000", -"ap c #000000", -"aq c #000000", -"ar c #000000", -"as c #000000", -"at c #000000", -"au c #000000", -"av c #000000", -"aw c #000000", -"ax c #000000", -"ay c #000000", -"az c #000000", -"aA c #000000", -"aB c #000000", -"aC c #000000", -"aD c #000000", -"aE c #000000", -"aF c #000000", -"aG c #000000", -"aH c #000000", -"aI c #000000", -"aJ c #000000", -"aK c #000000", -"aL c #000000", -"aM c #000000", -"aN c #000000", -"aO c #000000", -"aP c #000000", -"aQ c #000000", -"aR c #000000", -"aS c #000000", -"aT c #000000", -"aU c #000000", -"aV c #000000", -"aW c #000000", -"aX c #000000", -"aY c #000000", -"aZ c #000000", -"a0 c #000000", -"a1 c #000000", -"a2 c #000000", -"a3 c #000000", -"a4 c #000000", -"a5 c #000000", -"a6 c #000000", -"a7 c #000000", -"a8 c #000000", -/* pixels */ -"`k`k`k`k`k`E`E`E`E`K`K`K`n`k`k`k", -"`k`k`k`k`k`L`o`o`o`o`o`o`n`k`k`k", -"`k`k`k`k`k`L`o`o`o`o`o`o`n`k`k`k", -"`E`E`E`E`K`K`K`n`o`o`o`o`n`k`k`k", -"`L`o`o`o`o`o`o`n`o`o`o`o`n`k`k`k", -"`L`o`o`o`o`o`o`n`o`o`o`o`n`k`k`k", -"`L`o`o`o`o.A.t.z`m`m`m`m.u.v`i`T", -"`L`o`o`o`o.D.r.t.s`q`n.t.p`h`4`P", -"`L`o`o`o`o`o.w.f.o.C.B.m`f`3`O`k", -"`L`m`m`m`m`m`m.h`e.q.i.d.k`N`k`k", -"`k`v`r`r`r`l`C`I.y.#`j.g`M`k`k`k", -"`z`x`y`D`t`p`s`x.l.a`c.e`U`k`k`k", -"`t.E.F`J`A`w`u.c.``b`1..`d`S`k`k", -"`t`H`I`G`F`B.n`6`a`2`k.b`8.j`V`k", -"`k`x`x`x`x.x`.`#`0`k`k`k`7```g`R", -"`k`k`k`k`k`5`Z`W`k`k`k`k`k`X`Y`Q" -}; diff --git a/io/io/inc/TFile.h b/io/io/inc/TFile.h index dbf1c1ec2596f..669030ef432c6 100644 --- a/io/io/inc/TFile.h +++ b/io/io/inc/TFile.h @@ -229,10 +229,6 @@ class TFile : public TDirectoryFile { virtual InfoListRet GetStreamerInfoListImpl(bool lookupSICache); - // Creating projects - Int_t MakeProjectParMake(const char *packname, const char *filename); - Int_t MakeProjectParProofInf(const char *packname, const char *proofinfdir); - // Interface to basic system I/O routines virtual Int_t SysOpen(const char *pathname, Int_t flags, UInt_t mode); virtual Int_t SysClose(Int_t fd); diff --git a/io/io/src/TFile.cxx b/io/io/src/TFile.cxx index 4e75242a22f43..4565fa1f45019 100644 --- a/io/io/src/TFile.cxx +++ b/io/io/src/TFile.cxx @@ -2760,101 +2760,51 @@ void TFile::MakeProject(const char *dirname, const char * /*classes*/, { TString opt = option; opt.ToLower(); - Bool_t makepar = kFALSE; - TString parname, pardir; - if (opt.Contains("par")) { - // Create a PAR file - parname = gSystem->BaseName(dirname); - if (parname.EndsWith(".par")) parname.ReplaceAll(".par",""); - pardir = gSystem->GetDirName(dirname); - // Cleanup or prepare the dirs - TString path, filepath; - void *dir = gSystem->OpenDirectory(pardir); - if (dir) { - path.Form("%s/%s", pardir.Data(), parname.Data()); - void *dirp = gSystem->OpenDirectory(path); - if (dirp) { - path += "/PROOF-INF"; - void *dirinf = gSystem->OpenDirectory(path); - const char *afile = 0; - if (dirinf) { - while ((afile = gSystem->GetDirEntry(dirinf))) { - if (strcmp(afile,".") == 0) continue; - if (strcmp(afile,"..") == 0) continue; - filepath.Form("%s/%s", path.Data(), afile); - if (gSystem->Unlink(filepath)) - Warning("MakeProject", "1: problems unlinking '%s' ('%s', '%s')", filepath.Data(), path.Data(), afile); - } - gSystem->FreeDirectory(dirinf); - } - gSystem->Unlink(path); - path.Form("%s/%s", pardir.Data(), parname.Data()); - while ((afile = gSystem->GetDirEntry(dirp))) { - if (strcmp(afile,".") == 0) continue; - if (strcmp(afile,"..") == 0) continue; - filepath.Form("%s/%s", path.Data(), afile); - if (gSystem->Unlink(filepath)) - Warning("MakeProject", "2: problems unlinking '%s' ('%s', '%s')", filepath.Data(), path.Data(), afile); - } - gSystem->FreeDirectory(dirp); - if (gSystem->Unlink(path)) - Warning("MakeProject", "problems unlinking '%s'", path.Data()); - } - gSystem->FreeDirectory(dir); - } - // Make sure that the relevant dirs exists: this is mandatory, so we fail if unsuccessful - path.Form("%s/%s/PROOF-INF", pardir.Data(), parname.Data()); - if (gSystem->mkdir(path, kTRUE)) { - Error("MakeProject", "problems creating '%s'", path.Data()); - return; - } - makepar = kTRUE; - - } else { - void *dir = gSystem->OpenDirectory(dirname); - TString dirpath; - if (opt.Contains("update")) { - // check that directory exist, if not create it - if (!dir) { - gSystem->mkdir(dirname); - } + void *dir = gSystem->OpenDirectory(dirname); + TString dirpath; - } else if (opt.Contains("recreate")) { - // check that directory exist, if not create it - if (!dir) { - if (gSystem->mkdir(dirname) < 0) { - Error("MakeProject","cannot create directory '%s'",dirname); - return; - } - } - // clear directory - while (dir) { - const char *afile = gSystem->GetDirEntry(dir); - if (!afile) break; - if (strcmp(afile,".") == 0) continue; - if (strcmp(afile,"..") == 0) continue; - dirpath.Form("%s/%s",dirname,afile); - gSystem->Unlink(dirpath); - } + if (opt.Contains("update")) { + // check that directory exist, if not create it + if (!dir) { + gSystem->mkdir(dirname); + } - } else { - // new is assumed - // if directory already exist, print error message and return - if (dir) { - Error("MakeProject","cannot create directory %s, already existing",dirname); - gSystem->FreeDirectory(dir); - return; - } + } else if (opt.Contains("recreate")) { + // check that directory exist, if not create it + if (!dir) { if (gSystem->mkdir(dirname) < 0) { Error("MakeProject","cannot create directory '%s'",dirname); return; } } + // clear directory + while (dir) { + const char *afile = gSystem->GetDirEntry(dir); + if (!afile) break; + if (strcmp(afile,".") == 0) continue; + if (strcmp(afile,"..") == 0) continue; + dirpath.Form("%s/%s",dirname,afile); + gSystem->Unlink(dirpath); + } + + } else { + // new is assumed + // if directory already exist, print error message and return if (dir) { + Error("MakeProject","cannot create directory %s, already existing",dirname); gSystem->FreeDirectory(dir); + return; } + if (gSystem->mkdir(dirname) < 0) { + Error("MakeProject","cannot create directory '%s'",dirname); + return; + } + } + if (dir) { + gSystem->FreeDirectory(dir); } + Bool_t genreflex = opt.Contains("genreflex"); // we are now ready to generate the classes @@ -2867,7 +2817,6 @@ void TFile::MakeProject(const char *dirname, const char * /*classes*/, } TString clean_dirname(dirname); - if (makepar) clean_dirname.Form("%s/%s", pardir.Data(), parname.Data()); if (clean_dirname[clean_dirname.Length()-1]=='/') { clean_dirname.Remove(clean_dirname.Length()-1); } else if (clean_dirname[clean_dirname.Length()-1]=='\\') { @@ -2877,7 +2826,6 @@ void TFile::MakeProject(const char *dirname, const char * /*classes*/, } } TString subdirname( gSystem->BaseName(clean_dirname) ); - if (makepar) subdirname = parname; if (subdirname == "") { Error("MakeProject","Directory name must not be empty."); return; @@ -3029,7 +2977,7 @@ void TFile::MakeProject(const char *dirname, const char * /*classes*/, printf("MakeProject has generated %d classes in %s\n",ngener,clean_dirname.Data()); // generate the shared lib - if (!opt.Contains("+") && !makepar) { + if (!opt.Contains("+")) { delete list; filelist->Delete(); delete filelist; @@ -3038,26 +2986,24 @@ void TFile::MakeProject(const char *dirname, const char * /*classes*/, // Makefiles files FILE *fpMAKE = nullptr; - if (!makepar) { - // Create the MAKEP file by looping on all *.h files - // delete MAKEP if it already exists + // Create the MAKEP file by looping on all *.h files + // delete MAKEP if it already exists #ifdef WIN32 - path.Form("%s/makep.cmd",clean_dirname.Data()); + path.Form("%s/makep.cmd",clean_dirname.Data()); #else - path.Form("%s/MAKEP",clean_dirname.Data()); + path.Form("%s/MAKEP",clean_dirname.Data()); #endif #ifdef R__WINGCC - fpMAKE = fopen(path,"wb"); + fpMAKE = fopen(path,"wb"); #else - fpMAKE = fopen(path,"w"); + fpMAKE = fopen(path,"w"); #endif - if (!fpMAKE) { - Error("MakeProject", "cannot open file %s", path.Data()); - delete list; - filelist->Delete(); - delete filelist; - return; - } + if (!fpMAKE) { + Error("MakeProject", "cannot open file %s", path.Data()); + delete list; + filelist->Delete(); + delete filelist; + return; } // Add rootcint/genreflex statement generating ProjectDict.cxx @@ -3077,15 +3023,11 @@ void TFile::MakeProject(const char *dirname, const char * /*classes*/, return; } - if (!makepar) { - if (genreflex) { - fprintf(fpMAKE,"genreflex %sProjectHeaders.h -o %sProjectDict.cxx --comments --iocomments %s ",subdirname.Data(),subdirname.Data(),gSystem->GetIncludePath()); - path.Form("%s/%sSelection.xml",clean_dirname.Data(),subdirname.Data()); - } else { - fprintf(fpMAKE,"rootcint -v1 -f %sProjectDict.cxx %s ", subdirname.Data(), gSystem->GetIncludePath()); - path.Form("%s/%sLinkDef.h",clean_dirname.Data(),subdirname.Data()); - } + if (genreflex) { + fprintf(fpMAKE,"genreflex %sProjectHeaders.h -o %sProjectDict.cxx --comments --iocomments %s ",subdirname.Data(),subdirname.Data(),gSystem->GetIncludePath()); + path.Form("%s/%sSelection.xml",clean_dirname.Data(),subdirname.Data()); } else { + fprintf(fpMAKE,"rootcint -v1 -f %sProjectDict.cxx %s ", subdirname.Data(), gSystem->GetIncludePath()); path.Form("%s/%sLinkDef.h",clean_dirname.Data(),subdirname.Data()); } @@ -3259,108 +3201,44 @@ void TFile::MakeProject(const char *dirname, const char * /*classes*/, fclose(fp); fclose(ifp); - if (!makepar) { - // add compilation line - TString sdirname(subdirname); - - TString cmd = gSystem->GetMakeSharedLib(); - TString sources = TString::Format("%sProjectSource.cxx ", sdirname.Data()); - cmd.ReplaceAll("$SourceFiles",sources.Data()); - TString object = TString::Format("%sProjectSource.", sdirname.Data()); - object.Append( gSystem->GetObjExt() ); - cmd.ReplaceAll("$ObjectFiles", object.Data()); - cmd.ReplaceAll("$IncludePath",TString(gSystem->GetIncludePath()) + " -I" + clean_dirname.Data()); - cmd.ReplaceAll("$SharedLib",sdirname+"."+gSystem->GetSoExt()); - cmd.ReplaceAll("$LinkedLibs",gSystem->GetLibraries("","SDL")); - cmd.ReplaceAll("$LibName",sdirname); - cmd.ReplaceAll("$BuildDir","."); - TString sOpt; - TString rootbuild = ROOTBUILD; - if (rootbuild.Index("debug",0,TString::kIgnoreCase)==kNPOS) { - sOpt = gSystem->GetFlagsOpt(); - } else { - sOpt = gSystem->GetFlagsDebug(); - } - cmd.ReplaceAll("$Opt", sOpt); - - if (genreflex) { - fprintf(fpMAKE,"-s %sSelection.xml \n",subdirname.Data()); - } else { - fprintf(fpMAKE,"%sProjectHeaders.h ",subdirname.Data()); - fprintf(fpMAKE,"%sLinkDef.h \n",subdirname.Data()); - } - - fprintf(fpMAKE,"%s\n",cmd.Data()); - - printf("%s/MAKEP file has been generated\n", clean_dirname.Data()); - - fclose(fpMAKE); + // add compilation line + TString sdirname(subdirname); + + TString cmd = gSystem->GetMakeSharedLib(); + TString sources = TString::Format("%sProjectSource.cxx ", sdirname.Data()); + cmd.ReplaceAll("$SourceFiles",sources.Data()); + TString object = TString::Format("%sProjectSource.", sdirname.Data()); + object.Append( gSystem->GetObjExt() ); + cmd.ReplaceAll("$ObjectFiles", object.Data()); + cmd.ReplaceAll("$IncludePath",TString(gSystem->GetIncludePath()) + " -I" + clean_dirname.Data()); + cmd.ReplaceAll("$SharedLib",sdirname+"."+gSystem->GetSoExt()); + cmd.ReplaceAll("$LinkedLibs",gSystem->GetLibraries("","SDL")); + cmd.ReplaceAll("$LibName",sdirname); + cmd.ReplaceAll("$BuildDir","."); + TString sOpt; + TString rootbuild = ROOTBUILD; + if (rootbuild.Index("debug",0,TString::kIgnoreCase)==kNPOS) { + sOpt = gSystem->GetFlagsOpt(); + } else { + sOpt = gSystem->GetFlagsDebug(); + } + cmd.ReplaceAll("$Opt", sOpt); + if (genreflex) { + fprintf(fpMAKE,"-s %sSelection.xml \n",subdirname.Data()); } else { + fprintf(fpMAKE,"%sProjectHeaders.h ",subdirname.Data()); + fprintf(fpMAKE,"%sLinkDef.h \n",subdirname.Data()); + } - // Create the Makefile - TString filemake = TString::Format("%s/Makefile", clean_dirname.Data()); - if (MakeProjectParMake(parname, filemake.Data()) != 0) { - Error("MakeProject", "problems creating PAR make file '%s'", filemake.Data()); - delete list; - filelist->Delete(); - delete filelist; - return; - } - // Get Makefile.arch - TString mkarchsrc = TString::Format("%s/Makefile.arch", TROOT::GetEtcDir().Data()); - if (gSystem->ExpandPathName(mkarchsrc)) - Warning("MakeProject", "problems expanding '%s'", mkarchsrc.Data()); - TString mkarchdst = TString::Format("%s/Makefile.arch", clean_dirname.Data()); - if (gSystem->CopyFile(mkarchsrc.Data(), mkarchdst.Data(), kTRUE) != 0) { - Error("MakeProject", "problems retrieving '%s' to '%s'", mkarchsrc.Data(), mkarchdst.Data()); - delete list; - filelist->Delete(); - delete filelist; - return; - } - // Create the Makefile - TString proofinf = TString::Format("%s/PROOF-INF", clean_dirname.Data()); - if (MakeProjectParProofInf(parname, proofinf.Data()) != 0) { - Error("MakeProject", "problems creating BUILD.sh and/or SETUP.C under '%s'", proofinf.Data()); - delete list; - filelist->Delete(); - delete filelist; - return; - } + fprintf(fpMAKE,"%s\n",cmd.Data()); - // Make sure BUILD.sh is executable and create SETUP.C - TString cmod = TString::Format("chmod +x %s/PROOF-INF/BUILD.sh", clean_dirname.Data()); -#ifndef WIN32 - gSystem->Exec(cmod.Data()); -#else - // not really needed for Windows but it would work both both Unix and NT - chmod(cmod.Data(), 00700); -#endif - Printf("Files Makefile, Makefile.arch, PROOF-INF/BUILD.sh and" - " PROOF-INF/SETUP.C have been generated under '%s'", clean_dirname.Data()); + printf("%s/MAKEP file has been generated\n", clean_dirname.Data()); - // Generate the PAR file, if not Windows -#ifndef WIN32 - TString curdir = gSystem->WorkingDirectory(); - if (gSystem->ChangeDirectory(pardir)) { - TString cmd = TString::Format("tar czvf %s.par %s", parname.Data(), parname.Data()); - gSystem->Exec(cmd.Data()); - if (gSystem->ChangeDirectory(curdir)) { - Info("MakeProject", "PAR file %s.par generated", clean_dirname.Data()); - } else { - Warning("MakeProject", "problems changing directory back to '%s'", curdir.Data()); - } - } else { - Error("MakeProject", "problems changing directory to '%s' - skipping PAR file generation", pardir.Data()); - } -#else - Warning("MakeProject", "on Windows systems the PAR file cannot be generated out of the package directory!"); -#endif - } + fclose(fpMAKE); - if (!makepar && !opt.Contains("nocompilation")) { + if (!opt.Contains("nocompilation")) { // now execute the generated script compiling and generating the shared lib path = gSystem->WorkingDirectory(); gSystem->ChangeDirectory(clean_dirname.Data()); @@ -3388,245 +3266,6 @@ void TFile::MakeProject(const char *dirname, const char * /*classes*/, delete filelist; } -//////////////////////////////////////////////////////////////////////////////// -/// Create makefile at 'filemake' for PAR package 'pack'. -/// -/// Called by MakeProject when option 'par' is given. -/// Return 0 on success, -1 on error. - -Int_t TFile::MakeProjectParMake(const char *pack, const char *filemake) -{ - // Output file path must be defined - if (!filemake || (filemake && strlen(filemake) <= 0)) { - Error("MakeProjectParMake", "path for output file undefined!"); - return -1; - } - - // Package name must be defined - if (!pack || (pack && strlen(pack) <= 0)) { - Error("MakeProjectParMake", "package name undefined!"); - return -1; - } - -#ifdef R__WINGCC - FILE *fmk = fopen(filemake, "wb"); -#else - FILE *fmk = fopen(filemake, "w"); -#endif - if (!fmk) { - Error("MakeProjectParMake", "cannot create file '%s' (errno: %d)", filemake, TSystem::GetErrno()); - return -1; - } - - // Fill the file now - fprintf(fmk, "# Makefile for the ROOT test programs.\n"); - fprintf(fmk, "# This Makefile shows how to compile and link applications\n"); - fprintf(fmk, "# using the ROOT libraries on all supported platforms.\n"); - fprintf(fmk, "#\n"); - fprintf(fmk, "# Copyright (c) 2000 Rene Brun and Fons Rademakers\n"); - fprintf(fmk, "#\n"); - fprintf(fmk, "# Author: this makefile has been automatically generated via TFile::MakeProject\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "include Makefile.arch\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "#------------------------------------------------------------------------------\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "PACKO = %sProjectSource.$(ObjSuf)\n", pack); - fprintf(fmk, "PACKS = %sProjectSource.$(SrcSuf) %sProjectDict.$(SrcSuf)\n", pack, pack); - fprintf(fmk, "PACKSO = lib%s.$(DllSuf)\n", pack); - fprintf(fmk, "\n"); - fprintf(fmk, "ifeq ($(PLATFORM),win32)\n"); - fprintf(fmk, "PACKLIB = lib%s.lib\n", pack); - fprintf(fmk, "else\n"); - fprintf(fmk, "PACKLIB = $(PACKSO)\n"); - fprintf(fmk, "endif\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "OBJS = $(PACKO)\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "PROGRAMS =\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "#------------------------------------------------------------------------------\n"); - fprintf(fmk, "\n"); - fprintf(fmk, ".SUFFIXES: .$(SrcSuf) .$(ObjSuf) .$(DllSuf)\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "all: $(PACKLIB)\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "$(PACKSO): $(PACKO)\n"); - fprintf(fmk, "ifeq ($(ARCH),aix)\n"); - fprintf(fmk, "\t\t/usr/ibmcxx/bin/makeC++SharedLib $(OutPutOpt) $@ $(LIBS) -p 0 $^\n"); - fprintf(fmk, "else\n"); - fprintf(fmk, "ifeq ($(ARCH),aix5)\n"); - fprintf(fmk, "\t\t/usr/vacpp/bin/makeC++SharedLib $(OutPutOpt) $@ $(LIBS) -p 0 $^\n"); - fprintf(fmk, "else\n"); - fprintf(fmk, "ifeq ($(PLATFORM),macosx)\n"); - fprintf(fmk, "# We need to make both the .dylib and the .so\n"); - fprintf(fmk, "\t\t$(LD) $(SOFLAGS)$@ $(LDFLAGS) $^ $(OutPutOpt) $@ $(LIBS)\n"); - fprintf(fmk, "ifneq ($(subst $(MACOSX_MINOR),,1234),1234)\n"); - fprintf(fmk, "ifeq ($(MACOSX_MINOR),4)\n"); - fprintf(fmk, "\t\tln -sf $@ $(subst .$(DllSuf),.so,$@)\n"); - fprintf(fmk, "else\n"); - fprintf(fmk, "\t\t$(LD) -bundle -undefined $(UNDEFOPT) $(LDFLAGS) $^ \\\n"); - fprintf(fmk, "\t\t $(OutPutOpt) $(subst .$(DllSuf),.so,$@)\n"); - fprintf(fmk, "endif\n"); - fprintf(fmk, "endif\n"); - fprintf(fmk, "else\n"); - fprintf(fmk, "ifeq ($(PLATFORM),win32)\n"); - fprintf(fmk, "\t\tbindexplib $* $^ > $*.def\n"); - fprintf(fmk, "\t\tlib -nologo -MACHINE:IX86 $^ -def:$*.def \\\n"); - fprintf(fmk, "\t\t $(OutPutOpt)$(PACKLIB)\n"); - fprintf(fmk, "\t\t$(LD) $(SOFLAGS) $(LDFLAGS) $^ $*.exp $(LIBS) \\\n"); - fprintf(fmk, "\t\t $(OutPutOpt)$@\n"); - fprintf(fmk, "else\n"); - fprintf(fmk, "\t\t$(LD) $(SOFLAGS) $(LDFLAGS) $^ $(OutPutOpt) $@ $(LIBS) $(EXPLLINKLIBS)\n"); - fprintf(fmk, "endif\n"); - fprintf(fmk, "endif\n"); - fprintf(fmk, "endif\n"); - fprintf(fmk, "endif\n"); - fprintf(fmk, "\t\t@echo \"$@ done\"\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "clean:\n"); - fprintf(fmk, "\t\t@rm -f $(OBJS) core\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "distclean: clean\n"); - fprintf(fmk, "\t\t@rm -f $(PROGRAMS) $(PACKSO) $(PACKLIB) *Dict.* *.def *.exp \\\n"); - fprintf(fmk, "\t\t *.so *.lib *.dll *.d *.log .def so_locations\n"); - fprintf(fmk, "\t\t@rm -rf cxx_repository\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "# Dependencies\n"); - fprintf(fmk, "\n"); - fprintf(fmk, "%sProjectSource.$(ObjSuf): %sProjectHeaders.h %sLinkDef.h %sProjectDict.$(SrcSuf)\n", pack, pack, pack, pack); - fprintf(fmk, "\n"); - fprintf(fmk, "%sProjectDict.$(SrcSuf): %sProjectHeaders.h %sLinkDef.h\n", pack, pack, pack); - fprintf(fmk, "\t\t@echo \"Generating dictionary $@...\"\n"); - fprintf(fmk, "\t\t@rootcint -f $@ $^\n"); - fprintf(fmk, "\n"); - fprintf(fmk, ".$(SrcSuf).$(ObjSuf):\n"); - fprintf(fmk, "\t\t$(CXX) $(CXXFLAGS) -c $<\n"); - fprintf(fmk, "\n"); - - // Close the file - fclose(fmk); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create BUILD.sh and SETUP.C under 'proofinf' for PAR package 'pack'. -/// Called by MakeProject when option 'par' is given. -/// Return 0 on success, -1 on error. - -Int_t TFile::MakeProjectParProofInf(const char *pack, const char *proofinf) -{ - // Output directory path must be defined ... - if (!proofinf || (proofinf && strlen(proofinf) <= 0)) { - Error("MakeProjectParProofInf", "directory path undefined!"); - return -1; - } - - // ... and exist and be a directory - Int_t rcst = 0; - FileStat_t st; - if ((rcst = gSystem->GetPathInfo(proofinf, st)) != 0 || !R_ISDIR(st.fMode)) { - Error("MakeProjectParProofInf", "path '%s' %s", proofinf, - ((rcst == 0) ? "is not a directory" : "does not exist")); - return -1; - } - - // Package name must be defined - if (!pack || (pack && strlen(pack) <= 0)) { - Error("MakeProjectParProofInf", "package name undefined!"); - return -1; - } - - TString path; - - // The BUILD.sh first - path.Form("%s/BUILD.sh", proofinf); -#ifdef R__WINGCC - FILE *f = fopen(path.Data(), "wb"); -#else - FILE *f = fopen(path.Data(), "w"); -#endif - if (!f) { - Error("MakeProjectParProofInf", "cannot create file '%s' (errno: %d)", - path.Data(), TSystem::GetErrno()); - return -1; - } - - fprintf(f, "#! /bin/sh\n"); - fprintf(f, "# Build libEvent library.\n"); - fprintf(f, "\n"); - fprintf(f, "#\n"); - fprintf(f, "# The environment variables ROOTPROOFLITE and ROOTPROOFCLIENT can be used to\n"); - fprintf(f, "# adapt the script to the calling environment\n"); - fprintf(f, "#\n"); - fprintf(f, "# if test ! \"x$ROOTPROOFLITE\" = \"x\"; then\n"); - fprintf(f, "# echo \"event-BUILD: PROOF-Lite node (session has $ROOTPROOFLITE workers)\"\n"); - fprintf(f, "# elif test ! \"x$ROOTPROOFCLIENT\" = \"x\"; then\n"); - fprintf(f, "# echo \"event-BUILD: PROOF client\"\n"); - fprintf(f, "# else\n"); - fprintf(f, "# echo \"event-BUILD: standard PROOF node\"\n"); - fprintf(f, "# fi\n"); - fprintf(f, "\n"); - fprintf(f, "if [ \"\" = \"clean\" ]; then\n"); - fprintf(f, " make distclean\n"); - fprintf(f, " exit 0\n"); - fprintf(f, "fi\n"); - fprintf(f, "\n"); - fprintf(f, "make\n"); - fprintf(f, "rc=$?\n"); - fprintf(f, "echo \"rc=$?\"\n"); - fprintf(f, "if [ $? != \"0\" ] ; then\n"); - fprintf(f, " exit 1\n"); - fprintf(f, "fi\n"); - fprintf(f, "exit 0\n"); - - // Close the file - fclose(f); - - // Then SETUP.C - path.Form("%s/SETUP.C", proofinf); -#ifdef R__WINGCC - f = fopen(path.Data(), "wb"); -#else - f = fopen(path.Data(), "w"); -#endif - if (!f) { - Error("MakeProjectParProofInf", "cannot create file '%s' (errno: %d)", - path.Data(), TSystem::GetErrno()); - return -1; - } - - fprintf(f, "Int_t SETUP()\n"); - fprintf(f, "{\n"); - fprintf(f, "\n"); - fprintf(f, "//\n"); - fprintf(f, "// The environment variables ROOTPROOFLITE and ROOTPROOFCLIENT can be used to\n"); - fprintf(f, "// adapt the macro to the calling environment\n"); - fprintf(f, "//\n"); - fprintf(f, "// if (gSystem->Getenv(\"ROOTPROOFLITE\")) {\n"); - fprintf(f, "// Printf(\"event-SETUP: PROOF-Lite node (session has %%s workers)\",\n"); - fprintf(f, "// gSystem->Getenv(\"ROOTPROOFLITE\"));\n"); - fprintf(f, "// } else if (gSystem->Getenv(\"ROOTPROOFCLIENT\")) {\n"); - fprintf(f, "// Printf(\"event-SETUP: PROOF client\");\n"); - fprintf(f, "// } else {\n"); - fprintf(f, "// Printf(\"event-SETUP: standard PROOF node\");\n"); - fprintf(f, "// }\n"); - fprintf(f, "\n"); - fprintf(f, " if (gSystem->Load(\"lib%s\") == -1)\n", pack); - fprintf(f, " return -1;\n"); - fprintf(f, " return 0;\n"); - fprintf(f, "}\n"); - fprintf(f, "\n"); - - // Close the file - fclose(f); - - // Done - return 0; -} - //////////////////////////////////////////////////////////////////////////////// /// Read the list of StreamerInfo from this file. /// diff --git a/main/CMakeLists.txt b/main/CMakeLists.txt index c274a8f400755..91fbdc47ad284 100644 --- a/main/CMakeLists.txt +++ b/main/CMakeLists.txt @@ -30,10 +30,6 @@ if(MSVC) add_custom_command(TARGET root.exe POST_BUILD COMMAND mt -manifest ${ROOT_MANIFEST} -outputresource:${CMAKE_BINARY_DIR}/bin/root.exe) endif() -if(proof) - ROOT_EXECUTABLE(proofserv.exe pmain.cxx LIBRARIES Core MathCore CMAKENOEXPORT) -endif() - if(MSVC) ROOT_EXECUTABLE(hadd hadd.cxx LIBRARIES Core RIO Net Hist Graf Graf3d Gpad Tree Matrix MathCore CMAKENOEXPORT) diff --git a/main/src/pmain.cxx b/main/src/pmain.cxx deleted file mode 100644 index 6b1cefd2a7f60..0000000000000 --- a/main/src/pmain.cxx +++ /dev/null @@ -1,260 +0,0 @@ -// @(#)root/main:$Id$ -// Author: Fons Rademakers 15/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -////////////////////////////////////////////////////////////////////////// -// // -// PMain // -// // -// Main program used to create PROOF server application. // -// // -////////////////////////////////////////////////////////////////////////// -#include -#include - -#ifdef WIN32 -#include -#else -#include -#include -#endif -#include -#include - -#include -#include "RConfigure.h" -#include "TApplication.h" -#include "TInterpreter.h" -#include "TROOT.h" -#include "TSystem.h" -#include "strlcpy.h" - - -static Int_t gLogLevel = 0; - -// Special type for the hook to the TXProofServ constructor, needed to avoid -// using the plugin manager -typedef TApplication *(*TProofServ_t)(Int_t *argc, char **argv, FILE *flog); - -//////////////////////////////////////////////////////////////////////////////// -/// Read envs from file 'envfile' and add them to the env space - -static void ReadPutEnvs(const char *envfile) -{ - // Check inputs - if (!envfile || strlen(envfile) <= 0) return; - - // Open the file - FILE *fenv = fopen(envfile, "r"); - if (!fenv) return; - - // Read lines - char ln[4096]; - while (fgets(ln, sizeof(ln), fenv)) { - int l = strlen(ln); - // Strip '\n' - if (l > 0 && ln[l-1] == '\n') { ln[l-1] = '\0'; l--; } - // Skip comments or empty line - if (l <= 0 || ln[0] == '#') continue; - // Skip lines not in the form '=' - if (!strchr(ln, '=')) continue; - // Good line - char *ev = new char[l+1]; - strlcpy(ev, ln, l+1); - putenv(ev); // NOLINT: do not release memory in ev, will be used by environment variable - } - - // Close the file - fclose(fenv); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Redirect stdout to 'logfile'. This log file will be flushed to the -/// client or master after each command. -/// If donotredir != 0 just reopen the file for usage in TProofServ (already redirected). -/// On success return a pointer to the open log file. Return 0 on failure. - -static FILE *RedirectOutput(const char *logfile, const char *loc, Int_t donotredir) -{ - if (loc) - fprintf(stderr, "%s: RedirectOutput: enter: %s (do-not-redir: %d)\n", - loc, logfile, donotredir); - - if (donotredir == 0) { - if (!logfile || strlen(logfile) <= 0) { - fprintf(stderr,"%s: RedirectOutput: logfile path undefined\n", loc); - return 0; - } - - if (loc) - fprintf(stderr,"%s: RedirectOutput: reopen %s\n", loc, logfile); - FILE *flog = freopen(logfile, "a", stdout); - if (!flog) { - fprintf(stderr,"%s: RedirectOutput: could not freopen stdout\n", loc); - return 0; - } - - if (loc) - fprintf(stderr,"%s: RedirectOutput: dup2 ...\n", loc); - if ((dup2(fileno(stdout), fileno(stderr))) < 0) { - fprintf(stderr,"%s: RedirectOutput: could not redirect stderr\n", loc); - return 0; - } - } - - if (loc) - fprintf(stderr,"%s: RedirectOutput: read open ...\n", loc); - FILE *fLog = fopen(logfile, "r"); - if (!fLog) { - fprintf(stderr,"%s: RedirectOutput: could not open logfile %s\n", loc, logfile); - return 0; - } - - if (loc) - fprintf(stderr,"%s: RedirectOutput: done!\n", loc); - // We are done - return fLog; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set limits on the address space (virtual memory) if required. - -static void SetMaxMemLimits(const char *prog) -{ -#ifndef WIN32 - const char *assoft = gSystem->Getenv("ROOTPROOFASSOFT"); - const char *ashard = gSystem->Getenv("ROOTPROOFASHARD"); - - if (assoft || ashard) { - struct rlimit aslim, aslimref; - if (getrlimit(RLIMIT_AS, &aslimref) != 0) { - fprintf(stderr,"%s: problems getting RLIMIT_AS values (errno: %d)\n", prog, errno); - exit(1); - } - if (gLogLevel > 0) - fprintf(stderr, "%s: memory limits currently set to %lld (soft) and %lld (hard) bytes\n", - prog, (Long64_t)aslimref.rlim_cur, (Long64_t)aslimref.rlim_max); - aslim.rlim_cur = aslimref.rlim_cur; - aslim.rlim_max = aslimref.rlim_max; - if (assoft) { - Long_t rlim_cur = strtol(assoft, 0, 10); - if (rlim_cur < kMaxLong && rlim_cur > 0) - aslim.rlim_cur = (rlim_t) rlim_cur * (1024 * 1024); - } - if (ashard) { - Long_t rlim_max = strtol(ashard, 0, 10); - if (rlim_max < kMaxLong && rlim_max > 0) - aslim.rlim_max = (rlim_t) rlim_max * (1024 * 1024); - } - // Change the limits, if required - if ((aslim.rlim_cur != aslimref.rlim_cur) || (aslim.rlim_max != aslimref.rlim_max)) { - fprintf(stderr, "%s: setting memory limits to %lld (soft) and %lld (hard) bytes\n", - prog, (Long64_t)aslim.rlim_cur, (Long64_t)aslim.rlim_max); - if (setrlimit(RLIMIT_AS, &aslim) != 0) { - fprintf(stderr,"%s: problems setting RLIMIT_AS values (errno: %d)\n", prog, errno); - exit(1); - } - } - } -#endif -} - -//////////////////////////////////////////////////////////////////////////////// -/// PROOF server main program. - -int main(int argc, char **argv) -{ -#ifdef R__DEBUG - int debug = 1; - while (debug) - ; -#endif - if (argc >= 6) { - // Read and put system envs - ReadPutEnvs(argv[5]); - } - - gLogLevel = (argc >= 5) ? strtol(argv[4], 0, 10) : -1; - if (gLogLevel < 0 && gSystem->Getenv("ROOTPROOFLOGLEVEL")) - gLogLevel = atoi(gSystem->Getenv("ROOTPROOFLOGLEVEL")); - if (gLogLevel > 0) - fprintf(stderr,"%s: starting %s\n", argv[1], argv[0]); - - // Redirect the output - FILE *fLog = 0; - const char *loc = 0; - const char *logfile = gSystem->Getenv("ROOTPROOFLOGFILE"); - Int_t donotredir = 0; - if (gSystem->Getenv("ROOTPROOFDONOTREDIR")) { - donotredir++; - TString anr(gSystem->Getenv("ROOTPROOFDONOTREDIR")); - if (anr.IsDigit()) donotredir = anr.Atoi(); - } - if (logfile && donotredir != 1) { - loc = (gLogLevel > 0) ? argv[1] : 0; - if (gLogLevel > 0) - fprintf(stderr,"%s: redirecting output to %s\n", argv[1], logfile); - if (!(fLog = RedirectOutput(logfile, loc, donotredir))) { - fprintf(stderr,"%s: problems redirecting output to file %s\n", argv[1], logfile); - exit(1); - } - } - if (gLogLevel > 0) - fprintf(stderr,"%s: output redirected to: %s\n", - argv[1], (logfile ? logfile : "+++not redirected+++")); - - SetMaxMemLimits(argv[1]); - - gROOT->SetBatch(); - TApplication *theApp = 0; - - TString getter("GetTXProofServ"); - TString prooflib = "libProofx"; - if (argc > 2) { - if (!strcmp(argv[2], "lite")) { - // Lite version for local processing: this is in libProof and has its own getter - prooflib = "libProof"; - getter = "GetTProofServLite"; - } else if (strcmp(argv[2], "xpd")) { - // Not XPD: set the appropriate names for library and getter - prooflib = "libProof"; - getter = "GetTProofServ"; - } - } - char *p = 0; - if ((p = gSystem->DynamicPathName(prooflib, kTRUE))) { - delete[] p; - if (gSystem->Load(prooflib) == -1) { - fprintf(stderr,"%s: can't load %s\n", argv[1], prooflib.Data()); - exit(1); - } - } else { - fprintf(stderr,"%s: can't locate %s\n", argv[1], prooflib.Data()); - exit(1); - } - - // Locate constructor - Func_t f = gSystem->DynFindSymbol(prooflib, getter); - if (f) { - theApp = (TApplication *) (*((TProofServ_t)f))(&argc, argv, fLog); - } else { - fprintf(stderr,"%s: can't find %s\n", argv[1], getter.Data()); - exit(1); - } - - // Ready to run - if (gLogLevel > 0) - fprintf(stderr,"%s: running the TProofServ application\n", argv[1]); - - theApp->Run(); - - // We can exit now - gSystem->Exit(0); -} diff --git a/man/man1/proofserv.1 b/man/man1/proofserv.1 deleted file mode 100644 index ff05f2beb568d..0000000000000 --- a/man/man1/proofserv.1 +++ /dev/null @@ -1,76 +0,0 @@ -.\" -.\" $Id: proofserv.1,v 1.3 2006/02/28 16:38:23 rdm Exp $ -.\" -.TH PROOFSERV 1 "Version 3" "ROOT" -.\" NAME should be all caps, SECTION should be 1-8, maybe w/ subsection -.\" other parms are allowed: see man(7), man(1) -.SH NAME -proofserv \- PROOF (The Parallel ROOT Facility) -.SH "DESCRIPTION" -Using \fBPROOF\fI (The \fIP\fRarallel \fIROO\fRT \fIF\fRacility) one -can analyze trees in parallel on a cluster of computers. -.PP -The \fIproofserv\fR(1) is a basically the same as the \fBROOT\fR -interactive module \fIroot\fR(1), except that it reads its commands -from a socket instead of from the terminal. -.PP -Since it is a remote server it will not do graphics and therefore is -not linked with any graphics libraries (Motif, X11, etc.). -.PP -\fBproofserv\fR is started by the daemon \fBproofd\fR(1) on request by -the user. It is not a command to be executed by the user directly. -.PP -Note, that if \fBproofserv\fR is passed any arguments, it will use the -\fBlibProofx\fB interface, rather than the standard \fBlibProof\fR -interface. -.SH "SEE ALSO" -.IR root (1) -.PP -More information can be found at the \fBROOT\fR website: -.UR http://root.cern -\fIhttp://root.cern\fB -.UE -.SH "ORIGINAL AUTHORS" -The ROOT team (see web page above): -.RS -\fBRene Brun\fR and \fBFons Rademakers\fR -.RE -.SH "COPYRIGHT" -This library is free software; you can redistribute it and/or modify -it under the terms of the GNU Lesser General Public License as -published by the Free Software Foundation; either version 2.1 of the -License, or (at your option) any later version. -.P -This library is distributed in the hope that it will be useful, but -WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -Lesser General Public License for more details. -.P -You should have received a copy of the GNU Lesser General Public -License along with this library; if not, write to the Free Software -Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA -.SH AUTHOR -This manual page was written by Christian Holm Christensen -, for the Debian GNU/Linux system (but may be used by -others). -.\" -.\" $Log: proofserv.1,v $ -.\" Revision 1.3 2006/02/28 16:38:23 rdm -.\" From Christian Holm: -.\" many mods in the Debian and RedHat packaging system to ready ROOT for -.\" inclusion in Debian and RedHat distributions. -.\" -.\" Revision 1.2 2005/09/07 10:23:23 rdm -.\" From Christian Holm: -.\" fixes for Debian packages (due to the license change ROOT can be moved -.\" out of the non-free tree). Also updates in the man pages reflecting the -.\" new license. -.\" -.\" Revision 1.1 2001/08/15 13:30:48 rdm -.\" move man files to new subdir man1. This makes it possible to add -.\" $ROOTSYS/man to MANPATH and have "man root" work. -.\" -.\" Revision 1.1 2000/12/08 17:41:01 rdm -.\" man pages of all ROOT executables provided by Christian Holm. -.\" -.\" diff --git a/net/auth/inc/TAuthenticate.h b/net/auth/inc/TAuthenticate.h index 571abe65604f5..9199f278e44c8 100644 --- a/net/auth/inc/TAuthenticate.h +++ b/net/auth/inc/TAuthenticate.h @@ -17,8 +17,7 @@ // // // TAuthenticate // // // -// An authentication module for ROOT based network services, like rootd // -// and proofd. // +// An authentication module for ROOT based network services, like rootd.// // // // ////////////////////////////////////////////////////////////////////////// @@ -63,7 +62,7 @@ friend class TSocket; TString fDetails; // logon details (method dependent ...) THostAuth *fHostAuth; // pointer to relevant authentication info TString fPasswd; // user's password - TString fProtocol; // remote service (rootd, proofd) + TString fProtocol; // remote service (rootd) Bool_t fPwHash; // kTRUE if fPasswd is a passwd hash TString fRemote; // remote host to which we want to connect Int_t fRSAKey; // Type of RSA key used @@ -85,7 +84,6 @@ friend class TSocket; Bool_t GetUserPasswd(TString &user, TString &passwd, Bool_t &pwhash, Bool_t srppwd); char *GetRandString(Int_t Opt,Int_t Len); - Int_t ProofAuthSetup(); Int_t RfioAuth(TString &user); void SetEnvironment(); Int_t SshAuth(TString &user); @@ -103,7 +101,6 @@ friend class TSocket; static TString fgPasswd; static TPluginHandler *fgPasswdDialog; // Passwd dialog GUI plugin static Bool_t fgPromptUser; // kTRUE if user prompt required - static TList *fgProofAuthInfo; // Specific lists of THostAuth fro proof static Bool_t fgPwHash; // kTRUE if fgPasswd is a passwd hash static Bool_t fgReadHomeAuthrc; // kTRUE to look for $HOME/.rootauthrc static TString fgRootAuthrc; // Path to last rootauthrc-like file read @@ -122,7 +119,6 @@ friend class TSocket; static Bool_t CheckHost(const char *Host, const char *host); static void FileExpand(const char *fin, FILE *ftmp); - static Int_t ProofAuthSetup(TSocket *sock, Bool_t client); static void RemoveSecContext(TRootSecContext *ctx); public: @@ -149,7 +145,6 @@ friend class TSocket; void SetSecContext(TRootSecContext *ctx) { fSecContext = ctx; } static void AuthError(const char *where, Int_t error); - static Bool_t CheckProofAuth(Int_t cSec, TString &det); static Int_t DecodeRSAPublic(const char *rsapubexport, R__rsa_NUMBER &n, R__rsa_NUMBER &d, char **rsassl = nullptr); @@ -170,7 +165,6 @@ friend class TSocket; Option_t *opt = "R", Int_t *Exact = nullptr); static const char *GetKrb5Principal(); static Bool_t GetPromptUser(); - static TList *GetProofAuthInfo(); static Int_t GetRSAInit(); static const char *GetRSAPubExport(Int_t key = 0); static THostAuth *HasHostAuth(const char *host, const char *user, @@ -197,7 +191,6 @@ friend class TSocket; static void SetKrb5AuthHook(Krb5Auth_t func); static void SetPromptUser(Bool_t promptuser); static void SetDefaultRSAKeyType(Int_t key); - static void SetReadHomeAuthrc(Bool_t readhomeauthrc); // for PROOF static void SetRSAInit(Int_t init = 1); static Int_t SetRSAPublic(const char *rsapubexport, Int_t klen); static void SetSecureAuthHook(SecureAuth_t func); diff --git a/net/auth/inc/THostAuth.h b/net/auth/inc/THostAuth.h index 6e906f430d533..5f773709bfb4f 100644 --- a/net/auth/inc/THostAuth.h +++ b/net/auth/inc/THostAuth.h @@ -35,7 +35,7 @@ class THostAuth : public TObject { private: TString fHost; // Host - Char_t fServer; // Server (kSOCKD,kROOTD,kPROOFD) + Char_t fServer; // Server (kSOCKD,kROOTD) TString fUser; // Username Int_t fNumMethods; // Number of AuthMethods Int_t fMethods[kMAXSEC]; // AuthMethods @@ -59,7 +59,6 @@ class THostAuth : public TObject { const char *details); THostAuth(const char *host, Int_t server, const char *user, Int_t authmeth, const char *details); - THostAuth(const char *asstring); THostAuth(THostAuth &ha); virtual ~THostAuth(); diff --git a/net/auth/src/TAuthenticate.cxx b/net/auth/src/TAuthenticate.cxx index 884b21abb0df3..39ecad52c84f7 100644 --- a/net/auth/src/TAuthenticate.cxx +++ b/net/auth/src/TAuthenticate.cxx @@ -13,8 +13,7 @@ // // // TAuthenticate // // // -// An authentication module for ROOT based network services, like rootd // -// and proofd. // +// An authentication module for ROOT based network services, like rootd // // // // ////////////////////////////////////////////////////////////////////////// @@ -99,7 +98,6 @@ TDatime TAuthenticate::fgLastAuthrc; // Time of last reading of fgRoo TString TAuthenticate::fgPasswd; TPluginHandler *TAuthenticate::fgPasswdDialog = (TPluginHandler *)(-1); Bool_t TAuthenticate::fgPromptUser; -TList *TAuthenticate::fgProofAuthInfo = 0; Bool_t TAuthenticate::fgPwHash; Bool_t TAuthenticate::fgReadHomeAuthrc = kTRUE; // on/off search for $HOME/.rootauthrc TString TAuthenticate::fgRootAuthrc; // Path to last rootauthrc-like file read @@ -168,10 +166,6 @@ TAuthenticate::TAuthenticate(TSocket *sock, const char *remote, Info("Authenticate", "locking mutex (pid: %d)",gSystem->GetPid()); R__LOCKGUARD2(gAuthenticateMutex); - // In PROOF decode the buffer sent by the client, if any - if (gROOT->IsProofServ()) - ProofAuthSetup(); - // Use the ID of the starting thread as unique identifier if (fgProcessID < 0) fgProcessID = gSystem->GetPid(); @@ -215,20 +209,6 @@ TAuthenticate::TAuthenticate(TSocket *sock, const char *remote, } servtype = TSocket::kROOTD; } - if (strstr(sproto, "proof") != 0) { - if (rproto < 11) { - fVersion = 4; - if (rproto < 10) { - fVersion = 3; - if (rproto < 8) { - fVersion = 2; - if (rproto < 7) - fVersion = 1; - } - } - } - servtype = TSocket::kPROOFD; - } if (gDebug > 3) Info("TAuthenticate", "service: %s (remote protocol: %d): fVersion: %d", sproto, @@ -312,7 +292,6 @@ TAuthenticate::TAuthenticate(TSocket *sock, const char *remote, Int_t sec = -1; TString tmp = fProtocol; tmp.ReplaceAll("root",4,"",0); - tmp.ReplaceAll("proof",5,"",0); tmp.ReplaceAll("sock",4,"",0); if (!strncmp(tmp.Data(),"up",2)) sec = 0; @@ -363,7 +342,7 @@ void TAuthenticate::CatchTimeOut() } //////////////////////////////////////////////////////////////////////////////// -/// Authenticate to remote rootd or proofd server. Return kTRUE if +/// Authenticate to remote rootd server. Return kTRUE if /// authentication succeeded. Bool_t TAuthenticate::Authenticate() @@ -755,13 +734,6 @@ void TAuthenticate::SetEnvironment() if (!strncasecmp(pt, "yes",3) || !strncmp(pt, "1", 1)) fgPromptUser = kTRUE; - // Set ReUse flag - if (!gROOT->IsProofServ()) { - fgAuthReUse = kTRUE; - if (!strncasecmp(ru, "no",2) || !strncmp(ru, "0",1)) - fgAuthReUse = kFALSE; - } - // Set Expiring date fgExpDate = TDatime(); fgExpDate.Set(fgExpDate.Convert() + hh*3600 + mm*60); @@ -1095,9 +1067,9 @@ Int_t TAuthenticate::GetAuthMethodIdx(const char *meth) //////////////////////////////////////////////////////////////////////////////// /// Static method to prompt for the user name to be used for authentication -/// to rootd or proofd. User is asked to type user name. +/// to rootd. User is asked to type user name. /// Returns user name (which must be deleted by caller) or 0. -/// If non-interactive run (eg ProofServ) returns default user. +/// If non-interactive run returns default user. char *TAuthenticate::PromptUser(const char *remote) { @@ -1135,9 +1107,9 @@ char *TAuthenticate::PromptUser(const char *remote) //////////////////////////////////////////////////////////////////////////////// /// Static method to prompt for the user's passwd to be used for -/// authentication to rootd or proofd. Uses non-echoing command line +/// authentication to rootd. Uses non-echoing command line /// to get passwd. Returns passwd (which must de deleted by caller) or 0. -/// If non-interactive run (eg ProofServ) returns -1 +/// If non-interactive run returns -1 char *TAuthenticate::PromptPasswd(const char *prompt) { @@ -1242,19 +1214,6 @@ TList *TAuthenticate::GetAuthInfo() return fgAuthInfo; } -//////////////////////////////////////////////////////////////////////////////// -/// Static method returning the list with authentication directives -/// to be sent to proof. - -TList *TAuthenticate::GetProofAuthInfo() -{ - R__LOCKGUARD2(gAuthenticateMutex); - - if (!fgProofAuthInfo) - fgProofAuthInfo = new TList; - return fgProofAuthInfo; -} - //////////////////////////////////////////////////////////////////////////////// /// Print error string depending on error code. @@ -1290,7 +1249,7 @@ void TAuthenticate::AuthError(const char *where, Int_t err) } //////////////////////////////////////////////////////////////////////////////// -/// Set global user name to be used for authentication to rootd or proofd. +/// Set global user name to be used for authentication to rootd. void TAuthenticate::SetGlobalUser(const char *user) { @@ -1304,7 +1263,7 @@ void TAuthenticate::SetGlobalUser(const char *user) } //////////////////////////////////////////////////////////////////////////////// -/// Set global passwd to be used for authentication to rootd or proofd. +/// Set global passwd to be used for authentication to rootd. void TAuthenticate::SetGlobalPasswd(const char *passwd) { @@ -1318,7 +1277,7 @@ void TAuthenticate::SetGlobalPasswd(const char *passwd) } //////////////////////////////////////////////////////////////////////////////// -/// Set global passwd hash flag to be used for authentication to rootd or proofd. +/// Set global passwd hash flag to be used for authentication to rootd. void TAuthenticate::SetGlobalPwHash(Bool_t pwhash) { @@ -1326,23 +1285,13 @@ void TAuthenticate::SetGlobalPwHash(Bool_t pwhash) } //////////////////////////////////////////////////////////////////////////////// -/// Set global SRP passwd flag to be used for authentication to rootd or proofd. +/// Set global SRP passwd flag to be used for authentication to rootd. void TAuthenticate::SetGlobalSRPPwd(Bool_t) { ::Error("SetGlobalSRPPwd", "SRP no longer supported by ROOT"); } -//////////////////////////////////////////////////////////////////////////////// -/// Set flag controlling the reading of $HOME/.rootauthrc. -/// In PROOF the administrator may want to switch off private settings. -/// Always true, may only be set false via option to proofd. - -void TAuthenticate::SetReadHomeAuthrc(Bool_t readhomeauthrc) -{ - fgReadHomeAuthrc = readhomeauthrc; -} - //////////////////////////////////////////////////////////////////////////////// /// Set default expiring date for new validity contexts @@ -1851,14 +1800,6 @@ Int_t TAuthenticate::ClearAuth(TString &user, TString &passwd, Bool_t &pwdhash) if (kind == kROOTD_AUTH && stat >= 1) { - if (stat == 5 && fSocket->GetServType() == TSocket::kPROOFD) - // AFS: we cannot reuse the token because remotely the - // daemon token must be re-initialized; for PROOF, we - // just flag the entry as AFS; this allows to skip reusing - // but to keep the session key for password forwarding - fSecContext->SetID("AFS authentication"); - return 1; - } else { fgPasswd = ""; if (kind == kROOTD_ERR) AuthError("ClearAuth", stat); @@ -1889,8 +1830,6 @@ Int_t TAuthenticate::ClearAuth(TString &user, TString &passwd, Bool_t &pwdhash) TString server = "sockd"; if (fProtocol.Contains("root")) server = "rootd"; - if (fProtocol.Contains("proof")) - server = "proofd"; if (stat == kErrConnectionRefused) { if (gDebug > 0) Error("ClearAuth", @@ -1966,11 +1905,10 @@ Int_t TAuthenticate::ClearAuth(TString &user, TString &passwd, Bool_t &pwdhash) //////////////////////////////////////////////////////////////////////////////// /// Sets fUser=user and search fgAuthInfo for the entry pertaining to /// (host,user), setting fHostAuth accordingly. -/// If opt = "P" use fgProofAuthInfo list instead /// If no entry is found fHostAuth is not changed THostAuth *TAuthenticate::GetHostAuth(const char *host, const char *user, - Option_t *opt, Int_t *exact) + Option_t */*opt*/, Int_t *exact) { if (exact) *exact = 0; @@ -2000,10 +1938,6 @@ THostAuth *TAuthenticate::GetHostAuth(const char *host, const char *user, // Check list of auth info for already loaded info about this host TIter *next = new TIter(GetAuthInfo()); - if (!strncasecmp(opt,"P",1)) { - SafeDelete(next); - next = new TIter(GetProofAuthInfo()); - } THostAuth *ai; Bool_t notFound = kTRUE; @@ -2043,11 +1977,10 @@ THostAuth *TAuthenticate::GetHostAuth(const char *host, const char *user, //////////////////////////////////////////////////////////////////////////////// /// Checks if a THostAuth with exact match for {host,user} exists /// in the fgAuthInfo list -/// If opt = "P" use ProofAuthInfo list instead /// Returns pointer to it or 0 THostAuth *TAuthenticate::HasHostAuth(const char *host, const char *user, - Option_t *opt) + Option_t */*opt*/) { if (gDebug > 2) ::Info("TAuthenticate::HasHostAuth", "enter ... %s ... %s", host, user); @@ -2068,10 +2001,6 @@ THostAuth *TAuthenticate::HasHostAuth(const char *host, const char *user, } TIter *next = new TIter(GetAuthInfo()); - if (!strncasecmp(opt,"P",1)) { - SafeDelete(next); - next = new TIter(GetProofAuthInfo()); - } THostAuth *ai; while ((ai = (THostAuth *) (*next)())) { @@ -2200,12 +2129,9 @@ char *TAuthenticate::GetDefaultDetails(int sec, int opt, const char *usr) //////////////////////////////////////////////////////////////////////////////// /// Remove THostAuth instance from the list -void TAuthenticate::RemoveHostAuth(THostAuth * ha, Option_t *opt) +void TAuthenticate::RemoveHostAuth(THostAuth * ha, Option_t */*opt*/) { - if (!strncasecmp(opt,"P",1)) - GetProofAuthInfo()->Remove(ha); - else - GetAuthInfo()->Remove(ha); + GetAuthInfo()->Remove(ha); // ... destroy it delete ha; } @@ -2213,14 +2139,13 @@ void TAuthenticate::RemoveHostAuth(THostAuth * ha, Option_t *opt) //////////////////////////////////////////////////////////////////////////////// /// Print info about the authentication sector. /// If 'opt' contains 's' or 'S' prints information about established TSecContext, -/// else prints information about THostAuth (if 'opt' is 'p' or 'P', prints -/// Proof related information) +/// else prints information about THostAuth void TAuthenticate::Show(Option_t *opt) { TString sopt(opt); - if (sopt.Contains("s",TString::kIgnoreCase)) { + if (sopt.Contains("s", TString::kIgnoreCase)) { // Print established security contexts TIter next(gROOT->GetListOfSecContexts()); @@ -2230,40 +2155,19 @@ void TAuthenticate::Show(Option_t *opt) } else { - ::Info("::Print", - " +--------------------------- BEGIN --------------------------------+"); - ::Info("::Print", - " + +"); - if (sopt.Contains("p",TString::kIgnoreCase)) { - ::Info("::Print", - " + List fgProofAuthInfo has %4d members +", - GetProofAuthInfo()->GetSize()); - ::Info("::Print", - " + +"); - ::Info("::Print", - " +------------------------------------------------------------------+"); - TIter next(GetProofAuthInfo()); - THostAuth *ai; - while ((ai = (THostAuth *) next())) { - ai->Print(); - } - } else { - ::Info("::Print", - " + List fgAuthInfo has %4d members +", - GetAuthInfo()->GetSize()); - ::Info("::Print", - " + +"); - ::Info("::Print", - " +------------------------------------------------------------------+"); - TIter next(GetAuthInfo()); - THostAuth *ai; - while ((ai = (THostAuth *) next())) { - ai->Print(); - ai->PrintEstablished(); - } + ::Info("::Print", " +--------------------------- BEGIN --------------------------------+"); + ::Info("::Print", " + +"); + ::Info("::Print", " + List fgAuthInfo has %4d members +", + GetAuthInfo()->GetSize()); + ::Info("::Print", " + +"); + ::Info("::Print", " +------------------------------------------------------------------+"); + TIter next(GetAuthInfo()); + THostAuth *ai; + while ((ai = (THostAuth *)next())) { + ai->Print(); + ai->PrintEstablished(); } - ::Info("::Print", - " +---------------------------- END ---------------------------------+"); + ::Info("::Print", " +---------------------------- END ---------------------------------+"); } } @@ -2342,8 +2246,7 @@ Int_t TAuthenticate::AuthExists(TString username, Int_t method, const char *opti // NB: not backward compatible with dev version 4.00.02: switch // off 'reuse' for such servers to avoid hanging at this point. Int_t rproto = fSocket->GetRemoteProtocol(); - Bool_t oldsrv = ((fProtocol.BeginsWith("root") && rproto == 9) || - (fProtocol.BeginsWith("proof") && rproto == 8)); + Bool_t oldsrv = ((fProtocol.BeginsWith("root") && rproto == 9)); Int_t stat = 1, kind; if (!oldsrv) { if (fSocket->Recv(stat, kind) < 0) @@ -2412,8 +2315,6 @@ Int_t TAuthenticate::AuthExists(TString username, Int_t method, const char *opti TString server = "sockd"; if (fSocket->GetServType() == TSocket::kROOTD) server = "rootd"; - if (fSocket->GetServType() == TSocket::kPROOFD) - server = "proofd"; if (stat == kErrConnectionRefused) { Error("AuthExists","%s@%s does not accept connections from %s@%s", server.Data(),fRemote.Data(),fUser.Data(),gSystem->HostName()); @@ -3207,7 +3108,6 @@ Int_t TAuthenticate::SendRSAPublicKey(TSocket *socket, Int_t key) /// Read authentication directives from $ROOTAUTHRC, $HOME/.rootauthrc or /// `/system.rootauthrc` and create related THostAuth objects. /// Files are read only if they changed since last reading -/// If 'proofconf' is defined, check also file proofconf for directives Int_t TAuthenticate::ReadRootAuthrc() { @@ -3258,7 +3158,6 @@ Int_t TAuthenticate::ReadRootAuthrc() // THostAuth lists TList *authinfo = TAuthenticate::GetAuthInfo(); - TList *proofauthinfo = TAuthenticate::GetProofAuthInfo(); // Expand File into temporary file name and open it int expand = 1; @@ -3291,8 +3190,6 @@ Int_t TAuthenticate::ReadRootAuthrc() // Now scan file for meaningful directives TList tmpAuthInfo; char line[kMAXPATHLEN]; - Bool_t cont = kFALSE; - TString proofserv; while (fgets(line, sizeof(line), fd) != 0) { // Skip comment lines @@ -3319,26 +3216,6 @@ Int_t TAuthenticate::ReadRootAuthrc() strlcpy(tmp, line, tmpSize); char *nxt = strtok(tmp," "); - if (!strcmp(nxt, "proofserv") || cont) { - - // Building the list of data servers for proof (analyzed at the end) - char *ph = 0; - if (cont) - ph = nxt; - else - ph = strtok(0," "); - while (ph) { - if (*ph != 92) { - proofserv += TString((const char *)ph); - proofserv += TString(" "); - cont = kFALSE; - } else { - cont = kTRUE; - } - ph = strtok(0," "); - } - - } else { TString hostsrv = nxt; TString host = hostsrv; @@ -3354,8 +3231,6 @@ Int_t TAuthenticate::ReadRootAuthrc() srvtyp = TSocket::kSOCKD; else if (server == "1" || server.BeginsWith("root")) srvtyp = TSocket::kROOTD; - else if (server == "2" || server.BeginsWith("proof")) - srvtyp = TSocket::kPROOFD; } // Line with host info directives @@ -3432,7 +3307,6 @@ Int_t TAuthenticate::ReadRootAuthrc() ha->AddMethod(met,det); } } - } if (tmp) delete [] tmp; } // Close file and remove it if temporary @@ -3449,138 +3323,9 @@ Int_t TAuthenticate::ReadRootAuthrc() if (gDebug > 2) TAuthenticate::Show(); - // Now create the list of THostAuth to be sent over to - // the Master/Slaves, if requested ... - TList tmpproofauthinfo; - if (proofserv.Length() > 0) { - char *tmps = new char[proofserv.Length()+1]; - strlcpy(tmps,proofserv.Data(),proofserv.Length()+1); - char *nxt = strtok(tmps," "); - while (nxt) { - TString tmp((const char *)nxt); - Int_t pdd = -1; - // host - TString host; - if ((pdd = tmp.Index(":")) == -1) { - host = tmp; - } else { - host = tmp; - host.Resize(pdd); - if (!host.Length()) - host = "*"; - tmp.Remove(0,pdd+1); - } - // user - TString user; - if ((pdd = tmp.Index(":")) == -1) { - user = tmp; - } else { - user = tmp; - user.Resize(pdd); - if (!user.Length()) - user = "*"; - tmp.Remove(0,pdd+1); - } - // method(s) - TString meth; - Int_t nm = 0, me[kMAXSEC] = {0}, met = -1; - while (tmp.Length() > 0) { - meth = tmp; - if ((pdd = tmp.Index(":")) > -1) - meth.Resize(pdd); - if (meth.Length() > 1) { - // Method passed as string: translate it to number - met = GetAuthMethodIdx(meth.Data()); - if (met == -1 && gDebug > 2) - ::Info("TAuthenticate::ReadRootAuthrc", - "unrecognized method (%s): ",meth.Data()); - } else if (meth.Length() == 1) { - met = atoi(meth.Data()); - if (met > -1 && met < kMAXSEC) - me[nm++] = met; - } - if (pdd > -1) - tmp.Remove(0,pdd+1); - else - tmp.Resize(0); - } - - // Get related THostAuth, if exists, or create a new one - THostAuth *ha = 0; - THostAuth *hatmp = TAuthenticate::GetHostAuth(host,user); - if (!hatmp) { - ha = new THostAuth(host,user,nm,me,0); - } else { - // Create an empty THostAuth - ha = new THostAuth(host,user); - // Update with hatmp info - ha->Update(hatmp); - // ReOrder following new directives - ha->ReOrder(nm,me); - } - // Add to the tmp list - tmpproofauthinfo.Add(ha); - // Go to next - nxt = strtok(0," "); - } - delete [] tmps; - } - - // Update proofauthinfo with new info found - TAuthenticate::MergeHostAuthList(proofauthinfo,&tmpproofauthinfo,"P"); - // Print those, if requested ... - if (gDebug > 2) - TAuthenticate::Show("P"); - return authinfo->GetSize(); } -//////////////////////////////////////////////////////////////////////////////// -/// Check if the authentication method can be attempted for the client. - -Bool_t TAuthenticate::CheckProofAuth(Int_t cSec, TString &out) -{ - Bool_t rc = kFALSE; - const char netrc[2][20] = { "/.netrc", "/.rootnetrc" }; - TString user; - - // Get user logon name - UserGroup_t *pw = gSystem->GetUserInfo(); - if (pw) { - user = TString(pw->fUser); - delete pw; - } else { - ::Info("CheckProofAuth", - "not properly logged on (getpwuid unable to find relevant info)!"); - out = ""; - return rc; - } - - // UsrPwd - if (cSec == (Int_t) TAuthenticate::kClear) { - Int_t i = 0; - for (; i < 2; i++) { - TString infofile = TString(gSystem->HomeDirectory())+TString(netrc[i]); - if (!gSystem->AccessPathName(infofile, kReadPermission)) - rc = kTRUE; - } - if (rc) - out.Form("pt:0 ru:1 us:%s",user.Data()); - } - - if (gDebug > 3) { - if (strlen(out) > 0) - ::Info("CheckProofAuth", - "meth: %d ... is available: details: %s", cSec, out.Data()); - else - ::Info("CheckProofAuth", - "meth: %d ... is NOT available", cSec); - } - - // return - return rc; -} - //////////////////////////////////////////////////////////////////////////////// /// Standard version of CheckSecCtx to be passed to TAuthenticate::AuthExists /// Check if User is matches the one in Ctx @@ -3600,13 +3345,12 @@ Int_t StdCheckSecCtx(const char *user, TRootSecContext *ctx) } //////////////////////////////////////////////////////////////////////////////// -/// Tool for updating fgAuthInfo or fgProofAuthInfo +/// Tool for updating fgAuthInfo /// 'nin' contains list of last input information through (re)reading /// of a rootauthrc-alike file. 'nin' info has priority. /// 'std' is cleaned from inactive members. /// 'nin' members used to update existing members in 'std' are /// removed from 'nin', do that they do not leak -/// opt = "P" for proofauthinfo. void TAuthenticate::MergeHostAuthList(TList *std, TList *nin, Option_t *opt) { @@ -3656,7 +3400,7 @@ void TAuthenticate::MergeHostAuthList(TList *std, TList *nin, Option_t *opt) //////////////////////////////////////////////////////////////////////////////// /// Tool for removing SecContext ctx from THostAuth listed in -/// fgAuthInfo or fgProofAuthInfo +/// fgAuthInfo void TAuthenticate::RemoveSecContext(TRootSecContext *ctx) { @@ -3674,213 +3418,6 @@ void TAuthenticate::RemoveSecContext(TRootSecContext *ctx) } } } - - // proofauthinfo second - TIter nxpa(GetProofAuthInfo()); - while ((ha = (THostAuth *)nxpa())) { - TIter next(ha->Established()); - TRootSecContext *lctx = 0; - while ((lctx = (TRootSecContext *) next())) { - if (lctx == ctx) { - ha->Established()->Remove(ctx); - break; - } - } - } - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Authentication related stuff setup in TProofServ. -/// This is the place where the buffer send by the client / master is -/// decoded. It contains also password information, if the case requires. -/// Return 0 on success, -1 on failure. - -Int_t TAuthenticate::ProofAuthSetup() -{ - static Bool_t done = kFALSE; - - // Only once - if (done) - return 0; - done = kTRUE; - - // Localise the buffer and decode it - const char *p = gSystem->Getenv("ROOTPROOFAUTHSETUP"); - if (!p) { - if (gDebug > 2) - Info("ProofAuthSetup","Buffer not found: nothing to do"); - return 0; - } - TString mbuf = TBase64::Decode(p); - - // Create the message - TMessage *mess = new TMessage((void*)mbuf.Data(), mbuf.Length()+sizeof(UInt_t)); - - // Extract the information - TString user = ""; - TString passwd = ""; - Bool_t pwhash = kFALSE; - Bool_t srppwd = kFALSE; - Int_t rsakey = -1; - *mess >> user >> passwd >> pwhash >> srppwd >> rsakey; - - // Set Globals for later use - TAuthenticate::SetGlobalUser(user); - TAuthenticate::SetGlobalPasswd(passwd); - TAuthenticate::SetGlobalPwHash(pwhash); - TAuthenticate::SetDefaultRSAKeyType(rsakey); - const char *h = gSystem->Getenv("ROOTHOMEAUTHRC"); - if (h) { - Bool_t rha = (Bool_t)(strtol(h, (char **)0, 10)); - TAuthenticate::SetReadHomeAuthrc(rha); - } - - // Extract the list of THostAuth - TList *pha = (TList *)mess->ReadObject(TList::Class()); - if (!pha) { - if (gDebug > 0) - Info("ProofAuthSetup","List of THostAuth not found"); - return 0; - } - - Bool_t master = gROOT->IsProofServ(); - TIter next(pha); - THostAuth *ha = 0; - while ((ha = (THostAuth *)next())) { - - // Check if there is already one compatible - Int_t kExact = 0; - THostAuth *haex = 0; - Bool_t fromProofAI = kFALSE; - if (master) { - // Look first in the proof list - haex = TAuthenticate::GetHostAuth(ha->GetHost(),ha->GetUser(),"P",&kExact); - // If nothing found, look also in the standard list - if (!haex) { - haex = - TAuthenticate::GetHostAuth(ha->GetHost(),ha->GetUser(),"R",&kExact); - } else - fromProofAI = kTRUE; - } else { - // For slaves look first in the standard list only - haex = TAuthenticate::GetHostAuth(ha->GetHost(),ha->GetUser(),"R",&kExact); - } - - if (haex) { - // If yes, action depends on whether it matches exactly or not - if (kExact == 1) { - // Update info in authinfo if Slave or in proofauthinfo - // if Master and the entry was already in proofauthinfo - if (!master || fromProofAI) { - // update this existing one with the information found in - // in the new one, if needed - haex->Update(ha); - // Delete temporary THostAuth - SafeDelete(ha); - } else - // Master, entry not already in proofauthinfo, - // Add it to the list - TAuthenticate::GetProofAuthInfo()->Add(ha); - } else { - // update this new one with the information found in - // in the existing one (if needed) and ... - Int_t i = 0; - for (; i < haex->NumMethods(); i++) { - Int_t met = haex->GetMethod(i); - if (!ha->HasMethod(met)) - ha->AddMethod(met,haex->GetDetails(met)); - } - if (master) - // ... add the new one to the list - TAuthenticate::GetProofAuthInfo()->Add(ha); - else - // We add this one to the standard list - TAuthenticate::GetAuthInfo()->Add(ha); - } - } else { - if (master) - // We add this one to the list for forwarding - TAuthenticate::GetProofAuthInfo()->Add(ha); - else - // We add this one to the standard list - TAuthenticate::GetAuthInfo()->Add(ha); - } - } - - // We are done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Setup of authetication related stuff in PROOF run after a -/// successful authentication. -/// Return 0 on success, -1 on failure. - -Int_t TAuthenticate::ProofAuthSetup(TSocket *sock, Bool_t /* client */) -{ - // Fill some useful info - TSecContext *sc = sock->GetSecContext(); - TString user = sc->GetUser(); - Int_t remoteOffSet = sc->GetOffSet(); - - // send user name to remote host - // for UsrPwd method send also passwd, rsa encoded - TMessage pubkey; - TString passwd = ""; - Bool_t pwhash = kFALSE; - Bool_t srppwd = kFALSE; - - Bool_t upwd = sc->IsA("UsrPwd"); - - TPwdCtx *pwdctx = 0; - if (remoteOffSet > -1 && upwd) - pwdctx = (TPwdCtx *)(sc->GetContext()); - - if (upwd && pwdctx) { - passwd = pwdctx->GetPasswd(); - pwhash = pwdctx->IsPwHash(); - } - - Int_t keytyp = ((TRootSecContext *)sc)->GetRSAKey(); - - // Prepare buffer - TMessage mess; - mess << user << passwd << pwhash << srppwd << keytyp; - - // Add THostAuth info - mess.WriteObject(TAuthenticate::GetProofAuthInfo()); - - // Get buffer as a base 64 string - char *mbuf = mess.Buffer(); - Int_t mlen = mess.Length(); - TString messb64 = TBase64::Encode(mbuf, mlen); - - if (gDebug > 2) - ::Info("ProofAuthSetup","sending %d bytes", messb64.Length()); - - // Send it over - if (remoteOffSet > -1) { - if (TAuthenticate::SecureSend(sock, 1, keytyp, messb64.Data()) == -1) { - ::Error("ProofAuthSetup","problems secure-sending message buffer"); - return -1; - } - } else { - // There is no encryption key: send it plain - char buflen[20]; - snprintf(buflen,20, "%d", messb64.Length()); - if (sock->Send(buflen, kMESS_ANY) < 0) { - ::Error("ProofAuthSetup","plain: problems sending message length"); - return -1; - } - if (sock->SendRaw(messb64.Data(), messb64.Length()) < 0) { - ::Error("ProofAuthSetup","problems sending message buffer"); - return -1; - } - } - - // We are done - return 0; } //////////////////////////////////////////////////////////////////////////////// @@ -3891,385 +3428,3 @@ Int_t TAuthenticate::GetClientProtocol() return TSocket::GetClientProtocol(); } -// -// The code below is needed by TSlave and TProofServ for backward -// compatibility. -// - -//////////////////////////////////////////////////////////////////////////////// -/// Sends the list of the relevant THostAuth objects to the master or -/// to the active slaves, typically data servers external to the proof -/// cluster. The list is of THostAuth to be sent is specified by -/// TAuthenticate::fgProofAuthInfo after directives found in the -/// .rootauthrc family files ('proofserv' key) -/// Returns -1 if a problem sending THostAuth has occured, -2 in case -/// of problems closing the transmission. - -static Int_t SendHostAuth(TSocket *s) -{ - Int_t retval = 0, ns = 0; - - if (!s) { - Error("SendHostAuth","invalid input: socket undefined"); - return -1; - } - - - TIter next(TAuthenticate::GetProofAuthInfo()); - THostAuth *ha; - while ((ha = (THostAuth *)next())) { - TString buf; - ha->AsString(buf); - if((ns = s->Send(buf, kPROOF_HOSTAUTH)) < 1) { - retval = -1; - break; - } - if (gDebug > 2) - Info("SendHostAuth","sent %d bytes (%s)",ns,buf.Data()); - } - - // End of transmission ... - if ((ns = s->Send("END", kPROOF_HOSTAUTH)) < 1) - retval = -2; - if (gDebug > 2) - Info("SendHostAuth","sent %d bytes for closing",ns); - - return retval; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Receive from client/master directives for authentications, create -/// related THostAuth and add them to the TAuthenticate::ProofAuthInfo -/// list. Opt = "M" or "m" if Master, "S" or "s" if Proof slave. -/// The 'proofconf' file is read only if Master - -static Int_t RecvHostAuth(TSocket *s, Option_t *opt) -{ - if (!s) { - Error("RecvHostAuth","invalid input: socket undefined"); - return -1; - } - - // Check if Master - Bool_t master = !strncasecmp(opt,"M",1) ? kTRUE : kFALSE; - - // First read directives from , and alike files - TAuthenticate::ReadRootAuthrc(); - - // Receive buffer - Int_t kind; - char buf[kMAXSECBUF]; - Int_t nr = s->Recv(buf, kMAXSECBUF, kind); - if (nr < 0 || kind != kPROOF_HOSTAUTH) { - Error("RecvHostAuth", "received: kind: %d (%d bytes)", kind, nr); - return -1; - } - if (gDebug > 2) - Info("RecvHostAuth","received %d bytes (%s)",nr,buf); - - while (strcmp(buf, "END")) { - // Clean buffer - Int_t nc = (nr >= kMAXSECBUF) ? kMAXSECBUF - 1 : nr ; - buf[nc] = '\0'; - - // Create THostAuth - THostAuth *ha = new THostAuth((const char *)&buf); - - // Check if there is already one compatible - Int_t kExact = 0; - THostAuth *haex = 0; - Bool_t fromProofAI = kFALSE; - if (master) { - // Look first in the proof list - haex = TAuthenticate::GetHostAuth(ha->GetHost(),ha->GetUser(),"P",&kExact); - // If nothing found, look also in the standard list - if (!haex) { - haex = - TAuthenticate::GetHostAuth(ha->GetHost(),ha->GetUser(),"R",&kExact); - } else - fromProofAI = kTRUE; - } else { - // For slaves look first in the standard list only - haex = TAuthenticate::GetHostAuth(ha->GetHost(),ha->GetUser(),"R",&kExact); - } - - if (haex) { - // If yes, action depends on whether it matches exactly or not - if (kExact == 1) { - // Update info in authinfo if Slave or in proofauthinfo - // if master and the entry was already in proofauthinfo - if (!master || fromProofAI) { - // update this existing one with the information found in - // in the new one, if needed - haex->Update(ha); - // Delete temporary THostAuth - SafeDelete(ha); - } else - // master, entry not already in proofauthinfo, - // Add it to the list - TAuthenticate::GetProofAuthInfo()->Add(ha); - } else { - // update this new one with the information found in - // in the existing one (if needed) and ... - Int_t i = 0; - for (; i < haex->NumMethods(); i++) { - Int_t met = haex->GetMethod(i); - if (!ha->HasMethod(met)) - ha->AddMethod(met,haex->GetDetails(met)); - } - if (master) - // ... add the new one to the list - TAuthenticate::GetProofAuthInfo()->Add(ha); - else - // We add this one to the standard list - TAuthenticate::GetAuthInfo()->Add(ha); - } - } else { - if (master) - // We add this one to the list for forwarding - TAuthenticate::GetProofAuthInfo()->Add(ha); - else - // We add this one to the standard list - TAuthenticate::GetAuthInfo()->Add(ha); - } - - - // Get the next one - nr = s->Recv(buf, kMAXSECBUF, kind); - if (nr < 0 || kind != kPROOF_HOSTAUTH) { - Info("RecvHostAuth","Error: received: kind: %d (%d bytes)", kind, nr); - return -1; - } - if (gDebug > 2) - Info("RecvHostAuth","received %d bytes (%s)",nr,buf); - } - - return 0; -} - -extern "C" { - -//////////////////////////////////////////////////////////////////////////////// -/// Setup of authetication in PROOF run after successful opening -/// of the socket. Provided for backward compatibility. -/// Return 0 on success, -1 on failure. - -Int_t OldSlaveAuthSetup(TSocket *sock, - Bool_t /* master */, TString ord, TString conf) -{ - - // Fill some useful info - TSecContext *sc = sock->GetSecContext(); - TString user = sc->GetUser(); - Int_t proofdProto = sock->GetRemoteProtocol(); - Int_t remoteOffSet = sc->GetOffSet(); - - // send user name to remote host - // for UsrPwd method send also passwd, rsa encoded - TMessage pubkey; - TString passwd = ""; - Bool_t pwhash = kFALSE; - Bool_t srppwd = kFALSE; - - Bool_t upwd = sc->IsA("UsrPwd"); - - TPwdCtx *pwdctx = 0; - if (remoteOffSet > -1 && upwd) - pwdctx = (TPwdCtx *)(sc->GetContext()); - - if (upwd && pwdctx) { - - // Send offset to identify remotely the public part of RSA key - if (sock->Send(remoteOffSet, kROOTD_RSAKEY) != 2*sizeof(Int_t)) { - Error("OldAuthSetup", "failed to send offset in RSA key"); - return -1; - } - - if (pwdctx) { - passwd = pwdctx->GetPasswd(); - pwhash = pwdctx->IsPwHash(); - } - - Int_t keytyp = ((TRootSecContext *)sc)->GetRSAKey(); - if (TAuthenticate::SecureSend(sock, 1, keytyp, passwd.Data()) == -1) { - if (remoteOffSet > -1) - Warning("OldAuthSetup","problems secure-sending pass hash %s", - "- may result in failures"); - // If non RSA encoding available try passwd inversion - if (upwd) { - for (int i = 0; i < passwd.Length(); i++) { - char inv = ~passwd(i); - passwd.Replace(i, 1, inv); - } - TMessage mess; - mess << passwd; - if (sock->Send(mess) < 0) { - Error("OldAuthSetup", "failed to send inverted password"); - return -1; - } - } - } - - } else { - - // Send notification of no offset to be sent ... - if (sock->Send(-2, kROOTD_RSAKEY) != 2*sizeof(Int_t)) { - Error("OldAuthSetup", "failed to send no offset notification in RSA key"); - return -1; - } - } - - // Send ordinal (and config) info to slave (or master) - TMessage mess; - mess << user << pwhash << srppwd << ord << conf; - - if (sock->Send(mess) < 0) { - Error("OldAuthSetup", "failed to send ordinal and config info"); - return -1; - } - - if (proofdProto > 6) { - // Now we send authentication details to access, e.g., data servers - // not in the proof cluster and to be propagated to slaves. - // This is triggered by the 'proofserv ...' - // line in .rootauthrc - if (SendHostAuth(sock) < 0) { - Error("OldAuthSetup", "failed to send HostAuth info"); - return -1; - } - } - - // We are done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Authentication related setup in TProofServ run after successful -/// startup. Provided for backward compatibility. -/// Return 0 on success, -1 on failure. - -Int_t OldProofServAuthSetup(TSocket *sock, Bool_t master, Int_t protocol, - TString &user, TString &ord, TString &conf) -{ - // First receive, decode and store the public part of RSA key - Int_t retval, kind; - if (sock->Recv(retval, kind) != 2*sizeof(Int_t)) { - //other side has closed connection - Info("OldProofServAuthSetup", - "socket has been closed due to protocol mismatch - Exiting"); - return -1; - } - - Int_t rsakey = 0; - TString passwd; - if (kind == kROOTD_RSAKEY) { - - if (retval > -1) { - if (gSystem->Getenv("ROOTKEYFILE")) { - - TString keyfile = gSystem->Getenv("ROOTKEYFILE"); - keyfile += retval; - - FILE *fKey = 0; - char pubkey[kMAXPATHLEN] = { 0 }; - if (!gSystem->AccessPathName(keyfile.Data(), kReadPermission)) { - if ((fKey = fopen(keyfile.Data(), "r"))) { - Int_t klen = fread((void *)pubkey,1,sizeof(pubkey),fKey); - if (klen <= 0) { - Error("OldProofServAuthSetup", - "failed to read public key from '%s'", keyfile.Data()); - fclose(fKey); - return -1; - } - pubkey[klen] = 0; - // Set RSA key - rsakey = TAuthenticate::SetRSAPublic(pubkey,klen); - fclose(fKey); - } else { - Error("OldProofServAuthSetup", "failed to open '%s'", keyfile.Data()); - return -1; - } - } - } - - // Receive passwd - char *pwd = 0; - if (TAuthenticate::SecureRecv(sock, 2, rsakey, &pwd) < 0) { - Error("OldProofServAuthSetup", "failed to receive password"); - return -1; - } - passwd = pwd; - delete[] pwd; - - } else if (retval == -1) { - - // Receive inverted passwd - TMessage *mess; - if ((sock->Recv(mess) <= 0) || !mess) { - Error("OldProofServAuthSetup", "failed to receive inverted password"); - return -1; - } - (*mess) >> passwd; - delete mess; - - for (Int_t i = 0; i < passwd.Length(); i++) { - char inv = ~passwd(i); - passwd.Replace(i, 1, inv); - } - - } - } - - // Receive final information - TMessage *mess; - if ((sock->Recv(mess) <= 0) || !mess) { - Error("OldProofServAuthSetup", "failed to receive ordinal and config info"); - return -1; - } - - // Decode it - Bool_t pwhash, srppwd; - if (master) { - if (protocol < 4) { - (*mess) >> user >> pwhash >> srppwd >> conf; - ord = "0"; - } else { - (*mess) >> user >> pwhash >> srppwd >> ord >> conf; - } - } else { - if (protocol < 4) { - Int_t iord; - (*mess) >> user >> pwhash >> srppwd >> iord; - ord = "0."; - ord += iord; - } else { - (*mess) >> user >> pwhash >> srppwd >> ord >> conf; - } - } - delete mess; - - // Set Globals for later use - TAuthenticate::SetGlobalUser(user); - TAuthenticate::SetGlobalPasswd(passwd); - TAuthenticate::SetGlobalPwHash(pwhash); - TAuthenticate::SetDefaultRSAKeyType(rsakey); - const char *h = gSystem->Getenv("ROOTHOMEAUTHRC"); - if (h) { - Bool_t rha = (Bool_t)(strtol(h, (char **)0, 10)); - TAuthenticate::SetReadHomeAuthrc(rha); - } - - // Read user or system authentication directives and - // receive auth info transmitted from the client - Int_t harc = master ? RecvHostAuth(sock, "M") : RecvHostAuth(sock, "S"); - - if (harc < 0) { - Error("OldProofServAuthSetup", "failed to receive HostAuth info"); - return -1; - } - - // We are done - return 0; -} - -} // extern "C" diff --git a/net/auth/src/THostAuth.cxx b/net/auth/src/THostAuth.cxx index 0b364b212d4a6..d7491b84dc2e7 100644 --- a/net/auth/src/THostAuth.cxx +++ b/net/auth/src/THostAuth.cxx @@ -43,8 +43,8 @@ ClassImp(THostAuth); /// Create hostauth object. /// 'host' may contain also the server for whicb these directives /// are valid in the form 'host:server' or 'server://host' -/// with server either "sock[d]", "root[d]", "proof[d]" or -/// 0, 1, 2, respectively. +/// with server either "sock[d]", "root[d]" or +/// 0, 1, respectively. THostAuth::THostAuth(const char *host, const char *user, Int_t nmeth, Int_t *authmeth, char **details) : TObject() @@ -56,8 +56,8 @@ THostAuth::THostAuth(const char *host, const char *user, Int_t nmeth, /// Create hostauth object. /// 'host' may contain also the server for whicb these directives /// are valid in the form 'host:server' or 'server://host' -/// with server either "sock[d]", "root[d]", "proof[d]" or -/// 0, 1, 2, respectively. +/// with server either "sock[d]", "root[d]" or +/// 0, 1, respectively. THostAuth::THostAuth(const char *host, Int_t server, const char *user, Int_t nmeth, Int_t *authmeth, char **details) : TObject() @@ -94,8 +94,8 @@ THostAuth::THostAuth(const char *host, Int_t server, const char *user, /// Create hostauth object. /// 'host' may contain also the server for whicb these directives /// are valid in the form 'host:server' or 'server://host' -/// with server either "sock[d]", "root[d]", "proof[d]" or -/// 0, 1, 2, respectively. +/// with server either "sock[d]", "root[d]" or +/// 0, 1, respectively. void THostAuth::Create(const char *host, const char *user, Int_t nmeth, Int_t *authmeth, char **details) @@ -123,8 +123,6 @@ void THostAuth::Create(const char *host, const char *user, Int_t nmeth, fServer = TSocket::kSOCKD; else if (srv == "1" || srv.BeginsWith("root")) fServer = TSocket::kROOTD; - else if (srv == "2" || srv.BeginsWith("proof")) - fServer = TSocket::kPROOFD; } // Check and save the host FQDN ... @@ -184,72 +182,6 @@ void THostAuth::Create(const char *host, const char *user, Int_t nmeth, fActive = kTRUE; } - -//////////////////////////////////////////////////////////////////////////////// -/// Create hostauth object from directives given as a compact string -/// See THostAuth::AsString(). -/// Used in proof context only; fServer not set; to be set by hand -/// with SetServer() method if really needed - -THostAuth::THostAuth(const char *asstring) : TObject() -{ - fServer = -1; - - TString strtmp(asstring); - char *tmp = strdup(asstring); - - fHost = TString((const char *)strtok(tmp," ")); - strtmp.ReplaceAll(fHost,""); - fHost.Remove(0,fHost.Index(":")+1); - - fUser = TString((const char *)strtok(0," ")); - strtmp.ReplaceAll(fUser,""); - fUser.Remove(0,fUser.Index(":")+1); - - TString fNmet; - fNmet = TString((const char *)strtok(0," ")); - strtmp.ReplaceAll(fNmet,""); - fNmet.Remove(0,fNmet.Index(":")+1); - - free(tmp); - - fNumMethods = atoi(fNmet.Data()); - Int_t i = 0; - for (; i < fNumMethods; i++) { - TString det = strtmp; - det.Remove(0,det.Index("'")+1); - det.Resize(det.Index("'")); - // Remove leading spaces, if - char cmet[20]; - sscanf(det.Data(),"%10s",cmet); - Int_t met = atoi(cmet); - if (met > -1 && met < kMAXSEC) { - det.ReplaceAll(cmet,""); - while (det.First(' ') == 0) - det.Remove(0,1); - while (det.Last(' ') == (det.Length() - 1)) - det.Resize(det.Length() - 1); - fMethods[i] = met; - fSuccess[i] = 0; - fFailure[i] = 0; - fDetails[i] = det; - } - strtmp.Remove(0,strtmp.Index("'",strtmp.Index("'")+1)+1); - } - for (i = fNumMethods; i < kMAXSEC ; i++) { - fMethods[i] = -1; - fSuccess[i] = -1; - fFailure[i] = -1; - } - - // List of TSecContext - fSecContexts = new TList; - - // Active when created - fActive = kTRUE; -} - - //////////////////////////////////////////////////////////////////////////////// /// Copy ctor ... @@ -423,10 +355,10 @@ void THostAuth::SetDetails(Int_t level, const char *details) void THostAuth::Print(Option_t *proc) const { - char srvnam[5][8] = { "any", "sockd", "rootd", "proofd", "???" }; + char srvnam[4][8] = { "any", "sockd", "rootd", "???" }; - Int_t isrv = (fServer >= -1 && fServer <= TSocket::kPROOFD) ? - fServer+1 : TSocket::kPROOFD+2; + Int_t isrv = (fServer >= -1 && fServer <= TSocket::kROOTD) ? + fServer+1 : TSocket::kROOTD+2; Info("Print", "%s +------------------------------------------------------------------+",proc); diff --git a/net/auth/src/TRootAuth.cxx b/net/auth/src/TRootAuth.cxx index 984155a5ba97b..401ce66034e39 100644 --- a/net/auth/src/TRootAuth.cxx +++ b/net/auth/src/TRootAuth.cxx @@ -33,10 +33,9 @@ /// Returns 1 on success, 0 on failure. TSecContext *TRootAuth::Authenticate(TSocket *s, const char *host, - const char *user, Option_t *opts) + const char *user, Option_t */*opts*/) { TSecContext *ctx = 0; - Int_t rc = 0; Int_t rproto = s->GetRemoteProtocol() % 1000; if (s->GetServType() == (Int_t)TSocket::kROOTD) { @@ -52,15 +51,11 @@ TSecContext *TRootAuth::Authenticate(TSocket *s, const char *host, } } - Bool_t isPROOF = (s->GetServType() == (Int_t)TSocket::kPROOFD); - Bool_t isPROOFserv = (opts[0] == 'P') ? kTRUE : kFALSE; - // Build the protocol string for TAuthenticate TString proto = TUrl(s->GetUrl()).GetProtocol(); if (proto == "") { proto = "root"; - } else if (proto.Contains("sockd") || proto.Contains("rootd") || - proto.Contains("proofd")) { + } else if (proto.Contains("sockd") || proto.Contains("rootd")) { proto.ReplaceAll("d",1,"",0); } proto += Form(":%d",rproto); @@ -78,12 +73,7 @@ TSecContext *TRootAuth::Authenticate(TSocket *s, const char *host, else Error("Authenticate", "authentication failed for %s@%s", auth->GetUser(), host); - // This is to terminate properly remote proofd in case of failure - if (isPROOF) - s->Send(Form("%d %s", gSystem->GetPid(), host), kROOTD_CLEANUP); } else { - // Set return flag; - rc = 1; // Search pointer to relevant TSecContext ctx = auth->GetSecContext(); s->SetSecContext(ctx); @@ -91,15 +81,6 @@ TSecContext *TRootAuth::Authenticate(TSocket *s, const char *host, // Cleanup delete auth; - // If we are talking to a recent proofd send over a buffer with the - // remaining authentication related stuff - if (rc && isPROOF && rproto > 11) { - Bool_t client = !isPROOFserv; - if (TAuthenticate::ProofAuthSetup(s, client) !=0 ) { - Error("Authenticate", "PROOF: failed to finalize setup"); - } - } - // We are done return ctx; } diff --git a/net/auth/src/TRootSecContext.cxx b/net/auth/src/TRootSecContext.cxx index 06a2fd6eb26dd..c3ac5b9cd7130 100644 --- a/net/auth/src/TRootSecContext.cxx +++ b/net/auth/src/TRootSecContext.cxx @@ -128,11 +128,9 @@ Bool_t TRootSecContext::CleanupSecContext(Bool_t all) Int_t srvtyp = nscc->GetType(); Int_t rproto = nscc->GetProtocol(); Int_t level = 2; - if ((srvtyp == TSocket::kROOTD && rproto < 10) || - (srvtyp == TSocket::kPROOFD && rproto < 9)) + if ((srvtyp == TSocket::kROOTD && rproto < 10)) level = 1; - if ((srvtyp == TSocket::kROOTD && rproto < 8) || - (srvtyp == TSocket::kPROOFD && rproto < 7)) + if ((srvtyp == TSocket::kROOTD && rproto < 8)) level = 0; if (level) { Int_t port = nscc->GetPort(); @@ -140,11 +138,7 @@ Bool_t TRootSecContext::CleanupSecContext(Bool_t all) TSocket *news = new TSocket(fHost.Data(),port,-1); if (news && news->IsValid()) { - if (srvtyp == TSocket::kPROOFD) { - news->SetOption(kNoDelay, 1); - news->Send("cleaning request"); - } else - news->SetOption(kNoDelay, 0); + news->SetOption(kNoDelay, 0); // Backward compatibility: send socket size if (srvtyp == TSocket::kROOTD && level == 1) @@ -164,7 +158,7 @@ Bool_t TRootSecContext::CleanupSecContext(Bool_t all) } } if (cleaned && gDebug > 2) { - char srvname[3][10] = {"sockd", "rootd", "proofd"}; + char srvname[3][10] = {"sockd", "rootd"}; Info("CleanupSecContext", "remote %s notified for cleanup (%s,%d)", srvname[srvtyp],fHost.Data(),port); @@ -187,7 +181,7 @@ Bool_t TRootSecContext::CleanupSecContext(Bool_t all) /// If opt is "" print in special form for calls within THostAuth /// with cardinality "" /// If opt is "S" prints short in-line form for calls within TFTP, -/// TSlave, TProof ... +/// TSlave ... void TRootSecContext::Print(Option_t *opt) const { diff --git a/net/net/inc/NetErrors.h b/net/net/inc/NetErrors.h index 7b5018baf36a0..3426b5ea12f79 100644 --- a/net/net/inc/NetErrors.h +++ b/net/net/inc/NetErrors.h @@ -17,7 +17,7 @@ // // // NetErrors // // // -// This header file defines error codes generated by rootd/proofd. // +// This header file defines error codes generated by rootd. // // // ////////////////////////////////////////////////////////////////////////// diff --git a/net/net/inc/TSecContext.h b/net/net/inc/TSecContext.h index dc3cb57355cf2..7b427d3ae7ef1 100644 --- a/net/net/inc/TSecContext.h +++ b/net/net/inc/TSecContext.h @@ -110,7 +110,7 @@ class TSecContextCleanup : public TObject { private: Int_t fPort; Int_t fServerProtocol; - Int_t fServerType; // 0 = sockd, 1 = rootd, 2 = proofd + Int_t fServerType; // 0 = sockd, 1 = rootd public: TSecContextCleanup(Int_t port, Int_t proto, Int_t type) : diff --git a/net/net/inc/TSocket.h b/net/net/inc/TSocket.h index 6f4fcbee79339..d79ac9b25579e 100644 --- a/net/net/inc/TSocket.h +++ b/net/net/inc/TSocket.h @@ -41,15 +41,13 @@ class THostAuth; class TSocket : public TNamed { friend class TServerSocket; -friend class TProofServ; // to be able to call SetDescriptor(), RecvHostAuth() -friend class TSlave; // to be able to call SendHostAuth() public: enum EStatusBits { kIsUnix = BIT(16), // set if unix socket kBrokenConn = BIT(17) // set if conn reset by peer or broken }; enum EInterest { kRead = 1, kWrite = 2 }; - enum EServiceType { kSOCKD, kROOTD, kPROOFD }; + enum EServiceType { kSOCKD, kROOTD }; protected: enum ESocketErrors { diff --git a/net/net/inc/TUDPSocket.h b/net/net/inc/TUDPSocket.h index c8a01bf9564c1..36606633985b7 100644 --- a/net/net/inc/TUDPSocket.h +++ b/net/net/inc/TUDPSocket.h @@ -43,7 +43,7 @@ friend class TServerSocket; kBrokenConn = BIT(17) // set if conn reset by peer or broken }; enum EInterest { kRead = 1, kWrite = 2 }; - enum EServiceType { kSOCKD, kROOTD, kPROOFD }; + enum EServiceType { kSOCKD, kROOTD }; protected: TInetAddress fAddress; // remote internet address and port # diff --git a/net/net/src/NetErrors.cxx b/net/net/src/NetErrors.cxx index e460c5671e79f..a356a38a22272 100644 --- a/net/net/src/NetErrors.cxx +++ b/net/net/src/NetErrors.cxx @@ -14,7 +14,7 @@ // NetErrors // // // // This file defines error strings mapped to the error codes generated // -// by rootd/proofd. // +// by rootd. // // // ////////////////////////////////////////////////////////////////////////// diff --git a/net/net/src/TApplicationServer.cxx b/net/net/src/TApplicationServer.cxx index bcd8c9693c105..610f6f46b5c16 100644 --- a/net/net/src/TApplicationServer.cxx +++ b/net/net/src/TApplicationServer.cxx @@ -1023,7 +1023,7 @@ Int_t TApplicationServer::BrowseKey(const char *keyname) } //////////////////////////////////////////////////////////////////////////////// -/// Terminate the proof server. +/// Terminate the server. void TApplicationServer::Terminate(Int_t status) { diff --git a/net/net/src/TSQLMonitoring.cxx b/net/net/src/TSQLMonitoring.cxx index 8053bc38e8ad1..1e5fe25e6845f 100644 --- a/net/net/src/TSQLMonitoring.cxx +++ b/net/net/src/TSQLMonitoring.cxx @@ -1,4 +1,3 @@ -// @(#)root/proofplayer:$Id$ // Author: J.F. Grosse-Oetringhaus, G.Ganis /************************************************************************* diff --git a/net/net/src/TSecContext.cxx b/net/net/src/TSecContext.cxx index 575c8c4300f1e..61c14f197d033 100644 --- a/net/net/src/TSecContext.cxx +++ b/net/net/src/TSecContext.cxx @@ -243,8 +243,7 @@ Bool_t TSecContext::IsActive() const /// If opt is "F" (default) print object content. /// If opt is "" print in special form for calls within THostAuth /// with cardinality "" -/// If opt is "S" prints short in-line form for calls within TFTP, -/// TSlave, TProof ... +/// If opt is "S" prints short in-line form for calls within TFTP and similar void TSecContext::Print(Option_t *opt) const { diff --git a/net/net/src/TSocket.cxx b/net/net/src/TSocket.cxx index af2f9eb8222fa..c6ecbc6685b4d 100644 --- a/net/net/src/TSocket.cxx +++ b/net/net/src/TSocket.cxx @@ -53,7 +53,7 @@ ULong64_t TSocket::fgBytesRecv = 0; // 10: added support for authenticated socket via TSocket::CreateAuthSocket(...) // 11: modified SSH protocol + support for server 'no authentication' mode // 12: add random tags to avoid reply attacks (password+token) -// 13: authentication re-organization; cleanup in PROOF +// 13: LEGACY: authentication re-organization; cleanup in PROOF // 14: support for SSH authentication via SSH tunnel // 15: cope with fixes in TUrl::GetFile // 16: add env setup message exchange @@ -86,8 +86,6 @@ TSocket::TSocket(TInetAddress addr, const char *service, Int_t tcpwindowsize) fServType = kSOCKD; if (fService.Contains("root")) fServType = kROOTD; - if (fService.Contains("proof")) - fServType = kPROOFD; fAddress = addr; fAddress.fPort = gSystem->GetServiceByName(service); fBytesSent = 0; @@ -131,8 +129,6 @@ TSocket::TSocket(TInetAddress addr, Int_t port, Int_t tcpwindowsize) fServType = kSOCKD; if (fService.Contains("root")) fServType = kROOTD; - if (fService.Contains("proof")) - fServType = kPROOFD; fAddress = addr; fAddress.fPort = port; SetTitle(fService); @@ -174,8 +170,6 @@ TSocket::TSocket(const char *host, const char *service, Int_t tcpwindowsize) fServType = kSOCKD; if (fService.Contains("root")) fServType = kROOTD; - if (fService.Contains("proof")) - fServType = kPROOFD; fAddress = gSystem->GetHostByName(host); fAddress.fPort = gSystem->GetServiceByName(service); SetName(fAddress.GetHostName()); @@ -222,8 +216,6 @@ TSocket::TSocket(const char *url, Int_t port, Int_t tcpwindowsize) fServType = kSOCKD; if (fUrl.Contains("root")) fServType = kROOTD; - if (fUrl.Contains("proof")) - fServType = kPROOFD; fAddress = gSystem->GetHostByName(host); fAddress.fPort = port; SetName(fAddress.GetHostName()); @@ -1107,27 +1099,12 @@ Bool_t TSocket::Authenticate(const char *user) { Bool_t rc = kFALSE; - // Parse protocol name, for PROOF, send message with server role + // Parse protocol name TString sproto = TUrl(fUrl).GetProtocol(); if (sproto.Contains("sockd")) { fServType = kSOCKD; } else if (sproto.Contains("rootd")) { fServType = kROOTD; - } else if (sproto.Contains("proofd")) { - fServType = kPROOFD; - // Parse options - TString opt(TUrl(fUrl).GetOptions()); - //First letter in Opt describes type of proofserv to invoke - if (!strncasecmp(opt, "S", 1)) { - if (Send("slave") < 0) return rc; - } else if (!strncasecmp(opt, "M", 1)) { - if (Send("master") < 0) return rc; - } else { - Warning("Authenticate", - "called by TSlave: unknown option '%c' %s", - opt[0], " - assuming Slave"); - if (Send("slave") < 0) return rc; - } } if (gDebug > 2) Info("Authenticate","Local protocol: %s",sproto.Data()); @@ -1192,7 +1169,7 @@ Bool_t TSocket::Authenticate(const char *user) if (gDebug > 1) Info("Authenticate", "class for '%s' authentication loaded", alib.Data()); - Option_t *opts = (gROOT->IsProofServ()) ? "P" : ""; + Option_t *opts = ""; if (!(auth->Authenticate(this, host, user, opts))) { Error("Authenticate", "authentication attempt failed for %s@%s", user, host.Data()); @@ -1249,22 +1226,17 @@ Bool_t TSocket::Authenticate(const char *user) /// Creates a socket or a parallel socket and authenticates to the /// remote server. /// -/// url: [[proto][p][auth]://][user@]host[:port][/service][?options] +/// url: [[proto][p][auth]://][user@]host[:port][/service] /// -/// where proto = "sockd", "rootd", "proofd" +/// where proto = "sockd", "rootd" /// indicates the type of remote server; /// if missing "sockd" is assumed ("sockd" indicates /// any remote server session using TServerSocket) -/// [p] = for parallel sockets (forced internally for -/// rootd; ignored for proofd) /// [auth] = "up" or "k" to force UsrPwd or Krb5 authentication /// [port] = is the remote port number /// [service] = service name used to determine the port /// (for backward compatibility, specification of /// port as priority) -/// options = "m" or "s", when proto=proofd indicates whether -/// we are master or slave (used internally by -/// TSlave) /// /// An already opened connection can be used by passing its socket /// in opensock. @@ -1313,9 +1285,8 @@ TSocket *TSocket::CreateAuthSocket(const char *url, Int_t size, Int_t tcpwindows proto.Resize(proto.Length()-1); } - // Find out if parallel (ignore if proofd, force if rootd) - if (((proto.EndsWith("p") || size > 1) && - !proto.BeginsWith("proof")) || + // Find out if parallel (force if rootd) + if ((proto.EndsWith("p") || size > 1) || proto.BeginsWith("root") ) { parallel = kTRUE; if (proto.EndsWith("p")) @@ -1323,7 +1294,7 @@ TSocket *TSocket::CreateAuthSocket(const char *url, Int_t size, Int_t tcpwindows } // Force "sockd" if the rest is not recognized - if (!proto.BeginsWith("sock") && !proto.BeginsWith("proof") && + if (!proto.BeginsWith("sock") && !proto.BeginsWith("root")) proto = "sockd"; @@ -1396,17 +1367,13 @@ TSocket *TSocket::CreateAuthSocket(const char *url, Int_t size, Int_t tcpwindows /// Creates a socket or a parallel socket and authenticates to the /// remote server specified in 'url' on remote 'port' as 'user'. /// -/// url: [[proto][p][auth]://]host[/?options] +/// url: [[proto][auth]://]host /// -/// where proto = "sockd", "rootd", "proofd" +/// where proto = "sockd", "rootd" /// indicates the type of remote server /// if missing "sockd" is assumed ("sockd" indicates /// any remote server session using TServerSocket) -/// [p] = for parallel sockets (forced internally for -/// rootd) /// [auth] = "up" or "k" to force UsrPwd or Krb5 authentication -/// [options] = "m" or "s", when proto=proofd indicates whether -/// we are master or slave (used internally by TSlave) /// /// An already opened connection can be used by passing its socket /// in opensock. diff --git a/net/net/src/TUDPSocket.cxx b/net/net/src/TUDPSocket.cxx index 144f170fbfc32..c8d824189b884 100644 --- a/net/net/src/TUDPSocket.cxx +++ b/net/net/src/TUDPSocket.cxx @@ -64,8 +64,6 @@ TUDPSocket::TUDPSocket(TInetAddress addr, const char *service) fServType = kSOCKD; if (fService.Contains("root")) fServType = kROOTD; - if (fService.Contains("proof")) - fServType = kPROOFD; fAddress = addr; fAddress.fPort = gSystem->GetServiceByName(service); fBytesSent = 0; @@ -110,8 +108,6 @@ TUDPSocket::TUDPSocket(TInetAddress addr, Int_t port) fServType = kSOCKD; if (fService.Contains("root")) fServType = kROOTD; - if (fService.Contains("proof")) - fServType = kPROOFD; fAddress = addr; fAddress.fPort = port; SetTitle(fService); @@ -153,8 +149,6 @@ TUDPSocket::TUDPSocket(const char *host, const char *service) fServType = kSOCKD; if (fService.Contains("root")) fServType = kROOTD; - if (fService.Contains("proof")) - fServType = kPROOFD; fAddress = gSystem->GetHostByName(host); fAddress.fPort = gSystem->GetServiceByName(service); SetName(fAddress.GetHostName()); @@ -201,8 +195,6 @@ TUDPSocket::TUDPSocket(const char *url, Int_t port) fServType = kSOCKD; if (fUrl.Contains("root")) fServType = kROOTD; - if (fUrl.Contains("proof")) - fServType = kPROOFD; fAddress = gSystem->GetHostByName(host); fAddress.fPort = port; SetName(fAddress.GetHostName()); diff --git a/net/rpdutils/res/rpdconn.h b/net/rpdutils/res/rpdconn.h index ab15909e86d63..32ac8da5f4b89 100644 --- a/net/rpdutils/res/rpdconn.h +++ b/net/rpdutils/res/rpdconn.h @@ -17,7 +17,7 @@ // rpdconn // // // // This header file contains the definition of some utility classes // -// used for process communication between xproofd, rootd, proofexecv. // +// used for process communication between rootd. // // // ////////////////////////////////////////////////////////////////////////// diff --git a/net/rpdutils/res/rpddefs.h b/net/rpdutils/res/rpddefs.h index 5d4122f3801a6..fc09d26961d8c 100644 --- a/net/rpdutils/res/rpddefs.h +++ b/net/rpdutils/res/rpddefs.h @@ -37,6 +37,6 @@ const unsigned int kDMN_SYSLOG = 0x4; // Log messages to syslog i.o. stderr // // type of service -enum EService { kSOCKD = 0, kROOTD, kPROOFD }; +enum EService { kSOCKD = 0, kROOTD }; #endif diff --git a/net/rpdutils/res/rpderr.h b/net/rpdutils/res/rpderr.h index cb510dcc6ca99..665db0b2bc410 100644 --- a/net/rpdutils/res/rpderr.h +++ b/net/rpdutils/res/rpderr.h @@ -17,7 +17,7 @@ // // // rpderr // // // -// This header file defines error codes generated by rootd/proofd. // +// This header file defines error codes generated by rootd. // // NB: In case of change update also strings in net/inc/ErrStr.h // // // ////////////////////////////////////////////////////////////////////////// diff --git a/net/rpdutils/res/rpdp.h b/net/rpdutils/res/rpdp.h index 47922c4c25a0c..ba2a6a881f758 100644 --- a/net/rpdutils/res/rpdp.h +++ b/net/rpdutils/res/rpdp.h @@ -83,10 +83,6 @@ void DaemonStart(int ignsigcld, int fdkeep, EService service); // rpdutils.cxx void RpdAuthCleanup(const char *sstr, int opt); int RpdGenRSAKeys(int); -int RpdGetAuthProtocol(); -const char *RpdGetKeyRoot(); -int RpdGetClientProtocol(); -int RpdGetOffSet(); int RpdInitSession(int, std::string &, int &); int RpdInitSession(int, std::string &, int &, int &, std::string &); int RpdInitSession(int, std::string &, int &, int &, int &, std::string &); diff --git a/net/rpdutils/src/DaemonUtils.cxx b/net/rpdutils/src/DaemonUtils.cxx index 78bd4b2d81cda..d8b7e352e4d4c 100644 --- a/net/rpdutils/src/DaemonUtils.cxx +++ b/net/rpdutils/src/DaemonUtils.cxx @@ -573,7 +573,7 @@ namespace ROOT { fflush(stdout); // Actions are defined by the specific error handler ( - // see rootd.cxx and proofd.cxx) + // see rootd.cxx) if (func) (*func)(code,(const char *)buf, sizeof(buf)); } diff --git a/net/rpdutils/src/daemon.cxx b/net/rpdutils/src/daemon.cxx index 0d25986dc466a..f43656f567829 100644 --- a/net/rpdutils/src/daemon.cxx +++ b/net/rpdutils/src/daemon.cxx @@ -167,7 +167,7 @@ void DaemonStart(int ignsigcld, int fdkeep, EService service) out: // Close any open file descriptors for (fd = 0; fd < NOFILE; fd++) { - if ((fd != fdkeep) || (service == kPROOFD)) close(fd); + if ((fd != fdkeep)) close(fd); } ResetErrno(); // probably got set to EBADF from a close diff --git a/net/rpdutils/src/error.cxx b/net/rpdutils/src/error.cxx index 67980e399155e..b13b25647e2a9 100644 --- a/net/rpdutils/src/error.cxx +++ b/net/rpdutils/src/error.cxx @@ -15,7 +15,6 @@ // error // // // // Set of error handling routines for daemon process. // -// Merging of rootd and proofd/src/error.cxx // // // ////////////////////////////////////////////////////////////////////////// @@ -119,7 +118,7 @@ void Error(ErrorHandler_t func, int code, const char *va_(fmt), ...) } // Actions are defined by the specific error handler function - // (see rootd.cxx and proofd.cxx) + // (see rootd.cxx) if (func) (*func)(code,(const char *)buf, sizeof(buf)); } diff --git a/net/rpdutils/src/net.cxx b/net/rpdutils/src/net.cxx index b532848985762..f7c184fc3e723 100644 --- a/net/rpdutils/src/net.cxx +++ b/net/rpdutils/src/net.cxx @@ -472,8 +472,6 @@ int NetInit(EService servtype, int port1, int port2, int tcpwindowsize) if (!sp) { if (servtype == kROOTD) { port1 = 1094; - } else if (servtype == kPROOFD) { - port1 = 1093; } else { fprintf(stderr,"NetInit: unknown service: %s/tcp\n", service.data()); Error(gErrFatal, kErrFatal, diff --git a/net/rpdutils/src/rpdconn.cxx b/net/rpdutils/src/rpdconn.cxx index 7fb6fa8941440..e1009e50d4669 100644 --- a/net/rpdutils/src/rpdconn.cxx +++ b/net/rpdutils/src/rpdconn.cxx @@ -14,7 +14,7 @@ // rpdconn // // // // This header file contains the definition of some utility classes // -// used for process communication between xproofd, rootd, proofexecv. // +// used for process communication between rootd. // // // ////////////////////////////////////////////////////////////////////////// diff --git a/net/rpdutils/src/rpdutils.cxx b/net/rpdutils/src/rpdutils.cxx index 9ed8f3ce270ba..afcdb004aeff8 100644 --- a/net/rpdutils/src/rpdutils.cxx +++ b/net/rpdutils/src/rpdutils.cxx @@ -13,7 +13,7 @@ // // // rpdutils // // // -// Set of utilities for rootd/proofd daemon authentication. // +// Set of utilities for rootd daemon authentication. // // // ////////////////////////////////////////////////////////////////////////// @@ -221,7 +221,7 @@ ErrorHandler_t gErrSys = 0; ErrorHandler_t gErrFatal = 0; ErrorHandler_t gErr = 0; bool gSysLog = 0; -std::string gServName[3] = { "sockd", "rootd", "proofd" }; +std::string gServName[3] = { "sockd", "rootd" }; // // Local global consts @@ -493,41 +493,6 @@ void RpdSetMethInitFlag(int methinit) if (gDebug > 2) ErrorInfo("RpdSetMethInitFlag: gMethInit set to %d", gMethInit); } -//////////////////////////////////////////////////////////////////////////////// -/// Return pointer to the root string for key files -/// Used by proofd. - -const char *RpdGetKeyRoot() -{ - return (const char *)gRpdKeyRoot.c_str(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return protocol version run by the client. -/// Used by proofd. - -int RpdGetClientProtocol() -{ - return gClientProtocol; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return authentication protocol used for the handshake. -/// Used by proofd. - -int RpdGetAuthProtocol() -{ - return gAuthProtocol; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return offset in the authtab file. -/// Used by proofd. - -int RpdGetOffSet() -{ - return gOffSet; -} //////////////////////////////////////////////////////////////////////////////// @@ -796,7 +761,7 @@ int RpdUpdateAuthTab(int opt, const char *line, char **token, int ilck) retval = lseek(itab, 0, SEEK_END); // Save first RSA public key into file for later use by the - // same or other rootd/proofd; we will update the tab file + // same or other rootd; we will update the tab file // only if this operation is successful int ntry = 10; int rs = 0; @@ -2401,7 +2366,7 @@ int RpdPass(const char *pass, int errheq) SPrintf(line, kMAXPATHLEN, "0 1 %d %d %s %s", gRSAKey, gRemPid, gOpenHost.c_str(), gUser); - if (!afs_auth || gService == kPROOFD) + if (!afs_auth) offset = RpdUpdateAuthTab(1, line, &token); if (gDebug > 2) ErrorInfo("RpdPass: got offset %d", offset); @@ -2603,8 +2568,7 @@ int RpdCheckDaemon(const char *daemon) cmd[i++] = ch; } else { cmd[i] = '\0'; - if (strstr(cmd, "grep") == 0 && strstr(cmd, "rootd") == 0 - && strstr(cmd, "proofd") == 0) { + if (strstr(cmd, "grep") == 0 && strstr(cmd, "rootd") == 0) { cnt++; } i = 0; @@ -3196,7 +3160,7 @@ int RpdGetRSAKeys(const char *pubkey, int Opt) } //////////////////////////////////////////////////////////////////////////////// -/// Save RSA public key into file for later use by other rootd/proofd. +/// Save RSA public key into file for later use by other rootd. /// Return: 0 if ok /// 1 if not ok /// 2 if not ok because file already exists and cannot be @@ -3802,7 +3766,7 @@ int RpdAuthenticate() } } - // Guess the client procotol if not received via Rootd/ProofdProtocol + // Guess the client procotol if not received via Rootd if (gClientProtocol == 0) gClientProtocol = RpdGuessClientProt(buf, kind); @@ -4219,7 +4183,7 @@ int RpdInitSession(int servtype, std::string &user, /// - Inquire protocol /// - authenticate the client /// - login the client -/// Returns 1 for a PROOF master server, 0 otherwise +/// Returns 0 in presence of a server. /// Returns logged-in user, the remote client procotol cproto, the /// client kind of user anon and, if anonymous user, the client passwd. /// If TServerSocket (servtype==kSOCKD), the protocol number is returned @@ -4240,21 +4204,6 @@ int RpdInitSession(int servtype, std::string &user, // Get Host name NetGetRemoteHost(gOpenHost); - if (servtype == kPROOFD) { - - // find out if we are supposed to be a master or a slave server - char msg[80]; - if (NetRecv(msg, sizeof(msg)) < 0) { - ErrorInfo("RpdInitSession: Cannot receive master/slave status"); - return -1; - } - - retval = !strcmp(msg, "master") ? 1 : 0; - - if (gDebug > 0) - ErrorInfo("RpdInitSession: PROOF master/slave = %s", msg); - } - // Get protocol first // Failure typically indicate special actions like cleanup // which do not need additional work @@ -4284,7 +4233,7 @@ int RpdInitSession(int servtype, std::string &user, auth = RpdNoAuth(servtype); } - // Login the user (if in rootd/proofd environment) + // Login the user (if in rootd environment) if (gDoLogin > 0) { if (RpdLogin(servtype,auth) != 0) { ErrorInfo("RpdInitSession: unsuccessful login attempt"); @@ -4323,7 +4272,7 @@ int RpdInitSession(int servtype, std::string &user, /// - Inquire protocol /// - authenticate the client /// - login the client -/// Returns 1 for a PROOF master server, 0 otherwise +/// Returns 0 in presence of a master server. /// Returns logged-in user and remote process id in rid /// Called just after opening the connection @@ -4351,7 +4300,7 @@ int RpdNoAuth(int servtype) int auth = 0; // Receive target username - if (servtype == kROOTD || servtype == kPROOFD) { + if (servtype == kROOTD) { char buf[kMAXPATHLEN]; EMessageTypes kind; diff --git a/proof/CMakeLists.txt b/proof/CMakeLists.txt deleted file mode 100644 index 74c5f7eea9e38..0000000000000 --- a/proof/CMakeLists.txt +++ /dev/null @@ -1,11 +0,0 @@ -# Copyright (C) 1995-2019, Rene Brun and Fons Rademakers. -# All rights reserved. -# -# For the licensing terms see $ROOTSYS/LICENSE. -# For the list of contributors see $ROOTSYS/README/CREDITS. - -add_subdirectory(proof) -add_subdirectory(proofplayer) # special CMakeListst.txt -if(NOT WIN32) - add_subdirectory(proofbench) # special CMakeListst.txt -endif() diff --git a/proof/doc/confman/ConfigProofPoD.md b/proof/doc/confman/ConfigProofPoD.md deleted file mode 100644 index 14301009be4de..0000000000000 --- a/proof/doc/confman/ConfigProofPoD.md +++ /dev/null @@ -1,194 +0,0 @@ -Setup a static PROOF cluster with PROOF on Demand -================================================= - -Introduction ------------- - -Using PROOF on Demand is our current recommended way of running a PROOF -cluster. The usage of PoD is in particular helpful for the following -reasons: - -- **Sandboxing.** Each user get their own personal PROOF cluster, - separated from the others: a problem occurring on one personal - cluster does not affect the workflow of other users. - -- **Easier administration and self-servicing.** A user can restart their - personal PROOF cluster in case of troubles without waiting for a - system administrator's intervention. - -- **Efficient multiuser scheduling.** PROOF on Demand makes PROOF run on - top of an existing resource management system, moving the problem of - scheduling many concurrent users outside of PROOF. - -This guide particularly refers to the setup of a static PROOF cluster -running on physical hosts: the recommended setup is in practice the same -as the ready-to-go Virtual Analysis Facility. If you want to use PROOF -on the clouds there is no configuration to go through. - -Setup a resource management system ----------------------------------- - -Although PROOF on Demand can run on a cluster of nodes without using a -resource management system (using `pod-ssh`), it is recommended to setup a -dedicated one to benefit from the scheduling in a multiuser environment, or a -dedicated queue on an existing one. - -As there's a variety of resource management systems, this guide does not cover -their setup. The RMS preconfigured for the Virtual Analysis Facility is -[HTCondor](http://research.cs.wisc.edu/htcondor/), which we recommend primarily -because it has dynamic addition of workers built in. - -Configuration steps for all nodes ---------------------------------- - -### Setup CernVM-FS - -[CernVM-FS](http://cernvm.cern.ch/portal/filesystem) should be installed -on all machines as the preferred method for software distribution. - -> Configuration instructions for the latest CernVM-FS can be found -> [here](http://cernvm.cern.ch/portal/filesystem/techinformation). - -A brief step-by-step procedure to install CernVM-FS is hereby described. - -- Download and install the latest stable version from - [here](http://cernvm.cern.ch/portal/filesystem): pick one which is - appropriate to your operating system. You need the `cvmfs` package, - you *don't* need the `cvmfs-devel` or `cvmfs-server` ones. - -- As root user, run: - - # cvmfs_config setup - -- Start the `autofs` service: how to to this depends on your operating - system. - - On Ubuntu using Upstart: - - # restart autofs - - On RHEL-based or older Ubuntus: - - # service autofs restart - -- Prepare a `/etc/cvmfs/default.local` file (create it if it does not - exists) with the following configuration bits: - - ``` {.bash} - CVMFS_HTTP_PROXY=http://your-proxy-server.domain.ch:3128,DIRECT - CVMFS_REPOSITORIES=your-experiment.cern.ch,sft.cern.ch - CVMFS_QUOTA_LIMIT=50000 - ``` - - You need to properly specify your closest HTTP caching proxy: - separate many of them via commas. The last fallback value, `DIRECT`, - tells cvmfs to connect directly without using any proxy at all. - - Among the list of repositories (comma-separated), always specify - `sft.cern.ch` and the one containing the software to your experiment - (e.g., `cms.cern.ch`). - - The quota limit is, in Megabytes, the amount of local disk space to - use as cache. - -- Check the configuration and repositories with: - - # cvmfs_config chksetup - OK - # cvmfs_config probe - Probing /cvmfs/cms.cern.ch... OK - Probing /cvmfs/sft.cern.ch... OK - -> You might need special configurations for some custom software -> repositories! Special cases are not covered in this guide. - -### Firewall configuration - -[PROOF on Demand](http://pod.gsi.de/) is very flexible in handling -various cases of network topologies. The best solution would be to allow -all TCP communications between the cluster machines. - -No other incoming communication is required from the outside. - -Configuration steps for the head node only ------------------------------------------- - -### Setup HTTPS+SSH (sshcertauth) authentication - -> Latest recommended sshcertauth version is 0.8.5. -> -> [Download](https://github.com/dberzano/sshcertauth/archive/v0.8.5.zip) -> and [read the -> instructions](http://newton.ph.unito.it/~berzano/w/doku.php?id=proof:sshcertauth). - -If you want your users to connect to the PROOF cluster using their Grid -user certificate and private key you might be interested in installing -sshcertauth. Please refer to the [installation -guide](http://newton.ph.unito.it/~berzano/w/doku.php?id=proof:sshcertauth) -for further information. - -### PROOF on Demand - -> Latest recommended PROOF on Demand version is 3.12. -> -> **On CernVM-FS:** `/cvmfs/sft.cern.ch/lcg/external/PoD/3.12` -> -> **Source code:** [PoD download page](http://pod.gsi.de/download.html) -> and [Installation -> instructions](http://pod.gsi.de/doc/3.12/Installation.html) - -[PROOF on Demand](http://pod.gsi.de/) is required on the head node and on the -user's client. - -In case your experiment provides a version of PoD on CernVM-FS you can use -that one. Experiment-independent versions are available from the PH-SFT -cvmfs repository. - -Only if you have specific reasons while you want to use a customly built -PoD version, download the source code and compile it using the -installation instructions. - -Please note that [CMake](http://www.cmake.org/) and -[Boost](http://www.boost.org/) are required to build PoD. - -- After you have built PoD, install it with: - - make install - -- After installing PoD, run: - - pod-server getbins - - This has to be done only once and downloads the binary packages that - will be dynamically transferred to the worker nodes as binary - payload, and prevents us from installing PoD on each cluster node. - - It is important to do this step now, because in case PoD has been - installed in a directory where the user has no write privileges, as - in the case of system-wide installations, the user won't be able to - download those required packages in the PoD binary directory. - -> There is no need to "configure" PoD for your specific cluster: it is -> just enough to install it on your head node. -> -> PoD does not have any system-wide persistent daemon running or any -> system-wide configuration to be performed. Also, no part of PoD will -> be ever run as root. -> -> Do not worry about environment or software configuration at this time: -> there is no system configuration for that. All the environment for -> your software dependencies will be set via proper scripts from the PoD -> client. -> -> PoD client configuration and running is properly covered in the -> appropriate manual page. - -### Firewall configuration - -The head node only requires **TCP ports 22 (SSH) and 443 (HTTPS)** to accept -connections from the outside. Users will get an authentication "token" -from port 443 and all PROOF traffic will be automatically tunneled in a -SSH connection on port 22 by PoD. - -In case you are not using the HTTPS+SSH token+authentication method, access to -the sole port 22 is all you need. diff --git a/proof/doc/confman/DatasetStager.md b/proof/doc/confman/DatasetStager.md deleted file mode 100644 index 1ca55af56c668..0000000000000 --- a/proof/doc/confman/DatasetStager.md +++ /dev/null @@ -1,179 +0,0 @@ -The Dataset Stager -================== - -Overview --------- - -The [Dataset Stager (afdsmgrd)](http://afdsmgrd.googlecode.com/) is -a daemon that coordinates the transfer of data from a remote storage -to your local storage. - -For each file to transfer, a script is called. The script can be -customized to support your source and destination protocol. - -Staging requests are issued from the ROOT console, where you can also -control the progress of your staging. - -Installation ------------- - -The Dataset Stager is distributed both on a repository on its own and as -part of ROOT. The easiest way to compile it is to do it inside ROOT. - -Installing from ROOT --------------------- - -When configuring the ROOT source, enable the Dataset Stager by adding -`--enable-afdsmgrd`. Check in the list of enabled features if you have -"afdsmgrd". - -After running `make` (and, optionally, `make install`) you'll find the -daemon in the same directory of `root.exe`. - -The configuration file and init.d startup script will be in -`$ROOTSYS/etc/proof`. The daemon can and **must** run as unprivileged -user. - -Configuration -------------- - -The Dataset Stager can share its configuration file with PROOF, as -some directives are the same and unknown directives are just ignored. - -Directives are one per line and lines beginning with a pound sign (`#`) -are used for comments. - -> The configuration file is automatically checked at each loop: this -> means you can change configuration without restarting the daemon or -> stopping your current transfers. - -A detailed description of each directive follows. - -set *VARIABLE=value* -: This statement will substitute every occurrence of `$VARIABLE` with - its *value* in the rest of the configuration file. You can have - multiple `set` statements. - -xpd.stagereqrepo [dir:]*directory* -: This directive is shared with PROOF: *directory* is the full path to - the dataset repository. **Defaults to empty:** without this - directive the daemon is not operative. - - The `dir:` prefix is optional. - -dsmgrd.purgenoopds *true|false* -: Set it to *true* **(default is false)** to remove a dataset when no file to stage - is found. If no file to stage is found, but corrupted files exist, the - dataset is kept to signal failures. Used in combination with `xpd.stagereqrepo` - makes it "disposable": only the datasets effectively needed for signaling - the staging status will be kept, improving scalability and stability. - -dsmgrd.urlregex *regex* *subst* -: Each source URL present in the datasets will be matched to *regex* - and substituted to *subst*. *regex* supports grouping using - parentheses, and groups can be referenced in order using the dollar - sign with a number (`$1` for instance) in *subst*. - - Matching and substitution for multiple URL schemas are supported by - using in addition directives `dsmgrd.urlregex2` up to - `dsmgrd.urlregex4` which have the same syntax of this one. - - Example of URL translation via regexp: - - > - Configuration line: - > - > dsmgrd.urlregex alien://(.*)$ root://xrd.cern.ch/$1 - > - > - Source URL: - > - > alien:///alice/data/2012/LHC12b/000178209/ESDs/pass1/12000178209061.17/AliESDs.root - > - > - Resulting URL: - > - > root://xrd.cern.ch//alice/data/2012/LHC12b/000178209/ESDs/pass1/12000178209061.17/AliESDs.root - > -dsmgrd.sleepsecs *secs* -: Seconds to sleep between each loop. The dataset stager checks at - each loop the status of the managed transfers. Defaults to **30 - seconds**. - -dsmgrd.scandseveryloops *n* -: Every `n` loops, the dataset repository is checked for newly - incoming staging requests. Defaults to **10**. - -dsmgrd.parallelxfrs *n* -: Number of concurrent transfers. Defaults to **8**. - -dsmgrd.stagecmd *shell\_command* -: Command to run in order to stage each file. It might be whatever you - want (executable, shell script...). If you add `$URLTOSTAGE` and/or - `$TREENAME` in the *shell\_command*, they'll be substituted - respectively with the destination URL and the default ROOT tree name - in the file (as specified in the dataset staging request from ROOT). - - An example: - - dsmgrd.stagecmd /path/to/afdsmgrd-xrd-stage-verify.sh "$URLTOSTAGE" "$TREENAME" - - Return value of the command is ignored: standard output is - considered, as explained here. - - Defaults to `/bin/false`. - -dsmgrd.cmdtimeoutsecs *secs* -: Timeout on staging command, expressed in seconds: after this - timeout, the command is considered failed and it is killed (in first - place with `SIGSTOP`, then if it is unresponsive with `SIGKILL`). - Defaults to **0 (no timeout)**. - -dsmgrd.corruptafterfails *n* -: Set this to a number above zero to tell the daemon to mark files as - corrupted after a certain number of either download or verification - failures. A value of **0 (default)** tells the daemon to retry - forever. - -Configuring the MonALISA monitoring plugin ------------------------------------------- - -The Dataset Stager supports generic monitoring plugins. The only plugin -distributed with the stager is the MonALISA monitoring plugin. - -dsmgrd.notifyplugin */path/to/libafdsmgrd\_notify\_apmon.so* -: Set it to the path of the MonALISA plugin shared object. By default, - notification plugin is disabled. - -dsmgrd.apmonurl *apmon://apmon.cern.ch* -: This variable tells the ApMon notification plugin how to contact one - or more MonALISA server(s) to activate monitoring via ApMon. It - supports two kinds of URLs: - - - `http[s]://host/path/configuration_file.conf` (a remote file - where to fetch the list of servers from) - - - `apmon://[:password@]monalisahost[:8884]` (a single server to - contact directly) - - If the variable is not set, yet the plugin is loaded, MonALISA - monitoring is inhibited until a valid configuration variable is - provided. - -dsmgrd.apmonprefix *MY::CLUSTER::PREFIX* -: Since MonALISA organizes information in "clusters" and "hosts", here - you can specify what to use as cluster prefix for monitoring - datasets information and daemon status. If this variable is not set, - MonALISA monitoring is inhibited. Please note that the suffix - `_datasets` or `_status` is appended for each of the two types of - monitoring. - -A sample configuration file ---------------------------- - - xpd.stagereqrepo /opt/aaf/var/proof/datasets - dsmgrd.purgenoopds true - dsmgrd.urlregex alien://(.*)$ /storage$1 - dsmgrd.sleepsecs 20 - dsmgrd.scandseveryloops 30 - dsmgrd.parallelxfrs 10 - dsmgrd.stagecmd /opt/aaf/bin/af-xrddm-verify.sh "$URLTOSTAGE" "$TREENAME" - dsmgrd.cmdtimeoutsecs 3600 - dsmgrd.corruptafterfails 0 diff --git a/proof/doc/confman/DeployVirtualAnalysisFacility.md b/proof/doc/confman/DeployVirtualAnalysisFacility.md deleted file mode 100644 index a4f5430d817c9..0000000000000 --- a/proof/doc/confman/DeployVirtualAnalysisFacility.md +++ /dev/null @@ -1,220 +0,0 @@ -Deploying the Virtual Analysis Facility -======================================= - -Introduction ------------- - -Thanks to CernVM and PROOF on Demand, it is possible to deploy a ready -to use Virtual Analysis Facility on your cloud (either public, private -or even your desktop computer). - -On the server side, "configuring" the Virtual Analysis Facility is -simply a matter of starting a certain number of CernVM virtual machines -that will become part of your PROOF cluster. CernVM uses -contextualization to specialize each virtual machine to be either a head -node or a worker node. - -The Virtual Analysis Facility comes with many preconfigured things: - -- a HTCondor cluster capable of running PROOF on Demand - -- certificate authentication - -- your experiment's software (if available on CernVM-FS) - -Obtain the CernVM image and contextualization ---------------------------------------------- - -### Download the CernVM bare image - -The Virtual Analysis Facility currently works with *CernVM Batch 2.7.1 -64-bit*. This means that you need to have this CernVM image available -either on your local hard disk (in case of a desktop deployment) or in -your cloud's image repository. - -> For convenience we provide the direct link for the working versions: -> -> - [CernVM 2.7.1 batch 64-bit for -> **KVM**](https://cernvm.cern.ch/releases/19/cernvm-batch-node-2.7.1-2-3-x86_64.hdd.gz) -> -> - [CernVM 2.7.1 batch 64-bit for -> **Xen**](https://cernvm.cern.ch/releases/19/cernvm-batch-node-2.7.1-2-3-x86_64.ext3.gz) -> -> Images are gzipped. In most cases you'll need to gunzip them before -> registering to your image repository. - -### Create VM configuration profiles - -CernVM images are base images supporting boot-time customization via -configuration profiles called "contexts". Context creation can be -performed through the [CernVM Online](https://cernvm-online.cern.ch/) -website. The site is immediately accessible if you have a CERN account. - -Go to your [CernVM Online -Dashboard](https://cernvm-online.cern.ch/dashboard), click on the -**Create new context...** dropdown and select **Virtual Analysis Facility -node**. - -There's only a few parameters to configure. - -Context name -: A name for your context (such as *VAF Master for ATLAS*). Any name - will work. - -Role -: Use this to configure either a *master* or a *slave*. - -VAF master (only available when configuring a slave) -: IP address or FQDN of the Virtual Analysis Facility master. - -Auth method -: Choose between *ALICE LDAP* (useful only for ALICE users) or *Pool - accounts* (good for authenticating all the other Grid users). - -Num. pool accounts (only available when using pool accounts auth) -: Number of pool accounts to create. - -Proxy for CVMFS -: An URL specifying the proxy server for CernVM-FS, such as - `http://ca-proxy.cern.ch:3128/`. If you leave it empty, proxy will - be automatically discovered. - -HTCondor shared secret -: VMs part of the same cluster should have the same value of this - field. It is used to mutually authenticate VMs and it is used like a - password. - -Context password -: Current profile will be saved on the [CernVM Online - repository](http://cernvm-online.cern.ch/). If you don't want the - information there to be publicly available to other users, type in - a value for protecting the context with an encryption password. - -You will have to create a profile for the **master** and the **slave**. Since -most of the configuration variables are the same (like the *HTCondor -shared secret*) you can create one, clone it and change only what's -needed to change. - -Deploy it on the cloud ----------------------- - -Provided you have access to a certain cloud API, you'll need to -instantiate a certain number of CernVM batch images with proper -contextualization: one for the master, as many as you want as slaves. - -CernVM supports contextualization through the "user data" field -supported by all cloud infrastructures. - -Each cloud infrastructure has a different method of setting the "user -data". The following description will focus on: - -- [OpenNebula](http://opennebula.org/) - -- OpenStack (such as the [CERN Agile - infrastructure](https://openstack.cern.ch/)) - -- [Amazon EC2](http://aws.amazon.com/ec2/)-compatible interfaces via - the open [Eucalyptus](http://www.eucalyptus.com/) - [Euca2ools](http://www.eucalyptus.com/eucalyptus-cloud/tools): many popular - clouds support such interface and tools - -### Download the CernVM Online contextualizations - -Go to the CernVM Online Dashboard page where you have previously -customized the contexts for your master and your slaves. - -Click on the rightmost button on the line of the desired context and -select **Get rendered context** from the dropdown: save the output to a -text file (such as `my_vaf_context.txt`, the name we will use in the -examples that follow). This file will be subsequently passed as the so -called "user-data" file to the cloud API. - -> Repeat the operation for both the master context and the slave -> context. - -### OpenStack API: nova - -Example of a CernVM instantiation using `nova`: - -``` {.bash} -nova boot \ - --flavor m1.xlarge \ - --image cernvm-batch-node-2.6.0-4-1-x86_64 \ - --key-name my_default_keyparir \ - --user-data my_vaf_context.txt \ - Name-Of-My-New-VM -``` - -The `--user-data` option requires the context file we've just -downloaded. - -### EC2 API: euca-tools - -Example of a CernVM instantiation using `euca-tools`: - -``` {.bash} -euca-run-instances \ - --instance-type m1.xlarge \ - --key my_default_keyparir \ - --user-data-file my_vaf_context.txt \ - cernvm-batch-node-2.6.0-4-1-x86_64 -``` - -The `--user-data-file` option is the context file we've just downloaded. - -### OpenNebula - -An example VM definition follows: - -``` {.ruby} -CONTEXT=[ - EC2_USER_DATA="", -] -CPU="6" -VCPU="6" -DISK=[ - IMAGE="cernvm-batch-node-2.6.0-4-1-x86_64", - TARGET="vda" ] -MEMORY="16000" -NAME="CernVM-VAF-Node" -NIC=[ - NETWORK="My-OpenNebula-VNet" ] -OS=[ - ARCH="x86_64" ] -``` - -The `` requires the base64 version of the whole -downloaded context definition. You can obtain it by running: - - cat my_vaf_context.txt | base64 | tr -d '\n' - -Network security groups ------------------------ - -In order to make the Virtual Analysis Facility work properly, the -firewall of your infrastructure must be configured to allow some -connections. - -Some ports need to allow "external" connections while other ports might -be safely opened to allow only connections from other nodes of the -Virtual Analysis Facility. - -### Ports to open on all nodes - -HTCondor ports -: Allow **TCP and UDP range 9600-9700** only between nodes of the Virtual - Analysis Facility. - -Only HTCondor and PoD communication is needed between the nodes. No HTCondor -ports need to be opened to the world. - -### Additional ports to open on the front end node - -HTTPS -: Allow **TCP 443** from all - -SSH -: Allow **TCP 22** from all - -No other ports need to be opened from the outside. Your definition of -*allow from all* might vary. diff --git a/proof/doc/confman/Makefile b/proof/doc/confman/Makefile deleted file mode 100644 index dcd913d9572b2..0000000000000 --- a/proof/doc/confman/Makefile +++ /dev/null @@ -1,53 +0,0 @@ -# -# Makefile for creating HTML out of Markdown for the PROOF configuration -# manual. -# -# Pandoc is used for the conversion. Thanks to Olivier Couet for the Markdown -# ROOT documentation generation scripts. -# -# Pandoc sample installation on OSX (using Homebrew): -# -# $> brew install haskell-platform -# $> cabal install pandoc -# $> export PATH=$HOME/.cabal/bin:$PATH -# - -PANDOC_OPTS := -s -S --toc --chapters --number-sections -f markdown -H css/github.css -PANDOC_OPTS_INDEX := -s -S -f markdown -H css/github.css - -INPUT_MDS := \ - ConfigProofPoD.md \ - DeployVirtualAnalysisFacility.md \ - UsingVirtualAnalysisFacility.md \ - DatasetStager.md - -INDEX_MD := index.md -INDEX_HTML := index.html - -OUTPUT_HTML = $(INPUT_MDS:.md=.html) - -.PHONY: all html clean - -all: html - -html: idx $(OUTPUT_HTML) - -idx: - @echo Generating HTML index - @(echo "Index" > $(INDEX_MD) ; \ - echo "=====" >> $(INDEX_MD) ; \ - echo "" >> $(INDEX_MD) ; \ - for Md in $(INPUT_MDS) ; do \ - echo "1. [$$(head -n1 $$Md)]($${Md%.*}.html)" ; \ - done >> $(INDEX_MD) ) - @pandoc $(PANDOC_OPTS_INDEX) $(INDEX_MD) -o $(INDEX_HTML) - -%.html: %.md - @echo Generating HTML: $@ - @(OFFSET=$$(for F in $(INPUT_MDS) ; do echo $$F ; done | grep -n $< | cut -d: -f1) ; \ - let OFFSET-- ; \ - pandoc $(PANDOC_OPTS) --number-offset $$OFFSET $< -o $@) - -clean: - @echo Cleaning up - @rm -f $(OUTPUT_HTML) $(INDEX_MD) $(INDEX_HTML) diff --git a/proof/doc/confman/UsingVirtualAnalysisFacility.md b/proof/doc/confman/UsingVirtualAnalysisFacility.md deleted file mode 100644 index 5ff30df111bd3..0000000000000 --- a/proof/doc/confman/UsingVirtualAnalysisFacility.md +++ /dev/null @@ -1,379 +0,0 @@ -Using the Virtual Analysis Facility -=================================== - -Introduction ------------- - -The Virtual Analysis Facility can be easily used by having installed on -your client the following software: - -- [ROOT](http://root.cern.ch/) - -- [PROOF on Demand](http://pod.gsi.de/) - -- The VAF client *(see below)*: a convenience tool that sets up the - environment for your experiment's software both on your client and - on the PROOF worker nodes - -> If you are the end user, you'll probably might skip the part that -> concerns how to configure the VAF client: your system administrator -> has probably and conveniently set it up for you. - -The Virtual Analysis Facility client ------------------------------------- - -The Virtual Analysis Facility client takes care of setting the -environment for the end user required by your software's experiment. The -environment will both be set on the client and on each PROOF node. - -Technically it is a Bash shell script which provides shortcuts for PROOF -on Demand commands and ensures local and remote environment consistency: -by executing it you enter a new clean environment where all your -software dependencies have already been set up. - -Local and remote environment configuration is split into a series of -files, which give the possibility to: - -- have a system-wide, sysadmin-provided experiment configuration - -- execute user actions either *before* or *after* the execution of the - system-wide script (for instance, choosing the preferred version of - the experiment's software) - -- transfer a custom user **payload** on each PROOF worker (for instance, - user's client-generated Grid credentials to make PROOF workers - capable of accessing a remote authenticated storage) - -Configuration files are searched for in two different locations: - -- a system-wide directory: `/etc` - -- user's home directory: `~/.vaf` - -> A system-wide configuration file always has precedence over user's -> configuration. It is thus possible for the sysadmin to enforce a -> policy where some scripts cannot ever be overridden. - -Thanks to this separation, users can maintain an uncluttered directory -with very simple configuration files that contain only what really needs -or is allowed to be customized: for instance, user might specify a single line -containing the needed ROOT version, while all the technicalities to set -up the environment are taken care of inside system-installed scripts, -leaving the user's configuration directory clean and uncluttered. - -### Local environment configuration - -All the local environment files are loaded at the time of the -client's startup following a certain order - -- `common.before` - -- `local.before` - -- `local.conf` - -- `$VafConf_LocalPodLocation/PoD_env.sh` - -- `common.after` - -- `local.after` - -The `common.*` files are sourced both for the local and the remote -environment. This might be convenient to avoid repeating the same -configuration in different places. - -Each file is looked for first in the system-wide directory and then in -the user's directory. If a configuration file does not exist, it is -silently skipped. - -The `$VafConf_LocalPodLocation/PoD_env.sh` environment script, provided -with each PROOF on Demand installation, *must exist*: without this file, -the VAF client won't start. - -### List of VAF-specific variables - -There are some special variables that need to be set in one of the above -configuration files. - -`$VafConf_LocalPodLocation` -: Full path to the PoD installation on the client. - - > The `$VafConf_LocalPodLocation` variable must be set before the - > `PoD_env.sh` script gets sourced, so set it either in - > `common.before`, `local.before` or `local.conf`. Since PoD is - > usually system-wide installed, its location is normally - > system-wide set in either the `local.conf` file by the system - > administrator. - -`$VafConf_RemotePodLocation` -: Full path to the PoD installation on the VAF master node. - - *Note: this variable should be set in the configuration files for - the local environment despite it refers to a software present on the - remote nodes.* - -`$VafConf_PodRms` *(optional)* -: Name of the Resource Management System used for submitting PoD jobs. - Run `pod-submit -l` to see the possible values. - - If not set, defaults to `condor`. - -`$VafConf_PodQueue` *(optional)* -: Queue name where to submit PoD jobs. - - If no queue has been given, the default one configured on your RMS - will be used. - -### Remote environment configuration - -All the PoD commands sent to the VAF master will live in the environment -loaded via using the following scripts. - -Similarly to the local environment, configuration is split in different files -to allow for a system-wide configuration, which has precedence over -user's configuration in the home directory. If a script cannot be found, -it will be silently skipped. - -- `` - -- `common.before` - -- `remote.before` - -- `remote.conf` - -- `common.after` - -- `remote.after` - -For an explanation on how to pass extra data to the workers safely -through the payload, see below. - -### Payload: sending local files to the remote nodes - -In many cases it is necessary to send some local data to the remote -workers: it is very common, for instance, to distribute a local Grid -authentication proxy on the remote workers to let them authenticate to -access a data storage. - -The `payload` file must be an executable generating some output that -will be prepended to the remote environment preparation. Differently -than the other environment scripts, it is not executed: instead, it is -first run, then *the output it produces will be executed*. - -Let's see a practical example to better understand how it works. We need -to send our Grid proxy to the master node. - -This is our `payload` executable script: - -``` {.bash} -#!/bin/bash -echo "echo '`cat /tmp/x509up_u$UID | base64 | tr -d '\r\n'`'" \ - "| base64 -d > /tmp/x509up_u\$UID" -``` - -This script will be executed locally, providing another "script line" as -output: - -``` {.bash} -echo 'VGhpcyBpcyB0aGUgZmFrZSBjb250ZW50IG9mIG91ciBHcmlkIHByb3h5IGZpbGUuCg==' | base64 -d > /tmp/x509up_u$UID -``` - -This line will be prepended to the remote environment script and will be -executed before anything else on the remote node: it will effectively -decode the Base64 string back to the proxy file and write it into the -`/tmp` directory. Note also that the first `$UID` is not escaped and -will be substituted *locally* with your user ID *on your client -machine*, while the second one has the dollar escaped (`\$UID`) and will -be substituted *remotely* with your user ID *on the remote node*. - -> It is worth noting that the remote environment scripts will be sent to -> the remote node using a secure connection (SSH), thus there is no -> concern in placing sensitive user data there. - -Installing the Virtual Analysis Facility client ------------------------------------------------ - -### Download the client from Git - -The Virtual Analysis Facility client is available on -[GitHub](https://github.com/dberzano/virtual-analysis-facility): - -``` {.bash} -git clone git://github.com/dberzano/virtual-analysis-facility.git /dest/dir -``` - -The client will be found in `/dest/dir/client/bin/vaf-enter`: it is -convenient to add it to the `$PATH` so that the users might simply start -it by typing `vaf-enter`. - -### Install the experiment's configuration files system-wide - -A system administrator might find convenient to install the experiment -environment scripts system-wide. - -Configuration scripts for LHC experiments are shipped with the VAF -client and can be found in -`/dest/dir/client/config-samples/`. To make them used -by default by the VAF client, place them in the `/dest/dir/etc` -directory like this: - -``` {.bash} -rsync -a /dest/dir/client/config-samples// /dest/dir/etc/ -``` - -Remember that the trailing slash in the source directory name has a -meaning in `rsync` and must not be omitted. - -> Remember that system-wide configuration files will always have -> precedence over user's configuration files, so *don't place there -> files that are supposed to be provided by the user!* - -Entering the Virtual Analysis Facility environment --------------------------------------------------- - -The Virtual Analysis Facility client is a wrapper around commands sent -to the remote host by means of PROOF on Demand's `pod-remote`. The VAF -client takes care of setting up passwordless SSH from your client node -to the VAF master. - -### Getting the credentials - -> You can skip this paragraph if the remote server wasn't configured for -> HTTPS+SSH authentication. - -In our example we will assume that the remote server's name is -`cloud-gw-213.to.infn.it`: substitute it with your remote endpoint. - -First, check that you have your Grid certificate and private key -installed both in your browser and in the home directory of your -client. - -Point your browser to `https://cloud-gw-213.to.infn.it/auth/`: you'll -probably be asked for a certificate to choose for authentication. Pick -one and you'll be presented with the following web page: - -![Web authentication with sshcertauth](img/sshcertauth-web.png) - -The webpage clearly explains you what to do next. - -### Customizing user's configuration - -Before entering the VAF environment, you should customize the user's -configuration. How to do so depends on your experiment, but usually you -should essentially specify the version of the experiment's software you -need. - -For instance, in the CMS use case, only one file is needed: -`~/.vaf/common.before`, which contains something like: - -``` {.bash} -# Version of CMSSW (as reported by "scram list") -export VafCmsswVersion='CMSSW_5_3_9_sherpa2beta2' -``` - -### Entering the VAF environment - -Open a terminal on your client machine (can be either your local -computer or a remote user interface) and type: - - vaf-enter @cloud-gw-213.to.infn.it - -You'll substitute `` with the username that either your system -administrator or the web authentication (if you used it) provided you. - -You'll be presented with a neat shell which looks like the following: - - Entering VAF environment: dberzano@cloud-gw-213.to.infn.it - Remember: you are still in a shell on your local computer! - pod://dberzano@cloud-gw-213.to.infn.it [~] > - -This shell runs on your local computer and it has the environment -properly set up. - -PoD and PROOF workflow ----------------------- - -> The following operations are valid inside the `vaf-enter` environment. - -### Start your PoD server - -With PROOF on Demand, each user has the control of its own personal -PROOF cluster. The first thing to do is to start the PoD server and the -PROOF master like this: - - vafctl --start - -A successful output will be similar to: - - ** Starting remote PoD server on dberzano@cloud-gw-213.to.infn.it:/cvmfs/sft.cern.ch/lcg/external/PoD/3.12/x86_64-slc5-gcc41-python24-boost1.53 - ** Server is started. Use "pod-info -sd" to check the status of the server. - -### Request and wait for workers - -Now the server is started but you don't have any worker available. To -request for `` workers, do: - - vafreq - -To check how many workers became available for use: - - pod-info -n - -To continuously update the check (`Ctrl-C` to terminate): - - vafcount - -Example of output: - - Updating every 5 seconds. Press Ctrl-C to stop monitoring... - [20130411-172235] 0 - [20130411-172240] 0 - [20130411-172245] 12 - [20130411-172250] 12 - ... - -To execute a command after a certain number of workers is available (in -the example we wait for 5 workers then start ROOT): - - vafwait 5 && root -l - -> Workers take some time before becoming available. Also, it is possible -> that not all the requested workers will be satisfied. - -### Start ROOT and use PROOF - -When you are satisfied with the available number of active workers, you -may start your PROOF analysis. Start ROOT, and from its prompt connect -to PROOF like this: - - root [0] TProof::Open("pod://"); - -Example of output: - - Starting master: opening connection ... - Starting master: OK - Opening connections to workers: OK (12 workers) - Setting up worker servers: OK (12 workers) - PROOF set to parallel mode (12 workers) - -### Stop or restart your PoD cluster - -At the end of your session, remember to free the workers by stopping -your PoD server: - - vafctl --stop - -> PoD will stop the PROOF master and the workers after detecting they've -> been idle for a certain amount of time anyway, but it is a good habit -> to stop it for yourself when you're finished using it, so that you are -> immediately freeing resources and let them be available for other -> users. - -In case of a major PROOF failure (i.e., crash), you can simply restart -your personal PROOF cluster by running: - - vafctl --start - -PoD will stop and restart the PROOF master. You'll need to request the -workers again at this point. diff --git a/proof/doc/confman/css/github.css b/proof/doc/confman/css/github.css deleted file mode 100644 index 13903c0a74982..0000000000000 --- a/proof/doc/confman/css/github.css +++ /dev/null @@ -1,300 +0,0 @@ - diff --git a/proof/doc/confman/img/sshcertauth-web.png b/proof/doc/confman/img/sshcertauth-web.png deleted file mode 100644 index 801117cc588cc..0000000000000 Binary files a/proof/doc/confman/img/sshcertauth-web.png and /dev/null differ diff --git a/proof/doc/v520/index.html b/proof/doc/v520/index.html deleted file mode 100644 index ab7d036cbfb6e..0000000000000 --- a/proof/doc/v520/index.html +++ /dev/null @@ -1,113 +0,0 @@ -
-
- -

PROOF

- -
    - -
  • New functionality - -
      - -
    • Dataset management - -
        - -
      • New class TProofDataSetManager definining the interface -of PROOF to dataset metainfo database
        • - -
      • New class TProofDataSetManagerFile implementating -TProofDataSetManager using the file system as back-end;  the -separation is needed to load dataset menagers using different backends; -for example ATLAS foresees to have a MySQL-based implementation.
        • - -
      • The instance of the appropriate TProofDataSetManager is -instantiated via the plugin manager; by default an instance -of TProofDataSetManagerFile -managing the <sandbox>/datasets -area is created. The directive 'Proof.DataSetManager' can be used to -modify the settings for TProofDataSetManagerFile or to load a -different dataset manager; for example, to '/pool/datasets' as area for -the dataset information, the following directive can be added to the -xrootd config file -

        xpd.putrc Proof.DataSetManager file dir:/pool/datasets

        - -
        • - -
      • - -
    • Interface to TProofMgr::GetSessionLogs() in the dialog -box. The graphics layout of the logbox has been re-designed, with new -buttons to grep the logs and to save them to a file. It is also -possible to choose the range of lines to be displayed and the subset of -nodes. -
      • - -
    • Support for connection control base on the UNIX group -(new directive 'xpd.allowedgroups -<grp1>,<grp2>, ...')
      • - -
    • - -
  • Improvements: - -
      - -
    • In the case of mismatch between the expected and actual -number of processed events, send back to the client the list of failed -packets.
      • - -
    • Implement the classic strategy of the TPacketizer in -TPacketizerAdaptive; the strategy can be changed from adaptive -(default) to TPacketizer with: "PROOF_PacketizerStrategy" parameter to -PROOF
      • - -
    • The max workers per node can now be also set in the -xrootd config file with
      - -       xpd.putrc  -Packetizer.MaxWorkersPerNode: <desired number>
      • - -
    • -Make fCacheDir and fPackageDir controllable via directive -
      • - - -
    • - -
  • Fixes - -
      - -
    • Two memory leaks in TProofServ affecting repeated runs -withing the same session
      • - -
    • -Fix a problem cleaning-up the input list on the workers -
      • - -
    • The type of "PROOF_MaxSlavesPerNode", - "PROOF_ForceLocal" and - "PROOF_PacketAsAFraction" parameters has been changed from -Long_t to Int_t. 
      • - -
    • TProofCondor plug-in: -
        - -
      • Adapt the signatures of the main constructors of -TProofCondor and TProofPEAC -(and of the related plug-in handlers) to the one of TProof.
        • - -
      • Add the possibility to trigger the load of a generic -TProof-derived plug-in via -a directive the xrootd config file 'xpd.proofplugin', e.g. -'xpd.proofplugin condor:'
        • - -
      - -
      • - -
    • - - -
diff --git a/proof/doc/v522/index.html b/proof/doc/v522/index.html deleted file mode 100644 index 7c56270375250..0000000000000 --- a/proof/doc/v522/index.html +++ /dev/null @@ -1,211 +0,0 @@ - - - PROOF release notes - - -
- -
- -

PROOF

- -
    - -
  • New functionality
    • - -
      - -
    • PROOF-Lite
      • 2-tier -realization of PROOF intended for multi-core machines; the client -starts directly the workers; no daemon is required. To start a session -just use TProof::Open("") or TProof::Open("lite"). From there on -everything should be as in normal PROOF, though some functionality may -not have been ported yet. To start a standard PROOF -session (i.e. via daemons) on the localhost use -TProof::Open("localhost").
    • XrdProofd plug-in
      • - -
        - -
      • Possibility to define the list worker directly in the -xrootd config file (new directive xpd.worker, see Wiki reference pages)
        • - -
      • Support for automatic reconnections in the case xrootd -is restarted
        • - -
      • Dedicated admin area (under <xrd.admin>/.xproofd.<port>) to -keep information about active and terminated sessions, and active -clients. This is used to reguraly check the client and session -activity, to cleanup orphalin sessions and to shutdown inactive client -connections. 
        • - -
      • domain + level control of printout message
        • - -
      - -
    • Dynamic "per-query" scheduling
      • -
        - - -
      • Dynamic worker startup. It can be enabled by the cluster - administrator with the 'xpd.putrc Proof.DynamicStartup 1' directive - in the config file. The effect is that a session starts only on - the master. When a query is submitted (call to TProof::Process), - the session master contacts the scheduler. - In response it receives a list of workers and starts the worker - processes. The environment is copied from the master to the workers. - It consist of: the include and library paths, the set of enabled - packages as well as the macros loaded by the user.
        • - -
      -
    • Flexible and fault-tolerant workers
      • -
        -
      • A packet resubmitting mechanism. When a worker dies all the - packets that it processed are resubmitted.
        • -
      • Added the possibility to handle dynamically removed workers and partly processed - packets (when a worker is stopped while processing a packet it finishes - the current event and the rest of the packet is reassigned to another workers). - It's done by a new method TPacketizerAdaptive::AddProcessed(TSlave *sl, - TProofProgressStatus *st, TList **) and TPacketizerAdaptive::ReassignPacket.
        • - -
    • Memory control
      • Add -the possibility to display the memory footprint on workers and master as a -function of the entry processed (workers) or of the merging step -(master). A new button has been added to the PROOF dialog box to -retrieve and display the memory usage. On the workers about 100 -measurements are recorded by default; this number can be changed with 'proof->SetParameter("PROOF_MemLogFreq", memlogfreq)';
      • Add -the possibility to set upper limits on the virtual memory used by -processes; the session gets firts a warning when it reaches 80% of -the limit, and then processing is stopped whenit exceeds 95% of the -limit, sending back the results. Also, the memory footprint is notified -when the session is terminated. The limit in MBs is set by the -environment variable "ROOTPROOFASSOFT". An hard limit can be set via the -env "ROOTPROOFASHARD" (also in MBs): the process is automatically -killed by the system if it reaches this limit. Envs variables for the -PROOF processes can be set using the directive 'xpd.putenv' in the -xrootd config file.
    • Input data
      • Introduce the -concept of 'input data': these are objects that are distributed in -optimal way to the workers, which are available via the input list, but -which are not saved in the TQueryResult object. These are meant for big -objects whic can create a big overload when distributed via the -standard input list (which should mostly be used for job control -parameters).  To add an input-data object just use -TProof::AddInputData(TObject *); if the input-data objects are in a -file you can use TProof::SetInputDataFile(const char *file); the final -set of input-data objects is assembled from the objects added via -AddInputData and those found in the file defined bySetInputDataFile. 
      - -
    - -
      - -
    - -
  • Improvements:
    • - -
      - -
    • More -complete set of tests in test/stressProof . To run with PROOF-Lite pass -the argument 'lite' as master URL, e.g. './stressProof lite'.
    • Possibility -to control on the client via rc variable the location of the sandbox, -package directory, cache and dataset directory (the latters two only -for PROOF-Lite); the variable names are 'Proof.Sandbox',  -'Proof.PackageDir', 'Proof.CacheDir' and 'Proof.DataSetDir'. The default location of the sandbox has been changed from "~/proof" to "~/.proof" to avoid interferences with possible users' working areas.
    • XrdProofd plug-in
      • - -
        - -
      • Overall refactorization for easier -maintainance and improved solidity
        • - -
      • Improved format of printout messages: all information -messages contain now the tag 'xpd-I' and all error messages the -tag 'xpd-E', so that they can easily be grepped out from the -log file. -
        • - -
      - -
    • Log sending
      • - -
        - -
      • -Implement selective sending of logs from workers to master to avoid duplicating - too many text lines on the master log. Logs are now sent only after Exec, Print -requests and in case an error (level >= kError) occured. Of course, the full - logs can always be retrieved via TProofMgr::GetSessionLogs -
        • - -
      - -
    • Log retrieval:
      • - -
        - -
      • for 'grep' operations, use the system 'grep' command -via 'popen' -instead of a handmade filtering; this implies that the full grep -functionality is now available
        • - -
      • set the default number of displayed lines to 100 -instead of 10
        • - -
      - -
    • Improve diagnostic in case of worker death: clients will -now -receive a message containing the low level reason for the failure and a -hint for getting more information
    • In -TProofOutputFile, support the "<user>" and "<group>" -placeholders in the output file name to automatically re-direct the -output to an area specific to the logged user.
      • - -
    • Addition of a new class TProofProgressStatus, which is used to keep - the query progress stauts in all the TProofPlayer objects and in the - TPacketizerAdaptive. It is also send in kPROOF_GETPACKET and - kPROOF_STOPPROCESS messages.
      • -
    • The class TPacketizerProgressive is removed.
      • - -
    - -
      - -
    - -
  • Fixes
    • - -
      - -
    • Enable -the max number of sessions ('mxsess' parameter in the xpd.schedparam -directive); users are just refused to start a session if this limit is -reached.
    • Make sure to collect consistently input messages when running in asynchronous mode
    • Fix -a few problems with TProof::SendFile (used by UploadPackage, Load) -appearing when a rapid sequence of these commands was submitted
    • Invalidate the TProofMgr when the physical connection is -closed; avoids -crashing when trying to get the logs after a failure.
      • - -
    • Fix a memory leak in log retrieval (the TProofLog object -was never -deleted)
      • - -
    • Add protections for the cases the manager cannot be -initialized
      • - -
    • Fix a race condition possibly affecting the handling of -workers death
      • - -
    • Avoid duplicating worker logs in the master log file -unless -when explicitly needed by the request (Exec(...), Print(...)) or when -an error occured
    • Fix -problem with the determination and transmission of the name of the -object to be processed. The problem appeared when processing files -containing >1 trees in changing order.
    • Fix problem with TProof::Load loading the macro to one worker only per machine
    • Fix wrong return code preventing the correct propagation of the full ClearPackage to workers
    • Fix a problem causing the whole query to stop even in the case a worker was terminated gently with SIGTERM.
    • -Fix a problem triggering full re-build of a package upon change of a -single file; the version info file was wrongly reset; this should -happen only after a re-build.
    • Make sure that in case multiple TProofOutputFile are present, each get merged correctly
    • Fix problem in TProofServLogHandler::Notify due to bad usage of Form(...).
      - -
    - - diff --git a/proof/doc/v524/index.html b/proof/doc/v524/index.html deleted file mode 100644 index 2efbfa70d9b21..0000000000000 --- a/proof/doc/v524/index.html +++ /dev/null @@ -1,291 +0,0 @@ - - - - -
    -
    -

    PROOF

    -
      -
    • Warning
      • -
        -
      • The -classes -TProofDataSetManager -and TProofDataSetManagerFile have been renamed -TDataSetManager -and TDataSetManagerFile
        • -
      -
    • New -functionality
      • -
        -
      • Add support for session -queuing in the scheduler. This -allows to control the number of sessions allowed to process queries -concurrently. The feature is enabled by a new parameter 'queue:fifo' in -the 'xpd.schedparam' -directive. In case of static worker assignment -(default, random, -round-robin) the max number of running sessions can be limited by -another new parameter 'mxrun'; -for -example
        -            -    xpd.schedparam default -mxrun:3 queue:fifo
        -will run concurrently only 3 sessions. Additional requests are queued -and run as soon as one of the running -sessions goes idle. The current policy is FIFO, so that there is a -rotation among queued -sessions. In the case of load-based worker assignment, the max number -of running -queries is determined dynamically.
        • -
      • Add support for repeat functionality in the xrd.worker -directive. To avoid repeating the same line N times -one can just add -'repeat=N' -in the line; for -example
        -            -    xpd.worker worker -proofwrks:2093 repeat=4
        -will define 4 workers on port 2093 of machine 'proofwrks'.
        • -
      • Add support for port specification via the directive -'xpd.port'
        • -
      • Enable variable -substitution in 'xpd.' directives using the standard -Scalla mechanism described in -http://xrootd.slac.stanford.edu/doc/dev/Syntax_config.htm .
        • -
      • Build also a binary named 'xproofd' which runs -a xrootd -daemon with only the XrdProofdProtocol (i.e. no data serving).
        -This simplifies setups when data serving is not needed and also allows -to better disantagle problems related to one specific protocol. The new -binary accepts the same arguments as 'xrootd' and parses the same -directives form the same configuration file, with the exception of -'xpd.protocol xproofd libXrdProofd.so' which should now be dropped. AN -alternative port can be specified via the new 'xpd.port' directive (see -above).
        • -
      • Add support for 'MasterOnly' mode in starting a PROOF -session. This avoids starting the workers when one wants just to browse -the datasets or retrieve results. To start a session in 'MasterOnly' -mode enter "masteronly" -as second argument to TProof::Open, e.g.
        -  -         root[] -TProof *p = TProof::Open("<masterurl>", "masteronly")
        • -
      • Add full support for placeholders -<uid>, -<gid>, <group> and <homedir> -for the directives specified via 'xpd.putenv'
        • -
      • Add the configuration directive 'proofservparents' to -allow specifying a different list of parent names for the 'proofserv' -tasks. This is needed to avoid untimely killing of 'proofserv' -instances in test setups when multiple instances of the daemons are -running on the same machines under different names.
        • -
      • Add the possibility to switch to asynchronous mode while -running synchronously. A new button "Run -in background" has been added -to the dialog box. The behaviour of Ctrl-C has also been modified: the -user is prompted for a choice among continuing asynchronously, stopping -(terminating) or aborting the query.
        • -
      • Add the possibility to define the dataset information -sources via the directive 'xpd.datasetsrc'. -In this way the permissions -should be set correctly and the related problems disappear.
        • -
      • Record the logs from the ROOT version validation tests -(proofserv forked in test mode). In case of failure - or if the debug -flag is on - the log files are kept under -<xproof_adminpath>/rootsysvalidation/root.<tag>.log -(the <tag> has all the '/' replaced by '-'). This should -facilitate understanding the problems when in case of validation -failures.
        • -
      • Add support for automatic -running of PROOF sessions in valgrind. The second -argument of TProof::Open is used to trigger the relevant -settings. To valgrind the master session start PROOF -with TProof::Open("<master>","valgrind=master"); -to valgrind two workers sessions use -TProof::Open("<master>","valgrind=workers"); to valgrind -master and 2 workers, use -TProof::Open("<master>","valgrind=master+workers"). Other -combinations are available.
        -The valgrind logs are available with the tag -'<ordinal>-valgrind' in the log dialog or form -TProofMgr::GetSessionLogs() .
        -To add options to valgrind execute -TProof::AddEnvVar("PROOF_WRAPPERCMD", -"valgrind_opts:<options>") before starting the -session. 
        • -
      • Add new static TProof::LogViewer("<master>") -to graphically browse the session logs independently of the progress -dialog. The improved log window allows to chose a different master -and/or session  and displays human readable information about -the starting time of the session being browsed.
        • -
      • A set of scripts for quick interaction with a dataset -manager via PROOF are available under $ROOTSYS/etc/proof/utils/pq2 . -The scripts are prefixed -pq2 (proof -quick query - or -proof-dq2) -and allow to {browse, register, remove, verify} datasets on a given -PROOF master. See $ROOTSYS/etc/proof/utils/pq2/README for more -information.
        • -
      -
    • Improvements
      • -
        -
      • Enable by default schema evolution in TMessage; can be -disabled setting 'Proof.SchemaEvolution: -0' .
        • -
      • Extend the functionality of the dataset API to obtaine -information on per-server base; add also two new methods:
        • -
          -
        • TProof::SetDataSetTreeName(<dataset>,<treename>): -set/change the default tree name in the TFileCollection;
          • -
        • TProof::ExistsDataSet(<dataset>): -check -by-name the availability of a given dataset;
          • -
        -
      • In ProofBench, 
        • -
          -
        • Load the macro before executing it. This allows to -circumvent a problem recently fixed giving less dependency on the -server version.
          • -
        • In make_dset.C, simplification of the body and of the -signature, eliminating one redundant argument
          • -
        -
      • In TProofOutputFile, improve flexibility in defining the -URL for the local files server. The "LOCALDATASERVER" env is tested, -which can defined with placeholders via the xpd.putenv directive in the -xrootd/xproofd config files.
        • -
      • Improving parsing of lines with memory info. -This solves occasional crashes while generating the memory -plots.
        • -
      • In TProofMgr::GetSessionLogs:
        • -
          -
        • add the possibility to postpone the retrieval of the -logs files when the TProofLog object is created. This improved -functionality is exploited in the log window.
          • -
        • add decoding of the session starting time and full -information about the master URL
          • -
        -
      • Enable new xrootd configuration options, including the -possibility to set the compiler and linker
        • -
      • Cleanup of the TProofMgr functions DetachSession and -ShutdownSession, and better handling of the internal list registration, -to fix potential segvs when reopening a PROOF session inside the same -ROOT session.
        • -
      • Optimize the way results are transferred and merged:
        • -
          -
        • Output objects are added to the same TMessage until a -HWM is reached (default 1MB; controlled by 'ProofServ.MsgSizeHWM'); -this limits the number of transfers in the case of large numbers of -small objects.
          • -
        • Reasonably small histograms (GetSize() < -MsgSizeHWM) are merged in one-go at the end instead of one-by-one to -exploit, for example, the better performance of TH1::Merge on the full -list of histos.
          • -
        • Add possibility to compress the messages; this is -controlled by ProofServ.CompressMessage -<compression_level>
          -The default is still 'no compression' but this will allow to study the -impact of compression.
          • -
        -
      • Add sort of 'progress' counter for merging is now shown -on the client:
        -   -    root [n] p->Process(...)
        -      ...
        -      Mst-0: merging output objects ... / (4 -workers still sending) -
        -This asserts socket activity and fixes the timeout -problems during long merging phases reported in a few cases.
        • -
      • In TFileMerger, create directly the output file at the -final destination do not make a local copy in the temp directory first -(if needed, one can always set the temporary destination to temp -followed by a TFile::Cp to the final destination); this allows to avoid -reported problems with small temp partitions (see Forum).
        • -
      • In XrdProofConn, enable cycling through the -authentication protocol presented by the server. This only holds for -the choice of the protocol, because the server currently supports only -one full handshake.
        • -
      • In test/stressProof.cxx, avoid interferences between the -settings used for the PROOF tutorial and possible local settings -(daemon, dataset manager).
        • -
      • Add possibility to control the automatic re-loading of -the <proof.conf> file via the keyword -'reload:1'/'reload:0' -in the xpd.resource directive.
        • -
      • Move the validation of <proof.conf> at the -moment of use; this allows to specify a file path and to dynamically -create/modify/destroy the file; used by PoD.
        • -
      • Improve displaying speed of large log files
        • -
      -
    • Fixes
      • -
        -
      • Fix two severe -bugs in the way TTreeCache -was used in PROOF: one bug was de facto disactivating the cache; the -other was causing a std::bad_alloc exception to be thrown on workers -when opening a remote file after a local one.   
        • -
      • Fix several problems in TChain::Draw including
        • -
      -
        -
          -
        • drawing into an existing histogram, i.e. -chain->Draw("var>>myhist");
          • -
        -
          -
        • treatment of histogram merging in case of small -statistics, i.e. when
          -the autobinning is not or only partially active;
          • -
        -
          -
        • usage of existing canvases when different histogram -names are specified;
          -
          • -
        -
      -
        -
      • Fix a problem causing a duplication of the final feedback -object
        • -
      -
        -
      • Fix problem with determining the subdir name in -TFileMerger::MergeRecursive on Windows
        • -
      • Make sure that the default sandbox is under $HOME/.proof
        • -
      • Fix a problem with dataset validation in multi-level -master setups
        • -
      • Fix a problem with ordinal numbers in multi-master setups
        • -
      • Fix a problem with defining the internal paths for -executables when configuring with '--prefix'
        • -
      • Fix backward-incompatibility issue giving the error -message  "unknown action code: 5112"
        • -
      • Fix a few problems with file retrieval from the cache
        • -
      • Fix a problem with iteration of a std::list occasionally -causing seg-violations in TXSocket
        • -
      • Fix a few problems preventing correct usage of entry -lists in PROOF
        • -
      • Fix a problem with the permissions of the credentials -files created under <sandbox>/.creds
        • -
      • Fix a potential problem while determining the log paths -in log retrieval
        -
        • -
      • Do not use vnsprintf in the XrdProofd plug-in, potential -source of deadlocks.
        • -
      • Fix a problem overwriting the local environment settings -for the xrootd sec modules
        • -
      • In XrdProofdProofServMgr::Destroy, fix segv in message -creation when all sessions are destroyed at once
        • -
      • Fix a problem determining the relative time order of old -sessions for log retrieval
        • -
      • In TProof::HandleInputMessage, fix possible double delete -after kPROOF_STOPPROCESS
        • -
      • Fix a couple of issues on reconnection to a running -session (some dialog buttons not in the correct state; logs not -correctly redirected)
        • -
      • Fix a problem creating spurious warnings during 'draw' -queries
        • -
      -
    - \ No newline at end of file diff --git a/proof/doc/v526/index.html b/proof/doc/v526/index.html deleted file mode 100644 index 4a040d2e72bf8..0000000000000 --- a/proof/doc/v526/index.html +++ /dev/null @@ -1,313 +0,0 @@ - -
    -
    -

    Proof

    -
      -
    • New functionality -
        -
      • TProofMgr -
          -
        • Add support for the following functionality: -
            -
          • sandbox file listing and browsing
            • -
          • sandbox file removal
            • -
          • file upload, download
            • -
          • -
        • -
      • TProofDraw -
          -
        • Allow to set a color, size, size, width for lines, -area, markers; the attributes are transmitted via the input list and -automatically derived from the ones of the chain
          • -
        • -
      • Support automatic creation of a dataset out of files -created on the worker nodes by worker processes. The implementation is -an extension of the functionality of the class TProofOutputFile used -for merging via file.
        • -
      • Add the possibility to enable/disable the tree cache and -to change its size on per-query base; two new parameters are available: -
          -
        • PROOF_UseTreeCache    -Int_t        -Enable (0) or Disable (1) the tree cache (default 1)
          • -
        • PROOF_CacheSize       -Long64_t     Cache size in bytes -(default 10000000)
          • -
        -Examples:
        -        -a) to disable the cache for the next run enter:
        -                                 -proof->SetParameter("PROOF_UseTreeCache", 0)
        -        -b) to set the cache size to 20M
        -                                 -proof->SetParameter("PROOF_CacheSize", 20000000)
        • -
      •  Add the parameter -PROOF_UseParallelUnzip to toggle the use of the parallel unzip -(default off for now); to enable it add the following call
        -            -            -        - proof->SetParameter("PROOF_UseParallelUnzip", 1) -
        • -
      •  Add the possibility to give indications about -the number of workers at startup.
        - E.g.
        -       1. To -start max 5 workers
        -             -TProof::Open("<master>","workers=5")
        -       2. To -start max 2 workers per physical machine
        -             -TProof::Open("<master>","workers=2x")
        -     This is useful in general when -running tests (equivalent but quicker then full startup
        -     followed by -TProof::SetParallel(n) or TProof::DeactivateWorker(...)).
        • -
      • Add support for the worker SysInfo_t in TSlaveInfo -(obtained via TProof::GetListOfSlaveInfos())
        • -
      • Add new submerger functionality to speed up the merging -phase. At the end of the query, a set of workers are promoted -submergers and assigned a sub-set of workers to merge. Once each -sub-merger has merged its sub-set of workers, it sends its result to -the master, which merges the partial results into the final
        -set of results.
        -The determination of the sub-mergers is always done dynamically, based -on the recent performance of workers. An optimal (i.e. giving the -highest speed-up) number can be calculated analytically under simple -assumptions.
        -Merging via submergers is by default disabled. To enable it, with the -optimal number of sub-mergers, one should set the integer parameter -'PROOF_UseMergers' to 0, i.e.
        -
        -                     -proof->SetParameter("PROOF_UseMergers", 0)
        -
        -To force S sub-mergers (regardless of the optimal number) do
        -
        -                     -proof->SetParameter("PROOF_UseMergers", S)
        -
        -The new functionality can be tested in tutorials by adding the argument -'submergers' to runProof, e.g.
        -
        -        -        -     root [0] .L -tutorials/proof/runProof.C+
        -        -        -     root [1] -runProof("simple(nhist=10000,submergers)")
        -
        -(see the top of tutorials/proof/runProof.C for additional options). -
        -A test for the submerger functionality has also been added to -test/stressProof.cxx .
        • -
      • In PROOF-Lite, add the possibility for the administrator -to control the number of workers. This is done using -the rootrc variable ProofLite.MaxWorkers, which is read out of -/etc/system.rootrc and cannot be overwritten by users. Setting the -value to 0 disables PROOF-Lite.
        • - -
      • -
    • Improvements -
        -
      • TFileMerger -
          -
        • A few improvements on the way to make TFileMerger and -hadd totally equivalent: -
            -
          • import from hadd an optimization of key hashing
            • -
          • import from hadd a better way to invoke Merge for -generic objects
            • -
          • add option to merge histograms in one go, instead of -one-by-one as for generic objects (this option is not yet supported by -hadd).
            • -
          • -
        • -
      • TProofOutputFile -
          -
        • Add support for the placeholder <file> -the definition of the outputfile. This allows to have complete URL and -to pass options to TFile::Open.
          • -
        • -
      • XrdProofd plugin -
          -
        • Add automatically the line 'Path.ForceRemote 1' to the -session rootrc file if the ROOT version is < 5.24/00 ; this acts -as a workaround for the wrong TTreeCache initialization at the -transition between local and remote files fixed in 5.24/00 .
          • -
        • -
      • Enable mass storage domain settings when working with -TChain's -in multi-master mode. The Mass Storage Domain must be specified as -option in the URL
        -
        -              -chain.AddFile("root:// .....?msd=CERN")
        -
        - and the string must match the value specified in defining the -submaster node.
        • -
      • Improved performance monitoring: the 'Rate plot' button -in the dialog box has been renamed 'Performance Plot' and now shows up -to 4 plots as a function of the processing time: -
          -
        • Instantaneous processing rate, which is now better -estimated by a better estimation of the normalizing times
          • -
        • Average read chunck size, defined as -TFile::GetFileBytesRead() / TFile::GetFileReadCalls() during the last -unit of time; this allows to monitor the usage of the cache; this plot -is present only if some I/O is done, i.e. not for pure CPU tasks.
          • -
        • The number of active workers
          • -
        • The number of total and effecive sessions running -concurrently on the cluster (started by the same daemon); this plot is -present only is the number is at least onec different from 1.
          • -
        • -
      • If enabled, send monitoring information from the master -at each GetNextPacket (at each call of TPerfStat::PacketEvent) to allow -extrnal real-time progress monitoring.
        • -
      • Save the status of a 'proofserv' session into a new file -in the 'activesessions' area. The full path of the new file is
        -          -<admin_path>/.xproofd.<port>/activesessions/<user>.<group>.<pid>.status
        -
        -The status indicates whether the session is idle, running or queued.
        -The status is updated every 'checkfq' secs (see xpd.proofservmgr; -default 30 s). The status is dumped by the reader thread of TXProofServ -and therefore its r/w access is protected.
        -
        • -
      • Enable the use of the tree cache also for local files, -adapting the default settings for the cache to the recent changes
        • -
      • In the XrdProofd plug-in -
          -
        • Improve synchronization between parent and child during -fork
          • -
        • Optimize loops over directory entries
          • -
        • Improve error and notification messages
          • -
        • -
      • Improved handling of Ctrl-C; this follows from a fix in -TMonitor and an improved handling of non-finished query state in the -workers (results are not send to master if the query was aborted)
        • -
      • -
    • Fixes -
        -
      • TFileMerger -
          -
        • Fix a problem preventing correct transmission of all -non-mergeable objects (fixes bug #52886)
          • -
        • Remove the argument isdir from the function -MergeRecursive
          • -
        • Do not remove the first file in the list when returning -from MergeRecursive (fixes bug #54591)
          • -
        • Fix a major leak when merging files with collections -written using kSingleKey option.  The merger was reading each -key in memory and deleted the object at the end, but the container is -not owner by default, so all objects inside leaked.
          • -
        • -
      • PROOF-Lite -
          -
        • Fix a couple of memory leaks showing up when running -repeated queries
          • -
        • Fix a problem in TProofServ::CopyFromCache affecting -the case where the sandbox dir has a '.' and the macro name has no '.', -e.g. compiled selectors in PROOF-Lite.
          • -
        • -
      • TProofOutputFile -
          -
        • Fix a problem with the determination of the fDir member -affecting mostly PROOF-Lite
          • -
        • Fix a serious issue whose net effect was to delete the -outputfile just after having open it
          • -
        • -
      • XrdProofd plugin -
          -
        • Make sure that the limit on the number of old -sessions is applied whenever a new session is started and not only when -the daemon is started.
          • -
        • Fix the behaviour of the xpd.allowedusers directive: if -at least one of these directives is present, users in the password file -are not allowed by default but must be explicitly appear in one -xpd.allowedusers directive  
          • -
        • Fix a source for memory leak in -XrdProofdProtocol::SendMsg
          • -
        • Optimize the usage of strings in a few places
          • -
        • -
      • DataSet manager -
          -
        • Correctly classify as TTree all TTree derived classes -(e.g. TNtuple's)
          • -
        • Fix a problem in saving the end-point URL for local -files
          • -
        • Improve realtime notification during 'verify'
          • -
        • -
      • TProofDraw -
          -
        • Fix a problem with the axis ranges of the underlying -histogram in PolyMarker3D
          • -
        • Allow to use the default pad instead of forcing -creation of one pad per object
          • -
        • Add wrapper to handle the feedback default canvas
          • -
        • -
      • TEventIter -
          -
        • Fix a problem with changing the tree cache size: the -size was reset to the default value after the first file.
          • -
        • -
      • TDataSetManagerFile -
          -
        • Solve a consistency problem in checking URLs for -duplication when adding them to the relevant TFileInfo
          • -
        • During dataset validation, do not fail on duplications -but notify and add them to the bad file list
          • -
        • -
      • TPacketizerAdaptive, TPacketizer -
          -
        • Improve data node / worker matching by always using the -host FQDN
          • -
        • -
      • TPacketizerUnit, TEventIter -
          -
        • Make sure that the entry -number passed to TSelector::Process is unique and in increasing order -for non-data driven processing (packetizer TPacketizerUnit). This -allows to give a meaning to this variable, for example to related it to -one dimension of an integration.
          • -
        • -
      • Fixes in PROOF-Lite: -
          -
        • Make sure that with envs settings via TProof::AddEnvVar -are effective; this enables, for example, the automatic valgrind setup -introduced in 5.24/00 or the experiment specific settings via the -script defined by the env PROOF_INIT
          • -
        • Fix a problem with TProof::Load so that now it can be -also be used for PROOF-Lite
          • -
        • -
      • TProofPlayerRemote -
          -
        • In SendSelector, add misisng -option kCpBin when sending the selector source; the binary files were - never retrieved, even if present and valid
          • -
        • -
      • TProofPlayerSlave -
          -
        • In -Process, fix a problem with cache directory locking while building -the selector;  the net effect was that each worker process was -re-buidling its own selector binary.
          • -
        • -
      • TProofServ -
        • Fix -the order in which the log file is sent in asynchronous processing; the -wrong order was screwing up an immediate synchronous query submission -after an asynchronous run; this case occured, for example, in -'stressProof' .
          • -
        • -
      • -
    - diff --git a/proof/doc/v528/index.html b/proof/doc/v528/index.html deleted file mode 100644 index 2d073a7e959cf..0000000000000 --- a/proof/doc/v528/index.html +++ /dev/null @@ -1,265 +0,0 @@ - - -
    - -

    Proof

    -
      -
    • New functionality -
        -
      • Add support for processing -many datasets in one go in TProof::Process(const char -*dataset, ...).
        -Two options are provided:
        - - 'grand -dataset':   the datasets are added up -and considered as a single dataset (syntax: -"dataset1|dataset2|...")
        - - 'keep -separated': -the datasets are processed one after the other; the user is notified in -the selector of the change of dataset so they have the opportunity to -separate the results. A new packetizer, TPacketizerMulti, has been -developed for this case: it basically contains a list of standard -packetizers (one for each dataset) and loops over them (syntax: -"dataset1,dataset2,..." or dataset1 dataset2 ...").
        -In -both cases, entry-list can be applied using the syntax -"dataset<<entrylist", e.g. -"dataset1<<el1|dataset2<<el2|".
        -The datasets to be processed can also be specified on one or multiple lines in a text file.
        • -
      • Add -support for automatic download of a package when available on the -master but not locally. The downloaded packages are store under <sandbox>/packages/downloaded -and automatically checked for updates against the master repository. If -a local version of the same package is created (using the -UploadPackage) the entry in downloaded is -cleared, so that the behaviour is unchanged.
        • -
      • Add -the possibility to remap the server for the files in a dataset. This -allows, for example, to reuse the dataset information for the same -files stored in a different cluster.
        • -
      • Add a local cache for -TDataSetManagerFile. This is mainly used to improve the speed of -TDataSetManager::ShowDataSets, which is run very often by users and may -be very slow if the number of dataset is large. The cache is also used -to cache frequently received dataset objects.
      • Add the possibility to audit the activity on the nodes via syslog. .
        • -
      • New packetizer TPacketizerFile generating packets which contain a single -file path to be used in processing single files. Used, for example, in -tasks generating files. The files are specified into a TMap - named -'PROOF_FilesToProcess' - containing the list of files to be generated -per host (the key is the host name, the value the TList of TObjString -(or TFileInfo) with the files names - or a TFileCollection: the output -of TFileCollection::GetFilesPerServer() can be directly passed as files -map). Workers are first assigned files belonging to -the list with host name matching the worker name. The map is -distributed to the master via the input list.
      • Add support for -automatic setting of pointer data members to the relevant object in the -output list. The use of fOutputList->FindObject("name") in -TSelector::Terminate is not needed anymore for pointer data members, -e.g. histograms.
        • -
      • Add the possibility to define an external list of environment -variables to be transmitted to the master and workers. This is done via -the environment variable PROOF_ENVVARS. This addition allows to change -the variables wthout changing the macro or application running -TProof::Open.
        • -
      • Add the possibility to save the perfomance information shown -by the dialog into a small ntuple included in the output list. The -ntuple contains 5 floats (processing time, number of active workers, -event rate, MBytes read, number of effective sessions on the cluster) -and it is filled each time the number of active workers changes or at -max 100 regular intervals at least 5 secs apart; in this way the ntuple -has at most O(100 entries + number of workers). To enable the saving of -the ntuple execute the following:
        -        proof->SetParameter("PROOF_SaveProgressPerf", "yes");
        -before running the query. The ntuple is called 'PROOF_ProgressPerfNtuple'.
        • -
      • Add support for worker autodiscovery in PROOF using the -Avahi/Bonjour technology. The new functionality is supported on Mac -(MacOsX >= 10.4; no need of additional installs) and linux (requires -the Avahi framework, available by default on most of the -distributions). To use this functionality (instead-of or in-addition-to -the the static worker configuration via proof.conf or xpd.worker) the -new directive 'xpd.bonjour' must be used.
        • - -
      -
      • -
    • Improvements -
        -
      • Improve support for valgrind runs in PROOF-Lite
        • -
      • Add the possibility to add files to a dataset. This is -achieved with a new option 'U' (for update) to RegisterDataSet.
        • -
      • Add -methof TProof::GetStatistics to allow the client to retrieve the -correct values of fBytesRead, fRealTime and fCpuTime at any moment; -this will be used to setup a sort of ROOTmarks in stressProof .
        • -
      • Several improvements in the test program 'stressProof' -and in the tutorials under 'tutorials/proof'
        • -
      • Avoid -contacting the DNS when initializing TProofMgr as base class of -TProofMgrLite: it is not needed and it may introduce long startup -delays.
        • -
      • Make TProof::LogViewer("") start the viewer for -a Lite session, in parallel to whats happen for TProof::Open("").
        • -
      • Several -improvements in the handling of wild cards in the dataset manager; for -example, issuing a GetDataSet(...) on a dataset URI containign wild -cards will return a grand dataset sum of all the datasets matching the -URI.
        • -
      • Add options to get a list of all dataset registered names -from ScanDataSets (option kList; the result is a TMap of {TObjString, -TObjString} with the second TObjString empty).
      • Improved version of the PQ2 scripts; the scripts now invoke a dedicated ROOT application (named pq2) available under $ROOTSYS/bin .
      • Add -support for recursive reading of group config files via the 'include -sub-file' directive. This allows to have a common part and, for -example, customize differently the quotas.
      • Fix an issue with TTreeFriends. New tutorial showing how to use friends in PROOF.
      • Package -management: add support for arguments in the SETUP function: it is -possible now to pass a string or a list of objects. The -TProof::EnablePackage interface has been extended to support this.
      • Optimize -the validation step in the case not all the entries are required. The -validation step is stopped as soon as the requested number of events is -reached. If the parameter "PROOF_ValidateByFile" is set to 1, the -number of files is exactly what needed; otherwise the number of files -may exceed the number of files needed by (Number_Of_Workers - 1) .
        • -
      • New directive 'xpd.datadir' to better control the user data directories and their permission settings. 
      • In TPacketizerUnit, add the possibility to exactly share the number of cycles between the workers. -See the parameter PROOF_PacketizerFixedNum.
      • Implement -a timer to terminate idle sessions. The timeout value is controlled by -the variable ProofServ.IdleTimeout (value in seconds). This variable -can be set for all sessions in the xproofd config file via the 'xpd.putrc' directive.
        • -
      • Add the possibility to control the use of sub-mergers with -the ROOTrc variable Proof.SubMergers. It has the same meaning of the -parameter 'PROOF_UseMergers'. The capabilities of the latter have been -extended: now -1 means disable the use of submergers (before  negative values were ignored and there was no way for the user to disable the use of submergers).
        -
        • -
      • Packetizer optimizations: improved worked distribution when -the number of files left to be processed is smaller than the number of -workers and at least one file has a number of events significantly -larger than the average; better apply the upper/lower limits on the -expected packet processing time.
        • -
      • Add the possibility to single-out disk partitions in the -packetizer; this works adding the beginning of a path in the name -defining a new TFileNode (e.g. 'host://disk1' instead of 'host' only as -it was so far). These feature can be enabled by defining the rootrc -variable 'Packetizer.Partitions', e.g.
        -           Packetizer.Partitions  /disk1,/disk2,/disk3
        • -
      • Add to the output list the parameters used by the active packetizer.
        -
        • -
      • In the PrintProgress function used to display a text progress -bar, show also the average reading rate in [k,M,G}bytes/s in addition -to the event processing rate. This is useful to have a feeling of the -rate when running of a remote machine in batch mode.
        • -
      • Add the possibility to control the resident and virtual -memory of a proofserv using 'ulimit', which has less limitations and -more flexibility than setrlimit.
        • -
      • Deactivate workers when the requested packages could not be enabled properly.
        • -
      • Add support for reconfiguring the group manager and the -{env,rootrc} settings. The related configuration files are checked for -changes during the regular checks done by the XrdProofdManager.
        • -
      • Add support for selective definition of env and rootrc -variables. Different values can be set for different users, groups, SVN -versions or ROOT versions.
        • -
      • Improve the diagnostic in case of exceptions. Information -about the event and file being processed at the moment the exception -was raised is sent to the client, e.g.
        -   0.5: caught exception triggered by signal '1' while -processing dset:'EventTree', -file:'http://root.cern.ch/files/data/event_3.root', event:1 - check -logs for possible stacktrace
        -The patch also fixes a problem with submergers observed when a worker -was stopped because above the memory limits: this worker was -established as merger but could not do the work, for obvious reasons, -freezing the session.
        • -
      • Add two new methods to TProof: ShowMissingFiles() to facilitate -the display of the list of missing files; and GetMissingFiles() to get -a TFileCollection (dataset) with the missing files for further -processing.
        -
        -
        • - - -
      • -
    • Fixes -
        -
      • Fix a bug in error status transmission which avoid -session freezing in some cases
        • -
      • FIx -a few issues in libXrdProofd.so with handling of connection used for -admin operation: this should solve some cases where the daemon was not -responding.
        • -
      • Fix a few memory leaks showing up when -running several queries in the same session
        • -
      • Fix a few issues affecting the new sub-merging option
        • -
      • Fix an issue preventing proper real-time notification -during VerifyDataSet
        • -
      • Fix an issue with TQueryResult ordering (was causing -random 'stressProof' failures)
        • -
      • Fix -an issue with TProof::AskStatistics (fBytesRead, fRealTime and fCpuTime -were not correctly filled on the client; the values on the master, -displayed by TProof::Print were correct).
        • -
      • Fix several small issues affecting the handling of global -package directories
        • -
      • Fix an issue with socket handling in the main event-loop -while sendign or receiving files via TProofMgr.
        • -
      • Fix -a problem counting valid nodes in sequential or 'masteronly' mode, -generating the fake error message "GoParallel: attaching to candidate!"
        • -
      • Fix a few issues with the length of Unix socket paths -affecting PROOF-Lite and xproofd on MacOsX
        • -
      • Fix an issue with the release of file descriptors when -recovering sessions .
        • -
      • Fix an issue with a fake error message ("Error in -<TROOT::cd>: No such file root:/") in PROOF-Lite when -issuing TProof::SetParallel().
        • -
      • Fix a problem with negative values for 'workers still -sending' in PROOF-Lite .
        • -
      • Fix locking issue while building packages locally.
        • -
      • Fix issue setting permission and ownership of the dataset -user directories.
      • Fix -a subtle bug affecting the (possibly rare) case when not all entries -are required and # entries does not correspond to an complete subset of -files (e.g. # entries = 1001000 with files of 100000 entries each). The -effect was uncomplete processing (skipped events, magenta bar) or a -session freeze.
        • -
      • Fix problem with packet re-assignment in case of a worker death (some packets were processed twice or more times).
        • -
      • Fix problem with the transmission of non-default file -attributes (e.g. the number of entries) from TChainElement to -TDSetElement during TChain processing in PROOF
        • -
      • Fix problem in the default packetizer with validating the -exact number of needed files when the information about the entries is -already available.
        • -
      • Fix problem with 'xpd.putenv' and 'xpd.putrc' occuring when the variables themselves contain commas.
        • -
      • Avoid resolving the workers FQDN when running in PROOF-Lite, -creating unnecessary delays when running PROOF-Lite within virtual -machines.
        • -
      • Fix problem with the permissions of the user data directory.
        • -
      • Add files to the list of files to process only when finally validated.
        • -
      • Fix -problem with canvases when the feedback canvas and the final canvas are -the same (do not delete the feedback canvas at the end of processing)
        • -
      • Make sure that TProof::Load, TProofPlayer::SendSelector and -TSelector::GetSelector treat consistently the extensions of the -implementation files.
        • -
      • Unlock the cache after failure to load a selector; prevents session freezing
        • -
      • Correctly update the number of submergers when workers die
        • -
      • Add missing protection causing a crash in submergers when the output list contained TProofOutputFile objects.
        • -
      • Move the creation and start of the idle timeout from the end -of SetupCommon to the end of CreateServer, so that the timeout is not -active during worker setup.
        • -
      • Make sure that the TProof instance on the client is invalidated after an idle timeout.
        • -
      • Fix an old issue with DeactivateWorker("*") (the session is -was terminated because no worker was active; this call coudl not be -used as intermediate step to select a small number of workers).
        • -
      • Consistently check both Proof.Sandbox and ProofLite.Sandbox for sandbox non-default location as done in TProofLite
        • -
      • Fix a problem with the registration of missing files in the -'MissingFiles' list (files which could not be open on the workers were -not always added to the list).
        -
        -
        -
        • - - -
      • -
    - - diff --git a/proof/doc/v530/index.html b/proof/doc/v530/index.html deleted file mode 100644 index 943360b0171df..0000000000000 --- a/proof/doc/v530/index.html +++ /dev/null @@ -1,180 +0,0 @@ - - - - - - - -
    - -
    - - -

    Proof

    -
      -
    • New functionality -
        -
      • In TProof::Load, add the possibility to define a list of files to be - sent because needed by the main macro file. The list is comma-separated - and the first file is considered as the main one. For example
        -
        -         - proof->Load("<macropath>/mymacro.C+,<thispath>/thisheader.h,<thatpath>/thatheader.h")
        -
        - will make sure that the files 'thisheader.h' and 'thatheader.h', needed - by 'mymacro.C' are available in the sandbox on the worker machines. - Note that 'thisheader.h' and 'thatheader.h' will be available remotely - in the sandbox, as 'mymacro.C'; so they should be included directly by - 'mymacro.C', e.g. '#include "thisheader.h"' .
        • -
      • Import the dataset stager daemon 'afdsmgrd' into ROOT; this is used - to manage data staging based on the dataset information (see - http://code.google.com/p/afdsmgrd/ for more info). The daemon is - located under $ROOTSYS/proof/afdsmgrd .
        • -
      • New PROOF bench suite, a framework to run CPU and IO benchmarks with - default selectors/data or with user-provided ones. The code is located - under proof/proofbench.
        • -
      • Add the possibility to access the files on the workers via the same - port used by PROOF. This is useful for cases when it is not possible to - start a file server daemon on a different port (because, for eample, of - a firewall or just inconvenience) and workers do not share a file - system. Internally this works by forking a 'rootd' after identifying a - file request and trasferring the connection to it. The client side is a - TNetFile and it is triggered by the protocol "rootd://" (the just - implemented etc/plugins/TFile/P120_TNetFile.C includes this - protocol).
        • -
      • Add support for log file truncation. Truncation is disabled by - default. Enabling is controlled by the rootrc variable
        -
        -            - ProofServ.LogFileMaxSize   - {<bytes>|<kilobytes>K|<megabytes>M|<gigabytes>G}
        -
        -  indicating the max number of bytes. The number can be followed by - a {K,M,G} to indicate KBytes, MBytes or GBytes, respectively.
        • -
      • Add new derivation of TList (TProofOutputList) to be used on the - PROOF client to filter out PROOF internal objects when displaying or - printing the list. By default objects was names start with 'PROOF_' are - not shown. The presence of a non empty missing file list is - notified.
        • -
      • In the PROOF monitoring to: send additional information about memory - usage during the query, the name and size (# of files) of the dataset - processed (if any); add possibility to send the information to multiple - monitoring collectors.
        • -
      • Add support for block activation/deactivation of workers.
        • -
      • Add possibility to start the proofserv with 'system()' instead of - 'fork()' as done in PROOF-Lite. A new switch 'usefrk' has been added to - 'xpd.proofservmgr' to control that. Default is still fork().
        • -
      -
      • -
    • Improvements -
        -
      • In TProof::ClearPackages, use the manager to execute the command on - all known worker machines. Improves the consistency when re-istalling - packages.
        -
        • -
      • In TProof::GetDataSets, add support for option ':lite:'; this allows - to fill the map with only the summary information about the datasets - (the header of TFileCollections), significantly increasing the speed - and the memory footprint when the number of datasets is very large.
        • -
      • Accept '.' in user names.
        • -
      • Add switch to control caching of the files read on MacOsX. A call to - fcntl(fd, F_NOCACHE, 1) is done after opening the file.
        • -
      • Add export of the envs ROOTPROOFCLIENT and ROOTPROOFLITE when - appropriate. These allow to steer building and/or enabling of PAR files - in PROOF-INF/BUILD.sh and/or PROOF-INF/SETUP.C, improving transparency - between normal ROOT and PROOF. The example PAR - 'tutorials/proof/event.par' has been modified to check the two - variables.
        • -
      • Fix a few issues in SQL PROOF monitoring: in - TSQLMonitoringWriter::SendParameters, drop ''' around field names in - the INSERT string; also use TString::Format(...) instead of Form(...) - where relevant.  In TPerfStats: call 'proofgroup' instead of - 'group' the field with the PROOF group (interference with the 'group' - keyword in SQL); add new field 'querytag' VARCHAR(64) with the unique - query tag; in WriteQueryLog fill also the field 'totevents'; in - PacketEvent, add switch to control whether to send te information to - the monitoring system on per packet level (may be too much for SQL). - The switch is called fMonitorPerPacket and it is globally controlled by - the rootrc variable 'Proof.MonitorPerPacket' and at session level with - the parameter PROOF_MonitorPerPacket .
        • -
      • Improve treatment of the case when temporary files are asked to be - created on a shared file system not containing the sandboxes. This - case, which seems to be a rather common one, should be now fully - supported.
        • -
      • Correctly honour selector abort status settings - TSelector::kAbortProcess and TSelector::kAbortFile.
        • -
      • Improve reporting of the non-processed {files, events} in the final - 'MissingFiles' list. 
        • -
      • Improved algorithm for TPacketizerUnit to fix issue with non - homogeneous machines.
        • -
      • Improve the way the information about log files is saved in case of - failures. The log paths for these failing now should be now correctly - saved and accessible via TProofLog.
        • -
      • Improve merging of histograms. Just use TH1::Add whne the axis are - equal; much faster than TH1::Merge.
        • -
      -
      • -
    • Fixes -
        -
      • In TDataSetManagerFile::NotifyUpdate fix handling of the case when - the global list file does not exist yet (new dataset directory). Fixes - error messages during editing dataset operations.
        • -
      • Fix issue with machine names consistency when working on a local - machine ('localhost' or 'localhost.localdomain' are mapped to - gSystem->HostName()); solves possible matching problems in the - packetizer.
        • -
      • In TProofServ, fill the "grand total" message with more blanks, so - that no remnants of the previous message are visible on the screen.
        • -
      • In the autoconf/bonjour interface, fix issue preventing the correct - port (the protocol one, which may be different from the application - default) being used when registering the service.
        -
        • -
      • In TProofPlayer::AddQueryResult, fix a bug affecting the order in - whihc query results are registered when the start time is within 1 - second.
        • -
      • Fix worker name in TSlaveLite.
        • -
      • Fix problem with enabling packages with option 'notOnClient' in - PROOF-Lite .
        • -
      • Make sure the log file is flushed at the end of startup to avoid - spurious log messages on next commands .
        • -
      • In CreateSession(), fix an issue with the validity check for existing - sessions .
        • -
      • In TProofLite: fix problem with passing the 'varexp' and 'selection' - strings for processing, preventing correct usage of the operators '|' - and '||' in TTreeFormula.
        • -
      • In the TProofOutputFile constructor, remove the 'localroot' prefix - only if present in the path. Fixes possible truncation problems - occuring when the paths are not under the localroot scope.
        • -
      • In TXSocket and TXSlave: fix problem with the way collection over a - socket just marked as 'bad' was interrupted; the interrupt was de facto - ineffective, so that collection stayed always until the timeout expired - (default: 5 minutes). Should solve some of the cases were slow response - was experienced.
        • -
      • Fix a problem with log path transmission when the node dies early or - not even starts. The log path was empty and wrong was filled in when - retrieving the log buffers, disorienting debugging.
        • -
      • Fix a bug checking the first event which rendered ineffective the - request for processing a subset of events in a given dataset or - chain.
        • -
      • In pq2-ana-dist, fix problem with the labels of the distribution - histo occuring when machines are represented by IPs instead of - names.
        • -
      • Add missing calls to closedir() and TSystem::FreeDirectory, cuasing a - large number of filedescriptors remaining opened after xproofd - initialization.
        • -
      • Fix a problem with the final update of the progress information - affecting occasionally cases with skipped events.
        • -
      • Fix merging of TproofOutputFile when using submergers (the - intermediate files were not correctly handled).
        • -
      • Fix the way TChain weights are transmitted to TProofDraw in - DrawSelect operations.
        • -
      -
      • -
    • AoB -
        -
      • The class TFileMerger has been moved to 'io/io'
        • -
      -
      • -
    - - diff --git a/proof/doc/v532/index.html b/proof/doc/v532/index.html deleted file mode 100644 index d942ff1949614..0000000000000 --- a/proof/doc/v532/index.html +++ /dev/null @@ -1,87 +0,0 @@ -
    -
    - -

    PROOF System

    -NB: Starting with version 5.32/00, Xrootd -is not distributed any longer with ROOT but has become an external package. -If not avaibable the PROOF modules 'proofx' and 'proofd' will not be built. -The PROOF core modules, however, are built. Namely, PROOF-Lite will be -available even when Xrootd is not. -

    New functionality

    -
      -
    • Creating PAR packages from ROOT data files: it is now possible to -use TFile::MakeProject to create a PAR file to read the file.
      • -
    • Add support for backend-dependent record formatting of PROOF monitoring. -This is achieved by introducing a new layer, described by the abstract -interface TProofMonSender, with the record format defined in the backend -implemenation (currently TProofMonSenderML, for MonaLisa, and -TProofMonSenderSQL, for SQL backends). Currently three types of records -are sent: 'summary' (derived from what was currently posted), 'dataset', -with entries per dataset processed in the query, and 'files', with -entries per file processed in the query. In SQL terms, each of this -records corresponds to a different table. Sending of any of the three -records can be toggled independently.
      • -
    • In TProofMgr, add 'ping' functionality to test in non-blocking way if -a PROOF service is listening at a given port of a given host.
    -

    Improvements

    -
      -
    • In PROOF-Bench, file generation, add the possibility to change -only the generating function, passed as TMacro. Add also check on the -free space on the device and skip file generation if less than 10% or -less than 1 GB.
      • -
    • Record in TStatus also the max memory usage on the master and printed -via TStatus::Print; this allow a quick visualisation of the overall -memory usage at the end of the query.
      • -
    • Import version 0.9.6 of afdsmgrd
      • -
    • Make sure that the name(s) of the processed dataset(s) are registered -in the TFileInfo objects being processed, so that it can be used for -monitoring.
      • -
    • In XrdProofd, add possibility to skip the checks for the data -directories during session startup, as they may significantly slowdown -the startup process is the medium is busy. In such a case, admins -are responsible to create the directories in advance; the session -releated part fo the path is created by the session once up.
      • -
    • In XrdProofd, move the check for the username after authentication. -This is because authentication may run some credentials-to-user mapping -which can modify the requested username. This way we really check the -final username and not the one requested by the client, which may even -not exist on the machines. Side modification: when the mapping function -returns more usernames, the username specified by the client is used to -help choosing the effective username among the available choices; if not -match is found the handshake does any longer fail, the first mapped -username is chosen instead.
      • -
    • In XrdProofd, allow 'xpd.allowedgroups' to control also PROOF -groups, not only UNIX ones.
    • In XrdProofd, simplify error -messages in case of login failure because of non-authorization.
      • -
    • Remove hardcoded additions of dirname(COMPILER) and of -'/bin:/usr/bin:/usr/local/bin' in front of PATH. These uncontrolled -additions could hide specific settings in PATH and be the source of -weird problems appearing in PROOF only.
      • -
    • Add more flexibility to the definition of the library path seen by -proofserv. So far to avoid ambiguites in some cases, $ROOTSYS/lib was -removed and the one of the ROOT version chosen was added later on in -front, which proved to be to aggressive in some cases.
      • -
    • All changes (and fixes) needed to build against the version of Xrootd, -now always installed as external.
    -

    Fixes

    -
      -
    • Fix GetSessionLogs in PROOF-Lite
      • -
    • Restore correct parsing of "workers=N" in PROOF-Lite
      • -
    • In Proof-Bench, make sure that it can be run from any directory -and no matter how ROOT was installed
      • -
    • Fix issue in TProofPlayer::HandleHistogram preventing proper -histogram cleaning right after merging when using TH1::Add; histogram -were still destroyed at the end of the query, but there was no -memory advantage in TH1::Add wrt TH1::Merge.
      • -
    • Make sure that the performance tree is removed from the output -list when saved to the output file. Solves a segv at quit.
      • -
    • Decouple from registered TChains in already TProof::Close(); allows -to avoid possible crash at exit ('.q') occuring after the recent -revision of the socket cleanup policy.
      • -
    • In XrdProofd, fix a few issues with option 'xpd.multiuser'.
      • -
    • In TXSocket::ProcessUnsolicitedMsg, fix an issue preventig server -messages to be displayed during setup, i.e. when the XrdClientConn -instance is not yet defined.
      • -
    • In XrdProofd, fix the behavior of the 'xpd.allowedusers' and -'xpd.allowedgroups' directives.
      • -
    diff --git a/proof/doc/v534/index.html b/proof/doc/v534/index.html deleted file mode 100644 index 0b78918f37440..0000000000000 --- a/proof/doc/v534/index.html +++ /dev/null @@ -1,102 +0,0 @@ - -
    - -

    PROOF System

    -
      -
    • Added functionality -
        -
      • Added interface to simplify the creation of the performance -tree: two new methods TProof::SetPerfTree("<file>") and -TProof::SavePerfTree("<file>", "<queryref>") allow set -and/or save the information to a given file path. The perfomance tree -settim=ngs are diabled after each query, so they need to be enabled -each time.
        • -
      • Add support for a command line test run of 'proofserv'; this is -useful to test that the environment is setup correctly.
        • -
      • In TProofBench::DrawCPU, add possibility to extract of a couple -of numbers supposed to give an idea of the computing specs of the -cluster being benchmarked. These are the maximum rate for the standard -CPU intensive task and the normalized, per worker, rate. Both are -expressed in RNGPS (RaNdom Generation Per Second).
        • -
      • Add class TProofPerfAnalysis collecting a set of tools to -analyse the performance tree.
        • -
      • Add support for selector-by-object processing in PROOF. The -selector object, created and configured locally by the user, is added -to the input list and recuperated from there on the worker machines for -processing. Any input list setting in the selector itself is not -streamed but temporarly moved to then standard input list, so that user -can use the selector input list as container of processing information -if they find convenient to do so. Process(...) methods with the file -name argument replaced by 'TSelector *' have  introduced where -relevant (TProof, TProofPlayer and their derivatives, TDSet). 
        • -
      • Add the possibility to force submerging at node level, i.e. one -submerger per physical machine. In this way the network traffic can be -minimized, for example when merging large output files. The new feature -is enabled by setting the Int_t parameter 'PROOF_MergersByHost' (or the -directive 'Proof.MergersByHost') to a non-null value.
        • -
      • Simplify enabling of basic feedback. In TProof::Process, add -support for switches "fb=name1,name2,name3,... " or -"feedback=name1,name2,name3,... " in the option field. This enables -feedback for the specified objects, creating a TDrawFeedback object -attached to the session. Feedback for the specified objects is disabled -at the end of the query and the created TDrawFeedback is destroyed. The -special name 'stats' enables the three feedback histograms required by -a dedicated new class TStatsFeedback, and uses a TStatsFeedback instead -of TDrawFeedback to display the feedback.
        • -
      -
      • -
    • Improvements -
        -
      • Add to TProofMgr two static functions to upload files. These -functions are both called UploadFiles and differ in the first argument, -which is used to pass the files to be uploaded. These can be given as a -list (of TFileInfo or TObjString), a directory or specified in a text -file.
        • -
      • Add support for paralell dataset verification. This is -implemented via a dedicated TSelector (TSelVerifyDataSet) which is run -over the list of files in the dataset via TPacketizerFile. The file -order is preserved using the recently introduced index in TFileInfo.
        • -
      • In TProofOutputFile, add switch to control the way histograms -are merged by TFileMerger, i.e. one-by-one or all-in-one-go. The -default is one-by-one which requires much less memory. Merging in -one-go (the previous default) can be activated by passing 'H' in the -constructor options.
        • -
      • In ProofBench, add possibility to change the location of the -generated files via the third argument of TProofBench::MakeDataSet.
        • -
      • Several optimizations in the low level PROOF event loop -(TProofPlayer::Process),  allowing to reduce dramatically the -overhead introduced by the operations PROOF needs to perform during the -event loop. A measurement of the overhead can be obtained from a very -light computational task, for example, generating one random number and -filling one histogram; executing this task within a PROOF-Lite session -with 1 worker now takes only 1.8 times the time required by a straight -loop in the parent ROOT session; the same number before was about 13.
        • -
      • In TDrawFeedback::Feedback, call method Draw() of objects not -identified as TH1 derivation. This allows user-defined objects -implementing Draw to be displayed via this utility class.
        • -
      • In TProof::LoadPackageOnClient, do not create a symlink -'pack_name' to the package dir, but add directly the package dir to the -include path. This solves the longstanding annoying problem of failure -when a directory or file with the name of the package did already exist -in the local working directory.
        • -
      -
      • -
    • Fixes -
        -
      • Fix merging issue affecting automatic dataset creation when -only one worker is active.
        • -
      • Fix the realtime reported by TProof::GetRealTime() for masters -(it was overwritten with the ones coming from workers).
        • -
      • Fix serious problem with TProof::Load: additional files were -not copied in the master sandbox but left in the cache. A workaround -for backward compatibility has also been implemented.
        • -
      • Fix a problem preventing actions requiring access to worker -nodes (log file retrieval, reset) to work on workers where the username -is different from the one o the master, e.g. PoD on gLite.
        • -
      • Fix issue with the specification of the working directory -template in 'xpd.multiuser'. -
        • -
      -
      • -
    - diff --git a/proof/doc/v600/index.md b/proof/doc/v600/index.md deleted file mode 100644 index c9f7b00e08c86..0000000000000 --- a/proof/doc/v600/index.md +++ /dev/null @@ -1,49 +0,0 @@ -## PROOF System - -All the fixes and improvements in the PROOF system occured since the release of 5.34/00 are available both in the latest 5.34 tags and in 6.00/00. -The following is a summary of the major modifications since 5.34 . - -### New developments/functionality - -- Several improvements in the merging phase; in particular: - - Modification of output sending protocol to control memory usage, significantly reducing the memory footprint on the master, in particular when merging - large numbers of histograms. - - Use an hash table for the output list to significantly speed up names lookups during merging. -- Add support for dynamic addition of workers to a currently running process (currently supported by the unit packetizer). -- Automatization of the usage of file-based technology to handle outputs. -- [Improved dataset management model](https://root.cern/doc/v628/classTDataSetManagerAliEn.html) - where the PROOF (ROOT) dataset manager is a light frontend to the experiment file catalogs; TDataSetManagerFile is still - used as local cache of the experiment information or to store the work-in-progress status of the dataset manager daemon. This model addresses the scalability issues observed at ALICE AFs. -- Improvements in [TProofBench](https://root.cern.ch/doc/master/classTProofBench.html): - - Recording and display of the maximum rate during query, CPU efficiency calculation for PROOF-Lite runs, better measurement of wall time. - - Support for dynamic startup mode - -- Test program xpdtest to test the status of xproofd (see also man page under $ROOTSYS/man/man1): - -``` {.sh} - $ xpdtest [options] - --help, -h - Gives a short list of options avaliable, and exit - -t - type of test to be run: - 0 ping the daemon (includes process existence check if pid specified; see below) - 1 ping the daemon and check connection for default user - 2 ping the daemon and check connection for the default user and all recent users -... -``` -- Interface with **igprof** for fast statistic profiling. Like valgrind, it can be specified as option to TProof::Open and the output is available via the log viewer technology: - -``` {.cpp} - root[] p = TProof::Open("master", "igprof-pp") -``` -- Miscellanea: - - Added functions [Getenv](http://root.cern.ch/root/htmldoc/TProof.html#TProof:Getenv) and [GetRC](http://root.cern.ch/root/htmldoc/TProof.html#TProof:GetRC) - in TProof to retrieve environment information from the nodes, typically from the master. - - Add support unix secondary groups in group access control. This allows more flexibility in, for example, assigning group-shared credential files to the daemon. - - Several new tests and options in the test program _stressProof_. - -### Bug fixes - -Several consolidation fixes in several parts of the system (see the [5.34 patch release notes for details](https://root.cern/install/all_releases/root-version-v5-34-00-patch-release-notes/)). In particular, those for 'xproofd' were provided by B. Butler and M. Swiatlowski and greatly contributed to consolidate the daemon. - - diff --git a/proof/doc/v602/index.md b/proof/doc/v602/index.md deleted file mode 100644 index 0519ecba6ea91..0000000000000 --- a/proof/doc/v602/index.md +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/proof/proof/CMakeLists.txt b/proof/proof/CMakeLists.txt deleted file mode 100644 index 9ac8dffafc916..0000000000000 --- a/proof/proof/CMakeLists.txt +++ /dev/null @@ -1,86 +0,0 @@ -# Copyright (C) 1995-2019, Rene Brun and Fons Rademakers. -# All rights reserved. -# -# For the licensing terms see $ROOTSYS/LICENSE. -# For the list of contributors see $ROOTSYS/README/CREDITS. - -############################################################################ -# CMakeLists.txt file for building ROOT proof/proof package -# @author Pere Mato, CERN -############################################################################ - -ROOT_STANDARD_LIBRARY_PACKAGE(Proof - HEADERS - TCondor.h - TDataSetManagerFile.h - TDataSetManager.h - TDSet.h - TDSetProxy.h - TLockPath.h - TPackMgr.h - TProofChain.h - TProofCondor.h - TProofDebug.h - TProof.h - TProofLite.h - TProofLog.h - TProofMgr.h - TProofMgrLite.h - TProofNodeInfo.h - TProofOutputFile.h - TProofOutputList.h - TProofProgressStatus.h - TProofQueryResult.h - TProofResources.h - TProofResourcesStatic.h - TProofServ.h - TProofServLite.h - TProofSuperMaster.h - TQueryResultManager.h - TSelVerifyDataSet.h - TSlave.h - TSlaveLite.h - TVirtualProofPlayer.h - TVirtualPacketizer.h - ${PROOF_EXTRA_HEADERS} - SOURCES - src/TCondor.cxx - src/TDataSetManager.cxx - src/TDataSetManagerFile.cxx - src/TDSet.cxx - src/TDSetProxy.cxx - src/TLockPath.cxx - src/TPackMgr.cxx - src/TProofChain.cxx - src/TProofCondor.cxx - src/TProof.cxx - src/TProofDebug.cxx - src/TProofLite.cxx - src/TProofLog.cxx - src/TProofMgr.cxx - src/TProofMgrLite.cxx - src/TProofNodeInfo.cxx - src/TProofOutputFile.cxx - src/TProofOutputList.cxx - src/TProofProgressStatus.cxx - src/TProofQueryResult.cxx - src/TProofResourcesStatic.cxx - src/TProofServ.cxx - src/TProofServLite.cxx - src/TProofSuperMaster.cxx - src/TQueryResultManager.cxx - src/TSelVerifyDataSet.cxx - src/TSlave.cxx - src/TSlaveLite.cxx - src/TVirtualProofPlayer.cxx - src/TVirtualPacketizer.cxx - ${PROOF_EXTRA_SOURCES} - DEPENDENCIES - MathCore - Net - RIO - Thread - Tree - LIBRARIES - ${ROOT_ATOMIC_LIBS} -) diff --git a/proof/proof/doc/index.txt b/proof/proof/doc/index.txt deleted file mode 100644 index c104ad5c031e8..0000000000000 --- a/proof/proof/doc/index.txt +++ /dev/null @@ -1,12 +0,0 @@ -This directory contains the PROOF classes. -BEGIN_HTML - See: - -END_HTML diff --git a/proof/proof/inc/LinkDef.h b/proof/proof/inc/LinkDef.h deleted file mode 100644 index 9e98d672ed3a1..0000000000000 --- a/proof/proof/inc/LinkDef.h +++ /dev/null @@ -1,70 +0,0 @@ -/* @(#)root/proof:$Id$ */ - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifdef __CLING__ - -#pragma link off all globals; -#pragma link off all classes; -#pragma link off all functions; - -#pragma link C++ nestedtypedefs; -#pragma link C++ nestedclasses; - -#pragma link C++ global gProof; -#pragma link C++ global gProofServ; -#pragma link C++ global gProofDebugMask; -#pragma link C++ global gProofDebugLevel; - -#pragma link C++ class TDSet-; -#pragma link C++ class TDSetElement-; -#pragma link C++ class TLockPath+; -#pragma link C++ class TPackMgr+; -#pragma link C++ class TProofChain+; -#pragma link C++ class TProofMgr; -#pragma link C++ class TProofMgrLite; -#pragma link C++ class TProofDesc; -#pragma link C++ class TProof; -#pragma link C++ class TProofCondor; -#pragma link C++ class TProofLite; -#pragma link C++ class TProofSuperMaster; -#pragma link C++ class TSlaveInfo+; -#pragma link C++ class TProofServ; -#pragma link C++ class TProofServLite; -#pragma link C++ class TProofDebug; -#pragma link C++ class TProofLog; -#pragma link C++ class TProofLogElem; -#pragma link C++ class TSlave; -#pragma link C++ class TSlaveLite; -#pragma link C++ class TVirtualPacketizer+; -#pragma link C++ class TVirtualProofPlayer+; -#pragma link C++ class TProofQueryResult+; -#pragma link C++ class TQueryResultManager+; -#pragma link C++ class TDSetProxy+; -#pragma link C++ class TCondor+; -#pragma link C++ class TCondorSlave+; -#pragma link C++ class TProofNodeInfo; -#pragma link C++ class TProofResources; -#pragma link C++ class TProofResourcesStatic; -#pragma link C++ class TProofProgressStatus; -#pragma link C++ class TMergerInfo; -#pragma link C++ class TProofProgressInfo; -#pragma link C++ class TProofOutputList+; -#pragma link C++ class TProofOutputFile+; - -#pragma link C++ class TDataSetManager; -#pragma link C++ class TDataSetManagerFile; - -#pragma link C++ class TSelVerifyDataSet+; - -// For backward compatibility with old client / masters -#pragma link C++ class std::pair; -#pragma link C++ class std::list >; - -#endif diff --git a/proof/proof/inc/TCondor.h b/proof/proof/inc/TCondor.h deleted file mode 100644 index ed8d52490e94e..0000000000000 --- a/proof/proof/inc/TCondor.h +++ /dev/null @@ -1,91 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Maarten Ballintijn 06/12/03 - -/************************************************************************* - * Copyright (C) 1995-2003, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TCondor -#define ROOT_TCondor - -////////////////////////////////////////////////////////////////////////// -// // -// TCondor // -// // -// Interface to the Condor system. TCondor provides a (partial) API for // -// querying and controlling the Condor system, including experimental // -// extensions like COD (computing on demand) // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TString.h" - - -class TList; - - -//------------------------------------------------------------------------ - -class TCondorSlave : public TObject { -public: - TString fHostname; - Int_t fPort; - Int_t fPerfIdx; - TString fImage; - TString fClaimID; - TString fOrdinal; - TString fWorkDir; - - void Print(Option_t *option="") const override; - - ClassDefOverride(TCondorSlave,0) // Describes a claimed slave -}; - - -//------------------------------------------------------------------------ - -class TCondor : public TObject { -public: - enum EState { kFree, kSuspended, kActive }; - -private: - - Bool_t fValid; //access to Condor - TString fPool; //the condor pool to be accessed - EState fState; //our claim state - TList *fClaims; //list of claims we manage - -protected: - TCondorSlave *ClaimVM(const char *vm, const char *cmd); - -public: - TCondor(const char *pool = ""); - ~TCondor() override; - - - void Print(Option_t *option="") const override; - Bool_t IsValid() const { return fValid; } - - TList *GetVirtualMachines() const; - - TList *Claim(Int_t n, const char *cmd); - TCondorSlave *Claim(const char *vmname, const char *cmd); - Bool_t SetState(EState state); - EState GetState() const {return fState;} - Bool_t Suspend(); - Bool_t Resume(); - Bool_t Release(); - - Bool_t GetVmInfo(const char *vm, TString &image, Int_t &perfidx) const; - TString GetImage(const char *host) const; - - - ClassDefOverride(TCondor,0) // Interface to the Condor System -}; - -#endif diff --git a/proof/proof/inc/TDSet.h b/proof/proof/inc/TDSet.h deleted file mode 100644 index 3adf89d5c8e97..0000000000000 --- a/proof/proof/inc/TDSet.h +++ /dev/null @@ -1,265 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 11/01/02 - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TDSet -#define ROOT_TDSet - - -////////////////////////////////////////////////////////////////////////// -// // -// TDSet // -// // -// This class implements a data set to be used for PROOF processing. // -// The TDSet defines the class of which objects will be processed, // -// the directory in the file where the objects of that type can be // -// found and the list of files to be processed. The files can be // -// specified as logical file names (LFN's) or as physical file names // -// (PFN's). In case of LFN's the resolution to PFN's will be done // -// according to the currently active GRID interface. // -// Examples: // -// TDSet treeset("TTree", "AOD"); // -// treeset.Add("lfn:/alien.cern.ch/alice/prod2002/file1"); // -// ... // -// treeset.AddFriend(friendset); // -// // -// or // -// // -// TDSet objset("MyEvent", "*", "/events"); // -// objset.Add("root://cms.cern.ch/user/prod2002/hprod_1.root"); // -// ... // -// objset.Add(set2003); // -// // -// Validity of file names will only be checked at processing time // -// (typically on the PROOF master server), not at creation time. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TNamed.h" - -class TChain; -class TCollection; -class TCut; -class TDSet; -class TEventList; -class TEntryList; -class TFileInfo; -class THashList; -class TIter; -class TList; -class TProof; -class TProofChain; -class TSelector; -class TTree; - -// For backward compatibility (handle correctly requests from old clients) -#include -#include - -class TDSetElement : public TNamed { -public: - typedef std::list > FriendsList_t; - // TDSetElement status bits - enum EStatusBits { - kHasBeenLookedUp = BIT(15), - kWriteV3 = BIT(16), - kEmpty = BIT(17), - kCorrupted = BIT(18), - kNewRun = BIT(19), - kNewPacket = BIT(20) - }; - -private: - TString fDirectory; // directory in file where to look for objects - Long64_t fFirst; // first entry to process - Long64_t fNum; // number of entries to process - TString fMsd; // mass storage domain name - Long64_t fTDSetOffset;// the global offset in the TDSet of the first - // entry in this element - TObject *fEntryList; // entry (or event) list to be used in processing - Bool_t fValid; // whether or not the input values are valid - Long64_t fEntries; // total number of possible entries in file - TList *fFriends; // friend elements - - TString fDataSet; // Name of the dataset of which this element is part - TList *fAssocObjList; // List of objects associated to this element - // (e.g. TObjString describing associated files) - Float_t fMaxProcTime; // Max processing time in secs; -1 no limit - - Bool_t HasBeenLookedUp() const { return TestBit(kHasBeenLookedUp); } - - TDSetElement& operator=(const TDSetElement &); // Not implemented - -public: - TDSetElement(); - TDSetElement(const char *file, const char *objname = 0, - const char *dir = 0, Long64_t first = 0, Long64_t num = -1, - const char *msd = 0, const char *dataset = 0); - TDSetElement(const TDSetElement& elem); - ~TDSetElement() override; - - virtual TList *GetListOfFriends() const { return fFriends; } - virtual void AddFriend(TDSetElement *friendElement, const char *alias); - virtual void DeleteFriends(); - const char *GetFileName() const { return GetName(); } - Long64_t GetFirst() const { return fFirst; } - void SetFirst(Long64_t first) { fFirst = first; } - Long64_t GetNum() const { return fNum; } - Long64_t GetEntries(Bool_t istree = kTRUE, Bool_t openfile = kTRUE); - void SetEntries(Long64_t ent) { fEntries = ent; } - const char *GetMsd() const { return fMsd; } - void SetNum(Long64_t num) { fNum = num; } - Bool_t GetValid() const { return fValid; } - const char *GetObjName() const { return GetTitle(); } - const char *GetDirectory() const; - const char *GetDataSet() const { return fDataSet; } - void SetDataSet(const char *dataset) { fDataSet = dataset; } - void AddAssocObj(TObject *assocobj); - TList *GetListOfAssocObjs() const { return fAssocObjList; } - TObject *GetAssocObj(Long64_t i, Bool_t isentry = kFALSE); - void Print(Option_t *options="") const override; - Long64_t GetTDSetOffset() const { return fTDSetOffset; } - void SetTDSetOffset(Long64_t offset) { fTDSetOffset = offset; } - void SetEntryList(TObject *aList, Long64_t first = -1, Long64_t num = -1); - TObject *GetEntryList() const { return fEntryList; } - void Validate(Bool_t isTree); - void Validate(TDSetElement *elem); - void Invalidate() { fValid = kFALSE; } - void SetValid() { fValid = kTRUE; } - Int_t Compare(const TObject *obj) const override; - Bool_t IsSortable() const override { return kTRUE; } - Int_t Lookup(Bool_t force = kFALSE); - void SetLookedUp() { SetBit(kHasBeenLookedUp); } - TFileInfo *GetFileInfo(const char *type = "TTree"); - - void Reset(); - - Float_t GetMaxProcTime() const { return fMaxProcTime; } - void SetMaxProcTime(Float_t mpt) { fMaxProcTime = mpt; } - - Int_t MergeElement(TDSetElement *elem); - - ClassDefOverride(TDSetElement,9) // A TDSet element -}; - - -class TDSet : public TNamed { - -public: - // TDSet status bits - enum EStatusBits { - kWriteV3 = BIT(16), - kEmpty = BIT(17), - kValidityChecked = BIT(18), // Set if elements validiy has been checked - kSomeInvalid = BIT(19), // Set if at least one element is invalid - kMultiDSet = BIT(20), // Set if fElements is a list of datasets - kIsLocal = BIT(21) // Set if local, non-PROOF, processing - }; - -private: - Bool_t fIsTree; // true if type is a TTree (or TTree derived) - TObject *fEntryList; //! entry (or event) list for processing - TProofChain *fProofChain; //! for browsing purposes - - void SplitEntryList(); //Split entry list between elements - - TDSet(const TDSet &); // not implemented - void operator=(const TDSet &); // not implemented - -protected: - TString fDir; // name of the directory - TString fType; // type of objects (e.g. TTree); - TString fObjName; // name of objects to be analyzed (e.g. TTree name) - THashList *fElements; //-> list of TDSetElements (or TDSets, if in multi mode) - TIter *fIterator; //! iterator on fElements - TDSetElement *fCurrent; //! current element - TList *fSrvMaps; //! list for mapping server coordinates for files - TIter *fSrvMapsIter; //! iterator on fSrvMaps - -public: - TDSet(); - TDSet(const char *name, const char *objname = "*", - const char *dir = "/", const char *type = 0); - TDSet(const TChain &chain, Bool_t withfriends = kTRUE); - ~TDSet() override; - - virtual Bool_t Add(const char *file, const char *objname = 0, - const char *dir = 0, Long64_t first = 0, - Long64_t num = -1, const char *msd = 0); - virtual Bool_t Add(TDSet *set); - virtual Bool_t Add(TCollection *fileinfo, const char *meta = 0, - Bool_t availableOnly = kFALSE, TCollection *badlist = 0); - virtual Bool_t Add(TFileInfo *fileinfo, const char *meta = 0); - virtual void AddFriend(TDSet *friendset, const char *alias); - - virtual Long64_t Process(TSelector *selector, Option_t *option = "", - Long64_t nentries = -1, - Long64_t firstentry = 0, - TObject *enl = 0); // *MENU* - virtual Long64_t Process(const char *selector, Option_t *option = "", - Long64_t nentries = -1, - Long64_t firstentry = 0, - TObject *enl = 0); // *MENU* - virtual Long64_t Draw(const char *varexp, const char *selection, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0); // *MENU* - virtual Long64_t Draw(const char *varexp, const TCut &selection, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0); // *MENU* - void Draw(Option_t *opt) override { Draw(opt, "", "", -1, 0); } - - Int_t ExportFileList(const char *filepath, Option_t *opt = ""); - - void Print(Option_t *option="") const override; - - void SetObjName(const char *objname); - void SetDirectory(const char *dir); - - Bool_t IsTree() const { return fIsTree; } - Bool_t IsValid() const { return !fType.IsNull(); } - Bool_t ElementsValid(); - const char *GetType() const { return fType; } - const char *GetObjName() const { return fObjName; } - const char *GetDirectory() const { return fDir; } - TList *GetListOfElements() const { return (TList *)fElements; } - Int_t GetNumOfFiles(); - - Int_t Remove(TDSetElement *elem, Bool_t deleteElem = kTRUE); - - virtual void Reset(); - virtual TDSetElement *Next(Long64_t totalEntries = -1); - TDSetElement *Current() const { return fCurrent; }; - - static Long64_t GetEntries(Bool_t isTree, const char *filename, - const char *path, TString &objname); - - void AddInput(TObject *obj); - void ClearInput(); - TObject *GetOutput(const char *name); - TList *GetOutputList(); - virtual void StartViewer(); // *MENU* - - virtual TTree *GetTreeHeader(TProof *proof); - virtual void SetEntryList(TObject *aList); - TObject *GetEntryList() const { return fEntryList; } - void Validate(); - void Validate(TDSet *dset); - - void Lookup(Bool_t removeMissing = kFALSE, TList **missingFiles = 0); - void SetLookedUp(); - - void SetSrvMaps(TList *srvmaps = 0); - - void SetWriteV3(Bool_t on = kTRUE); - - ClassDefOverride(TDSet,9) // Data set for remote processing (PROOF) -}; - -#endif diff --git a/proof/proof/inc/TDSetProxy.h b/proof/proof/inc/TDSetProxy.h deleted file mode 100644 index 957d0ed199027..0000000000000 --- a/proof/proof/inc/TDSetProxy.h +++ /dev/null @@ -1,46 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Maarten Ballintijn 12/03/02 - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TDSetProxy -#define ROOT_TDSetProxy - - -////////////////////////////////////////////////////////////////////////// -// // -// TDSetProxy // -// // -// TDSet proxy for use on slaves. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TDSet.h" - -class TProofServ; - - -class TDSetProxy : public TDSet { - -private: - TProofServ *fServ; //! - -public: - TDSetProxy(); - TDSetProxy(const char *type, const char *objname = "*", const char *dir = "/"); - - void Reset() override; - TDSetElement *Next(Long64_t totalEntries = -1) override; - - void SetProofServ(TProofServ *serv); - - ClassDefOverride(TDSetProxy,1) // TDSet proxy for use on slaves -}; - -#endif diff --git a/proof/proof/inc/TDataSetManager.h b/proof/proof/inc/TDataSetManager.h deleted file mode 100644 index 007c1c3b44d32..0000000000000 --- a/proof/proof/inc/TDataSetManager.h +++ /dev/null @@ -1,161 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Jan Fiete Grosse-Oetringhaus, 08.08.07 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TDataSetManager -#define ROOT_TDataSetManager - -////////////////////////////////////////////////////////////////////////// -// // -// TDataSetManager // -// // -// This class contains functions to handle datasets in PROOF // -// It is the layer between TProofServ and the file system that stores // -// the datasets. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TString.h" -#include "TMap.h" -#include "TUri.h" - - -class TFileCollection; -class TFileInfo; -class TMD5; -class TUrl; -class TVirtualMonitoringWriter; -class TFileStager; - -class TDataSetManager : public TObject { - -private: - TDataSetManager(const TDataSetManager&); // not implemented - TDataSetManager& operator=(const TDataSetManager&); // not implemented - -protected: - TString fGroup; // Group to which the owner of this session belongs - TString fUser; // Owner of the session - TString fCommonUser; // User that stores the COMMON datasets - TString fCommonGroup; // Group that stores the COMMON datasets - - TUri fBase; // Base URI used to parse dataset names - - TMap fGroupQuota; // Group quotas (read from config file) - TMap fGroupUsed; // --> (TParameter) - TMap fUserUsed; // --> --> - - Long64_t fAvgFileSize; // Average file size to be used to estimate the dataset size (in MB) - - Int_t fNTouchedFiles; // Number of files touched in the last ScanDataSet operation - Int_t fNOpenedFiles; // Number of files opened in the last ScanDataSet operation - Int_t fNDisappearedFiles; // Number of files disappared in the last ScanDataSet operation - - TString fGroupConfigFile; // Path to the group config file - Long_t fMTimeGroupConfig; // Last modification of the group config file - - static TString fgCommonDataSetTag; // Name for common datasets, default: COMMON - - static TList *fgDataSetSrvMaps; // List of TPair(TRegexp, TObjString) for mapping server coordinates - // for dataset files (init from DataSet.SrvMap) - - virtual TMap *GetGroupUsedMap() { return &fGroupUsed; } - virtual TMap *GetUserUsedMap() { return &fUserUsed; } - Int_t GetNTouchedFiles() const { return fNTouchedFiles; } - Int_t GetNOpenedFiles() const { return fNOpenedFiles; } - Int_t GetNDisapparedFiles() const { return fNDisappearedFiles; } - void GetQuota(const char *group, const char *user, const char *dsName, TFileCollection *dataset); - void PrintDataSet(TFileCollection *fc, Int_t popt = 0); - void PrintUsedSpace(); - Bool_t ReadGroupConfig(const char *cf = 0); - virtual void UpdateUsedSpace(); - - static Long64_t ToBytes(const char *size = 0); - -public: - enum EDataSetStatusBits { - kCheckQuota = BIT(15), // quota checking enabled - kAllowRegister = BIT(16), // allow registration of a new dataset - kAllowVerify = BIT(17), // allow verification of a dataset (requires registration permit) - kTrustInfo = BIT(18), // during registration, trust the available information provided by the user - kIsSandbox = BIT(19), // dataset dir is in the user sandbox (simplified naming) - kUseCache = BIT(20), // force the usage of cache - kDoNotUseCache = BIT(21) // disable the cache - }; - - enum EDataSetWorkOpts { // General (bits 1-8) - kDebug = 0x1, kShowDefault = 0x2, kPrint = 0x4, kExport = 0x8, - kQuotaUpdate = 0x10, kSetDefaultTree = 0x20, kForceScan = 0x40, - kNoHeaderPrint = 0x80, - // File-based specific (bits 9-16) - kReopen = 0x100, kTouch = 0x200, kMaxFiles = 0x400, kReadShort = 0x800, - kFileMustExist = 0x1000, - kNoAction = 0x2000, kLocateOnly = 0x4000, kStageOnly = 0x8000, - // Auxilliary bits (bits 17-) - kNoCacheUpdate = 0x10000, kRefreshLs = 0x20000, kList = 0x40000, - kAllFiles = 0x80000, kStagedFiles = 0x100000, kNoStagedCheck = 0x200000 - }; - - TDataSetManager(const char *group = 0, const char *user = 0, const char *options = 0); - ~TDataSetManager() override; - - virtual Int_t ClearCache(const char *uri); - virtual Long64_t GetAvgFileSize() const { return fAvgFileSize; } - virtual TFileCollection *GetDataSet(const char *uri, const char *server = 0); - virtual TMap *GetDataSets(const char *uri, UInt_t /*option*/ = TDataSetManager::kExport); - virtual TMap *GetSubDataSets(const char *uri, const char *excludeservers); - - virtual Long64_t GetGroupQuota(const char *group); - virtual TMap *GetGroupQuotaMap() { return &fGroupQuota; } - virtual Long64_t GetGroupUsed(const char *group); - virtual Bool_t ExistsDataSet(const char *uri); - virtual void MonitorUsedSpace(TVirtualMonitoringWriter *monitoring); - virtual Int_t NotifyUpdate(const char *group = 0, const char *user = 0, - const char *dspath = 0, Long_t mtime = 0, const char *checksum = 0); - Bool_t ParseUri(const char *uri, TString *dsGroup = 0, TString *dsUser = 0, - TString *dsName = 0, TString *dsTree = 0, - Bool_t onlyCurrent = kFALSE, Bool_t wildcards = kFALSE); - virtual void ParseInitOpts(const char *opts); - virtual Bool_t RemoveDataSet(const char *uri); - virtual Int_t RegisterDataSet(const char *uri, TFileCollection *dataSet, const char *opt); - Int_t ScanDataSet(const char *uri, const char *opt); - virtual Int_t ScanDataSet(const char *uri, UInt_t option = kReopen | kDebug); - void SetScanCounters(Int_t t = -1, Int_t o = -1, Int_t d = -1); - virtual Int_t ShowCache(const char *uri); - virtual void ShowQuota(const char *opt); - - virtual void ShowDataSets(const char *uri = "*", const char *opt = ""); - - static TString CreateUri(const char *dsGroup = 0, const char *dsUser = 0, - const char *dsName = 0, const char *dsTree = 0); - static Bool_t CheckDataSetSrvMaps(TUrl *furl, TString &fn, TList *srvmaplist = 0); - static TList *GetDataSetSrvMaps(); - static TList *ParseDataSetSrvMaps(const TString &srvmaps); - static Int_t ScanDataSet(TFileCollection *dataset, Int_t fopt, Int_t sopt = 0, Int_t ropt = 0, - Bool_t dbg = kFALSE, - Int_t *touched = 0, Int_t *opened = 0, Int_t *disappeared = 0, - TList *flist = 0, Long64_t avgsz = -1, const char *mss = 0, - Int_t maxfiles = -1, const char *stageopts = 0); - static Bool_t CheckStagedStatus(TFileInfo *fileInfo, Int_t fopt, Int_t maxfiles, - Int_t newstagedfiles, TFileStager* stager, Bool_t createStager, - Bool_t dbg, Bool_t &changed, Bool_t& touched, - Bool_t& disappeared); - static void ProcessFile(TFileInfo *fileInfo, Int_t sopt, Bool_t checkstg, Bool_t doall, - TFileStager* stager, Bool_t createStager, const char* stageopts, - Bool_t dbg, Bool_t &changed, Bool_t& opened); - - static Int_t ScanFile(TFileInfo *fileinfo, Bool_t notify); - static Int_t FillMetaData(TFileInfo *fi, TDirectory *d, const char *rdir = "/"); - - ClassDefOverride(TDataSetManager, 0) // Abstract data set manager class -}; - -#endif diff --git a/proof/proof/inc/TDataSetManagerFile.h b/proof/proof/inc/TDataSetManagerFile.h deleted file mode 100644 index 1a2a91523abeb..0000000000000 --- a/proof/proof/inc/TDataSetManagerFile.h +++ /dev/null @@ -1,103 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Jan Fiete Grosse-Oetringhaus, 08.08.07 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TDataSetManagerFile -#define ROOT_TDataSetManagerFile - -////////////////////////////////////////////////////////////////////////// -// // -// TDataSetManagerFile // -// // -// Implementation of TDataSetManager handling datasets from root // -// files under a specific directory path // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TDataSetManager.h" - -const char* const kDataSet_LocalCache = "dataset.cache"; // default cache subdirectory -const char* const kDataSet_DataSetList = "dataset.list"; // file with info about all datasets -const char* const kDataSet_LockLocation = "lock.location"; // location of the lock file - -class TDataSetManagerFile : public TDataSetManager { - -private: - TString fDataSetDir; // Location of datasets - TString fMSSUrl; // URL for the Mass Storage System - TString fStageOpts; // Option string to be used in issuing staging requests - TString fDataSetLockFile; // Dataset lock file - Int_t fLockFileTimeLimit; // Limit in seconds after a lock automatically expires - TString fListFile; // File to check repository updates - Bool_t fIsRemote; // True if the repository is remote - Bool_t fUseCache; // True if the cache is used for browsing remote repositories - TString fLocalCacheDir; // Local cache when the repository is remote - Int_t fCacheUpdatePeriod; // Period for checking for new updated information - Bool_t fOpenPerms; // True if file permissions must be open - - // Local cache handling - void InitLocalCache(); - Int_t CheckLocalCache(const char *group, const char *user, const char *dsName = "ls", UInt_t option = 0); - -protected: - const char *GetDataSetPath(const char *group, const char *user, const char *dsName); - const char *GetDataSetPath(const char *group, const char *user, const char *dsName, - TString &md5path, Bool_t local = kFALSE); - void Init(); - Bool_t BrowseDataSets(const char *group, const char *user, const char *dsName, - UInt_t option, TObject *target); - - Bool_t RemoveDataSet(const char *group, const char *user, const char *dsName); - Bool_t ExistsDataSet(const char *group, const char *user, const char *dsName); - - Int_t ScanDataSet(const char *group, const char *user, const char *dsName, UInt_t option = kReopen | kDebug); - - Int_t ChecksumDataSet(const char *path, const char *md5path, TString &checksum); - - Int_t CreateLsFile(const char *group, const char *user, Long_t &mtime, TString &checksum); - Int_t FillLsDataSet(const char *group, const char *user, const char *dsName, TList *out, UInt_t option); - - void UpdateUsedSpace() override; - -public: - TDataSetManagerFile() : TDataSetManager(0, 0, 0) { } - TDataSetManagerFile(const char *group, const char *user, const char *ins); - TDataSetManagerFile(const char *ins); - ~TDataSetManagerFile() override { } - - void ParseInitOpts(const char *opts) override; - - Int_t ClearCache(const char *uri = 0) override; - TFileCollection *GetDataSet(const char *uri, const char *srv = 0) override; - TMap *GetDataSets(const char *uri, UInt_t option = TDataSetManager::kExport) override; - Bool_t ExistsDataSet(const char *uri) override; - Bool_t RemoveDataSet(const char *uri) override; - - Int_t RegisterDataSet(const char *uri, TFileCollection *dataSet, const char *opt) override; - Int_t ScanDataSet(const char *uri, UInt_t option = kReopen | kDebug) override; - Int_t NotifyUpdate(const char *group, const char *user, - const char *dspath, Long_t mtime, const char *checksum = 0) override; - Int_t ShowCache(const char *uri = 0) override; - - // These should / could be private but they are used directly by the external daemon - TFileCollection *GetDataSet(const char *group, const char *user, const char *dsName, - UInt_t option = 0, TMD5 **checksum = 0); - TMap *GetDataSets(const char *group, const char *user, const char *dsName = 0, - UInt_t option = 0); - const char *GetMSSUrl() const { return fMSSUrl; } - const char *GetStageOpts() const { return fStageOpts; } - Int_t WriteDataSet(const char *group, const char *user, const char *dsName, - TFileCollection *dataset, UInt_t option = 0, TMD5 *checksum = 0); - Long_t GetModTime(const char *uri); - - ClassDefOverride(TDataSetManagerFile, 0) // DataSet manager for files -}; - -#endif diff --git a/proof/proof/inc/TLockPath.h b/proof/proof/inc/TLockPath.h deleted file mode 100644 index f3b6243474e84..0000000000000 --- a/proof/proof/inc/TLockPath.h +++ /dev/null @@ -1,56 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis, Oct 2015 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TLockPath -#define ROOT_TLockPath - -////////////////////////////////////////////////////////////////////////// -// // -// TLockPath // -// // -// Path locking class allowing shared and exclusive locks // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TString.h" - -class TLockPath : public TObject { -private: - TString fName; // path to lock - Int_t fLockId; // file id of dir lock - -public: - TLockPath(const char *path = ""); - ~TLockPath() override { if (IsLocked()) Unlock(); } - - const char *GetName() const override { return fName; } - void SetName(const char *path) { fName = path; } - - Int_t Lock(Bool_t shared = kFALSE); - Int_t Unlock(); - - Bool_t IsLocked() const { return (fLockId > -1); } - - ClassDefOverride(TLockPath, 0) // Path locking class -}; - -class TLockPathGuard { -private: - TLockPath *fLocker; //locker instance - -public: - TLockPathGuard(TLockPath *l, Bool_t shared = kFALSE) { - fLocker = l; fLocker->Lock(shared); } - ~TLockPathGuard() { fLocker->Unlock(); } -}; - -#endif diff --git a/proof/proof/inc/TPackMgr.h b/proof/proof/inc/TPackMgr.h deleted file mode 100644 index a3a7fd56a8a49..0000000000000 --- a/proof/proof/inc/TPackMgr.h +++ /dev/null @@ -1,109 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis, Oct 2011 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TPackMgr -#define ROOT_TPackMgr - - -////////////////////////////////////////////////////////////////////////// -// // -// TPackMgr // -// // -// The PROOF manager interacts with the PROOF server coordinator to // -// create or destroy a PROOF session, attach to or detach from // -// existing one, and to monitor any client activity on the cluster. // -// At most one manager instance per server is allowed. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TLockPath.h" -#include "TNamed.h" -#include "TMD5.h" -#include "TList.h" -#include "TString.h" - -typedef void (*TPackMgrLog_t)(const char *); - -class TList; -class THashList; -class TPackMgr : public TNamed { -public: - enum ECheckVersionOpt { kDontCheck = 0, kCheckROOT = 1, kCheckGIT = 2}; - -private: - TPackMgrLog_t fLogger; // Logger - TString fName; // Key identifying this package manager - TString fDir; // Directory with packages - TLockPath fLock; // Locker - TString fPfx; // Prefix for notifications, if any - TList *fEnabledPackages; // List of packages enabled - - static THashList *fgGlobalPackMgrList; // list of package managers for global packages - -private: - TPackMgr(const TPackMgr&); // Not implemented - TPackMgr& operator=(const TPackMgr&); // Not implemented - - void Log(const char *msg); - -public: - TPackMgr(const char *dir, const char *key = "L0"); - ~TPackMgr() override; - - const char *GetName() const override { return fName.Data(); } - void SetName(const char *name) override { fName = name; } - const char *GetTitle() const override { return GetDir(); } - - TLockPath *GetLock() { return &fLock; } - - void SetLogger(TPackMgrLog_t logger) { fLogger = logger; } - void SetPrefix(const char *pfx) { fPfx = pfx; } - - Int_t Build(const char *pack, Int_t opt = TPackMgr::kCheckROOT); - Int_t Load(const char *pack, TList *optls = 0); - Int_t Load(const char *pack, const char *opts); - Int_t Unload(const char *pack); - - Bool_t Has(const char *pack); - Bool_t IsInDir(const char *path); - const char *GetDir() const { return fDir.Data(); } - Int_t GetPackDir(const char *pack, TString &pdir); - Int_t GetParPath(const char *pack, TString &path); - Int_t GetDownloadDir(TString &dldir); - void GetEnabledPackages(TString &packlist); - Bool_t IsPackageEnabled(const char *pack) { - return (fEnabledPackages && - fEnabledPackages->FindObject(pack) ? kTRUE : kFALSE); } - - void Show(const char *title = 0); - Int_t Clean(const char *pack); - Int_t Remove(const char *pack = 0, Bool_t dolock = kTRUE); - TList *GetList() const; - - void ShowEnabled(const char *title = 0); - TList* GetListOfEnabled() const; - - TMD5 *GetMD5(const char *pack); - TMD5 *ReadMD5(const char *pack); - - Int_t Install(const char *par, Bool_t rmold = kFALSE); - Int_t Unpack(const char *pack, TMD5 *sum = 0); - - // Static methods - static TPackMgr *GetPackMgr(const char *pack, TPackMgr *packmgr = nullptr); - static Int_t RegisterGlobalPath(const char *paths); - static Int_t FindParPath(TPackMgr *packmgr, const char *pack, TString &par); - static Bool_t IsEnabled(const char *pack, TPackMgr *packmgr = nullptr); - - ClassDefOverride(TPackMgr,0) // Package manager interface -}; - -#endif diff --git a/proof/proof/inc/TProof.h b/proof/proof/inc/TProof.h deleted file mode 100644 index 7bcdb3deb49f7..0000000000000 --- a/proof/proof/inc/TProof.h +++ /dev/null @@ -1,1079 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 13/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProof -#define ROOT_TProof - - -////////////////////////////////////////////////////////////////////////// -// // -// TProof // -// // -// This class controls a Parallel ROOT Facility, PROOF, cluster. // -// It fires the worker servers, it keeps track of how many workers are // -// running, it keeps track of the workers running status, it broadcasts // -// messages to all workers, it collects results, etc. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProofMgr.h" -#include "TProofDebug.h" -#include "TString.h" -#include "TMacro.h" -#include "MessageTypes.h" -#include "TMD5.h" -#include "TRegexp.h" -#include "TSysEvtHandler.h" -#include "TUrl.h" -#include "TProofOutputList.h" -#include "TStopwatch.h" -#include "TVirtualMutex.h" -#include "TPackMgr.h" - -#include -#include - -#define CANNOTUSE(x) Info(x,"Not manager: cannot use this method") - -class TChain; -class TCondor; -class TCondorSlave; -class TDrawFeedback; -class TDSet; -class TEventList; -class THashList; -class TList; -class TCollection; -class TMessage; -class TMonitor; -class TPluginHandler; -class TProof; -class TProofInputHandler; -class TProofInterruptHandler; -class TProofLockPath; -class TVirtualProofPlayer; -class TProofPlayer; -class TProofPlayerRemote; -class TProofProgressDialog; -class TProofServ; -class TQueryResult; -class TSignalHandler; -class TSlave; -class TSocket; -class TTree; -class TFileCollection; -class TMap; -class TDataSetManager; -class TDataSetManagerFile; -class TMacro; -class TSelector; - -// protocol changes: -// 1 -> 2: new arguments for Process() command, option added -// 2 -> 3: package manager enabling protocol changed -// 3 -> 4: introduction of multi-level-master support -// 4 -> 5: added friends support -// 5 -> 6: drop TFTP, support for asynchronous queries -// 6 -> 7: support for multisessions, archieve, retrieve, ... -// 7 -> 8: return number of entries in GetNextPacket -// 8 -> 9: support for stateless connection via xproofd -// 9 -> 10: new features requested, tested at CAF -// 10 -> 11: new merging strategy -// 11 -> 12: new progress message -// 12 -> 13: exchange version/architecture/compiler info -// 13 -> 14: new proofserv environment setting -// 14 -> 15: add support for entry lists; new version of TFileInfo -// 15 -> 16: add support for generic non-data based processing -// 16 -> 17: new dataset handling system; support for TFileCollection processing -// 17 -> 18: support for reconnection on daemon restarts -// 18 -> 19: TProofProgressStatus used in kPROOF_PROGRESS, kPROOF_STOPPROCESS -// and kPROOF_GETNEXTPACKET messages in Master - worker communication -// 19 -> 20: Fix the asynchronous mode (required changes in some messages) -// 20 -> 21: Add support for session queuing -// 21 -> 22: Add support for switching from sync to async while running ('Ctrl-Z' functionality) -// 22 -> 23: New dataset features (default tree name; classification per fileserver) -// 23 -> 24: Merging optimization -// 24 -> 25: Handling of 'data' dir; group information -// 25 -> 26: Use new TProofProgressInfo class -// 26 -> 27: Use new file for updating the session status -// 27 -> 28: Support for multi-datasets, fix global pack dirs, fix AskStatistics, -// package download, dataset caching -// 28 -> 29: Support for config parameters in EnablePackage, idle-timeout -// 29 -> 30: Add information about data dir in TSlaveInfo -// 30 -> 31: Development cycle 5.29 -// 31 -> 32: New log path trasmission -// 32 -> 33: Development cycle 5.29/04 (fixed worker activation, new startup technology, ...) -// 33 -> 34: Development cycle 5.33/02 (fix load issue, ...) -// 34 -> 35: Development cycle 5.99/01 (PLite on workers, staging requests in separate dsmgr...) -// 35 -> 36: SetParallel in dynamic mode (changes default in GoParallel), cancel staging requests -// 36 -> 37: Support for remote (web) PAR packages - -// PROOF magic constants -const Int_t kPROOF_Protocol = 37; // protocol version number -const Int_t kPROOF_Port = 1093; // IANA registered PROOF port -const char* const kPROOF_ConfFile = "proof.conf"; // default config file -const char* const kPROOF_ConfDir = "/usr/local/root"; // default config dir -const char* const kPROOF_WorkDir = ".proof"; // default working directory -const char* const kPROOF_CacheDir = "cache"; // file cache dir, under WorkDir -const char* const kPROOF_PackDir = "packages"; // package dir, under WorkDir -const char* const kPROOF_PackDownloadDir = "downloaded"; // subdir with downloaded PARs, under PackDir -const char* const kPROOF_QueryDir = "queries"; // query dir, under WorkDir -const char* const kPROOF_DataSetDir = "datasets"; // dataset dir, under WorkDir -const char* const kPROOF_DataDir = "data"; // dir for produced data, under WorkDir -const char* const kPROOF_CacheLockFile = "proof-cache-lock-"; // cache lock file -const char* const kPROOF_PackageLockFile = "proof-package-lock-"; // package lock file -const char* const kPROOF_QueryLockFile = "proof-query-lock-"; // query lock file -const char* const kPROOF_TerminateWorker = "+++ terminating +++"; // signal worker termination in MarkBad -const char* const kPROOF_WorkerIdleTO = "+++ idle-timeout +++"; // signal worker idle timeout in MarkBad -const char* const kPROOF_InputDataFile = "inputdata.root"; // Default input data file name -const char* const kPROOF_MissingFiles = "MissingFiles"; // Missingfile list name -const Long64_t kPROOF_DynWrkPollInt_s = 10; // minimum number of seconds between two polls for dyn wrks - -#ifndef R__WIN32 -const char* const kCP = "/bin/cp -fp"; -const char* const kRM = "/bin/rm -rf"; -const char* const kLS = "/bin/ls -l"; -const char* const kUNTAR = "%s -c %s/%s | (cd %s; tar xf -)"; -const char* const kUNTAR2 = "%s -c %s | (cd %s; tar xf -)"; -const char* const kUNTAR3 = "%s -c %s | (tar xf -)"; -const char* const kGUNZIP = "gunzip"; -#else -const char* const kCP = "copy"; -const char* const kRM = "delete"; -const char* const kLS = "dir"; -const char* const kUNTAR = "..."; -const char* const kUNTAR2 = "..."; -const char* const kUNTAR3 = "..."; -const char* const kGUNZIP = "gunzip"; -#endif - -typedef void (*PrintProgress_t)(Long64_t tot, Long64_t proc, Float_t proctime, Long64_t bytes); - -// Structure for the progress information -class TProofProgressInfo : public TObject { -public: - Long64_t fTotal; // Total number of events to process - Long64_t fProcessed; // Number of events processed - Long64_t fBytesRead; // Number of bytes read - Float_t fInitTime; // Time for initialization - Float_t fProcTime; // Time for processing - Float_t fEvtRateI; // Instantaneous event rate - Float_t fMBRateI; // Instantaneous byte read rate - Int_t fActWorkers; // Numebr of workers still active - Int_t fTotSessions; // Numebr of PROOF sessions running currently on the clusters - Float_t fEffSessions; // Number of effective sessions running on the machines allocated to this session - TProofProgressInfo(Long64_t tot = 0, Long64_t proc = 0, Long64_t bytes = 0, - Float_t initt = -1., Float_t proct = -1., - Float_t evts = -1., Float_t mbs = -1., - Int_t actw = 0, Int_t tsess = 0, Float_t esess = 0.) : - fTotal(tot), fProcessed(proc), fBytesRead(bytes), - fInitTime(initt), fProcTime(proct), fEvtRateI(evts), fMBRateI(mbs), - fActWorkers(actw), fTotSessions(tsess), fEffSessions(esess) { } - ~TProofProgressInfo() override { } - ClassDefOverride(TProofProgressInfo, 1); // Progress information -}; - -// PROOF Interrupt signal handler -class TProofInterruptHandler : public TSignalHandler { -private: - TProof *fProof; - - TProofInterruptHandler(const TProofInterruptHandler&); // Not implemented - TProofInterruptHandler& operator=(const TProofInterruptHandler&); // Not implemented -public: - TProofInterruptHandler(TProof *p) - : TSignalHandler(kSigInterrupt, kFALSE), fProof(p) { } - Bool_t Notify() override; -}; - -// Input handler for messages from TProofServ -class TProofInputHandler : public TFileHandler { -private: - TSocket *fSocket; - TProof *fProof; - - TProofInputHandler(const TProofInputHandler&); // Not implemented - TProofInputHandler& operator=(const TProofInputHandler&); // Not implemented -public: - TProofInputHandler(TProof *p, TSocket *s); - Bool_t Notify() override; - Bool_t ReadNotify() override { return Notify(); } -}; - -// Slaves info class -class TSlaveInfo : public TObject { -public: - enum ESlaveStatus { kActive, kNotActive, kBad }; - - TString fOrdinal; //slave ordinal - TString fHostName; //hostname this slave is running on - TString fMsd; //mass storage domain slave is in - TString fDataDir; //directory for user data - Int_t fPerfIndex; //relative performance of this slave - SysInfo_t fSysInfo; //Infomation about its hardware - ESlaveStatus fStatus; //slave status - - TSlaveInfo(const char *ordinal = "", const char *host = "", Int_t perfidx = 0, - const char *msd = "", const char *datadir = "") : - fOrdinal(ordinal), fHostName(host), fMsd(msd), fDataDir(datadir), - fPerfIndex(perfidx), fSysInfo(), fStatus(kNotActive) { } - - const char *GetDataDir() const { return fDataDir; } - const char *GetMsd() const { return fMsd; } - const char *GetName() const override { return fHostName; } - const char *GetOrdinal() const { return fOrdinal; } - SysInfo_t GetSysInfo() const { return fSysInfo; } - void SetStatus(ESlaveStatus stat) { fStatus = stat; } - void SetSysInfo(SysInfo_t si); - void SetOrdinal(const char *ord) { fOrdinal = ord; } - - Int_t Compare(const TObject *obj) const override; - Bool_t IsSortable() const override { return kTRUE; } - void Print(Option_t *option="") const override; - Bool_t IsEqual(const TObject* obj) const override; - - ClassDefOverride(TSlaveInfo,4) //basic info on workers -}; - -// Merger info class -class TMergerInfo : public TObject { -private: - - TSlave *fMerger; // Slave that acts as merger - Int_t fPort; // Port number, on which it accepts outputs from other workers - Int_t fMergedObjects; // Total number of objects it must accept from other workers - // (-1 == not set yet) - Int_t fWorkersToMerge; // Number of workers that are merged on this merger - // (does not change during time) - Int_t fMergedWorkers; // Current number of already merged workers - // (does change during time as workers are being merged) - - TList *fWorkers; // List of already assigned workers - Bool_t fIsActive; // Merger state - - TMergerInfo(const TMergerInfo&); // Not implemented - TMergerInfo& operator=(const TMergerInfo&); // Not implemented - -public: - TMergerInfo(TSlave *t, Int_t port, Int_t forHowManyWorkers) : - fMerger(t), fPort(port), fMergedObjects(0), fWorkersToMerge(forHowManyWorkers), - fMergedWorkers(0), fWorkers(0), fIsActive(kTRUE) { } - ~TMergerInfo() override; - - void AddWorker(TSlave *sl); - TList *GetWorkers() { return fWorkers; } - - TSlave *GetMerger() { return fMerger; } - Int_t GetPort() { return fPort; } - - Int_t GetWorkersToMerge() { return fWorkersToMerge; } - Int_t GetMergedWorkers() { return fMergedWorkers; } - Int_t GetMergedObjects() { return fMergedObjects; } - - void SetMergedWorker(); - void AddMergedObjects(Int_t objects) { fMergedObjects += objects; } - - Bool_t AreAllWorkersAssigned(); - Bool_t AreAllWorkersMerged(); - - void Deactivate() { fIsActive = kFALSE; } - Bool_t IsActive() { return fIsActive; } - - ClassDefOverride(TMergerInfo,0) // Basic info on merger, i.e. worker serving as merger -}; - -// Small auxiliary class for merging progress notification -class TProofMergePrg { -private: - TString fExp; - Int_t fIdx; - Int_t fNWrks; - Int_t fLastNWrks; - static char fgCr[4]; -public: - TProofMergePrg() : fExp(), fIdx(-1), fNWrks(-1), fLastNWrks(-1) { } - - const char *Export(Bool_t &changed) { - fExp.Form("%c (%d workers still sending) ", fgCr[fIdx], fNWrks); - changed = (fLastNWrks != fNWrks || fLastNWrks == -1) ? kTRUE : kFALSE; - fLastNWrks = fNWrks; - return fExp.Data(); } - void DecreaseNWrks() { fNWrks--; } - void IncreaseNWrks() { fNWrks++; } - void IncreaseIdx() { fIdx++; if (fIdx == 4) fIdx = 0; } - void Reset(Int_t n = -1) { fIdx = -1; SetNWrks(n); } - void SetNWrks(Int_t n) { fNWrks = n; } -}; - -class TProof : public TNamed, public TQObject { - -friend class TPacketizer; -friend class TPacketizerDev; -friend class TPacketizerAdaptive; -friend class TProofLite; -friend class TDataSetManager; -friend class TProofServ; -friend class TProofInputHandler; -friend class TProofInterruptHandler; -friend class TProofPlayer; -friend class TProofPlayerLite; -friend class TProofPlayerRemote; -friend class TProofProgressDialog; -friend class TSlave; -friend class TSlaveLite; -friend class TVirtualPacketizer; -friend class TXSlave; -friend class TXSocket; // to access kPing -friend class TXSocketHandler; // to access fCurrentMonitor and CollectInputFrom -friend class TXProofMgr; // to access EUrgent -friend class TXProofServ; // to access EUrgent - -public: - // PROOF status bits - enum EStatusBits { - kUsingSessionGui = BIT(14), - kNewInputData = BIT(15), - kIsClient = BIT(16), - kIsMaster = BIT(17), - kIsTopMaster = BIT(18), - kUseProgressDialog = BIT(19) - }; - enum EQueryMode { - kSync = 0, - kAsync = 1 - }; - enum EUploadOpt { - kAppend = 0x1, - kOverwriteDataSet = 0x2, - kNoOverwriteDataSet = 0x4, - kOverwriteAllFiles = 0x8, - kOverwriteNoFiles = 0x10, - kAskUser = 0x0 - }; - enum ERegisterOpt { - kFailIfExists = 0, - kOverwriteIfExists = 1, - kMergeIfExists = 2 - }; - enum EUploadPackageOpt { - kUntar = 0x0, //Untar over existing dir [default] - kRemoveOld = 0x1 //Remove existing dir with same name - }; - enum ERunStatus { - kRunning = 0, // Normal status - kStopped = 1, // After the stop button has been pressed - kAborted = 2 // After the abort button has been pressed - }; - - enum ESubMerger { - kOutputSize = 1, //Number of objects in worker's output list - kSendOutput = 2, //Naster asks worker for its output list - kBeMerger = 3, //Master tells worker to be a merger - kMergerDown = 4, //Merger cannot serve - kStopMerging = 5, //Master tells worker to stop merging (and return output) - kOutputSent = 6 //Worker reports sending its output to given worker - }; - - enum EProofClearData { - kPurge = 0x1, - kUnregistered = 0x2, - kDataset = 0x4, - kForceClear = 0x8 - }; - -private: - enum EUrgent { - kLocalInterrupt = -1, - kPing = 0, - kHardInterrupt = 1, - kSoftInterrupt, - kShutdownInterrupt - }; - enum EProofCacheCommands { - kShowCache = 1, - kClearCache = 2, - kShowPackages = 3, - kClearPackages = 4, - kClearPackage = 5, - kBuildPackage = 6, - kLoadPackage = 7, - kShowEnabledPackages = 8, - kShowSubCache = 9, - kClearSubCache = 10, - kShowSubPackages = 11, - kDisableSubPackages = 12, - kDisableSubPackage = 13, - kBuildSubPackage = 14, - kUnloadPackage = 15, - kDisablePackage = 16, - kUnloadPackages = 17, - kDisablePackages = 18, - kListPackages = 19, - kListEnabledPackages = 20, - kLoadMacro = 21 - }; - enum EProofDataSetCommands { - kUploadDataSet = 1, //Upload a dataset - kCheckDataSetName = 2, //Check wheter dataset of this name exists - kGetDataSets = 3, //List datasets saved on the master node - kRegisterDataSet = 4, //Save a TList object as a dataset - kGetDataSet = 5, //Get a TFileCollection of TFileInfo objects - kVerifyDataSet = 6, //Try open all files from a dataset and report results - kRemoveDataSet = 7, //Remove a dataset but leave files belonging to it - kMergeDataSet = 8, //Add new files to an existing dataset - kShowDataSets = 9, //Shows datasets, returns formatted output - kGetQuota = 10, //Get quota info per group - kShowQuota = 11, //Show quotas - kSetDefaultTreeName = 12, //Set the default tree name - kCache = 13, //Show/clear cache - kRequestStaging = 14, //Request staging of a dataset - kStagingStatus = 15, //Obtain staging status for the given dataset - kCancelStaging = 16 //Cancels dataset staging request - }; - enum ESendFileOpt { - kAscii = 0x0, - kBinary = 0x1, - kForce = 0x2, - kForward = 0x4, - kCpBin = 0x8, - kCp = 0x10 - }; - enum EProofWrkListAction { - kActivateWorker = 1, - kDeactivateWorker = 2 - }; - enum EBuildPackageOpt { - kDontBuildOnClient = -2, - kBuildOnSlavesNoWait = -1, - kBuildAll = 0, - kCollectBuildResults = 1 - }; - enum EProofShowQuotaOpt { - kPerGroup = 0x1, - kPerUser = 0x2 - }; - - Bool_t fValid; //is this a valid proof object - Bool_t fTty; //TRUE if connected to a terminal - TString fMaster; //master server ("" if a master); used in the browser - TString fWorkDir; //current work directory on remote servers - TString fGroup; //PROOF group of this user - Int_t fLogLevel; //server debug logging level - Int_t fStatus; //remote return status (part of kPROOF_LOGDONE) - Int_t fCheckFileStatus; //remote return status after kPROOF_CHECKFILE - TList *fRecvMessages; //Messages received during collect not yet processed - TList *fSlaveInfo; //!list returned by kPROOF_GETSLAVEINFO - Bool_t fSendGroupView; //if true send new group view - Bool_t fIsPollingWorkers; //will be set to kFALSE to prevent recursive dyn workers check in dyn mode - Long64_t fLastPollWorkers_s; //timestamp (in seconds) of last poll for workers, -1 if never checked - TList *fActiveSlaves; //list of active slaves (subset of all slaves) - TString fActiveSlavesSaved;// comma-separated list of active slaves (before last call to - // SetParallel or Activate/DeactivateWorkers) - TList *fInactiveSlaves; //list of inactive slaves (good but not used for processing) - TList *fUniqueSlaves; //list of all active slaves with unique file systems - TList *fAllUniqueSlaves; //list of all active slaves with unique file systems, including all submasters - TList *fNonUniqueMasters; //list of all active masters with a nonunique file system - TMonitor *fActiveMonitor; //monitor activity on all active slave sockets - TMonitor *fUniqueMonitor; //monitor activity on all unique slave sockets - TMonitor *fAllUniqueMonitor; //monitor activity on all unique slave sockets, including all submasters - TMonitor *fCurrentMonitor; //currently active monitor - Long64_t fBytesRead; //bytes read by all slaves during the session - Float_t fRealTime; //realtime spent by all slaves during the session - Float_t fCpuTime; //CPU time spent by all slaves during the session - TSignalHandler *fIntHandler; //interrupt signal handler (ctrl-c) - TPluginHandler *fProgressDialog; //progress dialog plugin - Bool_t fProgressDialogStarted; //indicates if the progress dialog is up - TVirtualProofPlayer *fPlayer; //current player - TList *fFeedback; //list of names to be returned as feedback - TList *fChains; //chains with this proof set - struct MD5Mod_t { - TMD5 fMD5; //file's md5 - Long_t fModtime; //file's modification time - }; - typedef std::map FileMap_t; - FileMap_t fFileMap; //map keeping track of a file's md5 and mod time - TDSet *fDSet; //current TDSet being validated - - Int_t fNotIdle; //Number of non-idle sub-nodes - Bool_t fSync; //true if type of currently processed query is sync - ERunStatus fRunStatus; //run status - Bool_t fIsWaiting; //true if queries have been enqueued - - Bool_t fRedirLog; //redirect received log info - TString fLogFileName; //name of the temp file for redirected logs - FILE *fLogFileW; //temp file to redirect logs - FILE *fLogFileR; //temp file to read redirected logs - Bool_t fLogToWindowOnly; //send log to window only - - Bool_t fSaveLogToMacro; // Whether to save received logs to TMacro fMacroLog (use with care) - TMacro fMacroLog; // Macro with the saved (last) log - - TProofMergePrg fMergePrg; //Merging progress - - TList *fWaitingSlaves; //stores a TPair of the slaves's TSocket and TMessage - TList *fQueries; //list of TProofQuery objects - Int_t fOtherQueries; //number of queries in list from previous sessions - Int_t fDrawQueries; //number of draw queries during this sessions - Int_t fMaxDrawQueries; //max number of draw queries kept - Int_t fSeqNum; //Remote sequential # of the last query submitted - - Int_t fSessionID; //remote ID of the session - - Bool_t fEndMaster; //true for a master in direct contact only with workers - - TPackMgr *fPackMgr; // Default package manager - TList *fEnabledPackagesOnCluster; //list of enabled packages - - TList *fInputData; //Input data objects sent over via file - TString fInputDataFile; //File with input data objects - - TProofOutputList fOutputList; // TList implementation filtering ls(...) and Print(...) - - PrintProgress_t fPrintProgress; //Function function to display progress info in batch mode - - std::recursive_mutex fCloseMutex; // Avoid crashes in MarkBad or alike while closing - - TList *fLoadedMacros; // List of loaded macros (just file names) - static TList *fgProofEnvList; // List of TNameds defining environment - // variables to pass to proofserv - - Bool_t fMergersSet; // Indicates, if the following variables have been initialized properly - Bool_t fMergersByHost; // Mergers assigned by host name - Int_t fMergersCount; - Int_t fWorkersToMerge; // Current total number of workers, which have not been yet assigned to any merger - Int_t fLastAssignedMerger; - TList *fMergers; - Bool_t fFinalizationRunning; - Int_t fRedirectNext; - - TString fPerfTree; // If non-null triggers saving of the performance info into fPerfTree - - TList *fWrksOutputReady; // List of workers ready to send output (in control output sending mode) - - static TPluginHandler *fgLogViewer; // Log dialog box plugin - -protected: - enum ESlaves { kAll, kActive, kUnique, kAllUnique }; - - Bool_t fMasterServ; //true if we are a master server - TUrl fUrl; //Url of the master - TString fConfFile; //file containing config information - TString fConfDir; //directory containing cluster config information - TString fImage; //master's image name - Int_t fProtocol; //remote PROOF server protocol version number - TList *fSlaves; //list of all slave servers as in config file - TList *fTerminatedSlaveInfos; //list of unique infos of terminated slaves - TList *fBadSlaves; //dead slaves (subset of all slaves) - TMonitor *fAllMonitor; //monitor activity on all valid slave sockets - Bool_t fDataReady; //true if data is ready to be analyzed - Long64_t fBytesReady; //number of bytes staged - Long64_t fTotalBytes; //number of bytes to be analyzed - TList *fAvailablePackages; //list of available packages - TList *fEnabledPackages; //list of enabled packages - TList *fRunningDSets; // Temporary datasets used for async running - - Int_t fCollectTimeout; // Timeout for (some) collect actions - - TString fDataPoolUrl; // default data pool entry point URL - TProofMgr::EServType fServType; // type of server: proofd, XrdProofd - TProofMgr *fManager; // manager to which this session belongs (if any) - EQueryMode fQueryMode; // default query mode - Bool_t fDynamicStartup; // are the workers started dynamically? - - TSelector *fSelector; // Selector to be processed, if any - - TStopwatch fQuerySTW; // Stopwatch to measure query times - Float_t fPrepTime; // Preparation time - -private: - TProof(const TProof &); // not implemented - void operator=(const TProof &); // idem - - void CleanGDirectory(TList *ol); - - Int_t Exec(const char *cmd, ESlaves list, Bool_t plusMaster); - Int_t SendCommand(const char *cmd, ESlaves list = kActive); - Int_t SendCurrentState(ESlaves list = kActive); - Int_t SendCurrentState(TList *list); - Bool_t CheckFile(const char *file, TSlave *sl, Long_t modtime, Int_t cpopt = (kCp | kCpBin)); - Int_t SendObject(const TObject *obj, ESlaves list = kActive); - Int_t SendGroupView(); - Int_t SendInitialState(); - Int_t SendPrint(Option_t *option=""); - Int_t Ping(ESlaves list); - void Interrupt(EUrgent type, ESlaves list = kActive); - void AskStatistics(); - void AskParallel(); - Int_t GoParallel(Int_t nodes, Bool_t accept = kFALSE, Bool_t random = kFALSE); - Int_t GoMoreParallel(Int_t nWorkersToAdd); - Int_t SetParallelSilent(Int_t nodes, Bool_t random = kFALSE); - void RecvLogFile(TSocket *s, Int_t size); - void NotifyLogMsg(const char *msg, const char *sfx = "\n"); - - Int_t BuildPackage(const char *package, EBuildPackageOpt opt = kBuildAll, Int_t chkveropt = TPackMgr::kCheckROOT, TList *workers = 0); - Int_t LoadPackage(const char *package, Bool_t notOnClient = kFALSE, TList *loadopts = 0, TList *workers = 0); - Int_t UnloadPackage(const char *package); - Int_t UnloadPackages(); - Int_t DisablePackage(const char *package); - Int_t DisablePackages(); - - void Activate(TList *slaves = 0); - Int_t Broadcast(const TMessage &mess, TList *slaves); - Int_t Broadcast(const TMessage &mess, ESlaves list = kActive); - Int_t Broadcast(const char *mess, Int_t kind, TList *slaves); - Int_t Broadcast(const char *mess, Int_t kind = kMESS_STRING, ESlaves list = kActive); - Int_t Broadcast(Int_t kind, TList *slaves) { return Broadcast(0, kind, slaves); } - Int_t Broadcast(Int_t kind, ESlaves list = kActive) { return Broadcast(0, kind, list); } - Int_t BroadcastFile(const char *file, Int_t opt, const char *rfile, TList *wrks); - Int_t BroadcastFile(const char *file, Int_t opt, const char *rfile = 0, ESlaves list = kAllUnique); - Int_t BroadcastGroupPriority(const char *grp, Int_t priority, ESlaves list = kAllUnique); - Int_t BroadcastGroupPriority(const char *grp, Int_t priority, TList *workers); - Int_t BroadcastObject(const TObject *obj, Int_t kind, TList *slaves); - Int_t BroadcastObject(const TObject *obj, Int_t kind = kMESS_OBJECT, ESlaves list = kActive); - Int_t BroadcastRaw(const void *buffer, Int_t length, TList *slaves); - Int_t BroadcastRaw(const void *buffer, Int_t length, ESlaves list = kActive); - Int_t Collect(const TSlave *sl, Long_t timeout = -1, Int_t endtype = -1, Bool_t deactonfail = kFALSE); - Int_t Collect(TMonitor *mon, Long_t timeout = -1, Int_t endtype = -1, Bool_t deactonfail = kFALSE); - Int_t CollectInputFrom(TSocket *s, Int_t endtype = -1, Bool_t deactonfail = kFALSE); - Int_t HandleInputMessage(TSlave *wrk, TMessage *m, Bool_t deactonfail = kFALSE); - void HandleSubmerger(TMessage *mess, TSlave *sl); - void SetMonitor(TMonitor *mon = 0, Bool_t on = kTRUE); - - void ReleaseMonitor(TMonitor *mon); - - virtual void FindUniqueSlaves(); - TSlave *FindSlave(TSocket *s) const; - TList *GetListOfSlaves() const { return fSlaves; } - TList *GetListOfInactiveSlaves() const { return fInactiveSlaves; } - TList *GetListOfUniqueSlaves() const { return fUniqueSlaves; } - TList *GetListOfBadSlaves() const { return fBadSlaves; } - Int_t GetNumberOfSlaves() const; - Int_t GetNumberOfActiveSlaves() const; - Int_t GetNumberOfInactiveSlaves() const; - Int_t GetNumberOfUniqueSlaves() const; - Int_t GetNumberOfBadSlaves() const; - - Bool_t IsEndMaster() const { return fEndMaster; } - Int_t ModifyWorkerLists(const char *ord, Bool_t add, Bool_t save); - Int_t RestoreActiveList(); - void SaveActiveList(); - - Bool_t IsSync() const { return fSync; } - void InterruptCurrentMonitor(); - - void SetRunStatus(ERunStatus rst) { fRunStatus = rst; } - - void MarkBad(TSlave *wrk, const char *reason = 0); - void MarkBad(TSocket *s, const char *reason = 0); - void TerminateWorker(TSlave *wrk); - void TerminateWorker(const char *ord); - - void ActivateAsyncInput(); - void DeActivateAsyncInput(); - - Int_t GetQueryReference(Int_t qry, TString &ref); - void PrintProgress(Long64_t total, Long64_t processed, - Float_t procTime = -1., Long64_t bytesread = -1); - - // Managing mergers - Bool_t CreateMerger(TSlave *sl, Int_t port); - void RedirectWorker(TSocket *s, TSlave * sl, Int_t output_size); - Int_t GetActiveMergersCount(); - Int_t FindNextFreeMerger(); - void ResetMergers() { fMergersSet = kFALSE; } - void AskForOutput(TSlave *sl); - - void FinalizationDone() { fFinalizationRunning = kFALSE; } - - void ResetMergePrg(); - void ParseConfigField(const char *config); - - Bool_t Prompt(const char *p); - void ClearDataProgress(Int_t r, Int_t t); - - static TList *GetDataSetSrvMaps(const TString &srvmaps); - -protected: - TProof(); // For derived classes to use - void InitMembers(); - Int_t Init(const char *masterurl, const char *conffile, - const char *confdir, Int_t loglevel, - const char *alias = 0); - virtual Bool_t StartSlaves(Bool_t attach = kFALSE); - Int_t AddWorkers(TList *wrks); - Int_t RemoveWorkers(TList *wrks); - void SetupWorkersEnv(TList *wrks, Bool_t increasingpool = kFALSE); - - void SetPlayer(TVirtualProofPlayer *player); - TVirtualProofPlayer *GetPlayer() const { return fPlayer; } - virtual TVirtualProofPlayer *MakePlayer(const char *player = 0, TSocket *s = 0); - - void UpdateDialog(); - - void HandleLibIncPath(const char *what, Bool_t add, const char *dirs); - - TList *GetListOfActiveSlaves() const { return fActiveSlaves; } - TSlave *CreateSlave(const char *url, const char *ord, - Int_t perf, const char *image, const char *workdir); - TSlave *CreateSubmaster(const char *url, const char *ord, - const char *image, const char *msd, Int_t nwk = 1); - - virtual Int_t PollForNewWorkers(); - virtual void SaveWorkerInfo(); - - Int_t Collect(ESlaves list = kActive, Long_t timeout = -1, Int_t endtype = -1, Bool_t deactonfail = kFALSE); - Int_t Collect(TList *slaves, Long_t timeout = -1, Int_t endtype = -1, Bool_t deactonfail = kFALSE); - - TList *GetEnabledPackages() const { return fEnabledPackagesOnCluster; } - - void SetDSet(TDSet *dset) { fDSet = dset; } - virtual void ValidateDSet(TDSet *dset); - - Int_t VerifyDataSetParallel(const char *uri, const char *optStr); - - TPluginHandler *GetProgressDialog() const { return fProgressDialog; } - - Int_t AssertPath(const char *path, Bool_t writable); - Int_t GetSandbox(TString &sb, Bool_t assert = kFALSE, const char *rc = 0); - - void PrepareInputDataFile(TString &dataFile); - virtual void SendInputDataFile(); - Int_t SendFile(const char *file, Int_t opt = (kBinary | kForward | kCp | kCpBin), - const char *rfile = 0, TSlave *sl = 0); - - // Fast enable/disable feedback from Process - void SetFeedback(TString &opt, TString &optfb, Int_t action); - // Output file handling during Process - Int_t HandleOutputOptions(TString &opt, TString &target, Int_t action); - - static void *SlaveStartupThread(void *arg); - - static Int_t AssertDataSet(TDSet *dset, TList *input, - TDataSetManager *mgr, TString &emsg); - static void AssertMacroPath(const char *macro); - - // Input data handling - static Int_t GetInputData(TList *input, const char *cachedir, TString &emsg); - static Int_t SaveInputData(TQueryResult *qr, const char *cachedir, TString &emsg); - static Int_t SendInputData(TQueryResult *qr, TProof *p, TString &emsg); - - // Parse CINT commands - static Bool_t GetFileInCmd(const char *cmd, TString &fn); - - // Pipe execution of commands - static void SystemCmd(const char *cmd, Int_t fdout); - -public: - TProof(const char *masterurl, const char *conffile = kPROOF_ConfFile, - const char *confdir = kPROOF_ConfDir, Int_t loglevel = 0, - const char *alias = 0, TProofMgr *mgr = 0); - ~TProof() override; - - void cd(Int_t id = -1); - - Int_t Ping(); - void Touch(); - Int_t Exec(const char *cmd, Bool_t plusMaster = kFALSE); - Int_t Exec(const char *cmd, const char *ord, Bool_t logtomacro = kFALSE); - - TString Getenv(const char *env, const char *ord = "0"); - Int_t GetRC(const char *RCenv, Int_t &env, const char *ord = "0"); - Int_t GetRC(const char *RCenv, Double_t &env, const char *ord = "0"); - Int_t GetRC(const char *RCenv, TString &env, const char *ord = "0"); - - virtual Long64_t Process(TDSet *dset, const char *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0); - virtual Long64_t Process(TFileCollection *fc, const char *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0); - virtual Long64_t Process(const char *dsetname, const char *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0, TObject *enl = 0); - virtual Long64_t Process(const char *selector, Long64_t nentries, - Option_t *option = ""); - // Process via TSelector - virtual Long64_t Process(TDSet *dset, TSelector *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0); - virtual Long64_t Process(TFileCollection *fc, TSelector *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0); - virtual Long64_t Process(const char *dsetname, TSelector *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0, TObject *enl = 0); - virtual Long64_t Process(TSelector *selector, Long64_t nentries, - Option_t *option = ""); - - virtual Long64_t DrawSelect(TDSet *dset, const char *varexp, - const char *selection = "", - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0); - Long64_t DrawSelect(const char *dsetname, const char *varexp, - const char *selection = "", - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0, TObject *enl = 0); - Int_t Archive(Int_t query, const char *url); - Int_t Archive(const char *queryref, const char *url = 0); - Int_t CleanupSession(const char *sessiontag); - Long64_t Finalize(Int_t query = -1, Bool_t force = kFALSE); - Long64_t Finalize(const char *queryref, Bool_t force = kFALSE); - Int_t Remove(Int_t query, Bool_t all = kFALSE); - Int_t Remove(const char *queryref, Bool_t all = kFALSE); - Int_t Retrieve(Int_t query, const char *path = 0); - Int_t Retrieve(const char *queryref, const char *path = 0); - - void DisableGoAsyn(); - void GoAsynchronous(); - void StopProcess(Bool_t abort, Int_t timeout = -1); - void Browse(TBrowser *b) override; - - virtual Int_t Echo(const TObject *obj); - virtual Int_t Echo(const char *str); - - Int_t SetParallel(Int_t nodes = -1, Bool_t random = kFALSE); - void SetLogLevel(Int_t level, UInt_t mask = TProofDebug::kAll); - - void Close(Option_t *option=""); - void Print(Option_t *option="") const override; - - //-- cache and package management - virtual void ShowCache(Bool_t all = kFALSE); - virtual void ClearCache(const char *file = 0); - TList *GetListOfPackages(); - TList *GetListOfEnabledPackages(); - void ShowPackages(Bool_t all = kFALSE, Bool_t redirlog = kFALSE); - void ShowEnabledPackages(Bool_t all = kFALSE); - Int_t ClearPackages(); - Int_t ClearPackage(const char *package); - Int_t DownloadPackage(const char *par, const char *dstdir = 0); - Int_t EnablePackage(const char *package, Bool_t notOnClient = kFALSE, TList *workers = 0); - Int_t EnablePackage(const char *package, const char *loadopts, - Bool_t notOnClient = kFALSE, TList *workers = 0); - Int_t EnablePackage(const char *package, TList *loadopts, - Bool_t notOnClient = kFALSE, TList *workers = 0); - Int_t UploadPackage(const char *par, EUploadPackageOpt opt = kUntar, TList *workers = 0); - virtual Int_t Load(const char *macro, Bool_t notOnClient = kFALSE, Bool_t uniqueOnly = kTRUE, - TList *wrks = 0); - - Int_t AddDynamicPath(const char *libpath, Bool_t onClient = kFALSE, TList *wrks = 0, Bool_t doCollect = kTRUE); - Int_t AddIncludePath(const char *incpath, Bool_t onClient = kFALSE, TList *wrks = 0, Bool_t doCollect = kTRUE); - Int_t RemoveDynamicPath(const char *libpath, Bool_t onClient = kFALSE); - Int_t RemoveIncludePath(const char *incpath, Bool_t onClient = kFALSE); - - //-- dataset management - Int_t UploadDataSet(const char *, TList *, const char * = 0, Int_t = 0, TList * = 0); - Int_t UploadDataSet(const char *, const char *, const char * = 0, Int_t = 0, TList * = 0); - Int_t UploadDataSetFromFile(const char *, const char *, const char * = 0, Int_t = 0, TList * = 0); - virtual Bool_t RegisterDataSet(const char *name, - TFileCollection *dataset, const char* optStr = ""); - virtual TMap *GetDataSets(const char *uri = "", const char* optStr = ""); - virtual void ShowDataSets(const char *uri = "", const char* optStr = ""); - - TMap *GetDataSetQuota(const char* optStr = ""); - void ShowDataSetQuota(Option_t* opt = 0); - - virtual Bool_t ExistsDataSet(const char *dataset); - void ShowDataSet(const char *dataset = "", const char* opt = "filter:SsCc"); - virtual Int_t RemoveDataSet(const char *dataset, const char* optStr = ""); - virtual Int_t VerifyDataSet(const char *dataset, const char* optStr = ""); - virtual TFileCollection *GetDataSet(const char *dataset, const char* optStr = ""); - TList *FindDataSets(const char *searchString, const char* optStr = ""); - virtual Bool_t RequestStagingDataSet(const char *dataset); - virtual TFileCollection *GetStagingStatusDataSet(const char *dataset); - virtual void ShowStagingStatusDataSet(const char *dataset, const char *optStr = "filter:SsCc"); - virtual Bool_t CancelStagingDataSet(const char *dataset); - - virtual Int_t SetDataSetTreeName( const char *dataset, const char *treename); - - virtual void ShowDataSetCache(const char *dataset = 0); - virtual void ClearDataSetCache(const char *dataset = 0); - - virtual void ShowData(); - void ClearData(UInt_t what = kUnregistered, const char *dsname = 0); - - const char *GetMaster() const { return fMaster; } - const char *GetConfDir() const { return fConfDir; } - const char *GetConfFile() const { return fConfFile; } - const char *GetUser() const { return fUrl.GetUser(); } - const char *GetGroup() const { return fGroup; } - const char *GetWorkDir() const { return fWorkDir; } - const char *GetSessionTag() const { return GetName(); } - const char *GetImage() const { return fImage; } - const char *GetUrl() { return fUrl.GetUrl(); } - Int_t GetPort() const { return fUrl.GetPort(); } - Int_t GetRemoteProtocol() const { return fProtocol; } - Int_t GetClientProtocol() const { return kPROOF_Protocol; } - Int_t GetStatus() const { return fStatus; } - Int_t GetLogLevel() const { return fLogLevel; } - Int_t GetParallel() const; - Int_t GetSeqNum() const { return fSeqNum; } - Int_t GetSessionID() const { return fSessionID; } - TList *GetListOfSlaveInfos(); - Bool_t UseDynamicStartup() const { return fDynamicStartup; } - - EQueryMode GetQueryMode(Option_t *mode = 0) const; - void SetQueryMode(EQueryMode mode); - - void SetRealTimeLog(Bool_t on = kTRUE); - - void GetStatistics(Bool_t verbose = kFALSE); - Long64_t GetBytesRead() const { return fBytesRead; } - Float_t GetRealTime() const { return fRealTime; } - Float_t GetCpuTime() const { return fCpuTime; } - - Bool_t IsLite() const { return (fServType == TProofMgr::kProofLite) ? kTRUE : kFALSE; } - Bool_t IsProofd() const { return (fServType == TProofMgr::kProofd) ? kTRUE : kFALSE; } - Bool_t IsFolder() const override { return kTRUE; } - Bool_t IsMaster() const { return fMasterServ; } - Bool_t IsValid() const { return fValid; } - Bool_t IsTty() const { return fTty; } - Bool_t IsParallel() const { return GetParallel() > 0 ? kTRUE : kFALSE; } - Bool_t IsIdle() const { return (fNotIdle <= 0) ? kTRUE : kFALSE; } - Bool_t IsWaiting() const { return fIsWaiting; } - - ERunStatus GetRunStatus() const { return fRunStatus; } - TList *GetLoadedMacros() const { return fLoadedMacros; } - - //-- input list parameter handling - void SetParameter(const char *par, const char *value); - void SetParameter(const char *par, Int_t value); - void SetParameter(const char *par, Long_t value); - void SetParameter(const char *par, Long64_t value); - void SetParameter(const char *par, Double_t value); - TObject *GetParameter(const char *par) const; - void DeleteParameters(const char *wildcard); - void ShowParameters(const char *wildcard = "PROOF_*") const; - - void AddInput(TObject *obj); - void ClearInput(); - TList *GetInputList(); - TObject *GetOutput(const char *name); - TList *GetOutputList(); - static TObject *GetOutput(const char *name, TList *out); - - void ShowMissingFiles(TQueryResult *qr = 0); - TFileCollection *GetMissingFiles(TQueryResult *qr = 0); - - void AddInputData(TObject *obj, Bool_t push = kFALSE); - void SetInputDataFile(const char *datafile); - void ClearInputData(TObject *obj = 0); - void ClearInputData(const char *name); - - void AddFeedback(const char *name); - void RemoveFeedback(const char *name); - void ClearFeedback(); - void ShowFeedback() const; - TList *GetFeedbackList() const; - - virtual TList *GetListOfQueries(Option_t *opt = ""); - Int_t GetNumberOfQueries(); - Int_t GetNumberOfDrawQueries() { return fDrawQueries; } - TList *GetQueryResults(); - TQueryResult *GetQueryResult(const char *ref = 0); - void GetMaxQueries(); - void SetMaxDrawQueries(Int_t max); - void ShowQueries(Option_t *opt = ""); - - Bool_t IsDataReady(Long64_t &totalbytes, Long64_t &bytesready); - - void SetActive(Bool_t /*active*/ = kTRUE) { } - - void LogMessage(const char *msg, Bool_t all); //*SIGNAL* - void Progress(Long64_t total, Long64_t processed); //*SIGNAL* - void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti); // *SIGNAL* - void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti, - Int_t actw, Int_t tses, Float_t eses); // *SIGNAL* - void Feedback(TList *objs); //*SIGNAL* - void QueryResultReady(const char *ref); //*SIGNAL* - void CloseProgressDialog(); //*SIGNAL* - void ResetProgressDialog(const char *sel, Int_t sz, - Long64_t fst, Long64_t ent); //*SIGNAL* - void StartupMessage(const char *msg, Bool_t status, Int_t done, - Int_t total); //*SIGNAL* - void DataSetStatus(const char *msg, Bool_t status, - Int_t done, Int_t total); //*SIGNAL* - - void SendDataSetStatus(const char *msg, UInt_t n, UInt_t tot, Bool_t st); - - void GetLog(Int_t start = -1, Int_t end = -1); - TMacro *GetLastLog(); - void PutLog(TQueryResult *qr); - void ShowLog(Int_t qry = -1); - void ShowLog(const char *queryref); - Bool_t SendingLogToWindow() const { return fLogToWindowOnly; } - void SendLogToWindow(Bool_t mode) { fLogToWindowOnly = mode; } - - TMacro *GetMacroLog() { return &fMacroLog; } - - void ResetProgressDialogStatus() { fProgressDialogStarted = kFALSE; } - - virtual TTree *GetTreeHeader(TDSet *tdset); - TList *GetOutputNames(); - - void AddChain(TChain *chain); - void RemoveChain(TChain *chain); - - TDrawFeedback *CreateDrawFeedback(); - void SetDrawFeedbackOption(TDrawFeedback *f, Option_t *opt); - void DeleteDrawFeedback(TDrawFeedback *f); - - void Detach(Option_t *opt = ""); - - virtual void SetAlias(const char *alias=""); - - TProofMgr *GetManager() { return fManager; } - void SetManager(TProofMgr *mgr); - - Int_t ActivateWorker(const char *ord, Bool_t save = kTRUE); - Int_t DeactivateWorker(const char *ord, Bool_t save = kTRUE); - - const char *GetDataPoolUrl() const { return fManager ? fManager->GetMssUrl() : 0; } - void SetDataPoolUrl(const char *url) { if (fManager) fManager->SetMssUrl(url); } - - void SetPrintProgress(PrintProgress_t pp) { fPrintProgress = pp; } - - void SetProgressDialog(Bool_t on = kTRUE); - - // Enable the performance tree - Int_t SavePerfTree(const char *pf = 0, const char *qref = 0); - void SetPerfTree(const char *pf = "perftree.root", Bool_t withWrks = kFALSE); - - // Opening and managing PROOF connections - static TProof *Open(const char *url = 0, const char *conffile = 0, - const char *confdir = 0, Int_t loglevel = 0); - static void LogViewer(const char *url = 0, Int_t sessionidx = 0); - static TProofMgr *Mgr(const char *url); - static void Reset(const char *url, Bool_t hard = kFALSE); - - static void AddEnvVar(const char *name, const char *value); - static void DelEnvVar(const char *name); - static const TList *GetEnvVars(); - static void ResetEnvVars(); - - // Input/output list utilities - static Int_t GetParameter(TCollection *c, const char *par, TString &value); - static Int_t GetParameter(TCollection *c, const char *par, Int_t &value); - static Int_t GetParameter(TCollection *c, const char *par, Long_t &value); - static Int_t GetParameter(TCollection *c, const char *par, Long64_t &value); - static Int_t GetParameter(TCollection *c, const char *par, Double_t &value); - - ClassDefOverride(TProof,0) //PROOF control class -} R__DEPRECATED(6,38, "Legacy interactive distributed processing engine. Use RDataFrame instead."); - -// Global object with default PROOF session -R__EXTERN TProof *gProof; - -#endif diff --git a/proof/proof/inc/TProofChain.h b/proof/proof/inc/TProofChain.h deleted file mode 100644 index 8e490cd612355..0000000000000 --- a/proof/proof/inc/TProofChain.h +++ /dev/null @@ -1,92 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis Nov 2006 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofChain -#define ROOT_TProofChain - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofChain // -// // -// A TChain proxy on PROOF. // -// Uses an internal TDSet to handle processing. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TChain.h" - -class TDSet; -class TList; -class TProof; - -class TProofChain : public TChain { - -public: - // TProofChain constants - enum { kOwnsChain = BIT(19) }; - -private: - void AddAliases(); - void FillDrawAttributes(TProof *p); - -protected: - TChain *fChain; // mother chain: needed for the browsing list - TDSet *fSet; // TDSet - -public: - TProofChain(); - TProofChain(TChain *chain, Bool_t gettreeheader); - TProofChain(TDSet *dset, Bool_t gettreeheader); - ~TProofChain() override; - - void Browse(TBrowser *b) override; - Int_t Debug() const {return fDebug;} - Long64_t Draw(const char *varexp, const TCut &selection, Option_t *option="" - ,Long64_t nentries=TTree::kMaxEntries, Long64_t firstentry=0) override; - Long64_t Draw(const char *varexp, const char *selection, Option_t *option="" - ,Long64_t nentries=TTree::kMaxEntries, Long64_t firstentry=0) override; // *MENU* - void Draw(Option_t *opt) override { Draw(opt, "", "", TTree::kMaxEntries, 0); } - TBranch *FindBranch(const char *name) override; - TLeaf *FindLeaf(const char *name) override; - TBranch *GetBranch(const char *name) override; - Bool_t GetBranchStatus(const char *branchname) const override; - Long64_t GetEntries() const override; - Long64_t GetEntries(const char *sel) override; - TList *GetListOfClones() override { return 0; } - TObjArray *GetListOfBranches() override {return (fTree ? fTree->GetListOfBranches() : (TObjArray *)0); } - TObjArray *GetListOfLeaves() override {return (fTree ? fTree->GetListOfLeaves() : (TObjArray *)0);} - TList *GetListOfFriends() const override {return 0;} - TList *GetListOfAliases() const override {return 0;} - - // GetMakeClass is left non-virtual for efficiency reason. - // Making it virtual affects the performance of the I/O - Int_t GetMakeClass() const {return fMakeClass;} - - TVirtualTreePlayer *GetPlayer(); - Long64_t GetReadEntry() const override; - Bool_t HasTreeHeader() const { return (fTree ? kTRUE : kFALSE); } - Long64_t Process(const char *filename, Option_t *option="", - Long64_t nentries=TTree::kMaxEntries, Long64_t firstentry=0) override; // *MENU* - virtual void Progress(Long64_t total, Long64_t processed); - Long64_t Process(TSelector *selector, Option_t *option="", - Long64_t nentries=TTree::kMaxEntries, Long64_t firstentry=0) override; - void SetDebug(Int_t level=1, Long64_t min=0, Long64_t max=9999999) override; // *MENU* - void SetEventList(TEventList *evlist) override { fEventList = evlist; } - void SetEntryList(TEntryList *enlist, const Option_t *) override { fEntryList = enlist; } - void SetName(const char *name) override; // *MENU* - virtual void ConnectProof(); - virtual void ReleaseProof(); - - ClassDefOverride(TProofChain,0) //TChain proxy for running chains on PROOF -}; - -#endif diff --git a/proof/proof/inc/TProofCondor.h b/proof/proof/inc/TProofCondor.h deleted file mode 100644 index e4b7b1524ed0a..0000000000000 --- a/proof/proof/inc/TProofCondor.h +++ /dev/null @@ -1,56 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 13/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofCondor -#define ROOT_TProofCondor - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofCondor // -// // -// This class controls a Parallel ROOT Facility, PROOF, cluster. // -// It fires the slave servers, it keeps track of how many slaves are // -// running, it keeps track of the slaves running status, it broadcasts // -// messages to all slaves, it collects results, etc. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProof.h" -#include "TString.h" - -class TCondor; -class TTimer; - -class TProofCondor : public TProof { - -friend class TCondor; - -private: - TCondor *fCondor; //proxy for our Condor pool - TTimer *fTimer; //timer for delayed Condor COD suspend - -protected: - Bool_t StartSlaves(Bool_t) override; - TString GetJobAd(); - -public: - TProofCondor(const char *masterurl, const char *conffile = kPROOF_ConfFile, - const char *confdir = kPROOF_ConfDir, Int_t loglevel = 0, - const char *alias = 0, TProofMgr *mgr = 0); - ~TProofCondor() override; - virtual void SetActive() { TProof::SetActive(); } - virtual void SetActive(Bool_t active); - - ClassDefOverride(TProofCondor,0) //PROOF control class for slaves allocated by condor -}; - -#endif diff --git a/proof/proof/inc/TProofDebug.h b/proof/proof/inc/TProofDebug.h deleted file mode 100644 index d7d022ac54ad9..0000000000000 --- a/proof/proof/inc/TProofDebug.h +++ /dev/null @@ -1,59 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Maarten Ballintijn 19/6/2002 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofDebug -#define ROOT_TProofDebug - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofDebug // -// // -// Detailed logging / debug scheme. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "Rtypes.h" - -// Remember to update the related mapping in tutorials/proof/runProof.C ! - -class TProofDebug { -public: - enum EProofDebugMask { - kNone = 0, - kPacketizer = 1, - kLoop = 2, - kSelector = 4, - kOutput = 8, - kInput = 16, - kGlobal = 32, - kPackage = 64, - kFeedback = 128, - kCondor = 256, - kDraw = 512, - kAsyn = 1024, - kCache = 2048, - kCollect = 4096, - kDataset = 8192, - kSubmerger = 16384, - kMonitoring = 32768, - - kAll = 0xFFFFFFFF - }; -}; - -R__EXTERN TProofDebug::EProofDebugMask gProofDebugMask; -R__EXTERN Int_t gProofDebugLevel; - -#define PDB(mask,level) \ -if ((TProofDebug::mask & gProofDebugMask) && gProofDebugLevel >= (level)) - -#endif diff --git a/proof/proof/inc/TProofLite.h b/proof/proof/inc/TProofLite.h deleted file mode 100644 index 4d12698049f51..0000000000000 --- a/proof/proof/inc/TProofLite.h +++ /dev/null @@ -1,173 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis March 2008 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofLite -#define ROOT_TProofLite - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofLite // -// // -// This class starts a PROOF session on the local machine: no daemons, // -// client and master merged, communications via UNIX-like sockets. // -// By default the number of workers started is NumberOfCores+1; a // -// different number can be forced on construction. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProof.h" - -class TDSet; -class TList; -class TQueryResultManager; -class TDataSetManager; -class TProofLockPath; -class TProofMgr; -class TProofQueryResult; -class TServerSocket; -class TSelector; -class TPMERegexp; - -class TProofLite : public TProof { - -friend class TProofPlayerLite; - -private: - Int_t fNWorkers; // Number of workers - TString fSandbox; // PROOF sandbox root dir - TString fCacheDir; // Directory containing cache of user files - TString fQueryDir; // Directory containing query results and status - TString fDataSetDir; // Directory containing info about known data sets - TString fSockPath; // UNIX socket path for communication with workers - TServerSocket *fServSock; // Server socket to accept call backs - Bool_t fForkStartup; // Startup N-1 workers forking the first worker - - Int_t fDynamicStartupStep; // Dyn Startup simulation: increment at each call - Int_t fDynamicStartupNMax; // Dyn Startup simulation: max number of workers - - TString fVarExp; // Internal variable to pass drawing options - TString fSelection; // Internal variable to pass drawing options - - TProofLockPath *fCacheLock; // Cache dir locker - TProofLockPath *fQueryLock; // Query dir locker - TQueryResultManager *fQMgr; // Query-result manager - - TDataSetManager *fDataSetManager; // Dataset manager - TDataSetManagerFile *fDataSetStgRepo; // Dataset manager for staging requests - - TPMERegexp *fReInvalid; // Regular expression matching invalid dataset URIs - - static Int_t fgWrksMax; // Max number of workers - - TProofLite(const TProofLite &); // not implemented - void operator=(const TProofLite &); // idem - - Int_t CleanupSandbox(); - Int_t CreateSandbox(); - void FindUniqueSlaves() override; - void NotifyStartUp(const char *action, Int_t done, Int_t tot); - Int_t SetProofServEnv(const char *ord); - Int_t InitDataSetManager(); - - void ResolveKeywords(TString &s, const char *ord, const char *logfile); - - void SendInputDataFile() override; - void ShowDataDir(const char *dirname); - -protected: - TProofLite() : TProof() { } // For derived classes to use - - Int_t Init(const char *masterurl, const char *conffile, - const char *confdir, Int_t loglevel, - const char *alias = 0); - TProofQueryResult *MakeQueryResult(Long64_t nent, const char *opt, - Long64_t fst, TDSet *dset, - const char *selec); - void SetQueryRunning(TProofQueryResult *pq); - Int_t SetupWorkers(Int_t opt = 0, TList *wrks = 0); - Int_t CopyMacroToCache(const char *macro, Int_t headerRequired = 0, - TSelector **selector = 0, Int_t opt = 0, TList *wrks = 0); - - Int_t PollForNewWorkers() override; - -public: - TProofLite(const char *masterurl, const char *conffile = kPROOF_ConfFile, - const char *confdir = kPROOF_ConfDir, Int_t loglevel = 0, - const char *alias = 0, TProofMgr *mgr = 0); - ~TProofLite() override; - - void Print(Option_t *option="") const override; - - Long64_t DrawSelect(TDSet *dset, const char *varexp, - const char *selection = "", - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0) override; - Long64_t Process(TDSet *dset, const char *sel, Option_t *o = "", - Long64_t nent = -1, Long64_t fst = 0) override; - Long64_t Process(TFileCollection *fc, const char *sel, Option_t *o = "", - Long64_t nent = -1, Long64_t fst = 0) override - { return TProof::Process(fc, sel, o, nent, fst); } - Long64_t Process(const char *dsname, const char *sel, Option_t *o = "", - Long64_t nent = -1, Long64_t fst = 0, TObject *enl = 0) override - { return TProof::Process(dsname, sel, o, nent, fst, enl); } - Long64_t Process(const char *sel, Long64_t nent, Option_t *o = "") override - { return TProof::Process(sel, nent, o); } - // Process via TSelector - Long64_t Process(TDSet *dset, TSelector *sel, Option_t *o = "", - Long64_t nent = -1, Long64_t fst = 0) override - { return TProof::Process(dset, sel, o, nent, fst); } - Long64_t Process(TFileCollection *fc, TSelector *sel, Option_t *o = "", - Long64_t nent = -1, Long64_t fst = 0) override - { return TProof::Process(fc, sel, o, nent, fst); } - Long64_t Process(const char *dsname, TSelector *sel, Option_t *o = "", - Long64_t nent = -1, Long64_t fst = 0, TObject *enl = 0) override - { return TProof::Process(dsname, sel, o, nent, fst, enl); } - Long64_t Process(TSelector* sel, Long64_t nent, Option_t *o = "") override - { return TProof::Process(sel, nent, o); } - - // Cache management - void ShowCache(Bool_t all = kFALSE) override; - void ClearCache(const char *file = 0) override; - Int_t Load(const char *macro, Bool_t notOnClient = kFALSE, Bool_t uniqueOnly = kTRUE, - TList *wrks = 0) override; - - // Data management - void ShowData() override; - - // Query management - TList *GetListOfQueries(Option_t *opt = "") override; - Int_t Remove(const char *ref, Bool_t all); - - // Dataset handling - Bool_t RegisterDataSet(const char *dsName, TFileCollection *ds, const char *opt = "") override; - Bool_t ExistsDataSet(const char *uri) override; - TMap *GetDataSets(const char *uri = "", const char * = 0) override; - void ShowDataSets(const char *uri = "", const char * = 0) override; - TFileCollection *GetDataSet(const char *uri, const char * = 0) override; - Int_t RemoveDataSet(const char *uri, const char * = 0) override; - Bool_t RequestStagingDataSet(const char *dataset) override; - Bool_t CancelStagingDataSet(const char *dataset) override; - TFileCollection *GetStagingStatusDataSet(const char *dataset) override; - Int_t VerifyDataSet(const char *uri, const char * = 0) override; - Int_t SetDataSetTreeName( const char *dataset, const char *treename) override; - void ShowDataSetCache(const char *dataset = 0) override; - void ClearDataSetCache(const char *dataset = 0) override; - - // Browsing - TTree *GetTreeHeader(TDSet *tdset) override; - - static Int_t GetNumberOfWorkers(const char *url = 0); - - ClassDefOverride(TProofLite,0) //PROOF-Lite control class -} R__DEPRECATED(6,38, "Legacy interactive distributed processing engine. Use RDataFrame instead."); - -#endif diff --git a/proof/proof/inc/TProofLog.h b/proof/proof/inc/TProofLog.h deleted file mode 100644 index 57a7b67d94f35..0000000000000 --- a/proof/proof/inc/TProofLog.h +++ /dev/null @@ -1,117 +0,0 @@ -// @(#)root/proof:$Id: 5d579564fccbadad9cd6f81ccb7726dddea80e0d $ -// Author: G. Ganis 31/08/06 - -/************************************************************************* - * Copyright (C) 1995-2006, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofLog -#define ROOT_TProofLog - -////////////////////////////////////////////////////////////////////////// -// // -// TProofLog // -// // -// Implementation of the PROOF session log handler // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TDatime.h" -#include "TNamed.h" -#include "TQObject.h" - -class TMacro; -class TProofLogElem; -class TProofMgr; - - -class TProofLog : public TNamed, public TQObject { - -friend class TProofLogElem; -friend class TProofMgrLite; -friend class TXProofMgr; - -private: - TProofMgr *fMgr; // parent TProofMgr - void *fFILE; // pointer logging file, if any - TList *fElem; // list of TProofLogElem objects - TDatime fStartTime; // Time at which this session started - - TProofLogElem *Add(const char *ord, const char *url); - -public: - // Screen or GUI box logging - enum ELogLocationBit { - kLogToBox = BIT(16) - }; - enum ERetrieveOpt { kLeading = 0x1, kTrailing = 0x2, - kAll = 0x3, kGrep = 0x4 }; - - TProofLog(const char *stag, const char *url, TProofMgr *mgr); - ~TProofLog() override; - - void Display(const char *ord = "*", Int_t from = -10, Int_t to = -1); - TList *GetListOfLogs() const { return fElem; } - Int_t Grep(const char *txt, Int_t from = 0); - void Print(Option_t *opt = 0) const override; - void Prt(const char *what, Bool_t newline = kTRUE); - Int_t Retrieve(const char *ord = "*", - TProofLog::ERetrieveOpt opt = TProofLog::kTrailing, - const char *fname = 0, const char *pattern = 0); - Int_t Save(const char *ord = "*", const char *fname = 0, Option_t *opt="w"); - - TDatime StartTime() { return fStartTime; } - - // Where to log - void SetLogToBox(Bool_t lgbox = kFALSE) { SetBit(kLogToBox, lgbox); } - Bool_t LogToBox() { return (TestBit(kLogToBox)) ? kTRUE : kFALSE; } - - static void SetMaxTransferSize(Long64_t maxsz); - - ClassDefOverride(TProofLog,0) // PROOF session log handler -}; - - -class TProofLogElem : public TNamed { - -private: - TProofLog *fLogger; // parent TProofLog - TMacro *fMacro; // container for the log lines - Long64_t fSize; // best knowledge of the log file size - Long64_t fFrom; // starting offset of the current content - Long64_t fTo; // end offset of the current content - TString fRole; // role (master-submaster-worker) - - static Long64_t fgMaxTransferSize; - - //the name of TProofLogElem is the ordinal number of the corresp. worker - //the title is the url - -public: - TProofLogElem(const char *ord, const char *url, - TProofLog *logger); - ~TProofLogElem() override; - - void Display(Int_t from = 0, Int_t to = -1); - TMacro *GetMacro() const { return fMacro; } - const char * GetRole() { return fRole.Data(); } - Int_t Grep(const char *txt, TString &res, Int_t from = 0); - Bool_t IsMaster() const { return (fRole == "master") ? kTRUE : kFALSE; } - Bool_t IsSubMaster() const { return (fRole == "submaster") ? kTRUE : kFALSE; } - Bool_t IsWorker() const { return (fRole == "worker") ? kTRUE : kFALSE; } - void Print(Option_t *opt = 0) const override; - void Prt(const char *what); - Int_t Retrieve(TProofLog::ERetrieveOpt opt = TProofLog::kTrailing, - const char *pattern = 0); - - static Long64_t GetMaxTransferSize(); - static void SetMaxTransferSize(Long64_t maxsz); - - ClassDefOverride(TProofLogElem,0) // PROOF session log element -}; - -#endif diff --git a/proof/proof/inc/TProofMgr.h b/proof/proof/inc/TProofMgr.h deleted file mode 100644 index 2d3a71b8b62b5..0000000000000 --- a/proof/proof/inc/TProofMgr.h +++ /dev/null @@ -1,185 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis, Nov 2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofMgr -#define ROOT_TProofMgr - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofMgr // -// // -// The PROOF manager interacts with the PROOF server coordinator to // -// create or destroy a PROOF session, attach to or detach from // -// existing one, and to monitor any client activity on the cluster. // -// At most one manager instance per server is allowed. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TList.h" -#include "TNamed.h" -#include "TUrl.h" -#include "TString.h" -#include "TSystem.h" - -class TFileCollection; -class TObjString; -class TProof; -class TProofDesc; -class TProofLog; -class TProofMgr; -class TSignalHandler; - -typedef TProofMgr *(*TProofMgr_t)(const char *, Int_t, const char *); - -class TProofMgr : public TNamed { - -public: - enum EServType { kProofd = 0, kXProofd = 1, kProofLite = 2 }; - -private: - TProofMgr(const TProofMgr&); // Not implemented - TProofMgr& operator=(const TProofMgr&); // Not implemented - - static TProofMgr_t fgTXProofMgrHook; // Constructor hooks for TXProofMgr - static TProofMgr_t GetXProofMgrHook(); - - static void ReplaceSubdirs(const char *fn, TString &fdst, TList &dirph); - -protected: - TString fMssUrl; // URL for the remote data pool, if any - Int_t fRemoteProtocol; // Protocol number run by the daemon server - EServType fServType; // Type of server: old-proofd, XrdProofd - TList *fSessions; // PROOF session managed by this server - TUrl fUrl; // Server URL - - TSignalHandler *fIntHandler; // Interrupt signal handler (ctrl-c) - - static TList fgListOfManagers; // Sub-list of TROOT::ListOfProofs for managers - - TProofMgr() : fRemoteProtocol(-1), - fServType(kXProofd), fSessions(0), fUrl(), fIntHandler(0) { } - -public: - TProofMgr(const char *url, Int_t loglevel = -1, const char *alias = ""); - ~TProofMgr() override; - - virtual Bool_t IsLite() const { return (fServType == kProofLite); } - virtual Bool_t IsProofd() const { return (fServType == kProofd); } - virtual Bool_t IsValid() const { return kTRUE; } - virtual void SetInvalid() { } - void Close() { SetInvalid(); } - - virtual TProof *AttachSession(Int_t, Bool_t = kFALSE); - virtual TProof *AttachSession(TProofDesc *, Bool_t = kFALSE); - virtual TProof *CreateSession(const char * = 0, const char * = 0, Int_t = -1); - virtual void DetachSession(Int_t, Option_t * = ""); - virtual void DetachSession(TProof *, Option_t * = ""); - virtual void DiscardSession(TProof *p); - virtual const char *GetMssUrl(Bool_t = kFALSE) { return fMssUrl.Data(); } - virtual TProofDesc *GetProofDesc(Int_t id); - virtual TProofDesc *GetProofDesc(TProof *p); - virtual Int_t GetRemoteProtocol() const { return fRemoteProtocol; } - virtual TProofLog *GetSessionLogs(Int_t = 0, const char * = 0, - const char * = "-v \"| SvcMsg\"", Bool_t = kFALSE) - { return (TProofLog *)0; } - virtual const char *GetUrl() { return fUrl.GetUrl(); } - virtual Bool_t MatchUrl(const char *url); - virtual TList *QuerySessions(Option_t *opt = "S"); - virtual TObjString *ReadBuffer(const char *, Long64_t, Int_t) - { return (TObjString *)0; } - virtual TObjString *ReadBuffer(const char *, const char *) - { return (TObjString *)0; } - virtual Int_t Reset(Bool_t hard = kFALSE, const char *usr = 0); - virtual void ShowWorkers(); - virtual Int_t SendMsgToUsers(const char *, const char * = 0); - virtual void SetAlias(const char *alias="") { TNamed::SetTitle(alias); } - virtual void SetMssUrl(const char *mss) { fMssUrl = mss; } - virtual Int_t SetROOTVersion(const char *) { return 0; } - virtual void ShowROOTVersions() { } - virtual void ShutdownSession(Int_t id) { DetachSession(id,"S"); } - virtual void ShutdownSession(TProof *p) { DetachSession(p,"S"); } - - // Remote file system actions - virtual Int_t Cp(const char *, const char * = 0, const char * = 0) { return -1; } - virtual void Find(const char * = "~/", const char * = 0, const char * = 0) { } - virtual void Grep(const char *, const char * = 0, const char * = 0) { } - virtual void Ls(const char * = "~/", const char * = 0, const char * = 0) { } - virtual void More(const char *, const char * = 0, const char * = 0) { } - virtual Int_t Rm(const char *, const char * = 0, const char * = 0); - virtual void Tail(const char *, const char * = 0, const char * = 0) { } - virtual Int_t Md5sum(const char *, TString &, const char * = 0) { return -1; } - virtual Int_t Stat(const char *, FileStat_t &, const char * = 0) { return -1; } - - virtual Int_t GetFile(const char *, const char *, const char * = 0) { return -1; } - virtual Int_t PutFile(const char *, const char *, const char * = 0) { return -1; } - - static TList *GetListOfManagers(); - - static void SetTXProofMgrHook(TProofMgr_t pmh); - - static TProofMgr *Create(const char *url, Int_t loglevel = -1, - const char *alias = 0, Bool_t xpd = kTRUE); - static Int_t Ping(const char *url, Bool_t checkxrd = kFALSE); - - static TFileCollection *UploadFiles(TList *src, const char *mss, const char *dest = 0); - static TFileCollection *UploadFiles(const char *txtfile, const char *mss, const char *dest = 0); - - ClassDefOverride(TProofMgr,0) // Abstract PROOF manager interface -}; - -// -// Metaclass describing the essentials of a PROOF session -// -class TProofDesc : public TNamed { -public: - enum EStatus { kUnknown = -1, kIdle = 0, kRunning =1, kShutdown = 2}; - -private: - TProofDesc(const TProofDesc&); // Not implemented - TProofDesc& operator=(const TProofDesc&); // Not implemented - - Int_t fLocalId; // ID in the local list - Int_t fStatus; // Session status (see EStatus) - TProof *fProof; // Related instance of TProof - Int_t fRemoteId; // Remote ID assigned by the coordinator to the proofserv - TString fUrl; // Url of the connection - -public: - TProofDesc(const char *tag = 0, const char *alias = 0, const char *url = 0, - Int_t id = -1, Int_t remid = -1, Int_t status = kIdle, TProof *p = 0) - : TNamed(tag, alias), - fLocalId(id), fStatus(0), fProof(p), fRemoteId(remid), fUrl(url) { SetStatus(status); } - ~TProofDesc() override { } - - Int_t GetLocalId() const { return fLocalId; } - TProof *GetProof() const { return fProof; } - Int_t GetRemoteId() const { return fRemoteId; } - Int_t GetStatus() const { return fStatus; } - const char *GetUrl() const { return fUrl; } - - Bool_t IsIdle() const { return (fStatus == kIdle) ? kTRUE : kFALSE; } - Bool_t IsRunning() const { return (fStatus == kRunning) ? kTRUE : kFALSE; } - Bool_t IsShuttingDown() const { return (fStatus == kShutdown) ? kTRUE : kFALSE; } - - Bool_t MatchId(Int_t id) const { return (fLocalId == id); } - - void Print(Option_t *opt = "") const override; - - void SetStatus(Int_t st) { fStatus = (st < kIdle || st > kShutdown) ? -1 : st; } - - void SetProof(TProof *p) { fProof = p; } - void SetRemoteId(Int_t id) { fRemoteId = id; } - - ClassDefOverride(TProofDesc,1) // Small class describing a proof session -}; - -#endif diff --git a/proof/proof/inc/TProofMgrLite.h b/proof/proof/inc/TProofMgrLite.h deleted file mode 100644 index 63a1809f194a2..0000000000000 --- a/proof/proof/inc/TProofMgrLite.h +++ /dev/null @@ -1,42 +0,0 @@ -// @(#)root/proofx:$Id$ -// Author: G. Ganis, Apr 2008 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofMgrLite -#define ROOT_TProofMgrLite - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofMgrLite // -// // -// Basic functionality implementtaion in the case of Lite sessions // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProofMgr.h" -#include "TUrl.h" - -class TProofMgrLite : public TProofMgr { - -public: - TProofMgrLite(const char *url, Int_t loglevel = -1, const char *alias = ""); - ~TProofMgrLite() override { } - - TProof *CreateSession(const char * = 0, const char * = 0, Int_t = -1) override; - TProofLog *GetSessionLogs(Int_t ridx = 0, const char *stag = 0, - const char *pattern = "-v | SvcMsg", Bool_t rescan = kFALSE) override; - TObjString *ReadBuffer(const char *file, Long64_t ofs, Int_t len) override; - TObjString *ReadBuffer(const char *file, const char *pattern) override; - - ClassDefOverride(TProofMgrLite,0) // XrdProofd PROOF manager interface -}; - -#endif diff --git a/proof/proof/inc/TProofNodeInfo.h b/proof/proof/inc/TProofNodeInfo.h deleted file mode 100644 index 7aea4d69c467d..0000000000000 --- a/proof/proof/inc/TProofNodeInfo.h +++ /dev/null @@ -1,90 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Paul Nilsson 7/12/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofNodeInfo -#define ROOT_TProofNodeInfo - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofNodeInfo // -// // -// Implementation of PROOF node info. // -// The purpose of this class is to provide a complete node description // -// for masters, submasters and workers. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TString.h" - -class TProofResourcesStatic; - - -class TProofNodeInfo : public TObject { - -friend class TProofResourcesStatic; - -public: - enum ENodeType { kMaster, kSubMaster, kWorker }; - -private: - ENodeType fNodeType; // Distinction between master, submaster and worker - TString fName; // Tag of the node (name:port) - TString fNodeName; // Name of the node - TString fWorkDir; // Working directory - TString fOrdinal; // Worker ordinal number - TString fImage; // File system image - TString fId; // Id number - TString fConfig; // Configuration file name [for submasters] - TString fMsd; // Msd value [for submasters] - Int_t fPort; // Port number - Int_t fPerfIndex; // Performance index - Int_t fNWrks; // Number of workers (when kSubMaster) - - void operator=(const TProofNodeInfo &); // idem - -public: - TProofNodeInfo(); - TProofNodeInfo(const char *str); - TProofNodeInfo(const TProofNodeInfo &nodeInfo); - ~TProofNodeInfo() override { } - - const char *GetName() const override { return fName; } - ENodeType GetNodeType() const { return fNodeType; } - const TString &GetNodeName() const { return fNodeName; } - const TString &GetWorkDir() const { return fWorkDir; } - const TString &GetOrdinal() const { return fOrdinal; } - const TString &GetImage() const { return fImage; } - const TString &GetId() const { return fId; } - const TString &GetConfig() const { return fConfig; } - const TString &GetMsd() const { return fMsd; } - Int_t GetPort() const { return fPort; } - Int_t GetPerfIndex() const { return fPerfIndex; } - Int_t GetNWrks() const { return fNWrks; } - - Bool_t IsMaster() const { return (fNodeType == kMaster) ? kTRUE : kFALSE; } - Bool_t IsSubMaster() const { return (fNodeType == kSubMaster) ? kTRUE : kFALSE; } - Bool_t IsWorker() const { return (fNodeType == kWorker) ? kTRUE : kFALSE; } - - void SetNodeType(ENodeType nt) { fNodeType = nt; } - void SetNWrks(Int_t nw) { fNWrks = nw; } - - void Assign(const TProofNodeInfo &n); - - void Print(const Option_t *) const override; - - static ENodeType GetNodeType(const TString &type); - - ClassDefOverride(TProofNodeInfo,1) // Class describing a PROOF node -}; - -#endif diff --git a/proof/proof/inc/TProofOutputFile.h b/proof/proof/inc/TProofOutputFile.h deleted file mode 100644 index b2cb341cae694..0000000000000 --- a/proof/proof/inc/TProofOutputFile.h +++ /dev/null @@ -1,127 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Long Tran-Thanh 14/09/07 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofOutputFile -#define ROOT_TProofOutputFile - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofOutputFile // -// // -// Small class to steer the merging of files produced on workers // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TNamed.h" - -class TCollection; -class TString; -class TList; -class TFile; -class TFileCollection; -class TFileMerger; - -class TProofOutputFile : public TNamed { - -friend class TProof; -friend class TProofPlayer; -friend class TProofPlayerRemote; - -public: - enum ERunType { kMerge = 1, // Type of run: merge or dataset creation - kDataset = 2}; - enum ETypeOpt { kRemote = 1, // Merge from original copies - kLocal = 2, // Make local copies before merging - kCreate = 4, // Create dataset - kRegister = 8, // Register dataset - kOverwrite = 16, // Force dataset replacement during registration - kVerify = 32}; // Verify the registered dataset - -private: - TProofOutputFile(const TProofOutputFile&); // Not implemented - TProofOutputFile& operator=(const TProofOutputFile&); // Not implemented - - TString fDir; // name of the directory to be exported - TString fRawDir; // name of the local directory where to create the file - TString fFileName; - TString fOptionsAnchor; // options and anchor string including delimiters, e.g. "?myopts#myanchor" - TString fOutputFileName; - TString fWorkerOrdinal; - TString fLocalHost; // Host where the file was created - Bool_t fIsLocal; // kTRUE if the file is in the sandbox - Bool_t fMerged; - ERunType fRunType; // Type of run (see enum ERunType) - UInt_t fTypeOpt; // Option (see enum ETypeOpt) - Bool_t fMergeHistosOneGo; // If true merge histos in one go (argument to TFileMerger) - - TFileCollection *fDataSet; // Instance of the file collection in 'dataset' mode - TFileMerger *fMerger; // Instance of the file merger in 'merge' mode - - void Init(const char *path, const char *dsname); - void SetFileName(const char* name) { fFileName = name; } - void SetDir(const char* dir, Bool_t raw = kFALSE) { if (raw) { fRawDir = dir; } else { fDir = dir; } } - void SetMerged(Bool_t merged = kTRUE) { fMerged = merged; } - void SetWorkerOrdinal(const char* ordinal) { fWorkerOrdinal = ordinal; } - - void AddFile(TFileMerger *merger, const char *path); - void NotifyError(const char *errmsg); - void Unlink(const char *path); - -protected: - -public: - enum EStatusBits { - kOutputFileNameSet = BIT(16), - kRetrieve = BIT(17), // If set, the file is copied to the final destination via the client - kSwapFile = BIT(18) // Set when the represented file is the result of the automatic - // save-to-file functionality - }; - TProofOutputFile() : fDir(), fRawDir(), fFileName(), fOptionsAnchor(), fOutputFileName(), - fWorkerOrdinal(), fLocalHost(), fIsLocal(kFALSE), fMerged(kFALSE), - fRunType(kMerge), fTypeOpt(kRemote), fMergeHistosOneGo(kFALSE), - fDataSet(0), fMerger(0) { } - TProofOutputFile(const char *path, const char *option = "M", const char *dsname = 0); - TProofOutputFile(const char *path, ERunType type, UInt_t opt = kRemote, const char *dsname = 0); - ~TProofOutputFile() override; - - const char *GetDir(Bool_t raw = kFALSE) const { return (raw) ? fRawDir : fDir; } - TFileCollection *GetFileCollection(); - TFileMerger *GetFileMerger(Bool_t local = kFALSE); - const char *GetFileName() const { return fFileName; } - const char *GetLocalHost() const { return fLocalHost; } - const char *GetOptionsAnchor() const { return fOptionsAnchor; } - const char *GetOutputFileName() const { return fOutputFileName; } - const char *GetWorkerOrdinal() const { return fWorkerOrdinal; } - - ERunType GetRunType() const { return fRunType; } - UInt_t GetTypeOpt() const { return fTypeOpt; } - Bool_t IsMerge() const { return (fRunType == kMerge) ? kTRUE : kFALSE; } - Bool_t IsMerged() const { return fMerged; } - Bool_t IsRegister() const { return ((fTypeOpt & kRegister) || (fTypeOpt & kVerify)) ? kTRUE : kFALSE; } - - Bool_t IsRetrieve() const { return (TestBit(TProofOutputFile::kRetrieve)) ? kTRUE : kFALSE; } - void SetRetrieve(Bool_t on = kTRUE) { if (on) { SetBit(TProofOutputFile::kRetrieve); - } else { ResetBit(TProofOutputFile::kRetrieve); }} - - Int_t AdoptFile(TFile *f); // Adopt a TFile already open - TFile* OpenFile(const char *opt); // Open a file with the specified name in fFileName1 - Long64_t Merge(TCollection *list); - void Print(Option_t *option = "") const override; - void SetOutputFileName(const char *name); - void ResetFileCollection() { fDataSet = 0; } - - static Int_t AssertDir(const char *dirpath); - - ClassDefOverride(TProofOutputFile,5) // Wrapper class to steer the merging of files produced on workers -}; - -#endif diff --git a/proof/proof/inc/TProofOutputList.h b/proof/proof/inc/TProofOutputList.h deleted file mode 100644 index 962866b4cbd20..0000000000000 --- a/proof/proof/inc/TProofOutputList.h +++ /dev/null @@ -1,51 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G. Ganis 04/08/2010 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofOutputList -#define ROOT_TProofOutputList - -////////////////////////////////////////////////////////////////////////// -// // -// TProofList // -// // -// Derivation of TList with an overload of ls() and Print() allowing // -// to filter out some of the variables. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TList.h" - -class TProofOutputList : public TList { - -private: - TList *fDontShow; // list of reg expression defining what should not be shown - -public: - TProofOutputList(const char *dontshow = "PROOF_*"); - ~TProofOutputList() override; - - void AttachList(TList *alist); - - void ls(Option_t *option="") const override ; - void Print(Option_t *option="") const override; - void Print(Option_t *option, Int_t recurse) const override - { TCollection::Print(option, recurse); } - void Print(Option_t *option, const char* wildcard, Int_t recurse=1) const override - { TCollection::Print(option, wildcard, recurse); } - void Print(Option_t *option, TPRegexp& regexp, Int_t recurse=1) const override - { TCollection::Print(option, regexp, recurse);} - - TList *GetDontShowList() { return fDontShow; } - - ClassDefOverride(TProofOutputList, 1); // Output list specific TList derivation -}; - -#endif diff --git a/proof/proof/inc/TProofProgressStatus.h b/proof/proof/inc/TProofProgressStatus.h deleted file mode 100644 index aa646fb51a56d..0000000000000 --- a/proof/proof/inc/TProofProgressStatus.h +++ /dev/null @@ -1,79 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Jan Iwaszkiewicz 08/08/08 - -/************************************************************************* - * Copyright (C) 1995-2008, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofProgressStatus -#define ROOT_TProofProgressStatus - -#include "TObject.h" - -////////////////////////////////////////////////////////////////////////// -// // -// TProofProgressStatus // -// // -// Small class including processing statistics // -// // -////////////////////////////////////////////////////////////////////////// - -class TProofProgressStatus : public TObject { -public: - enum EStatusBits { kFileNotOpen = BIT(15), - kFileCorrupted = BIT(16) - }; -private: - Long64_t fLastEntries; // Last chunck - Long64_t fEntries; - Long64_t fBytesRead; - Long64_t fReadCalls; - Double_t fLearnTime; // the time spent in learning phase - Double_t fLastProcTime; // the wallclock time of the last addition - Double_t fProcTime; // the wallclock time - Double_t fCPUTime; - Double_t fLastUpdate; // Timestamp of last change -public: - TProofProgressStatus(Long64_t fEntries = 0, Long64_t fBytesRead = 0, - Long64_t fReadCalls = 0, - Double_t fProcTime = 0, Double_t fCPUTime = 0); - void Reset() { fEntries = 0, - fBytesRead = 0, fProcTime = 0, fCPUTime = 0; SetLastUpdate();} - - inline Long64_t GetEntries() const { return fEntries; } - inline Long64_t GetBytesRead() const { return fBytesRead; } - inline Long64_t GetReadCalls() const { return fReadCalls; } - inline Double_t GetLearnTime() const { return fLearnTime; } - inline Double_t GetProcTime() const { return fProcTime; } - inline Double_t GetCPUTime() const { return fCPUTime; } - inline Double_t GetLastUpdate() const { return fLastUpdate; } - inline Double_t GetRate() const { return ((fProcTime > 0) ? fEntries/fProcTime : 0); } - Double_t GetCurrentRate() const; - inline void SetLastEntries(Long64_t entries) { fLastEntries = entries; } - inline void SetEntries(Long64_t entries) { fEntries = entries; } - inline void IncEntries(Long64_t entries = 1) { fLastEntries = entries; fEntries += entries; } - inline void IncBytesRead(Long64_t bytesRead) { fBytesRead += bytesRead; } - inline void SetBytesRead(Long64_t bytesRead) { fBytesRead = bytesRead; } - inline void IncReadCalls(Long64_t readCalls) { fReadCalls += readCalls; } - inline void SetReadCalls(Long64_t readCalls) { fReadCalls = readCalls; } - inline void SetLearnTime(Double_t learnTime) { fLearnTime = learnTime; } - inline void SetLastProcTime(Double_t procTime) { fLastProcTime = procTime; } - inline void SetProcTime(Double_t procTime) { fProcTime = procTime; } - inline void IncProcTime(Double_t procTime) { fLastProcTime = procTime; fProcTime += procTime; } - inline void SetCPUTime(Double_t procTime) { fCPUTime = procTime; } - inline void IncCPUTime(Double_t procTime) { fCPUTime += procTime; } - void SetLastUpdate(Double_t updtTime = 0); - void Print(Option_t* option = "") const override; - - TProofProgressStatus operator-(TProofProgressStatus &st); - TProofProgressStatus &operator+=(const TProofProgressStatus &st); - TProofProgressStatus &operator-=(const TProofProgressStatus &st); - - ClassDefOverride(TProofProgressStatus,2) // Proof progress status class -}; - -#endif diff --git a/proof/proof/inc/TProofQueryResult.h b/proof/proof/inc/TProofQueryResult.h deleted file mode 100644 index a191e615a755b..0000000000000 --- a/proof/proof/inc/TProofQueryResult.h +++ /dev/null @@ -1,55 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G Ganis Aug 2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofQueryResult -#define ROOT_TProofQueryResult - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofQueryResult // -// // -// TQueryResult version adapted to PROOF neeeds. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TQueryResult.h" - -class TDSet; - -class TProofQueryResult : public TQueryResult { - -friend class TProofLite; -friend class TProofServ; -friend class TQueryResultManager; - -private: - Int_t fStartLog; //log file offset at start - - TProofQueryResult(Int_t seqnum, const char *opt, TList *inlist, - Long64_t entries, Long64_t first, TDSet *dset, - const char *selec, TObject *elist = 0); - - void RecordEnd(EQueryStatus status, TList *outlist = 0) override - { TQueryResult::RecordEnd(status, outlist); } - - void SetFinalized() override { TQueryResult::SetFinalized(); } - void SetResultFile(const char *rf) { fResultFile = rf; } - void SetRunning(Int_t startlog, const char *par, Int_t nwrks); - -public: - TProofQueryResult() : TQueryResult(), fStartLog(-1) { } - ~TProofQueryResult() override { } - - ClassDefOverride(TProofQueryResult,1) //Class describing a PROOF query -}; - -#endif diff --git a/proof/proof/inc/TProofResources.h b/proof/proof/inc/TProofResources.h deleted file mode 100644 index 2749af89166cb..0000000000000 --- a/proof/proof/inc/TProofResources.h +++ /dev/null @@ -1,51 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Paul Nilsson 7/12/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofResources -#define ROOT_TProofResources - -////////////////////////////////////////////////////////////////////////// -// // -// TProofResources // -// // -// Abstract base class for PROOF resources. // -// The class contains common method declarations for derived classes // -// such as TProofResourcesStatic which reads and interprets static // -// config files, and returns master, submaster and worker information // -// using TProofNodeInfo objects. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" - -class TList; -class TString; -class TProofNodeInfo; - - -class TProofResources : public TObject { - -protected: - Bool_t fValid; // kTRUE if resource information was processed correctly - -public: - TProofResources() : fValid(kFALSE) { } - ~TProofResources() override { } - - virtual TProofNodeInfo *GetMaster() = 0; - virtual TList *GetSubmasters() = 0; - virtual TList *GetWorkers() = 0; - virtual Bool_t IsValid() const { return fValid; } - - ClassDefOverride(TProofResources,0) // Abstract class describing PROOF resources -}; - -#endif diff --git a/proof/proof/inc/TProofResourcesStatic.h b/proof/proof/inc/TProofResourcesStatic.h deleted file mode 100644 index c5000c6f498e4..0000000000000 --- a/proof/proof/inc/TProofResourcesStatic.h +++ /dev/null @@ -1,71 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Paul Nilsson 7/12/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofResourcesStatic -#define ROOT_TProofResourcesStatic - -////////////////////////////////////////////////////////////////////////// -// // -// TProofResourcesStatic // -// // -// Implementation of PROOF static resources. // -// The purpose of this class is to provide a standard interface to // -// static config files. It interprets Proof config files (proof.conf) // -// and sorts the contents into TProofNodeInfo objects. Master info will // -// be placed in fMaster (of type TProofNodeInfo). Submaster info will // -// be put in fSubmasterList (a TList of TProofNodeInfo objects), while // -// workers (and condorworkers) will be placed in fWorkerList (a TList // -// of TProofNodeInfo objects). // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProofResources.h" -#include "TString.h" - -class TList; -class TProofNodeInfo; - - -class TProofResourcesStatic : public TProofResources { - -public: - enum EInfoType { kNodeType, kOption, kHost }; - -private: - TProofNodeInfo *fMaster; // Master node info - TList *fSubmasterList; // Node info list with all submasters - TList *fWorkerList; // Node info list with all workers - Bool_t fFoundMaster; // kTRUE if config file has master info - TString fFileName; // Config file name - - void InitResources(); - Bool_t ReadConfigFile(const char *confDir, const char *fileName); - - static EInfoType GetInfoType(const TString &word); - static void SetOption(TProofNodeInfo *nodeinfo, const TString &option, - const TString &value); - static TProofNodeInfo *CreateNodeInfo(const TString &name); - -public: - TProofResourcesStatic(); - TProofResourcesStatic(const char *confDir, const char *fileName); - ~TProofResourcesStatic() override; - - TProofNodeInfo *GetMaster() override; - TList *GetSubmasters() override; - TList *GetWorkers() override; - TString GetFileName() const { return fFileName; } - - ClassDefOverride(TProofResourcesStatic,0) // Class to handle PROOF static config -}; - -#endif - diff --git a/proof/proof/inc/TProofServ.h b/proof/proof/inc/TProofServ.h deleted file mode 100644 index 603572fa19fab..0000000000000 --- a/proof/proof/inc/TProofServ.h +++ /dev/null @@ -1,469 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 16/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -#ifndef ROOT_TProofServ -#define ROOT_TProofServ - -////////////////////////////////////////////////////////////////////////// -// // -// TProofServ // -// // -// TProofServ is the PROOF server. It can act either as the master // -// server or as a slave server, depending on its startup arguments. It // -// receives and handles message coming from the client or from the // -// master server. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TApplication.h" -#include "TString.h" -#include "TSysEvtHandler.h" -#include "TStopwatch.h" -#include "TTimer.h" -#include "TPackMgr.h" -#include "TProofQueryResult.h" - -#include - -class TDataSetManager; -class TDataSetManagerFile; -class TDSet; -class TDSetElement; -class TFileCollection; -class TFileHandler; -class THashList; -class TIdleTOTimer; -class TList; -class TMap; -class TMessage; -class TMonitor; -class TPackMgr; -class TProof; -class TProofLockPath; -class TQueryResultManager; -class TReaperTimer; -class TServerSocket; -class TShutdownTimer; -class TSocket; -class TVirtualProofPlayer; - -// Hook to external function setting up authentication related stuff -// for old versions. -// For backward compatibility -typedef Int_t (*OldProofServAuthSetup_t)(TSocket *, Bool_t, Int_t, - TString &, TString &, TString &); - - -class TProofServ : public TApplication { - -friend class TProofServLite; -friend class TXProofServ; - -public: - enum EStatusBits { kHighMemory = BIT(17) }; - enum EQueryAction { kQueryOK, kQueryModify, kQueryStop, kQueryEnqueued }; - -private: - TString fService; //service we are running, either "proofserv" or "proofslave" - TString fUser; //user as which we run - TString fGroup; //group the user belongs to - TString fConfDir; //directory containing cluster config information - TString fConfFile; //file containing config information - TString fWorkDir; //directory containing all proof related info - TString fImage; //image name of the session - TString fSessionTag; //tag for the server session - TString fTopSessionTag; //tag for the global session - TString fSessionDir; //directory containing session dependent files - TPackMgr *fPackMgr; // Default package manager - TString fCacheDir; //directory containing cache of user files - TString fQueryDir; //directory containing query results and status - TString fDataSetDir; //directory containing info about known data sets - TString fDataDir; //directory containing data files produced during queries - TString fDataDirOpts; //Url type options for fDataDir - TString fAdminPath; //admin path for this session - TString fOutputFile; //path with the temporary results of the current or last query - TProofLockPath *fCacheLock; //cache dir locker - TProofLockPath *fQueryLock; //query dir locker - TString fArchivePath; //default archive path - TSocket *fSocket; //socket connection to client - TProof *fProof; //PROOF talking to slave servers - TVirtualProofPlayer *fPlayer; //actual player - FILE *fLogFile; //log file - Int_t fLogFileDes; //log file descriptor - Long64_t fLogFileMaxSize; //max size for log files (enabled if > 0) - Int_t fProtocol; //protocol version number - TString fOrdinal; //slave ordinal number - Int_t fGroupId; //slave unique id in the active slave group - Int_t fGroupSize; //size of the active slave group - Int_t fLogLevel; //debug logging level - Int_t fNcmd; //command history number - Int_t fGroupPriority; //priority of group the user belongs to (0 - 100) - Bool_t fEndMaster; //true for a master in direct contact only with workers - Bool_t fMasterServ; //true if we are a master server - Bool_t fInterrupt; //if true macro execution will be stopped - Float_t fRealTime; //real time spent executing commands - Float_t fCpuTime; //CPU time spent executing commands - TStopwatch fLatency; //measures latency of packet requests - TStopwatch fCompute; //measures time spent processing a packet - TStopwatch fSaveOutput; //measures time spent saving the partial result - Int_t fQuerySeqNum; //sequential number of the current or last query - - Int_t fTotSessions; //Total number of PROOF sessions on the cluster - Int_t fActSessions; //Total number of active PROOF sessions on the cluster - Float_t fEffSessions; //Effective Number of PROOF sessions on the assigned machines - - TFileHandler *fInputHandler; //Input socket handler - - TQueryResultManager *fQMgr; //Query-result manager - - TList *fWaitingQueries; //list of TProofQueryResult waiting to be processed - Bool_t fIdle; //TRUE if idle - std::recursive_mutex fQMtx; // To protect async msg queue - - TList *fQueuedMsg; //list of messages waiting to be processed - - TString fPrefix; //Prefix identifying the node - - Bool_t fRealTimeLog; //TRUE if log messages should be send back in real-time - - TShutdownTimer *fShutdownTimer; // Timer used to shutdown out-of-control sessions - TReaperTimer *fReaperTimer; // Timer used to control children state - TIdleTOTimer *fIdleTOTimer; // Timer used to control children state - - Int_t fCompressMsg; // Compression level for messages - - TDataSetManager* fDataSetManager; // dataset manager - TDataSetManagerFile *fDataSetStgRepo; // repository for staging requests - - Bool_t fSendLogToMaster; // On workers, controls logs sending to master - - TServerSocket *fMergingSocket; // Socket used for merging outputs if submerger - TMonitor *fMergingMonitor; // Monitor for merging sockets - Int_t fMergedWorkers; // Number of workers merged - - // Quotas (-1 to disable) - Int_t fMaxQueries; //Max number of queries fully kept - Long64_t fMaxBoxSize; //Max size of the sandbox - Long64_t fHWMBoxSize; //High-Water-Mark on the sandbox size - - // Memory limits (-1 to disable) set by envs ROOTPROFOASHARD, PROOF_VIRTMEMMAX, PROOF_RESMEMMAX - static Long_t fgVirtMemMax; //Hard limit enforced by the system (in kB) - static Long_t fgResMemMax; //Hard limit on the resident memory checked - //in TProofPlayer::Process (in kB) - static Float_t fgMemHWM; // Threshold fraction of max for warning and finer monitoring - static Float_t fgMemStop; // Fraction of max for stop processing - - // In bytes; default is 1MB - Long64_t fMsgSizeHWM; //High-Water-Mark on the size of messages with results - - static FILE *fgErrorHandlerFile; // File where to log - static Int_t fgRecursive; // Keep track of recursive inputs during processing - - // Control sending information to syslog - static Int_t fgLogToSysLog; // >0 sent to syslog too - static TString fgSysLogService; // name of the syslog service (eg: proofm-0, proofw-0.67) - static TString fgSysLogEntity; // logging entity (:) - - Int_t GetCompressionLevel() const; - - void RedirectOutput(const char *dir = 0, const char *mode = "w"); - Int_t CatMotd(); - Int_t UnloadPackage(const char *package); - Int_t UnloadPackages(); - Int_t OldAuthSetup(TString &wconf); - Int_t GetPriority(); - - // Query handlers - TProofQueryResult *MakeQueryResult(Long64_t nentries, const char *opt, - TList *inl, Long64_t first, TDSet *dset, - const char *selec, TObject *elist); - void SetQueryRunning(TProofQueryResult *pq); - - // Results handling - Int_t SendResults(TSocket *sock, TList *outlist = 0, TQueryResult *pq = 0); - Bool_t AcceptResults(Int_t connections, TVirtualProofPlayer *mergerPlayer); - - // Waiting queries handlers - void SetIdle(Bool_t st = kTRUE); - Bool_t IsWaiting(); - Int_t WaitingQueries(); - Int_t QueueQuery(TProofQueryResult *pq); - TProofQueryResult *NextQuery(); - Int_t CleanupWaitingQueries(Bool_t del = kTRUE, TList *qls = 0); - -protected: - virtual void HandleArchive(TMessage *mess, TString *slb = 0); - virtual Int_t HandleCache(TMessage *mess, TString *slb = 0); - virtual void HandleCheckFile(TMessage *mess, TString *slb = 0); - virtual Int_t HandleDataSets(TMessage *mess, TString *slb = 0); - virtual void HandleSubmerger(TMessage *mess); - virtual void HandleFork(TMessage *mess); - virtual Int_t HandleLibIncPath(TMessage *mess); - virtual void HandleProcess(TMessage *mess, TString *slb = 0); - virtual void HandleQueryList(TMessage *mess); - virtual void HandleRemove(TMessage *mess, TString *slb = 0); - virtual void HandleRetrieve(TMessage *mess, TString *slb = 0); - virtual Int_t HandleWorkerLists(TMessage *mess); - - virtual void ProcessNext(TString *slb = 0); - virtual Int_t Setup(); - Int_t SetupCommon(); - virtual void MakePlayer(); - virtual void DeletePlayer(); - - virtual Int_t Fork(); - Int_t GetSessionStatus(); - Bool_t IsIdle(); - Bool_t UnlinkDataDir(const char *path); - - static TString fgLastMsg; // Message about status before exception - static Long64_t fgLastEntry; // Last entry before exception - -public: - TProofServ(Int_t *argc, char **argv, FILE *flog = 0); - ~TProofServ() override; - - virtual Int_t CreateServer(); - - TProof *GetProof() const { return fProof; } - const char *GetService() const { return fService; } - const char *GetConfDir() const { return fConfDir; } - const char *GetConfFile() const { return fConfFile; } - const char *GetUser() const { return fUser; } - const char *GetGroup() const { return fGroup; } - const char *GetWorkDir() const { return fWorkDir; } - const char *GetImage() const { return fImage; } - const char *GetSessionTag() const { return fSessionTag; } - const char *GetTopSessionTag() const { return fTopSessionTag; } - const char *GetSessionDir() const { return fSessionDir; } - const char *GetCacheDir() const { return fCacheDir; } - TPackMgr *GetPackMgr() const { return fPackMgr; } - const char *GetDataDir() const { return fDataDir; } - const char *GetDataDirOpts() const { return fDataDirOpts; } - Int_t GetProtocol() const { return fProtocol; } - const char *GetOrdinal() const { return fOrdinal; } - Int_t GetGroupId() const { return fGroupId; } - Int_t GetGroupSize() const { return fGroupSize; } - Int_t GetLogLevel() const { return fLogLevel; } - TSocket *GetSocket() const { return fSocket; } - Float_t GetRealTime() const { return fRealTime; } - Float_t GetCpuTime() const { return fCpuTime; } - Int_t GetQuerySeqNum() const { return fQuerySeqNum; } - - Int_t GetTotSessions() const { return fTotSessions; } - Int_t GetActSessions() const { return fActSessions; } - Float_t GetEffSessions() const { return fEffSessions; } - - void GetOptions(Int_t *argc, char **argv) override; - TList *GetEnabledPackages() const { return fPackMgr->GetListOfEnabled(); } - - static Long_t GetVirtMemMax(); - static Long_t GetResMemMax(); - static Float_t GetMemHWM(); - static Float_t GetMemStop(); - - Long64_t GetMsgSizeHWM() const { return fMsgSizeHWM; } - - const char *GetPrefix() const { return fPrefix; } - - void FlushLogFile(); - void TruncateLogFile(); // Called also by TDSetProxy::Next() - - TProofLockPath *GetCacheLock() { return fCacheLock; } //cache dir locker; used by TProofPlayer - - virtual EQueryAction GetWorkers(TList *workers, Int_t &prioritychange, - Bool_t resume = kFALSE); - void HandleException(Int_t sig) override; - virtual Int_t HandleSocketInput(TMessage *mess, Bool_t all); - virtual void HandleSocketInput(); - virtual void HandleUrgentData(); - virtual void HandleSigPipe(); - virtual void HandleTermination() { Terminate(0); } - void Interrupt() { fInterrupt = kTRUE; } - Bool_t IsEndMaster() const { return fEndMaster; } - Bool_t IsMaster() const { return fMasterServ; } - Bool_t IsParallel() const; - Bool_t IsTopMaster() const { return fOrdinal == "0"; } - - void Run(Bool_t retrn = kFALSE) override; - - void Print(Option_t *option="") const override; - - void RestartComputeTime(); - - TObject *Get(const char *namecycle); - TDSetElement *GetNextPacket(Long64_t totalEntries = -1); - virtual void ReleaseWorker(const char *) { } - void Reset(const char *dir); - Int_t ReceiveFile(const char *file, Bool_t bin, Long64_t size); - void SendAsynMessage(const char *msg, Bool_t lf = kTRUE); - virtual void SendLogFile(Int_t status = 0, Int_t start = -1, Int_t end = -1); - void SendStatistics(); - void SendParallel(Bool_t async = kFALSE); - - Int_t UpdateSessionStatus(Int_t xst = -1); - - // Disable / Enable read timeout - virtual void DisableTimeout() { } - virtual void EnableTimeout() { } - - void Terminate(Int_t status) override; - - // Log control - void LogToMaster(Bool_t on = kTRUE) { fSendLogToMaster = on; } - - static FILE *SetErrorHandlerFile(FILE *ferr); - static void ErrorHandler(Int_t level, Bool_t abort, const char *location, - const char *msg); - - static void ResolveKeywords(TString &fname, const char *path = 0); - - static void SetLastMsg(const char *lastmsg); - static void SetLastEntry(Long64_t lastentry); - - // To handle local data server related paths - static void FilterLocalroot(TString &path, const char *url = "root://dum/"); - static void GetLocalServer(TString &dsrv); - - // To prepara ethe map of files to process - static TMap *GetDataSetNodeMap(TFileCollection *fc, TString &emsg); - static Int_t RegisterDataSets(TList *in, TList *out, TDataSetManager *dsm, TString &e); - - static Bool_t IsActive(); - static TProofServ *This(); - - ClassDefOverride(TProofServ,0) //PROOF Server Application Interface -}; - -R__EXTERN TProofServ *gProofServ; - -class TProofLockPath : public TNamed { -private: - Int_t fLockId; //file id of dir lock - -public: - TProofLockPath(const char *path) : TNamed(path,path), fLockId(-1) { } - ~TProofLockPath() override { if (IsLocked()) Unlock(); } - - Int_t Lock(); - Int_t Unlock(); - - Bool_t IsLocked() const { return (fLockId > -1); } -}; - -class TProofLockPathGuard { -private: - TProofLockPath *fLocker; //locker instance - -public: - TProofLockPathGuard(TProofLockPath *l) { fLocker = l; if (fLocker) fLocker->Lock(); } - ~TProofLockPathGuard() { if (fLocker) fLocker->Unlock(); } -}; - -//----- Handles output from commands executed externally via a pipe. ---------// -//----- The output is redirected one level up (i.e., to master or client). ---// -//______________________________________________________________________________ -class TProofServLogHandler : public TFileHandler { -private: - TSocket *fSocket; // Socket where to redirect the message - FILE *fFile; // File connected with the open pipe - TString fPfx; // Prefix to be prepended to messages - - static TString fgPfx; // Default prefix to be prepended to messages - static Int_t fgCmdRtn; // Return code of the command execution (available only - // after closing the pipe) -public: - enum EStatusBits { kFileIsPipe = BIT(23) }; - TProofServLogHandler(const char *cmd, TSocket *s, const char *pfx = ""); - TProofServLogHandler(FILE *f, TSocket *s, const char *pfx = ""); - ~TProofServLogHandler() override; - - Bool_t IsValid() { return ((fFile && fSocket) ? kTRUE : kFALSE); } - - Bool_t Notify() override; - Bool_t ReadNotify() override { return Notify(); } - - static void SetDefaultPrefix(const char *pfx); - static Int_t GetCmdRtn(); -}; - -//--- Guard class: close pipe, deactivatethe related descriptor --------------// -//______________________________________________________________________________ -class TProofServLogHandlerGuard { - -private: - TProofServLogHandler *fExecHandler; - -public: - TProofServLogHandlerGuard(const char *cmd, TSocket *s, - const char *pfx = "", Bool_t on = kTRUE); - TProofServLogHandlerGuard(FILE *f, TSocket *s, - const char *pfx = "", Bool_t on = kTRUE); - virtual ~TProofServLogHandlerGuard(); -}; - -//--- Special timer to control delayed shutdowns -//______________________________________________________________________________ -class TShutdownTimer : public TTimer { -private: - TProofServ *fProofServ; - Int_t fTimeout; - -public: - TShutdownTimer(TProofServ *p, Int_t delay); - - Bool_t Notify() override; -}; - -//--- Synchronous timer used to reap children processes change of state -//______________________________________________________________________________ -class TReaperTimer : public TTimer { -private: - TList *fChildren; // List of children (forked) processes - -public: - TReaperTimer(Long_t frequency = 1000) : TTimer(frequency, kTRUE), fChildren(0) { } - ~TReaperTimer() override; - - void AddPid(Int_t pid); - Bool_t Notify() override; -}; - -//--- Special timer to terminate idle sessions -//______________________________________________________________________________ -class TIdleTOTimer : public TTimer { -private: - TProofServ *fProofServ; - -public: - TIdleTOTimer(TProofServ *p, Int_t delay) : TTimer(delay, kTRUE), fProofServ(p) { } - - Bool_t Notify() override; -}; -//______________________________________________________________________________ -class TIdleTOTimerGuard { - -private: - TIdleTOTimer *fIdleTOTimer; - -public: - TIdleTOTimerGuard(TIdleTOTimer *t) : fIdleTOTimer(t) { if (fIdleTOTimer) fIdleTOTimer->Stop(); } - virtual ~TIdleTOTimerGuard() { if (fIdleTOTimer) fIdleTOTimer->Start(-1, kTRUE); } -}; - -//______________________________________________________________________________ -inline Int_t TProofServ::GetCompressionLevel() const -{ - return (fCompressMsg < 0) ? -1 : fCompressMsg % 100; -} - -#endif diff --git a/proof/proof/inc/TProofServLite.h b/proof/proof/inc/TProofServLite.h deleted file mode 100644 index 4a4d4934225c4..0000000000000 --- a/proof/proof/inc/TProofServLite.h +++ /dev/null @@ -1,58 +0,0 @@ -// @(#)root/proofx:$Id$ -// Author: G. Ganis Oct 2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofServLite -#define ROOT_TProofServLite - -////////////////////////////////////////////////////////////////////////// -// // -// TProofServLite // -// // -// TProofServLite is the version of the PROOF worker server for local // -// running. The client starts directly the desired number of these // -// workers; the master and daemons are eliminated, optimizing the number// -// of messages exchanged and created / destroyed. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProofServ.h" - -class TProofServLiteInterruptHandler; - -class TProofServLite : public TProofServ { - -private: - TProofServLiteInterruptHandler *fInterruptHandler; - TString fSockPath; // unix socket path - - Bool_t fTerminated; //true if Terminate() has been already called - - Int_t Setup() override; - Int_t SetupOnFork(const char *ord); - -public: - TProofServLite(Int_t *argc, char **argv, FILE *flog = 0); - ~TProofServLite() override; - - Int_t CreateServer() override; - - void HandleFork(TMessage *mess) override; - - //void HandleUrgentData(); - void HandleSigPipe() override; - void HandleTermination() override; - - void Terminate(Int_t status) override; - - ClassDefOverride(TProofServLite,0) //PROOF-Lite Server Application Interface -}; - -#endif diff --git a/proof/proof/inc/TProofSuperMaster.h b/proof/proof/inc/TProofSuperMaster.h deleted file mode 100644 index b0a97fa3a5c48..0000000000000 --- a/proof/proof/inc/TProofSuperMaster.h +++ /dev/null @@ -1,78 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 13/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofSuperMaster -#define ROOT_TProofSuperMaster - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofSuperMaster // -// // -// This class controls a Parallel ROOT Facility, PROOF, cluster. // -// It fires the slave servers, it keeps track of how many slaves are // -// running, it keeps track of the slaves running status, it broadcasts // -// messages to all slaves, it collects results, etc. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProof.h" -#include "TString.h" - -class TVirtualProofPlayer; -class TDSet; - -class TProofSuperMaster : public TProof { - -friend class TProofPlayerSuperMaster; - -protected: - Bool_t StartSlaves(Bool_t) override; - void ValidateDSet(TDSet *dset) override; - TVirtualProofPlayer *MakePlayer(const char *player = 0, TSocket *s = 0) override; - -public: - TProofSuperMaster(const char *masterurl, const char *conffile = kPROOF_ConfFile, - const char *confdir = kPROOF_ConfDir, Int_t loglevel = 0, - const char *alias = 0, TProofMgr *mgr = 0); - ~TProofSuperMaster() override { } - - Long64_t Process(TDSet *set, const char *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0) override; - Long64_t Process(TFileCollection *fc, const char *sel, Option_t *o = "", - Long64_t nent = -1, Long64_t fst = 0) override - { return TProof::Process(fc, sel, o, nent, fst); } - Long64_t Process(const char *dsname, const char *sel, - Option_t *o = "", Long64_t nent = -1, - Long64_t fst = 0, TObject *enl = 0) override - { return TProof::Process(dsname, sel, o, nent, fst, enl); } - Long64_t Process(const char *sel, Long64_t nent, Option_t *o = "") override - { return TProof::Process(sel, nent, o); } - // Process via TSelector - Long64_t Process(TDSet *set, TSelector *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0) override - { return TProof::Process(set, selector, option, nentries, firstentry); } - Long64_t Process(TFileCollection *fc, TSelector *sel, Option_t *o = "", - Long64_t nent = -1, Long64_t fst = 0) override - { return TProof::Process(fc, sel, o, nent, fst); } - Long64_t Process(const char *dsname, TSelector *sel, - Option_t *o = "", Long64_t nent = -1, - Long64_t fst = 0, TObject *enl = 0) override - { return TProof::Process(dsname, sel, o, nent, fst, enl); } - Long64_t Process(TSelector *sel, Long64_t nent, Option_t *o = "") override - { return TProof::Process(sel, nent, o); } - - ClassDefOverride(TProofSuperMaster,0) //PROOF control class for making submasters -}; - -#endif diff --git a/proof/proof/inc/TQueryResultManager.h b/proof/proof/inc/TQueryResultManager.h deleted file mode 100644 index 6bff031284b2b..0000000000000 --- a/proof/proof/inc/TQueryResultManager.h +++ /dev/null @@ -1,88 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis Mar 2008 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -#ifndef ROOT_TQueryResultManager -#define ROOT_TQueryResultManager - -////////////////////////////////////////////////////////////////////////// -// // -// TQueryResultManager // -// // -// This class manages the query-result area. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TStopwatch.h" -#include "TString.h" - -class TList; -class TProof; -class TProofLockPath; -class TProofQueryResult; -class TQueryResult; -class TVirtualProofPlayer; - -class TQueryResultManager : public TObject { - -private: - TString fQueryDir; //directory containing query results and status - TString fSessionTag; //tag for the session - TString fSessionDir; //directory containing session dependent files - Int_t fSeqNum; //sequential number of last processed query - Int_t fDrawQueries; //number of draw queries processed - Int_t fKeptQueries; //number of queries fully in memory and in dir - TList *fQueries; //list of TProofQueryResult objects - TList *fPreviousQueries; //list of TProofQueryResult objects from previous sections - TProofLockPath *fLock; //dir locker - FILE *fLogFile; //log file - TStopwatch fCompute; //measures time spend processing a query on the master - - void AddLogFile(TProofQueryResult *pq); - -public: - TQueryResultManager(const char *qdir, const char *stag, const char *sdir, - TProofLockPath *lck, FILE *logfile = 0); - ~TQueryResultManager() override; - - const char *QueryDir() const { return fQueryDir.Data(); } - Int_t SeqNum() const { return fSeqNum; } - Int_t DrawQueries() const { return fDrawQueries; } - Int_t KeptQueries() const { return fKeptQueries; } - TList *Queries() const { return fQueries; } - TList *PreviousQueries() const { return fPreviousQueries; } - - void IncrementSeqNum() { fSeqNum++; } - void IncrementDrawQueries() { fDrawQueries++; } - - Int_t ApplyMaxQueries(Int_t mxq); - Int_t CleanupQueriesDir(); - Bool_t FinalizeQuery(TProofQueryResult *pq, - TProof *proof, TVirtualProofPlayer *player); - Float_t GetCpuTime() { return fCompute.CpuTime(); } - Float_t GetRealTime() { return fCompute.RealTime(); } - TProofQueryResult *LocateQuery(TString queryref, Int_t &qry, TString &qdir); - void RemoveQuery(TQueryResult *qr, Bool_t soft = kFALSE); - void RemoveQuery(const char *queryref, TList *otherlist = 0); - void ResetTime() { fCompute.Start(); } - void SaveQuery(TProofQueryResult *qr, const char *fout = 0); - void SaveQuery(TProofQueryResult *qr, Int_t mxq); - - Int_t LockSession(const char *sessiontag, TProofLockPath **lck); - Int_t CleanupSession(const char *sessiontag); - void ScanPreviousQueries(const char *dir); - - ClassDefOverride(TQueryResultManager,0) //PROOF query result manager -}; - -#endif - diff --git a/proof/proof/inc/TSelVerifyDataSet.h b/proof/proof/inc/TSelVerifyDataSet.h deleted file mode 100644 index fedad2eb0f4aa..0000000000000 --- a/proof/proof/inc/TSelVerifyDataSet.h +++ /dev/null @@ -1,90 +0,0 @@ -// Author: Sangsu Ryu 28/06/2011 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TSelVerifyDataSet -#define ROOT_TSelVerifyDataSet - -////////////////////////////////////////////////////////////////////////// -// // -// TSelVerifyDataSet // -// // -// PROOF selector to parallel-process dataset on workers // -// // -////////////////////////////////////////////////////////////////////////// - -#include -#include - -class TTree; -class TFileCollection; - -class TSelVerifyDataSet : public TSelector { - -private: - - Int_t fFopt; - Int_t fSopt; - Int_t fRopt; - - // File selection, Reopen and Touch options - Bool_t fAllf; - Bool_t fCheckstg; - Bool_t fNonStgf; - Bool_t fReopen; - Bool_t fTouch; - Bool_t fStgf; - - // File processing options - Bool_t fNoaction; - Bool_t fFullproc; - Bool_t fLocateonly; - Bool_t fStageonly; - - // Run options - Bool_t fDoall; - Bool_t fGetlistonly; - Bool_t fScanlist; - - Bool_t fDbg; - - TString fMss; - TString fStageopts; - - Bool_t fChangedDs; - Int_t fTouched; - Int_t fOpened; - Int_t fDisappeared; - - TFileCollection *fSubDataSet; // Sub-dataset being verified - - void InitMembers(); - -public : - - TSelVerifyDataSet(TTree *); - TSelVerifyDataSet(); - ~TSelVerifyDataSet() override {} - Int_t Version() const override {return 1;} - void Begin(TTree *) override { } - void SlaveBegin(TTree *tree) override; - void Init(TTree *) override { } - Bool_t Notify() override { return kTRUE; } - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override {fInput = input;} - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override { } - - ClassDefOverride(TSelVerifyDataSet,0) //PROOF selector for parallel dataset verification -}; - -#endif diff --git a/proof/proof/inc/TSlave.h b/proof/proof/inc/TSlave.h deleted file mode 100644 index 5b925f3f9e06e..0000000000000 --- a/proof/proof/inc/TSlave.h +++ /dev/null @@ -1,170 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 14/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TSlave -#define ROOT_TSlave - - -////////////////////////////////////////////////////////////////////////// -// // -// TSlave // -// // -// This class describes a PROOF slave server. // -// It contains information like the slaves host name, ordinal number, // -// performance index, socket, etc. Objects of this class can only be // -// created via TProof member functions. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TString.h" - -class TFileHandler; -class TObjString; -class TProof; -class TSlave; -class TSocket; - -// Special type for the hook to external function setting up authentication -// related stuff for old versions. For backward compatibility. -typedef Int_t (*OldSlaveAuthSetup_t)(TSocket *, Bool_t, TString, TString); - -// Special type for the hook to the TSlave constructor, needed to avoid -// using the plugin manager -typedef TSlave *(*TSlave_t)(const char *url, const char *ord, Int_t perf, - const char *image, TProof *proof, Int_t stype, - const char *workdir, const char *msd, Int_t nwk); - -class TSlave : public TObject { - -friend class TProof; -friend class TProofLite; -friend class TSlaveLite; -friend class TXSlave; - -public: - - enum ESlaveType { kMaster, kSlave }; - enum ESlaveStatus { kInvalid, kActive, kInactive }; - enum EStatusBits { - kOutputRequested = BIT(15) // If output has been requested - }; - -private: - - static TSlave_t fgTXSlaveHook; - - TSlave(const TSlave &s) : TObject(s) { } - TSlave(const char *host, const char *ord, Int_t perf, - const char *image, TProof *proof, Int_t stype, - const char *workdir, const char *msd, Int_t = 1); - - Int_t OldAuthSetup(Bool_t master, TString wconf); - void Init(const char *host, Int_t port, Int_t stype); - void operator=(const TSlave &) { } - - static TSlave *Create(const char *url, const char *ord, Int_t perf, - const char *image, TProof *proof, Int_t stype, - const char *workdir, const char *msd, Int_t nwk = 1); - -protected: - TString fName; //slave's hostname - TString fImage; //slave's image name - TString fProofWorkDir; //base proofserv working directory (info obtained from slave) - TString fWorkDir; //slave's working directory (info obtained from slave) - TString fUser; //slave's user id - TString fGroup; //slave's group id - Int_t fPort; //slave's port number - TString fOrdinal; //slave's ordinal number - Int_t fPerfIdx; //relative CPU performance index - Int_t fProtocol; //slave's protocol level - TSocket *fSocket; //socket to slave - TProof *fProof; //proof cluster to which slave belongs - TFileHandler *fInput; //input handler related to this slave - Long64_t fBytesRead; //bytes read by slave (info is obtained from slave) - Float_t fRealTime; //real time spent executing commands (info obtained from slave) - Float_t fCpuTime; //CPU time spent executing commands (info obtained from slave) - ESlaveType fSlaveType; //type of slave (either kMaster or kSlave) - Int_t fStatus; //remote return status - Int_t fParallel; //number of active slaves - TString fMsd; //mass storage domain of slave - TString fSessionTag;//unique tag for ths worker process - - TString fROOTVers; //ROOT version run by worker - TString fArchComp; //Build architecture, compiler on worker (e.g. linux-gcc345) - - TSlave(); - virtual void FlushSocket() { } - void Init(TSocket *s, Int_t stype); - virtual void Interrupt(Int_t type); - virtual Int_t Ping(); - virtual TObjString *SendCoordinator(Int_t kind, const char *msg = 0, Int_t int2 = 0); - virtual Int_t SendGroupPriority(const char * /*grp*/, Int_t /*priority*/) { return 0; } - virtual void SetAlias(const char *alias); - void SetSocket(TSocket *s) { fSocket = s; } - virtual void SetStatus(Int_t st) { fStatus = st; } - virtual void StopProcess(Bool_t abort, Int_t timeout); - -public: - ~TSlave() override; - - virtual void Close(Option_t *opt = ""); - - Int_t Compare(const TObject *obj) const override; - Bool_t IsSortable() const override { return kTRUE; } - - const char *GetName() const override { return fName; } - const char *GetImage() const { return fImage; } - const char *GetProofWorkDir() const { return fProofWorkDir; } - const char *GetWorkDir() const { return fWorkDir; } - const char *GetUser() const { return fUser; } - const char *GetGroup() const { return fGroup; } - Int_t GetPort() const { return fPort; } - const char *GetOrdinal() const { return fOrdinal; } - Int_t GetPerfIdx() const { return fPerfIdx; } - Int_t GetProtocol() const { return fProtocol; } - TSocket *GetSocket() const { return fSocket; } - TProof *GetProof() const { return fProof; } - Long64_t GetBytesRead() const { return fBytesRead; } - Float_t GetRealTime() const { return fRealTime; } - Float_t GetCpuTime() const { return fCpuTime; } - Int_t GetSlaveType() const { return (Int_t)fSlaveType; } - Int_t GetStatus() const { return fStatus; } - Int_t GetParallel() const { return fParallel; } - const char *GetMsd() const { return fMsd; } - const char *GetSessionTag() const { return fSessionTag; } - TFileHandler *GetInputHandler() const { return fInput; } - void SetInputHandler(TFileHandler *ih); - - const char *GetArchCompiler() const { return fArchComp; } - const char *GetROOTVersion() const { return fROOTVers; } - - virtual Bool_t IsValid() const { return fSocket ? kTRUE : kFALSE; } - - void Print(Option_t *option="") const override; - - virtual Int_t SetupServ(Int_t stype, const char *conffile); - - virtual void SetInterruptHandler(Bool_t /* on */) { } - - void SetArchCompiler(const char *ac) { fArchComp = ac; } - void SetROOTVersion(const char *rv) { fROOTVers = rv; } - - void SetSessionTag(const char *st) { fSessionTag = st; } - - static void SetTXSlaveHook(TSlave_t xslavehook); - - virtual void Touch() { } - - ClassDefOverride(TSlave,0) //PROOF slave server -}; - -#endif diff --git a/proof/proof/inc/TSlaveLite.h b/proof/proof/inc/TSlaveLite.h deleted file mode 100644 index 4b359098deb9d..0000000000000 --- a/proof/proof/inc/TSlaveLite.h +++ /dev/null @@ -1,57 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis Mar 2008 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TSlaveLite -#define ROOT_TSlaveLite - - -////////////////////////////////////////////////////////////////////////// -// // -// TSlaveLite // -// // -// This is the version of TSlave for local worker servers. // -// See TSlave for details. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TSlave.h" - -class TObjString; -class TSocket; -class TSignalHandler; - -class TSlaveLite : public TSlave { - -friend class TProof; - -private: - Bool_t fValid; - TSignalHandler *fIntHandler; //interrupt signal handler (ctrl-c) - - void Init(); - -public: - TSlaveLite(const char *ord, Int_t perf, - const char *image, TProof *proof, Int_t stype, - const char *workdir, const char *msd, Int_t = 1); - ~TSlaveLite() override; - - void Close(Option_t *opt = "") override; - void DoError(int level, const char *location, const char *fmt, - va_list va) const override; - - void Print(Option_t *option="") const override; - Int_t SetupServ(Int_t stype, const char *conffile) override; - - ClassDefOverride(TSlaveLite, 0) //PROOF lite worker server -}; - -#endif diff --git a/proof/proof/inc/TVirtualPacketizer.h b/proof/proof/inc/TVirtualPacketizer.h deleted file mode 100644 index fc678889a4a46..0000000000000 --- a/proof/proof/inc/TVirtualPacketizer.h +++ /dev/null @@ -1,177 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 9/7/2002 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TVirtualPacketizer -#define ROOT_TVirtualPacketizer - -////////////////////////////////////////////////////////////////////////// -// // -// TVirtualPacketizer // -// // -// Packetizer is a load balancing object created for each query. // -// It generates packets to be processed on PROOF worker servers. // -// A packet is an event range (begin entry and number of entries) or // -// object range (first object and number of objects) in a TTree // -// (entries) or a directory (objects) in a file. // -// Packets are generated taking into account the performance of the // -// remote machine, the time it took to process a previous packet on // -// the remote machine, the locality of the database files, etc. // -// // -// TVirtualPacketizer includes common parts of PROOF packetizers. // -// Look in subclasses for details. // -// The default packetizer is TPacketizerAdaptive. // -// To use an alternative one, for instance - the TPacketizer, call: // -// proof->SetParameter("PROOF_Packetizer", "TPacketizer"); // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TSlave.h" -#include "TProofProgressStatus.h" -#include "TTime.h" - - -class TDSet; -class TDSetElement; -class TList; -class TMap; -class TMessage; -class TNtuple; -class TNtupleD; -class TProofProgressInfo; -class TSlave; - - -class TVirtualPacketizer : public TObject { - -public: // public because of Sun CC bug - class TVirtualSlaveStat; - -protected: - enum EUseEstOpt { // Option for usage of estimated values - kEstOff = 0, - kEstCurrent = 1, - kEstAverage = 2 - }; - - // General configuration parameters - Double_t fMinPacketTime; // minimum packet time - Double_t fMaxPacketTime; // maximum packet time - TList *fConfigParams; // List of configuration parameters - - TMap *fSlaveStats; // slave status, keyed by correspondig TSlave - - TProofProgressStatus *fProgressStatus; // pointer to status in the player. - TTimer *fProgress; // progress updates timer - - Long64_t fTotalEntries; // total number of entries to be distributed; - // not used in the progressive packetizer - TList *fFailedPackets;// a list of packets that failed while processing - - // Members for progress info - TTime fStartTime; // time offset - Float_t fInitTime; // time before processing - Float_t fProcTime; // time since start of processing - Float_t fTimeUpdt; // time between updates - TNtupleD *fCircProg; // Keeps circular info for "instantenous" - // rate calculations - Long_t fCircN; // Circularity - - TNtuple *fProgressPerf; // {Active workers, evt rate, MBs read} as a function of processing time - Float_t fProcTimeLast; // Time of the last measurement - Int_t fActWrksLast; // Active workers at fProcTimeLast - Float_t fEvtRateLast; // Evt rate at fProcTimeLast - Float_t fMBsReadLast; // MBs read at fProcTimeLast - Float_t fEffSessLast; // Number of effective sessions at fProcTimeLast - Bool_t fAWLastFill; // Whether to fill the last measurement - Float_t fReportPeriod; // Time between reports if nothing changes (estimated proc time / 100) - - EUseEstOpt fUseEstOpt; // Control usage of estimated values for the progress info - - Bool_t fValid; // Constructed properly? - Bool_t fStop; // Termination of Process() requested? - - TString fDataSet; // Name of the dataset being processed (for dataset-driven runs) - - TList *fInput; // Input list - - TVirtualPacketizer(TList *input, TProofProgressStatus *st = 0); - TVirtualPacketizer(const TVirtualPacketizer &); // no implementation, will generate - void operator=(const TVirtualPacketizer &); // error on accidental usage - - TDSetElement *CreateNewPacket(TDSetElement* base, Long64_t first, Long64_t num); - Long64_t GetEntries(Bool_t tree, TDSetElement *e); // Num of entries or objects - Bool_t HandleTimer(TTimer *timer) override; - -public: - enum EStatusBits { kIsInitializing = BIT(16), kIsDone = BIT(17), kIsTree = BIT(18) }; - ~TVirtualPacketizer() override; - - virtual Int_t AssignWork(TDSet* /*dset*/, Long64_t /*first*/, Long64_t /*num*/) { return -1; } - Bool_t IsValid() const { return fValid; } - Long64_t GetEntriesProcessed() const { return (fProgressStatus? fProgressStatus->GetEntries() : 0); } - virtual Int_t GetEstEntriesProcessed(Float_t, Long64_t &ent, Long64_t &bytes, Long64_t &calls) - { ent = GetEntriesProcessed(); bytes = GetBytesRead(); calls = GetReadCalls(); return 0; } - virtual Float_t GetCurrentRate(Bool_t &all) { all = kTRUE; return (fProgressStatus? fProgressStatus->GetCurrentRate() : 0.); } - Long64_t GetTotalEntries() const { return fTotalEntries; } - virtual TDSetElement *GetNextPacket(TSlave *sl, TMessage *r); - virtual void SetInitTime(); - virtual void StopProcess(Bool_t abort, Bool_t stoptimer = kFALSE); - TList *GetFailedPackets() { return fFailedPackets; } - void SetFailedPackets(TList *list) { fFailedPackets = list; } - virtual Int_t AddWorkers(TList *workers); - - Long64_t GetBytesRead() const { return (fProgressStatus? fProgressStatus->GetBytesRead() : 0); } - Long64_t GetReadCalls() const { return (fProgressStatus? fProgressStatus->GetReadCalls() : 0); } - Double_t GetCumProcTime() const { return fProgressStatus->GetProcTime(); } - Float_t GetInitTime() const { return fInitTime; } - Float_t GetProcTime() const { return fProcTime; } - TNtuple *GetProgressPerf(Bool_t steal = kFALSE) { if (steal) { TNtuple *n = fProgressPerf; fProgressPerf = 0; return n; - } else { return fProgressPerf;} } - TList *GetConfigParams(Bool_t steal = kFALSE) { if (steal) { TList *l = fConfigParams; fConfigParams = 0; return l; - } else { return fConfigParams;} } - virtual void MarkBad(TSlave * /*s*/, TProofProgressStatus * /*status*/, TList ** /*missingFiles*/) { return; } - virtual Int_t AddProcessed(TSlave * /*sl*/, TProofProgressStatus * /*st*/, - Double_t /*lat*/, TList ** /*missingFiles*/) { return 0; } - TProofProgressStatus *GetStatus() { return fProgressStatus; } - void SetProgressStatus(TProofProgressStatus *st) { fProgressStatus = st; } - void SetTotalEntries(Long64_t ent) { fTotalEntries = ent; } - - TMap *GetSlaveStats() const { return fSlaveStats; } - - virtual Int_t GetActiveWorkers() { return -1; } - - ClassDefOverride(TVirtualPacketizer,0) //Generate work packets for parallel processing -}; - -//------------------------------------------------------------------------------ - -class TVirtualPacketizer::TVirtualSlaveStat : public TObject { - -friend class TPacketizerAdaptive; -friend class TPacketizer; - -protected: - TString fWrkFQDN; // Worker FQDN - TSlave *fSlave; // corresponding TSlave record - TProofProgressStatus *fStatus; // status as of the last finished packet - -public: - const char *GetName() const override { return fWrkFQDN.Data(); } - const char *GetOrdinal() const { return fSlave->GetOrdinal(); } - Long64_t GetEntriesProcessed() const { return fStatus?fStatus->GetEntries():-1; } - Double_t GetProcTime() const { return fStatus?fStatus->GetProcTime():-1; } - Float_t GetAvgRate() { return fStatus->GetRate(); } - TProofProgressStatus *GetProgressStatus() { return fStatus; } - virtual TProofProgressStatus *AddProcessed(TProofProgressStatus *st) = 0; -}; - -#endif diff --git a/proof/proof/inc/TVirtualProofPlayer.h b/proof/proof/inc/TVirtualProofPlayer.h deleted file mode 100644 index 153a3e5375a49..0000000000000 --- a/proof/proof/inc/TVirtualProofPlayer.h +++ /dev/null @@ -1,145 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 15/03/07 - -/************************************************************************* - * Copyright (C) 1995-2007, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TVirtualProofPlayer -#define ROOT_TVirtualProofPlayer - - -////////////////////////////////////////////////////////////////////////// -// // -// TVirtualProofPlayer // -// // -// Abstract interface for the PROOF player. // -// See the concrete implementations under 'proofplayer' for details. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TQObject.h" - -class TDSet; -class TDSetElement; -class TEventList; -class TQueryResult; -class TDrawFeedback; -class TList; -class TSlave; -class TMessage; -class TProof; -class TSocket; -class TVirtualPacketizer; -class TProofProgressStatus; -class TProofProgressInfo; -class TSelector; - -class TVirtualProofPlayer : public TObject, public TQObject { - -public: - enum EStatusBits { kIsSubmerger = BIT(14) }; - // TDSet status bits - enum EExitStatus { kFinished, kStopped, kAborted }; - - TVirtualProofPlayer() { ResetBit(TVirtualProofPlayer::kIsSubmerger); } - ~TVirtualProofPlayer() override { } - - virtual Long64_t Process(TDSet *set, - const char *selector, Option_t *option = "", - Long64_t nentries = -1, Long64_t firstentry = 0) = 0; - virtual Long64_t Process(TDSet *set, - TSelector *selector, Option_t *option = "", - Long64_t nentries = -1, Long64_t firstentry = 0) = 0; - virtual Bool_t JoinProcess(TList *workers) = 0; - virtual Long64_t Finalize(Bool_t force = kFALSE, Bool_t sync = kFALSE) = 0; - virtual Long64_t Finalize(TQueryResult *qr) = 0; - virtual Long64_t DrawSelect(TDSet *set, const char *varexp, - const char *selection, Option_t *option = "", - Long64_t nentries = -1, Long64_t firstentry = 0) = 0; - virtual Int_t GetDrawArgs(const char *var, const char *sel, Option_t *opt, - TString &selector, TString &objname) = 0; - virtual void HandleGetTreeHeader(TMessage *mess) = 0; - virtual void HandleRecvHisto(TMessage *mess) = 0; - - virtual void StopProcess(Bool_t abort, Int_t timeout = -1) = 0; - virtual void AddInput(TObject *inp) = 0; - virtual void ClearInput() = 0; - virtual TObject *GetOutput(const char *name) const = 0; - virtual TList *GetOutputList() const = 0; - virtual TList *GetInputList() const = 0; - virtual TList *GetListOfResults() const = 0; - virtual void AddQueryResult(TQueryResult *q) = 0; - virtual TQueryResult *GetCurrentQuery() const = 0; - virtual TQueryResult *GetQueryResult(const char *ref) = 0; - virtual void RemoveQueryResult(const char *ref) = 0; - virtual void SetCurrentQuery(TQueryResult *q) = 0; - virtual void SetMaxDrawQueries(Int_t max) = 0; - virtual void RestorePreviousQuery() =0 ; - virtual Int_t AddOutputObject(TObject *obj) = 0; - virtual void AddOutput(TList *out) = 0; // Incorporate a list - virtual void StoreOutput(TList *out) = 0; // Adopts the list - virtual void StoreFeedback(TObject *slave, TList *out) = 0; // Adopts the list - virtual void Progress(Long64_t total, Long64_t processed) = 0; // *SIGNAL* - virtual void Progress(TSlave *, Long64_t total, Long64_t processed) = 0; - virtual void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) = 0; // *SIGNAL* - virtual void Progress(TSlave *, Long64_t total, Long64_t processed, - Long64_t bytesread, Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) = 0; // *SIGNAL* - virtual void Progress(TProofProgressInfo *) = 0; // *SIGNAL* - virtual void Progress(TSlave *, TProofProgressInfo *) = 0; // *SIGNAL* - virtual void Feedback(TList *objs) = 0; // *SIGNAL* - - virtual TDrawFeedback *CreateDrawFeedback(TProof *p) = 0; - virtual void SetDrawFeedbackOption(TDrawFeedback *f, Option_t *opt) = 0; - virtual void DeleteDrawFeedback(TDrawFeedback *f) = 0; - - virtual TDSetElement *GetNextPacket(TSlave *slave, TMessage *r) = 0; - - virtual Int_t ReinitSelector(TQueryResult *qr) = 0; - - virtual void UpdateAutoBin(const char *name, - Double_t& xmin, Double_t& xmax, - Double_t& ymin, Double_t& ymax, - Double_t& zmin, Double_t& zmax) = 0; - - virtual void MergeOutput(Bool_t = kFALSE) = 0; - - virtual Bool_t IsClient() const = 0; - - virtual EExitStatus GetExitStatus() const = 0; - virtual void SetExitStatus(EExitStatus) = 0; - virtual Long64_t GetEventsProcessed() const = 0; - virtual void AddEventsProcessed(Long64_t ev) = 0; - virtual TProofProgressStatus* GetProgressStatus() const = 0; - - virtual void SetDispatchTimer(Bool_t on = kTRUE) = 0; - virtual void SetStopTimer(Bool_t on = kTRUE, - Bool_t abort = kFALSE, Int_t timeout = 0) = 0; - virtual void SetInitTime() = 0; - - virtual void SetMerging(Bool_t on = kTRUE) = 0; - - virtual Long64_t GetCacheSize() = 0; - virtual Int_t GetLearnEntries() = 0; - - virtual void UpdateProgressInfo() = 0; - - virtual TVirtualPacketizer *GetPacketizer() const { return 0; } - - virtual void SetOutputFilePath(const char *fp) = 0; - virtual Int_t SavePartialResults(Bool_t queryend = kFALSE, Bool_t force = kFALSE) = 0; - - static TVirtualProofPlayer *Create(const char *player, TProof *p, TSocket *s = 0); - - ClassDefOverride(TVirtualProofPlayer,0) // Abstract PROOF player -}; - -#endif diff --git a/proof/proof/src/TCondor.cxx b/proof/proof/src/TCondor.cxx deleted file mode 100644 index e2c28a9a9a991..0000000000000 --- a/proof/proof/src/TCondor.cxx +++ /dev/null @@ -1,485 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Maarten Ballintijn 06/12/03 - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -////////////////////////////////////////////////////////////////////////// -// // -// TCondor // -// // -// Interface to the Condor system. TCondor provides a (partial) API for // -// querying and controlling the Condor system, including experimental // -// extensions like COD (computing on demand) // -// // -////////////////////////////////////////////////////////////////////////// - -#include - -#include "TCondor.h" -#include "TList.h" -#include "TSystem.h" -#include "TObjString.h" -#include "TRegexp.h" -#include "TProofDebug.h" -#include "Riostream.h" -#include "TEnv.h" -#include "TClass.h" - -ClassImp(TCondorSlave); -ClassImp(TCondor); - - -//////////////////////////////////////////////////////////////////////////////// -/// Create Condor interface object. Uses Condor apps since there is no -/// API yet. - -TCondor::TCondor(const char *pool) : fPool(pool), fState(kFree) -{ - fClaims = new TList; - - // Setup Condor - - TString condorHome = gEnv->GetValue("Proof.CondorHome", (char*)0); - if (condorHome != "") { - TString path = gSystem->Getenv("PATH"); - path = condorHome + "/bin:" + path; - gSystem->Setenv("PATH",path); - } - - TString condorConf = gEnv->GetValue("Proof.CondorConfig", (char*)0); - if (condorConf != "") { - gSystem->Setenv("CONDOR_CONFIG",condorConf); - } - - char *loc = gSystem->Which(gSystem->Getenv("PATH"), "condor_cod", - kExecutePermission); - - if (loc) { - fValid = kTRUE; - delete [] loc; - } else { - fValid = kFALSE; - } -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup Condor interface. - -TCondor::~TCondor() -{ - PDB(kCondor,1) Info("~TCondor","fState %d", fState ); - - if (fState != kFree) { - Release(); - } - delete fClaims; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Print master status - -void TCondor::Print(Option_t * opt) const -{ - std::cout << "OBJ: " << IsA()->GetName() - << "\tPool: \"" << fPool << "\"" - << "\tState: " << fState << std::endl; - fClaims->Print(opt); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Claim a VirtualMachine for PROOF usage. - -TCondorSlave *TCondor::ClaimVM(const char *vm, const char *cmd) -{ - Int_t port = 0; - - TString claimCmd = Form("condor_cod request -name %s -timeout 10 2>>%s/condor.proof.%d", - vm, gSystem->TempDirectory(), gSystem->GetUid() ); - - PDB(kCondor,2) Info("ClaimVM","command: %s", claimCmd.Data()); - FILE *pipe = gSystem->OpenPipe(claimCmd, "r"); - - if (!pipe) { - SysError("ClaimVM","cannot run command: %s", claimCmd.Data()); - return 0; - } - - TString claimId; - TString line; - while (line.Gets(pipe)) { - PDB(kCondor,3) Info("ClaimVM","line = %s", line.Data()); - - if (line.BeginsWith("ClaimId = \"")) { - line.Remove(0, line.Index("\"")+1); - line.Chop(); // remove trailing " - claimId = line; - PDB(kCondor,1) Info("ClaimVM","claim = '%s'", claimId.Data()); - TRegexp r("[0-9]*$"); - TString num = line(r); - port = 37000 + atoi(num.Data()); - PDB(kCondor,1) Info("ClaimVM","port = %d", port); - } - } - - Int_t r = gSystem->ClosePipe(pipe); - if (r) { - Error("ClaimVM","command: %s returned %d", claimCmd.Data(), r); - return 0; - } else { - PDB(kCondor,1) Info("ClaimVM","command: %s returned %d", claimCmd.Data(), r); - } - - TString jobad("jobad"); - FILE *jf = gSystem->TempFileName(jobad); - - if (jf == 0) return 0; - - TString str(cmd); - str.ReplaceAll("$(Port)", Form("%d", port)); - fputs(str, jf); - - fclose(jf); - - TString activateCmd = Form("condor_cod activate -id '%s' -jobad %s", - claimId.Data(), jobad.Data() ); - - PDB(kCondor,2) Info("ClaimVM","command: %s", activateCmd.Data()); - pipe = gSystem->OpenPipe(activateCmd, "r"); - - if (!pipe) { - SysError("ClaimVM","cannot run command: %s", activateCmd.Data()); - return 0; - } - - while (line.Gets(pipe)) { - PDB(kCondor,3) Info("ClaimVM","Activate: line = %s", line.Data()); - } - - r = gSystem->ClosePipe(pipe); - if (r) { - Error("ClaimVM","command: %s returned %d", activateCmd.Data(), r); - } else { - PDB(kCondor,1) Info("ClaimVM","command: %s returned %d", activateCmd.Data(), r); - } - - gSystem->Unlink(jobad); - - // TODO: get info at the start for all nodes ... - TCondorSlave *claim = new TCondorSlave; - claim->fClaimID = claimId; - TString node(vm); - node = node.Remove(0, node.Index("@")+1); - claim->fHostname = node; - claim->fPort = port; - claim->fPerfIdx = 100; //set performance index to 100 by default - claim->fImage = node; //set image to hostname by default - - return claim; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Get the names of the virtual machines in the pool. -/// Return a TList of TObjString or 0 in case of failure - -TList *TCondor::GetVirtualMachines() const -{ - TString poolopt = fPool ? Form("-pool %s", fPool.Data()) : ""; - TString cmd = Form("condor_status %s -format \"%%s\\n\" Name", poolopt.Data()); - - PDB(kCondor,2) Info("GetVirtualMachines","command: %s", cmd.Data()); - - FILE *pipe = gSystem->OpenPipe(cmd, "r"); - - if (!pipe) { - SysError("GetVirtualMachines","cannot run command: %s", cmd.Data()); - return 0; - } - - TString line; - TList *l = new TList; - while (line.Gets(pipe)) { - PDB(kCondor,3) Info("GetVirtualMachines","line = %s", line.Data()); - if (line != "") l->Add(new TObjString(line)); - } - - Int_t r = gSystem->ClosePipe(pipe); - if (r) { - delete l; - Error("GetVirtualMachines","command: %s returned %d", cmd.Data(), r); - return 0; - } else { - PDB(kCondor,1) Info("GetVirtualMachines","command: %s returned %d", cmd.Data(), r); - } - - return l; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Claim n virtual machines -/// This function figures out the image and performance index before returning -/// the list of condor slaves - -TList *TCondor::Claim(Int_t n, const char *cmd) -{ - if (fState != kFree) { - Error("Claim","not in state Free"); - return 0; - } - - TList *vms = GetVirtualMachines(); - TIter next(vms); - TObjString *vm; - for(Int_t i=0; i < n && (vm = (TObjString*) next()) != 0; i++ ) { - TCondorSlave *claim = ClaimVM(vm->GetName(), cmd); - if (claim != 0) { - if ( !GetVmInfo(vm->GetName(), claim->fImage, claim->fPerfIdx) ) { - // assume vm is gone - delete claim; - } else { - fClaims->Add(claim); - fState = kActive; - } - } - } - - vms->Delete(); - delete vms; - - return fClaims; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Claim virtual machine with name vmname -/// This function does not figure out the image and performance index before -/// returning the condor slave - -TCondorSlave *TCondor::Claim(const char *vmname, const char *cmd) -{ - if (fState != kFree && fState != kActive) { - Error("Claim","not in state Free or Active"); - return 0; - } - - TCondorSlave *claim = ClaimVM(vmname, cmd); - if (claim != 0) { - fClaims->Add(claim); - fState = kActive; - } - - return claim; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Set the state of workers - -Bool_t TCondor::SetState(EState state) -{ - PDB(kCondor,1) Info("SetState","state: %s (%lld)", - state == kSuspended ? "kSuspended" : "kActive", Long64_t(gSystem->Now())); - TIter next(fClaims); - TCondorSlave *claim; - while((claim = (TCondorSlave*) next()) != 0) { - TString cmd = Form("condor_cod %s -id '%s'", - state == kSuspended ? "suspend" : "resume", - claim->fClaimID.Data()); - - PDB(kCondor,2) Info("SetState","command: %s", cmd.Data()); - FILE *pipe = gSystem->OpenPipe(cmd, "r"); - - if (!pipe) { - SysError("SetState","cannot run command: %s", cmd.Data()); - return kFALSE; - } - - TString line; - while (line.Gets(pipe)) { - PDB(kCondor,3) Info("SetState","line = %s", line.Data()); - } - - Int_t r = gSystem->ClosePipe(pipe); - if (r) { - Error("SetState","command: %s returned %d", cmd.Data(), r); - return kFALSE; - } else { - PDB(kCondor,1) Info("SetState","command: %s returned %d", cmd.Data(), r); - } - } - - fState = state; - return kTRUE; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Suspend worker - -Bool_t TCondor::Suspend() -{ - if (fState != kActive) { - Error("Suspend","not in state Active"); - return kFALSE; - } - - return SetState(kSuspended); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Resume worker - -Bool_t TCondor::Resume() -{ - if (fState != kSuspended) { - Error("Suspend","not in state Suspended"); - return kFALSE; - } - - return SetState(kActive); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Release worker - -Bool_t TCondor::Release() -{ - if (fState == kFree) { - Error("Suspend","not in state Active or Suspended"); - return kFALSE; - } - - TCondorSlave *claim; - while((claim = (TCondorSlave*) fClaims->First()) != 0) { - TString cmd = Form("condor_cod release -id '%s'", claim->fClaimID.Data()); - - PDB(kCondor,2) Info("SetState","command: %s", cmd.Data()); - FILE *pipe = gSystem->OpenPipe(cmd, "r"); - - if (!pipe) { - SysError("Release","cannot run command: %s", cmd.Data()); - return kFALSE; - } - - TString line; - while (line.Gets(pipe)) { - PDB(kCondor,3) Info("Release","line = %s", line.Data()); - } - - Int_t r = gSystem->ClosePipe(pipe); - if (r) { - Error("Release","command: %s returned %d", cmd.Data(), r); - return kFALSE; - } else { - PDB(kCondor,1) Info("Release","command: %s returned %d", cmd.Data(), r); - } - - fClaims->Remove(claim); - delete claim; - } - - fState = kFree; - return kTRUE; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Get info about worker status - -Bool_t TCondor::GetVmInfo(const char *vm, TString &image, Int_t &perfidx) const -{ - TString cmd = Form("condor_status -format \"%%d:\" Mips -format \"%%s\\n\" FileSystemDomain " - "-const 'Name==\"%s\"'", vm); - - PDB(kCondor,2) Info("GetVmInfo","command: %s", cmd.Data()); - FILE *pipe = gSystem->OpenPipe(cmd, "r"); - - if (!pipe) { - SysError("GetVmInfo","cannot run command: %s", cmd.Data()); - return kFALSE; - } - - TString line; - while (line.Gets(pipe)) { - PDB(kCondor,3) Info("GetVmInfo","line = %s", line.Data()); - if (line != "") { - TString amips = line(TRegexp("^[0-9]*")); - perfidx = atoi(amips); - image = line(TRegexp("[^:]+$")); - break; - } - } - - Int_t r = gSystem->ClosePipe(pipe); - if (r) { - Error("GetVmInfo","command: %s returned %d", cmd.Data(), r); - return kFALSE; - } else { - PDB(kCondor,1) Info("GetVmInfo","command: %s returned %d", cmd.Data(), r); - } - - return kTRUE; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Get image of the worker - -TString TCondor::GetImage(const char *host) const -{ - TString cmd = Form("condor_status -direct %s -format \"Image:%%s\\n\" " - "FileSystemDomain", host); - - PDB(kCondor,2) Info("GetImage","command: %s", cmd.Data()); - - FILE *pipe = gSystem->OpenPipe(cmd, "r"); - - if (!pipe) { - SysError("GetImage","cannot run command: %s", cmd.Data()); - return ""; - } - - TString image; - TString line; - while (line.Gets(pipe)) { - PDB(kCondor,3) Info("GetImage","line = %s", line.Data()); - if (line != "") { - image = line(TRegexp("[^:]+$")); - break; - } - } - - Int_t r = gSystem->ClosePipe(pipe); - if (r) { - Error("GetImage","command: %s returned %d", cmd.Data(), r); - return ""; - } else { - PDB(kCondor,1) Info("GetImage","command: %s returned %d", cmd.Data(), r); - } - - return image; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Print worker status - -void TCondorSlave::Print(Option_t * /*opt*/ ) const -{ - std::cout << "OBJ: " << IsA()->GetName() - << " " << fHostname << ":" << fPort - << " Perf: " << fPerfIdx - << " Image: " << fImage << std::endl; -} diff --git a/proof/proof/src/TDSet.cxx b/proof/proof/src/TDSet.cxx deleted file mode 100644 index 75213c1ccf588..0000000000000 --- a/proof/proof/src/TDSet.cxx +++ /dev/null @@ -1,2034 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 11/01/02 - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TDSetElement -\ingroup proofkernel -Manages an element of a TDSet. - -See TDSet. -*/ - -#include "TDSet.h" - -#include "Riostream.h" -#include "TBuffer.h" -#include "TChain.h" -#include "TClass.h" -#include "TClassTable.h" -#include "TCut.h" -#include "TDataSetManager.h" -#include "TError.h" -#include "TEntryList.h" -#include "TEnv.h" -#include "TEventList.h" -#include "TFile.h" -#include "TFileInfo.h" -#include "TFileStager.h" -#include "TFriendElement.h" -#include "TKey.h" -#include "THashList.h" -#include "TMap.h" -#include "TROOT.h" -#include "TTimeStamp.h" -#include "TTree.h" -#include "TUrl.h" -#include "TRegexp.h" -#include "TVirtualPerfStats.h" -#include "TProof.h" -#include "TProofChain.h" -#include "TProofServ.h" -#include "TPluginManager.h" -#include "TChainElement.h" -#include "TSystem.h" -#include "TSelector.h" -#include "TObjString.h" - -#include "TVirtualStreamerInfo.h" -#include "TClassRef.h" - -ClassImp(TDSetElement); -ClassImp(TDSet); - -//////////////////////////////////////////////////////////////////////////////// -/// Default constructor - -TDSetElement::TDSetElement() : TNamed("",""), - fDirectory(), fFirst(0), fNum(0), fMsd(), - fTDSetOffset(0), fEntryList(0), fValid(kFALSE), - fEntries(0), fFriends(0), fDataSet(), fAssocObjList(0), - fMaxProcTime(-1) -{ - ResetBit(kWriteV3); - ResetBit(kHasBeenLookedUp); - ResetBit(kEmpty); - ResetBit(kCorrupted); - ResetBit(kNewRun); - ResetBit(kNewPacket); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a TDSet element. - -TDSetElement::TDSetElement(const char *file, const char *objname, const char *dir, - Long64_t first, Long64_t num, - const char *msd, const char *dataset) - : TNamed(file, objname) -{ - if (first < 0) { - Warning("TDSetElement", "first must be >= 0, %lld is not allowed - setting to 0", first); - fFirst = 0; - } else { - fFirst = first; - } - if (num < -1) { - Warning("TDSetElement", "num must be >= -1, %lld is not allowed - setting to -1", num); - fNum = -1; - } else { - fNum = num; - } - fMsd = msd; - fTDSetOffset = 0; - fEntryList = 0; - fFriends = 0; - fValid = kFALSE; - fEntries = -1; - fDataSet = dataset; - fAssocObjList = 0; - if (dir) - fDirectory = dir; - fMaxProcTime = -1.; - - ResetBit(kWriteV3); - ResetBit(kHasBeenLookedUp); - ResetBit(kEmpty); - ResetBit(kCorrupted); - ResetBit(kNewRun); - ResetBit(kNewPacket); -} - -//////////////////////////////////////////////////////////////////////////////// -/// copy constructor - -TDSetElement::TDSetElement(const TDSetElement& elem) - : TNamed(elem.GetFileName(), elem.GetObjName()) -{ - fDirectory = elem.GetDirectory(); - fFirst = elem.fFirst; - fNum = elem.fNum; - fMsd = elem.fMsd; - fTDSetOffset = elem.fTDSetOffset; - fEntryList = 0; - fValid = elem.fValid; - fEntries = elem.fEntries; - fFriends = 0; - fDataSet = elem.fDataSet; - fAssocObjList = 0; - fMaxProcTime = elem.fMaxProcTime; - ResetBit(kWriteV3); - ResetBit(kHasBeenLookedUp); - ResetBit(kEmpty); - ResetBit(kCorrupted); - ResetBit(kNewRun); - ResetBit(kNewPacket); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clean up the element. - -TDSetElement::~TDSetElement() -{ - DeleteFriends(); - if (fAssocObjList) { - fAssocObjList->SetOwner(kTRUE); - SafeDelete(fAssocObjList); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset TDSet element. - -void TDSetElement::Reset() -{ - fFirst = 0; - fNum = -1; - fTDSetOffset = 0; - fEntryList = 0; - fValid = kFALSE; - fEntries = -1; - fMaxProcTime = -1.; - - ResetBit(kHasBeenLookedUp); - ResetBit(kEmpty); - ResetBit(kNewRun); - ResetBit(kNewPacket); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check if 'elem' is overlapping or subsequent and, if the case, return -/// a merged element. -/// Returns: -/// 1 if the elements are overlapping -/// 0 if the elements are subsequent -/// -1 if the elements are neither overlapping nor subsequent - -Int_t TDSetElement::MergeElement(TDSetElement *elem) -{ - // The element must be defined - if (!elem) return -1; - - // The file names and object names must be the same - if (strcmp(GetName(), elem->GetName()) || strcmp(GetTitle(), elem->GetTitle())) - return -1; - - Int_t rc = -1; - // Check the overlap or subsequency - if (fFirst == 0 && fNum == -1) { - // Overlap, since we cover already the full range - rc = 1; - } else if (elem->GetFirst() == 0 && elem->GetNum() == -1) { - // Overlap, since 'elem' cover already the full range - fFirst = 0; - fNum = -1; - fEntries = elem->GetEntries(); - rc = 1; - } else if (fFirst >= 0 && fNum > 0 && elem->GetFirst() >= 0 && elem->GetNum() > 0) { - Long64_t last = fFirst + fNum - 1, lastref = 0; - Long64_t lastelem = elem->GetFirst() + elem->GetNum() - 1; - if (elem->GetFirst() == last + 1) { - lastref = lastelem; - rc = 0; - } else if (fFirst == lastelem + 1) { - fFirst += elem->GetFirst(); - lastref = last; - rc = 0; - } else if (elem->GetFirst() < last + 1 && elem->GetFirst() >= fFirst) { - lastref = (lastelem > last) ? lastelem : last; - rc = 1; - } else if (fFirst < lastelem + 1 && fFirst >= elem->GetFirst()) { - fFirst += elem->GetFirst(); - lastref = (lastelem > last) ? lastelem : last; - rc = 1; - } - fNum = lastref - fFirst + 1; - } - if (rc >= 0 && fEntries < 0 && elem->GetEntries() > 0) fEntries = elem->GetEntries(); - - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the content of this element in the form of a TFileInfo - -TFileInfo *TDSetElement::GetFileInfo(const char *type) -{ - // Create the TFileInfoMeta object - TFileInfoMeta *meta = 0; - Long64_t entries = (fEntries < 0 && fNum > 0) ? fNum : fEntries; - Printf("entries: %lld (%lld)", entries, fNum); - if (!strcmp(type, "TTree")) { - meta = new TFileInfoMeta(GetTitle(), "TTree", entries, fFirst, - fFirst + entries - 1); - } else { - meta = new TFileInfoMeta(GetTitle(), fDirectory, type, entries, fFirst, - fFirst + entries - 1); - } - TFileInfo *fi = new TFileInfo(GetName(), 0, 0, 0, meta); - if (!fDataSet.IsNull()) fi->SetTitle(fDataSet.Data()); - if (TestBit(TDSetElement::kCorrupted)) fi->SetBit(TFileInfo::kCorrupted); - return fi; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return directory where to look for object. - -const char *TDSetElement::GetDirectory() const -{ - return fDirectory; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print a TDSetElement. When option="a" print full data. - -void TDSetElement::Print(Option_t *opt) const -{ - if (opt && opt[0] == 'a') { - Printf("%s file=\"%s\" dir=\"%s\" obj=\"%s\" first=%lld num=%lld msd=\"%s\"", - IsA()->GetName(), GetName(), fDirectory.Data(), GetTitle(), - fFirst, fNum, fMsd.Data()); - } else { - Printf("\tLFN: %s", GetName()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Validate by opening the file. - -void TDSetElement::Validate(Bool_t isTree) -{ - Long64_t entries = GetEntries(isTree); - if (entries < 0) return; // Error should be reported by GetEntries() - if (fFirst < entries) { - if (fNum == -1) { - fNum = entries - fFirst; - fValid = kTRUE; - } else { - if (fNum <= entries - fFirst) { - fValid = kTRUE; - } else { - Error("Validate", "TDSetElement has only %lld entries starting" - " with entry %lld, while %lld were requested", - entries - fFirst, fFirst, fNum); - } - } - } else { - Error("Validate", "TDSetElement has only %lld entries with" - " first entry requested as %lld", entries, fFirst); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Validate by checking against another element. - -void TDSetElement::Validate(TDSetElement *elem) -{ - // NOTE: Since this function validates against another TDSetElement, - // if the other TDSetElement (elem) did not use -1 to request all - // entries, this TDSetElement may get less than all entries if it - // requests all (with -1). For the application it was developed for - // (TProofSuperMaster::ValidateDSet) it works, since the design was - // to send the elements to their mass storage domain and let them - // look at the file and send the info back to the supermaster. The - // ability to set fValid was also required to be only exist in - // TDSetElement through certain function and not be set externally. - // TDSetElement may need to be extended for more general applications. - - if (!elem || !elem->GetValid()) { - Error("Validate", "TDSetElement to validate against is not valid"); - return; - } - - TString name = TUrl(GetFileName()).GetFileAndOptions(); - TString elemname = TUrl(elem->GetFileName()).GetFileAndOptions(); - if ((name == elemname) && - !strcmp(GetDirectory(), elem->GetDirectory()) && - !strcmp(GetObjName(), elem->GetObjName())) { - Long64_t entries = elem->fFirst + elem->fNum; - if (fFirst < entries) { - if (fNum == -1) { - fNum = entries - fFirst; - fValid = kTRUE; - } else { - if (fNum <= entries - fFirst) { - fValid = kTRUE; - } else { - Error("Validate", "TDSetElement requests %lld entries starting" - " with entry %lld, while TDSetElement to validate against" - " has only %lld entries", fNum, fFirst, entries); - } - } - } else { - Error("Validate", "TDSetElement to validate against has only %lld" - " entries, but this TDSetElement requested %lld as its first" - " entry", entries, fFirst); - } - } else { - Error("Validate", "TDSetElements do not refer to same objects"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -///Compare elements by filename (and the fFirst). - -Int_t TDSetElement::Compare(const TObject *obj) const -{ - if (this == obj) return 0; - - const TDSetElement *elem = dynamic_cast(obj); - if (!elem) { - if (obj) - return (strncmp(GetName(),obj->GetName(),strlen(GetName()))) ? 1 : 0; - return -1; - } - - Int_t order = strncmp(GetName(),elem->GetFileName(),strlen(GetName())); - if (order == 0) { - if (GetFirst() < elem->GetFirst()) - return -1; - else if (GetFirst() > elem->GetFirst()) - return 1; - return 0; - } - return order; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add friend TDSetElement to this set. The friend element will be copied to this object. - -void TDSetElement::AddFriend(TDSetElement *friendElement, const char *alias) -{ - if (!friendElement) { - Error("AddFriend", "The friend TDSetElement is null!"); - return; - } - if (!fFriends) { - fFriends = new TList(); - fFriends->SetOwner(); - } - // Add alias (if any) as option 'friend_alias=|' - if (alias && strlen(alias) > 0) { - TUrl uf(friendElement->GetName()); - TString uo(uf.GetOptions()); - uo += TString::Format("friend_alias=%s|", alias); - uf.SetOptions(uo); - friendElement->SetName(uf.GetUrl()); - } - fFriends->Add(new TDSetElement(*friendElement)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Deletes the list of friends and all the friends on the list. - -void TDSetElement::DeleteFriends() -{ - if (!fFriends) - return; - - fFriends->SetOwner(kTRUE); - delete fFriends; - fFriends = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns next TDSetElement. - -TDSetElement *TDSet::Next(Long64_t /*totalEntries*/) -{ - if (!fIterator) { - fIterator = new TIter(fElements); - } - - fCurrent = (TDSetElement *) fIterator->Next(); - return fCurrent; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns number of entries in tree or objects in file. -/// If not yet defined and 'openfile' is TRUE, get the number from the file -/// (may considerably slow down the application). -/// Returns -1 in case of error. - -Long64_t TDSetElement::GetEntries(Bool_t isTree, Bool_t openfile) -{ - if (fEntries > -1 || !openfile) - return fEntries; - - Double_t start = 0; - if (gPerfStats) start = TTimeStamp(); - - // Take into account possible prefixes - TFile::EFileType typ = TFile::kDefault; - TString fname = gEnv->GetValue("Path.Localroot",""), pfx(fname); - // Get the locality (disable warnings or errors in connection attempts) - Int_t oldLevel = gErrorIgnoreLevel; - gErrorIgnoreLevel = kError+1; - if ((typ = TFile::GetType(GetName(), "", &fname)) != TFile::kLocal) fname = GetName(); - gErrorIgnoreLevel = oldLevel; - // Open the file - TFile *file = TFile::Open(fname); - - if (gPerfStats) - gPerfStats->FileOpenEvent(file, GetName(), start); - - if (file == 0) { - ::SysError("TDSetElement::GetEntries", - "cannot open file %s (type: %d, pfx: %s)", GetName(), typ, pfx.Data()); - return -1; - } - - // Record end-point Url and mark as looked-up; be careful to change - // nothing in the file name, otherwise some cross-checks may fail. - // The lookup is only performed if not yet done - if (Lookup(kFALSE) != 0) Warning("GetEntries", "lookup problems for %s", GetName()); - - TDirectory *dirsave = gDirectory; - if (!file->cd(fDirectory)) { - Error("GetEntries", "cannot cd to %s", fDirectory.Data()); - delete file; - return -1; - } - - TDirectory *dir = gDirectory; - dirsave->cd(); - - if (isTree) { - - TString on(GetTitle()); - TString sreg(GetTitle()); - // If a wild card we will use the first object of the type - // requested compatible with the reg expression we got - if (sreg.Length() <= 0 || sreg == "" || sreg.Contains("*")) { - if (sreg.Contains("*")) - sreg.ReplaceAll("*", ".*"); - else - sreg = ".*"; - TRegexp re(sreg); - if (dir->GetListOfKeys()) { - TIter nxk(dir->GetListOfKeys()); - TKey *k = 0; - Bool_t notfound = kTRUE; - while ((k = (TKey *) nxk())) { - if (!strcmp(k->GetClassName(), "TTree")) { - TString kn(k->GetName()); - if (kn.Index(re) != kNPOS) { - if (notfound) { - on = kn; - notfound = kFALSE; - } else if (kn != on) { - Warning("GetEntries", - "additional tree found in the file: %s", kn.Data()); - } - } - } - } - } - } - - TKey *key = dir->GetKey(on); - if (key == 0) { - Error("GetEntries", "cannot find tree \"%s\" in %s", - GetTitle(), GetName()); - delete file; - return -1; - } - TTree *tree = (TTree *) key->ReadObj(); - if (tree == 0) { - // Error always reported? - delete file; - return -1; - } - fEntries = tree->GetEntries(); - delete tree; - - } else { - TList *keys = dir->GetListOfKeys(); - fEntries = keys->GetSize(); - } - - delete file; - return fEntries; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Resolve end-point URL for this element -/// Return 0 on success and -1 otherwise - -Int_t TDSetElement::Lookup(Bool_t force) -{ - static Int_t xNetPluginOK = -1; - static TFileStager *xStager = 0; - Int_t retVal = 0; - - // Check if required - if (!force && HasBeenLookedUp()) - return retVal; - - TUrl url(GetName()); - // Save current options and anchor to be set ofthe final end URL - TString anch = url.GetAnchor(); - TString opts = url.GetOptions(); - // The full path - TString name(url.GetUrl()); - - // Depending on the type of backend, it might not make any sense to lookup - Bool_t doit = kFALSE; - TFile::EFileType type = TFile::GetType(name, ""); - if (type == TFile::kNet) { - TPluginHandler *h = 0; - // Network files via XROOTD - if (xNetPluginOK == -1) { - // Check the plugin the first time - xNetPluginOK = 0; - if ((h = gROOT->GetPluginManager()->FindHandler("TFile", name)) && - !strcmp(h->GetClass(),"TNetXNGFile") && h->LoadPlugin() == 0) - xNetPluginOK = 1; - } - doit = (xNetPluginOK == 1) ? kTRUE : kFALSE; - } - - // Locate the file - if (doit) { - if (!xStager || !xStager->Matches(name)) { - SafeDelete(xStager); - if (!(xStager = TFileStager::Open(name))) { - Error("Lookup", "TFileStager instance cannot be instantiated"); - retVal = -1; - } - } - if (xStager && xStager->Locate(name.Data(), name) == 0) { - // Get the effective end-point Url - url.SetUrl(name); - // Restore original options and anchor, if any - url.SetOptions(opts); - url.SetAnchor(anch); - // Save it into the element - fName = url.GetUrl(); - } else { - // Failure - Error("Lookup", "couldn't lookup %s", name.Data()); - retVal = -1; - } - } - - // Mark has looked-up - SetBit(kHasBeenLookedUp); - return retVal; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set entry (or event) list for this element - -void TDSetElement::SetEntryList(TObject *aList, Long64_t first, Long64_t num) -{ - if (!aList) { - // Nothing to do, except making sure to disable any previous setting - fEntryList = 0; - return; - } - - // Link the proper object - TEventList *evl = 0; - TEntryList *enl = dynamic_cast(aList); - if (!enl) - evl = dynamic_cast(aList); - if (!enl && !evl) { - Error("SetEntryList", "type of input object must be either TEntryList " - "or TEventList (found: '%s' - do nothing", aList->ClassName()); - return; - } - - // Action depends on the type - if (enl) { - enl->SetEntriesToProcess(num); - } else { - for (; num > 0; num--, first++) - evl->Enter(evl->GetEntry((Int_t)first)); - } - fEntryList = aList; - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add an associated object to the list - -void TDSetElement::AddAssocObj(TObject *assocobj) -{ - if (assocobj) { - if (!fAssocObjList) fAssocObjList = new TList; - if (fAssocObjList) fAssocObjList->Add(assocobj); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get i-th associated object. -/// If 'isentry' fFirst is subtracted, so that i == fFirst returns the first -/// object in the list. -/// If there are not enough elements in the list, the element i%list_size is -/// returned (if the list has only one element this only one element is always -/// returned. -/// This method is used when packet processing consist in processing the objects -/// in the associated object list. - -TObject *TDSetElement::GetAssocObj(Long64_t i, Bool_t isentry) -{ - TObject *o = 0; - if (!fAssocObjList || fAssocObjList->GetSize() <= 0) return o; - - TString s; - Int_t pos = -1; - if (isentry) { - if (i < fFirst) return o; - s.Form("%lld", i - fFirst); - } else { - if (i < 0) return o; - s.Form("%lld", i); - } - if (!(s.IsDigit())) return o; - pos = s.Atoi(); - if (pos > fAssocObjList->GetSize() - 1) pos %= fAssocObjList->GetSize(); - return fAssocObjList->At(pos); -} - - -/** \class TDSet -\ingroup proofkernel - -This class implements a data set to be used for PROOF processing. -The TDSet defines the class of which objects will be processed, -the directory in the file where the objects of that type can be -found and the list of files to be processed. The files can be -specified as logical file names (LFN's) or as physical file names -(PFN's). In case of LFN's the resolution to PFN's will be done -according to the currently active GRID interface. -Examples: - TDSet treeset("TTree", "AOD"); - treeset.Add("lfn:/alien.cern.ch/alice/prod2002/file1"); - ... - treeset.AddFriend(friendset); - -or - - TDSet objset("MyEvent", "*", "/events"); - objset.Add("root://cms.cern.ch/user/prod2002/hprod_1.root"); - ... - objset.Add(set2003); - -Validity of file names will only be checked at processing time -(typically on the PROOF master server), not at creation time. - -*/ - -//////////////////////////////////////////////////////////////////////////////// -/// Default ctor. - -TDSet::TDSet() -{ - fElements = new THashList; - fElements->SetOwner(); - fIsTree = kFALSE; - fIterator = 0; - fCurrent = 0; - fEntryList = 0; - fProofChain = 0; - fSrvMaps = 0; - fSrvMapsIter = 0; - ResetBit(kWriteV3); - ResetBit(kEmpty); - ResetBit(kValidityChecked); - ResetBit(kSomeInvalid); - ResetBit(kMultiDSet); - - // Add to the global list - gROOT->GetListOfDataSets()->Add(this); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a named TDSet object. The "type" defines the class of which objects -/// will be processed (default 'TTree'). The optional "objname" argument -/// specifies the name of the objects of the specified class. -/// If the "objname" is not given the behaviour depends on the 'type': -/// for 'TTree' the first TTree is analyzed; for other types, all objects of -/// the class found in the specified directory are processed. -/// The "dir" argument specifies in which directory the objects are -/// to be found, the top level directory ("/") is the default. -/// Directories can be specified using wildcards, e.g. "*" or "/*" -/// means to look in all top level directories, "/dir/*" in all -/// directories under "/dir", and "/*/*" to look in all directories -/// two levels deep. -/// For backward compatibility the type can also be passed via 'name', -/// in which case 'type' is ignored. - -TDSet::TDSet(const char *name, - const char *objname, const char *dir, const char *type) -{ - fElements = new THashList; - fElements->SetOwner(); - fIterator = 0; - fCurrent = 0; - fEntryList = 0; - fProofChain = 0; - fSrvMaps = 0; - fSrvMapsIter = 0; - ResetBit(kWriteV3); - ResetBit(kEmpty); - ResetBit(kValidityChecked); - ResetBit(kSomeInvalid); - ResetBit(kMultiDSet); - - fType = "TTree"; - TClass *c = 0; - // Check name - if (name && strlen(name) > 0) { - // In the old constructor signature it was the 'type' - if (!type) { - TString cn(name); - if (cn.Contains(':')) cn.Remove(0, cn.Index(":")+1); - if (TClass::GetClass(cn)) - fType = cn; - else - // Default type is 'TTree' - fName = name; - } else { - // Set name - fName = name; - // Check type - if (strlen(type) > 0) - if (TClass::GetClass(type)) - fType = type; - } - } else if (type && strlen(type) > 0) { - // Check the type - if (TClass::GetClass(type)) - fType = type; - } - // The correct class type - c = TClass::GetClass(fType); - - fIsTree = (c->InheritsFrom(TTree::Class())) ? kTRUE : kFALSE; - - if (objname) - fObjName = objname; - - if (dir) - fDir = dir; - - // Default name is the object name - if (fName.Length() <= 0) - fName = TString::Format("TDSet:%s", fObjName.Data()); - // We set the default title to the 'type' - fTitle = fType; - - // Add to the global list - gROOT->GetListOfDataSets()->Add(this); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a named TDSet object from existing TChain 'chain'. -/// If 'withfriends' is kTRUE add also friends. -/// This constructor substituted the static methods TChain::MakeTDSet -/// removing any residual dependence of 'tree' on 'proof'. - -TDSet::TDSet(const TChain &chain, Bool_t withfriends) -{ - fElements = new THashList; - fElements->SetOwner(); - fIterator = 0; - fCurrent = 0; - fEntryList = 0; - fProofChain = 0; - fSrvMaps = 0; - fSrvMapsIter = 0; - ResetBit(kWriteV3); - ResetBit(kEmpty); - ResetBit(kValidityChecked); - ResetBit(kSomeInvalid); - ResetBit(kMultiDSet); - - fType = "TTree"; - fIsTree = kTRUE; - fObjName = chain.GetName(); - fName = TString::Format("TChain:%s", chain.GetName()); - - // First fill elements without friends() - TIter next(chain.GetListOfFiles()); - TChainElement *elem = 0; - TString key; - while ((elem = (TChainElement *)next())) { - TString file(elem->GetTitle()); - TString tree(elem->GetName()); - Int_t isl = tree.Last('/'); - TString dir = "/"; - if (isl >= 0) { - // Copy the tree name specification - TString behindSlash = tree(isl + 1, tree.Length() - isl - 1); - // and remove it from basename - tree.Remove(isl); - dir = tree; - tree = behindSlash; - } - // Find MSD if any - TString msd(TUrl(file).GetOptions()); - Int_t imsd = kNPOS; - if ((imsd = msd.Index("msd=")) != kNPOS) { - msd.Remove(0, imsd+4); - } else { - // Not an MSD option - msd = ""; - } - Long64_t nent = (elem->GetEntries() > 0 && - elem->GetEntries() != TTree::kMaxEntries) ? elem->GetEntries() : -1; - if (Add(file, tree, dir, 0, nent, ((msd.IsNull()) ? 0 : msd.Data()))) { - if (elem->HasBeenLookedUp()) { - // Save lookup information, if any - TDSetElement *dse = (TDSetElement *) fElements->Last(); - if (dse) dse->SetLookedUp(); - } - } - } - SetDirectory(0); - - // Add friends now, if requested - if (withfriends) { - TList processed; - TList chainsQueue; - chainsQueue.Add((TObject *)&chain); - processed.Add((TObject *)&chain); - while (chainsQueue.GetSize() > 0) { - TChain *c = (TChain *) chainsQueue.First(); - chainsQueue.Remove(c); - TIter friendsIter(c->GetListOfFriends()); - while(TFriendElement *fe = dynamic_cast (friendsIter()) ) { - if (TChain *fc = dynamic_cast(fe->GetTree())) { - if (!processed.FindObject(fc)) { // if not yet processed - processed.AddFirst(fc); - AddFriend(new TDSet((const TChain &)(*fc), kFALSE), fe->GetName()); - chainsQueue.Add(fc); // for further processing - } - } else { - Reset(); - Error("TDSet", "Only TChains supported. Found illegal tree %s", - fe->GetTree()->GetName()); - return; - } - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup. - -TDSet::~TDSet() -{ - SafeDelete(fElements); - SafeDelete(fIterator); - SafeDelete(fProofChain); - fSrvMaps = 0; - fSrvMapsIter = 0; - - gROOT->GetListOfDataSets()->Remove(this); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process TDSet on currently active PROOF session. -/// The last argument 'enl' specifies an entry- or event-list to be used as -/// event selection. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TDSet::Process(TSelector *selector, Option_t *option, Long64_t nentries, - Long64_t first, TObject *enl) -{ - if (!IsValid() || !fElements->GetSize()) { - Error("Process", "not a correctly initialized TDSet"); - return -1; - } - - // Set entry list - SetEntryList(enl); - - if (gProof) - return gProof->Process(this, selector, option, nentries, first); - - Error("Process", "no active PROOF session"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process TDSet on currently active PROOF session. -/// The last argument 'enl' specifies an entry- or event-list to be used as -/// event selection. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TDSet::Process(const char *selector, Option_t *option, Long64_t nentries, - Long64_t first, TObject *enl) -{ - if (!IsValid() || !fElements->GetSize()) { - Error("Process", "not a correctly initialized TDSet"); - return -1; - } - - // Set entry list - SetEntryList(enl); - - if (gProof) - return gProof->Process(this, selector, option, nentries, first); - - Error("Process", "no active PROOF session"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add objects that might be needed during the processing of -/// the selector (see Process()). - -void TDSet::AddInput(TObject *obj) -{ - if (gProof) { - gProof->AddInput(obj); - } else { - Error("AddInput","No PROOF session active"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear input object list. - -void TDSet::ClearInput() -{ - if (gProof) - gProof->ClearInput(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get specified object that has been produced during the processing -/// (see Process()). - -TObject *TDSet::GetOutput(const char *name) -{ - if (gProof) - return gProof->GetOutput(name); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get list with all object created during processing (see Process()). - -TList *TDSet::GetOutputList() -{ - if (gProof) - return gProof->GetOutputList(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print TDSet basic or full data. When option="a" print full data. - -void TDSet::Print(const Option_t *opt) const -{ - const char *clnm = (IsA()) ? IsA()->GetName() : "TDSet"; - Printf("OBJ: %s\ttype %s\t%s\tin %s\telements %d", clnm, GetName(), - fObjName.Data(), GetTitle(), GetListOfElements()->GetSize()); - - if (opt && opt[0] == 'a') { - TIter next(GetListOfElements()); - TObject *obj; - while ((obj = next())) { - obj->Print(opt); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set/change object name. - -void TDSet::SetObjName(const char *objname) -{ - if (objname) { - fObjName = objname; - TIter next(GetListOfElements()); - TDSetElement *e; - while ((e = (TDSetElement *) next())) { - e->SetTitle(objname); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set/change directory. - -void TDSet::SetDirectory(const char *dir) -{ - if (dir) - fDir = dir; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add file to list of files to be analyzed. Optionally with the -/// objname and dir arguments the default, TDSet wide, objname and -/// dir can be overridden. - -Bool_t TDSet::Add(const char *file, const char *objname, const char *dir, - Long64_t first, Long64_t num, const char *msd) -{ - if (!file || !*file) { - Error("Add", "file name must be specified"); - return kFALSE; - } - - TString fn = file; - if (gProof && gProof->IsLite()) { - TUrl u(file, kTRUE); - if (!strcmp(u.GetProtocol(), "file")) { - fn = u.GetFileAndOptions(); - gSystem->ExpandPathName(fn); - if (!gSystem->IsAbsoluteFileName(fn)) - gSystem->PrependPathName(gSystem->WorkingDirectory(), fn); - } - } - - // check, if it already exists in the TDSet - TDSetElement *el = (TDSetElement *) fElements->FindObject(fn); - if (!el) { - if (!objname) - objname = GetObjName(); - if (!dir) - dir = GetDirectory(); - fElements->Add(new TDSetElement(fn, objname, dir, first, num, msd)); - } else { - TString msg; - msg.Form("duplication detected: %40s is already in dataset - ignored", fn.Data()); - Warning("Add", "%s", msg.Data()); - if (gProofServ) { - msg.Insert(0, "WARNING: "); - gProofServ->SendAsynMessage(msg); - } - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add specified data set to the this set. - -Bool_t TDSet::Add(TDSet *dset) -{ - if (!dset) - return kFALSE; - - if (TestBit(TDSet::kMultiDSet)) { - fElements->Add(dset); - return kTRUE; - } - - if (fType != dset->GetType()) { - Error("Add", "cannot add a set with a different type"); - return kFALSE; - } - - TDSetElement *el; - TIter next(dset->fElements); - TObject *last = (dset == this) ? fElements->Last() : 0; - while ((el = (TDSetElement*) next())) { - Add(el->GetFileName(), el->GetObjName(), el->GetDirectory(), - el->GetFirst(), el->GetNum(), el->GetMsd()); - if (el == last) break; - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add files passed as list of TFileInfo, TUrl or TObjString objects . -/// If TFileInfo, the first entry and the number of entries are also filled. -/// The argument 'meta' can be used to specify one of the subsets in the -/// file as described in the metadata of TFileInfo. By default the first one -/// is taken. -/// If 'availableOnly' is true only files available ('staged' and non corrupted) -/// are taken: those not satisfying this requirement are added to 'badlist', if -/// the latter is defined. By default availableOnly is false. - -Bool_t TDSet::Add(TCollection *filelist, const char *meta, Bool_t availableOnly, - TCollection *badlist) -{ - if (!filelist) - return kFALSE; - - TObject *o = 0; - TIter next(filelist); - while ((o = next())) { - TString cn(o->ClassName()); - if (cn == "TFileInfo") { - TFileInfo *fi = (TFileInfo *)o; - if (!availableOnly || - (fi->TestBit(TFileInfo::kStaged) && - !fi->TestBit(TFileInfo::kCorrupted))) { - Int_t nf = fElements->GetSize(); - if (!Add(fi, meta)) return kFALSE; - // Duplications count as bad files - if (fElements->GetSize() <= nf && badlist) badlist->Add(fi); - } else if (badlist) { - // Return list of non-usable files - badlist->Add(fi); - } - } else if (cn == "TUrl") { - Add(((TUrl *)o)->GetUrl()); - } else if (cn == "TObjString") { - Add(((TObjString *)o)->GetName()); - } else { - Warning("Add","found object fo unexpected type %s - ignoring", cn.Data()); - } - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set (or unset) the list for mapping servers coordinate for files. -/// Reinitialize the related iterator if needed. -/// Used by TProof. - -void TDSet::SetSrvMaps(TList *srvmaps) -{ - fSrvMaps = srvmaps; - SafeDelete(fSrvMapsIter); - if (fSrvMaps) fSrvMapsIter = new TIter(fSrvMaps); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add file described by 'fi' to list of files to be analyzed. -/// The argument 'meta' can be used to specify a subsets in the -/// file as described in the metadata of TFileInfo. By default the first one -/// is taken. - -Bool_t TDSet::Add(TFileInfo *fi, const char *meta) -{ - if (!fi) { - Error("Add", "TFileInfo object name must be specified"); - return kFALSE; - } - TString msg; - - // Check if a remap of the server coordinates is requested - const char *file = fi->GetFirstUrl()->GetUrl(); - Bool_t setLookedUp = kTRUE; - TString file1; - if (TDataSetManager::CheckDataSetSrvMaps(fi->GetFirstUrl(), file1, fSrvMaps) && - !(file1.IsNull())) { - file = file1.Data(); - setLookedUp = kFALSE; - } - // Check if it already exists in the TDSet - if (fElements->FindObject(file)) { - msg.Form("duplication detected: %40s is already in dataset - ignored", file); - Warning("Add", "%s", msg.Data()); - if (gProofServ) { - msg.Insert(0, "WARNING: "); - gProofServ->SendAsynMessage(msg); - } - return kTRUE; - } - - // If more than one metadata info require the specification of the objpath; - // the order in which they appear is not guaranteed and the error may be - // very difficult to find. - TFileInfoMeta *m = 0; - if (!meta || strlen(meta) <= 0 || !strcmp(meta, "/")) { - TList *fil = 0; - if ((fil = fi->GetMetaDataList()) && fil->GetSize() > 1) { - msg.Form("\n Object name unspecified and several objects available.\n"); - msg += " Please choose one from the list below:\n"; - TIter nx(fil); - while ((m = (TFileInfoMeta *) nx())) { - TString nm(m->GetName()); - if (nm.BeginsWith("/")) nm.Remove(0,1); - msg += Form(" %s -> TProof::Process(\"%s#%s\",...)\n", - nm.Data(), GetName(), nm.Data()); - } - if (gProofServ) - gProofServ->SendAsynMessage(msg); - else - Warning("Add", "%s", msg.Data()); - return kFALSE; - } - } - - // Get the metadata, if any - m = fi->GetMetaData(meta); - - // Create the element - const char *objname = 0; - const char *dir = 0; - Long64_t first = 0; - Long64_t num = -1; - if (!m) { - objname = GetObjName(); - dir = GetDirectory(); - } else { - objname = (m->GetObject() && strlen(m->GetObject())) ? m->GetObject() : GetObjName(); - dir = (m->GetDirectory() && strlen(m->GetDirectory())) ? m->GetDirectory() : GetDirectory(); - first = m->GetFirst(); - num = m->GetEntries(); - } - const char *dataset = 0; - if (strcmp(fi->GetTitle(), "TFileInfo")) dataset = fi->GetTitle(); - TDSetElement *el = new TDSetElement(file, objname, dir, first, -1, 0, dataset); - el->SetEntries(num); - - // Set looked-up bit - if (fi->TestBit(TFileInfo::kStaged) && setLookedUp) - el->SetBit(TDSetElement::kHasBeenLookedUp); - if (fi->TestBit(TFileInfo::kCorrupted)) - el->SetBit(TDSetElement::kCorrupted); - - // Add the element - fElements->Add(el); - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Export TDSetElements files as list of TFileInfo objects in file -/// 'fpath'. If the file exists already the action fails, unless -/// 'opt == "F"'. -/// Return 0 on success, -1 otherwise - -Int_t TDSet::ExportFileList(const char *fpath, Option_t *opt) -{ - if (!fElements) - return -1; - if (fElements->GetSize() <= 0) - return 0; - - Bool_t force = (opt[0] == 'F' || opt[0] == 'f'); - - if (gSystem->AccessPathName(fpath, kFileExists) == kFALSE) { - if (force) { - // Try removing the file - if (gSystem->Unlink(fpath)) { - Info("ExportFileList","error removing dataset file: %s", fpath); - return -1; - } - } - } - - // Create the file list - TList *fileinfo = new TList; - fileinfo->SetOwner(); - - TDSetElement *dse = 0; - TIter next(fElements); - while ((dse = (TDSetElement *) next())) { - TFileInfoMeta *m = new TFileInfoMeta(dse->GetTitle(), dse->GetDirectory(), GetType(), - dse->GetNum(), dse->GetFirst()); - TFileInfo *fi = new TFileInfo(dse->GetFileName()); - fi->AddMetaData(m); - fileinfo->Add(fi); - } - - // Write to file - TFile *f = TFile::Open(fpath, "RECREATE"); - if (f) { - f->cd(); - fileinfo->Write("fileList", TObject::kSingleKey); - f->Close(); - } else { - Info("ExportFileList","error creating dataset file: %s", fpath); - SafeDelete(fileinfo); - return -1; - } - - // Cleanup - SafeDelete(f); - SafeDelete(fileinfo); - - // We are done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add friend dataset to this set. Only possible if the TDSet type is -/// a TTree or derived class. The friendset will be owned by this class -/// and deleted in its destructor. - -void TDSet::AddFriend(TDSet *friendset, const char* alias) -{ - if (!friendset) { - Error("AddFriend", "The friend TDSet is null!"); - return; - } - - if (!fIsTree) { - Error("AddFriend", "a friend set can only be added to a TTree TDSet"); - return; - } - TList *thisList = GetListOfElements(); - TList *friendsList = friendset->GetListOfElements(); - if (thisList->GetSize() != friendsList->GetSize() && friendsList->GetSize() != 1) { - Error("AddFriend", "the friend dataset has %d elements while the main one has %d", - thisList->GetSize(), friendsList->GetSize()); - return; - } - TIter next(thisList); - TIter next2(friendsList); - TDSetElement *friendElem = 0; - if (friendsList->GetSize() == 1) - friendElem = dynamic_cast (friendsList->First()); - while(TDSetElement* e = dynamic_cast (next())) { - if (friendElem) // just one elem in the friend TDSet - e->AddFriend(friendElem, alias); - else - e->AddFriend(dynamic_cast (next2()), alias); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset or initialize access to the elements. - -void TDSet::Reset() -{ - if (!fIterator) { - fIterator = new TIter(fElements); - } else { - fIterator->Reset(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns number of entries in tree or objects in file. Returns -1 in -/// case of error. - -Long64_t TDSet::GetEntries(Bool_t isTree, const char *filename, const char *path, - TString &objname) -{ - Double_t start = 0; - if (gPerfStats) start = TTimeStamp(); - - // Take into acoount possible prefixes - TFile::EFileType typ = TFile::kDefault; - TString fname = gEnv->GetValue("Path.Localroot",""), pfx(fname); - // Get the locality (disable warnings or errors in connection attempts) - Int_t oldLevel = gErrorIgnoreLevel; - gErrorIgnoreLevel = kError+1; - if ((typ = TFile::GetType(filename, "", &fname)) != TFile::kLocal) fname = filename; - gErrorIgnoreLevel = oldLevel; - // Open the file - TFile *file = TFile::Open(fname); - - if (gPerfStats) - gPerfStats->FileOpenEvent(file, filename, start); - - if (file == 0) { - ::SysError("TDSet::GetEntries", - "cannot open file %s (type: %d, pfx: %s)", filename, typ, pfx.Data()); - return -1; - } - - TDirectory *dirsave = gDirectory; - if (!file->cd(path)) { - ::Error("TDSet::GetEntries", "cannot cd to %s", path); - delete file; - return -1; - } - - TDirectory *dir = gDirectory; - dirsave->cd(); - - Long64_t entries; - Bool_t fillname = kFALSE; - if (isTree) { - - TString on(objname); - TString sreg(objname); - // If a wild card we will use the first object of the type - // requested compatible with the reg expression we got - if (sreg.Length() <= 0 || sreg == "" || sreg.Contains("*")) { - fillname = kTRUE; - if (sreg.Contains("*")) - sreg.ReplaceAll("*", ".*"); - else - sreg = ".*"; - TRegexp re(sreg); - if (dir->GetListOfKeys()) { - TIter nxk(dir->GetListOfKeys()); - TKey *k = 0; - Bool_t notfound = kTRUE; - while ((k = (TKey *) nxk())) { - if (!strcmp(k->GetClassName(), "TTree")) { - TString kn(k->GetName()); - if (kn.Index(re) != kNPOS) { - if (notfound) { - on = kn; - notfound = kFALSE; - } else if (kn != on) { - ::Warning("TDSet::GetEntries", - "additional tree found in the file: %s", kn.Data()); - } - } - } - } - } - } - - TKey *key = dir->GetKey(on); - if (key == 0) { - ::Error("TDSet::GetEntries", "cannot find tree \"%s\" in %s", - objname.Data(), filename); - delete file; - return -1; - } - TTree *tree = (TTree *) key->ReadObj(); - if (tree == 0) { - // Error always reported? - delete file; - return -1; - } - entries = tree->GetEntries(); - delete tree; - - // Return full name in case of wildcards - objname = (fillname) ? on : objname; - - } else { - TList *keys = dir->GetListOfKeys(); - entries = keys->GetSize(); - } - - delete file; - return entries; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw expression varexp for specified entries. -/// Returns -1 in case of error or number of selected events in case of success. -/// This function accepts a TCut objects as argument. -/// Use the operator+ to concatenate cuts. -/// Example: -/// dset.Draw("x",cut1+cut2+cut3); - -Long64_t TDSet::Draw(const char *varexp, const TCut &selection, Option_t *option, - Long64_t nentries, Long64_t firstentry) -{ - return Draw(varexp, selection.GetTitle(), option, nentries, firstentry); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw expression varexp for specified entries. -/// Returns -1 in case of error or number of selected events in case of success. -/// For more see TTree::Draw(). - -Long64_t TDSet::Draw(const char *varexp, const char *selection, Option_t *option, - Long64_t nentries, Long64_t firstentry) -{ - if (!IsValid() || !fElements->GetSize()) { - Error("Draw", "not a correctly initialized TDSet"); - return -1; - } - - if (gProof) - return gProof->DrawSelect(this, varexp, selection, option, nentries, - firstentry); - - Error("Draw", "no active PROOF session"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Start the TTreeViewer on this TTree. - -void TDSet::StartViewer() -{ - if (gROOT->IsBatch()) { - Warning("StartViewer", "viewer cannot run in batch mode"); - return; - } - - if (!gProof) { - Error("StartViewer", "no PROOF found"); - return; - } - if (!IsTree()) { - Error("StartViewer", "TDSet contents should be of type TTree (or subtype)"); - return; - } - fProofChain = new TProofChain(this, kTRUE); - - const char *hname = gEnv->GetValue("TreeViewer.Name", "TTreeViewer"); - if (auto h = gROOT->GetPluginManager()->FindHandler("TVirtualTreeViewer", hname)) { - if (h->LoadPlugin() != -1) - h->ExecPlugin(1,fProofChain); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns a tree header containing the branches' structure of the dataset. - -TTree* TDSet::GetTreeHeader(TProof* proof) -{ - return proof->GetTreeHeader(this); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check if all elements are valid. - -Bool_t TDSet::ElementsValid() -{ - if (TestBit(TDSet::kValidityChecked)) - return (TestBit(TDSet::kSomeInvalid) ? kFALSE : kTRUE); - - SetBit(TDSet::kValidityChecked); - ResetBit(TDSet::kSomeInvalid); - TIter nextElem(GetListOfElements()); - while (TDSetElement *elem = dynamic_cast(nextElem())) { - if (!elem->GetValid()) { - SetBit(TDSet::kSomeInvalid); - return kFALSE; - } - } - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove TDSetElement 'elem' from the list. -/// Return 0 on success, -1 if the element is not in the list - -Int_t TDSet::Remove(TDSetElement *elem, Bool_t deleteElem) -{ - if (!elem || !(((THashList *)(GetListOfElements()))->Remove(elem))) - return -1; - - if (deleteElem) - SafeDelete(elem); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Validate the TDSet by opening files. - -void TDSet::Validate() -{ - TIter nextElem(GetListOfElements()); - while (TDSetElement *elem = dynamic_cast(nextElem())) { - if (!elem->GetValid()) - elem->Validate(IsTree()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Resolve the end-point URL for the current elements of this data set -/// If the removeMissing option is set to kTRUE, remove the TDSetElements -/// that can not be located. -/// The method returns the list of removed TDSetElements in *listOfMissingFiles -/// if the latter is defined (the list must be created outside). - -void TDSet::Lookup(Bool_t removeMissing, TList **listOfMissingFiles) -{ - // If an entry- or event- list has been given, assign the relevant portions - // to each element; this allows to look-up only for the elements which have - // something to be processed, so it is better to do it before the real look-up - // operations. - SplitEntryList(); - - TString msg("Looking up for exact location of files"); - UInt_t n = 0; - UInt_t ng = 0; - UInt_t tot = GetListOfElements()->GetSize(); - UInt_t n2 = (tot > 50) ? (UInt_t) tot / 50 : 1; - Bool_t st = kTRUE; - TIter nextElem(GetListOfElements()); - while (TDSetElement *elem = dynamic_cast(nextElem())) { - if (elem->GetNum() != 0) { // -1 means "all entries" - ng++; - if (!elem->GetValid()) - if (elem->Lookup(kFALSE)) - if (removeMissing) { - if (Remove(elem, kFALSE)) - Error("Lookup", "Error removing a missing file"); - if (listOfMissingFiles) - (*listOfMissingFiles)->Add(elem->GetFileInfo(fType)); - } - } - n++; - // Notify the client - if (gProof && (n > 0 && !(n % n2))) - gProof->SendDataSetStatus(msg, n, tot, st); - // Break if we have been asked to stop - if (gProof && gProof->GetRunStatus() != TProof::kRunning) - break; - } - // Notify the client if not all the files have entries to be processed - // (which may happen if an entry-list is used) - if (ng < tot && gProofServ) { - msg = Form("Files with entries to be processed: %d (out of %d)\n", ng, tot); - gProofServ->SendAsynMessage(msg); - } else { - // Final notification to the client - if (gProof) gProof->SendDataSetStatus(msg, n, tot, st); - } - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Flag all the elements as looked-up, so to avoid opening the files -/// if the functionality is not supported - -void TDSet::SetLookedUp() -{ - TIter nextElem(GetListOfElements()); - while (TDSetElement *elem = dynamic_cast(nextElem())) - elem->SetLookedUp(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Validate the TDSet against another TDSet. -/// Only validates elements in common from input TDSet. - -void TDSet::Validate(TDSet* dset) -{ - THashList bestElements; - bestElements.SetOwner(); - TList namedHolder; - namedHolder.SetOwner(); - TIter nextOtherElem(dset->GetListOfElements()); - while (TDSetElement *elem = dynamic_cast(nextOtherElem())) { - if (!elem->GetValid()) continue; - TString dir_file_obj = elem->GetDirectory(); - dir_file_obj += "_"; - dir_file_obj += TUrl(elem->GetFileName()).GetFileAndOptions(); - dir_file_obj += "_"; - dir_file_obj += elem->GetObjName(); - TPair *p = dynamic_cast(bestElements.FindObject(dir_file_obj)); - if (p) { - TDSetElement *prevelem = dynamic_cast(p->Value()); - if (prevelem) { - Long64_t entries = prevelem->GetFirst()+prevelem->GetNum(); - if (entriesGetFirst()+elem->GetNum()) { - bestElements.Remove(p); - bestElements.Add(new TPair(p->Key(), elem)); - delete p; - } - } - } else { - TNamed* named = new TNamed(dir_file_obj, dir_file_obj); - namedHolder.Add(named); - bestElements.Add(new TPair(named, elem)); - } - } - - TIter nextElem(GetListOfElements()); - while (TDSetElement *elem = dynamic_cast(nextElem())) { - if (!elem->GetValid()) { - TString dir_file_obj = elem->GetDirectory(); - dir_file_obj += "_"; - dir_file_obj += TUrl(elem->GetFileName()).GetFileAndOptions(); - dir_file_obj += "_"; - dir_file_obj += elem->GetObjName(); - if (TPair *p = dynamic_cast(bestElements.FindObject(dir_file_obj))) { - TDSetElement* validelem = dynamic_cast(p->Value()); - elem->Validate(validelem); - } - } - } -} - -// -// To handle requests coming from version 3 client / masters we need -// a special streamer -//////////////////////////////////////////////////////////////////////////////// -/// Stream an object of class TDSetElement. - -void TDSetElement::Streamer(TBuffer &R__b) -{ - if (R__b.IsReading()) { - UInt_t R__s, R__c; - Version_t R__v = R__b.ReadVersion(&R__s, &R__c); - ResetBit(kWriteV3); - if (R__v > 4) { - R__b.ReadClassBuffer(TDSetElement::Class(), this, R__v, R__s, R__c); - } else { - // For version 3 client / masters we need a special streamer - SetBit(kWriteV3); - if (R__v > 3) { - TNamed::Streamer(R__b); - } else { - // Old versions were not deriving from TNamed and had the - // file name and the object type name in the first two members - TObject::Streamer(R__b); - TString name, title; - R__b >> name >> title; - SetNameTitle(name, title); - } - // Now we read the standard part - R__b >> fDirectory; - R__b >> fFirst; - R__b >> fNum; - R__b >> fMsd; - R__b >> fTDSetOffset; - TEventList *evl; - R__b >> evl; - R__b >> fValid; - R__b >> fEntries; - - // Special treatment waiting for proper retrieving of stl containers - FriendsList_t *friends = new FriendsList_t; - static TClassRef classFriendsList = TClass::GetClass(typeid(FriendsList_t)); - R__b.ReadClassBuffer( classFriendsList, friends, classFriendsList->GetClassVersion(), 0, 0); - if (friends) { - // Convert friends to a TList (to be written) - fFriends = new TList(); - fFriends->SetOwner(); - for (FriendsList_t::iterator i = friends->begin(); - i != friends->end(); ++i) { - TDSetElement *dse = (TDSetElement *) i->first->Clone(); - fFriends->Add(new TPair(dse, new TObjString(i->second.Data()))); - } - } - // the value for fIsTree (only older versions are sending it) - Bool_t tmpIsTree; - R__b >> tmpIsTree; - R__b.CheckByteCount(R__s, R__c, TDSetElement::IsA()); - } - } else { - if (TestBit(kWriteV3)) { - // For version 3 client / masters we need a special streamer - R__b << Version_t(3); - TObject::Streamer(R__b); - R__b << TString(GetName()); - R__b << TString(GetTitle()); - R__b << fDirectory; - R__b << fFirst; - R__b << fNum; - R__b << fMsd; - R__b << fTDSetOffset; - R__b << (TEventList *)0; - R__b << fValid; - R__b << fEntries; - - // Special treatment waiting for proper retrieving of stl containers - FriendsList_t *friends = new FriendsList_t; - if (fFriends) { - TIter nxf(fFriends); - TPair *p = 0; - while ((p = (TPair *)nxf())) - friends->push_back(std::make_pair((TDSetElement *)p->Key(), - TString(((TObjString *)p->Value())->GetName()))); - } - static TClassRef classFriendsList = TClass::GetClass(typeid(FriendsList_t)); - R__b.WriteClassBuffer( classFriendsList, &friends ); - - // Older versions had an unused boolean called fIsTree: we fill it - // with its default value - R__b << kFALSE; - } else { - R__b.WriteClassBuffer(TDSetElement::Class(),this); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Stream an object of class TDSet. - -void TDSet::Streamer(TBuffer &R__b) -{ - if (R__b.IsReading()) { - UInt_t R__s, R__c; - Version_t R__v = R__b.ReadVersion(&R__s, &R__c); - ResetBit(kWriteV3); - if (R__v > 3) { - R__b.ReadClassBuffer(TDSet::Class(), this, R__v, R__s, R__c); - } else { - // For version 3 client / masters we need a special streamer - SetBit(kWriteV3); - TNamed::Streamer(R__b); - R__b >> fDir; - R__b >> fType; - R__b >> fObjName; - TList elems; - elems.Streamer(R__b); - elems.SetOwner(kFALSE); - if (elems.GetSize() > 0) { - fElements = new THashList; - fElements->SetOwner(); - TDSetElement *e = 0; - TIter nxe(&elems); - while ((e = (TDSetElement *)nxe())) { - fElements->Add(e); - } - } else { - fElements = 0; - } - R__b >> fIsTree; - } - } else { - if (TestBit(kWriteV3)) { - // For version 3 client / masters we need a special streamer - R__b << Version_t(3); - TNamed::Streamer(R__b); - R__b << fDir; - R__b << fType; - R__b << fObjName; - TList elems; - if (fElements) { - elems.SetOwner(kFALSE); - if (fElements->GetSize() > 0) { - TDSetElement *e = 0; - TIter nxe(fElements); - while ((e = (TDSetElement *)nxe())) - elems.Add(e); - } - } - elems.Streamer(R__b); - R__b << fIsTree; - } else { - R__b.WriteClassBuffer(TDSet::Class(),this); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set/Reset the 'OldStreamer' bit in this instance and its elements. -/// Needed for backward compatibility in talking to old client / masters. - -void TDSet::SetWriteV3(Bool_t on) -{ - if (on) - SetBit(TDSet::kWriteV3); - else - ResetBit(TDSet::kWriteV3); - // Loop over dataset elements - TIter nxe(GetListOfElements()); - TObject *o = 0; - while ((o = nxe())) - if (on) - o->SetBit(TDSetElement::kWriteV3); - else - o->ResetBit(TDSetElement::kWriteV3); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set entry (or event) list for this data set - -void TDSet::SetEntryList(TObject *aList) -{ - if (!aList) { - // Nothing to do, except making sure to disable any previous setting - fEntryList = 0; - // Reset the element lists - // TEntryList - TIter next(fElements); - TDSetElement *el=0; - while ((el=(TDSetElement*)next())){ - el->SetEntryList(aList); - el->Reset(); - } - return; - } - - if (TestBit(TDSet::kMultiDSet)) { - - // Global entry list for all the datasets - TIter nxds(fElements); - TDSet *ds = 0; - while ((ds = (TDSet *) nxds())) - ds->SetEntryList(aList); - - } else { - - // Link the proper object - TEventList *evl = 0; - TEntryList *enl = dynamic_cast(aList); - if (!enl) - evl = dynamic_cast(aList); - if (!enl && !evl) { - Error("SetEntryList", "type of input object must be either TEntryList " - "or TEventList (found: '%s' - do nothing", aList->ClassName()); - return; - } - - // Action depends on the type - fEntryList = (enl) ? enl : (TEntryList *)evl; - } - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Splits the main entry (or event) list into sub-lists for the elements of -/// thet data set - -void TDSet::SplitEntryList() -{ - if (TestBit(TDSet::kMultiDSet)) { - // Global entry list for all the datasets - TIter nxds(fElements); - TDSet *ds = 0; - while ((ds = (TDSet *) nxds())) - ds->SplitEntryList(); - // Done - return; - } - - if (!fEntryList) { - if (gDebug > 0) - Info("SplitEntryList", "no entry- (or event-) list to split - do nothing"); - return; - } - - // Action depend on type of list - TEntryList *enl = dynamic_cast(fEntryList); - if (enl) { - // TEntryList - TIter next(fElements); - TDSetElement *el=0; - TEntryList *sublist = 0; - while ((el=(TDSetElement*)next())){ - sublist = enl->GetEntryList(el->GetObjName(), el->GetFileName()); - if (sublist){ - el->SetEntryList(sublist); - el->SetNum(sublist->GetN()); - } else { - sublist = new TEntryList("", ""); - el->SetEntryList(sublist); - el->SetNum(0); - } - } - } else { - TEventList *evl = dynamic_cast(fEntryList); - if (evl) { - // TEventList - TIter next(fElements); - TDSetElement *el, *prev; - - prev = dynamic_cast (next()); - if (!prev) - return; - Long64_t low = prev->GetTDSetOffset(); - Long64_t high = low; - Long64_t currPos = 0; - do { - el = dynamic_cast (next()); - // kMaxLong64 means infinity - high = (el == 0) ? kMaxLong64 : el->GetTDSetOffset(); -#ifdef DEBUG - while (currPos < evl->GetN() && evl->GetEntry(currPos) < low) { - Error("SplitEntryList", - "TEventList: event outside of the range of any of the TDSetElements"); - currPos++; // unnecessary check - } -#endif - TEventList* nevl = new TEventList(); - while (currPos < evl->GetN() && evl->GetEntry((Int_t)currPos) < high) { - nevl->Enter(evl->GetEntry((Int_t)currPos) - low); - currPos++; - } - prev->SetEntryList(nevl); - prev->SetNum(nevl->GetN()); - low = high; - prev = el; - } while (el); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the number of files in the dataset - -Int_t TDSet::GetNumOfFiles() -{ - Int_t nf = -1; - if (fElements) { - nf = 0; - if (TestBit(TDSet::kMultiDSet)) { - TIter nxds(fElements); - TDSet *ds = 0; - while ((ds = (TDSet *) nxds())) - if (ds->GetListOfElements()) nf += ds->GetListOfElements()->GetSize(); - } else { - nf = fElements->GetSize(); - } - } - // Done - return nf; -} diff --git a/proof/proof/src/TDSetProxy.cxx b/proof/proof/src/TDSetProxy.cxx deleted file mode 100644 index 02d404924924f..0000000000000 --- a/proof/proof/src/TDSetProxy.cxx +++ /dev/null @@ -1,73 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Maarten Ballintijn 12/03/02 - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -////////////////////////////////////////////////////////////////////////// -// // -// TDSetProxy // -// // -// TDSet proxy for use on slaves. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TDSetProxy.h" -#include "TProofServ.h" - - -ClassImp(TDSetProxy); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TDSetProxy::TDSetProxy() -{ - fServ = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TDSetProxy::TDSetProxy(const char *type, const char *objname, const char *dir) - : TDSet(type,objname,dir) -{ - fServ = 0; - fCurrent = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the reference TProofServ instance - -void TDSetProxy::SetProofServ(TProofServ *serv) -{ - fServ = serv; - fCurrent = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset this instance - -void TDSetProxy::Reset() -{ - delete fCurrent; fCurrent = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the next packet - -TDSetElement *TDSetProxy::Next(Long64_t totalEntries) -{ - fCurrent = fServ->GetNextPacket(totalEntries); - - // Check log file length (before processing the next packet, so we have the - // chance to keep the latest logs) - fServ->TruncateLogFile(); - - return fCurrent; -} diff --git a/proof/proof/src/TDataSetManager.cxx b/proof/proof/src/TDataSetManager.cxx deleted file mode 100644 index 767502b52c537..0000000000000 --- a/proof/proof/src/TDataSetManager.cxx +++ /dev/null @@ -1,1986 +0,0 @@ -// @(#)root/base:$Id$ -// Author: Jan Fiete Grosse-Oetringhaus, 04.06.07 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -////////////////////////////////////////////////////////////////////////// -// // -// TDataSetManager // -// // -// This class contains functions to handle datasets in PROOF // -// It is the layer between TProofServ and the file system that stores // -// the datasets. // -// // -////////////////////////////////////////////////////////////////////////// - - -#include "TDataSetManager.h" - -#include "Riostream.h" - -#include "TEnv.h" -#include "TError.h" -#include "TFile.h" -#include "TFileCollection.h" -#include "TFileInfo.h" -#include "TFileStager.h" -#include "TMD5.h" -#include "THashList.h" -#include "TKey.h" -#include "TObjString.h" -#include "TParameter.h" -#include "TPRegexp.h" -#include "TRegexp.h" -#include "TSystem.h" -#include "TTree.h" -#include "TUrl.h" -#include "TVirtualMonitoring.h" - -// One Gigabyte -#define DSM_ONE_GB (1073741824) - -// Name for common datasets -TString TDataSetManager::fgCommonDataSetTag = "COMMON"; -TList *TDataSetManager::fgDataSetSrvMaps = 0; - -ClassImp(TDataSetManager); - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Main constructor - -TDataSetManager::TDataSetManager(const char *group, const char *user, - const char *options) - : fGroup(group), - fUser(user), fCommonUser(), fCommonGroup(), - fGroupQuota(), fGroupUsed(), - fUserUsed(), fNTouchedFiles(0), fNOpenedFiles(0), - fNDisappearedFiles(0), fMTimeGroupConfig(-1) -{ - // Fill default group and user if none is given - if (fGroup.IsNull()) - fGroup = "default"; - if (fUser.IsNull()) { - fUser = "--nouser--"; - // Get user logon name - UserGroup_t *pw = gSystem->GetUserInfo(); - if (pw) { - fUser = pw->fUser; - delete pw; - } - } - - fGroupQuota.SetOwner(); - fGroupUsed.SetOwner(); - fUserUsed.SetOwner(); - - fCommonUser = "COMMON"; - fCommonGroup = "COMMON"; - - fNTouchedFiles = -1; - fNOpenedFiles = -1; - fNDisappearedFiles = -1; - fMTimeGroupConfig = -1; - - fAvgFileSize = 50000000; // Default 50 MB per file - - // Parse options - ParseInitOpts(options); - - if (!fUser.IsNull() && !fGroup.IsNull()) { - - // If not in sandbox, construct the base URI using session defaults - // (group, user) (syntax: /group/user/dsname[#[subdir/]objname]) - if (!TestBit(TDataSetManager::kIsSandbox)) - fBase.SetUri(TString(Form("/%s/%s/", fGroup.Data(), fUser.Data()))); - - } - - // List of dataset server mapping instructions - TString srvmaps(gEnv->GetValue("DataSet.SrvMaps","")); - TString srvmapsenv(gSystem->Getenv("DATASETSRVMAPS")); - if (!(srvmapsenv.IsNull())) { - if (srvmapsenv.BeginsWith("+")) { - if (!(srvmaps.IsNull())) srvmaps += ","; - srvmaps += srvmapsenv(1,srvmapsenv.Length()); - } else { - srvmaps = srvmapsenv; - } - } - if (!(srvmaps.IsNull()) && !(fgDataSetSrvMaps = ParseDataSetSrvMaps(srvmaps))) - Warning("TDataSetManager", "problems parsing DataSet.SrvMaps input info (%s)" - " - ignoring", srvmaps.Data()); - - // Read config file - ReadGroupConfig(gEnv->GetValue("Proof.GroupFile", "")); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TDataSetManager::~TDataSetManager() -{ - // Clear used space - fGroupQuota.DeleteAll(); - fGroupUsed.DeleteAll(); - fUserUsed.DeleteAll(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Parse the opts string and set the init bits accordingly -/// Available options: -/// Cq: set kCheckQuota -/// Ar: set kAllowRegister -/// Av: set kAllowVerify -/// Ti: set kTrustInfo -/// Sb: set kIsSandbox -/// Ca: set kUseCache or kDoNotUseCache -/// The opts string may also contain additional unrelated info: in such a case -/// the field delimited by the prefix "opt:" is analyzed, e.g. if opts is -/// "/tmp/dataset opt:Cq:-Ar: root://lxb6046.cern.ch" only the substring -/// "Cq:-Ar:" will be parsed . - -void TDataSetManager::ParseInitOpts(const char *opts) -{ - // Default option bits - ResetBit(TDataSetManager::kCheckQuota); - SetBit(TDataSetManager::kAllowRegister); - SetBit(TDataSetManager::kAllowVerify); - SetBit(TDataSetManager::kTrustInfo); - ResetBit(TDataSetManager::kIsSandbox); - ResetBit(TDataSetManager::kUseCache); - ResetBit(TDataSetManager::kDoNotUseCache); - - if (opts && strlen(opts) > 0) { - TString opt(opts); - // If it contains the prefix "opt:", isolate the related field - Int_t ip = opt.Index("opt:"); - if (ip != kNPOS) opt.Remove(0, ip + 4); - ip = opt.Index(" "); - if (ip != kNPOS) opt.Remove(ip); - // Check the content, now - if (opt.Contains("Cq:") && !opt.Contains("-Cq:")) - SetBit(TDataSetManager::kCheckQuota); - if (opt.Contains("-Ar:")) - ResetBit(TDataSetManager::kAllowRegister); - if (opt.Contains("-Av:")) - ResetBit(TDataSetManager::kAllowVerify); - if (opt.Contains("-Ti:")) - ResetBit(TDataSetManager::kTrustInfo); - if (opt.Contains("Sb:") && !opt.Contains("-Sb:")) - SetBit(TDataSetManager::kIsSandbox); - if (opt.Contains("Ca:")) - SetBit(TDataSetManager::kUseCache); - if (opt.Contains("-Ca:")) - SetBit(TDataSetManager::kDoNotUseCache); - } - - // Check dependencies - if (TestBit(TDataSetManager::kAllowVerify)) { - // Dataset verification or requires registration permition - SetBit(TDataSetManager::kAllowRegister); - } - // UseCache has priority - if (TestBit(TDataSetManager::kUseCache) && TestBit(TDataSetManager::kDoNotUseCache)) - ResetBit(TDataSetManager::kDoNotUseCache); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Read group config file 'cf'. -/// If cf == 0 re-read, if changed, the file pointed by fGroupConfigFile . -/// -/// expects the following directives: -/// Group definition: -/// group `` ``+ -/// disk quota -/// property `` diskquota `` -/// average filesize (to be used when the file size is not available) -/// averagefilesize ``{G,g,M,m,K,k} - -Bool_t TDataSetManager::ReadGroupConfig(const char *cf) -{ - // Validate input - FileStat_t st; - if (!cf || (strlen(cf) <= 0) || !strcmp(cf, fGroupConfigFile.Data())) { - // If this is the first time we cannot do anything - if (fGroupConfigFile.IsNull()) { - if (gDebug > 0) - Info("ReadGroupConfig", "path to config file undefined - nothing to do"); - return kFALSE; - } - // Check if fGroupConfigFile has changed - if (gSystem->GetPathInfo(fGroupConfigFile, st)) { - Error("ReadGroupConfig", "could not stat %s", fGroupConfigFile.Data()); - return kFALSE; - } - if (st.fMtime <= fMTimeGroupConfig) { - if (gDebug > 0) - Info("ReadGroupConfig","file has not changed - do nothing"); - return kTRUE; - } - } - - // Either new file or the file has changed - if (cf && (strlen(cf) > 0)) { - // The file must exist and be readable - if (gSystem->GetPathInfo(cf, st)) { - Error("ReadGroupConfig", "could not stat %s", cf); - return kFALSE; - } - if (gSystem->AccessPathName(cf, kReadPermission)) { - Error("ReadGroupConfig", "cannot read %s", cf); - return kFALSE; - } - // Ok - fGroupConfigFile = cf; - fMTimeGroupConfig = st.fMtime; - } - - if (gDebug > 0) - Info("ReadGroupConfig","reading group config from %s", cf); - - // Open the config file - std::ifstream in; - in.open(cf); - if (!in.is_open()) { - Error("ReadGroupConfig", "could not open config file %s", cf); - return kFALSE; - } - - // Container for the global common user - TString tmpCommonUser; - - // Go through - TString line; - while (in.good()) { - // Read new line - line.ReadLine(in); - // Explicitely skip comment lines - if (line[0] == '#') continue; - // Parse it - Ssiz_t from = 0; - TString key; - if (!line.Tokenize(key, from, " ")) // No token - continue; - // Parsing depends on the key - if (key == "property") { - // Read group - TString grp; - if (!line.Tokenize(grp, from, " ")) {// No token - if (gDebug > 0) - Info("ReadGroupConfig","incomplete line: '%s'", line.Data()); - continue; - } - // Read type of property - TString type; - if (!line.Tokenize(type, from, " ")) // No token - continue; - if (type == "diskquota") { - // Read diskquota - TString sdq; - if (!line.Tokenize(sdq, from, " ")) // No token - continue; - // Enforce GigaBytes as default - if (sdq.IsDigit()) sdq += "G"; - Long64_t quota = ToBytes(sdq); - if (quota > -1) { - fGroupQuota.Add(new TObjString(grp), - new TParameter ("group quota", quota)); - } else { - Warning("ReadGroupConfig", - "problems parsing string: wrong or unsupported suffix? %s", - sdq.Data()); - } - } else if (type == "commonuser") { - // Read common user for this group - TString comusr; - if (!line.Tokenize(comusr, from, " ")) // No token - continue; - - } - - } else if (key == "dataset") { - // Read type - TString type; - if (!line.Tokenize(type, from, " ")) {// No token - if (gDebug > 0) - Info("ReadGroupConfig","incomplete line: '%s'", line.Data()); - continue; - } - if (type == "commonuser") { - // Read global common user - TString comusr; - if (!line.Tokenize(comusr, from, " ")) // No token - continue; - fCommonUser = comusr; - } else if (type == "commongroup") { - // Read global common group - TString comgrp; - if (!line.Tokenize(comgrp, from, " ")) // No token - continue; - fCommonGroup = comgrp; - } else if (type == "diskquota") { - // Quota check switch - TString on; - if (!line.Tokenize(on, from, " ")) // No token - continue; - if (on == "on") { - SetBit(TDataSetManager::kCheckQuota); - } else if (on == "off") { - ResetBit(TDataSetManager::kCheckQuota); - } - } - - } else if (key == "averagefilesize") { - - // Read average size - TString avgsize; - if (!line.Tokenize(avgsize, from, " ")) {// No token - if (gDebug > 0) - Info("ReadGroupConfig","incomplete line: '%s'", line.Data()); - continue; - } - Long64_t avgsz = ToBytes(avgsize); - if (avgsz > -1) { - fAvgFileSize = avgsz; - } else { - Warning("ReadGroupConfig", - "problems parsing string: wrong or unsupported suffix? %s", - avgsize.Data()); - } - } else if (key == "include") { - - // Read file to include - TString subfn; - if (!line.Tokenize(subfn, from, " ")) {// No token - if (gDebug > 0) - Info("ReadGroupConfig","incomplete line: '%s'", line.Data()); - continue; - } - // The file must be readable - if (gSystem->AccessPathName(subfn, kReadPermission)) { - Error("ReadGroupConfig", "request to parse file '%s' which is not readable", - subfn.Data()); - continue; - } - if (!ReadGroupConfig(subfn)) - Error("ReadGroupConfig", "problems parsing include file '%s'", subfn.Data()); - } - } - in.close(); - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static utility function to gt the number of bytes from a string -/// representation in the form "````" with `` = {"", "k", "M", "G", -/// "T", "P"} (case insensitive). -/// Returns -1 if the format is wrong. - -Long64_t TDataSetManager::ToBytes(const char *size) -{ - Long64_t lsize = -1; - - // Check if valid - if (!size || strlen(size) <= 0) return lsize; - - TString s(size); - // Determine factor - Long64_t fact = 1; - if (!s.IsDigit()) { - const char *unit[5] = { "k", "M", "G", "T", "P"}; - fact = 1024; - Int_t jj = 0; - while (jj <= 4) { - if (s.EndsWith(unit[jj], TString::kIgnoreCase)) { - s.Remove(s.Length()-1); - break; - } - fact *= 1024; - jj++; - } - } - // Apply factor now - if (s.IsDigit()) - lsize = s.Atoi() * fact; - - // Done - return lsize; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Utility function used in various methods for user dataset upload. - -TFileCollection *TDataSetManager::GetDataSet(const char *, const char *) -{ - AbstractMethod("GetDataSet"); - return (TFileCollection *)0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Removes the indicated dataset - -Bool_t TDataSetManager::RemoveDataSet(const char *) -{ - AbstractMethod("RemoveDataSet"); - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Checks if the indicated dataset exits - -Bool_t TDataSetManager::ExistsDataSet(const char *) -{ - AbstractMethod("ExistsDataSet"); - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Returns all datasets for the `` and `` specified by ``. -/// If `` is 0, it returns all datasets for the given ``. -/// If `` is 0, it returns all datasets. -/// The returned TMap contains: -/// `` --> `` --> `` --> `` (TFileCollection) -/// -/// The unsigned int 'option' is forwarded to GetDataSet and BrowseDataSet. -/// Available options (to be .or.ed): -/// kShowDefault a default selection is shown that include the ones from -/// the current user, the ones from the group and the common ones -/// kPrint print the dataset content -/// kQuotaUpdate update quotas -/// kExport use export naming -/// -/// NB1: options "kPrint", "kQuoatUpdate" and "kExport" are mutually exclusive -/// NB2: for options "kPrint" and "kQuoatUpdate" return is null. - -TMap *TDataSetManager::GetDataSets(const char *, UInt_t) -{ - AbstractMethod("GetDataSets"); - - return (TMap *)0; -} -//////////////////////////////////////////////////////////////////////////////// -/// Scans the dataset indicated by 'uri' following the 'opts' directives -/// -/// The 'opts' string contains up to 4 directive fields separated by ':' -/// -/// 'selection' field : -/// A, allfiles: process all files -/// D, staged: process only staged (on Disk) files (if 'allfiles:' is not specified -/// the default is to process only files marked as non-staged) -/// 'pre-action field': -/// O, open: open the files marked as staged when processing only files -/// marked as non-staged -/// T, touch: open and touch the files marked as staged when processing -/// only files marked as non-staged -/// I, nostagedcheck: do not check the actual stage status on selected files -/// -/// 'process' field: -/// N, noaction: do nothing on the selected files -/// P, fullproc: open the selected files and extract the meta information -/// L, locateonly: only locate the selected files -/// S, stageonly: issue a stage request for the selected files not yet staged -/// -/// 'auxiliary' field -/// V, verbose: notify the actions -/// -/// Returns 0 on success, -1 if any failure occurs. - -Int_t TDataSetManager::ScanDataSet(const char *uri, const char *opts) -{ - // Extract the directives - UInt_t o = 0; - if (opts && strlen(opts) > 0) { - // Selection options - if (strstr(opts, "allfiles:") || strchr(opts, 'A')) - o |= kAllFiles; - else if (strstr(opts, "staged:") || strchr(opts, 'D')) - o |= kStagedFiles; - // Pre-action options - if (strstr(opts, "open:") || strchr(opts, 'O')) - o |= kReopen; - if (strstr(opts, "touch:") || strchr(opts, 'T')) - o |= kTouch; - if (strstr(opts, "nostagedcheck:") || strchr(opts, 'I')) - o |= kNoStagedCheck; - // Process options - if (strstr(opts, "noaction:") || strchr(opts, 'N')) - o |= kNoAction; - if (strstr(opts, "locateonly:") || strchr(opts, 'L')) - o |= kLocateOnly; - if (strstr(opts, "stageonly:") || strchr(opts, 'S')) - o |= kStageOnly; - // Auxilliary options - if (strstr(opts, "verbose:") || strchr(opts, 'V')) - o |= kDebug; - } else { - // Default - o = kReopen | kDebug; - } - - // Run - return ScanDataSet(uri, o); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Scans the dataset indicated by `` and returns the number of missing files. -/// Returns -1 if any failure occurs. -/// For more details, see documentation of -/// ScanDataSet(TFileCollection *dataset, const char *option) - -Int_t TDataSetManager::ScanDataSet(const char *, UInt_t) -{ - AbstractMethod("ScanDataSet"); - - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Gets quota information from this dataset - -void TDataSetManager::GetQuota(const char *group, const char *user, - const char *dsName, TFileCollection *dataset) -{ - if (gDebug > 0) - Info("GetQuota", "processing dataset %s %s %s", group, user, dsName); - - if (dataset->GetTotalSize() > 0) { - TParameter *size = - dynamic_cast*> (fGroupUsed.GetValue(group)); - if (!size) { - size = new TParameter ("group used", 0); - fGroupUsed.Add(new TObjString(group), size); - } - - size->SetVal(size->GetVal() + dataset->GetTotalSize()); - - TMap *userMap = dynamic_cast (fUserUsed.GetValue(group)); - if (!userMap) { - userMap = new TMap; - fUserUsed.Add(new TObjString(group), userMap); - } - - size = dynamic_cast*> (userMap->GetValue(user)); - if (!size) { - size = new TParameter ("user used", 0); - userMap->Add(new TObjString(user), size); - } - - size->SetVal(size->GetVal() + dataset->GetTotalSize()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display quota information - -void TDataSetManager::ShowQuota(const char *opt) -{ - UpdateUsedSpace(); - - TMap *groupQuotaMap = GetGroupQuotaMap(); - TMap *userUsedMap = GetUserUsedMap(); - if (!groupQuotaMap || !userUsedMap) - return; - - Bool_t noInfo = kTRUE; - TIter iter(groupQuotaMap); - TObjString *group = 0; - while ((group = dynamic_cast (iter.Next()))) { - noInfo = kFALSE; - Long64_t groupQuota = GetGroupQuota(group->String()); - Long64_t groupUsed = GetGroupUsed(group->String()); - - Printf(" +++ Group %s uses %.1f GB out of %.1f GB", group->String().Data(), - (Float_t) groupUsed / DSM_ONE_GB, - (Float_t) groupQuota / DSM_ONE_GB); - - // display also user information - if (opt && !TString(opt).Contains("U", TString::kIgnoreCase)) - continue; - - TMap *userMap = dynamic_cast (userUsedMap->GetValue(group->String())); - if (!userMap) - continue; - - TIter iter2(userMap); - TObjString *user = 0; - while ((user = dynamic_cast (iter2.Next()))) { - TParameter *size2 = - dynamic_cast*> (userMap->GetValue(user->String().Data())); - if (!size2) - continue; - - Printf(" +++ User %s uses %.1f GB", user->String().Data(), - (Float_t) size2->GetVal() / DSM_ONE_GB); - } - - Printf("------------------------------------------------------"); - } - // Check if something has been printed - if (noInfo) { - Printf(" +++ Quota check enabled but no quota info available +++ "); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Prints the quota - -void TDataSetManager::PrintUsedSpace() -{ - Info("PrintUsedSpace", "listing used space"); - - TIter iter(&fUserUsed); - TObjString *group = 0; - while ((group = dynamic_cast (iter.Next()))) { - TMap *userMap = dynamic_cast (fUserUsed.GetValue(group->String())); - - TParameter *size = - dynamic_cast*> (fGroupUsed.GetValue(group->String())); - - if (userMap && size) { - Printf("Group %s: %lld B = %.2f GB", group->String().Data(), size->GetVal(), - (Float_t) size->GetVal() / DSM_ONE_GB); - - TIter iter2(userMap); - TObjString *user = 0; - while ((user = dynamic_cast (iter2.Next()))) { - TParameter *size2 = - dynamic_cast*> (userMap->GetValue(user->String().Data())); - if (size2) - Printf(" User %s: %lld B = %.2f GB", user->String().Data(), size2->GetVal(), - (Float_t) size2->GetVal() / DSM_ONE_GB); - } - - Printf("------------------------------------------------------"); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Log info to the monitoring server - -void TDataSetManager::MonitorUsedSpace(TVirtualMonitoringWriter *monitoring) -{ - Info("MonitorUsedSpace", "sending used space to monitoring server"); - - TIter iter(&fUserUsed); - TObjString *group = 0; - while ((group = dynamic_cast (iter.Next()))) { - TMap *userMap = dynamic_cast (fUserUsed.GetValue(group->String())); - TParameter *size = - dynamic_cast*> (fGroupUsed.GetValue(group->String())); - - if (!userMap || !size) - continue; - - TList *list = new TList; - list->SetOwner(); - list->Add(new TParameter("_TOTAL_", size->GetVal())); - Long64_t groupQuota = GetGroupQuota(group->String()); - if (groupQuota != -1) - list->Add(new TParameter("_QUOTA_", groupQuota)); - - TIter iter2(userMap); - TObjString *user = 0; - while ((user = dynamic_cast (iter2.Next()))) { - TParameter *size2 = - dynamic_cast*> (userMap->GetValue(user->String().Data())); - if (!size2) - continue; - list->Add(new TParameter(user->String().Data(), size2->GetVal())); - } - - if (!monitoring->SendParameters(list, group->String())) - Warning("MonitorUsedSpace", "problems sending monitoring parameters"); - delete list; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Returns the used space of that group - -Long64_t TDataSetManager::GetGroupUsed(const char *group) -{ - if (fgCommonDataSetTag == group) - group = fCommonGroup; - - TParameter *size = - dynamic_cast*> (fGroupUsed.GetValue(group)); - if (!size) { - if (gDebug > 0) - Info("GetGroupUsed", "group %s not found", group); - return 0; - } - - return size->GetVal(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// returns the quota a group is allowed to have - -Long64_t TDataSetManager::GetGroupQuota(const char *group) -{ - if (fgCommonDataSetTag == group) - group = fCommonGroup; - - TParameter *value = - dynamic_cast*> (fGroupQuota.GetValue(group)); - if (!value) { - if (gDebug > 0) - Info("GetGroupQuota", "group %s not found", group); - return 0; - } - return value->GetVal(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// updates the used space maps - -void TDataSetManager::UpdateUsedSpace() -{ - AbstractMethod("UpdateUsedSpace"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Register a dataset, perfoming quota checkings, if needed. -/// Returns 0 on success, -1 on failure - -Int_t TDataSetManager::RegisterDataSet(const char *, - TFileCollection *, const char *) -{ - AbstractMethod("RegisterDataSet"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save into the `/dataset.list` file the name of the last updated -/// or created or modified dataset -/// Returns 0 on success, -1 on error - -Int_t TDataSetManager::NotifyUpdate(const char * /*group*/, - const char * /*user*/, - const char * /*dspath*/, - Long_t /*mtime*/, - const char * /*checksum*/) -{ - AbstractMethod("NotifyUpdate"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear cached information matching uri - -Int_t TDataSetManager::ClearCache(const char * /*uri*/) -{ - AbstractMethod("ClearCache"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show cached information matching uri - -Int_t TDataSetManager::ShowCache(const char * /*uri*/) -{ - AbstractMethod("ShowCache"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Creates URI for the dataset manger in the form '[[/dsGroup/]dsUser/]dsName[#%dsObjPath]', -/// The optional dsObjPath can be in the form [subdir/]objname]'. - -TString TDataSetManager::CreateUri(const char *dsGroup, const char *dsUser, - const char *dsName, const char *dsObjPath) -{ - TString uri; - - if (dsGroup && strlen(dsGroup) > 0) { - if (dsUser && strlen(dsUser) > 0) { - uri += Form("/%s/%s/", dsGroup, dsUser); - } else { - uri += Form("/%s/*/", dsGroup); - } - } else if (dsUser && strlen(dsUser) > 0) { - uri += Form("%s/", dsUser); - } - if (dsName && strlen(dsName) > 0) - uri += dsName; - if (dsObjPath && strlen(dsObjPath) > 0) - uri += Form("#%s", dsObjPath); - - // Done - return uri; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Parses a (relative) URI that describes a DataSet on the cluster. -/// The input 'uri' should be in the form '[[/group/]user/]dsname[#[subdir/]objname]', -/// where 'objname' is the name of the object (e.g. the tree name) and the 'subdir' -/// is the directory in the file wher it should be looked for. -/// After resolving against a base URI consisting of proof://masterhost/group/user/ -/// - meaning masterhost, group and user of the current session - -/// the path is checked to contain exactly three elements separated by '/': -/// group/user/dsname -/// If wildcards, '*' is allowed in group and user and dsname is allowed to be empty. -/// If onlyCurrent, only group and user of current session are allowed. -/// Only non-null parameters are filled by this function. -/// Returns kTRUE in case of success. - -Bool_t TDataSetManager::ParseUri(const char *uri, - TString *dsGroup, TString *dsUser, - TString *dsName, TString *dsTree, - Bool_t onlyCurrent, Bool_t wildcards) -{ - TString uristr(uri); - - // If URI contains fields in the form "Field=Value;" it is a virtual URI and - // should be treated differently - if ((uristr.Index('=') >= 0) && (uristr.Index(';') >= 0)) { - - // URI is composed of two parts: a name (dsName), and the tree after the - // pound sign - - Warning("ParseUri", - "Dataset URI looks like a virtual URI, treating it as such. " - "No group and user will be parsed!"); - - TPMERegexp reVirtualUri("^([^#]+)(#(.*))?$"); - Int_t nm = reVirtualUri.Match(uristr); - - if (nm >= 2) { - if (dsGroup) *dsGroup = ""; - if (dsUser) *dsUser = ""; - if (dsName) *dsName = reVirtualUri[1]; - if (dsTree) { - if (nm == 4) *dsTree = reVirtualUri[3]; - else *dsTree = ""; - } - } - else return kFALSE; // should never happen! - - return kTRUE; - } - - // Append trailing slash if missing when wildcards are enabled - Int_t pc = 0; - if (wildcards && uristr.Length() > 0) { - pc = uristr.CountChar('/'); - Bool_t endsl = uristr.EndsWith("/") ? kTRUE : kFALSE; - Bool_t beginsl = uristr.BeginsWith("/") ? kTRUE : kFALSE; - if (beginsl) { - if (pc == 1) uristr += "/*/"; - if (pc == 2 && endsl) uristr += "*/"; - if (pc == 2 && !endsl) uristr += "/"; - } - } - - // Resolve given URI agains the base - TUri resolved = TUri::Transform(uristr, fBase); - if (resolved.HasQuery()) - Info ("ParseUri", "URI query part <%s> ignored", resolved.GetQuery().Data()); - - TString path(resolved.GetPath()); - // Must be in the form /group/user/dsname - if ((pc = path.CountChar('/')) != 3) { - if (!TestBit(TDataSetManager::kIsSandbox)) { - Error ("ParseUri", "illegal dataset path: '%s'", uri); - return kFALSE; - } else if (pc >= 0 && pc < 3) { - // Add missing slashes - TString sls("/"); - if (pc == 2) { - sls = "/"; - } else if (pc == 1) { - sls.Form("/%s/", fGroup.Data()); - } else if (pc == 0) { - sls.Form("/%s/%s/", fGroup.Data(), fUser.Data()); - } - path.Insert(0, sls); - } - } - if (gDebug > 1) - Info("ParseUri", "path: '%s'", path.Data()); - - // Get individual values from tokens - Int_t from = 1; - TString group, user, name; - if (path.Tokenize(group, from, "/")) { - if (path.Tokenize(user, from, "/")) { - if (!path.Tokenize(name, from, "/")) - if (gDebug > 0) Info("ParseUri", "'name' missing"); - } else { - if (gDebug > 0) Info("ParseUri", "'user' missing"); - } - } else { - if (gDebug > 1) Info("ParseUri", "'group' missing"); - } - - // The fragment may contain the subdir and the object name in the form '[subdir/]objname' - TString tree = resolved.GetFragment(); - if (tree.EndsWith("/")) - tree.Remove(tree.Length()-1); - - if (gDebug > 1) - Info("ParseUri", "group: '%s', user: '%s', dsname:'%s', seg: '%s'", - group.Data(), user.Data(), name.Data(), tree.Data()); - - // Check for unwanted use of wildcards - if ((user == "*" || group == "*") && !wildcards) { - Error ("ParseUri", "no wildcards allowed for user/group in this context (uri: '%s')", uri); - return kFALSE; - } - - // dsname may only be empty if wildcards expected - if (name.IsNull() && !wildcards) { - Error ("ParseUri", "DataSet name is empty"); - return kFALSE; - } - - // Construct regexp whitelist for checking illegal characters in user/group - TPRegexp wcExp (wildcards ? "^(?:[A-Za-z0-9-*_.]*|[*])$" : "^[A-Za-z0-9-_.]*$"); - - // Check for illegal characters in all components - if (!wcExp.Match(group)) { - Error("ParseUri", "illegal characters in group (uri: '%s', group: '%s')", uri, group.Data()); - return kFALSE; - } - - if (!wcExp.Match(user)) { - Error("ParseUri", "illegal characters in user (uri: '%s', user: '%s')", uri, user.Data()); - return kFALSE; - } - - // Construct regexp whitelist for checking illegal characters in name - if (!wcExp.Match(name)) { - Error("ParseUri", "illegal characters in name (uri: '%s', name: '%s')", uri, name.Data()); - return kFALSE; - } - - if (tree.Contains(TRegexp("[^A-Za-z0-9-/_]"))) { - Error("ParseUri", "Illegal characters in subdir/object name (uri: '%s', obj: '%s')", uri, tree.Data()); - return kFALSE; - } - - // Check user & group - if (onlyCurrent && (group.CompareTo(fGroup) || user.CompareTo(fUser))) { - Error("ParseUri", "only datasets from your group/user allowed"); - return kFALSE; - } - - // fill parameters passed by reference, if defined - if (dsGroup) - *dsGroup = group; - if (dsUser) - *dsUser = user; - if (dsName) - *dsName = name; - if (dsTree) - *dsTree = tree; - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Partition dataset 'ds' accordingly to the servers. -/// The returned TMap contains: -/// `` --> `` (TFileCollection) -/// where `` is the subset of 'ds' on `` -/// The partitioning is done using all the URLs in the TFileInfo's, so the -/// resulting datasets are not mutually exclusive. -/// The string 'exclude' contains a comma-separated list of servers to exclude -/// from the map. - -TMap *TDataSetManager::GetSubDataSets(const char *ds, const char *exclude) -{ - TMap *map = (TMap *)0; - - if (!ds || strlen(ds) <= 0) { - Info("GetDataSets", "dataset name undefined!"); - return map; - } - - // Get the dataset - TFileCollection *fc = GetDataSet(ds); - if (!fc) { - Info("GetDataSets", "could not retrieve the dataset '%s'", ds); - return map; - } - - // Get the subset - if (!(map = fc->GetFilesPerServer(exclude))) { - if (gDebug > 0) - Info("GetDataSets", "could not get map for '%s'", ds); - } - - // Cleanup - delete fc; - - // Done - return map; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Formatted printout of the content of TFileCollection 'fc'. -/// Options in the form -/// popt = u * 10 + f -/// f 0 => header only, 1 => header + files -/// when printing files -/// u 0 => print file name only, 1 => print full URL - -void TDataSetManager::PrintDataSet(TFileCollection *fc, Int_t popt) -{ - if (!fc) return; - - Int_t f = popt%10; - Int_t u = popt - 10 * f; - - Printf("+++"); - if (fc->GetTitle() && (strlen(fc->GetTitle()) > 0)) { - Printf("+++ Dumping: %s: ", fc->GetTitle()); - } else { - Printf("+++ Dumping: %s: ", fc->GetName()); - } - Printf("%s", fc->ExportInfo("+++ Summary:", 1)->GetName()); - if (f == 1) { - Printf("+++ Files:"); - Int_t nf = 0; - TIter nxfi(fc->GetList()); - TFileInfo *fi = 0; - while ((fi = (TFileInfo *)nxfi())) { - if (u == 1) - Printf("+++ %5d. %s", ++nf, fi->GetCurrentUrl()->GetUrl()); - else - Printf("+++ %5d. %s", ++nf, fi->GetCurrentUrl()->GetFile()); - } - } - Printf("+++"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Prints formatted information about the dataset 'uri'. -/// The type and format of output is driven by 'opt': -/// -/// 1. opt = "server:srv1[,srv2[,srv3[,...]]]" -/// Print info about the subsets of 'uri' on servers srv1, srv2, ... -/// 2. opt = "servers[:exclude:srv1[,srv2[,srv3[,...]]]]" -/// Print info about the subsets of 'uri' on all servers, except -/// the ones in the exclude list srv1, srv2, ... -/// 3. opt = `` -/// Print info about all datasets matching 'uri' -/// -/// If 'opt' contains 'full:' the list of files in the datasets are also printed. -/// In case 3. this is enabled only if 'uri' matches a single dataset. -/// -/// In case 3, if 'opt' contains -/// 'full:' the list of files in the datasets are also printed. -/// 'forcescan:' the dataset are open to get the information; otherwise the -/// pre-processed information is used. -/// 'noheader:' the labelling header is not printed; usefull when to chain -/// several printouts -/// 'noupdate:' do not update the cache (which may be slow on very remote -/// servers) -/// 'refresh:' refresh the information (requires appropriate credentials; -/// typically it can be done only for owned datasets) - -void TDataSetManager::ShowDataSets(const char *uri, const char *opt) -{ - TFileCollection *fc = 0; - TString o(opt); - Int_t popt = 0; - if (o.Contains("full:")) { - o.ReplaceAll("full:",""); - popt = 1; - } - if (o.BeginsWith("server:")) { - o.ReplaceAll("server:", ""); - TString srv; - Int_t from = 0; - while ((o.Tokenize(srv, from, ","))) { - fc = GetDataSet(uri, srv.Data()); - PrintDataSet(fc, popt); - delete fc; - } - } else if (o.BeginsWith("servers")) { - o.ReplaceAll("servers", ""); - if (o.BeginsWith(":exclude:")) - o.ReplaceAll(":exclude:", ""); - else - o = ""; - TMap *dsmap = GetSubDataSets(uri, o.Data()); - if (dsmap) { - TIter nxk(dsmap); - TObject *k = 0; - while ((k = nxk()) && (fc = (TFileCollection *) dsmap->GetValue(k))) { - PrintDataSet(fc, popt); - } - delete dsmap; - } - } else { - TString u(uri), grp, usr, dsn; - // Support for "*" or "/*" - if (u == "" || u == "*" || u == "/*" || u == "/*/" || u == "/*/*") u = "/*/*/"; - if (!ParseUri(u.Data(), &grp, &usr, &dsn, 0, kFALSE, kTRUE)) - Warning("ShowDataSets", "problems parsing URI '%s'", uri); - // Scan the existing datasets and print the content - UInt_t xopt = (UInt_t)(TDataSetManager::kPrint); - if (o.Contains("forcescan:")) xopt |= (UInt_t)(TDataSetManager::kForceScan); - if (o.Contains("noheader:")) xopt |= (UInt_t)(TDataSetManager::kNoHeaderPrint); - if (o.Contains("noupdate:")) xopt |= (UInt_t)(TDataSetManager::kNoCacheUpdate); - if (o.Contains("refresh:")) xopt |= (UInt_t)(TDataSetManager::kRefreshLs); - if (!u.IsNull() && !u.Contains("*") && !grp.IsNull() && !usr.IsNull() && !dsn.IsNull()) { - if (ExistsDataSet(uri)) { - // Single dataset - if (popt == 0) { - // Quick listing - GetDataSets(u.Data(), xopt); - } else if ((fc = GetDataSet(uri))) { - // Full print option - PrintDataSet(fc, 10 + popt); - delete fc; - } - return; - } - // Try all the directories - TRegexp reg(grp, kTRUE), reu(usr, kTRUE); - if (u.Index(reg) == kNPOS) grp = "*"; - if (u.Index(reu) == kNPOS) usr = "*"; - // Rebuild the uri - u.Form("/%s/%s/%s", grp.Data(), usr.Data(), dsn.Data()); - } - GetDataSets(u.Data(), xopt); - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Go through the files in the specified dataset, selecting files according to -/// 'fopt' and doing on these files the actions described by 'sopt'. -/// If required, the information in 'dataset' is updated. -/// -/// The int fopt controls which files have to be processed (or added to the list -/// if ropt is 1 - see below); 'fopt' is defined in term of csopt and fsopt: -/// fopt = sign(fsopt) * csopt * 100 + fsopt -/// where 'fsopt' controls the actual selection -/// -1 all files in the dataset -/// 0 process only files marked as 'non-staged' -/// >=1 as 0 but files that are marked 'staged' are open -/// >=2 as 1 but files that are marked 'staged' are touched -/// 10 process only files marked as 'staged'; files marked as 'non-staged' -/// are ignored -/// and 'csopt' controls if an actual check on the staged status (via TFileStager) is done -/// 0 check that the file is staged using TFileStager -/// 1 do not hard check the staged status -/// (example: use fopt = -101 to check the staged status of all the files, or fopt = 110 -/// to re-check the stage status of all the files marked as staged) -/// -/// If 'dbg' is true, some information about the ongoing operations is reguraly -/// printed; this can be useful when processing very large datasets, an operation -/// which can take a very long time. -/// -/// The int 'sopt' controls what is done on the selected files (this is effective only -/// if ropt is 0 or 2 - see below): -/// -1 no action (fopt = 2 and sopt = -1 touches all staged files) -/// 0 do the full process: open the files and fill the meta-information -/// in the TFileInfo object, including the end-point URL -/// 1 only locate the files, by updating the end-point URL (uses TFileStager::Locate -/// which is must faster of an TFile::Open) -/// 2 issue a stage request on the files -/// -/// The int 'ropt' controls which actions are performed: -/// 0 do the full process: get list of files to process and process them -/// 1 get the list of files to be scanned and return it in flist -/// 2 process the files in flist (according to sopt) -/// When defined flist is under the responsability the caller. -/// -/// If avgsz > 0 it is used for the final update of the dataset global counters. -/// -/// If 'mss' is defined use it to initialize the stager (instead of the Url in the -/// TFileInfo objects) -/// -/// If maxfiles > 0, select for processing a maximum of 'filesmax' files (but if fopt is 1 or 2 -/// all files marked as 'staged' are still open or touched) -/// -/// Return code -/// 1 dataset was not changed -/// 2 dataset was changed -/// -/// The number of touched, opened and disappeared files are returned in the respective -/// variables, if these are defined. - -Int_t TDataSetManager::ScanDataSet(TFileCollection *dataset, - Int_t fopt, Int_t sopt, Int_t ropt, Bool_t dbg, - Int_t *touched, Int_t *opened, Int_t *disappeared, - TList *flist, Long64_t avgsz, const char *mss, - Int_t maxfiles, const char *stageopts) -{ - // Max number of files - if (maxfiles > -1 && dbg) - ::Info("TDataSetManager::ScanDataSet", "processing a maximum of %d files", maxfiles); - - // File selection, Reopen and Touch options - Bool_t checkstg = (fopt >= 100 || fopt < -1) ? kFALSE : kTRUE; - - // File processing options - Bool_t noaction = (sopt == -1) ? kTRUE : kFALSE; - //Bool_t fullproc = (sopt == 0) ? kTRUE : kFALSE; - Bool_t locateonly = (sopt == 1) ? kTRUE : kFALSE; - Bool_t stageonly = (sopt == 2) ? kTRUE : kFALSE; - - // Run options - Bool_t doall = (ropt == 0) ? kTRUE : kFALSE; - Bool_t getlistonly = (ropt == 1) ? kTRUE : kFALSE; - Bool_t scanlist = (ropt == 2) ? kTRUE : kFALSE; - - if (scanlist && !flist) { - ::Error("TDataSetManager::ScanDataSet", "input list is mandatory for option 'scan file list'"); - return -1; - } - - Int_t ftouched = 0; - Int_t fopened = 0; - Int_t fdisappeared = 0; - - Bool_t bchanged_ds = kFALSE; - - TList *newStagedFiles = 0; - TFileInfo *fileInfo = 0; - TFileStager *stager = 0; - Bool_t createStager = kFALSE; - - if (doall || getlistonly) { - - // Point to the list - newStagedFiles = (!doall && getlistonly && flist) ? flist : new TList; - if (newStagedFiles != flist) newStagedFiles->SetOwner(kFALSE); - - stager = (mss && strlen(mss) > 0) ? TFileStager::Open(mss) : 0; - createStager = (stager) ? kFALSE : kTRUE; - - Bool_t bchanged_fi = kFALSE; - Bool_t btouched = kFALSE; - Bool_t bdisappeared = kFALSE; - - // Check which files have been staged, this can be replaced by a bulk command, - // once it exists in the xrdclient - TIter iter(dataset->GetList()); - while ((fileInfo = (TFileInfo *) iter())) { - - // For real time monitoring - gSystem->DispatchOneEvent(kTRUE); - - bchanged_fi = kFALSE; - btouched = kFALSE; - bdisappeared = kFALSE; - Bool_t newlystaged = CheckStagedStatus(fileInfo, fopt, maxfiles, newStagedFiles->GetEntries(), - stager, createStager, dbg, bchanged_fi, btouched, - bdisappeared); - - if (bchanged_fi) bchanged_ds = kTRUE; - if (btouched) ftouched++; - if (bdisappeared) fdisappeared++; - - // Notify - if (dbg && (ftouched+fdisappeared) % 100 == 0) - ::Info("TDataSetManager::ScanDataSet", "opening %d: file: %s", - ftouched + fdisappeared, fileInfo->GetCurrentUrl()->GetUrl()); - - // Register the newly staged file - if (!noaction && newlystaged) newStagedFiles->Add(fileInfo); - } - SafeDelete(stager); - - // If required to only get the list we are done - if (getlistonly) { - if (dbg && newStagedFiles->GetEntries() > 0) - ::Info("TDataSetManager::ScanDataSet", " %d files appear to be newly staged", - newStagedFiles->GetEntries()); - if (!flist) SafeDelete(newStagedFiles); - return ((bchanged_ds) ? 2 : 1); - } - } - - if (!noaction && (doall || scanlist)) { - - // Point to the list - newStagedFiles = (!doall && scanlist && flist) ? flist : newStagedFiles; - if (newStagedFiles != flist) newStagedFiles->SetOwner(kFALSE); - - // loop over now staged files - if (dbg && newStagedFiles->GetEntries() > 0) - ::Info("TDataSetManager::ScanDataSet", "opening %d files that appear to be newly staged", - newStagedFiles->GetEntries()); - - // If staging files, prepare the stager - if (locateonly || stageonly) { - stager = (mss && strlen(mss) > 0) ? TFileStager::Open(mss) : 0; - createStager = (stager) ? kFALSE : kTRUE; - } - - // Notify each 'fqnot' files (min 1, max 100) - Int_t fqnot = (newStagedFiles->GetSize() > 10) ? newStagedFiles->GetSize() / 10 : 1; - if (fqnot > 100) fqnot = 100; - Int_t count = 0; - Bool_t bchanged_fi = kFALSE; - Bool_t bopened = kFALSE; - TIter iter(newStagedFiles); - while ((fileInfo = (TFileInfo *) iter())) { - - if (dbg && (count%fqnot == 0)) - ::Info("TDataSetManager::ScanDataSet", "processing %d.'new' file: %s", - count, fileInfo->GetCurrentUrl()->GetUrl()); - count++; - - // For real time monitoring - gSystem->DispatchOneEvent(kTRUE); - bchanged_fi = kFALSE; - bopened = kFALSE; - - ProcessFile(fileInfo, sopt, checkstg, doall, stager, createStager, - stageopts, dbg, bchanged_fi, bopened); - - bchanged_ds |= bchanged_fi; - if (bopened) fopened++; - } - if (newStagedFiles != flist) SafeDelete(newStagedFiles); - - dataset->RemoveDuplicates(); - dataset->Update(avgsz); - } - - Int_t result = (bchanged_ds) ? 2 : 1; - if (result > 0 && dbg) - ::Info("TDataSetManager::ScanDataSet", "%d files 'new'; %d files touched;" - " %d files disappeared", fopened, ftouched, fdisappeared); - - // Fill outputs, if required - if (touched) *touched = ftouched; - if (opened) *opened = fopened; - if (disappeared) *disappeared = fdisappeared; - - // For real time monitoring - gSystem->DispatchOneEvent(kTRUE); - - return result; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check stage status of the file described by "fileInfo". -/// fopt is same as "fopt" in TDataSetManager::ScanDataSet, which is repeated below: -/// The int fopt controls which files have to be processed (or added to the list -/// if ropt is 1 - see below); 'fopt' is defined in term of csopt and fsopt: -/// fopt = sign(fsopt) * csopt * 100 + fsopt -/// where 'fsopt' controls the actual selection -/// -1 all files in the dataset -/// 0 process only files marked as 'non-staged' -/// >=1 as 0 but files that are marked 'staged' are open -/// >=2 as 1 but files that are marked 'staged' are touched -/// 10 process only files marked as 'staged'; files marked as 'non-staged' -/// are ignored -/// and 'csopt' controls if an actual check on the staged status (via TFileStager) is done -/// 0 check that the file is staged using TFileStager -/// 1 do not hard check the staged status -/// (example: use fopt = -101 to check the staged status of all the files, or fopt = 110 -/// to re-check the stage status of all the files marked as staged) -/// -/// If 'dbg' is true, some information about the ongoing operations is reguraly -/// printed; this can be useful when processing very large datasets, an operation -/// which can take a very long time. -/// -/// If maxfiles > 0, select for processing a maximum of 'filesmax' files (but if fopt is 1 or 2 -/// all files marked as 'staged' are still open or touched) -/// -/// Return code -/// kTRUE the file appears newly staged -/// kFALSE otherwise -/// -/// changed is true if the fileinfo is modified -/// touched is true if the file is open and read -/// disappeared is true if the file is marked staged but actually not staged - -Bool_t TDataSetManager::CheckStagedStatus(TFileInfo *fileInfo, Int_t fopt, Int_t maxfiles, - Int_t newstagedfiles, TFileStager* stager, - Bool_t createStager, Bool_t dbg, Bool_t& changed, - Bool_t& touched, Bool_t& disappeared) -{ - // File selection, Reopen and Touch options - Bool_t allf = (fopt == -1) ? kTRUE : kFALSE; - Bool_t checkstg = (fopt >= 100 || fopt < -1) ? kFALSE : kTRUE; - if (fopt >= 0) fopt %= 100; - Bool_t nonstgf = (fopt >= 0 && fopt < 10) ? kTRUE : kFALSE; - Bool_t reopen = (fopt >= 1 && fopt < 10) ? kTRUE : kFALSE; - Bool_t touch = (fopt >= 2 && fopt < 10) ? kTRUE : kFALSE; - Bool_t stgf = (fopt == 10) ? kTRUE : kFALSE; - - changed = kFALSE; - touched = kFALSE; - disappeared = kFALSE; - - // Check which files have been staged, this can be replaced by a bulk command, - // once it exists in the xrdclient - - if (!allf) { - - fileInfo->ResetUrl(); - if (!fileInfo->GetCurrentUrl()) { - ::Error("TDataSetManager::CheckStagedStatus", "GetCurrentUrl() returned 0 for %s", - fileInfo->GetFirstUrl()->GetUrl()); - return kFALSE; - } - - if (nonstgf && fileInfo->TestBit(TFileInfo::kStaged)) { - - // Skip files flagged as corrupted - if (fileInfo->TestBit(TFileInfo::kCorrupted)) return kFALSE; - - // Skip if we are not asked to re-open the staged files - if (!reopen) return kFALSE; - - // Set the URL removing the anchor (e.g. #AliESDs.root) because IsStaged() - // and TFile::Open() with filetype=raw do not accept anchors - TUrl *curl = fileInfo->GetCurrentUrl(); - const char *furl = curl->GetUrl(); - TString urlmod; - if (TDataSetManager::CheckDataSetSrvMaps(curl, urlmod) && !(urlmod.IsNull())) - furl = urlmod.Data(); - TUrl url(furl); - url.SetAnchor(""); - - // Check if file is still available, if touch is set actually read from the file - TString uopt(url.GetOptions()); - uopt += "filetype=raw&mxredir=2"; - url.SetOptions(uopt.Data()); - TFile *file = TFile::Open(url.GetUrl()); - if (file) { - if (touch) { - // Actually access the file - char tmpChar = 0; - if (file->ReadBuffer(&tmpChar, 1)) - ::Warning("TDataSetManager::CheckStagedStatus", "problems reading 1 byte from open file"); - // Count - touched = kTRUE; - } - file->Close(); - delete file; - } else { - // File could not be opened, reset staged bit - if (dbg) ::Info("TDataSetManager::CheckStagedStatus", "file %s disappeared", url.GetUrl()); - fileInfo->ResetBit(TFileInfo::kStaged); - disappeared = kTRUE; - changed = kTRUE; - - // Remove invalid URL, if other one left... - if (fileInfo->GetNUrls() > 1) - fileInfo->RemoveUrl(curl->GetUrl()); - } - // Go to next - return kFALSE; - } else if (stgf && !(fileInfo->TestBit(TFileInfo::kStaged))) { - // All staged files are processed: skip non staged - return kFALSE; - } - } - - // Only open maximum number of 'new' files - if (maxfiles > 0 && newstagedfiles >= maxfiles) - return kFALSE; - - // Hard check of the staged status, if required - if (checkstg) { - // Set the URL removing the anchor (e.g. #AliESDs.root) because IsStaged() - // and TFile::Open() with filetype=raw do not accept anchors - TUrl *curl = fileInfo->GetCurrentUrl(); - const char *furl = curl->GetUrl(); - TString urlmod; - Bool_t mapped = kFALSE; - if (TDataSetManager::CheckDataSetSrvMaps(curl, urlmod) && !(urlmod.IsNull())) { - furl = urlmod.Data(); - mapped = kTRUE; - } - TUrl url(furl); - url.SetAnchor(""); - - // Get the stager (either the global one or from the URL) - stager = createStager ? TFileStager::Open(url.GetUrl()) : stager; - - Bool_t result = kFALSE; - if (stager) { - result = stager->IsStaged(url.GetUrl()); - if (gDebug > 0) - ::Info("TDataSetManager::CheckStagedStatus", "IsStaged: %s: %d", url.GetUrl(), result); - if (createStager) - SafeDelete(stager); - } else { - ::Warning("TDataSetManager::CheckStagedStatus", - "could not get stager instance for '%s'", url.GetUrl()); - } - - // Go to next in case of failure - if (!result) { - if (fileInfo->TestBit(TFileInfo::kStaged)) { - // Reset the bit - fileInfo->ResetBit(TFileInfo::kStaged); - changed = kTRUE; - } - return kFALSE; - } else { - if (!(fileInfo->TestBit(TFileInfo::kStaged))) { - // Set the bit - fileInfo->SetBit(TFileInfo::kStaged); - changed = kTRUE; - } - } - - // If the url was re-mapped add the new url in front of the list - if (mapped) { - url.SetOptions(curl->GetOptions()); - url.SetAnchor(curl->GetAnchor()); - fileInfo->AddUrl(url.GetUrl(), kTRUE); - } - } - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Locate, stage, or fully validate file "fileInfo". - -void TDataSetManager::ProcessFile(TFileInfo *fileInfo, Int_t sopt, Bool_t checkstg, Bool_t doall, - TFileStager* stager, Bool_t createStager, const char *stageopts, - Bool_t dbg, Bool_t& changed, Bool_t& opened) -{ - // File processing options - //Bool_t noaction = (sopt == -1) ? kTRUE : kFALSE; - Bool_t fullproc = (sopt == 0) ? kTRUE : kFALSE; - Bool_t locateonly = (sopt == 1) ? kTRUE : kFALSE; - Bool_t stageonly = (sopt == 2) ? kTRUE : kFALSE; - - changed = kFALSE; - opened = kFALSE; - Int_t rc = -1; - - // Set the URL removing the anchor (e.g. #AliESDs.root) because IsStaged() - // and TFile::Open() with filetype=raw do not accept anchors - TUrl *curl = fileInfo->GetCurrentUrl(); - const char *furl = curl->GetUrl(); - TString urlmod; - //Bool_t mapped = kFALSE; - if (TDataSetManager::CheckDataSetSrvMaps(curl, urlmod) && !(urlmod.IsNull())) { - furl = urlmod.Data(); - //mapped = kTRUE; - } - TUrl url(furl); - url.SetOptions(""); - url.SetAnchor(""); - - if (createStager){ - if (!stager || (stager && !stager->Matches(url.GetUrl()))) { - SafeDelete(stager); - if (!(stager = TFileStager::Open(url.GetUrl())) || !(stager->IsValid())) { - ::Error("TDataSetManager::ProcessFile", - "could not get valid stager instance for '%s'", url.GetUrl()); - return; - } - } - } - // Locate the file, if just requested so - if (locateonly) { - TString eurl; - if (stager && stager->Locate(url.GetUrl(), eurl) == 0) { - TString opts(curl->GetOptions()); - TString anch(curl->GetAnchor()); - // Get the effective end-point Url - curl->SetUrl(eurl); - // Restore original options and anchor, if any - curl->SetOptions(opts); - curl->SetAnchor(anch); - // Flag and count - changed = kTRUE; - opened = kTRUE; - } else { - // Failure - ::Error("TDataSetManager::ProcessFile", "could not locate %s", url.GetUrl()); - } - - } else if (stageonly) { - TString eurl; - if (stager && !(stager->IsStaged(url.GetUrl()))) { - if (!(stager->Stage(url.GetUrl(), stageopts))) { - // Failure - ::Error("TDataSetManager::ProcessFile", - "problems issuing stage request for %s", url.GetUrl()); - } - } - } else if (fullproc) { - TString eurl; - // Full file validation - rc = -2; - Bool_t doscan = kTRUE; - if (checkstg) { - doscan = kFALSE; - if ((doall && fileInfo->TestBit(TFileInfo::kStaged)) || - (stager && stager->IsStaged(url.GetUrl()))) doscan = kTRUE; - } - if (doscan) { - if ((rc = TDataSetManager::ScanFile(fileInfo, dbg)) < -1) return; - changed = kTRUE; - } else if (stager) { - ::Warning("TDataSetManager::ProcessFile", - "required file '%s' does not look as being online (staged)", url.GetUrl()); - } - if (rc < 0) return; - // Count - opened = kTRUE; - } - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Open the file described by 'fileinfo' to extract the relevant meta-information. -/// Return 0 if OK, -2 if the file cannot be open, -1 if it is corrupted - -Int_t TDataSetManager::ScanFile(TFileInfo *fileinfo, Bool_t dbg) -{ - Int_t rc = -2; - // We need an input - if (!fileinfo) { - ::Error("TDataSetManager::ScanFile", "undefined input (!)"); - return rc; - } - - TUrl *url = fileinfo->GetCurrentUrl(); - - TFile *file = 0; - Bool_t anchor = kFALSE; - - // Get timeout settings (default none) - Int_t timeout = gEnv->GetValue("DataSet.ScanFile.OpenTimeout", -1); - TString fileopt; - if (timeout > 0) fileopt.Form("TIMEOUT=%d", timeout); - - // To determine the size we have to open the file without the anchor - // (otherwise we get the size of the contained file - in case of a zip archive) - // We open in raw mode which makes sure that the opening succeeds, even if - // the file is corrupted - const char *furl = url->GetUrl(); - TString urlmod; - if (TDataSetManager::CheckDataSetSrvMaps(url, urlmod) && !(urlmod.IsNull())) - furl = urlmod.Data(); - if (strlen(url->GetAnchor()) > 0) { - anchor = kTRUE; - // We need a raw open firts to get the real size of the file - TUrl urlNoAnchor(furl); - urlNoAnchor.SetAnchor(""); - TString unaopts = urlNoAnchor.GetOptions(); - if (!unaopts.IsNull()) { - unaopts += "&filetype=raw"; - } else { - unaopts = "filetype=raw"; - } - urlNoAnchor.SetOptions(unaopts); - // Wait max 5 secs per file - if (!(file = TFile::Open(urlNoAnchor.GetUrl(), fileopt))) return rc; - - // Save some relevant info - if (file->GetSize() > 0) fileinfo->SetSize(file->GetSize()); - fileinfo->SetBit(TFileInfo::kStaged); - - fileinfo->SetUUID(file->GetUUID().AsString()); - - // Add url of the disk server in front of the list - if (file->GetEndpointUrl()) { - // add endpoint url if it is not a local file - TUrl eurl(*(file->GetEndpointUrl())); - - if (strcmp(eurl.GetProtocol(), "file") || - !strcmp(eurl.GetProtocol(), url->GetProtocol())) { - - eurl.SetOptions(url->GetOptions()); - eurl.SetAnchor(url->GetAnchor()); - - // Fix the hostname - if (!strcmp(eurl.GetHost(), "localhost") || !strcmp(eurl.GetHost(), "127.0.0.1") || - !strcmp(eurl.GetHost(), "localhost.localdomain")) { - eurl.SetHost(TUrl(gSystem->HostName()).GetHostFQDN()); - } - // Add only if different - if (strcmp(eurl.GetUrl(), url->GetUrl())) - fileinfo->AddUrl(eurl.GetUrl(), kTRUE); - - if (gDebug > 0) ::Info("TDataSetManager::ScanFile", "added URL %s", eurl.GetUrl()); - } - } else { - ::Warning("TDataSetManager::ScanFile", "end-point URL undefined for file %s", file->GetName()); - } - - file->Close(); - delete file; - } - - // OK, set the relevant flags - rc = -1; - - // Disable warnings when reading a tree without loading the corresponding library - Int_t oldLevel = gErrorIgnoreLevel; - gErrorIgnoreLevel = kError+1; - - // Wait max 5 secs per file - if (!(file = TFile::Open(url->GetUrl(), fileopt))) { - // If the file could be opened before, but fails now it is corrupt... - if (dbg) ::Info("TDataSetManager::ScanFile", "marking %s as corrupt", url->GetUrl()); - fileinfo->SetBit(TFileInfo::kCorrupted); - // Set back old warning level - gErrorIgnoreLevel = oldLevel; - return rc; - } else if (!anchor) { - // Do the relevant settings - if (file->GetSize() > 0) fileinfo->SetSize(file->GetSize()); - fileinfo->SetBit(TFileInfo::kStaged); - - // Add url of the disk server in front of the list if it is not a local file - TUrl eurl(*(file->GetEndpointUrl())); - - if (strcmp(eurl.GetProtocol(), "file") || - !strcmp(eurl.GetProtocol(), url->GetProtocol())) { - - eurl.SetOptions(url->GetOptions()); - eurl.SetAnchor(url->GetAnchor()); - - // Fix the hostname - if (!strcmp(eurl.GetHost(), "localhost") || !strcmp(eurl.GetHost(), "127.0.0.1") || - !strcmp(eurl.GetHost(), "localhost.localdomain")) { - eurl.SetHost(TUrl(gSystem->HostName()).GetHostFQDN()); - } - // Add only if different - if (strcmp(eurl.GetUrl(), url->GetUrl())) - fileinfo->AddUrl(eurl.GetUrl(), kTRUE); - - if (gDebug > 0) ::Info("TDataSetManager::ScanFile", "added URL %s", eurl.GetUrl()); - } - fileinfo->SetUUID(file->GetUUID().AsString()); - } - rc = 0; - - // Loop over all entries and create/update corresponding metadata. - // TODO If we cannot read some of the trees, is the file corrupted as well? - if ((rc = TDataSetManager::FillMetaData(fileinfo, file, "/")) != 0) { - ::Error("TDataSetManager::ScanFile", - "problems processing the directory tree in looking for metainfo"); - fileinfo->SetBit(TFileInfo::kCorrupted); - rc = -1; - } - // Set back old warning level - gErrorIgnoreLevel = oldLevel; - - file->Close(); - delete file; - - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Navigate the directory 'd' (and its subdirectories) looking for TTree objects. -/// Fill in the relevant metadata information in 'fi'. The name of the TFileInfoMeta -/// metadata entry will be "/dir1/dir2/.../tree_name". -/// Return 0 on success, -1 if any problem happens (object found in keys cannot be read, -/// for example) - -Int_t TDataSetManager::FillMetaData(TFileInfo *fi, TDirectory *d, const char *rdir) -{ - // Check inputs - if (!fi || !d || !rdir) { - ::Error("TDataSetManager::FillMetaData", - "some inputs are invalid (fi:%p,d:%p,r:%s)", fi, d, rdir); - return -1; - } - - if (d->GetListOfKeys()) { - TIter nxk(d->GetListOfKeys()); - TKey *k = 0; - while ((k = dynamic_cast (nxk()))) { - - if (TClass::GetClass(k->GetClassName())->InheritsFrom(TDirectory::Class())) { - // Get the directory - TDirectory *sd = (TDirectory *) d->Get(k->GetName()); - if (!sd) { - ::Error("TDataSetManager::FillMetaData", "cannot get sub-directory '%s'", k->GetName()); - return -1; - } - if (TDataSetManager::FillMetaData(fi, sd, TString::Format("%s%s/", rdir, k->GetName())) != 0) { - ::Error("TDataSetManager::FillMetaData", "problems processing sub-directory '%s'", k->GetName()); - return -1; - } - - } else { - // We process only trees - if (!TClass::GetClass(k->GetClassName())->InheritsFrom(TTree::Class())) continue; - - TString ks; - ks.Form("%s%s", rdir, k->GetName()); - - TFileInfoMeta *md = fi->GetMetaData(ks); - if (!md) { - // Create it - md = new TFileInfoMeta(ks, k->GetClassName()); - fi->AddMetaData(md); - if (gDebug > 0) - ::Info("TDataSetManager::FillMetaData", "created meta data for tree %s", ks.Data()); - } - // Fill values - TTree *t = dynamic_cast (d->Get(k->GetName())); - if (t) { - if (t->GetEntries() >= 0) { - md->SetEntries(t->GetEntries()); - if (t->GetTotBytes() >= 0) - md->SetTotBytes(t->GetTotBytes()); - if (t->GetZipBytes() >= 0) - md->SetZipBytes(t->GetZipBytes()); - } - } else { - ::Error("TDataSetManager::FillMetaData", "could not get tree '%s'", k->GetName()); - return -1; - } - } - } - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a server mapping list from the content of 'srvmaps' -/// Return the list (owned by the caller) or 0 if no valid info could be found) - -TList *TDataSetManager::ParseDataSetSrvMaps(const TString &srvmaps) -{ - TList *srvmapslist = 0; - if (srvmaps.IsNull()) { - ::Warning("TDataSetManager::ParseDataSetSrvMaps", - "called with an empty string! - nothing to do"); - return srvmapslist; - } - TString srvmap, sf, st; - Int_t from = 0, from1 = 0; - while (srvmaps.Tokenize(srvmap, from, " ")) { - sf = ""; st = ""; - if (srvmap.Contains("|")) { - from1 = 0; - if (srvmap.Tokenize(sf, from1, "|")) - if (srvmap.Tokenize(st, from1, "|")) { } - } else { - st = srvmap; - } - if (st.IsNull()) { - ::Warning("TDataSetManager::ParseDataSetSrvMaps", - "parsing DataSet.SrvMaps: target must be defined" - " (token: %s) - ignoring", srvmap.Data()); - continue; - } else if (!(st.EndsWith("/"))) { - st += "/"; - } - // TUrl if wildcards or TObjString - TString sp; - TUrl *u = 0; - if (!(sf.IsNull()) && sf.Contains("*")) { - u = new TUrl(sf); - if (!(sf.BeginsWith(u->GetProtocol()))) u->SetProtocol("root"); - sp.Form(":%d", u->GetPort()); - if (!(sf.Contains(sp))) u->SetPort(1094); - if (!TString(u->GetHost()).Contains("*")) SafeDelete(u); - } - if (!srvmapslist) srvmapslist = new TList; - if (u) { - srvmapslist->Add(new TPair(u, new TObjString(st))); - } else { - srvmapslist->Add(new TPair(new TObjString(sf), new TObjString(st))); - } - } - // Done - if (srvmapslist) srvmapslist->SetOwner(kTRUE); - return srvmapslist; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static getter for server mapping list - -TList *TDataSetManager::GetDataSetSrvMaps() -{ - return fgDataSetSrvMaps; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check if the dataset server mappings apply to the url defined by 'furl'. -/// Use srvmaplist if defined, else use the default list. -/// If yes, resolve the mapping into file1 and return kTRUE. -/// Otherwise return kFALSE. - -Bool_t TDataSetManager::CheckDataSetSrvMaps(TUrl *furl, TString &file1, TList *srvmaplist) -{ - Bool_t replaced = kFALSE; - if (!furl) return replaced; - - const char *file = furl->GetUrl(); - TList *mlist = (srvmaplist) ? srvmaplist : fgDataSetSrvMaps; - if (mlist && mlist->GetSize() > 0) { - TIter nxm(mlist); - TPair *pr = 0; - while ((pr = (TPair *) nxm())) { - Bool_t replace = kFALSE; - // If TUrl apply reg exp on host - TUrl *u = dynamic_cast(pr->Key()); - if (u) { - if (!strcmp(u->GetProtocol(), furl->GetProtocol())) { - Ssiz_t len; - if (!strcmp(u->GetProtocol(), "file")) { - TRegexp re(u->GetFileAndOptions(), kTRUE); - if (re.Index(furl->GetFileAndOptions(), &len) == 0) replace = kTRUE; - } else { - if (u->GetPort() == furl->GetPort()) { - TRegexp re(u->GetHost(), kTRUE); - if (re.Index(furl->GetHost(), &len) == 0) replace = kTRUE; - } - } - } - } else { - TObjString *os = dynamic_cast(pr->Key()); - if (os) { - if (os->GetString().IsNull() || - !strncmp(file, os->GetName(), os->GetString().Length())) replace = kTRUE; - } - } - if (replace) { - TObjString *ost = dynamic_cast(pr->Value()); - if (ost) { - file1.Form("%s%s", ost->GetName(), furl->GetFileAndOptions()); - replaced = kTRUE; - break; - } - } - } - } - // Done - return replaced; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update scan counters - -void TDataSetManager::SetScanCounters(Int_t t, Int_t o, Int_t d) -{ - fNTouchedFiles = (t > -1) ? t : fNTouchedFiles; - fNOpenedFiles = (o > -1) ? o : fNOpenedFiles; - fNDisappearedFiles = (d > -1) ? d : fNDisappearedFiles; -} diff --git a/proof/proof/src/TDataSetManagerFile.cxx b/proof/proof/src/TDataSetManagerFile.cxx deleted file mode 100644 index 163d5391e4640..0000000000000 --- a/proof/proof/src/TDataSetManagerFile.cxx +++ /dev/null @@ -1,1989 +0,0 @@ -// @(#)root/base:$Id$ -// Author: Jan Fiete Grosse-Oetringhaus, 04.06.07 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -////////////////////////////////////////////////////////////////////////// -// // -// TDataSetManagerFile // -// // -// Implementation of TDataSetManager handling datasets from root // -// files under a specific directory path // -// // -////////////////////////////////////////////////////////////////////////// -#include - -#include "TDataSetManagerFile.h" - -#include "Riostream.h" -#include "TDatime.h" -#include "TEnv.h" -#include "TFileCollection.h" -#include "TFileInfo.h" -#include "TFile.h" -#include "TFileStager.h" -#include "TLockFile.h" -#include "TMap.h" -#include "TRegexp.h" -#include "TMD5.h" -#include "TMacro.h" -#include "TMessage.h" -#include "TObjString.h" -#include "TSystem.h" -#include "TError.h" -#include "TPRegexp.h" -#include "TVirtualMonitoring.h" -#include "THashList.h" -#include "TKey.h" -#include "TTree.h" -#include "TParameter.h" - -struct LsTreeEntry_t { - TObjString *fGrp; // Group - TObjString *fUsr; // User - TObjString *fMd5; // Checksum - TObjString *fLss; // 'ls' string - Long64_t fMtime; // modification time - LsTreeEntry_t(const char *g, const char *u, const char *cs, const char *ls, Long64_t m) : - fMtime(m) { fGrp = new TObjString(g); fUsr = new TObjString(u); - fMd5 = new TObjString(cs); fLss = new TObjString(ls); } - ~LsTreeEntry_t() { SafeDelete(fGrp); SafeDelete(fUsr); SafeDelete(fMd5); SafeDelete(fLss);} -}; - -ClassImp(TDataSetManagerFile); - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Main constructor - -TDataSetManagerFile::TDataSetManagerFile(const char *group, - const char *user, const char *ins) - : TDataSetManager(group, user, ins) -{ - // Parse options - ParseInitOpts(ins); - - // Init the instance - Init(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Main constructor - -TDataSetManagerFile::TDataSetManagerFile(const char *ins) - : TDataSetManager("", "", ins) -{ - // Parse options - ParseInitOpts(ins); - - // Init the instance - Init(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Do the real inititialization - -void TDataSetManagerFile::Init() -{ - fIsRemote = kFALSE; - if (!fUser.IsNull() && !fGroup.IsNull() && !fDataSetDir.IsNull()) { - - // Make sure that the dataset dir exists - TString dir; - dir.Form("%s/%s/%s", fDataSetDir.Data(), fGroup.Data(), fUser.Data()); - if (gSystem->AccessPathName(dir)) { - if (gSystem->mkdir(dir, kTRUE) != 0) { - TString emsg = dir; - // Read only dataset info system: switch to COMMON - fUser = fCommonUser; - fGroup = fCommonGroup; - ResetBit(TDataSetManager::kCheckQuota); - ResetBit(TDataSetManager::kAllowRegister); - ResetBit(TDataSetManager::kAllowVerify); - ResetBit(TDataSetManager::kTrustInfo); - dir.Form("%s/%s/%s", fDataSetDir.Data(), fGroup.Data(), fUser.Data()); - if (gSystem->AccessPathName(dir)) { - Error("Init", - "could not attach to a valid the dataset dir; paths tried:"); - Error("Init", " %s", emsg.Data()); - Error("Init", " %s", dir.Data()); - SetBit(TObject::kInvalidObject); - return; - } - } - else if (fOpenPerms) { - - // Directory creation was OK: let's open permissions if requested - TString t; - Int_t rr = 0; - - t.Form("%s/%s/%s", fDataSetDir.Data(), fGroup.Data(), fUser.Data()); - rr += gSystem->Chmod(t.Data(), 0777); - - t.Form("%s/%s", fDataSetDir.Data(), fGroup.Data()); - rr += gSystem->Chmod(t.Data(), 0777); - - rr += gSystem->Chmod(fDataSetDir.Data(), 0777); - - if (rr < 0) { - t.Form("%s/%s/%s", fDataSetDir.Data(), fGroup.Data(), - fUser.Data()); - Warning("Init", - "problems setting perms of dataset directory %s (#%d)", - t.Data(), TSystem::GetErrno()); - } - - } - } - - // If not in sandbox, construct the base URI using session defaults - // (group, user) (syntax: /group/user/dsname[#[subdir/]objname]) - if (!TestBit(TDataSetManager::kIsSandbox)) - fBase.SetUri(TString(Form("/%s/%s/", fGroup.Data(), fUser.Data()))); - - // Local or remote? - TString locPath; - TFile::EFileType pathType = TFile::GetType(fDataSetDir, "READ", &locPath); - if (pathType == TFile::kLocal) { - // Remote Url pointing to this machine - fDataSetDir = locPath; - if (gDebug > 0) - Info("Init", "repository '%s' is local", fDataSetDir.Data()); - } else if (pathType != TFile::kDefault && pathType != TFile::kFile) { - fIsRemote = kTRUE; - if (gDebug > 0) - Info("Init", "repository '%s' is remote", fDataSetDir.Data()); - } - - // Read locking path from kDataSet_LockLocation - TString lockloc = TString::Format("%s/%s", fDataSetDir.Data(), kDataSet_LockLocation); - if (!gSystem->AccessPathName(lockloc, kReadPermission)) { - // Open the file in RAW mode - lockloc += "?filetype=raw"; - TFile *f = TFile::Open(lockloc); - if (f && !(f->IsZombie())) { - const Int_t blen = 8192; - char buf[blen]; - Long64_t rest = f->GetSize(); - while (rest > 0) { - Long64_t len = (rest > blen - 1) ? blen - 1 : rest; - if (f->ReadBuffer(buf, len)) { - fDataSetLockFile = ""; - break; - } - buf[len] = '\0'; - fDataSetLockFile += buf; - rest -= len; - } - f->Close(); - SafeDelete(f); - fDataSetLockFile.ReplaceAll("\n",""); - } else { - lockloc.ReplaceAll("?filetype=raw", ""); - Warning("Init", "could not open remore file '%s' with the lock location", lockloc.Data()); - } - } - if (fDataSetLockFile.IsNull()) { - fDataSetLockFile.Form("%s-dataset-lock", fDataSetDir.Data()); - fDataSetLockFile.ReplaceAll("/","%"); - fDataSetLockFile.ReplaceAll(":","%"); - fDataSetLockFile.Insert(0, TString::Format("%s/", gSystem->TempDirectory())); - } - if (!fDataSetLockFile.IsNull() && fIsRemote) { - TUrl lu(fDataSetLockFile, kTRUE); - if (!strcmp(lu.GetProtocol(), "file")) { - // Add host and port - TUrl u(fDataSetDir); - TString srv(fDataSetDir); - srv.Remove(srv.Index(u.GetFile())); - fDataSetLockFile.Insert(0, srv); - } - } - } - - // Limit in seconds after a lock automatically expires - fLockFileTimeLimit = 120; - - // Default validity of the cache - fCacheUpdatePeriod = gEnv->GetValue("ProofDataSet.CacheUpdatePeriod", 0); - - // If the MSS url was not given, check if one is defined via env - if (fMSSUrl.IsNull()) - fMSSUrl = gEnv->GetValue("ProofDataSet.MSSUrl", ""); - // Default highest priority for xrd-backends - fStageOpts = gEnv->GetValue("DataSet.StageOpts", "p=3"); - - // File to check updates and its locking path - fListFile.Form("%s/%s", fDataSetDir.Data(), kDataSet_DataSetList); - - // Init the local cache directory if the repository is remote - fUseCache = kFALSE; - fLocalCacheDir = ""; - InitLocalCache(); -} -//////////////////////////////////////////////////////////////////////////////// -/// Init the local cache if required - -void TDataSetManagerFile::InitLocalCache() -{ - fUseCache = (fIsRemote) ? kTRUE : kFALSE; - - // Check if the caller has given specific instructions - TString useCache; - if (TestBit(TDataSetManager::kUseCache)) useCache = "yes"; - if (TestBit(TDataSetManager::kDoNotUseCache)) useCache = "no"; - if (useCache.IsNull()) useCache = gEnv->GetValue("DataSet.UseCache", ""); - if (useCache.IsNull() && gSystem->Getenv("DATASETCACHE")) - useCache = gSystem->Getenv("DATASETCACHE"); - useCache.ToLower(); - if (!useCache.IsNull()) - fUseCache = (useCache == "no" || useCache == "0") ? kFALSE : kTRUE; - - if (fUseCache) { - fLocalCacheDir = gSystem->Getenv("DATASETLOCALCACHEDIR"); - if (fLocalCacheDir.IsNull()) - fLocalCacheDir = gEnv->GetValue("DataSet.LocalCacheDir", ""); - if (!fLocalCacheDir.IsNull()) { - // Make sure that the non-default local cache directory exists and is writable - if (gSystem->AccessPathName(fLocalCacheDir)) { - if (gSystem->mkdir(fLocalCacheDir, kTRUE) != 0) { - // Switch to default - Warning("InitLocalCache", - "non-default local cache directory '%s' could not be created" - " - switching to default", fLocalCacheDir.Data()); - fLocalCacheDir = ""; - } - } - if (!fLocalCacheDir.IsNull() && - gSystem->AccessPathName(fLocalCacheDir, kWritePermission)) { - Warning("InitLocalCache", - "non-default local cache directory '%s' is not writable" - " - switching to default", - fDataSetDir.Data()); - fLocalCacheDir = ""; - } - } - // If not defined yet try the (unique) default - if (fLocalCacheDir.IsNull()) { - // Add something related to fDataSetDir - TString uds(fDataSetDir.Data()); - uds.ReplaceAll("/","%"); - uds.ReplaceAll(":","%"); - if (TString(gSystem->TempDirectory()).EndsWith(fUser.Data())) { - fLocalCacheDir.Form("%s/%s/%s", gSystem->TempDirectory(), - kDataSet_LocalCache, uds.Data()); - } else { - fLocalCacheDir.Form("%s/%s/%s/%s", gSystem->TempDirectory(), - fUser.Data(), kDataSet_LocalCache, uds.Data()); - } - // Make sure that the local cache dir exists and is writable - if (gSystem->AccessPathName(fLocalCacheDir) && gSystem->mkdir(fLocalCacheDir, kTRUE) != 0) { - // Disable - Warning("InitLocalCache", - "local cache directory '%s' could not be created" - " - disabling cache", fLocalCacheDir.Data()); - fUseCache = kFALSE; - } - if (!fLocalCacheDir.IsNull() && - gSystem->AccessPathName(fLocalCacheDir, kWritePermission)) { - Warning("InitLocalCache", - "local cache directory '%s' is not writable - disabling cache", - fDataSetDir.Data()); - fUseCache = kFALSE; - } - if (!fUseCache) fLocalCacheDir = ""; - } - } - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Parse the input string and set the init bits accordingly -/// Format is -/// dir:`` [mss:``] [opt:``] -/// The `` is mandatory. -/// See TDataSetManager::ParseInitOpts for the available -/// base options. -/// The base options are already initialized by the base constructor - -void TDataSetManagerFile::ParseInitOpts(const char *ins) -{ - SetBit(TObject::kInvalidObject); - fOpenPerms = kFALSE; - - // Needs something in - if (!ins || strlen(ins) <= 0) return; - - // Extract elements - Int_t from = 0; - TString s(ins), tok; - while (s.Tokenize(tok, from, " ")) { - if (tok.BeginsWith("dir:")) - fDataSetDir = tok(4, tok.Length()); - if (tok.BeginsWith("mss:")) - fMSSUrl = tok(4, tok.Length()); - if (tok == "perms:open") - fOpenPerms = kTRUE; - } - - // The directory is mandatory - if (fDataSetDir.IsNull()) return; - - // Object is valid - ResetBit(TObject::kInvalidObject); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns path of the indicated dataset. The extension is '.root' for all files -/// except for 'dsName==ls' which have extension '.txt'. -/// If 'local' is kTRUE the local cache path is returned instead in the form -/// ``/``.``.``.``. -/// NB: contains a static TString for result, so copy result before using twice. - -const char *TDataSetManagerFile::GetDataSetPath(const char *group, - const char *user, - const char *dsName, - TString &md5path, Bool_t local) -{ - if (fgCommonDataSetTag == group) - group = fCommonGroup; - - if (fgCommonDataSetTag == user) - user = fCommonUser; - - const char *ext = (!strcmp(dsName, "ls")) ? ".txt" : ".root"; - static TString result; - if (local) { - result.Form("%s/%s.%s.%s%s", fLocalCacheDir.Data(), group, user, dsName, ext); - md5path.Form("%s/%s.%s.%s.md5sum", fLocalCacheDir.Data(), group, user, dsName); - } else { - result.Form("%s/%s/%s/%s%s", fDataSetDir.Data(), group, user, dsName, ext); - md5path.Form("%s/%s/%s/%s.md5sum", fDataSetDir.Data(), group, user, dsName); - } - if (gDebug > 0) - Info("GetDataSetPath","paths: %s, %s ", result.Data(), md5path.Data()); - return result; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save into the `/kDataSet_DataSetList` file the name of the updated -/// or created or modified dataset. For still existing datasets, fill the -/// modification date in seconds anf the checksum. -/// Returns 0 on success, -1 on error - -Int_t TDataSetManagerFile::NotifyUpdate(const char *group, const char *user, - const char *dsName, Long_t mtime, const char *checksum) -{ - // Update / create list for the owner - Long_t lsmtime = 0; - TString lschecksum; - Int_t lsrc = -1; - if ((lsrc = CreateLsFile(group, user, lsmtime, lschecksum)) < 0) { - Warning("NotifyUpdate", "problems (re-)creating the dataset lists for '/%s/%s'", - group, user); - } - - { TLockFile lock(fDataSetLockFile, fLockFileTimeLimit); - TString dspath = TString::Format("/%s/%s/%s", group, user, dsName); - // Check if the global file exists - Bool_t hasListFile = gSystem->AccessPathName(fListFile) ? kFALSE : kTRUE; - // Load the info in form of TMacro - TMD5 *oldMd5 = 0, *newMd5 = 0; - if (hasListFile && !(oldMd5 = TMD5::FileChecksum(fListFile.Data()))) { - Error("NotifyUpdate", "problems calculating old checksum of %s", fListFile.Data()); - return -1; - } - // Create the TMacro object, filling it with the existing file, if any - TMacro mac; - if (hasListFile) mac.ReadFile(fListFile.Data()); - // Locate the line to change or delete - TObjString *os = mac.GetLineWith(dspath); - if (os) { - // Delete the line if required so - if (!strcmp(checksum, "removed")) { - mac.GetListOfLines()->Remove(os); - SafeDelete(os); - } else { - // Update the information - os->SetString(TString::Format("%ld %s %s", mtime, dspath.Data(), checksum)); - } - } else { - if (!strcmp(checksum, "removed")) { - Warning("NotifyUpdate", "entry for removed dataset '%s' not found!", dspath.Data()); - } else { - // Add new line - mac.AddLine(TString::Format("%ld %s %s", mtime, dspath.Data(), checksum)); - } - } - // Locate the ls line now, is needed - TString lspath = TString::Format("/%s/%s/ls", group, user); - os = mac.GetLineWith(lspath); - if (os) { - // Delete the line if required so - if (lsrc == 1) { - mac.GetListOfLines()->Remove(os); - SafeDelete(os); - } else { - // Update the information - os->SetString(TString::Format("%ld %s %s", lsmtime, lspath.Data(), lschecksum.Data())); - } - } else { - if (lsrc == 0) { - // Add new line - mac.AddLine(TString::Format("%ld %s %s", lsmtime, lspath.Data(), lschecksum.Data())); - } - } - // Write off the new content - mac.SaveSource(fListFile.Data()); - if (fOpenPerms) { - if (gSystem->Chmod(fListFile.Data(), 0666) < 0) { - Warning("NotifyUpdate", - "can't set permissions of dataset list file %s (#%d)", - fListFile.Data(), TSystem::GetErrno()); - } - } - if (!(newMd5 = TMD5::FileChecksum(fListFile.Data()))) { - Error("NotifyUpdate", "problems calculating new checksum of %s", fListFile.Data()); - SafeDelete(oldMd5); - return -1; - } - if (oldMd5 && (*newMd5 == *oldMd5)) - Warning("NotifyUpdate", "checksum for %s did not change!", fListFile.Data()); - // Cleanup - SafeDelete(oldMd5); - SafeDelete(newMd5); - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create or recreate the dataset lists for 'uri'. -/// The list are saved in text form in 'uri'/ls.txt for fast browsing and in -/// 'uri'/ls.root in form of TMacro for optimized and portable transfer. -/// Return 0 on success, 1 if the file was empty, -1 on error - -Int_t TDataSetManagerFile::CreateLsFile(const char *group, const char *user, - Long_t &mtime, TString &checksum) -{ - mtime = 0; - checksum = ""; - // Create temporary file; we cannot lock now because we would (dead-)lock - // during ShowDataSets - TString tmpfile; - tmpfile.Form("%s/%s/%s/ls.tmp.txt", fDataSetDir.Data(), group, user); - - // Redirect output to 'tmpfile' - RedirectHandle_t rh; - if (gSystem->RedirectOutput(tmpfile.Data(), "w", &rh) != 0) { - Error("CreateLsFile", "problems redirecting output to %s (errno: %d)", - tmpfile.Data(), TSystem::GetErrno()); - return -1; - } - // Create the list - TString uri; - uri.Form("/%s/%s", group, user); - ShowDataSets(uri, "forcescan:noheader:"); - // Restore output to standard streams - if (gSystem->RedirectOutput(0, 0, &rh) != 0) { - Error("CreateLsFile", "problems restoring output to standard streams (errno: %d)", - TSystem::GetErrno()); - return -1; - } - // We can lock now - TLockFile lock(fDataSetLockFile, fLockFileTimeLimit); - // Rename the temp file - TString lsfile; - lsfile.Form("%s/%s/%s/ls.txt", fDataSetDir.Data(), group, user); - // Remove the file, if existing - if (!gSystem->AccessPathName(lsfile) && gSystem->Unlink(lsfile) != 0) { - Error("CreateLsFile", "problems unlinking old file '%s' (errno: %d)", - lsfile.Data(), TSystem::GetErrno()); - return -1; - } - // Save the new file only if non empty - FileStat_t st; - if (gSystem->GetPathInfo(tmpfile, st) == 0 && st.fSize > 0) { - if (gSystem->Rename(tmpfile, lsfile) != 0) { - Error("CreateLsFile", "problems renaming '%s' to '%s' (errno: %d)", - tmpfile.Data(), lsfile.Data(), TSystem::GetErrno()); - return -1; - } -#ifndef WIN32 - // Make sure that the ownership and permissions are those expected - FileStat_t udirst; - if (!fIsRemote && gSystem->GetPathInfo(gSystem->GetDirName(tmpfile), udirst) == 0) { - if (chown(lsfile.Data(), udirst.fUid, udirst.fGid) != 0) { - Warning("CreateLsFile", "problems setting ownership on file '%s' (errno: %d)", - lsfile.Data(), TSystem::GetErrno()); - } - if (fOpenPerms) { - if (gSystem->Chmod(lsfile.Data(), 0666) < 0) { - Warning("NotifyUpdate", - "can't set permissions of list file %s (#%d)", - lsfile.Data(), TSystem::GetErrno()); - } - } - else if (chmod(lsfile.Data(), 0644) != 0) { - Warning("CreateLsFile", "problems setting permissions on file '%s' (errno: %d)", - lsfile.Data(), TSystem::GetErrno()); - } - } -#endif - mtime = st.fMtime; - TMD5 *md5 = TMD5::FileChecksum(lsfile); - if (!md5) { - Error("CreateLsFile", "problems calculating checksum for '%s'", lsfile.Data()); - } else { - checksum = md5->AsString(); - SafeDelete(md5); - } - } else { - if (!gSystem->AccessPathName(tmpfile) && gSystem->Unlink(tmpfile) != 0) { - Error("CreateLsFile", "problems unlinking temp file '%s' (errno: %d)", - tmpfile.Data(), TSystem::GetErrno()); - return -1; - } - // No datasets anymore - return 1; - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Adds the dataset in the folder of group, user to the list in target. -/// If dsName is defined, only the information about the specified dataset -/// is processed. -/// -/// The unsigned int 'option' is forwarded to GetDataSet and BrowseDataSet. -/// Available options (to be .or.ed): -/// kPrint print the dataset content -/// kQuotaUpdate update quotas -/// kExport use export naming -/// kList get a list of dataset names -/// -/// NB1: options "kPrint", "kQuoatUpdate" and "kExport" are mutually exclusive -/// NB2: for options "kPrint" and "kQuotaUpdate" return is null. - -Bool_t TDataSetManagerFile::BrowseDataSets(const char *group, const char *user, - const char *dsName, - UInt_t option, TObject *target) -{ - TString userDirPath; - userDirPath.Form("%s/%s/%s", fDataSetDir.Data(), group, user); - void *userDir = gSystem->OpenDirectory(userDirPath); - if (!userDir) - return kFALSE; - - // Options - Bool_t printing = (option & kPrint) ? kTRUE : kFALSE; - Bool_t exporting = (option & kExport) ? kTRUE : kFALSE; - Bool_t updating = (option & kQuotaUpdate) ? kTRUE : kFALSE; - Bool_t printout = (printing && (option & kDebug)) ? kTRUE : kFALSE; - Bool_t listing = (option & kList) ? kTRUE : kFALSE; - - // If printing is required add kReadShort to the options - if (printing || updating) - option |= kReadShort; - - // The last three options are mutually exclusive - if (((Int_t)printing + (Int_t)exporting + (Int_t)updating + (Int_t)listing) > 1) { - Error("BrowseDataSets", - "only one of kPrint, kQuotaUpdate, kExport or kList can be specified at once"); - return kFALSE; - } - Bool_t fillmap = (!exporting && !printing && !updating) ? kTRUE : kFALSE; - - // Output object - TMap *outmap = (fillmap || exporting || listing) ? (TMap *)target : (TMap *)0; - TList *outlist = (printing) ? (TList *)target : (TList *)0; - - TRegexp rg("^[^./][^/]*.root$"); //check that it is a root file, not starting with "." - - TRegexp *reds = 0; - if (dsName && strlen(dsName) > 0) reds = new TRegexp(dsName, kTRUE); - - TMap *userMap = 0, *datasetMap = 0; - // loop over datasets - const char *dsEnt = 0; - while ((dsEnt = gSystem->GetDirEntry(userDir))) { - TString datasetFile(dsEnt); - if (datasetFile.Index(rg) != kNPOS) { - TString datasetName(datasetFile(0, datasetFile.Length()-5)); - - // Check dataset name, if required - if (reds && datasetName.Index(*reds) == kNPOS) continue; - - if (gDebug > 0) - Info("GetDataSets", "found dataset %s of user %s in group %s", - datasetName.Data(), user, group); - - TFileCollection *fileList = GetDataSet(group, user, datasetName, option); - if (!fileList) { - Error("GetDataSets", "dataset %s (user %s, group %s) could not be opened", - datasetName.Data(), user, group); - continue; - } - if (gDebug > 0) - fileList->Print(); - - // We found a dataset, now add it to the map - - // COMMON dataset transition - const char *mapGroup = group; - if (fCommonGroup == mapGroup) - mapGroup = fgCommonDataSetTag.Data(); - const char *mapUser = user; - if (fCommonUser == mapUser) - mapUser = fgCommonDataSetTag.Data(); - - if (fillmap && !listing && outmap) { - if (!(userMap = dynamic_cast (outmap->GetValue(mapGroup)))) { - userMap = new TMap; - userMap->SetOwner(); - outmap->Add(new TObjString(mapGroup), userMap); - } - - if (!(datasetMap = dynamic_cast (userMap->GetValue(mapUser)))) { - datasetMap = new TMap; - datasetMap->SetOwner(); - userMap->Add(new TObjString(mapUser), datasetMap); - } - } - - // Action depends on option - if (exporting) { - - // Just format the dataset name with group and user - TString dsNameFormatted(Form("/%s/%s/%s", mapGroup, - mapUser, datasetName.Data())); - if (outmap) - outmap->Add(new TObjString(dsNameFormatted), fileList); - - } else if (updating) { - - // Update quotas - GetQuota(mapGroup, mapUser, datasetName.Data(), fileList); - - } else if (printing) { - - // Prepare the output list - if (outlist) { - TString dsNameFormatted(Form("/%s/%s/%s", mapGroup, - mapUser, datasetName.Data())); - // Magic number? - if (dsNameFormatted.Length() < 42) - dsNameFormatted.Resize(42); - - // Done - outlist->Add(fileList->ExportInfo(dsNameFormatted)); - if (printout) { - // Recreating the LS file - TObjString *os = (TObjString *) outlist->Last(); - if (os) Printf("%s", os->GetName()); - } - } - } else if (listing) { - - // Just a list of available datasets - if (outmap) { - outmap->Add(new TObjString(TString::Format("/%s/%s/%s", mapGroup, mapUser, datasetName.Data())), - new TObjString("")); - } - } else { - if (fillmap && datasetMap) - datasetMap->Add(new TObjString(datasetName), fileList); - } - } - } - gSystem->FreeDirectory(userDir); - SafeDelete(reds); - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// General purpose call to go through the existing datasets. -/// If `` is 0 or "*", act on all datasets for the given ``. -/// If `` is 0 or "*", act on all datasets. -/// If `` is defined, only the information about the specified dataset -/// is processed. -/// Action depends on option; available options: -/// -/// kExport Return a TMap object containing all the information about -/// datasets in the form: -/// { ``, `` } -/// | -/// { ``, ``} -/// (`` are TFileCollection objects) -/// kShowDefault as kExport with in addition a default selection including -/// the datasets from the current user, the ones from the group -/// and the common ones -/// -/// kPrint print the dataset content; no output is returned -/// kList get a list of available dataset names -/// kForceScan Re-open files while processing kPrint (do not use the -/// pre-processed information) -/// kNoHeaderPrint Labelling header is not printed -/// kQuotaUpdate update {group, user} quotas; no output is returned -/// -/// NB1: options "kPrint", "kQuoatUpdate" and "kExport" are mutually exclusive -/// NB2: for options "kPrint" and "kQuoatUpdate" return is null. - -TMap *TDataSetManagerFile::GetDataSets(const char *group, const char *user, - const char *dsName, UInt_t option) -{ - if (group && fgCommonDataSetTag == group) - group = fCommonGroup; - - if (user && fgCommonDataSetTag == user) - user = fCommonUser; - - // Special treatment for the COMMON user - Bool_t notCommonUser = kTRUE; - if ((user && fCommonUser == user) && - (group && fCommonGroup == group)) notCommonUser = kFALSE; - - // convert * to "nothing" - if (group && (strcmp(group, "*") == 0 || !group[0])) - group = 0; - if (user && (strcmp(user, "*") == 0 || !user[0])) - user = 0; - - Bool_t printing = (option & kPrint) ? kTRUE : kFALSE; - Bool_t forcescan = (option & kForceScan) ? kTRUE : kFALSE; - Bool_t printheader = (option & kNoHeaderPrint) ? kFALSE : kTRUE; - Bool_t exporting = (option & kExport) ? kTRUE : kFALSE; - Bool_t updating = (option & kQuotaUpdate) ? kTRUE : kFALSE; - Bool_t refreshingls = (option & kRefreshLs) ? kTRUE : kFALSE; - Bool_t listing = (option & kList) ? kTRUE : kFALSE; - - // The last three options are mutually exclusive - if (((Int_t)printing + (Int_t)exporting + (Int_t)updating + (Int_t)listing) > 1) { - Error("GetDataSets", "only one of '?P', '?Q', '?E' or '?L' can be specified at once"); - return 0; - } - - TObject *result = 0; - if (printing) { - // The output is a list of strings - TList *ol = new TList(); - ol->SetOwner(); - result = ol; - } else if (exporting || !updating || listing) { - TMap *om = new TMap; - om->SetOwner(); - result = om; - } - - if (gDebug > 0) - Info("GetDataSets", "opening dir %s", fDataSetDir.Data()); - - Long_t m; - TString s; - if (option & kShowDefault) { - // Add the common ones - if (refreshingls) { - if (CreateLsFile(fCommonGroup, fCommonUser, m, s) != 0) - Warning("GetDataSets", "problems recreating 'ls' info for {%s,%s}", - fCommonGroup.Data(), fCommonUser.Data()); - } else if (!printing || forcescan || (printing && - FillLsDataSet(fCommonGroup, fCommonUser, dsName, (TList *)result, option) != 0)) { - BrowseDataSets(fCommonGroup, fCommonUser, dsName, option, result); - } - user = 0; - } else { - // Fill the information at least once - if (!notCommonUser) notCommonUser = kTRUE; - } - - // Fill the information only once - if (notCommonUser) { - // group, user defined, no looping needed - if (user && group && strchr(user, '*') && strchr(group, '*')) { - if (refreshingls) { - if (CreateLsFile(group, user, m, s) != 0) - Warning("GetDataSets", "problems recreating 'ls' info for {%s,%s}", - group, user); - } else if (!printing || forcescan || (printing && - FillLsDataSet(group, user, dsName, (TList *)result, option) != 0)) { - BrowseDataSets(group, user, dsName, option, result); - } - if (!printing) return (TMap *)result; - } else { - TRegexp *reg = (group && strlen(group) > 0) ? new TRegexp(group, kTRUE) : 0; - TRegexp *reu = (user && strlen(user) > 0) ? new TRegexp(user, kTRUE) : 0; - // Loop needed, either on the local cache or on the real thing - if (printing && !forcescan && - fUseCache && CheckLocalCache(group, user, 0, option) == 0) { - // Loop on the local cache - Int_t from = 0; - TString locupdate, dsn, grp, usr; - locupdate.Form("%s/%s", fLocalCacheDir.Data(), kDataSet_DataSetList); - TMacro uptmac(locupdate); - TIter nxl(uptmac.GetListOfLines()); - TObjString *os = 0; - while ((os = (TObjString *) nxl())) { - if (!(os->GetString().Contains("/ls"))) continue; - from = 0; - if (!(os->GetString().Tokenize(dsn, from, " "))) continue; - if (!(os->GetString().Tokenize(dsn, from, " "))) continue; - from = 0; - // Get group and apply filter on group - if (!(dsn.Tokenize(grp, from, "/")) || (reg && (grp.Index(*reg) == kNPOS))) continue; - // Get user and apply filter on user - if (!(dsn.Tokenize(usr, from, "/")) || (reu && (usr.Index(*reu) == kNPOS))) continue; - // Now get the info - if (FillLsDataSet(grp, usr, dsName, (TList *)result, option) != 0) { - // Use the standard way opening all the files - BrowseDataSets(grp, usr, dsName, option, result); - } - } - } else { - // Loop needed on the real thing - void *dataSetDir = 0; - if ((dataSetDir = gSystem->OpenDirectory(fDataSetDir))) { - // loop over groups - const char *eg = 0; - while ((eg = gSystem->GetDirEntry(dataSetDir))) { - - if (strcmp(eg, ".") == 0 || strcmp(eg, "..") == 0) - continue; - - if (reg && (TString(eg).Index(*reg) == kNPOS)) - continue; - - TString groupDirPath; - groupDirPath.Form("%s/%s", fDataSetDir.Data(), eg); - - // Make sure it is a directory - FileStat_t dirSt; - if (gSystem->GetPathInfo(groupDirPath, dirSt) != 0 || !R_ISDIR(dirSt.fMode)) - continue; - - void *groupDir = gSystem->OpenDirectory(groupDirPath); - if (!groupDir) - continue; - - // loop over users - const char *eu = 0; - while ((eu = gSystem->GetDirEntry(groupDir))) { - - if (strcmp(eu, ".") == 0 || strcmp(eu, "..") == 0) - continue; - - if (reu && (TString(eu).Index(*reu) == kNPOS)) - continue; - - // If we have the ls.macro use that - if (refreshingls) { - if (CreateLsFile(eg, eu, m, s) != 0) - Warning("GetDataSets", "problems recreating 'ls' info for {%s,%s}", - eg, eu); - } else if (!printing || forcescan || (printing && - FillLsDataSet(eg, eu, dsName, (TList *)result, option) != 0)) { - // Use the standard way opening all the files - BrowseDataSets(eg, eu, dsName, option, result); - } - } - gSystem->FreeDirectory(groupDir); - } - gSystem->FreeDirectory(dataSetDir); - } - } - SafeDelete(reg); - SafeDelete(reu); - } - } - // Print the result, if required - if (printing) { - TList *output = (TList *)result; - output->Sort(); - if (printheader) { - Printf("Dataset repository: %s", fDataSetDir.Data()); - Printf("Dataset URI | # Files | Default tree | # Events | Disk | Staged"); - } - TIter iter4(output); - TObjString *os = 0; - while ((os = dynamic_cast (iter4()))) { - if (os->GetString().BeginsWith("file:")) { - // Path of the file to be browsed - TString path(os->GetString()(5, os->GetString().Length())); - RedirectHandle_t rh(path.Data()); - gSystem->ShowOutput(&rh); - fflush(stderr); - } else { - // Simple line - Printf("%s", os->String().Data()); - } - } - // Cleanup - SafeDelete(output); - result = 0; - } - - return (TMap *)result; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check for the 'ls.txt' for 'group' and 'user' and fill the path for the -/// ls file in 'out'. -/// If 'dsname' is defined, open the file and extract the relevant line. -/// Return 0 on success, -1 on failure - -Int_t TDataSetManagerFile::FillLsDataSet(const char *group, const char *user, - const char *dsname, TList *out, UInt_t option) -{ - // Check inputs - if (!group || strlen(group) <= 0 || !user || strlen(user) <= 0 || !out) { - Error("FillLsDataSet", "at least one of the inputs is invalid (%s,%s,%p)", group, user, out); - return -1; - } - - // File with the TMacro - Int_t crc = -1; - TString lsfile, lsmd5file; - if (!fUseCache || (fUseCache && (crc = CheckLocalCache(group, user, "ls", option)) <= 0)) { - Bool_t local = (crc == 0) ? kTRUE : kFALSE; - lsfile = GetDataSetPath(group, user, "ls", lsmd5file, local); - } else { - // The dataset does not exist anymore - return 0; - } - - if (gSystem->AccessPathName(lsfile, kFileExists)) { - if (gDebug > 0) - Info("FillLsDataSet", "file '%s' does not exists", lsfile.Data()); - return -1; - } - if (gSystem->AccessPathName(lsfile, kReadPermission)) { - Warning("FillLsDataSet", "file '%s' exists cannot be read (permission denied)", lsfile.Data()); - return -1; - } - - if (dsname && strlen(dsname) > 0) { - // Read the macro - TMacro *mac = new TMacro(lsfile.Data()); - if (!mac) { - Error("FillLsDataSet", "could not initialize TMacro from '%s'", lsfile.Data()); - return -1; - } - // Prepare the string to search for - TString fullname = TString::Format("/%s/%s/%s", group, user, dsname); - Bool_t wc = (fullname.Contains("*")) ? kTRUE : kFALSE; - if (wc) fullname.ReplaceAll("*", ".*"); - TRegexp reds(fullname); - TIter nxl(mac->GetListOfLines()); - TObjString *o; - Int_t nf = 0; - while ((o = (TObjString *) nxl())) { - if (o->GetString().Index(reds) != kNPOS) { - out->Add(o->Clone()); - nf++; - if (!wc) break; - } - } - if (nf > 0 && gDebug > 0) - Info("FillLsDataSet", "no match for dataset uri '/%s/%s/%s'", group, user, dsname); - // Delete the macro - SafeDelete(mac); - } else { - // Fill in the file information - out->Add(new TObjString(TString::Format("file:%s", lsfile.Data()))); - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Returns the dataset `` of user `` in group ``. -/// If checksum is non-zero, it will contain the pointer to a TMD5 sum object -/// with the checksum of the file, has to be deleted by the user. -/// If option has the bi kReadShort set, the shortobject is read, that does not -/// contain the list of files. This is much faster. - -TFileCollection *TDataSetManagerFile::GetDataSet(const char *group, - const char *user, - const char *dsName, - UInt_t option, - TMD5 **checksum) -{ - TFileCollection *fileList = 0; - Bool_t readshort = (option & kReadShort) ? kTRUE : kFALSE; - // Check is the file is in the cache - Int_t crc = -1; - TString path, md5path; - if (readshort || !fUseCache || - (!readshort && fUseCache && (crc = CheckLocalCache(group, user, dsName, option)) <= 0)) { - Bool_t local = (crc == 0) ? kTRUE : kFALSE; - path = GetDataSetPath(group, user, dsName, md5path, local); - } else { - // The dataset does not exist (anymore?) - if (gDebug > 0) - Info("GetDataSet", "dataset %s does not exist", path.Data()); - return fileList; - } - - // Now we lock because we are going to use the file, if there - TLockFile lock(fDataSetLockFile, fLockFileTimeLimit); - - // Check if the file can be opened for reading - if (gSystem->AccessPathName(path, kFileExists)) { - if (gDebug > 0) - Info("GetDataSet", "file '%s' does not exists", path.Data()); - return fileList; - } - if (gSystem->AccessPathName(path, kReadPermission)) { - Warning("GetDataSet", "file '%s' exists cannot be read (permission denied)", path.Data()); - return fileList; - } - - // Get checksum - if (checksum) { - // save md5 sum - *checksum = TMD5::ReadChecksum(md5path); - if (!(*checksum)) { - Error("GetDataSet", "could not get checksum of %s from %s", path.Data(), md5path.Data()); - return fileList; - } - } - - TFile *f = TFile::Open(path.Data()); - if (!f) { - Error("GetDataSet", "could not open file %s", path.Data()); - if (checksum) SafeDelete(*checksum); - return fileList; - } - - if (option & kReadShort) - fileList = dynamic_cast (f->Get("dataset_short")); - - if (!fileList) - fileList = dynamic_cast (f->Get("dataset")); - - f->Close(); - SafeDelete(f); - - return fileList; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check if the local cache information for group, user, dsName is up-to-date -/// If not, make the relevant updates -/// Return 0 if OK, 1 if the dataset does not exists anymore, -1 on failure - -Int_t TDataSetManagerFile::CheckLocalCache(const char *group, const char *user, - const char *dsName, UInt_t option) -{ - // Check first if the global update info is uptodate - static TMacro *uptmac = 0; - Bool_t need_last_update = (option & kNoCacheUpdate) ? kFALSE : kTRUE; - TString locupdtim, locupdate, remupdate; - locupdtim.Form("%s/%s.update", fLocalCacheDir.Data(), kDataSet_DataSetList); - locupdate.Form("%s/%s", fLocalCacheDir.Data(), kDataSet_DataSetList); - remupdate.Form("%s/%s", fDataSetDir.Data(), kDataSet_DataSetList); - need_last_update = (gSystem->AccessPathName(locupdate)) ? kTRUE : need_last_update; - TDatime now; - UInt_t tnow = now.Convert(); - FileStat_t timst, locst, remst; - if (need_last_update && !gSystem->AccessPathName(locupdtim)) { - if (gSystem->GetPathInfo(locupdtim, timst) == 0) { - need_last_update = kFALSE; - if ((Int_t)tnow > timst.fMtime + fCacheUpdatePeriod) need_last_update = kTRUE; - } - } - if (need_last_update) { - if (gSystem->GetPathInfo(remupdate, remst) != 0) { - Error("CheckLocalCache", "cannot get info for remote file '%s' - ignoring", remupdate.Data()); - return -1; - } - if (gSystem->GetPathInfo(locupdate, locst) == 0) { - need_last_update = kFALSE; - if (remst.fMtime > locst.fMtime) { - need_last_update = kTRUE; - } else { - if (!gSystem->AccessPathName(locupdtim)) - if (gSystem->Utime(locupdtim, tnow, 0) != 0) - Warning("CheckLocalCache", - "cannot set modification time on file '%s' (errno: %d)", - locupdtim.Data(), TSystem::GetErrno()); - } - } - } - // Get the file, if needed - if (need_last_update) { - if (!TFile::Cp(remupdate, locupdate, kFALSE)) { - Error("CheckLocalCache", "cannot get remote file '%s' - ignoring", remupdate.Data()); - return -1; - } - // Set the modification time - if (gSystem->Utime(locupdate, remst.fMtime, 0) != 0) { - Warning("CheckLocalCache", "cannot set modification time on file '%s' (errno: %d)", - locupdate.Data(), TSystem::GetErrno()); - } - // Touch or create the file to control updates - if (gSystem->AccessPathName(locupdtim)) { - FILE *ftim = fopen(locupdtim.Data(), "w"); - if (!ftim) { - Warning("CheckLocalCache", "problems create file '%s' (errno: %d)", - locupdtim.Data(), TSystem::GetErrno()); - } else { - if (fclose(ftim) != 0) - Warning("CheckLocalCache", "problems close file '%s' (errno: %d)", - locupdtim.Data(), TSystem::GetErrno()); - if (gSystem->Utime(locupdtim, now.Convert(), 0) != 0) - Warning("CheckLocalCache", - "cannot set modification time on file '%s' (errno: %d)", - locupdtim.Data(), TSystem::GetErrno()); - } - } - // Update macro info - SafeDelete(uptmac); - uptmac = new TMacro(locupdate); - } else { - // Touch or create the file to control updates - if (gSystem->AccessPathName(locupdtim)) { - FILE *ftim = fopen(locupdtim.Data(), "w"); - if (!ftim) { - Warning("CheckLocalCache", "problems create file '%s' (errno: %d)", - locupdtim.Data(), TSystem::GetErrno()); - } else { - if (fclose(ftim) != 0) - Warning("CheckLocalCache", "problems close file '%s' (errno: %d)", - locupdtim.Data(), TSystem::GetErrno()); - if (gSystem->GetPathInfo(locupdate, locst) == 0) { - if (gSystem->Utime(locupdtim, locst.fMtime, 0) != 0) - Warning("CheckLocalCache", - "cannot set modification time on file '%s' (errno: %d)", - locupdtim.Data(), TSystem::GetErrno()); - } else { - Warning("CheckLocalCache", "cannot get info for file '%s'" - " - will not touch '%s'", locupdate.Data(), locupdtim.Data()); - } - } - } - if (!uptmac) uptmac = new TMacro(locupdate); - } - - // If we are just interested in the global dataset list we are done - if (!dsName || strlen(dsName) <= 0) - return 0; - - // Read the information - TString ds, locpath, path, locmd5path, md5path, remmd5s; - TMD5 *locmd5 = 0; - // The paths ... - path = GetDataSetPath(group, user, dsName, md5path); - locpath = GetDataSetPath(group, user, dsName, locmd5path, kTRUE); - ds.Form("/%s/%s/%s", group, user, dsName); - TObjString *os = uptmac->GetLineWith(ds); - if (!os) { - // DataSet does not exist anymore - if (strcmp(dsName, "ls")) - Warning("CheckLocalCache", "dataset '%s' does not exists anymore", ds.Data()); - return 1; - } - // Extract the relevant information - TString s; - Int_t from = 0; - while (os->GetString().Tokenize(s, from, " ")) { - if (!s.IsDigit() && s != ds) { - remmd5s = s; - } - } - if (remmd5s == "---") { - // DataSet does not exist anymore - if (strcmp(dsName, "ls")) - Warning("CheckLocalCache", "dataset '%s' does not exists anymore", ds.Data()); - return 1; - } - Bool_t need_update = (option & kNoCacheUpdate) ? kFALSE : kTRUE; - if (!gSystem->AccessPathName(locpath)) { - if (need_update) { - need_update = kFALSE; - locmd5 = TMD5::ReadChecksum(locmd5path); - if (!locmd5 && !(locmd5 = TMD5::FileChecksum(locpath))) { - Warning("CheckLocalCache", "cannot get checksum of '%s' - assuming match failed", ds.Data()); - need_update = kTRUE; - } else { - if (remmd5s != locmd5->AsString()) need_update = kTRUE; - } - } - } else { - need_update = kTRUE; - } - // Get the file, if needed - if (need_update) { - SafeDelete(locmd5); - if (!TFile::Cp(path, locpath, kFALSE)) { - Error("CheckLocalCache", "cannot get remote file '%s' - ignoring", path.Data()); - return -1; - } - // Calculate and save the new checksum - locmd5 = TMD5::FileChecksum(locpath); - if (locmd5) { - if (remmd5s != locmd5->AsString()) - Warning("CheckLocalCache", "checksum for freshly downloaded file '%s' does not match the" - " one posted in '%s'", locpath.Data(), kDataSet_DataSetList); - if (TMD5::WriteChecksum(locmd5path, locmd5) != 0) - Warning("CheckLocalCache", "problems saving checksum to '%s' (errno: %d)", - locmd5path.Data(), TSystem::GetErrno()); - } else { - Warning("CheckLocalCache", "problems calculating checksum for '%s'", locpath.Data()); - } - } - SafeDelete(locmd5); - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear cached information matching uri - -Int_t TDataSetManagerFile::ClearCache(const char *uri) -{ - // Open the top directory - void *dirp = gSystem->OpenDirectory(fLocalCacheDir.Data()); - if (!dirp) { - Error("ClearCache", "cannot open directory '%s' (errno: %d)", - fLocalCacheDir.Data(), TSystem::GetErrno()); - return -1; - } - TRegexp *re = 0; - if (uri && strlen(uri) > 0) { - if (strcmp(uri, "*") && strcmp(uri, "/*") && strcmp(uri, "/*/") && - strcmp(uri, "/*/*") && strcmp(uri, "/*/*/") && strcmp(uri, "/*/*/*")) { - TString u(uri); - // Remove leading '/' - if (u(0) == '/') u.Remove(0,1); - // Change '/' to '%' - u.ReplaceAll("/", "."); - // Init the regular expression - u.ReplaceAll("*", ".*"); - re = new TRegexp(u.Data()); - } - } - - Printf(" Dataset repository: %s", fDataSetDir.Data()); - Printf(" Local cache directory: %s", fLocalCacheDir.Data()); - - Long64_t totsz = 0, nf = 0; - FileStat_t st; - TString path; - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - // Skip basic entries - if (!strcmp(e,".") || !strcmp(e,"..")) continue; - // Apply regular expression, if requested - if (re && TString(e).Index(*re) == kNPOS) continue; - // Group directory - path.Form("%s/%s", fLocalCacheDir.Data(), e); - // Get file information - if (gSystem->GetPathInfo(path, st) != 0) { - Warning("ShowCache", "problems 'stat'-ing '%s' (errno: %d)", - path.Data(), TSystem::GetErrno()); - continue; - } - // Count - totsz += st.fSize; - nf++; - // Remove the file - if (gSystem->Unlink(path) != 0) { - Warning("ClearCache", "problems unlinking '%s' (errno: %d)", - path.Data(), TSystem::GetErrno()); - } - } - gSystem->FreeDirectory(dirp); - SafeDelete(re); - - // Notify totals - Printf(" %lld bytes (%lld files) have been freed", totsz, nf); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show cached information matching uri - -Int_t TDataSetManagerFile::ShowCache(const char *uri) -{ - // Open the top directory - void *dirp = gSystem->OpenDirectory(fLocalCacheDir.Data()); - if (!dirp) { - Error("ShowCache", "cannot open directory '%s' (errno: %d)", - fLocalCacheDir.Data(), TSystem::GetErrno()); - return -1; - } - TRegexp *re = 0; - if (uri && strlen(uri) > 0) { - if (strcmp(uri, "*") && strcmp(uri, "/*") && strcmp(uri, "/*/") && - strcmp(uri, "/*/*") && strcmp(uri, "/*/*/") && strcmp(uri, "/*/*/*")) { - TString u(uri); - // Remove leading '/' - if (u(0) == '/') u.Remove(0,1); - // Change '/' to '%' - u.ReplaceAll("/", "."); - // Init the regular expression - u.ReplaceAll("*", ".*"); - re = new TRegexp(u.Data()); - } - } - - Printf(" Dataset repository: %s", fDataSetDir.Data()); - Printf(" Local cache directory: %s", fLocalCacheDir.Data()); - Printf(" Last modified Size(bytes) File"); - - Long64_t totsz = 0, nf = 0; - FileStat_t st; - TString path, sz; - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - // Skip basic entries - if (!strcmp(e,".") || !strcmp(e,"..")) continue; - // Apply regular expression, if requested - if (re && TString(e).Index(*re) == kNPOS) continue; - // Group directory - path.Form("%s/%s", fLocalCacheDir.Data(), e); - // Get file information - if (gSystem->GetPathInfo(path, st) != 0) { - Warning("ShowCache", "problems 'stat'-ing '%s' (errno: %d)", - path.Data(), TSystem::GetErrno()); - continue; - } - // Count - totsz += st.fSize; - nf++; - // Get modification time in human readable form - TDatime tmod(st.fMtime); - sz.Form("%lld", st.fSize); - sz.Resize(12); - Printf(" %s %s %s", tmod.AsSQLString(), sz.Data(), e); - } - gSystem->FreeDirectory(dirp); - SafeDelete(re); - - // Notify totals - Printf(" %lld files, %lld bytes", nf, totsz); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Writes indicated dataset. -/// If option has the bit kFileMustExist set, the file must still exist, -/// otherwise the new dataset is not written (returns 3 in this case). -/// If checksum is non-zero the files current checksum is checked against it, -/// if it does not match the file is not written (the function returns 2 in this -/// case, if the file has disappeared it is also not written (i.e. checksum -/// implies the bit kFileMustExist set in option). -/// Returns != 0 for success, 0 for error - -Int_t TDataSetManagerFile::WriteDataSet(const char *group, const char *user, - const char *dsName, TFileCollection *dataset, - UInt_t option, TMD5 *checksum) -{ - TString md5path, path, md5sum; - Long_t mtime = 0; - { TLockFile lock(fDataSetLockFile, fLockFileTimeLimit); - - Bool_t checkIfExists = ((option & kFileMustExist) || checksum) ? kTRUE : kFALSE; - - path = GetDataSetPath(group, user, dsName, md5path); - - if (checkIfExists) { - // check if file still exists, otherwise it was deleted in the meanwhile and is not written here - Long_t tmp; - if (gSystem->GetPathInfo(path, 0, (Long_t*) 0, 0, &tmp) != 0) { - if (gDebug > 0) - Info("WriteDataSet", "Dataset disappeared. Discarding update."); - return 3; - } - } - - if (checksum) { - // verify md5 sum, otherwise the file was changed in the meanwhile and is not overwritten here - TMD5 *checksum2 = TMD5::FileChecksum(path); - if (!checksum2) { - Error("WriteDataSet", "Could not get checksum of %s", path.Data()); - return 0; - } - - Bool_t checksumAgrees = (*checksum == *checksum2); - delete checksum2; - - if (!checksumAgrees) { - if (gDebug > 0) - Info("WriteDataSet", "Dataset changed. Discarding update."); - return 2; - } - } - - // write first in "." then rename to recover from crash during writing - TString tempFile(path); - Int_t index = -1; - while (tempFile.Index("/", index+1) >= 0) - index = tempFile.Index("/", index+1); - - tempFile.Insert(index+1, "."); - - TFile *f = TFile::Open(tempFile, "RECREATE"); - if (!f) { - Error("WriteDataSet", "Could not open dataset for writing %s", tempFile.Data()); - return 0; - } - - // write full TFileCollection - dataset->Write("dataset", TObject::kSingleKey | TObject::kOverwrite); - - // write only metadata - THashList *list = dataset->GetList(); - dataset->SetList(0); - dataset->Write("dataset_short", TObject::kSingleKey | TObject::kOverwrite); - - f->Close(); - delete f; - - // Restore full list - dataset->SetList(list); - - // file is written, rename to real filename - if (gSystem->Rename(tempFile, path) != 0) { - Error("WriteDataSet", "renaming %s to %s failed; dataset might be corrupted", - tempFile.Data(), path.Data()); - // Cleanup any MD5 sum information - if (!gSystem->AccessPathName(md5path, kWritePermission) && gSystem->Unlink(md5path) != 0) - Error("WriteDataSet", "unlink of %s failed", md5path.Data()); - return 0; - } - else if (fOpenPerms) { - if (gSystem->Chmod(path.Data(), 0666) < 0) { - Warning("NotifyUpdate", - "can't set permissions of dataset file %s (#%d)", - path.Data(), TSystem::GetErrno()); - } - } - - // Save md5 sum, otherwise the file was changed in the meanwhile and is not overwritten here - if (ChecksumDataSet(path, md5path, md5sum) != 0) { - Error("WriteDataSet", "problems calculating checksum of %s", path.Data()); - return 0; - } - else if (fOpenPerms) { - if (gSystem->Chmod(md5path.Data(), 0666) < 0) { - Warning("NotifyUpdate", - "can't set permissions of dataset MD5 checksum file %s (#%d)", - md5path.Data(), TSystem::GetErrno()); - } - } - - FileStat_t st; - if (gSystem->GetPathInfo(path, st) != 0) { - Error("WriteDataSet", "could not 'stat' the version of '%s'!", path.Data()); - return 0; - } - mtime= st.fMtime; - } - - // The repository was updated - if (NotifyUpdate(group, user, dsName, mtime, md5sum) != 0) - Warning("WriteDataSet", "problems notifying update with 'NotifyUpdate'"); - - return 1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Calculate the checksum of the indicated dataset at 'path' and save it to the -/// appropriate file 'md5path'. The MD5 string is returned in 'md5sum'. -/// Return 0 on success, -1 on error. - -Int_t TDataSetManagerFile::ChecksumDataSet(const char *path, - const char *md5path, TString &checksum) -{ - checksum = ""; - // Check inputs - if (!path || strlen(path) <= 0 || !md5path || strlen(md5path) <= 0) { - Error("ChecksumDataSet", "one or more inputs are invalid ('%s','%s')", - path, md5path); - return -1; - } - // Calculate md5 sum - TMD5 *md5sum = TMD5::FileChecksum(path); - if (!md5sum) { - Error("ChecksumDataSet", "problems calculating checksum of '%s'", path); - return -1; - } - // Save it to a file - if (TMD5::WriteChecksum(md5path, md5sum) != 0) { - Error("ChecksumDataSet", "problems saving checksum to '%s'", md5path); - SafeDelete(md5sum); - return -1; - } - // Fill output - checksum = md5sum->AsString(); - // Done - SafeDelete(md5sum); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Removes the indicated dataset - -Bool_t TDataSetManagerFile::RemoveDataSet(const char *group, const char *user, - const char *dsName) -{ - TString md5path, path; - { TLockFile lock(fDataSetLockFile, fLockFileTimeLimit); - - path = GetDataSetPath(group, user, dsName, md5path); - Int_t rc = 0; - // Remove the main file - if ((rc = gSystem->Unlink(path)) != 0) - Warning("RemoveDataSet", "problems removing main file '%s' (errno: %d)", - path.Data(), TSystem::GetErrno()); - // Remove the checksum file - if (gSystem->Unlink(md5path) != 0) - Warning("RemoveDataSet", "problems removing chcksum file '%s' (errno: %d)", - md5path.Data(), TSystem::GetErrno()); - } - - // The repository was updated - if (gSystem->AccessPathName(path, kFileExists)) { - if (NotifyUpdate(group, user, dsName, 0, "removed") != 0) - Warning("RemoveDataSet", "problems notifying update with 'NotifyUpdate'"); - // Success - return kTRUE; - } - // Failure - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Checks if the indicated dataset exits - -Bool_t TDataSetManagerFile::ExistsDataSet(const char *group, const char *user, - const char *dsName) -{ - TLockFile lock(fDataSetLockFile, fLockFileTimeLimit); - - TString md5path, path(GetDataSetPath(group, user, dsName, md5path)); - - return (gSystem->AccessPathName(path) == kFALSE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Register a dataset, perfoming quota checkings and verification, if required. -/// If a dataset with the same name already exists the action fails unless 'opts' -/// contains 'O', in which case the old dataset is overwritten, or contains 'U', -/// in which case 'newDataSet' is added to the existing dataset (duplications are -/// ignored, if any). -/// If 'opts' contains 'V' the dataset files are also verified (if the dataset manager -/// is configured to allow so). By default the dataset is not verified. -/// If 'opts' contains 'T' the in the dataset object (status bits, meta,...) -/// is trusted, i.e. not reset (if the dataset manager is configured to allow so). -/// Returns 0 on success, -1 on failure - -Int_t TDataSetManagerFile::RegisterDataSet(const char *uri, - TFileCollection *newDataSet, - const char *opts) -{ - if (!TestBit(TDataSetManager::kAllowRegister)) - return -1; - - // Get the dataset name - TString dsName; - if (ParseUri(uri, 0, 0, &dsName, 0, kTRUE) == kFALSE) { - Error("RegisterDataSet", "problem parsing uri: %s", uri); - return -1; - } - - // The dataset - TFileCollection *dataSet = newDataSet; - // Check option - TString opt(opts); - // If in update mode, retrieve the existing dataset, if any - if (opt.Contains("U", TString::kIgnoreCase)) { - // Fail if it exists already - if (ExistsDataSet(fGroup, fUser, dsName)) { - // Retrieve the dataset - if (!(dataSet = GetDataSet(fGroup, fUser, dsName))) { - // Dataset name does exist - Warning("RegisterDataSet", - "dataset '%s' claimed to exists but retrieval failed - ignoring", uri); - dataSet = newDataSet; - } else { - // Add new dataset to existing one - dataSet->Add(newDataSet); - } - } - } else if (!opt.Contains("O", TString::kIgnoreCase)) { - // Fail if it exists already - if (ExistsDataSet(fGroup, fUser, dsName)) { - //Dataset name does exist - Error("RegisterDataSet", "dataset '%s' exists already", uri); - return -1; - } - } - - // A temporary list to hold the unique members (i.e. the very set) - TList *uniqueFileList = new TList(); - TIter nextFile(dataSet->GetList()); - TFileInfo *prevFile = (TFileInfo*)nextFile(); - uniqueFileList->Add(prevFile); - while (TFileInfo *obj = (TFileInfo*)nextFile()) { - // Add entities only once to the temporary list - if (!uniqueFileList->FindObject(obj->GetFirstUrl()->GetUrl())) - uniqueFileList->Add(obj); - } - - // Clear dataSet and add contents of uniqueFileList needed otherwise - // THashList deletes the objects even when nodelete is set - dataSet->GetList()->SetOwner(0); - dataSet->GetList()->Clear("nodelete"); - dataSet->GetList()->SetOwner(1); - dataSet->GetList()->AddAll(uniqueFileList); - uniqueFileList->SetOwner(kFALSE); - delete uniqueFileList; - - // Enforce certain settings - Bool_t reset = kTRUE; - if (opt.Contains("T", TString::kIgnoreCase)) { - if (!TestBit(TDataSetManager::kTrustInfo)) { - Warning("RegisterDataSet", "configured to not trust the information" - " provided by users: ignoring request"); - } else { - reset = kFALSE; - } - } - if (reset) { - dataSet->SetName(dsName); - dataSet->ResetBitAll(TFileInfo::kStaged); - dataSet->ResetBitAll(TFileInfo::kCorrupted); - dataSet->RemoveMetaData(); - } - - // Verify the dataset if required - if (opt.Contains("V", TString::kIgnoreCase)) { - if (TestBit(TDataSetManager::kAllowVerify)) { - // Reopen files and notify - if (TDataSetManager::ScanDataSet(dataSet, 1, 0, 0, kTRUE ) < 0) { - Error("RegisterDataSet", "problems verifying the dataset"); - return -1; - } - } else { - Warning("RegisterDataSet", "user-driven verification not allowed: ignoring request"); - } - } - - // Update accumulated information - dataSet->Update(fAvgFileSize); - - if (TestBit(TDataSetManager::kCheckQuota)) { - if (dataSet->GetTotalSize() <= 0) { - Error("RegisterDataSet", "datasets without size information are not accepted:"); - if (fAvgFileSize < 0) { - Error("RegisterDataSet", "you may want to define an average" - " file size to get an estimated dataset size"); - } - return -1; - } - // now check the quota - UpdateUsedSpace(); - Long64_t used = GetGroupUsed(fGroup) + dataSet->GetTotalSize(); - - Info("RegisterDataSet", "your group %s uses %.1f GB + %.1f GB for the new dataset; " - "the available quota is %.1f GB", fGroup.Data(), - (Float_t) GetGroupUsed(fGroup) / 1073741824, - (Float_t) dataSet->GetTotalSize() / 1073741824, - (Float_t) GetGroupQuota(fGroup) / 1073741824); - if (used > GetGroupQuota(fGroup)) { - Error("RegisterDataSet", "quota exceeded"); - return -1; - } - } - - Bool_t success = WriteDataSet(fGroup, fUser, dsName, dataSet); - if (!success) - Error("RegisterDataSet", "could not write dataset: %s", dsName.Data()); - - // Done - return ((success) ? 0 : -1); -} -//////////////////////////////////////////////////////////////////////////////// -/// Scans the dataset indicated by `` and returns the number of missing files. -/// Returns -1 if any failure occurs, >= 0 on success. -/// For more details, see documentation of -/// ScanDataSet(TFileCollection *dataset, const char *option) - -Int_t TDataSetManagerFile::ScanDataSet(const char *uri, UInt_t opt) -{ - TString dsName, dsTree; - if ((opt & kSetDefaultTree)) { - if (TestBit(TDataSetManager::kAllowRegister)) { - if (ParseUri(uri, 0, 0, &dsName, &dsTree, kTRUE)) { - TFileCollection *dataset = GetDataSet(fGroup, fUser, dsName); - if (!dataset) return -1; - dataset->SetDefaultTreeName(dsTree.Data()); - Int_t rc = WriteDataSet(fGroup, fUser, dsName, dataset); - delete dataset; - return (rc == 0) ? -1 : 0; - } - } - } else { - if (TestBit(TDataSetManager::kAllowVerify)) { - if (ParseUri(uri, 0, 0, &dsName, 0, kTRUE, kTRUE)) { - if (!(dsName.Contains("*"))) { - if (ScanDataSet(fGroup, fUser, dsName, opt) > 0) - return GetNDisapparedFiles(); - } else { - TString luri = TString::Format("/%s/%s/%s", fGroup.Data(), fUser.Data(), dsName.Data()); - TMap *fcs = GetDataSets(luri, kList); - if (!fcs) return -1; - fcs->Print(); - Int_t ndisappeared = 0; - TIter nxd(fcs); - TObjString *d = 0; - while ((d = (TObjString *) nxd())) { - if (!(d->GetString().IsNull())) { - TString dsn(d->GetName()); - if (dsn.Contains("/")) dsn.Remove(0, dsn.Last('/') + 1); - if (ScanDataSet(fGroup, fUser, dsn, opt) > 0) { - ndisappeared += GetNDisapparedFiles(); - } else { - Warning("ScanDataSet", "problems processing dataset: %s", d->GetName()); - } - } else { - Warning("ScanDataSet", "empty string found in map while processing: %s", uri); - } - } - SafeDelete(fcs); - return ndisappeared; - } - } - } - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// See documentation of ScanDataSet(TFileCollection *dataset, UInt_t option) - -Int_t TDataSetManagerFile::ScanDataSet(const char *group, const char *user, - const char *dsName, UInt_t option) -{ - if (!TestBit(TDataSetManager::kAllowVerify)) - return -1; - - TFileCollection *dataset = GetDataSet(group, user, dsName); - if (!dataset) - return -1; - - // File selection - Int_t fopt = ((option & kAllFiles)) ? -1 : 0; - if (fopt >= 0) { - if ((option & kStagedFiles)) { - fopt = 10; - } else { - if ((option & kReopen)) fopt++; - if ((option & kTouch)) fopt++; - } - if ((option & kNoStagedCheck)) fopt += 100; - } else { - if ((option & kStagedFiles) || (option & kReopen) || (option & kTouch)) { - Warning("ScanDataSet", "kAllFiles mode: ignoring kStagedFiles or kReopen" - " or kTouch requests"); - } - if ((option & kNoStagedCheck)) fopt -= 100; - } - - // Type of action - Int_t sopt = ((option & kNoAction)) ? -1 : 0; - if (sopt >= 0) { - if ((option & kLocateOnly) && (option & kStageOnly)) { - Error("ScanDataSet", "kLocateOnly and kStageOnly cannot be processed concurrently"); - return -1; - } - if ((option & kLocateOnly)) sopt = 1; - if ((option & kStageOnly)) sopt = 2; - } else if ((option & kLocateOnly) || (option & kStageOnly)) { - Warning("ScanDataSet", "kNoAction mode: ignoring kLocateOnly or kStageOnly requests"); - } - - Bool_t dbg = ((option & kDebug)) ? kTRUE : kFALSE; - // Do the scan - Int_t result = TDataSetManager::ScanDataSet(dataset, fopt, sopt, 0, dbg, - &fNTouchedFiles, &fNOpenedFiles, &fNDisappearedFiles, - (TList *)0, fAvgFileSize, fMSSUrl.Data(), -1, fStageOpts.Data()); - if (result == 2) { - if (WriteDataSet(group, user, dsName, dataset) == 0) { - delete dataset; - return -2; - } - } - delete dataset; - - return result; -} - -//////////////////////////////////////////////////////////////////////////////// -/// -/// Returns all datasets for the `` and `` specified by ``. -/// If `` is 0, it returns all datasets for the given ``. -/// If `` is 0, it returns all datasets. -/// The returned TMap contains: -/// `` --> `` --> `` --> `` (TFileCollection) -/// -/// The unsigned int 'option' is forwarded to GetDataSet and BrowseDataSet. -/// Available options (to be .or.ed): -/// kShowDefault a default selection is shown that include the ones from -/// the current user, the ones from the group and the common ones -/// kPrint print the dataset content -/// kQuotaUpdate update quotas -/// kExport use export naming -/// -/// NB1: options "kPrint", "kQuoatUpdate" and "kExport" are mutually exclusive -/// NB2: for options "kPrint" and "kQuoatUpdate" return is null. - -TMap *TDataSetManagerFile::GetDataSets(const char *uri, UInt_t option) -{ - TString dsUser, dsGroup, dsName; - - if (((option & kPrint) || (option & kExport)) && strlen(uri) <= 0) - option |= kShowDefault; - - if (ParseUri(uri, &dsGroup, &dsUser, &dsName, 0, kFALSE, kTRUE)) - return GetDataSets(dsGroup, dsUser, dsName, option); - return (TMap *)0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Utility function used in various methods for user dataset upload. - -TFileCollection *TDataSetManagerFile::GetDataSet(const char *uri, const char *opts) -{ - TString dsUser, dsGroup, dsName, ss(opts); - - TFileCollection *fc = 0; - if (!strchr(uri, '*')) { - if (!ParseUri(uri, &dsGroup, &dsUser, &dsName)) return fc; - UInt_t opt = (ss.Contains("S:") || ss.Contains("short:")) ? kReadShort : 0; - ss.ReplaceAll("S:",""); - ss.ReplaceAll("short:",""); - fc = GetDataSet(dsGroup, dsUser, dsName, opt); - } else { - TMap *fcs = GetDataSets(uri); - if (!fcs) return fc; - TIter nxd(fcs); - TObject *k = 0; - TFileCollection *xfc = 0; - while ((k = nxd()) && (xfc = (TFileCollection *) fcs->GetValue(k))) { - if (!fc) { - // The first one - fc = xfc; - fcs->Remove(k); - } else { - // Add - fc->Add(xfc); - } - } - } - - if (fc && !ss.IsNull()) { - // Build up the subset - TFileCollection *sfc = 0; - TString s; - Int_t from = 0; - while (ss.Tokenize(s, from, ",")) { - TFileCollection *xfc = fc->GetFilesOnServer(s.Data()); - if (xfc) { - if (sfc) { - sfc->Add(xfc); - delete xfc; - } else { - sfc = xfc; - } - } - } - // Cleanup - delete fc; - fc = sfc; - } - // Done - return fc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Removes the indicated dataset - -Bool_t TDataSetManagerFile::RemoveDataSet(const char *uri) -{ - TString dsName; - - if (TestBit(TDataSetManager::kAllowRegister)) { - if (ParseUri(uri, 0, 0, &dsName, 0, kTRUE)) { - Bool_t rc = RemoveDataSet(fGroup, fUser, dsName); - if (rc) return kTRUE; - Error("RemoveDataSet", "error removing dataset %s", dsName.Data()); - } - } - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Checks if the indicated dataset exits - -Bool_t TDataSetManagerFile::ExistsDataSet(const char *uri) -{ - TString dsUser, dsGroup, dsName; - - if (ParseUri(uri, &dsGroup, &dsUser, &dsName)) - return ExistsDataSet(dsGroup, dsUser, dsName); - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// updates the used space maps - -void TDataSetManagerFile::UpdateUsedSpace() -{ - // Clear used space entries - fGroupUsed.DeleteAll(); - fUserUsed.DeleteAll(); - - // Scan the existing datasets - GetDataSets(0, 0, 0, (UInt_t)kQuotaUpdate); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Gets last dataset modification time. Returns -1 on error, or number of -/// seconds since epoch on success - -Long_t TDataSetManagerFile::GetModTime(const char *uri) -{ - TString group, user, name, md5path; - if (!ParseUri(uri, &group, &user, &name)) { - return -1; - } - - TString path( GetDataSetPath(group, user, name, md5path) ); - - Long_t modTime; - if (gSystem->GetPathInfo(path.Data(), - (Long_t *)0, (Long_t *)0, (Long_t *)0, &modTime)) { - return -1; - } - - return modTime; -} diff --git a/proof/proof/src/TLockPath.cxx b/proof/proof/src/TLockPath.cxx deleted file mode 100644 index 60c50f7dea4b1..0000000000000 --- a/proof/proof/src/TLockPath.cxx +++ /dev/null @@ -1,105 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis, Oct 2011 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TLockPath -\ingroup proofkernel - -Path locking class allowing shared and exclusive locks - -*/ - -#include "TLockPath.h" -#include "TSystem.h" -#if defined(R__WIN32) && !defined(R__WINGCC) -#include -#define lseek _lseek -#define close _close -#define open _open -#define O_CREAT _O_CREAT -#define O_RDWR _O_RDWR -#else -#include -#endif - -//////////////////////////////////////////////////////////////////////////////// -/// Locks the directory. Waits if lock is hold by an other process. -/// Returns 0 on success, -1 in case of error. - -TLockPath::TLockPath(const char *path) : fName(path), fLockId(-1) -{ - // Work with full names - if (gSystem->ExpandPathName(fName)) - Warning("TLockPath", "problems expanding path '%s'", fName.Data()); -} - -Int_t TLockPath::Lock(Bool_t shared) -{ - const char *pname = GetName(); - - if (gSystem->AccessPathName(pname)) - fLockId = open(pname, O_CREAT | O_RDWR, 0644); - else - fLockId = open(pname, O_RDWR); - - if (fLockId == -1) { - SysError("Lock", "cannot open lock file %s", pname); - return -1; - } - - if (gDebug > 1) - Info("Lock", "%d: locking file %s ...", gSystem->GetPid(), pname); - // lock the file -#if !defined(R__WIN32) && !defined(R__WINGCC) - int op = (shared) ? LOCK_SH : LOCK_EX ; - if (flock(fLockId, op) == -1) { - SysError("Lock", "error locking %s", pname); - close(fLockId); - fLockId = -1; - return -1; - } -#endif - - if (gDebug > 1) - Info("Lock", "%d: file %s locked", gSystem->GetPid(), pname); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Unlock the directory. Returns 0 in case of success, -/// -1 in case of error. - -Int_t TLockPath::Unlock() -{ - if (!IsLocked()) - return 0; - - if (gDebug > 1) - Info("Unlock", "%d: unlocking file %s ...", gSystem->GetPid(), GetName()); - // unlock the file - lseek(fLockId, 0, SEEK_SET); -#if !defined(R__WIN32) && !defined(R__WINGCC) - if (flock(fLockId, LOCK_UN) == -1) { - SysError("Unlock", "error unlocking %s", GetName()); - close(fLockId); - fLockId = -1; - return -1; - } -#endif - - if (gDebug > 1) - Info("Unlock", "%d: file %s unlocked", gSystem->GetPid(), GetName()); - - close(fLockId); - fLockId = -1; - - return 0; -} diff --git a/proof/proof/src/TPackMgr.cxx b/proof/proof/src/TPackMgr.cxx deleted file mode 100644 index 3f163a5720172..0000000000000 --- a/proof/proof/src/TPackMgr.cxx +++ /dev/null @@ -1,975 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis, Oct 2015 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TPackMgr -\ingroup proofkernel - -The PROOF package manager contains tools to manage packages. -This class has been created to eliminate duplications, and to allow for -standalone usage. - -*/ - -#include "TPackMgr.h" - -#include "TError.h" -#include "TFile.h" -#include "TFunction.h" -#include "THashList.h" -#include "TList.h" -#include "TMacro.h" -#include "TMD5.h" -#include "TMethodArg.h" -#include "TMethodCall.h" -#include "TObjString.h" -#include "TParameter.h" -#include "TMap.h" -#include "TProof.h" // for constants such as kRM and kLS. -#include "TROOT.h" -#include "TSystem.h" - -ClassImp(TPackMgr); - - -static void DefaultLogger(const char *msg) { Printf("%s", msg); } - -THashList *TPackMgr::fgGlobalPackMgrList = 0; // list of package managers for global packages - -//////////////////////////////////////////////////////////////////////////////// -/// Create a PROOF package manager - -TPackMgr::TPackMgr(const char *dir, const char *key) - : fLogger(DefaultLogger), fName(key), fDir(dir), fLock(dir), fEnabledPackages(0) -{ - // Work with full names - if (gSystem->ExpandPathName(fDir)) - Warning("TPackMgr", "problems expanding path '%s'", fDir.Data()); - // The lock file in temp - TString lockname = TString::Format("%s/packdir-lock-%s", - gSystem->TempDirectory(), TString(fDir).ReplaceAll("/","%").Data()); - fLock.SetName(lockname); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destroy a TPackMgr instance - -TPackMgr::~TPackMgr() -{ - // Destroy the lock file - if (fEnabledPackages) delete fEnabledPackages; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Wrapper to notofuer / logger - -void TPackMgr::Log(const char *msg) -{ - if (fLogger) { - if (fPfx.IsNull()) - (*fLogger)(msg); - else - (*fLogger)(TString::Format("%s: %s", fPfx.Data(), msg)); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Method to build a package. -/// Return -1 on error, 0 otherwise - -Int_t TPackMgr::Build(const char *pack, Int_t opt) -{ - Int_t rc = 0; - - TLockPathGuard lp(&fLock); - - if (gDebug > 0) - Info("Build", "building package %s ...", pack); - - TString ocwd = gSystem->WorkingDirectory(); - TString pdir = TString::Format("%s/%s", fDir.Data(), pack); - gSystem->ChangeDirectory(pdir); - - // check for BUILD.sh and execute - if (!gSystem->AccessPathName("PROOF-INF/BUILD.sh")) { - // Notify the upper level - Log(TString::Format("building %s ...", pack)); - - // Read version from file proofvers.txt, and if current version is - // not the same do a "BUILD.sh clean" - Bool_t goodver = kTRUE; - Bool_t savever = kFALSE; - TString v, r; - FILE *f = fopen("PROOF-INF/proofvers.txt", "r"); - if (f) { - v.Gets(f); - r.Gets(f); - fclose(f); - if (opt == TPackMgr::kCheckROOT && v != gROOT->GetVersion()) goodver = kFALSE; - } - if (!f || !goodver) { - if (!gSystem->AccessPathName(pdir, kWritePermission)) { - savever = kTRUE; - Log(TString::Format("%s: version change" - " (current: %s, build: %s): cleaning ... ", - pack, gROOT->GetVersion(), v.Data())); - // Hard cleanup: go up the dir tree - gSystem->ChangeDirectory(fDir); - // remove package directory - gSystem->Exec(TString::Format("%s %s", kRM, pdir.Data())); - // find gunzip... - char *gunzip = gSystem->Which(gSystem->Getenv("PATH"), kGUNZIP, - kExecutePermission); - if (gunzip) { - TString par; - par.Form("%s.par", pdir.Data()); - // untar package - TString cmd; - cmd.Form(kUNTAR3, gunzip, par.Data()); - rc = gSystem->Exec(cmd); - if (rc != 0) { - Error("Build", "failure executing: %s", cmd.Data()); - } else { - // Store md5 in package/PROOF-INF/md5.txt - TMD5 *md5local = TMD5::FileChecksum(par); - if (md5local) { - TString md5f = pdir + "/PROOF-INF/md5.txt"; - TMD5::WriteChecksum(md5f, md5local); - // Go down to the package directory - gSystem->ChangeDirectory(pdir); - // Cleanup - SafeDelete(md5local); - } else { - Warning("Build", "failure calculating/saving MD5sum for '%s'", par.Data()); - } - } - delete [] gunzip; - } else { - Error("Build", "%s not found", kGUNZIP); - rc = -1; - } - } else { - Log(TString::Format("%s: ROOT version inconsistency (current: %s, build: %s):" - " directory not writable: cannot re-build!!! ", - pack, gROOT->GetVersion(), v.Data())); - rc = -1; - } - - if (rc == 0) { - // To build the package we execute PROOF-INF/BUILD.sh via a pipe - // so that we can send back the log in (almost) real-time to the - // (impatient) client. Note that this operation will block, so - // the messages from builds on the workers will reach the client - // shortly after the master ones. - TString ipath(gSystem->GetIncludePath()); - ipath.ReplaceAll("\"",""); - TString cmd; - cmd.Form("export ROOTINCLUDEPATH=\"%s\" ; PROOF-INF/BUILD.sh", ipath.Data()); - rc = gSystem->Exec(cmd); - if (rc != 0) { - Error("Build", "failure executing: %s", cmd.Data()); - } else { - // Success: write version file - if (savever) { - f = fopen("PROOF-INF/proofvers.txt", "w"); - if (f) { - fputs(gROOT->GetVersion(), f); - fputs(TString::Format("\n%s", gROOT->GetGitCommit()), f); - fclose(f); - } - } - } - } - } else { - // Notify the user - if (gDebug > 0) - Info("Build", "no PROOF-INF/BUILD.sh found for package %s", pack); - } - } - // Always return to the initial directory - gSystem->ChangeDirectory(ocwd); - - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Method to load a package taking an option const char * -/// Return -1 on error, 0 otherwise - -Int_t TPackMgr::Load(const char *pack, const char *opts) -{ - TList *optls = new TList; - optls->Add(new TObjString(opts)); - Int_t rc = Load(pack, optls); - optls->SetOwner(); - delete optls; - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Method to load a package taking an option list -/// Return -1 on error, 0 otherwise - -Int_t TPackMgr::Load(const char *pack, TList *optls) -{ - Int_t rc = 0; - TString emsg; - - // If already loaded don't do it again - if (fEnabledPackages && fEnabledPackages->FindObject(pack)) { - Log(TString::Format("error: TPackMgr::Load: package %s already loaded", pack)); - return 0; - } - - // Which pack mgr has the package? - if (!Has(pack)) { - // Check the global packages - TPackMgr *packmgr = 0; - if (!(packmgr = TPackMgr::GetPackMgr(pack, nullptr))) { - // Package not found - Log(TString::Format("error: TPackMgr::Load: failure locating %s ...", pack)); - return -1; - } - // Load from there - return packmgr->Load(pack, optls); - } - - // We have the package - TString pdir = TString::Format("%s/%s", fDir.Data(), pack); - - // Check dependencies - TString deps = TString::Format("%s/PROOF-INF/depends", pdir.Data()); - if (!gSystem->AccessPathName(deps)) { - TMacro mdeps("deps"); - if (mdeps.ReadFile(deps) > 0) { - Log(TString::Format("info: TPackMgr::Load: checking dependencies for package %s ...", pack)); - TIter nxl(mdeps.GetListOfLines()); - TObjString *os = 0; - while ((os = (TObjString *)nxl())) { - if (!TPackMgr::IsEnabled(os->GetName(), this)) { - if (Load(os->GetName(), optls) < 0) { - // Package loading failure - Log(TString::Format("error: TPackMgr::Load: failure loading dep %s ...", os->GetName())); - return -1; - } - } - } - } - } - - // Make sure it has been build - Int_t chkveropt = kCheckROOT; - if (optls) { - TParameter *pcv = (TParameter *) optls->FindObject("PROOF_Package_CheckVersion"); - if (pcv) { - chkveropt = pcv->GetVal(); - optls->Remove(pcv); - delete pcv; - } - } - if (Build(pack, chkveropt) < 0) { - // Package not found - Log(TString::Format("error: TPackMgr::Load: package %s oes not build! ", pack)); - return -1; - } - - TString ocwd = gSystem->WorkingDirectory(); - gSystem->ChangeDirectory(pdir); - - // Shared lock from here - TLockPathGuard lp(&fLock, kTRUE); - - // Check for SETUP.C and execute - if (!gSystem->AccessPathName("PROOF-INF/SETUP.C")) { - // We need to change the name of the function to avoid problems when we load more packages - TString setup; - setup.Form("SETUP_%d_%x", gSystem->GetPid(), TString(pack).Hash()); - // Remove special characters - TMacro setupmc("PROOF-INF/SETUP.C"); - TObjString *setupline = setupmc.GetLineWith("SETUP("); - if (setupline) { - TString setupstring(setupline->GetString()); - setupstring.ReplaceAll("SETUP(", TString::Format("%s(", setup.Data())); - setupline->SetString(setupstring); - } else { - // Macro does not contain SETUP() - Log(TString::Format("warning: macro '%s/PROOF-INF/SETUP.C' does not contain a SETUP()" - " function", pack)); - } - - // Load the macro - if (!setupmc.Load()) { - // Macro could not be loaded - Log(TString::Format("error: macro '%s/PROOF-INF/SETUP.C' could not be loaded:" - " cannot continue", pack)); - rc = -1; - } else { - // Check the signature - TFunction *fun = (TFunction *) gROOT->GetListOfGlobalFunctions()->FindObject(setup); - if (!fun) { - // Notify the upper level - Log(TString::Format("error: function SETUP() not found in macro '%s/PROOF-INF/SETUP.C':" - " cannot continue", pack)); - rc = -1; - } else { - TMethodCall callEnv; - // Check the number of arguments - if (fun->GetNargs() == 0) { - // No arguments (basic signature) - callEnv.Init(fun); - if (optls && optls->GetSize() > 0) { - Log(TString::Format("warning: loaded SETUP() for '%s' does not take any argument:" - " the specified argument will be ignored", pack)); - } - } else if (fun->GetNargs() == 1) { - TMethodArg *arg = (TMethodArg *) fun->GetListOfMethodArgs()->First(); - if (arg) { - callEnv.Init(fun); - // Check argument type - TString argsig(arg->GetTitle()); - if (argsig.BeginsWith("TList")) { - callEnv.ResetParam(); - callEnv.SetParam((Longptr_t) optls); - } else if (argsig.BeginsWith("const char")) { - callEnv.ResetParam(); - TObjString *os = optls ? dynamic_cast(optls->First()) : 0; - if (os) { - callEnv.SetParam((Longptr_t) os->GetName()); - } else { - if (optls && optls->First()) { - Log(TString::Format("warning: found object argument of type %s:" - " SETUP expects 'const char *': ignoring", - optls->First()->ClassName())); - } - callEnv.SetParam((Long_t) 0); - } - } else { - // Notify the upper level - Log(TString::Format("error: unsupported SETUP signature: SETUP(%s)" - " cannot continue", arg->GetTitle())); - rc = -1; - } - } else { - // Notify the upper level - Log("error: cannot get information about the SETUP() argument:" - " cannot continue"); - rc = -1; - } - } else if (fun->GetNargs() > 1) { - // Notify the upper level - Log("error: function SETUP() can have at most a 'TList *' argument:" - " cannot continue"); - rc = -1; - } - // Execute - Longptr_t setuprc = (rc == 0) ? 0 : -1; - if (rc == 0) { - callEnv.Execute(setuprc); - if (setuprc < 0) rc = -1; - } - } - } - } - - gSystem->ChangeDirectory(ocwd); - - if (rc == 0) { - // create link to package in working directory - gSystem->Symlink(pdir, pack); - - // add package to list of include directories to be searched - // by ACliC - gSystem->AddIncludePath(TString::Format("-I%s", pack)); - - // add package to list of include directories to be searched by CINT - gROOT->ProcessLine(TString::Format(".I %s", pack)); - - TPair *pck = (optls && optls->GetSize() > 0) ? new TPair(new TObjString(pack), optls->Clone()) - : new TPair(new TObjString(pack), 0); - if (!fEnabledPackages) { - fEnabledPackages = new TList; - fEnabledPackages->SetOwner(); - } - fEnabledPackages->Add(pck); - } - - return rc; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Method to unload a package. -/// Return -1 on error, 0 otherwise - -Int_t TPackMgr::Unload(const char *pack) -{ - Int_t rc = 0; - - if (fEnabledPackages && fEnabledPackages->GetSize() > 0) { - TPair *ppack = 0; - if (pack && strlen(pack) > 0) { - if ((ppack = (TPair *) fEnabledPackages->FindObject(pack))) { - - // Remove entry from include path - TString aclicincpath = gSystem->GetIncludePath(); - TString cintincpath = gInterpreter->GetIncludePath(); - // remove interpreter part of gSystem->GetIncludePath() - aclicincpath.Remove(aclicincpath.Length() - cintincpath.Length() - 1); - // remove package's include path - aclicincpath.ReplaceAll(TString(" -I") + pack, ""); - gSystem->SetIncludePath(aclicincpath); - - //TODO reset interpreter include path - - // remove entry from enabled packages list - delete fEnabledPackages->Remove(ppack); - } - - // Cleanup the link, if there - if (!gSystem->AccessPathName(pack)) - if (gSystem->Unlink(pack) != 0) rc = -1; - - } else { - - // Iterate over packages and remove each package - TIter nxp(fEnabledPackages); - while ((ppack = (TPair *) nxp())) { - if (Unload(ppack->GetName()) != 0) rc = -1; - } - - } - } - - // We are done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Method to check if this package manager has package 'pack'. -/// Return kTRUE or kFALSE - -Bool_t TPackMgr::Has(const char *pack) -{ - // always follows BuildPackage so no need to check for PROOF-INF - TString pdir = TString::Format("%s/%s", fDir.Data(), pack); - - // Shared lock from here - TLockPathGuard lp(&fLock, kTRUE); - - if (gSystem->AccessPathName(pdir, kReadPermission) || - gSystem->AccessPathName(pdir + "/PROOF-INF", kReadPermission)) - return kFALSE; - - // Relevant directories exist and ar readable - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Method to check if 'path' is in the managed directory -/// Return kTRUE or kFALSE - -Bool_t TPackMgr::IsInDir(const char *path) -{ - return strncmp(fDir.Data(), path, fDir.Length()) ? kFALSE : kTRUE ; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Method to get the path of the dir for package 'pack'. -/// Return -1 in case of error (not found), 0 otherwise - -Int_t TPackMgr::GetPackDir(const char *pack, TString &pdir) -{ - // Make sure the extension is not ".par" - TString pn(pack); - if (strstr(pack, ".par")) pn.Remove(pn.Last('.')); - pdir.Form("%s/%s", fDir.Data(), pn.Data()); - if (gSystem->AccessPathName(pdir, kReadPermission)) return -1; - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Method to get a semi-colon separated list with the names of the enabled -/// packages. - -void TPackMgr::GetEnabledPackages(TString &packlist) -{ - packlist = ""; - if (!fEnabledPackages) return; - - TIter nxp(fEnabledPackages); - TPair *pck= 0; - while ((pck = (TPair *)nxp())) { - if (packlist.Length() <= 0) - packlist = pck->GetName(); - else - packlist += TString::Format(";%s", pck->GetName()); - } - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Method to get the path of the PAR file for package 'pack'. -/// Return -1 in case of error (not found), 0 otherwise - -Int_t TPackMgr::GetParPath(const char *pack, TString &path) -{ - // Make sure the extension is ".par" - const char *fm = (strstr(pack, ".par")) ? "%s/%s" : "%s/%s.par"; - path.Form(fm, fDir.Data(), pack); - if (gSystem->AccessPathName(path, kReadPermission)) return -1; - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Method to get the download dir; create if not existing -/// Return -1 in case of error (not found; not created), 0 otherwise - -Int_t TPackMgr::GetDownloadDir(TString &dldir) -{ - dldir.Form("%s/downloaded", fDir.Data()); - if (gSystem->AccessPathName(dldir, kReadPermission)) { - if (gSystem->mkdir(dldir, kTRUE) != 0) return -1; - if (gSystem->AccessPathName(dldir, kReadPermission)) return -1; - } - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show available packages -/// - -void TPackMgr::Show(const char *title) -{ - if (fgGlobalPackMgrList && fgGlobalPackMgrList->GetSize() > 0) { - // Scan the list of global packages dirs - TIter nxpm(fgGlobalPackMgrList); - TPackMgr *pm = 0; - while ((pm = (TPackMgr *)nxpm())) { - pm->Show(TString::Format("*** Global Package cache %s %s:%s ***\n", - pm->GetName(), gSystem->HostName(), pm->GetTitle())); - } - } - - if (title && strlen(title) > 0) - printf("%s\n", title); - else - printf("*** Package cache %s:%s ***\n", gSystem->HostName(), fDir.Data()); - fflush(stdout); - // Shared lock from here - TLockPathGuard lp(&fLock, kTRUE); - gSystem->Exec(TString::Format("%s %s", kLS, fDir.Data())); - printf("\n"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clean dir for package 'pack' -/// Return -1 in case of error, 0 otherwise -/// - -Int_t TPackMgr::Clean(const char *pack) -{ - // Shared lock from here - TLockPathGuard lp(&fLock); - Int_t rc = 0; - if (pack && strlen(pack)) { - // remove package directory and par file - rc = gSystem->Exec(TString::Format("%s %s/%s/*", kRM, fDir.Data(), pack)); - } - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove package 'pack' -/// If 'pack' is null or empty all packages are cleared -/// - -Int_t TPackMgr::Remove(const char *pack, Bool_t dolock) -{ - // Shared lock from here - if (dolock) fLock.Lock(); - Int_t rc1 = 0, rc2 = 0, rc3 = 0; - if (pack && strlen(pack)) { - // remove package directory and par file - TString path = TString::Format("%s/downloaded/%s.par", fDir.Data(), pack); - gSystem->Exec(TString::Format("%s %s", kRM, path.Data())); - if (!gSystem->AccessPathName(path, kFileExists)) rc1 = -1; - path.ReplaceAll("/downloaded/", "/"); - gSystem->Exec(TString::Format("%s %s", kRM, path.Data())); - if (!gSystem->AccessPathName(path, kFileExists)) rc2 = -1; - path.Remove(path.Last('.')); - gSystem->Exec(TString::Format("%s %s", kRM, path.Data())); - if (!gSystem->AccessPathName(path, kFileExists)) rc3 = -1; - } else { - // Clear all packages - rc1 = gSystem->Exec(TString::Format("%s %s/*", kRM, fDir.Data())); - } - if (dolock) fLock.Unlock(); - return (rc1 + rc2 + rc3); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get list of available packages -/// Returns a pointer to a TList object, transferring ownership to the caller - -TList *TPackMgr::GetList() const -{ - TList *plist = new TList; - void *dir = gSystem->OpenDirectory(fDir); - if (dir) { - TString pac(gSystem->GetDirEntry(dir)); - while (pac.Length() > 0) { - if (pac.EndsWith(".par")) { - pac.ReplaceAll(".par",""); - plist->Add(new TObjString(pac.Data())); - } - pac = gSystem->GetDirEntry(dir); - } - } - gSystem->FreeDirectory(dir); - - return plist; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get list of enabled packages -/// Returns a pointer to a TList object, transferring ownership to the caller - -TList *TPackMgr::GetListOfEnabled() const -{ - TList *epl = nullptr; - if (fEnabledPackages && fEnabledPackages->GetSize() > 0) { - epl = new TList; - TIter nxp(fEnabledPackages); - TObject *o = 0; - while ((o = nxp())) { - epl->Add(new TObjString(o->GetName())); - } - } - return epl; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show enabled packages -/// - -void TPackMgr::ShowEnabled(const char *title) -{ - if (fgGlobalPackMgrList && fgGlobalPackMgrList->GetSize() > 0) { - // Scan the list of global packages dirs - TIter nxpm(fgGlobalPackMgrList); - TPackMgr *pm = 0; - while ((pm = (TPackMgr *)nxpm())) { - pm->ShowEnabled(TString::Format("*** Global Package cache %s %s:%s ***\n", - pm->GetName(), gSystem->HostName(), pm->GetTitle())); - } - } - - if (!fEnabledPackages || fEnabledPackages->GetSize() <= 0) return; - - if (title && strlen(title) > 0) - printf("%s\n", title); - else - printf("*** Package enabled on %s ***\n", gSystem->HostName()); - fflush(stdout); - - TIter next(fEnabledPackages); - while (TPair *pck = (TPair *) next()) { - printf("%s\n", pck->GetName()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get MD5 checksum of the PAR file corresponding to given package -/// Returns a pointer to a TMD5 object, transferring ownership to the caller - -TMD5 *TPackMgr::GetMD5(const char *pack) -{ - // Shared lock from here - TLockPathGuard lp(&fLock, kTRUE); - // PAR file path - const char *fm = (strstr(pack, ".par")) ? "%s/%s" : "%s/%s.par"; - TString parfile = TString::Format(fm, fDir.Data(), pack); - - return TMD5::FileChecksum(parfile); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Read MD5 checksum of the PAR file from the PROOF-INF/md5.txt file. -/// Returns a pointer to a TMD5 object, transferring ownership to the caller - -TMD5 *TPackMgr::ReadMD5(const char *pack) -{ - TString pn(pack); - if (pn.EndsWith(".par")) pn.Remove(pn.Last('.')); - - TString md5f = TString::Format("%s/%s/PROOF-INF/md5.txt", fDir.Data(), pn.Data()); - TLockPathGuard lp(&fLock, kTRUE); - return TMD5::ReadChecksum(md5f); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Read MD5 checksum of the PAR file from the PROOF-INF/md5.txt file. -/// Returns a pointer to a TMD5 object, transferring ownership to the caller - -Int_t TPackMgr::Unpack(const char *pack, TMD5 *sum) -{ - Int_t rc = 0; - TString fn(pack), pn(pack); - if (!fn.EndsWith(".par")) fn += ".par"; - if (pn.EndsWith(".par")) pn.Remove(pn.Last('.')); - - // Find gunzip... - char *gunzip = gSystem->Which(gSystem->Getenv("PATH"), kGUNZIP, kExecutePermission); - if (gunzip) { - // untar package - TString cmd; - cmd.Form(kUNTAR, gunzip, fDir.Data(), fn.Data(), fDir.Data()); - rc = gSystem->Exec(cmd); - if (rc != 0) - Error("Unpack", "failure executing: %s (rc: %d)", cmd.Data(), rc); - delete [] gunzip; - } else { - Error("Unpack", "%s not found", kGUNZIP); - rc = -2; - } - // check that fDir/pack now exists - if (gSystem->AccessPathName(TString::Format("%s/%s", fDir.Data(), pn.Data()), kWritePermission)) { - // par file did not unpack itself in the expected directory, failure - rc = -1; - Error("Unpack", "package %s did not unpack into %s", fn.Data(), pn.Data()); - } else { - // store md5 in package/PROOF-INF/md5.txt - if (sum) { - TString md5f = TString::Format("%s/%s/PROOF-INF/md5.txt", fDir.Data(), pn.Data()); - TMD5::WriteChecksum(md5f, sum); - } - } - - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Install package from par (unpack the file in the directory); par can be an -/// URL for remote retrieval. If rmold is kTRUE an existing version of the package -/// is removed if existing. -/// Returns 0 on success, <0 otherwise - -Int_t TPackMgr::Install(const char *parpath, Bool_t rmold) -{ - Int_t rc = 0; - - Info("Install", "installing %s ...", parpath); - TString par = parpath; - gSystem->ExpandPathName(par); - - // Does par exists? - if (gSystem->AccessPathName(par.Data(), kReadPermission)) { - Error("Install", "%s is invalid", par.Data()); - return -1; - } - TString parname = gSystem->BaseName(par.Data()); - TString pack = parname(0, parname.Last('.')); - TString dest = TString::Format("%s/%s", fDir.Data(), parname.Data()); - TString psrc = par, ssrc; - TMD5 *sums = 0, *md5 = 0, *md5d = 0; - - // Check if we need to download: get the remote checksum - // Retrieve the checksum of the file, if available - // Dowload checksum file, if available - TString dldir; - if (GetDownloadDir(dldir) != 0) { - Error("Install", "could not create/get download directory"); - return -1; - } - - TLockPathGuard lp(&fLock, kFALSE); - - TString parsum(par); - parsum.ReplaceAll(".par", ".md5sum"); - if (!gSystem->AccessPathName(parsum, kReadPermission)) { - ssrc.Form("%s/%s", dldir.Data(), gSystem->BaseName(parsum)); - if (!TFile::Cp(parsum, ssrc)) { - Warning("Install", "could not retrieve %s", parsum.Data()); - } else { - md5 = TMD5::ReadChecksum(ssrc); - } - } - - // Do we have already the file? - Bool_t parexists = (!gSystem->AccessPathName(dest)) ? kTRUE : kFALSE; - - Bool_t install = kTRUE; - // If yes and we are asked to clean the old one, do it - if (parexists) { - install = kFALSE; - if (rmold) { - // Asked to remove: do it - if (Remove(pack, kFALSE) < 0) { - Error("Install", "could not remove existing version of '%s'", pack.Data()); - if (md5) delete md5; - return -1; - } - install = kTRUE; - } else { - if (!md5) { - TFile::EFileType ft = TFile::GetType(par.Data()); - if (ft == TFile::kWeb || ft == TFile::kNet) { - psrc.Form("%s/%s", dldir.Data(), parname.Data()); - if (!TFile::Cp(par.Data(), psrc)) { - Error("Install", "could not retrieve %s", par.Data()); - return -1; - } - } - // psrc is either the original par or the downloaded path - md5 = TMD5::FileChecksum(psrc); - } - // Now we need to compare with the local one - sums = TMD5::FileChecksum(dest); - if (sums && md5 && (*sums != *md5)) install = kTRUE; - } - } - if (sums) delete sums; - - // Install if required - if (install) { - if (!TFile::Cp(psrc, dest)) { - Error("Install", "could not copy %s to %s", psrc.Data(), dest.Data()); - if (md5) delete md5; - return -1; - } - } - md5d = TMD5::FileChecksum(dest); - - if (md5 && *md5 != *md5d) - Warning("Install", "checksums do not match:\n\tdownloaded:\t%s\n\texpected:\t%s", - md5d->AsString(), md5->AsString()); - if (Unpack(pack, md5d) != 0) { - Error("Install", "could not unpack %s", dest.Data()); - rc = -1; - } - if (md5) delete md5; - if (md5d) delete md5d; - return rc; -} - -//--------------------------------------------------------------------------------------------------- -// Static methods -//--------------------------------------------------------------------------------------------------- - -//////////////////////////////////////////////////////////////////////////////// -/// Parse one or more paths as possible sources of packages -/// Returns number of paths added; or -1 in case of problems - -Int_t TPackMgr::RegisterGlobalPath(const char *paths) -{ - Int_t ng = 0; - // List of directories where to look for global packages - TString globpack(paths); - if (globpack.Length() > 0) { - Int_t from = 0; - TString ldir; - while (globpack.Tokenize(ldir, from, ":")) { - if (gSystem->AccessPathName(ldir, kReadPermission)) { - ::Warning("TPackMgr::RegisterGlobalPath", - "directory for global packages %s does not" - " exist or is not readable", ldir.Data()); - } else { - // Add to the list, key will be "G", i.e. "G0", "G1", ... - TString key; - key.Form("G%d", ng++); - if (!fgGlobalPackMgrList) { - fgGlobalPackMgrList = new THashList(); - fgGlobalPackMgrList->SetOwner(); - } - TPackMgr *pmgr = new TPackMgr(ldir); - pmgr->SetName(key); - fgGlobalPackMgrList->Add(pmgr); - ::Info("TPackMgr::RegisterGlobalPath", - "manager for global packages directory %s added to the list", - ldir.Data()); - } - } - } - // Number of registered packages - return ng; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Get the package manager having 'pack'; priority is given to packmgr, if -/// defined. -/// Returns packmgr or nullptr - -TPackMgr *TPackMgr::GetPackMgr(const char *pack, TPackMgr *packmgr) -{ - if (packmgr && packmgr->Has(pack)) return packmgr; - - if (fgGlobalPackMgrList && fgGlobalPackMgrList->GetSize() > 0) { - // Scan the list of global packages managers - TIter nxpm(fgGlobalPackMgrList); - TPackMgr *pm = 0; - while ((pm = (TPackMgr *)nxpm())) { - if (pm->Has(pack)) return pm; - } - } - return nullptr; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Get the full path to PAR, looking also in the global dirs. -/// Returns -1 if not found, 0 if available in global dirs, 1 if it can be -/// uploaded from the local package dir. -/// For the cases >= 0, par is filled with the path of the PAR file - -Int_t TPackMgr::FindParPath(TPackMgr *packmgr, const char *pack, TString &par) -{ - // Try the package dir - if (packmgr && packmgr->GetParPath(pack, par) == 0) return 1; - - // Try global package dirs - if (fgGlobalPackMgrList && fgGlobalPackMgrList->GetSize() > 0) { - // Scan the list of global packages dirs - TIter nxpm(fgGlobalPackMgrList); - TPackMgr *pm = 0; - while ((pm = (TPackMgr *)nxpm())) { - if (pm->GetParPath(pack, par) == 0) { - // Package found, stop searching - break; - } - par = ""; - } - if (par.Length() > 0) return 0; - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check if the package is enabled; priority is given to packmgr, if -/// defined. -/// Returns kTRUE if enabled - -Bool_t TPackMgr::IsEnabled(const char *pack, TPackMgr *packmgr) -{ - if (packmgr && packmgr->IsPackageEnabled(pack)) return kTRUE; - - if (fgGlobalPackMgrList && fgGlobalPackMgrList->GetSize() > 0) { - // Scan the list of global packages managers - TIter nxpm(fgGlobalPackMgrList); - TPackMgr *pm = 0; - while ((pm = (TPackMgr *)nxpm())) { - if (pm->IsPackageEnabled(pack)) return kTRUE; - } - } - // Not Enabled - return kFALSE; -} diff --git a/proof/proof/src/TProof.cxx b/proof/proof/src/TProof.cxx deleted file mode 100644 index 0315a9e97f441..0000000000000 --- a/proof/proof/src/TProof.cxx +++ /dev/null @@ -1,12705 +0,0 @@ -// @(#)root/proof:$Id: a2a50e759072c37ccbc65ecbcce735a76de86e95 $ -// Author: Fons Rademakers 13/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** -\defgroup proof PROOF - -Classes defining the Parallel ROOT Facility, PROOF, a framework for parallel analysis of ROOT TTrees. - -\deprecated -We keep PROOF for those who still need it for legacy use cases. -PROOF is not developed anymore and receiving only limited support. -%ROOT has since a few years moved to RDataFrame and related products as multi-core/multi-processing engines. - -*/ - -/** -\defgroup proofkernel PROOF kernel Libraries -\ingroup proof - -The PROOF kernel libraries (libProof, libProofPlayer, libProofDraw) contain the classes defining -the kernel of the PROOF facility, i.e. the protocol and the utilities to steer data processing -and handling of results. - -*/ - -/** \class TProof -\ingroup proofkernel - -This class controls a Parallel ROOT Facility, PROOF, cluster. -It fires the worker servers, it keeps track of how many workers are -running, it keeps track of the workers running status, it broadcasts -messages to all workers, it collects results, etc. - -*/ - -#include -#include -#include -#ifdef WIN32 -# include -# include -# include -# include "snprintf.h" -#else -# include -#endif -#include - -#include "RConfigure.h" -#include "Riostream.h" -#include "Getline.h" -#include "TBrowser.h" -#include "TChain.h" -#include "TCondor.h" -#include "TDSet.h" -#include "TError.h" -#include "TEnv.h" -#include "TEntryList.h" -#include "TEventList.h" -#include "TFile.h" -#include "TFileInfo.h" -#include "TFTP.h" -#include "THashList.h" -#include "TInterpreter.h" -#include "TKey.h" -#include "TMap.h" -#include "TMath.h" -#include "TMessage.h" -#include "TMonitor.h" -#include "TObjArray.h" -#include "TObjString.h" -#include "TParameter.h" -#include "TProof.h" -#include "TProofNodeInfo.h" -#include "TProofOutputFile.h" -#include "TVirtualProofPlayer.h" -#include "TVirtualPacketizer.h" -#include "TProofServ.h" -#include "TPluginManager.h" -#include "TQueryResult.h" -#include "TRandom.h" -#include "TRegexp.h" -#include "TROOT.h" -#include "TSlave.h" -#include "TSocket.h" -#include "TSortedList.h" -#include "TSystem.h" -#include "TTree.h" -#include "TUrl.h" -#include "TFileCollection.h" -#include "TDataSetManager.h" -#include "TDataSetManagerFile.h" -#include "TMacro.h" -#include "TSelector.h" -#include "TPRegexp.h" -#include "TPackMgr.h" - -#include - -TProof *gProof = 0; - -// Rotating indicator -char TProofMergePrg::fgCr[4] = {'-', '\\', '|', '/'}; - -TList *TProof::fgProofEnvList = 0; // List of env vars for proofserv -TPluginHandler *TProof::fgLogViewer = 0; // Log viewer handler - -ClassImp(TProof); - -//----- PROOF Interrupt signal handler ----------------------------------------- -//////////////////////////////////////////////////////////////////////////////// -/// TProof interrupt handler. - -Bool_t TProofInterruptHandler::Notify() -{ - if (!fProof->IsTty() || fProof->GetRemoteProtocol() < 22) { - - // Cannot ask the user : abort any remote processing - fProof->StopProcess(kTRUE); - - } else { - // Real stop or request to switch to asynchronous? - const char *a = 0; - if (fProof->GetRemoteProtocol() < 22) { - a = Getline("\nSwitch to asynchronous mode not supported remotely:" - "\nEnter S/s to stop, Q/q to quit, any other key to continue: "); - } else { - a = Getline("\nEnter A/a to switch asynchronous, S/s to stop, Q/q to quit," - " any other key to continue: "); - } - if (a[0] == 'Q' || a[0] == 'S' || a[0] == 'q' || a[0] == 's') { - - Info("Notify","Processing interrupt signal ... %c", a[0]); - - // Stop or abort any remote processing - Bool_t abort = (a[0] == 'Q' || a[0] == 'q') ? kTRUE : kFALSE; - fProof->StopProcess(abort); - - } else if ((a[0] == 'A' || a[0] == 'a') && fProof->GetRemoteProtocol() >= 22) { - // Stop any remote processing - fProof->GoAsynchronous(); - } - } - - return kTRUE; -} - -//----- Input handler for messages from TProofServ ----------------------------- -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TProofInputHandler::TProofInputHandler(TProof *p, TSocket *s) - : TFileHandler(s->GetDescriptor(),1), - fSocket(s), fProof(p) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle input - -Bool_t TProofInputHandler::Notify() -{ - fProof->CollectInputFrom(fSocket); - return kTRUE; -} - - -//------------------------------------------------------------------------------ - -ClassImp(TSlaveInfo); - -//////////////////////////////////////////////////////////////////////////////// -/// Used to sort slaveinfos by ordinal. - -Int_t TSlaveInfo::Compare(const TObject *obj) const -{ - if (!obj) return 1; - - const TSlaveInfo *si = dynamic_cast(obj); - - if (!si) return fOrdinal.CompareTo(obj->GetName()); - - const char *myord = GetOrdinal(); - const char *otherord = si->GetOrdinal(); - while (myord && otherord) { - Int_t myval = atoi(myord); - Int_t otherval = atoi(otherord); - if (myval < otherval) return 1; - if (myval > otherval) return -1; - myord = strchr(myord, '.'); - if (myord) myord++; - otherord = strchr(otherord, '.'); - if (otherord) otherord++; - } - if (myord) return -1; - if (otherord) return 1; - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Used to compare slaveinfos by ordinal. - -Bool_t TSlaveInfo::IsEqual(const TObject* obj) const -{ - if (!obj) return kFALSE; - const TSlaveInfo *si = dynamic_cast(obj); - if (!si) return kFALSE; - return (strcmp(GetOrdinal(), si->GetOrdinal()) == 0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print slave info. If opt = "active" print only the active -/// slaves, if opt="notactive" print only the not active slaves, -/// if opt = "bad" print only the bad slaves, else -/// print all slaves. - -void TSlaveInfo::Print(Option_t *opt) const -{ - TString stat = fStatus == kActive ? "active" : - fStatus == kBad ? "bad" : - "not active"; - - Bool_t newfmt = kFALSE; - TString oo(opt); - if (oo.Contains("N")) { - newfmt = kTRUE; - oo.ReplaceAll("N",""); - } - if (oo == "active" && fStatus != kActive) return; - if (oo == "notactive" && fStatus != kNotActive) return; - if (oo == "bad" && fStatus != kBad) return; - - if (newfmt) { - TString msd, si, datadir; - if (!(fMsd.IsNull())) msd.Form("| msd: %s ", fMsd.Data()); - if (!(fDataDir.IsNull())) datadir.Form("| datadir: %s ", fDataDir.Data()); - if (fSysInfo.fCpus > 0) { - si.Form("| %s, %d cores, %d MB ram", fHostName.Data(), - fSysInfo.fCpus, fSysInfo.fPhysRam); - } else { - si.Form("| %s", fHostName.Data()); - } - Printf("Worker: %9s %s %s%s| %s", fOrdinal.Data(), si.Data(), msd.Data(), datadir.Data(), stat.Data()); - - } else { - TString msd = fMsd.IsNull() ? "" : fMsd.Data(); - - std::cout << "Slave: " << fOrdinal - << " hostname: " << fHostName - << " msd: " << msd - << " perf index: " << fPerfIndex - << " " << stat - << std::endl; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Setter for fSysInfo - -void TSlaveInfo::SetSysInfo(SysInfo_t si) -{ - fSysInfo.fOS = si.fOS; // OS - fSysInfo.fModel = si.fModel; // computer model - fSysInfo.fCpuType = si.fCpuType; // type of cpu - fSysInfo.fCpus = si.fCpus; // number of cpus - fSysInfo.fCpuSpeed = si.fCpuSpeed; // cpu speed in MHz - fSysInfo.fBusSpeed = si.fBusSpeed; // bus speed in MHz - fSysInfo.fL2Cache = si.fL2Cache; // level 2 cache size in KB - fSysInfo.fPhysRam = si.fPhysRam; // Physical RAM -} - -ClassImp(TProof); - -//------------------------------------------------------------------------------ - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TMergerInfo::~TMergerInfo() -{ - // Just delete the list, the objects are owned by other list - if (fWorkers) { - fWorkers->SetOwner(kFALSE); - SafeDelete(fWorkers); - } -} -//////////////////////////////////////////////////////////////////////////////// -/// Increase number of already merged workers by 1 - -void TMergerInfo::SetMergedWorker() -{ - if (AreAllWorkersMerged()) - Error("SetMergedWorker", "all workers have been already merged before!"); - else - fMergedWorkers++; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add new worker to the list of workers to be merged by this merger - -void TMergerInfo::AddWorker(TSlave *sl) -{ - if (!fWorkers) - fWorkers = new TList(); - if (fWorkersToMerge == fWorkers->GetSize()) { - Error("AddWorker", "all workers have been already assigned to this merger"); - return; - } - fWorkers->Add(sl); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return if merger has already merged all workers, i.e. if it has finished its merging job - -Bool_t TMergerInfo::AreAllWorkersMerged() -{ - return (fWorkersToMerge == fMergedWorkers); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return if the determined number of workers has been already assigned to this merger - -Bool_t TMergerInfo::AreAllWorkersAssigned() -{ - if (!fWorkers) - return kFALSE; - - return (fWorkers->GetSize() == fWorkersToMerge); -} - -//////////////////////////////////////////////////////////////////////////////// -/// This a private API function. -/// It checks whether the connection string contains a PoD cluster protocol. -/// If it does, then the connection string will be changed to reflect -/// a real PROOF connection string for a PROOF cluster managed by PoD. -/// PoD: http://pod.gsi.de . -/// Return -1 if the PoD request failed; return 0 otherwise. - -static Int_t PoDCheckUrl(TString *_cluster) -{ - if ( !_cluster ) - return 0; - - // trim spaces from both sides of the string - *_cluster = _cluster->Strip( TString::kBoth ); - // PoD protocol string - const TString pod_prot("pod"); - - // URL test - // TODO: The URL test is to support remote PoD servers (not managed by pod-remote) - TUrl url( _cluster->Data() ); - if( pod_prot.CompareTo(url.GetProtocol(), TString::kIgnoreCase) ) - return 0; - - // PoD cluster is used - // call pod-info in a batch mode (-b). - // pod-info will find either a local PoD cluster or - // a remote one, manged by pod-remote. - *_cluster = gSystem->GetFromPipe("pod-info -c -b"); - if( 0 == _cluster->Length() ) { - Error("PoDCheckUrl", "PoD server is not running"); - return -1; - } - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a PROOF environment. Starting PROOF involves either connecting -/// to a master server, which in turn will start a set of slave servers, or -/// directly starting as master server (if master = ""). Masterurl is of -/// the form: [proof[s]://]host[:port]. Conffile is the name of the config -/// file describing the remote PROOF cluster (this argument alows you to -/// describe different cluster configurations). -/// The default is proof.conf. Confdir is the directory where the config -/// file and other PROOF related files are (like motd and noproof files). -/// Loglevel is the log level (default = 1). User specified custom config -/// files will be first looked for in $HOME/.conffile. - -TProof::TProof(const char *masterurl, const char *conffile, const char *confdir, - Int_t loglevel, const char *alias, TProofMgr *mgr) - : fUrl(masterurl) -{ - // Default initializations - InitMembers(); - - // This may be needed during init - fManager = mgr; - - // Default server type - fServType = TProofMgr::kXProofd; - - // Default query mode - fQueryMode = kSync; - - // Parse the main URL, adjusting the missing fields and setting the relevant - // bits - ResetBit(TProof::kIsClient); - ResetBit(TProof::kIsMaster); - - // Protocol and Host - if (!masterurl || strlen(masterurl) <= 0) { - fUrl.SetProtocol("proof"); - fUrl.SetHost("__master__"); - } else if (!(strstr(masterurl, "://"))) { - fUrl.SetProtocol("proof"); - } - // Port - if (fUrl.GetPort() == TUrl(" ").GetPort()) - fUrl.SetPort(TUrl("proof:// ").GetPort()); - - // Make sure to store the FQDN, so to get a solid reference for subsequent checks - if (!strcmp(fUrl.GetHost(), "__master__")) - fMaster = fUrl.GetHost(); - else if (!strlen(fUrl.GetHost())) - fMaster = gSystem->GetHostByName(gSystem->HostName()).GetHostName(); - else - fMaster = gSystem->GetHostByName(fUrl.GetHost()).GetHostName(); - - // Server type - if (strlen(fUrl.GetOptions()) > 0) { - TString opts(fUrl.GetOptions()); - if (!(strncmp(fUrl.GetOptions(),"std",3))) { - fServType = TProofMgr::kProofd; - opts.Remove(0,3); - fUrl.SetOptions(opts.Data()); - } else if (!(strncmp(fUrl.GetOptions(),"lite",4))) { - fServType = TProofMgr::kProofLite; - opts.Remove(0,4); - fUrl.SetOptions(opts.Data()); - } - } - - // Instance type - fMasterServ = kFALSE; - SetBit(TProof::kIsClient); - ResetBit(TProof::kIsMaster); - if (fMaster == "__master__") { - fMasterServ = kTRUE; - ResetBit(TProof::kIsClient); - SetBit(TProof::kIsMaster); - } else if (fMaster == "prooflite") { - // Client and master are merged - fMasterServ = kTRUE; - SetBit(TProof::kIsMaster); - } - // Flag that we are a client - if (TestBit(TProof::kIsClient)) - if (!gSystem->Getenv("ROOTPROOFCLIENT")) gSystem->Setenv("ROOTPROOFCLIENT",""); - - Init(masterurl, conffile, confdir, loglevel, alias); - - // If the user was not set, get it from the master - if (strlen(fUrl.GetUser()) <= 0) { - TString usr, emsg; - if (Exec("gProofServ->GetUser()", "0", kTRUE) == 0) { - TObjString *os = fMacroLog.GetLineWith("const char"); - if (os) { - Ssiz_t fst = os->GetString().First('\"'); - Ssiz_t lst = os->GetString().Last('\"'); - usr = os->GetString()(fst+1, lst-fst-1); - } else { - emsg = "could not find 'const char *' string in macro log"; - } - } else { - emsg = "could not retrieve user info"; - } - if (!emsg.IsNull()) { - // Get user logon name - UserGroup_t *pw = gSystem->GetUserInfo(); - if (pw) { - usr = pw->fUser; - delete pw; - } - Warning("TProof", "%s: using local default %s", emsg.Data(), usr.Data()); - } - // Set the user name in the main URL - fUrl.SetUser(usr.Data()); - } - - // If called by a manager, make sure it stays in last position - // for cleaning - if (mgr) { - R__LOCKGUARD(gROOTMutex); - gROOT->GetListOfSockets()->Remove(mgr); - gROOT->GetListOfSockets()->Add(mgr); - } - - // Old-style server type: we add this to the list and set the global pointer - if (IsProofd() || TestBit(TProof::kIsMaster)) - if (!gROOT->GetListOfProofs()->FindObject(this)) - gROOT->GetListOfProofs()->Add(this); - - // Still needed by the packetizers: needs to be changed - gProof = this; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Protected constructor to be used by classes deriving from TProof -/// (they have to call Init themselves and override StartSlaves -/// appropriately). -/// -/// This constructor simply closes any previous gProof and sets gProof -/// to this instance. - -TProof::TProof() : fUrl(""), fServType(TProofMgr::kXProofd) -{ - // Default initializations - InitMembers(); - - if (!gROOT->GetListOfProofs()->FindObject(this)) - gROOT->GetListOfProofs()->Add(this); - - gProof = this; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Default initializations - -void TProof::InitMembers() -{ - fValid = kFALSE; - fTty = kFALSE; - fRecvMessages = 0; - fSlaveInfo = 0; - fMasterServ = kFALSE; - fSendGroupView = kFALSE; - fIsPollingWorkers = kFALSE; - fLastPollWorkers_s = -1; - fActiveSlaves = 0; - fInactiveSlaves = 0; - fUniqueSlaves = 0; - fAllUniqueSlaves = 0; - fNonUniqueMasters = 0; - fActiveMonitor = 0; - fUniqueMonitor = 0; - fAllUniqueMonitor = 0; - fCurrentMonitor = 0; - fBytesRead = 0; - fRealTime = 0; - fCpuTime = 0; - fIntHandler = 0; - fProgressDialog = 0; - fProgressDialogStarted = kFALSE; - SetBit(kUseProgressDialog); - fPlayer = 0; - fFeedback = 0; - fChains = 0; - fDSet = 0; - fNotIdle = 0; - fSync = kTRUE; - fRunStatus = kRunning; - fIsWaiting = kFALSE; - fRedirLog = kFALSE; - fLogFileW = 0; - fLogFileR = 0; - fLogToWindowOnly = kFALSE; - fSaveLogToMacro = kFALSE; - fMacroLog.SetName("ProofLogMacro"); - - fWaitingSlaves = 0; - fQueries = 0; - fOtherQueries = 0; - fDrawQueries = 0; - fMaxDrawQueries = 1; - fSeqNum = 0; - - fSessionID = -1; - fEndMaster = kFALSE; - - fPackMgr = 0; - fEnabledPackagesOnCluster = 0; - - fInputData = 0; - - fPrintProgress = 0; - - fLoadedMacros = 0; - - fProtocol = -1; - fSlaves = 0; - fTerminatedSlaveInfos = 0; - fBadSlaves = 0; - fAllMonitor = 0; - fDataReady = kFALSE; - fBytesReady = 0; - fTotalBytes = 0; - fAvailablePackages = 0; - fEnabledPackages = 0; - fRunningDSets = 0; - - fCollectTimeout = -1; - - fManager = 0; - fQueryMode = kSync; - fDynamicStartup = kFALSE; - - fMergersSet = kFALSE; - fMergersByHost = kFALSE; - fMergers = 0; - fMergersCount = -1; - fLastAssignedMerger = 0; - fWorkersToMerge = 0; - fFinalizationRunning = kFALSE; - - fPerfTree = ""; - - fWrksOutputReady = 0; - - fSelector = 0; - - fPrepTime = 0.; - - // Check if the user defined a list of environment variables to send over: - // include them into the dedicated list - if (gSystem->Getenv("PROOF_ENVVARS")) { - TString envs(gSystem->Getenv("PROOF_ENVVARS")), env, envsfound; - Int_t from = 0; - while (envs.Tokenize(env, from, ",")) { - if (!env.IsNull()) { - if (!gSystem->Getenv(env)) { - Warning("Init", "request for sending over undefined environemnt variable '%s' - ignoring", env.Data()); - } else { - if (!envsfound.IsNull()) envsfound += ","; - envsfound += env; - TProof::DelEnvVar(env); - TProof::AddEnvVar(env, gSystem->Getenv(env)); - } - } - } - if (envsfound.IsNull()) { - Warning("Init", "none of the requested env variables were found: '%s'", envs.Data()); - } else { - Info("Init", "the following environment variables have been added to the list to be sent to the nodes: '%s'", envsfound.Data()); - } - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clean up PROOF environment. - -TProof::~TProof() -{ - if (fChains) { - while (TChain *chain = dynamic_cast (fChains->First()) ) { - // remove "chain" from list - chain->SetProof(0); - RemoveChain(chain); - } - } - - // remove links to packages enabled on the client - if (TestBit(TProof::kIsClient)) { - // iterate over all packages - TList *epl = fPackMgr->GetListOfEnabled(); - if (epl) { - TIter nxp(epl); - while (TObjString *pck = (TObjString *)(nxp())) { - FileStat_t stat; - if (gSystem->GetPathInfo(pck->String(), stat) == 0) { - // check if symlink, if so unlink - // NOTE: GetPathInfo() returns 1 in case of symlink that does not point to - // existing file or to a directory, but if fIsLink is true the symlink exists - if (stat.fIsLink) - gSystem->Unlink(pck->String()); - } - } - epl->Delete(); - delete epl; - } - } - - Close(); - SafeDelete(fIntHandler); - SafeDelete(fSlaves); - SafeDelete(fActiveSlaves); - SafeDelete(fInactiveSlaves); - SafeDelete(fUniqueSlaves); - SafeDelete(fAllUniqueSlaves); - SafeDelete(fNonUniqueMasters); - SafeDelete(fTerminatedSlaveInfos); - SafeDelete(fBadSlaves); - SafeDelete(fAllMonitor); - SafeDelete(fActiveMonitor); - SafeDelete(fUniqueMonitor); - SafeDelete(fAllUniqueMonitor); - SafeDelete(fSlaveInfo); - SafeDelete(fChains); - SafeDelete(fPlayer); - SafeDelete(fFeedback); - SafeDelete(fWaitingSlaves); - SafeDelete(fAvailablePackages); - SafeDelete(fEnabledPackages); - SafeDelete(fLoadedMacros); - SafeDelete(fPackMgr); - SafeDelete(fRecvMessages); - SafeDelete(fInputData); - SafeDelete(fRunningDSets); - SafeDelete(fEnabledPackagesOnCluster); - if (fWrksOutputReady) { - fWrksOutputReady->SetOwner(kFALSE); - delete fWrksOutputReady; - } - if (fQueries) { - fQueries->Delete(); - delete fQueries; - } - - // remove file with redirected logs - if (TestBit(TProof::kIsClient)) { - if (fLogFileR) - fclose(fLogFileR); - if (fLogFileW) - fclose(fLogFileW); - if (fLogFileName.Length() > 0) - gSystem->Unlink(fLogFileName); - } - - // Remove for the global list - gROOT->GetListOfProofs()->Remove(this); - // ... and from the manager list - if (fManager && fManager->IsValid()) - fManager->DiscardSession(this); - - if (gProof && gProof == this) { - // Set previous as default - TIter pvp(gROOT->GetListOfProofs(), kIterBackward); - while ((gProof = (TProof *)pvp())) { - if (gProof->InheritsFrom(TProof::Class())) - break; - } - } - - // For those interested in our destruction ... - Emit("~TProof()"); - Emit("CloseWindow()"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Start the PROOF environment. Starting PROOF involves either connecting -/// to a master server, which in turn will start a set of slave servers, or -/// directly starting as master server (if master = ""). For a description -/// of the arguments see the TProof ctor. Returns the number of started -/// master or slave servers, returns 0 in case of error, in which case -/// fValid remains false. - -Int_t TProof::Init(const char *, const char *conffile, - const char *confdir, Int_t loglevel, const char *alias) -{ - R__ASSERT(gSystem); - - fValid = kFALSE; - - // Connected to terminal? - fTty = (isatty(0) == 0 || isatty(1) == 0) ? kFALSE : kTRUE; - - // If in attach mode, options is filled with additional info - Bool_t attach = kFALSE; - if (strlen(fUrl.GetOptions()) > 0) { - attach = kTRUE; - // A flag from the GUI - TString opts = fUrl.GetOptions(); - if (opts.Contains("GUI")) { - SetBit(TProof::kUsingSessionGui); - opts.Remove(opts.Index("GUI")); - fUrl.SetOptions(opts); - } - } - - if (TestBit(TProof::kIsMaster)) { - // Fill default conf file and conf dir - if (!conffile || !conffile[0]) - fConfFile = kPROOF_ConfFile; - if (!confdir || !confdir[0]) - fConfDir = kPROOF_ConfDir; - // The group; the client receives it in the kPROOF_SESSIONTAG message - if (gProofServ) fGroup = gProofServ->GetGroup(); - } else { - fConfDir = confdir; - fConfFile = conffile; - } - - // Analysise the conffile field - if (fConfFile.Contains("workers=0")) fConfFile.ReplaceAll("workers=0", "masteronly"); - ParseConfigField(fConfFile); - - fWorkDir = gSystem->WorkingDirectory(); - fLogLevel = loglevel; - fProtocol = kPROOF_Protocol; - fSendGroupView = kTRUE; - fImage = fMasterServ ? "" : ""; - fIntHandler = 0; - fStatus = 0; - fRecvMessages = new TList; - fRecvMessages->SetOwner(kTRUE); - fSlaveInfo = 0; - fChains = new TList; - fAvailablePackages = 0; - fEnabledPackages = 0; - fRunningDSets = 0; - fEndMaster = TestBit(TProof::kIsMaster) ? kTRUE : kFALSE; - fInputData = 0; - ResetBit(TProof::kNewInputData); - fPrintProgress = 0; - - // Timeout for some collect actions - fCollectTimeout = gEnv->GetValue("Proof.CollectTimeout", -1); - - // Should the workers be started dynamically; default: no - fDynamicStartup = gEnv->GetValue("Proof.DynamicStartup", kFALSE); - - // Default entry point for the data pool is the master - if (TestBit(TProof::kIsClient)) - fDataPoolUrl.Form("root://%s", fMaster.Data()); - else - fDataPoolUrl = ""; - - fProgressDialog = 0; - fProgressDialogStarted = kFALSE; - - // Default alias is the master name - TString al = (alias) ? alias : fMaster.Data(); - SetAlias(al); - - // Client logging of messages from the master and slaves - fRedirLog = kFALSE; - if (TestBit(TProof::kIsClient)) { - fLogFileName.Form("%s/ProofLog_%d", gSystem->TempDirectory(), gSystem->GetPid()); - if ((fLogFileW = fopen(fLogFileName, "w")) == 0) - Error("Init", "could not create temporary logfile"); - if ((fLogFileR = fopen(fLogFileName, "r")) == 0) - Error("Init", "could not open temp logfile for reading"); - } - fLogToWindowOnly = kFALSE; - - // Status of cluster - fNotIdle = 0; - // Query type - fSync = (attach) ? kFALSE : kTRUE; - // Not enqueued - fIsWaiting = kFALSE; - - // Counters - fBytesRead = 0; - fRealTime = 0; - fCpuTime = 0; - - // List of queries - fQueries = 0; - fOtherQueries = 0; - fDrawQueries = 0; - fMaxDrawQueries = 1; - fSeqNum = 0; - - // Remote ID of the session - fSessionID = -1; - - // Part of active query - fWaitingSlaves = 0; - - // Make remote PROOF player - fPlayer = 0; - MakePlayer(); - - fFeedback = new TList; - fFeedback->SetOwner(); - fFeedback->SetName("FeedbackList"); - AddInput(fFeedback); - - // sort slaves by descending performance index - fSlaves = new TSortedList(kSortDescending); - fActiveSlaves = new TList; - fInactiveSlaves = new TList; - fUniqueSlaves = new TList; - fAllUniqueSlaves = new TList; - fNonUniqueMasters = new TList; - fBadSlaves = new TList; - fAllMonitor = new TMonitor; - fActiveMonitor = new TMonitor; - fUniqueMonitor = new TMonitor; - fAllUniqueMonitor = new TMonitor; - fCurrentMonitor = 0; - - fTerminatedSlaveInfos = new TList; - fTerminatedSlaveInfos->SetOwner(kTRUE); - - fLoadedMacros = 0; - fPackMgr = 0; - - // Enable optimized sending of streamer infos to use embedded backward/forward - // compatibility support between different ROOT versions and different versions of - // users classes - Bool_t enableSchemaEvolution = gEnv->GetValue("Proof.SchemaEvolution",1); - if (enableSchemaEvolution) { - TMessage::EnableSchemaEvolutionForAll(); - } else { - Info("TProof", "automatic schema evolution in TMessage explicitly disabled"); - } - - if (IsMaster()) { - // to make UploadPackage() method work on the master as well. - if (gProofServ) fPackMgr = gProofServ->GetPackMgr(); - } else { - - TString sandbox; - if (GetSandbox(sandbox, kTRUE) != 0) { - Error("Init", "failure asserting sandbox directory %s", sandbox.Data()); - return 0; - } - - // Package Dir - TString packdir = gEnv->GetValue("Proof.PackageDir", ""); - if (packdir.IsNull()) - packdir.Form("%s/%s", sandbox.Data(), kPROOF_PackDir); - if (AssertPath(packdir, kTRUE) != 0) { - Error("Init", "failure asserting directory %s", packdir.Data()); - return 0; - } - fPackMgr = new TPackMgr(packdir); - if (gDebug > 0) - Info("Init", "package directory set to %s", packdir.Data()); - } - - if (!IsMaster()) { - // List of directories where to look for global packages - TString globpack = gEnv->GetValue("Proof.GlobalPackageDirs",""); - TProofServ::ResolveKeywords(globpack); - Int_t nglb = TPackMgr::RegisterGlobalPath(globpack); - if (gDebug > 0) - Info("Init", " %d global package directories registered", nglb); - } - - // Master may want dynamic startup - if (fDynamicStartup) { - if (!IsMaster()) { - // If on client - start the master - if (!StartSlaves(attach)) - return 0; - } - } else { - - // Master Only mode (for operations requiring only the master, e.g. dataset browsing, - // result retrieving, ...) - Bool_t masterOnly = gEnv->GetValue("Proof.MasterOnly", kFALSE); - if (!IsMaster() || !masterOnly) { - // Start slaves (the old, static, per-session way) - if (!StartSlaves(attach)) - return 0; - // Client: Is Master in dynamic startup mode? - if (!IsMaster()) { - Int_t dyn = 0; - GetRC("Proof.DynamicStartup", dyn); - if (dyn != 0) fDynamicStartup = kTRUE; - } - } - } - // we are now properly initialized - fValid = kTRUE; - - // De-activate monitor (will be activated in Collect) - fAllMonitor->DeActivateAll(); - - // By default go into parallel mode - Int_t nwrk = GetRemoteProtocol() > 35 ? -1 : 9999; - TNamed *n = 0; - if (TProof::GetEnvVars() && - (n = (TNamed *) TProof::GetEnvVars()->FindObject("PROOF_NWORKERS"))) { - TString s(n->GetTitle()); - if (s.IsDigit()) nwrk = s.Atoi(); - } - GoParallel(nwrk, attach); - - // Send relevant initial state to slaves - if (!attach) - SendInitialState(); - else if (!IsIdle()) - // redirect log - fRedirLog = kTRUE; - - // Done at this point, the alias will be communicated to the coordinator, if any - if (TestBit(TProof::kIsClient)) - SetAlias(al); - - SetActive(kFALSE); - - if (IsValid()) { - - // Activate input handler - ActivateAsyncInput(); - - R__LOCKGUARD(gROOTMutex); - gROOT->GetListOfSockets()->Add(this); - } - - AskParallel(); - - return fActiveSlaves->GetSize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the sandbox path from ' Proof.Sandbox' or the alternative var 'rc'. -/// Use the existing setting or the default if nothing is found. -/// If 'assert' is kTRUE, make also sure that the path exists. -/// Return 0 on success, -1 on failure - -Int_t TProof::GetSandbox(TString &sb, Bool_t assert, const char *rc) -{ - // Get it from 'rc', if defined - if (rc && strlen(rc)) sb = gEnv->GetValue(rc, sb); - // Or use the default 'rc' - if (sb.IsNull()) sb = gEnv->GetValue("Proof.Sandbox", ""); - // If nothing found , use the default - if (sb.IsNull()) sb.Form("~/%s", kPROOF_WorkDir); - // Expand special settings - if (sb == ".") { - sb = gSystem->pwd(); - } else if (sb == "..") { - sb = gSystem->GetDirName(gSystem->pwd()); - } - gSystem->ExpandPathName(sb); - - // Assert the path, if required - if (assert && AssertPath(sb, kTRUE) != 0) return -1; - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// The config file field may contain special instructions which need to be -/// parsed at the beginning, e.g. for debug runs with valgrind. -/// Several options can be given separated by a ',' - -void TProof::ParseConfigField(const char *config) -{ - TString sconf(config), opt; - Ssiz_t from = 0; - #ifdef R__LINUX - Bool_t cpuPin = kFALSE; - #endif - - // Analysise the field - const char *cq = (IsLite()) ? "\"" : ""; - while (sconf.Tokenize(opt, from, ",")) { - if (opt.IsNull()) continue; - - if (opt.BeginsWith("valgrind")) { - // Any existing valgrind setting? User can give full settings, which we fully respect, - // or pass additional options for valgrind by prefixing 'valgrind_opts:'. For example, - // TProof::AddEnvVar("PROOF_MASTER_WRAPPERCMD", "valgrind_opts:--time-stamp --leak-check=full" - // will add option "--time-stamp --leak-check=full" to our default options - TString mst, top, sub, wrk, all; - TList *envs = fgProofEnvList; - TNamed *n = 0; - if (envs) { - if ((n = (TNamed *) envs->FindObject("PROOF_WRAPPERCMD"))) - all = n->GetTitle(); - if ((n = (TNamed *) envs->FindObject("PROOF_MASTER_WRAPPERCMD"))) - mst = n->GetTitle(); - if ((n = (TNamed *) envs->FindObject("PROOF_TOPMASTER_WRAPPERCMD"))) - top = n->GetTitle(); - if ((n = (TNamed *) envs->FindObject("PROOF_SUBMASTER_WRAPPERCMD"))) - sub = n->GetTitle(); - if ((n = (TNamed *) envs->FindObject("PROOF_SLAVE_WRAPPERCMD"))) - wrk = n->GetTitle(); - } - if (all != "" && mst == "") mst = all; - if (all != "" && top == "") top = all; - if (all != "" && sub == "") sub = all; - if (all != "" && wrk == "") wrk = all; - if (all != "" && all.BeginsWith("valgrind_opts:")) { - // The field is used to add an option Reset the setting - Info("ParseConfigField","valgrind run: resetting 'PROOF_WRAPPERCMD':" - " must be set again for next run , if any"); - TProof::DelEnvVar("PROOF_WRAPPERCMD"); - } - TString var, cmd; - cmd.Form("%svalgrind -v --suppressions=/etc/valgrind-root.supp", cq); - TString mstlab("NO"), wrklab("NO"); - Bool_t doMaster = (opt == "valgrind" || (opt.Contains("master") && - !opt.Contains("topmaster") && !opt.Contains("submaster"))) - ? kTRUE : kFALSE; - if (doMaster) { - if (!IsLite()) { - // Check if we have to add a var - if (mst == "" || mst.BeginsWith("valgrind_opts:")) { - mst.ReplaceAll("valgrind_opts:",""); - var.Form("%s --log-file=.valgrind.log %s", cmd.Data(), mst.Data()); - TProof::AddEnvVar("PROOF_MASTER_WRAPPERCMD", var); - mstlab = "YES"; - } else if (mst != "") { - mstlab = "YES"; - } - } else { - if (opt.Contains("master")) { - Warning("ParseConfigField", - "master valgrinding does not make sense for PROOF-Lite: ignoring"); - opt.ReplaceAll("master", ""); - if (!opt.Contains("workers")) return; - } - if (opt == "valgrind" || opt == "valgrind=") opt = "valgrind=workers"; - } - } - if (opt.Contains("topmaster")) { - // Check if we have to add a var - if (top == "" || top.BeginsWith("valgrind_opts:")) { - top.ReplaceAll("valgrind_opts:",""); - var.Form("%s --log-file=.valgrind.log %s", cmd.Data(), top.Data()); - TProof::AddEnvVar("PROOF_TOPMASTER_WRAPPERCMD", var); - mstlab = "YES"; - } else if (top != "") { - mstlab = "YES"; - } - } - if (opt.Contains("submaster")) { - // Check if we have to add a var - if (sub == "" || sub.BeginsWith("valgrind_opts:")) { - sub.ReplaceAll("valgrind_opts:",""); - var.Form("%s --log-file=.valgrind.log %s", cmd.Data(), sub.Data()); - TProof::AddEnvVar("PROOF_SUBMASTER_WRAPPERCMD", var); - mstlab = "YES"; - } else if (sub != "") { - mstlab = "YES"; - } - } - if (opt.Contains("=workers") || opt.Contains("+workers")) { - // Check if we have to add a var - if (wrk == "" || wrk.BeginsWith("valgrind_opts:")) { - wrk.ReplaceAll("valgrind_opts:",""); - var.Form("%s --log-file=.__valgrind__.log %s%s", cmd.Data(), wrk.Data(), cq); - TProof::AddEnvVar("PROOF_SLAVE_WRAPPERCMD", var); - TString nwrks("2"); - Int_t inw = opt.Index('#'); - if (inw != kNPOS) { - nwrks = opt(inw+1, opt.Length()); - if (!nwrks.IsDigit()) nwrks = "2"; - } - // Set the relevant variables - if (!IsLite()) { - TProof::AddEnvVar("PROOF_NWORKERS", nwrks); - } else { - gEnv->SetValue("ProofLite.Workers", nwrks.Atoi()); - } - wrklab = nwrks; - // Register the additional worker log in the session file - // (for the master this is done automatically) - TProof::AddEnvVar("PROOF_ADDITIONALLOG", "__valgrind__.log*"); - } else if (wrk != "") { - wrklab = "ALL"; - } - } - // Increase the relevant timeouts - if (!IsLite()) { - TProof::AddEnvVar("PROOF_INTWAIT", "5000"); - gEnv->SetValue("Proof.SocketActivityTimeout", 6000); - } else { - gEnv->SetValue("ProofLite.StartupTimeOut", 5000); - } - // Warn for slowness - Printf(" "); - if (!IsLite()) { - Printf(" ---> Starting a debug run with valgrind (master:%s, workers:%s)", mstlab.Data(), wrklab.Data()); - } else { - Printf(" ---> Starting a debug run with valgrind (workers:%s)", wrklab.Data()); - } - Printf(" ---> Please be patient: startup may be VERY slow ..."); - Printf(" ---> Logs will be available as special tags in the log window (from the progress dialog or TProof::LogViewer()) "); - Printf(" ---> (Reminder: this debug run makes sense only if you are running a debug version of ROOT)"); - Printf(" "); - - } else if (opt.BeginsWith("igprof-pp")) { - - // IgProf profiling on master and worker. PROOF does not set the - // environment for you: proper environment variables (like PATH and - // LD_LIBRARY_PATH) should be set externally - - Printf("*** Requested IgProf performance profiling ***"); - TString addLogExt = "__igprof.pp__.log"; - TString addLogFmt = "igprof -pk -pp -t proofserv.exe -o %s.%s"; - TString tmp; - - if (IsLite()) { - addLogFmt.Append("\""); - addLogFmt.Prepend("\""); - } - - tmp.Form(addLogFmt.Data(), "", addLogExt.Data()); - TProof::AddEnvVar("PROOF_MASTER_WRAPPERCMD", tmp.Data()); - - tmp.Form(addLogFmt.Data(), "", addLogExt.Data()); - TProof::AddEnvVar("PROOF_SLAVE_WRAPPERCMD", tmp.Data() ); - - TProof::AddEnvVar("PROOF_ADDITIONALLOG", addLogExt.Data()); - - } else if (opt.BeginsWith("cpupin=")) { - // Enable CPU pinning. Takes as argument the list of processor IDs - // that will be used in order. Processor IDs are numbered from 0, - // use likwid to see how they are organized. A possible parameter - // format would be: - // - // cpupin=3+4+0+9+10+22+7 - // - // Only the specified processor IDs will be used in a round-robin - // fashion, dealing with the fact that you can request more workers - // than the number of processor IDs you have specified. - // - // To use all available processors in their order: - // - // cpupin=* - - opt.Remove(0, 7); - - // Remove any char which is neither a number nor a plus '+' - for (Ssiz_t i=0; i '9'))) - opt[i] = '_'; - } - opt.ReplaceAll("_", ""); - TProof::AddEnvVar("PROOF_SLAVE_CPUPIN_ORDER", opt); - #ifdef R__LINUX - cpuPin = kTRUE; - #endif - } else if (opt.BeginsWith("workers=")) { - - // Request for a given number of workers (within the max) or worker - // startup combination: - // workers=5 start max 5 workers (or less, if less are assigned) - // workers=2x start max 2 workers per node (or less, if less are assigned) - opt.ReplaceAll("workers=",""); - TProof::AddEnvVar("PROOF_NWORKERS", opt); - } - } - - // In case of PROOF-Lite, enable CPU pinning when requested (Linux only) - #ifdef R__LINUX - if (IsLite() && cpuPin) { - Printf("*** Requested CPU pinning ***"); - const TList *ev = GetEnvVars(); - const char *pinCmd = "taskset -c "; - TString val; - TNamed *p; - if (ev && (p = dynamic_cast(ev->FindObject("PROOF_SLAVE_WRAPPERCMD")))) { - val = p->GetTitle(); - val.Insert(val.Length()-1, " "); - val.Insert(val.Length()-1, pinCmd); - } - else { - val.Form("\"%s\"", pinCmd); - } - TProof::AddEnvVar("PROOF_SLAVE_WRAPPERCMD", val.Data()); - } - #endif -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that 'path' exists; if 'writable' is kTRUE, make also sure -/// that the path is writable - -Int_t TProof::AssertPath(const char *inpath, Bool_t writable) -{ - if (!inpath || strlen(inpath) <= 0) { - Error("AssertPath", "undefined input path"); - return -1; - } - - TString path(inpath); - gSystem->ExpandPathName(path); - - if (gSystem->AccessPathName(path, kFileExists)) { - if (gSystem->mkdir(path, kTRUE) != 0) { - Error("AssertPath", "could not create path %s", path.Data()); - return -1; - } - } - // It must be writable - if (gSystem->AccessPathName(path, kWritePermission) && writable) { - if (gSystem->Chmod(path, 0666) != 0) { - Error("AssertPath", "could not make path %s writable", path.Data()); - return -1; - } - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set manager and schedule its destruction after this for clean -/// operations. - -void TProof::SetManager(TProofMgr *mgr) -{ - fManager = mgr; - - if (mgr) { - R__LOCKGUARD(gROOTMutex); - gROOT->GetListOfSockets()->Remove(mgr); - gROOT->GetListOfSockets()->Add(mgr); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Works on the master node only. -/// It starts workers on the machines in workerList and sets the paths, -/// packages and macros as on the master. -/// It is a subbstitute for StartSlaves(...) -/// The code is mostly the master part of StartSlaves, -/// with the parallel startup removed. - -Int_t TProof::AddWorkers(TList *workerList) -{ - if (!IsMaster()) { - Error("AddWorkers", "AddWorkers can only be called on the master!"); - return -1; - } - - if (!workerList || !(workerList->GetSize())) { - Error("AddWorkers", "empty list of workers!"); - return -2; - } - - // Code taken from master part of StartSlaves with the parllel part removed - - fImage = gProofServ->GetImage(); - if (fImage.IsNull()) - fImage.Form("%s:%s", TUrl(gSystem->HostName()).GetHostFQDN(), gProofServ->GetWorkDir()); - - // Get all workers - UInt_t nSlaves = workerList->GetSize(); - UInt_t nSlavesDone = 0; - Int_t ord = 0; - - // Loop over all new workers and start them (if we had already workers it means we are - // increasing parallelism or that is not the first time we are called) - Bool_t goMoreParallel = (fSlaves->GetEntries() > 0) ? kTRUE : kFALSE; - - // A list of TSlave objects for workers that are being added - TList *addedWorkers = new TList(); - if (!addedWorkers) { - // This is needed to silence Coverity ... - Error("AddWorkers", "cannot create new list for the workers to be added"); - return -2; - } - addedWorkers->SetOwner(kFALSE); - TListIter next(workerList); - TObject *to; - TProofNodeInfo *worker; - TSlaveInfo *dummysi = new TSlaveInfo(); - while ((to = next())) { - // Get the next worker from the list - worker = (TProofNodeInfo *)to; - - // Read back worker node info - const Char_t *image = worker->GetImage().Data(); - const Char_t *workdir = worker->GetWorkDir().Data(); - Int_t perfidx = worker->GetPerfIndex(); - Int_t sport = worker->GetPort(); - if (sport == -1) - sport = fUrl.GetPort(); - - // Create worker server - TString fullord; - if (worker->GetOrdinal().Length() > 0) { - fullord.Form("%s.%s", gProofServ->GetOrdinal(), worker->GetOrdinal().Data()); - } else { - fullord.Form("%s.%d", gProofServ->GetOrdinal(), ord); - } - - // Remove worker from the list of workers terminated gracefully - dummysi->SetOrdinal(fullord); - TSlaveInfo *rmsi = (TSlaveInfo *)fTerminatedSlaveInfos->Remove(dummysi); - SafeDelete(rmsi); - - // Create worker server - TString wn(worker->GetNodeName()); - if (wn == "localhost" || wn.BeginsWith("localhost.")) wn = gSystem->HostName(); - TUrl u(TString::Format("%s:%d", wn.Data(), sport)); - // Add group info in the password firdl, if any - if (strlen(gProofServ->GetGroup()) > 0) { - // Set also the user, otherwise the password is not exported - if (strlen(u.GetUser()) <= 0) - u.SetUser(gProofServ->GetUser()); - u.SetPasswd(gProofServ->GetGroup()); - } - TSlave *slave = 0; - if (worker->IsWorker()) { - slave = CreateSlave(u.GetUrl(), fullord, perfidx, image, workdir); - } else { - slave = CreateSubmaster(u.GetUrl(), fullord, - image, worker->GetMsd(), worker->GetNWrks()); - } - - // Add to global list (we will add to the monitor list after - // finalizing the server startup) - Bool_t slaveOk = kTRUE; - fSlaves->Add(slave); - if (slave->IsValid()) { - addedWorkers->Add(slave); - } else { - slaveOk = kFALSE; - fBadSlaves->Add(slave); - Warning("AddWorkers", "worker '%s' is invalid", slave->GetOrdinal()); - } - - PDB(kGlobal,3) - Info("AddWorkers", "worker on host %s created" - " and added to list (ord: %s)", worker->GetName(), slave->GetOrdinal()); - - // Notify opening of connection - nSlavesDone++; - TMessage m(kPROOF_SERVERSTARTED); - m << TString("Opening connections to workers") << nSlaves - << nSlavesDone << slaveOk; - gProofServ->GetSocket()->Send(m); - - ord++; - } //end of the worker loop - SafeDelete(dummysi); - - // Cleanup - SafeDelete(workerList); - - nSlavesDone = 0; - - // Here we finalize the server startup: in this way the bulk - // of remote operations are almost parallelized - TIter nxsl(addedWorkers); - TSlave *sl = 0; - while ((sl = (TSlave *) nxsl())) { - - // Finalize setup of the server - if (sl->IsValid()) - sl->SetupServ(TSlave::kSlave, 0); - - // Monitor good slaves - Bool_t slaveOk = kTRUE; - if (sl->IsValid()) { - fAllMonitor->Add(sl->GetSocket()); - PDB(kGlobal,3) - Info("AddWorkers", "worker on host %s finalized" - " and added to list", sl->GetOrdinal()); - } else { - slaveOk = kFALSE; - fBadSlaves->Add(sl); - } - - // Notify end of startup operations - nSlavesDone++; - TMessage m(kPROOF_SERVERSTARTED); - m << TString("Setting up worker servers") << nSlaves - << nSlavesDone << slaveOk; - gProofServ->GetSocket()->Send(m); - } - - // Now set new state on the added workers (on all workers for simplicity) - // use fEnabledPackages, fLoadedMacros, - // gSystem->GetDynamicPath() and gSystem->GetIncludePath() - // no need to load packages that are only loaded and not enabled (dyn mode) - Int_t nwrk = GetRemoteProtocol() > 35 ? -1 : 9999; - TNamed *n = 0; - if (TProof::GetEnvVars() && - (n = (TNamed *) TProof::GetEnvVars()->FindObject("PROOF_NWORKERS"))) { - TString s(n->GetTitle()); - if (s.IsDigit()) nwrk = s.Atoi(); - } - - if (fDynamicStartup && goMoreParallel) { - - PDB(kGlobal, 3) - Info("AddWorkers", "will invoke GoMoreParallel()"); - Int_t nw = GoMoreParallel(nwrk); - PDB(kGlobal, 3) - Info("AddWorkers", "GoMoreParallel()=%d", nw); - - } - else { - // Not in Dynamic Workers mode - PDB(kGlobal, 3) - Info("AddWorkers", "will invoke GoParallel()"); - GoParallel(nwrk, kFALSE, 0); - } - - // Set worker processing environment - SetupWorkersEnv(addedWorkers, goMoreParallel); - - // Update list of current workers - PDB(kGlobal, 3) - Info("AddWorkers", "will invoke SaveWorkerInfo()"); - SaveWorkerInfo(); - - // Inform the client that the number of workers has changed - if (fDynamicStartup && gProofServ) { - PDB(kGlobal, 3) - Info("AddWorkers", "will invoke SendParallel()"); - gProofServ->SendParallel(kTRUE); - - if (goMoreParallel && fPlayer) { - // In case we are adding workers dynamically to an existing process, we - // should invoke a special player's Process() to set only added workers - // to the proper state - PDB(kGlobal, 3) - Info("AddWorkers", "will send the PROCESS message to selected workers"); - fPlayer->JoinProcess(addedWorkers); - // Update merger counters (new workers are not yet active) - fMergePrg.SetNWrks(fActiveSlaves->GetSize() + addedWorkers->GetSize()); - } - } - - // Cleanup - delete addedWorkers; - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set up packages, loaded macros, include and lib paths ... - -void TProof::SetupWorkersEnv(TList *addedWorkers, Bool_t increasingWorkers) -{ - TList *server_packs = gProofServ ? gProofServ->GetEnabledPackages() : nullptr; - // Packages - TList *packs = server_packs ? server_packs : GetEnabledPackages(); - if (packs && packs->GetSize() > 0) { - TIter nxp(packs); - TPair *pck = 0; - while ((pck = (TPair *) nxp())) { - // Upload and Enable methods are intelligent and avoid - // re-uploading or re-enabling of a package to a node that has it. - if (fDynamicStartup && increasingWorkers) { - // Upload only on added workers - PDB(kGlobal, 3) - Info("SetupWorkersEnv", "will invoke UploadPackage() and EnablePackage() on added workers"); - if (UploadPackage(pck->GetName(), kUntar, addedWorkers) >= 0) - EnablePackage(pck->GetName(), (TList *) pck->Value(), kTRUE, addedWorkers); - } else { - PDB(kGlobal, 3) - Info("SetupWorkersEnv", "will invoke UploadPackage() and EnablePackage() on all workers"); - if (UploadPackage(pck->GetName()) >= 0) - EnablePackage(pck->GetName(), (TList *) pck->Value(), kTRUE); - } - } - } - - if (server_packs) { - server_packs->Delete(); - delete server_packs; - } - - // Loaded macros - if (fLoadedMacros) { - TIter nxp(fLoadedMacros); - TObjString *os = 0; - while ((os = (TObjString *) nxp())) { - PDB(kGlobal, 3) { - Info("SetupWorkersEnv", "will invoke Load() on selected workers"); - Printf("Loading a macro : %s", os->GetName()); - } - Load(os->GetName(), kTRUE, kTRUE, addedWorkers); - } - } - - // Dynamic path - TString dyn = gSystem->GetDynamicPath(); - dyn.ReplaceAll(":", " "); - dyn.ReplaceAll("\"", " "); - PDB(kGlobal, 3) - Info("SetupWorkersEnv", "will invoke AddDynamicPath() on selected workers"); - AddDynamicPath(dyn, kFALSE, addedWorkers, kFALSE); // Do not Collect - - // Include path - TString inc = gSystem->GetIncludePath(); - inc.ReplaceAll("-I", " "); - inc.ReplaceAll("\"", " "); - PDB(kGlobal, 3) - Info("SetupWorkersEnv", "will invoke AddIncludePath() on selected workers"); - AddIncludePath(inc, kFALSE, addedWorkers, kFALSE); // Do not Collect - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Used for shuting down the workres after a query is finished. -/// Sends each of the workers from the workerList, a kPROOF_STOP message. -/// If the workerList == 0, shutdown all the workers. - -Int_t TProof::RemoveWorkers(TList *workerList) -{ - if (!IsMaster()) { - Error("RemoveWorkers", "RemoveWorkers can only be called on the master!"); - return -1; - } - - fFileMap.clear(); // This could be avoided if CopyFromCache was used in SendFile - - if (!workerList) { - // shutdown all the workers - TIter nxsl(fSlaves); - TSlave *sl = 0; - while ((sl = (TSlave *) nxsl())) { - // Shut down the worker assumig that it is not processing - TerminateWorker(sl); - } - - } else { - if (!(workerList->GetSize())) { - Error("RemoveWorkers", "The list of workers should not be empty!"); - return -2; - } - - // Loop over all the workers and stop them - TListIter next(workerList); - TObject *to; - TProofNodeInfo *worker; - while ((to = next())) { - TSlave *sl = 0; - if (!strcmp(to->ClassName(), "TProofNodeInfo")) { - // Get the next worker from the list - worker = (TProofNodeInfo *)to; - TIter nxsl(fSlaves); - while ((sl = (TSlave *) nxsl())) { - // Shut down the worker assumig that it is not processing - if (sl->GetName() == worker->GetNodeName()) - break; - } - } else if (to->InheritsFrom(TSlave::Class())) { - sl = (TSlave *) to; - } else { - Warning("RemoveWorkers","unknown object type: %s - it should be" - " TProofNodeInfo or inheriting from TSlave", to->ClassName()); - } - // Shut down the worker assumig that it is not processing - if (sl) { - if (gDebug > 0) - Info("RemoveWorkers","terminating worker %s", sl->GetOrdinal()); - TerminateWorker(sl); - } - } - } - - // Update also the master counter - if (gProofServ && fSlaves->GetSize() <= 0) gProofServ->ReleaseWorker("master"); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Start up PROOF slaves. - -Bool_t TProof::StartSlaves(Bool_t attach) -{ - // If this is a master server, find the config file and start slave - // servers as specified in the config file - if (TestBit(TProof::kIsMaster)) { - - Int_t pc = 0; - TList *workerList = new TList; - // Get list of workers - if (gProofServ->GetWorkers(workerList, pc) == TProofServ::kQueryStop) { - TString emsg("no resource currently available for this session: please retry later"); - if (gDebug > 0) Info("StartSlaves", "%s", emsg.Data()); - gProofServ->SendAsynMessage(emsg.Data()); - return kFALSE; - } - // Setup the workers - if (AddWorkers(workerList) < 0) - return kFALSE; - - } else { - - // create master server - Printf("Starting master: opening connection ..."); - TSlave *slave = CreateSubmaster(fUrl.GetUrl(), "0", "master", 0); - - if (slave->IsValid()) { - - // Notify - fprintf(stderr,"Starting master:" - " connection open: setting up server ... \r"); - StartupMessage("Connection to master opened", kTRUE, 1, 1); - - if (!attach) { - - // Set worker interrupt handler - slave->SetInterruptHandler(kTRUE); - - // Finalize setup of the server - slave->SetupServ(TSlave::kMaster, fConfFile); - - if (slave->IsValid()) { - - // Notify - Printf("Starting master: OK "); - StartupMessage("Master started", kTRUE, 1, 1); - - // check protocol compatibility - // protocol 1 is not supported anymore - if (fProtocol == 1) { - Error("StartSlaves", - "client and remote protocols not compatible (%d and %d)", - kPROOF_Protocol, fProtocol); - slave->Close("S"); - delete slave; - return kFALSE; - } - - fSlaves->Add(slave); - fAllMonitor->Add(slave->GetSocket()); - - // Unset worker interrupt handler - slave->SetInterruptHandler(kFALSE); - - // Set interrupt PROOF handler from now on - fIntHandler = new TProofInterruptHandler(this); - - // Give-up after 5 minutes - Int_t rc = Collect(slave, 300); - Int_t slStatus = slave->GetStatus(); - if (slStatus == -99 || slStatus == -98 || rc == 0) { - fSlaves->Remove(slave); - fAllMonitor->Remove(slave->GetSocket()); - if (slStatus == -99) - Error("StartSlaves", "no resources available or problems setting up workers (check logs)"); - else if (slStatus == -98) - Error("StartSlaves", "could not setup output redirection on master"); - else - Error("StartSlaves", "setting up master"); - slave->Close("S"); - delete slave; - return 0; - } - - if (!slave->IsValid()) { - fSlaves->Remove(slave); - fAllMonitor->Remove(slave->GetSocket()); - slave->Close("S"); - delete slave; - Error("StartSlaves", - "failed to setup connection with PROOF master server"); - return kFALSE; - } - - if (!gROOT->IsBatch() && TestBit(kUseProgressDialog)) { - if ((fProgressDialog = - gROOT->GetPluginManager()->FindHandler("TProofProgressDialog"))) - if (fProgressDialog->LoadPlugin() == -1) - fProgressDialog = 0; - } - } else { - // Notify - Printf("Starting master: failure"); - } - } else { - - // Notify - Printf("Starting master: OK "); - StartupMessage("Master attached", kTRUE, 1, 1); - - if (!gROOT->IsBatch() && TestBit(kUseProgressDialog)) { - if ((fProgressDialog = - gROOT->GetPluginManager()->FindHandler("TProofProgressDialog"))) - if (fProgressDialog->LoadPlugin() == -1) - fProgressDialog = 0; - } - - fSlaves->Add(slave); - fIntHandler = new TProofInterruptHandler(this); - } - - } else { - delete slave; - // Notify only if verbosity is on: most likely the failure has already been notified - if (gDebug > 0) - Error("StartSlaves", "failed to create (or connect to) the PROOF master server"); - return kFALSE; - } - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close all open slave servers. -/// Client can decide to shutdown the remote session by passing option is 'S' -/// or 's'. Default for clients is detach, if supported. Masters always -/// shutdown the remote counterpart. - -void TProof::Close(Option_t *opt) -{ - { std::lock_guard lock(fCloseMutex); - - fValid = kFALSE; - if (fSlaves) { - if (fIntHandler) - fIntHandler->Remove(); - - TIter nxs(fSlaves); - TSlave *sl = 0; - while ((sl = (TSlave *)nxs())) - sl->Close(opt); - - fActiveSlaves->Clear("nodelete"); - fUniqueSlaves->Clear("nodelete"); - fAllUniqueSlaves->Clear("nodelete"); - fNonUniqueMasters->Clear("nodelete"); - fBadSlaves->Clear("nodelete"); - fInactiveSlaves->Clear("nodelete"); - fSlaves->Delete(); - } - } - - { R__LOCKGUARD(gROOTMutex); - gROOT->GetListOfSockets()->Remove(this); - - if (fChains) { - while (TChain *chain = dynamic_cast (fChains->First()) ) { - // remove "chain" from list - chain->SetProof(0); - RemoveChain(chain); - } - } - - if (IsProofd()) { - - gROOT->GetListOfProofs()->Remove(this); - if (gProof && gProof == this) { - // Set previous proofd-related as default - TIter pvp(gROOT->GetListOfProofs(), kIterBackward); - while ((gProof = (TProof *)pvp())) { - if (gProof->IsProofd()) - break; - } - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a new TSlave of type TSlave::kSlave. -/// Note: creation of TSlave is private with TProof as a friend. -/// Derived classes must use this function to create slaves. - -TSlave *TProof::CreateSlave(const char *url, const char *ord, - Int_t perf, const char *image, const char *workdir) -{ - TSlave* sl = TSlave::Create(url, ord, perf, image, - this, TSlave::kSlave, workdir, 0); - - if (sl->IsValid()) { - sl->SetInputHandler(new TProofInputHandler(this, sl->GetSocket())); - // must set fParallel to 1 for slaves since they do not - // report their fParallel with a LOG_DONE message - sl->fParallel = 1; - } - - return sl; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Create a new TSlave of type TSlave::kMaster. -/// Note: creation of TSlave is private with TProof as a friend. -/// Derived classes must use this function to create slaves. - -TSlave *TProof::CreateSubmaster(const char *url, const char *ord, - const char *image, const char *msd, Int_t nwk) -{ - TSlave *sl = TSlave::Create(url, ord, 100, image, this, - TSlave::kMaster, 0, msd, nwk); - - if (sl->IsValid()) { - sl->SetInputHandler(new TProofInputHandler(this, sl->GetSocket())); - } - - return sl; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Find slave that has TSocket s. Returns 0 in case slave is not found. - -TSlave *TProof::FindSlave(TSocket *s) const -{ - TSlave *sl; - TIter next(fSlaves); - - while ((sl = (TSlave *)next())) { - if (sl->IsValid() && sl->GetSocket() == s) - return sl; - } - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add to the fUniqueSlave list the active slaves that have a unique -/// (user) file system image. This information is used to transfer files -/// only once to nodes that share a file system (an image). Submasters -/// which are not in fUniqueSlaves are put in the fNonUniqueMasters -/// list. That list is used to trigger the transferring of files to -/// the submaster's unique slaves without the need to transfer the file -/// to the submaster. - -void TProof::FindUniqueSlaves() -{ - fUniqueSlaves->Clear(); - fUniqueMonitor->RemoveAll(); - fAllUniqueSlaves->Clear(); - fAllUniqueMonitor->RemoveAll(); - fNonUniqueMasters->Clear(); - - TIter next(fActiveSlaves); - - while (TSlave *sl = dynamic_cast(next())) { - if (fImage == sl->fImage) { - if (sl->GetSlaveType() == TSlave::kMaster) { - fNonUniqueMasters->Add(sl); - fAllUniqueSlaves->Add(sl); - fAllUniqueMonitor->Add(sl->GetSocket()); - } - continue; - } - - TIter next2(fUniqueSlaves); - TSlave *replace_slave = 0; - Bool_t add = kTRUE; - while (TSlave *sl2 = dynamic_cast(next2())) { - if (sl->fImage == sl2->fImage) { - add = kFALSE; - if (sl->GetSlaveType() == TSlave::kMaster) { - if (sl2->GetSlaveType() == TSlave::kSlave) { - // give preference to master - replace_slave = sl2; - add = kTRUE; - } else if (sl2->GetSlaveType() == TSlave::kMaster) { - fNonUniqueMasters->Add(sl); - fAllUniqueSlaves->Add(sl); - fAllUniqueMonitor->Add(sl->GetSocket()); - } else { - Error("FindUniqueSlaves", "TSlave is neither Master nor Slave"); - R__ASSERT(0); - } - } - break; - } - } - - if (add) { - fUniqueSlaves->Add(sl); - fAllUniqueSlaves->Add(sl); - fUniqueMonitor->Add(sl->GetSocket()); - fAllUniqueMonitor->Add(sl->GetSocket()); - if (replace_slave) { - fUniqueSlaves->Remove(replace_slave); - fAllUniqueSlaves->Remove(replace_slave); - fUniqueMonitor->Remove(replace_slave->GetSocket()); - fAllUniqueMonitor->Remove(replace_slave->GetSocket()); - } - } - } - - // will be actiavted in Collect() - fUniqueMonitor->DeActivateAll(); - fAllUniqueMonitor->DeActivateAll(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return number of slaves as described in the config file. - -Int_t TProof::GetNumberOfSlaves() const -{ - return fSlaves->GetSize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return number of active slaves, i.e. slaves that are valid and in -/// the current computing group. - -Int_t TProof::GetNumberOfActiveSlaves() const -{ - return fActiveSlaves->GetSize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return number of inactive slaves, i.e. slaves that are valid but not in -/// the current computing group. - -Int_t TProof::GetNumberOfInactiveSlaves() const -{ - return fInactiveSlaves->GetSize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return number of unique slaves, i.e. active slaves that have each a -/// unique different user files system. - -Int_t TProof::GetNumberOfUniqueSlaves() const -{ - return fUniqueSlaves->GetSize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return number of bad slaves. This are slaves that we in the config -/// file, but refused to startup or that died during the PROOF session. - -Int_t TProof::GetNumberOfBadSlaves() const -{ - return fBadSlaves->GetSize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ask the for the statistics of the slaves. - -void TProof::AskStatistics() -{ - if (!IsValid()) return; - - Broadcast(kPROOF_GETSTATS, kActive); - Collect(kActive, fCollectTimeout); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get statistics about CPU time, real time and bytes read. -/// If verbose, print the resuls (always available via GetCpuTime(), GetRealTime() -/// and GetBytesRead() - -void TProof::GetStatistics(Bool_t verbose) -{ - if (fProtocol > 27) { - // This returns the correct result - AskStatistics(); - } else { - // AskStatistics is buggy: parse the output of Print() - RedirectHandle_t rh; - gSystem->RedirectOutput(fLogFileName, "a", &rh); - Print(); - gSystem->RedirectOutput(0, 0, &rh); - TMacro *mp = GetLastLog(); - if (mp) { - // Look for global directories - TIter nxl(mp->GetListOfLines()); - TObjString *os = 0; - while ((os = (TObjString *) nxl())) { - TString s(os->GetName()); - if (s.Contains("Total MB's processed:")) { - s.ReplaceAll("Total MB's processed:", ""); - if (s.IsFloat()) fBytesRead = (Long64_t) s.Atof() * (1024*1024); - } else if (s.Contains("Total real time used (s):")) { - s.ReplaceAll("Total real time used (s):", ""); - if (s.IsFloat()) fRealTime = s.Atof(); - } else if (s.Contains("Total CPU time used (s):")) { - s.ReplaceAll("Total CPU time used (s):", ""); - if (s.IsFloat()) fCpuTime = s.Atof(); - } - } - delete mp; - } - } - - if (verbose) { - Printf(" Real/CPU time (s): %.3f / %.3f; workers: %d; processed: %.2f MBs", - GetRealTime(), GetCpuTime(), GetParallel(), float(GetBytesRead())/(1024*1024)); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ask the for the number of parallel slaves. - -void TProof::AskParallel() -{ - if (!IsValid()) return; - - Broadcast(kPROOF_GETPARALLEL, kActive); - Collect(kActive, fCollectTimeout); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ask the master for the list of queries. - -TList *TProof::GetListOfQueries(Option_t *opt) -{ - if (!IsValid() || TestBit(TProof::kIsMaster)) return (TList *)0; - - Bool_t all = ((strchr(opt,'A') || strchr(opt,'a'))) ? kTRUE : kFALSE; - TMessage m(kPROOF_QUERYLIST); - m << all; - Broadcast(m, kActive); - Collect(kActive, fCollectTimeout); - - // This should have been filled by now - return fQueries; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Number of queries processed by this session - -Int_t TProof::GetNumberOfQueries() -{ - if (fQueries) - return fQueries->GetSize() - fOtherQueries; - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set max number of draw queries whose results are saved - -void TProof::SetMaxDrawQueries(Int_t max) -{ - if (max > 0) { - if (fPlayer) - fPlayer->SetMaxDrawQueries(max); - fMaxDrawQueries = max; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get max number of queries whose full results are kept in the -/// remote sandbox - -void TProof::GetMaxQueries() -{ - TMessage m(kPROOF_MAXQUERIES); - m << kFALSE; - Broadcast(m, kActive); - Collect(kActive, fCollectTimeout); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return pointer to the list of query results in the player - -TList *TProof::GetQueryResults() -{ - return (fPlayer ? fPlayer->GetListOfResults() : (TList *)0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return pointer to the full TQueryResult instance owned by the player -/// and referenced by 'ref'. If ref = 0 or "", return the last query result. - -TQueryResult *TProof::GetQueryResult(const char *ref) -{ - return (fPlayer ? fPlayer->GetQueryResult(ref) : (TQueryResult *)0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ask the master for the list of queries. -/// Options: -/// "A" show information about all the queries known to the -/// server, i.e. even those processed by other sessions -/// "L" show only information about queries locally available -/// i.e. already retrieved. If "L" is specified, "A" is -/// ignored. -/// "F" show all details available about queries -/// "H" print help menu -/// Default "" - -void TProof::ShowQueries(Option_t *opt) -{ - Bool_t help = ((strchr(opt,'H') || strchr(opt,'h'))) ? kTRUE : kFALSE; - if (help) { - - // Help - - Printf("+++"); - Printf("+++ Options: \"A\" show all queries known to server"); - Printf("+++ \"L\" show retrieved queries"); - Printf("+++ \"F\" full listing of query info"); - Printf("+++ \"H\" print this menu"); - Printf("+++"); - Printf("+++ (case insensitive)"); - Printf("+++"); - Printf("+++ Use Retrieve(<#>) to retrieve the full" - " query results from the master"); - Printf("+++ e.g. Retrieve(8)"); - - Printf("+++"); - - return; - } - - if (!IsValid()) return; - - Bool_t local = ((strchr(opt,'L') || strchr(opt,'l'))) ? kTRUE : kFALSE; - - TObject *pq = 0; - if (!local) { - GetListOfQueries(opt); - - if (!fQueries) return; - - TIter nxq(fQueries); - - // Queries processed by other sessions - if (fOtherQueries > 0) { - Printf("+++"); - Printf("+++ Queries processed during other sessions: %d", fOtherQueries); - Int_t nq = 0; - while (nq++ < fOtherQueries && (pq = nxq())) - pq->Print(opt); - } - - // Queries processed by this session - Printf("+++"); - Printf("+++ Queries processed during this session: selector: %d, draw: %d", - GetNumberOfQueries(), fDrawQueries); - while ((pq = nxq())) - pq->Print(opt); - - } else { - - // Queries processed by this session - Printf("+++"); - Printf("+++ Queries processed during this session: selector: %d, draw: %d", - GetNumberOfQueries(), fDrawQueries); - - // Queries available locally - TList *listlocal = fPlayer ? fPlayer->GetListOfResults() : (TList *)0; - if (listlocal) { - Printf("+++"); - Printf("+++ Queries available locally: %d", listlocal->GetSize()); - TIter nxlq(listlocal); - while ((pq = nxlq())) - pq->Print(opt); - } - } - Printf("+++"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See if the data is ready to be analyzed. - -Bool_t TProof::IsDataReady(Long64_t &totalbytes, Long64_t &bytesready) -{ - if (!IsValid()) return kFALSE; - - TList submasters; - TIter nextSlave(GetListOfActiveSlaves()); - while (TSlave *sl = dynamic_cast(nextSlave())) { - if (sl->GetSlaveType() == TSlave::kMaster) { - submasters.Add(sl); - } - } - - fDataReady = kTRUE; //see if any submasters set it to false - fBytesReady = 0; - fTotalBytes = 0; - //loop over submasters and see if data is ready - if (submasters.GetSize() > 0) { - Broadcast(kPROOF_DATA_READY, &submasters); - Collect(&submasters); - } - - bytesready = fBytesReady; - totalbytes = fTotalBytes; - - EmitVA("IsDataReady(Long64_t,Long64_t)", 2, totalbytes, bytesready); - - PDB(kGlobal,2) - Info("IsDataReady", "%lld / %lld (%s)", - bytesready, totalbytes, fDataReady?"READY":"NOT READY"); - - return fDataReady; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send interrupt to master or slave servers. - -void TProof::Interrupt(EUrgent type, ESlaves list) -{ - if (!IsValid()) return; - - TList *slaves = 0; - if (list == kAll) slaves = fSlaves; - if (list == kActive) slaves = fActiveSlaves; - if (list == kUnique) slaves = fUniqueSlaves; - if (list == kAllUnique) slaves = fAllUniqueSlaves; - - if (slaves->GetSize() == 0) return; - - TSlave *sl; - TIter next(slaves); - - while ((sl = (TSlave *)next())) { - if (sl->IsValid()) { - - // Ask slave to progate the interrupt request - sl->Interrupt((Int_t)type); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns number of slaves active in parallel mode. Returns 0 in case -/// there are no active slaves. Returns -1 in case of error. - -Int_t TProof::GetParallel() const -{ - if (!IsValid()) return -1; - - // iterate over active slaves and return total number of slaves - TIter nextSlave(GetListOfActiveSlaves()); - Int_t nparallel = 0; - while (TSlave* sl = dynamic_cast(nextSlave())) - if (sl->GetParallel() >= 0) - nparallel += sl->GetParallel(); - - return nparallel; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns list of TSlaveInfo's. In case of error return 0. - -TList *TProof::GetListOfSlaveInfos() -{ - if (!IsValid()) return 0; - - if (fSlaveInfo == 0) { - fSlaveInfo = new TSortedList(kSortDescending); - fSlaveInfo->SetOwner(); - } else { - fSlaveInfo->Delete(); - } - - TList masters; - TIter next(GetListOfSlaves()); - TSlave *slave; - - while ((slave = (TSlave *) next()) != 0) { - if (slave->GetSlaveType() == TSlave::kSlave) { - const char *name = IsLite() ? gSystem->HostName() : slave->GetName(); - TSlaveInfo *slaveinfo = new TSlaveInfo(slave->GetOrdinal(), - name, - slave->GetPerfIdx()); - fSlaveInfo->Add(slaveinfo); - - TIter nextactive(GetListOfActiveSlaves()); - TSlave *activeslave; - while ((activeslave = (TSlave *) nextactive())) { - if (TString(slaveinfo->GetOrdinal()) == activeslave->GetOrdinal()) { - slaveinfo->SetStatus(TSlaveInfo::kActive); - break; - } - } - - TIter nextbad(GetListOfBadSlaves()); - TSlave *badslave; - while ((badslave = (TSlave *) nextbad())) { - if (TString(slaveinfo->GetOrdinal()) == badslave->GetOrdinal()) { - slaveinfo->SetStatus(TSlaveInfo::kBad); - break; - } - } - // Get system info if supported - if (slave->IsValid()) { - if (slave->GetSocket()->Send(kPROOF_GETSLAVEINFO) == -1) - MarkBad(slave, "could not send kPROOF_GETSLAVEINFO message"); - else - masters.Add(slave); - } - - } else if (slave->GetSlaveType() == TSlave::kMaster) { - if (slave->IsValid()) { - if (slave->GetSocket()->Send(kPROOF_GETSLAVEINFO) == -1) - MarkBad(slave, "could not send kPROOF_GETSLAVEINFO message"); - else - masters.Add(slave); - } - } else { - Error("GetSlaveInfo", "TSlave is neither Master nor Slave"); - R__ASSERT(0); - } - } - if (masters.GetSize() > 0) Collect(&masters); - - return fSlaveInfo; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Activate slave server list. - -void TProof::Activate(TList *slaves) -{ - TMonitor *mon = fAllMonitor; - mon->DeActivateAll(); - - slaves = !slaves ? fActiveSlaves : slaves; - - TIter next(slaves); - TSlave *sl; - while ((sl = (TSlave*) next())) { - if (sl->IsValid()) - mon->Activate(sl->GetSocket()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Activate (on == TRUE) or deactivate (on == FALSE) all sockets -/// monitored by 'mon'. - -void TProof::SetMonitor(TMonitor *mon, Bool_t on) -{ - TMonitor *m = (mon) ? mon : fCurrentMonitor; - if (m) { - if (on) - m->ActivateAll(); - else - m->DeActivateAll(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast the group priority to all workers in the specified list. Returns -/// the number of workers the message was successfully sent to. -/// Returns -1 in case of error. - -Int_t TProof::BroadcastGroupPriority(const char *grp, Int_t priority, TList *workers) -{ - if (!IsValid()) return -1; - - if (workers->GetSize() == 0) return 0; - - int nsent = 0; - TIter next(workers); - - TSlave *wrk; - while ((wrk = (TSlave *)next())) { - if (wrk->IsValid()) { - if (wrk->SendGroupPriority(grp, priority) == -1) - MarkBad(wrk, "could not send group priority"); - else - nsent++; - } - } - - return nsent; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast the group priority to all workers in the specified list. Returns -/// the number of workers the message was successfully sent to. -/// Returns -1 in case of error. - -Int_t TProof::BroadcastGroupPriority(const char *grp, Int_t priority, ESlaves list) -{ - TList *workers = 0; - if (list == kAll) workers = fSlaves; - if (list == kActive) workers = fActiveSlaves; - if (list == kUnique) workers = fUniqueSlaves; - if (list == kAllUnique) workers = fAllUniqueSlaves; - - return BroadcastGroupPriority(grp, priority, workers); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset the merge progress notificator - -void TProof::ResetMergePrg() -{ - fMergePrg.Reset(fActiveSlaves->GetSize()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast a message to all slaves in the specified list. Returns -/// the number of slaves the message was successfully sent to. -/// Returns -1 in case of error. - -Int_t TProof::Broadcast(const TMessage &mess, TList *slaves) -{ - if (!IsValid()) return -1; - - if (!slaves || slaves->GetSize() == 0) return 0; - - int nsent = 0; - TIter next(slaves); - - TSlave *sl; - while ((sl = (TSlave *)next())) { - if (sl->IsValid()) { - if (sl->GetSocket()->Send(mess) == -1) - MarkBad(sl, "could not broadcast request"); - else - nsent++; - } - } - - return nsent; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast a message to all slaves in the specified list (either -/// all slaves or only the active slaves). Returns the number of slaves -/// the message was successfully sent to. Returns -1 in case of error. - -Int_t TProof::Broadcast(const TMessage &mess, ESlaves list) -{ - TList *slaves = 0; - if (list == kAll) slaves = fSlaves; - if (list == kActive) slaves = fActiveSlaves; - if (list == kUnique) slaves = fUniqueSlaves; - if (list == kAllUnique) slaves = fAllUniqueSlaves; - - return Broadcast(mess, slaves); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast a character string buffer to all slaves in the specified -/// list. Use kind to set the TMessage what field. Returns the number of -/// slaves the message was sent to. Returns -1 in case of error. - -Int_t TProof::Broadcast(const char *str, Int_t kind, TList *slaves) -{ - TMessage mess(kind); - if (str) mess.WriteString(str); - return Broadcast(mess, slaves); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast a character string buffer to all slaves in the specified -/// list (either all slaves or only the active slaves). Use kind to -/// set the TMessage what field. Returns the number of slaves the message -/// was sent to. Returns -1 in case of error. - -Int_t TProof::Broadcast(const char *str, Int_t kind, ESlaves list) -{ - TMessage mess(kind); - if (str) mess.WriteString(str); - return Broadcast(mess, list); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast an object to all slaves in the specified list. Use kind to -/// set the TMEssage what field. Returns the number of slaves the message -/// was sent to. Returns -1 in case of error. - -Int_t TProof::BroadcastObject(const TObject *obj, Int_t kind, TList *slaves) -{ - TMessage mess(kind); - mess.WriteObject(obj); - return Broadcast(mess, slaves); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast an object to all slaves in the specified list. Use kind to -/// set the TMEssage what field. Returns the number of slaves the message -/// was sent to. Returns -1 in case of error. - -Int_t TProof::BroadcastObject(const TObject *obj, Int_t kind, ESlaves list) -{ - TMessage mess(kind); - mess.WriteObject(obj); - return Broadcast(mess, list); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast a raw buffer of specified length to all slaves in the -/// specified list. Returns the number of slaves the buffer was sent to. -/// Returns -1 in case of error. - -Int_t TProof::BroadcastRaw(const void *buffer, Int_t length, TList *slaves) -{ - if (!IsValid()) return -1; - - if (slaves->GetSize() == 0) return 0; - - int nsent = 0; - TIter next(slaves); - - TSlave *sl; - while ((sl = (TSlave *)next())) { - if (sl->IsValid()) { - if (sl->GetSocket()->SendRaw(buffer, length) == -1) - MarkBad(sl, "could not send broadcast-raw request"); - else - nsent++; - } - } - - return nsent; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast a raw buffer of specified length to all slaves in the -/// specified list. Returns the number of slaves the buffer was sent to. -/// Returns -1 in case of error. - -Int_t TProof::BroadcastRaw(const void *buffer, Int_t length, ESlaves list) -{ - TList *slaves = 0; - if (list == kAll) slaves = fSlaves; - if (list == kActive) slaves = fActiveSlaves; - if (list == kUnique) slaves = fUniqueSlaves; - if (list == kAllUnique) slaves = fAllUniqueSlaves; - - return BroadcastRaw(buffer, length, slaves); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast file to all workers in the specified list. Returns the number of workers -/// the buffer was sent to. -/// Returns -1 in case of error. - -Int_t TProof::BroadcastFile(const char *file, Int_t opt, const char *rfile, TList *wrks) -{ - if (!IsValid()) return -1; - - if (wrks->GetSize() == 0) return 0; - - int nsent = 0; - TIter next(wrks); - - TSlave *wrk; - while ((wrk = (TSlave *)next())) { - if (wrk->IsValid()) { - if (SendFile(file, opt, rfile, wrk) < 0) - Error("BroadcastFile", - "problems sending file to worker %s (%s)", - wrk->GetOrdinal(), wrk->GetName()); - else - nsent++; - } - } - - return nsent; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Broadcast file to all workers in the specified list. Returns the number of workers -/// the buffer was sent to. -/// Returns -1 in case of error. - -Int_t TProof::BroadcastFile(const char *file, Int_t opt, const char *rfile, ESlaves list) -{ - TList *wrks = 0; - if (list == kAll) wrks = fSlaves; - if (list == kActive) wrks = fActiveSlaves; - if (list == kUnique) wrks = fUniqueSlaves; - if (list == kAllUnique) wrks = fAllUniqueSlaves; - - return BroadcastFile(file, opt, rfile, wrks); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Release the used monitor to be used, making sure to delete newly created -/// monitors. - -void TProof::ReleaseMonitor(TMonitor *mon) -{ - if (mon && (mon != fAllMonitor) && (mon != fActiveMonitor) - && (mon != fUniqueMonitor) && (mon != fAllUniqueMonitor)) { - delete mon; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Collect responses from slave sl. Returns the number of slaves that -/// responded (=1). -/// If timeout >= 0, wait at most timeout seconds (timeout = -1 by default, -/// which means wait forever). -/// If defined (>= 0) endtype is the message that stops this collection. - -Int_t TProof::Collect(const TSlave *sl, Long_t timeout, Int_t endtype, Bool_t deactonfail) -{ - Int_t rc = 0; - - TMonitor *mon = 0; - if (!sl->IsValid()) return 0; - - if (fCurrentMonitor == fAllMonitor) { - mon = new TMonitor; - } else { - mon = fAllMonitor; - mon->DeActivateAll(); - } - mon->Activate(sl->GetSocket()); - - rc = Collect(mon, timeout, endtype, deactonfail); - ReleaseMonitor(mon); - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Collect responses from the slave servers. Returns the number of slaves -/// that responded. -/// If timeout >= 0, wait at most timeout seconds (timeout = -1 by default, -/// which means wait forever). -/// If defined (>= 0) endtype is the message that stops this collection. - -Int_t TProof::Collect(TList *slaves, Long_t timeout, Int_t endtype, Bool_t deactonfail) -{ - Int_t rc = 0; - - TMonitor *mon = 0; - - if (fCurrentMonitor == fAllMonitor) { - mon = new TMonitor; - } else { - mon = fAllMonitor; - mon->DeActivateAll(); - } - TIter next(slaves); - TSlave *sl; - while ((sl = (TSlave*) next())) { - if (sl->IsValid()) - mon->Activate(sl->GetSocket()); - } - - rc = Collect(mon, timeout, endtype, deactonfail); - ReleaseMonitor(mon); - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Collect responses from the slave servers. Returns the number of slaves -/// that responded. -/// If timeout >= 0, wait at most timeout seconds (timeout = -1 by default, -/// which means wait forever). -/// If defined (>= 0) endtype is the message that stops this collection. - -Int_t TProof::Collect(ESlaves list, Long_t timeout, Int_t endtype, Bool_t deactonfail) -{ - Int_t rc = 0; - TMonitor *mon = 0; - - if (list == kAll) mon = fAllMonitor; - if (list == kActive) mon = fActiveMonitor; - if (list == kUnique) mon = fUniqueMonitor; - if (list == kAllUnique) mon = fAllUniqueMonitor; - if (fCurrentMonitor == mon) { - // Get a copy - mon = new TMonitor(*mon); - } - mon->ActivateAll(); - - rc = Collect(mon, timeout, endtype, deactonfail); - ReleaseMonitor(mon); - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Collect responses from the slave servers. Returns the number of messages -/// received. Can be 0 if there are no active slaves. -/// If timeout >= 0, wait at most timeout seconds (timeout = -1 by default, -/// which means wait forever). -/// If defined (>= 0) endtype is the message that stops this collection. -/// Collect also stops its execution from time to time to check for new -/// workers in Dynamic Startup mode. - -Int_t TProof::Collect(TMonitor *mon, Long_t timeout, Int_t endtype, Bool_t deactonfail) -{ - Int_t collectId = gRandom->Integer(9999); - - PDB(kCollect, 3) - Info("Collect", ">>>>>> Entering collect responses #%04d", collectId); - - // Reset the status flag and clear the messages in the list, if any - fStatus = 0; - fRecvMessages->Clear(); - - Long_t actto = (Long_t)(gEnv->GetValue("Proof.SocketActivityTimeout", -1) * 1000); - - if (!mon->GetActive(actto)) return 0; - - DeActivateAsyncInput(); - - // Used by external code to know what we are monitoring - TMonitor *savedMonitor = 0; - if (fCurrentMonitor) { - savedMonitor = fCurrentMonitor; - fCurrentMonitor = mon; - } else { - fCurrentMonitor = mon; - fBytesRead = 0; - fRealTime = 0.0; - fCpuTime = 0.0; - } - - // We want messages on the main window during synchronous collection, - // but we save the present status to restore it at the end - Bool_t saveRedirLog = fRedirLog; - if (!IsIdle() && !IsSync()) - fRedirLog = kFALSE; - - int cnt = 0, rc = 0; - - // Timeout counter - Long_t nto = timeout; - PDB(kCollect, 2) - Info("Collect","#%04d: active: %d", collectId, mon->GetActive()); - - // On clients, handle Ctrl-C during collection - if (fIntHandler) - fIntHandler->Add(); - - // Sockets w/o activity during the last 'sto' millisecs are deactivated - Int_t nact = 0; - Long_t sto = -1; - Int_t nsto = 60; - Int_t pollint = gEnv->GetValue("Proof.DynamicStartupPollInt", (Int_t) kPROOF_DynWrkPollInt_s); - mon->ResetInterrupt(); - while ((nact = mon->GetActive(sto)) && (nto < 0 || nto > 0)) { - - // Dump last waiting sockets, if in debug mode - PDB(kCollect, 2) { - if (nact < 4) { - TList *al = mon->GetListOfActives(); - if (al && al->GetSize() > 0) { - Info("Collect"," %d node(s) still active:", al->GetSize()); - TIter nxs(al); - TSocket *xs = 0; - while ((xs = (TSocket *)nxs())) { - TSlave *wrk = FindSlave(xs); - if (wrk) - Info("Collect"," %s (%s)", wrk->GetName(), wrk->GetOrdinal()); - else - Info("Collect"," %p: %s:%d", xs, xs->GetInetAddress().GetHostName(), - xs->GetInetAddress().GetPort()); - } - } - delete al; - } - } - - // Preemptive poll for new workers on the master only in Dynamic Mode and only - // during processing (TODO: should work on Top Master only) - if (TestBit(TProof::kIsMaster) && !IsIdle() && fDynamicStartup && !fIsPollingWorkers && - ((fLastPollWorkers_s == -1) || (time(0)-fLastPollWorkers_s >= pollint))) { - fIsPollingWorkers = kTRUE; - if (PollForNewWorkers() > 0) DeActivateAsyncInput(); - fLastPollWorkers_s = time(0); - fIsPollingWorkers = kFALSE; - PDB(kCollect, 1) - Info("Collect","#%04d: now active: %d", collectId, mon->GetActive()); - } - - // Wait for a ready socket - PDB(kCollect, 3) - Info("Collect", "Will invoke Select() #%04d", collectId); - TSocket *s = mon->Select(1000); - - if (s && s != (TSocket *)(-1)) { - // Get and analyse the info it did receive - rc = CollectInputFrom(s, endtype, deactonfail); - if (rc == 1 || (rc == 2 && !savedMonitor)) { - // Deactivate it if we are done with it - mon->DeActivate(s); - TList *al = mon->GetListOfActives(); - PDB(kCollect, 2) - Info("Collect","#%04d: deactivating %p (active: %d, %p)", collectId, - s, mon->GetActive(), - al->First()); - delete al; - } else if (rc == 2) { - // This end message was for the saved monitor - // Deactivate it if we are done with it - if (savedMonitor) { - savedMonitor->DeActivate(s); - TList *al = mon->GetListOfActives(); - PDB(kCollect, 2) - Info("Collect","save monitor: deactivating %p (active: %d, %p)", - s, savedMonitor->GetActive(), - al->First()); - delete al; - } - } - - // Update counter (if no error occured) - if (rc >= 0) - cnt++; - } else { - // If not timed-out, exit if not stopped or not aborted - // (player exits status is finished in such a case); otherwise, - // we still need to collect the partial output info - if (!s) - if (fPlayer && (fPlayer->GetExitStatus() == TVirtualProofPlayer::kFinished)) - mon->DeActivateAll(); - // Decrease the timeout counter if requested - if (s == (TSocket *)(-1) && nto > 0) - nto--; - } - - // Check if there are workers with ready output to be sent and ask the first to send it - if (IsMaster() && fWrksOutputReady && fWrksOutputReady->GetSize() > 0) { - // Maximum number of concurrent sendings - Int_t mxws = gEnv->GetValue("Proof.ControlSendOutput", 1); - if (TProof::GetParameter(fPlayer->GetInputList(), "PROOF_ControlSendOutput", mxws) != 0) - mxws = gEnv->GetValue("Proof.ControlSendOutput", 1); - TIter nxwr(fWrksOutputReady); - TSlave *wrk = 0; - while (mxws && (wrk = (TSlave *) nxwr())) { - if (!wrk->TestBit(TSlave::kOutputRequested)) { - // Ask worker for output - TMessage sendoutput(kPROOF_SENDOUTPUT); - PDB(kCollect, 2) - Info("Collect", "worker %s was asked to send its output to master", - wrk->GetOrdinal()); - if (wrk->GetSocket()->Send(sendoutput) != 1) { - wrk->SetBit(TSlave::kOutputRequested); - mxws--; - } - } else { - // Count - mxws--; - } - } - } - - // Check if we need to check the socket activity (we do it every 10 cycles ~ 10 sec) - sto = -1; - if (--nsto <= 0) { - sto = (Long_t) actto; - nsto = 60; - } - - } // end loop over active monitors - - // If timed-out, deactivate the remaining sockets - if (nto == 0) { - TList *al = mon->GetListOfActives(); - if (al && al->GetSize() > 0) { - // Notify the name of those which did timeout - Info("Collect"," %d node(s) went in timeout:", al->GetSize()); - TIter nxs(al); - TSocket *xs = 0; - while ((xs = (TSocket *)nxs())) { - TSlave *wrk = FindSlave(xs); - if (wrk) - Info("Collect"," %s", wrk->GetName()); - else - Info("Collect"," %p: %s:%d", xs, xs->GetInetAddress().GetHostName(), - xs->GetInetAddress().GetPort()); - } - } - delete al; - mon->DeActivateAll(); - } - - // Deactivate Ctrl-C special handler - if (fIntHandler) - fIntHandler->Remove(); - - // make sure group view is up to date - SendGroupView(); - - // Restore redirection setting - fRedirLog = saveRedirLog; - - // Restore the monitor - fCurrentMonitor = savedMonitor; - - ActivateAsyncInput(); - - PDB(kCollect, 3) - Info("Collect", "<<<<<< Exiting collect responses #%04d", collectId); - - return cnt; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Asks the PROOF Serv for new workers in Dynamic Startup mode and activates -/// them. Returns the number of new workers found, or <0 on errors. - -Int_t TProof::PollForNewWorkers() -{ - // Requests for worker updates - Int_t dummy = 0; - TList *reqWorkers = new TList(); - reqWorkers->SetOwner(kFALSE); - - if (!TestBit(TProof::kIsMaster)) { - Error("PollForNewWorkers", "Can't invoke: not on a master -- should not happen!"); - return -1; - } - if (!gProofServ) { - Error("PollForNewWorkers", "No ProofServ available -- should not happen!"); - return -1; - } - - gProofServ->GetWorkers(reqWorkers, dummy, kTRUE); // last 2 are dummy - - // List of new workers only (TProofNodeInfo) - TList *newWorkers = new TList(); - newWorkers->SetOwner(kTRUE); - - TIter next(reqWorkers); - TProofNodeInfo *ni; - TString fullOrd; - while (( ni = dynamic_cast(next()) )) { - - // Form the full ordinal - fullOrd.Form("%s.%s", gProofServ->GetOrdinal(), ni->GetOrdinal().Data()); - - TIter nextInner(fSlaves); - TSlave *sl; - Bool_t found = kFALSE; - while (( sl = dynamic_cast(nextInner()) )) { - if ( strcmp(sl->GetOrdinal(), fullOrd.Data()) == 0 ) { - found = kTRUE; - break; - } - } - - if (found) delete ni; - else { - newWorkers->Add(ni); - PDB(kGlobal, 1) - Info("PollForNewWorkers", "New worker found: %s:%s", - ni->GetNodeName().Data(), fullOrd.Data()); - } - } - - delete reqWorkers; // not owner - - Int_t nNewWorkers = newWorkers->GetEntries(); - - // Add the new workers - if (nNewWorkers > 0) { - PDB(kGlobal, 1) - Info("PollForNewWorkers", "Requesting to add %d new worker(s)", newWorkers->GetEntries()); - Int_t rv = AddWorkers(newWorkers); - if (rv < 0) { - Error("PollForNewWorkers", "Call to AddWorkers() failed (got %d < 0)", rv); - return -1; - } - // Don't delete newWorkers: AddWorkers() will do that - } - else { - PDB(kGlobal, 2) - Info("PollForNewWorkers", "No new worker found"); - delete newWorkers; - } - - return nNewWorkers; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove links to objects in list 'ol' from gDirectory - -void TProof::CleanGDirectory(TList *ol) -{ - if (ol) { - TIter nxo(ol); - TObject *o = 0; - while ((o = nxo())) - gDirectory->RecursiveRemove(o); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Collect and analyze available input from socket s. -/// Returns 0 on success, -1 if any failure occurs. - -Int_t TProof::CollectInputFrom(TSocket *s, Int_t endtype, Bool_t deactonfail) -{ - TMessage *mess; - - Int_t recvrc = 0; - if ((recvrc = s->Recv(mess)) < 0) { - PDB(kCollect,2) - Info("CollectInputFrom","%p: got %d from Recv()", s, recvrc); - Bool_t bad = kTRUE; - if (recvrc == -5) { - // Broken connection: try reconnection - if (fCurrentMonitor) fCurrentMonitor->Remove(s); - if (s->Reconnect() == 0) { - if (fCurrentMonitor) fCurrentMonitor->Add(s); - bad = kFALSE; - } - } - if (bad) - MarkBad(s, "problems receiving a message in TProof::CollectInputFrom(...)"); - // Ignore this wake up - return -1; - } - if (!mess) { - // we get here in case the remote server died - MarkBad(s, "undefined message in TProof::CollectInputFrom(...)"); - return -1; - } - Int_t rc = 0; - - Int_t what = mess->What(); - TSlave *sl = FindSlave(s); - rc = HandleInputMessage(sl, mess, deactonfail); - if (rc == 1 && (endtype >= 0) && (what != endtype)) - // This message was for the base monitor in recursive case - rc = 2; - - // We are done successfully - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Analyze the received message. -/// Returns 0 on success (1 if this the last message from this socket), -1 if -/// any failure occurs. - -Int_t TProof::HandleInputMessage(TSlave *sl, TMessage *mess, Bool_t deactonfail) -{ - char str[512]; - TObject *obj; - Int_t rc = 0; - - if (!mess || !sl) { - Warning("HandleInputMessage", "given an empty message or undefined worker"); - return -1; - } - Bool_t delete_mess = kTRUE; - TSocket *s = sl->GetSocket(); - if (!s) { - Warning("HandleInputMessage", "worker socket is undefined"); - return -1; - } - - // The message type - Int_t what = mess->What(); - - PDB(kCollect,3) - Info("HandleInputMessage", "got type %d from '%s'", what, sl->GetOrdinal()); - - switch (what) { - - case kMESS_OK: - // Add the message to the list - fRecvMessages->Add(mess); - delete_mess = kFALSE; - break; - - case kMESS_OBJECT: - if (fPlayer) fPlayer->HandleRecvHisto(mess); - break; - - case kPROOF_FATAL: - { TString msg; - if ((mess->BufferSize() > mess->Length())) - (*mess) >> msg; - if (msg.IsNull()) { - MarkBad(s, "received kPROOF_FATAL"); - } else { - MarkBad(s, msg); - } - } - if (fProgressDialogStarted) { - // Finalize the progress dialog - Emit("StopProcess(Bool_t)", kTRUE); - } - break; - - case kPROOF_STOP: - // Stop collection from this worker - Info("HandleInputMessage", "received kPROOF_STOP from %s: disabling any further collection this worker", - sl->GetOrdinal()); - rc = 1; - break; - - case kPROOF_GETTREEHEADER: - // Add the message to the list - fRecvMessages->Add(mess); - delete_mess = kFALSE; - rc = 1; - break; - - case kPROOF_TOUCH: - // send a request for touching the remote admin file - { - sl->Touch(); - } - break; - - case kPROOF_GETOBJECT: - // send slave object it asks for - mess->ReadString(str, sizeof(str)); - obj = gDirectory->Get(str); - if (obj) - s->SendObject(obj); - else - s->Send(kMESS_NOTOK); - break; - - case kPROOF_GETPACKET: - { - PDB(kGlobal,2) - Info("HandleInputMessage","%s: kPROOF_GETPACKET", sl->GetOrdinal()); - TDSetElement *elem = 0; - elem = fPlayer ? fPlayer->GetNextPacket(sl, mess) : 0; - - if (elem != (TDSetElement*) -1) { - TMessage answ(kPROOF_GETPACKET); - answ << elem; - s->Send(answ); - - while (fWaitingSlaves != 0 && fWaitingSlaves->GetSize()) { - TPair *p = (TPair*) fWaitingSlaves->First(); - s = (TSocket*) p->Key(); - TMessage *m = (TMessage*) p->Value(); - - elem = fPlayer ? fPlayer->GetNextPacket(sl, m) : 0; - if (elem != (TDSetElement*) -1) { - TMessage a(kPROOF_GETPACKET); - a << elem; - s->Send(a); - // remove has to happen via Links because TPair does not have - // a Compare() function and therefore RemoveFirst() and - // Remove(TObject*) do not work - fWaitingSlaves->Remove(fWaitingSlaves->FirstLink()); - delete p; - delete m; - } else { - break; - } - } - } else { - if (fWaitingSlaves == 0) fWaitingSlaves = new TList; - fWaitingSlaves->Add(new TPair(s, mess)); - delete_mess = kFALSE; - } - } - break; - - case kPROOF_LOGFILE: - { - Int_t size; - (*mess) >> size; - PDB(kGlobal,2) - Info("HandleInputMessage","%s: kPROOF_LOGFILE: size: %d", sl->GetOrdinal(), size); - RecvLogFile(s, size); - } - break; - - case kPROOF_LOGDONE: - (*mess) >> sl->fStatus >> sl->fParallel; - PDB(kCollect,2) - Info("HandleInputMessage","%s: kPROOF_LOGDONE: status %d parallel %d", - sl->GetOrdinal(), sl->fStatus, sl->fParallel); - if (sl->fStatus != 0) { - // Return last nonzero status - fStatus = sl->fStatus; - // Deactivate the worker, if required - if (deactonfail) DeactivateWorker(sl->fOrdinal); - } - // Remove from the workers-ready list - if (fWrksOutputReady && fWrksOutputReady->FindObject(sl)) { - sl->ResetBit(TSlave::kOutputRequested); - fWrksOutputReady->Remove(sl); - } - rc = 1; - break; - - case kPROOF_GETSTATS: - { - (*mess) >> sl->fBytesRead >> sl->fRealTime >> sl->fCpuTime - >> sl->fWorkDir >> sl->fProofWorkDir; - PDB(kCollect,2) - Info("HandleInputMessage", "kPROOF_GETSTATS: %s", sl->fWorkDir.Data()); - TString img; - if ((mess->BufferSize() > mess->Length())) - (*mess) >> img; - // Set image - if (img.IsNull()) { - if (sl->fImage.IsNull()) - sl->fImage.Form("%s:%s", TUrl(sl->fName).GetHostFQDN(), - sl->fProofWorkDir.Data()); - } else { - sl->fImage = img; - } - PDB(kGlobal,2) - Info("HandleInputMessage", - "kPROOF_GETSTATS:%s image: %s", sl->GetOrdinal(), sl->GetImage()); - - fBytesRead += sl->fBytesRead; - fRealTime += sl->fRealTime; - fCpuTime += sl->fCpuTime; - rc = 1; - } - break; - - case kPROOF_GETPARALLEL: - { - Bool_t async = kFALSE; - (*mess) >> sl->fParallel; - if ((mess->BufferSize() > mess->Length())) - (*mess) >> async; - rc = (async) ? 0 : 1; - } - break; - - case kPROOF_CHECKFILE: - { // New servers (>= 5.22) send the status - if ((mess->BufferSize() > mess->Length())) { - (*mess) >> fCheckFileStatus; - } else { - // Form old servers this meant success (failure was signaled with the - // dangerous kPROOF_FATAL) - fCheckFileStatus = 1; - } - rc = 1; - } - break; - - case kPROOF_SENDFILE: - { // New server: signals ending of sendfile operation - rc = 1; - } - break; - - case kPROOF_PACKAGE_LIST: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_PACKAGE_LIST: enter"); - Int_t type = 0; - (*mess) >> type; - switch (type) { - case TProof::kListEnabledPackages: - SafeDelete(fEnabledPackages); - fEnabledPackages = (TList *) mess->ReadObject(TList::Class()); - if (fEnabledPackages) { - fEnabledPackages->SetOwner(); - } else { - Error("HandleInputMessage", - "kPROOF_PACKAGE_LIST: kListEnabledPackages: TList not found in message!"); - } - break; - case TProof::kListPackages: - SafeDelete(fAvailablePackages); - fAvailablePackages = (TList *) mess->ReadObject(TList::Class()); - if (fAvailablePackages) { - fAvailablePackages->SetOwner(); - } else { - Error("HandleInputMessage", - "kPROOF_PACKAGE_LIST: kListPackages: TList not found in message!"); - } - break; - default: - Error("HandleInputMessage", "kPROOF_PACKAGE_LIST: unknown type: %d", type); - } - } - break; - - case kPROOF_SENDOUTPUT: - { - // We start measuring the merging time - fPlayer->SetMerging(); - - // Worker is ready to send output: make sure the relevant bit is reset - sl->ResetBit(TSlave::kOutputRequested); - PDB(kGlobal,2) - Info("HandleInputMessage","kPROOF_SENDOUTPUT: enter (%s)", sl->GetOrdinal()); - // Create the list if not yet done - if (!fWrksOutputReady) { - fWrksOutputReady = new TList; - fWrksOutputReady->SetOwner(kFALSE); - } - fWrksOutputReady->Add(sl); - } - break; - - case kPROOF_OUTPUTOBJECT: - { - // We start measuring the merging time - fPlayer->SetMerging(); - - PDB(kGlobal,2) - Info("HandleInputMessage","kPROOF_OUTPUTOBJECT: enter"); - Int_t type = 0; - const char *prefix = gProofServ ? gProofServ->GetPrefix() : "Lite-0"; - if (!TestBit(TProof::kIsClient) && !fMergersSet && !fFinalizationRunning) { - Info("HandleInputMessage", "finalization on %s started ...", prefix); - fFinalizationRunning = kTRUE; - } - - while ((mess->BufferSize() > mess->Length())) { - (*mess) >> type; - // If a query result header, add it to the player list - if (fPlayer) { - if (type == 0) { - // Retrieve query result instance (output list not filled) - TQueryResult *pq = - (TQueryResult *) mess->ReadObject(TQueryResult::Class()); - if (pq) { - // Add query to the result list in TProofPlayer - fPlayer->AddQueryResult(pq); - fPlayer->SetCurrentQuery(pq); - // And clear the output list, as we start merging a new set of results - if (fPlayer->GetOutputList()) - fPlayer->GetOutputList()->Clear(); - // Add the unique query tag as TNamed object to the input list - // so that it is available in TSelectors for monitoring - TString qid = TString::Format("%s:%s",pq->GetTitle(),pq->GetName()); - if (fPlayer->GetInputList()->FindObject("PROOF_QueryTag")) - fPlayer->GetInputList()->Remove(fPlayer->GetInputList()->FindObject("PROOF_QueryTag")); - fPlayer->AddInput(new TNamed("PROOF_QueryTag", qid.Data())); - } else { - Warning("HandleInputMessage","kPROOF_OUTPUTOBJECT: query result missing"); - } - } else if (type > 0) { - // Read object - TObject *o = mess->ReadObject(TObject::Class()); - // Increment counter on the client side - fMergePrg.IncreaseIdx(); - TString msg; - Bool_t changed = kFALSE; - msg.Form("%s: merging output objects ... %s", prefix, fMergePrg.Export(changed)); - if (gProofServ) { - gProofServ->SendAsynMessage(msg.Data(), kFALSE); - } else if (IsTty() || changed) { - fprintf(stderr, "%s\r", msg.Data()); - } - // Add or merge it - if ((fPlayer->AddOutputObject(o) == 1)) { - // Remove the object if it has been merged - SafeDelete(o); - } - if (type > 1) { - // Update the merger progress info - fMergePrg.DecreaseNWrks(); - if (TestBit(TProof::kIsClient) && !IsLite()) { - // In PROOFLite this has to be done once only in TProofLite::Process - TQueryResult *pq = fPlayer->GetCurrentQuery(); - if (pq) { - pq->SetOutputList(fPlayer->GetOutputList(), kFALSE); - // Add input objects (do not override remote settings, if any) - TObject *xo = 0; - TIter nxin(fPlayer->GetInputList()); - // Servers prior to 5.28/00 do not create the input list in the TQueryResult - if (!pq->GetInputList()) pq->SetInputList(new TList()); - while ((xo = nxin())) - if (!pq->GetInputList()->FindObject(xo->GetName())) - pq->AddInput(xo->Clone()); - // If the last object, notify the GUI that the result arrived - QueryResultReady(TString::Format("%s:%s", pq->GetTitle(), pq->GetName())); - } - // Processing is over - UpdateDialog(); - } - } - } - } else { - Warning("HandleInputMessage", "kPROOF_OUTPUTOBJECT: player undefined!"); - } - } - } - break; - - case kPROOF_OUTPUTLIST: - { - // We start measuring the merging time - - PDB(kGlobal,2) - Info("HandleInputMessage","%s: kPROOF_OUTPUTLIST: enter", sl->GetOrdinal()); - TList *out = 0; - if (fPlayer) { - fPlayer->SetMerging(); - if (TestBit(TProof::kIsMaster) || fProtocol < 7) { - out = (TList *) mess->ReadObject(TList::Class()); - } else { - TQueryResult *pq = - (TQueryResult *) mess->ReadObject(TQueryResult::Class()); - if (pq) { - // Add query to the result list in TProofPlayer - fPlayer->AddQueryResult(pq); - fPlayer->SetCurrentQuery(pq); - // To avoid accidental cleanups from anywhere else - // remove objects from gDirectory and clone the list - out = pq->GetOutputList(); - CleanGDirectory(out); - out = (TList *) out->Clone(); - // Notify the GUI that the result arrived - QueryResultReady(TString::Format("%s:%s", pq->GetTitle(), pq->GetName())); - } else { - PDB(kGlobal,2) - Info("HandleInputMessage", - "%s: kPROOF_OUTPUTLIST: query result missing", sl->GetOrdinal()); - } - } - if (out) { - out->SetOwner(); - fPlayer->AddOutput(out); // Incorporate the list - SafeDelete(out); - } else { - PDB(kGlobal,2) - Info("HandleInputMessage", - "%s: kPROOF_OUTPUTLIST: outputlist is empty", sl->GetOrdinal()); - } - } else { - Warning("HandleInputMessage", - "%s: kPROOF_OUTPUTLIST: player undefined!", sl->GetOrdinal()); - } - // On clients at this point processing is over - if (TestBit(TProof::kIsClient) && !IsLite()) - UpdateDialog(); - } - break; - - case kPROOF_QUERYLIST: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_QUERYLIST: enter"); - (*mess) >> fOtherQueries >> fDrawQueries; - if (fQueries) { - fQueries->Delete(); - delete fQueries; - fQueries = 0; - } - fQueries = (TList *) mess->ReadObject(TList::Class()); - } - break; - - case kPROOF_RETRIEVE: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_RETRIEVE: enter"); - TQueryResult *pq = - (TQueryResult *) mess->ReadObject(TQueryResult::Class()); - if (pq && fPlayer) { - fPlayer->AddQueryResult(pq); - // Notify the GUI that the result arrived - QueryResultReady(TString::Format("%s:%s", pq->GetTitle(), pq->GetName())); - } else { - PDB(kGlobal,2) - Info("HandleInputMessage", - "kPROOF_RETRIEVE: query result missing or player undefined"); - } - } - break; - - case kPROOF_MAXQUERIES: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_MAXQUERIES: enter"); - Int_t max = 0; - - (*mess) >> max; - Printf("Number of queries fully kept remotely: %d", max); - } - break; - - case kPROOF_SERVERSTARTED: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_SERVERSTARTED: enter"); - - UInt_t tot = 0, done = 0; - TString action; - Bool_t st = kTRUE; - - (*mess) >> action >> tot >> done >> st; - - if (TestBit(TProof::kIsClient)) { - if (tot) { - TString type = (action.Contains("submas")) ? "submasters" - : "workers"; - Int_t frac = (Int_t) (done*100.)/tot; - char msg[512] = {0}; - if (frac >= 100) { - snprintf(msg, 512, "%s: OK (%d %s) \n", - action.Data(),tot, type.Data()); - } else { - snprintf(msg, 512, "%s: %d out of %d (%d %%)\r", - action.Data(), done, tot, frac); - } - if (fSync) - fprintf(stderr,"%s", msg); - else - NotifyLogMsg(msg, 0); - } - // Notify GUIs - StartupMessage(action.Data(), st, (Int_t)done, (Int_t)tot); - } else { - - // Just send the message one level up - TMessage m(kPROOF_SERVERSTARTED); - m << action << tot << done << st; - gProofServ->GetSocket()->Send(m); - } - } - break; - - case kPROOF_DATASET_STATUS: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_DATASET_STATUS: enter"); - - UInt_t tot = 0, done = 0; - TString action; - Bool_t st = kTRUE; - - (*mess) >> action >> tot >> done >> st; - - if (TestBit(TProof::kIsClient)) { - if (tot) { - TString type = "files"; - Int_t frac = (Int_t) (done*100.)/tot; - char msg[512] = {0}; - if (frac >= 100) { - snprintf(msg, 512, "%s: OK (%d %s) \n", - action.Data(),tot, type.Data()); - } else { - snprintf(msg, 512, "%s: %d out of %d (%d %%)\r", - action.Data(), done, tot, frac); - } - if (fSync) - fprintf(stderr,"%s", msg); - else - NotifyLogMsg(msg, 0); - } - // Notify GUIs - DataSetStatus(action.Data(), st, (Int_t)done, (Int_t)tot); - } else { - - // Just send the message one level up - TMessage m(kPROOF_DATASET_STATUS); - m << action << tot << done << st; - gProofServ->GetSocket()->Send(m); - } - } - break; - - case kPROOF_STARTPROCESS: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_STARTPROCESS: enter"); - - // For Proof-Lite this variable is the number of workers and is set - // by the player - if (!IsLite()) { - fNotIdle = 1; - fIsWaiting = kFALSE; - } - - // Redirect the output, if needed - fRedirLog = (fSync) ? fRedirLog : kTRUE; - - // The signal is used on masters by XrdProofdProtocol to catch - // the start of processing; on clients it allows to update the - // progress dialog - if (!TestBit(TProof::kIsMaster)) { - - // This is the end of preparation - fQuerySTW.Stop(); - fPrepTime = fQuerySTW.RealTime(); - PDB(kGlobal,2) Info("HandleInputMessage","Preparation time: %f s", fPrepTime); - - TString selec; - Int_t dsz = -1; - Long64_t first = -1, nent = -1; - (*mess) >> selec >> dsz >> first >> nent; - // Start or reset the progress dialog - if (!gROOT->IsBatch()) { - if (fProgressDialog && - !TestBit(kUsingSessionGui) && TestBit(kUseProgressDialog)) { - if (!fProgressDialogStarted) { - fProgressDialog->ExecPlugin(5, this, - selec.Data(), dsz, first, nent); - fProgressDialogStarted = kTRUE; - } else { - ResetProgressDialog(selec, dsz, first, nent); - } - } - ResetBit(kUsingSessionGui); - } - } - } - break; - - case kPROOF_ENDINIT: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_ENDINIT: enter"); - - if (TestBit(TProof::kIsMaster)) { - if (fPlayer) - fPlayer->SetInitTime(); - } - } - break; - - case kPROOF_SETIDLE: - { - PDB(kGlobal,2) - Info("HandleInputMessage","kPROOF_SETIDLE from '%s': enter (%d)", sl->GetOrdinal(), fNotIdle); - - // The session is idle - if (IsLite()) { - if (fNotIdle > 0) { - fNotIdle--; - PDB(kGlobal,2) - Info("HandleInputMessage", "%s: got kPROOF_SETIDLE", sl->GetOrdinal()); - } else { - Warning("HandleInputMessage", - "%s: got kPROOF_SETIDLE but no running workers ! protocol error?", - sl->GetOrdinal()); - } - } else { - fNotIdle = 0; - // Check if the query has been enqueued - if ((mess->BufferSize() > mess->Length())) - (*mess) >> fIsWaiting; - } - } - break; - - case kPROOF_QUERYSUBMITTED: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_QUERYSUBMITTED: enter"); - - // We have received the sequential number - (*mess) >> fSeqNum; - Bool_t sync = fSync; - if ((mess->BufferSize() > mess->Length())) - (*mess) >> sync; - if (sync != fSync && fSync) { - // The server required to switch to asynchronous mode - Activate(); - fSync = kFALSE; - } - DisableGoAsyn(); - // Check if the query has been enqueued - fIsWaiting = kTRUE; - // For Proof-Lite this variable is the number of workers and is set by the player - if (!IsLite()) - fNotIdle = 1; - - rc = 1; - } - break; - - case kPROOF_SESSIONTAG: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_SESSIONTAG: enter"); - - // We have received the unique tag and save it as name of this object - TString stag; - (*mess) >> stag; - SetName(stag); - // In the TSlave object - sl->SetSessionTag(stag); - // Server may have also sent the group - if ((mess->BufferSize() > mess->Length())) - (*mess) >> fGroup; - // Server may have also sent the user - if ((mess->BufferSize() > mess->Length())) { - TString usr; - (*mess) >> usr; - if (!usr.IsNull()) fUrl.SetUser(usr.Data()); - } - } - break; - - case kPROOF_FEEDBACK: - { - PDB(kGlobal,2) - Info("HandleInputMessage","kPROOF_FEEDBACK: enter"); - TList *out = (TList *) mess->ReadObject(TList::Class()); - out->SetOwner(); - if (fPlayer) - fPlayer->StoreFeedback(sl, out); // Adopts the list - else - // Not yet ready: stop collect asap - rc = 1; - } - break; - - case kPROOF_AUTOBIN: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_AUTOBIN: enter"); - - TString name; - Double_t xmin, xmax, ymin, ymax, zmin, zmax; - - (*mess) >> name >> xmin >> xmax >> ymin >> ymax >> zmin >> zmax; - - if (fPlayer) fPlayer->UpdateAutoBin(name,xmin,xmax,ymin,ymax,zmin,zmax); - - TMessage answ(kPROOF_AUTOBIN); - - answ << name << xmin << xmax << ymin << ymax << zmin << zmax; - - s->Send(answ); - } - break; - - case kPROOF_PROGRESS: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_PROGRESS: enter"); - - if (GetRemoteProtocol() > 25) { - // New format - TProofProgressInfo *pi = 0; - (*mess) >> pi; - fPlayer->Progress(sl,pi); - } else if (GetRemoteProtocol() > 11) { - Long64_t total, processed, bytesread; - Float_t initTime, procTime, evtrti, mbrti; - (*mess) >> total >> processed >> bytesread - >> initTime >> procTime - >> evtrti >> mbrti; - if (fPlayer) - fPlayer->Progress(sl, total, processed, bytesread, - initTime, procTime, evtrti, mbrti); - - } else { - // Old format - Long64_t total, processed; - (*mess) >> total >> processed; - if (fPlayer) - fPlayer->Progress(sl, total, processed); - } - } - break; - - case kPROOF_STOPPROCESS: - { - // This message is sent from a worker that finished processing. - // We determine whether it was asked to finish by the - // packetizer or stopped during processing a packet - // (by TProof::RemoveWorkers() or by an external signal). - // In the later case call packetizer->MarkBad. - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_STOPPROCESS: enter"); - - Long64_t events = 0; - Bool_t abort = kFALSE; - TProofProgressStatus *status = 0; - - if ((mess->BufferSize() > mess->Length()) && (fProtocol > 18)) { - (*mess) >> status >> abort; - } else if ((mess->BufferSize() > mess->Length()) && (fProtocol > 8)) { - (*mess) >> events >> abort; - } else { - (*mess) >> events; - } - if (fPlayer) { - if (fProtocol > 18) { - TList *listOfMissingFiles = 0; - if (!(listOfMissingFiles = (TList *)GetOutput("MissingFiles"))) { - listOfMissingFiles = new TList(); - listOfMissingFiles->SetName("MissingFiles"); - if (fPlayer) - fPlayer->AddOutputObject(listOfMissingFiles); - } - if (fPlayer->GetPacketizer()) { - Int_t ret = - fPlayer->GetPacketizer()->AddProcessed(sl, status, 0, &listOfMissingFiles); - if (ret > 0) - fPlayer->GetPacketizer()->MarkBad(sl, status, &listOfMissingFiles); - // This object is now owned by the packetizer - status = 0; - } - if (status) fPlayer->AddEventsProcessed(status->GetEntries()); - } else { - fPlayer->AddEventsProcessed(events); - } - } - SafeDelete(status); - if (!TestBit(TProof::kIsMaster)) - Emit("StopProcess(Bool_t)", abort); - break; - } - - case kPROOF_SUBMERGER: - { - PDB(kGlobal,2) Info("HandleInputMessage", "kPROOF_SUBMERGER: enter"); - HandleSubmerger(mess, sl); - } - break; - - case kPROOF_GETSLAVEINFO: - { - PDB(kGlobal,2) Info("HandleInputMessage", "kPROOF_GETSLAVEINFO: enter"); - - Bool_t active = (GetListOfActiveSlaves()->FindObject(sl) != 0); - Bool_t bad = (GetListOfBadSlaves()->FindObject(sl) != 0); - TList* tmpinfo = 0; - (*mess) >> tmpinfo; - if (tmpinfo == 0) { - Error("HandleInputMessage", "kPROOF_GETSLAVEINFO: no list received!"); - } else { - tmpinfo->SetOwner(kFALSE); - Int_t nentries = tmpinfo->GetSize(); - for (Int_t i=0; i(tmpinfo->At(i)); - if (slinfo) { - // If PROOF-Lite - if (IsLite()) slinfo->fHostName = gSystem->HostName(); - // Check if we have already a instance for this worker - TIter nxw(fSlaveInfo); - TSlaveInfo *ourwi = 0; - while ((ourwi = (TSlaveInfo *)nxw())) { - if (!strcmp(ourwi->GetOrdinal(), slinfo->GetOrdinal())) { - ourwi->SetSysInfo(slinfo->GetSysInfo()); - ourwi->fHostName = slinfo->GetName(); - if (slinfo->GetDataDir() && (strlen(slinfo->GetDataDir()) > 0)) - ourwi->fDataDir = slinfo->GetDataDir(); - break; - } - } - if (!ourwi) { - fSlaveInfo->Add(slinfo); - } else { - slinfo = ourwi; - } - if (slinfo->fStatus != TSlaveInfo::kBad) { - if (!active) slinfo->SetStatus(TSlaveInfo::kNotActive); - if (bad) slinfo->SetStatus(TSlaveInfo::kBad); - } - if (sl->GetMsd() && (strlen(sl->GetMsd()) > 0)) - slinfo->fMsd = sl->GetMsd(); - } - } - delete tmpinfo; - rc = 1; - } - } - break; - - case kPROOF_VALIDATE_DSET: - { - PDB(kGlobal,2) - Info("HandleInputMessage", "kPROOF_VALIDATE_DSET: enter"); - TDSet* dset = 0; - (*mess) >> dset; - if (!fDSet) - Error("HandleInputMessage", "kPROOF_VALIDATE_DSET: fDSet not set"); - else - fDSet->Validate(dset); - delete dset; - } - break; - - case kPROOF_DATA_READY: - { - PDB(kGlobal,2) Info("HandleInputMessage", "kPROOF_DATA_READY: enter"); - Bool_t dataready = kFALSE; - Long64_t totalbytes, bytesready; - (*mess) >> dataready >> totalbytes >> bytesready; - fTotalBytes += totalbytes; - fBytesReady += bytesready; - if (dataready == kFALSE) fDataReady = dataready; - } - break; - - case kPROOF_PING: - // do nothing (ping is already acknowledged) - break; - - case kPROOF_MESSAGE: - { - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_MESSAGE: enter"); - - // We have received the unique tag and save it as name of this object - TString msg; - (*mess) >> msg; - Bool_t lfeed = kTRUE; - if ((mess->BufferSize() > mess->Length())) - (*mess) >> lfeed; - - if (TestBit(TProof::kIsClient)) { - - if (fSync) { - // Notify locally - fprintf(stderr,"%s%c", msg.Data(), (lfeed ? '\n' : '\r')); - } else { - // Notify locally taking care of redirection, windows logs, ... - NotifyLogMsg(msg, (lfeed ? "\n" : "\r")); - } - } else { - - // The message is logged for debugging purposes. - fprintf(stderr,"%s%c", msg.Data(), (lfeed ? '\n' : '\r')); - if (gProofServ) { - // We hide it during normal operations - gProofServ->FlushLogFile(); - - // And send the message one level up - gProofServ->SendAsynMessage(msg, lfeed); - } - } - } - break; - - case kPROOF_VERSARCHCOMP: - { - TString vac; - (*mess) >> vac; - PDB(kGlobal,2) Info("HandleInputMessage","kPROOF_VERSARCHCOMP: %s", vac.Data()); - Int_t from = 0; - TString vers, archcomp; - if (vac.Tokenize(vers, from, "|")) - vac.Tokenize(archcomp, from, "|"); - sl->SetArchCompiler(archcomp); - vers.ReplaceAll(":","|"); - sl->SetROOTVersion(vers); - } - break; - - default: - { - Error("HandleInputMessage", "unknown command received from '%s' (what = %d)", - sl->GetOrdinal(), what); - } - break; - } - - // Cleanup - if (delete_mess) - delete mess; - - // We are done successfully - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process a message of type kPROOF_SUBMERGER - -void TProof::HandleSubmerger(TMessage *mess, TSlave *sl) -{ - // Message sub-type - Int_t type = 0; - (*mess) >> type; - TSocket *s = sl->GetSocket(); - - switch (type) { - case kOutputSent: - { - if (IsEndMaster()) { - Int_t merger_id = -1; - (*mess) >> merger_id; - - PDB(kSubmerger, 2) - Info("HandleSubmerger", "kOutputSent: Worker %s:%d:%s had sent its output to merger #%d", - sl->GetName(), sl->GetPort(), sl->GetOrdinal(), merger_id); - - if (!fMergers || fMergers->GetSize() <= merger_id) { - Error("HandleSubmerger", "kOutputSize: #%d not in list ", merger_id); - break; - } - TMergerInfo * mi = (TMergerInfo *) fMergers->At(merger_id); - mi->SetMergedWorker(); - if (mi->AreAllWorkersMerged()) { - mi->Deactivate(); - if (GetActiveMergersCount() == 0) { - fMergers->Clear(); - delete fMergers; - fMergersSet = kFALSE; - fMergersCount = -1; - fLastAssignedMerger = 0; - PDB(kSubmerger, 2) Info("HandleSubmerger", "all mergers removed ... "); - } - } - } else { - PDB(kSubmerger, 2) Error("HandleSubmerger","kOutputSent: received not on endmaster!"); - } - } - break; - - case kMergerDown: - { - Int_t merger_id = -1; - (*mess) >> merger_id; - - PDB(kSubmerger, 2) Info("HandleSubmerger", "kMergerDown: #%d ", merger_id); - - if (!fMergers || fMergers->GetSize() <= merger_id) { - Error("HandleSubmerger", "kMergerDown: #%d not in list ", merger_id); - break; - } - - TMergerInfo * mi = (TMergerInfo *) fMergers->At(merger_id); - if (!mi->IsActive()) { - break; - } else { - mi->Deactivate(); - } - - // Stop the invalid merger in the case it is still listening - TMessage stop(kPROOF_SUBMERGER); - stop << Int_t(kStopMerging); - stop << 0; - s->Send(stop); - - // Ask for results from merger (only original results from this node as worker are returned) - AskForOutput(mi->GetMerger()); - - // Ask for results from all workers assigned to this merger - TIter nxo(mi->GetWorkers()); - TObject * o = 0; - while ((o = nxo())) { - AskForOutput((TSlave *)o); - } - PDB(kSubmerger, 2) Info("HandleSubmerger", "kMergerDown:%d: exit", merger_id); - } - break; - - case kOutputSize: - { - if (IsEndMaster()) { - PDB(kSubmerger, 2) - Info("HandleSubmerger", "worker %s reported as finished ", sl->GetOrdinal()); - - const char *prefix = gProofServ ? gProofServ->GetPrefix() : "Lite-0"; - if (!fFinalizationRunning) { - Info("HandleSubmerger", "finalization on %s started ...", prefix); - fFinalizationRunning = kTRUE; - } - - Int_t output_size = 0; - Int_t merging_port = 0; - (*mess) >> output_size >> merging_port; - - PDB(kSubmerger, 2) Info("HandleSubmerger", - "kOutputSize: Worker %s:%d:%s reports %d output objects (+ available port %d)", - sl->GetName(), sl->GetPort(), sl->GetOrdinal(), output_size, merging_port); - TString msg; - if (!fMergersSet) { - - Int_t activeWorkers = fCurrentMonitor ? fCurrentMonitor->GetActive() : GetNumberOfActiveSlaves(); - - // First pass - setting number of mergers according to user or dynamically - fMergersCount = -1; // No mergers used if not set by user - TParameter *mc = dynamic_cast *>(GetParameter("PROOF_UseMergers")); - if (mc) fMergersCount = mc->GetVal(); // Value set by user - TParameter *mh = dynamic_cast *>(GetParameter("PROOF_MergersByHost")); - if (mh) fMergersByHost = (mh->GetVal() != 0) ? kTRUE : kFALSE; // Assign submergers by hostname - - // Mergers count specified by user but not valid - if (fMergersCount < 0 || (fMergersCount > (activeWorkers/2) )) { - msg.Form("%s: Invalid request: cannot start %d mergers for %d workers", - prefix, fMergersCount, activeWorkers); - if (gProofServ) - gProofServ->SendAsynMessage(msg); - else - Printf("%s",msg.Data()); - fMergersCount = 0; - } - // Mergers count will be set dynamically - if ((fMergersCount == 0) && (!fMergersByHost)) { - if (activeWorkers > 1) { - fMergersCount = TMath::Nint(TMath::Sqrt(activeWorkers)); - if (activeWorkers / fMergersCount < 2) - fMergersCount = (Int_t) TMath::Sqrt(activeWorkers); - } - if (fMergersCount > 1) - msg.Form("%s: Number of mergers set dynamically to %d (for %d workers)", - prefix, fMergersCount, activeWorkers); - else { - msg.Form("%s: No mergers will be used for %d workers", - prefix, activeWorkers); - fMergersCount = -1; - } - if (gProofServ) - gProofServ->SendAsynMessage(msg); - else - Printf("%s",msg.Data()); - } else if (fMergersByHost) { - // We force mergers at host level to minimize network traffic - if (activeWorkers > 1) { - fMergersCount = 0; - THashList hosts; - TIter nxwk(fSlaves); - TObject *wrk = 0; - while ((wrk = nxwk())) { - if (!hosts.FindObject(wrk->GetName())) { - hosts.Add(new TObjString(wrk->GetName())); - fMergersCount++; - } - } - } - if (fMergersCount > 1) - msg.Form("%s: Number of mergers set to %d (for %d workers), one for each slave host", - prefix, fMergersCount, activeWorkers); - else { - msg.Form("%s: No mergers will be used for %d workers", - prefix, activeWorkers); - fMergersCount = -1; - } - if (gProofServ) - gProofServ->SendAsynMessage(msg); - else - Printf("%s",msg.Data()); - } else { - msg.Form("%s: Number of mergers set by user to %d (for %d workers)", - prefix, fMergersCount, activeWorkers); - if (gProofServ) - gProofServ->SendAsynMessage(msg); - else - Printf("%s",msg.Data()); - } - - // We started merging; we call it here because fMergersCount is still the original number - // and can be saved internally - fPlayer->SetMerging(kTRUE); - - // Update merger counters (new workers are not yet active) - fMergePrg.SetNWrks(fMergersCount); - - if (fMergersCount > 0) { - - fMergers = new TList(); - fLastAssignedMerger = 0; - // Total number of workers, which will not act as mergers ('pure workers') - fWorkersToMerge = (activeWorkers - fMergersCount); - // Establish the first merger - if (!CreateMerger(sl, merging_port)) { - // Cannot establish first merger - AskForOutput(sl); - fWorkersToMerge--; - fMergersCount--; - } - if (IsLite()) fMergePrg.SetNWrks(fMergersCount); - } else { - AskForOutput(sl); - } - fMergersSet = kTRUE; - } else { - // Multiple pass - if (fMergersCount == -1) { - // No mergers. Workers send their outputs directly to master - AskForOutput(sl); - } else { - if ((fRedirectNext > 0 ) && (!fMergersByHost)) { - RedirectWorker(s, sl, output_size); - fRedirectNext--; - } else { - Bool_t newMerger = kTRUE; - if (fMergersByHost) { - TIter nxmg(fMergers); - TMergerInfo *mgi = 0; - while ((mgi = (TMergerInfo *) nxmg())) { - if (!strcmp(sl->GetName(), mgi->GetMerger()->GetName())) { - newMerger = kFALSE; - break; - } - } - } - if ((fMergersCount > fMergers->GetSize()) && newMerger) { - // Still not enough mergers established - if (!CreateMerger(sl, merging_port)) { - // Cannot establish a merger - AskForOutput(sl); - fWorkersToMerge--; - fMergersCount--; - } - } else - RedirectWorker(s, sl, output_size); - } - } - } - } else { - Error("HandleSubMerger","kOutputSize received not on endmaster!"); - } - } - break; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Redirect output of worker sl to some merger - -void TProof::RedirectWorker(TSocket *s, TSlave * sl, Int_t output_size) -{ - Int_t merger_id = -1; - - if (fMergersByHost) { - for (Int_t i = 0; i < fMergers->GetSize(); i++) { - TMergerInfo *mgi = (TMergerInfo *)fMergers->At(i); - if (!strcmp(sl->GetName(), mgi->GetMerger()->GetName())) { - merger_id = i; - break; - } - } - } else { - merger_id = FindNextFreeMerger(); - } - - if (merger_id == -1) { - // No free merger (probably it had crashed before) - AskForOutput(sl); - } else { - TMessage sendoutput(kPROOF_SUBMERGER); - sendoutput << Int_t(kSendOutput); - PDB(kSubmerger, 2) - Info("RedirectWorker", "redirecting worker %s to merger %d", sl->GetOrdinal(), merger_id); - - PDB(kSubmerger, 2) Info("RedirectWorker", "redirecting output to merger #%d", merger_id); - if (!fMergers || fMergers->GetSize() <= merger_id) { - Error("RedirectWorker", "#%d not in list ", merger_id); - return; - } - TMergerInfo * mi = (TMergerInfo *) fMergers->At(merger_id); - - TString hname = (IsLite()) ? "localhost" : mi->GetMerger()->GetName(); - sendoutput << merger_id; - sendoutput << hname; - sendoutput << mi->GetPort(); - s->Send(sendoutput); - mi->AddMergedObjects(output_size); - mi->AddWorker(sl); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return a merger, which is both active and still accepts some workers to be -/// assigned to it. It works on the 'round-robin' basis. - -Int_t TProof::FindNextFreeMerger() -{ - while (fLastAssignedMerger < fMergers->GetSize() && - (!((TMergerInfo*)fMergers->At(fLastAssignedMerger))->IsActive() || - ((TMergerInfo*)fMergers->At(fLastAssignedMerger))->AreAllWorkersAssigned())) { - fLastAssignedMerger++; - } - - if (fLastAssignedMerger == fMergers->GetSize()) { - fLastAssignedMerger = 0; - } else { - return fLastAssignedMerger++; - } - - while (fLastAssignedMerger < fMergers->GetSize() && - (!((TMergerInfo*)fMergers->At(fLastAssignedMerger))->IsActive() || - ((TMergerInfo*)fMergers->At(fLastAssignedMerger))->AreAllWorkersAssigned())) { - fLastAssignedMerger++; - } - - if (fLastAssignedMerger == fMergers->GetSize()) { - return -1; - } else { - return fLastAssignedMerger++; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Master asks for output from worker sl - -void TProof::AskForOutput(TSlave *sl) -{ - TMessage sendoutput(kPROOF_SUBMERGER); - sendoutput << Int_t(kSendOutput); - - PDB(kSubmerger, 2) Info("AskForOutput", - "worker %s was asked to send its output to master", - sl->GetOrdinal()); - - sendoutput << -1; - sendoutput << TString("master"); - sendoutput << -1; - sl->GetSocket()->Send(sendoutput); - if (IsLite()) fMergePrg.IncreaseNWrks(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Final update of the progress dialog - -void TProof::UpdateDialog() -{ - if (!fPlayer) return; - - // Handle abort ... - if (fPlayer->GetExitStatus() == TVirtualProofPlayer::kAborted) { - if (fSync) - Info("UpdateDialog", - "processing was aborted - %lld events processed", - fPlayer->GetEventsProcessed()); - - if (GetRemoteProtocol() > 11) { - // New format - Progress(-1, fPlayer->GetEventsProcessed(), -1, -1., -1., -1., -1.); - } else { - Progress(-1, fPlayer->GetEventsProcessed()); - } - Emit("StopProcess(Bool_t)", kTRUE); - } - - // Handle stop ... - if (fPlayer->GetExitStatus() == TVirtualProofPlayer::kStopped) { - if (fSync) - Info("UpdateDialog", - "processing was stopped - %lld events processed", - fPlayer->GetEventsProcessed()); - - if (GetRemoteProtocol() > 25) { - // New format - Progress(-1, fPlayer->GetEventsProcessed(), -1, -1., -1., -1., -1., -1, -1, -1.); - } else if (GetRemoteProtocol() > 11) { - Progress(-1, fPlayer->GetEventsProcessed(), -1, -1., -1., -1., -1.); - } else { - Progress(-1, fPlayer->GetEventsProcessed()); - } - Emit("StopProcess(Bool_t)", kFALSE); - } - - // Final update of the dialog box - if (GetRemoteProtocol() > 25) { - // New format - EmitVA("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)", - 10, (Long64_t)(-1), (Long64_t)(-1), (Long64_t)(-1),(Float_t)(-1.),(Float_t)(-1.), - (Float_t)(-1.),(Float_t)(-1.),(Int_t)(-1),(Int_t)(-1),(Float_t)(-1.)); - } else if (GetRemoteProtocol() > 11) { - // New format - EmitVA("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - 7, (Long64_t)(-1), (Long64_t)(-1), (Long64_t)(-1), - (Float_t)(-1.),(Float_t)(-1.),(Float_t)(-1.),(Float_t)(-1.)); - } else { - EmitVA("Progress(Long64_t,Long64_t)", 2, (Long64_t)(-1), (Long64_t)(-1)); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Activate the a-sync input handler. - -void TProof::ActivateAsyncInput() -{ - TIter next(fSlaves); - TSlave *sl; - - while ((sl = (TSlave*) next())) - if (sl->GetInputHandler()) - sl->GetInputHandler()->Add(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// De-activate a-sync input handler. - -void TProof::DeActivateAsyncInput() -{ - TIter next(fSlaves); - TSlave *sl; - - while ((sl = (TSlave*) next())) - if (sl->GetInputHandler()) - sl->GetInputHandler()->Remove(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the active mergers count - -Int_t TProof::GetActiveMergersCount() -{ - if (!fMergers) return 0; - - Int_t active_mergers = 0; - - TIter mergers(fMergers); - TMergerInfo *mi = 0; - while ((mi = (TMergerInfo *)mergers())) { - if (mi->IsActive()) active_mergers++; - } - - return active_mergers; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a new merger - -Bool_t TProof::CreateMerger(TSlave *sl, Int_t port) -{ - PDB(kSubmerger, 2) - Info("CreateMerger", "worker %s will be merger ", sl->GetOrdinal()); - - PDB(kSubmerger, 2) Info("CreateMerger","Begin"); - - if (port <= 0) { - PDB(kSubmerger,2) - Info("CreateMerger", "cannot create merger on port %d - exit", port); - return kFALSE; - } - - Int_t workers = -1; - if (!fMergersByHost) { - Int_t mergersToCreate = fMergersCount - fMergers->GetSize(); - // Number of pure workers, which are not simply divisible by mergers - Int_t rest = fWorkersToMerge % mergersToCreate; - // We add one more worker for each of the first 'rest' mergers being established - if (rest > 0 && fMergers->GetSize() < rest) { - rest = 1; - } else { - rest = 0; - } - workers = (fWorkersToMerge / mergersToCreate) + rest; - } else { - Int_t workersOnHost = 0; - for (Int_t i = 0; i < fActiveSlaves->GetSize(); i++) { - if(!strcmp(sl->GetName(), fActiveSlaves->At(i)->GetName())) workersOnHost++; - } - workers = workersOnHost - 1; - } - - TString msg; - msg.Form("worker %s on host %s will be merger for %d additional workers", sl->GetOrdinal(), sl->GetName(), workers); - - if (gProofServ) { - gProofServ->SendAsynMessage(msg); - } else { - Printf("%s",msg.Data()); - } - TMergerInfo * merger = new TMergerInfo(sl, port, workers); - - TMessage bemerger(kPROOF_SUBMERGER); - bemerger << Int_t(kBeMerger); - bemerger << fMergers->GetSize(); - bemerger << workers; - sl->GetSocket()->Send(bemerger); - - PDB(kSubmerger,2) Info("CreateMerger", - "merger #%d (port: %d) for %d workers started", - fMergers->GetSize(), port, workers); - - fMergers->Add(merger); - fWorkersToMerge = fWorkersToMerge - workers; - - fRedirectNext = workers / 2; - - PDB(kSubmerger, 2) Info("CreateMerger", "exit"); - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add a bad slave server to the bad slave list and remove it from -/// the active list and from the two monitor objects. Assume that the work -/// done by this worker was lost and ask packerizer to reassign it. - -void TProof::MarkBad(TSlave *wrk, const char *reason) -{ - std::lock_guard lock(fCloseMutex); - - // We may have been invalidated in the meanwhile: nothing to do in such a case - if (!IsValid()) return; - - if (!wrk) { - Error("MarkBad", "worker instance undefined: protocol error? "); - return; - } - - // Local URL - static TString thisurl; - if (thisurl.IsNull()) { - if (IsMaster()) { - Int_t port = gEnv->GetValue("ProofServ.XpdPort",-1); - thisurl = TUrl(gSystem->HostName()).GetHostFQDN(); - if (port > 0) thisurl += TString::Format(":%d", port); - } else { - thisurl.Form("%s@%s:%d", fUrl.GetUser(), fUrl.GetHost(), fUrl.GetPort()); - } - } - - if (!reason || (strcmp(reason, kPROOF_TerminateWorker) && strcmp(reason, kPROOF_WorkerIdleTO))) { - // Message for notification - const char *mastertype = (gProofServ && gProofServ->IsTopMaster()) ? "top master" : "master"; - TString src = IsMaster() ? Form("%s at %s", mastertype, thisurl.Data()) : "local session"; - TString msg; - msg.Form("\n +++ Message from %s : marking %s:%d (%s) as bad\n +++ Reason: %s", - src.Data(), wrk->GetName(), wrk->GetPort(), wrk->GetOrdinal(), - (reason && strlen(reason)) ? reason : "unknown"); - Info("MarkBad", "%s", msg.Data()); - // Notify one level up, if the case - // Add some hint for diagnostics - if (gProofServ) { - msg += TString::Format("\n\n +++ Most likely your code crashed on worker %s at %s:%d.\n", - wrk->GetOrdinal(), wrk->GetName(), wrk->GetPort()); - } else { - msg += TString::Format("\n\n +++ Most likely your code crashed\n"); - } - msg += TString::Format(" +++ Please check the session logs for error messages either using\n"); - msg += TString::Format(" +++ the 'Show logs' button or executing\n"); - msg += TString::Format(" +++\n"); - if (gProofServ) { - msg += TString::Format(" +++ root [] TProof::Mgr(\"%s\")->GetSessionLogs()->" - "Display(\"%s\",0)\n\n", thisurl.Data(), wrk->GetOrdinal()); - gProofServ->SendAsynMessage(msg, kTRUE); - } else { - msg += TString::Format(" +++ root [] TProof::Mgr(\"%s\")->GetSessionLogs()->" - "Display(\"*\")\n\n", thisurl.Data()); - Printf("%s", msg.Data()); - } - } else if (reason) { - if (gDebug > 0 && strcmp(reason, kPROOF_WorkerIdleTO)) { - Info("MarkBad", "worker %s at %s:%d asked to terminate", - wrk->GetOrdinal(), wrk->GetName(), wrk->GetPort()); - } - } - - if (IsMaster() && reason) { - if (strcmp(reason, kPROOF_TerminateWorker)) { - // if the reason was not a planned termination - TList *listOfMissingFiles = 0; - if (!(listOfMissingFiles = (TList *)GetOutput("MissingFiles"))) { - listOfMissingFiles = new TList(); - listOfMissingFiles->SetName("MissingFiles"); - if (fPlayer) - fPlayer->AddOutputObject(listOfMissingFiles); - } - // If a query is being processed, assume that the work done by - // the worker was lost and needs to be reassigned. - TVirtualPacketizer *packetizer = fPlayer ? fPlayer->GetPacketizer() : 0; - if (packetizer) { - // the worker was lost so do resubmit the packets - packetizer->MarkBad(wrk, 0, &listOfMissingFiles); - } - } else { - // Tell the coordinator that we are gone - if (gProofServ) { - TString ord(wrk->GetOrdinal()); - Int_t id = ord.Last('.'); - if (id != kNPOS) ord.Remove(0, id+1); - gProofServ->ReleaseWorker(ord.Data()); - } - } - } else if (TestBit(TProof::kIsClient) && reason && !strcmp(reason, kPROOF_WorkerIdleTO)) { - // We are invalid after this - fValid = kFALSE; - } - - fActiveSlaves->Remove(wrk); - FindUniqueSlaves(); - - fAllMonitor->Remove(wrk->GetSocket()); - fActiveMonitor->Remove(wrk->GetSocket()); - - fSendGroupView = kTRUE; - - if (IsMaster()) { - if (reason && !strcmp(reason, kPROOF_TerminateWorker)) { - // if the reason was a planned termination then delete the worker and - // remove it from all the lists - fSlaves->Remove(wrk); - fBadSlaves->Remove(wrk); - fActiveSlaves->Remove(wrk); - fInactiveSlaves->Remove(wrk); - fUniqueSlaves->Remove(wrk); - fAllUniqueSlaves->Remove(wrk); - fNonUniqueMasters->Remove(wrk); - - // we add it to the list of terminated slave infos instead, so that it - // stays available in the .workers persistent file - TSlaveInfo *si = new TSlaveInfo( - wrk->GetOrdinal(), - Form("%s@%s:%d", wrk->GetUser(), wrk->GetName(), wrk->GetPort()), - 0, "", wrk->GetWorkDir()); - if (!fTerminatedSlaveInfos->Contains(si)) fTerminatedSlaveInfos->Add(si); - else delete si; - - delete wrk; - } else { - fBadSlaves->Add(wrk); - fActiveSlaves->Remove(wrk); - fUniqueSlaves->Remove(wrk); - fAllUniqueSlaves->Remove(wrk); - fNonUniqueMasters->Remove(wrk); - if (fCurrentMonitor) fCurrentMonitor->DeActivate(wrk->GetSocket()); - wrk->Close(); - // Update the mergers count, if needed - if (fMergersSet) { - Int_t mergersCount = -1; - TParameter *mc = dynamic_cast *>(GetParameter("PROOF_UseMergers")); - if (mc) mergersCount = mc->GetVal(); // Value set by user - // Mergers count is set dynamically: recalculate it - if (mergersCount == 0) { - Int_t activeWorkers = fCurrentMonitor ? fCurrentMonitor->GetActive() : GetNumberOfActiveSlaves(); - if (activeWorkers > 1) { - fMergersCount = TMath::Nint(TMath::Sqrt(activeWorkers)); - if (activeWorkers / fMergersCount < 2) - fMergersCount = (Int_t) TMath::Sqrt(activeWorkers); - } - } - } - } - - // Update session workers files - SaveWorkerInfo(); - } else { - // On clients the proof session should be removed from the lists - // and deleted, since it is not valid anymore - fSlaves->Remove(wrk); - if (fManager) - fManager->DiscardSession(this); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add slave with socket s to the bad slave list and remove if from -/// the active list and from the two monitor objects. - -void TProof::MarkBad(TSocket *s, const char *reason) -{ - std::lock_guard lock(fCloseMutex); - - // We may have been invalidated in the meanwhile: nothing to do in such a case - if (!IsValid()) return; - - TSlave *wrk = FindSlave(s); - MarkBad(wrk, reason); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ask an active worker 'wrk' to terminate, i.e. to shutdown - -void TProof::TerminateWorker(TSlave *wrk) -{ - if (!wrk) { - Warning("TerminateWorker", "worker instance undefined: protocol error? "); - return; - } - - // Send stop message - if (wrk->GetSocket() && wrk->GetSocket()->IsValid()) { - TMessage mess(kPROOF_STOP); - wrk->GetSocket()->Send(mess); - } else { - if (gDebug > 0) - Info("TerminateWorker", "connection to worker is already down: cannot" - " send termination message"); - } - - // This is a bad worker from now on - MarkBad(wrk, kPROOF_TerminateWorker); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ask an active worker 'ord' to terminate, i.e. to shutdown - -void TProof::TerminateWorker(const char *ord) -{ - if (ord && strlen(ord) > 0) { - Bool_t all = (ord[0] == '*') ? kTRUE : kFALSE; - if (IsMaster()) { - TIter nxw(fSlaves); - TSlave *wrk = 0; - while ((wrk = (TSlave *)nxw())) { - if (all || !strcmp(wrk->GetOrdinal(), ord)) { - TerminateWorker(wrk); - if (!all) break; - } - } - } else { - TMessage mess(kPROOF_STOP); - mess << TString(ord); - Broadcast(mess); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ping PROOF. Returns 1 if master server responded. - -Int_t TProof::Ping() -{ - return Ping(kActive); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ping PROOF slaves. Returns the number of slaves that responded. - -Int_t TProof::Ping(ESlaves list) -{ - TList *slaves = 0; - if (list == kAll) slaves = fSlaves; - if (list == kActive) slaves = fActiveSlaves; - if (list == kUnique) slaves = fUniqueSlaves; - if (list == kAllUnique) slaves = fAllUniqueSlaves; - - if (slaves->GetSize() == 0) return 0; - - int nsent = 0; - TIter next(slaves); - - TSlave *sl; - while ((sl = (TSlave *)next())) { - if (sl->IsValid()) { - if (sl->Ping() == -1) { - MarkBad(sl, "ping unsuccessful"); - } else { - nsent++; - } - } - } - - return nsent; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ping PROOF slaves. Returns the number of slaves that responded. - -void TProof::Touch() -{ - TList *slaves = fSlaves; - - if (slaves->GetSize() == 0) return; - - TIter next(slaves); - - TSlave *sl; - while ((sl = (TSlave *)next())) { - if (sl->IsValid()) { - sl->Touch(); - } - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print status of PROOF cluster. - -void TProof::Print(Option_t *option) const -{ - TString secCont; - - if (TestBit(TProof::kIsClient)) { - Printf("Connected to: %s (%s)", GetMaster(), - IsValid() ? "valid" : "invalid"); - Printf("Port number: %d", GetPort()); - Printf("User: %s", GetUser()); - Printf("ROOT version|rev: %s|%s", gROOT->GetVersion(), gROOT->GetGitCommit()); - Printf("Architecture-Compiler: %s-%s", gSystem->GetBuildArch(), - gSystem->GetBuildCompilerVersion()); - TSlave *sl = (TSlave *)fActiveSlaves->First(); - if (sl) { - TString sc; - if (sl->GetSocket()->GetSecContext()) - Printf("Security context: %s", - sl->GetSocket()->GetSecContext()->AsString(sc)); - Printf("Proofd protocol version: %d", sl->GetSocket()->GetRemoteProtocol()); - } else { - Printf("Security context: Error - No connection"); - Printf("Proofd protocol version: Error - No connection"); - } - Printf("Client protocol version: %d", GetClientProtocol()); - Printf("Remote protocol version: %d", GetRemoteProtocol()); - Printf("Log level: %d", GetLogLevel()); - Printf("Session unique tag: %s", IsValid() ? GetSessionTag() : ""); - Printf("Default data pool: %s", IsValid() ? GetDataPoolUrl() : ""); - if (IsValid()) - const_cast(this)->SendPrint(option); - } else { - const_cast(this)->AskStatistics(); - if (IsParallel()) - Printf("*** Master server %s (parallel mode, %d workers):", - gProofServ->GetOrdinal(), GetParallel()); - else - Printf("*** Master server %s (sequential mode):", - gProofServ->GetOrdinal()); - - Printf("Master host name: %s", gSystem->HostName()); - Printf("Port number: %d", GetPort()); - if (strlen(gProofServ->GetGroup()) > 0) { - Printf("User/Group: %s/%s", GetUser(), gProofServ->GetGroup()); - } else { - Printf("User: %s", GetUser()); - } - TString ver; - ver.Form("%s|%s", gROOT->GetVersion(), gROOT->GetGitCommit()); - if (gSystem->Getenv("ROOTVERSIONTAG")) - ver.Form("%s|%s", gROOT->GetVersion(), gSystem->Getenv("ROOTVERSIONTAG")); - Printf("ROOT version|rev|tag: %s", ver.Data()); - Printf("Architecture-Compiler: %s-%s", gSystem->GetBuildArch(), - gSystem->GetBuildCompilerVersion()); - Printf("Protocol version: %d", GetClientProtocol()); - Printf("Image name: %s", GetImage()); - Printf("Working directory: %s", gSystem->WorkingDirectory()); - Printf("Config directory: %s", GetConfDir()); - Printf("Config file: %s", GetConfFile()); - Printf("Log level: %d", GetLogLevel()); - Printf("Number of workers: %d", GetNumberOfSlaves()); - Printf("Number of active workers: %d", GetNumberOfActiveSlaves()); - Printf("Number of unique workers: %d", GetNumberOfUniqueSlaves()); - Printf("Number of inactive workers: %d", GetNumberOfInactiveSlaves()); - Printf("Number of bad workers: %d", GetNumberOfBadSlaves()); - Printf("Total MB's processed: %.2f", float(GetBytesRead())/(1024*1024)); - Printf("Total real time used (s): %.3f", GetRealTime()); - Printf("Total CPU time used (s): %.3f", GetCpuTime()); - if (TString(option).Contains("a", TString::kIgnoreCase) && GetNumberOfSlaves()) { - Printf("List of workers:"); - TList masters; - TIter nextslave(fSlaves); - while (TSlave* sl = dynamic_cast(nextslave())) { - if (!sl->IsValid()) continue; - - if (sl->GetSlaveType() == TSlave::kSlave) { - sl->Print(option); - } else if (sl->GetSlaveType() == TSlave::kMaster) { - TMessage mess(kPROOF_PRINT); - mess.WriteString(option); - if (sl->GetSocket()->Send(mess) == -1) - const_cast(this)->MarkBad(sl, "could not send kPROOF_PRINT request"); - else - masters.Add(sl); - } else { - Error("Print", "TSlave is neither Master nor Worker"); - R__ASSERT(0); - } - } - const_cast(this)->Collect(&masters, fCollectTimeout); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Extract from opt information about output handling settings. -/// The understood keywords are: -/// of=``, outfile=`` output file location -/// ds=``, dataset=`` dataset name ('of' and 'ds' are -/// mutually exclusive,execution stops -/// if both are found) -/// sft[=``], savetofile[=``] control saving to file -/// -/// For 'mvf', the `` integer has the following meaning: -/// `` = ``*10 + `` -/// `` = 0 save to file if memory threshold is reached -/// (the memory threshold is set by the cluster -/// admin); in case an output file is defined, the -/// files are merged at the end; -/// 1 save results to file. -/// `` = 0 save at the end of the query -/// 1 save results after each packet (to reduce the -/// loss in case of crash). -/// -/// Setting 'ds' automatically sets 'mvf=1'; it is still possible to set 'mvf=11' -/// to save results after each packet. -/// -/// The separator from the next option is either a ' ' or a ';' -/// -/// All recognized settings are removed from the input string opt. -/// If action == 0, set up the output file accordingly, if action == 1 clean related -/// output file settings. -/// If the final target file is local then 'target' is set to the final local path -/// when action == 0 and used to retrieve the file with TFile::Cp when action == 1. -/// -/// Output file settings are in the form -/// -/// ``of=name `` -/// ``outfile=name,...;`` -/// -/// The separator from the next option is either a ' ' or a ';' -/// Called interanally by TProof::Process. -/// -/// Returns 0 on success, -1 on error. - -Int_t TProof::HandleOutputOptions(TString &opt, TString &target, Int_t action) -{ - TString outfile, dsname, stfopt; - if (action == 0) { - TString tagf, tagd, tags, oo; - Ssiz_t from = 0, iof = kNPOS, iod = kNPOS, ios = kNPOS; - while (opt.Tokenize(oo, from, "[; ]")) { - if (oo.BeginsWith("of=")) { - tagf = "of="; - iof = opt.Index(tagf); - } else if (oo.BeginsWith("outfile=")) { - tagf = "outfile="; - iof = opt.Index(tagf); - } else if (oo.BeginsWith("ds")) { - tagd = "ds"; - iod = opt.Index(tagd); - } else if (oo.BeginsWith("dataset")) { - tagd = "dataset"; - iod = opt.Index(tagd); - } else if (oo.BeginsWith("stf")) { - tags = "stf"; - ios = opt.Index(tags); - } else if (oo.BeginsWith("savetofile")) { - tags = "savetofile"; - ios = opt.Index(tags); - } - } - // Check consistency - if (iof != kNPOS && iod != kNPOS) { - Error("HandleOutputOptions", "options 'of'/'outfile' and 'ds'/'dataset' are incompatible!"); - return -1; - } - - // Check output file first - if (iof != kNPOS) { - from = iof + tagf.Length(); - if (!opt.Tokenize(outfile, from, "[; ]") || outfile.IsNull()) { - Error("HandleOutputOptions", "could not extract output file settings string! (%s)", opt.Data()); - return -1; - } - // For removal from original options string - tagf += outfile; - } - // Check dataset - if (iod != kNPOS) { - from = iod + tagd.Length(); - if (!opt.Tokenize(dsname, from, "[; ]")) - if (gDebug > 0) Info("HandleOutputOptions", "no dataset name found: use default"); - // For removal from original options string - tagd += dsname; - // The name may be empty or beginning with a '=' - if (dsname.BeginsWith("=")) dsname.Replace(0, 1, ""); - if (dsname.Contains("|V")) { - target = "ds|V"; - dsname.ReplaceAll("|V", ""); - } - if (dsname.IsNull()) dsname = "dataset_"; - } - // Check stf - if (ios != kNPOS) { - from = ios + tags.Length(); - if (!opt.Tokenize(stfopt, from, "[; ]")) - if (gDebug > 0) Info("HandleOutputOptions", "save-to-file not found: use default"); - // For removal from original options string - tags += stfopt; - // It must be digit - if (!stfopt.IsNull()) { - if (stfopt.BeginsWith("=")) stfopt.Replace(0,1,""); - if (!stfopt.IsNull()) { - if (!stfopt.IsDigit()) { - Error("HandleOutputOptions", "save-to-file option must be a digit! (%s)", stfopt.Data()); - return -1; - } - } else { - // Default - stfopt = "1"; - } - } else { - // Default - stfopt = "1"; - } - } - // Remove from original options string - opt.ReplaceAll(tagf, ""); - opt.ReplaceAll(tagd, ""); - opt.ReplaceAll(tags, ""); - } - - // Parse now - if (action == 0) { - // Output file - if (!outfile.IsNull()) { - if (!outfile.BeginsWith("master:")) { - if (gSystem->AccessPathName(gSystem->GetDirName(outfile.Data()), kWritePermission)) { - Warning("HandleOutputOptions", - "directory '%s' for the output file does not exists or is not writable:" - " saving to master", gSystem->GetDirName(outfile.Data()).Data()); - outfile.Form("master:%s", gSystem->BaseName(outfile.Data())); - } else { - if (!IsLite()) { - // The target file is local, so we need to retrieve it - target = outfile; - if (!stfopt.IsNull()) { - outfile.Form("master:%s", gSystem->BaseName(target.Data())); - } else { - outfile = ""; - } - } - } - } - if (outfile.BeginsWith("master:")) { - outfile.ReplaceAll("master:", ""); - if (outfile.IsNull() || !gSystem->IsAbsoluteFileName(outfile)) { - // Get the master data dir - TString ddir, emsg; - if (!IsLite()) { - if (Exec("gProofServ->GetDataDir()", "0", kTRUE) == 0) { - TObjString *os = fMacroLog.GetLineWith("const char"); - if (os) { - Ssiz_t fst = os->GetString().First('\"'); - Ssiz_t lst = os->GetString().Last('\"'); - ddir = os->GetString()(fst+1, lst-fst-1); - } else { - emsg = "could not find 'const char *' string in macro log! cannot continue"; - } - } else { - emsg = "could not retrieve master data directory info! cannot continue"; - } - if (!emsg.IsNull()) { - Error("HandleOutputOptions", "%s", emsg.Data()); - return -1; - } - } - if (!ddir.IsNull()) ddir += "/"; - if (outfile.IsNull()) { - outfile.Form("%s", ddir.Data()); - } else { - outfile.Insert(0, TString::Format("%s", ddir.Data())); - } - } - } - // Set the parameter - if (!outfile.IsNull()) { - if (!outfile.BeginsWith("of:")) outfile.Insert(0, "of:"); - SetParameter("PROOF_DefaultOutputOption", outfile.Data()); - } - } - // Dataset creation - if (!dsname.IsNull()) { - dsname.Insert(0, "ds:"); - // Set the parameter - SetParameter("PROOF_DefaultOutputOption", dsname.Data()); - // Check the Save-To-File option - if (!stfopt.IsNull()) { - Int_t ostf = (Int_t) stfopt.Atoi(); - if (ostf%10 <= 0) { - Warning("HandleOutputOptions", "Dataset required bu Save-To-File disabled: enabling!"); - stfopt.Form("%d", ostf+1); - } - } else { - // Minimal setting - stfopt = "1"; - } - } - // Save-To-File options - if (!stfopt.IsNull()) { - // Set the parameter - SetParameter("PROOF_SavePartialResults", (Int_t) stfopt.Atoi()); - } - } else { - // Retrieve the file, if required - if (GetOutputList()) { - if (target == "ds|V") { - // Find the dataset - dsname = ""; - TIter nxo(GetOutputList()); - TObject *o = 0; - while ((o = nxo())) { - if (o->InheritsFrom(TFileCollection::Class())) { - VerifyDataSet(o->GetName()); - dsname = o->GetName(); - break; - } - } - if (!dsname.IsNull()) { - TFileCollection *fc = GetDataSet(dsname); - if (fc) { - fc->Print(); - } else { - Warning("HandleOutputOptions", "could not retrieve TFileCollection for dataset '%s'", dsname.Data()); - } - } else { - Warning("HandleOutputOptions", "dataset not found!"); - } - } else { - Bool_t targetcopied = kFALSE; - TProofOutputFile *pf = 0; - if (!target.IsNull()) - pf = (TProofOutputFile *) GetOutputList()->FindObject(gSystem->BaseName(target.Data())); - if (pf) { - // Copy the file - if (strcmp(TUrl(pf->GetOutputFileName(), kTRUE).GetUrl(), - TUrl(target, kTRUE).GetUrl())) { - if (TFile::Cp(pf->GetOutputFileName(), target)) { - Printf(" Output successfully copied to %s", target.Data()); - targetcopied = kTRUE; - } else { - Warning("HandleOutputOptions", "problems copying output to %s", target.Data()); - } - } - } - TFile *fout = 0; - TObject *o = 0; - TIter nxo(GetOutputList()); - Bool_t swapcopied = kFALSE; - while ((o = nxo())) { - TProofOutputFile *pof = dynamic_cast(o); - if (pof) { - if (pof->TestBit(TProofOutputFile::kSwapFile) && !target.IsNull()) { - if (pof == pf && targetcopied) continue; - // Copy the file - if (strcmp(TUrl(pf->GetOutputFileName(), kTRUE).GetUrl(), - TUrl(target, kTRUE).GetUrl())) { - if (TFile::Cp(pof->GetOutputFileName(), target)) { - Printf(" Output successfully copied to %s", target.Data()); - swapcopied = kTRUE; - } else { - Warning("HandleOutputOptions", "problems copying output to %s", target.Data()); - } - } - } else if (pof->IsRetrieve()) { - // Retrieve this file to the local path indicated in the title - if (strcmp(TUrl(pf->GetOutputFileName(), kTRUE).GetUrl(), - TUrl(pof->GetTitle(), kTRUE).GetUrl())) { - if (TFile::Cp(pof->GetOutputFileName(), pof->GetTitle())) { - Printf(" Output successfully copied to %s", pof->GetTitle()); - } else { - Warning("HandleOutputOptions", - "problems copying %s to %s", pof->GetOutputFileName(), pof->GetTitle()); - } - } - } - } - } - if (!target.IsNull() && !swapcopied) { - if (!fout && !pf) { - fout = TFile::Open(target, "RECREATE"); - if (!fout || (fout && fout->IsZombie())) { - SafeDelete(fout); - Warning("HandleOutputOptions", "problems opening output file %s", target.Data()); - } - } - if (fout) { - nxo.Reset(); - while ((o = nxo())) { - TProofOutputFile *pof = dynamic_cast(o); - if (!pof) { - // Write the object to the open output file - o->Write(); - } - } - } - } - // Clean-up - if (fout) { - fout->Close(); - SafeDelete(fout); - Printf(" Output saved to %s", target.Data()); - } - } - } - // Remove the parameter - DeleteParameters("PROOF_DefaultOutputOption"); - // Remove the parameter - DeleteParameters("PROOF_SavePartialResults"); - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Extract from opt in optfb information about wanted feedback settings. -/// Feedback are removed from the input string opt. -/// If action == 0, set up feedback accordingly, if action == 1 clean related -/// feedback settings (using info in optfb, if available, or reparsing opt). -/// -/// Feedback requirements are in the form -/// -/// fb=name1,name2,name3,... -/// feedback=name1,name2,name3,...; -/// -/// The special name 'stats' triggers feedback about events and packets. -/// The separator from the next option is either a ' ' or a ';'. -/// Called interanally by TProof::Process. - -void TProof::SetFeedback(TString &opt, TString &optfb, Int_t action) -{ - Ssiz_t from = 0; - if (action == 0 || (action == 1 && optfb.IsNull())) { - TString tag("fb="); - Ssiz_t ifb = opt.Index(tag); - if (ifb == kNPOS) { - tag = "feedback="; - ifb = opt.Index(tag); - } - if (ifb == kNPOS) return; - from = ifb + tag.Length(); - - if (!opt.Tokenize(optfb, from, "[; ]") || optfb.IsNull()) { - Warning("SetFeedback", "could not extract feedback string! Ignoring ..."); - return; - } - // Remove from original options string - tag += optfb; - opt.ReplaceAll(tag, ""); - } - - // Parse now - TString nm, startdraw, stopdraw; - from = 0; - while (optfb.Tokenize(nm, from, ",")) { - // Special name first - if (nm == "stats") { - if (action == 0) { - startdraw.Form("gDirectory->Add(new TStatsFeedback((TProof *)%p))", this); - gROOT->ProcessLine(startdraw.Data()); - SetParameter("PROOF_StatsHist", ""); - AddFeedback("PROOF_EventsHist"); - AddFeedback("PROOF_PacketsHist"); - AddFeedback("PROOF_ProcPcktHist"); - } else { - stopdraw.Form("TObject *o = gDirectory->FindObject(\"%s\"); " - " if (o && strcmp(o->ClassName(), \"TStatsFeedback\")) " - " { gDirectory->Remove(o); delete o; }", GetSessionTag()); - gROOT->ProcessLine(stopdraw.Data()); - DeleteParameters("PROOF_StatsHist"); - RemoveFeedback("PROOF_EventsHist"); - RemoveFeedback("PROOF_PacketsHist"); - RemoveFeedback("PROOF_ProcPcktHist"); - } - } else { - if (action == 0) { - // Enable or - AddFeedback(nm); - startdraw.Form("gDirectory->Add(new TDrawFeedback((TProof *)%p))", this); - gROOT->ProcessLine(startdraw.Data()); - } else { - // ... or disable - RemoveFeedback(nm); - stopdraw.Form("TObject *o = gDirectory->FindObject(\"%s\"); " - " if (o && strcmp(o->ClassName(), \"TDrawFeedback\")) " - " { gDirectory->Remove(o); delete o; }", GetSessionTag()); - gROOT->ProcessLine(stopdraw.Data()); - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process a data set (TDSet) using the specified selector (.C) file or -/// Tselector object -/// Entry- or event-lists should be set in the data set object using -/// TDSet::SetEntryList. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProof::Process(TDSet *dset, const char *selector, Option_t *option, - Long64_t nentries, Long64_t first) -{ - if (!IsValid() || !fPlayer) return -1; - - // Set PROOF to running state - SetRunStatus(TProof::kRunning); - - TString opt(option), optfb, outfile; - // Enable feedback, if required - if (opt.Contains("fb=") || opt.Contains("feedback=")) SetFeedback(opt, optfb, 0); - // Define output file, either from 'opt' or the default one - if (HandleOutputOptions(opt, outfile, 0) != 0) return -1; - - // Resolve query mode - fSync = (GetQueryMode(opt) == kSync); - - if (fSync && (!IsIdle() || IsWaiting())) { - // Already queued or processing queries: switch to asynchronous mode - Info("Process", "session is in waiting or processing status: switch to asynchronous mode"); - fSync = kFALSE; - opt.ReplaceAll("SYNC",""); - opt += "ASYN"; - } - - // Cleanup old temporary datasets - if ((IsIdle() && !IsWaiting()) && fRunningDSets && fRunningDSets->GetSize() > 0) { - fRunningDSets->SetOwner(kTRUE); - fRunningDSets->Delete(); - } - - // deactivate the default application interrupt handler - // ctrl-c's will be forwarded to PROOF to stop the processing - TSignalHandler *sh = 0; - if (fSync) { - if (gApplication) - sh = gSystem->RemoveSignalHandler(gApplication->GetSignalHandler()); - } - - // Make sure we get a fresh result - fOutputList.Clear(); - - // Make sure that the workers ready list is empty - if (fWrksOutputReady) { - fWrksOutputReady->SetOwner(kFALSE); - fWrksOutputReady->Clear(); - } - - // Make sure the selector path is in the macro path - TProof::AssertMacroPath(selector); - - // Reset time measurements - fQuerySTW.Reset(); - - Long64_t rv = -1; - if (selector && strlen(selector)) { - rv = fPlayer->Process(dset, selector, opt.Data(), nentries, first); - } else if (fSelector) { - rv = fPlayer->Process(dset, fSelector, opt.Data(), nentries, first); - } else { - Error("Process", "neither a selecrot file nor a selector object have" - " been specified: cannot process!"); - } - - // This is the end of merging - fQuerySTW.Stop(); - Float_t rt = fQuerySTW.RealTime(); - // Update the query content - TQueryResult *qr = GetQueryResult(); - if (qr) { - qr->SetTermTime(rt); - qr->SetPrepTime(fPrepTime); - } - - // Disable feedback, if required - if (!optfb.IsNull()) SetFeedback(opt, optfb, 1); - // Finalise output file settings (opt is ignored in here) - if (HandleOutputOptions(opt, outfile, 1) != 0) return -1; - - // Retrieve status from the output list - if (rv >= 0) { - TParameter *sst = - (TParameter *) fOutputList.FindObject("PROOF_SelectorStatus"); - if (sst) rv = sst->GetVal(); - } - - if (fSync) { - // reactivate the default application interrupt handler - if (sh) - gSystem->AddSignalHandler(sh); - // Save the performance info, if required - if (!fPerfTree.IsNull()) { - if (SavePerfTree() != 0) Error("Process", "saving performance info ..."); - // Must be re-enabled each time - SetPerfTree(0); - } - } - - return rv; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process a data set (TFileCollection) using the specified selector (.C) file -/// or TSelector object. -/// The default tree is analyzed (i.e. the first one found). To specify another -/// tree, the default tree can be changed using TFileCollection::SetDefaultMetaData . -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProof::Process(TFileCollection *fc, const char *selector, - Option_t *option, Long64_t nentries, Long64_t first) -{ - if (!IsValid() || !fPlayer) return -1; - - if (fProtocol < 17) { - Info("Process", "server version < 5.18/00:" - " processing of TFileCollection not supported"); - return -1; - } - - // We include the TFileCollection to the input list and we create a - // fake TDSet with infor about it - TDSet *dset = new TDSet(TString::Format("TFileCollection:%s", fc->GetName()), 0, 0, ""); - fPlayer->AddInput(fc); - - - Long64_t retval = -1; - if (selector && strlen(selector)) { - retval = Process(dset, selector, option, nentries, first); - } else if (fSelector) { - retval = Process(dset, fSelector, option, nentries, first); - } else { - Error("Process", "neither a selecrot file nor a selector object have" - " been specified: cannot process!"); - } - fPlayer->GetInputList()->Remove(fc); // To avoid problems in future - - // Cleanup - if (IsLite() && !fSync) { - if (!fRunningDSets) fRunningDSets = new TList; - fRunningDSets->Add(dset); - } else { - delete dset; - } - - return retval; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process a dataset which is stored on the master with name 'dsetname'. -/// The syntax for dsetname is name[#[dir/]objname], e.g. -/// "mydset" analysis of the first tree in the top dir of the dataset -/// named "mydset" -/// "mydset#T" analysis tree "T" in the top dir of the dataset -/// named "mydset" -/// "mydset#adir/T" analysis tree "T" in the dir "adir" of the dataset -/// named "mydset" -/// "mydset#adir/" analysis of the first tree in the dir "adir" of the -/// dataset named "mydset" -/// The component 'name' in its more general form contains also the group and -/// user name following "///". Each of these components -/// can contain one or more wildcards '*', in which case all the datasets matching -/// the expression are added together as a global dataset (wildcard support has -/// been added in version 5.27/02). -/// The last argument 'elist' specifies an entry- or event-list to be used as -/// event selection. -/// It is also possible (starting w/ version 5.27/02) to run on multiple datasets -/// at once in a more flexible way that the one provided by wildcarding. There -/// are three possibilities: -/// 1) specifying the dataset names separated by the OR operator '|', e.g. -/// dsetname = "|||..." -/// in this case the datasets are a seen as a global unique dataset -/// 2) specifying the dataset names separated by a ',' or a ' ', e.g. -/// dsetname = ", ,..." -/// in this case the datasets are processed one after the other and the -/// selector is notified when switching dataset via a bit in the current -/// processed element. -/// 3) giving the path of a textfile where the dataset names are specified -/// on one or multiple lines; the lines found are joined as in 1), unless -/// the filepath is followed by a ',' (i.e. p->Process("datasets.txt,",...) -/// with the dataset names listed in 'datasets.txt') in which case they are -/// treated as in 2); the file is open in raw mode with TFile::Open and -/// therefore it cane be remote, e.g. on a Web server. -/// Each `` has the format specified above for the single dataset processing, -/// included wildcarding (the name of the tree and subdirectory must be same for -/// all the datasets). -/// In the case of multiple datasets, 'elist' is treated a global entry list. -/// It is possible to specify per-dataset entry lists using the syntax -/// "mydset[#adir/[T]]?enl=entrylist" -/// or -/// "mydset[#adir/[T]]<AccessPathName(fname, kReadPermission))) { - TUrl uf(fname, kTRUE); - uf.SetOptions(TString::Format("%sfiletype=raw", uf.GetOptions())); - TFile *f = TFile::Open(uf.GetUrl()); - if (f && !(f->IsZombie())) { - const Int_t blen = 8192; - char buf[blen]; - Long64_t rest = f->GetSize(); - while (rest > 0) { - Long64_t len = (rest > blen - 1) ? blen - 1 : rest; - if (f->ReadBuffer(buf, len)) { - Error("Process", "problems reading from file '%s'", fname.Data()); - dsname = ""; - break; - } - buf[len] = '\0'; - dsname += buf; - rest -= len; - } - f->Close(); - SafeDelete(f); - // We fail if a failure occured - if (rest > 0) return -1; - } else { - Error("Process", "could not open file '%s'", fname.Data()); - return -1; - } - } - if (dsname.IsNull()) { - dsname = dsetname; - } else { - // Remove trailing '\n' - if (dsname.EndsWith("\n")) dsname.Remove(dsname.Length()-1, 1); - // Replace all '\n' with the proper separator - dsname.ReplaceAll("\n", separator); - if (gDebug > 0) { - Info("Process", "processing multi-dataset read from file '%s':", fname.Data()); - Info("Process", " '%s'", dsname.Data()); - } - } - - TString names(dsname), name, enl, newname; - // If multi-dataset check if server supports it - if (fProtocol < 28 && names.Index(TRegexp("[, |]")) != kNPOS) { - Info("Process", "multi-dataset processing not supported by the server"); - return -1; - } - - TEntryList *el = 0; - TString dsobj, dsdir; - Int_t from = 0; - while (names.Tokenize(name, from, "[, |]")) { - - newname = name; - // Extract the specific entry-list, if any - enl = ""; - Int_t ienl = name.Index("?enl="); - if (ienl == kNPOS) { - ienl = name.Index("<<"); - if (ienl != kNPOS) { - newname.Remove(ienl); - ienl += strlen("<<"); - } - } else { - newname.Remove(ienl); - ienl += strlen("?enl="); - } - - // Check the name syntax first - TString obj, dir("/"); - Int_t idxc = newname.Index("#"); - if (idxc != kNPOS) { - Int_t idxs = newname.Index("/", 1, idxc, TString::kExact); - if (idxs != kNPOS) { - obj = newname(idxs+1, newname.Length()); - dir = newname(idxc+1, newname.Length()); - dir.Remove(dir.Index("/") + 1); - newname.Remove(idxc); - } else { - obj = newname(idxc+1, newname.Length()); - newname.Remove(idxc); - } - } else if (newname.Index(":") != kNPOS && newname.Index("://") == kNPOS) { - // protection against using ':' instead of '#' - Error("Process", "bad name syntax (%s): please use" - " a '#' after the dataset name", name.Data()); - dsname.ReplaceAll(name, ""); - continue; - } - if (dsobj.IsNull() && dsdir.IsNull()) { - // The first one specifies obj and dir - dsobj = obj; - dsdir = dir; - } else if (obj != dsobj || dir != dsdir) { - // Inconsistent specification: not supported - Warning("Process", "'obj' or 'dir' specification not consistent w/ the first given: ignore"); - } - // Process the entry-list name, if any - if (ienl != kNPOS) { - // Get entrylist name or path - enl = name(ienl, name.Length()); - el = 0; - TObject *oel = 0; - // If not in the input list ... - TList *inpl = GetInputList(); - if (inpl && (oel = inpl->FindObject(enl))) el = dynamic_cast(oel); - // ... check the heap - if (!el && gDirectory && (oel = gDirectory->FindObject(enl))) { - if ((el = dynamic_cast(oel))) { - // Add to the input list (input data not available on master where - // this info will be processed) - if (fProtocol >= 28) - if (!(inpl->FindObject(el->GetName()))) AddInput(el); - } - } - // If not in the heap, check a file, if any - if (!el) { - if (!gSystem->AccessPathName(enl)) { - TFile *f = TFile::Open(enl); - if (f && !(f->IsZombie()) && f->GetListOfKeys()) { - TIter nxk(f->GetListOfKeys()); - TKey *k = 0; - while ((k = (TKey *) nxk())) { - if (!strcmp(k->GetClassName(), "TEntryList")) { - if (!el) { - if ((el = dynamic_cast(f->Get(k->GetName())))) { - // Add to the input list (input data not available on master where - // this info will be processed) - if (fProtocol >= 28) { - if (!(inpl->FindObject(el->GetName()))) { - el = (TEntryList *) el->Clone(); - AddInput(el); - } - } else { - el = (TEntryList *) el->Clone(); - } - } - } else if (strcmp(el->GetName(), k->GetName())) { - Warning("Process", "multiple entry lists found in file '%s': the first one is taken;\n" - "if this is not what you want, load first the content in memory" - "and select it by name ", enl.Data()); - } - } - } - } else { - Warning("Process","file '%s' cannot be open or is empty - ignoring", enl.Data()); - } - } - } - // Transmit the information - if (fProtocol >= 28) { - newname += "?enl="; - if (el) { - // An entry list object is avalaible in the input list: add its name - newname += el->GetName(); - } else { - // The entry list object was not found: send the name, the future entry list manager will - // find it on the server side - newname += enl; - } - } - } - // Adjust the name for this dataset - dsname.ReplaceAll(name, newname); - } - - // Create the dataset object - TDSet *dset = new TDSet(dsname, dsobj, dsdir); - // Set entry list - if (el && fProtocol < 28) { - dset->SetEntryList(el); - } else { - dset->SetEntryList(elist); - } - // Run - Long64_t retval = -1; - if (selector && strlen(selector)) { - retval = Process(dset, selector, option, nentries, first); - } else if (fSelector) { - retval = Process(dset, fSelector, option, nentries, first); - } else { - Error("Process", "neither a selector file nor a selector object have" - " been specified: cannot process!"); - } - // Cleanup - if (IsLite() && !fSync) { - if (!fRunningDSets) fRunningDSets = new TList; - fRunningDSets->Add(dset); - } else { - delete dset; - } - - return retval; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Generic (non-data based) selector processing: the Process() method of the -/// specified selector (.C) or TSelector object is called 'n' times. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProof::Process(const char *selector, Long64_t n, Option_t *option) -{ - if (!IsValid()) return -1; - - if (fProtocol < 16) { - Info("Process", "server version < 5.17/04: generic processing not supported"); - return -1; - } - - // Fake data set - TDSet *dset = new TDSet; - dset->SetBit(TDSet::kEmpty); - - Long64_t retval = -1; - if (selector && strlen(selector)) { - retval = Process(dset, selector, option, n); - } else if (fSelector) { - retval = Process(dset, fSelector, option, n); - } else { - Error("Process", "neither a selector file nor a selector object have" - " been specified: cannot process!"); - } - - // Cleanup - if (IsLite() && !fSync) { - if (!fRunningDSets) fRunningDSets = new TList; - fRunningDSets->Add(dset); - } else { - delete dset; - } - return retval; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process a data set (TDSet) using the specified selector object. -/// Entry- or event-lists should be set in the data set object using -/// TDSet::SetEntryList. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProof::Process(TDSet *dset, TSelector *selector, Option_t *option, - Long64_t nentries, Long64_t first) -{ - if (fProtocol < 34) { - Error("Process", "server version < 5.33/02:" - "processing by object not supported"); - return -1; - } - if (!selector) { - Error("Process", "selector object undefined!"); - return -1; - } - fSelector = selector; - Long64_t rc = Process(dset, (const char*)0, option, nentries, first); - fSelector = 0; - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process a data set (TFileCollection) using the specified selector object -/// The default tree is analyzed (i.e. the first one found). To specify another -/// tree, the default tree can be changed using TFileCollection::SetDefaultMetaData . -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProof::Process(TFileCollection *fc, TSelector *selector, - Option_t *option, Long64_t nentries, Long64_t first) -{ - if (fProtocol < 34) { - Error("Process", "server version < 5.33/02:" - "processing by object not supported"); - return -1; - } - if (!selector) { - Error("Process", "selector object undefined!"); - return -1; - } - fSelector = selector; - Long64_t rc = Process(fc, (const char*)0, option, nentries, first); - fSelector = 0; - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process with name of dataset and TSelector object - -Long64_t TProof::Process(const char *dsetname, TSelector *selector, - Option_t *option, Long64_t nentries, - Long64_t first, TObject *elist) -{ - if (fProtocol < 34) { - Error("Process", "server version < 5.33/02:" - "processing by object not supported"); - return -1; - } - if (!selector) { - Error("Process", "selector object undefined!"); - return -1; - } - fSelector = selector; - Long64_t rc = Process(dsetname, (const char*)0, option, nentries, first, elist); - fSelector = 0; - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Generic (non-data based) selector processing: the Process() method of the -/// specified selector is called 'n' times. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProof::Process(TSelector *selector, Long64_t n, Option_t *option) -{ - if (fProtocol < 34) { - Error("Process", "server version < 5.33/02:" - "processing by object not supported"); - return -1; - } - if (!selector) { - Error("Process", "selector object undefined!"); - return -1; - } - fSelector = selector; - Long64_t rc = Process((const char*)0, n, option); - fSelector = 0; - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get reference for the qry-th query in fQueries (as -/// displayed by ShowQueries). - -Int_t TProof::GetQueryReference(Int_t qry, TString &ref) -{ - ref = ""; - if (qry > 0) { - if (!fQueries) - GetListOfQueries(); - if (fQueries) { - TIter nxq(fQueries); - TQueryResult *qr = 0; - while ((qr = (TQueryResult *) nxq())) - if (qr->GetSeqNum() == qry) { - ref.Form("%s:%s", qr->GetTitle(), qr->GetName()); - return 0; - } - } - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize the qry-th query in fQueries. -/// If force, force retrieval if the query is found in the local list -/// but has already been finalized (default kFALSE). -/// If query < 0, finalize current query. -/// Return 0 on success, -1 on error - -Long64_t TProof::Finalize(Int_t qry, Bool_t force) -{ - if (fPlayer) { - if (qry > 0) { - TString ref; - if (GetQueryReference(qry, ref) == 0) { - return Finalize(ref, force); - } else { - Info("Finalize", "query #%d not found", qry); - } - } else { - // The last query - return Finalize("", force); - } - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize query with reference ref. -/// If force, force retrieval if the query is found in the local list -/// but has already been finalized (default kFALSE). -/// If ref = 0, finalize current query. -/// Return 0 on success, -1 on error - -Long64_t TProof::Finalize(const char *ref, Bool_t force) -{ - if (fPlayer) { - // Get the pointer to the query - TQueryResult *qr = (ref && strlen(ref) > 0) ? fPlayer->GetQueryResult(ref) - : GetQueryResult(); - Bool_t retrieve = kFALSE; - TString xref(ref); - if (!qr) { - if (!xref.IsNull()) { - retrieve = kTRUE; - } - } else { - if (qr->IsFinalized()) { - if (force) { - retrieve = kTRUE; - } else { - Info("Finalize","query already finalized:" - " use Finalize(,kTRUE) to force new retrieval"); - qr = 0; - } - } else { - retrieve = kTRUE; - xref.Form("%s:%s", qr->GetTitle(), qr->GetName()); - } - } - if (retrieve) { - Retrieve(xref.Data()); - qr = fPlayer->GetQueryResult(xref.Data()); - } - if (qr) - return fPlayer->Finalize(qr); - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send retrieve request for the qry-th query in fQueries. -/// If path is defined save it to path. - -Int_t TProof::Retrieve(Int_t qry, const char *path) -{ - if (qry > 0) { - TString ref; - if (GetQueryReference(qry, ref) == 0) - return Retrieve(ref, path); - else - Info("Retrieve", "query #%d not found", qry); - } else { - Info("Retrieve","positive argument required - do nothing"); - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send retrieve request for the query specified by ref. -/// If path is defined save it to path. -/// Generic method working for all queries known by the server. - -Int_t TProof::Retrieve(const char *ref, const char *path) -{ - if (ref) { - TMessage m(kPROOF_RETRIEVE); - m << TString(ref); - Broadcast(m, kActive); - Collect(kActive, fCollectTimeout); - - // Archive it locally, if required - if (path) { - - // Get pointer to query - TQueryResult *qr = fPlayer ? fPlayer->GetQueryResult(ref) : 0; - - if (qr) { - - TFile *farc = TFile::Open(path,"UPDATE"); - if (!farc || (farc && !(farc->IsOpen()))) { - Info("Retrieve", "archive file cannot be open (%s)", path); - return 0; - } - farc->cd(); - - // Update query status - qr->SetArchived(path); - - // Write to file - qr->Write(); - - farc->Close(); - SafeDelete(farc); - - } else { - Info("Retrieve", "query not found after retrieve"); - return -1; - } - } - - return 0; - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send remove request for the qry-th query in fQueries. - -Int_t TProof::Remove(Int_t qry, Bool_t all) -{ - if (qry > 0) { - TString ref; - if (GetQueryReference(qry, ref) == 0) - return Remove(ref, all); - else - Info("Remove", "query #%d not found", qry); - } else { - Info("Remove","positive argument required - do nothing"); - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send remove request for the query specified by ref. -/// If all = TRUE remove also local copies of the query, if any. -/// Generic method working for all queries known by the server. -/// This method can be also used to reset the list of queries -/// waiting to be processed: for that purpose use ref == "cleanupqueue". - -Int_t TProof::Remove(const char *ref, Bool_t all) -{ - if (all) { - // Remove also local copies, if any - if (fPlayer) - fPlayer->RemoveQueryResult(ref); - } - - if (IsLite()) return 0; - - if (ref) { - TMessage m(kPROOF_REMOVE); - m << TString(ref); - Broadcast(m, kActive); - Collect(kActive, fCollectTimeout); - return 0; - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send archive request for the qry-th query in fQueries. - -Int_t TProof::Archive(Int_t qry, const char *path) -{ - if (qry > 0) { - TString ref; - if (GetQueryReference(qry, ref) == 0) - return Archive(ref, path); - else - Info("Archive", "query #%d not found", qry); - } else { - Info("Archive","positive argument required - do nothing"); - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send archive request for the query specified by ref. -/// Generic method working for all queries known by the server. -/// If ref == "Default", path is understood as a default path for -/// archiving. - -Int_t TProof::Archive(const char *ref, const char *path) -{ - if (ref) { - TMessage m(kPROOF_ARCHIVE); - m << TString(ref) << TString(path); - Broadcast(m, kActive); - Collect(kActive, fCollectTimeout); - return 0; - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send cleanup request for the session specified by tag. - -Int_t TProof::CleanupSession(const char *sessiontag) -{ - if (sessiontag) { - TMessage m(kPROOF_CLEANUPSESSION); - m << TString(sessiontag); - Broadcast(m, kActive); - Collect(kActive, fCollectTimeout); - return 0; - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Change query running mode to the one specified by 'mode'. - -void TProof::SetQueryMode(EQueryMode mode) -{ - fQueryMode = mode; - - if (gDebug > 0) - Info("SetQueryMode","query mode is set to: %s", fQueryMode == kSync ? - "Sync" : "Async"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Find out the query mode based on the current setting and 'mode'. - -TProof::EQueryMode TProof::GetQueryMode(Option_t *mode) const -{ - EQueryMode qmode = fQueryMode; - - if (mode && (strlen(mode) > 0)) { - TString m(mode); - m.ToUpper(); - if (m.Contains("ASYN")) { - qmode = kAsync; - } else if (m.Contains("SYNC")) { - qmode = kSync; - } - } - - if (gDebug > 0) - Info("GetQueryMode","query mode is set to: %s", qmode == kSync ? - "Sync" : "Async"); - - return qmode; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Execute the specified drawing action on a data set (TDSet). -/// Event- or Entry-lists should be set in the data set object using -/// TDSet::SetEntryList. -/// Returns -1 in case of error or number of selected events otherwise. - -Long64_t TProof::DrawSelect(TDSet *dset, const char *varexp, - const char *selection, Option_t *option, - Long64_t nentries, Long64_t first) -{ - if (!IsValid() || !fPlayer) return -1; - - // Make sure that asynchronous processing is not active - if (!IsIdle()) { - Info("DrawSelect","not idle, asynchronous Draw not supported"); - return -1; - } - TString opt(option); - Int_t idx = opt.Index("ASYN", 0, TString::kIgnoreCase); - if (idx != kNPOS) - opt.Replace(idx,4,""); - - return fPlayer->DrawSelect(dset, varexp, selection, opt, nentries, first); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Execute the specified drawing action on a data set which is stored on the -/// master with name 'dsetname'. -/// The syntax for dsetname is name[#[dir/]objname], e.g. -/// "mydset" analysis of the first tree in the top dir of the dataset -/// named "mydset" -/// "mydset#T" analysis tree "T" in the top dir of the dataset -/// named "mydset" -/// "mydset#adir/T" analysis tree "T" in the dir "adir" of the dataset -/// named "mydset" -/// "mydset#adir/" analysis of the first tree in the dir "adir" of the -/// dataset named "mydset" -/// The last argument 'enl' specifies an entry- or event-list to be used as -/// event selection. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProof::DrawSelect(const char *dsetname, const char *varexp, - const char *selection, Option_t *option, - Long64_t nentries, Long64_t first, TObject *enl) -{ - if (fProtocol < 13) { - Info("Process", "processing 'by name' not supported by the server"); - return -1; - } - - TString name(dsetname); - TString obj; - TString dir = "/"; - Int_t idxc = name.Index("#"); - if (idxc != kNPOS) { - Int_t idxs = name.Index("/", 1, idxc, TString::kExact); - if (idxs != kNPOS) { - obj = name(idxs+1, name.Length()); - dir = name(idxc+1, name.Length()); - dir.Remove(dir.Index("/") + 1); - name.Remove(idxc); - } else { - obj = name(idxc+1, name.Length()); - name.Remove(idxc); - } - } else if (name.Index(":") != kNPOS && name.Index("://") == kNPOS) { - // protection against using ':' instead of '#' - Error("DrawSelect", "bad name syntax (%s): please use" - " a '#' after the dataset name", dsetname); - return -1; - } - - TDSet *dset = new TDSet(name, obj, dir); - // Set entry-list, if required - dset->SetEntryList(enl); - Long64_t retval = DrawSelect(dset, varexp, selection, option, nentries, first); - delete dset; - return retval; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send STOPPROCESS message to master and workers. - -void TProof::StopProcess(Bool_t abort, Int_t timeout) -{ - PDB(kGlobal,2) - Info("StopProcess","enter %d", abort); - - if (!IsValid()) - return; - - // Flag that we have been stopped - ERunStatus rst = abort ? TProof::kAborted : TProof::kStopped; - SetRunStatus(rst); - - if (fPlayer) - fPlayer->StopProcess(abort, timeout); - - // Stop any blocking 'Collect' request; on masters we do this only if - // aborting; when stopping, we still need to receive the results - if (TestBit(TProof::kIsClient) || abort) - InterruptCurrentMonitor(); - - if (fSlaves->GetSize() == 0) - return; - - // Notify the remote counterpart - TSlave *sl; - TIter next(fSlaves); - while ((sl = (TSlave *)next())) - if (sl->IsValid()) - // Ask slave to progate the stop/abort request - sl->StopProcess(abort, timeout); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Signal to disable related switches - -void TProof::DisableGoAsyn() -{ - Emit("DisableGoAsyn()"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send GOASYNC message to the master. - -void TProof::GoAsynchronous() -{ - if (!IsValid()) return; - - if (GetRemoteProtocol() < 22) { - Info("GoAsynchronous", "functionality not supported by the server - ignoring"); - return; - } - - if (fSync && !IsIdle()) { - TMessage m(kPROOF_GOASYNC); - Broadcast(m); - } else { - Info("GoAsynchronous", "either idle or already in asynchronous mode - ignoring"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Receive the log file of the slave with socket s. - -void TProof::RecvLogFile(TSocket *s, Int_t size) -{ - const Int_t kMAXBUF = 16384; //32768 //16384 //65536; - char buf[kMAXBUF]; - - // If macro saving is enabled prepare macro - if (fSaveLogToMacro && fMacroLog.GetListOfLines()) { - fMacroLog.GetListOfLines()->SetOwner(kTRUE); - fMacroLog.GetListOfLines()->Clear(); - } - - // Append messages to active logging unit - Int_t fdout = -1; - if (!fLogToWindowOnly) { - fdout = (fRedirLog) ? fileno(fLogFileW) : fileno(stdout); - if (fdout < 0) { - Warning("RecvLogFile", "file descriptor for outputs undefined (%d):" - " will not log msgs", fdout); - return; - } - lseek(fdout, (off_t) 0, SEEK_END); - } - - Int_t left, rec, r; - Long_t filesize = 0; - - while (filesize < size) { - left = Int_t(size - filesize); - if (left >= kMAXBUF) - left = kMAXBUF-1; - rec = s->RecvRaw(&buf, left); - filesize = (rec > 0) ? (filesize + rec) : filesize; - if (!fLogToWindowOnly && !fSaveLogToMacro) { - if (rec > 0) { - - char *p = buf; - r = rec; - while (r) { - Int_t w; - - w = write(fdout, p, r); - - if (w < 0) { - SysError("RecvLogFile", "error writing to unit: %d", fdout); - break; - } - r -= w; - p += w; - } - } else if (rec < 0) { - Error("RecvLogFile", "error during receiving log file"); - break; - } - } - if (rec > 0) { - buf[rec] = 0; - EmitVA("LogMessage(const char*,Bool_t)", 2, buf, kFALSE); - // If macro saving is enabled add to TMacro - if (fSaveLogToMacro) fMacroLog.AddLine(buf); - } - } - - // If idle restore logs to main session window - if (fRedirLog && IsIdle() && !TestBit(TProof::kIsMaster)) - fRedirLog = kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Notify locally 'msg' to the appropriate units (file, stdout, window) -/// If defined, 'sfx' is added after 'msg' (typically a line-feed); - -void TProof::NotifyLogMsg(const char *msg, const char *sfx) -{ - // Must have somenthing to notify - Int_t len = 0; - if (!msg || (len = strlen(msg)) <= 0) - return; - - // Get suffix length if any - Int_t lsfx = (sfx) ? strlen(sfx) : 0; - - // Append messages to active logging unit - Int_t fdout = -1; - if (!fLogToWindowOnly) { - fdout = (fRedirLog) ? fileno(fLogFileW) : fileno(stdout); - if (fdout < 0) { - Warning("NotifyLogMsg", "file descriptor for outputs undefined (%d):" - " will not notify msgs", fdout); - return; - } - lseek(fdout, (off_t) 0, SEEK_END); - } - - if (!fLogToWindowOnly) { - // Write to output unit (stdout or a log file) - if (len > 0) { - char *p = (char *)msg; - Int_t r = len; - while (r) { - Int_t w = write(fdout, p, r); - if (w < 0) { - SysError("NotifyLogMsg", "error writing to unit: %d", fdout); - break; - } - r -= w; - p += w; - } - // Add a suffix, if requested - if (lsfx > 0) - if (write(fdout, sfx, lsfx) != lsfx) - SysError("NotifyLogMsg", "error writing to unit: %d", fdout); - } - } - if (len > 0) { - // Publish the message to the separate window (if the latter is missing - // the message will just get lost) - EmitVA("LogMessage(const char*,Bool_t)", 2, msg, kFALSE); - } - - // If idle restore logs to main session window - if (fRedirLog && IsIdle()) - fRedirLog = kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Log a message into the appropriate window by emitting a signal. - -void TProof::LogMessage(const char *msg, Bool_t all) -{ - PDB(kGlobal,1) - Info("LogMessage","Enter ... %s, 'all: %s", msg ? msg : "", - all ? "true" : "false"); - - if (gROOT->IsBatch()) { - PDB(kGlobal,1) Info("LogMessage","GUI not started - use TProof::ShowLog()"); - return; - } - - if (msg) - EmitVA("LogMessage(const char*,Bool_t)", 2, msg, all); - - // Re-position at the beginning of the file, if requested. - // This is used by the dialog when it re-opens the log window to - // provide all the session messages - if (all) - lseek(fileno(fLogFileR), (off_t) 0, SEEK_SET); - - const Int_t kMAXBUF = 32768; - char buf[kMAXBUF]; - Int_t len; - do { - while ((len = read(fileno(fLogFileR), buf, kMAXBUF-1)) < 0 && - TSystem::GetErrno() == EINTR) - TSystem::ResetErrno(); - - if (len < 0) { - Error("LogMessage", "error reading log file"); - break; - } - - if (len > 0) { - buf[len] = 0; - EmitVA("LogMessage(const char*,Bool_t)", 2, buf, kFALSE); - } - - } while (len > 0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send to all active slaves servers the current slave group size -/// and their unique id. Returns number of active slaves. -/// Returns -1 in case of error. - -Int_t TProof::SendGroupView() -{ - if (!IsValid()) return -1; - if (TestBit(TProof::kIsClient)) return 0; - if (!fSendGroupView) return 0; - fSendGroupView = kFALSE; - - TIter next(fActiveSlaves); - TSlave *sl; - - int bad = 0, cnt = 0, size = GetNumberOfActiveSlaves(); - char str[32]; - - while ((sl = (TSlave *)next())) { - snprintf(str, 32, "%d %d", cnt, size); - if (sl->GetSocket()->Send(str, kPROOF_GROUPVIEW) == -1) { - MarkBad(sl, "could not send kPROOF_GROUPVIEW message"); - bad++; - } else - cnt++; - } - - // Send the group view again in case there was a change in the - // group size due to a bad slave - - if (bad) SendGroupView(); - - return GetNumberOfActiveSlaves(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static method to extract the filename (if any) form a CINT command. -/// Returns kTRUE and the filename in 'fn'; returns kFALSE if not found or not -/// appliable. - -Bool_t TProof::GetFileInCmd(const char *cmd, TString &fn) -{ - TString s = cmd; - s = s.Strip(TString::kBoth); - - if (s.Length() > 0 && - (s.BeginsWith(".L") || s.BeginsWith(".x") || s.BeginsWith(".X"))) { - TString file = s(2, s.Length()); - TString acm, arg, io; - fn = gSystem->SplitAclicMode(file, acm, arg, io); - if (!fn.IsNull()) - return kTRUE; - } - - // Not found - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send command to be executed on the PROOF master and/or slaves. -/// If plusMaster is kTRUE then exeucte on slaves and master too. -/// Command can be any legal command line command. Commands like -/// ".x file.C" or ".L file.C" will cause the file file.C to be send -/// to the PROOF cluster. Returns -1 in case of error, >=0 in case of -/// succes. - -Int_t TProof::Exec(const char *cmd, Bool_t plusMaster) -{ - return Exec(cmd, kActive, plusMaster); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send command to be executed on the PROOF master and/or slaves. -/// Command can be any legal command line command. Commands like -/// ".x file.C" or ".L file.C" will cause the file file.C to be send -/// to the PROOF cluster. Returns -1 in case of error, >=0 in case of -/// succes. - -Int_t TProof::Exec(const char *cmd, ESlaves list, Bool_t plusMaster) -{ - if (!IsValid()) return -1; - - TString s = cmd; - s = s.Strip(TString::kBoth); - - if (!s.Length()) return 0; - - // check for macro file and make sure the file is available on all slaves - TString filename; - if (TProof::GetFileInCmd(s.Data(), filename)) { - char *fn = gSystem->Which(TROOT::GetMacroPath(), filename, kReadPermission); - if (fn) { - if (GetNumberOfUniqueSlaves() > 0) { - if (SendFile(fn, kAscii | kForward | kCpBin) < 0) { - Error("Exec", "file %s could not be transfered", fn); - delete [] fn; - return -1; - } - } else { - TString scmd = s(0,3) + fn; - Int_t n = SendCommand(scmd, list); - delete [] fn; - return n; - } - } else { - Error("Exec", "macro %s not found", filename.Data()); - return -1; - } - delete [] fn; - } - - if (plusMaster) { - if (IsLite()) { - gROOT->ProcessLine(cmd); - } else { - DeactivateWorker("*"); - Int_t res = SendCommand(cmd, list); - ActivateWorker("restore"); - if (res < 0) - return res; - } - } - return SendCommand(cmd, list); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send command to be executed on node of ordinal 'ord' (use "0" for master). -/// Command can be any legal command line command. Commands like -/// ".x file.C" or ".L file.C" will cause the file file.C to be send -/// to the PROOF cluster. -/// If logtomacro is TRUE the text result of the action is saved in the fMacroLog -/// TMacro, accessible via TMacro::GetMacroLog(); -/// Returns -1 in case of error, >=0 in case of succes. - -Int_t TProof::Exec(const char *cmd, const char *ord, Bool_t logtomacro) -{ - if (!IsValid()) return -1; - - TString s = cmd; - s = s.Strip(TString::kBoth); - - if (!s.Length()) return 0; - - Int_t res = 0; - if (IsLite()) { - gROOT->ProcessLine(cmd); - } else { - Bool_t oldRedirLog = fRedirLog; - fRedirLog = kTRUE; - // Deactivate all workers - DeactivateWorker("*"); - fRedirLog = kFALSE; - // Reactivate the target ones, if needed - if (strcmp(ord, "master") && strcmp(ord, "0")) ActivateWorker(ord); - // Honour log-to-macro-saving settings - Bool_t oldSaveLog = fSaveLogToMacro; - fSaveLogToMacro = logtomacro; - res = SendCommand(cmd, kActive); - fSaveLogToMacro = oldSaveLog; - fRedirLog = kTRUE; - ActivateWorker("restore"); - fRedirLog = oldRedirLog; - } - // Done - return res; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send command to be executed on the PROOF master and/or slaves. -/// Command can be any legal command line command, however commands -/// like ".x file.C" or ".L file.C" will not cause the file.C to be -/// transfered to the PROOF cluster. In that case use TProof::Exec(). -/// Returns the status send by the remote server as part of the -/// kPROOF_LOGDONE message. Typically this is the return code of the -/// command on the remote side. Returns -1 in case of error. - -Int_t TProof::SendCommand(const char *cmd, ESlaves list) -{ - if (!IsValid()) return -1; - - Broadcast(cmd, kMESS_CINT, list); - Collect(list); - - return fStatus; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get value of environment variable 'env' on node 'ord' - -TString TProof::Getenv(const char *env, const char *ord) -{ - // The command to be executed - TString cmd = TString::Format("gSystem->Getenv(\"%s\")", env); - if (Exec(cmd.Data(), ord, kTRUE) != 0) return TString(""); - // Get the line - TObjString *os = fMacroLog.GetLineWith("const char"); - if (os) { - TString info; - Ssiz_t from = 0; - os->GetString().Tokenize(info, from, "\""); - os->GetString().Tokenize(info, from, "\""); - if (gDebug > 0) Printf("%s: '%s'", env, info.Data()); - return info; - } - return TString(""); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get into 'env' the value of integer RC env variable 'rcenv' on node 'ord' - -Int_t TProof::GetRC(const char *rcenv, Int_t &env, const char *ord) -{ - // The command to be executed - TString cmd = TString::Format("if (gEnv->Lookup(\"%s\")) { gEnv->GetValue(\"%s\",\"\"); }", rcenv, rcenv); - // Exectute the command saving the logs to macro - if (Exec(cmd.Data(), ord, kTRUE) != 0) return -1; - // Get the line - TObjString *os = fMacroLog.GetLineWith("const char"); - Int_t rc = -1; - if (os) { - Ssiz_t fst = os->GetString().First('\"'); - Ssiz_t lst = os->GetString().Last('\"'); - TString info = os->GetString()(fst+1, lst-fst-1); - if (info.IsDigit()) { - env = info.Atoi(); - rc = 0; - if (gDebug > 0) - Printf("%s: %d", rcenv, env); - } - } - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get into 'env' the value of double RC env variable 'rcenv' on node 'ord' - -Int_t TProof::GetRC(const char *rcenv, Double_t &env, const char *ord) -{ - // The command to be executed - TString cmd = TString::Format("if (gEnv->Lookup(\"%s\")) { gEnv->GetValue(\"%s\",\"\"); }", rcenv, rcenv); - // Exectute the command saving the logs to macro - if (Exec(cmd.Data(), ord, kTRUE) != 0) return -1; - // Get the line - TObjString *os = fMacroLog.GetLineWith("const char"); - Int_t rc = -1; - if (os) { - Ssiz_t fst = os->GetString().First('\"'); - Ssiz_t lst = os->GetString().Last('\"'); - TString info = os->GetString()(fst+1, lst-fst-1); - if (info.IsFloat()) { - env = info.Atof(); - rc = 0; - if (gDebug > 0) - Printf("%s: %f", rcenv, env); - } - } - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get into 'env' the value of string RC env variable 'rcenv' on node 'ord' - -Int_t TProof::GetRC(const char *rcenv, TString &env, const char *ord) -{ - // The command to be executed - TString cmd = TString::Format("if (gEnv->Lookup(\"%s\")) { gEnv->GetValue(\"%s\",\"\"); }", rcenv, rcenv); - // Exectute the command saving the logs to macro - if (Exec(cmd.Data(), ord, kTRUE) != 0) return -1; - // Get the line - TObjString *os = fMacroLog.GetLineWith("const char"); - Int_t rc = -1; - if (os) { - Ssiz_t fst = os->GetString().First('\"'); - Ssiz_t lst = os->GetString().Last('\"'); - env = os->GetString()(fst+1, lst-fst-1); - rc = 0; - if (gDebug > 0) - Printf("%s: %s", rcenv, env.Data()); - } - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Transfer the current state of the master to the active slave servers. -/// The current state includes: the current working directory, etc. -/// Returns the number of active slaves. Returns -1 in case of error. - -Int_t TProof::SendCurrentState(TList *list) -{ - if (!IsValid()) return -1; - - // Go to the new directory, reset the interpreter environment and - // tell slave to delete all objects from its new current directory. - Broadcast(gDirectory->GetPath(), kPROOF_RESET, list); - - return GetParallel(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Transfer the current state of the master to the active slave servers. -/// The current state includes: the current working directory, etc. -/// Returns the number of active slaves. Returns -1 in case of error. - -Int_t TProof::SendCurrentState(ESlaves list) -{ - if (!IsValid()) return -1; - - // Go to the new directory, reset the interpreter environment and - // tell slave to delete all objects from its new current directory. - Broadcast(gDirectory->GetPath(), kPROOF_RESET, list); - - return GetParallel(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Transfer the initial (i.e. current) state of the master to all -/// slave servers. Currently the initial state includes: log level. -/// Returns the number of active slaves. Returns -1 in case of error. - -Int_t TProof::SendInitialState() -{ - if (!IsValid()) return -1; - - SetLogLevel(fLogLevel, gProofDebugMask); - - return GetNumberOfActiveSlaves(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check if a file needs to be send to the slave. Use the following -/// algorithm: -/// - check if file appears in file map -/// - if yes, get file's modtime and check against time in map, -/// if modtime not same get md5 and compare against md5 in map, -/// if not same return kTRUE. -/// - if no, get file's md5 and modtime and store in file map, ask -/// slave if file exists with specific md5, if yes return kFALSE, -/// if no return kTRUE. -/// The options 'cpopt' define if to copy things from cache to sandbox and what. -/// To retrieve from the cache the binaries associated with the file TProof::kCpBin -/// must be set in cpopt; the default is copy everything. -/// Returns kTRUE in case file needs to be send, returns kFALSE in case -/// file is already on remote node. - -Bool_t TProof::CheckFile(const char *file, TSlave *slave, Long_t modtime, Int_t cpopt) -{ - Bool_t sendto = kFALSE; - - // create worker based filename - TString sn = slave->GetName(); - sn += ":"; - sn += slave->GetOrdinal(); - sn += ":"; - sn += gSystem->BaseName(file); - - // check if file is in map - FileMap_t::const_iterator it; - if ((it = fFileMap.find(sn)) != fFileMap.end()) { - // file in map - MD5Mod_t md = (*it).second; - if (md.fModtime != modtime) { - TMD5 *md5 = TMD5::FileChecksum(file); - if (md5) { - if ((*md5) != md.fMD5) { - sendto = kTRUE; - md.fMD5 = *md5; - md.fModtime = modtime; - fFileMap[sn] = md; - // When on the master, the master and/or slaves may share - // their file systems and cache. Therefore always make a - // check for the file. If the file already exists with the - // expected md5 the kPROOF_CHECKFILE command will cause the - // file to be copied from cache to slave sandbox. - if (TestBit(TProof::kIsMaster)) { - sendto = kFALSE; - TMessage mess(kPROOF_CHECKFILE); - mess << TString(gSystem->BaseName(file)) << md.fMD5 << cpopt; - slave->GetSocket()->Send(mess); - - fCheckFileStatus = 0; - Collect(slave, fCollectTimeout, kPROOF_CHECKFILE); - sendto = (fCheckFileStatus == 0) ? kTRUE : kFALSE; - } - } - delete md5; - } else { - Error("CheckFile", "could not calculate local MD5 check sum - dont send"); - return kFALSE; - } - } - } else { - // file not in map - TMD5 *md5 = TMD5::FileChecksum(file); - MD5Mod_t md; - if (md5) { - md.fMD5 = *md5; - md.fModtime = modtime; - fFileMap[sn] = md; - delete md5; - } else { - Error("CheckFile", "could not calculate local MD5 check sum - dont send"); - return kFALSE; - } - TMessage mess(kPROOF_CHECKFILE); - mess << TString(gSystem->BaseName(file)) << md.fMD5 << cpopt; - slave->GetSocket()->Send(mess); - - fCheckFileStatus = 0; - Collect(slave, fCollectTimeout, kPROOF_CHECKFILE); - sendto = (fCheckFileStatus == 0) ? kTRUE : kFALSE; - } - - return sendto; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send a file to master or slave servers. Returns number of slaves -/// the file was sent to, maybe 0 in case master and slaves have the same -/// file system image, -1 in case of error. -/// If defined, send to worker 'wrk' only. -/// If defined, the full path of the remote path will be rfile. -/// If rfile = "cache" the file is copied to the remote cache instead of the sandbox -/// (to copy to the cache on a different name use rfile = "cache:newname"). -/// The mask 'opt' is an or of ESendFileOpt: -/// -/// kAscii (0x0) if set true ascii file transfer is used -/// kBinary (0x1) if set true binary file transfer is used -/// kForce (0x2) if not set an attempt is done to find out -/// whether the file really needs to be downloaded -/// (a valid copy may already exist in the cache -/// from a previous run); the bit is set by -/// UploadPackage, since the check is done elsewhere. -/// kForward (0x4) if set, ask server to forward the file to slave -/// or submaster (meaningless for slave servers). -/// kCpBin (0x8) Retrieve from the cache the binaries associated -/// with the file -/// kCp (0x10) Retrieve the files from the cache -/// - -Int_t TProof::SendFile(const char *file, Int_t opt, const char *rfile, TSlave *wrk) -{ - if (!IsValid()) return -1; - - // Use the active slaves list ... - TList *slaves = (rfile && !strcmp(rfile, "cache")) ? fUniqueSlaves : fActiveSlaves; - // ... or the specified slave, if any - if (wrk) { - slaves = new TList(); - slaves->Add(wrk); - } - - if (slaves->GetSize() == 0) return 0; - -#ifndef R__WIN32 - Int_t fd = open(file, O_RDONLY); -#else - Int_t fd = open(file, O_RDONLY | O_BINARY); -#endif - if (fd < 0) { - SysError("SendFile", "cannot open file %s", file); - return -1; - } - - // Get info about the file - Long64_t size = -1; - Long_t id, flags, modtime = 0; - if (gSystem->GetPathInfo(file, &id, &size, &flags, &modtime) == 1) { - Error("SendFile", "cannot stat file %s", file); - close(fd); - return -1; - } - if (size == 0) { - Error("SendFile", "empty file %s", file); - close(fd); - return -1; - } - - // Decode options - Bool_t bin = (opt & kBinary) ? kTRUE : kFALSE; - Bool_t force = (opt & kForce) ? kTRUE : kFALSE; - Bool_t fw = (opt & kForward) ? kTRUE : kFALSE; - - // Copy options - Int_t cpopt = 0; - if ((opt & kCp)) cpopt |= kCp; - if ((opt & kCpBin)) cpopt |= (kCp | kCpBin); - - const Int_t kMAXBUF = 32768; //16384 //65536; - char buf[kMAXBUF]; - Int_t nsl = 0; - - TIter next(slaves); - TSlave *sl; - TString fnam(rfile); - if (fnam == "cache") { - fnam += TString::Format(":%s", gSystem->BaseName(file)); - } else if (fnam.IsNull()) { - fnam = gSystem->BaseName(file); - } - // List on which we will collect the results - fStatus = 0; - while ((sl = (TSlave *)next())) { - if (!sl->IsValid()) - continue; - - Bool_t sendto = force ? kTRUE : CheckFile(file, sl, modtime, cpopt); - // Don't send the kPROOF_SENDFILE command to real slaves when sendto - // is false. Masters might still need to send the file to newly added - // slaves. - PDB(kPackage,2) { - const char *snd = (sl->fSlaveType == TSlave::kSlave && sendto) ? "" : "not"; - Info("SendFile", "%s sending file %s to: %s:%s (%d)", snd, - file, sl->GetName(), sl->GetOrdinal(), sendto); - } - if (sl->fSlaveType == TSlave::kSlave && !sendto) - continue; - // The value of 'size' is used as flag remotely, so we need to - // reset it to 0 if we are not going to send the file - Long64_t siz = sendto ? size : 0; - snprintf(buf, kMAXBUF, "%s %d %lld %d", fnam.Data(), bin, siz, fw); - if (sl->GetSocket()->Send(buf, kPROOF_SENDFILE) == -1) { - MarkBad(sl, "could not send kPROOF_SENDFILE request"); - continue; - } - - if (sendto) { - - lseek(fd, 0, SEEK_SET); - - Int_t len; - do { - while ((len = read(fd, buf, kMAXBUF)) < 0 && TSystem::GetErrno() == EINTR) - TSystem::ResetErrno(); - - if (len < 0) { - SysError("SendFile", "error reading from file %s", file); - Interrupt(kSoftInterrupt, kActive); - close(fd); - return -1; - } - - if (len > 0 && sl->GetSocket()->SendRaw(buf, len) == -1) { - SysError("SendFile", "error writing to slave %s:%s (now offline)", - sl->GetName(), sl->GetOrdinal()); - MarkBad(sl, "sendraw failure"); - sl = 0; - break; - } - - } while (len > 0); - - nsl++; - } - // Wait for the operation to be done - if (sl) - Collect(sl, fCollectTimeout, kPROOF_SENDFILE); - } - - close(fd); - - // Cleanup temporary list, if any - if (slaves != fActiveSlaves && slaves != fUniqueSlaves) - SafeDelete(slaves); - - // We return failure is at least one unique worker failed - return (fStatus != 0) ? -1 : nsl; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Sends an object to master and workers and expect them to send back a -/// message with the output of its TObject::Print(). Returns -1 on error, the -/// number of workers that received the objects on success. - -Int_t TProof::Echo(const TObject *obj) -{ - if (!IsValid() || !obj) return -1; - TMessage mess(kPROOF_ECHO); - mess.WriteObject(obj); - return Broadcast(mess); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Sends a string to master and workers and expect them to echo it back to -/// the client via a message. It is a special case of the generic Echo() -/// that works with TObjects. Returns -1 on error, the number of workers that -/// received the message on success. - -Int_t TProof::Echo(const char *str) -{ - TObjString *os = new TObjString(str); - Int_t rv = Echo(os); - delete os; - return rv; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send object to master or slave servers. Returns number of slaves object -/// was sent to, -1 in case of error. - -Int_t TProof::SendObject(const TObject *obj, ESlaves list) -{ - if (!IsValid() || !obj) return -1; - - TMessage mess(kMESS_OBJECT); - - mess.WriteObject(obj); - return Broadcast(mess, list); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send print command to master server. Returns number of slaves message -/// was sent to. Returns -1 in case of error. - -Int_t TProof::SendPrint(Option_t *option) -{ - if (!IsValid()) return -1; - - Broadcast(option, kPROOF_PRINT, kActive); - return Collect(kActive, fCollectTimeout); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set server logging level. - -void TProof::SetLogLevel(Int_t level, UInt_t mask) -{ - char str[32]; - fLogLevel = level; - gProofDebugLevel = level; - gProofDebugMask = (TProofDebug::EProofDebugMask) mask; - snprintf(str, 32, "%d %u", level, mask); - Broadcast(str, kPROOF_LOGLEVEL, kAll); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Switch ON/OFF the real-time logging facility. When this option is -/// ON, log messages from processing are sent back as they come, instead of -/// being sent back at the end in one go. This may help debugging or monitoring -/// in some cases, but, depending on the amount of log, it may have significant -/// consequencies on the load over the network, so it must be used with care. - -void TProof::SetRealTimeLog(Bool_t on) -{ - if (IsValid()) { - TMessage mess(kPROOF_REALTIMELOG); - mess << on; - Broadcast(mess); - } else { - Warning("SetRealTimeLog","session is invalid - do nothing"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Tell PROOF how many slaves to use in parallel. If random is TRUE a random -/// selection is done (if nodes is less than the available nodes). -/// Returns the number of parallel slaves. Returns -1 in case of error. - -Int_t TProof::SetParallelSilent(Int_t nodes, Bool_t random) -{ - if (!IsValid()) return -1; - - if (TestBit(TProof::kIsMaster)) { - if (!fDynamicStartup) GoParallel(nodes, kFALSE, random); - return SendCurrentState(); - } else { - if (nodes < 0) { - PDB(kGlobal,1) Info("SetParallelSilent", "request all nodes"); - } else { - PDB(kGlobal,1) Info("SetParallelSilent", "request %d node%s", nodes, - nodes == 1 ? "" : "s"); - } - TMessage mess(kPROOF_PARALLEL); - mess << nodes << random; - Broadcast(mess); - Collect(kActive, fCollectTimeout); - Int_t n = GetParallel(); - PDB(kGlobal,1) Info("SetParallelSilent", "got %d node%s", n, n == 1 ? "" : "s"); - return n; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Tell PROOF how many slaves to use in parallel. Returns the number of -/// parallel slaves. Returns -1 in case of error. - -Int_t TProof::SetParallel(Int_t nodes, Bool_t random) -{ - // If delayed startup reset settings, if required - if (fDynamicStartup && nodes < 0) { - if (gSystem->Getenv("PROOF_NWORKERS")) gSystem->Unsetenv("PROOF_NWORKERS"); - } - - Int_t n = SetParallelSilent(nodes, random); - if (TestBit(TProof::kIsClient)) { - if (n < 1) { - Printf("PROOF set to sequential mode"); - } else { - TString subfix = (n == 1) ? "" : "s"; - if (random) - subfix += ", randomly selected"; - Printf("PROOF set to parallel mode (%d worker%s)", n, subfix.Data()); - } - } else if (fDynamicStartup && nodes >= 0) { - if (gSystem->Getenv("PROOF_NWORKERS")) gSystem->Unsetenv("PROOF_NWORKERS"); - gSystem->Setenv("PROOF_NWORKERS", TString::Format("%d", nodes)); - } - return n; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add nWorkersToAdd workers to current list of workers. This function is -/// works on the master only, and only when an analysis is ongoing. A message -/// is sent back to the client when we go "more" parallel. -/// Returns -1 on error, number of total (not added!) workers on success. - -Int_t TProof::GoMoreParallel(Int_t nWorkersToAdd) -{ - if (!IsValid() || !IsMaster() || IsIdle()) { - Error("GoMoreParallel", "can't invoke here -- should not happen!"); - return -1; - } - if (!gProofServ && !IsLite()) { - Error("GoMoreParallel", "no ProofServ available nor Lite -- should not happen!"); - return -1; - } - - TSlave *sl = 0x0; - TIter next( fSlaves ); - Int_t nAddedWorkers = 0; - - while (((nAddedWorkers < nWorkersToAdd) || (nWorkersToAdd == -1)) && - (( sl = dynamic_cast( next() ) ))) { - - // If worker is of an invalid type, break everything: it should not happen! - if ((sl->GetSlaveType() != TSlave::kSlave) && - (sl->GetSlaveType() != TSlave::kMaster)) { - Error("GoMoreParallel", "TSlave is neither a Master nor a Slave: %s:%s", - sl->GetName(), sl->GetOrdinal()); - R__ASSERT(0); - } - - // Skip current worker if it is not a good candidate - if ((!sl->IsValid()) || (fBadSlaves->FindObject(sl)) || - (strcmp("IGNORE", sl->GetImage()) == 0)) { - PDB(kGlobal, 2) - Info("GoMoreParallel", "Worker %s:%s won't be considered", - sl->GetName(), sl->GetOrdinal()); - continue; - } - - // Worker is good but it is already active: skip it - if (fActiveSlaves->FindObject(sl)) { - Info("GoMoreParallel", "Worker %s:%s is already active: skipping", - sl->GetName(), sl->GetOrdinal()); - continue; - } - - // - // From here on: worker is a good candidate - // - - if (sl->GetSlaveType() == TSlave::kSlave) { - sl->SetStatus(TSlave::kActive); - fActiveSlaves->Add(sl); - fInactiveSlaves->Remove(sl); - fActiveMonitor->Add(sl->GetSocket()); - nAddedWorkers++; - PDB(kGlobal, 2) - Info("GoMoreParallel", "Worker %s:%s marked as active!", - sl->GetName(), sl->GetOrdinal()); - } - else { - // Can't add masters dynamically: this should not happen! - Error("GoMoreParallel", "Dynamic addition of master is not supported"); - R__ASSERT(0); - } - - } // end loop over all slaves - - // Get slave status (will set the slaves fWorkDir correctly) - PDB(kGlobal, 3) - Info("GoMoreParallel", "Will invoke AskStatistics() -- implies a Collect()"); - AskStatistics(); - - // Find active slaves with unique image - PDB(kGlobal, 3) - Info("GoMoreParallel", "Will invoke FindUniqueSlaves()"); - FindUniqueSlaves(); - - // Send new group-view to slaves - PDB(kGlobal, 3) - Info("GoMoreParallel", "Will invoke SendGroupView()"); - SendGroupView(); - - PDB(kGlobal, 3) - Info("GoMoreParallel", "Will invoke GetParallel()"); - Int_t nTotalWorkers = GetParallel(); - - // Notify the client that we've got more workers, and print info on - // Master's log as well - TString s; - s.Form("PROOF just went more parallel (%d additional worker%s, %d worker%s total)", - nAddedWorkers, (nAddedWorkers == 1) ? "" : "s", - nTotalWorkers, (nTotalWorkers == 1) ? "" : "s"); - if (gProofServ) gProofServ->SendAsynMessage(s); - Info("GoMoreParallel", "%s", s.Data()); - - return nTotalWorkers; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Go in parallel mode with at most "nodes" slaves. Since the fSlaves -/// list is sorted by slave performace the active list will contain first -/// the most performant nodes. Returns the number of active slaves. -/// If random is TRUE, and nodes is less than the number of available workers, -/// a random selection is done. -/// Returns -1 in case of error. - -Int_t TProof::GoParallel(Int_t nodes, Bool_t attach, Bool_t random) -{ - if (!IsValid()) return -1; - - fActiveSlaves->Clear(); - fActiveMonitor->RemoveAll(); - - // Prepare the list of candidates first. - // Algorithm depends on random option. - TSlave *sl = 0; - TList *wlst = new TList; - TIter nxt(fSlaves); - fInactiveSlaves->Clear(); - while ((sl = (TSlave *)nxt())) { - if (sl->IsValid() && !fBadSlaves->FindObject(sl)) { - if (strcmp("IGNORE", sl->GetImage()) == 0) continue; - if ((sl->GetSlaveType() != TSlave::kSlave) && - (sl->GetSlaveType() != TSlave::kMaster)) { - Error("GoParallel", "TSlave is neither Master nor Slave"); - R__ASSERT(0); - } - // Good candidate - wlst->Add(sl); - // Set it inactive - fInactiveSlaves->Add(sl); - sl->SetStatus(TSlave::kInactive); - } - } - Int_t nwrks = (nodes < 0 || nodes > wlst->GetSize()) ? wlst->GetSize() : nodes; - int cnt = 0; - fEndMaster = TestBit(TProof::kIsMaster) ? kTRUE : kFALSE; - while (cnt < nwrks) { - // Random choice, if requested - if (random) { - Int_t iwrk = (Int_t) (gRandom->Rndm() * wlst->GetSize()); - sl = (TSlave *) wlst->At(iwrk); - } else { - // The first available - sl = (TSlave *) wlst->First(); - } - if (!sl) { - Error("GoParallel", "attaching to candidate!"); - break; - } - // Remove from the list - wlst->Remove(sl); - - Int_t slavenodes = 0; - if (sl->GetSlaveType() == TSlave::kSlave) { - sl->SetStatus(TSlave::kActive); - fActiveSlaves->Add(sl); - fInactiveSlaves->Remove(sl); - fActiveMonitor->Add(sl->GetSocket()); - slavenodes = 1; - } else if (sl->GetSlaveType() == TSlave::kMaster) { - fEndMaster = kFALSE; - TMessage mess(kPROOF_PARALLEL); - if (!attach) { - Int_t nn = (nodes < 0) ? -1 : nodes-cnt; - mess << nn; - } else { - // To get the number of slaves - mess.SetWhat(kPROOF_LOGFILE); - mess << -1 << -1; - } - if (sl->GetSocket()->Send(mess) == -1) { - MarkBad(sl, "could not send kPROOF_PARALLEL or kPROOF_LOGFILE request"); - slavenodes = 0; - } else { - Collect(sl, fCollectTimeout); - if (sl->IsValid()) { - sl->SetStatus(TSlave::kActive); - fActiveSlaves->Add(sl); - fInactiveSlaves->Remove(sl); - fActiveMonitor->Add(sl->GetSocket()); - if (sl->GetParallel() > 0) { - slavenodes = sl->GetParallel(); - } else { - // Sequential mode: the master acts as a worker - slavenodes = 1; - } - } else { - MarkBad(sl, "collect failed after kPROOF_PARALLEL or kPROOF_LOGFILE request"); - slavenodes = 0; - } - } - } - // 'slavenodes' may be different than 1 in multimaster setups - cnt += slavenodes; - } - - // Cleanup list - wlst->SetOwner(0); - SafeDelete(wlst); - - // Get slave status (will set the slaves fWorkDir correctly) - AskStatistics(); - - // Find active slaves with unique image - FindUniqueSlaves(); - - // Send new group-view to slaves - if (!attach) - SendGroupView(); - - Int_t n = GetParallel(); - - if (TestBit(TProof::kIsClient)) { - if (n < 1) - printf("PROOF set to sequential mode\n"); - else - printf("PROOF set to parallel mode (%d worker%s)\n", - n, n == 1 ? "" : "s"); - } - - PDB(kGlobal,1) Info("GoParallel", "got %d node%s", n, n == 1 ? "" : "s"); - return n; -} - -//////////////////////////////////////////////////////////////////////////////// -/// List contents of the data directory in the sandbox. -/// This is the place where files produced by the client queries are kept - -void TProof::ShowData() -{ - if (!IsValid() || !fManager) return; - - // This is run via the manager - fManager->Find("~/data", "-type f", "all"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove files for the data directory. -/// The option 'what' can take the values: -/// kPurge remove all files and directories under '~/data' -/// kUnregistered remove only files not in registered datasets (default) -/// kDataset remove files belonging to dataset 'dsname' -/// User is prompt for confirmation, unless kForceClear is ORed with the option - -void TProof::ClearData(UInt_t what, const char *dsname) -{ - if (!IsValid() || !fManager) return; - - // Check whether we need to prompt - TString prompt, a("Y"); - Bool_t force = (what & kForceClear) ? kTRUE : kFALSE; - Bool_t doask = (!force && IsTty()) ? kTRUE : kFALSE; - - // If all just send the request - if ((what & TProof::kPurge)) { - // Prompt, if requested - if (doask && !Prompt("Do you really want to remove all data files")) return; - if (fManager->Rm("~/data/*", "-rf", "all") < 0) - Warning("ClearData", "problems purging data directory"); - return; - } else if ((what & TProof::kDataset)) { - // We must have got a name - if (!dsname || strlen(dsname) <= 0) { - Error("ClearData", "dataset name mandatory when removing a full dataset"); - return; - } - // Check if the dataset is registered - if (!ExistsDataSet(dsname)) { - Error("ClearData", "dataset '%s' does not exists", dsname); - return; - } - // Get the file content - TFileCollection *fc = GetDataSet(dsname); - if (!fc) { - Error("ClearData", "could not retrieve info about dataset '%s'", dsname); - return; - } - // Prompt, if requested - TString pmpt = TString::Format("Do you really want to remove all data files" - " of dataset '%s'", dsname); - if (doask && !Prompt(pmpt.Data())) return; - - // Loop through the files - Bool_t rmds = kTRUE; - TIter nxf(fc->GetList()); - TFileInfo *fi = 0; - Int_t rfiles = 0, nfiles = fc->GetList()->GetSize(); - while ((fi = (TFileInfo *) nxf())) { - // Fill the host info - TString host, file; - // Take info from the current url - if (!(fi->GetFirstUrl())) { - Error("ClearData", "GetFirstUrl() returns NULL for '%s' - skipping", - fi->GetName()); - continue; - } - TUrl uf(*(fi->GetFirstUrl())); - file = uf.GetFile(); - host = uf.GetHost(); - // Now search for any "file:" url - Int_t nurl = fi->GetNUrls(); - fi->ResetUrl(); - TUrl *up = 0; - while (nurl-- && fi->NextUrl()) { - up = fi->GetCurrentUrl(); - if (!strcmp(up->GetProtocol(), "file")) { - TString opt(up->GetOptions()); - if (opt.BeginsWith("node=")) { - host=opt; - host.ReplaceAll("node=",""); - file = up->GetFile(); - break; - } - } - } - // Issue a remove request now - if (fManager->Rm(file.Data(), "-f", host.Data()) != 0) { - Error("ClearData", "problems removing '%s'", file.Data()); - // Some files not removed: keep the meta info about this dataset - rmds = kFALSE; - } - rfiles++; - ClearDataProgress(rfiles, nfiles); - } - fprintf(stderr, "\n"); - if (rmds) { - // All files were removed successfully: remove also the dataset meta info - RemoveDataSet(dsname); - } - } else if (what & TProof::kUnregistered) { - - // Get the existing files - TString outtmp("ProofClearData_"); - FILE *ftmp = gSystem->TempFileName(outtmp); - if (!ftmp) { - Error("ClearData", "cannot create temp file for logs"); - return; - } - fclose(ftmp); - RedirectHandle_t h; - gSystem->RedirectOutput(outtmp.Data(), "w", &h); - ShowData(); - gSystem->RedirectOutput(0, 0, &h); - // Parse the output file now - std::ifstream in; - in.open(outtmp.Data()); - if (!in.is_open()) { - Error("ClearData", "could not open temp file for logs: %s", outtmp.Data()); - gSystem->Unlink(outtmp); - return; - } - // Go through - Int_t nfiles = 0; - TMap *afmap = new TMap; - TString line, host, file; - Int_t from = 0; - while (in.good()) { - line.ReadLine(in); - if (line.IsNull()) continue; - while (line.EndsWith("\n")) { line.Strip(TString::kTrailing, '\n'); } - from = 0; - host = ""; - if (!line.Tokenize(host, from, "| ")) continue; - file = ""; - if (!line.Tokenize(file, from, "| ")) continue; - if (!host.IsNull() && !file.IsNull()) { - TList *fl = (TList *) afmap->GetValue(host.Data()); - if (!fl) { - fl = new TList(); - fl->SetName(host); - afmap->Add(new TObjString(host), fl); - } - fl->Add(new TObjString(file)); - nfiles++; - PDB(kDataset,2) - Info("ClearData", "added info for: h:%s, f:%s", host.Data(), file.Data()); - } else { - Warning("ClearData", "found incomplete line: '%s'", line.Data()); - } - } - // Close and remove the file - in.close(); - gSystem->Unlink(outtmp); - - // Get registered data files - TString sel = TString::Format("/%s/%s/", GetGroup(), GetUser()); - TMap *fcmap = GetDataSets(sel); - if (!fcmap || (fcmap && fcmap->GetSize() <= 0)) { - PDB(kDataset,1) - Warning("ClearData", "no dataset beloning to '%s'", sel.Data()); - SafeDelete(fcmap); - } - - // Go thorugh and prepare the lists per node - TString opt; - TObjString *os = 0; - if (fcmap) { - TIter nxfc(fcmap); - while ((os = (TObjString *) nxfc())) { - TFileCollection *fc = 0; - if ((fc = (TFileCollection *) fcmap->GetValue(os))) { - TFileInfo *fi = 0; - TIter nxfi(fc->GetList()); - while ((fi = (TFileInfo *) nxfi())) { - // Get special "file:" url - fi->ResetUrl(); - Int_t nurl = fi->GetNUrls(); - TUrl *up = 0; - while (nurl-- && fi->NextUrl()) { - up = fi->GetCurrentUrl(); - if (!strcmp(up->GetProtocol(), "file")) { - opt = up->GetOptions(); - if (opt.BeginsWith("node=")) { - host=opt; - host.ReplaceAll("node=",""); - file = up->GetFile(); - PDB(kDataset,2) - Info("ClearData", "found: host: %s, file: %s", host.Data(), file.Data()); - // Remove this from the full list, if there - TList *fl = (TList *) afmap->GetValue(host.Data()); - if (fl) { - TObjString *fn = (TObjString *) fl->FindObject(file.Data()); - if (fn) { - fl->Remove(fn); - SafeDelete(fn); - nfiles--; - } else { - Warning("ClearData", - "registered file '%s' not found in the full list!", - file.Data()); - } - } - break; - } - } - } - } - } - } - // Clean up the received map - if (fcmap) fcmap->SetOwner(kTRUE); - SafeDelete(fcmap); - } - // List of the files to be removed - Info("ClearData", "%d unregistered files to be removed:", nfiles); - afmap->Print(); - // Prompt, if requested - TString pmpt = TString::Format("Do you really want to remove all %d" - " unregistered data files", nfiles); - if (doask && !Prompt(pmpt.Data())) return; - - // Remove one by one; we may implement a bloc remove in the future - Int_t rfiles = 0; - TIter nxls(afmap); - while ((os = (TObjString *) nxls())) { - TList *fl = 0; - if ((fl = (TList *) afmap->GetValue(os))) { - TIter nxf(fl); - TObjString *fn = 0; - while ((fn = (TObjString *) nxf())) { - // Issue a remove request now - if (fManager->Rm(fn->GetName(), "-f", os->GetName()) != 0) { - Error("ClearData", "problems removing '%s' on host '%s'", - fn->GetName(), os->GetName()); - } - rfiles++; - ClearDataProgress(rfiles, nfiles); - } - } - } - fprintf(stderr, "\n"); - // Final cleanup - afmap->SetOwner(kTRUE); - SafeDelete(afmap); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Prompt the question 'p' requiring an answer y,Y,n,N -/// Return kTRUE is the answer was y or Y, kFALSE in all other cases. - -Bool_t TProof::Prompt(const char *p) -{ - TString pp(p); - if (!pp.Contains("?")) pp += "?"; - if (!pp.Contains("[y/N]")) pp += " [y/N]"; - TString a = Getline(pp.Data()); - if (a != "\n" && a[0] != 'y' && a[0] != 'Y' && a[0] != 'n' && a[0] != 'N') { - Printf("Please answer y, Y, n or N"); - // Unclear answer: assume negative - return kFALSE; - } else if (a == "\n" || a[0] == 'n' || a[0] == 'N') { - // Explicitly Negative answer - return kFALSE; - } - // Explicitly Positive answer - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Progress bar for clear data - -void TProof::ClearDataProgress(Int_t r, Int_t t) -{ - fprintf(stderr, "[TProof::ClearData] Total %5d files\t|", t); - for (Int_t l = 0; l < 20; l++) { - if (r > 0 && t > 0) { - if (l < 20*r/t) - fprintf(stderr, "="); - else if (l == 20*r/t) - fprintf(stderr, ">"); - else if (l > 20*r/t) - fprintf(stderr, "."); - } else - fprintf(stderr, "="); - } - fprintf(stderr, "| %.02f %% \r", 100.0*(t ? (r/t) : 1)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// List contents of file cache. If all is true show all caches also on -/// slaves. If everything is ok all caches are to be the same. - -void TProof::ShowCache(Bool_t all) -{ - if (!IsValid()) return; - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kShowCache) << all; - Broadcast(mess, kUnique); - - if (all) { - TMessage mess2(kPROOF_CACHE); - mess2 << Int_t(kShowSubCache) << all; - Broadcast(mess2, fNonUniqueMasters); - - Collect(kAllUnique, fCollectTimeout); - } else { - Collect(kUnique, fCollectTimeout); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove file from all file caches. If file is 0 or "" or "*", remove all -/// the files - -void TProof::ClearCache(const char *file) -{ - if (!IsValid()) return; - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kClearCache) << TString(file); - Broadcast(mess, kUnique); - - TMessage mess2(kPROOF_CACHE); - mess2 << Int_t(kClearSubCache) << TString(file); - Broadcast(mess2, fNonUniqueMasters); - - Collect(kAllUnique); - - // clear file map so files get send again to remote nodes - fFileMap.clear(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Exec system command 'cmd'. If fdout > -1, append the output to fdout. - -void TProof::SystemCmd(const char *cmd, Int_t fdout) -{ - if (fdout < 0) { - // Exec directly the command - gSystem->Exec(cmd); - } else { - // Exec via a pipe - FILE *fin = gSystem->OpenPipe(cmd, "r"); - if (fin) { - // Now we go - char line[2048]; - while (fgets(line, 2048, fin)) { - Int_t r = strlen(line); - if (r > 0) { - if (write(fdout, line, r) < 0) { - ::Warning("TProof::SystemCmd", - "errno %d writing to file descriptor %d", - TSystem::GetErrno(), fdout); - } - } else { - // Done - break; - } - } - gSystem->ClosePipe(fin); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// List contents of package directory. If all is true show all package -/// directories also on slaves. If everything is ok all package directories -/// should be the same. If redir is kTRUE the result is redirected to the log -/// file (option available for internal actions). - -void TProof::ShowPackages(Bool_t all, Bool_t redirlog) -{ - if (!IsValid()) return; - - Bool_t oldredir = fRedirLog; - if (redirlog) fRedirLog = kTRUE; - - // Active logging unit - FILE *fout = (fRedirLog) ? fLogFileW : stdout; - if (!fout) { - Warning("ShowPackages", "file descriptor for outputs undefined (%p):" - " will not log msgs", fout); - return; - } - lseek(fileno(fout), (off_t) 0, SEEK_END); - - if (TestBit(TProof::kIsClient)) { - fPackMgr->Show(); - } - - // Nothing more to do if we are a Lite-session - if (IsLite()) { - fRedirLog = oldredir; - return; - } - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kShowPackages) << all; - Broadcast(mess, kUnique); - - if (all) { - TMessage mess2(kPROOF_CACHE); - mess2 << Int_t(kShowSubPackages) << all; - Broadcast(mess2, fNonUniqueMasters); - - Collect(kAllUnique, fCollectTimeout); - } else { - Collect(kUnique, fCollectTimeout); - } - // Restore logging option - fRedirLog = oldredir; -} - -//////////////////////////////////////////////////////////////////////////////// -/// List which packages are enabled. If all is true show enabled packages -/// for all active slaves. If everything is ok all active slaves should -/// have the same packages enabled. - -void TProof::ShowEnabledPackages(Bool_t all) -{ - if (!IsValid()) return; - - if (TestBit(TProof::kIsClient)) { - fPackMgr->ShowEnabled(TString::Format("*** Enabled packages on client on %s\n", - gSystem->HostName())); - } - - // Nothing more to do if we are a Lite-session - if (IsLite()) return; - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kShowEnabledPackages) << all; - Broadcast(mess); - Collect(kActive, fCollectTimeout); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove all packages. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::ClearPackages() -{ - if (!IsValid()) return -1; - - if (UnloadPackages() == -1) - return -1; - - if (DisablePackages() == -1) - return -1; - - return fStatus; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove a specific package. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::ClearPackage(const char *package) -{ - if (!IsValid()) return -1; - - if (!package || !package[0]) { - Error("ClearPackage", "need to specify a package name"); - return -1; - } - - // if name, erroneously, is a par pathname strip off .par and path - TString pac = package; - if (pac.EndsWith(".par")) - pac.Remove(pac.Length()-4); - pac = gSystem->BaseName(pac); - - if (UnloadPackage(pac) == -1) - return -1; - - if (DisablePackage(pac) == -1) - return -1; - - return fStatus; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove a specific package. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::DisablePackage(const char *pack) -{ - if (!IsValid()) return -1; - - if (!pack || strlen(pack) <= 0) { - Error("DisablePackage", "need to specify a package name"); - return -1; - } - - // if name, erroneously, is a par pathname strip off .par and path - TString pac = pack; - if (pac.EndsWith(".par")) - pac.Remove(pac.Length()-4); - pac = gSystem->BaseName(pac); - - if (fPackMgr->Remove(pack) < 0) - Warning("DisablePackage", "problem removing locally package '%s'", pack); - - // Nothing more to do if we are a Lite-session - if (IsLite()) return 0; - - Int_t st = -1; - Bool_t done = kFALSE; - if (fManager) { - // Try to do it via XROOTD (new way) - TString path; - path.Form("~/packages/%s", pack); - if (fManager->Rm(path, "-rf", "all") != -1) { - path.Append(".par"); - if (fManager->Rm(path, "-f", "all") != -1) { - done = kTRUE; - st = 0; - } - } - } - if (!done) { - // Try via TProofServ (old way) - TMessage mess(kPROOF_CACHE); - mess << Int_t(kDisablePackage) << pac; - Broadcast(mess, kUnique); - - TMessage mess2(kPROOF_CACHE); - mess2 << Int_t(kDisableSubPackage) << pac; - Broadcast(mess2, fNonUniqueMasters); - - Collect(kAllUnique); - st = fStatus; - } - - // Done - return st; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove all packages. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::DisablePackages() -{ - if (!IsValid()) return -1; - - // remove all packages on client - if (fPackMgr->Remove(nullptr) < 0) - Warning("DisablePackages", "problem removing packages locally"); - - // Nothing more to do if we are a Lite-session - if (IsLite()) return 0; - - Int_t st = -1; - Bool_t done = kFALSE; - if (fManager) { - // Try to do it via XROOTD (new way) - if (fManager->Rm("~/packages/*", "-rf", "all") != -1) { - done = kTRUE; - st = 0; - } - } - if (!done) { - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kDisablePackages); - Broadcast(mess, kUnique); - - TMessage mess2(kPROOF_CACHE); - mess2 << Int_t(kDisableSubPackages); - Broadcast(mess2, fNonUniqueMasters); - - Collect(kAllUnique); - st = fStatus; - } - - // Done - return st; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build specified package. Executes the PROOF-INF/BUILD.sh -/// script if it exists on all unique nodes. If opt is kBuildOnSlavesNoWait -/// then submit build command to slaves, but don't wait -/// for results. If opt is kCollectBuildResults then collect result -/// from slaves. To be used on the master. -/// If opt = kBuildAll (default) then submit and wait for results -/// (to be used on the client). -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::BuildPackage(const char *package, - EBuildPackageOpt opt, Int_t chkveropt, TList *workers) -{ - if (!IsValid()) return -1; - - if (!package || !package[0]) { - Error("BuildPackage", "need to specify a package name"); - return -1; - } - - // if name, erroneously, is a par pathname strip off .par and path - TString pac = package; - if (pac.EndsWith(".par")) - pac.Remove(pac.Length()-4); - pac = gSystem->BaseName(pac); - - Bool_t buildOnClient = kTRUE; - if (opt == kDontBuildOnClient) { - buildOnClient = kFALSE; - opt = kBuildAll; - } - // Prepare the local package - TString pdir; - Int_t st = 0; - - if (opt <= kBuildAll && (!IsLite() || !buildOnClient)) { - if (workers) { - TMessage mess(kPROOF_CACHE); - mess << Int_t(kBuildPackage) << pac << chkveropt; - Broadcast(mess, workers); - - } else { - TMessage mess(kPROOF_CACHE); - mess << Int_t(kBuildPackage) << pac << chkveropt; - Broadcast(mess, kUnique); - - TMessage mess2(kPROOF_CACHE); - mess2 << Int_t(kBuildSubPackage) << pac << chkveropt; - Broadcast(mess2, fNonUniqueMasters); - } - } - - if (opt >= kBuildAll) { - // by first forwarding the build commands to the master and slaves - // and only then building locally we build in parallel - if (buildOnClient) { - st = fPackMgr->Build(pac, chkveropt); - } - - - fStatus = 0; - if (!IsLite() || !buildOnClient) { - - // On the master, workers that fail are deactivated - // Bool_t deactivateOnFailure = (IsMaster()) ? kTRUE : kFALSE; - if (workers) { -// Collect(workers, -1, -1, deactivateOnFailure); - Collect(workers); - } else { - Collect(kAllUnique); - } - } - - if (fStatus < 0 || st < 0) - return -1; - } - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load specified package. Executes the PROOF-INF/SETUP.C script -/// on all active nodes. If notOnClient = true, don't load package -/// on the client. The default is to load the package also on the client. -/// The argument 'loadopts' specify a list of objects to be passed to the SETUP. -/// The objects in the list must be streamable; the SETUP macro will be executed -/// like this: SETUP.C(loadopts). -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::LoadPackage(const char *package, Bool_t notOnClient, - TList *loadopts, TList *workers) -{ - if (!IsValid()) return -1; - - if (!package || !package[0]) { - Error("LoadPackage", "need to specify a package name"); - return -1; - } - - // if name, erroneously, is a par pathname strip off .par and path - TString pac = package; - if (pac.EndsWith(".par")) - pac.Remove(pac.Length()-4); - pac = gSystem->BaseName(pac); - - if (!notOnClient && TestBit(TProof::kIsClient)) - if (fPackMgr->Load(package, loadopts) == -1) return -1; - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kLoadPackage) << pac; - if (loadopts) mess << loadopts; - - // On the master, workers that fail are deactivated - Bool_t deactivateOnFailure = (IsMaster()) ? kTRUE : kFALSE; - - Bool_t doCollect = (fDynamicStartup && !IsIdle()) ? kFALSE : kTRUE; - - if (workers) { - PDB(kPackage, 3) - Info("LoadPackage", "Sending load message to selected workers only"); - Broadcast(mess, workers); - if (doCollect) Collect(workers, -1, -1, deactivateOnFailure); - } else { - Broadcast(mess); - Collect(kActive, -1, -1, deactivateOnFailure); - } - - return fStatus; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Unload specified package. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::UnloadPackage(const char *package) -{ - if (!IsValid()) return -1; - - if (!package || !package[0]) { - Error("UnloadPackage", "need to specify a package name"); - return -1; - } - - // if name, erroneously, is a par pathname strip off .par and path - TString pac = package; - if (pac.EndsWith(".par")) - pac.Remove(pac.Length()-4); - pac = gSystem->BaseName(pac); - - if (fPackMgr->Unload(package) < 0) - Warning("UnloadPackage", "unable to remove symlink to %s", package); - - // Nothing more to do if we are a Lite-session - if (IsLite()) return 0; - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kUnloadPackage) << pac; - Broadcast(mess); - Collect(); - - return fStatus; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Unload all packages. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::UnloadPackages() -{ - if (!IsValid()) return -1; - - if (TestBit(TProof::kIsClient)) { - if (fPackMgr->Unload(0) < 0) return -1; - } - - // Nothing more to do if we are a Lite-session - if (IsLite()) return 0; - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kUnloadPackages); - Broadcast(mess); - Collect(); - - return fStatus; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable specified package. Executes the PROOF-INF/BUILD.sh -/// script if it exists followed by the PROOF-INF/SETUP.C script. -/// In case notOnClient = true, don't enable the package on the client. -/// The default is to enable packages also on the client. -/// If specified, enables packages only on the specified workers. -/// Returns 0 in case of success and -1 in case of error. -/// Provided for backward compatibility. - -Int_t TProof::EnablePackage(const char *package, Bool_t notOnClient, - TList *workers) -{ - return EnablePackage(package, (TList *)0, notOnClient, workers); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable specified package. Executes the PROOF-INF/BUILD.sh -/// script if it exists followed by the PROOF-INF/SETUP.C script. -/// In case notOnClient = true, don't enable the package on the client. -/// The default is to enable packages also on the client. -/// It is is possible to specify options for the loading step via 'loadopts'; -/// the string will be passed passed as argument to SETUP. -/// Special option 'chkv=``' (or 'checkversion=``') can be used to control -/// plugin version checking during building: possible choices are: -/// off no check; failure may occur at loading -/// on check ROOT version [default] -/// svn check ROOT version and Git commit SHA1. -/// (Use ';', ' ' or '|' to separate 'chkv=``' from the rest.) -/// If specified, enables packages only on the specified workers. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::EnablePackage(const char *package, const char *loadopts, - Bool_t notOnClient, TList *workers) -{ - TList *optls = 0; - if (loadopts && strlen(loadopts)) { - if (fProtocol > 28) { - TObjString *os = new TObjString(loadopts); - // Filter out 'checkversion=off|on|svn' or 'chkv=...' - os->String().ReplaceAll("checkversion=", "chkv="); - Ssiz_t fcv = kNPOS, lcv = kNPOS; - if ((fcv = os->String().Index("chkv=")) != kNPOS) { - TRegexp re("[; |]"); - if ((lcv = os->String().Index(re, fcv)) == kNPOS) { - lcv = os->String().Length(); - } - TString ocv = os->String()(fcv, lcv - fcv); - Int_t cvopt = -1; - if (ocv.EndsWith("=off") || ocv.EndsWith("=0")) - cvopt = (Int_t) TPackMgr::kDontCheck; - else if (ocv.EndsWith("=on") || ocv.EndsWith("=1")) - cvopt = (Int_t) TPackMgr::kCheckROOT; - else - Warning("EnablePackage", "'checkversion' option unknown from argument: '%s' - ignored", ocv.Data()); - if (cvopt > -1) { - if (gDebug > 0) - Info("EnablePackage", "setting check version option from argument: %d", cvopt); - optls = new TList; - optls->Add(new TParameter("PROOF_Package_CheckVersion", (Int_t) cvopt)); - // Remove the special option from; we leave a separator if there were two (one before and one after) - if (lcv != kNPOS && fcv == 0) ocv += os->String()[lcv]; - if (fcv > 0 && os->String().Index(re, fcv - 1) == fcv - 1) os->String().Remove(fcv - 1, 1); - os->String().ReplaceAll(ocv.Data(), ""); - } - } - if (!os->String().IsNull()) { - if (!optls) optls = new TList; - optls->Add(new TObjString(os->String().Data())); - } - if (optls) optls->SetOwner(kTRUE); - } else { - // Notify - Warning("EnablePackage", "remote server does not support options: ignoring the option string"); - } - } - // Run - Int_t rc = EnablePackage(package, optls, notOnClient, workers); - // Clean up - SafeDelete(optls); - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable specified package. Executes the PROOF-INF/BUILD.sh -/// script if it exists followed by the PROOF-INF/SETUP.C script. -/// In case notOnClient = true, don't enable the package on the client. -/// The default is to enable packages also on the client. -/// It is is possible to specify a list of objects to be passed to the SETUP -/// functions via 'loadopts'; the objects must be streamable. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::EnablePackage(const char *package, TList *loadopts, - Bool_t notOnClient, TList *workers) -{ - if (!IsValid()) return -1; - - if (!package || !package[0]) { - Error("EnablePackage", "need to specify a package name"); - return -1; - } - - // if name, erroneously, is a par pathname strip off .par and path - TString pac = package; - if (pac.EndsWith(".par")) - pac.Remove(pac.Length()-4); - pac = gSystem->BaseName(pac); - - EBuildPackageOpt opt = kBuildAll; - if (notOnClient) - opt = kDontBuildOnClient; - - // Get check version option; user settings have priority - Int_t chkveropt = TPackMgr::kCheckROOT; - TString ocv = gEnv->GetValue("Proof.Package.CheckVersion", ""); - if (!ocv.IsNull()) { - if (ocv == "off" || ocv == "0") - chkveropt = (Int_t) TPackMgr::kDontCheck; - else if (ocv == "on" || ocv == "1") - chkveropt = (Int_t) TPackMgr::kCheckROOT; - else - Warning("EnablePackage", "'checkversion' option unknown from rootrc: '%s' - ignored", ocv.Data()); - } - if (loadopts) { - TParameter *pcv = (TParameter *) loadopts->FindObject("PROOF_Package_CheckVersion"); - if (pcv) { - chkveropt = pcv->GetVal(); - loadopts->Remove(pcv); - delete pcv; - } - } - if (gDebug > 0) - Info("EnablePackage", "using check version option: %d", chkveropt); - - if (BuildPackage(pac, opt, chkveropt, workers) == -1) - return -1; - - TList *optls = (loadopts && loadopts->GetSize() > 0) ? loadopts : 0; - if (optls && fProtocol <= 28) { - Warning("EnablePackage", "remote server does not support options: ignoring the option list"); - optls = 0; - } - - if (LoadPackage(pac, notOnClient, optls, workers) == -1) - return -1; - - // Record the information for later usage (simulation of dynamic start on PROOF-Lite) - if (!fEnabledPackagesOnCluster) { - fEnabledPackagesOnCluster = new TList; - fEnabledPackagesOnCluster->SetOwner(); - } - if (!fEnabledPackagesOnCluster->FindObject(pac)) { - TPair *pck = (optls && optls->GetSize() > 0) ? new TPair(new TObjString(pac), optls->Clone()) - : new TPair(new TObjString(pac), 0); - fEnabledPackagesOnCluster->Add(pck); - } - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Download a PROOF archive (PAR file) from the master package repository. -/// The PAR file is downloaded in the current directory or in the directory -/// specified by 'dstdir'. If a package with the same name already exists -/// at destination, a check on the MD5 sum is done and the user warned or -/// prompted for action, depending is the file is equal or different. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::DownloadPackage(const char *pack, const char *dstdir) -{ - if (!fManager || !(fManager->IsValid())) { - Error("DownloadPackage", "the manager is undefined!"); - return -1; - } - - // Create the default source and destination paths - TString parname(gSystem->BaseName(pack)), src, dst; - if (!parname.EndsWith(".par")) parname += ".par"; - src.Form("packages/%s", parname.Data()); - if (!dstdir || strlen(dstdir) <= 0) { - dst.Form("./%s", parname.Data()); - } else { - // Check the destination directory - FileStat_t st; - if (gSystem->GetPathInfo(dstdir, st) != 0) { - // Directory does not exit: create it - if (gSystem->mkdir(dstdir, kTRUE) != 0) { - Error("DownloadPackage", - "could not create the destination directory '%s' (errno: %d)", - dstdir, TSystem::GetErrno()); - return -1; - } - } else if (!R_ISDIR(st.fMode) && !R_ISLNK(st.fMode)) { - Error("DownloadPackage", - "destination path '%s' exist but is not a directory!", dstdir); - return -1; - } - dst.Form("%s/%s", dstdir, parname.Data()); - } - - // Make sure the source file exists - FileStat_t stsrc; - RedirectHandle_t rh; - if (gSystem->RedirectOutput(fLogFileName, "a", &rh) != 0) - Warning("DownloadPackage", "problems redirecting output to '%s'", fLogFileName.Data()); - Int_t rc = fManager->Stat(src, stsrc); - if (gSystem->RedirectOutput(0, 0, &rh) != 0) - Warning("DownloadPackage", "problems restoring output"); - if (rc != 0) { - // Check if there is another possible source - ShowPackages(kFALSE, kTRUE); - TMacro *mp = GetLastLog(); - if (mp) { - // Look for global directories - Bool_t isGlobal = kFALSE; - TIter nxl(mp->GetListOfLines()); - TObjString *os = 0; - TString globaldir; - while ((os = (TObjString *) nxl())) { - TString s(os->GetName()); - if (s.Contains("*** Global Package cache")) { - // Get the directory - s.Remove(0, s.Last(':') + 1); - s.Remove(s.Last(' ')); - globaldir = s; - isGlobal = kTRUE; - } else if (s.Contains("*** Package cache")) { - isGlobal = kFALSE; - globaldir = ""; - } - // Check for the package - if (isGlobal && s.Contains(parname)) { - src.Form("%s/%s", globaldir.Data(), parname.Data()); - break; - } - } - // Cleanup - delete mp; - } - } - - // Do it via the manager - if (fManager->GetFile(src, dst, "silent") != 0) { - Error("DownloadPackage", "problems downloading '%s' (src:%s, dst:%s)", - pack, src.Data(), dst.Data()); - return -1; - } else { - Info("DownloadPackage", "'%s' cross-checked against master repository (local path: %s)", - pack, dst.Data()); - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Upload a PROOF archive (PAR file). A PAR file is a compressed -/// tar file with one special additional directory, PROOF-INF -/// (blatantly copied from Java's jar format). It must have the extension -/// .par. A PAR file can be directly a binary or a source with a build -/// procedure. In the PROOF-INF directory there can be a build script: -/// BUILD.sh to be called to build the package, in case of a binary PAR -/// file don't specify a build script or make it a no-op. Then there is -/// SETUP.C which sets the right environment variables to use the package, -/// like LD_LIBRARY_PATH, etc. -/// The 'opt' allows to specify whether the .PAR should be just unpacked -/// in the existing dir (opt = kUntar, default) or a remove of the existing -/// directory should be executed (opt = kRemoveOld), so triggering a full -/// re-build. The option if effective only for PROOF protocol > 8 . -/// The lab 'dirlab' (e.g. 'G0') indicates that the package is to uploaded to -/// an alternative global directory for global usage. This may require special -/// privileges. -/// If download is kTRUE and the package is not found locally, then it is downloaded -/// from the master repository. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::UploadPackage(const char *pack, EUploadPackageOpt opt, - TList *workers) -{ - if (!IsValid()) return -1; - - // Remote PAR ? - TFile::EFileType ft = TFile::GetType(pack); - Bool_t remotepar = (ft == TFile::kWeb || ft == TFile::kNet) ? kTRUE : kFALSE; - - TString par(pack), base, name; - if (par.EndsWith(".par")) { - base = gSystem->BaseName(par); - name = base(0, base.Length() - strlen(".par")); - } else { - name = gSystem->BaseName(par); - base.Form("%s.par", name.Data()); - par += ".par"; - } - - // Default location is the local working dir; then the package dir - gSystem->ExpandPathName(par); - if (gSystem->AccessPathName(par, kReadPermission)) { - Int_t xrc = -1; - if (!remotepar) xrc = TPackMgr::FindParPath(fPackMgr, name, par); - if (xrc == 0) { - // Package is in the global dirs - if (gDebug > 0) - Info("UploadPackage", "global package found (%s): no upload needed", - par.Data()); - return 0; - } else if (xrc < 0) { - Error("UploadPackage", "PAR file '%s' not found", par.Data()); - return -1; - } - } - - // Strategy: - // On the client: - // get md5 of package and check if it is different - // from the one stored in the local package directory. If it is lock - // the package directory and copy the package, unlock the directory. - // On the masters: - // get md5 of package and check if it is different from the - // one stored on the remote node. If it is different lock the remote - // package directory and use TFTP or SendFile to ftp the package to the - // remote node, unlock the directory. - - - if (TestBit(TProof::kIsClient)) { - Bool_t rmold = (opt == TProof::kRemoveOld) ? kTRUE : kFALSE; - if (fPackMgr->Install(par, rmold) < 0) { - Error("UploadPackage", "installing '%s' failed", gSystem->BaseName(par)); - return -1; - } - } - - // Nothing more to do if we are a Lite-session - if (IsLite()) return 0; - - TMD5 *md5 = fPackMgr->ReadMD5(name); - - TString smsg; - if (remotepar && GetRemoteProtocol() > 36) { - smsg.Form("+%s", par.Data()); - } else { - smsg.Form("+%s", base.Data()); - } - - TMessage mess(kPROOF_CHECKFILE); - mess << smsg << (*md5); - TMessage mess2(kPROOF_CHECKFILE); - smsg.Replace(0, 1, "-"); - mess2 << smsg << (*md5); - TMessage mess3(kPROOF_CHECKFILE); - smsg.Replace(0, 1, "="); - mess3 << smsg << (*md5); - - delete md5; - - if (fProtocol > 8) { - // Send also the option - mess << (UInt_t) opt; - mess2 << (UInt_t) opt; - mess3 << (UInt_t) opt; - } - - // Loop over all slaves with unique fs image, or to a selected - // list of workers, if specified - if (!workers) - workers = fUniqueSlaves; - TIter next(workers); - TSlave *sl = 0; - while ((sl = (TSlave *) next())) { - if (!sl->IsValid()) - continue; - - sl->GetSocket()->Send(mess); - - fCheckFileStatus = 0; - Collect(sl, fCollectTimeout, kPROOF_CHECKFILE); - if (fCheckFileStatus == 0) { - - if (fProtocol > 5) { - // remote directory is locked, upload file over the open channel - smsg.Form("%s/%s/%s", sl->GetProofWorkDir(), kPROOF_PackDir, base.Data()); - if (SendFile(par, (kBinary | kForce | kCpBin | kForward), smsg.Data(), sl) < 0) { - Error("UploadPackage", "%s: problems uploading file %s", - sl->GetOrdinal(), par.Data()); - return -1; - } - } else { - // old servers receive it via TFTP - TFTP ftp(TString("root://")+sl->GetName(), 1); - if (!ftp.IsZombie()) { - smsg.Form("%s/%s", sl->GetProofWorkDir(), kPROOF_PackDir); - ftp.cd(smsg.Data()); - ftp.put(par, base.Data()); - } - } - - // install package and unlock dir - sl->GetSocket()->Send(mess2); - fCheckFileStatus = 0; - Collect(sl, fCollectTimeout, kPROOF_CHECKFILE); - if (fCheckFileStatus == 0) { - Error("UploadPackage", "%s: unpacking of package %s failed", - sl->GetOrdinal(), base.Data()); - return -1; - } - } - } - - // loop over all other master nodes - TIter nextmaster(fNonUniqueMasters); - TSlave *ma; - while ((ma = (TSlave *) nextmaster())) { - if (!ma->IsValid()) - continue; - - ma->GetSocket()->Send(mess3); - - fCheckFileStatus = 0; - Collect(ma, fCollectTimeout, kPROOF_CHECKFILE); - if (fCheckFileStatus == 0) { - // error -> package should have been found - Error("UploadPackage", "package %s did not exist on submaster %s", - base.Data(), ma->GetOrdinal()); - return -1; - } - } - - return 0; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that the directory path contained by macro is in the macro path - -void TProof::AssertMacroPath(const char *macro) -{ - static TString macrop(gROOT->GetMacroPath()); - if (macro && strlen(macro) > 0) { - TString dirn = gSystem->GetDirName(macro); - if (!macrop.Contains(dirn)) { - macrop += TString::Format("%s:", dirn.Data()); - gROOT->SetMacroPath(macrop); - } - } -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Load the specified macro on master, workers and, if notOnClient is -/// kFALSE, on the client. The macro file is uploaded if new or updated. -/// Additional files to be uploaded (or updated, if needed) can be specified -/// after a comma, e.g. "mymacro.C+,thisheader.h,thatheader.h". -/// If existing in the same directory, a header basename(macro).h or .hh, is also -/// uploaded. -/// The default is to load the macro also on the client; notOnClient can be used -/// to avoid loading on the client. -/// On masters, if uniqueWorkers is kTRUE, the macro is loaded on unique workers -/// only, and collection is not done; if uniqueWorkers is kFALSE, collection -/// from the previous request is done, and broadcasting + collection from the -/// other workers is done. -/// The wrks arg can be used on the master to limit the set of workers. -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProof::Load(const char *macro, Bool_t notOnClient, Bool_t uniqueWorkers, - TList *wrks) -{ - if (!IsValid()) return -1; - - if (!macro || !macro[0]) { - Error("Load", "need to specify a macro name"); - return -1; - } - - // Make sure the path is in the macro path - TProof::AssertMacroPath(macro); - - if (TestBit(TProof::kIsClient) && !wrks) { - - // Extract the file implementation name first - TString addsname, implname = macro; - Ssiz_t icom = implname.Index(","); - if (icom != kNPOS) { - addsname = implname(icom + 1, implname.Length()); - implname.Remove(icom); - } - TString basemacro = gSystem->BaseName(implname), mainmacro(implname); - TString bmsg(basemacro), acmode, args, io; - implname = gSystem->SplitAclicMode(implname, acmode, args, io); - - // Macro names must have a standard format - Int_t dot = implname.Last('.'); - if (dot == kNPOS) { - Info("Load", "macro '%s' does not contain a '.': do nothing", macro); - return -1; - } - - // Is there any associated header file - Bool_t hasHeader = kTRUE; - TString headname = implname; - headname.Remove(dot); - headname += ".h"; - if (gSystem->AccessPathName(headname, kReadPermission)) { - TString h = headname; - headname.Remove(dot); - headname += ".hh"; - if (gSystem->AccessPathName(headname, kReadPermission)) { - hasHeader = kFALSE; - if (gDebug > 0) - Info("Load", "no associated header file found: tried: %s %s", - h.Data(), headname.Data()); - } - } - - // Is there any additional file ? - TString addincs; - TList addfiles; - if (!addsname.IsNull()) { - TString fn; - Int_t from = 0; - while (addsname.Tokenize(fn, from, ",")) { - if (gSystem->AccessPathName(fn, kReadPermission)) { - Error("Load", "additional file '%s' not found", fn.Data()); - return -1; - } - // Create the additional include statement - if (!notOnClient) { - TString dirn = gSystem->GetDirName(fn); - if (addincs.IsNull()) { - addincs.Form("-I%s", dirn.Data()); - } else if (!addincs.Contains(dirn)) { - addincs += TString::Format(" -I%s", dirn.Data()); - } - } - // Remember these files ... - addfiles.Add(new TObjString(fn)); - } - } - - // Send files now; the md5 check is run here; see SendFile for more - // details. - if (SendFile(implname, kAscii | kForward , "cache") == -1) { - Error("Load", "problems sending implementation file %s", implname.Data()); - return -1; - } - if (hasHeader) - if (SendFile(headname, kAscii | kForward , "cache") == -1) { - Error("Load", "problems sending header file %s", headname.Data()); - return -1; - } - // Additional files - if (addfiles.GetSize() > 0) { - TIter nxfn(&addfiles); - TObjString *os = 0; - while ((os = (TObjString *) nxfn())) { - // These files need to be available everywhere, cache and sandbox - if (SendFile(os->GetName(), kAscii | kForward, "cache") == -1) { - Error("Load", "problems sending additional file %s", os->GetName()); - return -1; - } - // Add the base names to the message broadcasted - bmsg += TString::Format(",%s", gSystem->BaseName(os->GetName())); - } - addfiles.SetOwner(kTRUE); - } - - // The files are now on the workers: now we send the loading request - TMessage mess(kPROOF_CACHE); - if (GetRemoteProtocol() < 34) { - mess << Int_t(kLoadMacro) << basemacro; - // This may be needed - AddIncludePath("../../cache"); - } else { - mess << Int_t(kLoadMacro) << bmsg; - } - Broadcast(mess, kActive); - - // Load locally, if required - if (!notOnClient) { - // Mofify the include path - TString oldincs = gSystem->GetIncludePath(); - if (!addincs.IsNull()) gSystem->AddIncludePath(addincs); - - // By first forwarding the load command to the master and workers - // and only then loading locally we load/build in parallel - gROOT->ProcessLine(TString::Format(".L %s", mainmacro.Data())); - - // Restore include path - if (!addincs.IsNull()) gSystem->SetIncludePath(oldincs); - - // Update the macro path - TString mp(TROOT::GetMacroPath()); - TString np = gSystem->GetDirName(macro); - if (!np.IsNull()) { - np += ":"; - if (!mp.BeginsWith(np) && !mp.Contains(":"+np)) { - Int_t ip = (mp.BeginsWith(".:")) ? 2 : 0; - mp.Insert(ip, np); - TROOT::SetMacroPath(mp); - if (gDebug > 0) - Info("Load", "macro path set to '%s'", TROOT::GetMacroPath()); - } - } - } - - // Wait for master and workers to be done - Collect(kActive); - - if (IsLite()) { - PDB(kGlobal, 1) Info("Load", "adding loaded macro: %s", macro); - if (!fLoadedMacros) { - fLoadedMacros = new TList(); - fLoadedMacros->SetOwner(); - } - // if wrks is specified the macro should already be loaded on the master. - fLoadedMacros->Add(new TObjString(macro)); - } - - } else { - // On master - - // The files are now on the workers: now we send the loading request first - // to the unique workers, so that the eventual compilation occurs only once. - TString basemacro = gSystem->BaseName(macro); - TMessage mess(kPROOF_CACHE); - - if (uniqueWorkers) { - mess << Int_t(kLoadMacro) << basemacro; - if (wrks) { - Broadcast(mess, wrks); - Collect(wrks); - } else { - Broadcast(mess, kUnique); - } - } else { - // Wait for the result of the previous sending - Collect(kUnique); - - // We then send a tuned loading request to the other workers - TList others; - TSlave *wrk = 0; - TIter nxw(fActiveSlaves); - while ((wrk = (TSlave *)nxw())) { - if (!fUniqueSlaves->FindObject(wrk)) { - others.Add(wrk); - } - } - - // Do not force compilation, if it was requested - Int_t ld = basemacro.Last('.'); - if (ld != kNPOS) { - Int_t lpp = basemacro.Index("++", ld); - if (lpp != kNPOS) basemacro.Replace(lpp, 2, "+"); - } - mess << Int_t(kLoadMacro) << basemacro; - Broadcast(mess, &others); - Collect(&others); - } - - PDB(kGlobal, 1) Info("Load", "adding loaded macro: %s", macro); - if (!fLoadedMacros) { - fLoadedMacros = new TList(); - fLoadedMacros->SetOwner(); - } - // if wrks is specified the macro should already be loaded on the master. - if (!wrks) - fLoadedMacros->Add(new TObjString(macro)); - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add 'libpath' to the lib path search. -/// Multiple paths can be specified at once separating them with a comma or -/// a blank. -/// Return 0 on success, -1 otherwise - -Int_t TProof::AddDynamicPath(const char *libpath, Bool_t onClient, TList *wrks, - Bool_t doCollect) -{ - if ((!libpath || !libpath[0])) { - if (gDebug > 0) - Info("AddDynamicPath", "list is empty - nothing to do"); - return 0; - } - - // Do it also on clients, if required - if (onClient) - HandleLibIncPath("lib", kTRUE, libpath); - - TMessage m(kPROOF_LIB_INC_PATH); - m << TString("lib") << (Bool_t)kTRUE; - - // Add paths - if (libpath && strlen(libpath)) { - m << TString(libpath); - } else { - m << TString("-"); - } - - // Tell the server to send back or not - m << (Int_t)doCollect; - - // Forward the request - if (wrks) { - Broadcast(m, wrks); - if (doCollect) - Collect(wrks, fCollectTimeout); - } else { - Broadcast(m); - Collect(kActive, fCollectTimeout); - } - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add 'incpath' to the inc path search. -/// Multiple paths can be specified at once separating them with a comma or -/// a blank. -/// Return 0 on success, -1 otherwise - -Int_t TProof::AddIncludePath(const char *incpath, Bool_t onClient, TList *wrks, - Bool_t doCollect) -{ - if ((!incpath || !incpath[0])) { - if (gDebug > 0) - Info("AddIncludePath", "list is empty - nothing to do"); - return 0; - } - - // Do it also on clients, if required - if (onClient) - HandleLibIncPath("inc", kTRUE, incpath); - - TMessage m(kPROOF_LIB_INC_PATH); - m << TString("inc") << (Bool_t)kTRUE; - - // Add paths - if (incpath && strlen(incpath)) { - m << TString(incpath); - } else { - m << TString("-"); - } - - // Tell the server to send back or not - m << (Int_t)doCollect; - - // Forward the request - if (wrks) { - Broadcast(m, wrks); - if (doCollect) - Collect(wrks, fCollectTimeout); - } else { - Broadcast(m); - Collect(kActive, fCollectTimeout); - } - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove 'libpath' from the lib path search. -/// Multiple paths can be specified at once separating them with a comma or -/// a blank. -/// Return 0 on success, -1 otherwise - -Int_t TProof::RemoveDynamicPath(const char *libpath, Bool_t onClient) -{ - if ((!libpath || !libpath[0])) { - if (gDebug > 0) - Info("RemoveDynamicPath", "list is empty - nothing to do"); - return 0; - } - - // Do it also on clients, if required - if (onClient) - HandleLibIncPath("lib", kFALSE, libpath); - - TMessage m(kPROOF_LIB_INC_PATH); - m << TString("lib") <<(Bool_t)kFALSE; - - // Add paths - if (libpath && strlen(libpath)) - m << TString(libpath); - else - m << TString("-"); - - // Forward the request - Broadcast(m); - Collect(kActive, fCollectTimeout); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove 'incpath' from the inc path search. -/// Multiple paths can be specified at once separating them with a comma or -/// a blank. -/// Return 0 on success, -1 otherwise - -Int_t TProof::RemoveIncludePath(const char *incpath, Bool_t onClient) -{ - if ((!incpath || !incpath[0])) { - if (gDebug > 0) - Info("RemoveIncludePath", "list is empty - nothing to do"); - return 0; - } - - // Do it also on clients, if required - if (onClient) - HandleLibIncPath("in", kFALSE, incpath); - - TMessage m(kPROOF_LIB_INC_PATH); - m << TString("inc") << (Bool_t)kFALSE; - - // Add paths - if (incpath && strlen(incpath)) - m << TString(incpath); - else - m << TString("-"); - - // Forward the request - Broadcast(m); - Collect(kActive, fCollectTimeout); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle lib, inc search paths modification request - -void TProof::HandleLibIncPath(const char *what, Bool_t add, const char *dirs) -{ - TString type(what); - TString path(dirs); - - // Check type of action - if ((type != "lib") && (type != "inc")) { - Error("HandleLibIncPath","unknown action type: %s - protocol error?", type.Data()); - return; - } - - // Separators can be either commas or blanks - path.ReplaceAll(","," "); - - // Decompose lists - TObjArray *op = 0; - if (path.Length() > 0 && path != "-") { - if (!(op = path.Tokenize(" "))) { - Warning("HandleLibIncPath","decomposing path %s", path.Data()); - return; - } - } - - if (add) { - - if (type == "lib") { - - // Add libs - TIter nxl(op, kIterBackward); - TObjString *lib = 0; - while ((lib = (TObjString *) nxl())) { - // Expand path - TString xlib = lib->GetName(); - gSystem->ExpandPathName(xlib); - // Add to the dynamic lib search path if it exists and can be read - if (!gSystem->AccessPathName(xlib, kReadPermission)) { - TString newlibpath = gSystem->GetDynamicPath(); - // In the first position after the working dir - Int_t pos = 0; - if (newlibpath.BeginsWith(".:")) - pos = 2; - if (newlibpath.Index(xlib) == kNPOS) { - newlibpath.Insert(pos,TString::Format("%s:", xlib.Data())); - gSystem->SetDynamicPath(newlibpath); - } - } else { - if (gDebug > 0) - Info("HandleLibIncPath", - "libpath %s does not exist or cannot be read - not added", xlib.Data()); - } - } - - } else { - - // Add incs - TIter nxi(op); - TObjString *inc = 0; - while ((inc = (TObjString *) nxi())) { - // Expand path - TString xinc = inc->GetName(); - gSystem->ExpandPathName(xinc); - // Add to the dynamic lib search path if it exists and can be read - if (!gSystem->AccessPathName(xinc, kReadPermission)) { - TString curincpath = gSystem->GetIncludePath(); - if (curincpath.Index(xinc) == kNPOS) - gSystem->AddIncludePath(TString::Format("-I%s", xinc.Data())); - } else - if (gDebug > 0) - Info("HandleLibIncPath", - "incpath %s does not exist or cannot be read - not added", xinc.Data()); - } - } - - - } else { - - if (type == "lib") { - - // Remove libs - TIter nxl(op); - TObjString *lib = 0; - while ((lib = (TObjString *) nxl())) { - // Expand path - TString xlib = lib->GetName(); - gSystem->ExpandPathName(xlib); - // Remove from the dynamic lib search path - TString newlibpath = gSystem->GetDynamicPath(); - newlibpath.ReplaceAll(TString::Format("%s:", xlib.Data()),""); - gSystem->SetDynamicPath(newlibpath); - } - - } else { - - // Remove incs - TIter nxi(op); - TObjString *inc = 0; - while ((inc = (TObjString *) nxi())) { - TString newincpath = gSystem->GetIncludePath(); - newincpath.ReplaceAll(TString::Format("-I%s", inc->GetName()),""); - // Remove the interpreter path (added anyhow internally) - newincpath.ReplaceAll(gInterpreter->GetIncludePath(),""); - gSystem->SetIncludePath(newincpath); - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get from the master the list of names of the packages available. - -TList *TProof::GetListOfPackages() -{ - if (!IsValid()) - return (TList *)0; - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kListPackages); - Broadcast(mess); - Collect(kActive, fCollectTimeout); - - return fAvailablePackages; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get from the master the list of names of the packages enabled. - -TList *TProof::GetListOfEnabledPackages() -{ - if (!IsValid()) - return (TList *)0; - - TMessage mess(kPROOF_CACHE); - mess << Int_t(kListEnabledPackages); - Broadcast(mess); - Collect(kActive, fCollectTimeout); - - return fEnabledPackages; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print a progress bar on stderr. Used in batch mode. - -void TProof::PrintProgress(Long64_t total, Long64_t processed, - Float_t procTime, Long64_t bytesread) -{ - if (fPrintProgress) { - Bool_t redirlog = fRedirLog; - fRedirLog = kFALSE; - // Call the external function - (*fPrintProgress)(total, processed, procTime, bytesread); - fRedirLog = redirlog; - return; - } - - fprintf(stderr, "[TProof::Progress] Total %lld events\t|", total); - - for (int l = 0; l < 20; l++) { - if (total > 0) { - if (l < 20*processed/total) - fprintf(stderr, "="); - else if (l == 20*processed/total) - fprintf(stderr, ">"); - else if (l > 20*processed/total) - fprintf(stderr, "."); - } else - fprintf(stderr, "="); - } - Float_t evtrti = (procTime > 0. && processed > 0) ? processed / procTime : -1.; - Float_t mbsrti = (procTime > 0. && bytesread > 0) ? bytesread / procTime : -1.; - TString sunit("B/s"); - if (evtrti > 0.) { - Float_t remainingTime = (total >= processed) ? (total - processed) / evtrti : -1; - if (mbsrti > 0.) { - const Float_t toK = 1024., toM = 1048576., toG = 1073741824.; - if (mbsrti >= toG) { - mbsrti /= toG; - sunit = "GB/s"; - } else if (mbsrti >= toM) { - mbsrti /= toM; - sunit = "MB/s"; - } else if (mbsrti >= toK) { - mbsrti /= toK; - sunit = "kB/s"; - } - fprintf(stderr, "| %.02f %% [%.1f evts/s, %.1f %s, time left: %.1f s]\r", - (total ? ((100.0*processed)/total) : 100.0), evtrti, mbsrti, sunit.Data(), remainingTime); - } else { - fprintf(stderr, "| %.02f %% [%.1f evts/s, time left: %.1f s]\r", - (total ? ((100.0*processed)/total) : 100.0), evtrti, remainingTime); - } - } else { - fprintf(stderr, "| %.02f %%\r", - (total ? ((100.0*processed)/total) : 100.0)); - } - if (processed >= total) { - fprintf(stderr, "\n Query processing time: %.1f s\n", procTime); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get query progress information. Connect a slot to this signal -/// to track progress. - -void TProof::Progress(Long64_t total, Long64_t processed) -{ - if (fPrintProgress) { - // Call the external function - return (*fPrintProgress)(total, processed, -1., -1); - } - - PDB(kGlobal,1) - Info("Progress","%2f (%lld/%lld)", 100.*processed/total, processed, total); - - if (gROOT->IsBatch()) { - // Simple progress bar - if (total > 0) - PrintProgress(total, processed); - } else { - EmitVA("Progress(Long64_t,Long64_t)", 2, total, processed); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get query progress information. Connect a slot to this signal -/// to track progress. - -void TProof::Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) -{ - PDB(kGlobal,1) - Info("Progress","%lld %lld %lld %f %f %f %f", total, processed, bytesread, - initTime, procTime, evtrti, mbrti); - - if (gROOT->IsBatch()) { - // Simple progress bar - if (total > 0) - PrintProgress(total, processed, procTime, bytesread); - } else { - EmitVA("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t)", - 7, total, processed, bytesread, initTime, procTime, evtrti, mbrti); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get query progress information. Connect a slot to this signal -/// to track progress. - -void TProof::Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti, Int_t actw, Int_t tses, Float_t eses) -{ - PDB(kGlobal,1) - Info("Progress","%lld %lld %lld %f %f %f %f %d %f", total, processed, bytesread, - initTime, procTime, evtrti, mbrti, actw, eses); - - if (gROOT->IsBatch()) { - // Simple progress bar - if (total > 0) - PrintProgress(total, processed, procTime, bytesread); - } else { - EmitVA("Progress(Long64_t,Long64_t,Long64_t,Float_t,Float_t,Float_t,Float_t,Int_t,Int_t,Float_t)", - 10, total, processed, bytesread, initTime, procTime, evtrti, mbrti, actw, tses, eses); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get list of feedback objects. Connect a slot to this signal -/// to monitor the feedback object. - -void TProof::Feedback(TList *objs) -{ - PDB(kGlobal,1) - Info("Feedback","%d objects", objs->GetSize()); - PDB(kFeedback,1) { - Info("Feedback","%d objects", objs->GetSize()); - objs->ls(); - } - - Emit("Feedback(TList *objs)", (Longptr_t) objs); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close progress dialog. - -void TProof::CloseProgressDialog() -{ - PDB(kGlobal,1) - Info("CloseProgressDialog", - "called: have progress dialog: %d", fProgressDialogStarted); - - // Nothing to do if not there - if (!fProgressDialogStarted) - return; - - Emit("CloseProgressDialog()"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset progress dialog. - -void TProof::ResetProgressDialog(const char *sel, Int_t sz, Long64_t fst, - Long64_t ent) -{ - PDB(kGlobal,1) - Info("ResetProgressDialog","(%s,%d,%lld,%lld)", sel, sz, fst, ent); - - EmitVA("ResetProgressDialog(const char*,Int_t,Long64_t,Long64_t)", - 4, sel, sz, fst, ent); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send startup message. - -void TProof::StartupMessage(const char *msg, Bool_t st, Int_t done, Int_t total) -{ - PDB(kGlobal,1) - Info("StartupMessage","(%s,%d,%d,%d)", msg, st, done, total); - - EmitVA("StartupMessage(const char*,Bool_t,Int_t,Int_t)", - 4, msg, st, done, total); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send dataset preparation status. - -void TProof::DataSetStatus(const char *msg, Bool_t st, Int_t done, Int_t total) -{ - PDB(kGlobal,1) - Info("DataSetStatus","(%s,%d,%d,%d)", msg, st, done, total); - - EmitVA("DataSetStatus(const char*,Bool_t,Int_t,Int_t)", - 4, msg, st, done, total); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send or notify data set status - -void TProof::SendDataSetStatus(const char *action, UInt_t done, - UInt_t tot, Bool_t st) -{ - if (IsLite()) { - if (tot) { - TString type = "files"; - Int_t frac = (Int_t) (done*100.)/tot; - char msg[512] = {0}; - if (frac >= 100) { - snprintf(msg, 512, "%s: OK (%d %s) \n", - action,tot, type.Data()); - } else { - snprintf(msg, 512, "%s: %d out of %d (%d %%)\r", - action, done, tot, frac); - } - if (fSync) - fprintf(stderr,"%s", msg); - else - NotifyLogMsg(msg, 0); - } - return; - } - - if (TestBit(TProof::kIsMaster)) { - TMessage mess(kPROOF_DATASET_STATUS); - mess << TString(action) << tot << done << st; - gProofServ->GetSocket()->Send(mess); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Notify availability of a query result. - -void TProof::QueryResultReady(const char *ref) -{ - PDB(kGlobal,1) - Info("QueryResultReady","ref: %s", ref); - - Emit("QueryResultReady(const char*)",ref); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Validate a TDSet. - -void TProof::ValidateDSet(TDSet *dset) -{ - if (dset->ElementsValid()) return; - - TList nodes; - nodes.SetOwner(); - - TList slholder; - slholder.SetOwner(); - TList elemholder; - elemholder.SetOwner(); - - // build nodelist with slaves and elements - TIter nextSlave(GetListOfActiveSlaves()); - while (TSlave *sl = dynamic_cast(nextSlave())) { - TList *sllist = 0; - TPair *p = dynamic_cast(nodes.FindObject(sl->GetName())); - if (!p) { - sllist = new TList; - sllist->SetName(sl->GetName()); - slholder.Add(sllist); - TList *elemlist = new TList; - elemlist->SetName(TString(sl->GetName())+"_elem"); - elemholder.Add(elemlist); - nodes.Add(new TPair(sllist, elemlist)); - } else { - sllist = dynamic_cast(p->Key()); - } - if (sllist) sllist->Add(sl); - } - - // add local elements to nodes - TList nonLocal; // list of nonlocal elements - // make two iterations - first add local elements - then distribute nonlocals - for (Int_t i = 0; i < 2; i++) { - Bool_t local = i>0?kFALSE:kTRUE; - TIter nextElem(local ? dset->GetListOfElements() : &nonLocal); - while (TDSetElement *elem = dynamic_cast(nextElem())) { - if (elem->GetValid()) continue; - TPair *p = dynamic_cast(local?nodes.FindObject(TUrl(elem->GetFileName()).GetHost()):nodes.At(0)); - if (p) { - TList *eli = dynamic_cast(p->Value()); - TList *sli = dynamic_cast(p->Key()); - if (eli && sli) { - eli->Add(elem); - - // order list by elements/slave - TPair *p2 = p; - Bool_t stop = kFALSE; - while (!stop) { - TPair *p3 = dynamic_cast(nodes.After(p2->Key())); - if (p3) { - TList *p3v = dynamic_cast(p3->Value()); - TList *p3k = dynamic_cast(p3->Key()); - if (p3v && p3k) { - Int_t nelem = p3v->GetSize(); - Int_t nsl = p3k->GetSize(); - if (nelem*sli->GetSize() < eli->GetSize()*nsl) p2 = p3; - else stop = kTRUE; - } - } else { - stop = kTRUE; - } - } - - if (p2!=p) { - nodes.Remove(p->Key()); - nodes.AddAfter(p2->Key(), p); - } - } else { - Warning("ValidateDSet", "invalid values from TPair! Protocol error?"); - continue; - } - - } else { - if (local) { - nonLocal.Add(elem); - } else { - Warning("ValidateDSet", "no node to allocate TDSetElement to - ignoring"); - } - } - } - } - - // send to slaves - TList usedslaves; - TIter nextNode(&nodes); - SetDSet(dset); // set dset to be validated in Collect() - while (TPair *node = dynamic_cast(nextNode())) { - TList *slaves = dynamic_cast(node->Key()); - TList *setelements = dynamic_cast(node->Value()); - if (!slaves || !setelements) continue; - // distribute elements over the slaves - Int_t nslaves = slaves->GetSize(); - Int_t nelements = setelements->GetSize(); - for (Int_t i=0; iGetType(), dset->GetObjName(), - dset->GetDirectory()); - for (Int_t j = (i*nelements)/nslaves; - j < ((i+1)*nelements)/nslaves; - j++) { - TDSetElement *elem = - dynamic_cast(setelements->At(j)); - if (elem) { - copyset.Add(elem->GetFileName(), elem->GetObjName(), - elem->GetDirectory(), elem->GetFirst(), - elem->GetNum(), elem->GetMsd()); - } - } - - if (copyset.GetListOfElements()->GetSize()>0) { - TMessage mesg(kPROOF_VALIDATE_DSET); - mesg << ©set; - - TSlave *sl = dynamic_cast(slaves->At(i)); - if (sl) { - PDB(kGlobal,1) Info("ValidateDSet", - "Sending TDSet with %d elements to slave %s" - " to be validated", - copyset.GetListOfElements()->GetSize(), - sl->GetOrdinal()); - sl->GetSocket()->Send(mesg); - usedslaves.Add(sl); - } - } - } - } - - PDB(kGlobal,1) - Info("ValidateDSet","Calling Collect"); - Collect(&usedslaves); - SetDSet(0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add data objects that might be needed during the processing of -/// the selector (see Process()). This object can be very large, so they -/// are distributed in an optimized way using a dedicated file. -/// If push is TRUE the input data are sent over even if no apparent change -/// occured to the list. - -void TProof::AddInputData(TObject *obj, Bool_t push) -{ - if (obj) { - if (!fInputData) fInputData = new TList; - if (!fInputData->FindObject(obj)) { - fInputData->Add(obj); - SetBit(TProof::kNewInputData); - } - } - if (push) SetBit(TProof::kNewInputData); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove obj form the input data list; if obj is null (default), clear the -/// input data info. - -void TProof::ClearInputData(TObject *obj) -{ - if (!obj) { - if (fInputData) { - fInputData->SetOwner(kTRUE); - SafeDelete(fInputData); - } - ResetBit(TProof::kNewInputData); - - // Also remove any info about input data in the input list - TObject *o = 0; - TList *in = GetInputList(); - while ((o = GetInputList()->FindObject("PROOF_InputDataFile"))) - in->Remove(o); - while ((o = GetInputList()->FindObject("PROOF_InputData"))) - in->Remove(o); - - // ... and reset the file - fInputDataFile = ""; - gSystem->Unlink(kPROOF_InputDataFile); - - } else if (fInputData) { - Int_t sz = fInputData->GetSize(); - while (fInputData->FindObject(obj)) - fInputData->Remove(obj); - // Flag for update, if anything changed - if (sz != fInputData->GetSize()) - SetBit(TProof::kNewInputData); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove obj 'name' form the input data list; - -void TProof::ClearInputData(const char *name) -{ - TObject *obj = (fInputData && name) ? fInputData->FindObject(name) : 0; - if (obj) ClearInputData(obj); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the file to be used to optimally distribute the input data objects. -/// If the file exists the object in the file are added to those in the -/// fInputData list. If the file path is null, a default file will be created -/// at the moment of sending the processing request with the content of -/// the fInputData list. See also SendInputDataFile. - -void TProof::SetInputDataFile(const char *datafile) -{ - if (datafile && strlen(datafile) > 0) { - if (fInputDataFile != datafile && strcmp(datafile, kPROOF_InputDataFile)) - SetBit(TProof::kNewInputData); - fInputDataFile = datafile; - } else { - if (!fInputDataFile.IsNull()) - SetBit(TProof::kNewInputData); - fInputDataFile = ""; - } - // Make sure that the chosen file is readable - if (fInputDataFile != kPROOF_InputDataFile && !fInputDataFile.IsNull() && - gSystem->AccessPathName(fInputDataFile, kReadPermission)) { - fInputDataFile = ""; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send the input data objects to the master; the objects are taken from the -/// dedicated list and / or the specified file. -/// If the fInputData is empty the specified file is sent over. -/// If there is no specified file, a file named "inputdata.root" is created locally -/// with the content of fInputData and sent over to the master. -/// If both fInputData and the specified file are not empty, a copy of the file -/// is made locally and augmented with the content of fInputData. - -void TProof::SendInputDataFile() -{ - // Prepare the file - TString dataFile; - PrepareInputDataFile(dataFile); - - // Send it, if not empty - if (dataFile.Length() > 0) { - - Info("SendInputDataFile", "broadcasting %s", dataFile.Data()); - BroadcastFile(dataFile.Data(), kBinary, "cache", kActive); - - // Set the name in the input list - TString t = TString::Format("cache:%s", gSystem->BaseName(dataFile)); - AddInput(new TNamed("PROOF_InputDataFile", t.Data())); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Prepare the file with the input data objects to be sent the master; the -/// objects are taken from the dedicated list and / or the specified file. -/// If the fInputData is empty the specified file is sent over. -/// If there is no specified file, a file named "inputdata.root" is created locally -/// with the content of fInputData and sent over to the master. -/// If both fInputData and the specified file are not empty, a copy of the file -/// is made locally and augmented with the content of fInputData. - -void TProof::PrepareInputDataFile(TString &dataFile) -{ - // Save info about new data for usage in this call; - Bool_t newdata = TestBit(TProof::kNewInputData) ? kTRUE : kFALSE; - // Next time we need some change - ResetBit(TProof::kNewInputData); - - // Check the list - Bool_t list_ok = (fInputData && fInputData->GetSize() > 0) ? kTRUE : kFALSE; - // Check the file - Bool_t file_ok = kFALSE; - if (fInputDataFile != kPROOF_InputDataFile && !fInputDataFile.IsNull() && - !gSystem->AccessPathName(fInputDataFile, kReadPermission)) { - // It must contain something - TFile *f = TFile::Open(fInputDataFile); - if (f && f->GetListOfKeys() && f->GetListOfKeys()->GetSize() > 0) - file_ok = kTRUE; - } - - // Remove any info about input data in the input list - TObject *o = 0; - TList *in = GetInputList(); - while ((o = GetInputList()->FindObject("PROOF_InputDataFile"))) - in->Remove(o); - while ((o = GetInputList()->FindObject("PROOF_InputData"))) - in->Remove(o); - - // We must have something to send - dataFile = ""; - if (!list_ok && !file_ok) return; - - // Three cases: - if (file_ok && !list_ok) { - // Just send the file - dataFile = fInputDataFile; - } else if (!file_ok && list_ok) { - fInputDataFile = kPROOF_InputDataFile; - // Nothing to do, if no new data - if (!newdata && !gSystem->AccessPathName(fInputDataFile)) return; - // Create the file first - TFile *f = TFile::Open(fInputDataFile, "RECREATE"); - if (f) { - f->cd(); - TIter next(fInputData); - TObject *obj; - while ((obj = next())) { - obj->Write(0, TObject::kSingleKey, 0); - } - f->Close(); - SafeDelete(f); - } else { - Error("PrepareInputDataFile", "could not (re-)create %s", fInputDataFile.Data()); - return; - } - dataFile = fInputDataFile; - } else if (file_ok && list_ok) { - dataFile = kPROOF_InputDataFile; - // Create the file if not existing or there are new data - if (newdata || gSystem->AccessPathName(dataFile)) { - // Cleanup previous file if obsolete - if (!gSystem->AccessPathName(dataFile)) - gSystem->Unlink(dataFile); - if (dataFile != fInputDataFile) { - // Make a local copy first - if (gSystem->CopyFile(fInputDataFile, dataFile, kTRUE) != 0) { - Error("PrepareInputDataFile", "could not make local copy of %s", fInputDataFile.Data()); - return; - } - } - // Add the input data list - TFile *f = TFile::Open(dataFile, "UPDATE"); - if (f) { - f->cd(); - TIter next(fInputData); - TObject *obj = 0; - while ((obj = next())) { - obj->Write(0, TObject::kSingleKey, 0); - } - f->Close(); - SafeDelete(f); - } else { - Error("PrepareInputDataFile", "could not open %s for updating", dataFile.Data()); - return; - } - } - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add objects that might be needed during the processing of -/// the selector (see Process()). - -void TProof::AddInput(TObject *obj) -{ - if (fPlayer) fPlayer->AddInput(obj); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear input object list. - -void TProof::ClearInput() -{ - if (fPlayer) fPlayer->ClearInput(); - - // the system feedback list is always in the input list - AddInput(fFeedback); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get input list. - -TList *TProof::GetInputList() -{ - return (fPlayer ? fPlayer->GetInputList() : (TList *)0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get specified object that has been produced during the processing -/// (see Process()). - -TObject *TProof::GetOutput(const char *name) -{ - - if (TestBit(TProof::kIsMaster)) - // Can be called by MarkBad on the master before the player is initialized - return (fPlayer) ? fPlayer->GetOutput(name) : (TObject *)0; - - // This checks also associated output files - return (GetOutputList()) ? GetOutputList()->FindObject(name) : (TObject *)0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Find object 'name' in list 'out' or in the files specified in there - -TObject *TProof::GetOutput(const char *name, TList *out) -{ - TObject *o = 0; - if (!name || (name && strlen(name) <= 0) || - !out || (out && out->GetSize() <= 0)) return o; - if ((o = out->FindObject(name))) return o; - - // For the time being we always check for all the files; this may require - // some caching - TProofOutputFile *pf = 0; - TIter nxo(out); - while ((o = nxo())) { - if ((pf = dynamic_cast (o))) { - TFile *f = 0; - if (!(f = (TFile *) gROOT->GetListOfFiles()->FindObject(pf->GetOutputFileName()))) { - TString fn = TString::Format("%s/%s", pf->GetDir(), pf->GetFileName()); - f = TFile::Open(fn.Data()); - if (!f || (f && f->IsZombie())) { - ::Warning("TProof::GetOutput", "problems opening file %s", fn.Data()); - } - } - if (f && (o = f->Get(name))) return o; - } - } - - // Done, unsuccessfully - return o; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get list with all object created during processing (see Process()). - -TList *TProof::GetOutputList() -{ - if (fOutputList.GetSize() > 0) return &fOutputList; - if (fPlayer) { - fOutputList.AttachList(fPlayer->GetOutputList()); - return &fOutputList; - } - return (TList *)0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set input list parameter. If the parameter is already -/// set it will be set to the new value. - -void TProof::SetParameter(const char *par, const char *value) -{ - if (!fPlayer) { - Warning("SetParameter", "player undefined! Ignoring"); - return; - } - - TList *il = fPlayer->GetInputList(); - TObject *item = il->FindObject(par); - if (item) { - il->Remove(item); - delete item; - } - il->Add(new TNamed(par, value)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set an input list parameter. - -void TProof::SetParameter(const char *par, Int_t value) -{ - if (!fPlayer) { - Warning("SetParameter", "player undefined! Ignoring"); - return; - } - - TList *il = fPlayer->GetInputList(); - TObject *item = il->FindObject(par); - if (item) { - il->Remove(item); - delete item; - } - il->Add(new TParameter(par, value)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set an input list parameter. - -void TProof::SetParameter(const char *par, Long_t value) -{ - if (!fPlayer) { - Warning("SetParameter", "player undefined! Ignoring"); - return; - } - - TList *il = fPlayer->GetInputList(); - TObject *item = il->FindObject(par); - if (item) { - il->Remove(item); - delete item; - } - il->Add(new TParameter(par, value)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set an input list parameter. - -void TProof::SetParameter(const char *par, Long64_t value) -{ - if (!fPlayer) { - Warning("SetParameter", "player undefined! Ignoring"); - return; - } - - TList *il = fPlayer->GetInputList(); - TObject *item = il->FindObject(par); - if (item) { - il->Remove(item); - delete item; - } - il->Add(new TParameter(par, value)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set an input list parameter. - -void TProof::SetParameter(const char *par, Double_t value) -{ - if (!fPlayer) { - Warning("SetParameter", "player undefined! Ignoring"); - return; - } - - TList *il = fPlayer->GetInputList(); - TObject *item = il->FindObject(par); - if (item) { - il->Remove(item); - delete item; - } - il->Add(new TParameter(par, value)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get specified parameter. A parameter set via SetParameter() is either -/// a TParameter or a TNamed or 0 in case par is not defined. - -TObject *TProof::GetParameter(const char *par) const -{ - if (!fPlayer) { - Warning("GetParameter", "player undefined! Ignoring"); - return (TObject *)0; - } - - TList *il = fPlayer->GetInputList(); - return il->FindObject(par); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete the input list parameters specified by a wildcard (e.g. PROOF_*) -/// or exact name (e.g. PROOF_MaxSlavesPerNode). - -void TProof::DeleteParameters(const char *wildcard) -{ - if (!fPlayer) return; - - if (!wildcard) wildcard = ""; - TRegexp re(wildcard, kTRUE); - Int_t nch = strlen(wildcard); - - TList *il = fPlayer->GetInputList(); - if (il) { - TObject *p = 0; - TIter next(il); - while ((p = next())) { - TString s = p->GetName(); - if (nch && s != wildcard && s.Index(re) == kNPOS) continue; - il->Remove(p); - delete p; - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show the input list parameters specified by the wildcard. -/// Default is the special PROOF control parameters (PROOF_*). - -void TProof::ShowParameters(const char *wildcard) const -{ - if (!fPlayer) return; - - if (!wildcard) wildcard = ""; - TRegexp re(wildcard, kTRUE); - Int_t nch = strlen(wildcard); - - TList *il = fPlayer->GetInputList(); - TObject *p; - TIter next(il); - while ((p = next())) { - TString s = p->GetName(); - if (nch && s != wildcard && s.Index(re) == kNPOS) continue; - if (p->IsA() == TNamed::Class()) { - Printf("%s\t\t\t%s", s.Data(), p->GetTitle()); - } else if (p->IsA() == TParameter::Class()) { - Printf("%s\t\t\t%ld", s.Data(), dynamic_cast*>(p)->GetVal()); - } else if (p->IsA() == TParameter::Class()) { - Printf("%s\t\t\t%lld", s.Data(), dynamic_cast*>(p)->GetVal()); - } else if (p->IsA() == TParameter::Class()) { - Printf("%s\t\t\t%f", s.Data(), dynamic_cast*>(p)->GetVal()); - } else { - Printf("%s\t\t\t%s", s.Data(), p->GetTitle()); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add object to feedback list. - -void TProof::AddFeedback(const char *name) -{ - PDB(kFeedback, 3) - Info("AddFeedback", "Adding object \"%s\" to feedback", name); - if (fFeedback->FindObject(name) == 0) - fFeedback->Add(new TObjString(name)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove object from feedback list. - -void TProof::RemoveFeedback(const char *name) -{ - TObject *obj = fFeedback->FindObject(name); - if (obj != 0) { - fFeedback->Remove(obj); - delete obj; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear feedback list. - -void TProof::ClearFeedback() -{ - fFeedback->Delete(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show items in feedback list. - -void TProof::ShowFeedback() const -{ - if (fFeedback->GetSize() == 0) { - Info("","no feedback requested"); - return; - } - - fFeedback->Print(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return feedback list. - -TList *TProof::GetFeedbackList() const -{ - return fFeedback; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Creates a tree header (a tree with nonexisting files) object for -/// the DataSet. - -TTree *TProof::GetTreeHeader(TDSet *dset) -{ - TList *l = GetListOfActiveSlaves(); - TSlave *sl = (TSlave*) l->First(); - if (sl == 0) { - Error("GetTreeHeader", "No connection"); - return 0; - } - - TSocket *soc = sl->GetSocket(); - TMessage msg(kPROOF_GETTREEHEADER); - - msg << dset; - - soc->Send(msg); - - TMessage *reply; - Int_t d = -1; - if (fProtocol >= 20) { - Collect(sl, fCollectTimeout, kPROOF_GETTREEHEADER); - reply = (TMessage *) fRecvMessages->First(); - } else { - d = soc->Recv(reply); - } - if (!reply) { - Error("GetTreeHeader", "Error getting a replay from the master.Result %d", (int) d); - return 0; - } - - TString s1; - TTree *t = 0; - (*reply) >> s1; - if (s1 == "Success") - (*reply) >> t; - - PDB(kGlobal, 1) { - if (t) { - Info("GetTreeHeader", "%s, message size: %d, entries: %d", - s1.Data(), reply->BufferSize(), (int) t->GetMaxEntryLoop()); - } else { - Info("GetTreeHeader", "tree header retrieval failed"); - } - } - delete reply; - - return t; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw feedback creation proxy. When accessed via TProof avoids -/// link dependency on libProofPlayer. - -TDrawFeedback *TProof::CreateDrawFeedback() -{ - return (fPlayer ? fPlayer->CreateDrawFeedback(this) : (TDrawFeedback *)0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set draw feedback option. - -void TProof::SetDrawFeedbackOption(TDrawFeedback *f, Option_t *opt) -{ - if (fPlayer) fPlayer->SetDrawFeedbackOption(f, opt); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete draw feedback object. - -void TProof::DeleteDrawFeedback(TDrawFeedback *f) -{ - if (fPlayer) fPlayer->DeleteDrawFeedback(f); -} - -//////////////////////////////////////////////////////////////////////////////// -/// FIXME: to be written - -TList *TProof::GetOutputNames() -{ - return 0; -/* - TMessage msg(kPROOF_GETOUTPUTLIST); - TList* slaves = fActiveSlaves; - Broadcast(msg, slaves); - TMonitor mon; - TList* outputList = new TList(); - - TIter si(slaves); - TSlave *slave; - while ((slave = (TSlave*)si.Next()) != 0) { - PDB(kGlobal,4) Info("GetOutputNames","Socket added to monitor: %p (%s)", - slave->GetSocket(), slave->GetName()); - mon.Add(slave->GetSocket()); - } - mon.ActivateAll(); - ((TProof*)gProof)->DeActivateAsyncInput(); - ((TProof*)gProof)->fCurrentMonitor = &mon; - - while (mon.GetActive() != 0) { - TSocket *sock = mon.Select(); - if (!sock) { - Error("GetOutputList","TMonitor::.Select failed!"); - break; - } - mon.DeActivate(sock); - TMessage *reply; - if (sock->Recv(reply) <= 0) { - MarkBad(slave, "receive failed after kPROOF_GETOUTPUTLIST request"); -// Error("GetOutputList","Recv failed! for slave-%d (%s)", -// slave->GetOrdinal(), slave->GetName()); - continue; - } - if (reply->What() != kPROOF_GETOUTPUTNAMES ) { -// Error("GetOutputList","unexpected message %d from slawe-%d (%s)", reply->What(), -// slave->GetOrdinal(), slave->GetName()); - MarkBad(slave, "wrong reply to kPROOF_GETOUTPUTLIST request"); - continue; - } - TList* l; - - (*reply) >> l; - TIter next(l); - TNamed *n; - while ( (n = dynamic_cast (next())) ) { - if (!outputList->FindObject(n->GetName())) - outputList->Add(n); - } - delete reply; - } - ((TProof*)gProof)->fCurrentMonitor = 0; - - return outputList; -*/ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build the PROOF's structure in the browser. - -void TProof::Browse(TBrowser *b) -{ - b->Add(fActiveSlaves, fActiveSlaves->Class(), "fActiveSlaves"); - b->Add(&fMaster, fMaster.Class(), "fMaster"); - b->Add(fFeedback, fFeedback->Class(), "fFeedback"); - b->Add(fChains, fChains->Class(), "fChains"); - - if (fPlayer) { - b->Add(fPlayer->GetInputList(), fPlayer->GetInputList()->Class(), "InputList"); - if (fPlayer->GetOutputList()) - b->Add(fPlayer->GetOutputList(), fPlayer->GetOutputList()->Class(), "OutputList"); - if (fPlayer->GetListOfResults()) - b->Add(fPlayer->GetListOfResults(), - fPlayer->GetListOfResults()->Class(), "ListOfResults"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set a new PROOF player. - -void TProof::SetPlayer(TVirtualProofPlayer *player) -{ - if (fPlayer) - delete fPlayer; - fPlayer = player; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Construct a TProofPlayer object. The player string specifies which -/// player should be created: remote, slave, sm (supermaster) or base. -/// Default is remote. Socket is needed in case a slave player is created. - -TVirtualProofPlayer *TProof::MakePlayer(const char *player, TSocket *s) -{ - if (!player) - player = "remote"; - - SetPlayer(TVirtualProofPlayer::Create(player, this, s)); - return GetPlayer(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add chain to data set - -void TProof::AddChain(TChain *chain) -{ - fChains->Add(chain); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove chain from data set - -void TProof::RemoveChain(TChain *chain) -{ - fChains->Remove(chain); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ask for remote logs in the range [start, end]. If start == -1 all the -/// messages not yet received are sent back. - -void TProof::GetLog(Int_t start, Int_t end) -{ - if (!IsValid() || TestBit(TProof::kIsMaster)) return; - - TMessage msg(kPROOF_LOGFILE); - - msg << start << end; - - Broadcast(msg, kActive); - Collect(kActive, fCollectTimeout); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fill a TMacro with the log lines since the last reading (fLogFileR) -/// Return (TMacro *)0 if no line was logged. -/// The returned TMacro must be deleted by the caller. - -TMacro *TProof::GetLastLog() -{ - TMacro *maclog = 0; - - // Save present offset - off_t nowlog = lseek(fileno(fLogFileR), (off_t) 0, SEEK_CUR); - if (nowlog < 0) { - SysError("GetLastLog", - "problem lseeking log file to current position (errno: %d)", TSystem::GetErrno()); - return maclog; - } - - // Get extremes - off_t startlog = nowlog; - off_t endlog = lseek(fileno(fLogFileR), (off_t) 0, SEEK_END); - if (endlog < 0) { - SysError("GetLastLog", - "problem lseeking log file to end position (errno: %d)", TSystem::GetErrno()); - return maclog; - } - - // Perhaps nothing to log - UInt_t tolog = (UInt_t)(endlog - startlog); - if (tolog <= 0) return maclog; - - // Set starting point - if (lseek(fileno(fLogFileR), startlog, SEEK_SET) < 0) { - SysError("GetLastLog", - "problem lseeking log file to start position (errno: %d)", TSystem::GetErrno()); - return maclog; - } - - // Create the output object - maclog = new TMacro; - - // Now we go - char line[2048]; - Int_t wanted = (tolog > sizeof(line)) ? sizeof(line) : tolog; - while (fgets(line, wanted, fLogFileR)) { - Int_t r = strlen(line); - if (r > 0) { - if (line[r-1] == '\n') line[r-1] = '\0'; - maclog->AddLine(line); - } else { - // Done - break; - } - tolog -= r; - wanted = (tolog > sizeof(line)) ? sizeof(line) : tolog; - } - - // Restore original pointer - if (lseek(fileno(fLogFileR), nowlog, SEEK_SET) < 0) { - Warning("GetLastLog", - "problem lseeking log file to original position (errno: %d)", TSystem::GetErrno()); - } - - // Done - return maclog; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display log of query pq into the log window frame - -void TProof::PutLog(TQueryResult *pq) -{ - if (!pq) return; - - TList *lines = pq->GetLogFile()->GetListOfLines(); - if (lines) { - TIter nxl(lines); - TObjString *l = 0; - while ((l = (TObjString *)nxl())) - EmitVA("LogMessage(const char*,Bool_t)", 2, l->GetName(), kFALSE); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display on screen the content of the temporary log file for query -/// in reference - -void TProof::ShowLog(const char *queryref) -{ - // Make sure we have all info (GetListOfQueries retrieves the - // head info only) - Retrieve(queryref); - - if (fPlayer) { - if (queryref) { - if (fPlayer->GetListOfResults()) { - TIter nxq(fPlayer->GetListOfResults()); - TQueryResult *qr = 0; - while ((qr = (TQueryResult *) nxq())) - if (strstr(queryref, qr->GetTitle()) && - strstr(queryref, qr->GetName())) - break; - if (qr) { - PutLog(qr); - return; - } - - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display on screen the content of the temporary log file. -/// If qry == -2 show messages from the last (current) query. -/// If qry == -1 all the messages not yet displayed are shown (default). -/// If qry == 0, all the messages in the file are shown. -/// If qry > 0, only the messages related to query 'qry' are shown. -/// For qry != -1 the original file offset is restored at the end - -void TProof::ShowLog(Int_t qry) -{ - // Save present offset - off_t nowlog = lseek(fileno(fLogFileR), (off_t) 0, SEEK_CUR); - if (nowlog < 0) { - SysError("ShowLog", "problem lseeking log file (errno: %d)", TSystem::GetErrno()); - return; - } - - // Get extremes - off_t startlog = nowlog; - off_t endlog = lseek(fileno(fLogFileR), (off_t) 0, SEEK_END); - if (endlog < 0) { - SysError("ShowLog", "problem lseeking log file (errno: %d)", TSystem::GetErrno()); - return; - } - - lseek(fileno(fLogFileR), nowlog, SEEK_SET); - if (qry == 0) { - startlog = 0; - lseek(fileno(fLogFileR), (off_t) 0, SEEK_SET); - } else if (qry != -1) { - - TQueryResult *pq = 0; - if (qry == -2) { - // Pickup the last one - pq = (GetQueryResults()) ? ((TQueryResult *)(GetQueryResults()->Last())) : 0; - if (!pq) { - GetListOfQueries(); - if (fQueries) - pq = (TQueryResult *)(fQueries->Last()); - } - } else if (qry > 0) { - TList *queries = GetQueryResults(); - if (queries) { - TIter nxq(queries); - while ((pq = (TQueryResult *)nxq())) - if (qry == pq->GetSeqNum()) - break; - } - if (!pq) { - queries = GetListOfQueries(); - TIter nxq(queries); - while ((pq = (TQueryResult *)nxq())) - if (qry == pq->GetSeqNum()) - break; - } - } - if (pq) { - PutLog(pq); - return; - } else { - if (gDebug > 0) - Info("ShowLog","query %d not found in list", qry); - qry = -1; - } - } - - // Number of bytes to log - UInt_t tolog = (UInt_t)(endlog - startlog); - - // Perhaps nothing - if (tolog <= 0) { - // Set starting point - lseek(fileno(fLogFileR), startlog, SEEK_SET); - } - - // Now we go - Int_t np = 0; - char line[2048]; - Int_t wanted = (tolog > sizeof(line)) ? sizeof(line) : tolog; - while (fgets(line, wanted, fLogFileR)) { - - Int_t r = strlen(line); - if (!SendingLogToWindow()) { - if (line[r-1] != '\n') line[r-1] = '\n'; - if (r > 0) { - char *p = line; - while (r) { - Int_t w = write(fileno(stdout), p, r); - if (w < 0) { - SysError("ShowLog", "error writing to stdout"); - break; - } - r -= w; - p += w; - } - } - tolog -= strlen(line); - np++; - - // Ask if more is wanted - if (!(np%10)) { - const char *opt = Getline("More (y/n)? [y]"); - if (opt[0] == 'n') - break; - } - - // We may be over - if (tolog <= 0) - break; - - // Update wanted bytes - wanted = (tolog > sizeof(line)) ? sizeof(line) : tolog; - } else { - // Log to window - if (line[r-1] == '\n') line[r-1] = 0; - LogMessage(line, kFALSE); - } - } - if (!SendingLogToWindow()) { - // Avoid screwing up the prompt - if (write(fileno(stdout), "\n", 1) != 1) - SysError("ShowLog", "error writing to stdout"); - } - - // Restore original pointer - if (qry > -1) - lseek(fileno(fLogFileR), nowlog, SEEK_SET); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set session with 'id' the default one. If 'id' is not found in the list, -/// the current session is set as default - -void TProof::cd(Int_t id) -{ - if (GetManager()) { - TProofDesc *d = GetManager()->GetProofDesc(id); - if (d) { - if (d->GetProof()) { - gProof = d->GetProof(); - return; - } - } - - // Id not found or undefined: set as default this session - gProof = this; - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Detach this instance to its proofserv. -/// If opt is 'S' or 's' the remote server is shutdown - -void TProof::Detach(Option_t *opt) -{ - // Nothing to do if not in contact with proofserv - if (!IsValid()) return; - - // Get worker and socket instances - TSlave *sl = (TSlave *) fActiveSlaves->First(); - TSocket *s = 0; - if (!sl || !(sl->IsValid()) || !(s = sl->GetSocket())) { - Error("Detach","corrupted worker instance: wrk:%p, sock:%p", sl, s); - return; - } - - Bool_t shutdown = (strchr(opt,'s') || strchr(opt,'S')) ? kTRUE : kFALSE; - - // If processing, try to stop processing first - if (shutdown && !IsIdle()) { - // Remove pending requests - Remove("cleanupqueue"); - // Do not wait for ever, but al least 20 seconds - Long_t timeout = gEnv->GetValue("Proof.ShutdownTimeout", 60); - timeout = (timeout > 20) ? timeout : 20; - // Send stop signal - StopProcess(kFALSE, (Long_t) (timeout / 2)); - // Receive results - Collect(kActive, timeout); - } - - // Avoid spurious messages: deactivate new inputs ... - DeActivateAsyncInput(); - - // ... and discard existing ones - sl->FlushSocket(); - - // Close session (we always close the connection) - Close(opt); - - // Close the progress dialog, if any - if (fProgressDialogStarted) - CloseProgressDialog(); - - // Update info in the table of our manager, if any - if (GetManager() && GetManager()->QuerySessions("L")) { - TIter nxd(GetManager()->QuerySessions("L")); - TProofDesc *d = 0; - while ((d = (TProofDesc *)nxd())) { - if (d->GetProof() == this) { - d->SetProof(0); - GetManager()->QuerySessions("L")->Remove(d); - break; - } - } - } - - // Invalidate this instance - fValid = kFALSE; - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set an alias for this session. If reconnection is supported, the alias -/// will be communicated to the remote coordinator so that it can be recovered -/// when reconnecting - -void TProof::SetAlias(const char *alias) -{ - // Set it locally - TNamed::SetTitle(alias); - if (TestBit(TProof::kIsMaster)) - // Set the name at the same value - TNamed::SetName(alias); - - // Nothing to do if not in contact with coordinator - if (!IsValid()) return; - - if (!IsProofd() && TestBit(TProof::kIsClient)) { - TSlave *sl = (TSlave *) fActiveSlaves->First(); - if (sl) - sl->SetAlias(alias); - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// *** This function is deprecated and will disappear in future versions *** -/// *** It is just a wrapper around TFile::Cp. -/// *** Please use TProofMgr::UploadFiles. -/// -/// Upload a set of files and save the list of files by name dataSetName. -/// The 'files' argument is a list of TFileInfo objects describing the files -/// as first url. -/// The mask 'opt' is a combination of EUploadOpt: -/// kAppend (0x1) if set true files will be appended to -/// the dataset existing by given name -/// kOverwriteDataSet (0x2) if dataset with given name exited it -/// would be overwritten -/// kNoOverwriteDataSet (0x4) do not overwirte if the dataset exists -/// kOverwriteAllFiles (0x8) overwrite all files that may exist -/// kOverwriteNoFiles (0x10) overwrite none -/// kAskUser (0x0) ask user before overwriteng dataset/files -/// The default value is kAskUser. -/// The user will be asked to confirm overwriting dataset or files unless -/// specified opt provides the answer! -/// If kOverwriteNoFiles is set, then a pointer to TList must be passed as -/// skippedFiles argument. The function will add to this list TFileInfo -/// objects describing all files that existed on the cluster and were -/// not uploaded. -/// -/// Communication Summary -/// Client Master -/// |------------>DataSetName----------->| -/// |<-------kMESS_OK/kMESS_NOTOK<-------| (Name OK/file exist) -/// (*)|-------> call RegisterDataSet ------->| -/// (*) - optional - -Int_t TProof::UploadDataSet(const char *, TList *, const char *, Int_t, TList *) -{ - Printf(" *** WARNING: this function is obsolete: it has been replaced by TProofMgr::UploadFiles ***"); - - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// *** This function is deprecated and will disappear in future versions *** -/// *** It is just a wrapper around TFile::Cp. -/// *** Please use TProofMgr::UploadFiles. -/// -/// Upload a set of files and save the list of files by name dataSetName. -/// The mask 'opt' is a combination of EUploadOpt: -/// kAppend (0x1) if set true files will be appended to -/// the dataset existing by given name -/// kOverwriteDataSet (0x2) if dataset with given name exited it -/// would be overwritten -/// kNoOverwriteDataSet (0x4) do not overwirte if the dataset exists -/// kOverwriteAllFiles (0x8) overwrite all files that may exist -/// kOverwriteNoFiles (0x10) overwrite none -/// kAskUser (0x0) ask user before overwriteng dataset/files -/// The default value is kAskUser. -/// The user will be asked to confirm overwriting dataset or files unless -/// specified opt provides the answer! -/// If kOverwriteNoFiles is set, then a pointer to TList must be passed as -/// skippedFiles argument. The function will add to this list TFileInfo -/// objects describing all files that existed on the cluster and were -/// not uploaded. -/// - -Int_t TProof::UploadDataSet(const char *, const char *, const char *, Int_t, TList *) -{ - Printf(" *** WARNING: this function is obsolete: it has been replaced by TProofMgr::UploadFiles ***"); - - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// *** This function is deprecated and will disappear in future versions *** -/// *** It is just a wrapper around TFile::Cp. -/// *** Please use TProofMgr::UploadFiles. -/// -/// Upload files listed in "file" to PROOF cluster. -/// Where file = name of file containing list of files and -/// dataset = dataset name and opt is a combination of EUploadOpt bits. -/// Each file description (line) can include wildcards. -/// Check TFileInfo compatibility - -Int_t TProof::UploadDataSetFromFile(const char *, const char *, const char *, Int_t, TList *) -{ - Printf(" *** WARNING: this function is obsolete: it has been replaced by TProofMgr::UploadFiles ***"); - - // Done - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Register the 'dataSet' on the cluster under the current -/// user, group and the given 'dataSetName'. -/// If a dataset with the same name already exists the action fails unless 'opts' -/// contains 'O', in which case the old dataset is overwritten, or contains 'U', -/// in which case 'newDataSet' is added to the existing dataset (duplications are -/// ignored, if any). -/// If 'opts' contains 'V' the dataset files are also verified (if the dataset manager -/// is configured to allow so). By default the dataset is not verified. -/// If 'opts' contains 'T' the in the dataset object (status bits, meta,...) -/// is trusted, i.e. not reset (if the dataset manager is configured to allow so). -/// If 'opts' contains 'S' validation would be run serially (meaningful only if -/// validation is required). -/// Returns kTRUE on success. - -Bool_t TProof::RegisterDataSet(const char *dataSetName, - TFileCollection *dataSet, const char *optStr) -{ - // Check TFileInfo compatibility - if (fProtocol < 17) { - Info("RegisterDataSet", - "functionality not available: the server does not have dataset support"); - return kFALSE; - } - - if (!dataSetName || strlen(dataSetName) <= 0) { - Info("RegisterDataSet", "specifying a dataset name is mandatory"); - return kFALSE; - } - - Bool_t parallelverify = kFALSE; - TString sopt(optStr); - if (sopt.Contains("V") && fProtocol >= 34 && !sopt.Contains("S")) { - // We do verification in parallel later on; just register for now - parallelverify = kTRUE; - sopt.ReplaceAll("V", ""); - } - // This would screw up things remotely, make sure is not there - sopt.ReplaceAll("S", ""); - - TMessage mess(kPROOF_DATASETS); - mess << Int_t(kRegisterDataSet); - mess << TString(dataSetName); - mess << sopt; - mess.WriteObject(dataSet); - Broadcast(mess); - - Bool_t result = kTRUE; - Collect(); - if (fStatus != 0) { - Error("RegisterDataSet", "dataset was not saved"); - result = kFALSE; - return result; - } - - // If old server or not verifying in parallel we are done - if (!parallelverify) return result; - - // If we are here it means that we will verify in parallel - sopt += "V"; - if (VerifyDataSet(dataSetName, sopt) < 0){ - Error("RegisterDataSet", "problems verifying dataset '%s'", dataSetName); - return kFALSE; - } - - // We are done - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set/Change the name of the default tree. The tree name may contain -/// subdir specification in the form "subdir/name". -/// Returns 0 on success, -1 otherwise. - -Int_t TProof::SetDataSetTreeName(const char *dataset, const char *treename) -{ - // Check TFileInfo compatibility - if (fProtocol < 23) { - Info("SetDataSetTreeName", "functionality not supported by the server"); - return -1; - } - - if (!dataset || strlen(dataset) <= 0) { - Info("SetDataSetTreeName", "specifying a dataset name is mandatory"); - return -1; - } - - if (!treename || strlen(treename) <= 0) { - Info("SetDataSetTreeName", "specifying a tree name is mandatory"); - return -1; - } - - TUri uri(dataset); - TString fragment(treename); - if (!fragment.BeginsWith("/")) fragment.Insert(0, "/"); - uri.SetFragment(fragment); - - TMessage mess(kPROOF_DATASETS); - mess << Int_t(kSetDefaultTreeName); - mess << uri.GetUri(); - Broadcast(mess); - - Collect(); - if (fStatus != 0) { - Error("SetDataSetTreeName", "some error occured: default tree name not changed"); - return -1; - } - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Lists all datasets that match given uri. -/// The 'optStr' can contain a comma-separated list of servers for which the -/// information is wanted. If ':lite:' (case insensitive) is specified in 'optStr' -/// only the global information in the TFileCollection is retrieved; useful to only -/// get the list of available datasets. - -TMap *TProof::GetDataSets(const char *uri, const char *optStr) -{ - if (fProtocol < 15) { - Info("GetDataSets", - "functionality not available: the server does not have dataset support"); - return 0; - } - if (fProtocol < 31 && strstr(optStr, ":lite:")) - Warning("GetDataSets", "'lite' option not supported by the server"); - - TMessage mess(kPROOF_DATASETS); - mess << Int_t(kGetDataSets); - mess << TString(uri ? uri : ""); - mess << TString(optStr ? optStr : ""); - Broadcast(mess); - Collect(kActive, fCollectTimeout); - - TMap *dataSetMap = 0; - if (fStatus != 0) { - Error("GetDataSets", "error receiving datasets information"); - } else { - // Look in the list - TMessage *retMess = (TMessage *) fRecvMessages->First(); - if (retMess && retMess->What() == kMESS_OK) { - if (!(dataSetMap = (TMap *)(retMess->ReadObject(TMap::Class())))) - Error("GetDataSets", "error receiving datasets"); - } else - Error("GetDataSets", "message not found or wrong type (%p)", retMess); - } - - return dataSetMap; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Shows datasets in locations that match the uri. -/// By default shows the user's datasets and global ones - -void TProof::ShowDataSets(const char *uri, const char* optStr) -{ - if (fProtocol < 15) { - Info("ShowDataSets", - "functionality not available: the server does not have dataset support"); - return; - } - - TMessage mess(kPROOF_DATASETS); - mess << Int_t(kShowDataSets); - mess << TString(uri ? uri : ""); - mess << TString(optStr ? optStr : ""); - Broadcast(mess); - - Collect(kActive, fCollectTimeout); - if (fStatus != 0) - Error("ShowDataSets", "error receiving datasets information"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns kTRUE if 'dataset' exists, kFALSE otherwise - -Bool_t TProof::ExistsDataSet(const char *dataset) -{ - if (fProtocol < 15) { - Info("ExistsDataSet", "functionality not available: the server has an" - " incompatible version of TFileInfo"); - return kFALSE; - } - - if (!dataset || strlen(dataset) <= 0) { - Error("ExistsDataSet", "dataset name missing"); - return kFALSE; - } - - TMessage msg(kPROOF_DATASETS); - msg << Int_t(kCheckDataSetName) << TString(dataset); - Broadcast(msg); - Collect(kActive, fCollectTimeout); - if (fStatus == -1) { - // The dataset exists - return kTRUE; - } - // The dataset does not exists - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear the content of the dataset cache, if any (matching 'dataset', if defined). - -void TProof::ClearDataSetCache(const char *dataset) -{ - if (fProtocol < 28) { - Info("ClearDataSetCache", "functionality not available on server"); - return; - } - - TMessage msg(kPROOF_DATASETS); - msg << Int_t(kCache) << TString(dataset) << TString("clear"); - Broadcast(msg); - Collect(kActive, fCollectTimeout); - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display the content of the dataset cache, if any (matching 'dataset', if defined). - -void TProof::ShowDataSetCache(const char *dataset) -{ - if (fProtocol < 28) { - Info("ShowDataSetCache", "functionality not available on server"); - return; - } - - TMessage msg(kPROOF_DATASETS); - msg << Int_t(kCache) << TString(dataset) << TString("show"); - Broadcast(msg); - Collect(kActive, fCollectTimeout); - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get a list of TFileInfo objects describing the files of the specified -/// dataset. -/// To get the short version (containing only the global meta information) -/// specify optStr = "S:" or optStr = "short:". -/// To get the sub-dataset of files located on a given server(s) specify -/// the list of servers (comma-separated) in the 'optStr' field. - -TFileCollection *TProof::GetDataSet(const char *uri, const char *optStr) -{ - if (fProtocol < 15) { - Info("GetDataSet", "functionality not available: the server has an" - " incompatible version of TFileInfo"); - return 0; - } - - if (!uri || strlen(uri) <= 0) { - Info("GetDataSet", "specifying a dataset name is mandatory"); - return 0; - } - - TMessage nameMess(kPROOF_DATASETS); - nameMess << Int_t(kGetDataSet); - nameMess << TString(uri); - nameMess << TString(optStr ? optStr: ""); - if (Broadcast(nameMess) < 0) - Error("GetDataSet", "sending request failed"); - - Collect(kActive, fCollectTimeout); - TFileCollection *fileList = 0; - if (fStatus != 0) { - Error("GetDataSet", "error receiving datasets information"); - } else { - // Look in the list - TMessage *retMess = (TMessage *) fRecvMessages->First(); - if (retMess && retMess->What() == kMESS_OK) { - if (!(fileList = (TFileCollection*)(retMess->ReadObject(TFileCollection::Class())))) - Error("GetDataSet", "error reading list of files"); - } else - Error("GetDataSet", "message not found or wrong type (%p)", retMess); - } - - return fileList; -} - -//////////////////////////////////////////////////////////////////////////////// -/// display meta-info for given dataset usi - -void TProof::ShowDataSet(const char *uri, const char* opt) -{ - TFileCollection *fileList = 0; - if ((fileList = GetDataSet(uri))) { - fileList->Print(opt); - delete fileList; - } else - Warning("ShowDataSet","no such dataset: %s", uri); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove the specified dataset from the PROOF cluster. -/// Files are not deleted. - -Int_t TProof::RemoveDataSet(const char *uri, const char* optStr) -{ - TMessage nameMess(kPROOF_DATASETS); - nameMess << Int_t(kRemoveDataSet); - nameMess << TString(uri?uri:""); - nameMess << TString(optStr?optStr:""); - if (Broadcast(nameMess) < 0) - Error("RemoveDataSet", "sending request failed"); - Collect(kActive, fCollectTimeout); - - if (fStatus != 0) - return -1; - else - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Find datasets, returns in a TList all found datasets. - -TList* TProof::FindDataSets(const char* /*searchString*/, const char* /*optStr*/) -{ - Error ("FindDataSets", "not yet implemented"); - return (TList *) 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Allows users to request staging of a particular dataset. Requests are -/// saved in a special dataset repository and must be honored by the endpoint. - -Bool_t TProof::RequestStagingDataSet(const char *dataset) -{ - if (fProtocol < 35) { - Error("RequestStagingDataSet", - "functionality not supported by the server"); - return kFALSE; - } - - TMessage mess(kPROOF_DATASETS); - mess << Int_t(kRequestStaging); - mess << TString(dataset); - Broadcast(mess); - - Collect(); - if (fStatus != 0) { - Error("RequestStagingDataSet", "staging request was unsuccessful"); - return kFALSE; - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cancels a dataset staging request. Returns kTRUE on success, kFALSE on -/// failure. Dataset not found equals to a failure. - -Bool_t TProof::CancelStagingDataSet(const char *dataset) -{ - if (fProtocol < 36) { - Error("CancelStagingDataSet", - "functionality not supported by the server"); - return kFALSE; - } - - TMessage mess(kPROOF_DATASETS); - mess << Int_t(kCancelStaging); - mess << TString(dataset); - Broadcast(mess); - - Collect(); - if (fStatus != 0) { - Error("CancelStagingDataSet", "cancel staging request was unsuccessful"); - return kFALSE; - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Obtains a TFileCollection showing the staging status of the specified -/// dataset. A valid dataset manager and dataset staging requests repository -/// must be present on the endpoint. - -TFileCollection *TProof::GetStagingStatusDataSet(const char *dataset) -{ - if (fProtocol < 35) { - Error("GetStagingStatusDataSet", - "functionality not supported by the server"); - return NULL; - } - - TMessage nameMess(kPROOF_DATASETS); - nameMess << Int_t(kStagingStatus); - nameMess << TString(dataset); - if (Broadcast(nameMess) < 0) { - Error("GetStagingStatusDataSet", "sending request failed"); - return NULL; - } - - Collect(kActive, fCollectTimeout); - TFileCollection *fc = NULL; - - if (fStatus < 0) { - Error("GetStagingStatusDataSet", "problem processing the request"); - } - else if (fStatus == 0) { - TMessage *retMess = (TMessage *)fRecvMessages->First(); - if (retMess && (retMess->What() == kMESS_OK)) { - fc = (TFileCollection *)( - retMess->ReadObject(TFileCollection::Class()) ); - if (!fc) - Error("GetStagingStatusDataSet", "error reading list of files"); - } - else { - Error("GetStagingStatusDataSet", - "response message not found or wrong type (%p)", retMess); - } - } - //else {} - - return fc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Like GetStagingStatusDataSet, but displays results immediately. - -void TProof::ShowStagingStatusDataSet(const char *dataset, const char *opt) -{ - TFileCollection *fc = GetStagingStatusDataSet(dataset); - if (fc) { - fc->Print(opt); - delete fc; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Verify if all files in the specified dataset are available. -/// Print a list and return the number of missing files. -/// Returns -1 in case of error. - -Int_t TProof::VerifyDataSet(const char *uri, const char *optStr) -{ - if (fProtocol < 15) { - Info("VerifyDataSet", "functionality not available: the server has an" - " incompatible version of TFileInfo"); - return -1; - } - - // Sanity check - if (!uri || (uri && strlen(uri) <= 0)) { - Error("VerifyDataSet", "dataset name is is mandatory"); - return -1; - } - - Int_t nmissingfiles = 0; - - TString sopt(optStr); - if (fProtocol < 34 || sopt.Contains("S")) { - sopt.ReplaceAll("S", ""); - Info("VerifyDataSet", "Master-only verification"); - TMessage nameMess(kPROOF_DATASETS); - nameMess << Int_t(kVerifyDataSet); - nameMess << TString(uri); - nameMess << sopt; - Broadcast(nameMess); - - Collect(kActive, fCollectTimeout); - - if (fStatus < 0) { - Info("VerifyDataSet", "no such dataset %s", uri); - return -1; - } else - nmissingfiles = fStatus; - return nmissingfiles; - } - - // Request for parallel verification: can only be done if we have workers - if (!IsParallel() && !fDynamicStartup) { - Error("VerifyDataSet", "PROOF is in sequential mode (no workers): cannot do parallel verification."); - Error("VerifyDataSet", "Either start PROOF with some workers or force sequential adding 'S' as option."); - return -1; - } - - // Do parallel verification - return VerifyDataSetParallel(uri, optStr); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Internal function for parallel dataset verification used TProof::VerifyDataSet and -/// TProofLite::VerifyDataSet - -Int_t TProof::VerifyDataSetParallel(const char *uri, const char *optStr) -{ - Int_t nmissingfiles = 0; - - // Let PROOF master prepare node-files map - SetParameter("PROOF_FilesToProcess", Form("dataset:%s", uri)); - - // Use TPacketizerFile - TString oldpack; - if (TProof::GetParameter(GetInputList(), "PROOF_Packetizer", oldpack) != 0) oldpack = ""; - SetParameter("PROOF_Packetizer", "TPacketizerFile"); - - // Add dataset name - SetParameter("PROOF_VerifyDataSet", uri); - // Add options - SetParameter("PROOF_VerifyDataSetOption", optStr); - SetParameter("PROOF_SavePartialResults", (Int_t)0); - Int_t oldifiip = -1; - if (TProof::GetParameter(GetInputList(), "PROOF_IncludeFileInfoInPacket", oldifiip) != 0) oldifiip = -1; - SetParameter("PROOF_IncludeFileInfoInPacket", (Int_t)1); - - // TO DO : figure out mss and stageoption - const char* mss=""; - SetParameter("PROOF_MSS", mss); - const char* stageoption=""; - SetParameter("PROOF_StageOption", stageoption); - - // Process verification in parallel - Process("TSelVerifyDataSet", (Long64_t) 1); - - // Restore packetizer - if (!oldpack.IsNull()) - SetParameter("PROOF_Packetizer", oldpack); - else - DeleteParameters("PROOF_Packetizer"); - - // Delete or restore parameters - DeleteParameters("PROOF_FilesToProcess"); - DeleteParameters("PROOF_VerifyDataSet"); - DeleteParameters("PROOF_VerifyDataSetOption"); - DeleteParameters("PROOF_MSS"); - DeleteParameters("PROOF_StageOption"); - if (oldifiip > -1) { - SetParameter("PROOF_IncludeFileInfoInPacket", oldifiip); - } else { - DeleteParameters("PROOF_IncludeFileInfoInPacket"); - } - DeleteParameters("PROOF_SavePartialResults"); - - // Merge outputs - Int_t nopened = 0; - Int_t ntouched = 0; - Bool_t changed_ds = kFALSE; - - TIter nxtout(GetOutputList()); - TObject* obj; - TList *lfiindout = new TList; - while ((obj = nxtout())) { - TList *l = dynamic_cast(obj); - if (l && TString(l->GetName()).BeginsWith("PROOF_ListFileInfos_")) { - TIter nxt(l); - TFileInfo *fiindout = 0; - while ((fiindout = (TFileInfo*) nxt())) { - lfiindout->Add(fiindout); - } - } - // Add up number of disppeared files - TParameter* pdisappeared = dynamic_cast*>(obj); - if ( pdisappeared && TString(pdisappeared->GetName()).BeginsWith("PROOF_NoFilesDisppeared_")) { - nmissingfiles += pdisappeared->GetVal(); - } - TParameter* pnopened = dynamic_cast*>(obj); - if (pnopened && TString(pnopened->GetName()).BeginsWith("PROOF_NoFilesOpened_")) { - nopened += pnopened->GetVal(); - } - TParameter* pntouched = dynamic_cast*>(obj); - if (pntouched && TString(pntouched->GetName()).BeginsWith("PROOF_NoFilesTouched_")) { - ntouched += pntouched->GetVal(); - } - TParameter* pchanged_ds = dynamic_cast*>(obj); - if (pchanged_ds && TString(pchanged_ds->GetName()).BeginsWith("PROOF_DataSetChanged_")) { - if (pchanged_ds->GetVal() == kTRUE) changed_ds = kTRUE; - } - } - - Info("VerifyDataSetParallel", "%s: changed? %d (# files opened = %d, # files touched = %d," - " # missing files = %d)", - uri, changed_ds, nopened, ntouched, nmissingfiles); - // Done - return nmissingfiles; -} - -//////////////////////////////////////////////////////////////////////////////// -/// returns a map of the quotas of all groups - -TMap *TProof::GetDataSetQuota(const char* optStr) -{ - if (IsLite()) { - Info("UploadDataSet", "Lite-session: functionality not implemented"); - return (TMap *)0; - } - - TMessage mess(kPROOF_DATASETS); - mess << Int_t(kGetQuota); - mess << TString(optStr?optStr:""); - Broadcast(mess); - - Collect(kActive, fCollectTimeout); - TMap *groupQuotaMap = 0; - if (fStatus < 0) { - Info("GetDataSetQuota", "could not receive quota"); - } else { - // Look in the list - TMessage *retMess = (TMessage *) fRecvMessages->First(); - if (retMess && retMess->What() == kMESS_OK) { - if (!(groupQuotaMap = (TMap*)(retMess->ReadObject(TMap::Class())))) - Error("GetDataSetQuota", "error getting quotas"); - } else - Error("GetDataSetQuota", "message not found or wrong type (%p)", retMess); - } - - return groupQuotaMap; -} - -//////////////////////////////////////////////////////////////////////////////// -/// shows the quota and usage of all groups -/// if opt contains "U" shows also distribution of usage on user-level - -void TProof::ShowDataSetQuota(Option_t* opt) -{ - if (fProtocol < 15) { - Info("ShowDataSetQuota", - "functionality not available: the server does not have dataset support"); - return; - } - - if (IsLite()) { - Info("UploadDataSet", "Lite-session: functionality not implemented"); - return; - } - - TMessage mess(kPROOF_DATASETS); - mess << Int_t(kShowQuota); - mess << TString(opt?opt:""); - Broadcast(mess); - - Collect(); - if (fStatus != 0) - Error("ShowDataSetQuota", "error receiving quota information"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// If in active in a monitor set ready state - -void TProof::InterruptCurrentMonitor() -{ - if (fCurrentMonitor) - fCurrentMonitor->Interrupt(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that the worker identified by the ordinal number 'ord' is -/// in the active list. The request will be forwarded to the master -/// in direct contact with the worker. If needed, this master will move -/// the worker from the inactive to the active list and rebuild the list -/// of unique workers. -/// Use ord = "*" to activate all inactive workers. -/// The string 'ord' can also be a comma-separated list of ordinal numbers the -/// status of which will be modified at once. -/// Return <0 if something went wrong (-2 if at least one worker was not found) -/// or the number of workers with status change (on master; 0 on client). - -Int_t TProof::ActivateWorker(const char *ord, Bool_t save) -{ - return ModifyWorkerLists(ord, kTRUE, save); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove the worker identified by the ordinal number 'ord' from the -/// the active list. The request will be forwarded to the master -/// in direct contact with the worker. If needed, this master will move -/// the worker from the active to the inactive list and rebuild the list -/// of unique workers. -/// Use ord = "*" to deactivate all active workers. -/// The string 'ord' can also be a comma-separated list of ordinal numbers the -/// status of which will be modified at once. -/// Return <0 if something went wrong (-2 if at least one worker was not found) -/// or the number of workers with status change (on master; 0 on client). - -Int_t TProof::DeactivateWorker(const char *ord, Bool_t save) -{ - return ModifyWorkerLists(ord, kFALSE, save); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Modify the worker active/inactive list by making the worker identified by -/// the ordinal number 'ord' active (add == TRUE) or inactive (add == FALSE). -/// The string 'ord' can also be a comma-separated list of ordinal numbers the -/// status of which will be modified at once. -/// If needed, the request will be forwarded to the master in direct contact -/// with the worker. The end-master will move the worker from one list to the -/// other active and rebuild the list of unique active workers. -/// Use ord = "*" to deactivate all active workers. -/// If save is TRUE the current active list is saved before any modification is -/// done; re-running with ord = "restore" restores the saved list -/// Return <0 if something went wrong (-2 if at least one worker was not found) -/// or the number of workers with status change (on master; 0 on client). - -Int_t TProof::ModifyWorkerLists(const char *ord, Bool_t add, Bool_t save) -{ - // Make sure the input make sense - if (!ord || strlen(ord) <= 0) { - Info("ModifyWorkerLists", - "an ordinal number - e.g. \"0.4\" or \"*\" for all - is required as input"); - return -1; - } - if (gDebug > 0) - Info("ModifyWorkerLists", "ord: '%s' (add: %d, save: %d)", ord, add, save); - - Int_t nwc = 0; - Bool_t restoring = !strcmp(ord, "restore") ? kTRUE : kFALSE; - if (IsEndMaster()) { - if (restoring) { - // We are asked to restore the previous settings - nwc = RestoreActiveList(); - } else { - if (save) SaveActiveList(); - } - } - - Bool_t allord = strcmp(ord, "*") ? kFALSE : kTRUE; - - // Check if this is for us - if (TestBit(TProof::kIsMaster) && gProofServ) { - if (!allord && - strncmp(ord, gProofServ->GetOrdinal(), strlen(gProofServ->GetOrdinal()))) - return 0; - } - - Bool_t fw = kTRUE; // Whether to forward one step down - Bool_t rs = kFALSE; // Whether to rescan for unique workers - - // Appropriate list pointing - TList *in = (add) ? fInactiveSlaves : fActiveSlaves; - TList *out = (add) ? fActiveSlaves : fInactiveSlaves; - - if (IsEndMaster() && !restoring) { - // Create the hash list of ordinal numbers - THashList *ords = 0; - if (!allord) { - ords = new THashList(); - const char *masterord = (gProofServ) ? gProofServ->GetOrdinal() : "0"; - TString oo(ord), o; - Int_t from = 0; - while(oo.Tokenize(o, from, ",")) - if (o.BeginsWith(masterord)) ords->Add(new TObjString(o)); - } - // We do not need to send forward - fw = kFALSE; - // Look for the worker in the initial list - TObject *os = 0; - TSlave *wrk = 0; - if (in->GetSize() > 0) { - TIter nxw(in); - while ((wrk = (TSlave *) nxw())) { - os = 0; - if (allord || (ords && (os = ords->FindObject(wrk->GetOrdinal())))) { - // Add it to the final list - if (!out->FindObject(wrk)) { - out->Add(wrk); - if (add) - fActiveMonitor->Add(wrk->GetSocket()); - } - // Remove it from the initial list - in->Remove(wrk); - if (!add) { - fActiveMonitor->Remove(wrk->GetSocket()); - wrk->SetStatus(TSlave::kInactive); - } else - wrk->SetStatus(TSlave::kActive); - // Count - nwc++; - // Nothing to forward (ord is unique) - fw = kFALSE; - // Rescan for unique workers (active list modified) - rs = kTRUE; - // We may be done, if not option 'all' - if (!allord && ords) { - if (os) ords->Remove(os); - if (ords->GetSize() == 0) break; - SafeDelete(os); - } - } - } - } - // If some worker not found, notify it if at the end - if (!fw && ords && ords->GetSize() > 0) { - TString oo; - TIter nxo(ords); - while ((os = nxo())) { - TIter nxw(out); - while ((wrk = (TSlave *) nxw())) - if (!strcmp(os->GetName(), wrk->GetOrdinal())) break; - if (!wrk) { - if (!oo.IsNull()) oo += ","; - oo += os->GetName(); - } - } - if (!oo.IsNull()) { - Warning("ModifyWorkerLists", "worker(s) '%s' not found!", oo.Data()); - nwc = -2; - } - } - // Cleanup hash list - if (ords) { - ords->Delete(); - SafeDelete(ords); - } - } - - // Rescan for unique workers - if (rs) - FindUniqueSlaves(); - - // Forward the request one step down, if needed - Int_t action = (add) ? (Int_t) kActivateWorker : (Int_t) kDeactivateWorker; - if (fw) { - if (fProtocol > 32) { - TMessage mess(kPROOF_WORKERLISTS); - mess << action << TString(ord); - Broadcast(mess); - Collect(kActive, fCollectTimeout); - if (fStatus != 0) { - nwc = (fStatus < nwc) ? fStatus : nwc; - if (fStatus == -2) { - if (gDebug > 0) - Warning("ModifyWorkerLists", "request not completely full filled"); - } else { - Error("ModifyWorkerLists", "request failed"); - } - } - } else { - TString oo(ord), o; - if (oo.Contains(",")) - Warning("ModifyWorkerLists", "block request not supported by server: splitting into pieces ..."); - Int_t from = 0; - while(oo.Tokenize(o, from, ",")) { - TMessage mess(kPROOF_WORKERLISTS); - mess << action << o; - Broadcast(mess); - Collect(kActive, fCollectTimeout); - } - } - } - // Done - return nwc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save current list of active workers - -void TProof::SaveActiveList() -{ - if (!fActiveSlavesSaved.IsNull()) fActiveSlavesSaved = ""; - if (fInactiveSlaves->GetSize() == 0) { - fActiveSlavesSaved = "*"; - } else { - TIter nxw(fActiveSlaves); - TSlave *wk = 0; - while ((wk = (TSlave *)nxw())) { fActiveSlavesSaved += TString::Format("%s,", wk->GetOrdinal()); } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Restore saved list of active workers - -Int_t TProof::RestoreActiveList() -{ - // Clear the current active list - DeactivateWorker("*", kFALSE); - // Restore the previous active list - if (!fActiveSlavesSaved.IsNull()) - return ActivateWorker(fActiveSlavesSaved, kFALSE); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Start a PROOF session on a specific cluster. If cluster is 0 (the -/// default) then the PROOF Session Viewer GUI pops up and 0 is returned. -/// If cluster is "lite://" we start a PROOF-lite session. -/// If cluster is "" (empty string) then we connect to the cluster specified -/// by 'Proof.LocalDefault', defaulting to "lite://". -/// If cluster is "pod://" (case insensitive), then we connect to a PROOF cluster -/// managed by PROOF on Demand (PoD, http://pod.gsi.de ). -/// Via conffile a specific PROOF config file in the confir directory can be specified. -/// Use loglevel to set the default loging level for debugging. -/// The appropriate instance of TProofMgr is created, if not -/// yet existing. The instantiated TProof object is returned. -/// Use TProof::cd() to switch between PROOF sessions. -/// For more info on PROOF see the TProof ctor. - -TProof *TProof::Open(const char *cluster, const char *conffile, - const char *confdir, Int_t loglevel) -{ - const char *pn = "TProof::Open"; - - // Make sure libProof and dependents are loaded and TProof can be created, - // dependents are loaded via the information in the [system].rootmap file - if (!cluster) { - - TPluginManager *pm = gROOT->GetPluginManager(); - if (!pm) { - ::Error(pn, "plugin manager not found"); - return 0; - } - - if (gROOT->IsBatch()) { - ::Error(pn, "we are in batch mode, cannot show PROOF Session Viewer"); - return 0; - } - // start PROOF Session Viewer - TPluginHandler *sv = pm->FindHandler("TSessionViewer", ""); - if (!sv) { - ::Error(pn, "no plugin found for TSessionViewer"); - return 0; - } - if (sv->LoadPlugin() == -1) { - ::Error(pn, "plugin for TSessionViewer could not be loaded"); - return 0; - } - sv->ExecPlugin(0); - return 0; - - } else { - - TString clst(cluster); - - // Check for PoD cluster - if (PoDCheckUrl( &clst ) < 0) return 0; - - if (clst.BeginsWith("workers=")) clst.Insert(0, "lite:///?"); - if (clst.BeginsWith("tunnel=")) clst.Insert(0, "/?"); - - // Parse input URL - TUrl u(clst); - - // *** GG, 060711: this does not seem to work any more (at XrdClient level) - // *** to be investigated (it is not really needed; static tunnels work). - // Dynamic tunnel: - // Parse any tunning info ("/?tunnel=[:]tunnel_port) - TString opts(u.GetOptions()); - if (!opts.IsNull()) { - Int_t it = opts.Index("tunnel="); - if (it != kNPOS) { - TString sport = opts(it + strlen("tunnel="), opts.Length()); - TString host("127.0.0.1"); - Int_t port = -1; - Int_t ic = sport.Index(":"); - if (ic != kNPOS) { - // Isolate the host - host = sport(0, ic); - sport.Remove(0, ic + 1); - } - if (!sport.IsDigit()) { - // Remove the non digit part - TRegexp re("[^0-9]"); - Int_t ind = sport.Index(re); - if (ind != kNPOS) - sport.Remove(ind); - } - // Set the port - if (sport.IsDigit()) - port = sport.Atoi(); - if (port > 0) { - // Set the relevant variables - ::Info("TProof::Open","using tunnel at %s:%d", host.Data(), port); - gEnv->SetValue("XNet.SOCKS4Host", host); - gEnv->SetValue("XNet.SOCKS4Port", port); - } else { - // Warn parsing problems - ::Warning("TProof::Open", - "problems parsing tunnelling info from options: %s", opts.Data()); - } - } - } - - // Find out if we are required to attach to a specific session - Int_t locid = -1; - Bool_t create = kFALSE; - if (opts.Length() > 0) { - if (opts.BeginsWith("N",TString::kIgnoreCase)) { - create = kTRUE; - opts.Remove(0,1); - u.SetOptions(opts); - } else if (opts.IsDigit()) { - locid = opts.Atoi(); - } - } - - // Attach-to or create the appropriate manager - TProofMgr *mgr = TProofMgr::Create(u.GetUrl()); - - TProof *proof = 0; - if (mgr && mgr->IsValid()) { - - // If XProofd we always attempt an attach first (unless - // explicitly not requested). - Bool_t attach = (create || mgr->IsProofd() || mgr->IsLite()) ? kFALSE : kTRUE; - if (attach) { - TProofDesc *d = 0; - if (locid < 0) - // Get the list of sessions - d = (TProofDesc *) mgr->QuerySessions("")->First(); - else - d = (TProofDesc *) mgr->GetProofDesc(locid); - if (d) { - proof = (TProof*) mgr->AttachSession(d); - if (!proof || !proof->IsValid()) { - if (locid) - ::Error(pn, "new session could not be attached"); - SafeDelete(proof); - } - } - } - - // start the PROOF session - if (!proof) { - proof = (TProof*) mgr->CreateSession(conffile, confdir, loglevel); - if (!proof || !proof->IsValid()) { - ::Error(pn, "new session could not be created"); - SafeDelete(proof); - } - } - } - return proof; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get instance of the effective manager for 'url' -/// Return 0 on failure. - -TProofMgr *TProof::Mgr(const char *url) -{ - if (!url) - return (TProofMgr *)0; - - // Attach or create the relevant instance - return TProofMgr::Create(url); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Wrapper around TProofMgr::Reset(...). - -void TProof::Reset(const char *url, Bool_t hard) -{ - if (url) { - TProofMgr *mgr = TProof::Mgr(url); - if (mgr && mgr->IsValid()) - mgr->Reset(hard); - else - ::Error("TProof::Reset", - "unable to initialize a valid manager instance"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get environemnt variables. - -const TList *TProof::GetEnvVars() -{ - return fgProofEnvList; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add an variable to the list of environment variables passed to proofserv -/// on the master and slaves - -void TProof::AddEnvVar(const char *name, const char *value) -{ - if (gDebug > 0) ::Info("TProof::AddEnvVar","%s=%s", name, value); - - if (fgProofEnvList == 0) { - // initialize the list if needed - fgProofEnvList = new TList; - fgProofEnvList->SetOwner(); - } else { - // replace old entries with the same name - TObject *o = fgProofEnvList->FindObject(name); - if (o != 0) { - fgProofEnvList->Remove(o); - } - } - fgProofEnvList->Add(new TNamed(name, value)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove an variable from the list of environment variables passed to proofserv -/// on the master and slaves - -void TProof::DelEnvVar(const char *name) -{ - if (fgProofEnvList == 0) return; - - TObject *o = fgProofEnvList->FindObject(name); - if (o != 0) { - fgProofEnvList->Remove(o); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear the list of environment variables passed to proofserv -/// on the master and slaves - -void TProof::ResetEnvVars() -{ - if (fgProofEnvList == 0) return; - - SafeDelete(fgProofEnvList); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save information about the worker set in the file .workers in the working -/// dir. Called each time there is a change in the worker setup, e.g. by -/// TProof::MarkBad(). - -void TProof::SaveWorkerInfo() -{ - // We must be masters - if (TestBit(TProof::kIsClient)) - return; - - // We must have a server defined - if (!gProofServ) { - Error("SaveWorkerInfo","gProofServ undefined"); - return; - } - - // The relevant lists must be defined - if (!fSlaves && !fBadSlaves) { - Warning("SaveWorkerInfo","all relevant worker lists is undefined"); - return; - } - - // Create or truncate the file first - TString fnwrk = gSystem->GetDirName(gProofServ->GetSessionDir())+"/.workers"; - FILE *fwrk = fopen(fnwrk.Data(),"w"); - if (!fwrk) { - Error("SaveWorkerInfo", - "cannot open %s for writing (errno: %d)", fnwrk.Data(), errno); - return; - } - - // Do we need to register an additional line for another log? - TString addlogext; - TString addLogTag; - if (gSystem->Getenv("PROOF_ADDITIONALLOG")) { - addlogext = gSystem->Getenv("PROOF_ADDITIONALLOG"); - TPMERegexp reLogTag("^__(.*)__\\.log"); // $ - if (reLogTag.Match(addlogext) == 2) { - addLogTag = reLogTag[1]; - } - else { - addLogTag = "+++"; - } - if (gDebug > 0) - Info("SaveWorkerInfo", "request for additional line with ext: '%s'", addlogext.Data()); - } - - // Used to eliminate datetime and PID from workdir to obtain log file name - TPMERegexp re("(.*?)-[0-9]+-[0-9]+$"); - - // Loop over the list of workers (active is any worker not flagged as bad) - TIter nxa(fSlaves); - TSlave *wrk = 0; - TString logfile; - while ((wrk = (TSlave *) nxa())) { - Int_t status = (fBadSlaves && fBadSlaves->FindObject(wrk)) ? 0 : 1; - logfile = wrk->GetWorkDir(); - if (re.Match(logfile) == 2) logfile = re[1]; - else continue; // invalid (should not happen) - // Write out record for this worker - fprintf(fwrk,"%s@%s:%d %d %s %s.log\n", - wrk->GetUser(), wrk->GetName(), wrk->GetPort(), status, - wrk->GetOrdinal(), logfile.Data()); - // Additional line, if required - if (addlogext.Length() > 0) { - fprintf(fwrk,"%s@%s:%d %d %s(%s) %s.%s\n", - wrk->GetUser(), wrk->GetName(), wrk->GetPort(), status, - wrk->GetOrdinal(), addLogTag.Data(), logfile.Data(), addlogext.Data()); - } - - } - - // Loop also over the list of bad workers (if they failed to startup they are not in - // the overall list - TIter nxb(fBadSlaves); - while ((wrk = (TSlave *) nxb())) { - logfile = wrk->GetWorkDir(); - if (re.Match(logfile) == 2) logfile = re[1]; - else continue; // invalid (should not happen) - if (!fSlaves->FindObject(wrk)) { - // Write out record for this worker - fprintf(fwrk,"%s@%s:%d 0 %s %s.log\n", - wrk->GetUser(), wrk->GetName(), wrk->GetPort(), - wrk->GetOrdinal(), logfile.Data()); - } - } - - // Eventually loop over the list of gracefully terminated workers: we'll get - // logfiles from those workers as well. They'll be shown with a special - // status of "2" - TIter nxt(fTerminatedSlaveInfos); - TSlaveInfo *sli; - while (( sli = (TSlaveInfo *)nxt() )) { - logfile = sli->GetDataDir(); - if (re.Match(logfile) == 2) logfile = re[1]; - else continue; // invalid (should not happen) - fprintf(fwrk, "%s 2 %s %s.log\n", - sli->GetName(), sli->GetOrdinal(), logfile.Data()); - // Additional line, if required - if (addlogext.Length() > 0) { - fprintf(fwrk, "%s 2 %s(%s) %s.%s\n", - sli->GetName(), sli->GetOrdinal(), addLogTag.Data(), - logfile.Data(), addlogext.Data()); - } - } - - // Close file - fclose(fwrk); - - // We are done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the value from the specified parameter from the specified collection. -/// Returns -1 in case of error (i.e. list is 0, parameter does not exist -/// or value type does not match), 0 otherwise. - -Int_t TProof::GetParameter(TCollection *c, const char *par, TString &value) -{ - TObject *obj = c ? c->FindObject(par) : (TObject *)0; - if (obj) { - TNamed *p = dynamic_cast(obj); - if (p) { - value = p->GetTitle(); - return 0; - } - } - return -1; - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the value from the specified parameter from the specified collection. -/// Returns -1 in case of error (i.e. list is 0, parameter does not exist -/// or value type does not match), 0 otherwise. - -Int_t TProof::GetParameter(TCollection *c, const char *par, Int_t &value) -{ - TObject *obj = c ? c->FindObject(par) : (TObject *)0; - if (obj) { - TParameter *p = dynamic_cast*>(obj); - if (p) { - value = p->GetVal(); - return 0; - } - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the value from the specified parameter from the specified collection. -/// Returns -1 in case of error (i.e. list is 0, parameter does not exist -/// or value type does not match), 0 otherwise. - -Int_t TProof::GetParameter(TCollection *c, const char *par, Long_t &value) -{ - TObject *obj = c ? c->FindObject(par) : (TObject *)0; - if (obj) { - TParameter *p = dynamic_cast*>(obj); - if (p) { - value = p->GetVal(); - return 0; - } - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the value from the specified parameter from the specified collection. -/// Returns -1 in case of error (i.e. list is 0, parameter does not exist -/// or value type does not match), 0 otherwise. - -Int_t TProof::GetParameter(TCollection *c, const char *par, Long64_t &value) -{ - TObject *obj = c ? c->FindObject(par) : (TObject *)0; - if (obj) { - TParameter *p = dynamic_cast*>(obj); - if (p) { - value = p->GetVal(); - return 0; - } - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the value from the specified parameter from the specified collection. -/// Returns -1 in case of error (i.e. list is 0, parameter does not exist -/// or value type does not match), 0 otherwise. - -Int_t TProof::GetParameter(TCollection *c, const char *par, Double_t &value) -{ - TObject *obj = c ? c->FindObject(par) : (TObject *)0; - if (obj) { - TParameter *p = dynamic_cast*>(obj); - if (p) { - value = p->GetVal(); - return 0; - } - } - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that dataset is in the form to be processed. This may mean -/// retrieving the relevant info from the dataset manager or from the -/// attached input list. -/// Returns 0 on success, -1 on error - -Int_t TProof::AssertDataSet(TDSet *dset, TList *input, - TDataSetManager *mgr, TString &emsg) -{ - emsg = ""; - - // We must have something to process - if (!dset || !input || !mgr) { - emsg.Form("invalid inputs (%p, %p, %p)", dset, input, mgr); - return -1; - } - - TList *datasets = new TList; - TFileCollection *dataset = 0; - TString lookupopt; - TString dsname(dset->GetName()); - - // First extract the "entry list" part on the global name, if any - TString dsns(dsname), enlname; - Ssiz_t eli = dsns.Index("?enl="); - if (eli != kNPOS) { - enlname = dsns(eli + strlen("?enl="), dsns.Length()); - dsns.Remove(eli, dsns.Length()-eli); - } - - // The dataset maybe in the form of a TFileCollection in the input list - if (dsname.BeginsWith("TFileCollection:")) { - // Isolate the real name - dsname.ReplaceAll("TFileCollection:", ""); - // Get the object - dataset = (TFileCollection *) input->FindObject(dsname); - if (!dataset) { - emsg.Form("TFileCollection %s not found in input list", dset->GetName()); - return -1; - } - // Remove from everywhere - input->RecursiveRemove(dataset); - // Add it to the local list - datasets->Add(new TPair(dataset, new TObjString(enlname.Data()))); - // Make sure we lookup everything (unless the client or the administrator - // required something else) - if (TProof::GetParameter(input, "PROOF_LookupOpt", lookupopt) != 0) { - lookupopt = gEnv->GetValue("Proof.LookupOpt", "all"); - input->Add(new TNamed("PROOF_LookupOpt", lookupopt.Data())); - } - } - - // This is the name we parse for additional specifications, such directory - // and object name; for multiple datasets we assume that the directory and - // and object name are the same for all datasets - TString dsnparse; - // The received message included an empty dataset, with only the name - // defined: assume that a dataset, stored on the PROOF master by that - // name, should be processed. - if (!dataset) { - - TFileCollection *fc = nullptr; - - // Check if the entry list and dataset name are valid. If they have spaces, - // commas, or pipes, they are not considered as valid and we revert to the - // "multiple datasets" case - TRegexp rg("[, |]"); - Bool_t validEnl = (enlname.Index(rg) == kNPOS) ? kTRUE : kFALSE; - Bool_t validSdsn = (dsns.Index(rg) == kNPOS) ? kTRUE : kFALSE; - - if (validEnl && validSdsn && (( fc = mgr->GetDataSet(dsns) ))) { - - // - // String corresponds to ONE dataset only - // - - TIter nxfi(fc->GetList()); - TFileInfo *fi; - while (( fi = (TFileInfo *)nxfi() )) - fi->SetTitle(dsns.Data()); - dataset = fc; - dsnparse = dsns; // without entry list - - // Adds the entry list (or empty string if not specified) - datasets->Add( new TPair(dataset, new TObjString( enlname.Data() )) ); - - } else { - - // - // String does NOT correspond to one dataset: check if many datasets - // were specified instead - // - - dsns = dsname.Data(); - TString dsn1; - Int_t from1 = 0; - while (dsns.Tokenize(dsn1, from1, "[, ]")) { - TString dsn2; - Int_t from2 = 0; - while (dsn1.Tokenize(dsn2, from2, "|")) { - enlname = ""; - Int_t ienl = dsn2.Index("?enl="); - if (ienl != kNPOS) { - enlname = dsn2(ienl + 5, dsn2.Length()); - dsn2.Remove(ienl); - } - if ((fc = mgr->GetDataSet(dsn2.Data()))) { - // Save dataset name in TFileInfo's title to use it in TDset - TIter nxfi(fc->GetList()); - TFileInfo *fi; - while ((fi = (TFileInfo *) nxfi())) { fi->SetTitle(dsn2.Data()); } - dsnparse = dsn2; - if (!dataset) { - // This is our dataset - dataset = fc; - } else { - // Add it to the dataset - dataset->Add(fc); - SafeDelete(fc); - } - } - } - // The dataset name(s) in the first element - if (dataset) { - if (dataset->GetList()->First()) - ((TFileInfo *)(dataset->GetList()->First()))->SetTitle(dsn1.Data()); - // Add it to the local list - datasets->Add(new TPair(dataset, new TObjString(enlname.Data()))); - } - // Reset the pointer - dataset = 0; - } - - } - - // - // At this point the dataset(s) to be processed, if any, are found in the - // "datasets" variable - // - - if (!datasets || datasets->GetSize() <= 0) { - emsg.Form("no dataset(s) found on the master corresponding to: %s", dsname.Data()); - return -1; - } else { - // Make 'dataset' to point to the first one in the list - if (!(dataset = (TFileCollection *) ((TPair *)(datasets->First()))->Key())) { - emsg.Form("dataset pointer is null: corruption? - aborting"); - return -1; - } - } - // Apply the lookup option requested by the client or the administartor - // (by default we trust the information in the dataset) - if (TProof::GetParameter(input, "PROOF_LookupOpt", lookupopt) != 0) { - lookupopt = gEnv->GetValue("Proof.LookupOpt", "stagedOnly"); - input->Add(new TNamed("PROOF_LookupOpt", lookupopt.Data())); - } - } else { - // We were given a named, single, TFileCollection - dsnparse = dsname; - } - - // Logic for the subdir/obj names: try first to see if the dataset name contains - // some info; if not check the settings in the TDSet object itself; if still empty - // check the default tree name / path in the TFileCollection object; if still empty - // use the default as the flow will determine - TString dsTree; - // Get the [subdir/]tree, if any - mgr->ParseUri(dsnparse.Data(), 0, 0, 0, &dsTree); - if (dsTree.IsNull()) { - // Use what we have in the original dataset; we need this to locate the - // meta data information - dsTree += dset->GetDirectory(); - dsTree += dset->GetObjName(); - } - if (!dsTree.IsNull() && dsTree != "/") { - TString tree(dsTree); - Int_t idx = tree.Index("/"); - if (idx != kNPOS) { - TString dir = tree(0, idx+1); - tree.Remove(0, idx); - dset->SetDirectory(dir); - } - dset->SetObjName(tree); - } else { - // Use the default obj name from the TFileCollection - dsTree = dataset->GetDefaultTreeName(); - } - - // Pass dataset server mapping instructions, if any - TList *srvmapsref = TDataSetManager::GetDataSetSrvMaps(); - TList *srvmapslist = srvmapsref; - TString srvmaps; - if (TProof::GetParameter(input, "PROOF_DataSetSrvMaps", srvmaps) == 0) { - srvmapslist = TDataSetManager::ParseDataSetSrvMaps(srvmaps); - if (gProofServ) { - TString msg; - if (srvmapsref && !srvmapslist) { - msg.Form("+++ Info: dataset server mapping(s) DISABLED by user"); - } else if (srvmapsref && srvmapslist && srvmapslist != srvmapsref) { - msg.Form("+++ Info: dataset server mapping(s) modified by user"); - } else if (!srvmapsref && srvmapslist) { - msg.Form("+++ Info: dataset server mapping(s) added by user"); - } - gProofServ->SendAsynMessage(msg.Data()); - } - } - - // Flag multi-datasets - if (datasets->GetSize() > 1) dset->SetBit(TDSet::kMultiDSet); - // Loop over the list of datasets - TList *listOfMissingFiles = new TList; - TEntryList *entrylist = 0; - TPair *pair = 0; - TIter nxds(datasets); - while ((pair = (TPair *) nxds())) { - // File Collection - dataset = (TFileCollection *) pair->Key(); - // Entry list, if any - TEntryList *enl = 0; - TObjString *os = (TObjString *) pair->Value(); - if (strlen(os->GetName())) { - if (!(enl = dynamic_cast(input->FindObject(os->GetName())))) { - if (gProofServ) - gProofServ->SendAsynMessage(TString::Format("+++ Warning:" - " entry list %s not found", os->GetName())); - } - if (enl && (!(enl->GetLists()) || enl->GetLists()->GetSize() <= 0)) { - if (gProofServ) - gProofServ->SendAsynMessage(TString::Format("+++ Warning:" - " no sub-lists in entry-list!")); - } - } - TList *missingFiles = new TList; - TSeqCollection* files = dataset->GetList(); - if (gDebug > 0) files->Print(); - Bool_t availableOnly = (lookupopt != "all") ? kTRUE : kFALSE; - if (dset->TestBit(TDSet::kMultiDSet)) { - TDSet *ds = new TDSet(dataset->GetName(), dset->GetObjName(), dset->GetDirectory()); - ds->SetSrvMaps(srvmapslist); - if (!ds->Add(files, dsTree, availableOnly, missingFiles)) { - emsg.Form("error integrating dataset %s", dataset->GetName()); - continue; - } - // Add the TDSet object to the multi-dataset - dset->Add(ds); - // Add entry list if any - if (enl) ds->SetEntryList(enl); - } else { - dset->SetSrvMaps(srvmapslist); - if (!dset->Add(files, dsTree, availableOnly, missingFiles)) { - emsg.Form("error integrating dataset %s", dataset->GetName()); - continue; - } - if (enl) entrylist = enl; - } - if (missingFiles) { - // The missing files objects have to be removed from the dataset - // before delete. - TIter next(missingFiles); - TObject *file; - while ((file = next())) { - dataset->GetList()->Remove(file); - listOfMissingFiles->Add(file); - } - missingFiles->SetOwner(kFALSE); - missingFiles->Clear(); - } - SafeDelete(missingFiles); - } - // Cleanup; we need to do this because pairs do no delete their content - nxds.Reset(); - while ((pair = (TPair *) nxds())) { - if (pair->Key()) delete pair->Key(); - if (pair->Value()) delete pair->Value(); - } - datasets->SetOwner(kTRUE); - SafeDelete(datasets); - - // Cleanup the server mapping list, if created by the user - if (srvmapslist && srvmapslist != srvmapsref) { - srvmapslist->SetOwner(kTRUE); - SafeDelete(srvmapslist); - } - - // Set the global entrylist, if required - if (entrylist) dset->SetEntryList(entrylist); - - // Make sure it will be sent back merged with other similar lists created - // during processing; this list will be transferred by the player to the - // output list, once the latter has been created (see TProofPlayerRemote::Process) - if (listOfMissingFiles && listOfMissingFiles->GetSize() > 0) { - listOfMissingFiles->SetName("MissingFiles"); - input->Add(listOfMissingFiles); - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save input data file from 'cachedir' into the sandbox or create a the file -/// with input data objects - -Int_t TProof::SaveInputData(TQueryResult *qr, const char *cachedir, TString &emsg) -{ - TList *input = 0; - - // We must have got something to process - if (!qr || !(input = qr->GetInputList()) || - !cachedir || strlen(cachedir) <= 0) return 0; - - // There must be some input data or input data file - TNamed *data = (TNamed *) input->FindObject("PROOF_InputDataFile"); - TList *inputdata = (TList *) input->FindObject("PROOF_InputData"); - if (!data && !inputdata) return 0; - // Default dstination filename - if (!data) - input->Add((data = new TNamed("PROOF_InputDataFile", kPROOF_InputDataFile))); - - TString dstname(data->GetTitle()), srcname; - Bool_t fromcache = kFALSE; - if (dstname.BeginsWith("cache:")) { - fromcache = kTRUE; - dstname.ReplaceAll("cache:", ""); - srcname.Form("%s/%s", cachedir, dstname.Data()); - if (gSystem->AccessPathName(srcname)) { - emsg.Form("input data file not found in cache (%s)", srcname.Data()); - return -1; - } - } - - // If from cache, just move the cache file - if (fromcache) { - if (gSystem->CopyFile(srcname, dstname, kTRUE) != 0) { - emsg.Form("problems copying %s to %s", srcname.Data(), dstname.Data()); - return -1; - } - } else { - // Create the file - if (inputdata && inputdata->GetSize() > 0) { - TFile *f = TFile::Open(dstname.Data(), "RECREATE"); - if (f) { - f->cd(); - inputdata->Write(); - f->Close(); - delete f; - } else { - emsg.Form("could not create %s", dstname.Data()); - return -1; - } - } else { - emsg.Form("no input data!"); - return -1; - } - } - ::Info("TProof::SaveInputData", "input data saved to %s", dstname.Data()); - - // Save the file name and clean up the data list - data->SetTitle(dstname); - if (inputdata) { - input->Remove(inputdata); - inputdata->SetOwner(); - delete inputdata; - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send the input data file to the workers - -Int_t TProof::SendInputData(TQueryResult *qr, TProof *p, TString &emsg) -{ - TList *input = 0; - - // We must have got something to process - if (!qr || !(input = qr->GetInputList())) return 0; - - // There must be some input data or input data file - TNamed *inputdata = (TNamed *) input->FindObject("PROOF_InputDataFile"); - if (!inputdata) return 0; - - TString fname(inputdata->GetTitle()); - if (gSystem->AccessPathName(fname)) { - emsg.Form("input data file not found in sandbox (%s)", fname.Data()); - return -1; - } - - // PROOF session must available - if (!p || !p->IsValid()) { - emsg.Form("TProof object undefined or invalid: protocol error!"); - return -1; - } - - // Send to unique workers and submasters - p->BroadcastFile(fname, TProof::kBinary, "cache"); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the input data from the file defined in the input list - -Int_t TProof::GetInputData(TList *input, const char *cachedir, TString &emsg) -{ - // We must have got something to process - if (!input || !cachedir || strlen(cachedir) <= 0) return 0; - - // There must be some input data or input data file - TNamed *inputdata = (TNamed *) input->FindObject("PROOF_InputDataFile"); - if (!inputdata) return 0; - - TString fname; - fname.Form("%s/%s", cachedir, inputdata->GetTitle()); - if (gSystem->AccessPathName(fname)) { - emsg.Form("input data file not found in cache (%s)", fname.Data()); - return -1; - } - - // List of added objects (for proper cleaning ...) - TList *added = new TList; - added->SetName("PROOF_InputObjsFromFile"); - // Read the input data into the input list - TFile *f = TFile::Open(fname.Data()); - if (f) { - TList *keys = (TList *) f->GetListOfKeys(); - if (!keys) { - emsg.Form("could not get list of object keys from file"); - return -1; - } - TIter nxk(keys); - TKey *k = 0; - while ((k = (TKey *)nxk())) { - TObject *o = f->Get(k->GetName()); - if (o) { - input->Add(o); - added->Add(o); - } - } - // Add the file as last one - if (added->GetSize() > 0) { - added->Add(f); - input->Add(added); - } else { - // Cleanup the file now - f->Close(); - delete f; - } - } else { - emsg.Form("could not open %s", fname.Data()); - return -1; - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Start the log viewer window usign the plugin manager - -void TProof::LogViewer(const char *url, Int_t idx) -{ - if (!gROOT->IsBatch()) { - // Get the handler, if not yet done - if (!fgLogViewer) { - if ((fgLogViewer = - gROOT->GetPluginManager()->FindHandler("TProofProgressLog"))) { - if (fgLogViewer->LoadPlugin() == -1) { - fgLogViewer = 0; - ::Error("TProof::LogViewer", "cannot load the relevant plug-in"); - return; - } - } - } - if (fgLogViewer) { - // Execute the plug-in - TString u = (url && strlen(url) <= 0) ? "lite" : url; - fgLogViewer->ExecPlugin(2, u.Data(), idx); - } - } else { - if (url && strlen(url) > 0) { - ::Info("TProof::LogViewer", - "batch mode: use TProofLog *pl = TProof::Mgr(\"%s\")->GetSessionLogs(%d)", url, idx); - } else if (url && strlen(url) <= 0) { - ::Info("TProof::LogViewer", - "batch mode: use TProofLog *pl = TProof::Mgr(\"lite\")->GetSessionLogs(%d)", idx); - } else { - ::Info("TProof::LogViewer", - "batch mode: use TProofLog *pl = TProof::Mgr(\"\")->GetSessionLogs(%d)", idx); - } - } - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable/Disable the graphic progress dialog. -/// By default the dialog is enabled - -void TProof::SetProgressDialog(Bool_t on) -{ - if (on) - SetBit(kUseProgressDialog); - else - ResetBit(kUseProgressDialog); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show information about missing files during query described by 'qr' or the -/// last query if qr is null (default). -/// A short summary is printed in the end. - -void TProof::ShowMissingFiles(TQueryResult *qr) -{ - TQueryResult *xqr = (qr) ? qr : GetQueryResult(); - if (!xqr) { - Warning("ShowMissingFiles", "no (last) query found: do nothing"); - return; - } - - // Get the list, if any - TList *missing = (xqr->GetOutputList()) ? (TList *) xqr->GetOutputList()->FindObject("MissingFiles") : 0; - if (!missing) { - Info("ShowMissingFiles", "no files missing in query %s:%s", xqr->GetTitle(), xqr->GetName()); - return; - } - - Int_t nmf = 0, ncf = 0; - Long64_t msz = 0, mszzip = 0, mev = 0; - // Scan the list - TFileInfo *fi = 0; - TIter nxf(missing); - while ((fi = (TFileInfo *) nxf())) { - char status = 'M'; - if (fi->TestBit(TFileInfo::kCorrupted)) { - ncf++; - status = 'C'; - } else { - nmf++; - } - TFileInfoMeta *im = fi->GetMetaData(); - if (im) { - if (im->GetTotBytes() > 0) msz += im->GetTotBytes(); - if (im->GetZipBytes() > 0) mszzip += im->GetZipBytes(); - mev += im->GetEntries(); - Printf(" %d. (%c) %s %s %lld", ncf+nmf, status, fi->GetCurrentUrl()->GetUrl(), im->GetName(), im->GetEntries()); - } else { - Printf(" %d. (%c) %s '' -1", ncf+nmf, status, fi->GetCurrentUrl()->GetUrl()); - } - } - - // Final notification - if (msz <= 0) msz = -1; - if (mszzip <= 0) mszzip = -1; - Double_t xf = (Double_t)mev / (mev + xqr->GetEntries()) ; - if (msz > 0. || mszzip > 0.) { - Printf(" +++ %d file(s) missing, %d corrupted, i.e. %lld unprocessed events -->" - " about %.2f%% of the total (%lld bytes, %lld zipped)", - nmf, ncf, mev, xf * 100., msz, mszzip); - } else { - Printf(" +++ %d file(s) missing, %d corrupted, i.e. %lld unprocessed events -->" - " about %.2f%% of the total", nmf, ncf, mev, xf * 100.); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get a TFileCollection with the files missing in the query described by 'qr' -/// or the last query if qr is null (default). -/// Return a null pointer if none were found, for whatever reason. -/// The caller is responsible for the returned object. - -TFileCollection *TProof::GetMissingFiles(TQueryResult *qr) -{ - TFileCollection *fc = 0; - - TQueryResult *xqr = (qr) ? qr : GetQueryResult(); - if (!xqr) { - Warning("GetMissingFiles", "no (last) query found: do nothing"); - return fc; - } - - // Get the list, if any - TList *missing = (xqr->GetOutputList()) ? (TList *) xqr->GetOutputList()->FindObject("MissingFiles") : 0; - if (!missing) { - if (gDebug > 0) - Info("ShowMissingFiles", "no files missing in query %s:%s", xqr->GetTitle(), xqr->GetName()); - return fc; - } - - // Create collection: name is .m, where 'j' is the first giving a non existing name - TString fcname("unknown"); - TDSet *ds = (TDSet *) xqr->GetInputObject("TDSet"); - if (ds) { - fcname.Form("%s.m0", ds->GetName()); - Int_t j = 1; - while (gDirectory->FindObject(fcname) && j < 1000) - fcname.Form("%s.m%d", ds->GetName(), j++); - } - fc = new TFileCollection(fcname, "Missing Files"); - if (ds) fc->SetDefaultTreeName(ds->GetObjName()); - // Scan the list - TFileInfo *fi = 0; - TIter nxf(missing); - while ((fi = (TFileInfo *) nxf())) { - fc->Add((TFileInfo *) fi->Clone()); - } - fc->Update(); - // Done - return fc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable/Disable saving of the performance tree - -void TProof::SetPerfTree(const char *pf, Bool_t withWrks) -{ - if (pf && strlen(pf) > 0) { - fPerfTree = pf; - SetParameter("PROOF_StatsHist", ""); - SetParameter("PROOF_StatsTrace", ""); - if (withWrks) SetParameter("PROOF_SlaveStatsTrace", ""); - Info("SetPerfTree", "saving of the performance tree enabled (%s)", fPerfTree.Data()); - } else { - fPerfTree = ""; - DeleteParameters("PROOF_StatsHist"); - DeleteParameters("PROOF_StatsTrace"); - DeleteParameters("PROOF_SlaveStatsTrace"); - Info("SetPerfTree", "saving of the performance tree disabled"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save performance information from TPerfStats to file 'pf'. -/// If 'ref' is defined, do it for query 'ref'. -/// Return 0 on sucecss, -1 in case of any error - -Int_t TProof::SavePerfTree(const char *pf, const char *ref) -{ - if (!IsValid()) { - Error("SafePerfTree", "this TProof instance is invalid!"); - return -1; - } - - TList *outls = GetOutputList(); - TString sref; - if (ref && strlen(ref) > 0) { - if (!fPlayer) { - Error("SafePerfTree", "requested to use query '%s' but player instance undefined!", ref); - return -1; - } - TQueryResult *qr = fPlayer->GetQueryResult(ref); - if (!qr) { - Error("SafePerfTree", "TQueryResult instance for query '%s' could not be retrieved", ref); - return -1; - } - outls = qr->GetOutputList(); - sref.Form(" for requested query '%s'", ref); - } - if (!outls || (outls && outls->GetSize() <= 0)) { - Error("SafePerfTree", "outputlist%s undefined or empty", sref.Data()); - return -1; - } - - TString fn = fPerfTree; - if (pf && strlen(pf)) fn = pf; - if (fn.IsNull()) fn = "perftree.root"; - - TFile f(fn, "RECREATE"); - if (f.IsZombie()) { - Error("SavePerfTree", "could not open file '%s' for writing", fn.Data()); - } else { - f.cd(); - TIter nxo(outls); - TObject* obj = 0; - while ((obj = nxo())) { - TString objname(obj->GetName()); - if (objname.BeginsWith("PROOF_")) { - // Must list the objects since other PROOF_ objects exist - // besides timing objects - if (objname == "PROOF_PerfStats" || - objname == "PROOF_PacketsHist" || - objname == "PROOF_EventsHist" || - objname == "PROOF_NodeHist" || - objname == "PROOF_LatencyHist" || - objname == "PROOF_ProcTimeHist" || - objname == "PROOF_CpuTimeHist") - obj->Write(); - } - } - f.Close(); - } - Info("SavePerfTree", "performance information%s saved in %s ...", sref.Data(), fn.Data()); - - // Done - return 0; -} diff --git a/proof/proof/src/TProofChain.cxx b/proof/proof/src/TProofChain.cxx deleted file mode 100644 index 396f15ad17d9a..0000000000000 --- a/proof/proof/src/TProofChain.cxx +++ /dev/null @@ -1,461 +0,0 @@ - -// @(#)root/proof:$Id$ -// Author: G. Ganis Nov 2006 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -////////////////////////////////////////////////////////////////////////// -// // -// TProofChain // -// // -// A TChain proxy on PROOF. // -// Uses an internal TDSet to handle processing. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProofChain.h" -#include "TDSet.h" -#include "TList.h" -#include "TProof.h" -#include "TROOT.h" -#include "TEventList.h" -#include "TEntryList.h" - -ClassImp(TProofChain); - -//////////////////////////////////////////////////////////////////////////////// -/// Crates a new PROOF chain proxy. - -TProofChain::TProofChain() : TChain() -{ - fChain = 0; - fTree = 0; - fSet = 0; - fDirectory = gDirectory; - ResetBit(kOwnsChain); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Crates a new PROOF chain proxy containing the files from the chain. - -TProofChain::TProofChain(TChain *chain, Bool_t gettreeheader) : TChain() -{ - fChain = chain; - fTree = 0; - fSet = chain ? new TDSet((const TChain &)(*chain)) : 0; - fDirectory = gDirectory; - if (gProof) { - gProof->AddChain(chain); - ConnectProof(); - if (gProof->IsLite()) { - SetBit(kProofLite); - fTree = fChain; - } else { - if (gettreeheader && fSet) - fTree = gProof->GetTreeHeader(fSet); - } - } - ResetBit(kOwnsChain); - fEntryList = (chain) ? chain->GetEntryList() : 0; - fEventList = (chain) ? chain->GetEventList() : 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Creates a new PROOF chain proxy containing the files from the dset. - -TProofChain::TProofChain(TDSet *dset, Bool_t gettreeheader) : TChain() -{ - fChain = 0; - fTree = 0; - fSet = dset; - fDirectory = gDirectory; - if (gProof) { - ConnectProof(); - if (gettreeheader && dset) - fTree = gProof->GetTreeHeader(dset); - if (gProof->IsLite()) - SetBit(kProofLite); - } - if (fTree && fSet) { - fChain = new TChain(fTree->GetName()); - TIter nxe(fSet->GetListOfElements()); - TDSetElement *e = 0; - while ((e = (TDSetElement *) nxe())) { - fChain->AddFile(e->GetName()); - } - SetBit(kOwnsChain); - if (TestBit(kProofLite)) - fTree = fChain; - } - TObject *en = (dset) ? dset->GetEntryList() : 0; - if (en) { - if (en->InheritsFrom("TEntryList")) { - fEntryList = (TEntryList *) en; - } else { - fEventList = (TEventList *) en; - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TProofChain::~TProofChain() -{ - if (fChain) { - SafeDelete(fSet); - // Remove the chain from the private lists in the TProof objects - TIter nxp(gROOT->GetListOfSockets()); - TObject *o = 0; - TProof *p = 0; - while ((o = nxp())) - if ((p = dynamic_cast(o))) - p->RemoveChain(fChain); - if (fTree == fChain) fTree = 0; - if (TestBit(kOwnsChain)) { - SafeDelete(fChain); - } else { - fChain = 0; - } - } else { - // Not owner - fSet = 0; - } - SafeDelete(fTree); - fDirectory = 0; - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Forwards the execution to the dummy tree header. -/// See TTree::Browse(). - -void TProofChain::Browse(TBrowser *b) -{ - fSet->Browse(b); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Forwards the execution to the TDSet. -/// Returns -1 in case of error or number of selected events in case of success. -/// See TDSet::Browse(). - -Long64_t TProofChain::Draw(const char *varexp, const TCut &selection, - Option_t *option, Long64_t nentries, Long64_t firstentry) -{ - if (!gProof) { - Error("Draw", "no active PROOF session"); - return -1; - } - ConnectProof(); - - fReadEntry = firstentry; - - // Set either the entry-list (priority) or the event-list - if (fEntryList) { - fSet->SetEntryList(fEntryList); - } else if (fEventList) { - fSet->SetEntryList(fEventList); - } else { - // Disable previous settings, if any - fSet->SetEntryList(0); - } - - // Fill drawing attributes - FillDrawAttributes(gProof); - - // Add alias information, if any - AddAliases(); - - Long64_t rv = fSet->Draw(varexp, selection, option, nentries, firstentry); - return rv; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Forwards the execution to the TDSet. -/// Returns -1 in case of error or number of selected events in case of success. -/// See TDSet::Browse(). - -Long64_t TProofChain::Draw(const char *varexp, const char *selection, - Option_t *option,Long64_t nentries, Long64_t firstentry) -{ - if (!gProof) { - Error("Draw", "no active PROOF session"); - return -1; - } - ConnectProof(); - - fReadEntry = firstentry; - - // Set either the entry-list (priority) or the event-list - if (fEntryList) { - fSet->SetEntryList(fEntryList); - } else if (fEventList) { - fSet->SetEntryList(fEventList); - } else { - // Disable previous settings, if any - fSet->SetEntryList(0); - } - - // Fill drawing attributes - FillDrawAttributes(gProof); - - // Add alias information, if any - AddAliases(); - - Long64_t rv = fSet->Draw(varexp, selection, option, nentries, firstentry); - return rv; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Aliases are added to the input list. The names are comma-separated in the -/// TNamed 'PROOF_ListOfAliases'. For each name, there is an trey named `alias:`. - -void TProofChain::AddAliases() -{ - TList *al = fChain->GetListOfAliases(); - if (al && al->GetSize() > 0) { - TIter nxa(al); - TNamed *nm = 0, *nmo = 0; - TString names, nma; - while ((nm = (TNamed *)nxa())) { - names += nm->GetName(); - names += ","; - nma.Form("alias:%s", nm->GetName()); - nmo = (TNamed *)((gProof->GetInputList()) ? gProof->GetInputList()->FindObject(nma) : 0); - if (nmo) { - nmo->SetTitle(nm->GetTitle()); - } else { - gProof->AddInput(new TNamed(nma.Data(), nm->GetTitle())); - } - } - nmo = (TNamed *)((gProof->GetInputList()) ? gProof->GetInputList()->FindObject("PROOF_ListOfAliases") : 0); - if (nmo) { - nmo->SetTitle(names.Data()); - } else { - gProof->AddInput(new TNamed("PROOF_ListOfAliases", names.Data())); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Communicate the drawing attributes for this chain to the input list -/// so that the draw selectors can use them, in case of need. -/// The drawing attributes are: -/// -/// LineColor Line color -/// LineStyle Line style -/// LineWidth Line width -/// MarkerColor Marker color index -/// MarkerSize Marker size -/// MarkerStyle Marker style -/// FillColor Area fill color -/// FillStyle Area fill style - -void TProofChain::FillDrawAttributes(TProof *p) -{ - if (!p || !fChain) { - Error("FillDrawAttributes", "invalid PROOF or mother chain pointers!"); - return; - } - - // Weight - p->SetParameter("PROOF_ChainWeight", fChain->GetWeight()); - - // Line Attributes - p->SetParameter("PROOF_LineColor", (Int_t) fChain->GetLineColor()); - p->SetParameter("PROOF_LineStyle", (Int_t) fChain->GetLineStyle()); - p->SetParameter("PROOF_LineWidth", (Int_t) fChain->GetLineWidth()); - - // Marker Attributes - p->SetParameter("PROOF_MarkerColor", (Int_t) fChain->GetMarkerColor()); - p->SetParameter("PROOF_MarkerSize", (Int_t) fChain->GetMarkerSize()*1000); - p->SetParameter("PROOF_MarkerStyle", (Int_t) fChain->GetMarkerStyle()); - - // Area fill attributes - p->SetParameter("PROOF_FillColor", (Int_t) fChain->GetFillColor()); - p->SetParameter("PROOF_FillStyle", (Int_t) fChain->GetFillStyle()); - - if (gDebug > 0) { - Info("FillDrawAttributes","line: color:%d, style:%d, width:%d", - fChain->GetLineColor(), fChain->GetLineStyle(), fChain->GetLineWidth()); - Info("FillDrawAttributes","marker: color:%d, style:%d, size:%f", - fChain->GetMarkerColor(), fChain->GetMarkerStyle(), fChain->GetMarkerSize()); - Info("FillDrawAttributes","area: color:%d, style:%d", - fChain->GetFillColor(), fChain->GetFillStyle()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Forwards the execution to the dummy tree header. -/// See TTree::FindBranch(). - -TBranch *TProofChain::FindBranch(const char* branchname) -{ - return (fTree ? fTree->FindBranch(branchname) : (TBranch *)0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Forwards the execution to the dummy tree header. -/// See TTree::FindLeaf(). - -TLeaf *TProofChain::FindLeaf(const char* searchname) -{ - return (fTree ? fTree->FindLeaf(searchname) : (TLeaf *)0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Forwards the execution to the dummy tree header. -/// See TTree::GetBranch(). - -TBranch *TProofChain::GetBranch(const char *name) -{ - return (fTree ? fTree->GetBranch(name) : (TBranch *)0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Forwards the execution to the dummy tree header. -/// See TTree::GetBranchStatus(). - -Bool_t TProofChain::GetBranchStatus(const char *branchname) const -{ - return (fTree ? fTree->GetBranchStatus(branchname) : kFALSE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Forwards the execution to the dummy tree header. -/// See TTree::GetPlayer(). - -TVirtualTreePlayer *TProofChain::GetPlayer() -{ - return (fTree ? fTree->GetPlayer() : (TVirtualTreePlayer *)0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Forwards the execution to the TDSet. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. -/// See TDSet::Process(). - -Long64_t TProofChain::Process(const char *filename, Option_t *option, - Long64_t nentries, Long64_t firstentry) -{ - // Set either the entry-list (priority) or the event-list - TObject *enl = 0; - if (fEntryList) { - enl = fEntryList; - } else if (fEventList) { - enl = fEventList; - } - - return fSet->Process(filename, option, nentries, firstentry, enl); -} - -//////////////////////////////////////////////////////////////////////////////// -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProofChain::Process(TSelector *selector, Option_t *option, - Long64_t nentries, Long64_t firstentry) -{ - // Set either the entry-list (priority) or the event-list - TObject *enl = 0; - if (fEntryList) { - enl = fEntryList; - } else if (fEventList) { - enl = fEventList; - } - - return fSet->Process(selector, option, nentries, firstentry, enl); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TTree::SetDebug - -void TProofChain::SetDebug(Int_t level, Long64_t min, Long64_t max) -{ - TTree::SetDebug(level, min, max); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TTree::GetName. - -void TProofChain::SetName(const char *name) -{ - TTree::SetName(name); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns the total number of entries in the TProofChain, which is -/// the number of entries in the TDSet that it holds. - -Long64_t TProofChain::GetEntries() const -{ - // this was used for holding the total number of entries - if (TestBit(kProofLite)) { - return (fTree ? fTree->GetEntries() : (Long64_t)(-1)); - } else { - return (fTree ? fTree->GetMaxEntryLoop() : (Long64_t)(-1)); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TTree::GetEntries(const char *selection) -/// Not implemented in TProofChain. Shouldn't be used. - -Long64_t TProofChain::GetEntries(const char *selection) -{ - if (TestBit(kProofLite)) { - return (fTree ? fTree->GetEntries(selection) : (Long64_t)(-1)); - } else { - Warning("GetEntries", "GetEntries(selection) not yet implemented"); - return ((Long64_t)(-1)); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Changes the number of processed entries. - -void TProofChain::Progress(Long64_t total, Long64_t processed) -{ - if (gROOT->IsInterrupted() && gProof) - gProof->StopProcess(kTRUE); - if (total) { } - - fReadEntry = processed; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns the number of processed entries. - -Long64_t TProofChain::GetReadEntry() const -{ - return fReadEntry; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Releases PROOF. Disconnect the "progress" signal. - -void TProofChain::ReleaseProof() -{ - if (!gProof) - return; - gProof->Disconnect("Progress(Long64_t,Long64_t)", - this, "Progress(Long64_t,Long64_t)"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Connects the proof "Progress" signal. - -void TProofChain::ConnectProof() -{ - if (gProof) - gProof->Connect("Progress(Long64_t,Long64_t)", "TProofChain", - this, "Progress(Long64_t,Long64_t)"); -} diff --git a/proof/proof/src/TProofCondor.cxx b/proof/proof/src/TProofCondor.cxx deleted file mode 100644 index 46aeaa3f43b78..0000000000000 --- a/proof/proof/src/TProofCondor.cxx +++ /dev/null @@ -1,322 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 13/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -////////////////////////////////////////////////////////////////////////// -// // -// TProof // -// // -// This class controls a Parallel ROOT Facility, PROOF, cluster. // -// It fires the slave servers, it keeps track of how many slaves are // -// running, it keeps track of the slaves running status, it broadcasts // -// messages to all slaves, it collects results, etc. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProofCondor.h" - -#include "TCondor.h" -#include "TList.h" -#include "TMap.h" -#include "TMessage.h" -#include "TMonitor.h" -#include "TProofNodeInfo.h" -#include "TProofResourcesStatic.h" -#include "TProofServ.h" -#include "TSlave.h" -#include "TSocket.h" -#include "TString.h" -#include "TTimer.h" - -ClassImp(TProofCondor); - -//////////////////////////////////////////////////////////////////////////////// -/// Start proof using condor - -TProofCondor::TProofCondor(const char *masterurl, const char *conffile, - const char *confdir, Int_t loglevel, - const char *, TProofMgr *mgr) - : fCondor(0), fTimer(0) -{ - // Default initializations - InitMembers(); - - // This may be needed during init - fManager = mgr; - - fUrl = TUrl(masterurl); - - if (!conffile || !conffile[0]) { - conffile = kPROOF_ConfFile; - } else if (!strncasecmp(conffile, "condor:", 7)) { - conffile+=7; - } - - if (!confdir || !confdir[0]) { - confdir = kPROOF_ConfDir; - } - - Init(masterurl, conffile, confdir, loglevel); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clean up Condor PROOF environment. - -TProofCondor::~TProofCondor() -{ - SafeDelete(fCondor); - SafeDelete(fTimer); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Setup Condor workers using dynamic information - -Bool_t TProofCondor::StartSlaves(Bool_t) -{ - fCondor = new TCondor; - TString jobad = GetJobAd(); - - fImage = fCondor->GetImage(gSystem->HostName()); - if (fImage.Length() == 0) { - Error("StartSlaves", "Empty Condor image found for system %s", - gSystem->HostName()); - return kFALSE; - } - - TList claims; - if (fConfFile.IsNull()) { - // startup all slaves if no config file given - TList *condorclaims = fCondor->Claim(9999, jobad); - TIter nextclaim(condorclaims); - while (TObject *o = nextclaim()) claims.Add(o); - } else { - // parse config file - TProofResourcesStatic *resources = new TProofResourcesStatic(fConfDir, fConfFile); - fConfFile = resources->GetFileName(); // Update the global file name (with path) - PDB(kGlobal,1) Info("StartSlaves", "using PROOF config file: %s", fConfFile.Data()); - - // Get all workers - TList *workerList = resources->GetWorkers(); - if (workerList->GetSize() == 0) { - Error("StartSlaves", "Found no condorworkers in %s", fConfFile.Data()); - return kFALSE; - } - - // check for valid slave lines and claim condor nodes - Int_t ord = 0; - - // Loop over all workers and start them - TListIter next(workerList); - TObject *to; - TProofNodeInfo *worker; - int nSlavesDone = 0; - while ((to = next())) { - // Get the next worker from the list - worker = (TProofNodeInfo *)to; - - // Read back worker node info - const Char_t *image = worker->GetImage().Data(); - const Char_t *workdir = worker->GetWorkDir().Data(); - Int_t perfidx = worker->GetPerfIndex(); - - gSystem->Sleep(10 /* ms */); - TCondorSlave* csl = fCondor->Claim(worker->GetNodeName().Data(), jobad); - if (csl) { - csl->fPerfIdx = perfidx; - csl->fImage = image; - csl->fWorkDir = workdir; - gSystem->ExpandPathName(csl->fWorkDir); - TString fullord = TString(gProofServ->GetOrdinal()) + "." + ((Long_t) ord); - csl->fOrdinal = fullord.Data(); - claims.Add(csl); - ord++; - } - - // Notify claim creation - nSlavesDone++; - TMessage m(kPROOF_SERVERSTARTED); - m << TString("Creating COD Claim") << workerList->GetSize() - << nSlavesDone << (csl != 0); - gProofServ->GetSocket()->Send(m); - - } // end while (worker loop) - - // Cleanup - delete resources; - resources = 0; - } // end else (parse config file) - - Long_t delay = 500; // timer delay 0.5s - Int_t ntries = 20; // allow 20 tries (must be > 1 for algorithm to work) - Int_t trial = 1; - Int_t idx = 0; - - int nClaims = claims.GetSize(); - int nClaimsDone = 0; - while (claims.GetSize() > 0) { - TCondorSlave* c = 0; - - // Get Condor Slave - if (trial == 1) { - c = dynamic_cast(claims.At(idx)); - } else { - TPair *p = dynamic_cast(claims.At(idx)); - if (p) { - TTimer *t = dynamic_cast(p->Value()); - if (t) { - // wait remaining time - Long64_t wait = t->GetAbsTime()-gSystem->Now(); - if (wait > 0) gSystem->Sleep((UInt_t)wait); - c = dynamic_cast(p->Key()); - } - } - } - - // create slave - TSlave *slave = 0; - if (c) slave = CreateSlave(Form("%s:%d", c->fHostname.Data(), c->fPort), c->fOrdinal, - c->fPerfIdx, c->fImage, c->fWorkDir); - - // add slave to appropriate list - if (trial < ntries) { - if (slave && slave->IsValid()) { - fSlaves->Add(slave); - if (trial == 1) { - claims.Remove(c); - } else { - TPair *p = dynamic_cast(claims.Remove(c)); - if (p) { - TTimer *xt = dynamic_cast(p->Value()); - if (xt) delete xt; - delete p; - } - } - nClaimsDone++; - TMessage m(kPROOF_SERVERSTARTED); - m << TString("Opening connections to workers") << nClaims - << nClaimsDone << kTRUE; - gProofServ->GetSocket()->Send(m); - } else if (slave) { - if (trial == 1) { - TTimer* timer = new TTimer(delay); - TPair *p = new TPair(c, timer); - claims.RemoveAt(idx); - claims.AddAt(p, idx); - } else { - TPair *p = dynamic_cast(claims.At(idx)); - if (p && p->Value()) { - TTimer *xt = dynamic_cast(p->Value()); - if (xt) xt->Reset(); - } - } - delete slave; - idx++; - } else { - Warning("StartSlaves", "could not create TSlave object!"); - } - } else { - if (slave) { - fSlaves->Add(slave); - TPair *p = dynamic_cast(claims.Remove(c)); - if (p && p->Value()) { - TTimer *xt = dynamic_cast(p->Value()); - delete xt; - } - if (p) delete p; - - nClaimsDone++; - TMessage m(kPROOF_SERVERSTARTED); - m << TString("Opening connections to workers") << nClaims - << nClaimsDone << slave->IsValid(); - gProofServ->GetSocket()->Send(m); - } else { - Warning("StartSlaves", "could not create TSlave object!"); - } - } - - if (idx>=claims.GetSize()) { - trial++; - idx = 0; - } - } - - // Here we finalize the server startup: in this way the bulk - // of remote operations are almost parallelized - TIter nxsl(fSlaves); - TSlave *sl = 0; - int nSlavesDone = 0, nSlavesTotal = fSlaves->GetSize(); - while ((sl = (TSlave *) nxsl())) { - - // Finalize setup of the server - if (sl->IsValid()) { - sl->SetupServ(TSlave::kSlave, 0); - } - - if (sl->IsValid()) { - fAllMonitor->Add(sl->GetSocket()); - } else { - fBadSlaves->Add(sl); - } - - // Notify end of startup operations - nSlavesDone++; - TMessage m(kPROOF_SERVERSTARTED); - Bool_t wrkvalid = sl->IsValid() ? kTRUE : kFALSE; - m << TString("Setting up worker servers") << nSlavesTotal - << nSlavesDone << wrkvalid; - gProofServ->GetSocket()->Send(m); - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Suspend or resume PROOF via Condor. - -void TProofCondor::SetActive(Bool_t active) -{ - if (fTimer == 0) { - fTimer = new TTimer(); - } - if (active) { - PDB(kCondor,1) Info("SetActive","-- Condor Resume --"); - fTimer->Stop(); - if (fCondor->GetState() == TCondor::kSuspended) - fCondor->Resume(); - } else { -#if 1 - return; // don't suspend for the moment -#else - Int_t delay = 60000; // milli seconds - PDB(kCondor,1) Info("SetActive","-- Delayed Condor Suspend (%d msec / to %lld) --", - delay, delay + Long64_t(gSystem->Now())); - fTimer->Connect("Timeout()", "TCondor", fCondor, "Suspend()"); - fTimer->Start(10000, kTRUE); // single shot -#endif - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get job Ad - -TString TProofCondor::GetJobAd() -{ - TString ad; - - ad = "JobUniverse = 5\n"; // vanilla - ad += Form("Cmd = \"%s/bin/proofd\"\n", GetConfDir()); - ad += Form("Iwd = \"%s\"\n", gSystem->TempDirectory()); - ad += "In = \"/dev/null\"\n"; - ad += Form("Out = \"%s/proofd.out.$(Port)\"\n", gSystem->TempDirectory()); - ad += Form("Err = \"%s/proofd.err.$(Port)\"\n", gSystem->TempDirectory()); - ad += Form("Args = \"-f -p $(Port) -d %d %s\"\n", GetLogLevel(), GetConfDir()); - - return ad; -} diff --git a/proof/proof/src/TProofDebug.cxx b/proof/proof/src/TProofDebug.cxx deleted file mode 100644 index 00acee8ffea8a..0000000000000 --- a/proof/proof/src/TProofDebug.cxx +++ /dev/null @@ -1,24 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Maarten Ballintijn 19/6/2002 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -////////////////////////////////////////////////////////////////////////// -// // -// TProofDebug // -// // -// Detailed logging / debug scheme. // -// // -////////////////////////////////////////////////////////////////////////// - - -#include "TProofDebug.h" - -TProofDebug::EProofDebugMask gProofDebugMask = TProofDebug::kAll; -Int_t gProofDebugLevel = 0; diff --git a/proof/proof/src/TProofLite.cxx b/proof/proof/src/TProofLite.cxx deleted file mode 100644 index 3db5e1a30a6fe..0000000000000 --- a/proof/proof/src/TProofLite.cxx +++ /dev/null @@ -1,2696 +0,0 @@ -// @(#)root/proof:$Id: 7735e42a1b96a9f40ae76bd884acac883a178dee $ -// Author: G. Ganis March 2008 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofLite -\ingroup proofkernel - -This class starts a PROOF session on the local machine: no daemons, -client and master merged, communications via UNIX-like sockets. -By default the number of workers started is NumberOfCores+1; a -different number can be forced on construction. - -*/ - -#include "TProofLite.h" - -#ifdef WIN32 -# include -# include "snprintf.h" -#endif -#include "RConfigure.h" -#include "TDSet.h" -#include "TEnv.h" -#include "TError.h" -#include "TFile.h" -#include "TFileCollection.h" -#include "TFileInfo.h" -#include "THashList.h" -#include "TMessage.h" -#include "TMonitor.h" -#include "TObjString.h" -#include "TPluginManager.h" -#include "TDataSetManager.h" -#include "TDataSetManagerFile.h" -#include "TParameter.h" -#include "TPRegexp.h" -#include "TProofQueryResult.h" -#include "TProofServ.h" -#include "TQueryResultManager.h" -#include "TROOT.h" -#include "TServerSocket.h" -#include "TSlave.h" -#include "TSortedList.h" -#include "TTree.h" -#include "TVirtualProofPlayer.h" -#include "TSelector.h" -#include "TPackMgr.h" - -ClassImp(TProofLite); - -Int_t TProofLite::fgWrksMax = -2; // Unitialized max number of workers - -//////////////////////////////////////////////////////////////////////////////// -/// Create a PROOF environment. Starting PROOF involves either connecting -/// to a master server, which in turn will start a set of slave servers, or -/// directly starting as master server (if master = ""). Masterurl is of -/// the form: [proof[s]://]host[:port]. Conffile is the name of the config -/// file describing the remote PROOF cluster (this argument alows you to -/// describe different cluster configurations). -/// The default is proof.conf. Confdir is the directory where the config -/// file and other PROOF related files are (like motd and noproof files). -/// Loglevel is the log level (default = 1). User specified custom config -/// files will be first looked for in $HOME/.conffile. - -TProofLite::TProofLite(const char *url, const char *conffile, const char *confdir, - Int_t loglevel, const char *alias, TProofMgr *mgr) -{ - fUrl.SetUrl(url); - - // Default initializations - fServSock = 0; - fCacheLock = 0; - fQueryLock = 0; - fQMgr = 0; - fDataSetManager = 0; - fDataSetStgRepo = 0; - fReInvalid = new TPMERegexp("[^A-Za-z0-9._-]"); - InitMembers(); - - // This may be needed during init - fManager = mgr; - - // Default server type - fServType = TProofMgr::kProofLite; - - // Default query mode - fQueryMode = kSync; - - // Client and master are merged - fMasterServ = kTRUE; - if (fManager) SetBit(TProof::kIsClient); - SetBit(TProof::kIsMaster); - - // Flag that we are a client - if (!gSystem->Getenv("ROOTPROOFCLIENT")) gSystem->Setenv("ROOTPROOFCLIENT",""); - - // Protocol and Host - fUrl.SetProtocol("proof"); - fUrl.SetHost("__lite__"); - fUrl.SetPort(1093); - - // User - if (strlen(fUrl.GetUser()) <= 0) { - // Get user logon name - UserGroup_t *pw = gSystem->GetUserInfo(); - if (pw) { - fUrl.SetUser(pw->fUser); - delete pw; - } - } - fMaster = gSystem->HostName(); - - // Analysise the conffile field - ParseConfigField(conffile); - - // Determine the number of workers giving priority to users request. - // Otherwise use the system information, if available, or just start - // the minimal number, i.e. 2 . - if ((fNWorkers = GetNumberOfWorkers(url)) > 0) { - - TString stup; - if (gProofServ) { - Int_t port = gEnv->GetValue("ProofServ.XpdPort", 1093); - stup.Form("%s @ %s:%d ", gProofServ->GetOrdinal(), gSystem->HostName(), port); - } - Printf(" +++ Starting PROOF-Lite %swith %d workers +++", stup.Data(), fNWorkers); - // Init the session now - Init(url, conffile, confdir, loglevel, alias); - } - - // For final cleanup - if (!gROOT->GetListOfProofs()->FindObject(this)) - gROOT->GetListOfProofs()->Add(this); - - // Still needed by the packetizers: needs to be changed - gProof = this; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Start the PROOF environment. Starting PROOF involves either connecting -/// to a master server, which in turn will start a set of slave servers, or -/// directly starting as master server (if master = ""). For a description -/// of the arguments see the TProof ctor. Returns the number of started -/// master or slave servers, returns 0 in case of error, in which case -/// fValid remains false. - -Int_t TProofLite::Init(const char *, const char *conffile, - const char *confdir, Int_t loglevel, const char *) -{ - R__ASSERT(gSystem); - - fValid = kFALSE; - - // Connected to terminal? - fTty = (isatty(0) == 0 || isatty(1) == 0) ? kFALSE : kTRUE; - - if (TestBit(TProof::kIsMaster)) { - // Fill default conf file and conf dir - if (!conffile || !conffile[0]) - fConfFile = kPROOF_ConfFile; - if (!confdir || !confdir[0]) - fConfDir = kPROOF_ConfDir; - } else { - fConfDir = confdir; - fConfFile = conffile; - } - - // The sandbox for this session - if (CreateSandbox() != 0) { - Error("Init", "could not create/assert sandbox for this session"); - return 0; - } - - // UNIX path for communication with workers - TString sockpathdir = gEnv->GetValue("ProofLite.SockPathDir", gSystem->TempDirectory()); - if (sockpathdir.IsNull()) sockpathdir = gSystem->TempDirectory(); - if (sockpathdir(sockpathdir.Length()-1) == '/') sockpathdir.Remove(sockpathdir.Length()-1); - fSockPath.Form("%s/plite-%d", sockpathdir.Data(), gSystem->GetPid()); - if (fSockPath.Length() > 104) { - // Sort of hardcoded limit length for Unix systems - Error("Init", "Unix socket path '%s' is too long (%d bytes):", - fSockPath.Data(), fSockPath.Length()); - Error("Init", "use 'ProofLite.SockPathDir' to create it under a directory different" - " from '%s'", sockpathdir.Data()); - return 0; - } - - fLogLevel = loglevel; - fProtocol = kPROOF_Protocol; - fSendGroupView = kTRUE; - fImage = ""; - fIntHandler = 0; - fStatus = 0; - fRecvMessages = new TList; - fRecvMessages->SetOwner(kTRUE); - fSlaveInfo = 0; - fChains = new TList; - fAvailablePackages = 0; - fEnabledPackages = 0; - fEndMaster = TestBit(TProof::kIsMaster) ? kTRUE : kFALSE; - fInputData = 0; - ResetBit(TProof::kNewInputData); - - fEnabledPackagesOnCluster = new TList; - fEnabledPackagesOnCluster->SetOwner(); - - // Timeout for some collect actions - fCollectTimeout = gEnv->GetValue("Proof.CollectTimeout", -1); - - // Should the workers be started dynamically; default: no - fDynamicStartup = kFALSE; - fDynamicStartupStep = -1; - fDynamicStartupNMax = -1; - TString dynconf = gEnv->GetValue("Proof.SimulateDynamicStartup", ""); - if (dynconf.Length() > 0) { - fDynamicStartup = kTRUE; - fLastPollWorkers_s = time(0); - // Extract parameters - Int_t from = 0; - TString p; - if (dynconf.Tokenize(p, from, ":")) - if (p.IsDigit()) fDynamicStartupStep = p.Atoi(); - if (dynconf.Tokenize(p, from, ":")) - if (p.IsDigit()) fDynamicStartupNMax = p.Atoi(); - } - - - fProgressDialog = 0; - fProgressDialogStarted = kFALSE; - - // Client logging of messages from the workers - fRedirLog = kFALSE; - if (TestBit(TProof::kIsClient)) { - fLogFileName = Form("%s/session-%s.log", fWorkDir.Data(), GetName()); - if ((fLogFileW = fopen(fLogFileName.Data(), "w")) == 0) - Error("Init", "could not create temporary logfile %s", fLogFileName.Data()); - if ((fLogFileR = fopen(fLogFileName.Data(), "r")) == 0) - Error("Init", "could not open logfile %s for reading", fLogFileName.Data()); - } - fLogToWindowOnly = kFALSE; - - fCacheLock = new TProofLockPath(TString::Format("%s/%s%s", gSystem->TempDirectory(), - kPROOF_CacheLockFile, - TString(fCacheDir).ReplaceAll("/","%").Data())); - - // Create 'queries' locker instance and lock it - fQueryLock = new TProofLockPath(TString::Format("%s/%s%s-%s", gSystem->TempDirectory(), - kPROOF_QueryLockFile, GetName(), - TString(fQueryDir).ReplaceAll("/","%").Data())); - fQueryLock->Lock(); - // Create the query manager - fQMgr = new TQueryResultManager(fQueryDir, GetName(), fWorkDir, - fQueryLock, fLogFileW); - - // Apply quotas, if any - Int_t maxq = gEnv->GetValue("ProofLite.MaxQueriesSaved", 10); - if (fQMgr && fQMgr->ApplyMaxQueries(maxq) != 0) - Warning("Init", "problems applying fMaxQueries"); - - if (InitDataSetManager() != 0) - Warning("Init", "problems initializing the dataset manager"); - - // Status of cluster - fNotIdle = 0; - - // Query type - fSync = kTRUE; - - // List of queries - fQueries = 0; - fOtherQueries = 0; - fDrawQueries = 0; - fMaxDrawQueries = 1; - fSeqNum = 0; - - // Remote ID of the session - fSessionID = -1; - - // Part of active query - fWaitingSlaves = 0; - - // Make remote PROOF player - fPlayer = 0; - MakePlayer("lite"); - - fFeedback = new TList; - fFeedback->SetOwner(); - fFeedback->SetName("FeedbackList"); - AddInput(fFeedback); - - // Sort workers by descending performance index - fSlaves = new TSortedList(kSortDescending); - fActiveSlaves = new TList; - fInactiveSlaves = new TList; - fUniqueSlaves = new TList; - fAllUniqueSlaves = new TList; - fNonUniqueMasters = new TList; - fBadSlaves = new TList; - fAllMonitor = new TMonitor; - fActiveMonitor = new TMonitor; - fUniqueMonitor = new TMonitor; - fAllUniqueMonitor = new TMonitor; - fCurrentMonitor = 0; - fServSock = 0; - - fTerminatedSlaveInfos = new TList; - fTerminatedSlaveInfos->SetOwner(kTRUE); - - // Control how to start the workers; copy-on-write (fork) is *very* - // experimental and available on Unix only. - fForkStartup = kFALSE; - if (gEnv->GetValue("ProofLite.ForkStartup", 0) != 0) { -#ifndef WIN32 - fForkStartup = kTRUE; -#else - Warning("Init", "fork-based workers startup is not available on Windows - ignoring"); -#endif - } - - fLoadedMacros = 0; - if (TestBit(TProof::kIsClient)) { - - // List of directories where to look for global packages - TString globpack = gEnv->GetValue("Proof.GlobalPackageDirs",""); - TProofServ::ResolveKeywords(globpack); - Int_t nglb = TPackMgr::RegisterGlobalPath(globpack); - if (gDebug > 0) - Info("Init", " %d global package directories registered", nglb); - } - - // Start workers - if (SetupWorkers(0) != 0) { - Error("Init", "problems setting up workers"); - return 0; - } - - // we are now properly initialized - fValid = kTRUE; - - // De-activate monitor (will be activated in Collect) - fAllMonitor->DeActivateAll(); - - // By default go into parallel mode - GoParallel(-1, kFALSE); - - // Send relevant initial state to slaves - SendInitialState(); - - SetActive(kFALSE); - - if (IsValid()) { - // Activate input handler - ActivateAsyncInput(); - // Set PROOF to running state - SetRunStatus(TProof::kRunning); - } - // We register the session as a socket so that cleanup is done properly - R__LOCKGUARD(gROOTMutex); - gROOT->GetListOfSockets()->Add(this); - - AskParallel(); - - return fActiveSlaves->GetSize(); -} -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TProofLite::~TProofLite() -{ - // Shutdown the workers - RemoveWorkers(0); - - if (!(fQMgr && fQMgr->Queries() && fQMgr->Queries()->GetSize())) { - // needed in case fQueryDir is on NFS ?! - gSystem->MakeDirectory(fQueryDir+"/.delete"); - gSystem->Exec(Form("%s %s", kRM, fQueryDir.Data())); - } - - // Remove lock file - if (fQueryLock) { - gSystem->Unlink(fQueryLock->GetName()); - fQueryLock->Unlock(); - } - - SafeDelete(fReInvalid); - SafeDelete(fDataSetManager); - SafeDelete(fDataSetStgRepo); - - // Cleanup the socket - SafeDelete(fServSock); - gSystem->Unlink(fSockPath); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static method to determine the number of workers giving priority to users request. -/// Otherwise use the system information, if available, or just start -/// the minimal number, i.e. 2 . - -Int_t TProofLite::GetNumberOfWorkers(const char *url) -{ - Bool_t notify = kFALSE; - if (fgWrksMax == -2) { - // Find the max number of workers, if any - TString sysname = "system.rootrc"; - char *s = gSystem->ConcatFileName(TROOT::GetEtcDir(), sysname); - TEnv sysenv(0); - sysenv.ReadFile(s, kEnvGlobal); - fgWrksMax = sysenv.GetValue("ProofLite.MaxWorkers", -1); - // Notify once the user if its will is changed - notify = kTRUE; - if (s) delete[] s; - } - if (fgWrksMax == 0) { - ::Error("TProofLite::GetNumberOfWorkers", - "PROOF-Lite disabled by the system administrator: sorry!"); - return 0; - } - - TString nw; - Int_t nWorkers = -1; - Bool_t urlSetting = kFALSE; - if (url && strlen(url)) { - nw = url; - Int_t in = nw.Index("workers="); - if (in != kNPOS) { - nw.Remove(0, in + strlen("workers=")); - while (!nw.IsDigit()) - nw.Remove(nw.Length()-1); - if (!nw.IsNull()) { - if ((nWorkers = nw.Atoi()) <= 0) { - ::Warning("TProofLite::GetNumberOfWorkers", - "number of workers specified by 'workers='" - " is non-positive: using default"); - } else { - urlSetting = kFALSE; - } - } - } - } - if (!urlSetting && fgProofEnvList) { - // Check PROOF_NWORKERS - TNamed *nm = (TNamed *) fgProofEnvList->FindObject("PROOF_NWORKERS"); - if (nm) { - nw = nm->GetTitle(); - if (nw.IsDigit()) { - if ((nWorkers = nw.Atoi()) == 0) { - ::Warning("TProofLite::GetNumberOfWorkers", - "number of workers specified by 'workers='" - " is non-positive: using default"); - } - } - } - } - if (nWorkers <= 0) { - nWorkers = gEnv->GetValue("ProofLite.Workers", -1); - if (nWorkers <= 0) { - SysInfo_t si; - if (gSystem->GetSysInfo(&si) == 0 && si.fCpus > 2) { - nWorkers = si.fCpus; - } else { - // Two workers by default - nWorkers = 2; - } - if (notify) notify = kFALSE; - } - } - // Apply the max, if any - if (fgWrksMax > 0 && fgWrksMax < nWorkers) { - if (notify) - ::Warning("TProofLite::GetNumberOfWorkers", "number of PROOF-Lite workers limited by" - " the system administrator to %d", fgWrksMax); - nWorkers = fgWrksMax; - } - - // Done - return nWorkers; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Start up PROOF workers. - -Int_t TProofLite::SetupWorkers(Int_t opt, TList *startedWorkers) -{ - // Create server socket on the assigned UNIX sock path - if (!fServSock) { - if ((fServSock = new TServerSocket(fSockPath))) { - R__LOCKGUARD(gROOTMutex); - // Remove from the list so that cleanup can be done in the correct order - gROOT->GetListOfSockets()->Remove(fServSock); - } - } - if (!fServSock || !fServSock->IsValid()) { - Error("SetupWorkers", - "unable to create server socket for internal communications"); - SetBit(kInvalidObject); - return -1; - } - - // Create a monitor and add the socket to it - TMonitor *mon = new TMonitor; - mon->Add(fServSock); - - TList started; - TSlave *wrk = 0; - Int_t nWrksDone = 0, nWrksTot = -1; - TString fullord; - - if (opt == 0) { - nWrksTot = fForkStartup ? 1 : fNWorkers; - // Now we create the worker applications which will call us back to finalize - // the setup - Int_t ord = 0; - for (; ord < nWrksTot; ord++) { - - // Ordinal for this worker server - const char *o = (gProofServ) ? gProofServ->GetOrdinal() : "0"; - fullord.Form("%s.%d", o, ord); - - // Create environment files - SetProofServEnv(fullord); - - // Create worker server and add to the list - if ((wrk = CreateSlave("lite", fullord, 100, fImage, fWorkDir))) - started.Add(wrk); - - // Notify - NotifyStartUp("Opening connections to workers", ++nWrksDone, nWrksTot); - - } //end of worker loop - } else { - if (!fForkStartup) { - Warning("SetupWorkers", "standard startup: workers already started"); - return -1; - } - nWrksTot = fNWorkers - 1; - // Now we create the worker applications which will call us back to finalize - // the setup - TString clones; - Int_t ord = 0; - for (; ord < nWrksTot; ord++) { - - // Ordinal for this worker server - const char *o = (gProofServ) ? gProofServ->GetOrdinal() : "0"; - fullord.Form("%s.%d", o, ord + 1); - if (!clones.IsNull()) clones += " "; - clones += fullord; - - // Create worker server and add to the list - if ((wrk = CreateSlave("lite", fullord, -1, fImage, fWorkDir))) - started.Add(wrk); - - // Notify - NotifyStartUp("Opening connections to workers", ++nWrksDone, nWrksTot); - - } //end of worker loop - - // Send the request - TMessage m(kPROOF_FORK); - m << clones; - Broadcast(m, kActive); - } - - // Wait for call backs - nWrksDone = 0; - nWrksTot = started.GetSize(); - Int_t nSelects = 0; - Int_t to = gEnv->GetValue("ProofLite.StartupTimeOut", 5) * 1000; - while (started.GetSize() > 0 && nSelects < nWrksTot) { - - // Wait for activity on the socket for max 5 secs - TSocket *xs = mon->Select(to); - - // Count attempts and check - nSelects++; - if (xs == (TSocket *) -1) continue; - - // Get the connection - TSocket *s = fServSock->Accept(); - if (s && s->IsValid()) { - // Receive ordinal - TMessage *msg = 0; - if (s->Recv(msg) < 0) { - Warning("SetupWorkers", "problems receiving message from accepted socket!"); - } else { - if (msg) { - TString ord; - *msg >> ord; - // Find who is calling back - if ((wrk = (TSlave *) started.FindObject(ord))) { - // Remove it from the started list - started.Remove(wrk); - - // Assign tis socket the selected worker - wrk->SetSocket(s); - // Remove socket from global TROOT socket list. Only the TProof object, - // representing all worker sockets, will be added to this list. This will - // ensure the correct termination of all proof servers in case the - // root session terminates. - { R__LOCKGUARD(gROOTMutex); - gROOT->GetListOfSockets()->Remove(s); - } - if (wrk->IsValid()) { - // Set the input handler - wrk->SetInputHandler(new TProofInputHandler(this, wrk->GetSocket())); - // Set fParallel to 1 for workers since they do not - // report their fParallel with a LOG_DONE message - wrk->fParallel = 1; - // Finalize setup of the server - wrk->SetupServ(TSlave::kSlave, 0); - } - - // Monitor good workers - fSlaves->Add(wrk); - if (wrk->IsValid()) { - if (opt == 1) fActiveSlaves->Add(wrk); - fAllMonitor->Add(wrk->GetSocket()); - // Record also in the list for termination - if (startedWorkers) startedWorkers->Add(wrk); - // Notify startup operations - NotifyStartUp("Setting up worker servers", ++nWrksDone, nWrksTot); - } else { - // Flag as bad - fBadSlaves->Add(wrk); - } - } - } else { - Warning("SetupWorkers", "received empty message from accepted socket!"); - } - } - } - } - - // Cleanup the monitor and the server socket - mon->DeActivateAll(); - delete mon; - - // Create Progress dialog, if needed - if (!gROOT->IsBatch() && !fProgressDialog) { - if ((fProgressDialog = - gROOT->GetPluginManager()->FindHandler("TProofProgressDialog"))) - if (fProgressDialog->LoadPlugin() == -1) - fProgressDialog = 0; - } - - if (opt == 1) { - // Collect replies - Collect(kActive); - // Update group view - SendGroupView(); - // By default go into parallel mode - SetParallel(-1, 0); - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Notify setting-up operation message - -void TProofLite::NotifyStartUp(const char *action, Int_t done, Int_t tot) -{ - Int_t frac = (Int_t) (done*100.)/tot; - char msg[512] = {0}; - if (frac >= 100) { - snprintf(msg, 512, "%s: OK (%d workers) \n", - action, tot); - } else { - snprintf(msg, 512, "%s: %d out of %d (%d %%)\r", - action, done, tot, frac); - } - fprintf(stderr,"%s", msg); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create environment files for worker 'ord' - -Int_t TProofLite::SetProofServEnv(const char *ord) -{ - // Check input - if (!ord || strlen(ord) <= 0) { - Error("SetProofServEnv", "ordinal string undefined"); - return -1; - } - - // ROOT env file - TString rcfile(Form("%s/worker-%s.rootrc", fWorkDir.Data(), ord)); - FILE *frc = fopen(rcfile.Data(), "w"); - if (!frc) { - Error("SetProofServEnv", "cannot open rc file %s", rcfile.Data()); - return -1; - } - - // The session working dir depends on the role - fprintf(frc,"# The session working dir\n"); - fprintf(frc,"ProofServ.SessionDir: %s/worker-%s\n", fWorkDir.Data(), ord); - - // The session unique tag - fprintf(frc,"# Session tag\n"); - fprintf(frc,"ProofServ.SessionTag: %s\n", GetName()); - - // Log / Debug level - fprintf(frc,"# Proof Log/Debug level\n"); - fprintf(frc,"Proof.DebugLevel: %d\n", gDebug); - - // Ordinal number - fprintf(frc,"# Ordinal number\n"); - fprintf(frc,"ProofServ.Ordinal: %s\n", ord); - - // ROOT Version tag - fprintf(frc,"# ROOT Version tag\n"); - fprintf(frc,"ProofServ.RootVersionTag: %s\n", gROOT->GetVersion()); - - // Work dir - TString sandbox = fSandbox; - if (GetSandbox(sandbox, kFALSE, "ProofServ.Sandbox") != 0) - Warning("SetProofServEnv", "problems getting sandbox string for worker"); - fprintf(frc,"# Users sandbox\n"); - fprintf(frc, "ProofServ.Sandbox: %s\n", sandbox.Data()); - - // Cache dir - fprintf(frc,"# Users cache\n"); - fprintf(frc, "ProofServ.CacheDir: %s\n", fCacheDir.Data()); - - // Package dir - fprintf(frc,"# Users packages\n"); - fprintf(frc, "ProofServ.PackageDir: %s\n", fPackMgr->GetDir()); - - // Image - fprintf(frc,"# Server image\n"); - fprintf(frc, "ProofServ.Image: %s\n", fImage.Data()); - - // Set Open socket - fprintf(frc,"# Open socket\n"); - fprintf(frc, "ProofServ.OpenSock: %s\n", fSockPath.Data()); - - // Client Protocol - fprintf(frc,"# Client Protocol\n"); - fprintf(frc, "ProofServ.ClientVersion: %d\n", kPROOF_Protocol); - - // ROOT env file created - fclose(frc); - - // System env file - TString envfile(Form("%s/worker-%s.env", fWorkDir.Data(), ord)); - FILE *fenv = fopen(envfile.Data(), "w"); - if (!fenv) { - Error("SetProofServEnv", "cannot open env file %s", envfile.Data()); - return -1; - } - // ROOTSYS - fprintf(fenv, "export ROOTSYS=%s\n", TROOT::GetRootSys().Data()); - // Conf dir - fprintf(fenv, "export ROOTCONFDIR=%s\n", TROOT::GetRootSys().Data()); - // TMPDIR - fprintf(fenv, "export TMPDIR=%s\n", gSystem->TempDirectory()); - // Log file in the log dir - TString logfile(Form("%s/worker-%s.log", fWorkDir.Data(), ord)); - fprintf(fenv, "export ROOTPROOFLOGFILE=%s\n", logfile.Data()); - // RC file - fprintf(fenv, "export ROOTRCFILE=%s\n", rcfile.Data()); - // ROOT version tag (needed in building packages) - fprintf(fenv, "export ROOTVERSIONTAG=%s\n", gROOT->GetVersion()); - // This flag can be used to identify the type of worker; for example, in BUILD.sh or SETUP.C ... - fprintf(fenv, "export ROOTPROOFLITE=%d\n", fNWorkers); - // Local files are on the local file system - fprintf(fenv, "export LOCALDATASERVER=\"file://\"\n"); - // Set the user envs - if (fgProofEnvList) { - TString namelist; - TIter nxenv(fgProofEnvList); - TNamed *env = 0; - while ((env = (TNamed *)nxenv())) { - TString senv(env->GetTitle()); - ResolveKeywords(senv, ord, logfile.Data()); - fprintf(fenv, "export %s=%s\n", env->GetName(), senv.Data()); - if (namelist.Length() > 0) - namelist += ','; - namelist += env->GetName(); - } - fprintf(fenv, "export PROOF_ALLVARS=%s\n", namelist.Data()); - } - - // System env file created - fclose(fenv); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Resolve some keywords in 's' -/// ``, ``, ``, `` - -void TProofLite::ResolveKeywords(TString &s, const char *ord, - const char *logfile) -{ - if (!logfile) return; - - // Log file - if (s.Contains("") && logfile) { - TString lfr(logfile); - if (lfr.EndsWith(".log")) lfr.Remove(lfr.Last('.')); - s.ReplaceAll("", lfr.Data()); - } - - // user - if (gSystem->Getenv("USER") && s.Contains("")) { - s.ReplaceAll("", gSystem->Getenv("USER")); - } - - // rootsys - if (gSystem->Getenv("ROOTSYS") && s.Contains("")) { - s.ReplaceAll("", gSystem->Getenv("ROOTSYS")); - } - - // cpupin: pin to this CPU num (from 0 to ncpus-1) - if (s.Contains("")) { - TString o = ord; - Int_t n = o.Index('.'); - if (n != kNPOS) { - - o.Remove(0, n+1); - n = o.Atoi(); // n is ord - - TString cpuPinList; - { - const TList *envVars = GetEnvVars(); - TNamed *var; - if (envVars) { - var = dynamic_cast(envVars->FindObject("PROOF_SLAVE_CPUPIN_ORDER")); - if (var) cpuPinList = var->GetTitle(); - } - } - - UInt_t nCpus = 1; - { - SysInfo_t si; - if (gSystem->GetSysInfo(&si) == 0 && (si.fCpus > 0)) - nCpus = si.fCpus; - else nCpus = 1; // fallback - } - - if (cpuPinList.IsNull() || (cpuPinList == "*")) { - // Use processors in order - n = n % nCpus; - } - else { - // Use processors in user's order - // n is now the ordinal, converting to idx - n = n % (cpuPinList.CountChar('+')+1); - TString tok; - Ssiz_t from = 0; - for (Int_t i=0; cpuPinList.Tokenize(tok, from, "\\+"); i++) { - if (i == n) { - n = (tok.Atoi() % nCpus); - break; - } - } - } - - o.Form("%d", n); - } - else { - o = "0"; // should not happen - } - s.ReplaceAll("", o); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create the sandbox for this session - -Int_t TProofLite::CreateSandbox() -{ - // Make sure the sandbox area exist and is writable - if (GetSandbox(fSandbox, kTRUE, "ProofLite.Sandbox") != 0) return -1; - - // Package Manager - TString packdir = gEnv->GetValue("Proof.PackageDir", ""); - if (packdir.IsNull()) - packdir.Form("%s/%s", fSandbox.Data(), kPROOF_PackDir); - if (AssertPath(packdir, kTRUE) != 0) return -1; - fPackMgr = new TPackMgr(packdir); - - // Cache Dir - fCacheDir = gEnv->GetValue("Proof.CacheDir", ""); - if (fCacheDir.IsNull()) - fCacheDir.Form("%s/%s", fSandbox.Data(), kPROOF_CacheDir); - if (AssertPath(fCacheDir, kTRUE) != 0) return -1; - - // Data Set Dir - fDataSetDir = gEnv->GetValue("Proof.DataSetDir", ""); - if (fDataSetDir.IsNull()) - fDataSetDir.Form("%s/%s", fSandbox.Data(), kPROOF_DataSetDir); - if (AssertPath(fDataSetDir, kTRUE) != 0) return -1; - - // Session unique tag (name of this TProof instance) - TString stag; - stag.Form("%s-%d-%d", gSystem->HostName(), (int)time(0), gSystem->GetPid()); - SetName(stag.Data()); - - Int_t subpath = gEnv->GetValue("ProofLite.SubPath", 1); - // Subpath for this session in the fSandbox (/path-to-working-dir) - TString sessdir; - if (subpath != 0) { - sessdir = gSystem->WorkingDirectory(); - sessdir.ReplaceAll(gSystem->HomeDirectory(),""); - sessdir.ReplaceAll("/","-"); - sessdir.Replace(0,1,"/",1); - sessdir.Insert(0, fSandbox.Data()); - } else { - // USe the sandbox - sessdir = fSandbox; - } - - // Session working and queries dir - fWorkDir.Form("%s/session-%s", sessdir.Data(), stag.Data()); - if (AssertPath(fWorkDir, kTRUE) != 0) return -1; - - // Create symlink to the last session - TString lastsess; - lastsess.Form("%s/last-lite-session", sessdir.Data()); - gSystem->Unlink(lastsess); - gSystem->Symlink(fWorkDir, lastsess); - - // Queries Dir: local to the working dir, unless required differently - fQueryDir = gEnv->GetValue("Proof.QueryDir", ""); - if (fQueryDir.IsNull()) - fQueryDir.Form("%s/%s", sessdir.Data(), kPROOF_QueryDir); - if (AssertPath(fQueryDir, kTRUE) != 0) return -1; - - // Cleanup old sessions dirs - CleanupSandbox(); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print status of PROOF-Lite cluster. - -void TProofLite::Print(Option_t *option) const -{ - TString ord; - if (gProofServ) ord.Form("%s ", gProofServ->GetOrdinal()); - if (IsParallel()) - Printf("*** PROOF-Lite cluster %s(parallel mode, %d workers):", ord.Data(), GetParallel()); - else - Printf("*** PROOF-Lite cluster %s(sequential mode)", ord.Data()); - - if (gProofServ) { - TString url(gSystem->HostName()); - // Add port to URL, if defined - Int_t port = gEnv->GetValue("ProofServ.XpdPort", 1093); - if (port > -1) url.Form("%s:%d",gSystem->HostName(), port); - Printf("URL: %s", url.Data()); - } else { - Printf("Host name: %s", gSystem->HostName()); - } - Printf("User: %s", GetUser()); - TString ver(gROOT->GetVersion()); - ver += TString::Format("|%s", gROOT->GetGitCommit()); - if (gSystem->Getenv("ROOTVERSIONTAG")) - ver += TString::Format("|%s", gSystem->Getenv("ROOTVERSIONTAG")); - Printf("ROOT version|rev|tag: %s", ver.Data()); - Printf("Architecture-Compiler: %s-%s", gSystem->GetBuildArch(), - gSystem->GetBuildCompilerVersion()); - Printf("Protocol version: %d", GetClientProtocol()); - Printf("Working directory: %s", gSystem->WorkingDirectory()); - Printf("Communication path: %s", fSockPath.Data()); - Printf("Log level: %d", GetLogLevel()); - Printf("Number of workers: %d", GetNumberOfSlaves()); - Printf("Number of active workers: %d", GetNumberOfActiveSlaves()); - Printf("Number of unique workers: %d", GetNumberOfUniqueSlaves()); - Printf("Number of inactive workers: %d", GetNumberOfInactiveSlaves()); - Printf("Number of bad workers: %d", GetNumberOfBadSlaves()); - Printf("Total MB's processed: %.2f", float(GetBytesRead())/(1024*1024)); - Printf("Total real time used (s): %.3f", GetRealTime()); - Printf("Total CPU time used (s): %.3f", GetCpuTime()); - if (TString(option).Contains("a", TString::kIgnoreCase) && GetNumberOfSlaves()) { - Printf("List of workers:"); - TIter nextslave(fSlaves); - while (TSlave* sl = dynamic_cast(nextslave())) { - if (sl->IsValid()) - sl->Print(option); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a TProofQueryResult instance for this query. - -TProofQueryResult *TProofLite::MakeQueryResult(Long64_t nent, const char *opt, - Long64_t fst, TDSet *dset, - const char *selec) -{ - // Increment sequential number - Int_t seqnum = -1; - if (fQMgr) { - fQMgr->IncrementSeqNum(); - seqnum = fQMgr->SeqNum(); - } - - // Create the instance and add it to the list - TProofQueryResult *pqr = new TProofQueryResult(seqnum, opt, - fPlayer->GetInputList(), nent, - fst, dset, selec, - (dset ? dset->GetEntryList() : 0)); - // Title is the session identifier - pqr->SetTitle(GetName()); - - return pqr; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set query in running state. - -void TProofLite::SetQueryRunning(TProofQueryResult *pq) -{ - // Record current position in the log file at start - fflush(fLogFileW); - Int_t startlog = lseek(fileno(fLogFileW), (off_t) 0, SEEK_END); - - // Add some header to logs - Printf(" "); - Info("SetQueryRunning", "starting query: %d", pq->GetSeqNum()); - - // Build the list of loaded PAR packages - TString parlist = ""; - fPackMgr->GetEnabledPackages(parlist); - - // Set in running state - pq->SetRunning(startlog, parlist, GetParallel()); - - // Bytes and CPU at start (we will calculate the differential at end) - AskStatistics(); - pq->SetProcessInfo(pq->GetEntries(), GetCpuTime(), GetBytesRead()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Execute the specified drawing action on a data set (TDSet). -/// Event- or Entry-lists should be set in the data set object using -/// TDSet::SetEntryList. -/// Returns -1 in case of error or number of selected events otherwise. - -Long64_t TProofLite::DrawSelect(TDSet *dset, const char *varexp, - const char *selection, Option_t *option, - Long64_t nentries, Long64_t first) -{ - if (!IsValid()) return -1; - - // Make sure that asynchronous processing is not active - if (!IsIdle()) { - Info("DrawSelect","not idle, asynchronous Draw not supported"); - return -1; - } - TString opt(option); - Int_t idx = opt.Index("ASYN", 0, TString::kIgnoreCase); - if (idx != kNPOS) - opt.Replace(idx,4,""); - - // Fill the internal variables - fVarExp = varexp; - fSelection = selection; - - return Process(dset, "draw:", opt, nentries, first); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process a data set (TDSet) using the specified selector (.C) file. -/// Entry- or event-lists should be set in the data set object using -/// TDSet::SetEntryList. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProofLite::Process(TDSet *dset, const char *selector, Option_t *option, - Long64_t nentries, Long64_t first) -{ - // For the time being cannot accept other queries if not idle, even if in async - // mode; needs to set up an event handler to manage that - - TString opt(option), optfb, outfile; - // Enable feedback, if required - if (opt.Contains("fb=") || opt.Contains("feedback=")) SetFeedback(opt, optfb, 0); - // Define output file, either from 'opt' or the default one - if (HandleOutputOptions(opt, outfile, 0) != 0) return -1; - - // Resolve query mode - fSync = (GetQueryMode(opt) == kSync); - if (!fSync) { - Info("Process","asynchronous mode not yet supported in PROOF-Lite"); - return -1; - } - - if (!IsIdle()) { - // Notify submission - Info("Process", "not idle: cannot accept queries"); - return -1; - } - - // Cleanup old temporary datasets - if (IsIdle() && fRunningDSets && fRunningDSets->GetSize() > 0) { - fRunningDSets->SetOwner(kTRUE); - fRunningDSets->Delete(); - } - - if (!IsValid() || !fQMgr || !fPlayer) { - Error("Process", "invalid sesion or query-result manager undefined!"); - return -1; - } - - // Make sure that all enabled workers get some work, unless stated - // differently - if (!fPlayer->GetInputList()->FindObject("PROOF_MaxSlavesPerNode")) - SetParameter("PROOF_MaxSlavesPerNode", (Long_t)0); - - Bool_t hasNoData = (!dset || (dset && dset->TestBit(TDSet::kEmpty))) ? kTRUE : kFALSE; - - // If just a name was given to identify the dataset, retrieve it from the - // local files - // Make sure the dataset contains the information needed - TString emsg; - if ((!hasNoData) && dset->GetListOfElements()->GetSize() == 0) { - if (TProof::AssertDataSet(dset, fPlayer->GetInputList(), fDataSetManager, emsg) != 0) { - Error("Process", "from AssertDataSet: %s", emsg.Data()); - return -1; - } - if (dset->GetListOfElements()->GetSize() == 0) { - Error("Process", "no files to process!"); - return -1; - } - } else if (hasNoData) { - // Check if we are required to process with TPacketizerFile a registered dataset - TNamed *ftp = dynamic_cast(fPlayer->GetInputList()->FindObject("PROOF_FilesToProcess")); - if (ftp) { - TString dsn(ftp->GetTitle()); - if (!dsn.Contains(":") || dsn.BeginsWith("dataset:")) { - dsn.ReplaceAll("dataset:", ""); - // Make sure we have something in input and a dataset manager - if (!fDataSetManager) { - emsg.Form("dataset manager not initialized!"); - } else { - TFileCollection *fc = 0; - // Get the dataset - if (!(fc = fDataSetManager->GetDataSet(dsn))) { - emsg.Form("requested dataset '%s' does not exists", dsn.Data()); - } else { - TMap *fcmap = TProofServ::GetDataSetNodeMap(fc, emsg); - if (fcmap) { - fPlayer->GetInputList()->Remove(ftp); - delete ftp; - fcmap->SetOwner(kTRUE); - fcmap->SetName("PROOF_FilesToProcess"); - fPlayer->GetInputList()->Add(fcmap); - } - } - } - if (!emsg.IsNull()) { - Error("HandleProcess", "%s", emsg.Data()); - return -1; - } - } - } - } - - TString selec(selector), varexp, selection, objname; - // If a draw query, extract the relevant info - if (selec.BeginsWith("draw:")) { - varexp = fVarExp; - selection = fSelection; - // Decode now the expression - if (fPlayer->GetDrawArgs(varexp, selection, opt, selec, objname) != 0) { - Error("Process", "draw query: error parsing arguments '%s', '%s', '%s'", - varexp.Data(), selection.Data(), opt.Data()); - return -1; - } - } - - // Create instance of query results (the data set is added after Process) - TProofQueryResult *pq = MakeQueryResult(nentries, opt, first, 0, selec); - - // Check if queries must be saved into files - // Automatic saving is controlled by ProofLite.AutoSaveQueries - Bool_t savequeries = - (!strcmp(gEnv->GetValue("ProofLite.AutoSaveQueries", "off"), "on")) ? kTRUE : kFALSE; - - // Keep queries in memory and how many (-1 = all, 0 = none, ...) - Int_t memqueries = gEnv->GetValue("ProofLite.MaxQueriesMemory", 1); - - // If not a draw action add the query to the main list - if (!(pq->IsDraw())) { - if (fQMgr->Queries()) { - if (memqueries != 0) fQMgr->Queries()->Add(pq); - if (memqueries >= 0 && fQMgr->Queries()->GetSize() > memqueries) { - // Remove oldest - TObject *qfst = fQMgr->Queries()->First(); - fQMgr->Queries()->Remove(qfst); - delete qfst; - } - } - // Also save it to queries dir - if (savequeries) fQMgr->SaveQuery(pq); - } - - // Set the query number - fSeqNum = pq->GetSeqNum(); - - // Set in running state - SetQueryRunning(pq); - - // Save to queries dir, if not standard draw - if (!(pq->IsDraw())) { - if (savequeries) fQMgr->SaveQuery(pq); - } else { - fQMgr->IncrementDrawQueries(); - } - - // Start or reset the progress dialog - if (!gROOT->IsBatch()) { - Int_t dsz = (dset && dset->GetListOfElements()) ? dset->GetListOfElements()->GetSize() : -1; - if (fProgressDialog && - !TestBit(kUsingSessionGui) && TestBit(kUseProgressDialog)) { - if (!fProgressDialogStarted) { - fProgressDialog->ExecPlugin(5, this, selec.Data(), dsz, - first, nentries); - fProgressDialogStarted = kTRUE; - } else { - ResetProgressDialog(selec.Data(), dsz, first, nentries); - } - } - ResetBit(kUsingSessionGui); - } - - // Add query results to the player lists - if (!(pq->IsDraw())) - fPlayer->AddQueryResult(pq); - - // Set query currently processed - fPlayer->SetCurrentQuery(pq); - - // Make sure the unique query tag is available as TNamed object in the - // input list so that it can be used in TSelectors for monitoring - TNamed *qtag = (TNamed *) fPlayer->GetInputList()->FindObject("PROOF_QueryTag"); - if (qtag) { - qtag->SetTitle(Form("%s:%s",pq->GetTitle(),pq->GetName())); - } else { - TObject *o = fPlayer->GetInputList()->FindObject("PROOF_QueryTag"); - if (o) fPlayer->GetInputList()->Remove(o); - fPlayer->AddInput(new TNamed("PROOF_QueryTag", - Form("%s:%s",pq->GetTitle(),pq->GetName()))); - } - - // Set PROOF to running state - SetRunStatus(TProof::kRunning); - - // deactivate the default application interrupt handler - // ctrl-c's will be forwarded to PROOF to stop the processing - TSignalHandler *sh = 0; - if (fSync) { - if (gApplication) - sh = gSystem->RemoveSignalHandler(gApplication->GetSignalHandler()); - } - - // Make sure we get a fresh result - fOutputList.Clear(); - - // Start the additional workers now if using fork-based startup - TList *startedWorkers = 0; - if (fForkStartup) { - startedWorkers = new TList; - startedWorkers->SetOwner(kFALSE); - SetupWorkers(1, startedWorkers); - } - - // This is the end of preparation - fQuerySTW.Reset(); - - Long64_t rv = 0; - if (!(pq->IsDraw())) { - if (selector && strlen(selector)) { - rv = fPlayer->Process(dset, selec, opt, nentries, first); - } else { - rv = fPlayer->Process(dset, fSelector, opt, nentries, first); - } - } else { - rv = fPlayer->DrawSelect(dset, varexp, selection, opt, nentries, first); - } - - // This is the end of merging - fQuerySTW.Stop(); - Float_t rt = fQuerySTW.RealTime(); - // Update the query content - TQueryResult *qr = GetQueryResult(); - if (qr) { - qr->SetTermTime(rt); - // Preparation time is always null in PROOF-Lite - } - - // Disable feedback, if required - if (!optfb.IsNull()) SetFeedback(opt, optfb, 1); - - if (fSync) { - - // Terminate additional workers if using fork-based startup - if (fForkStartup && startedWorkers) { - RemoveWorkers(startedWorkers); - SafeDelete(startedWorkers); - } - - // reactivate the default application interrupt handler - if (sh) - gSystem->AddSignalHandler(sh); - - // Return number of events processed - if (fPlayer->GetExitStatus() != TVirtualProofPlayer::kFinished) { - Bool_t abort = (fPlayer->GetExitStatus() == TVirtualProofPlayer::kAborted) - ? kTRUE : kFALSE; - if (abort) fPlayer->StopProcess(kTRUE); - Emit("StopProcess(Bool_t)", abort); - } - - // In PROOFLite this has to be done once only in TProofLite::Process - pq->SetOutputList(fPlayer->GetOutputList(), kFALSE); - // If the last object, notify the GUI that the result arrived - QueryResultReady(Form("%s:%s", pq->GetTitle(), pq->GetName())); - // Processing is over - UpdateDialog(); - - // Save the data set into the TQueryResult (should be done after Process to avoid - // improper deletion during collection) - if (rv == 0 && dset && !dset->TestBit(TDSet::kEmpty) && pq->GetInputList()) { - pq->GetInputList()->Add(dset); - if (dset->GetEntryList()) - pq->GetInputList()->Add(dset->GetEntryList()); - } - - // Register any dataset produced during this processing, if required - if (fDataSetManager && fPlayer->GetOutputList()) { - TNamed *psr = (TNamed *) fPlayer->GetOutputList()->FindObject("PROOFSERV_RegisterDataSet"); - if (psr) { - TString err; - if (TProofServ::RegisterDataSets(fPlayer->GetInputList(), - fPlayer->GetOutputList(), fDataSetManager, err) != 0) - Warning("ProcessNext", "problems registering produced datasets: %s", err.Data()); - fPlayer->GetOutputList()->Remove(psr); - delete psr; - } - } - - // Complete filling of the TQueryResult instance - AskStatistics(); - if (!(pq->IsDraw())) { - if (fQMgr->FinalizeQuery(pq, this, fPlayer)) { - if (savequeries) fQMgr->SaveQuery(pq, -1); - } - } - - // Remove aborted queries from the list - if (fPlayer && fPlayer->GetExitStatus() == TVirtualProofPlayer::kAborted) { - if (fPlayer->GetListOfResults()) fPlayer->GetListOfResults()->Remove(pq); - if (fQMgr) fQMgr->RemoveQuery(pq); - } else { - // If the last object, notify the GUI that the result arrived - QueryResultReady(Form("%s:%s", pq->GetTitle(), pq->GetName())); - // Keep in memory only light info about a query - if (!(pq->IsDraw()) && memqueries >= 0) { - if (fQMgr && fQMgr->Queries()) { - TQueryResult *pqr = pq->CloneInfo(); - if (pqr) fQMgr->Queries()->Add(pqr); - // Remove from the fQueries list - fQMgr->Queries()->Remove(pq); - } - } - // To get the prompt back - TString msg; - msg.Form("Lite-0: all output objects have been merged "); - fprintf(stderr, "%s\n", msg.Data()); - } - // Save the performance info, if required - if (!fPerfTree.IsNull()) { - if (SavePerfTree() != 0) Error("Process", "saving performance info ..."); - // Must be re-enabled each time - SetPerfTree(0); - } - } - // Finalise output file settings (opt is ignored in here) - if (HandleOutputOptions(opt, outfile, 1) != 0) return -1; - - // Retrieve status from the output list - if (rv >= 0) { - TParameter *sst = - (TParameter *) fOutputList.FindObject("PROOF_SelectorStatus"); - if (sst) rv = sst->GetVal(); - } - - - // Done - return rv; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Initialize the dataset manager from directives or from defaults -/// Return 0 on success, -1 on failure - -Int_t TProofLite::InitDataSetManager() -{ - fDataSetManager = 0; - - // Default user and group - TString user("???"), group("default"); - UserGroup_t *pw = gSystem->GetUserInfo(); - if (pw) { - user = pw->fUser; - delete pw; - } - - // Dataset manager instance via plug-in - TPluginHandler *h = 0; - TString dsm = gEnv->GetValue("Proof.DataSetManager", ""); - if (!dsm.IsNull()) { - // Get plugin manager to load the appropriate TDataSetManager - if (gROOT->GetPluginManager()) { - // Find the appropriate handler - h = gROOT->GetPluginManager()->FindHandler("TDataSetManager", dsm); - if (h && h->LoadPlugin() != -1) { - // make instance of the dataset manager - fDataSetManager = - reinterpret_cast(h->ExecPlugin(3, group.Data(), - user.Data(), dsm.Data())); - } - } - } - if (fDataSetManager && fDataSetManager->TestBit(TObject::kInvalidObject)) { - Warning("InitDataSetManager", "dataset manager plug-in initialization failed"); - SafeDelete(fDataSetManager); - } - - // If no valid dataset manager has been created we instantiate the default one - if (!fDataSetManager) { - TString opts("Av:"); - TString dsetdir = gEnv->GetValue("ProofServ.DataSetDir", ""); - if (dsetdir.IsNull()) { - // Use the default in the sandbox - dsetdir = fDataSetDir; - opts += "Sb:"; - } - // Find the appropriate handler - if (!h) { - h = gROOT->GetPluginManager()->FindHandler("TDataSetManager", "file"); - if (h && h->LoadPlugin() == -1) h = 0; - } - if (h) { - // make instance of the dataset manager - fDataSetManager = reinterpret_cast(h->ExecPlugin(3, - group.Data(), user.Data(), - Form("dir:%s opt:%s", dsetdir.Data(), opts.Data()))); - } - if (fDataSetManager && fDataSetManager->TestBit(TObject::kInvalidObject)) { - Warning("InitDataSetManager", "default dataset manager plug-in initialization failed"); - SafeDelete(fDataSetManager); - } - } - - if (gDebug > 0 && fDataSetManager) { - Info("InitDataSetManager", "datasetmgr Cq: %d, Ar: %d, Av: %d, Ti: %d, Sb: %d", - fDataSetManager->TestBit(TDataSetManager::kCheckQuota), - fDataSetManager->TestBit(TDataSetManager::kAllowRegister), - fDataSetManager->TestBit(TDataSetManager::kAllowVerify), - fDataSetManager->TestBit(TDataSetManager::kTrustInfo), - fDataSetManager->TestBit(TDataSetManager::kIsSandbox)); - } - - // Dataset manager for staging requests - TString dsReqCfg = gEnv->GetValue("Proof.DataSetStagingRequests", ""); - if (!dsReqCfg.IsNull()) { - TPMERegexp reReqDir("(^| )(dir:)?([^ ]+)( |$)"); - - if (reReqDir.Match(dsReqCfg) == 5) { - TString dsDirFmt; - dsDirFmt.Form("dir:%s perms:open", reReqDir[3].Data()); - fDataSetStgRepo = new TDataSetManagerFile("_stage_", "_stage_", dsDirFmt); - if (fDataSetStgRepo && fDataSetStgRepo->TestBit(TObject::kInvalidObject)) { - Warning("InitDataSetManager", "failed init of dataset staging requests repository"); - SafeDelete(fDataSetStgRepo); - } - } else { - Warning("InitDataSetManager", "specify, with [dir:], a valid path for staging requests"); - } - } else if (gDebug > 0) { - Warning("InitDataSetManager", "no repository for staging requests available"); - } - - // Done - return (fDataSetManager ? 0 : -1); -} - -//////////////////////////////////////////////////////////////////////////////// -/// List contents of file cache. If all is true show all caches also on -/// slaves. If everything is ok all caches are to be the same. - -void TProofLite::ShowCache(Bool_t) -{ - if (!IsValid()) return; - - Printf("*** Local file cache %s ***", fCacheDir.Data()); - gSystem->Exec(Form("%s %s", kLS, fCacheDir.Data())); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove files from all file caches. - -void TProofLite::ClearCache(const char *file) -{ - if (!IsValid()) return; - - fCacheLock->Lock(); - if (!file || strlen(file) <= 0) { - gSystem->Exec(Form("%s %s/*", kRM, fCacheDir.Data())); - } else { - gSystem->Exec(Form("%s %s/%s", kRM, fCacheDir.Data(), file)); - } - fCacheLock->Unlock(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Copy the specified macro in the cache directory. The macro file is -/// uploaded if new or updated. If existing, the corresponding header -/// basename(macro).h or .hh, is also uploaded. For the other arguments -/// see TProof::Load(). -/// Returns 0 in case of success and -1 in case of error. - -Int_t TProofLite::Load(const char *macro, Bool_t notOnClient, Bool_t uniqueOnly, - TList *wrks) -{ - if (!IsValid()) return -1; - - if (!macro || !macro[0]) { - Error("Load", "need to specify a macro name"); - return -1; - } - - TString macs(macro), mac; - Int_t from = 0; - while (macs.Tokenize(mac, from, ",")) { - if (IsIdle()) { - if (CopyMacroToCache(mac) < 0) return -1; - } else { - // The name - TString macn = gSystem->BaseName(mac); - macn.Remove(macn.Last('.')); - // Relevant pointers - TList cachedFiles; - TString cacheDir = fCacheDir; - gSystem->ExpandPathName(cacheDir); - void * dirp = gSystem->OpenDirectory(cacheDir); - if (dirp) { - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - if (!strncmp(e, macn.Data(), macn.Length())) { - TString fncache = Form("%s/%s", cacheDir.Data(), e); - cachedFiles.Add(new TObjString(fncache.Data())); - } - } - gSystem->FreeDirectory(dirp); - } - } - } - - return TProof::Load(macro, notOnClient, uniqueOnly, wrks); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Copy a macro, and its possible associated .h[h] file, -/// to the cache directory, from where the workers can get the file. -/// If headerRequired is 1, return -1 in case the header is not found. -/// If headerRequired is 0, try to copy header too. -/// If headerRequired is -1, don't look for header, only copy macro. -/// If the selector pionter is not 0, consider the macro to be a selector -/// and try to load the selector and set it to the pointer. -/// The mask 'opt' is an or of ESendFileOpt: -/// kCpBin (0x8) Retrieve from the cache the binaries associated -/// with the file -/// kCp (0x10) Retrieve the files from the cache -/// Return -1 in case of error, 0 otherwise. - -Int_t TProofLite::CopyMacroToCache(const char *macro, Int_t headerRequired, - TSelector **selector, Int_t opt, TList *) -{ - // Relevant pointers - TString cacheDir = fCacheDir; - gSystem->ExpandPathName(cacheDir); - TProofLockPath *cacheLock = fCacheLock; - - // Split out the aclic mode, if any - TString name = macro; - TString acmode, args, io; - name = gSystem->SplitAclicMode(name, acmode, args, io); - - PDB(kGlobal,1) - Info("CopyMacroToCache", "enter: names: %s, %s", macro, name.Data()); - - // Make sure that the file exists - if (gSystem->AccessPathName(name, kReadPermission)) { - Error("CopyMacroToCache", "file %s not found or not readable", name.Data()); - return -1; - } - - // Update the macro path - TString mp(TROOT::GetMacroPath()); - TString np = gSystem->GetDirName(name); - if (!np.IsNull()) { - np += ":"; - if (!mp.BeginsWith(np) && !mp.Contains(":"+np)) { - Int_t ip = (mp.BeginsWith(".:")) ? 2 : 0; - mp.Insert(ip, np); - TROOT::SetMacroPath(mp); - PDB(kGlobal,1) - Info("CopyMacroToCache", "macro path set to '%s'", TROOT::GetMacroPath()); - } - } - - // Check the header file - Int_t dot = name.Last('.'); - const char *hext[] = { ".h", ".hh", "" }; - TString hname, checkedext; - Int_t i = 0; - while (strlen(hext[i]) > 0) { - hname = name(0, dot); - hname += hext[i]; - if (!gSystem->AccessPathName(hname, kReadPermission)) - break; - if (!checkedext.IsNull()) checkedext += ","; - checkedext += hext[i]; - hname = ""; - i++; - } - if (hname.IsNull() && headerRequired == 1) { - Error("CopyMacroToCache", "header file for %s not found or not readable " - "(checked extensions: %s)", name.Data(), checkedext.Data()); - return -1; - } - if (headerRequired < 0) - hname = ""; - - cacheLock->Lock(); - - // Check these files with those in the cache (if any) - Bool_t useCacheBinaries = kFALSE; - TString cachedname = Form("%s/%s", cacheDir.Data(), gSystem->BaseName(name)); - TString cachedhname; - if (!hname.IsNull()) - cachedhname = Form("%s/%s", cacheDir.Data(), gSystem->BaseName(hname)); - if (!gSystem->AccessPathName(cachedname, kReadPermission)) { - TMD5 *md5 = TMD5::FileChecksum(name); - TMD5 *md5cache = TMD5::FileChecksum(cachedname); - if (md5 && md5cache && (*md5 == *md5cache)) - useCacheBinaries = kTRUE; - if (!hname.IsNull()) { - if (!gSystem->AccessPathName(cachedhname, kReadPermission)) { - TMD5 *md5h = TMD5::FileChecksum(hname); - TMD5 *md5hcache = TMD5::FileChecksum(cachedhname); - if (md5h && md5hcache && (*md5h != *md5hcache)) - useCacheBinaries = kFALSE; - SafeDelete(md5h); - SafeDelete(md5hcache); - } - } - SafeDelete(md5); - SafeDelete(md5cache); - } - - // Create version file name template - TString vername(Form(".%s", name.Data())); - dot = vername.Last('.'); - if (dot != kNPOS) - vername.Remove(dot); - vername += ".binversion"; - Bool_t savever = kFALSE; - - // Check binary version - if (useCacheBinaries) { - TString v, r; - FILE *f = fopen(Form("%s/%s", cacheDir.Data(), vername.Data()), "r"); - if (f) { - v.Gets(f); - r.Gets(f); - fclose(f); - } - if (!f || v != gROOT->GetVersion() || r != gROOT->GetGitCommit()) - useCacheBinaries = kFALSE; - } - - // Create binary name template - TString binname = gSystem->BaseName(name); - dot = binname.Last('.'); - if (dot != kNPOS) - binname.Replace(dot,1,"_"); - TString pcmname = TString::Format("%s_ACLiC_dict_rdict.pcm", binname.Data()); - binname += "."; - - FileStat_t stlocal, stcache; - void *dirp = 0; - if (useCacheBinaries) { - // Loop over binaries in the cache and copy them locally if newer then the local - // versions or there is no local version - dirp = gSystem->OpenDirectory(cacheDir); - if (dirp) { - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - if (!strncmp(e, binname.Data(), binname.Length()) || - !strncmp(e, pcmname.Data(), pcmname.Length())) { - TString fncache = Form("%s/%s", cacheDir.Data(), e); - Bool_t docp = kTRUE; - if (!gSystem->GetPathInfo(fncache, stcache)) { - Int_t rc = gSystem->GetPathInfo(e, stlocal); - if (rc == 0 && (stlocal.fMtime >= stcache.fMtime)) - docp = kFALSE; - // Copy the file, if needed - if (docp) { - gSystem->Exec(Form("%s %s", kRM, e)); - PDB(kGlobal,2) - Info("CopyMacroToCache", - "retrieving %s from cache", fncache.Data()); - gSystem->Exec(Form("%s %s %s", kCP, fncache.Data(), e)); - } - } - } - } - gSystem->FreeDirectory(dirp); - } - } - cacheLock->Unlock(); - - if (selector) { - // Now init the selector in optimized way - if (!(*selector = TSelector::GetSelector(macro))) { - Error("CopyMacroToCache", "could not create a selector from %s", macro); - return -1; - } - } - - cacheLock->Lock(); - - TList *cachedFiles = new TList; - // Save information in the cache now for later usage - dirp = gSystem->OpenDirectory("."); - if (dirp) { - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - if (!strncmp(e, binname.Data(), binname.Length()) || - !strncmp(e, pcmname.Data(), pcmname.Length())) { - Bool_t docp = kTRUE; - if (!gSystem->GetPathInfo(e, stlocal)) { - TString fncache = Form("%s/%s", cacheDir.Data(), e); - Int_t rc = gSystem->GetPathInfo(fncache, stcache); - if (rc == 0 && (stlocal.fMtime <= stcache.fMtime)) - docp = kFALSE; - // Copy the file, if needed - if (docp) { - gSystem->Exec(Form("%s %s", kRM, fncache.Data())); - PDB(kGlobal,2) - Info("CopyMacroToCache","caching %s ...", e); - gSystem->Exec(Form("%s %s %s", kCP, e, fncache.Data())); - savever = kTRUE; - } - if (opt & kCpBin) - cachedFiles->Add(new TObjString(fncache.Data())); - } - } - } - gSystem->FreeDirectory(dirp); - } - - // Save binary version if requested - if (savever) { - FILE *f = fopen(Form("%s/%s", cacheDir.Data(), vername.Data()), "w"); - if (f) { - fputs(gROOT->GetVersion(), f); - fputs(Form("\n%s", gROOT->GetGitCommit()), f); - fclose(f); - } - } - - // Save also the selector info, if needed - if (!useCacheBinaries) { - gSystem->Exec(Form("%s %s", kRM, cachedname.Data())); - PDB(kGlobal,2) - Info("CopyMacroToCache","caching %s ...", name.Data()); - gSystem->Exec(Form("%s %s %s", kCP, name.Data(), cachedname.Data())); - if (!hname.IsNull()) { - gSystem->Exec(Form("%s %s", kRM, cachedhname.Data())); - PDB(kGlobal,2) - Info("CopyMacroToCache","caching %s ...", hname.Data()); - gSystem->Exec(Form("%s %s %s", kCP, hname.Data(), cachedhname.Data())); - } - } - if (opt & kCp) { - cachedFiles->Add(new TObjString(cachedname.Data())); - if (!hname.IsNull()) - cachedFiles->Add(new TObjString(cachedhname.Data())); - } - - cacheLock->Unlock(); - - cachedFiles->SetOwner(); - delete cachedFiles; - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove old sessions dirs keep at most 'Proof.MaxOldSessions' (default 10) - -Int_t TProofLite::CleanupSandbox() -{ - Int_t maxold = gEnv->GetValue("Proof.MaxOldSessions", 1); - - if (maxold < 0) return 0; - - TSortedList *olddirs = new TSortedList(kFALSE); - - TString sandbox = gSystem->GetDirName(fWorkDir.Data()); - - void *dirp = gSystem->OpenDirectory(sandbox); - if (dirp) { - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - if (!strncmp(e, "session-", 8) && !strstr(e, GetName())) { - TString d(e); - Int_t i = d.Last('-'); - if (i != kNPOS) d.Remove(i); - i = d.Last('-'); - if (i != kNPOS) d.Remove(0,i+1); - TString path = Form("%s/%s", sandbox.Data(), e); - olddirs->Add(new TNamed(d, path)); - } - } - gSystem->FreeDirectory(dirp); - } - - // Clean it up, if required - Bool_t notify = kTRUE; - while (olddirs->GetSize() > maxold) { - if (notify && gDebug > 0) - Printf("Cleaning sandbox at: %s", sandbox.Data()); - notify = kFALSE; - TNamed *n = (TNamed *) olddirs->Last(); - if (n) { - gSystem->Exec(Form("%s %s", kRM, n->GetTitle())); - olddirs->Remove(n); - delete n; - } - } - - // Cleanup - olddirs->SetOwner(); - delete olddirs; - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the list of queries. - -TList *TProofLite::GetListOfQueries(Option_t *opt) -{ - Bool_t all = ((strchr(opt,'A') || strchr(opt,'a'))) ? kTRUE : kFALSE; - - TList *ql = new TList; - Int_t ntot = 0, npre = 0, ndraw= 0; - if (fQMgr) { - if (all) { - // Rescan - TString qdir = fQueryDir; - Int_t idx = qdir.Index("session-"); - if (idx != kNPOS) - qdir.Remove(idx); - fQMgr->ScanPreviousQueries(qdir); - // Gather also information about previous queries, if any - if (fQMgr->PreviousQueries()) { - TIter nxq(fQMgr->PreviousQueries()); - TProofQueryResult *pqr = 0; - while ((pqr = (TProofQueryResult *)nxq())) { - ntot++; - pqr->fSeqNum = ntot; - ql->Add(pqr); - } - } - } - - npre = ntot; - if (fQMgr->Queries()) { - // Add info about queries in this session - TIter nxq(fQMgr->Queries()); - TProofQueryResult *pqr = 0; - TQueryResult *pqm = 0; - while ((pqr = (TProofQueryResult *)nxq())) { - ntot++; - if ((pqm = pqr->CloneInfo())) { - pqm->fSeqNum = ntot; - ql->Add(pqm); - } else { - Warning("GetListOfQueries", "unable to clone TProofQueryResult '%s:%s'", - pqr->GetName(), pqr->GetTitle()); - } - } - } - // Number of draw queries - ndraw = fQMgr->DrawQueries(); - } - - fOtherQueries = npre; - fDrawQueries = ndraw; - if (fQueries) { - fQueries->Delete(); - delete fQueries; - fQueries = 0; - } - fQueries = ql; - - // This should have been filled by now - return fQueries; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Register the 'dataSet' on the cluster under the current -/// user, group and the given 'dataSetName'. -/// Fails if a dataset named 'dataSetName' already exists, unless 'optStr' -/// contains 'O', in which case the old dataset is overwritten. -/// If 'optStr' contains 'V' the dataset files are verified (default no -/// verification). -/// Returns kTRUE on success. - -Bool_t TProofLite::RegisterDataSet(const char *uri, - TFileCollection *dataSet, const char* optStr) -{ - if (!fDataSetManager) { - Info("RegisterDataSet", "dataset manager not available"); - return kFALSE; - } - - if (!uri || strlen(uri) <= 0) { - Info("RegisterDataSet", "specifying a dataset name is mandatory"); - return kFALSE; - } - - Bool_t parallelverify = kFALSE; - TString sopt(optStr); - if (sopt.Contains("V") && !sopt.Contains("S")) { - // We do verification in parallel later on; just register for now - parallelverify = kTRUE; - sopt.ReplaceAll("V", ""); - } - // This would screw up things remotely, make sure is not there - sopt.ReplaceAll("S", ""); - - Bool_t result = kTRUE; - if (fDataSetManager->TestBit(TDataSetManager::kAllowRegister)) { - // Check the list - if (!dataSet || dataSet->GetList()->GetSize() == 0) { - Error("RegisterDataSet", "can not save an empty list."); - result = kFALSE; - } - // Register the dataset (quota checks are done inside here) - result = (fDataSetManager->RegisterDataSet(uri, dataSet, sopt) == 0) - ? kTRUE : kFALSE; - } else { - Info("RegisterDataSet", "dataset registration not allowed"); - result = kFALSE; - } - - if (!result) - Error("RegisterDataSet", "dataset was not saved"); - - // If old server or not verifying in parallel we are done - if (!parallelverify) return result; - - // If we are here it means that we will verify in parallel - sopt += "V"; - if (VerifyDataSet(uri, sopt) < 0){ - Error("RegisterDataSet", "problems verifying dataset '%s'", uri); - return kFALSE; - } - - // Done - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set/Change the name of the default tree. The tree name may contain -/// subdir specification in the form "subdir/name". -/// Returns 0 on success, -1 otherwise. - -Int_t TProofLite::SetDataSetTreeName(const char *dataset, const char *treename) -{ - if (!fDataSetManager) { - Info("ExistsDataSet", "dataset manager not available"); - return kFALSE; - } - - if (!dataset || strlen(dataset) <= 0) { - Info("SetDataSetTreeName", "specifying a dataset name is mandatory"); - return -1; - } - - if (!treename || strlen(treename) <= 0) { - Info("SetDataSetTreeName", "specifying a tree name is mandatory"); - return -1; - } - - TUri uri(dataset); - TString fragment(treename); - if (!fragment.BeginsWith("/")) fragment.Insert(0, "/"); - uri.SetFragment(fragment); - - return fDataSetManager->ScanDataSet(uri.GetUri().Data(), - (UInt_t)TDataSetManager::kSetDefaultTree); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Returns kTRUE if 'dataset' described by 'uri' exists, kFALSE otherwise - -Bool_t TProofLite::ExistsDataSet(const char *uri) -{ - if (!fDataSetManager) { - Info("ExistsDataSet", "dataset manager not available"); - return kFALSE; - } - - if (!uri || strlen(uri) <= 0) { - Error("ExistsDataSet", "dataset name missing"); - return kFALSE; - } - - // Check if the dataset exists - return fDataSetManager->ExistsDataSet(uri); -} - -//////////////////////////////////////////////////////////////////////////////// -/// lists all datasets that match given uri - -TMap *TProofLite::GetDataSets(const char *uri, const char *srvex) -{ - if (!fDataSetManager) { - Info("GetDataSets", "dataset manager not available"); - return (TMap *)0; - } - - // Get the datasets and return the map - if (srvex && strlen(srvex) > 0) { - return fDataSetManager->GetSubDataSets(uri, srvex); - } else { - UInt_t opt = (UInt_t)TDataSetManager::kExport; - return fDataSetManager->GetDataSets(uri, opt); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Shows datasets in locations that match the uri -/// By default shows the user's datasets and global ones - -void TProofLite::ShowDataSets(const char *uri, const char *opt) -{ - if (!fDataSetManager) { - Info("GetDataSet", "dataset manager not available"); - return; - } - - fDataSetManager->ShowDataSets(uri, opt); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get a list of TFileInfo objects describing the files of the specified -/// dataset. - -TFileCollection *TProofLite::GetDataSet(const char *uri, const char *) -{ - if (!fDataSetManager) { - Info("GetDataSet", "dataset manager not available"); - return (TFileCollection *)0; - } - - if (!uri || strlen(uri) <= 0) { - Info("GetDataSet", "specifying a dataset name is mandatory"); - return 0; - } - - // Return the list - return fDataSetManager->GetDataSet(uri); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove the specified dataset from the PROOF cluster. -/// Files are not deleted. - -Int_t TProofLite::RemoveDataSet(const char *uri, const char *) -{ - if (!fDataSetManager) { - Info("RemoveDataSet", "dataset manager not available"); - return -1; - } - - if (fDataSetManager->TestBit(TDataSetManager::kAllowRegister)) { - if (!fDataSetManager->RemoveDataSet(uri)) { - // Failure - return -1; - } - } else { - Info("RemoveDataSet", "dataset creation / removal not allowed"); - return -1; - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Allows users to request staging of a particular dataset. Requests are -/// saved in a special dataset repository and must be honored by the endpoint. -/// This is the special PROOF-Lite re-implementation of the TProof function -/// and includes code originally implemented in TProofServ. - -Bool_t TProofLite::RequestStagingDataSet(const char *dataset) -{ - if (!dataset) { - Error("RequestStagingDataSet", "invalid dataset specified"); - return kFALSE; - } - - if (!fDataSetStgRepo) { - Error("RequestStagingDataSet", "no dataset staging request repository available"); - return kFALSE; - } - - TString dsUser, dsGroup, dsName, dsTree; - - // Transform input URI in a valid dataset name - TString validUri = dataset; - while (fReInvalid->Substitute(validUri, "_")) {} - - // Check if dataset exists beforehand: if it does, staging has already been requested - if (fDataSetStgRepo->ExistsDataSet(validUri.Data())) { - Warning("RequestStagingDataSet", "staging of %s already requested", dataset); - return kFALSE; - } - - // Try to get dataset from current manager - TFileCollection *fc = fDataSetManager->GetDataSet(dataset); - if (!fc || (fc->GetNFiles() == 0)) { - Error("RequestStagingDataSet", "empty dataset or no dataset returned"); - if (fc) delete fc; - return kFALSE; - } - - // Reset all staged bits and remove unnecessary URLs (all but last) - TIter it(fc->GetList()); - TFileInfo *fi; - while ((fi = dynamic_cast(it.Next()))) { - fi->ResetBit(TFileInfo::kStaged); - Int_t nToErase = fi->GetNUrls() - 1; - for (Int_t i=0; iRemoveUrlAt(0); - } - - fc->Update(); // absolutely necessary - - // Save request - fDataSetStgRepo->ParseUri(validUri, &dsGroup, &dsUser, &dsName); - if (fDataSetStgRepo->WriteDataSet(dsGroup, dsUser, dsName, fc) == 0) { - // Error, can't save dataset - Error("RequestStagingDataSet", "can't register staging request for %s", dataset); - delete fc; - return kFALSE; - } - - Info("RequestStagingDataSet", "Staging request registered for %s", dataset); - delete fc; - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cancels a dataset staging request. Returns kTRUE on success, kFALSE on -/// failure. Dataset not found equals to a failure. PROOF-Lite -/// re-implementation of the equivalent function in TProofServ. - -Bool_t TProofLite::CancelStagingDataSet(const char *dataset) -{ - if (!dataset) { - Error("CancelStagingDataSet", "invalid dataset specified"); - return kFALSE; - } - - if (!fDataSetStgRepo) { - Error("CancelStagingDataSet", "no dataset staging request repository available"); - return kFALSE; - } - - // Transform URI in a valid dataset name - TString validUri = dataset; - while (fReInvalid->Substitute(validUri, "_")) {} - - if (!fDataSetStgRepo->RemoveDataSet(validUri.Data())) - return kFALSE; - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Obtains a TFileCollection showing the staging status of the specified -/// dataset. A valid dataset manager and dataset staging requests repository -/// must be present on the endpoint. PROOF-Lite version of the equivalent -/// function from TProofServ. - -TFileCollection *TProofLite::GetStagingStatusDataSet(const char *dataset) -{ - if (!dataset) { - Error("GetStagingStatusDataSet", "invalid dataset specified"); - return 0; - } - - if (!fDataSetStgRepo) { - Error("GetStagingStatusDataSet", "no dataset staging request repository available"); - return 0; - } - - // Transform URI in a valid dataset name - TString validUri = dataset; - while (fReInvalid->Substitute(validUri, "_")) {} - - // Get the list - TFileCollection *fc = fDataSetStgRepo->GetDataSet(validUri.Data()); - if (!fc) { - // No such dataset (not an error) - Info("GetStagingStatusDataSet", "no pending staging request for %s", dataset); - return 0; - } - - // Dataset found: return it (must be cleaned by caller) - return fc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Verify if all files in the specified dataset are available. -/// Print a list and return the number of missing files. - -Int_t TProofLite::VerifyDataSet(const char *uri, const char *optStr) -{ - if (!fDataSetManager) { - Info("VerifyDataSet", "dataset manager not available"); - return -1; - } - - Int_t rc = -1; - TString sopt(optStr); - if (sopt.Contains("S")) { - - if (fDataSetManager->TestBit(TDataSetManager::kAllowVerify)) { - rc = fDataSetManager->ScanDataSet(uri); - } else { - Info("VerifyDataSet", "dataset verification not allowed"); - rc = -1; - } - return rc; - } - - // Done - return VerifyDataSetParallel(uri, optStr); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear the content of the dataset cache, if any (matching 'dataset', if defined). - -void TProofLite::ClearDataSetCache(const char *dataset) -{ - if (fDataSetManager) fDataSetManager->ClearCache(dataset); - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display the content of the dataset cache, if any (matching 'dataset', if defined). - -void TProofLite::ShowDataSetCache(const char *dataset) -{ - // For PROOF-Lite act locally - if (fDataSetManager) fDataSetManager->ShowCache(dataset); - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that the input data objects are available to the workers in a -/// dedicated file in the cache; the objects are taken from the dedicated list -/// and / or the specified file. -/// If the fInputData is empty the specified file is sent over. -/// If there is no specified file, a file named "inputdata.root" is created locally -/// with the content of fInputData and sent over to the master. -/// If both fInputData and the specified file are not empty, a copy of the file -/// is made locally and augmented with the content of fInputData. - -void TProofLite::SendInputDataFile() -{ - // Prepare the file - TString dataFile; - PrepareInputDataFile(dataFile); - - // Make sure it is in the cache, if not empty - if (dataFile.Length() > 0) { - - if (!dataFile.BeginsWith(fCacheDir)) { - // Destination - TString dst; - dst.Form("%s/%s", fCacheDir.Data(), gSystem->BaseName(dataFile)); - // Remove it first if it exists - if (!gSystem->AccessPathName(dst)) - gSystem->Unlink(dst); - // Copy the file - if (gSystem->CopyFile(dataFile, dst) != 0) - Warning("SendInputDataFile", "problems copying '%s' to '%s'", - dataFile.Data(), dst.Data()); - } - - // Set the name in the input list so that the workers can find it - AddInput(new TNamed("PROOF_InputDataFile", Form("%s", gSystem->BaseName(dataFile)))); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle remove request. - -Int_t TProofLite::Remove(const char *ref, Bool_t all) -{ - PDB(kGlobal, 1) - Info("Remove", "Enter: %s, %d", ref, all); - - if (all) { - // Remove also local copies, if any - if (fPlayer) - fPlayer->RemoveQueryResult(ref); - } - - TString queryref(ref); - - if (queryref == "cleanupdir") { - - // Cleanup previous sessions results - Int_t nd = (fQMgr) ? fQMgr->CleanupQueriesDir() : -1; - - // Notify - Info("Remove", "%d directories removed", nd); - // We are done - return 0; - } - - - if (fQMgr) { - TProofLockPath *lck = 0; - if (fQMgr->LockSession(queryref, &lck) == 0) { - - // Remove query - fQMgr->RemoveQuery(queryref, 0); - - // Unlock and remove the lock file - if (lck) { - gSystem->Unlink(lck->GetName()); - SafeDelete(lck); - } - - // We are done - return 0; - } - } else { - Warning("Remove", "query result manager undefined!"); - } - - // Notify failure - Info("Remove", - "query %s could not be removed (unable to lock session)", queryref.Data()); - - // Done - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Creates a tree header (a tree with nonexisting files) object for -/// the DataSet. - -TTree *TProofLite::GetTreeHeader(TDSet *dset) -{ - TTree *t = 0; - if (!dset) { - Error("GetTreeHeader", "undefined TDSet"); - return t; - } - - dset->Reset(); - TDSetElement *e = dset->Next(); - Long64_t entries = 0; - TFile *f = 0; - if (!e) { - PDB(kGlobal, 1) Info("GetTreeHeader", "empty TDSet"); - } else { - f = TFile::Open(e->GetFileName()); - t = 0; - if (f) { - t = (TTree*) f->Get(e->GetObjName()); - if (t) { - t->SetMaxVirtualSize(0); - t->DropBaskets(); - entries = t->GetEntries(); - - // compute #entries in all the files - while ((e = dset->Next()) != 0) { - TFile *f1 = TFile::Open(e->GetFileName()); - if (f1) { - TTree *t1 = (TTree*) f1->Get(e->GetObjName()); - if (t1) { - entries += t1->GetEntries(); - delete t1; - } - delete f1; - } - } - t->SetMaxEntryLoop(entries); // this field will hold the total number of entries ;) - } - } - } - // Done - return t; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add to the fUniqueSlave list the active slaves that have a unique -/// (user) file system image. This information is used to transfer files -/// only once to nodes that share a file system (an image). Submasters -/// which are not in fUniqueSlaves are put in the fNonUniqueMasters -/// list. That list is used to trigger the transferring of files to -/// the submaster's unique slaves without the need to transfer the file -/// to the submaster. - -void TProofLite::FindUniqueSlaves() -{ - fUniqueSlaves->Clear(); - fUniqueMonitor->RemoveAll(); - fAllUniqueSlaves->Clear(); - fAllUniqueMonitor->RemoveAll(); - fNonUniqueMasters->Clear(); - - if (fActiveSlaves->GetSize() <= 0) return; - - TSlave *wrk = dynamic_cast(fActiveSlaves->First()); - if (!wrk) { - Error("FindUniqueSlaves", "first object in fActiveSlaves not a TSlave: embarrasing!"); - return; - } - fUniqueSlaves->Add(wrk); - fAllUniqueSlaves->Add(wrk); - fUniqueMonitor->Add(wrk->GetSocket()); - fAllUniqueMonitor->Add(wrk->GetSocket()); - - // will be actiavted in Collect() - fUniqueMonitor->DeActivateAll(); - fAllUniqueMonitor->DeActivateAll(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// List contents of the data directory in the sandbox. -/// This is the place where files produced by the client queries are kept - -void TProofLite::ShowData() -{ - if (!IsValid()) return; - - // Get worker infos - TList *wrki = GetListOfSlaveInfos(); - TSlaveInfo *wi = 0; - TIter nxwi(wrki); - while ((wi = (TSlaveInfo *) nxwi())) { - ShowDataDir(wi->GetDataDir()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// List contents of the data directory 'dirname' - -void TProofLite::ShowDataDir(const char *dirname) -{ - if (!dirname) return; - - FileStat_t dirst; - if (gSystem->GetPathInfo(dirname, dirst) != 0) return; - if (!R_ISDIR(dirst.fMode)) return; - - void *dirp = gSystem->OpenDirectory(dirname); - TString fn; - const char *ent = 0; - while ((ent = gSystem->GetDirEntry(dirp))) { - fn.Form("%s/%s", dirname, ent); - FileStat_t st; - if (gSystem->GetPathInfo(fn.Data(), st) == 0) { - if (R_ISREG(st.fMode)) { - Printf("lite:0| %s", fn.Data()); - } else if (R_ISREG(st.fMode)) { - ShowDataDir(fn.Data()); - } - } - } - gSystem->FreeDirectory(dirp); - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Simulate dynamic addition, for test purposes. -/// Here we decide how many workers to add, we create them and set the -/// environment. -/// This call is called regularly by Collect if the opton is enabled. -/// Returns the number of new workers added, or <0 on errors. - -Int_t TProofLite::PollForNewWorkers() -{ - // Max workers - if (fDynamicStartupNMax <= 0) { - SysInfo_t si; - if (gSystem->GetSysInfo(&si) == 0 && si.fCpus > 2) { - fDynamicStartupNMax = si.fCpus; - } else { - fDynamicStartupNMax = 2; - } - } - if (fNWorkers >= fDynamicStartupNMax) { - // Max reached: disable - Info("PollForNewWorkers", "max reached: %d workers started", fNWorkers); - fDynamicStartup = kFALSE; - return 0; - } - - // Number of new workers - Int_t nAdd = (fDynamicStartupStep > 0) ? fDynamicStartupStep : 1; - - // Create a monitor and add the socket to it - TMonitor *mon = new TMonitor; - mon->Add(fServSock); - - TList started; - TSlave *wrk = 0; - Int_t nWrksDone = 0, nWrksTot = -1; - TString fullord; - - nWrksTot = fNWorkers + nAdd; - // Now we create the worker applications which will call us back to finalize - // the setup - Int_t ord = fNWorkers; - for (; ord < nWrksTot; ord++) { - - // Ordinal for this worker server - fullord = Form("0.%d", ord); - - // Create environment files - SetProofServEnv(fullord); - - // Create worker server and add to the list - if ((wrk = CreateSlave("lite", fullord, 100, fImage, fWorkDir))) - started.Add(wrk); - - PDB(kGlobal, 3) - Info("PollForNewWorkers", "additional worker '%s' started", fullord.Data()); - - // Notify - NotifyStartUp("Opening connections to workers", ++nWrksDone, nWrksTot); - - } //end of worker loop - fNWorkers = nWrksTot; - - // A list of TSlave objects for workers that are being added - TList *addedWorkers = new TList(); - addedWorkers->SetOwner(kFALSE); - - // Wait for call backs - nWrksDone = 0; - nWrksTot = started.GetSize(); - Int_t nSelects = 0; - Int_t to = gEnv->GetValue("ProofLite.StartupTimeOut", 5) * 1000; - while (started.GetSize() > 0 && nSelects < nWrksTot) { - - // Wait for activity on the socket for max 5 secs - TSocket *xs = mon->Select(to); - - // Count attempts and check - nSelects++; - if (xs == (TSocket *) -1) continue; - - // Get the connection - TSocket *s = fServSock->Accept(); - if (s && s->IsValid()) { - // Receive ordinal - TMessage *msg = 0; - if (s->Recv(msg) < 0) { - Warning("PollForNewWorkers", "problems receiving message from accepted socket!"); - } else { - if (msg) { - *msg >> fullord; - // Find who is calling back - if ((wrk = (TSlave *) started.FindObject(fullord))) { - // Remove it from the started list - started.Remove(wrk); - - // Assign tis socket the selected worker - wrk->SetSocket(s); - // Remove socket from global TROOT socket list. Only the TProof object, - // representing all worker sockets, will be added to this list. This will - // ensure the correct termination of all proof servers in case the - // root session terminates. - { R__LOCKGUARD(gROOTMutex); - gROOT->GetListOfSockets()->Remove(s); - } - if (wrk->IsValid()) { - // Set the input handler - wrk->SetInputHandler(new TProofInputHandler(this, wrk->GetSocket())); - // Set fParallel to 1 for workers since they do not - // report their fParallel with a LOG_DONE message - wrk->fParallel = 1; - // Finalize setup of the server - wrk->SetupServ(TSlave::kSlave, 0); - } - - // Monitor good workers - fSlaves->Add(wrk); - if (wrk->IsValid()) { - fActiveSlaves->Add(wrk); // Is this required? Check! - fAllMonitor->Add(wrk->GetSocket()); - // Record also in the list for termination - addedWorkers->Add(wrk); - // Notify startup operations - NotifyStartUp("Setting up added worker servers", ++nWrksDone, nWrksTot); - } else { - // Flag as bad - fBadSlaves->Add(wrk); - } - } - } else { - Warning("PollForNewWorkers", "received empty message from accepted socket!"); - } - } - } - } - - // Cleanup the monitor and the server socket - mon->DeActivateAll(); - delete mon; - - Broadcast(kPROOF_GETSTATS, addedWorkers); - Collect(addedWorkers, fCollectTimeout); - - // Update group view - // SendGroupView(); - - // By default go into parallel mode - // SetParallel(-1, 0); - SendCurrentState(addedWorkers); - - // Set worker processing environment - SetupWorkersEnv(addedWorkers, kTRUE); - - // We are adding workers dynamically to an existing process, we - // should invoke a special player's Process() to set only added workers - // to the proper state - if (fPlayer) { - PDB(kGlobal, 3) - Info("PollForNewWorkers", "Will send the PROCESS message to selected workers"); - fPlayer->JoinProcess(addedWorkers); - } - - // Cleanup fwhat remained from startup - Collect(addedWorkers); - - // Activate - TIter naw(addedWorkers); - while ((wrk = (TSlave *)naw())) { - fActiveMonitor->Add(wrk->GetSocket()); - } - // Cleanup - delete addedWorkers; - - // Done - return nWrksDone; -} diff --git a/proof/proof/src/TProofLog.cxx b/proof/proof/src/TProofLog.cxx deleted file mode 100644 index 2f6078bc83537..0000000000000 --- a/proof/proof/src/TProofLog.cxx +++ /dev/null @@ -1,580 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis 31/08/06 - -/************************************************************************* - * Copyright (C) 1995-2006, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofLog -\ingroup proofkernel - -Implementation of the PROOF session log handler. - -*/ - -#include "TFile.h" -#include "TMacro.h" -#include "TProofLog.h" -#include "TProofMgr.h" -#include "TObjString.h" -#include "TUrl.h" - -ClassImp(TProofLog); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor. - -TProofLog::TProofLog(const char *stag, const char *url, TProofMgr *mgr) - : TNamed(stag, url) -{ - SetLogToBox(); - fFILE = 0; - fElem = new TList; - fElem->SetOwner(); - fMgr = mgr; - // Set a fake starting time - fStartTime.Set((UInt_t)0); - // Extract real starting time - TString st(stag); - Int_t idx = st.Index('-'); - if (idx != kNPOS) { - st.Remove(0, idx+1); - idx = st.Index('-'); - if (idx != kNPOS) { - st.Remove(idx); - if (st.IsDigit()) { - fStartTime.Set(st.Atoi()); - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TProofLog::~TProofLog() -{ - SafeDelete(fElem); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add new entry to the list of elements. - -TProofLogElem *TProofLog::Add(const char *ord, const char *url) -{ - TProofLogElem *ple = new TProofLogElem(ord, url, this); - fElem->Add(ple); - // Done - return ple; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Retrieve the content of the log file associated with worker 'ord'. -/// If 'ord' is "*" (default), all the workers are retrieved. If 'all' -/// is true, the whole files are retrieved; else a max of -/// fgMaxTransferSize (about 1000 lines) per file is read, starting from -/// the end (i.e. the last ~1000 lines). -/// The received buffer is added to the file fname, if the latter is defined. -/// If opt == TProofLog::kGrep only the lines containing 'pattern' are -/// retrieved (remote grep functionality); to filter out a pattern 'pat' use -/// pattern = "-v pat". -/// Return 0 on success, -1 in case of any error. - -Int_t TProofLog::Retrieve(const char *ord, TProofLog::ERetrieveOpt opt, - const char *fname, const char *pattern) -{ - // Validate inputs - if (opt == TProofLog::kGrep && (!pattern || strlen(pattern) <= 0)) { - Error("Retrieve", "option 'Grep' requires a pattern"); - return -1; - } - - Int_t nel = (ord[0] == '*') ? fElem->GetSize() : 1; - // Iterate over the elements - TIter nxe(fElem); - TProofLogElem *ple = 0; - Int_t nd = 0, nb = 0; - TString msg; - while ((ple = (TProofLogElem *) nxe())) { - if (ord[0] == '*' || !strcmp(ord, ple->GetName())) { - if (ple->Retrieve(opt, pattern) != 0) { - nb++; - } else { - nd++; - } - Float_t frac = ((Float_t)nd + (Float_t)nb) * 100. / (Float_t)nel; - msg.Form("Retrieving logs: %d ok, %d not ok (%.0f%% processed)\r", nd, nb, frac); - Prt(msg.Data(), kFALSE); - } - } - Prt("\n"); - - // Save to file, if required - if (fname) - Save(ord, fname); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display the content associated with worker 'ord' from line 'from' -/// to line 'to' inclusive. A negative value -/// for 'from' indicates lines counted from the end (tail action); 'to' -/// is ignored in such a case. -/// If 'ord' is "*" (default), all the workers are displayed. - -void TProofLog::Display(const char *ord, Int_t from, Int_t to) -{ - TString msg; - if (ord[0] == '*') { - Int_t nel = (fElem) ? fElem->GetSize() : 0; - // Write global header - msg.Form("\n// --------- Displaying PROOF Session logs --------\n" - "// Server: %s \n// Session: %s \n// # of elements: %d \n" - "// ------------------------------------------------\n\n", - GetTitle(), GetName(), nel); - Prt(msg.Data()); - } - // Iterate over the elements - TIter nxe(fElem); - TProofLogElem *ple = 0; - while ((ple = (TProofLogElem *) nxe())) { - if (ord[0] == '*' || !strcmp(ord, ple->GetName())) - ple->Display(from, to); - } - if (ord[0] == '*') - // Write global tail - Prt("// --------- End of PROOF Session logs ---------\n"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print head info about the content - -void TProofLog::Print(Option_t *opt) const -{ - Int_t nel = (fElem) ? fElem->GetSize() : 0; - // Write global header - fprintf(stderr, "// --------- PROOF Session logs object --------\n"); - fprintf(stderr, "// Server: %s \n", GetTitle()); - fprintf(stderr, "// Session: %s \n", GetName()); - fprintf(stderr, "// # of elements: %d \n", nel); - fprintf(stderr, "// --------------------------------------------\n"); - - // Iterate over the elements - TIter nxe(fElem); - TProofLogElem *ple = 0; - while ((ple = (TProofLogElem *) nxe())) - ple->Print(opt); - - // Write global tail - fprintf(stderr, "// --------------------------------------------\n"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Special printing procedure - -void TProofLog::Prt(const char *what, Bool_t newline) -{ - if (what) { - if (LogToBox()) { - // Send to log box: - EmitVA("Prt(const char*)", 2, what, kFALSE); - } else { - FILE *where = (fFILE) ? (FILE *)fFILE : stderr; - fputs(what, where); - if (newline) fputc('\n', where); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save the content associated with worker 'ord' to finel 'fname'. -/// If 'ord' is "*" (default), the log from all the workers is saved. -/// If 'opt' is "a" the file is open in append mode; otherwise the file -/// is truncated. - -Int_t TProofLog::Save(const char *ord, const char *fname, Option_t *opt) -{ - // Make sure we got a file name - if (!fname) { - Warning("Save", "filename undefined - do nothing"); - return -1; - } - - // Open file to write header - // Check, if the option is to append - TString option = opt; - option.ToLower(); - FILE *fout=0; - if (option.Contains("a")){ - fout = fopen(fname, "a"); - } else { - fout = fopen(fname, "w"); - } - if (!fout) { - Warning("Save", "file could not be opened - do nothing"); - return -1; - } - fFILE = (void *) fout; - - TString msg; - if (ord[0] == '*') { - Int_t nel = (fElem) ? fElem->GetSize() : 0; - // Write global header - msg.Form("\n// --------- Displaying PROOF Session logs --------\n" - "// Server: %s \n// Session: %s \n// # of elements: %d \n" - "// ------------------------------------------------\n\n", - GetTitle(), GetName(), nel); - Prt(msg.Data()); - } - - // Iterate over the elements - TIter nxe(fElem); - TProofLogElem *ple = 0; - while ((ple = (TProofLogElem *) nxe())) { - if (ord[0] == '*' || !strcmp(ord, ple->GetName())) - ple->Display(0); - } - - if (ord[0] == '*') { - // Write global tail - Prt("// --------- End of PROOF Session logs ---------\n"); - } - - // Close file - fclose(fout); - fFILE = 0; - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Search lines containing 'txt', starting from line 'from'. -/// Print the lines where this happens. - -Int_t TProofLog::Grep(const char *txt, Int_t from) -{ - if (!txt || strlen(txt) <= 0) { - Warning("Grep", "text to be searched for is undefined - do nothing"); - return -1; - } - - Int_t nel = (fElem) ? fElem->GetSize() : 0; - // Write global header - TString msg; - msg.Form("\n// --------- Search in PROOF Session logs --------\n" - "// Server: %s \n// Session: %s \n// # of elements: %d \n" - "// Text searched for: \"%s\"", GetTitle(), GetName(), nel, txt); - Prt(msg.Data()); - if (from > 1) { - msg.Form("// starting from line %d \n", from); - } else { - msg = "\n"; - } - Prt(msg.Data()); - Prt("// ------------------------------------------------\n"); - - // Iterate over the elements - TIter nxe(fElem); - TProofLogElem *ple = 0; - while ((ple = (TProofLogElem *) nxe())) { - TString res; - Int_t nf = ple->Grep(txt, res, from); - if (nf > 0) { - msg.Form("// Ord: %s - line(s): %s\n", ple->GetName(), res.Data()); - Prt(msg.Data()); - } - } - - Prt("// ------------------------------------------------\n"); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set max transfer size. - -void TProofLog::SetMaxTransferSize(Long64_t maxsz) -{ - TProofLogElem::SetMaxTransferSize(maxsz); -} - -// -// TProofLogElem -// - -Long64_t TProofLogElem::fgMaxTransferSize = 100000; // about 1000 lines - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor. - -TProofLogElem::TProofLogElem(const char *ord, const char *url, - TProofLog *logger) - : TNamed(ord, url) -{ - fLogger = logger; - fMacro = new TMacro; - fSize = -1; - fFrom = -1; - fTo = -1; - - //Note the role here, don't redo at each call of Display() - if (strstr(GetTitle(), "worker-")) { - fRole = "worker"; - } else { - if (strchr(GetName(), '.')) { - fRole = "submaster"; - } else { - fRole = "master"; - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TProofLogElem::~TProofLogElem() -{ - SafeDelete(fMacro); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get max transfer size. - -Long64_t TProofLogElem::GetMaxTransferSize() -{ - return fgMaxTransferSize; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set max transfer size. - -void TProofLogElem::SetMaxTransferSize(Long64_t maxsz) -{ - fgMaxTransferSize = maxsz; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Retrieve the content of the associated file. The approximate number -/// of lines to be retrieved is given by 'lines', with the convention that -/// 0 means 'all', a positive number means the first 'lines' and a negative -/// number means the last '-lines'. Default is -1000. -/// If opt == TProofLog::kGrep only the lines containing 'pattern' are -/// retrieved (remote grep functionality); to filter out a pattern 'pat' use -/// pattern = "-v pat". -/// Return 0 on success, -1 in case of any error. - -Int_t TProofLogElem::Retrieve(TProofLog::ERetrieveOpt opt, const char *pattern) -{ - // Make sure we have a reference manager - if (!fLogger->fMgr || !fLogger->fMgr->IsValid()) { - Warning("Retrieve", "No reference manager: corruption?"); - return -1; - } - - // Print some info on the file - if (gDebug >= 2) { - Info("Retrieve", "Retrieving from ordinal %s file %s with pattern %s", - GetName(), GetTitle(), (pattern ? pattern : "(no pattern)")); - } - - // Determine offsets - if (opt == TProofLog::kAll) { - // Re-read everything - fFrom = 0; - fTo = -1; - if (gDebug >= 1) - Info("Retrieve", "Retrieving the whole file"); - } else if (opt == TProofLog::kLeading) { - // Read leading part - fFrom = 0; - fTo = fgMaxTransferSize; - if (gDebug >= 1) - Info("Retrieve", "Retrieving the leading %lld lines of file", fTo); - } else if (opt == TProofLog::kGrep) { - // Retrieve lines containing 'pattern', which must be defined - if (!pattern || strlen(pattern) <= 0) { - Error("Retrieve", "option 'Grep' requires a pattern"); - return -1; - } - if (gDebug >= 1) - Info("Retrieve", "Retrieving only lines filtered with %s", pattern); - } else { - // Read trailing part - fFrom = -fgMaxTransferSize; - fTo = -1; - if (gDebug >= 1) - Info("Retrieve", "Retrieving the last %lld lines of file", -fFrom); - } - - // Reset the macro - SafeDelete(fMacro); - fMacro = new TMacro; - - // Size to be read - Long64_t len = (fTo > fFrom) ? fTo - fFrom : -1; - - // Readout the buffer - TObjString *os = 0; - if (fLogger->fMgr) { - TString fileName = GetTitle(); - if (fileName.Contains("__igprof.pp__")) { - // File is an IgProf log. Override all patterns and preprocess it - if (gDebug >= 1) - Info("Retrieve", "Retrieving analyzed IgProf performance profile"); - TString analyzeAndFilter = \ - "|( T=`mktemp` && cat > \"$T\" ; igprof-analyse -d -g \"$T\" ; rm -f \"$T\" )"; - if (pattern && (*pattern == '|')) - analyzeAndFilter.Append(pattern); - os = fLogger->fMgr->ReadBuffer(fileName.Data(), analyzeAndFilter.Data()); - } - else if (opt == TProofLog::kGrep) - os = fLogger->fMgr->ReadBuffer(fileName.Data(), pattern); - else - os = fLogger->fMgr->ReadBuffer(fileName.Data(), fFrom, len); - } - if (os) { - // Loop over lines - TString ln; - Ssiz_t from = 0; - while (os->String().Tokenize(ln, from, "\n")) - fMacro->AddLine(ln.Data()); - - // Cleanup - delete os; - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display the current content starting from line 'from' to line 'to' -/// inclusive. -/// A negative value for 'from' indicates lines counted from the end -/// (tail action); 'to' is ignored in such a case. -/// TProofLog::Prt is called to display: the location (screen, file, box) -/// is defined there. -/// Return 0 on success, -1 in case of any error. - -void TProofLogElem::Display(Int_t from, Int_t to) -{ - Int_t nls = (fMacro->GetListOfLines()) ? - fMacro->GetListOfLines()->GetSize() : 0; - - // Starting line - Int_t i = 0; - Int_t ie = (to > -1 && to < nls) ? to : nls; - if (from > 1) { - if (from <= nls) - i = from - 1; - } else if (from < 0) { - // Tail action - if (-from <= nls) - i = nls + from; - ie = nls; - } - // Write header - TString msg; - Prt("// --------- Start of element log -----------------\n"); - msg.Form("// Ordinal: %s (role: %s)\n", GetName(), fRole.Data()); - Prt(msg.Data()); - // Separate out the submaster path, if any - TString path(GetTitle()); - Int_t ic = path.Index(","); - if (ic != kNPOS) { - TString subm(path); - path.Remove(0, ic+1); - subm.Remove(ic); - msg.Form("// Submaster: %s \n", subm.Data()); - Prt(msg.Data()); - } - msg.Form("// Path: %s \n// # of retrieved lines: %d ", path.Data(), nls); - Prt(msg.Data()); - if (i > 0 || ie < nls) { - msg.Form("(displaying lines: %d -> %d)\n", i+1, ie); - } else { - msg = "\n"; - } - Prt(msg.Data()); - Prt("// ------------------------------------------------\n"); - // Write lines - msg = ""; - if (fMacro->GetListOfLines()) { - TIter nxl(fMacro->GetListOfLines()); - TObjString *os = 0; - Int_t kk = 0; - while ((os = (TObjString *) nxl())) { - kk++; - if (kk > i) { - if (msg.Length() < 100000) { - if (msg.Length() > 0) msg += "\n"; - msg += os->GetName(); - } else { - Prt(msg.Data()); - msg = ""; - } - } - if (kk > ie) break; - } - } - if (msg.Length() > 0) Prt(msg.Data()); - // Write tail - Prt("// --------- End of element log -------------------\n\n"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print a line with the relevant info. - -void TProofLogElem::Print(Option_t *) const -{ - Int_t nls = (fMacro->GetListOfLines()) ? - fMacro->GetListOfLines()->GetSize() : 0; - const char *role = (strstr(GetTitle(), "worker-")) ? "worker" : "master"; - - fprintf(stderr, "Ord: %s Host: Role: %s lines: %d\n", GetName(), role, nls); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Special printing procedure. - -void TProofLogElem::Prt(const char *what) -{ - if (fLogger) - fLogger->Prt(what); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Search lines containing 'txt', starting from line 'from'. Return -/// their blanck-separated list into 'res'. -/// Return the number of lines found, or -1 in case of error. - -Int_t TProofLogElem::Grep(const char *txt, TString &res, Int_t from) -{ - Int_t nls = (fMacro->GetListOfLines()) ? - fMacro->GetListOfLines()->GetSize() : 0; - - Int_t nf = 0; - Int_t i = (from > 0) ? (from - 1) : 0; - for( ; i < nls; i++) { - TObjString *os = (TObjString *) fMacro->GetListOfLines()->At(i); - if (os) { - if (strstr(os->GetName(), txt)) { - if (res.Length() > 0) - res += " "; - res += (i + 1); - nf++; - } - } - } - - // Done - return nf; -} diff --git a/proof/proof/src/TProofMgr.cxx b/proof/proof/src/TProofMgr.cxx deleted file mode 100644 index 0f0c898c9f304..0000000000000 --- a/proof/proof/src/TProofMgr.cxx +++ /dev/null @@ -1,1114 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis, Nov 2005 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofMgr -\ingroup proofkernel - -The PROOF manager interacts with the PROOF server coordinator to -create or destroy a PROOF session, attach to or detach from -existing one, and to monitor any client activity on the cluster. -At most one manager instance per server is allowed. - -*/ - -#include "Bytes.h" -#include "TError.h" -#include "TEnv.h" -#include "TFile.h" -#include "TFileCollection.h" -#include "TFileInfo.h" -#include "TList.h" -#include "TParameter.h" -#include "TProof.h" -#include "TProofMgr.h" -#include "TProofMgrLite.h" -#include "TSocket.h" -#include "TROOT.h" -#include "TMath.h" -#include "TObjString.h" - -ClassImp(TProofMgr); - -// Sub-list of TROOT::fProofs with managers -TList TProofMgr::fgListOfManagers; -TProofMgr_t TProofMgr::fgTXProofMgrHook = 0; - -// Auxilliary structures for pinging -// The client request -typedef struct { - int first; - int second; - int third; - int fourth; - int fifth; -} clnt_HS_t; -// The body received after the first handshake's header -typedef struct { - int msglen; - int protover; - int msgval; -} srv_HS_t; - -//////////////////////////////////////////////////////////////////////////////// -/// Create a PROOF manager for the standard (old) environment. - -TProofMgr::TProofMgr(const char *url, Int_t, const char *alias) - : TNamed("",""), fRemoteProtocol(-1), fServType(kXProofd), - fSessions(0), fIntHandler(0) -{ - fServType = kProofd; - - // AVoid problems with empty URLs - fUrl = (!url || strlen(url) <= 0) ? TUrl("proof://localhost") : TUrl(url); - - // Correct URL protocol - if (!strcmp(fUrl.GetProtocol(), TUrl("a").GetProtocol())) - fUrl.SetProtocol("proof"); - - // Check port - if (fUrl.GetPort() == TUrl("a").GetPort()) { - // For the time being we use 'rootd' service as default. - // This will be changed to 'proofd' as soon as XRD will be able to - // accept on multiple ports - Int_t port = gSystem->GetServiceByName("proofd"); - if (port < 0) { - if (gDebug > 0) - Info("TProofMgr","service 'proofd' not found by GetServiceByName" - ": using default IANA assigned tcp port 1093"); - port = 1093; - } else { - if (gDebug > 1) - Info("TProofMgr","port from GetServiceByName: %d", port); - } - fUrl.SetPort(port); - } - - // Check and save the host FQDN ... - if (strcmp(fUrl.GetHost(), "__lite__")) { - if (strcmp(fUrl.GetHost(), fUrl.GetHostFQDN())) - fUrl.SetHost(fUrl.GetHostFQDN()); - } - - SetName(fUrl.GetUrl(kTRUE)); - if (alias) - SetAlias(alias); - else - SetAlias(fUrl.GetHost()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destroy a TProofMgr instance - -TProofMgr::~TProofMgr() -{ - SafeDelete(fSessions); - SafeDelete(fIntHandler); - - fgListOfManagers.Remove(this); - gROOT->GetListOfProofs()->Remove(this); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Dummy version provided for completeness. Just returns a pointer to -/// existing session 'id' (as shown by TProof::QuerySessions) or 0 if 'id' is -/// not valid. The boolena 'gui' should be kTRUE when invoked from the GUI. - -TProof *TProofMgr::AttachSession(Int_t id, Bool_t gui) -{ - TProofDesc *d = GetProofDesc(id); - if (d) - return AttachSession(d, gui); - - Info("AttachSession","invalid proofserv id (%d)", id); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Dummy version provided for completeness. Just returns a pointer to -/// existing session 'id' (as shown by TProof::QuerySessions) or 0 if 'id' is -/// not valid. - -TProof *TProofMgr::AttachSession(TProofDesc *d, Bool_t) -{ - if (!d) { - Warning("AttachSession","invalid description object - do nothing"); - return 0; - } - - if (d->GetProof()) - // Nothing to do if already in contact with proofserv - return d->GetProof(); - - Warning("AttachSession","session not available - do nothing"); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Detach session with 'id' from its proofserv. The 'id' is the number -/// shown by QuerySessions. The correspondent TProof object is deleted. -/// If id == 0 all the known sessions are detached. -/// Option opt="S" or "s" forces session shutdown. - -void TProofMgr::DetachSession(Int_t id, Option_t *opt) -{ - if (!IsValid()) { - Warning("DetachSession","invalid TProofMgr - do nothing"); - return; - } - - if (id > 0) { - - TProofDesc *d = GetProofDesc(id); - if (d) { - if (d->GetProof()) - d->GetProof()->Detach(opt); - TProof *p = d->GetProof(); - fSessions->Remove(d); - SafeDelete(p); - delete d; - } - - } else if (id == 0) { - - // Requesto to destroy all sessions - if (fSessions) { - // Delete PROOF sessions - TIter nxd(fSessions); - TProofDesc *d = 0; - while ((d = (TProofDesc *)nxd())) { - if (d->GetProof()) - d->GetProof()->Detach(opt); - TProof *p = d->GetProof(); - fSessions->Remove(d); - SafeDelete(p); - } - fSessions->Delete(); - } - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Detach session 'p' from its proofserv. The instance 'p' is invalidated -/// and should be deleted by the caller - -void TProofMgr::DetachSession(TProof *p, Option_t *opt) -{ - if (!IsValid()) { - Warning("DetachSession","invalid TProofMgr - do nothing"); - return; - } - - if (p) { - // Single session request - TProofDesc *d = GetProofDesc(p); - if (d) { - if (d->GetProof()) - // The session is closed here - d->GetProof()->Detach(opt); - fSessions->Remove(d); - delete d; - } - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get list of sessions accessible to this manager. - -TList *TProofMgr::QuerySessions(Option_t *opt) -{ - if (opt && !strncasecmp(opt,"L",1)) - // Just return the existing list - return fSessions; - - // Create list if not existing - if (!fSessions) { - fSessions = new TList(); - fSessions->SetOwner(); - } - - // Fill-in entries from the official list - if (gROOT->GetListOfProofs()) { - // Loop over - TIter nxp(gROOT->GetListOfProofs()); - TObject *o = 0; - TProof *p = 0; - Int_t ns = 0; - while ((o = nxp())) { - if (o->InheritsFrom(TProof::Class())) { - p = (TProof *)o; - // Only those belonging to this server - if (MatchUrl(p->GetUrl())) { - if (!(fSessions->FindObject(p->GetSessionTag()))) { - Int_t st = (p->IsIdle()) ? TProofDesc::kIdle - : TProofDesc::kRunning; - TProofDesc *d = - new TProofDesc(p->GetName(), p->GetTitle(), p->GetUrl(), - ++ns, p->GetSessionID(), st, p); - fSessions->Add(d); - } - } - } - } - } - - // Drop entries not existing any longer - if (fSessions->GetSize() > 0) { - TIter nxd(fSessions); - TProofDesc *d = 0; - while ((d = (TProofDesc *)nxd())) { - if (d->GetProof()) { - if (!(gROOT->GetListOfProofs()->FindObject(d->GetProof()))) { - fSessions->Remove(d); - SafeDelete(d); - } else { - if (opt && !strncasecmp(opt,"S",1)) - d->Print(""); - } - } - } - } - - // We are done - return fSessions; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send a message to connected users. Only superusers can do this. -/// The first argument specifies the message or the file from where to take -/// the message. -/// The second argument specifies the user to which to send the message: if -/// empty or null the message is send to all the connected users. -/// return 0 in case of success, -1 in case of error - -Int_t TProofMgr::SendMsgToUsers(const char *, const char *) -{ - Warning("SendMsgToUsers","functionality not supported"); - - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send a cleanup request for the sessions associated with the current -/// user. -/// Not supported. - -Int_t TProofMgr::Reset(Bool_t, const char *) -{ - Warning("Reset","functionality not supported"); - - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show available workers - -void TProofMgr::ShowWorkers() -{ - AbstractMethod("ShowWorkers"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get TProofDesc instance corresponding to 'id'. - -TProofDesc *TProofMgr::GetProofDesc(Int_t id) -{ - TProofDesc *d = 0; - if (id > 0) { - // Retrieve an updated list - QuerySessions(""); - if (fSessions) { - TIter nxd(fSessions); - while ((d = (TProofDesc *)nxd())) { - if (d->MatchId(id)) - return d; - } - } - } - - return d; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get TProofDesc instance corresponding to TProof object 'p'. - -TProofDesc *TProofMgr::GetProofDesc(TProof *p) -{ - TProofDesc *d = 0; - if (p) { - // Retrieve an updated list - QuerySessions(""); - if (fSessions) { - TIter nxd(fSessions); - while ((d = (TProofDesc *)nxd())) { - if (p == d->GetProof()) - return d; - } - } - } - - return d; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Discard TProofDesc of session 'p' from the internal list - -void TProofMgr::DiscardSession(TProof *p) -{ - if (p) { - TProofDesc *d = 0; - if (fSessions) { - TIter nxd(fSessions); - while ((d = (TProofDesc *)nxd())) { - if (p == d->GetProof()) { - fSessions->Remove(d); - delete d; - break; - } - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a new remote session (master and associated workers). - -TProof *TProofMgr::CreateSession(const char *cfg, - const char *cfgdir, Int_t loglevel) -{ - // Create - if (IsProofd()) - fUrl.SetOptions("std"); - - // Create the instance - TProof *p = new TProof(fUrl.GetUrl(), cfg, cfgdir, loglevel, 0, this); - - if (p && p->IsValid()) { - - // Save record about this session - Int_t ns = 1; - if (fSessions) { - // To avoid ambiguities in case of removal of some elements - if (fSessions->Last()) - ns = ((TProofDesc *)(fSessions->Last()))->GetLocalId() + 1; - } else { - // Create the list - fSessions = new TList; - } - - // Create the description class - Int_t st = (p->IsIdle()) ? TProofDesc::kIdle : TProofDesc::kRunning ; - TProofDesc *d = - new TProofDesc(p->GetName(), p->GetTitle(), p->GetUrl(), - ns, p->GetSessionID(), st, p); - fSessions->Add(d); - - } else { - // Session creation failed - if (gDebug > 0) Error("CreateSession", "PROOF session creation failed"); - SafeDelete(p); - } - - // We are done - return p; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Checks if 'url' refers to the same `user@host:port` entity as the URL -/// in memory - -Bool_t TProofMgr::MatchUrl(const char *url) -{ - TUrl u(url); - - // Correct URL protocol - if (!strcmp(u.GetProtocol(), TUrl("a").GetProtocol())) - u.SetProtocol("proof"); - - // Correct port - if (u.GetPort() == TUrl("a").GetPort()) { - Int_t port = gSystem->GetServiceByName("proofd"); - if (port < 0) - port = 1093; - u.SetPort(port); - } - - // Now we can check - if (!strcmp(u.GetHostFQDN(), fUrl.GetHostFQDN())) - if (u.GetPort() == fUrl.GetPort()) - if (strlen(u.GetUser()) <= 0 || !strcmp(u.GetUser(),fUrl.GetUser())) - return kTRUE; - - // Match failed - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Extract pointers to PROOF managers from TROOT::fProofs. - -TList *TProofMgr::GetListOfManagers() -{ - // Update the list with new entries - if (gROOT->GetListOfProofs()) { - TIter nxp(gROOT->GetListOfProofs()); - TObject *o = 0; - while ((o = nxp())) { - if (o->InheritsFrom(TProofMgr::Class()) && !fgListOfManagers.FindObject(o)) - fgListOfManagers.Add(o); - } - } - - // Get rid of invalid entries and notify - if (fgListOfManagers.GetSize() > 0) { - TIter nxp(&fgListOfManagers); - TObject *o = 0; - Int_t nm = 0; - while ((o = nxp())) { - if (!(gROOT->GetListOfProofs()->FindObject(o))) { - fgListOfManagers.Remove(o); - } else { - TProofMgr *p = (TProofMgr *)o; - if (gDebug > 0) - Printf("// #%d: \"%s\" (%s)", ++nm, p->GetName(), p->GetTitle()); - } - } - } else { - if (gDebug > 0) - Printf("No managers found"); - } - - // We are done - return &fgListOfManagers; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static method returning the appropriate TProofMgr object using -/// the plugin manager. - -TProofMgr *TProofMgr::Create(const char *uin, Int_t loglevel, - const char *alias, Bool_t xpd) -{ - TProofMgr *m= 0; - - Bool_t isLite = kFALSE; - - // Resolve url; if empty the actions depend of the default - TUrl u(uin); - TString proto = u.GetProtocol(); - if (proto.IsNull()) { - u.SetUrl(gEnv->GetValue("Proof.LocalDefault", "lite://")); - proto = u.GetProtocol(); - } - TString host = u.GetHost(); - if (proto == "lite" || host == "__lite__" ) { -#ifndef WIN32 - isLite = kTRUE; - u.SetHost("__lite__"); - u.SetProtocol("proof"); - u.SetPort(1093); -#else - ::Info("TProofMgr::Create","'lite' not yet supported on Windows"); - return m; -#endif - } - - if (!isLite) { - // in case user gave as url: "machine.dom.ain", replace - // "http" by "proof" and "80" by "1093" - if (!strcmp(u.GetProtocol(), TUrl("a").GetProtocol())) - u.SetProtocol("proof"); - if (u.GetPort() == TUrl("a").GetPort()) - u.SetPort(1093); - } - - // Avoid multiple calls to GetUrl - const char *url = u.GetUrl(); - - // Make sure we do not have already a manager for this URL - TList *lm = TProofMgr::GetListOfManagers(); - if (lm) { - TIter nxm(lm); - while ((m = (TProofMgr *)nxm())) { - if (m->IsValid()) { - if (m->MatchUrl(url)) return m; - } else { - fgListOfManagers.Remove(m); - SafeDelete(m); - break; - } - } - } - - if (isLite) { - // Init the lite version - return new TProofMgrLite(url, loglevel, alias); - } - - m = 0; - Bool_t trystd = kTRUE; - - // If required, we assume first that the remote server is based on XrdProofd - if (xpd) { - TProofMgr_t cm = TProofMgr::GetXProofMgrHook(); - if (cm) { - m = (TProofMgr *) (*cm)(url, loglevel, alias); - // Update trystd flag - trystd = (m && !(m->IsValid()) && m->IsProofd()) ? kTRUE : kFALSE; - } - } - - // If the first attempt failed, we instantiate an old interface - if (trystd) { - SafeDelete(m); - m = new TProofMgr(url, loglevel, alias); - } - - // Record the new manager, if any - if (m) { - fgListOfManagers.Add(m); - if (m->IsValid() && !(m->IsProofd())) { - R__LOCKGUARD(gROOTMutex); - gROOT->GetListOfProofs()->Add(m); - gROOT->GetListOfSockets()->Add(m); - } - } - - // We are done - return m; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the constructor hook fro TXProofMgr. -/// We do this without the plugin manager because it blocks the -/// CINT mutex breaking the parallel startup. - -TProofMgr_t TProofMgr::GetXProofMgrHook() -{ - if (!fgTXProofMgrHook) { - // Load the appropriate library ... - TString prooflib = "libProofx"; - char *p = 0; - if ((p = gSystem->DynamicPathName(prooflib, kTRUE))) { - delete[] p; - if (gSystem->Load(prooflib) == -1) - ::Error("TProofMgr::GetXProofMgrCtor", - "can't load %s", prooflib.Data()); - } else - ::Error("TProofMgr::GetXProofMgrCtor", - "can't locate %s", prooflib.Data()); - } - - // Done - return fgTXProofMgrHook; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set hook to TXProofMgr ctor - -void TProofMgr::SetTXProofMgrHook(TProofMgr_t pmh) -{ - fgTXProofMgrHook = pmh; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Non-blocking check for a PROOF (or Xrootd, if checkxrd) service at 'url' -/// Return -/// 0 if a XProofd (or Xrootd, if checkxrd) daemon is listening at 'url' -/// -1 if nothing is listening on the port (connection cannot be open) -/// 1 if something is listening but not XProofd (or not Xrootd, if checkxrd) - -Int_t TProofMgr::Ping(const char *url, Bool_t checkxrd) -{ - if (!url || (url && strlen(url) <= 0)) { - ::Error("TProofMgr::Ping", "empty url - fail"); - return -1; - } - - TUrl u(url); - // Check the port and set the defaults - if (!strcmp(u.GetProtocol(), "http") && u.GetPort() == 80) { - if (!checkxrd) { - u.SetPort(1093); - } else { - u.SetPort(1094); - } - } - - // Open the connection, disabling warnings ... - Int_t oldLevel = gErrorIgnoreLevel; - gErrorIgnoreLevel = kSysError+1; - TSocket s(u.GetHost(), u.GetPort()); - if (!(s.IsValid())) { - if (gDebug > 0) - ::Info("TProofMgr::Ping", "could not open connection to %s:%d", u.GetHost(), u.GetPort()); - gErrorIgnoreLevel = oldLevel; - return -1; - } - // Send the first bytes - int writeCount = -1; - clnt_HS_t initHS; - memset(&initHS, 0, sizeof(initHS)); - int len = sizeof(initHS); - if (checkxrd) { - initHS.fourth = (int)host2net((int)4); - initHS.fifth = (int)host2net((int)2012); - if ((writeCount = s.SendRaw(&initHS, len)) != len) { - if (gDebug > 0) - ::Info("TProofMgr::Ping", "1st: wrong number of bytes sent: %d (expected: %d)", - writeCount, len); - gErrorIgnoreLevel = oldLevel; - return 1; - } - } else { - initHS.third = (int)host2net((int)1); - if ((writeCount = s.SendRaw(&initHS, len)) != len) { - if (gDebug > 0) - ::Info("TProofMgr::Ping", "1st: wrong number of bytes sent: %d (expected: %d)", - writeCount, len); - gErrorIgnoreLevel = oldLevel; - return 1; - } - // These 8 bytes are need by 'proofd' and discarded by XPD - int dum[2]; - dum[0] = (int)host2net((int)4); - dum[1] = (int)host2net((int)2012); - if ((writeCount = s.SendRaw(&dum[0], sizeof(dum))) != sizeof(dum)) { - if (gDebug > 0) - ::Info("TProofMgr::Ping", "2nd: wrong number of bytes sent: %d (expected: %d)", - writeCount, (int) sizeof(dum)); - gErrorIgnoreLevel = oldLevel; - return 1; - } - } - // Read first server response - int type; - len = sizeof(type); - int readCount = s.RecvRaw(&type, len); // 4(2+2) bytes - if (readCount != len) { - if (gDebug > 0) - ::Info("TProofMgr::Ping", "1st: wrong number of bytes read: %d (expected: %d)", - readCount, len); - gErrorIgnoreLevel = oldLevel; - return 1; - } - // to host byte order - type = net2host(type); - // Check if the server is the eXtended proofd - if (type == 0) { - srv_HS_t xbody; - len = sizeof(xbody); - readCount = s.RecvRaw(&xbody, len); // 12(4+4+4) bytes - if (readCount != len) { - if (gDebug > 0) - ::Info("TProofMgr::Ping", "2nd: wrong number of bytes read: %d (expected: %d)", - readCount, len); - gErrorIgnoreLevel = oldLevel; - return 1; - } - xbody.protover = net2host(xbody.protover); - xbody.msgval = net2host(xbody.msglen); - xbody.msglen = net2host(xbody.msgval); - - } else if (type == 8) { - // Standard proofd - if (gDebug > 0) ::Info("TProofMgr::Ping", "server is old %s", (checkxrd ? "ROOTD" : "PROOFD")); - gErrorIgnoreLevel = oldLevel; - return 1; - } else { - // We don't know the server type - if (gDebug > 0) ::Info("TProofMgr::Ping", "unknown server type: %d", type); - gErrorIgnoreLevel = oldLevel; - return 1; - } - - // Restore ignore level - gErrorIgnoreLevel = oldLevel; - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Parse file name extracting the directory subcomponents in dirs, stored -/// as TObjStrings. - -void TProofMgr::ReplaceSubdirs(const char *fn, TString &fdst, TList &dirph) -{ - if (!fn || (fn && strlen(fn) <= 0)) return; - if (dirph.GetSize() <= 0) return; - - // Parse fn - TList dirs; - TString dd(fn), d; - Ssiz_t from = 0; - while (dd.Tokenize(d, from, "/")) { - if (!d.IsNull()) dirs.Add(new TObjString(d)); - } - if (dirs.GetSize() <= 0) return; - dirs.SetOwner(kTRUE); - - TIter nxph(&dirph); - TParameter *pi = 0; - while ((pi = (TParameter *) nxph())) { - if (pi->GetVal() < dirs.GetSize()) { - TObjString *os = (TObjString *) dirs.At(pi->GetVal()); - if (os) fdst.ReplaceAll(pi->GetName(), os->GetName()); - } else { - ::Warning("TProofMgr::ReplaceSubdirs", - "requested directory level '%s' is not available in the file path", - pi->GetName()); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Upload files provided via the list 'src' (as TFileInfo or TObjString) -/// to 'mss'. The path under 'mss' is determined by 'dest'; the following -/// place-holders can be used in 'dest': -/// -/// Place-holder | Meaning | -/// ----------------------------|------------------------------------ -/// \, \, \, ... | referring to the n-th sub-component of the src path -/// \ | basename in the source path -/// \ | basename sans extension -/// \ | Extension -/// \ | serial number of file in the list -/// \ | as \ but zero padded -/// \ | the full file path -/// \, \ | the local user and group names. -/// \ | the users PROOF group -/// \ | immediate parent directory -/// \ | next-to immediate parent directory -/// -/// So, for example, if the source filename for the 99-th file is -/// protosrc://host//d0/d1/d2/d3/d4/d5/myfile -/// then with dest = '/pool/user/\/\/\/\/\' and -/// mss = 'protodst://hostdst//nm/ -/// the corresponding destination path is -/// protodst://hostdst//nm/pool/user/d3/d4/d5/99/myfile -/// -/// If 'dest' is empty, \ is used. -/// -/// Returns a TFileCollection with the destination files created; this -/// TFileCollection is, for example, ready to be registered as dataset. - -TFileCollection *TProofMgr::UploadFiles(TList *src, - const char *mss, const char *dest) -{ - TFileCollection *ds = 0; - - // The inputs must be make sense - if (!src || (src && src->GetSize() <= 0)) { - ::Warning("TProofMgr::UploadFiles", "list is empty!"); - return ds; - } - if (!mss || (mss && strlen(mss) <= 0)) { - ::Warning("TProofMgr::UploadFiles", "MSS is undefined!"); - return ds; - } - - TList dirph; - - // If the destination is defined we need to understand if we have place-holders - if (dest && strlen(dest) > 0) { - TString dst(dest), dt; - Ssiz_t from = 0; - TRegexp re(""); - while (dst.Tokenize(dt, from, "/")) { - if (dt.Contains(re)) { - TParameter *pi = new TParameter(dt, -1); - dt.ReplaceAll("", ""); - if (dt.IsDigit()) { - pi->SetVal(dt.Atoi()); - dirph.Add(pi); - } else { - SafeDelete(pi); - } - } - } - dirph.SetOwner(kTRUE); - } - // Generate template for zero-padded serial numbers - TString sForm = TString::Format("%%0%dd", - Int_t(TMath::Log10(src->GetEntries()+1))); - - // Now we will actually copy files and create the TList object - ds = new TFileCollection(); - TIter nxf(src); - TObject *o = 0; - TObjString *os = 0; - TFileInfo *fi = 0; - Int_t kn = 0; - while ((o = nxf())) { - TUrl *furl = 0; - if (!strcmp(o->ClassName(), "TFileInfo")) { - if (!(fi = dynamic_cast(o))) { - ::Warning("TProofMgr::UploadFiles", - "object of class name '%s' does not cast to %s - ignore", - o->ClassName(), o->ClassName()); - continue; - } - furl = fi->GetFirstUrl(); - } else if (!strcmp(o->ClassName(), "TObjString")) { - if (!(os = dynamic_cast(o))) { - ::Warning("TProofMgr::UploadFiles", - "object of class name '%s' does not cast to %s - ignore", - o->ClassName(), o->ClassName()); - continue; - } - furl = new TUrl(os->GetName()); - } else { - ::Warning("TProofMgr::UploadFiles", - "object of unsupported class '%s' found in list - ignore", o->ClassName()); - continue; - } - - // The file must be accessible - if (gSystem->AccessPathName(furl->GetUrl()) == kFALSE) { - - // Create the destination path - TString fdst(mss); - if (dest && strlen(dest) > 0) { - fdst += dest; - } else { - fdst += TString::Format("/%s", furl->GetFile()); - } - - // Replace filename and basename - if (fdst.Contains("")) fdst.ReplaceAll("", gSystem->BaseName(furl->GetFile())); - if (fdst.Contains("")) fdst.ReplaceAll("", furl->GetFile()); - if (fdst.Contains("")) { - // Basename sans 'extension' - TString bs(gSystem->BaseName(furl->GetFile())); - Int_t idx = bs.Last('.'); - if (idx != kNPOS) bs.Remove(idx); - fdst.ReplaceAll("", bs.Data()); - } - if (fdst.Contains("")) { - // 'Extension' - that is the last part after the last '.' - TString ex(furl->GetFile()); - Int_t idx = ex.Last('.'); - if (idx != kNPOS) ex.Remove(0, idx+1); - else ex = ""; - fdst.ReplaceAll("", ex); - } - if (fdst.Contains("")) { - fdst.ReplaceAll("", - gSystem->BaseName(gSystem - ->DirName(furl->GetFile()))); - - } - if (fdst.Contains("")) { - fdst.ReplaceAll("", - gSystem->BaseName(gSystem - ->DirName(gSystem - ->DirName(furl->GetFile())))); - - } - - - // Replace serial number - if (fdst.Contains("")) { - TString skn = TString::Format("%d", kn); - fdst.ReplaceAll("", skn); - } - if (fdst.Contains("")) { - TString skn = TString::Format(sForm.Data(), kn); - fdst.ReplaceAll("", skn); - } - kn++; - - // Replace user and group name - UserGroup_t *pw = gSystem->GetUserInfo(); - if (pw) { - if (fdst.Contains("")) fdst.ReplaceAll("", pw->fUser); - if (fdst.Contains("")) fdst.ReplaceAll("", pw->fGroup); - delete pw; - } - if (gProof && fdst.Contains("")) - fdst.ReplaceAll("", gProof->GetGroup()); - - // Now replace the subdirs, if required - if (dirph.GetSize() > 0) - TProofMgr::ReplaceSubdirs(gSystem->GetDirName(furl->GetFile()), fdst, dirph); - - // Check double slashes in the file field (Turl sets things correctly inside) - TUrl u(fdst); - fdst = u.GetUrl(); - - // Copy the file now - ::Info("TProofMgr::UploadFiles", "uploading '%s' to '%s'", furl->GetUrl(), fdst.Data()); - if (TFile::Cp(furl->GetUrl(), fdst.Data())) { - // Build TFileCollection - ds->Add(new TFileInfo(fdst.Data())); - } else { - ::Error("TProofMgr::UploadFiles", "file %s was not copied", furl->GetUrl()); - } - } - } - - // Return the TFileCollection - return ds; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Upload to 'mss' the files listed in the text file 'srcfiles' or contained -/// in the directory 'srcfiles'. -/// In the case 'srcfiles' is a text file, the files must be specified one per -/// line, with line beginning by '#' ignored (i.e. considered comments). -/// The path under 'mss' is defined by 'dest'; the following -/// place-holders can be used in 'dest': -/// -/// Place-holder | Meaning | -/// ----------------------------|------------------------------------ -/// \, \, \, ... | referring to the n-th sub-component of the src path -/// \ | basename in the source path -/// \ | serial number of file in the list -/// \ | the full file path -/// \, \ | the local user and group names. -/// -/// So, for example, if the source filename for the 99-th file is -/// protosrc://host//d0/d1/d2/d3/d4/d5/myfile -/// then with dest = '/pool/user/\/\/\/\/\' and -/// mss = 'protodst://hostdst//nm/ -/// the corresponding destination path is -/// protodst://hostdst//nm/pool/user/d3/d4/d5/99/myfile -/// -/// If 'dest' is empty, \ is used. -/// -/// Returns a TFileCollection with the destination files created; this -/// TFileCollection is, for example, ready to be registered as dataset. - -TFileCollection *TProofMgr::UploadFiles(const char *srcfiles, - const char *mss, const char *dest) -{ - TFileCollection *ds = 0; - - // The inputs must be make sense - if (!srcfiles || (srcfiles && strlen(srcfiles) <= 0)) { - ::Error("TProofMgr::UploadFiles", "input text file or directory undefined!"); - return ds; - } - if (!mss || (mss && strlen(mss) <= 0)) { - ::Error("TProofMgr::UploadFiles", "MSS is undefined!"); - return ds; - } - - TString inpath = srcfiles; - gSystem->ExpandPathName(inpath); - - FileStat_t fst; - if (gSystem->GetPathInfo(inpath.Data(), fst)) { - ::Error("TProofMgr::UploadFiles", - "could not get information about the input path '%s':" - " make sure that it exists and is readable", srcfiles); - return ds; - } - - // Create the list to feed UploadFile(TList *, ...) - TList files; - files.SetOwner(); - - TString line; - if (R_ISREG(fst.fMode)) { - // Text file - std::ifstream f; - f.open(inpath.Data(), std::ifstream::out); - if (f.is_open()) { - while (f.good()) { - line.ReadToDelim(f); - line.Strip(TString::kTrailing, '\n'); - // Skip comments - if (line.BeginsWith("#")) continue; - if (gSystem->AccessPathName(line, kReadPermission) == kFALSE) - files.Add(new TFileInfo(line)); - } - f.close(); - } else { - ::Error("TProofMgr::UploadFiles", "unable to open file '%s'", srcfiles); - } - } else if (R_ISDIR(fst.fMode)) { - // Directory - void *dirp = gSystem->OpenDirectory(inpath.Data()); - if (dirp) { - const char *ent = 0; - while ((ent = gSystem->GetDirEntry(dirp))) { - if (!strcmp(ent, ".") || !strcmp(ent, "..")) continue; - line.Form("%s/%s", inpath.Data(), ent); - if (gSystem->AccessPathName(line, kReadPermission) == kFALSE) - files.Add(new TFileInfo(line)); - } - gSystem->FreeDirectory(dirp); - } else { - ::Error("TProofMgr::UploadFiles", "unable to open directory '%s'", inpath.Data()); - } - } else { - ::Error("TProofMgr::UploadFiles", - "input path '%s' is neither a regular file nor a directory!", inpath.Data()); - return ds; - } - if (files.GetSize() <= 0) { - ::Warning("TProofMgr::UploadFiles", "no files found in file or directory '%s'", inpath.Data()); - } else { - ds = TProofMgr::UploadFiles(&files, mss, dest); - } - // Done - return ds; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Run 'rm' on 'what'. Locally it is just a call to TSystem::Unlink . - -Int_t TProofMgr::Rm(const char *what, const char *, const char *) -{ - Int_t rc = -1; - // Nothing to do if not in contact with proofserv - if (!IsValid()) { - Error("Rm", "invalid TProofMgr - do nothing"); - return rc; - } - // Nothing to do if not in contact with proofserv - if (!what || (what && strlen(what) <= 0)) { - Error("Rm", "path undefined!"); - return rc; - } - - TUrl u(what); - if (!strcmp(u.GetProtocol(), "file")) { - rc = gSystem->Unlink(u.GetFile()); - } else { - rc = gSystem->Unlink(what); - } - // Done - return (rc == 0) ? 0 : -1; -} - -// -// TProofDesc -// - -ClassImp(TProofDesc); - -//////////////////////////////////////////////////////////////////////////////// -/// Dump the content to the screen. - -void TProofDesc::Print(Option_t *) const -{ - const char *st[] = { "unknown", "idle", "processing", "shutting down"}; - - Printf("// # %d", fLocalId); - Printf("// alias: %s, url: \"%s\"", GetTitle(), GetUrl()); - Printf("// tag: %s", GetName()); - Printf("// status: %s, attached: %s (remote ID: %d)",st[fStatus+1], (fProof ? "YES" : "NO"), fRemoteId); -} diff --git a/proof/proof/src/TProofMgrLite.cxx b/proof/proof/src/TProofMgrLite.cxx deleted file mode 100644 index 18931dd35ad38..0000000000000 --- a/proof/proof/src/TProofMgrLite.cxx +++ /dev/null @@ -1,407 +0,0 @@ -// @(#)root/proofx:$Id$ -// Author: Gerardo Ganis Apr 2008 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofMgrLite -\ingroup proofkernel - -Basic TProofMgr functionality implementation in the case of Lite session. - -*/ - -#include -#ifdef WIN32 -#include -#endif - -#include "TProofMgrLite.h" - -#include "Riostream.h" -#include "TEnv.h" -#include "TError.h" -#include "TObjString.h" -#include "TProofLite.h" -#include "TProofLog.h" -#include "TROOT.h" -#include "TRegexp.h" -#include "TSortedList.h" - -ClassImp(TProofMgrLite); - -//////////////////////////////////////////////////////////////////////////////// -/// Create a PROOF manager for the Lite environment. - -TProofMgrLite::TProofMgrLite(const char *url, Int_t dbg, const char *alias) - : TProofMgr(url, dbg, alias) -{ - // Set the correct servert type - fServType = kProofLite; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a new session - -TProof *TProofMgrLite::CreateSession(const char *cfg, - const char *, Int_t loglevel) -{ - TString c(fUrl.GetOptions()); - if (!c.Contains("workers=") && cfg && strstr(cfg, "workers=")) c = cfg; - Int_t nwrk = TProofLite::GetNumberOfWorkers(c); - if (nwrk == 0) return (TProof *)0; - - // Check if we have already a running session - if (gProof && gProof->IsLite()) { - if (gProof->IsValid()) { - if (nwrk > 0 && gProof->GetParallel() != nwrk) { - delete gProof; - gProof = 0; - } else { - // We have already a running session - return gProof; - } - } else { - // Remove existing instance - delete gProof; - gProof = 0; - } - } - - // Create the instance - TString u("lite"); - if (strlen(fUrl.GetOptions()) > 0) u.Form("lite/?%s", fUrl.GetOptions()); - TProof *p = new TProofLite(u, cfg, 0, loglevel, 0, this); - - if (p && p->IsValid()) { - - // Save record about this session - Int_t ns = 1; - if (fSessions) { - // To avoid ambiguities in case of removal of some elements - if (fSessions->Last()) - ns = ((TProofDesc *)(fSessions->Last()))->GetLocalId() + 1; - } else { - // Create the list - fSessions = new TList; - } - - // Create the description class - Int_t st = (p->IsIdle()) ? TProofDesc::kIdle : TProofDesc::kRunning ; - TProofDesc *d = - new TProofDesc(p->GetName(), p->GetTitle(), p->GetUrl(), - ns, p->GetSessionID(), st, p); - fSessions->Add(d); - - } else { - // Session creation failed - Error("CreateSession", "creating PROOF session"); - SafeDelete(p); - } - - // We are done - return p; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get logs or log tails from last session associated with this manager -/// instance. -/// The arguments allow to specify a session different from the last one: -/// isess specifies a position relative to the last one, i.e. 1 -/// for the next to last session; the absolute value is taken -/// so -1 and 1 are equivalent. -/// stag specifies the unique tag of the wanted session -/// The special value stag = "NR" allows to just initialize the TProofLog -/// object w/o retrieving the files; this may be useful when the number -/// of workers is large and only a subset of logs is required. -/// If 'stag' is specified 'isess' is ignored (unless stag = "NR"). -/// If 'pattern' is specified only the lines containing it are retrieved -/// (remote grep functionality); to filter out a pattern 'pat' use -/// pattern = "-v pat". -/// Returns a TProofLog object (to be deleted by the caller) on success, -/// 0 if something wrong happened. - -TProofLog *TProofMgrLite::GetSessionLogs(Int_t isess, const char *stag, - const char *pattern, Bool_t) -{ - TProofLog *pl = 0; - - // The absolute value of isess counts - isess = (isess < 0) ? -isess : isess; - - // Special option in stag - bool retrieve = 1; - TString tag(stag); - if (tag == "NR") { - retrieve = 0; - tag = ""; - } - - // The working dir - TString sandbox(gSystem->WorkingDirectory()); - sandbox.ReplaceAll(gSystem->HomeDirectory(),""); - sandbox.ReplaceAll("/","-"); - sandbox.Replace(0,1,"/",1); - if (strlen(gEnv->GetValue("ProofLite.Sandbox", "")) > 0) { - sandbox.Insert(0, gEnv->GetValue("ProofLite.Sandbox", "")); - } else if (strlen(gEnv->GetValue("Proof.Sandbox", "")) > 0) { - sandbox.Insert(0, gEnv->GetValue("Proof.Sandbox", "")); - } else { - TString sb; - sb.Form("~/%s", kPROOF_WorkDir); - sandbox.Insert(0, sb.Data()); - } - gSystem->ExpandPathName(sandbox); - - TString sessiondir; - if (tag.Length() > 0) { - sessiondir.Form("%s/session-%s", sandbox.Data(), tag.Data()); - if (gSystem->AccessPathName(sessiondir, kReadPermission)) { - Error("GetSessionLogs", "information for session '%s' not available", tag.Data()); - return (TProofLog *)0; - } - } else { - // Get the list of available dirs - TSortedList *olddirs = new TSortedList(kFALSE); - void *dirp = gSystem->OpenDirectory(sandbox); - if (dirp) { - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - if (!strncmp(e, "session-", 8)) { - TString d(e); - Int_t i = d.Last('-'); - if (i != kNPOS) d.Remove(i); - i = d.Last('-'); - if (i != kNPOS) d.Remove(0,i+1); - TString path = Form("%s/%s", sandbox.Data(), e); - olddirs->Add(new TNamed(d, path)); - } - } - gSystem->FreeDirectory(dirp); - } - - // Check isess - if (isess > olddirs->GetSize() - 1) { - Warning("GetSessionLogs", - "session index out of range (%d): take oldest available session", isess); - isess = olddirs->GetSize() - 1; - } - - // Locate the session dir - Int_t isx = isess; - TNamed *n = (TNamed *) olddirs->First(); - while (isx-- > 0) { - olddirs->Remove(n); - delete n; - n = (TNamed *) olddirs->First(); - } - if (!n) { - Error("GetSessionLogs", "cannot locate session dir for index '%d' under '%s':" - " cannot continue!", isess, sandbox.Data()); - return (TProofLog *)0; - } - sessiondir = n->GetTitle(); - tag = gSystem->BaseName(sessiondir); - tag.ReplaceAll("session-", ""); - - // Cleanup - olddirs->SetOwner(); - delete olddirs; - } - Info("GetSessionLogs", "analysing session dir %s", sessiondir.Data()); - - // Create the instance now - pl = new TProofLog(tag, "", this); - - void *dirp = gSystem->OpenDirectory(sessiondir); - if (dirp) { - TSortedList *logs = new TSortedList; - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - TString fn(e); - if (fn.EndsWith(".log") && fn.CountChar('-') > 0) { - TString ord, url; - if (fn.BeginsWith("session-")) { - ord = "-1"; - } else if (fn.BeginsWith("worker-")) { - ord = fn; - ord.ReplaceAll("worker-", ""); - Int_t id = ord.First('-'); - if (id != kNPOS) { - ord.Remove(id); - } else if (ord.Contains(".valgrind")) { - // Add to the list (special tag for valgrind outputs) - ord.ReplaceAll(".valgrind.log","-valgrind"); - } else { - // Not a good path - ord = ""; - } - if (!ord.IsNull()) ord.ReplaceAll("0.", ""); - } - if (!ord.IsNull()) { - url = Form("%s/%s", sessiondir.Data(), e); - // Add to the list - logs->Add(new TNamed(ord, url)); - // Notify - if (gDebug > 1) - Info("GetSessionLogs", "ord: %s, url: %s", ord.Data(), url.Data()); - } - } - } - gSystem->FreeDirectory(dirp); - - TIter nxl(logs); - TNamed *n = 0; - while ((n = (TNamed *) nxl())) { - TString ord = Form("0.%s", n->GetName()); - if (ord == "0.-1") ord = "0"; - // Add to the list - pl->Add(ord, n->GetTitle()); - } - - // Cleanup - logs->SetOwner(); - delete logs; - } - - // Retrieve the default part - if (pl && retrieve) { - const char *pat = pattern ? pattern : "-v \"| SvcMsg\""; - if (pat && strlen(pat) > 0) - pl->Retrieve("*", TProofLog::kGrep, 0, pat); - else - pl->Retrieve(); - } - - // Done - return pl; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Read 'len' bytes from offset 'ofs' of the local file 'fin'. -/// Returns a TObjString with the content or 0, in case of failure - -TObjString *TProofMgrLite::ReadBuffer(const char *fin, Long64_t ofs, Int_t len) -{ - if (!fin || strlen(fin) <= 0) { - Error("ReadBuffer", "undefined path!"); - return (TObjString *)0; - } - - // Open the file - TString fn = TUrl(fin).GetFile(); - Int_t fd = open(fn.Data(), O_RDONLY); - if (fd < 0) { - Error("ReadBuffer", "problems opening file %s", fn.Data()); - return (TObjString *)0; - } - - // Total size - off_t start = 0, end = lseek(fd, (off_t) 0, SEEK_END); - - // Set the offset - if (ofs > 0 && ofs < end) { - start = lseek(fd, (off_t) ofs, SEEK_SET); - } else { - start = lseek(fd, (off_t) 0, SEEK_SET); - } - if (len > (end - start + 1) || len <= 0) - len = end - start + 1; - - TString outbuf; - const Int_t kMAXBUF = 32768; - char buf[kMAXBUF]; - Int_t left = len; - Int_t wanted = (left > kMAXBUF - 1) ? kMAXBUF - 1 : left; - do { - while ((len = read(fd, buf, wanted)) < 0 && TSystem::GetErrno() == EINTR) - TSystem::ResetErrno(); - - if (len < 0) { - Error("ReadBuffer", "error reading file %s", fn.Data()); - close(fd); - return (TObjString *)0; - } else if (len > 0) { - if (len == wanted) - buf[len-1] = '\n'; - buf[len] = '\0'; - outbuf += buf; - } - - // Update counters - left -= len; - wanted = (left > kMAXBUF - 1) ? kMAXBUF - 1 : left; - - } while (len > 0 && left > 0); - - // Close file - close(fd); - - // Done - return new TObjString(outbuf.Data()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Read lines containing 'pattern' in 'file'. -/// Returns a TObjString with the content or 0, in case of failure - -TObjString *TProofMgrLite::ReadBuffer(const char *fin, const char *pattern) -{ - // If no pattern, read everything - if (!pattern || strlen(pattern) <= 0) - return (TObjString *)0; - - if (!fin || strlen(fin) <= 0) { - Error("ReadBuffer", "undefined path!"); - return (TObjString *)0; - } - TString fn = TUrl(fin).GetFile(); - - TString pat(pattern); - // Check if "-v" - Bool_t excl = kFALSE; - if (pat.Contains("-v ")) { - pat.ReplaceAll("-v ", ""); - excl = kTRUE; - } - pat = pat.Strip(TString::kLeading, ' '); - pat = pat.Strip(TString::kTrailing, ' '); - pat = pat.Strip(TString::kLeading, '\"'); - pat = pat.Strip(TString::kTrailing, '\"'); - - // Use a regular expression - TRegexp re(pat); - - // Open file with file info - std::ifstream in; - in.open(fn.Data()); - - TString outbuf; - - // Read the input list of files and add them to the chain - TString line; - while(in.good()) { - - // Read next line - line.ReadLine(in); - - // Keep only lines with pattern - if ((excl && line.Index(re) != kNPOS) || - (!excl && line.Index(re) == kNPOS)) continue; - - // Remove trailing '\n', if any - if (!line.EndsWith("\n")) line.Append('\n'); - - // Add to output - outbuf += line; - } - in.close(); - - // Done - return new TObjString(outbuf.Data()); -} diff --git a/proof/proof/src/TProofNodeInfo.cxx b/proof/proof/src/TProofNodeInfo.cxx deleted file mode 100644 index e2b0cb79b89f5..0000000000000 --- a/proof/proof/src/TProofNodeInfo.cxx +++ /dev/null @@ -1,162 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Paul Nilsson 7/12/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofNodeInfo -\ingroup proofkernel - -The purpose of this class is to provide a complete node description -for masters, submasters and workers. - -*/ - -#include "TProofNodeInfo.h" - -ClassImp(TProofNodeInfo); - -//////////////////////////////////////////////////////////////////////////////// -/// Default constructor. - -TProofNodeInfo::TProofNodeInfo() - : fNodeType(kWorker), fPort(-1), fPerfIndex(100), fNWrks(1) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor from a string containing all the information in a serialized -/// way. Used to decode thr information coming from the coordinator -/// ``|``|``|``|``|``|``|``|``|`` - -TProofNodeInfo::TProofNodeInfo(const char *str) - : fNodeType(kWorker), fPort(-1), fPerfIndex(100), fNWrks(1) -{ - // Needs a non empty string to do something - if (!str || strlen(str) <= 0) - return; - - TString ss(str), s; - Ssiz_t from = 0; - // NodeType - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - fNodeType = GetNodeType(s); - // NodeName - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - fNodeName = s; - // Port - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - if (s.IsDigit()) fPort = s.Atoi(); - // Ordinal - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - fOrdinal = s; - // ID string - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - fId = s; - // Performance - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - if (s.IsDigit()) fPerfIndex = s.Atoi(); - // Image - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - fImage = s; - // Working dir - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - fWorkDir = s; - // Mass Storage Domain - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - fMsd = s; - // Config file (master or submaster; for backward compatibility) - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - fConfig = s; - // Number of workers - if (ss.Tokenize(s, from, "|") && !s.IsNull() && s != "-") - if (s.IsDigit()) fNWrks = s.Atoi(); - - // Set the name - fName.Form("%s:%d", fNodeName.Data(), fPort); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Copy constructor. - -TProofNodeInfo::TProofNodeInfo(const TProofNodeInfo &nodeInfo) : TObject(nodeInfo) -{ - fName = nodeInfo.fName; - fNodeType = nodeInfo.fNodeType; - fNodeName = nodeInfo.fNodeName; - fWorkDir = nodeInfo.fWorkDir; - fOrdinal = nodeInfo.fOrdinal; - fImage = nodeInfo.fImage; - fId = nodeInfo.fId; - fConfig = nodeInfo.fConfig; - fMsd = nodeInfo.fMsd; - fPort = nodeInfo.fPort; - fPerfIndex = nodeInfo.fPerfIndex; - fNWrks = nodeInfo.fNWrks; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Asssign content of node n to this node - -void TProofNodeInfo::Assign(const TProofNodeInfo &n) -{ - fName = n.fName; - fNodeType = n.fNodeType; - fNodeName = n.fNodeName; - fWorkDir = n.fWorkDir; - fOrdinal = n.fOrdinal; - fImage = n.fImage; - fId = n.fId; - fConfig = n.fConfig; - fMsd = n.fMsd; - fPort = n.fPort; - fPerfIndex = n.fPerfIndex; - fNWrks = n.fNWrks; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print the TProofNodeInfo structure. - -void TProofNodeInfo::Print(const Option_t *opt) const -{ - if (opt[0] == 'c' || opt[0] == 'C') { - Printf("%d %s:%d %s %s", fNodeType, fNodeName.Data(), fPort, - fOrdinal.Data(), fWorkDir.Data()); - } else { - Printf(" +++ TProofNodeInfo: %s +++", fName.Data()); - Printf(" NodeName: %s, Port: %d, NodeType: %d, Ordinal: %s", - fNodeName.Data(), fPort, fNodeType, fOrdinal.Data()); - Printf(" WorkDir: %s, Image: %s", fWorkDir.Data(), fImage.Data()); - Printf(" Id: %s, Config: %s", fId.Data(), fConfig.Data()); - Printf(" Msd: %s", fMsd.Data()); - Printf(" Performance: %d", fPerfIndex); - Printf(" NumberOfWrks: %d", fNWrks); - Printf("+++++++++++++++++++++++++++++++++++++++++++"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static method returning node type. Allowed input: "master", "submaster", -/// or anything else which will be interpreted as worker. - -TProofNodeInfo::ENodeType TProofNodeInfo::GetNodeType(const TString &type) -{ - ENodeType enType; - - if (type == "M" || type == "master") { - enType = kMaster; - } - else if (type == "S" || type == "submaster") { - enType = kSubMaster; - } - else { // [worker/slave or condorworker] - enType = kWorker; - } - - return enType; -} diff --git a/proof/proof/src/TProofOutputFile.cxx b/proof/proof/src/TProofOutputFile.cxx deleted file mode 100644 index 42fb984d3e88e..0000000000000 --- a/proof/proof/src/TProofOutputFile.cxx +++ /dev/null @@ -1,549 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Long Tran-Thanh 14/09/07 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofOutputFile -\ingroup proofkernel - -Class to steer the merging of files produced on the workers - -*/ - -#include "TProofOutputFile.h" -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -ClassImp(TProofOutputFile); - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor - -TProofOutputFile::TProofOutputFile(const char *path, - ERunType type, UInt_t opt, const char *dsname) - : TNamed(path, ""), fRunType(type), fTypeOpt(opt) -{ - fIsLocal = kFALSE; - fMerged = kFALSE; - fMerger = 0; - fDataSet = 0; - ResetBit(TProofOutputFile::kRetrieve); - ResetBit(TProofOutputFile::kSwapFile); - - Init(path, dsname); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor with the old signature, kept for convenience and backard compatibility. -/// Options: -/// 'M' merge: finally merge the created files -/// 'L' local: copy locally the files before merging (implies 'M') -/// 'D' dataset: create a TFileCollection -/// 'R' register: dataset run with dataset registration -/// 'O' overwrite: force dataset replacement during registration -/// 'V' verify: verify the registered dataset -/// 'H' merge histograms in one go (option to TFileMerger) -/// Special 'option' values for backward compatibility: -/// "" equivalent to "M" -/// "LOCAL" equivalent to "ML" or "L" - -TProofOutputFile::TProofOutputFile(const char *path, - const char *option, const char *dsname) - : TNamed(path, "") -{ - fIsLocal = kFALSE; - fMerged = kFALSE; - fMerger = 0; - fDataSet = 0; - fMergeHistosOneGo = kFALSE; - - // Fill the run type and option type - fRunType = kMerge; - fTypeOpt = kRemote; - if (option && strlen(option) > 0) { - TString opt(option); - if (opt.Contains("L") || (opt == "LOCAL")) fTypeOpt = kLocal; - if (opt.Contains("H")) fMergeHistosOneGo = kTRUE; - if (!opt.Contains("M") && opt.Contains("D")) { - // Dataset creation mode - fRunType = kDataset; - fTypeOpt = kCreate; - if (opt.Contains("R")) fTypeOpt = (ETypeOpt) (fTypeOpt | kRegister); - if (opt.Contains("O")) fTypeOpt = (ETypeOpt) (fTypeOpt | kOverwrite); - if (opt.Contains("V")) fTypeOpt = (ETypeOpt) (fTypeOpt | kVerify); - } - } - - Init(path, dsname); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Initializer. Called by all constructors - -void TProofOutputFile::Init(const char *path, const char *dsname) -{ - fLocalHost = TUrl(gSystem->HostName()).GetHostFQDN(); - Int_t port = gEnv->GetValue("ProofServ.XpdPort", -1); - if (port > -1) { - fLocalHost += ":"; - fLocalHost += port; - } - - TString xpath(path); - // Resolve the relevant placeholders in fFileName (e.g. root://a.ser.ver//data/dir///file) - TProofServ::ResolveKeywords(xpath, 0); - TUrl u(xpath, kTRUE); - // File name - fFileName = u.GetFile(); - // The name is used to identify this entity - SetName(gSystem->BaseName(fFileName.Data())); - // The title is the dataset name in the case such option is chosen. - // In the merging case it can be the final location of the file on the client if the retrieve - // option is chosen; if the case, this set in TProofPlayer::MergeOutputFiles. - if (fRunType == kDataset) { - if (dsname && strlen(dsname) > 0) { - // This is the dataset name in case such option is chosen - SetTitle(dsname); - } else { - // Default dataset name - SetTitle(GetName()); - } - } - // Options and anchor, if any - if (u.GetOptions() && strlen(u.GetOptions()) > 0) - fOptionsAnchor += TString::Format("?%s", u.GetOptions()); - if (u.GetAnchor() && strlen(u.GetAnchor()) > 0) - fOptionsAnchor += TString::Format("#%s", u.GetAnchor()); - // Path - fIsLocal = kFALSE; - fDir = u.GetUrl(); - Int_t pos = fDir.Index(fFileName); - if (pos != kNPOS) fDir.Remove(pos); - fRawDir = fDir; - - if (fDir.BeginsWith("file:")) { - fIsLocal = kTRUE; - // For local files, the user is allowed to create files under the specified directory. - // If this is not the case, the file is rooted automatically to the assigned dir which - // is the datadir for dataset creation runs, and the working dir for merging runs - TString dirPath = gSystem->GetDirName(fFileName); - fFileName = gSystem->BaseName(fFileName); - if (AssertDir(dirPath) != 0) - Error("Init", "problems asserting path '%s'", dirPath.Data()); - TString dirData = (!IsMerge() && gProofServ) ? gProofServ->GetDataDir() - : gSystem->WorkingDirectory(); - if ((dirPath[0] == '/') && gSystem->AccessPathName(dirPath, kWritePermission)) { - Warning("Init", "not allowed to create files under '%s' - chrooting to '%s'", - dirPath.Data(), dirData.Data()); - dirPath.Insert(0, dirData); - } else if (dirPath.BeginsWith("..")) { - dirPath.Remove(0, 2); - if (dirPath[0] != '/') dirPath.Insert(0, "/"); - dirPath.Insert(0, dirData); - } else if (dirPath[0] == '.' || dirPath[0] == '~') { - dirPath.Remove(0, 1); - if (dirPath[0] != '/') dirPath.Insert(0, "/"); - dirPath.Insert(0, dirData); - } else if (dirPath.IsNull()) { - dirPath = dirData; - } - // Make sure that session-tag, ordinal and query sequential number are present otherwise - // we may override outputs from other workers - if (gProofServ) { - if (!IsMerge() || (!dirPath.BeginsWith(gProofServ->GetDataDir()) && - !dirPath.BeginsWith(gSystem->WorkingDirectory()))) { - if (!dirPath.Contains(gProofServ->GetOrdinal())) { - if (!dirPath.EndsWith("/")) dirPath += "/"; - dirPath += gProofServ->GetOrdinal(); - } - } - if (!IsMerge()) { - if (!dirPath.Contains(gProofServ->GetSessionTag())) { - if (!dirPath.EndsWith("/")) dirPath += "/"; - dirPath += gProofServ->GetSessionTag(); - } - if (!dirPath.Contains("")) { - if (!dirPath.EndsWith("/")) dirPath += "/"; - dirPath += ""; - } - // Resolve the relevant placeholders - TProofServ::ResolveKeywords(dirPath, 0); - } - } - // Save the raw directory - fRawDir = dirPath; - // Make sure the path exists - if (AssertDir(dirPath) != 0) - Error("Init", "problems asserting path '%s'", dirPath.Data()); - // Take into account local server settings - TProofServ::GetLocalServer(fDir); - TProofServ::FilterLocalroot(dirPath, fDir); - // The path to be used to address the file - fDir += dirPath; - } - // Notify - Info("Init", "dir: %s (raw: %s)", fDir.Data(), fRawDir.Data()); - - // Default output file name - ResetBit(TProofOutputFile::kOutputFileNameSet); - fOutputFileName = ""; - if (gEnv->Lookup("Proof.OutputFile")) { - fOutputFileName = gEnv->GetValue("Proof.OutputFile", ""); - SetBit(TProofOutputFile::kOutputFileNameSet); - } - // Add default file name - TString fileName = path; - if (!fileName.EndsWith(".root")) fileName += ".root"; - // Make sure that the file name was inserted (may not happen if the placeholder is missing) - if (!fOutputFileName.IsNull() && !fOutputFileName.Contains("")) { - if (!fOutputFileName.EndsWith("/")) fOutputFileName += "/"; - fOutputFileName += fileName; - } - // Resolve placeholders - fileName.ReplaceAll("",""); // No ordinal in the final merged file - TProofServ::ResolveKeywords(fOutputFileName, fileName); - Info("Init", "output file url: %s", fOutputFileName.Data()); - // Fill ordinal - fWorkerOrdinal = ""; - TProofServ::ResolveKeywords(fWorkerOrdinal, 0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Main destructor - -TProofOutputFile::~TProofOutputFile() -{ - if (fDataSet) delete fDataSet; - if (fMerger) delete fMerger; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the name of the output file; in the form of an Url. - -void TProofOutputFile::SetOutputFileName(const char *name) -{ - if (name && strlen(name) > 0) { - fOutputFileName = name; - TProofServ::ResolveKeywords(fOutputFileName); - PDB(kOutput,1) Info("SetOutputFileName", "output file url: %s", fOutputFileName.Data()); - } else { - fOutputFileName = ""; - } - SetBit(TProofOutputFile::kOutputFileNameSet); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Open the file using the unique temporary name - -TFile* TProofOutputFile::OpenFile(const char* opt) -{ - if (fFileName.IsNull()) return 0; - - // Create the path - TString fileLoc; - fileLoc.Form("%s/%s%s", fRawDir.Data(), fFileName.Data(), fOptionsAnchor.Data()); - - // Open the file - TFile *retFile = TFile::Open(fileLoc, opt); - - return retFile; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Adopt a file already open. -/// Return 0 if OK, -1 in case of failure - -Int_t TProofOutputFile::AdoptFile(TFile *f) -{ - if (!f || (f && f->IsZombie())) { - Error("AdoptFile", "file is undefined or zombie!"); - return -1; - } - const TUrl *u = f->GetEndpointUrl(); - if (!u) { - Error("AdoptFile", "file end-point url is undefined!"); - return -1; - } - - // Set the name and dir - fIsLocal = kFALSE; - if (!strcmp(u->GetProtocol(), "file")) { - fIsLocal = kTRUE; - fDir = u->GetFile(); - } else { - fDir = u->GetUrl(); - } - fFileName = gSystem->BaseName(fDir.Data()); - fDir.ReplaceAll(fFileName, ""); - fRawDir = fDir; - - // If local remove prefix, if any - if (fIsLocal) { - TString localDS; - TProofServ::GetLocalServer(localDS); - if (!localDS.IsNull()) { - TProofServ::FilterLocalroot(fDir, localDS); - fDir.Insert(0, localDS); - } - } - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Merge objects from the list into this object - -Long64_t TProofOutputFile::Merge(TCollection* list) -{ - PDB(kOutput,2) Info("Merge","enter: merge? %d", IsMerge()); - - // Needs somethign to merge - if(!list || list->IsEmpty()) return 0; - - if (IsMerge()) { - // Build-up the merger - TString fileLoc; - TString outputFileLoc = (fOutputFileName.IsNull()) ? fFileName : fOutputFileName; - // Get the file merger instance - Bool_t localMerge = (fRunType == kMerge && fTypeOpt == kLocal) ? kTRUE : kFALSE; - TFileMerger *merger = GetFileMerger(localMerge); - if (!merger) { - Error("Merge", "could not instantiate the file merger"); - return -1; - } - - if (!fMerged) { - merger->OutputFile(outputFileLoc); - fileLoc.Form("%s/%s", fDir.Data(), GetFileName()); - AddFile(merger, fileLoc); - fMerged = kTRUE; - } - - TIter next(list); - TObject *o = 0; - while((o = next())) { - TProofOutputFile *pFile = dynamic_cast(o); - if (pFile) { - fileLoc.Form("%s/%s", pFile->GetDir(), pFile->GetFileName()); - AddFile(merger, fileLoc); - } - } - } else { - // Get the reference MSS url, if any - TUrl mssUrl(gEnv->GetValue("ProofServ.PoolUrl","")); - // Build-up the TFileCollection - TFileCollection *dataset = GetFileCollection(); - if (!dataset) { - Error("Merge", "could not instantiate the file collection"); - return -1; - } - fMerged = kTRUE; - TString path; - TFileInfo *fi = 0; - // If new, add ourseelves - dataset->Update(); - PDB(kOutput,2) Info("Merge","dataset: %s (nfiles: %lld)", dataset->GetName(), dataset->GetNFiles()); - if (dataset->GetNFiles() == 0) { - // Save the export and raw urls - path.Form("%s/%s%s", GetDir(), GetFileName(), GetOptionsAnchor()); - fi = new TFileInfo(path); - // Add also an URL with the redirector path, if any - if (mssUrl.IsValid()) { - TUrl ur(fi->GetFirstUrl()->GetUrl()); - ur.SetProtocol(mssUrl.GetProtocol()); - ur.SetHost(mssUrl.GetHost()); - ur.SetPort(mssUrl.GetPort()); - if (mssUrl.GetUser() && strlen(mssUrl.GetUser()) > 0) - ur.SetUser(mssUrl.GetUser()); - fi->AddUrl(ur.GetUrl()); - } - // Add special local URL to keep track of the file - path.Form("%s/%s?node=%s", GetDir(kTRUE), GetFileName(), GetLocalHost()); - fi->AddUrl(path); - PDB(kOutput,2) fi->Print(); - // Now add to the dataset - dataset->Add(fi); - } - - TIter next(list); - TObject *o = 0; - while((o = next())) { - TProofOutputFile *pFile = dynamic_cast(o); - if (pFile) { - // Save the export and raw urls - path.Form("%s/%s%s", pFile->GetDir(), pFile->GetFileName(), pFile->GetOptionsAnchor()); - fi = new TFileInfo(path); - // Add also an URL with the redirector path, if any - if (mssUrl.IsValid()) { - TUrl ur(fi->GetFirstUrl()->GetUrl()); - ur.SetProtocol(mssUrl.GetProtocol()); - ur.SetHost(mssUrl.GetHost()); - ur.SetPort(mssUrl.GetPort()); - if (mssUrl.GetUser() && strlen(mssUrl.GetUser()) > 0) - ur.SetUser(mssUrl.GetUser()); - fi->AddUrl(ur.GetUrl()); - } - // Add special local URL to keep track of the file - path.Form("%s/%s?node=%s", pFile->GetDir(kTRUE), pFile->GetFileName(), pFile->GetLocalHost()); - fi->AddUrl(path); - PDB(kOutput,2) fi->Print(); - // Now add to the dataset - dataset->Add(fi); - } - } - } - PDB(kOutput,2) Info("Merge","Done"); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Dump the class content - -void TProofOutputFile::Print(Option_t *) const -{ - Info("Print","-------------- %s : start (%s) ------------", GetName(), fLocalHost.Data()); - Info("Print"," dir: %s", fDir.Data()); - Info("Print"," raw dir: %s", fRawDir.Data()); - Info("Print"," file name: %s%s", fFileName.Data(), fOptionsAnchor.Data()); - if (IsMerge()) { - Info("Print"," run type: create a merged file"); - Info("Print"," merging option: %s", - (fTypeOpt == kLocal) ? "local copy" : "keep remote"); - } else { - TString opt; - if ((fTypeOpt & kRegister)) opt += "R"; - if ((fTypeOpt & kOverwrite)) opt += "O"; - if ((fTypeOpt & kVerify)) opt += "V"; - Info("Print"," run type: create dataset (name: '%s', opt: '%s')", - GetTitle(), opt.Data()); - } - Info("Print"," output file name: %s", fOutputFileName.Data()); - Info("Print"," ordinal: %s", fWorkerOrdinal.Data()); - Info("Print","-------------- %s : done -------------", GetName()); - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Notify error message - -void TProofOutputFile::NotifyError(const char *msg) -{ - if (msg) { - if (gProofServ) - gProofServ->SendAsynMessage(msg); - else - Printf("%s", msg); - } else { - Info("NotifyError","called with empty message"); - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add file to merger, checking the result - -void TProofOutputFile::AddFile(TFileMerger *merger, const char *path) -{ - if (merger && path) { - if (!merger->AddFile(path)) - NotifyError(Form("TProofOutputFile::AddFile:" - " error from TFileMerger::AddFile(%s)", path)); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Unlink path - -void TProofOutputFile::Unlink(const char *path) -{ - if (path) { - if (!gSystem->AccessPathName(path)) { - if (gSystem->Unlink(path) != 0) - NotifyError(Form("TProofOutputFile::Unlink:" - " error from TSystem::Unlink(%s)", path)); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get instance of the file collection to be used in 'dataset' mode - -TFileCollection *TProofOutputFile::GetFileCollection() -{ - if (!fDataSet) - fDataSet = new TFileCollection(GetTitle()); - return fDataSet; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get instance of the file merger to be used in 'merge' mode - -TFileMerger *TProofOutputFile::GetFileMerger(Bool_t local) -{ - if (!fMerger) - fMerger = new TFileMerger(local, fMergeHistosOneGo); - return fMerger; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Assert directory path 'dirpath', with the ownership of the last already -/// existing subpath. -/// Return 0 on success, -1 on error - -Int_t TProofOutputFile::AssertDir(const char *dirpath) -{ - TString existsPath(dirpath); - TList subPaths; - while (existsPath != "/" && existsPath != "." && gSystem->AccessPathName(existsPath)) { - subPaths.AddFirst(new TObjString(gSystem->BaseName(existsPath))); - existsPath = gSystem->GetDirName(existsPath); - } - subPaths.SetOwner(kTRUE); - FileStat_t st; - if (gSystem->GetPathInfo(existsPath, st) == 0) { - TString xpath = existsPath; - TIter nxp(&subPaths); - TObjString *os = 0; - while ((os = (TObjString *) nxp())) { - xpath += TString::Format("/%s", os->GetName()); - if (gSystem->mkdir(xpath, kTRUE) == 0) { - if (gSystem->Chmod(xpath, (UInt_t) st.fMode) != 0) - ::Warning("TProofOutputFile::AssertDir", "problems setting mode on '%s'", xpath.Data()); - } else { - ::Error("TProofOutputFile::AssertDir", "problems creating path '%s'", xpath.Data()); - return -1; - } - } - } else { - ::Warning("TProofOutputFile::AssertDir", "could not get info for path '%s': will only try to create" - " the full path w/o trying to set the mode", existsPath.Data()); - if (gSystem->mkdir(existsPath, kTRUE) != 0) { - ::Error("TProofOutputFile::AssertDir", "problems creating path '%s'", existsPath.Data()); - return -1; - } - } - // Done - return 0; -} diff --git a/proof/proof/src/TProofOutputList.cxx b/proof/proof/src/TProofOutputList.cxx deleted file mode 100644 index b90b7ece2e244..0000000000000 --- a/proof/proof/src/TProofOutputList.cxx +++ /dev/null @@ -1,156 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G. Ganis 04/08/2010 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofOutputList -\ingroup proofkernel - -Derivation of TList with an overload of ls() and Print() allowing to filter -out some of the variables - -*/ - -#include "TObjString.h" -#include "TProof.h" -#include "TProofOutputList.h" -#include "TRegexp.h" -#include "TString.h" - -ClassImp(TProofOutputList); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TProofOutputList::TProofOutputList(const char *dontshow) : TList() -{ - fDontShow = new TList(); - TString regs(dontshow), reg; - Int_t from = 0; - while (regs.Tokenize(reg, from, ",")) { - fDontShow->Add(new TObjString(reg)); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TProofOutputList::~TProofOutputList() -{ - fDontShow->SetOwner(kTRUE); - SafeDelete(fDontShow); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Attach to list 'alist' - -void TProofOutputList::AttachList(TList *alist) -{ - if (!alist) return; - - if (GetSize() > 0) Clear(); - - TIter nxo(alist); - TObject *obj = 0; - while ((obj = nxo())) { Add(obj); } - SetOwner(kFALSE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// List the content of the list - -void TProofOutputList::ls(Option_t *option) const -{ - TString opt(option); - opt.ToUpper(); - if (opt.BeginsWith("ALL")) { - opt.Remove(0,3); - TList::ls(opt); - } else { - TIter nxos(fDontShow); - TObjString *os = 0; - TList doShow; - doShow.SetOwner(kFALSE); - - Bool_t hasmissing = kFALSE; - TIter nxo(this); - TObject *obj = 0; - while ((obj = nxo())) { - TString s = obj->GetName(); - if (s == kPROOF_MissingFiles) { - TList *mf = dynamic_cast (obj); - if (mf && mf->GetSize() > 0) hasmissing = kTRUE; - } else { - nxos.Reset(); - Bool_t doadd = kTRUE; - while ((os = (TObjString *) nxos())) { - TRegexp rg(os->GetName(), kTRUE); - if (s.Index(rg) != kNPOS) { - doadd = kFALSE; - break; - } - } - if (doadd) doShow.Add(obj); - } - } - doShow.ls(option); - // Notify if missing files were found - if (hasmissing) - Printf(" +++ Missing files list not empty: use ShowMissingFiles to display it +++"); - } - // Done - return; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Print the content of the list - -void TProofOutputList::Print(Option_t *option) const -{ - TString opt(option); - opt.ToUpper(); - if (opt.BeginsWith("ALL")) { - opt.Remove(0,3); - TList::Print(opt); - } else { - TIter nxos(fDontShow); - TObjString *os = 0; - TList doShow; - doShow.SetOwner(kFALSE); - - Bool_t hasmissing = kFALSE; - TIter nxo(this); - TObject *obj = 0; - while ((obj = nxo())) { - TString s = obj->GetName(); - if (s == kPROOF_MissingFiles) { - TList *mf = dynamic_cast (obj); - if (mf && mf->GetSize() > 0) hasmissing = kTRUE; - } else { - nxos.Reset(); - Bool_t doadd = kTRUE; - while ((os = (TObjString *) nxos())) { - TRegexp rg(os->GetName()); - if (s.Index(rg) != kNPOS) {; - doadd = kFALSE; - break; - } - } - if (doadd) doShow.Add(obj); - } - } - doShow.Print(option); - // Notify if missing files were found - if (hasmissing) - Printf(" +++ Missing files list not empty: use ShowMissingFiles to display it +++"); - } - // Done - return; -} diff --git a/proof/proof/src/TProofProgressStatus.cxx b/proof/proof/src/TProofProgressStatus.cxx deleted file mode 100644 index 11000cad51ebd..0000000000000 --- a/proof/proof/src/TProofProgressStatus.cxx +++ /dev/null @@ -1,123 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Jan Iwaszkiewicz 08/08/08 - -/************************************************************************* - * Copyright (C) 1995-2008, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofProgressStatus -\ingroup proofkernel - -Container class for processing statistics - -*/ - -#include "TProofProgressStatus.h" -#include "TObject.h" -#include "TString.h" -#include "TSystem.h" -#include "TTime.h" - -ClassImp(TProofProgressStatus); - -//////////////////////////////////////////////////////////////////////////////// -/// Main and default constructor - -TProofProgressStatus::TProofProgressStatus(Long64_t entries, - Long64_t bytesRead, - Long64_t readCalls, - Double_t procTime, - Double_t cpuTime): TObject() -{ - fLastEntries = 0; - fEntries = entries; - fBytesRead = bytesRead; - fReadCalls = readCalls; - fLearnTime = 0.; - fLastProcTime = 0; - fProcTime = procTime; - fCPUTime = cpuTime; - SetLastUpdate(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// '+=' operator - -TProofProgressStatus& TProofProgressStatus::operator+=(const TProofProgressStatus &st) -{ - fLastEntries += st.fEntries; - fEntries += st.fEntries; - fBytesRead += st.fBytesRead; - fReadCalls += st.fReadCalls; - if (st.fLearnTime > fLearnTime) - fLearnTime = st.fLearnTime; - fLastProcTime = st.fProcTime; - fProcTime += st.fProcTime; - fCPUTime += st.fCPUTime; - SetLastUpdate(); - return *this; -} - -//////////////////////////////////////////////////////////////////////////////// -/// '-=' operator - -TProofProgressStatus& TProofProgressStatus::operator-=(const TProofProgressStatus &st) -{ - fEntries -= st.fEntries; - fBytesRead -= st.fBytesRead; - fReadCalls -= st.fReadCalls; - if (st.fLearnTime < fLearnTime) - fLearnTime = st.fLearnTime; - fProcTime -= st.fProcTime; - fCPUTime -= st.fCPUTime; - SetLastUpdate(); - return *this; -} - -//////////////////////////////////////////////////////////////////////////////// -/// '-' operator - -TProofProgressStatus TProofProgressStatus::operator-(TProofProgressStatus &st) -{ - return TProofProgressStatus(*this) -= st; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Dump the content - -void TProofProgressStatus::Print(Option_t*option) const -{ - Printf("TProofProgressStatus:%s: Ents:(%lld,%lld), Bytes:%lld, Calls:%lld," - " Learn:%.3g s, Proc:(%.3g,%.3g) s, CPU:%.3g s", - option, fEntries, fLastEntries, fBytesRead, fReadCalls, - fLearnTime, fProcTime, fLastProcTime, fCPUTime); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update time stamp either with the passed value (if > 0) or with -/// the current time - -void TProofProgressStatus::SetLastUpdate(Double_t updtTime) -{ - if (updtTime > 0) { - fLastUpdate = updtTime; - } else { - TTime tnow = gSystem->Now(); - fLastUpdate = (Double_t) (Long64_t(tnow)) / (Double_t)1000.; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get current rate. Rteunr the average rate if the current is not defined - -Double_t TProofProgressStatus::GetCurrentRate() const -{ - if (fLastProcTime > 0) { - return fLastEntries / fLastProcTime; - } - return GetRate(); -} diff --git a/proof/proof/src/TProofQueryResult.cxx b/proof/proof/src/TProofQueryResult.cxx deleted file mode 100644 index fa1e3e9f91d54..0000000000000 --- a/proof/proof/src/TProofQueryResult.cxx +++ /dev/null @@ -1,65 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G Ganis Sep 2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofQueryResult -\ingroup proofkernel - -TQueryResult version adapted to PROOF neeeds - -*/ - -#include "TDSet.h" -#include "TError.h" -#include "TList.h" -#include "TProofQueryResult.h" -#include "TSystem.h" -#include "TMacro.h" - -ClassImp(TProofQueryResult); - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor. - -TProofQueryResult::TProofQueryResult(Int_t sn, const char *opt, TList *inlist, - Long64_t ent, Long64_t fst, TDSet *dset, - const char *sel, TObject *elist) - : TQueryResult(sn, opt, inlist, ent, fst, sel) -{ - fNumWrks = -1; - fStartLog = -1; - - // Add data sets and event lists to the input list - if (fInputList && dset) - fInputList->Add(dset); - if (fInputList && elist) - fInputList->Add(elist); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Call when running starts. - -void TProofQueryResult::SetRunning(Int_t startlog, const char *par, Int_t nwrks) -{ - Info("SetRunning", "nwrks: %d", nwrks); - - fStatus = kRunning; - fNumWrks = nwrks; - fStart.Set(); - fEnd.Set(fStart.Convert()-1); - fParList = (par && (strlen(par) > 0)) ? par : "-"; - fStartLog = startlog; - - // Add header to log file - fLogFile->AddLine("+++"); - fLogFile->AddLine(Form("+++ Start processing query # %d (log file offset: %d)", - fSeqNum, startlog)); - fLogFile->AddLine("+++"); -} diff --git a/proof/proof/src/TProofResourcesStatic.cxx b/proof/proof/src/TProofResourcesStatic.cxx deleted file mode 100644 index 54bbcc54a1823..0000000000000 --- a/proof/proof/src/TProofResourcesStatic.cxx +++ /dev/null @@ -1,386 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Paul Nilsson 7/12/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofResourcesStatic -\ingroup proofkernel - -The purpose of this class is to provide a standard interface to -static config files. It interprets Proof config files (proof.conf) -and sorts the contents into TProofNodeInfo objects. Master info will -be placed in fMaster (of type TProofNodeInfo). Submaster info will -be put in fSubmasterList (a TList of TProofNodeInfo objects), while -workers (and condorworkers) will be placed in fWorkerList (a TList -of TProofNodeInfo objects). - -*/ - -#include "Riostream.h" -#include "TProofResourcesStatic.h" -#include "TSystem.h" -#include "TInetAddress.h" -#include "TProofNodeInfo.h" -#include "TProofDebug.h" -#include "TUrl.h" -#include "TList.h" -#include "TObjArray.h" -#include "TObjString.h" -#include "TError.h" - -ClassImp(TProofResourcesStatic); - - -//////////////////////////////////////////////////////////////////////////////// -/// This ctor is used in TProofServ::Setup() in combination with GetWorkDir() -/// for a quick scan of the config file to retrieve the work directory. - -TProofResourcesStatic::TProofResourcesStatic() -{ - // Create master node info and submaster/worker lists, and set default values - InitResources(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Using this ctor will retrieve all information in the config file -/// and store it in fMaster, fSubmasterList and fWorkerList, -/// condorworkers will be stored in the fWorkerList. - -TProofResourcesStatic::TProofResourcesStatic(const char *confDir, - const char *fileName) -{ - // Create master node info and submaster/worker lists, and set default values - InitResources(); - - // Open and read the PROOF config file - if (!ReadConfigFile(confDir, fileName)) { - PDB(kAll,1) - Info("TProofResourcesStatic", "error encountered while reading config file"); - fValid = kFALSE; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TProofResourcesStatic::~TProofResourcesStatic() -{ - delete fSubmasterList; - delete fWorkerList; - delete fMaster; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create master node info and submaster/worker lists, -/// and set default values. - -void TProofResourcesStatic::InitResources() -{ - // Create master - fMaster = new TProofNodeInfo(); - fMaster->fNodeType = TProofNodeInfo::GetNodeType("master"); - fFoundMaster = kFALSE; // Set to kTRUE if the config file contains master info - - // Create workers - fWorkerList = new TList(); - fWorkerList->SetOwner(); - - // Create submaster - fSubmasterList = new TList(); - fSubmasterList->SetOwner(); - - // Assume that the config file will be ok - fValid = kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the master node. Only return the master info if it was set -/// in the config file. - -TProofNodeInfo *TProofResourcesStatic::GetMaster() -{ - if (fFoundMaster) - return fMaster; - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the list of submaster nodes. - -TList *TProofResourcesStatic::GetSubmasters() -{ - return fSubmasterList; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the list of worker nodes. - -TList *TProofResourcesStatic::GetWorkers(void) -{ - return fWorkerList; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Read the PROOF config file and fill the master and worker list. - -Bool_t TProofResourcesStatic::ReadConfigFile(const char *confDir, - const char *fileName) -{ - Bool_t status = kTRUE; - - // Skip prefix (e.g. "sm:") if any - const char *p = (const char *) strstr(fileName,":"); - if (p) - fileName = p+1; - - // Use file specified by the cluster administrator, if any - const char *cf = gSystem->Getenv("ROOTPROOFCONF"); - if (cf && !(gSystem->AccessPathName(cf, kReadPermission))) { - fFileName = cf; - } else { - if (cf) - PDB(kGlobal,1) - Info("ReadConfigFile", "file %s cannot be read:" - " check existence and/or permissions", cf); - if (fileName && strlen(fileName) > 0) { - // Use user defined file or default - // Add a proper path to the file name - fFileName.Form("%s/.%s", gSystem->HomeDirectory(), fileName); - PDB(kGlobal,2) - Info("ReadConfigFile", "checking PROOF config file %s", fFileName.Data()); - if (gSystem->AccessPathName(fFileName, kReadPermission)) { - fFileName.Form("%s/etc/proof/%s", confDir, fileName); - PDB(kGlobal,2) - Info("ReadConfigFile", "checking PROOF config file %s", fFileName.Data()); - if (gSystem->AccessPathName(fFileName, kReadPermission)) { - PDB(kAll,1) - Info("ReadConfigFile", "no PROOF config file found"); - return kFALSE; - } - } - } else { - PDB(kAll,1) - Info("ReadConfigFile", "no PROOF config file specified"); - return kFALSE; - } - } - PDB(kGlobal,1) - Info("ReadConfigFile", "using PROOF config file: %s", fFileName.Data()); - - // Open the config file - std::fstream infile(fFileName.Data(), std::ios::in); - if (infile.is_open()) { - Bool_t isMaster = kFALSE; - Bool_t isSubmaster = kFALSE; - Bool_t isWorker = kFALSE; - - // Each line in the file consists of several 'keywords', e.g. - // line = "master mypc image=local" - // keyword[0] = "master" - // keyword[1] = "mypc" - // keyword[2] = "image=local" - // The last keyword has an option "image" with value "local" - TString line = ""; - TString keyword = ""; - - // Read the entire file into the allLines object - TString allLines = ""; - allLines.ReadString(infile); - TObjArray *lines = allLines.Tokenize("\n"); - Int_t numberOfLines = lines->GetEntries(); - - // Process one line at the time - for (Int_t j = 0; j < numberOfLines; j++) { - TObjString *objLine = (TObjString *)lines->At(j); - line = objLine->GetString(); - line = line.Strip(TString::kBoth); - - // Unless this line was empty or a comment, interpret the line - if ( !((line(0,1) == "#") || (line == "")) ) { - TProofNodeInfo *nodeinfo = 0; - - // Reset boolean (condorworkers are treated as a workers) - isMaster = kFALSE; - isSubmaster = kFALSE; - isWorker = kFALSE; - - // Extract all words in the current line - TObjArray *tokens = line.Tokenize(" "); - Int_t n = tokens->GetEntries(); - TString option; - TString value; - for (Int_t i = 0; i < n; i++) { - - // Extrace one word from the current line - keyword = ((TObjString *)tokens->At(i))->GetString(); - - // Interpret this keyword - switch (GetInfoType(keyword)) { - case kNodeType: { - if (keyword == "master" || keyword == "node") { - nodeinfo = CreateNodeInfo(keyword); - isMaster = kTRUE; // will be reset - } - // [either submaster, worker or condorworker] - else if (keyword == "submaster") { - // Get a submaster info node - nodeinfo = CreateNodeInfo(keyword); - isSubmaster = kTRUE; - } else { - // Get a worker or condorworker info node - nodeinfo = CreateNodeInfo(keyword); - isWorker = kTRUE; - } - break; - } - case kHost: { - // Store the host name - if (nodeinfo) { - nodeinfo->fNodeName = keyword; - - // Set default image - if (isMaster) { - TString node = TUrl(nodeinfo->fNodeName).GetHost(); - nodeinfo->fImage = strstr(nodeinfo->fNodeName, node.Data()); - } else { - // If the node name contains an '@' sign, it should be removed - // before copying it into the [default] image info - // On what position is the '@' sign? (if there is one) - TString tmp = nodeinfo->fNodeName; - const Ssiz_t equalPosition = tmp.Index("@", 1, 0, TString::kExact); - - // Extract the host - nodeinfo->fImage = tmp(equalPosition + 1, tmp.Length()); - } - } else { - Error("ReadConfigFile","Command not recognized: %s (ignored)", - keyword.Data()); - } - break; - } - case kOption: { - // On what position is the '=' sign? - const Ssiz_t equalPosition = - keyword.Index("=", 1, 0, TString::kExact); - - // Extract the option and its value - TString tmp = keyword; - option = tmp(0, equalPosition); - value = tmp(equalPosition + 1, tmp.Length()); - - // Set the node info options - SetOption(nodeinfo, option, value); - break; - } - default: - break; - } // end switch - - } // end if - - // Check if we found a good master - if (isMaster) { - // Check if the master can run on this node - TString node = TUrl(nodeinfo->fNodeName).GetHost(); - TString host = gSystem->GetHostByName(gSystem->HostName()).GetHostName(); - TInetAddress inetaddr = gSystem->GetHostByName(node); - if (!host.CompareTo(inetaddr.GetHostName()) || (node == "localhost")) { - fFoundMaster = kTRUE; - fMaster->Assign(*nodeinfo); - } - } - - // Store the submaster, worker or condorworker - if (isWorker) { - fWorkerList->Add(nodeinfo); - } - else if (isSubmaster) { - fSubmasterList->Add(nodeinfo); - } - } // else - - } // while (! infile.eof() ) - infile.close(); - - // Did the config file contain appropriate master information? - if (!fFoundMaster) { - Error("ReadConfigFile","No master info found in config file"); - status = kFALSE; - } - } // end if (infile.is_open()) - else { - // Error: could not open file - status = kFALSE; - } - - return status; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Static method to set the node info options. - -void TProofResourcesStatic::SetOption(TProofNodeInfo *nodeinfo, - const TString &option, - const TString &value) -{ - if (!nodeinfo) return; - - if (option == "workdir") { - nodeinfo->fWorkDir = value; - } else if (option == "image") { - nodeinfo->fImage = value; - } else if (option == "perf") { - nodeinfo->fPerfIndex = value.Atoi(); - } else if (option == "config") { - nodeinfo->fConfig = value; - } else if (option == "msd") { - nodeinfo->fMsd = value; - } else if (option == "port") { - nodeinfo->fPort = value.Atoi(); - } else { - ::Error("SetOption","No such option [%s=%s]",option.Data(),value.Data()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static method to determine the info type. - -TProofResourcesStatic::EInfoType TProofResourcesStatic::GetInfoType(const TString &word) -{ - EInfoType type = kNodeType; - - if ((word == "node") || (word == "master") || (word == "submaster") || - (word == "worker") || (word == "slave") || - (word == "condorworker") || (word == "condorslave")) { - type = kNodeType; - } - else if (word.Contains("=", TString::kExact)) { - type = kOption; - } else { - type = kHost; - } - - return type; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fill out the preliminary TProofNodeInfo structure. - -TProofNodeInfo *TProofResourcesStatic::CreateNodeInfo(const TString &name) -{ - TProofNodeInfo *nodeInfo = new TProofNodeInfo(); - nodeInfo->fNodeType = TProofNodeInfo::GetNodeType(name); - nodeInfo->fNodeName = name; - nodeInfo->fPort = -1; - nodeInfo->fPerfIndex = 100; - - return nodeInfo; -} diff --git a/proof/proof/src/TProofServ.cxx b/proof/proof/src/TProofServ.cxx deleted file mode 100644 index 9b6f3a201da54..0000000000000 --- a/proof/proof/src/TProofServ.cxx +++ /dev/null @@ -1,7047 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 16/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofServ -\ingroup proofkernel - -Class providing the PROOF server. It can act either as the master -server or as a slave server, depending on its startup arguments. It -receives and handles message coming from the client or from the -master server. - -*/ - -#include "RConfigure.h" -#include -#include "Riostream.h" - -#ifdef WIN32 - #include - #include - #include "snprintf.h" - typedef long off_t; -#endif -#include -#include -#include -#include -#include -#ifndef WIN32 -#include -#endif -#include - -// To handle exceptions -#include -#include - -#if (defined(__FreeBSD__) && (__FreeBSD__ < 4)) || \ - (defined(__APPLE__) && (!defined(MAC_OS_X_VERSION_10_3) || \ - (MAC_OS_X_VERSION_MAX_ALLOWED < MAC_OS_X_VERSION_10_3))) -#include -#define lockf(fd, op, sz) flock((fd), (op)) -#ifndef F_LOCK -#define F_LOCK (LOCK_EX | LOCK_NB) -#endif -#ifndef F_ULOCK -#define F_ULOCK LOCK_UN -#endif -#endif - -#include "TProofServ.h" -#include "TDSetProxy.h" -#include "TEnv.h" -#include "TError.h" -#include "TEventList.h" -#include "TEntryList.h" -#include "TException.h" -#include "TFile.h" -#include "THashList.h" -#include "TInterpreter.h" -#include "TKey.h" -#include "TMessage.h" -#include "TVirtualPerfStats.h" -#include "TProofDebug.h" -#include "TProof.h" -#include "TVirtualProofPlayer.h" -#include "TProofQueryResult.h" -#include "TQueryResultManager.h" -#include "TRegexp.h" -#include "TROOT.h" -#include "TObjArray.h" -#include "TSocket.h" -#include "TStopwatch.h" -#include "TSystem.h" -#include "TTimeStamp.h" -#include "TUrl.h" -#include "TPluginManager.h" -#include "TObjString.h" -#include "compiledata.h" -#include "TProofResourcesStatic.h" -#include "TProofNodeInfo.h" -#include "TFileInfo.h" -#include "TClass.h" -#include "TSQLServer.h" -#include "TSQLResult.h" -#include "TSQLRow.h" -#include "TPRegexp.h" -#include "TParameter.h" -#include "TMap.h" -#include "TSortedList.h" -#include "TFileCollection.h" -#include "TLockFile.h" -#include "TDataSetManagerFile.h" -#include "TProofProgressStatus.h" -#include "TServerSocket.h" -#include "TMonitor.h" -#include "TProofOutputFile.h" -#include "TSelector.h" -#include "TPackMgr.h" - -// global proofserv handle -TProofServ *gProofServ = 0; - -// debug hook -static volatile Int_t gProofServDebug = 1; - -// Syslog control -Int_t TProofServ::fgLogToSysLog = 0; -TString TProofServ::fgSysLogService("proof"); -TString TProofServ::fgSysLogEntity("undef:default"); - -// File where to log: default stderr -FILE *TProofServ::fgErrorHandlerFile = 0; - -// Integrate with crash reporter. -#ifdef __APPLE__ -extern "C" { -static const char *__crashreporter_info__ = 0; -asm(".desc ___crashreporter_info__, 0x10"); -} -#endif - -// To control allowed actions while processing -Int_t TProofServ::fgRecursive = 0; - -// Last message and entry before exceptions -TString TProofServ::fgLastMsg(""); -Long64_t TProofServ::fgLastEntry = -1; - -// Memory controllers -Long_t TProofServ::fgVirtMemMax = -1; -Long_t TProofServ::fgResMemMax = -1; -Float_t TProofServ::fgMemHWM = 0.80; -Float_t TProofServ::fgMemStop = 0.95; - -// Async Logger -static void SendAsynMsg(const char *msg) { - if (gProofServ) gProofServ->SendAsynMessage(msg, kTRUE); -} - -//----- Termination signal handler --------------------------------------------- -//////////////////////////////////////////////////////////////////////////////// - -class TProofServTerminationHandler : public TSignalHandler { - TProofServ *fServ; -public: - TProofServTerminationHandler(TProofServ *s) - : TSignalHandler(kSigTermination, kFALSE) { fServ = s; } - Bool_t Notify() override; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Handle this interrupt - -Bool_t TProofServTerminationHandler::Notify() -{ - Printf("Received SIGTERM: terminating"); - fServ->HandleTermination(); - return kTRUE; -} - -//----- Interrupt signal handler ----------------------------------------------- -//////////////////////////////////////////////////////////////////////////////// - -class TProofServInterruptHandler : public TSignalHandler { - TProofServ *fServ; -public: - TProofServInterruptHandler(TProofServ *s) - : TSignalHandler(kSigUrgent, kFALSE) { fServ = s; } - Bool_t Notify() override; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Handle this interrupt - -Bool_t TProofServInterruptHandler::Notify() -{ - fServ->HandleUrgentData(); - if (TROOT::Initialized()) { - Throw(GetSignal()); - } - return kTRUE; -} - -//----- SigPipe signal handler ------------------------------------------------- -//////////////////////////////////////////////////////////////////////////////// - -class TProofServSigPipeHandler : public TSignalHandler { - TProofServ *fServ; -public: - TProofServSigPipeHandler(TProofServ *s) : TSignalHandler(kSigPipe, kFALSE) - { fServ = s; } - Bool_t Notify() override; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Handle this signal - -Bool_t TProofServSigPipeHandler::Notify() -{ - fServ->HandleSigPipe(); - return kTRUE; -} - -//----- Input handler for messages from parent or master ----------------------- -//////////////////////////////////////////////////////////////////////////////// - -class TProofServInputHandler : public TFileHandler { - TProofServ *fServ; -public: - TProofServInputHandler(TProofServ *s, Int_t fd) : TFileHandler(fd, 1) - { fServ = s; } - Bool_t Notify() override; - Bool_t ReadNotify() override { return Notify(); } -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Handle this input - -Bool_t TProofServInputHandler::Notify() -{ - fServ->HandleSocketInput(); - return kTRUE; -} - -TString TProofServLogHandler::fgPfx = ""; // Default prefix to be prepended to messages -Int_t TProofServLogHandler::fgCmdRtn = 0; // Return code of the command execution (available only - // after closing the pipe) -//////////////////////////////////////////////////////////////////////////////// -/// Execute 'cmd' in a pipe and handle output messages from the related file - -TProofServLogHandler::TProofServLogHandler(const char *cmd, - TSocket *s, const char *pfx) - : TFileHandler(-1, 1), fSocket(s), fPfx(pfx) -{ - ResetBit(kFileIsPipe); - fgCmdRtn = 0; - fFile = 0; - if (s && cmd) { - fFile = gSystem->OpenPipe(cmd, "r"); - if (fFile) { - SetFd(fileno(fFile)); - // Notify what already in the file - Notify(); - // Used in the destructor - SetBit(kFileIsPipe); - } else { - fSocket = 0; - Error("TProofServLogHandler", "executing command in pipe"); - fgCmdRtn = -1; - } - } else { - Error("TProofServLogHandler", - "undefined command (%p) or socket (%p)", (int *)cmd, s); - } -} -//////////////////////////////////////////////////////////////////////////////// -/// Handle available message from the open file 'f' - -TProofServLogHandler::TProofServLogHandler(FILE *f, TSocket *s, const char *pfx) - : TFileHandler(-1, 1), fSocket(s), fPfx(pfx) -{ - ResetBit(kFileIsPipe); - fgCmdRtn = 0; - fFile = 0; - if (s && f) { - fFile = f; - SetFd(fileno(fFile)); - // Notify what already in the file - Notify(); - } else { - Error("TProofServLogHandler", "undefined file (%p) or socket (%p)", f, s); - } -} -//////////////////////////////////////////////////////////////////////////////// -/// Handle available message in the open file - -TProofServLogHandler::~TProofServLogHandler() -{ - if (TestBit(kFileIsPipe) && fFile) { - Int_t rc = gSystem->ClosePipe(fFile); -#ifdef WIN32 - fgCmdRtn = rc; -#else - fgCmdRtn = WIFEXITED(rc) ? WEXITSTATUS(rc) : -1; -#endif - } - fFile = 0; - fSocket = 0; - ResetBit(kFileIsPipe); -} -//////////////////////////////////////////////////////////////////////////////// -/// Handle available message in the open file - -Bool_t TProofServLogHandler::Notify() -{ - if (IsValid()) { - TMessage m(kPROOF_MESSAGE); - // Read buffer - char line[4096]; - char *plf = 0; - while (fgets(line, sizeof(line), fFile)) { - if ((plf = strchr(line, '\n'))) - *plf = 0; - // Create log string - TString log; - if (fPfx.Length() > 0) { - // Prepend prefix specific to this instance - log.Form("%s: %s", fPfx.Data(), line); - } else if (fgPfx.Length() > 0) { - // Prepend default prefix - log.Form("%s: %s", fgPfx.Data(), line); - } else { - // Nothing to prepend - log = line; - } - // Send the message one level up - m.Reset(kPROOF_MESSAGE); - m << log; - fSocket->Send(m); - } - } - return kTRUE; -} -//////////////////////////////////////////////////////////////////////////////// -/// Static method to set the default prefix - -void TProofServLogHandler::SetDefaultPrefix(const char *pfx) -{ - fgPfx = pfx; -} -//////////////////////////////////////////////////////////////////////////////// -/// Static method to get the return code from the execution of a command via -/// the pipe. This is always 0 when the log handler is not used with a pipe - -Int_t TProofServLogHandler::GetCmdRtn() -{ - return fgCmdRtn; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Init a guard for executing a command in a pipe - -TProofServLogHandlerGuard::TProofServLogHandlerGuard(const char *cmd, TSocket *s, - const char *pfx, Bool_t on) -{ - fExecHandler = 0; - if (cmd && on) { - fExecHandler = new TProofServLogHandler(cmd, s, pfx); - if (fExecHandler->IsValid()) { - gSystem->AddFileHandler(fExecHandler); - } else { - Error("TProofServLogHandlerGuard","invalid handler"); - } - } else { - if (on) - Error("TProofServLogHandlerGuard","undefined command"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Init a guard for executing a command in a pipe - -TProofServLogHandlerGuard::TProofServLogHandlerGuard(FILE *f, TSocket *s, - const char *pfx, Bool_t on) -{ - fExecHandler = 0; - if (f && on) { - fExecHandler = new TProofServLogHandler(f, s, pfx); - if (fExecHandler->IsValid()) { - gSystem->AddFileHandler(fExecHandler); - } else { - Error("TProofServLogHandlerGuard","invalid handler"); - } - } else { - if (on) - Error("TProofServLogHandlerGuard","undefined file"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close a guard for executing a command in a pipe - -TProofServLogHandlerGuard::~TProofServLogHandlerGuard() -{ - if (fExecHandler && fExecHandler->IsValid()) { - gSystem->RemoveFileHandler(fExecHandler); - SafeDelete(fExecHandler); - } -} - -//--- Special timer to control delayed shutdowns ----------------------------// -//////////////////////////////////////////////////////////////////////////////// -/// Construtor - -TShutdownTimer::TShutdownTimer(TProofServ *p, Int_t delay) - : TTimer(delay, kFALSE), fProofServ(p) -{ - fTimeout = gEnv->GetValue("ProofServ.ShutdownTimeout", 20); - // Backward compaitibility: until 5.32 the variable was called ProofServ.ShutdonwTimeout - fTimeout = gEnv->GetValue("ProofServ.ShutdonwTimeout", fTimeout); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle expiration of the shutdown timer. In the case of low activity the -/// process will be aborted. - -Bool_t TShutdownTimer::Notify() -{ - if (gDebug > 0) - printf("TShutdownTimer::Notify: checking activity on the input socket\n"); - - // Check activity on the socket - TSocket *xs = 0; - if (fProofServ && (xs = fProofServ->GetSocket())) { - TTimeStamp now; - TTimeStamp ts = xs->GetLastUsage(); - Long_t dt = (Long_t)(now.GetSec() - ts.GetSec()) * 1000 + - (Long_t)(now.GetNanoSec() - ts.GetNanoSec()) / 1000000 ; - if (dt > fTimeout * 60000) { - printf("TShutdownTimer::Notify: input socket: %p: did not show any activity" - " during the last %d mins: aborting\n", xs, fTimeout); - // At this point we lost our controller: we need to abort to avoid - // hidden timeouts or loops - gSystem->Abort(); - } else { - if (gDebug > 0) - printf("TShutdownTimer::Notify: input socket: %p: show activity" - " %ld secs ago\n", xs, dt / 60000); - } - } - // Needed for the next shot - Reset(); - return kTRUE; -} - -//--- Synchronous timer used to reap children processes change of state ------// -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TReaperTimer::~TReaperTimer() -{ - if (fChildren) { - fChildren->SetOwner(kTRUE); - delete fChildren; - fChildren = 0; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add an entry for 'pid' in the internal list - -void TReaperTimer::AddPid(Int_t pid) -{ - if (pid > 0) { - if (!fChildren) - fChildren = new TList; - TString spid; - spid.Form("%d", pid); - fChildren->Add(new TParameter(spid.Data(), pid)); - TurnOn(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check if any of the registered children has changed its state. -/// Unregister those that are gone. - -Bool_t TReaperTimer::Notify() -{ - if (fChildren) { - TIter nxp(fChildren); - TParameter *p = 0; - while ((p = (TParameter *)nxp())) { - int status; -#ifndef WIN32 - pid_t pid; - do { - pid = waitpid(p->GetVal(), &status, WNOHANG); - } while (pid < 0 && errno == EINTR); -#else - intptr_t pid; - pid = _cwait(&status, (intptr_t)p->GetVal(), 0); -#endif - if (pid > 0 && pid == p->GetVal()) { - // Remove from the list - fChildren->Remove(p); - delete p; - } - } - } - - // Stop the timer if no children - if (!fChildren || fChildren->GetSize() <= 0) { - Stop(); - } else { - // Needed for the next shot - Reset(); - } - return kTRUE; -} - -//--- Special timer to terminate idle sessions ----------------------------// -//////////////////////////////////////////////////////////////////////////////// -/// Handle expiration of the idle timer. The session will just be terminated. - -Bool_t TIdleTOTimer::Notify() -{ - Info ("Notify", "session idle for more then %lld secs: terminating", Long64_t(fTime)/1000); - - if (fProofServ) { - // Set the status to timed-out - Int_t uss_rc = -1; - if ((uss_rc = fProofServ->UpdateSessionStatus(4)) != 0) - Warning("Notify", "problems updating session status (errno: %d)", -uss_rc); - // Send a terminate request - TString msg; - if (fProofServ->GetProtocol() < 29) { - msg.Form("\n//\n// PROOF session at %s (%s) terminated because idle for more than %lld secs\n" - "// Please IGNORE any error message possibly displayed below\n//", - gSystem->HostName(), fProofServ->GetSessionTag(), Long64_t(fTime)/1000); - } else { - msg.Form("\n//\n// PROOF session at %s (%s) terminated because idle for more than %lld secs\n//", - gSystem->HostName(), fProofServ->GetSessionTag(), Long64_t(fTime)/1000); - } - fProofServ->SendAsynMessage(msg.Data()); - fProofServ->Terminate(0); - Reset(); - Stop(); - } else { - Warning("Notify", "fProofServ undefined!"); - Start(-1, kTRUE); - } - return kTRUE; -} - -ClassImp(TProofServ); - -// Hook to the constructor. This is needed to avoid using the plugin manager -// which may create problems in multi-threaded environments. -extern "C" { - TApplication *GetTProofServ(Int_t *argc, char **argv, FILE *flog) - { return new TProofServ(argc, argv, flog); } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor. Create an application environment. The TProofServ -/// environment provides an eventloop via inheritance of TApplication. -/// Actual server creation work is done in CreateServer() to allow -/// overloading. - -TProofServ::TProofServ(Int_t *argc, char **argv, FILE *flog) - : TApplication("proofserv", argc, argv, 0, -1) -{ - // If test and tty, we are done - Bool_t xtest = (argc && *argc == 1) ? kTRUE : kFALSE; - if (xtest) { - Printf("proofserv: command line testing: OK"); - exit(0); - } - - // Read session specific rootrc file - TString rcfile = gSystem->Getenv("ROOTRCFILE") ? gSystem->Getenv("ROOTRCFILE") - : "session.rootrc"; - if (!gSystem->AccessPathName(rcfile, kReadPermission)) - gEnv->ReadFile(rcfile, kEnvChange); - - // Upper limit on Virtual Memory (in kB) - fgVirtMemMax = gEnv->GetValue("Proof.VirtMemMax",-1); - if (fgVirtMemMax < 0 && gSystem->Getenv("PROOF_VIRTMEMMAX")) { - Long_t mmx = strtol(gSystem->Getenv("PROOF_VIRTMEMMAX"), 0, 10); - if (mmx < kMaxLong && mmx > 0) - fgVirtMemMax = mmx * 1024; - } - // Old variable for backward compatibility - if (fgVirtMemMax < 0 && gSystem->Getenv("ROOTPROOFASHARD")) { - Long_t mmx = strtol(gSystem->Getenv("ROOTPROOFASHARD"), 0, 10); - if (mmx < kMaxLong && mmx > 0) - fgVirtMemMax = mmx * 1024; - } - // Upper limit on Resident Memory (in kB) - fgResMemMax = gEnv->GetValue("Proof.ResMemMax",-1); - if (fgResMemMax < 0 && gSystem->Getenv("PROOF_RESMEMMAX")) { - Long_t mmx = strtol(gSystem->Getenv("PROOF_RESMEMMAX"), 0, 10); - if (mmx < kMaxLong && mmx > 0) - fgResMemMax = mmx * 1024; - } - // Thresholds for warnings and stop processing - fgMemStop = gEnv->GetValue("Proof.MemStop", 0.95); - fgMemHWM = gEnv->GetValue("Proof.MemHWM", 0.80); - if (fgVirtMemMax > 0 || fgResMemMax > 0) { - if ((fgMemStop < 0.) || (fgMemStop > 1.)) { - Warning("TProofServ", "requested memory fraction threshold to stop processing" - " (MemStop) out of range [0,1] - ignoring"); - fgMemStop = 0.95; - } - if ((fgMemHWM < 0.) || (fgMemHWM > fgMemStop)) { - Warning("TProofServ", "requested memory fraction threshold for warning and finer monitoring" - " (MemHWM) out of range [0,MemStop] - ignoring"); - fgMemHWM = 0.80; - } - } - - // Wait (loop) to allow debugger to connect - Bool_t test = (argc && *argc >= 4 && !strcmp(argv[3], "test")) ? kTRUE : kFALSE; - if ((gEnv->GetValue("Proof.GdbHook",0) == 3 && !test) || - (gEnv->GetValue("Proof.GdbHook",0) == 4 && test)) { - while (gProofServDebug) - ; - } - - // Test instance - if (argc && *argc >= 4) - if (!strcmp(argv[3], "test")) - fService = "prooftest"; - - // crude check on number of arguments - if (argc && *argc < 2) { - Error("TProofServ", "Must have at least 1 arguments (see proofd)."); - exit(1); - } - - // Set global to this instance - gProofServ = this; - - // Log control flags - fSendLogToMaster = kFALSE; - - // Abort on higher than kSysError's and set error handler - gErrorAbortLevel = kSysError + 1; - SetErrorHandlerFile(stderr); - SetErrorHandler(ErrorHandler); - - fNcmd = 0; - fGroupPriority = 100; - fInterrupt = kFALSE; - fProtocol = 0; - fOrdinal = gEnv->GetValue("ProofServ.Ordinal", "-1"); - fGroupId = -1; - fGroupSize = 0; - fRealTime = 0.0; - fCpuTime = 0.0; - fProof = 0; - fPlayer = 0; - fSocket = 0; - - fTotSessions = -1; - fActSessions = -1; - fEffSessions = -1.; - fPackMgr = 0; - - fLogFile = flog; - fLogFileDes = -1; - - fArchivePath = ""; - // Init lockers - fCacheLock = 0; - fQueryLock = 0; - - fQMgr = 0; - fWaitingQueries = new TList; - fIdle = kTRUE; - fQuerySeqNum = -1; - - fQueuedMsg = new TList; - - fRealTimeLog = kFALSE; - - fShutdownTimer = 0; - fReaperTimer = 0; - fIdleTOTimer = 0; - - fDataSetManager = 0; // Initialized in Setup() - fDataSetStgRepo = 0; // Initialized in Setup() - - fInputHandler = 0; - - // Quotas disabled by default - fMaxQueries = -1; - fMaxBoxSize = -1; - fHWMBoxSize = -1; - - // Submerger quantities - fMergingSocket = 0; - fMergingMonitor = 0; - fMergedWorkers = 0; - - // Bit to flg high-memory footprint - ResetBit(TProofServ::kHighMemory); - - // Max message size - fMsgSizeHWM = gEnv->GetValue("ProofServ.MsgSizeHWM", 1000000); - - // Message compression - fCompressMsg = gEnv->GetValue("ProofServ.CompressMessage", 0); - - gProofDebugLevel = gEnv->GetValue("Proof.DebugLevel",0); - fLogLevel = gProofDebugLevel; - - gProofDebugMask = (TProofDebug::EProofDebugMask) gEnv->GetValue("Proof.DebugMask",~0); - if (gProofDebugLevel > 0) - Info("TProofServ", "DebugLevel %d Mask 0x%x", gProofDebugLevel, gProofDebugMask); - - // Max log file size - fLogFileMaxSize = -1; - TString logmx = gEnv->GetValue("ProofServ.LogFileMaxSize", ""); - if (!logmx.IsNull()) { - Long64_t xf = 1; - if (!logmx.IsDigit()) { - if (logmx.EndsWith("K")) { - xf = 1024; - logmx.Remove(TString::kTrailing, 'K'); - } else if (logmx.EndsWith("M")) { - xf = 1024*1024; - logmx.Remove(TString::kTrailing, 'M'); - } else if (logmx.EndsWith("G")) { - xf = 1024*1024*1024; - logmx.Remove(TString::kTrailing, 'G'); - } - } - if (logmx.IsDigit()) { - fLogFileMaxSize = logmx.Atoi() * xf; - if (fLogFileMaxSize > 0) - Info("TProofServ", "keeping the log file size within %lld bytes", fLogFileMaxSize); - } else { - logmx = gEnv->GetValue("ProofServ.LogFileMaxSize", ""); - Warning("TProofServ", "bad formatted log file size limit ignored: '%s'", logmx.Data()); - } - } - - // Parse options - GetOptions(argc, argv); - - // Default prefix in the form '-' - fPrefix = (IsMaster() ? "Mst-" : "Wrk-"); - if (test) fPrefix = "Test"; - if (fOrdinal != "-1") - fPrefix += fOrdinal; - TProofServLogHandler::SetDefaultPrefix(fPrefix); - - // Syslog control - TString slog = gEnv->GetValue("ProofServ.LogToSysLog", ""); - if (!(slog.IsNull())) { - if (slog.IsDigit()) { - fgLogToSysLog = slog.Atoi(); - } else { - char c = (slog[0] == 'M' || slog[0] == 'm') ? 'm' : 'a'; - c = (slog[0] == 'W' || slog[0] == 'w') ? 'w' : c; - Bool_t dosyslog = ((c == 'm' && IsMaster()) || - (c == 'w' && !IsMaster()) || c == 'a') ? kTRUE : kFALSE; - if (dosyslog) { - slog.Remove(0,1); - if (slog.IsDigit()) fgLogToSysLog = slog.Atoi(); - if (fgLogToSysLog <= 0) - Warning("TProofServ", "request for syslog logging ineffective!"); - } - } - } - // Initialize proper service if required - if (fgLogToSysLog > 0) { - fgSysLogService = (IsMaster()) ? "proofm" : "proofw"; - if (fOrdinal != "-1") fgSysLogService += TString::Format("-%s", fOrdinal.Data()); - gSystem->Openlog(fgSysLogService, kLogPid | kLogCons, kLogLocal5); - } - - // Enable optimized sending of streamer infos to use embedded backward/forward - // compatibility support between different ROOT versions and different versions of - // users classes - Bool_t enableSchemaEvolution = gEnv->GetValue("Proof.SchemaEvolution",1); - if (enableSchemaEvolution) { - TMessage::EnableSchemaEvolutionForAll(); - } else { - Info("TProofServ", "automatic schema evolution in TMessage explicitly disabled"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize the server setup. If master, create the TProof instance to talk -/// to the worker or submaster nodes. -/// Return 0 on success, -1 on error - -Int_t TProofServ::CreateServer() -{ - // Get socket to be used (setup in proofd) - TString opensock = gSystem->Getenv("ROOTOPENSOCK"); - if (opensock.Length() <= 0) - opensock = gEnv->GetValue("ProofServ.OpenSock", "-1"); - Int_t sock = opensock.Atoi(); - if (sock <= 0) { - Fatal("CreateServer", "Invalid socket descriptor number (%d)", sock); - return -1; - } - fSocket = new TSocket(sock); - - // Set compression level, if any - fSocket->SetCompressionSettings(fCompressMsg); - - // debug hooks - if (IsMaster()) { - // wait (loop) in master to allow debugger to connect - if (gEnv->GetValue("Proof.GdbHook",0) == 1) { - while (gProofServDebug) - ; - } - } else { - // wait (loop) in slave to allow debugger to connect - if (gEnv->GetValue("Proof.GdbHook",0) == 2) { - while (gProofServDebug) - ; - } - } - - if (gProofDebugLevel > 0) - Info("CreateServer", "Service %s ConfDir %s IsMaster %d\n", - GetService(), GetConfDir(), (Int_t)fMasterServ); - - if (Setup() != 0) { - // Setup failure - LogToMaster(); - SendLogFile(); - Terminate(0); - return -1; - } - - // Set the default prefix in the form '-' (it was already done - // in the constructor, but for standard PROOF the ordinal number is only set in - // Setup(), so we need to do it again here) - TString pfx = (IsMaster() ? "Mst-" : "Wrk-"); - pfx += GetOrdinal(); - TProofServLogHandler::SetDefaultPrefix(pfx); - - if (!fLogFile) { - RedirectOutput(); - // If for some reason we failed setting a redirection file for the logs - // we cannot continue - if (!fLogFile || (fLogFileDes = fileno(fLogFile)) < 0) { - LogToMaster(); - SendLogFile(-98); - Terminate(0); - return -1; - } - } else { - // Use the file already open by pmain - if ((fLogFileDes = fileno(fLogFile)) < 0) { - LogToMaster(); - SendLogFile(-98); - Terminate(0); - return -1; - } - } - - // Send message of the day to the client - if (IsMaster()) { - if (CatMotd() == -1) { - LogToMaster(); - SendLogFile(-99); - Terminate(0); - return -1; - } - } - - // Everybody expects std::iostream to be available, so load it... - ProcessLine("#include ", kTRUE); - ProcessLine("#include ",kTRUE); // for std::string std::iostream. - - // The following libs are also useful to have, make sure they are loaded... - //gROOT->LoadClass("TMinuit", "Minuit"); - //gROOT->LoadClass("TPostScript", "Postscript"); - - // Load user functions - const char *logon; - logon = gEnv->GetValue("Proof.Load", (char *)0); - if (logon) { - char *mac = gSystem->Which(TROOT::GetMacroPath(), logon, kReadPermission); - if (mac) - ProcessLine(TString::Format(".L %s", logon), kTRUE); - delete [] mac; - } - - // Execute logon macro - logon = gEnv->GetValue("Proof.Logon", (char *)0); - if (logon && !NoLogOpt()) { - char *mac = gSystem->Which(TROOT::GetMacroPath(), logon, kReadPermission); - if (mac) - ProcessFile(logon); - delete [] mac; - } - - // Save current interpreter context - gInterpreter->SaveContext(); - gInterpreter->SaveGlobalsContext(); - - // Install interrupt and message input handlers - gSystem->AddSignalHandler(new TProofServTerminationHandler(this)); - gSystem->AddSignalHandler(new TProofServInterruptHandler(this)); - fInputHandler = new TProofServInputHandler(this, sock); - gSystem->AddFileHandler(fInputHandler); - - // if master, start slave servers - if (IsMaster()) { - TString master = "proof://__master__"; - TInetAddress a = gSystem->GetSockName(sock); - if (a.IsValid()) { - master += ":"; - master += a.GetPort(); - } - - // Get plugin manager to load appropriate TProof from - TPluginManager *pm = gROOT->GetPluginManager(); - if (!pm) { - Error("CreateServer", "no plugin manager found"); - SendLogFile(-99); - Terminate(0); - return -1; - } - - // Find the appropriate handler - TPluginHandler *h = pm->FindHandler("TProof", fConfFile); - if (!h) { - Error("CreateServer", "no plugin found for TProof with a" - " config file of '%s'", fConfFile.Data()); - SendLogFile(-99); - Terminate(0); - return -1; - } - - // load the plugin - if (h->LoadPlugin() == -1) { - Error("CreateServer", "plugin for TProof could not be loaded"); - SendLogFile(-99); - Terminate(0); - return -1; - } - - // make instance of TProof - fProof = reinterpret_cast(h->ExecPlugin(5, master.Data(), - fConfFile.Data(), - GetConfDir(), - fLogLevel, 0)); - if (!fProof || !fProof->IsValid()) { - Error("CreateServer", "plugin for TProof could not be executed"); - SafeDelete(fProof); - SendLogFile(-99); - Terminate(0); - return -1; - } - // Find out if we are a master in direct contact only with workers - fEndMaster = fProof->IsEndMaster(); - - SendLogFile(); - } - - // Setup the shutdown timer - if (!fShutdownTimer) { - // Check activity on socket every 5 mins - fShutdownTimer = new TShutdownTimer(this, 300000); - fShutdownTimer->Start(-1, kFALSE); - } - - // Check if schema evolution is effective: clients running versions <=17 do not - // support that: send a warning message - if (fProtocol <= 17) { - TString msg; - msg.Form("Warning: client version is too old: automatic schema evolution is ineffective.\n" - " This may generate compatibility problems between streamed objects.\n" - " The advise is to move to ROOT >= 5.21/02 ."); - SendAsynMessage(msg.Data()); - } - - // Setup the idle timer - if (IsMaster() && !fIdleTOTimer) { - // Check activity on socket every 5 mins - Int_t idle_to = gEnv->GetValue("ProofServ.IdleTimeout", -1); - if (idle_to > 0) { - fIdleTOTimer = new TIdleTOTimer(this, idle_to * 1000); - fIdleTOTimer->Start(-1, kTRUE); - if (gProofDebugLevel > 0) - Info("CreateServer", " idle timer started (%d secs)", idle_to); - } else if (gProofDebugLevel > 0) { - Info("CreateServer", " idle timer not started (no idle timeout requested)"); - } - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup. Not really necessary since after this dtor there is no -/// live anyway. - -TProofServ::~TProofServ() -{ - SafeDelete(fWaitingQueries); - SafeDelete(fSocket); - SafeDelete(fPackMgr); - SafeDelete(fCacheLock); - SafeDelete(fQueryLock); - SafeDelete(fDataSetManager); - SafeDelete(fDataSetStgRepo); - close(fLogFileDes); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print message of the day (in the file pointed by the env PROOFMOTD -/// or from fConfDir/etc/proof/motd). The motd is not shown more than -/// once a day. If the file pointed by env PROOFNOPROOF exists (or the -/// file fConfDir/etc/proof/noproof exists), show its contents and close -/// the connection. - -Int_t TProofServ::CatMotd() -{ - TString lastname; - FILE *motd; - Bool_t show = kFALSE; - - // If we are disabled just print the message and close the connection - TString motdname(GetConfDir()); - // The env variable PROOFNOPROOF allows to put the file in an alternative - // location not overwritten by a new installation - if (gSystem->Getenv("PROOFNOPROOF")) { - motdname = gSystem->Getenv("PROOFNOPROOF"); - } else { - motdname += "/etc/proof/noproof"; - } - if ((motd = fopen(motdname, "r"))) { - Int_t c; - printf("\n"); - while ((c = getc(motd)) != EOF) - putchar(c); - fclose(motd); - printf("\n"); - - return -1; - } - - // get last modification time of the file ~/proof/.prooflast - lastname = TString(GetWorkDir()) + "/.prooflast"; - gSystem->ExpandPathName(lastname); - Long64_t size; - Long_t id, flags, modtime, lasttime = 0; - if (gSystem->GetPathInfo(lastname.Data(), &id, &size, &flags, &lasttime) == 1) - lasttime = 0; - - // show motd at least once per day - if (time(0) - lasttime > (time_t)86400) - show = kTRUE; - - // The env variable PROOFMOTD allows to put the file in an alternative - // location not overwritten by a new installation - if (gSystem->Getenv("PROOFMOTD")) { - motdname = gSystem->Getenv("PROOFMOTD"); - } else { - motdname = GetConfDir(); - motdname += "/etc/proof/motd"; - } - if (gSystem->GetPathInfo(motdname, &id, &size, &flags, &modtime) == 0) { - if (modtime > lasttime || show) { - if ((motd = fopen(motdname, "r"))) { - Int_t c; - printf("\n"); - while ((c = getc(motd)) != EOF) - putchar(c); - fclose(motd); - printf("\n"); - } - } - } - - if (lasttime) - gSystem->Unlink(lastname.Data()); - Int_t fd = creat(lastname.Data(), 0600); - if (fd >= 0) close(fd); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get object with name "name;cycle" (e.g. "aap;2") from master or client. -/// This method is called by TDirectory::Get() in case the object can not -/// be found locally. - -TObject *TProofServ::Get(const char *namecycle) -{ - if (fSocket->Send(namecycle, kPROOF_GETOBJECT) < 0) { - Error("Get", "problems sending request"); - return (TObject *)0; - } - - TObject *idcur = 0; - - Bool_t notdone = kTRUE; - while (notdone) { - TMessage *mess = 0; - if (fSocket->Recv(mess) < 0) - return 0; - Int_t what = mess->What(); - if (what == kMESS_OBJECT) { - idcur = mess->ReadObject(mess->GetClass()); - notdone = kFALSE; - } else { - Int_t xrc = HandleSocketInput(mess, kFALSE); - if (xrc == -1) { - Error("Get", "command %d cannot be executed while processing", what); - } else if (xrc == -2) { - Error("Get", "unknown command %d ! Protocol error?", what); - } - } - delete mess; - } - - return idcur; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset the compute time - -void TProofServ::RestartComputeTime() -{ - fCompute.Stop(); - if (fPlayer) { - TProofProgressStatus *status = fPlayer->GetProgressStatus(); - if (status) status->SetLearnTime(fCompute.RealTime()); - Info("RestartComputeTime", "compute time restarted after %f secs (%d entries)", - fCompute.RealTime(), fPlayer->GetLearnEntries()); - } - fCompute.Start(kFALSE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next range of entries to be processed on this server. - -TDSetElement *TProofServ::GetNextPacket(Long64_t totalEntries) -{ - Long64_t bytesRead = 0; - - if (gPerfStats) bytesRead = gPerfStats->GetBytesRead(); - - if (fCompute.Counter() > 0) - fCompute.Stop(); - - TMessage req(kPROOF_GETPACKET); - Double_t cputime = fCompute.CpuTime(); - Double_t realtime = fCompute.RealTime(); - - if (fProtocol > 18) { - req << fLatency.RealTime(); - TProofProgressStatus *status = 0; - if (fPlayer) { - fPlayer->UpdateProgressInfo(); - status = fPlayer->GetProgressStatus(); - } else { - Error("GetNextPacket", "no progress status object"); - return 0; - } - // the CPU and wallclock proc times are kept in the TProofServ and here - // added to the status object in the fPlayer. - if (status->GetEntries() > 0) { - PDB(kLoop, 2) status->Print(GetOrdinal()); - status->IncProcTime(realtime); - status->IncCPUTime(cputime); - } - // Flag cases with problems in opening files - if (totalEntries < 0) status->SetBit(TProofProgressStatus::kFileNotOpen); - // Add to the message - req << status; - // Send tree cache information - Long64_t cacheSize = (fPlayer) ? fPlayer->GetCacheSize() : -1; - Int_t learnent = (fPlayer) ? fPlayer->GetLearnEntries() : -1; - req << cacheSize << learnent; - - // Sent over the number of entries in the file, used by packetizer do not relying - // on initial validation. Also, -1 means that the file could not be open, which is - // used to flag files as missing - req << totalEntries; - - // Send the time spent in saving the partial result to file - if (fProtocol > 34) req << fSaveOutput.RealTime(); - - PDB(kLoop, 1) { - PDB(kLoop, 2) status->Print(); - Info("GetNextPacket","cacheSize: %lld, learnent: %d", cacheSize, learnent); - } - // Reset the status bits - status->ResetBit(TProofProgressStatus::kFileNotOpen); - status->ResetBit(TProofProgressStatus::kFileCorrupted); - status = 0; // status is owned by the player. - } else { - req << fLatency.RealTime() << realtime << cputime - << bytesRead << totalEntries; - if (fPlayer) - req << fPlayer->GetEventsProcessed(); - } - - fLatency.Start(); - Int_t rc = fSocket->Send(req); - if (rc <= 0) { - Error("GetNextPacket","Send() failed, returned %d", rc); - return 0; - } - - // Save the current output - if (fPlayer) { - fSaveOutput.Start(); - if (fPlayer->SavePartialResults(kFALSE) < 0) - Warning("GetNextPacket", "problems saving partial results"); - fSaveOutput.Stop(); - } - - TDSetElement *e = 0; - Bool_t notdone = kTRUE; - while (notdone) { - - TMessage *mess; - if ((rc = fSocket->Recv(mess)) <= 0) { - fLatency.Stop(); - Error("GetNextPacket","Recv() failed, returned %d", rc); - return 0; - } - - Int_t xrc = 0; - TString file, dir, obj; - - Int_t what = mess->What(); - - switch (what) { - case kPROOF_GETPACKET: - - fLatency.Stop(); - (*mess) >> e; - if (e != 0) { - fCompute.Start(); - PDB(kLoop, 2) Info("GetNextPacket", "'%s' '%s' '%s' %lld %lld", - e->GetFileName(), e->GetDirectory(), - e->GetObjName(), e->GetFirst(),e->GetNum()); - } else { - PDB(kLoop, 2) Info("GetNextPacket", "Done"); - } - notdone = kFALSE; - break; - - case kPROOF_STOPPROCESS: - // if a kPROOF_STOPPROCESS message is returned to kPROOF_GETPACKET - // GetNextPacket() will return 0 and the TPacketizer and hence - // TEventIter will be stopped - fLatency.Stop(); - PDB(kLoop, 2) Info("GetNextPacket:kPROOF_STOPPROCESS","received"); - break; - - default: - xrc = HandleSocketInput(mess, kFALSE); - if (xrc == -1) { - Error("GetNextPacket", "command %d cannot be executed while processing", what); - } else if (xrc == -2) { - Error("GetNextPacket", "unknown command %d ! Protocol error?", what); - } - break; - } - - delete mess; - - } - - // Done - return e; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get and handle command line options. Fixed format: -/// "proofserv"|"proofslave" `` - -void TProofServ::GetOptions(Int_t *argc, char **argv) -{ - Bool_t xtest = (argc && *argc > 3 && !strcmp(argv[3], "test")) ? kTRUE : kFALSE; - - // If test and tty - if (xtest && !(isatty(0) == 0 || isatty(1) == 0)) { - Printf("proofserv: command line testing: OK"); - exit(0); - } - - if (!argc || (argc && *argc <= 1)) { - Fatal("GetOptions", "Must be started from proofd with arguments"); - exit(1); - } - - if (!strcmp(argv[1], "proofserv")) { - fMasterServ = kTRUE; - fEndMaster = kTRUE; - } else if (!strcmp(argv[1], "proofslave")) { - fMasterServ = kFALSE; - fEndMaster = kFALSE; - } else { - Fatal("GetOptions", "Must be started as 'proofserv' or 'proofslave'"); - exit(1); - } - - fService = argv[1]; - - // Confdir - if (!(gSystem->Getenv("ROOTCONFDIR"))) { - Fatal("GetOptions", "ROOTCONFDIR shell variable not set"); - exit(1); - } - fConfDir = gSystem->Getenv("ROOTCONFDIR"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle input coming from the client or from the master server. - -void TProofServ::HandleSocketInput() -{ - // The idle timeout guard: stops the timer and restarts when we return from here - TIdleTOTimerGuard itg(fIdleTOTimer); - - Bool_t all = (fgRecursive > 0) ? kFALSE : kTRUE; - fgRecursive++; - - TMessage *mess; - Int_t rc = 0; - TString exmsg; - - // Check log file length (before the action, so we have the chance to keep the - // latest logs) - TruncateLogFile(); - - try { - - // Get message - if (fSocket->Recv(mess) <= 0 || !mess) { - // Pending: do something more intelligent here - // but at least get a message in the log file - Error("HandleSocketInput", "retrieving message from input socket"); - Terminate(0); - return; - } - Int_t what = mess->What(); - PDB(kCollect, 1) - Info("HandleSocketInput", "got type %d from '%s'", what, fSocket->GetTitle()); - - fNcmd++; - - if (fProof) fProof->SetActive(); - - Bool_t doit = kTRUE; - - while (doit) { - - // Process the message - rc = HandleSocketInput(mess, all); - if (rc < 0) { - TString emsg; - if (rc == -1) { - emsg.Form("HandleSocketInput: command %d cannot be executed while processing", what); - } else if (rc == -3) { - emsg.Form("HandleSocketInput: message %d undefined! Protocol error?", what); - } else { - emsg.Form("HandleSocketInput: unknown command %d! Protocol error?", what); - } - SendAsynMessage(emsg.Data()); - } else if (rc == 2) { - // Add to the queue - fQueuedMsg->Add(mess); - PDB(kGlobal, 1) - Info("HandleSocketInput", "message of type %d enqueued; sz: %d", - what, fQueuedMsg->GetSize()); - mess = 0; - } - - // Still something to do? - doit = 0; - if (fgRecursive == 1 && fQueuedMsg->GetSize() > 0) { - // Add to the queue - PDB(kCollect, 1) - Info("HandleSocketInput", "processing enqueued message of type %d; left: %d", - what, fQueuedMsg->GetSize()); - all = 1; - SafeDelete(mess); - mess = (TMessage *) fQueuedMsg->First(); - if (mess) fQueuedMsg->Remove(mess); - doit = 1; - } - } - - } catch (std::bad_alloc &) { - // Memory allocation problem: - exmsg.Form("caught exception 'bad_alloc' (memory leak?) %s %lld", - fgLastMsg.Data(), fgLastEntry); - } catch (std::exception &exc) { - // Standard exception caught - exmsg.Form("caught standard exception '%s' %s %lld", - exc.what(), fgLastMsg.Data(), fgLastEntry); - } catch (int i) { - // Other exception caught - exmsg.Form("caught exception throwing %d %s %lld", - i, fgLastMsg.Data(), fgLastEntry); - } catch (const char *str) { - // Other exception caught - exmsg.Form("caught exception throwing '%s' %s %lld", - str, fgLastMsg.Data(), fgLastEntry); - } catch (...) { - // Caught other exception - exmsg.Form("caught exception %s %lld", - fgLastMsg.Data(), fgLastEntry); - } - - // Terminate on exception - if (!exmsg.IsNull()) { - // Save info in the log file too - Error("HandleSocketInput", "%s", exmsg.Data()); - // Try to warn the user - SendAsynMessage(TString::Format("%s: %s", GetOrdinal(), exmsg.Data())); - // Terminate - Terminate(0); - } - - // Terminate also if a high memory footprint was detected before the related - // exception was thrwon - if (TestBit(TProofServ::kHighMemory)) { - // Save info in the log file too - exmsg.Form("high-memory footprint detected during Process(...) - terminating"); - Error("HandleSocketInput", "%s", exmsg.Data()); - // Try to warn the user - SendAsynMessage(TString::Format("%s: %s", GetOrdinal(), exmsg.Data())); - // Terminate - Terminate(0); - } - - fgRecursive--; - - if (fProof) { - // If something wrong went on during processing and we do not have - // any worker anymore, we shutdown this session - Bool_t masterOnly = gEnv->GetValue("Proof.MasterOnly", kFALSE); - Bool_t dynamicStartup = gEnv->GetValue("Proof.DynamicStartup", kFALSE); - Int_t ngwrks = fProof->GetListOfActiveSlaves()->GetSize() + fProof->GetListOfInactiveSlaves()->GetSize(); - if (rc == 0 && ngwrks == 0 && !masterOnly && !dynamicStartup) { - SendAsynMessage(" *** No workers left: cannot continue! Terminating ... *** "); - Terminate(0); - } - fProof->SetActive(kFALSE); - // Reset PROOF to running state - fProof->SetRunStatus(TProof::kRunning); - } - - // Cleanup - SafeDelete(mess); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process input coming from the client or from the master server. -/// If 'all' is kFALSE, process only those messages that can be handled -/// during query processing. -/// Returns -1 if the message could not be processed, <-1 if something went -/// wrong. Returns 1 if the action may have changed the parallel state. -/// Returns 2 if the message has to be enqueued. -/// Returns 0 otherwise - -Int_t TProofServ::HandleSocketInput(TMessage *mess, Bool_t all) -{ - static TStopwatch timer; - char str[2048]; - Bool_t aborted = kFALSE; - - if (!mess) return -3; - - Int_t what = mess->What(); - PDB(kCollect, 1) - Info("HandleSocketInput", "processing message type %d from '%s'", - what, fSocket->GetTitle()); - - timer.Start(); - - Int_t rc = 0, lirc = 0; - TString slb; - TString *pslb = (fgLogToSysLog > 0) ? &slb : (TString *)0; - - switch (what) { - - case kMESS_CINT: - if (all) { - mess->ReadString(str, sizeof(str)); - // Make sure that the relevant files are available - TString fn; - - Bool_t hasfn = TProof::GetFileInCmd(str, fn); - - if (IsParallel() && fProof && !fProof->UseDynamicStartup()) { - fProof->SendCommand(str); - } else { - PDB(kGlobal, 1) - Info("HandleSocketInput:kMESS_CINT", "processing: %s...", str); - TString ocwd; - if (hasfn) { - fCacheLock->Lock(); - ocwd = gSystem->WorkingDirectory(); - gSystem->ChangeDirectory(fCacheDir.Data()); - } - ProcessLine(str); - if (hasfn) { - gSystem->ChangeDirectory(ocwd); - fCacheLock->Unlock(); - } - } - - LogToMaster(); - } else { - rc = -1; - } - SendLogFile(); - if (pslb) slb = str; - break; - - case kMESS_STRING: - if (all) { - mess->ReadString(str, sizeof(str)); - } else { - rc = -1; - } - break; - - case kMESS_OBJECT: - if (all) { - mess->ReadObject(mess->GetClass()); - } else { - rc = -1; - } - break; - - case kPROOF_GROUPVIEW: - if (all) { - mess->ReadString(str, sizeof(str)); - // coverity[secure_coding] - sscanf(str, "%d %d", &fGroupId, &fGroupSize); - } else { - rc = -1; - } - break; - - case kPROOF_LOGLEVEL: - { UInt_t mask; - mess->ReadString(str, sizeof(str)); - sscanf(str, "%d %u", &fLogLevel, &mask); - Bool_t levelchanged = (fLogLevel != gProofDebugLevel) ? kTRUE : kFALSE; - gProofDebugLevel = fLogLevel; - gProofDebugMask = (TProofDebug::EProofDebugMask) mask; - if (levelchanged) - Info("HandleSocketInput:kPROOF_LOGLEVEL", "debug level set to %d (mask: 0x%x)", - gProofDebugLevel, gProofDebugMask); - if (IsMaster()) - fProof->SetLogLevel(fLogLevel, mask); - } - break; - - case kPROOF_PING: - { if (IsMaster()) - fProof->Ping(); - // do nothing (ping is already acknowledged) - } - break; - - case kPROOF_PRINT: - mess->ReadString(str, sizeof(str)); - Print(str); - LogToMaster(); - SendLogFile(); - break; - - case kPROOF_RESET: - if (all) { - mess->ReadString(str, sizeof(str)); - Reset(str); - } else { - rc = -1; - } - break; - - case kPROOF_STATUS: - Warning("HandleSocketInput:kPROOF_STATUS", - "kPROOF_STATUS message is obsolete"); - if (fSocket->Send(fProof->GetParallel(), kPROOF_STATUS) < 0) - Warning("HandleSocketInput:kPROOF_STATUS", "problem sending of request"); - break; - - case kPROOF_GETSTATS: - SendStatistics(); - break; - - case kPROOF_GETPARALLEL: - SendParallel(); - break; - - case kPROOF_STOP: - if (all) { - if (IsMaster()) { - TString ord; - *mess >> ord; - PDB(kGlobal, 1) - Info("HandleSocketInput:kPROOF_STOP", "request for worker %s", ord.Data()); - if (fProof) fProof->TerminateWorker(ord); - } else { - PDB(kGlobal, 1) - Info("HandleSocketInput:kPROOF_STOP", "got request to terminate"); - Terminate(0); - } - } else { - rc = -1; - } - break; - - case kPROOF_STOPPROCESS: - if (all) { - // this message makes only sense when the query is being processed, - // however the message can also be received if the user pressed - // ctrl-c, so ignore it! - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_STOPPROCESS","enter"); - } else { - Long_t timeout = -1; - (*mess) >> aborted; - if (fProtocol > 9) - (*mess) >> timeout; - PDB(kGlobal, 1) - Info("HandleSocketInput:kPROOF_STOPPROCESS", - "recursive mode: enter %d, %ld", aborted, timeout); - if (fProof) - // On the master: propagate further - fProof->StopProcess(aborted, timeout); - else - // Worker: actually stop processing - if (fPlayer) - fPlayer->StopProcess(aborted, timeout); - } - break; - - case kPROOF_PROCESS: - { - TProofServLogHandlerGuard hg(fLogFile, fSocket, "", fRealTimeLog); - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_PROCESS","enter"); - HandleProcess(mess, pslb); - // The log file is send either in HandleProcess or HandleSubmergers. - // The reason is that the order of various messages depend on the - // processing mode (sync/async) and/or merging mode - } - break; - - case kPROOF_SENDOUTPUT: - { - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_SENDOUTPUT", - "worker was asked to send output to master"); - Int_t sorc = 0; - if (SendResults(fSocket, fPlayer->GetOutputList()) != 0) { - Error("HandleSocketInput:kPROOF_SENDOUTPUT", "problems sending output list"); - sorc = 1; - } - // Signal the master that we are idle - fSocket->Send(kPROOF_SETIDLE); - SetIdle(kTRUE); - DeletePlayer(); - SendLogFile(sorc); - } - break; - - case kPROOF_QUERYLIST: - { - HandleQueryList(mess); - // Notify - SendLogFile(); - } - break; - - case kPROOF_REMOVE: - { - HandleRemove(mess, pslb); - // Notify - SendLogFile(); - } - break; - - case kPROOF_RETRIEVE: - { - HandleRetrieve(mess, pslb); - // Notify - SendLogFile(); - } - break; - - case kPROOF_ARCHIVE: - { - HandleArchive(mess, pslb); - // Notify - SendLogFile(); - } - break; - - case kPROOF_MAXQUERIES: - { PDB(kGlobal, 1) - Info("HandleSocketInput:kPROOF_MAXQUERIES", "Enter"); - TMessage m(kPROOF_MAXQUERIES); - m << fMaxQueries; - fSocket->Send(m); - // Notify - SendLogFile(); - } - break; - - case kPROOF_CLEANUPSESSION: - if (all) { - PDB(kGlobal, 1) - Info("HandleSocketInput:kPROOF_CLEANUPSESSION", "Enter"); - TString stag; - (*mess) >> stag; - if (fQMgr && fQMgr->CleanupSession(stag) == 0) { - Printf("Session %s cleaned up", stag.Data()); - } else { - Printf("Could not cleanup session %s", stag.Data()); - } - } else { - rc = -1; - } - // Notify - SendLogFile(); - break; - - case kPROOF_GETENTRIES: - { PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_GETENTRIES", "Enter"); - Bool_t isTree; - TString filename; - TString dir; - TString objname("undef"); - Long64_t entries = -1; - - if (all) { - (*mess) >> isTree >> filename >> dir >> objname; - PDB(kGlobal, 2) Info("HandleSocketInput:kPROOF_GETENTRIES", - "Report size of object %s (%s) in dir %s in file %s", - objname.Data(), isTree ? "T" : "O", - dir.Data(), filename.Data()); - entries = TDSet::GetEntries(isTree, filename, dir, objname); - PDB(kGlobal, 2) Info("HandleSocketInput:kPROOF_GETENTRIES", - "Found %lld %s", entries, isTree ? "entries" : "objects"); - } else { - rc = -1; - } - TMessage answ(kPROOF_GETENTRIES); - answ << entries << objname; - SendLogFile(); // in case of error messages - fSocket->Send(answ); - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_GETENTRIES", "Done"); - } - break; - - case kPROOF_CHECKFILE: - if (!all && fProtocol <= 19) { - // Come back later - rc = 2; - } else { - // Handle file checking request - HandleCheckFile(mess, pslb); - FlushLogFile(); // Avoid sending (error) messages at next action - } - break; - - case kPROOF_SENDFILE: - if (!all && fProtocol <= 19) { - // Come back later - rc = 2; - } else { - mess->ReadString(str, sizeof(str)); - Long_t size; - Int_t bin, fw = 1; - char name[1024]; - if (fProtocol > 5) { - sscanf(str, "%1023s %d %ld %d", name, &bin, &size, &fw); - } else { - sscanf(str, "%1023s %d %ld", name, &bin, &size); - } - TString fnam(name); - Bool_t copytocache = kTRUE; - if (fnam.BeginsWith("cache:")) { - fnam.ReplaceAll("cache:", TString::Format("%s/", fCacheDir.Data())); - copytocache = kFALSE; - } - - Int_t rfrc = 0; - if (size > 0) { - rfrc = ReceiveFile(fnam, bin ? kTRUE : kFALSE, size); - } else { - // Take it from the cache - if (!fnam.BeginsWith(fCacheDir.Data())) { - fnam.Insert(0, TString::Format("%s/", fCacheDir.Data())); - } - } - if (rfrc == 0) { - // copy file to cache if not a PAR file - if (copytocache && size > 0 && !fPackMgr->IsInDir(name)) - gSystem->Exec(TString::Format("%s %s %s", kCP, fnam.Data(), fCacheDir.Data())); - if (IsMaster() && fw == 1) { - Int_t opt = TProof::kForward | TProof::kCp; - if (bin) - opt |= TProof::kBinary; - PDB(kGlobal, 1) - Info("HandleSocketInput","forwarding file: %s", fnam.Data()); - if (fProof->SendFile(fnam, opt, (copytocache ? "cache" : "")) < 0) { - Error("HandleSocketInput", "forwarding file: %s", fnam.Data()); - } - } - if (fProtocol > 19) fSocket->Send(kPROOF_SENDFILE); - } else { - // There was an error - SendLogFile(1); - } - } - break; - - case kPROOF_LOGFILE: - { - Int_t start, end; - (*mess) >> start >> end; - PDB(kGlobal, 1) - Info("HandleSocketInput:kPROOF_LOGFILE", - "Logfile request - byte range: %d - %d", start, end); - - LogToMaster(); - SendLogFile(0, start, end); - } - break; - - case kPROOF_PARALLEL: - if (all) { - if (IsMaster()) { - Int_t nodes; - Bool_t random = kFALSE; - (*mess) >> nodes; - if ((mess->BufferSize() > mess->Length())) - (*mess) >> random; - if (fProof) fProof->SetParallel(nodes, random); - rc = 1; - } - } else { - rc = -1; - } - // Notify - SendLogFile(); - break; - - case kPROOF_CACHE: - if (!all && fProtocol <= 19) { - // Come back later - rc = 2; - } else { - TProofServLogHandlerGuard hg(fLogFile, fSocket, "", fRealTimeLog); - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_CACHE","enter"); - Int_t hcrc = HandleCache(mess, pslb); - // Notify - SendLogFile(hcrc); - } - break; - - case kPROOF_WORKERLISTS: - { Int_t wlrc = -1; - if (all) { - if (IsMaster()) - wlrc = HandleWorkerLists(mess); - else - Warning("HandleSocketInput:kPROOF_WORKERLISTS", - "Action meaning-less on worker nodes: protocol error?"); - } else { - rc = -1; - } - // Notify - SendLogFile(wlrc); - } - break; - - case kPROOF_GETSLAVEINFO: - if (all) { - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_GETSLAVEINFO", "Enter"); - if (IsMaster()) { - - Bool_t ok = kTRUE; - // if the session does not have workers and is in the dynamic mode - if (fProof->UseDynamicStartup()) { - ok = kFALSE; - // get the a list of workers and start them - Int_t pc = 0; - TList* workerList = new TList(); - EQueryAction retVal = GetWorkers(workerList, pc); - if (retVal != TProofServ::kQueryStop && retVal != TProofServ::kQueryEnqueued) { - Int_t ret = fProof->AddWorkers(workerList); - if (ret < 0) { - Error("HandleSocketInput:kPROOF_GETSLAVEINFO", - "adding a list of worker nodes returned: %d", ret); - } - } else { - Error("HandleSocketInput:kPROOF_GETSLAVEINFO", - "getting list of worker nodes returned: %d", retVal); - } - ok = kTRUE; - } - if (ok) { - TList *info = fProof->GetListOfSlaveInfos(); - TMessage answ(kPROOF_GETSLAVEINFO); - answ << info; - fSocket->Send(answ); - // stop the workers - if (IsMaster() && fProof->UseDynamicStartup()) fProof->RemoveWorkers(0); - } - } else { - TMessage answ(kPROOF_GETSLAVEINFO); - TList *info = new TList; - TSlaveInfo *wi = new TSlaveInfo(GetOrdinal(), TUrl(gSystem->HostName()).GetHostFQDN(), 0, "", GetDataDir()); - SysInfo_t si; - gSystem->GetSysInfo(&si); - wi->SetSysInfo(si); - info->Add(wi); - answ << (TList *)info; - fSocket->Send(answ); - info->SetOwner(kTRUE); - delete info; - } - - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_GETSLAVEINFO", "Done"); - } else { - TMessage answ(kPROOF_GETSLAVEINFO); - answ << (TList *)0; - fSocket->Send(answ); - rc = -1; - } - break; - - case kPROOF_GETTREEHEADER: - if (all) { - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_GETTREEHEADER", "Enter"); - - TVirtualProofPlayer *p = TVirtualProofPlayer::Create("slave", 0, fSocket); - if (p) { - p->HandleGetTreeHeader(mess); - delete p; - } else { - Error("HandleSocketInput:kPROOF_GETTREEHEADER", "could not create TProofPlayer instance!"); - } - - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_GETTREEHEADER", "Done"); - } else { - TMessage answ(kPROOF_GETTREEHEADER); - answ << TString("Failed") << (TObject *)0; - fSocket->Send(answ); - rc = -1; - } - break; - - case kPROOF_GETOUTPUTLIST: - { PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_GETOUTPUTLIST", "Enter"); - TList* outputList = 0; - if (IsMaster()) { - outputList = fProof->GetOutputList(); - if (!outputList) - outputList = new TList(); - } else { - outputList = new TList(); - if (fProof->GetPlayer()) { - TList *olist = fProof->GetPlayer()->GetOutputList(); - TIter next(olist); - TObject *o; - while ( (o = next()) ) { - outputList->Add(new TNamed(o->GetName(), "")); - } - } - } - outputList->SetOwner(); - TMessage answ(kPROOF_GETOUTPUTLIST); - answ << outputList; - fSocket->Send(answ); - delete outputList; - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_GETOUTPUTLIST", "Done"); - } - break; - - case kPROOF_VALIDATE_DSET: - if (all) { - PDB(kGlobal, 1) - Info("HandleSocketInput:kPROOF_VALIDATE_DSET", "Enter"); - - TDSet* dset = 0; - (*mess) >> dset; - - if (IsMaster()) fProof->ValidateDSet(dset); - else dset->Validate(); - - TMessage answ(kPROOF_VALIDATE_DSET); - answ << dset; - fSocket->Send(answ); - delete dset; - PDB(kGlobal, 1) - Info("HandleSocketInput:kPROOF_VALIDATE_DSET", "Done"); - } else { - rc = -1; - } - // Notify - SendLogFile(); - break; - - case kPROOF_DATA_READY: - if (all) { - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_DATA_READY", "Enter"); - TMessage answ(kPROOF_DATA_READY); - if (IsMaster()) { - Long64_t totalbytes = 0, bytesready = 0; - Bool_t dataready = fProof->IsDataReady(totalbytes, bytesready); - answ << dataready << totalbytes << bytesready; - } else { - Error("HandleSocketInput:kPROOF_DATA_READY", - "This message should not be sent to slaves"); - answ << kFALSE << Long64_t(0) << Long64_t(0); - } - fSocket->Send(answ); - PDB(kGlobal, 1) Info("HandleSocketInput:kPROOF_DATA_READY", "Done"); - } else { - TMessage answ(kPROOF_DATA_READY); - answ << kFALSE << Long64_t(0) << Long64_t(0); - fSocket->Send(answ); - rc = -1; - } - // Notify - SendLogFile(); - break; - - case kPROOF_DATASETS: - { Int_t dsrc = -1; - if (fProtocol > 16) { - dsrc = HandleDataSets(mess, pslb); - } else { - Error("HandleSocketInput", "old client: no or incompatible dataset support"); - } - SendLogFile(dsrc); - } - break; - - case kPROOF_SUBMERGER: - { HandleSubmerger(mess); - } - break; - - case kPROOF_LIB_INC_PATH: - if (all) { - lirc = HandleLibIncPath(mess); - } else { - rc = -1; - } - // Notify the client - if (lirc > 0) SendLogFile(); - break; - - case kPROOF_REALTIMELOG: - { Bool_t on; - (*mess) >> on; - PDB(kGlobal, 1) - Info("HandleSocketInput:kPROOF_REALTIMELOG", - "setting real-time logging %s", (on ? "ON" : "OFF")); - fRealTimeLog = on; - // Forward the request to lower levels - if (IsMaster()) - fProof->SetRealTimeLog(on); - } - break; - - case kPROOF_FORK: - if (all) { - HandleFork(mess); - LogToMaster(); - } else { - rc = -1; - } - SendLogFile(); - break; - - case kPROOF_STARTPROCESS: - if (all) { - // This message resumes the session; should not come during processing. - - if (WaitingQueries() == 0) { - Error("HandleSocketInput", "no queries enqueued"); - break; - } - - // Similar to handle process - // get the list of workers and start them - TList *workerList = (fProof->UseDynamicStartup()) ? new TList : (TList *)0; - Int_t pc = 0; - EQueryAction retVal = GetWorkers(workerList, pc, kTRUE); - - if (retVal == TProofServ::kQueryOK) { - Int_t ret = 0; - if (workerList && (ret = fProof->AddWorkers(workerList)) < 0) { - Error("HandleSocketInput", "adding a list of worker nodes returned: %d", ret); - } else { - ProcessNext(pslb); - // Set idle - SetIdle(kTRUE); - // Signal the client that we are idle - TMessage m(kPROOF_SETIDLE); - Bool_t waiting = (WaitingQueries() > 0) ? kTRUE : kFALSE; - m << waiting; - fSocket->Send(m); - } - } else { - if (retVal == TProofServ::kQueryStop) { - Error("HandleSocketInput", "error getting list of worker nodes"); - } else if (retVal != TProofServ::kQueryEnqueued) { - Warning("HandleSocketInput", "query was re-queued!"); - } else { - Error("HandleSocketInput", "unexpected answer: %d", retVal); - break; - } - } - - } - break; - - case kPROOF_GOASYNC: - { // The client requested to switch to asynchronous mode: - // communicate the sequential number of the running query for later - // identification, if any - if (!IsIdle() && fPlayer) { - // Get query currently being processed - TProofQueryResult *pq = (TProofQueryResult *) fPlayer->GetCurrentQuery(); - TMessage m(kPROOF_QUERYSUBMITTED); - m << pq->GetSeqNum() << kFALSE; - fSocket->Send(m); - } else { - // Idle or undefined: nothing to do; ignore - SendAsynMessage("Processing request to go asynchronous:" - " idle or undefined player - ignoring"); - } - } - break; - - case kPROOF_ECHO: - { // Echo request: an object has been sent along. If the object is a - // string, it is simply echoed back to the client from the master - // and each worker. Elsewhere, the output of TObject::Print() is - // sent. Received object is disposed after usage. - - TObject *obj = mess->ReadObject(0x0); // class type ignored - - if (IsMaster()) { - // We are on master - // dbTODO: forward on dynamic startup when wrks are up - if (IsParallel() && fProof && !fProof->UseDynamicStartup()) { - fProof->Echo(obj); // forward to lower layer - } - } - - TMessage rmsg(kPROOF_MESSAGE); - TString smsg; - - if (obj->InheritsFrom(TObjString::Class())) { - // It's a string: echo it - smsg.Form("Echo response from %s:%s: %s", - gSystem->HostName(), GetOrdinal(), - ((TObjString *)obj)->String().Data()); - } - else { - // Not a string: collect Print() output and send it - - // Output to tempfile - TString tmpfn = "echo-out-"; - FILE *tf = gSystem->TempFileName(tmpfn, fDataDir); - if (!tf || (gSystem->RedirectOutput(tmpfn.Data()) == -1)) { - Error("HandleSocketInput", "Can't redirect output"); - if (tf) { - fclose(tf); - gSystem->Unlink(tmpfn); - } - rc = -1; - delete obj; - break; - } - //cout << obj->ClassName() << endl; - obj->Print(); - gSystem->RedirectOutput(0x0); // restore - fclose(tf); - - // Read file back and send it via message - smsg.Form("*** Echo response from %s:%s ***\n", - gSystem->HostName(), GetOrdinal()); - TMacro *fr = new TMacro(); - fr->ReadFile(tmpfn); - TIter nextLine(fr->GetListOfLines()); - TObjString *line; - while (( line = (TObjString *)nextLine() )) { - smsg.Append( line->String() ); - } - - // Close the reader (TMacro) and remove file - delete fr; - gSystem->Unlink(tmpfn); - } - - // Send message and dispose object - rmsg << smsg; - GetSocket()->Send(rmsg); - delete obj; - } - break; - - default: - Error("HandleSocketInput", "unknown command %d", what); - rc = -2; - break; - } - - fRealTime += (Float_t)timer.RealTime(); - fCpuTime += (Float_t)timer.CpuTime(); - - if (!(slb.IsNull()) || fgLogToSysLog > 1) { - TString s; - s.Form("%s %d %.3f %.3f %s", fgSysLogEntity.Data(), - what, timer.RealTime(), timer.CpuTime(), slb.Data()); - gSystem->Syslog(kLogNotice, s.Data()); - } - - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Accept and merge results from a set of workers - -Bool_t TProofServ::AcceptResults(Int_t connections, TVirtualProofPlayer *mergerPlayer) -{ - TMessage *mess = new TMessage(); - Int_t mergedWorkers = 0; - - PDB(kSubmerger, 1) Info("AcceptResults", "enter"); - - // Overall result of this procedure - Bool_t result = kTRUE; - - fMergingMonitor = new TMonitor(); - fMergingMonitor->Add(fMergingSocket); - - Int_t numworkers = 0; - while (fMergingMonitor->GetActive() > 0 && mergedWorkers < connections) { - - TSocket *s = fMergingMonitor->Select(); - if (!s) { - Info("AcceptResults", "interrupt!"); - result = kFALSE; - break; - } - - if (s == fMergingSocket) { - // New incoming connection - TSocket *sw = fMergingSocket->Accept(); - if (sw && sw != (TSocket *)(-1)) { - fMergingMonitor->Add(sw); - - PDB(kSubmerger, 2) - Info("AcceptResults", "connection from a worker accepted on merger %s ", - fOrdinal.Data()); - // All assigned workers are connected - if (++numworkers >= connections) - fMergingMonitor->Remove(fMergingSocket); - } else { - PDB(kSubmerger, 1) - Info("AcceptResults", "spurious signal found of merging socket"); - } - } else { - if (s->Recv(mess) < 0) { - Error("AcceptResults", "problems receiving message"); - continue; - } - PDB(kSubmerger, 2) - Info("AcceptResults", "message received: %d ", (mess ? mess->What() : 0)); - if (!mess) { - Error("AcceptResults", "message received: %p ", mess); - continue; - } - Int_t type = 0; - - // Read output objec(s) from the received message - while ((mess->BufferSize() > mess->Length())) { - (*mess) >> type; - - PDB(kSubmerger, 2) Info("AcceptResults", " type %d ", type); - if (type == 2) { - mergedWorkers++; - PDB(kSubmerger, 2) - Info("AcceptResults", - "a new worker has been mergerd. Total merged workers: %d", - mergedWorkers); - } - TObject *o = mess->ReadObject(TObject::Class()); - if (mergerPlayer->AddOutputObject(o) == 1) { - // Remove the object if it has been merged - PDB(kSubmerger, 2) Info("AcceptResults", "removing %p (has been merged)", o); - SafeDelete(o); - } else - PDB(kSubmerger, 2) Info("AcceptResults", "%p not merged yet", o); - } - } - } - fMergingMonitor->DeActivateAll(); - - TList* sockets = fMergingMonitor->GetListOfDeActives(); - Int_t size = sockets->GetSize(); - for (Int_t i =0; i< size; ++i){ - ((TSocket*)(sockets->At(i)))->Close(); - PDB(kSubmerger, 2) Info("AcceptResults", "closing socket"); - delete ((TSocket*)(sockets->At(i))); - } - delete sockets; - - fMergingMonitor->RemoveAll(); - SafeDelete(fMergingMonitor); - - PDB(kSubmerger, 2) Info("AcceptResults", "exit: %d", result); - return result; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle Out-Of-Band data sent by the master or client. - -void TProofServ::HandleUrgentData() -{ - char oob_byte; - Int_t n, nch, wasted = 0; - - const Int_t kBufSize = 1024; - char waste[kBufSize]; - - // Real-time notification of messages - TProofServLogHandlerGuard hg(fLogFile, fSocket, "", fRealTimeLog); - - PDB(kGlobal, 5) - Info("HandleUrgentData", "handling oob..."); - - // Receive the OOB byte - while ((n = fSocket->RecvRaw(&oob_byte, 1, kOob)) < 0) { - if (n == -2) { // EWOULDBLOCK - // - // The OOB data has not yet arrived: flush the input stream - // - // In some systems (Solaris) regular recv() does not return upon - // receipt of the oob byte, which makes the below call to recv() - // block indefinitely if there are no other data in the queue. - // FIONREAD ioctl can be used to check if there are actually any - // data to be flushed. If not, wait for a while for the oob byte - // to arrive and try to read it again. - // - fSocket->GetOption(kBytesToRead, nch); - if (nch == 0) { - gSystem->Sleep(1000); - continue; - } - - if (nch > kBufSize) nch = kBufSize; - n = fSocket->RecvRaw(waste, nch); - if (n <= 0) { - Error("HandleUrgentData", "error receiving waste"); - break; - } - wasted = 1; - } else { - Error("HandleUrgentData", "error receiving OOB"); - return; - } - } - - PDB(kGlobal, 5) - Info("HandleUrgentData", "got OOB byte: %d\n", oob_byte); - - if (fProof) fProof->SetActive(); - - switch (oob_byte) { - - case TProof::kHardInterrupt: - Info("HandleUrgentData", "*** Hard Interrupt"); - - // If master server, propagate interrupt to slaves - if (IsMaster()) - fProof->Interrupt(TProof::kHardInterrupt); - - // Flush input socket - while (1) { - Int_t atmark; - - fSocket->GetOption(kAtMark, atmark); - - if (atmark) { - // Send the OOB byte back so that the client knows where - // to stop flushing its input stream of obsolete messages - n = fSocket->SendRaw(&oob_byte, 1, kOob); - if (n <= 0) - Error("HandleUrgentData", "error sending OOB"); - break; - } - - // find out number of bytes to read before atmark - fSocket->GetOption(kBytesToRead, nch); - if (nch == 0) { - gSystem->Sleep(1000); - continue; - } - - if (nch > kBufSize) nch = kBufSize; - n = fSocket->RecvRaw(waste, nch); - if (n <= 0) { - Error("HandleUrgentData", "error receiving waste (2)"); - break; - } - } - - SendLogFile(); - - break; - - case TProof::kSoftInterrupt: - Info("HandleUrgentData", "Soft Interrupt"); - - // If master server, propagate interrupt to slaves - if (IsMaster()) - fProof->Interrupt(TProof::kSoftInterrupt); - - if (wasted) { - Error("HandleUrgentData", "soft interrupt flushed stream"); - break; - } - - Interrupt(); - - SendLogFile(); - - break; - - case TProof::kShutdownInterrupt: - Info("HandleUrgentData", "Shutdown Interrupt"); - - // If master server, propagate interrupt to slaves - if (IsMaster()) - fProof->Interrupt(TProof::kShutdownInterrupt); - - Terminate(0); - - break; - - default: - Error("HandleUrgentData", "unexpected OOB byte"); - break; - } - - if (fProof) fProof->SetActive(kFALSE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Called when the client is not alive anymore (i.e. when kKeepAlive -/// has failed). - -void TProofServ::HandleSigPipe() -{ - // Real-time notification of messages - TProofServLogHandlerGuard hg(fLogFile, fSocket, "", fRealTimeLog); - - if (IsMaster()) { - // Check if we are here because client is closed. Try to ping client, - // if that works it we are here because some slave died - if (fSocket->Send(kPROOF_PING | kMESS_ACK) < 0) { - Info("HandleSigPipe", "keepAlive probe failed"); - // Tell slaves we are going to close since there is no client anymore - - fProof->SetActive(); - fProof->Interrupt(TProof::kShutdownInterrupt); - fProof->SetActive(kFALSE); - Terminate(0); - } - } else { - Info("HandleSigPipe", "keepAlive probe failed"); - Terminate(0); // will not return from here.... - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// True if in parallel mode. - -Bool_t TProofServ::IsParallel() const -{ - if (IsMaster() && fProof) - return fProof->IsParallel() || fProof->UseDynamicStartup() ; - - // false in case we are a slave - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print status of slave server. - -void TProofServ::Print(Option_t *option) const -{ - if (IsMaster() && fProof) - fProof->Print(option); - else - Printf("This is worker %s", gSystem->HostName()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Redirect stdout to a log file. This log file will be flushed to the -/// client or master after each command. - -void TProofServ::RedirectOutput(const char *dir, const char *mode) -{ - char logfile[512]; - - TString sdir = (dir && strlen(dir) > 0) ? dir : fSessionDir.Data(); - if (IsMaster()) { - snprintf(logfile, 512, "%s/master-%s.log", sdir.Data(), fOrdinal.Data()); - } else { - snprintf(logfile, 512, "%s/worker-%s.log", sdir.Data(), fOrdinal.Data()); - } - - if ((freopen(logfile, mode, stdout)) == 0) - SysError("RedirectOutput", "could not freopen stdout (%s)", logfile); - - if ((dup2(fileno(stdout), fileno(stderr))) < 0) - SysError("RedirectOutput", "could not redirect stderr"); - - if ((fLogFile = fopen(logfile, "r")) == 0) - SysError("RedirectOutput", "could not open logfile '%s'", logfile); - - // from this point on stdout and stderr are properly redirected - if (fProtocol < 4 && fWorkDir != TString::Format("~/%s", kPROOF_WorkDir)) { - Warning("RedirectOutput", "no way to tell master (or client) where" - " to upload packages"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset PROOF environment to be ready for execution of next command. - -void TProofServ::Reset(const char *dir) -{ - // First go to new directory. Check first that we got a reasonable path; - // in PROOF-Lite it may not be the case - TString dd(dir); - if (!dd.BeginsWith("proofserv")) { - Int_t ic = dd.Index(":"); - if (ic != kNPOS) - dd.Replace(0, ic, "proofserv"); - } - gDirectory->cd(dd.Data()); - - // Clear interpreter environment. - gROOT->Reset(); - - // Make sure current directory is empty (don't delete anything when - // we happen to be in the ROOT memory only directory!?) - if (gDirectory != gROOT) { - gDirectory->Delete(); - } - - if (IsMaster()) fProof->SendCurrentState(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Receive a file, either sent by a client or a master server. -/// If bin is true it is a binary file, other wise it is an ASCII -/// file and we need to check for Windows \\r tokens. Returns -1 in -/// case of error, 0 otherwise. - -Int_t TProofServ::ReceiveFile(const char *file, Bool_t bin, Long64_t size) -{ - if (size <= 0) return 0; - - // open file, overwrite already existing file - Int_t fd = open(file, O_CREAT | O_TRUNC | O_WRONLY, 0600); - if (fd < 0) { - SysError("ReceiveFile", "error opening file %s", file); - return -1; - } - - const Int_t kMAXBUF = 16384; //32768 //16384 //65536; - char buf[kMAXBUF], cpy[kMAXBUF]; - - Int_t left, r; - Long64_t filesize = 0; - - while (filesize < size) { - left = Int_t(size - filesize); - if (left > kMAXBUF) - left = kMAXBUF; - r = fSocket->RecvRaw(&buf, left); - if (r > 0) { - char *p = buf; - - filesize += r; - while (r) { - Int_t w; - - if (!bin) { - Int_t k = 0, i = 0, j = 0; - char *q; - while (i < r) { - if (p[i] == '\r') { - i++; - k++; - } - cpy[j++] = buf[i++]; - } - q = cpy; - r -= k; - w = write(fd, q, r); - } else { - w = write(fd, p, r); - } - - if (w < 0) { - SysError("ReceiveFile", "error writing to file %s", file); - close(fd); - return -1; - } - r -= w; - p += w; - } - } else if (r < 0) { - Error("ReceiveFile", "error during receiving file %s", file); - close(fd); - return -1; - } - } - - close(fd); - - if (chmod(file, 0644) != 0) - Warning("ReceiveFile", "error setting mode 0644 on file %s", file); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Main server eventloop. - -void TProofServ::Run(Bool_t retrn) -{ - // Setup the server - if (CreateServer() == 0) { - - // Run the main event loop - TApplication::Run(retrn); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send log file to master. -/// If start > -1 send only bytes in the range from start to end, -/// if end <= start send everything from start. - -void TProofServ::SendLogFile(Int_t status, Int_t start, Int_t end) -{ - // Determine the number of bytes left to be read from the log file. - fflush(stdout); - - // On workers we do not send the logs to masters (to avoid duplication of - // text) unless asked explicitly, e.g. after an Exec(...) request. - if (!IsMaster()) { - if (!fSendLogToMaster) { - FlushLogFile(); - } else { - // Decide case by case - LogToMaster(kFALSE); - } - } - - off_t ltot=0, lnow=0; - Int_t left = -1; - Bool_t adhoc = kFALSE; - - if (fLogFileDes > -1) { - ltot = lseek(fileno(stdout), (off_t) 0, SEEK_END); - lnow = lseek(fLogFileDes, (off_t) 0, SEEK_CUR); - - if (ltot >= 0 && lnow >= 0) { - if (start > -1) { - lseek(fLogFileDes, (off_t) start, SEEK_SET); - if (end <= start || end > ltot) - end = ltot; - left = (Int_t)(end - start); - if (end < ltot) - left++; - adhoc = kTRUE; - } else { - left = (Int_t)(ltot - lnow); - } - } - } - - if (left > 0) { - if (fSocket->Send(left, kPROOF_LOGFILE) < 0) { - SysError("SendLogFile", "error sending kPROOF_LOGFILE"); - return; - } - - const Int_t kMAXBUF = 32768; //16384 //65536; - char buf[kMAXBUF]; - Int_t wanted = (left > kMAXBUF) ? kMAXBUF : left; - Int_t len; - do { - while ((len = read(fLogFileDes, buf, wanted)) < 0 && - TSystem::GetErrno() == EINTR) - TSystem::ResetErrno(); - - if (len < 0) { - SysError("SendLogFile", "error reading log file"); - break; - } - - if (end == ltot && len == wanted) - buf[len-1] = '\n'; - - if (fSocket->SendRaw(buf, len) < 0) { - SysError("SendLogFile", "error sending log file"); - break; - } - - // Update counters - left -= len; - wanted = (left > kMAXBUF) ? kMAXBUF : left; - - } while (len > 0 && left > 0); - } - - // Restore initial position if partial send - if (adhoc && lnow >=0 ) - lseek(fLogFileDes, lnow, SEEK_SET); - - TMessage mess(kPROOF_LOGDONE); - if (IsMaster()) - mess << status << (fProof ? fProof->GetParallel() : 0); - else - mess << status << (Int_t) 1; - - if (fSocket->Send(mess) < 0) { - SysError("SendLogFile", "error sending kPROOF_LOGDONE"); - return; - } - - PDB(kGlobal, 1) Info("SendLogFile", "kPROOF_LOGDONE sent"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send statistics of slave server to master or client. - -void TProofServ::SendStatistics() -{ - Long64_t bytesread = TFile::GetFileBytesRead(); - Float_t cputime = fCpuTime, realtime = fRealTime; - if (IsMaster()) { - bytesread = fProof->GetBytesRead(); - cputime = fProof->GetCpuTime(); - } - - TMessage mess(kPROOF_GETSTATS); - TString workdir = gSystem->WorkingDirectory(); // expect TString on other side - mess << bytesread << realtime << cputime << workdir; - if (fProtocol >= 4) mess << TString(gProofServ->GetWorkDir()); - mess << TString(gProofServ->GetImage()); - fSocket->Send(mess); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send number of parallel nodes to master or client. - -void TProofServ::SendParallel(Bool_t async) -{ - Int_t nparallel = 0; - if (IsMaster()) { - PDB(kGlobal, 2) - Info("SendParallel", "Will invoke AskParallel()"); - fProof->AskParallel(); - PDB(kGlobal, 2) - Info("SendParallel", "Will invoke GetParallel()"); - nparallel = fProof->GetParallel(); - } else { - nparallel = 1; - } - - TMessage mess(kPROOF_GETPARALLEL); - mess << nparallel << async; - fSocket->Send(mess); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print the ProofServ logo on standard output. -/// Return 0 on success, -1 on failure - -Int_t TProofServ::Setup() -{ - char str[512]; - - if (IsMaster()) { - snprintf(str, 512, "**** Welcome to the PROOF server @ %s ****", gSystem->HostName()); - } else { - snprintf(str, 512, "**** PROOF slave server @ %s started ****", gSystem->HostName()); - } - - if (fSocket->Send(str) != 1+static_cast(strlen(str))) { - Error("Setup", "failed to send proof server startup message"); - return -1; - } - - // exchange protocol level between client and master and between - // master and slave - Int_t what; - if (fSocket->Recv(fProtocol, what) != 2*sizeof(Int_t)) { - Error("Setup", "failed to receive remote proof protocol"); - return -1; - } - if (fSocket->Send(kPROOF_Protocol, kROOTD_PROTOCOL) != 2*sizeof(Int_t)) { - Error("Setup", "failed to send local proof protocol"); - return -1; - } - - // If old version, setup authentication related stuff - if (fProtocol < 5) { - TString wconf; - if (OldAuthSetup(wconf) != 0) { - Error("Setup", "OldAuthSetup: failed to setup authentication"); - return -1; - } - if (IsMaster()) { - fConfFile = wconf; - fWorkDir.Form("~/%s", kPROOF_WorkDir); - } else { - if (fProtocol < 4) { - fWorkDir.Form("~/%s", kPROOF_WorkDir); - } else { - fWorkDir = wconf; - if (fWorkDir.IsNull()) fWorkDir.Form("~/%s", kPROOF_WorkDir); - } - } - } else { - - // Receive some useful information - TMessage *mess; - if ((fSocket->Recv(mess) <= 0) || !mess) { - Error("Setup", "failed to receive ordinal and config info"); - return -1; - } - if (IsMaster()) { - (*mess) >> fUser >> fOrdinal >> fConfFile; - fWorkDir = gEnv->GetValue("ProofServ.Sandbox", TString::Format("~/%s", kPROOF_WorkDir)); - } else { - (*mess) >> fUser >> fOrdinal >> fWorkDir; - if (fWorkDir.IsNull()) - fWorkDir = gEnv->GetValue("ProofServ.Sandbox", TString::Format("~/%s", kPROOF_WorkDir)); - } - // Set the correct prefix - if (fOrdinal != "-1") - fPrefix += fOrdinal; - TProofServLogHandler::SetDefaultPrefix(fPrefix); - delete mess; - } - - if (IsMaster()) { - - // strip off any prooftype directives - TString conffile = fConfFile; - conffile.Remove(0, 1 + conffile.Index(":")); - - // parse config file to find working directory - TProofResourcesStatic resources(fConfDir, conffile); - if (resources.IsValid()) { - if (resources.GetMaster()) { - TString tmpWorkDir = resources.GetMaster()->GetWorkDir(); - if (tmpWorkDir != "") - fWorkDir = tmpWorkDir; - } - } else { - Info("Setup", "invalid config file %s (missing or unreadable", - resources.GetFileName().Data()); - } - } - - // Set $HOME and $PATH. The HOME directory was already set to the - // user's home directory by proofd. - gSystem->Setenv("HOME", gSystem->HomeDirectory()); - - // Add user name in case of non default workdir - if (fWorkDir.BeginsWith("/") && - !fWorkDir.BeginsWith(gSystem->HomeDirectory())) { - if (!fWorkDir.EndsWith("/")) - fWorkDir += "/"; - UserGroup_t *u = gSystem->GetUserInfo(); - if (u) { - fWorkDir += u->fUser; - delete u; - } - } - - // Goto to the main PROOF working directory - gSystem->ExpandPathName(fWorkDir); - if (gProofDebugLevel > 0) - Info("Setup", "working directory set to %s", fWorkDir.Data()); - - // host first name - TString host = gSystem->HostName(); - if (host.Index(".") != kNPOS) - host.Remove(host.Index(".")); - - // Session tag - fSessionTag.Form("%s-%s-%ld-%d", fOrdinal.Data(), host.Data(), - (Long_t)TTimeStamp().GetSec(),gSystem->GetPid()); - fTopSessionTag = fSessionTag; - - // create session directory and make it the working directory - fSessionDir = fWorkDir; - if (IsMaster()) - fSessionDir += "/master-"; - else - fSessionDir += "/slave-"; - fSessionDir += fSessionTag; - - // Common setup - if (SetupCommon() != 0) { - Error("Setup", "common setup failed"); - return -1; - } - - // Incoming OOB should generate a SIGURG - fSocket->SetOption(kProcessGroup, gSystem->GetPid()); - - // Send packets off immediately to reduce latency - fSocket->SetOption(kNoDelay, 1); - - // Check every two hours if client is still alive - fSocket->SetOption(kKeepAlive, 1); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Common part (between TProofServ and TXProofServ) of the setup phase. -/// Return 0 on success, -1 on error - -Int_t TProofServ::SetupCommon() -{ - // deny write access for group and world - gSystem->Umask(022); - -#ifdef R__UNIX - // Add bindir to PATH - TString path(gSystem->Getenv("PATH")); - TString bindir(TROOT::GetBinDir()); - // Augment PATH, if required - // ^, , ^, - TString paths = gEnv->GetValue("ProofServ.BinPaths", ""); - if (paths.Length() > 0) { - Int_t icomp = 0; - if (paths.Contains("^")) - icomp = 1; - else if (paths.Contains("")) - icomp = -1; - if (icomp != 0) { -# ifdef COMPILER - TString compiler = COMPILER; - if (compiler.Index("is ") != kNPOS) - compiler.Remove(0, compiler.Index("is ") + 3); - compiler = gSystem->GetDirName(compiler); - if (icomp == 1) { - if (!bindir.IsNull()) bindir += ":"; - bindir += compiler; - } else if (icomp == -1) { - if (!path.IsNull()) path += ":"; - path += compiler; - } -#endif - } - Int_t isysb = 0; - if (paths.Contains("^")) - isysb = 1; - else if (paths.Contains("")) - isysb = -1; - if (isysb != 0) { - if (isysb == 1) { - if (!bindir.IsNull()) bindir += ":"; - bindir += "/bin:/usr/bin:/usr/local/bin"; - } else if (isysb == -1) { - if (!path.IsNull()) path += ":"; - path += "/bin:/usr/bin:/usr/local/bin"; - } - } - } - // Final insert - if (!bindir.IsNull()) bindir += ":"; - path.Insert(0, bindir); - gSystem->Setenv("PATH", path); -#endif - - if (gSystem->AccessPathName(fWorkDir)) { - gSystem->mkdir(fWorkDir, kTRUE); - if (!gSystem->ChangeDirectory(fWorkDir)) { - Error("SetupCommon", "can not change to PROOF directory %s", - fWorkDir.Data()); - return -1; - } - } else { - if (!gSystem->ChangeDirectory(fWorkDir)) { - gSystem->Unlink(fWorkDir); - gSystem->mkdir(fWorkDir, kTRUE); - if (!gSystem->ChangeDirectory(fWorkDir)) { - Error("SetupCommon", "can not change to PROOF directory %s", - fWorkDir.Data()); - return -1; - } - } - } - - // Set group - fGroup = gEnv->GetValue("ProofServ.ProofGroup", "default"); - - // Check and make sure "cache" directory exists - fCacheDir = gEnv->GetValue("ProofServ.CacheDir", - TString::Format("%s/%s", fWorkDir.Data(), kPROOF_CacheDir)); - ResolveKeywords(fCacheDir); - if (gSystem->AccessPathName(fCacheDir)) - gSystem->mkdir(fCacheDir, kTRUE); - if (gProofDebugLevel > 0) - Info("SetupCommon", "cache directory set to %s", fCacheDir.Data()); - fCacheLock = - new TProofLockPath(TString::Format("%s/%s%s", - gSystem->TempDirectory(), kPROOF_CacheLockFile, - TString(fCacheDir).ReplaceAll("/","%").Data())); - // Make also sure the cache path is in the macro path - TProof::AssertMacroPath(TString::Format("%s/.", fCacheDir.Data())); - - // Check and make sure "packages" directory exists - TString packdir = gEnv->GetValue("ProofServ.PackageDir", - TString::Format("%s/%s", fWorkDir.Data(), kPROOF_PackDir)); - ResolveKeywords(packdir); - if (gSystem->AccessPathName(packdir)) - gSystem->mkdir(packdir, kTRUE); - fPackMgr = new TPackMgr(packdir); - fPackMgr->SetLogger(SendAsynMsg); - // Notification message - TString noth; - const char *k = (IsMaster()) ? "Mst" : "Wrk"; - noth.Form("%s-%s", k, fOrdinal.Data()); - fPackMgr->SetPrefix(noth.Data()); - if (gProofDebugLevel > 0) - Info("SetupCommon", "package directory set to %s", packdir.Data()); - - // Check and make sure "data" directory exists - fDataDir = gEnv->GetValue("ProofServ.DataDir",""); - Ssiz_t isep = kNPOS; - if (fDataDir.IsNull()) { - // Use default - fDataDir.Form("%s/%s//", fWorkDir.Data(), kPROOF_DataDir); - } else if ((isep = fDataDir.Last(' ')) != kNPOS) { - fDataDirOpts = fDataDir(isep + 1, fDataDir.Length()); - fDataDir.Remove(isep); - } - ResolveKeywords(fDataDir); - if (gSystem->AccessPathName(fDataDir)) - if (gSystem->mkdir(fDataDir, kTRUE) != 0) { - Warning("SetupCommon", "problems creating path '%s' (errno: %d)", - fDataDir.Data(), TSystem::GetErrno()); - } - if (gProofDebugLevel > 0) - Info("SetupCommon", "data directory set to %s", fDataDir.Data()); - - // Check and apply possible options - TString dataDirOpts = gEnv->GetValue("ProofServ.DataDirOpts",""); - if (!dataDirOpts.IsNull()) { - // Do they apply to this server type - Bool_t doit = kTRUE; - if ((IsMaster() && !dataDirOpts.Contains("M")) || - (!IsMaster() && !dataDirOpts.Contains("W"))) doit = kFALSE; - if (doit) { - // Get the wanted mode - UInt_t m = 0755; - if (dataDirOpts.Contains("g")) m = 0775; - if (dataDirOpts.Contains("a") || dataDirOpts.Contains("o")) m = 0777; - if (gProofDebugLevel > 0) - Info("SetupCommon", "requested mode for data directories is '%o'", m); - // Loop over paths - FileStat_t st; - TString p, subp; - Int_t from = 0; - if (fDataDir.BeginsWith("/")) p = "/"; - while (fDataDir.Tokenize(subp, from, "/")) { - if (subp.IsNull()) continue; - p += subp; - if (gSystem->GetPathInfo(p, st) == 0) { - if (st.fUid == (Int_t) gSystem->GetUid() && st.fGid == (Int_t) gSystem->GetGid()) { - if (gSystem->Chmod(p.Data(), m) != 0) { - Warning("SetupCommon", "problems setting mode '%o' on path '%s' (errno: %d)", - m, p.Data(), TSystem::GetErrno()); - break; - } - } - p += "/"; - } else { - Warning("SetupCommon", "problems stat-ing path '%s' (errno: %d; datadir: %s)", - p.Data(), TSystem::GetErrno(), fDataDir.Data()); - break; - } - } - } - } - - // List of directories where to look for global packages - TString globpack = gEnv->GetValue("Proof.GlobalPackageDirs",""); - - ResolveKeywords(globpack); - Int_t nglb = TPackMgr::RegisterGlobalPath(globpack); - Info("SetupCommon", " %d global package directories registered", nglb); - FlushLogFile(); - - // Check the session dir - if (fSessionDir != gSystem->WorkingDirectory()) { - ResolveKeywords(fSessionDir); - if (gSystem->AccessPathName(fSessionDir)) - gSystem->mkdir(fSessionDir, kTRUE); - if (!gSystem->ChangeDirectory(fSessionDir)) { - Error("SetupCommon", "can not change to working directory '%s'", - fSessionDir.Data()); - return -1; - } - } - gSystem->Setenv("PROOF_SANDBOX", fSessionDir); - if (gProofDebugLevel > 0) - Info("SetupCommon", "session dir is '%s'", fSessionDir.Data()); - - // On masters, check and make sure that "queries" and "datasets" - // directories exist - if (IsMaster()) { - - // Make sure that the 'queries' dir exist - fQueryDir = fWorkDir; - fQueryDir += TString("/") + kPROOF_QueryDir; - ResolveKeywords(fQueryDir); - if (gSystem->AccessPathName(fQueryDir)) - gSystem->mkdir(fQueryDir, kTRUE); - fQueryDir += TString("/session-") + fTopSessionTag; - if (gSystem->AccessPathName(fQueryDir)) - gSystem->mkdir(fQueryDir, kTRUE); - if (gProofDebugLevel > 0) - Info("SetupCommon", "queries dir is %s", fQueryDir.Data()); - - // Create 'queries' locker instance and lock it - fQueryLock = new TProofLockPath(TString::Format("%s/%s%s-%s", - gSystem->TempDirectory(), - kPROOF_QueryLockFile, fSessionTag.Data(), - TString(fQueryDir).ReplaceAll("/","%").Data())); - fQueryLock->Lock(); - // Create the query manager - fQMgr = new TQueryResultManager(fQueryDir, fSessionTag, fSessionDir, - fQueryLock, 0); - } - - // Server image - fImage = gEnv->GetValue("ProofServ.Image", ""); - - // Get the group priority - if (IsMaster()) { - // Send session tag to client - TMessage m(kPROOF_SESSIONTAG); - m << fTopSessionTag << fGroup << fUser; - fSocket->Send(m); - // Group priority - fGroupPriority = GetPriority(); - // Dataset manager instance via plug-in - TPluginHandler *h = 0; - TString dsms = gEnv->GetValue("Proof.DataSetManager", ""); - if (!dsms.IsNull()) { - TString dsm; - Int_t from = 0; - while (dsms.Tokenize(dsm, from, ",")) { - if (fDataSetManager && !fDataSetManager->TestBit(TObject::kInvalidObject)) { - Warning("SetupCommon", "a valid dataset manager already initialized"); - Warning("SetupCommon", "support for multiple managers not yet available"); - break; - } - // Get plugin manager to load the appropriate TDataSetManager - if (gROOT->GetPluginManager()) { - // Find the appropriate handler - h = gROOT->GetPluginManager()->FindHandler("TDataSetManager", dsm); - if (h && h->LoadPlugin() != -1) { - // make instance of the dataset manager - fDataSetManager = - reinterpret_cast(h->ExecPlugin(3, fGroup.Data(), - fUser.Data(), dsm.Data())); - } - } - } - // Check the result of the dataset manager initialization - if (fDataSetManager && fDataSetManager->TestBit(TObject::kInvalidObject)) { - Warning("SetupCommon", "dataset manager plug-in initialization failed"); - SendAsynMessage("TXProofServ::SetupCommon: dataset manager plug-in initialization failed"); - SafeDelete(fDataSetManager); - } - } else { - // Initialize the default dataset manager - TString opts("Av:"); - TString dsetdir = gEnv->GetValue("ProofServ.DataSetDir", ""); - if (dsetdir.IsNull()) { - // Use the default in the sandbox - dsetdir.Form("%s/%s", fWorkDir.Data(), kPROOF_DataSetDir); - if (gSystem->AccessPathName(fDataSetDir)) - gSystem->MakeDirectory(fDataSetDir); - opts += "Sb:"; - } - // Find the appropriate handler - if (!h) { - h = gROOT->GetPluginManager()->FindHandler("TDataSetManager", "file"); - if (h && h->LoadPlugin() == -1) h = 0; - } - if (h) { - // make instance of the dataset manager - TString oo = TString::Format("dir:%s opt:%s", dsetdir.Data(), opts.Data()); - fDataSetManager = reinterpret_cast(h->ExecPlugin(3, - fGroup.Data(), fUser.Data(), oo.Data())); - } - if (fDataSetManager && fDataSetManager->TestBit(TObject::kInvalidObject)) { - Warning("SetupCommon", "default dataset manager plug-in initialization failed"); - SafeDelete(fDataSetManager); - } - } - // Dataset manager for staging requests - TString dsReqCfg = gEnv->GetValue("Proof.DataSetStagingRequests", ""); - if (!dsReqCfg.IsNull()) { - TPMERegexp reReqDir("(^| )(dir:)?([^ ]+)( |$)"); - - if (reReqDir.Match(dsReqCfg) == 5) { - TString dsDirFmt; - dsDirFmt.Form("dir:%s perms:open", reReqDir[3].Data()); - fDataSetStgRepo = new TDataSetManagerFile("_stage_", "_stage_", - dsDirFmt); - if (fDataSetStgRepo && - fDataSetStgRepo->TestBit(TObject::kInvalidObject)) { - Warning("SetupCommon", - "failed init of dataset staging requests repository"); - SafeDelete(fDataSetStgRepo); - } - } else { - Warning("SetupCommon", - "specify, with [dir:], a valid path for staging requests"); - } - } else if (gProofDebugLevel > 0) { - Warning("SetupCommon", "no repository for staging requests available"); - } - } - - // Quotas - TString quotas = gEnv->GetValue(TString::Format("ProofServ.UserQuotas.%s", fUser.Data()),""); - if (quotas.IsNull()) - quotas = gEnv->GetValue(TString::Format("ProofServ.UserQuotasByGroup.%s", fGroup.Data()),""); - if (quotas.IsNull()) - quotas = gEnv->GetValue("ProofServ.UserQuotas", ""); - if (!quotas.IsNull()) { - // Parse it; format ("maxquerykept=10 hwmsz=800m maxsz=1g") - TString tok; - Ssiz_t from = 0; - while (quotas.Tokenize(tok, from, " ")) { - // Set max number of query results to keep - if (tok.BeginsWith("maxquerykept=")) { - tok.ReplaceAll("maxquerykept=",""); - if (tok.IsDigit()) - fMaxQueries = tok.Atoi(); - else - Info("SetupCommon", - "parsing 'maxquerykept' :ignoring token %s : not a digit", tok.Data()); - } - // Set High-Water-Mark or max on the sandbox size - const char *ksz[2] = {"hwmsz=", "maxsz="}; - for (Int_t j = 0; j < 2; j++) { - if (tok.BeginsWith(ksz[j])) { - tok.ReplaceAll(ksz[j],""); - Long64_t fact = -1; - if (!tok.IsDigit()) { - // Parse (k, m, g) - tok.ToLower(); - const char *s[3] = {"k", "m", "g"}; - Int_t i = 0, ki = 1024; - while (fact < 0) { - if (tok.EndsWith(s[i++])) - fact = ki; - else - ki *= 1024; - } - tok.Remove(tok.Length()-1); - } - if (tok.IsDigit()) { - if (j == 0) - fHWMBoxSize = (fact > 0) ? tok.Atoi() * fact : tok.Atoi(); - else - fMaxBoxSize = (fact > 0) ? tok.Atoi() * fact : tok.Atoi(); - } else { - TString ssz(ksz[j], strlen(ksz[j])-1); - Info("SetupCommon", "parsing '%s' : ignoring token %s", ssz.Data(), tok.Data()); - } - } - } - } - } - - // Apply quotas, if any - if (IsMaster() && fQMgr) - if (fQMgr->ApplyMaxQueries(fMaxQueries) != 0) - Warning("SetupCommon", "problems applying fMaxQueries"); - - // Send "ROOTversion|ArchCompiler" flag - if (fProtocol > 12) { - TString vac = gROOT->GetVersion(); - vac += TString::Format(":%s", gROOT->GetGitCommit()); - TString rtag = gEnv->GetValue("ProofServ.RootVersionTag", ""); - if (rtag.Length() > 0) - vac += TString::Format(":%s", rtag.Data()); - vac += TString::Format("|%s-%s",gSystem->GetBuildArch(), gSystem->GetBuildCompilerVersion()); - TMessage m(kPROOF_VERSARCHCOMP); - m << vac; - fSocket->Send(m); - } - - // Set user vars in TProof - TString all_vars(gSystem->Getenv("PROOF_ALLVARS")); - TString name; - Int_t from = 0; - while (all_vars.Tokenize(name, from, ",")) { - if (!name.IsNull()) { - TString value = gSystem->Getenv(name); - TProof::AddEnvVar(name, value); - } - } - - if (fgLogToSysLog > 0) { - // Set the syslog entity (all the information is available now) - if (!(fUser.IsNull()) && !(fGroup.IsNull())) { - fgSysLogEntity.Form("%s:%s", fUser.Data(), fGroup.Data()); - } else if (!(fUser.IsNull()) && fGroup.IsNull()) { - fgSysLogEntity.Form("%s:default", fUser.Data()); - } else if (fUser.IsNull() && !(fGroup.IsNull())) { - fgSysLogEntity.Form("undef:%s", fGroup.Data()); - } - // Log the beginning of this session - TString s; - s.Form("%s 0 %.3f %.3f", fgSysLogEntity.Data(), fRealTime, fCpuTime); - gSystem->Syslog(kLogNotice, s.Data()); - } - - if (gProofDebugLevel > 0) - Info("SetupCommon", "successfully completed"); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Terminate the proof server. - -void TProofServ::Terminate(Int_t status) -{ - if (fgLogToSysLog > 0) { - TString s; - s.Form("%s -1 %.3f %.3f %d", fgSysLogEntity.Data(), fRealTime, fCpuTime, status); - gSystem->Syslog(kLogNotice, s.Data()); - } - - // Notify the memory footprint - ProcInfo_t pi; - if (!gSystem->GetProcInfo(&pi)){ - Info("Terminate", "process memory footprint: %ld/%ld kB virtual, %ld/%ld kB resident ", - pi.fMemVirtual, fgVirtMemMax, pi.fMemResident, fgResMemMax); - } - - // Cleanup session directory - if (status == 0) { - // make sure we remain in a "connected" directory - gSystem->ChangeDirectory("/"); - // needed in case fSessionDir is on NFS ?! - gSystem->MakeDirectory(fSessionDir+"/.delete"); - gSystem->Exec(TString::Format("%s %s", kRM, fSessionDir.Data())); - } - - // Cleanup queries directory if empty - if (IsMaster()) { - if (!(fQMgr && fQMgr->Queries() && fQMgr->Queries()->GetSize())) { - // make sure we remain in a "connected" directory - gSystem->ChangeDirectory("/"); - // needed in case fQueryDir is on NFS ?! - gSystem->MakeDirectory(fQueryDir+"/.delete"); - gSystem->Exec(TString::Format("%s %s", kRM, fQueryDir.Data())); - // Remove lock file - if (fQueryLock) - gSystem->Unlink(fQueryLock->GetName()); - } - - // Unlock the query dir owned by this session - if (fQueryLock) - fQueryLock->Unlock(); - } - - // Cleanup data directory if empty - if (!fDataDir.IsNull() && !gSystem->AccessPathName(fDataDir, kWritePermission)) { - if (UnlinkDataDir(fDataDir)) - Info("Terminate", "data directory '%s' has been removed", fDataDir.Data()); - } - - // Remove input handler to avoid spurious signals in socket - // selection for closing activities executed upon exit() - TIter next(gSystem->GetListOfFileHandlers()); - TObject *fh = 0; - while ((fh = next())) { - TProofServInputHandler *ih = dynamic_cast(fh); - if (ih) - gSystem->RemoveFileHandler(ih); - } - - // Stop processing events - gSystem->ExitLoop(); - - // Exit() is called in pmain -} - -//////////////////////////////////////////////////////////////////////////////// -/// Scan recursively the datadir and unlink it if empty -/// Return kTRUE if it can be unlinked, kFALSE otherwise - -Bool_t TProofServ::UnlinkDataDir(const char *path) -{ - if (!path || strlen(path) <= 0) return kFALSE; - - Bool_t dorm = kTRUE; - void *dirp = gSystem->OpenDirectory(path); - if (dirp) { - TString fpath; - const char *ent = 0; - while (dorm && (ent = gSystem->GetDirEntry(dirp))) { - if (!strcmp(ent, ".") || !strcmp(ent, "..")) continue; - fpath.Form("%s/%s", path, ent); - FileStat_t st; - if (gSystem->GetPathInfo(fpath, st) == 0 && R_ISDIR(st.fMode)) { - dorm = UnlinkDataDir(fpath); - } else { - dorm = kFALSE; - } - } - // Close the directory - gSystem->FreeDirectory(dirp); - } else { - // Cannot open the directory - dorm = kFALSE; - } - - // Do remove, if required - if (dorm && gSystem->Unlink(path) != 0) - Warning("UnlinkDataDir", "data directory '%s' is empty but could not be removed", path); - // done - return dorm; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static function that returns kTRUE in case we are a PROOF server. - -Bool_t TProofServ::IsActive() -{ - return gProofServ ? kTRUE : kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static function returning pointer to global object gProofServ. -/// Mainly for use via CINT, where the gProofServ symbol might be -/// deleted from the symbol table. - -TProofServ *TProofServ::This() -{ - return gProofServ; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Setup authentication related stuff for old versions. -/// Provided for backward compatibility. - -Int_t TProofServ::OldAuthSetup(TString &conf) -{ - OldProofServAuthSetup_t oldAuthSetupHook = 0; - - if (!oldAuthSetupHook) { - // Load libraries needed for (server) authentication ... - TString authlib = "libRootAuth"; - char *p = 0; - // The generic one - if ((p = gSystem->DynamicPathName(authlib, kTRUE))) { - delete[] p; - if (gSystem->Load(authlib) == -1) { - Error("OldAuthSetup", "can't load %s",authlib.Data()); - return kFALSE; - } - } else { - Error("OldAuthSetup", "can't locate %s",authlib.Data()); - return -1; - } - // - // Locate OldProofServAuthSetup - Func_t f = gSystem->DynFindSymbol(authlib,"OldProofServAuthSetup"); - if (f) - oldAuthSetupHook = (OldProofServAuthSetup_t)(f); - else { - Error("OldAuthSetup", "can't find OldProofServAuthSetup"); - return -1; - } - } - // - // Setup - return (*oldAuthSetupHook)(fSocket, IsMaster(), fProtocol, - fUser, fOrdinal, conf); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a TProofQueryResult instance for this query. - -TProofQueryResult *TProofServ::MakeQueryResult(Long64_t nent, - const char *opt, - TList *inlist, Long64_t fst, - TDSet *dset, const char *selec, - TObject *elist) -{ - // Increment sequential number - Int_t seqnum = -1; - if (fQMgr) { - fQMgr->IncrementSeqNum(); - seqnum = fQMgr->SeqNum(); - } - - // Locally we always use the current streamer - Bool_t olds = (dset && dset->TestBit(TDSet::kWriteV3)) ? kTRUE : kFALSE; - if (olds) - dset->SetWriteV3(kFALSE); - - // Create the instance and add it to the list - TProofQueryResult *pqr = new TProofQueryResult(seqnum, opt, inlist, nent, - fst, dset, selec, elist); - // Title is the session identifier - pqr->SetTitle(gSystem->BaseName(fQueryDir)); - - // Restore old streamer info - if (olds) - dset->SetWriteV3(kTRUE); - - return pqr; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set query in running state. - -void TProofServ::SetQueryRunning(TProofQueryResult *pq) -{ - // Record current position in the log file at start - fflush(stdout); - Int_t startlog = lseek(fileno(stdout), (off_t) 0, SEEK_END); - - // Add some header to logs - Printf(" "); - Info("SetQueryRunning", "starting query: %d", pq->GetSeqNum()); - - // Build the list of loaded PAR packages - TString parlist = ""; - fPackMgr->GetEnabledPackages(parlist); - - if (fProof) { - // Set in running state - pq->SetRunning(startlog, parlist, fProof->GetParallel()); - - // Bytes and CPU at start (we will calculate the differential at end) - pq->SetProcessInfo(pq->GetEntries(), - fProof->GetCpuTime(), fProof->GetBytesRead()); - } else { - // Set in running state - pq->SetRunning(startlog, parlist, -1); - - // Bytes and CPU at start (we will calculate the differential at end) - pq->SetProcessInfo(pq->GetEntries(), float(0.), 0); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle archive request. - -void TProofServ::HandleArchive(TMessage *mess, TString *slb) -{ - PDB(kGlobal, 1) - Info("HandleArchive", "Enter"); - - TString queryref; - TString path; - (*mess) >> queryref >> path; - - if (slb) slb->Form("%s %s", queryref.Data(), path.Data()); - - // If this is a set default action just save the default - if (queryref == "Default") { - fArchivePath = path; - Info("HandleArchive", - "default path set to %s", fArchivePath.Data()); - return; - } - - Int_t qry = -1; - TString qdir; - TProofQueryResult *pqr = fQMgr ? fQMgr->LocateQuery(queryref, qry, qdir) : 0; - TProofQueryResult *pqm = pqr; - - if (path.Length() <= 0) { - if (fArchivePath.Length() <= 0) { - Info("HandleArchive", - "archive paths are not defined - do nothing"); - return; - } - if (qry > 0) { - path.Form("%s/session-%s-%d.root", - fArchivePath.Data(), fTopSessionTag.Data(), qry); - } else { - path = queryref; - path.ReplaceAll(":q","-"); - path.Insert(0, TString::Format("%s/",fArchivePath.Data())); - path += ".root"; - } - } - - // Build file name for specific query - if (!pqr || qry < 0) { - TString fout = qdir; - fout += "/query-result.root"; - - TFile *f = TFile::Open(fout,"READ"); - pqr = 0; - if (f) { - f->ReadKeys(); - TIter nxk(f->GetListOfKeys()); - TKey *k = 0; - while ((k = (TKey *)nxk())) { - if (!strcmp(k->GetClassName(), "TProofQueryResult")) { - pqr = (TProofQueryResult *) f->Get(k->GetName()); - if (pqr) - break; - } - } - f->Close(); - delete f; - } else { - Info("HandleArchive", - "file cannot be open (%s)",fout.Data()); - return; - } - } - - if (pqr) { - - PDB(kGlobal, 1) Info("HandleArchive", - "archive path for query #%d: %s", - qry, path.Data()); - TFile *farc = 0; - if (gSystem->AccessPathName(path)) - farc = TFile::Open(path,"NEW"); - else - farc = TFile::Open(path,"UPDATE"); - if (!farc || !(farc->IsOpen())) { - Info("HandleArchive", - "archive file cannot be open (%s)",path.Data()); - return; - } - farc->cd(); - - // Update query status - pqr->SetArchived(path); - if (pqm) - pqm->SetArchived(path); - - // Write to file - pqr->Write(); - - // Update temporary files too - if (qry > -1 && fQMgr) - fQMgr->SaveQuery(pqr); - - // Notify - Info("HandleArchive", - "results of query %s archived to file %s", - queryref.Data(), path.Data()); - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get a map {server-name, list-of-files} for collection 'fc' to be used in -/// TPacketizerFile. Returns a pointer to the map (ownership of the caller). -/// Or (TMap *)0 and an error message in emsg. - -TMap *TProofServ::GetDataSetNodeMap(TFileCollection *fc, TString &emsg) -{ - TMap *fcmap = 0; - emsg = ""; - - // Sanity checks - if (!fc) { - emsg.Form("file collection undefined!"); - return fcmap; - } - - // Prepare data set map - fcmap = new TMap(); - - TIter nxf(fc->GetList()); - TFileInfo *fiind = 0; - TString key; - while ((fiind = (TFileInfo *)nxf())) { - TUrl *xurl = fiind->GetCurrentUrl(); - // Find the key for this server - key.Form("%s://%s", xurl->GetProtocol(), xurl->GetHostFQDN()); - if (xurl->GetPort() > 0) - key += TString::Format(":%d", xurl->GetPort()); - // Get the map entry for this key - TPair *ent = 0; - THashList* l = 0; - if ((ent = (TPair *) fcmap->FindObject(key.Data()))) { - // Attach to the list - l = (THashList *) ent->Value(); - } else { - // Create list - l = new THashList; - l->SetOwner(kTRUE); - // Add it to the map - fcmap->Add(new TObjString(key.Data()), l); - } - // Add fileinfo with index to list - l->Add(fiind); - } - - // Done - return fcmap; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle processing request. - -void TProofServ::HandleProcess(TMessage *mess, TString *slb) -{ - PDB(kGlobal, 1) - Info("HandleProcess", "Enter"); - - // Nothing to do for slaves if we are not idle - if (!IsTopMaster() && !IsIdle()) - return; - - TDSet *dset; - TString filename, opt; - TList *input; - Long64_t nentries, first; - TEventList *evl = 0; - TEntryList *enl = 0; - Bool_t sync; - - (*mess) >> dset >> filename >> input >> opt >> nentries >> first >> evl >> sync; - // Get entry list information, if any (support started with fProtocol == 15) - if ((mess->BufferSize() > mess->Length()) && fProtocol > 14) - (*mess) >> enl; - Bool_t hasNoData = (!dset || dset->TestBit(TDSet::kEmpty)) ? kTRUE : kFALSE; - - // Priority to the entry list - TObject *elist = (enl) ? (TObject *)enl : (TObject *)evl; - if (enl && evl) - // Cannot specify both at the same time - SafeDelete(evl); - if ((!hasNoData) && elist) - dset->SetEntryList(elist); - - if (IsTopMaster()) { - - TString emsg; - // Make sure the dataset contains the information needed - if ((!hasNoData) && dset->GetListOfElements()->GetSize() == 0) { - if (TProof::AssertDataSet(dset, input, fDataSetManager, emsg) != 0) { - SendAsynMessage(TString::Format("AssertDataSet on %s: %s", - fPrefix.Data(), emsg.Data())); - Error("HandleProcess", "AssertDataSet: %s", emsg.Data()); - // To terminate collection - if (sync) SendLogFile(); - return; - } - } else if (hasNoData) { - // Check if we are required to process with TPacketizerFile a registered dataset - TNamed *ftp = dynamic_cast(input->FindObject("PROOF_FilesToProcess")); - if (ftp) { - TString dsn(ftp->GetTitle()); - if (!dsn.Contains(":") || dsn.BeginsWith("dataset:")) { - dsn.ReplaceAll("dataset:", ""); - // Get the map for TPacketizerFile - // Make sure we have something in input and a dataset manager - if (!fDataSetManager) { - emsg.Form("dataset manager not initialized!"); - } else { - TFileCollection *fc = 0; - // Get the dataset - if (!(fc = fDataSetManager->GetDataSet(dsn))) { - emsg.Form("requested dataset '%s' does not exists", dsn.Data()); - } else { - TMap *fcmap = GetDataSetNodeMap(fc, emsg); - if (fcmap) { - input->Remove(ftp); - delete ftp; - fcmap->SetOwner(kTRUE); - fcmap->SetName("PROOF_FilesToProcess"); - input->Add(fcmap); - } - } - } - if (!emsg.IsNull()) { - SendAsynMessage(TString::Format("HandleProcess on %s: %s", - fPrefix.Data(), emsg.Data())); - Error("HandleProcess", "%s", emsg.Data()); - // To terminate collection - if (sync) SendLogFile(); - return; - } - } - } - } - - TProofQueryResult *pq = 0; - - // Create instance of query results; we set ownership of the input list - // to the TQueryResult object, to avoid too many instantiations - pq = MakeQueryResult(nentries, opt, 0, first, 0, filename, 0); - - // Prepare the input list and transfer it into the TQueryResult object - if (dset) input->Add(dset); - if (elist) input->Add(elist); - pq->SetInputList(input, kTRUE); - - // Clear the list - input->Clear("nodelete"); - SafeDelete(input); - - // Save input data, if any - if (TProof::SaveInputData(pq, fCacheDir.Data(), emsg) != 0) - Warning("HandleProcess", "could not save input data: %s", emsg.Data()); - - // If not a draw action add the query to the main list - if (!(pq->IsDraw())) { - if (fQMgr) { - if (fQMgr->Queries()) fQMgr->Queries()->Add(pq); - // Also save it to queries dir - fQMgr->SaveQuery(pq); - } - } - - // Add anyhow to the waiting lists - QueueQuery(pq); - - // Call get Workers - // if we are not idle the scheduler will just enqueue the query and - // send a resume message later. - - Bool_t enqueued = kFALSE; - Int_t pc = 0; - // if the session does not have workers and is in the dynamic mode - if (fProof->UseDynamicStartup()) { - // get the a list of workers and start them - TList* workerList = new TList(); - EQueryAction retVal = GetWorkers(workerList, pc); - if (retVal == TProofServ::kQueryStop) { - Error("HandleProcess", "error getting list of worker nodes"); - // To terminate collection - if (sync) SendLogFile(); - return; - } else if (retVal == TProofServ::kQueryEnqueued) { - // change to an asynchronous query - enqueued = kTRUE; - Info("HandleProcess", "query %d enqueued", pq->GetSeqNum()); - } else { - Int_t ret = fProof->AddWorkers(workerList); - if (ret < 0) { - Error("HandleProcess", "Adding a list of worker nodes returned: %d", - ret); - // To terminate collection - if (sync) SendLogFile(); - return; - } - } - } else { - EQueryAction retVal = GetWorkers(0, pc); - if (retVal == TProofServ::kQueryStop) { - Error("HandleProcess", "error getting list of worker nodes"); - // To terminate collection - if (sync) SendLogFile(); - return; - } else if (retVal == TProofServ::kQueryEnqueued) { - // change to an asynchronous query - enqueued = kTRUE; - Info("HandleProcess", "query %d enqueued", pq->GetSeqNum()); - } else if (retVal != TProofServ::kQueryOK) { - Error("HandleProcess", "unknown return value: %d", retVal); - // To terminate collection - if (sync) SendLogFile(); - return; - } - } - - // If the client submission was asynchronous, signal the submission of - // the query and communicate the assigned sequential number for later - // identification - TMessage m(kPROOF_QUERYSUBMITTED); - if (!sync || enqueued) { - m << pq->GetSeqNum() << kFALSE; - fSocket->Send(m); - } - - // Nothing more to do if we are not idle - if (!IsIdle()) { - // Notify submission - Info("HandleProcess", - "query \"%s:%s\" submitted", pq->GetTitle(), pq->GetName()); - return; - } - - // Process - // in the static mode, if a session is enqueued it will be processed after current query - // (there is no way to enqueue if idle). - // in the dynamic mode we will process here only if the session was idle and got workers! - Bool_t doprocess = kFALSE; - while (WaitingQueries() > 0 && !enqueued) { - doprocess = kTRUE; - // - ProcessNext(slb); - // avoid processing async queries sent during processing in dyn mode - if (fProof->UseDynamicStartup()) - enqueued = kTRUE; - - } // Loop on submitted queries - - // Set idle - SetIdle(kTRUE); - - // Reset mergers - fProof->ResetMergers(); - - // kPROOF_SETIDLE sets the client to idle; in asynchronous mode clients monitor - // TProof::IsIdle for to check the readiness of a query, so we need to send this - // before to be sure thatn everything about a query is received by the client - if (!sync) SendLogFile(); - - // Signal the client that we are idle - if (doprocess) { - m.Reset(kPROOF_SETIDLE); - Bool_t waiting = (WaitingQueries() > 0) ? kTRUE : kFALSE; - m << waiting; - fSocket->Send(m); - } - - // In synchronous mode TProof::Collect is terminated by the reception of the - // log file and subsequent submissions are controlled by TProof::IsIdle(), so - // this must be last one to be sent - if (sync) SendLogFile(); - - // Set idle - SetIdle(kTRUE); - - } else { - - // Reset compute stopwatch: we include all what done from now on - fCompute.Reset(); - fCompute.Start(); - - // Set not idle - SetIdle(kFALSE); - - // Cleanup the player - Bool_t deleteplayer = kTRUE; - MakePlayer(); - - // Setup data set - if (dset && (dset->IsA() == TDSetProxy::Class())) - ((TDSetProxy*)dset)->SetProofServ(this); - - // Get input data, if any - TString emsg; - if (TProof::GetInputData(input, fCacheDir.Data(), emsg) != 0) - Warning("HandleProcess", "could not get input data: %s", emsg.Data()); - - // Get query sequential number - if (TProof::GetParameter(input, "PROOF_QuerySeqNum", fQuerySeqNum) != 0) - Warning("HandleProcess", "could not get query sequential number!"); - - // Make the ordinal number available in the selector - TObject *nord = 0; - while ((nord = input->FindObject("PROOF_Ordinal"))) - input->Remove(nord); - input->Add(new TNamed("PROOF_Ordinal", GetOrdinal())); - - // Set input - TIter next(input); - TObject *o = 0; - while ((o = next())) { - PDB(kGlobal, 2) Info("HandleProcess", "adding: %s", o->GetName()); - fPlayer->AddInput(o); - } - - // Check if a TSelector object is passed via input list - TObject *obj = 0; - TSelector *selector_obj = 0; - TIter nxt(input); - while ((obj = nxt())){ - if (obj->InheritsFrom("TSelector")) { - selector_obj = (TSelector *) obj; - filename = selector_obj->ClassName(); - Info("HandleProcess", "selector obj for '%s' found", selector_obj->ClassName()); - break; - } - } - - // Signal the master that we are starting processing - fSocket->Send(kPROOF_STARTPROCESS); - - // Reset latency stopwatch - fLatency.Reset(); - fSaveOutput.Reset(); - - // Process - PDB(kGlobal, 1) Info("HandleProcess", "calling %s::Process()", fPlayer->IsA()->GetName()); - - if (selector_obj){ - Info("HandleProcess", "calling fPlayer->Process() with selector object: %s", selector_obj->ClassName()); - fPlayer->Process(dset, selector_obj, opt, nentries, first); - } - else { - Info("HandleProcess", "calling fPlayer->Process() with selector name: %s", filename.Data()); - fPlayer->Process(dset, filename, opt, nentries, first); - } - - // Return number of events processed - TMessage m(kPROOF_STOPPROCESS); - Bool_t abort = (fPlayer->GetExitStatus() != TVirtualProofPlayer::kAborted) ? kFALSE : kTRUE; - if (fProtocol > 18) { - TProofProgressStatus* status = - new TProofProgressStatus(fPlayer->GetEventsProcessed(), - gPerfStats?gPerfStats->GetBytesRead():0); - if (status) - m << status << abort; - if (slb) - slb->Form("%d %lld %lld", fPlayer->GetExitStatus(), - status->GetEntries(), status->GetBytesRead()); - SafeDelete(status); - } else { - m << fPlayer->GetEventsProcessed() << abort; - if (slb) - slb->Form("%d %lld -1", fPlayer->GetExitStatus(), fPlayer->GetEventsProcessed()); - } - - fSocket->Send(m); - PDB(kGlobal, 2) - Info("TProofServ::Handleprocess", - "worker %s has finished processing with %d objects in output list", - GetOrdinal(), fPlayer->GetOutputList()->GetEntries()); - - // Cleanup the input data set info - SafeDelete(dset); - SafeDelete(enl); - SafeDelete(evl); - - Bool_t outok = (fPlayer->GetExitStatus() != TVirtualProofPlayer::kAborted && - fPlayer->GetOutputList()) ? kTRUE : kFALSE; - if (outok) { - // Check if in controlled output sending mode or submerging - Int_t cso = 0; - Bool_t isSubMerging = kFALSE; - - // Check if we are in merging mode (i.e. parameter PROOF_UseMergers exists) - Int_t nm = 0; - if (TProof::GetParameter(input, "PROOF_UseMergers", nm) == 0) { - isSubMerging = (nm >= 0) ? kTRUE : kFALSE; - } - if (!isSubMerging) { - cso = gEnv->GetValue("Proof.ControlSendOutput", 1); - if (TProof::GetParameter(input, "PROOF_ControlSendOutput", cso) != 0) - cso = gEnv->GetValue("Proof.ControlSendOutput", 1); - } - - if (cso > 0) { - - // Control output sending mode: wait for the master to ask for the objects. - // Allows controls of memory usage on the master. - TMessage msg(kPROOF_SENDOUTPUT); - fSocket->Send(msg); - - // Set idle - SetIdle(kTRUE); - - // Do not cleanup the player yet: it will be used in sending output activities - deleteplayer = kFALSE; - - PDB(kGlobal, 1) - Info("HandleProcess", "controlled mode: worker %s has finished," - " sizes sent to master", fOrdinal.Data()); - } else { - - // Check if we are in merging mode (i.e. parameter PROOF_UseMergers exists) - if (TestBit(TProofServ::kHighMemory)) { - if (isSubMerging) - Info("HandleProcess", "submerging disabled because of high-memory case"); - isSubMerging = kFALSE; - } else { - PDB(kGlobal, 2) Info("HandleProcess", "merging mode check: %d", isSubMerging); - } - - if (!IsMaster() && isSubMerging) { - // Worker in merging mode. - //---------------------------- - // First, it reports only the size of its output to the master - // + port on which it can possibly accept outputs from other workers if it becomes a merger - // Master will later tell it where it should send the output (either to the master or to some merger) - // or if it should become a merger - - TMessage msg_osize(kPROOF_SUBMERGER); - msg_osize << Int_t(TProof::kOutputSize); - msg_osize << fPlayer->GetOutputList()->GetEntries(); - - fMergingSocket = new TServerSocket(0); - Int_t merge_port = 0; - if (fMergingSocket) { - PDB(kGlobal, 2) - Info("HandleProcess", "possible port for merging connections: %d", - fMergingSocket->GetLocalPort()); - merge_port = fMergingSocket->GetLocalPort(); - } - msg_osize << merge_port; - fSocket->Send(msg_osize); - - // Set idle - SetIdle(kTRUE); - - // Do not cleanup the player yet: it will be used in sub-merging activities - deleteplayer = kFALSE; - - PDB(kSubmerger, 2) Info("HandleProcess", "worker %s has finished", fOrdinal.Data()); - - } else { - // Sub-master OR worker not in merging mode - // --------------------------------------------- - PDB(kGlobal, 2) Info("HandleProcess", "sending result directly to master"); - if (SendResults(fSocket, fPlayer->GetOutputList()) != 0) - Warning("HandleProcess","problems sending output list"); - - // Masters reset the mergers, if any - if (IsMaster()) fProof->ResetMergers(); - - // Signal the master that we are idle - fSocket->Send(kPROOF_SETIDLE); - - // Set idle - SetIdle(kTRUE); - - // Notify the user - SendLogFile(); - } - - - - } - - } else { - // No output list - if (fPlayer->GetExitStatus() != TVirtualProofPlayer::kAborted) - Warning("HandleProcess","the output list is empty!"); - if (SendResults(fSocket) != 0) - Warning("HandleProcess", "problems sending output list"); - - // Masters reset the mergers, if any - if (IsMaster()) fProof->ResetMergers(); - - // Signal the master that we are idle - fSocket->Send(kPROOF_SETIDLE); - - // Set idle - SetIdle(kTRUE); - - // Notify the user - SendLogFile(); - } - - // Prevent from double-deleting in input - TIter nex(input); - while ((obj = nex())) { - if (obj->InheritsFrom("TSelector")) input->Remove(obj); - } - - // Make also sure the input list objects are deleted - fPlayer->GetInputList()->SetOwner(0); - - // Remove possible inputs from a file and the file, if any - TList *added = dynamic_cast(input->FindObject("PROOF_InputObjsFromFile")); - if (added) { - if (added->GetSize() > 0) { - // The file must be the last one - TFile *f = dynamic_cast(added->Last()); - if (f) { - added->Remove(f); - TIter nxo(added); - while ((o = nxo())) { input->Remove(o); } - input->Remove(added); - added->SetOwner(kFALSE); - added->Clear(); - f->Close(); - delete f; - } - } - SafeDelete(added); - } - input->SetOwner(); - SafeDelete(input); - - // Cleanup if required - if (deleteplayer) DeletePlayer(); - } - - PDB(kGlobal, 1) Info("HandleProcess", "done"); - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Sends all objects from the given list to the specified socket - -Int_t TProofServ::SendResults(TSocket *sock, TList *outlist, TQueryResult *pq) -{ - PDB(kOutput, 2) Info("SendResults", "enter"); - - TString msg; - if (fProtocol > 23 && outlist) { - // Send objects in bunches of max fMsgSizeHWM bytes to optimize transfer - // Objects are merged one-by-one by the client - // Messages for objects - TMessage mbuf(kPROOF_OUTPUTOBJECT); - // Objects in the output list - Int_t olsz = outlist->GetSize(); - if (IsTopMaster() && pq) { - msg.Form("%s: merging output objects ... done ", - fPrefix.Data()); - SendAsynMessage(msg.Data()); - // Message for the client - msg.Form("%s: objects merged; sending output: %d objs", fPrefix.Data(), olsz); - SendAsynMessage(msg.Data(), kFALSE); - // Send light query info - mbuf << (Int_t) 0; - mbuf.WriteObject(pq); - if (sock->Send(mbuf) < 0) return -1; - } - // Objects in the output list - Int_t ns = 0, np = 0; - TIter nxo(outlist); - TObject *o = 0; - Int_t totsz = 0, objsz = 0; - mbuf.Reset(); - while ((o = nxo())) { - if (mbuf.Length() > fMsgSizeHWM) { - PDB(kOutput, 1) - Info("SendResults", - "message has %d bytes: limit of %lld bytes reached - sending ...", - mbuf.Length(), fMsgSizeHWM); - // Compress the message, if required; for these messages we do it already - // here so we get the size; TXSocket does not do it twice. - if (GetCompressionLevel() > 0) { - mbuf.SetCompressionSettings(fCompressMsg); - mbuf.Compress(); - objsz = mbuf.CompLength(); - } else { - objsz = mbuf.Length(); - } - totsz += objsz; - if (IsTopMaster()) { - msg.Form("%s: objects merged; sending obj %d/%d (%d bytes) ", - fPrefix.Data(), ns, olsz, objsz); - SendAsynMessage(msg.Data(), kFALSE); - } - if (sock->Send(mbuf) < 0) return -1; - // Reset the message - mbuf.Reset(); - np = 0; - } - ns++; - np++; - mbuf << (Int_t) ((ns >= olsz) ? 2 : 1); - mbuf << o; - } - if (np > 0) { - // Compress the message, if required; for these messages we do it already - // here so we get the size; TXSocket does not do it twice. - if (GetCompressionLevel() > 0) { - mbuf.SetCompressionSettings(fCompressMsg); - mbuf.Compress(); - objsz = mbuf.CompLength(); - } else { - objsz = mbuf.Length(); - } - totsz += objsz; - if (IsTopMaster()) { - msg.Form("%s: objects merged; sending obj %d/%d (%d bytes) ", - fPrefix.Data(), ns, olsz, objsz); - SendAsynMessage(msg.Data(), kFALSE); - } - if (sock->Send(mbuf) < 0) return -1; - } - if (IsTopMaster()) { - // Send total size - msg.Form("%s: grand total: sent %d objects, size: %d bytes ", - fPrefix.Data(), olsz, totsz); - SendAsynMessage(msg.Data()); - } - } else if (fProtocol > 10 && outlist) { - - // Send objects one-by-one to optimize transfer and merging - // Messages for objects - TMessage mbuf(kPROOF_OUTPUTOBJECT); - // Objects in the output list - Int_t olsz = outlist->GetSize(); - if (IsTopMaster() && pq) { - msg.Form("%s: merging output objects ... done ", - fPrefix.Data()); - SendAsynMessage(msg.Data()); - // Message for the client - msg.Form("%s: objects merged; sending output: %d objs", fPrefix.Data(), olsz); - SendAsynMessage(msg.Data(), kFALSE); - // Send light query info - mbuf << (Int_t) 0; - mbuf.WriteObject(pq); - if (sock->Send(mbuf) < 0) return -1; - } - - Int_t ns = 0; - Int_t totsz = 0, objsz = 0; - TIter nxo(fPlayer->GetOutputList()); - TObject *o = 0; - while ((o = nxo())) { - ns++; - mbuf.Reset(); - Int_t type = (Int_t) ((ns >= olsz) ? 2 : 1); - mbuf << type; - mbuf.WriteObject(o); - // Compress the message, if required; for these messages we do it already - // here so we get the size; TXSocket does not do it twice. - if (GetCompressionLevel() > 0) { - mbuf.SetCompressionSettings(fCompressMsg); - mbuf.Compress(); - objsz = mbuf.CompLength(); - } else { - objsz = mbuf.Length(); - } - totsz += objsz; - if (IsTopMaster()) { - msg.Form("%s: objects merged; sending obj %d/%d (%d bytes) ", - fPrefix.Data(), ns, olsz, objsz); - SendAsynMessage(msg.Data(), kFALSE); - } - if (sock->Send(mbuf) < 0) return -1; - } - // Total size - if (IsTopMaster()) { - // Send total size - msg.Form("%s: grand total: sent %d objects, size: %d bytes ", - fPrefix.Data(), olsz, totsz); - SendAsynMessage(msg.Data()); - } - - } else if (IsTopMaster() && fProtocol > 6 && outlist) { - - // Buffer to be sent - TMessage mbuf(kPROOF_OUTPUTLIST); - mbuf.WriteObject(pq); - // Sizes - Int_t blen = mbuf.CompLength(); - Int_t olsz = outlist->GetSize(); - // Message for the client - msg.Form("%s: sending output: %d objs, %d bytes", fPrefix.Data(), olsz, blen); - SendAsynMessage(msg.Data(), kFALSE); - if (sock->Send(mbuf) < 0) return -1; - - } else { - if (outlist) { - PDB(kGlobal, 2) Info("SendResults", "sending output list"); - } else { - PDB(kGlobal, 2) Info("SendResults", "notifying failure or abort"); - } - if (sock->SendObject(outlist, kPROOF_OUTPUTLIST) < 0) return -1; - } - - PDB(kOutput,2) Info("SendResults", "done"); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// process the next query from the queue of submitted jobs. -/// to be called on the top master only. - -void TProofServ::ProcessNext(TString *slb) -{ - TDSet *dset = 0; - TString filename, opt; - TList *input = 0; - Long64_t nentries = -1, first = 0; - - // TObject *elist = 0; - TProofQueryResult *pq = 0; - - TObject* obj = 0; - TSelector* selector_obj = 0; - - // Process - - // Reset compute stopwatch: we include all what done from now on - fCompute.Reset(); - fCompute.Start(); - - // Get next query info (also removes query from the list) - pq = NextQuery(); - if (pq) { - - // Set not idle - SetIdle(kFALSE); - opt = pq->GetOptions(); - input = pq->GetInputList(); - nentries = pq->GetEntries(); - first = pq->GetFirst(); - filename = pq->GetSelecImp()->GetName(); - Ssiz_t id = opt.Last('#'); - if (id != kNPOS && id < opt.Length() - 1) { - filename += opt(id + 1, opt.Length()); - // Remove it from 'opt' so user found on the workers what they specified - opt.Remove(id); - } - // Attach to data set and entry- (or event-) list (if any) - TObject *o = 0; - if ((o = pq->GetInputObject("TDSet"))) { - dset = (TDSet *) o; - } else { - // Should never get here - Error("ProcessNext", "no TDset object: cannot continue"); - return; - } - // elist = 0; - // if ((o = pq->GetInputObject("TEntryList"))) - // elist = o; - // else if ((o = pq->GetInputObject("TEventList"))) - // elist = o; - - // Expand selector files - if (pq->GetSelecImp()) { - gSystem->Exec(TString::Format("%s %s", kRM, pq->GetSelecImp()->GetName())); - pq->GetSelecImp()->SaveSource(pq->GetSelecImp()->GetName()); - } - if (pq->GetSelecHdr() && - !strstr(pq->GetSelecHdr()->GetName(), "TProofDrawHist")) { - gSystem->Exec(TString::Format("%s %s", kRM, pq->GetSelecHdr()->GetName())); - pq->GetSelecHdr()->SaveSource(pq->GetSelecHdr()->GetName()); - } - - // Taking out a TSelector object from input list - TIter nxt(input); - while ((obj = nxt())){ - if (obj->InheritsFrom("TSelector") && - !strcmp(pq->GetSelecImp()->GetName(), obj->ClassName())) { - selector_obj = (TSelector *) obj; - Info("ProcessNext", "found object for selector '%s'", obj->ClassName()); - break; - } - } - - } else { - // Should never get here - Error("ProcessNext", "empty waiting queries list!"); - return; - } - - // Set in running state - SetQueryRunning(pq); - - // Save to queries dir, if not standard draw - if (fQMgr) { - if (!(pq->IsDraw())) - fQMgr->SaveQuery(pq); - else - fQMgr->IncrementDrawQueries(); - fQMgr->ResetTime(); - } - - // Signal the client that we are starting a new query - TMessage m(kPROOF_STARTPROCESS); - m << TString(pq->GetSelecImp()->GetName()) - << dset->GetNumOfFiles() - << pq->GetFirst() << pq->GetEntries(); - fSocket->Send(m); - - // Create player - MakePlayer(); - - // Add query results to the player lists - fPlayer->AddQueryResult(pq); - - // Set query currently processed - fPlayer->SetCurrentQuery(pq); - - // Setup data set - if (dset->IsA() == TDSetProxy::Class()) - ((TDSetProxy*)dset)->SetProofServ(this); - - // Add the unique query tag as TNamed object to the input list - // so that it is available in TSelectors for monitoring - TString qid = TString::Format("%s:%s",pq->GetTitle(),pq->GetName()); - input->Add(new TNamed("PROOF_QueryTag", qid.Data())); - // ... and the sequential number - fQuerySeqNum = pq->GetSeqNum(); - input->Add(new TParameter("PROOF_QuerySeqNum", fQuerySeqNum)); - - // Check whether we have to enforce the use of submergers, but only if the user did - // not express itself on the subject - if (gEnv->Lookup("Proof.UseMergers") && !input->FindObject("PROOF_UseMergers")) { - Int_t smg = gEnv->GetValue("Proof.UseMergers",-1); - if (smg >= 0) { - input->Add(new TParameter("PROOF_UseMergers", smg)); - PDB(kSubmerger, 2) Info("ProcessNext", "PROOF_UseMergers set to %d", smg); - if (gEnv->Lookup("Proof.MergersByHost")) { - Int_t mbh = gEnv->GetValue("Proof.MergersByHost", 0); - if (mbh != 0) { - // Administrator settings have the priority - TObject *o = 0; - if ((o = input->FindObject("PROOF_MergersByHost"))) { input->Remove(o); delete o; } - input->Add(new TParameter("PROOF_MergersByHost", mbh)); - PDB(kSubmerger, 2) Info("ProcessNext", "submergers setup by host/node"); - } - } - } - } - - // Set input - TIter next(input); - TObject *o = 0; - while ((o = next())) { - PDB(kGlobal, 2) Info("ProcessNext", "adding: %s", o->GetName()); - fPlayer->AddInput(o); - } - - // Remove the list of the missing files from the original list, if any - if ((o = input->FindObject("MissingFiles"))) input->Remove(o); - - // Process - PDB(kGlobal, 1) Info("ProcessNext", "calling %s::Process()", fPlayer->IsA()->GetName()); - if (selector_obj){ - Info("ProcessNext", "calling fPlayer->Process() with selector object: %s", selector_obj->ClassName()); - fPlayer->Process(dset, selector_obj, opt, nentries, first); - } - else { - Info("ProcessNext", "calling fPlayer->Process() with selector name: %s", filename.Data()); - fPlayer->Process(dset, filename, opt, nentries, first); - } - - // This is the end of merging - fPlayer->SetMerging(kFALSE); - - // Return number of events processed - Bool_t abort = - (fPlayer->GetExitStatus() == TVirtualProofPlayer::kAborted) ? kTRUE : kFALSE; - if (fPlayer->GetExitStatus() != TVirtualProofPlayer::kFinished) { - m.Reset(kPROOF_STOPPROCESS); - // message sent from worker to the master - if (fProtocol > 18) { - TProofProgressStatus* status = fPlayer->GetProgressStatus(); - m << status << abort; - status = 0; // the status belongs to the player. - } else if (fProtocol > 8) { - m << fPlayer->GetEventsProcessed() << abort; - } else { - m << fPlayer->GetEventsProcessed(); - } - fSocket->Send(m); - } - - // Register any dataset produced during this processing, if required - if (fDataSetManager && fPlayer->GetOutputList()) { - TNamed *psr = (TNamed *) fPlayer->GetOutputList()->FindObject("PROOFSERV_RegisterDataSet"); - if (psr) { - TString emsg; - if (RegisterDataSets(input, fPlayer->GetOutputList(), fDataSetManager, emsg) != 0) - Warning("ProcessNext", "problems registering produced datasets: %s", emsg.Data()); - do { - fPlayer->GetOutputList()->Remove(psr); - delete psr; - } while ((psr = (TNamed *) fPlayer->GetOutputList()->FindObject("PROOFSERV_RegisterDataSet"))); - } - } - - // Complete filling of the TQueryResult instance - if (fQMgr && !pq->IsDraw()) { - if (!abort) fProof->AskStatistics(); - if (fQMgr->FinalizeQuery(pq, fProof, fPlayer)) - fQMgr->SaveQuery(pq, fMaxQueries); - } - - // If we were requested to save results on the master and we are not in save-to-file mode - // then we save the results - if (IsTopMaster() && fPlayer->GetOutputList()) { - Bool_t save = kTRUE; - TIter nxo(fPlayer->GetOutputList()); - TObject *xo = 0; - while ((xo = nxo())) { - if (xo->InheritsFrom("TProofOutputFile") && xo->TestBit(TProofOutputFile::kSwapFile)) { - save = kFALSE; - break; - } - } - if (save) { - TNamed *nof = (TNamed *) input->FindObject("PROOF_DefaultOutputOption"); - if (nof) { - TString oopt(nof->GetTitle()); - if (oopt.BeginsWith("of:")) { - oopt.Replace(0, 3, ""); - if (!oopt.IsNull()) fPlayer->SetOutputFilePath(oopt); - fPlayer->SavePartialResults(kTRUE, kTRUE); - } - } - } - } - - // Send back the results - TQueryResult *pqr = pq->CloneInfo(); - // At least the TDSet name in the light object - Info("ProcessNext", "adding info about dataset '%s' in the light query result", dset->GetName()); - TList rin; - TDSet *ds = new TDSet(dset->GetName(), dset->GetObjName()); - rin.Add(ds); - if (pqr) pqr->SetInputList(&rin, kTRUE); - if (fPlayer->GetExitStatus() != TVirtualProofPlayer::kAborted && fPlayer->GetOutputList()) { - PDB(kGlobal, 2) - Info("ProcessNext", "sending results"); - TQueryResult *xpq = (pqr && fProtocol > 10) ? pqr : pq; - if (SendResults(fSocket, fPlayer->GetOutputList(), xpq) != 0) - Warning("ProcessNext", "problems sending output list"); - if (slb) slb->Form("%d %lld %lld %.3f", fPlayer->GetExitStatus(), pq->GetEntries(), - pq->GetBytes(), pq->GetUsedCPU()); - } else { - if (fPlayer->GetExitStatus() != TVirtualProofPlayer::kAborted) - Warning("ProcessNext","the output list is empty!"); - if (SendResults(fSocket, fPlayer->GetOutputList()) != 0) - Warning("ProcessNext", "problems sending output list"); - if (slb) slb->Form("%d -1 -1 %.3f", fPlayer->GetExitStatus(), pq->GetUsedCPU()); - } - - // Remove aborted queries from the list - if (fPlayer->GetExitStatus() == TVirtualProofPlayer::kAborted) { - SafeDelete(pqr); - if (fQMgr) fQMgr->RemoveQuery(pq); - } else { - // Keep in memory only light infor about a query - if (!(pq->IsDraw()) && pqr) { - if (fQMgr && fQMgr->Queries()) { - fQMgr->Queries()->Add(pqr); - // Remove from the fQueries list - fQMgr->Queries()->Remove(pq); - } - // These removes 'pq' from the internal player list and - // deletes it; in this way we do not attempt a double delete - // when destroying the player - fPlayer->RemoveQueryResult(TString::Format("%s:%s", - pq->GetTitle(), pq->GetName())); - } - } - - DeletePlayer(); - if (IsMaster() && fProof->UseDynamicStartup()) - // stop the workers - fProof->RemoveWorkers(0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Register TFileCollections in 'out' as datasets according to the rules in 'in' - -Int_t TProofServ::RegisterDataSets(TList *in, TList *out, - TDataSetManager *dsm, TString &msg) -{ - PDB(kDataset, 1) - ::Info("TProofServ::RegisterDataSets", - "enter: %d objs in the output list", (out ? out->GetSize() : -1)); - - if (!in || !out || !dsm) { - ::Error("TProofServ::RegisterDataSets", "invalid inputs: %p, %p, %p", in, out, dsm); - return 0; - } - msg = ""; - THashList tags; - TList torm; - TIter nxo(out); - TObject *o = 0; - while ((o = nxo())) { - // Only file collections TFileCollection - TFileCollection *ds = dynamic_cast (o); - if (ds) { - // Origin of this dataset - ds->SetTitle(gSystem->HostName()); - // The tag and register option - TNamed *fcn = 0; - TString tag = TString::Format("DATASET_%s", ds->GetName()); - if (!(fcn = (TNamed *) out->FindObject(tag))) continue; - // If this tag is in the list of processed tags, flag it for removal - if (tags.FindObject(tag)) { - torm.Add(o); - continue; - } - // Register option - TString regopt(fcn->GetTitle()); - // Sort according to the internal index, if required - if (regopt.Contains(":sortidx:")) { - ds->Sort(kTRUE); - regopt.ReplaceAll(":sortidx:", ""); - } - // Register this dataset - if (dsm->TestBit(TDataSetManager::kAllowRegister)) { - // Extract the list - if (ds->GetList()->GetSize() > 0) { - // Register the dataset (quota checks are done inside here) - const char *vfmsg = regopt.Contains("V") ? " and verifying" : ""; - msg.Form("Registering%s dataset '%s' ... ", vfmsg, ds->GetName()); - // Always allow verification for this action - Bool_t allowVerify = dsm->TestBit(TDataSetManager::kAllowVerify) ? kTRUE : kFALSE; - if (regopt.Contains("V") && !allowVerify) dsm->SetBit(TDataSetManager::kAllowVerify); - // Main action - Int_t rc = dsm->RegisterDataSet(ds->GetName(), ds, regopt); - // Reset to the previous state if needed - if (regopt.Contains("V") && !allowVerify) dsm->ResetBit(TDataSetManager::kAllowVerify); - if (rc != 0) { - ::Warning("TProofServ::RegisterDataSets", - "failure registering or verifying dataset '%s'", ds->GetName()); - msg.Form("Registering%s dataset '%s' ... failed! See log for more details", vfmsg, ds->GetName()); - } else { - ::Info("TProofServ::RegisterDataSets", "dataset '%s' successfully registered%s", - ds->GetName(), (strlen(vfmsg) > 0) ? " and verified" : ""); - msg.Form("Registering%s dataset '%s' ... OK", vfmsg, ds->GetName()); - // Add tag to the list of processed tags to avoid double processing - // (there may be more objects with the same name, created by each worker) - tags.Add(new TObjString(tag)); - } - // Notify - PDB(kDataset, 2) { - ::Info("TProofServ::RegisterDataSets", "printing collection"); - ds->Print("F"); - } - } else { - ::Warning("TProofServ::RegisterDataSets", "collection '%s' is empty", o->GetName()); - } - } else { - ::Info("TProofServ::RegisterDataSets", "dataset registration not allowed"); - return -1; - } - } - } - // Cleanup all temporary stuff possibly created by each worker - TIter nxrm(&torm); - while ((o = nxrm())) out->Remove(o); - torm.SetOwner(kTRUE); - // Remove tags - TIter nxtg(&tags); - while((o = nxtg())) { - TObject *oo = 0; - while ((oo = out->FindObject(o->GetName()))) { out->Remove(oo); } - } - tags.SetOwner(kTRUE); - - PDB(kDataset, 1) ::Info("TProofServ::RegisterDataSets", "exit"); - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle request for list of queries. - -void TProofServ::HandleQueryList(TMessage *mess) -{ - PDB(kGlobal, 1) - Info("HandleQueryList", "Enter"); - - Bool_t all; - (*mess) >> all; - - TList *ql = new TList; - Int_t ntot = 0, npre = 0, ndraw= 0; - if (fQMgr) { - if (all) { - // Rescan - TString qdir = fQueryDir; - Int_t idx = qdir.Index("session-"); - if (idx != kNPOS) - qdir.Remove(idx); - fQMgr->ScanPreviousQueries(qdir); - // Send also information about previous queries, if any - if (fQMgr->PreviousQueries()) { - TIter nxq(fQMgr->PreviousQueries()); - TProofQueryResult *pqr = 0; - while ((pqr = (TProofQueryResult *)nxq())) { - ntot++; - pqr->fSeqNum = ntot; - ql->Add(pqr); - } - } - } - - npre = ntot; - if (fQMgr->Queries()) { - // Add info about queries in this session - TIter nxq(fQMgr->Queries()); - TProofQueryResult *pqr = 0; - TQueryResult *pqm = 0; - while ((pqr = (TProofQueryResult *)nxq())) { - ntot++; - if ((pqm = pqr->CloneInfo())) { - pqm->fSeqNum = ntot; - ql->Add(pqm); - } else { - Warning("HandleQueryList", "unable to clone TProofQueryResult '%s:%s'", - pqr->GetName(), pqr->GetTitle()); - } - } - } - // Number of draw queries - ndraw = fQMgr->DrawQueries(); - } - - TMessage m(kPROOF_QUERYLIST); - m << npre << ndraw << ql; - fSocket->Send(m); - delete ql; - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle remove request. - -void TProofServ::HandleRemove(TMessage *mess, TString *slb) -{ - PDB(kGlobal, 1) - Info("HandleRemove", "Enter"); - - TString queryref; - (*mess) >> queryref; - - if (slb) *slb = queryref; - - if (queryref == "cleanupqueue") { - // Remove pending requests - Int_t pend = CleanupWaitingQueries(); - // Notify - Info("HandleRemove", "%d queries removed from the waiting list", pend); - // We are done - return; - } - - if (queryref == "cleanupdir") { - - // Cleanup previous sessions results - Int_t nd = (fQMgr) ? fQMgr->CleanupQueriesDir() : -1; - - // Notify - Info("HandleRemove", "%d directories removed", nd); - // We are done - return; - } - - - if (fQMgr) { - TProofLockPath *lck = 0; - if (fQMgr->LockSession(queryref, &lck) == 0) { - - // Remove query - TList qtorm; - fQMgr->RemoveQuery(queryref, &qtorm); - CleanupWaitingQueries(kFALSE, &qtorm); - - // Unlock and remove the lock file - if (lck) { - gSystem->Unlink(lck->GetName()); - SafeDelete(lck); - } - - // We are done - return; - } - } else { - Warning("HandleRemove", "query result manager undefined!"); - } - - // Notify failure - Info("HandleRemove", - "query %s could not be removed (unable to lock session)", queryref.Data()); - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle retrieve request. - -void TProofServ::HandleRetrieve(TMessage *mess, TString *slb) -{ - PDB(kGlobal, 1) - Info("HandleRetrieve", "Enter"); - - TString queryref; - (*mess) >> queryref; - - if (slb) *slb = queryref; - - // Parse reference string - Int_t qry = -1; - TString qdir; - if (fQMgr) fQMgr->LocateQuery(queryref, qry, qdir); - - TString fout = qdir; - fout += "/query-result.root"; - - TFile *f = TFile::Open(fout,"READ"); - TProofQueryResult *pqr = 0; - if (f) { - f->ReadKeys(); - TIter nxk(f->GetListOfKeys()); - TKey *k = 0; - while ((k = (TKey *)nxk())) { - if (!strcmp(k->GetClassName(), "TProofQueryResult")) { - pqr = (TProofQueryResult *) f->Get(k->GetName()); - // For backward compatibility - if (pqr && fProtocol < 13) { - TDSet *d = 0; - TObject *o = 0; - TIter nxi(pqr->GetInputList()); - while ((o = nxi())) - if ((d = dynamic_cast(o))) - break; - d->SetWriteV3(kTRUE); - } - if (pqr) { - - // Message for the client - Float_t qsz = (Float_t) f->GetSize(); - Int_t ilb = 0; - static const char *clb[4] = { "bytes", "KB", "MB", "GB" }; - while (qsz > 1000. && ilb < 3) { - qsz /= 1000.; - ilb++; - } - SendAsynMessage(TString::Format("%s: sending result of %s:%s (%.1f %s)", - fPrefix.Data(), pqr->GetTitle(), pqr->GetName(), - qsz, clb[ilb])); - fSocket->SendObject(pqr, kPROOF_RETRIEVE); - } else { - Info("HandleRetrieve", - "query not found in file %s",fout.Data()); - // Notify not found - fSocket->SendObject(0, kPROOF_RETRIEVE); - } - break; - } - } - f->Close(); - delete f; - } else { - Info("HandleRetrieve", - "file cannot be open (%s)",fout.Data()); - // Notify not found - fSocket->SendObject(0, kPROOF_RETRIEVE); - return; - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle lib, inc search paths modification request - -Int_t TProofServ::HandleLibIncPath(TMessage *mess) -{ - TString type; - Bool_t add; - TString path; - Int_t rc = 1; - (*mess) >> type >> add >> path; - if (mess->BufferSize() > mess->Length()) (*mess) >> rc; - - // Check type of action - if ((type != "lib") && (type != "inc")) { - Error("HandleLibIncPath","unknown action type: %s", type.Data()); - return rc; - } - - // Separators can be either commas or blanks - path.ReplaceAll(","," "); - - // Decompose lists - TObjArray *op = 0; - if (path.Length() > 0 && path != "-") { - if (!(op = path.Tokenize(" "))) { - Error("HandleLibIncPath","decomposing path %s", path.Data()); - return rc; - } - } - - if (add) { - - if (type == "lib") { - - // Add libs - TIter nxl(op, kIterBackward); - TObjString *lib = 0; - while ((lib = (TObjString *) nxl())) { - // Expand path - TString xlib = lib->GetName(); - gSystem->ExpandPathName(xlib); - // Add to the dynamic lib search path if it exists and can be read - if (!gSystem->AccessPathName(xlib, kReadPermission)) { - TString newlibpath = gSystem->GetDynamicPath(); - // In the first position after the working dir - Int_t pos = 0; - if (newlibpath.BeginsWith(".:")) - pos = 2; - if (newlibpath.Index(xlib) == kNPOS) { - newlibpath.Insert(pos,TString::Format("%s:", xlib.Data())); - gSystem->SetDynamicPath(newlibpath); - } - } else { - Info("HandleLibIncPath", - "libpath %s does not exist or cannot be read - not added", xlib.Data()); - } - } - - // Forward the request, if required - if (IsMaster()) - fProof->AddDynamicPath(path); - - } else { - - // Add incs - TIter nxi(op); - TObjString *inc = 0; - while ((inc = (TObjString *) nxi())) { - // Expand path - TString xinc = inc->GetName(); - gSystem->ExpandPathName(xinc); - // Add to the dynamic lib search path if it exists and can be read - if (!gSystem->AccessPathName(xinc, kReadPermission)) { - TString curincpath = gSystem->GetIncludePath(); - if (curincpath.Index(xinc) == kNPOS) - gSystem->AddIncludePath(TString::Format("-I%s", xinc.Data())); - } else - Info("HandleLibIncPath", - "incpath %s does not exist or cannot be read - not added", xinc.Data()); - } - - // Forward the request, if required - if (IsMaster()) - fProof->AddIncludePath(path); - } - - - } else { - - if (type == "lib") { - - // Remove libs - TIter nxl(op); - TObjString *lib = 0; - while ((lib = (TObjString *) nxl())) { - // Expand path - TString xlib = lib->GetName(); - gSystem->ExpandPathName(xlib); - // Remove from the dynamic lib search path - TString newlibpath = gSystem->GetDynamicPath(); - newlibpath.ReplaceAll(TString::Format("%s:", xlib.Data()),""); - gSystem->SetDynamicPath(newlibpath); - } - - // Forward the request, if required - if (IsMaster()) - fProof->RemoveDynamicPath(path); - - } else { - - // Remove incs - TIter nxi(op); - TObjString *inc = 0; - while ((inc = (TObjString *) nxi())) { - TString newincpath = gSystem->GetIncludePath(); - newincpath.ReplaceAll(TString::Format("-I%s", inc->GetName()),""); - // Remove the interpreter path (added anyhow internally) - newincpath.ReplaceAll(gInterpreter->GetIncludePath(),""); - gSystem->SetIncludePath(newincpath); - } - - // Forward the request, if required - if (IsMaster()) - fProof->RemoveIncludePath(path); - } - } - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle file checking request. - -void TProofServ::HandleCheckFile(TMessage *mess, TString *slb) -{ - TString filenam; - TMD5 md5; - UInt_t opt = TProof::kUntar; - - TMessage reply(kPROOF_CHECKFILE); - - // Parse message - (*mess) >> filenam >> md5; - if ((mess->BufferSize() > mess->Length()) && (fProtocol > 8)) - (*mess) >> opt; - - if (slb) *slb = filenam; - - if (filenam.BeginsWith("-")) { - // install package: - // compare md5's, untar, store md5 in PROOF-INF, remove par file - Int_t st = 0; - Bool_t err = kFALSE; - filenam = filenam.Strip(TString::kLeading, '-'); - TString packnam = filenam; - packnam.Remove(packnam.Length() - 4); // strip off ".par" - // compare md5's to check if transmission was ok - TMD5 *md5local = fPackMgr->GetMD5(packnam); - if (md5local && md5 == (*md5local)) { - if ((opt & TProof::kRemoveOld)) { - if ((st = fPackMgr->Clean(packnam))) - Error("HandleCheckFile", "failure cleaning %s", packnam.Data()); - } - // Unpack - st = fPackMgr->Unpack(packnam, md5local); - if (st == 0) { - // Notify the client - reply << (Int_t)1; - PDB(kPackage, 1) - Info("HandleCheckFile", - "package %s installed on node", filenam.Data()); - } else { - if (st == -2) { - Error("HandleCheckFile", "gunzip not found"); - } else { - Error("HandleCheckFile", "package %s did not unpack into %s", - filenam.Data(), packnam.Data()); - } - reply << (Int_t)0; - if (fProtocol <= 19) reply.Reset(kPROOF_FATAL); - err = kTRUE; - } - } else { - reply << (Int_t)0; - if (fProtocol <= 19) reply.Reset(kPROOF_FATAL); - err = kTRUE; - PDB(kPackage, 1) - Error("HandleCheckFile", - "package %s not yet on node", filenam.Data()); - } - - // Note: Originally an fPackageLock->Unlock() call was made - // after the if-else statement below. With multilevel masters, - // submasters still check to make sure the package exists with - // the correct md5 checksum and need to do a read lock there. - // As yet locking is not that sophisicated so the lock must - // be released below before the call to fProof->UploadPackage(). - if (err) { - // delete par file in case of error - fPackMgr->Remove(filenam); - } else if (IsMaster()) { - // forward to workers - TString parpath; - fPackMgr->GetParPath(filenam, parpath); - if (fProof->UploadPackage(parpath, - (TProof::EUploadPackageOpt)opt) != 0) - Info("HandleCheckFile", - "problems uploading package %s", parpath.Data()); - } - delete md5local; - fSocket->Send(reply); - - } else if (filenam.BeginsWith("+") || filenam.BeginsWith("=")) { - filenam.Remove(0,1); - - TString parname = filenam; - // If remote install it from there - TFile::EFileType ft = TFile::GetType(filenam); - if (ft == TFile::kWeb || ft == TFile::kNet) { - parname = gSystem->BaseName(filenam); - if (fPackMgr->Install(filenam) < 0) { - Warning("HandleCheckFile", - "problems installing package %s", filenam.Data()); - - } - } - - // check file in package directory - TMD5 *md5local = fPackMgr->ReadMD5(parname); - - if (md5local && md5 == (*md5local)) { - // package already on server, unlock directory - reply << (Int_t)1; - PDB(kPackage, 1) - Info("HandleCheckFile", - "package %s already on node", parname.Data()); - if (IsMaster()) { - Int_t xrc = 0; - TString par = filenam; - if (ft != TFile::kWeb && ft != TFile::kNet) { - xrc = fPackMgr->GetParPath(filenam, par); - } - if (xrc == 0) { - if (fProof->UploadPackage(par) != 0) - Warning("HandleCheckFile", - "problems uploading package %s", par.Data()); - } - } - - } else { - reply << (Int_t)0; - if (fProtocol <= 19) reply.Reset(kPROOF_FATAL); - PDB(kPackage, 1) - Info("HandleCheckFile", - "package %s not yet on node", filenam.Data()); - } - delete md5local; - fSocket->Send(reply); - - } else { - // check file in cache directory - TString cachef = fCacheDir + "/" + filenam; - fCacheLock->Lock(); - TMD5 *md5local = TMD5::FileChecksum(cachef); - - if (md5local && md5 == (*md5local)) { - reply << (Int_t)1; - PDB(kCache, 1) - Info("HandleCheckFile", "file %s already on node", filenam.Data()); - } else { - reply << (Int_t)0; - if (fProtocol <= 19) reply.Reset(kPROOF_FATAL); - PDB(kCache, 1) - Info("HandleCheckFile", "file %s not yet on node", filenam.Data()); - } - delete md5local; - fSocket->Send(reply); - fCacheLock->Unlock(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle here all cache and package requests. - -Int_t TProofServ::HandleCache(TMessage *mess, TString *slb) -{ - PDB(kGlobal, 1) - Info("HandleCache", "Enter"); - - Int_t status = 0; - Int_t type = 0; - Bool_t all = kFALSE; - TMessage msg; - Bool_t fromglobal = kFALSE; - Int_t chkveropt = TPackMgr::kCheckROOT; // Default: check ROOT version - TPackMgr *packmgr = 0; - - // Notification message - TString noth; - const char *k = (IsMaster()) ? "Mst" : "Wrk"; - noth.Form("%s-%s", k, fOrdinal.Data()); - - TList *optls = 0; - TString packagedir, package, pdir, ocwd, file; - (*mess) >> type; - switch (type) { - case TProof::kShowCache: - (*mess) >> all; - printf("*** File cache %s:%s ***\n", gSystem->HostName(), - fCacheDir.Data()); - fflush(stdout); - PDB(kCache, 1) { - gSystem->Exec(TString::Format("%s -a %s", kLS, fCacheDir.Data())); - } else { - gSystem->Exec(TString::Format("%s %s", kLS, fCacheDir.Data())); - } - if (IsMaster() && all) - fProof->ShowCache(all); - LogToMaster(); - if (slb) slb->Form("%d %d", type, all); - break; - case TProof::kClearCache: - file = ""; - if ((mess->BufferSize() > mess->Length())) (*mess) >> file; - fCacheLock->Lock(); - if (file.IsNull() || file == "*") { - gSystem->Exec(TString::Format("%s %s/* %s/.*.binversion", kRM, fCacheDir.Data(), fCacheDir.Data())); - } else { - gSystem->Exec(TString::Format("%s %s/%s", kRM, fCacheDir.Data(), file.Data())); - } - fCacheLock->Unlock(); - if (IsMaster()) - fProof->ClearCache(file); - if (slb) slb->Form("%d %s", type, file.Data()); - break; - case TProof::kShowPackages: - (*mess) >> all; - fPackMgr->Show(); - if (IsMaster() && all) - fProof->ShowPackages(all); - LogToMaster(); - if (slb) slb->Form("%d %d", type, all); - break; - case TProof::kClearPackages: - if ((status = fPackMgr->Unload(0)) == 0) { - fPackMgr->Remove(); - if (IsMaster()) - status = fProof->ClearPackages(); - } - if (slb) slb->Form("%d %d", type, status); - break; - case TProof::kClearPackage: - (*mess) >> package; - if ((status = fPackMgr->Unload(package)) == 0) { - fPackMgr->Remove(package); - if (IsMaster()) - status = fProof->ClearPackage(package); - } - if (slb) slb->Form("%d %s %d", type, package.Data(), status); - break; - case TProof::kBuildPackage: - (*mess) >> package; - if ((mess->BufferSize() > mess->Length())) (*mess) >> chkveropt; - - if (!(packmgr = TPackMgr::GetPackMgr(package.Data(), fPackMgr))) { - // Package not found - SendAsynMessage(TString::Format("%s: kBuildPackage: failure locating %s ...", - noth.Data(), package.Data())); - status = -1; - break; - } - fromglobal = (packmgr == fPackMgr) ? kFALSE : kTRUE; - packagedir = packmgr->GetTitle(); - - if (IsMaster() && !fromglobal) { - // Make sure package is available on all slaves, even new ones - TString par; - Int_t xrc = packmgr->GetParPath(package, par); - if (xrc != 0 || fProof->UploadPackage(par) != 0) { - Warning("HandleCache", - "kBuildPackage: problems forwarding package %s to workers", package.Data()); - SendAsynMessage(TString::Format("%s: kBuildPackage: problems forwarding package %s to workers ...", - noth.Data(), package.Data())); - } - } - - if (!status) { - - PDB(kPackage, 1) - Info("HandleCache", - "kBuildPackage: package %s exists and has PROOF-INF directory", package.Data()); - - // Forward build command to slaves, but don't wait for results - if (IsMaster()) - fProof->BuildPackage(package, TProof::kBuildOnSlavesNoWait); - - // Build here - status = packmgr->Build(package.Data(), chkveropt); - } - - if (status) { - // Notify the upper level - SendAsynMessage(TString::Format("%s: failure building %s ... (status: %d)", noth.Data(), package.Data(), status)); - } else { - // collect built results from slaves - if (IsMaster()) - status = fProof->BuildPackage(package, TProof::kCollectBuildResults); - PDB(kPackage, 1) - Info("HandleCache", "package %s successfully built", package.Data()); - } - if (slb) slb->Form("%d %s %d %d", type, package.Data(), status, chkveropt); - break; - - case TProof::kLoadPackage: - (*mess) >> package; - - // Get argument, if any - if ((mess->BufferSize() > mess->Length())) (*mess) >> optls; - // Load the package - if ((status = fPackMgr->Load(package.Data(), optls)) < 0) { - - // Notify the upper level - SendAsynMessage(TString::Format("%s: failure loading %s, args: %p (%d) ...", - noth.Data(), package.Data(), optls, - (optls && optls->GetSize() > 0) ? optls->GetSize() : 0)); - - } else { - - if (IsMaster()) { - if (optls && optls->GetSize() > 0) { - // List argument - status = fProof->LoadPackage(package, kFALSE, optls); - } else { - // No argument - status = fProof->LoadPackage(package); - } - } - - PDB(kPackage, 1) - Info("HandleCache", "package %s successfully loaded", package.Data()); - } - - if (slb) slb->Form("%d %s %d", type, package.Data(), status); - break; - - case TProof::kShowEnabledPackages: - (*mess) >> all; - { TString title("*** Enabled packages ***"); - if (!IsMaster() || all) { - title.Form("*** Enabled packages on %s %s on %s", - (IsMaster()) ? "master" : "worker", - fOrdinal.Data(), gSystem->HostName()); - } - fPackMgr->ShowEnabled(title); - } - if (IsMaster() && all) - fProof->ShowEnabledPackages(all); - LogToMaster(); - if (slb) slb->Form("%d %d", type, all); - break; - case TProof::kShowSubCache: - (*mess) >> all; - if (IsMaster() && all) - fProof->ShowCache(all); - LogToMaster(); - if (slb) slb->Form("%d %d", type, all); - break; - case TProof::kClearSubCache: - file = ""; - if ((mess->BufferSize() > mess->Length())) (*mess) >> file; - if (IsMaster()) - fProof->ClearCache(file); - if (slb) slb->Form("%d %s", type, file.Data()); - break; - case TProof::kShowSubPackages: - (*mess) >> all; - if (IsMaster() && all) - fProof->ShowPackages(all); - LogToMaster(); - if (slb) slb->Form("%d %d", type, all); - break; - case TProof::kDisableSubPackages: - if (IsMaster()) - fProof->DisablePackages(); - if (slb) slb->Form("%d", type); - break; - case TProof::kDisableSubPackage: - (*mess) >> package; - if (IsMaster()) - fProof->DisablePackage(package); - if (slb) slb->Form("%d %s", type, package.Data()); - break; - case TProof::kBuildSubPackage: - (*mess) >> package; - if ((mess->BufferSize() > mess->Length())) (*mess) >> chkveropt; - if (IsMaster()) - fProof->BuildPackage(package, TProof::kBuildAll, chkveropt); - if (slb) slb->Form("%d %s %d", type, package.Data(), chkveropt); - break; - case TProof::kUnloadPackage: - (*mess) >> package; - status = fPackMgr->Unload(package); - if (IsMaster() && status == 0) - status = fProof->UnloadPackage(package); - if (slb) slb->Form("%d %s %d", type, package.Data(), status); - break; - case TProof::kDisablePackage: - (*mess) >> package; - fPackMgr->Remove(package); - if (IsMaster()) - fProof->DisablePackage(package); - if (slb) slb->Form("%d %s", type, package.Data()); - break; - case TProof::kUnloadPackages: - status = fPackMgr->Unload(0); - if (IsMaster() && status == 0) - status = fProof->UnloadPackages(); - if (slb) slb->Form("%d %s %d", type, package.Data(), status); - break; - case TProof::kDisablePackages: - fPackMgr->Remove(); - if (IsMaster()) - fProof->DisablePackages(); - if (slb) slb->Form("%d %s", type, package.Data()); - break; - case TProof::kListEnabledPackages: - msg.Reset(kPROOF_PACKAGE_LIST); - { TList *epl = fPackMgr->GetListOfEnabled(); - msg << type << epl; - fSocket->Send(msg); - epl->SetOwner(); - delete epl; - } - if (slb) slb->Form("%d", type); - break; - case TProof::kListPackages: - { - TList *pack = fPackMgr->GetList(); - msg.Reset(kPROOF_PACKAGE_LIST); - msg << type << pack; - fSocket->Send(msg); - pack->SetOwner(kTRUE); - delete pack; - } - if (slb) slb->Form("%d", type); - break; - case TProof::kLoadMacro: - { - (*mess) >> package; - - // By first forwarding the load command to the unique workers - // and only then loading locally we load/build in parallel - if (IsMaster()) - fProof->Load(package, kFALSE, kTRUE); - - // Atomic action - fCacheLock->Lock(); - - TString originalCwd = gSystem->WorkingDirectory(); - gSystem->ChangeDirectory(fCacheDir.Data()); - - // Load the macro - TString pack(package); - Ssiz_t from = 0; - if ((from = pack.Index(",")) != kNPOS) pack.Remove(from); - Info("HandleCache", "loading macro %s ...", pack.Data()); - gROOT->ProcessLine(TString::Format(".L %s", pack.Data())); - - // Release atomicity - gSystem->ChangeDirectory(originalCwd.Data()); - fCacheLock->Unlock(); - - // Now we collect the result from the unique workers and send the load request - // to the other workers (no compilation) - if (IsMaster()) - fProof->Load(package, kFALSE, kFALSE); - - // Notify the upper level - LogToMaster(); - - if (slb) slb->Form("%d %s", type, package.Data()); - } - break; - default: - Error("HandleCache", "unknown type %d", type); - break; - } - - // We are done - return status; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle here all requests to modify worker lists - -Int_t TProofServ::HandleWorkerLists(TMessage *mess) -{ - PDB(kGlobal, 1) - Info("HandleWorkerLists", "Enter"); - - Int_t type = 0, rc = 0; - TString ord; - - (*mess) >> type; - - switch (type) { - case TProof::kActivateWorker: - (*mess) >> ord; - if (ord != "*" && !ord.BeginsWith(GetOrdinal()) && ord != "restore") break; - if (fProof) { - Int_t nact = fProof->GetListOfActiveSlaves()->GetSize(); - Int_t nactmax = fProof->GetListOfSlaves()->GetSize() - - fProof->GetListOfBadSlaves()->GetSize(); - if (nact < nactmax || !IsEndMaster()) { - Int_t nwc = fProof->ActivateWorker(ord); - Int_t nactnew = fProof->GetListOfActiveSlaves()->GetSize(); - if (ord == "*") { - if (nactnew == nactmax) { - PDB(kGlobal, 1) Info("HandleWorkerList", "all workers (re-)activated"); - } else { - if (IsEndMaster()) - PDB(kGlobal, 1) Info("HandleWorkerList", "%d workers could not be (re-)activated", nactmax - nactnew); - } - } else if (ord == "restore") { - if (nwc > 0) { - PDB(kGlobal, 1) Info("HandleWorkerList","active worker(s) restored"); - } else { - Error("HandleWorkerList", "some active worker(s) could not be restored; check logs"); - } - } else { - if (nactnew == (nact + nwc)) { - if (nwc > 0) - PDB(kGlobal, 1) Info("HandleWorkerList","worker(s) %s (re-)activated", ord.Data()); - } else { - if (nwc != -2 && IsEndMaster()) { - Error("HandleWorkerList", "some worker(s) could not be (re-)activated;" - " # of actives: %d --> %d (nwc: %d)", - nact, nactnew, nwc); - } - rc = (nwc < 0) ? nwc : -1; - } - } - } else { - PDB(kGlobal, 1) Info("HandleWorkerList","all workers are already active"); - } - } else { - Warning("HandleWorkerList","undefined PROOF session: protocol error?"); - } - break; - case TProof::kDeactivateWorker: - (*mess) >> ord; - if (ord != "*" && !ord.BeginsWith(GetOrdinal()) && ord != "restore") break; - if (fProof) { - Int_t nact = fProof->GetListOfActiveSlaves()->GetSize(); - if (nact > 0) { - Int_t nwc = fProof->DeactivateWorker(ord); - Int_t nactnew = fProof->GetListOfActiveSlaves()->GetSize(); - if (ord == "*") { - if (nactnew == 0) { - PDB(kGlobal, 1) Info("HandleWorkerList","all workers deactivated"); - } else { - if (IsEndMaster()) - PDB(kGlobal, 1) Info("HandleWorkerList","%d workers could not be deactivated", nactnew); - } - } else { - if (nactnew == (nact - nwc)) { - if (nwc > 0) - PDB(kGlobal, 1) Info("HandleWorkerList","worker(s) %s deactivated", ord.Data()); - } else { - if (nwc != -2 && IsEndMaster()) { - Error("HandleWorkerList", "some worker(s) could not be deactivated:" - " # of actives: %d --> %d (nwc: %d)", - nact, nactnew, nwc); - } - rc = (nwc < 0) ? nwc : -1; - } - } - } else { - PDB(kGlobal, 1) Info("HandleWorkerList","all workers are already inactive"); - } - } else { - Warning("HandleWorkerList","undefined PROOF session: protocol error?"); - } - break; - default: - Warning("HandleWorkerList","unknown action type (%d)", type); - rc = -1; - } - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get list of workers to be used from now on. -/// The list must be provided by the caller. - -TProofServ::EQueryAction TProofServ::GetWorkers(TList *workers, - Int_t & /* prioritychange */, - Bool_t /* resume */) -{ - // Parse the config file - TProofResourcesStatic *resources = - new TProofResourcesStatic(fConfDir, fConfFile); - fConfFile = resources->GetFileName(); // Update the global file name (with path) - PDB(kGlobal,1) - Info("GetWorkers", "using PROOF config file: %s", fConfFile.Data()); - - // Get the master - TProofNodeInfo *master = resources->GetMaster(); - if (!master) { - PDB(kAll,1) - Info("GetWorkers", - "no appropriate master line found in %s", fConfFile.Data()); - return kQueryStop; - } else { - // Set image if not yet done and available - if (fImage.IsNull() && strlen(master->GetImage()) > 0) - fImage = master->GetImage(); - } - - // Fill submaster or worker list - if (workers) { - if (resources->GetSubmasters() && resources->GetSubmasters()->GetSize() > 0) { - PDB(kAll,1) - resources->GetSubmasters()->Print(); - TProofNodeInfo *ni = 0; - TIter nw(resources->GetSubmasters()); - while ((ni = (TProofNodeInfo *) nw())) - workers->Add(new TProofNodeInfo(*ni)); - } else if (resources->GetWorkers() && resources->GetWorkers()->GetSize() > 0) { - PDB(kAll,1) - resources->GetWorkers()->Print(); - TProofNodeInfo *ni = 0; - TIter nw(resources->GetWorkers()); - while ((ni = (TProofNodeInfo *) nw())) - workers->Add(new TProofNodeInfo(*ni)); - } - } - - // We are done - return kQueryOK; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the file stream where to log (default stderr). -/// If ferr == 0 the default is restored. -/// Returns current setting. - -FILE *TProofServ::SetErrorHandlerFile(FILE *ferr) -{ - FILE *oldferr = fgErrorHandlerFile; - fgErrorHandlerFile = (ferr) ? ferr : stderr; - return oldferr; -} - -//////////////////////////////////////////////////////////////////////////////// -/// The PROOF error handler function. It prints the message on fgErrorHandlerFile and -/// if abort is set it aborts the application. - -void TProofServ::ErrorHandler(Int_t level, Bool_t abort, const char *location, - const char *msg) -{ - if (gErrorIgnoreLevel == kUnset) { - gErrorIgnoreLevel = 0; - if (gEnv) { - TString lvl = gEnv->GetValue("Root.ErrorIgnoreLevel", "Print"); - if (!lvl.CompareTo("Print", TString::kIgnoreCase)) - gErrorIgnoreLevel = kPrint; - else if (!lvl.CompareTo("Info", TString::kIgnoreCase)) - gErrorIgnoreLevel = kInfo; - else if (!lvl.CompareTo("Warning", TString::kIgnoreCase)) - gErrorIgnoreLevel = kWarning; - else if (!lvl.CompareTo("Error", TString::kIgnoreCase)) - gErrorIgnoreLevel = kError; - else if (!lvl.CompareTo("Break", TString::kIgnoreCase)) - gErrorIgnoreLevel = kBreak; - else if (!lvl.CompareTo("SysError", TString::kIgnoreCase)) - gErrorIgnoreLevel = kSysError; - else if (!lvl.CompareTo("Fatal", TString::kIgnoreCase)) - gErrorIgnoreLevel = kFatal; - } - } - - if (level < gErrorIgnoreLevel) - return; - - // Always communicate errors via SendLogFile - if (level >= kError && gProofServ) - gProofServ->LogToMaster(); - - Bool_t tosyslog = (fgLogToSysLog > 2) ? kTRUE : kFALSE; - - const char *type = 0; - ELogLevel loglevel = kLogInfo; - - Int_t ipos = (location) ? strlen(location) : 0; - - if (level >= kPrint) { - loglevel = kLogInfo; - type = "Print"; - } - if (level >= kInfo) { - loglevel = kLogInfo; - char *ps = location ? (char *) strrchr(location, '|') : (char *)0; - if (ps) { - ipos = (int)(ps - (char *)location); - type = "SvcMsg"; - } else { - type = "Info"; - } - } - if (level >= kWarning) { - loglevel = kLogWarning; - type = "Warning"; - } - if (level >= kError) { - loglevel = kLogErr; - type = "Error"; - } - if (level >= kBreak) { - loglevel = kLogErr; - type = "*** Break ***"; - } - if (level >= kSysError) { - loglevel = kLogErr; - type = "SysError"; - } - if (level >= kFatal) { - loglevel = kLogErr; - type = "Fatal"; - } - - - TString buf; - - // Time stamp - TTimeStamp ts; - TString st(ts.AsString("lc"),19); - - if (!location || ipos == 0 || - (level >= kPrint && level < kInfo) || - (level >= kBreak && level < kSysError)) { - fprintf(fgErrorHandlerFile, "%s %5d %s | %s: %s\n", st(11,8).Data(), - gSystem->GetPid(), - (gProofServ ? gProofServ->GetPrefix() : "proof"), - type, msg); - if (tosyslog) - buf.Form("%s: %s:%s", fgSysLogEntity.Data(), type, msg); - } else { - fprintf(fgErrorHandlerFile, "%s %5d %s | %s in <%.*s>: %s\n", st(11,8).Data(), - gSystem->GetPid(), - (gProofServ ? gProofServ->GetPrefix() : "proof"), - type, ipos, location, msg); - if (tosyslog) - buf.Form("%s: %s:<%.*s>: %s", fgSysLogEntity.Data(), type, ipos, location, msg); - } - fflush(fgErrorHandlerFile); - - if (tosyslog) - gSystem->Syslog(loglevel, buf); - -#ifdef __APPLE__ - if (__crashreporter_info__) - delete [] __crashreporter_info__; - __crashreporter_info__ = StrDup(buf); -#endif - - if (abort) { - - static Bool_t recursive = kFALSE; - - if (gProofServ != 0 && !recursive) { - recursive = kTRUE; - if (gProofServ->GetSocket()) gProofServ->GetSocket()->Send(kPROOF_FATAL); - recursive = kFALSE; - } - - fprintf(fgErrorHandlerFile, "aborting\n"); - fflush(fgErrorHandlerFile); - gSystem->StackTrace(); - gSystem->Abort(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make player instance. - -void TProofServ::MakePlayer() -{ - TVirtualProofPlayer *p = 0; - - // Cleanup first - DeletePlayer(); - - if (IsParallel()) { - // remote mode - p = fProof->MakePlayer(); - } else { - // slave or sequential mode - p = TVirtualProofPlayer::Create("slave", 0, fSocket); - if (IsMaster()) - fProof->SetPlayer(p); - } - - // set player - fPlayer = p; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete player instance. - -void TProofServ::DeletePlayer() -{ - if (IsMaster()) { - PDB(kGlobal, 1) { - fCompute.Stop(); - Printf(" +++ Latest processing times: %f s (CPU: %f s)", - fCompute.RealTime(), fCompute.CpuTime()); - } - if (fProof) fProof->SetPlayer(0); - } else { - SafeDelete(fPlayer); - } - fPlayer = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the processing priority for the group the user belongs too. This -/// priority is a number (0 - 100) determined by a scheduler (third -/// party process) based on some basic priority the group has, e.g. -/// we might want to give users in a specific group (e.g. promptana) -/// a higher priority than users in other groups, and on the analysis -/// of historical logging data (i.e. usage of CPU by the group in a -/// previous time slot, as recorded in TPerfStats::WriteQueryLog()). -/// -/// Currently the group priority is obtained by a query in a SQL DB -/// table proofpriority, which has the format: -/// CREATE TABLE proofpriority ( -/// id INT NOT NULL PRIMARY KEY AUTO_INCREMENT, -/// group VARCHAR(32) NOT NULL, -/// priority INT -///) - -Int_t TProofServ::GetPriority() -{ - TString sqlserv = gEnv->GetValue("ProofServ.QueryLogDB",""); - TString sqluser = gEnv->GetValue("ProofServ.QueryLogUser",""); - TString sqlpass = gEnv->GetValue("ProofServ.QueryLogPasswd",""); - - Int_t priority = 100; - - if (sqlserv == "") - return priority; - - TString sql; - sql.Form("SELECT priority WHERE group='%s' FROM proofpriority", fGroup.Data()); - - // open connection to SQL server - TSQLServer *db = TSQLServer::Connect(sqlserv, sqluser, sqlpass); - - if (!db || db->IsZombie()) { - Error("GetPriority", "failed to connect to SQL server %s as %s %s", - sqlserv.Data(), sqluser.Data(), sqlpass.Data()); - printf("%s\n", sql.Data()); - } else { - TSQLResult *res = db->Query(sql); - - if (!res) { - Error("GetPriority", "query into proofpriority failed"); - Printf("%s", sql.Data()); - } else { - TSQLRow *row = res->Next(); // first row is header - if (row) { - priority = atoi(row->GetField(0)); - delete row; - } else { - Error("GetPriority", "first row is header is NULL"); - } - } - delete res; - } - delete db; - - return priority; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send an asychronous message to the master / client . -/// Masters will forward up the message to the client. -/// The client prints 'msg' of stderr and adds a '\\n'/'\\r' depending on -/// 'lf' being kTRUE (default) or kFALSE. -/// Returns the return value from TSocket::Send(TMessage &) . - -void TProofServ::SendAsynMessage(const char *msg, Bool_t lf) -{ - static TMessage m(kPROOF_MESSAGE); - - // To leave a track in the output file ... if requested - // (clients will be notified twice) - PDB(kAsyn,1) - Info("SendAsynMessage","%s", (msg ? msg : "(null)")); - - if (fSocket && msg) { - m.Reset(kPROOF_MESSAGE); - m << TString(msg) << lf; - if (fSocket->Send(m) <= 0) - Warning("SendAsynMessage", "could not send message '%s'", msg); - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reposition the read pointer in the log file to the very end. -/// This allows to "hide" useful debug messages during normal operations -/// while preserving the possibility to have them in case of problems. - -void TProofServ::FlushLogFile() -{ - off_t lend = lseek(fileno(stdout), (off_t)0, SEEK_END); - if (lend >= 0) lseek(fLogFileDes, lend, SEEK_SET); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Truncate the log file to the 80% of the required max size if this -/// is set. - -void TProofServ::TruncateLogFile() -{ -#ifndef WIN32 - TString emsg; - if (fLogFileMaxSize > 0 && fLogFileDes > 0) { - fflush(stdout); - struct stat st; - if (fstat(fLogFileDes, &st) == 0) { - if (st.st_size >= fLogFileMaxSize) { - off_t truncsz = (off_t) (( fLogFileMaxSize * 80 ) / 100 ); - if (truncsz < 100) { - emsg.Form("+++ WARNING +++: %s: requested truncate size too small" - " (%lld,%lld) - ignore ", fPrefix.Data(), (Long64_t) truncsz, fLogFileMaxSize); - SendAsynMessage(emsg.Data()); - return; - } - TSystem::ResetErrno(); - while (ftruncate(fileno(stdout), truncsz) != 0 && - (TSystem::GetErrno() == EINTR)) { - TSystem::ResetErrno(); - } - if (TSystem::GetErrno() > 0) { - Error("TruncateLogFile", "truncating to %lld bytes; file size is %lld bytes (errno: %d)", - (Long64_t)truncsz, (Long64_t)st.st_size, TSystem::GetErrno()); - emsg.Form("+++ WARNING +++: %s: problems truncating log file to %lld bytes; file size is %lld bytes" - " (errno: %d)", fPrefix.Data(), (Long64_t)truncsz, (Long64_t)st.st_size, TSystem::GetErrno()); - SendAsynMessage(emsg.Data()); - } else { - Info("TruncateLogFile", "file truncated to %lld bytes (80%% of %lld); file size was %lld bytes ", - (Long64_t)truncsz, fLogFileMaxSize, (Long64_t)st.st_size); - emsg.Form("+++ WARNING +++: %s: log file truncated to %lld bytes (80%% of %lld)", - fPrefix.Data(), (Long64_t)truncsz, fLogFileMaxSize); - SendAsynMessage(emsg.Data()); - } - } - } else { - emsg.Form("+++ WARNING +++: %s: could not stat log file descriptor" - " for truncation (errno: %d)", fPrefix.Data(), TSystem::GetErrno()); - SendAsynMessage(emsg.Data()); - } - } -#endif -} - -//////////////////////////////////////////////////////////////////////////////// -/// Exception handler: we do not try to recover here, just exit. - -void TProofServ::HandleException(Int_t sig) -{ - Error("HandleException", "caugth exception triggered by signal '%d' %s %lld", - sig, fgLastMsg.Data(), fgLastEntry); - // Description - TString emsg; - emsg.Form("%s: caught exception triggered by signal '%d' %s %lld", - GetOrdinal(), sig, fgLastMsg.Data(), fgLastEntry); - // Try to warn the user - SendAsynMessage(emsg.Data()); - - gSystem->Exit(sig); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle here requests about datasets. - -Int_t TProofServ::HandleDataSets(TMessage *mess, TString *slb) -{ - if (gDebug > 0) - Info("HandleDataSets", "enter"); - - // We need a dataset manager - if (!fDataSetManager) { - Warning("HandleDataSets", "no data manager is available to fullfil the request"); - return -1; - } - - // Used in most cases - TString dsUser, dsGroup, dsName, dsTree, uri, opt; - Int_t rc = 0; - - // Invalid characters in dataset URI - TPMERegexp reInvalid("[^A-Za-z0-9._-]"); // from ParseUri - - // Message type - Int_t type = 0; - (*mess) >> type; - - switch (type) { - case TProof::kCheckDataSetName: - // - // Check whether this dataset exist - { - (*mess) >> uri; - if (slb) slb->Form("%d %s", type, uri.Data()); - if (fDataSetManager->ExistsDataSet(uri)) - // Dataset name does exist - return -1; - } - break; - case TProof::kRegisterDataSet: - // list size must be above 0 - { - if (fDataSetManager->TestBit(TDataSetManager::kAllowRegister)) { - (*mess) >> uri; - (*mess) >> opt; - if (slb) slb->Form("%d %s %s", type, uri.Data(), opt.Data()); - // Extract the list - TFileCollection *dataSet = - dynamic_cast ((mess->ReadObject(TFileCollection::Class()))); - if (!dataSet || dataSet->GetList()->GetSize() == 0) { - Error("HandleDataSets", "can not save an empty list."); - return -1; - } - // Register the dataset (quota checks are done inside here) - rc = fDataSetManager->RegisterDataSet(uri, dataSet, opt); - delete dataSet; - return rc; - } else { - Info("HandleDataSets", "dataset registration not allowed"); - if (slb) slb->Form("%d notallowed", type); - return -1; - } - } - break; - - case TProof::kRequestStaging: - { - (*mess) >> uri; // TString - - if (!fDataSetStgRepo) { - Error("HandleDataSets", - "no dataset staging request repository available"); - return -1; - } - - // Transform input URI in a valid dataset name - TString validUri = uri; - while (reInvalid.Substitute(validUri, "_")) {} - - // Check if dataset exists beforehand: if it does, staging has - // already been requested - if (fDataSetStgRepo->ExistsDataSet(validUri.Data())) { - Warning("HandleDataSets", "staging of %s already requested", - uri.Data()); - return -1; - } - - // Try to get dataset from current manager - TFileCollection *fc = fDataSetManager->GetDataSet(uri.Data()); - if (!fc || (fc->GetNFiles() == 0)) { - Error("HandleDataSets", "empty dataset or no dataset returned"); - if (fc) delete fc; - return -1; - } - - // Reset all staged bits and remove unnecessary URLs (all but last) - TIter it(fc->GetList()); - TFileInfo *fi; - while ((fi = dynamic_cast(it.Next()))) { - fi->ResetBit(TFileInfo::kStaged); - Int_t nToErase = fi->GetNUrls() - 1; - for (Int_t i=0; iRemoveUrlAt(0); - } - - fc->Update(); // absolutely necessary - - // Save request - fDataSetStgRepo->ParseUri(validUri, &dsGroup, &dsUser, &dsName); - if (fDataSetStgRepo->WriteDataSet(dsGroup, dsUser, - dsName, fc) == 0) { - // Error, can't save dataset - Error("HandleDataSets", - "can't register staging request for %s", uri.Data()); - delete fc; - return -1; - } - - Info("HandleDataSets", - "Staging request registered for %s", uri.Data()); - - delete fc; - return 0; // success (-1 == failure) - } - break; - - case TProof::kStagingStatus: - { - if (!fDataSetStgRepo) { - Error("HandleDataSets", - "no dataset staging request repository available"); - return -1; - } - - (*mess) >> uri; // TString - - // Transform URI in a valid dataset name - while (reInvalid.Substitute(uri, "_")) {} - - // Get the list - TFileCollection *fc = fDataSetStgRepo->GetDataSet(uri.Data()); - if (fc) { - fSocket->SendObject(fc, kMESS_OK); - delete fc; - return 0; - } - else { - // No such dataset: not an error, but don't send message - Info("HandleDataSets", "no pending staging request for %s", - uri.Data()); - return 0; - } - } - break; - - case TProof::kCancelStaging: - { - if (!fDataSetStgRepo) { - Error("HandleDataSets", - "no dataset staging request repository available"); - return -1; - } - - (*mess) >> uri; - - // Transform URI in a valid dataset name - while (reInvalid.Substitute(uri, "_")) {} - - if (!fDataSetStgRepo->RemoveDataSet(uri.Data())) - return -1; // failure - - return 0; // success - } - break; - - case TProof::kShowDataSets: - { - (*mess) >> uri >> opt; - if (slb) slb->Form("%d %s %s", type, uri.Data(), opt.Data()); - // Show content - fDataSetManager->ShowDataSets(uri, opt); - } - break; - - case TProof::kGetDataSets: - { - (*mess) >> uri >> opt; - if (slb) slb->Form("%d %s %s", type, uri.Data(), opt.Data()); - // Get the datasets and fill a map - UInt_t omsk = (UInt_t)TDataSetManager::kExport; - Ssiz_t kLite = opt.Index(":lite:", 0, TString::kIgnoreCase); - if (kLite != kNPOS) { - omsk |= (UInt_t)TDataSetManager::kReadShort; - opt.Remove(kLite, strlen(":lite:")); - } - TMap *returnMap = fDataSetManager->GetDataSets(uri, omsk); - // If defines, option gives the name of a server for which to extract the information - if (returnMap && !opt.IsNull()) { - // The return map will be in the form --> - TMap *rmap = new TMap; - TObject *k = 0; - TFileCollection *fc = 0, *xfc = 0; - TIter nxd(returnMap); - while ((k = nxd()) && (fc = (TFileCollection *) returnMap->GetValue(k))) { - // Get subset on specified server, if any - if ((xfc = fc->GetFilesOnServer(opt.Data()))) { - rmap->Add(new TObjString(k->GetName()), xfc); - } - } - returnMap->DeleteAll(); - if (rmap->GetSize() > 0) { - returnMap = rmap; - } else { - Info("HandleDataSets", "no dataset found on server '%s'", opt.Data()); - delete rmap; - returnMap = 0; - } - } - if (returnMap) { - // Send them back - fSocket->SendObject(returnMap, kMESS_OK); - returnMap->DeleteAll(); - } else { - // Failure - return -1; - } - } - break; - case TProof::kGetDataSet: - { - (*mess) >> uri >> opt; - if (slb) slb->Form("%d %s %s", type, uri.Data(), opt.Data()); - // Get the list - TFileCollection *fileList = fDataSetManager->GetDataSet(uri,opt); - if (fileList) { - fSocket->SendObject(fileList, kMESS_OK); - delete fileList; - } else { - // Failure - return -1; - } - } - break; - case TProof::kRemoveDataSet: - { - if (fDataSetManager->TestBit(TDataSetManager::kAllowRegister)) { - (*mess) >> uri; - if (slb) slb->Form("%d %s", type, uri.Data()); - if (!fDataSetManager->RemoveDataSet(uri)) { - // Failure - return -1; - } - } else { - Info("HandleDataSets", "dataset creation / removal not allowed"); - if (slb) slb->Form("%d notallowed", type); - return -1; - } - } - break; - case TProof::kVerifyDataSet: - { - if (fDataSetManager->TestBit(TDataSetManager::kAllowVerify)) { - (*mess) >> uri >> opt; - if (slb) slb->Form("%d %s %s", type, uri.Data(), opt.Data()); - TProofServLogHandlerGuard hg(fLogFile, fSocket); - rc = fDataSetManager->ScanDataSet(uri, opt); - // TODO: verify in parallel: - // - dataset = GetDataSet(uri) - // - TList flist; TDataSetManager::ScanDataSet(dataset, ..., &flist) - // - fPlayer->Process( ... flist ...) // needs to be developed - // - dataset->Integrate(flist) (perhaps automatic; flist object owned by dataset) - // - RegisterDataSet(uri, dataset, "OT") - } else { - Info("HandleDataSets", "dataset verification not allowed"); - return -1; - } - } - break; - case TProof::kGetQuota: - { - if (fDataSetManager->TestBit(TDataSetManager::kCheckQuota)) { - if (slb) slb->Form("%d", type); - TMap *groupQuotaMap = fDataSetManager->GetGroupQuotaMap(); - if (groupQuotaMap) { - // Send result - fSocket->SendObject(groupQuotaMap, kMESS_OK); - } else { - return -1; - } - } else { - Info("HandleDataSets", "quota control disabled"); - if (slb) slb->Form("%d disabled", type); - return -1; - } - } - break; - case TProof::kShowQuota: - { - if (fDataSetManager->TestBit(TDataSetManager::kCheckQuota)) { - if (slb) slb->Form("%d", type); - (*mess) >> opt; - // Display quota information - fDataSetManager->ShowQuota(opt); - } else { - Info("HandleDataSets", "quota control disabled"); - if (slb) slb->Form("%d disabled", type); - } - } - break; - case TProof::kSetDefaultTreeName: - { - if (fDataSetManager->TestBit(TDataSetManager::kAllowRegister)) { - (*mess) >> uri; - if (slb) slb->Form("%d %s", type, uri.Data()); - rc = fDataSetManager->ScanDataSet(uri, (UInt_t)TDataSetManager::kSetDefaultTree); - } else { - Info("HandleDataSets", "kSetDefaultTreeName: modification of dataset info not allowed"); - if (slb) slb->Form("%d notallowed", type); - return -1; - } - } - break; - case TProof::kCache: - { - (*mess) >> uri >> opt; - if (slb) slb->Form("%d %s %s", type, uri.Data(), opt.Data()); - if (opt == "show") { - // Show cache content - fDataSetManager->ShowCache(uri); - } else if (opt == "clear") { - // Clear cache content - fDataSetManager->ClearCache(uri); - } else { - Error("HandleDataSets", "kCache: unknown action: %s", opt.Data()); - } - } - break; - default: - rc = -1; - Error("HandleDataSets", "unknown type %d", type); - break; - } - - // We are done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle a message of type kPROOF_SUBMERGER - -void TProofServ::HandleSubmerger(TMessage *mess) -{ - // Message type - Int_t type = 0; - (*mess) >> type; - - TString msg; - switch (type) { - case TProof::kOutputSize: - break; - - case TProof::kSendOutput: - { - Bool_t deleteplayer = kTRUE; - if (!IsMaster()) { - if (fMergingMonitor) { - Info("HandleSubmerger", "kSendOutput: interrupting ..."); - fMergingMonitor->Interrupt(); - } - if (fMergingSocket) { - if (fMergingMonitor) fMergingMonitor->Remove(fMergingSocket); - fMergingSocket->Close(); - SafeDelete(fMergingSocket); - } - - TString name; - Int_t port = 0; - Int_t merger_id = -1; - (*mess) >> merger_id >> name >> port; - PDB(kSubmerger, 1) - Info("HandleSubmerger","worker %s redirected to merger #%d %s:%d", fOrdinal.Data(), merger_id, name.Data(), port); - - TSocket *t = 0; - if (name.Length() > 0 && port > 0 && (t = new TSocket(name, port)) && t->IsValid()) { - - PDB(kSubmerger, 2) Info("HandleSubmerger", - "kSendOutput: worker asked for sending output to merger #%d %s:%d", - merger_id, name.Data(), port); - - if (SendResults(t, fPlayer->GetOutputList()) != 0) { - msg.Form("worker %s cannot send results to merger #%d at %s:%d", GetPrefix(), merger_id, name.Data(), port); - PDB(kSubmerger, 2) Info("HandleSubmerger", - "kSendOutput: %s - inform the master", msg.Data()); - SendAsynMessage(msg); - // Results not send - TMessage answ(kPROOF_SUBMERGER); - answ << Int_t(TProof::kMergerDown); - answ << merger_id; - fSocket->Send(answ); - } else { - // Worker informs master that it had sent its output to the merger - TMessage answ(kPROOF_SUBMERGER); - answ << Int_t(TProof::kOutputSent); - answ << merger_id; - fSocket->Send(answ); - - PDB(kSubmerger, 2) Info("HandleSubmerger", "kSendOutput: worker sent its output"); - fSocket->Send(kPROOF_SETIDLE); - SetIdle(kTRUE); - SendLogFile(); - } - } else { - - if (name == "master") { - PDB(kSubmerger, 2) Info("HandleSubmerger", - "kSendOutput: worker was asked for sending output to master"); - if (SendResults(fSocket, fPlayer->GetOutputList()) != 0) - Warning("HandleSubmerger", "problems sending output list"); - // Signal the master that we are idle - fSocket->Send(kPROOF_SETIDLE); - SetIdle(kTRUE); - SendLogFile(); - - } else if (!t || !(t->IsValid())) { - msg.Form("worker %s could not open a valid socket to merger #%d at %s:%d", - GetPrefix(), merger_id, name.Data(), port); - PDB(kSubmerger, 2) Info("HandleSubmerger", - "kSendOutput: %s - inform the master", msg.Data()); - SendAsynMessage(msg); - // Results not send - TMessage answ(kPROOF_SUBMERGER); - answ << Int_t(TProof::kMergerDown); - answ << merger_id; - fSocket->Send(answ); - deleteplayer = kFALSE; - } - - SafeDelete(t); - - } - - } else { - Error("HandleSubmerger", "kSendOutput: received not on worker"); - } - - // Cleanup - if (deleteplayer) DeletePlayer(); - } - break; - case TProof::kBeMerger: - { - Bool_t deleteplayer = kTRUE; - if (!IsMaster()) { - Int_t merger_id = -1; - //Int_t merger_port = 0; - Int_t connections = 0; - (*mess) >> merger_id >> connections; - PDB(kSubmerger, 2) - Info("HandleSubmerger", "worker %s established as merger", fOrdinal.Data()); - - PDB(kSubmerger, 2) - Info("HandleSubmerger", - "kBeMerger: worker asked for being merger #%d for %d connections", - merger_id, connections); - - TVirtualProofPlayer *mergerPlayer = TVirtualProofPlayer::Create("remote",fProof,0); - - if (mergerPlayer) { - PDB(kSubmerger, 2) Info("HandleSubmerger", - "kBeMerger: mergerPlayer created (%p) ", mergerPlayer); - - // This may be used internally - mergerPlayer->SetBit(TVirtualProofPlayer::kIsSubmerger); - - // Accept results from assigned workers - if (AcceptResults(connections, mergerPlayer)) { - PDB(kSubmerger, 2) - Info("HandleSubmerger", "kBeMerger: all outputs from workers accepted"); - - PDB(kSubmerger, 2) - Info("","adding own output to the list on %s", fOrdinal.Data()); - - // Add own results to the output list. - // On workers the player does not own the output list, which is owned - // by the selector and deleted in there - // On workers the player does not own the output list, which is owned - // by the selector and deleted in there - TIter nxo(fPlayer->GetOutputList()); - TObject * o = 0; - while ((o = nxo())) { - if ((mergerPlayer->AddOutputObject(o) != 1)) { - // Remove the object if it has not been merged: it is owned - // now by the merger player (in its output list) - if (fPlayer->GetOutputList()) { - PDB(kSubmerger, 2) - Info("HandleSocketInput", "removing merged object (%p)", o); - fPlayer->GetOutputList()->Remove(o); - } - } - } - PDB(kSubmerger, 2) Info("HandleSubmerger","kBeMerger: own outputs added"); - PDB(kSubmerger, 2) Info("HandleSubmerger","starting delayed merging on %s", fOrdinal.Data()); - - // Delayed merging if neccessary - mergerPlayer->MergeOutput(kTRUE); - - PDB(kSubmerger, 2) mergerPlayer->GetOutputList()->Print("all"); - - PDB(kSubmerger, 2) Info("HandleSubmerger", "delayed merging on %s finished ", fOrdinal.Data()); - PDB(kSubmerger, 2) Info("HandleSubmerger", "%s sending results to master ", fOrdinal.Data()); - // Send merged results to master - if (SendResults(fSocket, mergerPlayer->GetOutputList()) != 0) - Warning("HandleSubmerger","kBeMerger: problems sending output list"); - if (mergerPlayer->GetOutputList()) - mergerPlayer->GetOutputList()->SetOwner(kTRUE); - - PDB(kSubmerger, 2) Info("HandleSubmerger","kBeMerger: results sent to master"); - // Signal the master that we are idle - fSocket->Send(kPROOF_SETIDLE); - SetIdle(kTRUE); - SendLogFile(); - } else { - // Results from all assigned workers not accepted - TMessage answ(kPROOF_SUBMERGER); - answ << Int_t(TProof::kMergerDown); - answ << merger_id; - fSocket->Send(answ); - deleteplayer = kFALSE; - } - // Reset - SafeDelete(mergerPlayer); - - } else { - Warning("HandleSubmerger","kBeMerger: problems craeting the merger player!"); - // Results from all assigned workers not accepted - TMessage answ(kPROOF_SUBMERGER); - answ << Int_t(TProof::kMergerDown); - answ << merger_id; - fSocket->Send(answ); - deleteplayer = kFALSE; - } - } else { - Error("HandleSubmerger","kSendOutput: received not on worker"); - } - - // Cleanup - if (deleteplayer) DeletePlayer(); - } - break; - - case TProof::kMergerDown: - break; - - case TProof::kStopMerging: - { - // Received only in case of forced termination of merger by master - PDB(kSubmerger, 2) Info("HandleSubmerger", "kStopMerging"); - if (fMergingMonitor) { - Info("HandleSubmerger", "kStopMerging: interrupting ..."); - fMergingMonitor->Interrupt(); - } - } - break; - - case TProof::kOutputSent: - break; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cloning itself via fork. Not implemented - -void TProofServ::HandleFork(TMessage *) -{ - Info("HandleFork", "fork cloning not implemented"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fork a child. -/// If successful, return 0 in the child process and the child pid in the parent -/// process. The child pid is registered for reaping. -/// Return <0 in the parent process in case of failure. - -Int_t TProofServ::Fork() -{ -#ifndef WIN32 - // Fork - pid_t pid; - if ((pid = fork()) < 0) { - Error("Fork", "failed to fork"); - return pid; - } - - // Nothing else to do in the child - if (!pid) return pid; - - // Make sure that the reaper timer is started - if (!fReaperTimer) { - fReaperTimer = new TReaperTimer(1000); - fReaperTimer->Start(-1); - } - - // Register the new child - fReaperTimer->AddPid(pid); - - // Done - return pid; -#else - Warning("Fork", "Functionality not provided under windows"); - return -1; -#endif -} - -//////////////////////////////////////////////////////////////////////////////// -/// Replace \, \, \, \, \, \, \, -/// \ and \ placeholders in fname. -/// Here, \ is the root version in integer form, e.g. 53403, and -/// \ a string includign version, architecture and compiler version, -/// e.g. '53403_linuxx8664gcc_gcc46' . - -void TProofServ::ResolveKeywords(TString &fname, const char *path) -{ - // Replace , if any - if (fname.Contains("")) { - if (gProofServ && gProofServ->GetUser() && strlen(gProofServ->GetUser())) { - fname.ReplaceAll("", gProofServ->GetUser()); - } else if (gProof && gProof->GetUser() && strlen(gProof->GetUser())) { - fname.ReplaceAll("", gProof->GetUser()); - } else { - fname.ReplaceAll("", "nouser"); - } - } - // Replace , if any - if (fname.Contains("")) { - if (gProofServ && gProofServ->GetUser() && strlen(gProofServ->GetUser())) { - TString u(gProofServ->GetUser()[0]); - fname.ReplaceAll("", u); - } else if (gProof && gProof->GetUser() && strlen(gProof->GetUser())) { - TString u(gProof->GetUser()[0]); - fname.ReplaceAll("", u); - } else { - fname.ReplaceAll("", "n"); - } - } - // Replace , if any - if (fname.Contains("")) { - if (gProofServ && gProofServ->GetGroup() && strlen(gProofServ->GetGroup())) { - fname.ReplaceAll("", gProofServ->GetGroup()); - } else if (gProof && gProof->GetGroup() && strlen(gProof->GetGroup())) { - fname.ReplaceAll("", gProof->GetGroup()); - } else { - fname.ReplaceAll("", "default"); - } - } - // Replace , if any - if (fname.Contains("")) { - if (gProofServ && gProofServ->GetSessionTag() && strlen(gProofServ->GetSessionTag())) { - fname.ReplaceAll("", gProofServ->GetSessionTag()); - } else if (gProof && gProof->GetSessionTag() && strlen(gProof->GetSessionTag())) { - fname.ReplaceAll("", gProof->GetSessionTag()); - } else { - ::Warning("TProofServ::ResolveKeywords", "session tag undefined: ignoring"); - } - } - // Replace , if any - if (fname.Contains("")) { - if (gProofServ && gProofServ->GetOrdinal() && strlen(gProofServ->GetOrdinal())) - fname.ReplaceAll("", gProofServ->GetOrdinal()); - else - ::Warning("TProofServ::ResolveKeywords", "ordinal number undefined: ignoring"); - } - // Replace , if any - if (fname.Contains("")) { - if (gProofServ && gProofServ->GetQuerySeqNum() && gProofServ->GetQuerySeqNum() > 0) - fname.ReplaceAll("", TString::Format("%d", gProofServ->GetQuerySeqNum()).Data()); - else - ::Warning("TProofServ::ResolveKeywords", "query seqeuntial number undefined: ignoring"); - } - // Replace , if any - if (fname.Contains("") && path && strlen(path) > 0) { - fname.ReplaceAll("", path); - } - // Replace , if any - if (fname.Contains("")) { - TString v = TString::Format("%d", gROOT->GetVersionInt()); - fname.ReplaceAll("", v); - } - // Replace , if any - if (fname.Contains("")) { - TString b = TString::Format("%d_%s_%s", gROOT->GetVersionInt(), gSystem->GetBuildArch(), - gSystem->GetBuildCompilerVersion()); - fname.ReplaceAll("", b); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the status of this session: -/// 0 idle -/// 1 running -/// 2 being terminated (currently unused) -/// 3 queued -/// 4 idle timed-out (not set in here but in TIdleTOTimer::Notify) -/// This is typically run in the reader thread, so access needs to be protected - -Int_t TProofServ::GetSessionStatus() -{ - std::lock_guard lock(fQMtx); - Int_t st = (fIdle) ? 0 : 1; - if (fIdle && fWaitingQueries->GetSize() > 0) st = 3; - return st; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update the session status in the relevant file. The status is taken from -/// GetSessionStatus() unless xst >= 0, in which case xst is used. -/// Return 0 on success, -errno if the file could not be opened. - -Int_t TProofServ::UpdateSessionStatus(Int_t xst) -{ - FILE *fs = fopen(fAdminPath.Data(), "w"); - if (fs) { - Int_t st = (xst < 0) ? GetSessionStatus() : xst; - fprintf(fs, "%d", st); - fclose(fs); - PDB(kGlobal, 2) - Info("UpdateSessionStatus", "status (=%d) update in path: %s", st, fAdminPath.Data()); - } else { - return -errno; - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the idle status - -Bool_t TProofServ::IsIdle() -{ - std::lock_guard lock(fQMtx); - return fIdle; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Change the idle status - -void TProofServ::SetIdle(Bool_t st) -{ - std::lock_guard lock(fQMtx); - fIdle = st; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return kTRUE if the session is waiting for the OK to start processing - -Bool_t TProofServ::IsWaiting() -{ - std::lock_guard lock(fQMtx); - if (fIdle && fWaitingQueries->GetSize() > 0) return kTRUE; - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the number of waiting queries - -Int_t TProofServ::WaitingQueries() -{ - std::lock_guard lock(fQMtx); - return fWaitingQueries->GetSize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add a query to the waiting list -/// Returns the number of queries in the list - -Int_t TProofServ::QueueQuery(TProofQueryResult *pq) -{ - std::lock_guard lock(fQMtx); - fWaitingQueries->Add(pq); - return fWaitingQueries->GetSize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get the next query from the waiting list. -/// The query is removed from the list. - -TProofQueryResult *TProofServ::NextQuery() -{ - std::lock_guard lock(fQMtx); - TProofQueryResult *pq = (TProofQueryResult *) fWaitingQueries->First(); - fWaitingQueries->Remove(pq); - return pq; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup the waiting queries list. The objects are deleted if 'del' is true. -/// If 'qls' is non null, only objects in 'qls' are removed. -/// Returns the number of cleanup queries - -Int_t TProofServ::CleanupWaitingQueries(Bool_t del, TList *qls) -{ - std::lock_guard lock(fQMtx); - Int_t ncq = 0; - if (qls) { - TIter nxq(qls); - TObject *o = 0; - while ((o = nxq())) { - if (fWaitingQueries->FindObject(o)) ncq++; - fWaitingQueries->Remove(o); - if (del) delete o; - } - } else { - ncq = fWaitingQueries->GetSize(); - fWaitingQueries->SetOwner(del); - fWaitingQueries->Delete(); - } - // Done - return ncq; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the message to be sent back in case of exceptions - -void TProofServ::SetLastMsg(const char *lastmsg) -{ - fgLastMsg = lastmsg; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the last entry before exception - -void TProofServ::SetLastEntry(Long64_t entry) -{ - fgLastEntry = entry; -} - -//////////////////////////////////////////////////////////////////////////////// -/// VirtMemMax getter - -Long_t TProofServ::GetVirtMemMax() -{ - return fgVirtMemMax; -} -//////////////////////////////////////////////////////////////////////////////// -/// ResMemMax getter - -Long_t TProofServ::GetResMemMax() -{ - return fgResMemMax; -} -//////////////////////////////////////////////////////////////////////////////// -/// MemHWM getter - -Float_t TProofServ::GetMemHWM() -{ - return fgMemHWM; -} -//////////////////////////////////////////////////////////////////////////////// -/// MemStop getter - -Float_t TProofServ::GetMemStop() -{ - return fgMemStop; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Extract LOCALDATASERVER info in 'dsrv' - -void TProofServ::GetLocalServer(TString &dsrv) -{ - // Check if a local data server has been specified - if (gSystem->Getenv("LOCALDATASERVER")) { - dsrv = gSystem->Getenv("LOCALDATASERVER"); - if (!dsrv.EndsWith("/")) dsrv += "/"; - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// If 'path' is local and 'dsrv' is Xrootd, apply 'path.Localroot' settings, -/// if any. -/// The final path via the server is dsrv+path. - -void TProofServ::FilterLocalroot(TString &path, const char *dsrv) -{ - TUrl u(path, kTRUE); - if (!strcmp(u.GetProtocol(), "file")) { - // Remove prefix, if any, if included and if Xrootd - TString pfx = gEnv->GetValue("Path.Localroot",""); - if (!pfx.IsNull() && !strncmp(u.GetFile(), pfx.Data(), pfx.Length())) { - TString srvp = TUrl(dsrv).GetProtocol(); - if (srvp == "root" || srvp == "xrd") path.Remove(0, pfx.Length()); - } - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Locks the directory. Waits if lock is hold by an other process. -/// Returns 0 on success, -1 in case of error. - -Int_t TProofLockPath::Lock() -{ - const char *pname = GetName(); - - if (gSystem->AccessPathName(pname)) - fLockId = open(pname, O_CREAT|O_RDWR, 0644); - else - fLockId = open(pname, O_RDWR); - - if (fLockId == -1) { - SysError("Lock", "cannot open lock file %s", pname); - return -1; - } - - PDB(kPackage, 2) - Info("Lock", "%d: locking file %s ...", gSystem->GetPid(), pname); - // lock the file -#if !defined(R__WIN32) && !defined(R__WINGCC) - if (lockf(fLockId, F_LOCK, (off_t) 1) == -1) { - SysError("Lock", "error locking %s", pname); - close(fLockId); - fLockId = -1; - return -1; - } -#endif - - PDB(kPackage, 2) - Info("Lock", "%d: file %s locked", gSystem->GetPid(), pname); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Unlock the directory. Returns 0 in case of success, -/// -1 in case of error. - -Int_t TProofLockPath::Unlock() -{ - if (!IsLocked()) - return 0; - - PDB(kPackage, 2) - Info("Lock", "%d: unlocking file %s ...", gSystem->GetPid(), GetName()); - // unlock the file - lseek(fLockId, 0, SEEK_SET); -#if !defined(R__WIN32) && !defined(R__WINGCC) - if (lockf(fLockId, F_ULOCK, (off_t)1) == -1) { - SysError("Unlock", "error unlocking %s", GetName()); - close(fLockId); - fLockId = -1; - return -1; - } -#endif - - PDB(kPackage, 2) - Info("Unlock", "%d: file %s unlocked", gSystem->GetPid(), GetName()); - - close(fLockId); - fLockId = -1; - - return 0; -} diff --git a/proof/proof/src/TProofServLite.cxx b/proof/proof/src/TProofServLite.cxx deleted file mode 100644 index 365b9bba25875..0000000000000 --- a/proof/proof/src/TProofServLite.cxx +++ /dev/null @@ -1,621 +0,0 @@ -// @(#)root/proofx:$Id$ -// Author: Gerardo Ganis 12/12/2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofServLite -\ingroup proofkernel - -Version of the PROOF worker server for local running. The client starts -directly the desired number of these workers; the master and daemons are -eliminated, optimizing the number of messages exchanged and created / destroyed. - -*/ - -////////////////////////////////////////////////////////////////////////// -// // -// TProofServLite // -// // -// TProofServLite is the version of the PROOF worker server for local // -// running. The client starts directly the desired number of these // -// workers; the master and daemons are eliminated, optimizing the number// -// of messages exchanged and created / destroyed. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "RConfigure.h" -#include -#include "Riostream.h" - -#ifdef WIN32 - #include - typedef long off_t; - #include -#else -#include -#endif -#include -#include - -#include "TProofServLite.h" -#include "TEnv.h" -#include "TError.h" -#include "TException.h" -#include "THashList.h" -#include "TInterpreter.h" -#include "TMessage.h" -#include "TProofDebug.h" -#include "TProof.h" -#include "TVirtualProofPlayer.h" -#include "TProofQueryResult.h" -#include "TRegexp.h" -#include "TClass.h" -#include "TROOT.h" -#include "TSystem.h" -#include "TPluginManager.h" -#include "TSocket.h" -#include "TTimeStamp.h" -#include "compiledata.h" - -// debug hook -static volatile Int_t gProofServDebug = 1; - -//----- Interrupt signal handler ----------------------------------------------- -//////////////////////////////////////////////////////////////////////////////// - -class TProofServLiteInterruptHandler : public TSignalHandler { - TProofServLite *fServ; -public: - TProofServLiteInterruptHandler(TProofServLite *s) - : TSignalHandler(kSigUrgent, kFALSE) { fServ = s; } - Bool_t Notify() override; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Handle urgent data - -Bool_t TProofServLiteInterruptHandler::Notify() -{ - fServ->HandleUrgentData(); - if (TROOT::Initialized()) { - Throw(GetSignal()); - } - return kTRUE; -} - -//----- SigPipe signal handler ------------------------------------------------- -//////////////////////////////////////////////////////////////////////////////// - -class TProofServLiteSigPipeHandler : public TSignalHandler { - TProofServLite *fServ; -public: - TProofServLiteSigPipeHandler(TProofServLite *s) : TSignalHandler(kSigPipe, kFALSE) - { fServ = s; } - Bool_t Notify() override; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Handle sig pipe - -Bool_t TProofServLiteSigPipeHandler::Notify() -{ - fServ->HandleSigPipe(); - return kTRUE; -} - -//----- Termination signal handler --------------------------------------------- -//////////////////////////////////////////////////////////////////////////////// - -class TProofServLiteTerminationHandler : public TSignalHandler { - TProofServLite *fServ; -public: - TProofServLiteTerminationHandler(TProofServLite *s) - : TSignalHandler(kSigTermination, kFALSE) { fServ = s; } - Bool_t Notify() override; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Handle termination - -Bool_t TProofServLiteTerminationHandler::Notify() -{ - Printf("TProofServLiteTerminationHandler::Notify: wake up!"); - - fServ->HandleTermination(); - return kTRUE; -} - -//----- Seg violation signal handler --------------------------------------------- -//////////////////////////////////////////////////////////////////////////////// - -class TProofServLiteSegViolationHandler : public TSignalHandler { - TProofServLite *fServ; -public: - TProofServLiteSegViolationHandler(TProofServLite *s) - : TSignalHandler(kSigSegmentationViolation, kFALSE) { fServ = s; } - Bool_t Notify() override; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Handle seg violation - -Bool_t TProofServLiteSegViolationHandler::Notify() -{ - Printf("**** "); - Printf("**** Segmentation violation: terminating ****"); - Printf("**** "); - fServ->HandleTermination(); - return kTRUE; -} - -//----- Input handler for messages from parent or master ----------------------- -//////////////////////////////////////////////////////////////////////////////// - -class TProofServLiteInputHandler : public TFileHandler { - TProofServLite *fServ; -public: - TProofServLiteInputHandler(TProofServLite *s, Int_t fd) : TFileHandler(fd, 1) - { fServ = s; } - Bool_t Notify() override; - Bool_t ReadNotify() override { return Notify(); } -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Handle input on the socket - -Bool_t TProofServLiteInputHandler::Notify() -{ - fServ->HandleSocketInput(); - - return kTRUE; -} - -ClassImp(TProofServLite); - -// Hook to the constructor. This is needed to avoid using the plugin manager -// which may create problems in multi-threaded environments. -extern "C" { - TApplication *GetTProofServLite(Int_t *argc, char **argv, FILE *flog) - { return new TProofServLite(argc, argv, flog); } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor - -TProofServLite::TProofServLite(Int_t *argc, char **argv, FILE *flog) - : TProofServ(argc, argv, flog) -{ - fInterruptHandler = 0; - fTerminated = kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize the server setup. If master, create the TProof instance to talk -/// the worker or submaster nodes. -/// Return 0 on success, -1 on error - -Int_t TProofServLite::CreateServer() -{ - if (gProofDebugLevel > 0) - Info("CreateServer", "starting server creation"); - - // Get file descriptor for log file - if (fLogFile) { - // Use the file already open by pmain - if ((fLogFileDes = fileno(fLogFile)) < 0) { - Error("CreateServer", "resolving the log file description number"); - return -1; - } - } - - // Get socket to be used to call back our xpd - fSockPath = gEnv->GetValue("ProofServ.OpenSock", ""); - if (fSockPath.Length() <= 0) { - Error("CreateServer", "Socket setup by xpd undefined"); - return -1; - } - TString entity = gEnv->GetValue("ProofServ.Entity", ""); - if (entity.Length() > 0) - fSockPath.Insert(0,TString::Format("%s/", entity.Data())); - - // Call back the client - fSocket = new TSocket(fSockPath); - if (!fSocket || !(fSocket->IsValid())) { - Error("CreateServer", "Failed to open connection to the client"); - return -1; - } - - // Send our ordinal, to allow the client to identify us - TMessage msg; - msg << fOrdinal; - fSocket->Send(msg); - - // Get socket descriptor - Int_t sock = fSocket->GetDescriptor(); - - // Install interrupt and message input handlers - fInterruptHandler = new TProofServLiteInterruptHandler(this); - gSystem->AddSignalHandler(fInterruptHandler); - gSystem->AddFileHandler(new TProofServLiteInputHandler(this, sock)); - - // Wait (loop) in worker node to allow debugger to connect - if (gEnv->GetValue("Proof.GdbHook",0) == 2) { - while (gProofServDebug) - ; - } - - if (gProofDebugLevel > 0) - Info("CreateServer", "Service: %s, ConfDir: %s, IsMaster: %d", - fService.Data(), fConfDir.Data(), (Int_t)fMasterServ); - - if (Setup() == -1) { - // Setup failure - Terminate(0); - SendLogFile(); - return -1; - } - - if (!fLogFile) { - RedirectOutput(); - // If for some reason we failed setting a redirection file for the logs - // we cannot continue - if (!fLogFile || (fLogFileDes = fileno(fLogFile)) < 0) { - Terminate(0); - SendLogFile(-98); - return -1; - } - } - - // Everybody expects std::iostream to be available, so load it... - ProcessLine("#include ", kTRUE); - ProcessLine("#include ",kTRUE); // for std::string std::iostream. - - // Load user functions - const char *logon; - logon = gEnv->GetValue("Proof.Load", (char *)0); - if (logon) { - char *mac = gSystem->Which(TROOT::GetMacroPath(), logon, kReadPermission); - if (mac) - ProcessLine(TString::Format(".L %s", logon), kTRUE); - delete [] mac; - } - - // Execute logon macro - logon = gEnv->GetValue("Proof.Logon", (char *)0); - if (logon && !NoLogOpt()) { - char *mac = gSystem->Which(TROOT::GetMacroPath(), logon, kReadPermission); - if (mac) - ProcessFile(logon); - delete [] mac; - } - - // Save current interpreter context - gInterpreter->SaveContext(); - gInterpreter->SaveGlobalsContext(); - - // Avoid spurious messages at first action - FlushLogFile(); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup. Not really necessary since after this dtor there is no -/// live anyway. - -TProofServLite::~TProofServLite() -{ - delete fSocket; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Called when the client is not alive anymore; terminate the session. - -void TProofServLite::HandleSigPipe() -{ - Terminate(0); // will not return from here.... -} - -//////////////////////////////////////////////////////////////////////////////// -/// Called when the client is not alive anymore; terminate the session. - -void TProofServLite::HandleTermination() -{ - Terminate(0); // will not return from here.... -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print the ProofServ logo on standard output. -/// Return 0 on success, -1 on error - -Int_t TProofServLite::Setup() -{ - char str[512]; - - if (IsMaster()) { - snprintf(str, 512, "**** Welcome to the PROOF server @ %s ****", gSystem->HostName()); - } else { - snprintf(str, 512, "**** PROOF worker server @ %s started ****", gSystem->HostName()); - } - - if (fSocket->Send(str) != 1+static_cast(strlen(str))) { - Error("Setup", "failed to send proof server startup message"); - return -1; - } - - // Get client protocol - if ((fProtocol = gEnv->GetValue("ProofServ.ClientVersion", -1)) < 0) { - Error("Setup", "remote proof protocol missing"); - return -1; - } - - // The local user - UserGroup_t *pw = gSystem->GetUserInfo(); - if (pw) { - fUser = pw->fUser; - delete pw; - } - - // Work dir and ... - fWorkDir = gEnv->GetValue("ProofServ.Sandbox", TString::Format("~/%s", kPROOF_WorkDir)); - Info("Setup", "fWorkDir: %s", fWorkDir.Data()); - - // Get Session tags - fTopSessionTag = gEnv->GetValue("ProofServ.SessionTag", "-1"); - fSessionTag.Form("%s-%s-%ld-%d", fOrdinal.Data(), gSystem->HostName(), - (Long_t)TTimeStamp().GetSec(), gSystem->GetPid()); - if (gProofDebugLevel > 0) - Info("Setup", "session tag is %s", fSessionTag.Data()); - if (fTopSessionTag.IsNull()) fTopSessionTag = fSessionTag; - - // Send session tag to client - TMessage m(kPROOF_SESSIONTAG); - m << fSessionTag; - fSocket->Send(m); - - // Get Session dir (sandbox) - if ((fSessionDir = gEnv->GetValue("ProofServ.SessionDir", "-1")) == "-1") { - Error("Setup", "Session dir missing"); - return -1; - } - - // Link the session tag to the log file - if (gSystem->Getenv("ROOTPROOFLOGFILE")) { - TString logfile = gSystem->Getenv("ROOTPROOFLOGFILE"); - Int_t iord = logfile.Index(TString::Format("-%s", fOrdinal.Data())); - if (iord != kNPOS) logfile.Remove(iord); - logfile += TString::Format("-%s.log", fSessionTag.Data()); - gSystem->Symlink(gSystem->Getenv("ROOTPROOFLOGFILE"), logfile); - } - - // Goto to the main PROOF working directory - gSystem->ExpandPathName(fWorkDir); - if (gProofDebugLevel > 0) - Info("Setup", "working directory set to %s", fWorkDir.Data()); - - // Common setup - if (SetupCommon() != 0) { - Error("Setup", "common setup failed"); - return -1; - } - - // Check every two hours if client is still alive - fSocket->SetOption(kKeepAlive, 1); - - // Install SigPipe handler to handle kKeepAlive failure - gSystem->AddSignalHandler(new TProofServLiteSigPipeHandler(this)); - - // Install Termination handler - gSystem->AddSignalHandler(new TProofServLiteTerminationHandler(this)); - - // Install seg violation handler - gSystem->AddSignalHandler(new TProofServLiteSegViolationHandler(this)); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Terminate the proof server. - -void TProofServLite::Terminate(Int_t status) -{ - if (fTerminated) - // Avoid doubling the exit operations - exit(1); - fTerminated = kTRUE; - - // Notify - Info("Terminate", "starting session termination operations ..."); - - // Cleanup session directory - if (status == 0) { - // make sure we remain in a "connected" directory - gSystem->ChangeDirectory("/"); - // needed in case fSessionDir is on NFS ?! - gSystem->MakeDirectory(fSessionDir+"/.delete"); - gSystem->Exec(TString::Format("%s %s", kRM, fSessionDir.Data())); - } - - // Cleanup data directory if empty - if (!fDataDir.IsNull() && !gSystem->AccessPathName(fDataDir, kWritePermission)) { - if (UnlinkDataDir(fDataDir)) - Info("Terminate", "data directory '%s' has been removed", fDataDir.Data()); - } - - // Remove input and signal handlers to avoid spurious "signals" - // for closing activities executed upon exit() - gSystem->RemoveSignalHandler(fInterruptHandler); - - // Stop processing events (set a flag to exit the event loop) - gSystem->ExitLoop(); - - // Notify - Printf("Terminate: termination operations ended: quitting!"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cloning itself via fork. - -void TProofServLite::HandleFork(TMessage *mess) -{ - if (!mess) { - Error("HandleFork", "empty message!"); - return; - } - - // Extract the ordinals of the clones - TString clones; - (*mess) >> clones; - PDB(kGlobal, 1) - Info("HandleFork", "cloning to %s", clones.Data()); - - TString clone; - Int_t from = 0; - while (clones.Tokenize(clone, from, " ")) { - - Int_t rc = 0; - // Fork - if ((rc = Fork()) < 0) { - Error("HandleFork", "failed to fork %s", clone.Data()); - return; - } - - // If the child, finalize the setup and return - if (rc == 0) { - SetupOnFork(clone.Data()); - return; - } - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize the server setup afetr forking. -/// Return 0 on success, -1 on error - -Int_t TProofServLite::SetupOnFork(const char *ord) -{ - if (gProofDebugLevel > 0) - Info("SetupOnFork", "finalizing setup of %s", ord); - - // Set the ordinal - fOrdinal = ord; - TString sord; - sord.Form("-%s", fOrdinal.Data()); - - // Close the current log file - if (fLogFile) { - fclose(fLogFile); - fLogFileDes = -1; - } - - TString sdir = gSystem->GetDirName(fSessionDir.Data()); - RedirectOutput(sdir.Data(), "a"); - // If for some reason we failed setting a redirection file for the logs - // we cannot continue - if (!fLogFile || (fLogFileDes = fileno(fLogFile)) < 0) { - Terminate(0); - return -1; - } - FlushLogFile(); - - // Eliminate existing symlink - void *dirp = gSystem->OpenDirectory(sdir); - if (dirp) { - TString ent; - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - ent.Form("%s/%s", sdir.Data(), e); - FileStat_t st; - if (gSystem->GetPathInfo(ent.Data(), st) == 0) { - if (st.fIsLink && ent.Contains(sord)) { - PDB(kGlobal, 1) - Info("SetupOnFork","unlinking: %s", ent.Data()); - gSystem->Unlink(ent); - } - } - } - gSystem->FreeDirectory(dirp); - } - - // The session tag - fSessionTag.Form("%s-%d-%d", gSystem->HostName(), (int)time(0), gSystem->GetPid()); - - // Create new symlink - TString logfile = gSystem->Getenv("ROOTPROOFLOGFILE"); - logfile.ReplaceAll("-0.0", sord.Data()); - gSystem->Setenv("ROOTPROOFLOGFILE", logfile); - Int_t iord = logfile.Index(sord.Data()); - if (iord != kNPOS) logfile.Remove(iord + sord.Length()); - logfile += TString::Format("-%s.log", fSessionTag.Data()); - gSystem->Symlink(gSystem->Getenv("ROOTPROOFLOGFILE"), logfile); - - // Get socket to be used to call back our xpd - fSockPath = gEnv->GetValue("ProofServ.OpenSock", ""); - if (fSockPath.Length() <= 0) { - Error("CreateServer", "Socket setup by xpd undefined"); - return -1; - } - TString entity = gEnv->GetValue("ProofServ.Entity", ""); - if (entity.Length() > 0) - fSockPath.Insert(0, TString::Format("%s/", entity.Data())); - - // Call back the client - fSocket = new TSocket(fSockPath); - if (!fSocket || !(fSocket->IsValid())) { - Error("CreateServer", "Failed to open connection to the client"); - return -1; - } - - // Send our ordinal, to allow the client to identify us - TMessage msg; - msg << fOrdinal; - fSocket->Send(msg); - - // Get socket descriptor - Int_t sock = fSocket->GetDescriptor(); - - // Install interrupt and message input handlers - fInterruptHandler = new TProofServLiteInterruptHandler(this); - gSystem->AddSignalHandler(fInterruptHandler); - gSystem->AddFileHandler(new TProofServLiteInputHandler(this, sock)); - - // Wait (loop) in worker node to allow debugger to connect - if (gEnv->GetValue("Proof.GdbHook",0) == 2) { - while (gProofServDebug) - ; - } - - if (gProofDebugLevel > 0) - Info("SetupOnFork", "Service: %s, ConfDir: %s, IsMaster: %d", - fService.Data(), fConfDir.Data(), (Int_t)fMasterServ); - - if (Setup() == -1) { - // Setup failure - Terminate(0); - SendLogFile(); - return -1; - } - - // Disallow the interpretation of Rtypes.h, TError.h and TGenericClassInfo.h - ProcessLine("#define ROOT_Rtypes 0", kTRUE); - ProcessLine("#define ROOT_TError 0", kTRUE); - ProcessLine("#define ROOT_TGenericClassInfo 0", kTRUE); - - // Save current interpreter context - gInterpreter->SaveContext(); - gInterpreter->SaveGlobalsContext(); - - // Done - return 0; -} diff --git a/proof/proof/src/TProofSuperMaster.cxx b/proof/proof/src/TProofSuperMaster.cxx deleted file mode 100644 index 63e3c28cdcce2..0000000000000 --- a/proof/proof/src/TProofSuperMaster.cxx +++ /dev/null @@ -1,363 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 13/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofSuperMaster -\ingroup proofkernel - -Implementation of TProof controlling PROOF federated clusters. - -*/ - -#include "TProofSuperMaster.h" -#include "TString.h" -#include "TObjString.h" -#include "TError.h" -#include "TList.h" -#include "TSortedList.h" -#include "TSlave.h" -#include "TMap.h" -#include "TProofServ.h" -#include "TSocket.h" -#include "TMonitor.h" -#include "TDSet.h" -#include "TPluginManager.h" -#include "TVirtualProofPlayer.h" -#include "TMessage.h" -#include "TUrl.h" -#include "TProofResourcesStatic.h" -#include "TProofNodeInfo.h" -#include "TROOT.h" - -ClassImp(TProofSuperMaster); - -//////////////////////////////////////////////////////////////////////////////// -/// Start super master PROOF session. - -TProofSuperMaster::TProofSuperMaster(const char *masterurl, const char *conffile, - const char *confdir, Int_t loglevel, - const char *alias, TProofMgr *mgr) -{ - // Default initializations - InitMembers(); - - // This may be needed during init - fManager = mgr; - - fUrl = TUrl(masterurl); - - if (!conffile || !conffile[0]) - conffile = kPROOF_ConfFile; - else if (!strncasecmp(conffile, "sm:", 3)) - conffile+=3; - if (!confdir || !confdir[0]) - confdir = kPROOF_ConfDir; - - // Instance type - fMasterServ = kTRUE; - ResetBit(TProof::kIsClient); - SetBit(TProof::kIsMaster); - SetBit(TProof::kIsTopMaster); - - Init(masterurl, conffile, confdir, loglevel, alias); - - // For Final cleanup - gROOT->GetListOfProofs()->Add(this); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Start up PROOF submasters. - -Bool_t TProofSuperMaster::StartSlaves(Bool_t) -{ - // If this is a supermaster server, find the config file and start - // submaster servers as specified in the config file. - // There is a difference in startup between a slave and a submaster - // in which the submaster will issue a kPROOF_LOGFILE and - // then a kPROOF_LOGDONE message (which must be collected) - // while slaves do not. - - Int_t pc = 0; - TList *submasterList = new TList; - // Get list of workers - if (gProofServ->GetWorkers(submasterList, pc) == TProofServ::kQueryStop) { - Error("StartSlaves", "getting list of submaster nodes"); - return kFALSE; - } - fImage = gProofServ->GetImage(); - if (fImage.IsNull()) - fImage = Form("%s:%s", TUrl(gSystem->HostName()).GetHostFQDN(), - gProofServ->GetWorkDir()); - - UInt_t nSubmasters = submasterList->GetSize(); - UInt_t nSubmastersDone = 0; - Int_t ord = 0; - TList validSubmasters; - TList validPairs; - validPairs.SetOwner(); - - // Loop over all submasters and start them - TListIter next(submasterList); - TObject *to; - TProofNodeInfo *submaster; - while ((to = next())) { - // Get the next submaster from the list - submaster = (TProofNodeInfo *)to; - const Char_t *conffile = submaster->GetConfig(); - const Char_t *image = submaster->GetImage(); - const Char_t *msd = submaster->GetMsd(); - Int_t sport = submaster->GetPort(); - if (sport == -1) - sport = fUrl.GetPort(); - - TString fullord = TString(gProofServ->GetOrdinal()) + "." + ((Long_t) ord); - - // create submaster server - TUrl u(Form("%s:%d", submaster->GetNodeName().Data(), sport)); - // Add group info in the password firdl, if any - if (strlen(gProofServ->GetGroup()) > 0) { - // Set also the user, otherwise the password is not exported - if (strlen(u.GetUser()) <= 0) - u.SetUser(gProofServ->GetUser()); - u.SetPasswd(gProofServ->GetGroup()); - } - TSlave *slave = - CreateSubmaster(u.GetUrl(), fullord, image, msd); - - // Add to global list (we will add to the monitor list after - // finalizing the server startup) - Bool_t submasterOk = kTRUE; - fSlaves->Add(slave); - if (slave->IsValid()) { - validPairs.Add(new TPair(slave, new TObjString(conffile))); - } else { - submasterOk = kFALSE; - fBadSlaves->Add(slave); - } - - PDB(kGlobal,3) - Info("StartSlaves","submaster on host %s created and" - " added to list", submaster->GetNodeName().Data()); - - // Notify opening of connection - nSubmastersDone++; - TMessage m(kPROOF_SERVERSTARTED); - m << TString("Opening connections to submasters") << nSubmasters - << nSubmastersDone << submasterOk; - gProofServ->GetSocket()->Send(m); - - ord++; - - } // end loop over all submasters - - // Cleanup - SafeDelete(submasterList); - - nSubmastersDone = 0; - - // Here we finalize the server startup: in this way the bulk - // of remote operations are almost parallelized - TIter nxsc(&validPairs); - TPair *sc = 0; - while ((sc = (TPair *) nxsc())) { - // Finalize setup of the server - TSlave *sl = (TSlave *) sc->Key(); - TObjString *cf = (TObjString *) sc->Value(); - sl->SetupServ(TSlave::kMaster, cf->GetName()); - - // Monitor good slaves - Bool_t submasterOk = kTRUE; - if (sl->IsValid()) { - // check protocol compatability - // protocol 1 is not supported anymore - if (fProtocol == 1) { - Error("StartSlaves", "master and submaster protocols" - " not compatible (%d and %d)", - kPROOF_Protocol, fProtocol); - submasterOk = kFALSE; - fBadSlaves->Add(sl); - } else { - fAllMonitor->Add(sl->GetSocket()); - validSubmasters.Add(sl); - } - } else { - submasterOk = kFALSE; - fBadSlaves->Add(sl); - } - - // Notify end of startup operations - nSubmastersDone++; - TMessage m(kPROOF_SERVERSTARTED); - m << TString("Setting up submasters") << nSubmasters - << nSubmastersDone << submasterOk; - gProofServ->GetSocket()->Send(m); - } - - Collect(kAll); //Get kPROOF_LOGFILE and kPROOF_LOGDONE messages - TIter nextSubmaster(&validSubmasters); - while (TSlave* sl = dynamic_cast(nextSubmaster())) { - if (sl->GetStatus() == -99) { - Error("StartSlaves", "not allowed to connect to PROOF master server"); - fBadSlaves->Add(sl); - continue; - } - - if (!sl->IsValid()) { - Error("StartSlaves", "failed to setup connection with PROOF master server"); - fBadSlaves->Add(sl); - continue; - } - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process a data set (TDSet) using the specified selector (.C) file. -/// Entry- or event-lists should be set in the data set object using -/// TDSet::SetEntryList. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProofSuperMaster::Process(TDSet *set, const char *selector, Option_t *option, - Long64_t nentries, Long64_t first) -{ - if (!IsValid()) return -1; - - R__ASSERT(GetPlayer()); - - if (GetProgressDialog()) - GetProgressDialog()->ExecPlugin(5, this, selector, set->GetListOfElements()->GetSize(), - first, nentries); - - return GetPlayer()->Process(set, selector, option, nentries, first); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Validate a TDSet. - -void TProofSuperMaster::ValidateDSet(TDSet *dset) -{ - if (dset->ElementsValid()) return; - - // We need to recheck after this - dset->ResetBit(TDSet::kValidityChecked); - dset->ResetBit(TDSet::kSomeInvalid); - - TList msds; - msds.SetOwner(); - - TList smholder; - smholder.SetOwner(); - TList elemholder; - elemholder.SetOwner(); - - // build nodelist with slaves and elements - TIter nextSubmaster(GetListOfActiveSlaves()); - while (TSlave *sl = dynamic_cast(nextSubmaster())) { - TList *smlist = 0; - TPair *p = dynamic_cast(msds.FindObject(sl->GetMsd())); - if (!p) { - smlist = new TList; - smlist->SetName(sl->GetMsd()); - - smholder.Add(smlist); - TList *elemlist = new TSortedList(kSortDescending); - elemlist->SetName(TString(sl->GetMsd())+"_elem"); - elemholder.Add(elemlist); - msds.Add(new TPair(smlist, elemlist)); - } else { - smlist = dynamic_cast(p->Key()); - } - if (smlist) smlist->Add(sl); - } - - TIter nextElem(dset->GetListOfElements()); - while (TDSetElement *elem = dynamic_cast(nextElem())) { - if (elem->GetValid()) continue; - TPair *p = dynamic_cast(msds.FindObject(elem->GetMsd())); - if (p && p->Value()) { - TList *xl = dynamic_cast(p->Value()); - if (xl) xl->Add(elem); - } else { - Error("ValidateDSet", "no mass storage domain '%s' associated" - " with available submasters", - elem->GetMsd()); - return; - } - } - - // send to slaves - TList usedsms; - TIter nextSM(&msds); - SetDSet(dset); // set dset to be validated in Collect() - while (TPair *msd = dynamic_cast(nextSM())) { - TList *sms = dynamic_cast(msd->Key()); - TList *setelements = dynamic_cast(msd->Value()); - - // distribute elements over the slaves - Int_t nsms = sms ? sms->GetSize() : -1; - Int_t nelements = setelements ? setelements->GetSize() : -1; - for (Int_t i=0; iGetType(), dset->GetObjName(), - dset->GetDirectory()); - for (Int_t j = (i*nelements)/nsms; - j < ((i+1)*nelements)/nsms; - j++) { - TDSetElement *elem = setelements ? - dynamic_cast(setelements->At(j)) : (TDSetElement *)0; - if (elem) { - set.Add(elem->GetFileName(), elem->GetObjName(), - elem->GetDirectory(), elem->GetFirst(), - elem->GetNum(), elem->GetMsd()); - } - } - - if (set.GetListOfElements()->GetSize()>0) { - TMessage mesg(kPROOF_VALIDATE_DSET); - mesg << &set; - - TSlave *sl = dynamic_cast(sms->At(i)); - if (sl) { - PDB(kGlobal,1) - Info("ValidateDSet", - "Sending TDSet with %d elements to worker %s" - " to be validated", set.GetListOfElements()->GetSize(), - sl->GetOrdinal()); - sl->GetSocket()->Send(mesg); - usedsms.Add(sl); - } else { - Warning("ValidateDSet", "not a TSlave object"); - } - } - } - } - - PDB(kGlobal,1) - Info("ValidateDSet","Calling Collect"); - Collect(&usedsms); - SetDSet(0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Construct a TProofPlayer object. The player string specifies which -/// player should be created: remote, slave, sm (supermaster) or base. -/// Default is sm. Socket is needed in case a slave player is created. - -TVirtualProofPlayer *TProofSuperMaster::MakePlayer(const char *player, TSocket *s) -{ - if (!player) - player = "sm"; - - SetPlayer(TVirtualProofPlayer::Create(player, this, s)); - return GetPlayer(); -} - diff --git a/proof/proof/src/TQueryResultManager.cxx b/proof/proof/src/TQueryResultManager.cxx deleted file mode 100644 index 8d73d21e3c825..0000000000000 --- a/proof/proof/src/TQueryResultManager.cxx +++ /dev/null @@ -1,734 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis Mar 2008 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TQueryResultManager -\ingroup proofkernel - -Class managing the query-result area - -*/ - -#include -#ifdef WIN32 -# include -#endif - -#include "TQueryResultManager.h" - -#include "TFile.h" -#include "THashList.h" -#include "TKey.h" -#include "TProofQueryResult.h" -#include "TObjString.h" -#include "TParameter.h" -#include "TProof.h" -#include "TProofServ.h" -#include "TRegexp.h" -#include "TSortedList.h" -#include "TSystem.h" -#include "TVirtualProofPlayer.h" - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TQueryResultManager::TQueryResultManager(const char *qdir, const char *stag, - const char *sdir, - TProofLockPath *lck, FILE *logfile) -{ - fQueryDir = qdir; - fSessionTag = stag; - fSessionDir = sdir; - fSeqNum = 0; - fDrawQueries = 0; - fKeptQueries = 0; - fQueries = new TList; - fPreviousQueries = 0; - fLock = lck; - fLogFile = (logfile) ? logfile : stdout; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup. Not really necessary since after this dtor there is no -/// live anyway. - -TQueryResultManager::~TQueryResultManager() -{ - SafeDelete(fQueries); - SafeDelete(fPreviousQueries); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add part of log file concerning TQueryResult pq to its macro -/// container. - -void TQueryResultManager::AddLogFile(TProofQueryResult *pq) -{ - if (!pq) - return; - - // Make sure everything is written to file - fflush(fLogFile); - - // Save current position - off_t lnow = 0; - if ((lnow = lseek(fileno(fLogFile), (off_t) 0, SEEK_CUR)) < 0) { - Error("AddLogFile", "problems lseeking current position on log file (errno: %d)", errno); - return; - } - - // The range we are interested in - Int_t start = pq->fStartLog; - if (start > -1) - lseek(fileno(fLogFile), (off_t) start, SEEK_SET); - - // Read the lines and add then to the internal container - const Int_t kMAXBUF = 4096; - char line[kMAXBUF]; - while (fgets(line, sizeof(line), fLogFile)) { - if (line[strlen(line)-1] == '\n') - line[strlen(line)-1] = 0; - pq->AddLogLine((const char *)line); - } - - // Restore initial position if partial send - if (lnow >= 0) lseek(fileno(fLogFile), lnow, SEEK_SET); -} -//////////////////////////////////////////////////////////////////////////////// -/// Remove all queries results referring to previous sessions - -Int_t TQueryResultManager::CleanupQueriesDir() -{ - Int_t nd = 0; - - // Cleanup previous stuff - if (fPreviousQueries) { - fPreviousQueries->Delete(); - SafeDelete(fPreviousQueries); - } - - // Loop over session dirs - TString queriesdir = fQueryDir; - queriesdir = queriesdir.Remove(queriesdir.Index(kPROOF_QueryDir) + - strlen(kPROOF_QueryDir)); - void *dirs = gSystem->OpenDirectory(queriesdir); - if (dirs) { - char *sess = 0; - while ((sess = (char *) gSystem->GetDirEntry(dirs))) { - - // We are interested only in "session-..." subdirs - if (strlen(sess) < 7 || strncmp(sess,"session",7)) - continue; - - // We do not want this session at this level - if (strstr(sess, fSessionTag)) - continue; - - // Remove the directory - TString qdir; - qdir.Form("%s/%s", queriesdir.Data(), sess); - PDB(kGlobal, 1) - Info("RemoveQuery", "removing directory: %s", qdir.Data()); - gSystem->Exec(Form("%s %s", kRM, qdir.Data())); - nd++; - } - // Close directory - gSystem->FreeDirectory(dirs); - } else { - Warning("RemoveQuery", "cannot open queries directory: %s", queriesdir.Data()); - } - - // Done - return nd; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Scan the queries directory for the results of previous queries. -/// The headers of the query results found are loaded in fPreviousQueries. -/// The full query result can be retrieved via TProof::Retrieve. - -void TQueryResultManager::ScanPreviousQueries(const char *dir) -{ - // Cleanup previous stuff - if (fPreviousQueries) { - fPreviousQueries->Delete(); - SafeDelete(fPreviousQueries); - } - - // Loop over session dirs - void *dirs = gSystem->OpenDirectory(dir); - char *sess = 0; - while ((sess = (char *) gSystem->GetDirEntry(dirs))) { - - // We are interested only in "session-..." subdirs - if (strlen(sess) < 7 || strncmp(sess,"session",7)) - continue; - - // We do not want this session at this level - if (strstr(sess, fSessionTag)) - continue; - - // Loop over query dirs - void *dirq = gSystem->OpenDirectory(Form("%s/%s", dir, sess)); - char *qry = 0; - while ((qry = (char *) gSystem->GetDirEntry(dirq))) { - - // We are interested only in "n/" subdirs - if (qry[0] == '.') - continue; - - // File with the query result - TString fn = Form("%s/%s/%s/query-result.root", dir, sess, qry); - TFile *f = TFile::Open(fn); - if (f) { - f->ReadKeys(); - TIter nxk(f->GetListOfKeys()); - TKey *k = 0; - TProofQueryResult *pqr = 0; - while ((k = (TKey *)nxk())) { - if (!strcmp(k->GetClassName(), "TProofQueryResult")) { - pqr = (TProofQueryResult *) f->Get(k->GetName()); - if (pqr) { - TQueryResult *qr = pqr->CloneInfo(); - if (qr) { - if (!fPreviousQueries) - fPreviousQueries = new TList; - if (qr->GetStatus() > TQueryResult::kRunning) { - fPreviousQueries->Add(qr); - } else { - // (For the time being) remove a non completed - // query if not owned by anybody - TProofLockPath *lck = 0; - if (LockSession(qr->GetTitle(), &lck) == 0) { - RemoveQuery(qr); - // Unlock and remove the lock file - SafeDelete(lck); - } - } - } else { - Warning("ScanPreviousQueries", "unable to clone TProofQueryResult '%s:%s'", - pqr->GetName(), pqr->GetTitle()); - } - } - } - } - f->Close(); - delete f; - } - } - gSystem->FreeDirectory(dirq); - } - gSystem->FreeDirectory(dirs); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Scan the queries directory and remove the oldest ones (and relative dirs, -/// if empty) in such a way only 'mxq' queries are kept. -/// Return 0 on success, -1 in case of problems - -Int_t TQueryResultManager::ApplyMaxQueries(Int_t mxq) -{ - // Nothing to do if mxq is -1. - if (mxq < 0) - return 0; - - // We will sort the entries using the creation time - TSortedList *sl = new TSortedList; - sl->SetOwner(); - // List with information - THashList *hl = new THashList; - hl->SetOwner(); - - // Keep track of the queries per session dir - TList *dl = new TList; - dl->SetOwner(); - - // Loop over session dirs - TString dir = fQueryDir; - Int_t idx = dir.Index("session-"); - if (idx != kNPOS) - dir.Remove(idx); - void *dirs = gSystem->OpenDirectory(dir); - char *sess = 0; - while ((sess = (char *) gSystem->GetDirEntry(dirs))) { - - // We are interested only in "session-..." subdirs - if (strlen(sess) < 7 || strncmp(sess,"session",7)) - continue; - - // We do not want this session at this level - if (strstr(sess, fSessionTag)) - continue; - - // Loop over query dirs - Int_t nq = 0; - void *dirq = gSystem->OpenDirectory(Form("%s/%s", dir.Data(), sess)); - char *qry = 0; - while ((qry = (char *) gSystem->GetDirEntry(dirq))) { - - // We are interested only in "n/" subdirs - if (qry[0] == '.') - continue; - - // File with the query result - TString fn = Form("%s/%s/%s/query-result.root", dir.Data(), sess, qry); - - FileStat_t st; - if (gSystem->GetPathInfo(fn, st)) { - PDB(kGlobal, 1) - Info("ApplyMaxQueries","file '%s' cannot be stated: remove it", fn.Data()); - gSystem->Unlink(gSystem->GetDirName(fn)); - continue; - } - - // Add the entry in the sorted list - sl->Add(new TObjString(TString::Format("%ld", st.fMtime))); - hl->Add(new TNamed((const char*)TString::Format("%ld",st.fMtime), fn.Data())); - nq++; - } - gSystem->FreeDirectory(dirq); - - if (nq > 0) - dl->Add(new TParameter(TString::Format("%s/%s", dir.Data(), sess), nq)); - else - // Remove it - gSystem->Exec(TString::Format("%s -fr %s/%s", kRM, dir.Data(), sess)); - } - gSystem->FreeDirectory(dirs); - - // Now we apply the quota - TIter nxq(sl, kIterBackward); - Int_t nqkept = 0; - TObjString *os = 0; - while ((os = (TObjString *)nxq())) { - if (nqkept < mxq) { - // Keep this and go to the next - nqkept++; - } else { - // Clean this - TNamed *nm = dynamic_cast(hl->FindObject(os->GetName())); - if (nm) { - gSystem->Unlink(nm->GetTitle()); - // Update dir counters - TString tdir = gSystem->GetDirName(nm->GetTitle()); - tdir = gSystem->GetDirName(tdir.Data()); - TParameter *nq = dynamic_cast*>(dl->FindObject(tdir)); - if (nq) { - Int_t val = nq->GetVal(); - nq->SetVal(--val); - if (nq->GetVal() <= 0) - // Remove the directory if empty - gSystem->Exec(Form("%s -fr %s", kRM, tdir.Data())); - } - } - } - } - - // Cleanup - delete sl; - delete hl; - delete dl; - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Try locking query area of session tagged sessiontag. -/// The id of the locking file is returned in fid and must be -/// unlocked via UnlockQueryFile(fid). - -Int_t TQueryResultManager::LockSession(const char *sessiontag, TProofLockPath **lck) -{ - // We do not need to lock our own session - if (strstr(sessiontag, fSessionTag)) - return 0; - - if (!lck) { - Error("LockSession","locker space undefined"); - return -1; - } - *lck = 0; - - // Check the format - TString stag = sessiontag; - TRegexp re("session-.*-.*-.*-.*"); - Int_t i1 = stag.Index(re); - if (i1 == kNPOS) { - Error("LockSession","bad format: %s", sessiontag); - return -1; - } - stag.ReplaceAll("session-",""); - - // Drop query number, if any - Int_t i2 = stag.Index(":q"); - if (i2 != kNPOS) - stag.Remove(i2); - - // Make sure that parent process does not exist anylonger - TString parlog = fSessionDir; - parlog = parlog.Remove(parlog.Index("master-")+strlen("master-")); - parlog += stag; - if (!gSystem->AccessPathName(parlog)) { - PDB(kGlobal, 1) - Info("LockSession", "parent still running: do nothing"); - return -1; - } - - // Lock the query lock file - if (fLock) { - TString qlock = fLock->GetName(); - qlock.ReplaceAll(fSessionTag, stag); - - if (!gSystem->AccessPathName(qlock)) { - *lck = new TProofLockPath(qlock); - if (((*lck)->Lock()) < 0) { - Error("LockSession","problems locking query lock file"); - SafeDelete(*lck); - return -1; - } - } - } - - // We are done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup query dir qdir. - -Int_t TQueryResultManager::CleanupSession(const char *sessiontag) -{ - if (!sessiontag) { - Error("CleanupSession","session tag undefined"); - return -1; - } - - // Query dir - TString qdir = fQueryDir; - qdir.ReplaceAll(Form("session-%s", fSessionTag.Data()), sessiontag); - Int_t idx = qdir.Index(":q"); - if (idx != kNPOS) - qdir.Remove(idx); - if (gSystem->AccessPathName(qdir)) { - Info("CleanupSession","query dir %s does not exist", qdir.Data()); - return -1; - } - - TProofLockPath *lck = 0; - if (LockSession(sessiontag, &lck) == 0) { - - // Cleanup now - gSystem->Exec(Form("%s %s", kRM, qdir.Data())); - - // Unlock and remove the lock file - if (lck) { - gSystem->Unlink(lck->GetName()); - SafeDelete(lck); // Unlocks, if necessary - } - - // We are done - return 0; - } - - // Notify failure - Info("CleanupSession", "could not lock session %s", sessiontag); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save current status of query 'qr' to file name fout. -/// If fout == 0 (default) use the default name. - -void TQueryResultManager::SaveQuery(TProofQueryResult *qr, const char *fout) -{ - if (!qr || qr->IsDraw()) - return; - - // Create dir for specific query - TString querydir = Form("%s/%d",fQueryDir.Data(), qr->GetSeqNum()); - - // Create dir, if needed - if (gSystem->AccessPathName(querydir)) - gSystem->MakeDirectory(querydir); - TString ofn = fout ? fout : Form("%s/query-result.root", querydir.Data()); - - // Recreate file and save query in its current status - TFile *f = TFile::Open(ofn, "RECREATE"); - if (f) { - f->cd(); - if (!(qr->IsArchived())) - qr->SetResultFile(ofn); - qr->Write(); - f->Close(); - delete f; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove everything about query queryref; if defined 'otherlist' will containe -/// the list of removed pointers (already deleted) - -void TQueryResultManager::RemoveQuery(const char *queryref, TList *otherlist) -{ - PDB(kGlobal, 1) - Info("RemoveQuery", "Enter"); - - // Parse reference string - Int_t qry = -1; - TString qdir; - TProofQueryResult *pqr = LocateQuery(queryref, qry, qdir); - // Remove instance in memory - if (pqr) { - if (qry > -1) { - fQueries->Remove(pqr); - if (otherlist) otherlist->Add(pqr); - } else - fPreviousQueries->Remove(pqr); - delete pqr; - pqr = 0; - } - - // Remove the directory - PDB(kGlobal, 1) - Info("RemoveQuery", "removing directory: %s", qdir.Data()); - gSystem->Exec(Form("%s %s", kRM, qdir.Data())); - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove everything about query qr. If soft = TRUE leave a track -/// in memory with the relevant info - -void TQueryResultManager::RemoveQuery(TQueryResult *qr, Bool_t soft) -{ - PDB(kGlobal, 1) - Info("RemoveQuery", "Enter"); - - if (!qr) - return; - - // Remove the directory - TString qdir = fQueryDir; - qdir = qdir.Remove(qdir.Index(kPROOF_QueryDir)+strlen(kPROOF_QueryDir)); - qdir = Form("%s/%s/%d", qdir.Data(), qr->GetTitle(), qr->GetSeqNum()); - PDB(kGlobal, 1) - Info("RemoveQuery", "removing directory: %s", qdir.Data()); - gSystem->Exec(Form("%s %s", kRM, qdir.Data())); - - // Remove from memory lists - if (soft) { - TQueryResult *qrn = qr->CloneInfo(); - Int_t idx = fQueries->IndexOf(qr); - if (idx > -1) - fQueries->AddAt(qrn, idx); - else - SafeDelete(qrn); - } - fQueries->Remove(qr); - SafeDelete(qr); - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Locate query referenced by queryref. Return pointer to instance -/// in memory, if any, or 0. Fills qdir with the query specific directory -/// and qry with the query number for queries processed by this session. - -TProofQueryResult *TQueryResultManager::LocateQuery(TString queryref, Int_t &qry, TString &qdir) -{ - TProofQueryResult *pqr = 0; - - // Find out if the request is a for a local query or for a - // previously processed one - qry = -1; - if (queryref.IsDigit()) { - qry = queryref.Atoi(); - } else if (queryref.Contains(fSessionTag)) { - Int_t i1 = queryref.Index(":q"); - if (i1 != kNPOS) { - queryref.Remove(0,i1+2); - qry = queryref.Atoi(); - } - } - - // Build dir name for specific query - qdir = ""; - if (qry > -1) { - - PDB(kGlobal, 1) - Info("LocateQuery", "local query: %d", qry); - - // Remove query from memory list - if (fQueries) { - TIter nxq(fQueries); - while ((pqr = (TProofQueryResult *) nxq())) { - if (pqr->GetSeqNum() == qry) { - // Dir for specific query - qdir = Form("%s/%d", fQueryDir.Data(), qry); - break; - } - } - } - - } else { - PDB(kGlobal, 1) - Info("LocateQuery", "previously processed query: %s", queryref.Data()); - - // Remove query from memory list - if (fPreviousQueries) { - TIter nxq(fPreviousQueries); - while ((pqr = (TProofQueryResult *) nxq())) { - if (queryref.Contains(pqr->GetTitle()) && - queryref.Contains(pqr->GetName())) - break; - } - } - - queryref.ReplaceAll(":q","/"); - qdir = fQueryDir; - qdir = qdir.Remove(qdir.Index(kPROOF_QueryDir)+strlen(kPROOF_QueryDir)); - qdir = Form("%s/%s", qdir.Data(), queryref.Data()); - } - - // We are done - return pqr; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Final steps after Process() to complete the TQueryResult instance. - -Bool_t TQueryResultManager::FinalizeQuery(TProofQueryResult *pq, - TProof *proof, TVirtualProofPlayer *player) -{ - if (!pq || !proof || !player) { - Warning("FinalizeQuery", "bad inputs: query = %p, proof = %p, player: %p ", - pq ? pq : 0, proof ? proof : 0, player ? player : 0); - return kFALSE; - } - - Int_t qn = pq->GetSeqNum(); - Long64_t np = player->GetEventsProcessed(); - TVirtualProofPlayer::EExitStatus est = player->GetExitStatus(); - TList *out = player->GetOutputList(); - - Float_t cpu = proof->GetCpuTime(); - Long64_t bytes = proof->GetBytesRead(); - - TQueryResult::EQueryStatus st = TQueryResult::kAborted; - - PDB(kGlobal, 2) Info("FinalizeQuery","query #%d", qn); - - PDB(kGlobal, 1) - Info("FinalizeQuery","%.1f %lld", cpu, bytes); - - // Some notification (useful in large logs) - Bool_t save = kTRUE; - switch (est) { - case TVirtualProofPlayer::kAborted: - PDB(kGlobal, 1) - Info("FinalizeQuery", "query %d has been ABORTED <====", qn); - out = 0; - save = kFALSE; - break; - case TVirtualProofPlayer::kStopped: - PDB(kGlobal, 1) - Info("FinalizeQuery", - "query %d has been STOPPED: %lld events processed", qn, np); - st = TQueryResult::kStopped; - break; - case TVirtualProofPlayer::kFinished: - PDB(kGlobal, 1) - Info("FinalizeQuery", - "query %d has been completed: %lld events processed", qn, np); - st = TQueryResult::kCompleted; - break; - default: - Warning("FinalizeQuery", - "query %d: unknown exit status (%d)", qn, player->GetExitStatus()); - } - - // Fill some variables; in the CPU time we do not include anymore the time - // used on the master for preparing and merging, because we want to measure - // the efficiency or farction of time useful for work doen by workers - PDB(kGlobal, 1) - Info("FinalizeQuery", "cpu: %.4f, saved: %.4f, master: %.4f", - cpu, pq->GetUsedCPU() ,GetCpuTime()); -// pq->SetProcessInfo(np, cpu - pq->GetUsedCPU() + GetCpuTime()); - // We take the difference because this is the total CPU time of the session - pq->SetProcessInfo(np, cpu - pq->GetUsedCPU()); - pq->RecordEnd(st, out); - - // Save the logs into the query result instance - AddLogFile(pq); - - // Done - return save; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save current query honouring the max number of queries allowed - -void TQueryResultManager::SaveQuery(TProofQueryResult *pq, Int_t mxq) -{ - // We may need some cleanup - if (mxq > -1) { - if (fQueries && fKeptQueries >= mxq) { - // Find oldest completed and archived query - TQueryResult *fcom = 0; - TQueryResult *farc = 0; - TIter nxq(fQueries); - TQueryResult *qr = 0; - while (fKeptQueries >= mxq) { - while ((qr = (TQueryResult *) nxq())) { - if (qr->IsArchived()) { - if (qr->GetOutputList() && !farc) - farc = qr; - } else if (qr->GetStatus() > TQueryResult::kRunning && !fcom) { - fcom = qr; - } - if (farc && fcom) - break; - } - if (!farc && !fcom) { - break; - } else if (farc) { - RemoveQuery(farc, kTRUE); - fKeptQueries--; - farc = 0; - } else if (fcom) { - RemoveQuery(fcom); - fKeptQueries--; - fcom = 0; - } - } - } - if (fKeptQueries < mxq) { - SaveQuery(pq); - fKeptQueries++; - } else { - TString emsg; - emsg.Form("Too many saved queries (%d): cannot save %s:%s", - fKeptQueries, pq->GetTitle(), pq->GetName()); - if (gProofServ) { - gProofServ->SendAsynMessage(emsg.Data()); - } else { - Warning("SaveQuery", "%s", emsg.Data()); - } - } - } else { - SaveQuery(pq); - fKeptQueries++; - } -} diff --git a/proof/proof/src/TSelVerifyDataSet.cxx b/proof/proof/src/TSelVerifyDataSet.cxx deleted file mode 100644 index 1b472e5c92698..0000000000000 --- a/proof/proof/src/TSelVerifyDataSet.cxx +++ /dev/null @@ -1,365 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 28/06/2011 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TSelVerifyDataSet -\ingroup proofkernel - -Selector to verify dataset in parallel on workers - -*/ - -#define TSelVerifyDataSet_cxx - -#include "TSelVerifyDataSet.h" -#include "TDataSetManager.h" -#include "TDSet.h" -#include "TParameter.h" -#include "TTree.h" -#include "TFile.h" -#include "TNamed.h" -#include "TSystem.h" -#include "TROOT.h" -#include "TFileStager.h" -#include "TProofDebug.h" -#include "TProofServ.h" -#include "TFileCollection.h" -#include "TFileInfo.h" - -ClassImp(TSelVerifyDataSet); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TSelVerifyDataSet::TSelVerifyDataSet(TTree *) -{ - InitMembers(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TSelVerifyDataSet::TSelVerifyDataSet() -{ - InitMembers(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Initialize members - -void TSelVerifyDataSet::InitMembers() -{ - fFopt = -1; - fSopt = 0; - fRopt = 0; - - fAllf = 0; - fCheckstg = 0; - fNonStgf = 0; - fReopen = 0; - fTouch = 0; - fStgf = 0; - fNoaction = 0; - fFullproc = 0; - fLocateonly = 0; - fStageonly = 0; - fDoall = 0; - fGetlistonly = 0; - fScanlist = 0; - fDbg = 0; - - fChangedDs = kFALSE; - fTouched = 0; - fOpened = 0; - fDisappeared = 0; - fSubDataSet = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Worker Begin - -void TSelVerifyDataSet::SlaveBegin(TTree *) -{ - TString dsname, opts; - - TNamed* par = dynamic_cast(fInput->FindObject("PROOF_VerifyDataSet")); - if (par) { - dsname = par->GetTitle(); - } else { - Abort("cannot find dataset name: cannot continue", kAbortProcess); - return; - } - - par = dynamic_cast(fInput->FindObject("PROOF_VerifyDataSetOption")); - if (par) { - opts = par->GetTitle(); - } else { - Abort("cannot find verify options: cannot continue", kAbortProcess); - return; - } - - par = dynamic_cast(fInput->FindObject("PROOF_MSS")); - if (par) { - fMss = par->GetTitle(); - PDB(kSelector, 2) Info("SlaveBegin", "dataset MSS: '%s'", fMss.Data()); - } - - par = dynamic_cast(fInput->FindObject("PROOF_StageOption")); - if (par) { - fStageopts = par->GetTitle(); - PDB(kSelector, 2) Info("SlaveBegin", "dataset stage options: '%s'", fStageopts.Data()); - } - - // Extract the directives - UInt_t o = 0; - if (!opts.IsNull()) { - // Selection options - if (strstr(opts, "allfiles:") || strchr(opts, 'A')) - o |= TDataSetManager::kAllFiles; - else if (strstr(opts, "staged:") || strchr(opts, 'D')) - o |= TDataSetManager::kStagedFiles; - // Pre-action options - if (strstr(opts, "open:") || strchr(opts, 'O')) - o |= TDataSetManager::kReopen; - if (strstr(opts, "touch:") || strchr(opts, 'T')) - o |= TDataSetManager::kTouch; - if (strstr(opts, "nostagedcheck:") || strchr(opts, 'I')) - o |= TDataSetManager::kNoStagedCheck; - // Process options - if (strstr(opts, "noaction:") || strchr(opts, 'N')) - o |= TDataSetManager::kNoAction; - if (strstr(opts, "locateonly:") || strchr(opts, 'L')) - o |= TDataSetManager::kLocateOnly; - if (strstr(opts, "stageonly:") || strchr(opts, 'S')) - o |= TDataSetManager::kStageOnly; - // Auxilliary options - if (strstr(opts, "verbose:") || strchr(opts, 'V')) - o |= TDataSetManager::kDebug; - } else { - // Default - o = TDataSetManager::kReopen | TDataSetManager::kDebug; - } - - PDB(kSelector, 1) Info("SlaveBegin", "o=%d", o); - // File selection - fFopt = ((o & TDataSetManager::kAllFiles)) ? -1 : 0; - if (fFopt >= 0) { - if ((o & TDataSetManager::kStagedFiles)) { - fFopt = 10; - } else { - if ((o & TDataSetManager::kReopen)) fFopt++; - if ((o & TDataSetManager::kTouch)) fFopt++; - } - if ((o & TDataSetManager::kNoStagedCheck)) fFopt += 100; - } else { - if ((o & TDataSetManager::kStagedFiles) || (o & TDataSetManager::kReopen) || (o & TDataSetManager::kTouch)) { - Warning("SlaveBegin", "kAllFiles mode: ignoring kStagedFiles or kReopen" - " or kTouch requests"); - } - if ((o & TDataSetManager::kNoStagedCheck)) fFopt -= 100; - } - PDB(kSelector, 1) Info("SlaveBegin", "fFopt=%d", fFopt); - - // Type of action - fSopt = ((o & TDataSetManager::kNoAction)) ? -1 : 0; - if (fSopt >= 0) { - if ((o & TDataSetManager::kLocateOnly) && (o & TDataSetManager::kStageOnly)) { - Error("SlaveBegin", "kLocateOnly and kStageOnly cannot be processed concurrently"); - return; - } - if ((o & TDataSetManager::kLocateOnly)) fSopt = 1; - if ((o & TDataSetManager::kStageOnly)) fSopt = 2; - } else if ((o & TDataSetManager::kLocateOnly) || (o & TDataSetManager::kStageOnly)) { - Warning("SlaveBegin", "kNoAction mode: ignoring kLocateOnly or kStageOnly requests"); - } - PDB(kSelector, 1) Info("SlaveBegin", "fSopt=%d", fSopt); - - fDbg = ((o & TDataSetManager::kDebug)) ? kTRUE : kFALSE; - - // File selection, Reopen and Touch options - fAllf = (fFopt == -1) ? kTRUE : kFALSE; - fCheckstg = (fFopt >= 100 || fFopt < -1) ? kFALSE : kTRUE; - if (fFopt >= 0) fFopt %= 100; - fNonStgf = (fFopt >= 0 && fFopt < 10) ? kTRUE : kFALSE; - fReopen = (fFopt >= 1 && fFopt < 10) ? kTRUE : kFALSE; - fTouch = (fFopt >= 2 && fFopt < 10) ? kTRUE : kFALSE; - fStgf = (fFopt == 10) ? kTRUE : kFALSE; - - PDB(kSelector, 1) Info("SlaveBegin", - "fAllf=%d fCheckstg=%d fNonStgf=%d fReopen=%d fTouch=%d fStgf=%d", - fAllf, fCheckstg, fNonStgf, fReopen, fTouch, fStgf); - - // File processing options - fNoaction = (fSopt == -1) ? kTRUE : kFALSE; - fFullproc = (fSopt == 0) ? kTRUE : kFALSE; - fLocateonly = (fSopt == 1) ? kTRUE : kFALSE; - fStageonly = (fSopt == 2) ? kTRUE : kFALSE; - - PDB(kSelector, 1) Info("SlaveBegin", - "fNoaction=%d fFullproc=%d fLocateonly=%d fStageonly=%d", - fNoaction, fFullproc, fLocateonly, fStageonly); - - // Run options - fDoall = (fRopt == 0) ? kTRUE : kFALSE; - fGetlistonly = (fRopt == 1) ? kTRUE : kFALSE; - fScanlist = (fRopt == 2) ? kTRUE : kFALSE; - - PDB(kSelector, 1) Info("SlaveBegin", - "fDoall=%d fGetlistonly=%d fScanlist=%d", - fDoall, fGetlistonly, fScanlist); - - TString hostname(TUrl(gSystem->HostName()).GetHostFQDN()); - TString thisordinal = gProofServ ? gProofServ->GetOrdinal() : "n.d"; - TString title = - TString::Format("TSelVerifyDataSet_%s_%s", hostname.Data(), thisordinal.Data()); - fSubDataSet= new TFileCollection(dsname, title); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process a single entry - -Bool_t TSelVerifyDataSet::Process(Long64_t entry) -{ - TDSetElement *fCurrent = 0; - TPair *elemPair = 0; - if (fInput && (elemPair = dynamic_cast - (fInput->FindObject("PROOF_CurrentElement")))) { - if ((fCurrent = dynamic_cast(elemPair->Value()))) - Info("Process", "entry %lld: file: '%s'", entry, fCurrent->GetName()); - } - if (!fCurrent) { - Error("Process", "entry %lld: current element not found!", entry); - return kFALSE; - } - - TFileInfo *fileInfo = dynamic_cast(fCurrent->GetAssocObj(0)); - if (!fileInfo) { - Error("Process", "can not get TFileInfo; returning"); - return kFALSE; - } - - PDB(kSelector, 1) { - Info("Process", "input fileinfo: "); - fileInfo->Print("L"); - } - - TFileStager *stager = 0; - Bool_t createStager = kFALSE; - - TFileInfo* newfileinfo = new TFileInfo(*fileInfo); - newfileinfo->SetIndex(fileInfo->GetIndex()); - - if (fDoall || fGetlistonly) { - - stager = (fMss && strlen(fMss) > 0) ? TFileStager::Open(fMss) : 0; - createStager = (stager) ? kFALSE : kTRUE; - - // Check which files have been staged, this can be replaced by a bulk command, - // once it exists in the xrdclient - - // For real time monitoring - gSystem->DispatchOneEvent(kTRUE); - - Bool_t changed = kFALSE; - Bool_t touched = kFALSE; - Bool_t disappeared = kFALSE; - - TDataSetManager::CheckStagedStatus(newfileinfo, fFopt, -1, 0, stager, createStager, - fDbg, changed, touched, disappeared); - - if (changed) fChangedDs = kTRUE; - if (touched) fTouched++; - if (disappeared) fDisappeared++; - - SafeDelete(stager); - - PDB(kSelector, 1) Info("Process", - "fChangedDs = %d, fTouched = %d disappeared = %d", - fChangedDs, fTouched, fDisappeared); - - // If required to only get the list we are done - if (fGetlistonly) { - Info("Process", "updated fileinfo: "); - newfileinfo->Print("F"); - fSubDataSet->Add(newfileinfo); - return kTRUE; - } - } - - if (!fNoaction && (fDoall || fScanlist)) { - - // Point to the fileinfo - //newStagedFiles = (!fDoall && fScanlist && flist) ? flist : newStagedFiles; - if (!fDoall && fScanlist) { - SafeDelete(newfileinfo); - newfileinfo = new TFileInfo(*fileInfo); - newfileinfo->SetIndex(fileInfo->GetIndex()); - } - - // Loop over now staged files - PDB(kSelector, 1) Info("Process", - "file appear to be newly staged; %s", - newfileinfo->GetFirstUrl()->GetUrl()); - - // If staging files, prepare the stager - if (fLocateonly || fStageonly) { - stager = (fMss && strlen(fMss) > 0) ? TFileStager::Open(fMss) : 0; - createStager = (stager) ? kFALSE : kTRUE; - } - - // Process the file - Bool_t changed = kFALSE; - Bool_t opened = kFALSE; - TDataSetManager::ProcessFile(newfileinfo, fSopt, fCheckstg, fDoall, stager, createStager, fStageopts, - fDbg, changed, opened); - - if (changed) fChangedDs = kTRUE; - if (opened) fOpened++; - } - - PDB(kSelector, 1) { - Info("Process", "updated fileinfo: "); - newfileinfo->Print("L"); - } - fSubDataSet->Add(newfileinfo); - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Worker Terminate - -void TSelVerifyDataSet::SlaveTerminate() -{ - if (fSubDataSet) { - fSubDataSet->Update(); - if (fSubDataSet->GetNFiles() > 0) { - fOutput->Add(fSubDataSet); - Info("SlaveTerminate", - "sub-dataset '%s' added to the output list (%lld files)", - fSubDataSet->GetTitle(), fSubDataSet->GetNFiles()); - } - // Add information for registration - fOutput->Add(new TNamed(TString::Format("DATASET_%s", fSubDataSet->GetName()).Data(),"OT:sortidx:")); - fOutput->Add(new TNamed("PROOFSERV_RegisterDataSet", "")); - } - - // Send the number of files disppeared, opened and mark 'changed'if any fileinfo in the dataset has changed - TString hostname(TUrl(gSystem->HostName()).GetHostFQDN()); - TString thisordinal = gProofServ ? gProofServ->GetOrdinal() : "n.d"; - TString sfdisppeared= TString::Format("PROOF_NoFilesDisppeared_%s_%s", hostname.Data(), thisordinal.Data()); - fOutput->Add(new TParameter(sfdisppeared.Data(), fDisappeared)); - TString sfOpened= TString::Format("PROOF_NoFilesOpened_%s_%s", hostname.Data(), thisordinal.Data()); - fOutput->Add(new TParameter(sfOpened.Data(), fOpened)); - TString sfTouched = TString::Format("PROOF_NoFilesTouched_%s_%s", hostname.Data(), thisordinal.Data()); - fOutput->Add(new TParameter(sfTouched.Data(), fTouched)); - TString schanged= TString::Format("PROOF_DataSetChanged_%s_%s", hostname.Data(), thisordinal.Data()); - fOutput->Add(new TParameter(schanged.Data(), fChangedDs)); -} diff --git a/proof/proof/src/TSlave.cxx b/proof/proof/src/TSlave.cxx deleted file mode 100644 index 20e59f5501ff8..0000000000000 --- a/proof/proof/src/TSlave.cxx +++ /dev/null @@ -1,668 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 14/02/97 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TSlave -\ingroup proofkernel - -Class describing a PROOF worker server. It contains information like the -workers host name, ordinal number, performance index, socket, etc. -Objects of this class can only be created via TProof member functions. - -*/ - -#include - -#include "RConfigure.h" -#include "TApplication.h" -#include "TSlave.h" -#include "TSlaveLite.h" -#include "TProof.h" -#include "TSystem.h" -#include "TROOT.h" -#include "TUrl.h" -#include "TMessage.h" -#include "TError.h" -#include "TVirtualMutex.h" -#include "TSocket.h" -#include "TObjString.h" - -ClassImp(TSlave); - -// Hook for the TXSlave constructor -TSlave_t TSlave::fgTXSlaveHook = 0; - -//////////////////////////////////////////////////////////////////////////////// -/// Create a PROOF slave object. Called via the TProof ctor. - -TSlave::TSlave(const char *url, const char *ord, Int_t perf, - const char *image, TProof *proof, Int_t stype, - const char *workdir, const char *msd, Int_t) - : fImage(image), fProofWorkDir(workdir), - fWorkDir(workdir), fPort(-1), - fOrdinal(ord), fPerfIdx(perf), - fProtocol(0), fSocket(0), fProof(proof), - fInput(0), fBytesRead(0), fRealTime(0), - fCpuTime(0), fSlaveType((ESlaveType)stype), fStatus(TSlave::kInvalid), - fParallel(0), fMsd(msd) -{ - fName = TUrl(url).GetHostFQDN(); - fPort = TUrl(url).GetPort(); - - Init(url, -1, stype); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Default constructor used by derived classes - -TSlave::TSlave() -{ - fPort = -1; - fOrdinal = "-1"; - fPerfIdx = -1; - fProof = 0; - fSlaveType = kMaster; - fProtocol = 0; - fSocket = 0; - fInput = 0; - fBytesRead = 0; - fRealTime = 0; - fCpuTime = 0; - fStatus = kInvalid; - fParallel = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Init a PROOF slave object. Called via the TSlave ctor. -/// The Init method is technology specific and is overwritten by derived -/// classes. - -void TSlave::Init(const char *host, Int_t port, Int_t stype) -{ - // The url contains information about the server type: make sure - // it is 'proofd' or alike - TString proto = fProof->fUrl.GetProtocol(); - proto.Insert(5, 'd'); - - TUrl hurl(host); - hurl.SetProtocol(proto); - if (port > 0) - hurl.SetPort(port); - - // Add information about our status (Client or Master) - TString iam; - if (fProof->IsMaster() && stype == kSlave) { - iam = "Master"; - hurl.SetOptions("SM"); - } else if (fProof->IsMaster() && stype == kMaster) { - iam = "Master"; - hurl.SetOptions("MM"); - } else if (!fProof->IsMaster() && stype == kMaster) { - iam = "Local Client"; - hurl.SetOptions("MC"); - } else { - Error("Init","Impossible PROOF <-> SlaveType Configuration Requested"); - R__ASSERT(0); - } - - // Open authenticated connection to remote PROOF slave server. - // If a connection was already open (fSocket != 0), re-use it - // to perform authentication (optimization needed to avoid a double - // opening in case this is called by TXSlave). - Int_t wsize = 65536; - fSocket = TSocket::CreateAuthSocket(hurl.GetUrl(), 0, wsize, fSocket); - - if (!fSocket || !fSocket->IsAuthenticated()) { - SafeDelete(fSocket); - return; - } - - // Remove socket from global TROOT socket list. Only the TProof object, - // representing all slave sockets, will be added to this list. This will - // ensure the correct termination of all proof servers in case the - // root session terminates. - { - R__LOCKGUARD(gROOTMutex); - gROOT->GetListOfSockets()->Remove(fSocket); - } - - // Fill some useful info - fUser = fSocket->GetSecContext()->GetUser(); - PDB(kGlobal,3) { - Info("Init","%s: fUser is .... %s", iam.Data(), fUser.Data()); - } - - if (fSocket->GetRemoteProtocol() >= 14 ) { - TMessage m(kPROOF_SETENV); - - const TList *envs = TProof::GetEnvVars(); - if (envs != 0 ) { - TIter next(envs); - for (TObject *o = next(); o != 0; o = next()) { - TNamed *env = dynamic_cast(o); - if (env != 0) { - TString def = Form("%s=%s", env->GetName(), env->GetTitle()); - const char *p = def.Data(); - m << p; - } - } - } - fSocket->Send(m); - } else { - Info("Init","** NOT ** Sending kPROOF_SETENV RemoteProtocol : %d", - fSocket->GetRemoteProtocol()); - } - - char buf[512]; - fSocket->Recv(buf, sizeof(buf)); - if (strcmp(buf, "Okay")) { - Printf("%s", buf); - SafeDelete(fSocket); - return; - } - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Init a PROOF slave object. Called via the TSlave ctor. -/// The Init method is technology specific and is overwritten by derived -/// classes. - -Int_t TSlave::SetupServ(Int_t stype, const char *conffile) -{ - // get back startup message of proofserv (we are now talking with - // the real proofserver and not anymore with the proofd front-end) - Int_t what; - char buf[512]; - if (fSocket->Recv(buf, sizeof(buf), what) <= 0) { - Error("SetupServ", "failed to receive slave startup message"); - SafeDelete(fSocket); - return -1; - } - - if (what == kMESS_NOTOK) { - SafeDelete(fSocket); - return -1; - } - - // exchange protocol level between client and master and between - // master and slave - if (fSocket->Send(kPROOF_Protocol, kROOTD_PROTOCOL) != 2*sizeof(Int_t)) { - Error("SetupServ", "failed to send local PROOF protocol"); - SafeDelete(fSocket); - return -1; - } - - if (fSocket->Recv(fProtocol, what) != 2*sizeof(Int_t)) { - Error("SetupServ", "failed to receive remote PROOF protocol"); - SafeDelete(fSocket); - return -1; - } - - // protocols less than 4 are incompatible - if (fProtocol < 4) { - Error("SetupServ", "incompatible PROOF versions (remote version" - " must be >= 4, is %d)", fProtocol); - SafeDelete(fSocket); - return -1; - } - - fProof->fProtocol = fProtocol; // protocol of last slave on master - - if (fProtocol < 5) { - // - // Setup authentication related stuff for ald versions - Bool_t isMaster = (stype == kMaster); - TString wconf = isMaster ? TString(conffile) : fProofWorkDir; - if (OldAuthSetup(isMaster, wconf) != 0) { - Error("SetupServ", "OldAuthSetup: failed to setup authentication"); - SafeDelete(fSocket); - return -1; - } - } else { - // - // Send ordinal (and config) info to slave (or master) - TMessage mess; - if (stype == kMaster) - mess << fUser << fOrdinal << TString(conffile); - else - mess << fUser << fOrdinal << fProofWorkDir; - - if (fSocket->Send(mess) < 0) { - Error("SetupServ", "failed to send ordinal and config info"); - SafeDelete(fSocket); - return -1; - } - } - - // set some socket options - fSocket->SetOption(kNoDelay, 1); - - // Set active state - fStatus = kActive; - - // We are done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Init a PROOF slave object using the connection opened via s. Used to -/// avoid double opening when an attempt via TXSlave found a remote proofd. - -void TSlave::Init(TSocket *s, Int_t stype) -{ - fSocket = s; - TSlave::Init(s->GetInetAddress().GetHostName(), s->GetPort(), stype); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destroy slave. - -TSlave::~TSlave() -{ - Close(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close slave socket. - -void TSlave::Close(Option_t *opt) -{ - if (fSocket) { - - // If local client ... - if (!(fProof->IsMaster()) && !strncasecmp(opt,"S",1)) { - // ... tell master and slaves to stop - Interrupt(TProof::kShutdownInterrupt); - } - - // deactivate used sec context if talking to proofd daemon running - // an old protocol (sec context disactivated remotely) - TSecContext *sc = fSocket->GetSecContext(); - if (sc && sc->IsActive()) { - TIter last(sc->GetSecContextCleanup(), kIterBackward); - TSecContextCleanup *nscc = 0; - while ((nscc = (TSecContextCleanup *)last())) { - if (nscc->GetType() == TSocket::kPROOFD && - nscc->GetProtocol() < 9) { - sc->DeActivate(""); - break; - } - } - } - } - - SafeDelete(fInput); - SafeDelete(fSocket); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Used to sort slaves by performance index. - -Int_t TSlave::Compare(const TObject *obj) const -{ - const TSlave *sl = dynamic_cast(obj); - - if (!sl) { - Error("Compare", "input is not a TSlave object"); - return 0; - } - - if (fPerfIdx > sl->GetPerfIdx()) return 1; - if (fPerfIdx < sl->GetPerfIdx()) return -1; - const char *myord = GetOrdinal(); - const char *otherord = sl->GetOrdinal(); - while (myord && otherord) { - Int_t myval = atoi(myord); - Int_t otherval = atoi(otherord); - if (myval < otherval) return 1; - if (myval > otherval) return -1; - myord = strchr(myord, '.'); - if (myord) myord++; - otherord = strchr(otherord, '.'); - if (otherord) otherord++; - } - if (myord) return -1; - if (otherord) return 1; - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Printf info about slave. - -void TSlave::Print(Option_t *) const -{ - TString sc; - - const char *sst[] = { "invalid" , "valid", "inactive" }; - Int_t st = fSocket ? ((fStatus == kInactive) ? 2 : 1) : 0; - - Printf("*** Worker %s (%s)", fOrdinal.Data(), sst[st]); - Printf(" Host name: %s", GetName()); - Printf(" Port number: %d", GetPort()); - Printf(" Worker session tag: %s", GetSessionTag()); - Printf(" ROOT version|rev|tag: %s", GetROOTVersion()); - Printf(" Architecture-Compiler: %s", GetArchCompiler()); - if (fSocket) { - if (strlen(GetGroup()) > 0) { - Printf(" User/Group: %s/%s", GetUser(), GetGroup()); - } else { - Printf(" User: %s", GetUser()); - } - if (fSocket->GetSecContext()) - Printf(" Security context: %s", fSocket->GetSecContext()->AsString(sc)); - Printf(" Proofd protocol version: %d", fSocket->GetRemoteProtocol()); - Printf(" Image name: %s", GetImage()); - Printf(" Working directory: %s", GetWorkDir()); - Printf(" Performance index: %d", GetPerfIdx()); - Printf(" MB's processed: %.2f", float(GetBytesRead())/(1024*1024)); - Printf(" MB's sent: %.2f", float(fSocket->GetBytesRecv())/(1024*1024)); - Printf(" MB's received: %.2f", float(fSocket->GetBytesSent())/(1024*1024)); - Printf(" Real time used (s): %.3f", GetRealTime()); - Printf(" CPU time used (s): %.3f", GetCpuTime()); - } else { - if (strlen(GetGroup()) > 0) { - Printf(" User/Group: %s/%s", GetUser(), GetGroup()); - } else { - Printf(" User: %s", GetUser()); - } - Printf(" Security context:"); - Printf(" Proofd protocol version:"); - Printf(" Image name: %s", GetImage()); - Printf(" Working directory: %s", GetWorkDir()); - Printf(" Performance index: %d", GetPerfIdx()); - Printf(" MB's processed: %.2f", float(GetBytesRead())/(1024*1024)); - Printf(" MB's sent:"); - Printf(" MB's received:"); - Printf(" Real time used (s): %.3f", GetRealTime()); - Printf(" CPU time used (s): %.3f", GetCpuTime()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Adopt and register input handler for this slave. Handler will be deleted -/// by the slave. - -void TSlave::SetInputHandler(TFileHandler *ih) -{ - fInput = ih; - fInput->Add(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Setup authentication related stuff for old versions. -/// Provided for backward compatibility. - -Int_t TSlave::OldAuthSetup(Bool_t master, TString wconf) -{ - static OldSlaveAuthSetup_t oldAuthSetupHook = 0; - - if (!oldAuthSetupHook) { - // Load libraries needed for (server) authentication ... - TString authlib = "libRootAuth"; - char *p = 0; - // The generic one - if ((p = gSystem->DynamicPathName(authlib, kTRUE))) { - delete[] p; - if (gSystem->Load(authlib) == -1) { - Error("OldAuthSetup", "can't load %s",authlib.Data()); - return kFALSE; - } - } else { - Error("OldAuthSetup", "can't locate %s",authlib.Data()); - return -1; - } - // - // Locate OldSlaveAuthSetup - Func_t f = gSystem->DynFindSymbol(authlib,"OldSlaveAuthSetup"); - if (f) - oldAuthSetupHook = (OldSlaveAuthSetup_t)(f); - else { - Error("OldAuthSetup", "can't find OldSlaveAuthSetup"); - return -1; - } - } - // - // Setup - return (*oldAuthSetupHook)(fSocket, master, fOrdinal, wconf); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static method returning the appropriate TSlave object for the remote -/// server. - -TSlave *TSlave::Create(const char *url, const char *ord, Int_t perf, - const char *image, TProof *proof, Int_t stype, - const char *workdir, const char *msd, Int_t nwk) -{ - TSlave *s = 0; - - // Check if we are setting up a lite version - if (!strcmp(url, "lite")) { - return new TSlaveLite(ord, perf, image, proof, stype, workdir, msd); - } - - // No need to try a XPD connection in some well defined cases - Bool_t tryxpd = kTRUE; - if (!(proof->IsMaster())) { - if (proof->IsProofd()) - tryxpd = kFALSE; - } else { - if (gApplication && (gApplication->Argc() < 3 || - (gApplication->Argc() > 2 && gApplication->Argv(2) && - strncmp(gApplication->Argv(2),"xpd",3)))) - tryxpd = kFALSE; - } - - // We do this without the plugin manager because it blocks the CINT mutex - // breaking the parallel startup - if (!fgTXSlaveHook) { - - // Load the library containing TXSlave ... - TString proofxlib = "libProofx"; - char *p = 0; - if ((p = gSystem->DynamicPathName(proofxlib, kTRUE))) { - delete[] p; - if (gSystem->Load(proofxlib) == -1) - ::Error("TSlave::Create", "can't load %s", proofxlib.Data()); - } else - ::Error("TSlave::Create", "can't locate %s", proofxlib.Data()); - } - - // Load the right class - if (fgTXSlaveHook && tryxpd) { - s = (*fgTXSlaveHook)(url, ord, perf, image, proof, stype, workdir, msd, nwk); - } else { - s = new TSlave(url, ord, perf, image, proof, stype, workdir, msd); - } - - return s; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Ping the remote master or slave servers. -/// Returns 0 if ok, -1 in case of error - -Int_t TSlave::Ping() -{ - if (!IsValid()) return -1; - - TMessage mess(kPROOF_PING | kMESS_ACK); - fSocket->Send(mess); - if (fSocket->Send(mess) == -1) { - Warning("Ping","%s: acknowledgement not received", GetOrdinal()); - return -1; - } - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send interrupt OOB byte to master or slave servers. -/// Returns 0 if ok, -1 in case of error - -void TSlave::Interrupt(Int_t type) -{ - if (!IsValid()) return; - - char oobc = (char) type; - const int kBufSize = 1024; - char waste[kBufSize]; - - // Send one byte out-of-band message to server - if (fSocket->SendRaw(&oobc, 1, kOob) <= 0) { - Error("Interrupt", "error sending oobc to slave %s", GetOrdinal()); - return; - } - - if (type == TProof::kHardInterrupt) { - char oob_byte; - int n, nch, nbytes = 0, nloop = 0; - - // Receive the OOB byte - while ((n = fSocket->RecvRaw(&oob_byte, 1, kOob)) < 0) { - if (n == -2) { // EWOULDBLOCK - // - // The OOB data has not yet arrived: flush the input stream - // - // In some systems (Solaris) regular recv() does not return upon - // receipt of the oob byte, which makes the below call to recv() - // block indefinitely if there are no other data in the queue. - // FIONREAD ioctl can be used to check if there are actually any - // data to be flushed. If not, wait for a while for the oob byte - // to arrive and try to read it again. - // - fSocket->GetOption(kBytesToRead, nch); - if (nch == 0) { - gSystem->Sleep(1000); - continue; - } - - if (nch > kBufSize) nch = kBufSize; - n = fSocket->RecvRaw(waste, nch); - if (n <= 0) { - Error("Interrupt", "error receiving waste from slave %s", - GetOrdinal()); - break; - } - nbytes += n; - } else if (n == -3) { // EINVAL - // - // The OOB data has not arrived yet - // - gSystem->Sleep(100); - if (++nloop > 100) { // 10 seconds time-out - Error("Interrupt", "server %s does not respond", GetOrdinal()); - break; - } - } else { - Error("Interrupt", "error receiving OOB from server %s", - GetOrdinal()); - break; - } - } - - // - // Continue flushing the input socket stream until the OOB - // mark is reached - // - while (1) { - int atmark; - - fSocket->GetOption(kAtMark, atmark); - - if (atmark) - break; - - // find out number of bytes to read before atmark - fSocket->GetOption(kBytesToRead, nch); - if (nch == 0) { - gSystem->Sleep(1000); - continue; - } - - if (nch > kBufSize) nch = kBufSize; - n = fSocket->RecvRaw(waste, nch); - if (n <= 0) { - Error("Interrupt", "error receiving waste (2) from slave %s", - GetOrdinal()); - break; - } - nbytes += n; - } - if (nbytes > 0) { - if (fProof->IsMaster()) - Info("Interrupt", "slave %s:%s synchronized: %d bytes discarded", - GetName(), GetOrdinal(), nbytes); - else - Info("Interrupt", "PROOF synchronized: %d bytes discarded", nbytes); - } - - // Get log file from master or slave after a hard interrupt - fProof->Collect(this); - - } else if (type == TProof::kSoftInterrupt) { - - // Get log file from master or slave after a soft interrupt - fProof->Collect(this); - - } else if (type == TProof::kShutdownInterrupt) { - - ; // nothing expected to be returned - - } else { - - // Unexpected message, just receive log file - fProof->Collect(this); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Sent stop/abort request to PROOF server. - -void TSlave::StopProcess(Bool_t abort, Int_t timeout) -{ - // Notify the remote counterpart - TMessage msg(kPROOF_STOPPROCESS); - msg << abort; - if (fProof->fProtocol > 9) - msg << timeout; - fSocket->Send(msg); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send message to intermediate coordinator. Only meaningful when there is one, -/// i.e. in XPD framework - -TObjString *TSlave::SendCoordinator(Int_t, const char *, Int_t) -{ - if (gDebug > 0) - Info("SendCoordinator","method not implemented for this communication layer"); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set an alias for this session. If reconnection is supported, the alias -/// will be communicated to the remote coordinator so that it can be recovered -/// when reconnecting - -void TSlave::SetAlias(const char *) -{ - if (gDebug > 0) - Info("SetAlias","method not implemented for this communication layer"); - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set hook to TXSlave ctor - -void TSlave::SetTXSlaveHook(TSlave_t xslavehook) -{ - fgTXSlaveHook = xslavehook; -} diff --git a/proof/proof/src/TSlaveLite.cxx b/proof/proof/src/TSlaveLite.cxx deleted file mode 100644 index 14d8475120175..0000000000000 --- a/proof/proof/src/TSlaveLite.cxx +++ /dev/null @@ -1,186 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Gerardo Ganis March 2008 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TSlaveLite -\ingroup proofkernel - -Version of TSlave for local worker servers. -See TSlave for details. - -*/ - - - -////////////////////////////////////////////////////////////////////////// -// // -// TSlaveLite // -// // -// This is the version of TSlave for local worker servers. // -// See TSlave for details. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "RConfigure.h" -#include "TSlaveLite.h" -#include "TProof.h" -#include "TProofServ.h" -#include "TSystem.h" -#include "TROOT.h" -#include "TUrl.h" -#include "TMessage.h" -#include "TMonitor.h" -#include "TError.h" -#include "TSocket.h" -#include "TSysEvtHandler.h" -#include "TVirtualMutex.h" - -ClassImp(TSlaveLite); - -//______________________________________________________________________________ -//---- error handling ---------------------------------------------------------- -//---- Needed to avoid blocking on the CINT mutex in printouts ----------------- - -//////////////////////////////////////////////////////////////////////////////// -/// Interface to ErrorHandler (protected). - -void TSlaveLite::DoError(int level, const char *location, - const char *fmt, va_list va) const -{ - ::ErrorHandler(level, Form("TSlaveLite::%s", location), fmt, va); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a PROOF slave object. Called via the TProof ctor. - -TSlaveLite::TSlaveLite(const char *ord, Int_t perf, - const char *image, TProof *proof, Int_t stype, - const char *workdir, const char *msd, Int_t) : TSlave() -{ - fName = ord; // Need this during the setup phase; see end of SetupServ - fImage = image; - fProofWorkDir = workdir; - fWorkDir = workdir; - fOrdinal = ord; - fPerfIdx = perf; - fProof = proof; - fSlaveType = (ESlaveType)stype; - fMsd = msd; - fIntHandler = 0; - fValid = kFALSE; - fProtocol = kPROOF_Protocol; - - if (fPerfIdx > 0) Init(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Init a PROOF worker object. Called via the TSlaveLite ctor. - -void TSlaveLite::Init() -{ - // Command to be executed - TString cmd; - cmd.Form(". %s/worker-%s.env; export ROOTBINDIR=\"%s\"; %s/proofserv proofslave lite %d %d 0&", - fWorkDir.Data(), fOrdinal.Data(), TROOT::GetBinDir().Data(), TROOT::GetBinDir().Data(), - gSystem->GetPid(), gDebug); - // Execute - if (gSystem->Exec(cmd) != 0) { - Error("Init", "an error occured while executing 'proofserv'"); - SetBit(kInvalidObject); - return; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Init a PROOF slave object. Called via the TSlaveLite ctor. -/// The Init method is technology specific and is overwritten by derived -/// classes. - -Int_t TSlaveLite::SetupServ(Int_t, const char *) -{ - // Get back startup message of proofserv (we are now talking with - // the real proofserver and not anymore with the proofd front-end) - Int_t what; - char buf[512]; - if (fSocket->Recv(buf, sizeof(buf), what) <= 0) { - Error("SetupServ", "failed to receive slave startup message"); - Close("S"); - SafeDelete(fSocket); - fValid = kFALSE; - return -1; - } - - if (what == kMESS_NOTOK) { - SafeDelete(fSocket); - fValid = kFALSE; - return -1; - } - - // Receive the unique tag and save it as name of this object - TMessage *msg = 0; - if (fSocket->Recv(msg) <= 0 || !msg || msg->What() != kPROOF_SESSIONTAG) { - Error("SetupServ", "failed to receive unique session tag"); - Close("S"); - SafeDelete(fSocket); - fValid = kFALSE; - return -1; - } - // Extract the unique tag - (*msg) >> fSessionTag; - - // Set the real name (temporarly set to ordinal for the setup) - fName = gSystem->HostName(); - - // We are done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destroy slave. - -TSlaveLite::~TSlaveLite() -{ - Close(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close slave socket. - -void TSlaveLite::Close(Option_t *opt) -{ - if (fSocket) - // Closing socket ... - fSocket->Close(opt); - - SafeDelete(fInput); - SafeDelete(fSocket); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Printf info about slave. - -void TSlaveLite::Print(Option_t *) const -{ - const char *sst[] = { "invalid" , "valid", "inactive" }; - Int_t st = fSocket ? ((fStatus == kInactive) ? 2 : 1) : 0; - - Printf("*** Worker %s (%s)", fOrdinal.Data(), sst[st]); - Printf(" Worker session tag: %s", GetSessionTag()); - Printf(" ROOT version|rev|tag: %s", GetROOTVersion()); - Printf(" Architecture-Compiler: %s", GetArchCompiler()); - if (fSocket) { - Printf(" Working directory: %s", GetWorkDir()); - Printf(" MB's processed: %.2f", float(GetBytesRead())/(1024*1024)); - Printf(" MB's sent: %.2f", float(fSocket->GetBytesRecv())/(1024*1024)); - Printf(" MB's received: %.2f", float(fSocket->GetBytesSent())/(1024*1024)); - Printf(" Real time used (s): %.3f", GetRealTime()); - Printf(" CPU time used (s): %.3f", GetCpuTime()); - } -} diff --git a/proof/proof/src/TVirtualPacketizer.cxx b/proof/proof/src/TVirtualPacketizer.cxx deleted file mode 100644 index 0434329344881..0000000000000 --- a/proof/proof/src/TVirtualPacketizer.cxx +++ /dev/null @@ -1,448 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 9/7/2002 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TVirtualPacketizer -\ingroup proofkernel - -The packetizer is a load balancing object created for each query. -It generates packets to be processed on PROOF worker servers. -A packet is an event range (begin entry and number of entries) or -object range (first object and number of objects) in a TTree -(entries) or a directory (objects) in a file. -Packets are generated taking into account the performance of the -remote machine, the time it took to process a previous packet on -the remote machine, the locality of the database files, etc. - -TVirtualPacketizer includes common parts of PROOF packetizers. -Look in subclasses for details. -The default packetizer is TPacketizerAdaptive (TPacketizer for Proof-Lite). -To use an alternative one, for instance - the TPacketizer, call: -proof->SetParameter("PROOF_Packetizer", "TPacketizer"); - -*/ - -#include "TVirtualPacketizer.h" -#include "TEnv.h" -#include "TFile.h" -#include "TTree.h" -#include "TKey.h" -#include "TDSet.h" -#include "TError.h" -#include "TEventList.h" -#include "TEntryList.h" -#include "TMap.h" -#include "TMessage.h" -#include "TParameter.h" - -#include "TProof.h" -#include "TProofDebug.h" -#include "TVirtualProofPlayer.h" -#include "TProofServ.h" -#include "TSlave.h" -#include "TSocket.h" -#include "TTimer.h" -#include "TUrl.h" -#include "TMath.h" -#include "TMonitor.h" -#include "TNtuple.h" -#include "TNtupleD.h" -#include "TVirtualPerfStats.h" - -ClassImp(TVirtualPacketizer); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor. - -TVirtualPacketizer::TVirtualPacketizer(TList *input, TProofProgressStatus *st) -{ - fInput = input; - // General configuration parameters - fMinPacketTime = 3; - Double_t minPacketTime = 0; - if (TProof::GetParameter(input, "PROOF_MinPacketTime", minPacketTime) == 0) { - Info("TVirtualPacketizer", "setting minimum time for a packet to %f", - minPacketTime); - fMinPacketTime = (Int_t) minPacketTime; - } - fMaxPacketTime = 20; - Double_t maxPacketTime = 0; - if (TProof::GetParameter(input, "PROOF_MaxPacketTime", maxPacketTime) == 0) { - Info("TVirtualPacketizer", "setting maximum packet time for a packet to %f", - maxPacketTime); - fMaxPacketTime = (Int_t) maxPacketTime; - } - ResetBit(TVirtualPacketizer::kIsTree); - - // Create the list to save them in the query result (each derived packetizer is - // responsible to update this coherently) - fConfigParams = new TList; - fConfigParams->SetName("PROOF_PacketizerConfigParams"); - fConfigParams->Add(new TParameter("PROOF_MinPacketTime", fMinPacketTime)); - fConfigParams->Add(new TParameter("PROOF_MaxPacketTime", fMaxPacketTime)); - - fProgressStatus = st; - if (!fProgressStatus) { - Error("TVirtualPacketizer", "No progress status"); - return; - } - fTotalEntries = 0; - fValid = kTRUE; - fStop = kFALSE; - fFailedPackets = 0; - fDataSet = ""; - fSlaveStats = 0; - - // Performance monitoring - fStartTime = gSystem->Now(); - SetBit(TVirtualPacketizer::kIsInitializing); - ResetBit(TVirtualPacketizer::kIsDone); - fInitTime = 0; - fProcTime = 0; - fTimeUpdt = -1.; - - // Init circularity ntple for performance calculations - fCircProg = new TNtupleD("CircNtuple","Circular progress info","tm:ev:mb:rc:al"); - fCircN = 5; - TProof::GetParameter(input, "PROOF_ProgressCircularity", fCircN); - fCircProg->SetCircular(fCircN); - fCircProg->SetDirectory(0); - - // Check if we need to start the progress timer (multi-packetizers do not want - // timers from the packetizers they control ...). Also submasters do not need - // that (the progress timer is the one at the top master). - TString startProgress("yes"); - TProof::GetParameter(input, "PROOF_StartProgressTimer", startProgress); - // If we are on a submaster, check if there is something else to do - if (gProofServ && gProofServ->IsMaster() && !gProofServ->IsTopMaster()) startProgress = "no"; - - // Init progress timer, if requested - // The timer is destroyed (and therefore stopped) by the relevant TPacketizer implementation - // in GetNextPacket when end of work is detected. - fProgress = 0; - if (startProgress == "yes") { - Long_t period = 500; - TProof::GetParameter(input, "PROOF_ProgressPeriod", period); - fProgress = new TTimer; - fProgress->SetObject(this); - fProgress->Start(period, kFALSE); - } - - // Init ntple to store active workers vs processing time - fProgressPerf = 0; - TString saveProgressPerf("no"); - if (TProof::GetParameter(input, "PROOF_SaveProgressPerf", saveProgressPerf) == 0) { - if (fProgress && saveProgressPerf == "yes") - fProgressPerf = new TNtuple("PROOF_ProgressPerfNtuple", - "{Active workers, evt rate, MB read} vs processing time", "tm:aw:er:mb:ns"); - } - fProcTimeLast = -1.; - fActWrksLast = -1; - fEvtRateLast = -1.; - fMBsReadLast = -1.; - fEffSessLast = -1.; - fAWLastFill = kFALSE; - fReportPeriod = -1.; - - // Whether to send estimated values for the progress info - TString estopt; - if (TProof::GetParameter(input, "PROOF_RateEstimation", estopt) != 0 || - estopt.IsNull()) { - // Parse option from the env - estopt = gEnv->GetValue("Proof.RateEstimation", ""); - } - fUseEstOpt = kEstOff; - if (estopt == "current") - fUseEstOpt = kEstCurrent; - else if (estopt == "average") - fUseEstOpt = kEstAverage; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TVirtualPacketizer::~TVirtualPacketizer() -{ - SafeDelete(fCircProg); - SafeDelete(fProgress); - SafeDelete(fFailedPackets); - SafeDelete(fConfigParams); - SafeDelete(fProgressPerf); - fProgressStatus = 0; // belongs to the player -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get entries. - -Long64_t TVirtualPacketizer::GetEntries(Bool_t tree, TDSetElement *e) -{ - Long64_t entries; - TFile *file = TFile::Open(e->GetFileName()); - - if (!file || (file && file->IsZombie())) { - const char *emsg = (file) ? strerror(file->GetErrno()) : ""; - Error("GetEntries","Cannot open file: %s (%s)", e->GetFileName(), emsg); - return -1; - } - - TDirectory *dirsave = gDirectory; - if ( ! file->cd(e->GetDirectory()) ) { - Error("GetEntries","Cannot cd to: %s", e->GetDirectory() ); - delete file; - return -1; - } - TDirectory *dir = gDirectory; - dirsave->cd(); - - if ( tree ) { - TKey *key = dir->GetKey(e->GetObjName()); - if ( key == 0 ) { - Error("GetEntries","Cannot find tree \"%s\" in %s", - e->GetObjName(), e->GetFileName() ); - delete file; - return -1; - } - TTree *t = (TTree *) key->ReadObj(); - if ( t == 0 ) { - // Error always reported? - delete file; - return -1; - } - entries = (Long64_t) t->GetEntries(); - delete t; - - } else { - TList *keys = dir->GetListOfKeys(); - entries = keys->GetSize(); - } - - delete file; - - return entries; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next packet. - -TDSetElement *TVirtualPacketizer::GetNextPacket(TSlave *, TMessage *) -{ - AbstractMethod("GetNextPacket"); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Stop process. - -void TVirtualPacketizer::StopProcess(Bool_t /*abort*/, Bool_t stoptimer) -{ - fStop = kTRUE; - if (stoptimer) HandleTimer(0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Creates a new TDSetElement from from base packet starting from -/// the first entry with num entries. -/// The function returns a new created objects which have to be deleted. - -TDSetElement* TVirtualPacketizer::CreateNewPacket(TDSetElement* base, - Long64_t first, Long64_t num) -{ - TDSetElement* elem = new TDSetElement(base->GetFileName(), base->GetObjName(), - base->GetDirectory(), first, num, - 0, fDataSet.Data()); - - // create TDSetElements for all the friends of elem. - TList *friends = base->GetListOfFriends(); - if (friends) { - TIter nxf(friends); - TDSetElement *fe = 0; - while ((fe = (TDSetElement *) nxf())) { - PDB(kLoop,2) - Info("CreateNewPacket", "friend: file '%s', obj:'%s'", - fe->GetFileName(), fe->GetObjName()); - TDSetElement *xfe = new TDSetElement(fe->GetFileName(), fe->GetObjName(), - fe->GetDirectory(), first, num); - // The alias, if any, is in the element name options ('friend_alias=|') - elem->AddFriend(xfe, 0); - } - } - - return elem; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send progress message to client. - -Bool_t TVirtualPacketizer::HandleTimer(TTimer *) -{ - PDB(kPacketizer,2) - Info("HandleTimer", "fProgress: %p, isDone: %d", - fProgress, TestBit(TVirtualPacketizer::kIsDone)); - - if (fProgress == 0 || TestBit(TVirtualPacketizer::kIsDone)) { - // Make sure that the timer is stopped - if (fProgress) fProgress->Stop(); - return kFALSE; - } - - // Prepare progress info - TTime tnow = gSystem->Now(); - Float_t now = Long64_t(tnow - fStartTime) / (Float_t)1000.; - Long64_t estent = GetEntriesProcessed(); - Long64_t estmb = GetBytesRead(); - Long64_t estrc = GetReadCalls(); - - // Times and counters - Float_t evtrti = -1., mbrti = -1.; - if (TestBit(TVirtualPacketizer::kIsInitializing)) { - // Initialization - fInitTime = now; - } else { - // Fill the reference as first - if (fCircProg->GetEntries() <= 0) { - fCircProg->Fill((Double_t)0., 0., 0., 0., 0.); - } - // Time between updates - fTimeUpdt = now - fProcTime; - // Update proc time - fProcTime = now - fInitTime; - // Get the last entry - Double_t *ar = fCircProg->GetArgs(); - fCircProg->GetEntry(fCircProg->GetEntries()-1); - // The current rate - Bool_t all = kTRUE; - evtrti = GetCurrentRate(all); - Double_t xall = (all) ? 1. : 0.; - GetEstEntriesProcessed(0, estent, estmb, estrc); - if (estent >= fTotalEntries) { - estent = GetEntriesProcessed(); - estmb = GetBytesRead(); - estrc = GetReadCalls(); - } - // Fill entry - Double_t evts = (Double_t) estent; - Double_t mbs = (estmb > 0) ? estmb / TMath::Power(2.,20.) : 0.; //--> MB - Double_t rcs = (Double_t) estrc; - fCircProg->Fill((Double_t)fProcTime, evts, mbs, rcs, xall); - fCircProg->GetEntry(fCircProg->GetEntries()-2); - if (all) { - Double_t dt = (Double_t)fProcTime - ar[0]; - Long64_t de = (evts > ar[1]) ? (Long64_t) (evts - ar[1]) : 0; - Long64_t db = (mbs > ar[2]) ? (Long64_t) ((mbs - ar[2])*TMath::Power(2.,20.)) : 0; - if (gPerfStats) - gPerfStats->RateEvent((Double_t)fProcTime, dt, de, db); - // Get the last to spot the cache readings - Double_t rc = (Double_t)estrc - ar[3]; - mbrti = (rc > 0 && mbs > ar[2]) ? (Float_t) (mbs - ar[2]) / rc : 0. ; - } - // Final report only once (to correctly determine the proc time) - if (fTotalEntries > 0 && GetEntriesProcessed() >= fTotalEntries) - SetBit(TVirtualPacketizer::kIsDone); - PDB(kPacketizer,2) - Info("HandleTimer", "ent:%lld, bytes:%lld, proct:%f, evtrti:%f, mbrti:%f (%f,%f)", - estent, estmb, fProcTime, evtrti, mbrti, mbs, ar[2]); - } - - if (gProofServ) { - // Message to be sent over - TMessage m(kPROOF_PROGRESS); - if (gProofServ->GetProtocol() > 25) { - Int_t actw = GetActiveWorkers(); - Int_t acts = gProofServ->GetActSessions(); - Float_t effs = gProofServ->GetEffSessions(); - if (fProgressPerf && estent > 0) { - // Estimated query time - if (fProcTime > 0.) { - fReportPeriod = (Float_t) fTotalEntries / (Double_t) estent * fProcTime / 100.; - if (fReportPeriod > 0. && fReportPeriod < 5.) fReportPeriod = 5.; - } - - if (fProgressPerf->GetEntries() <= 0) { - // Fill the first entry - fProgressPerf->Fill(fProcTime, (Float_t)actw, -1., -1., -1.); - } else { - // Fill only if changed since last entry filled - Float_t *far = fProgressPerf->GetArgs(); - fProgressPerf->GetEntry(fProgressPerf->GetEntries()-1); - Bool_t doReport = (fReportPeriod > 0. && - (fProcTime - far[0]) >= fReportPeriod) ? kTRUE : kFALSE; - Float_t mbsread = estmb / 1024. / 1024.; - if (TMath::Abs((Float_t)actw - far[1]) > 0.1) { - if (fAWLastFill) - fProgressPerf->Fill(fProcTimeLast, (Float_t)fActWrksLast, - fEvtRateLast, fMBsReadLast, fEffSessLast); - fProgressPerf->Fill(fProcTime, (Float_t)actw, evtrti, mbsread, effs); - fAWLastFill = kFALSE; - } else if (doReport) { - fProgressPerf->Fill(fProcTime, (Float_t)actw, evtrti, mbsread, effs); - fAWLastFill = kFALSE; - } else { - fAWLastFill = kTRUE; - } - fProcTimeLast = fProcTime; - fActWrksLast = actw; - fEvtRateLast = evtrti; - fMBsReadLast = mbsread; - fEffSessLast = effs; - } - } - // Fill the message now - TProofProgressInfo pi(fTotalEntries, estent, estmb, fInitTime, - fProcTime, evtrti, mbrti, actw, acts, effs); - m << π - } else if (gProofServ->GetProtocol() > 11) { - // Fill the message now - m << fTotalEntries << estent << estmb << fInitTime << fProcTime - << evtrti << mbrti; - } else { - // Old format - m << fTotalEntries << GetEntriesProcessed(); - } - // send message to client; - gProofServ->GetSocket()->Send(m); - - } else { - if (gProof && gProof->GetPlayer()) { - // Log locally - gProof->GetPlayer()->Progress(fTotalEntries, estent, estmb, - fInitTime, fProcTime, evtrti, mbrti); - } - } - - // Final report only once (to correctly determine the proc time) - if (fTotalEntries > 0 && GetEntriesProcessed() >= fTotalEntries) - SetBit(TVirtualPacketizer::kIsDone); - - return kFALSE; // ignored? -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the initialization time - -void TVirtualPacketizer::SetInitTime() -{ - if (TestBit(TVirtualPacketizer::kIsInitializing)) { - fInitTime = Long64_t(gSystem->Now() - fStartTime) / (Float_t)1000.; - ResetBit(TVirtualPacketizer::kIsInitializing); - PDB(kPacketizer,2) - Info("SetInitTime","fInitTime set to %f s", fInitTime); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Adds new workers. Must be implemented by each real packetizer properly. -/// Returns the number of workers added, or -1 on failure. - -Int_t TVirtualPacketizer::AddWorkers(TList *) -{ - Warning("AddWorkers", "Not implemented for this packetizer"); - - return -1; -} diff --git a/proof/proof/src/TVirtualProofPlayer.cxx b/proof/proof/src/TVirtualProofPlayer.cxx deleted file mode 100644 index f9bf039021b32..0000000000000 --- a/proof/proof/src/TVirtualProofPlayer.cxx +++ /dev/null @@ -1,49 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Fons Rademakers 15/03/07 - -/************************************************************************* - * Copyright (C) 1995-2007, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TVirtualProofPlayer -\ingroup proofkernel - -Abstract interface for the PROOF player. -See the concrete implementations under 'proofplayer' for details. - -*/ - -#include "TVirtualProofPlayer.h" -#include "TPluginManager.h" -#include "TROOT.h" -#include "TError.h" - -//////////////////////////////////////////////////////////////////////////////// -/// Create a PROOF player. - -TVirtualProofPlayer *TVirtualProofPlayer::Create(const char *player, - TProof *pr, TSocket *s) -{ - TPluginHandler *h; - TVirtualProofPlayer *p = 0; - - if (!player || !*player) { - ::Error("TVirtualProofPlayer::Create", "player name missing"); - return 0; - } - - if ((h = gROOT->GetPluginManager()->FindHandler("TVirtualProofPlayer", player))) { - if (h->LoadPlugin() == -1) - return 0; - if (!strcmp(player, "slave")) - p = (TVirtualProofPlayer *) h->ExecPlugin(1, s); - else - p = (TVirtualProofPlayer *) h->ExecPlugin(1, pr); - } - - return p; -} diff --git a/proof/proofbench/CMakeLists.txt b/proof/proofbench/CMakeLists.txt deleted file mode 100644 index 99a09135c1189..0000000000000 --- a/proof/proofbench/CMakeLists.txt +++ /dev/null @@ -1,60 +0,0 @@ -# Copyright (C) 1995-2019, Rene Brun and Fons Rademakers. -# All rights reserved. -# -# For the licensing terms see $ROOTSYS/LICENSE. -# For the list of contributors see $ROOTSYS/README/CREDITS. - -############################################################################ -# CMakeLists.txt file for building ROOT proof/proofbench package -############################################################################ - -ROOT_STANDARD_LIBRARY_PACKAGE(ProofBench - HEADERS - TProofBenchDataSet.h - TProofBench.h - TProofBenchRunCPU.h - TProofBenchRunDataRead.h - TProofBenchRun.h - TProofBenchTypes.h - TProofNodes.h - TProofPerfAnalysis.h - SOURCES - src/TProofBench.cxx - src/TProofBenchDataSet.cxx - src/TProofBenchRunCPU.cxx - src/TProofBenchRun.cxx - src/TProofBenchRunDataRead.cxx - src/TProofNodes.cxx - src/TProofPerfAnalysis.cxx - DEPENDENCIES - Core - Gpad - Hist - ProofPlayer - INSTALL_OPTIONS - FILTER "TSel" -) - -# Generation and installation of the PAR files required by the benchmark -add_custom_target(ProofBenchPARFiles ALL - DEPENDS - ${CMAKE_BINARY_DIR}/etc/proof/proofbench/ProofBenchCPUSel.par - ${CMAKE_BINARY_DIR}/etc/proof/proofbench/ProofBenchDataSel.par -) - -add_custom_command(OUTPUT - ${CMAKE_BINARY_DIR}/etc/proof/proofbench/ProofBenchCPUSel.par - ${CMAKE_BINARY_DIR}/etc/proof/proofbench/ProofBenchDataSel.par - DEPENDS - ${CMAKE_SOURCE_DIR}/etc/proof/utils/makepbenchpars.sh - COMMAND - ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/etc/proof/proofbench - COMMAND - ${CMAKE_SOURCE_DIR}/etc/proof/utils/makepbenchpars.sh ProofBenchCPUSel ${CMAKE_SOURCE_DIR} ${CMAKE_BINARY_DIR} - COMMAND - ${CMAKE_SOURCE_DIR}/etc/proof/utils/makepbenchpars.sh ProofBenchDataSel ${CMAKE_SOURCE_DIR} ${CMAKE_BINARY_DIR} - WORKING_DIRECTORY ${CMAKE_SOURCE_DIR} -) - -install(DIRECTORY ${CMAKE_BINARY_DIR}/etc/proof/proofbench - DESTINATION ${CMAKE_INSTALL_SYSCONFDIR}/proof USE_SOURCE_PERMISSIONS) diff --git a/proof/proofbench/inc/LinkDef.h b/proof/proofbench/inc/LinkDef.h deleted file mode 100644 index 9496be64c47d3..0000000000000 --- a/proof/proofbench/inc/LinkDef.h +++ /dev/null @@ -1,29 +0,0 @@ -/* @(#)root/proof:$Id$ */ - -/************************************************************************* - * Copyright (C) 1995-2004, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifdef __CLING__ - -#pragma link off all globals; -#pragma link off all classes; -#pragma link off all functions; - -#pragma link C++ class TProofBench+; -#pragma link C++ class TProofBenchDataSet+; -#pragma link C++ class TProofBenchRun+; -#pragma link C++ class TProofBenchRunCPU+; -#pragma link C++ class TProofBenchRunDataRead+; -#pragma link C++ class TProofNodes+; -#pragma link C++ class TPBReadType+; -#pragma link C++ class TPBHistType+; -#pragma link C++ class TPBHandleDSType+; - -#pragma link C++ class TProofPerfAnalysis+; - -#endif diff --git a/proof/proofbench/inc/TProofBench.h b/proof/proofbench/inc/TProofBench.h deleted file mode 100644 index 95b2743b43ee3..0000000000000 --- a/proof/proofbench/inc/TProofBench.h +++ /dev/null @@ -1,146 +0,0 @@ -// @(#)root/proofx:$Id$ -// Author: G.Ganis, S.Ryu Feb 2011 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofBench -#define ROOT_TProofBench - -////////////////////////////////////////////////////////////////////////// -// // -// TProofBench // -// // -// Steering class for PROOF benchmarks // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TProofBenchTypes.h" -#include "TString.h" - - -class TF1; -class TFile; -class TGraphErrors; -class TProfile; -class TProof; -class TProofBenchRunCPU; -class TProofBenchRunDataRead; -class TProofBenchDataSet; - -class TProofBench : public TObject { - -private: - Bool_t fUnlinkOutfile; // Whether to remove empty output files - -protected: - - TProof* fProof; // Proof - TProof* fProofDS; // Proof to be used for dataset actions - TFile *fOutFile; // Output file - TString fOutFileName; // Name of the output file - Int_t fNtries; // Number of times a measurement is repeated - TPBHistType *fHistType; // Type of histograms for CPU runs - Int_t fNHist; // Number of histograms to be created in default CPU runs - TPBReadType *fReadType; // Type of read (partial, full) - TString fDataSet; // Name of the dataset - Int_t fNFilesWrk; // Number of files generated files per worker - Int_t fNumWrkMax; // Max number of workers (required for dynamic setups) - Bool_t fReleaseCache; // Release cache for data reads between runs - - TString fCPUSel; // Selector to be used for CPU benchmarks - TString fCPUPar; // List of par files to be loaded for CPU benchmarks - TString fDataSel; // Selector to be used for data benchmarks - TString fDataPar; // List of par files to be loaded for data benchmarks - TString fDataGenSel; // Selector to be used for generate data for benchmarks - TString fDataGenPar; // List of par files to be loaded to generate data for benchmarks - TString fSelOption; // Option field for processing the selector - - TProofBenchRunCPU *fRunCPU; // Instance to run CPU scans - TProofBenchRunDataRead *fRunDS; // Instance to run data-read scans - TProofBenchDataSet *fDS; // Instance to handle datasets operations - - Bool_t fDebug; // Debug switch - - TNamed *fDescription; // Strings describing the cluster for this test (saved in the output file) - - static TGraphErrors *GetGraph(TDirectory *d, const char *pfn, - Int_t &nb, Double_t &xmi, Double_t &xmx, - Double_t &ymi, Double_t &ymx, Int_t &kmx, TProfile *&pf); - - static TF1 *fgFp1; // Simple 1st degree polynomial - static TF1 *fgFp1n; // Normalized 1st degree - static TF1 *fgFp2; // Simple 2nd degree polynomial - static TF1 *fgFp2n; // Normalized 2nd degree - static TF1 *fgFp3; // Function with varying Rcpu - static TF1 *fgFp3n; // Normalized Function with varying Rcpu - - static TF1 *fgFio; // Function used for I/O rate fits - static TF1 *fgFioV; // Function used for I/O rate fits with non-constant Rcpu - - static TList *fgGraphs; // List of TGraphErrors created by Draw actions - - static void AssertFittingFun(Double_t mi, Double_t mx); - -public: - - TProofBench(const char *url, const char *outfile = "", const char *proofopt = 0); - - ~TProofBench() override; - - Int_t RunCPU(Long64_t nevents=-1, Int_t start=-1, Int_t stop=-1, Int_t step=-1); - Int_t RunCPUx(Long64_t nevents=-1, Int_t start=-1, Int_t stop=-1); - Int_t RunDataSet(const char *dset = "BenchDataSet", - Int_t start = 1, Int_t stop = -1, Int_t step = 1); - Int_t RunDataSetx(const char *dset = "BenchDataSet", Int_t start = 1, Int_t stop = -1); - - Int_t CopyDataSet(const char *dset, const char *dsetdst, const char *destdir); - Int_t MakeDataSet(const char *dset = 0, Long64_t nevt = -1, const char *fnroot = "event", - Bool_t regenerate = kFALSE); - Int_t ReleaseCache(const char *dset); - Int_t RemoveDataSet(const char *dset); - - void CloseOutFile(); - Int_t OpenOutFile(Bool_t wrt = kFALSE, Bool_t verbose = kTRUE); - Int_t SetOutFile(const char *outfile, Bool_t verbose = kTRUE); - const char *GetOutFileName() const { return fOutFileName; } - void SetNFilesWrk(Int_t nf = 0) { fNFilesWrk = (nf > 0) ? nf : 4; } - void SetNTries(Int_t nt) { if (nt > 0) fNtries = nt; } - void SetHistType(TPBHistType *histtype) { fHistType = histtype; } - void SetNHist(Int_t nh) { fNHist = nh; } - void SetReadType(TPBReadType *readtype) { fReadType = readtype; } - void SetReleaseCache(Bool_t on = kTRUE) { fReleaseCache = on; } - - void SetCPUSel(const char *sel) { fCPUSel = sel; } - void SetCPUPar(const char *par) { fCPUPar = par; } - void SetDataSel(const char *sel) { fDataSel = sel; } - void SetDataPar(const char *par) { fDataPar = par; } - void SetDataGenSel(const char *sel) { fDataGenSel = sel; } - void SetDataGenPar(const char *par) { fDataGenPar = par; } - void SetSelOption(const char *opt) { fSelOption = opt; } - - void SetProofDS(TProof *p); - - void SetDebug(Bool_t debug = kTRUE) { fDebug = debug; } - - Bool_t GetDebug() { return fDebug; } - - static void DrawCPU(const char *outfile, const char *opt = "std:", Bool_t verbose = kFALSE, - Int_t dofit = 0, Int_t n0 = -1, Int_t n1 = -1); - static void DrawDataSet(const char *outfile, const char *opt = "std:", const char *type = "mbs", Bool_t verbose = kFALSE, - Int_t dofit = 0, Int_t n0 = -1, Int_t n1 = -1); - static void GetPerfSpecs(const char *path = ".", Int_t degfit = 1); - static void DrawEfficiency(const char *outfile, const char *opt = "", Bool_t verbose = kFALSE); - - static TList *GetGraphs() { return fgGraphs; } - - ClassDefOverride(TProofBench, 0) // Steering class for PROOF benchmarks -}; - -#endif diff --git a/proof/proofbench/inc/TProofBenchDataSet.h b/proof/proofbench/inc/TProofBenchDataSet.h deleted file mode 100644 index 98706968fb3f4..0000000000000 --- a/proof/proofbench/inc/TProofBenchDataSet.h +++ /dev/null @@ -1,50 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofBenchDataSet -#define ROOT_TProofBenchDataSet - -////////////////////////////////////////////////////////////////////////// -// // -// TProofBenchDataSet // -// // -// Handle operations on datasets used by ProofBench // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" - -class TProof; - -class TProofBenchDataSet: public TObject { - -private: - -protected: - TProof* fProof; //proof - - Int_t Handle(const char *dset, TObject *type); - -public: - - TProofBenchDataSet(TProof *proof = 0); - ~TProofBenchDataSet() override { } - - Bool_t IsProof(TProof *p) { return (p == fProof) ? kTRUE : kFALSE; } - - Int_t CopyFiles(const char *dset, const char *destdir); - Int_t ReleaseCache(const char *dset); - Int_t RemoveFiles(const char *dset); - - ClassDefOverride(TProofBenchDataSet,0) //Handle operations on datasets -}; - -#endif diff --git a/proof/proofbench/inc/TProofBenchRun.h b/proof/proofbench/inc/TProofBenchRun.h deleted file mode 100644 index 3e22c3a54aa61..0000000000000 --- a/proof/proofbench/inc/TProofBenchRun.h +++ /dev/null @@ -1,63 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofBenchRun -#define ROOT_TProofBenchRun - -////////////////////////////////////////////////////////////////////////// -// // -// TProofBenchRun // -// // -// Abstract base class for PROOF benchmark run. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" -#include "TProofBenchTypes.h" - -class TProof; - -class TProofBenchRun : public TObject { - -private: - -protected: - - TProof* fProof; // Proof - TString fSelName; // Name of the selector to be run - TString fParList; // List of PARs to be loaded - TString fSelOption; // Option field for processing the selector - -public: - - TProofBenchRun(TProof *proof = 0, const char *sel = 0); - - ~TProofBenchRun() override; - - virtual const char *GetSelName() { return fSelName; } - virtual const char *GetParList() { return fParList; } - virtual void SetSelName(const char *sel) { fSelName = sel; } - virtual void SetParList(const char *pars) { fParList = pars; } - virtual void SetSelOption(const char *opt) { fSelOption = opt; } - - virtual void Run(Long64_t nevents, Int_t start = -1, Int_t stop = -1, - Int_t step = -1, Int_t ntries = -1, Int_t debug = -1, - Int_t draw = -1) = 0; - virtual void Run(const char *dset, Int_t start = -1, Int_t stop = -1, - Int_t step = -1, Int_t ntries = -1, Int_t debug = -1, - Int_t draw = -1) = 0; - - void Print(Option_t *option = "") const override =0; - - ClassDefOverride(TProofBenchRun, 0) //Abstract base class for PROOF benchmark run -}; - -#endif diff --git a/proof/proofbench/inc/TProofBenchRunCPU.h b/proof/proofbench/inc/TProofBenchRunCPU.h deleted file mode 100644 index e2d1798f32e05..0000000000000 --- a/proof/proofbench/inc/TProofBenchRunCPU.h +++ /dev/null @@ -1,136 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofBenchRunCPU -#define ROOT_TProofBenchRunCPU - -////////////////////////////////////////////////////////////////////////// -// // -// TProofBenchRunCPU // -// // -// CPU-intensive PROOF benchmark test generates events and fill 1, 2, // -//or 3-D histograms. No I/O activity is involved. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TString.h" - -#include "TProofBenchRun.h" - - -class TCanvas; -class TList; -class TProof; -class TProfile; -class TLegend; -class TH2; -class TTree; - -class TProofBenchMode; -class TProofNodes; -class TPBHistType; - -class TProofBenchRunCPU : public TProofBenchRun { - -private: - - TPBHistType *fHistType; //histogram type - Int_t fNHists; //number of histograms - - Long64_t fNEvents; //number of events to generate - Int_t fNTries; //number of tries - - Int_t fStart; //start number of workers to scan - Int_t fStop; //stop number of workers to scan - Int_t fStep; //test to be performed every fStep workers - - Int_t fDraw; //draw switch - Int_t fDebug; //debug switch - - TDirectory* fDirProofBench; //directory for proof outputs - - TProofNodes* fNodes; //node information - - TList* fListPerfPlots; //list of performance plots - TCanvas* fCanvas; //canvas for performance plots - - TProfile *fProfile_perfstat_event; - TH2 *fHist_perfstat_event; - TProfile *fProfile_perfstat_evtmax; - TProfile *fNorm_perfstat_evtmax; - TProfile *fProfile_queryresult_event; - TProfile *fNorm_queryresult_event; - TProfile *fProfile_cpu_eff; - - TLegend *fProfLegend; // Legend for profiles - TLegend *fNormLegend; // Legend for norms - - TString fName; //name of CPU run - - void BuildHistos(Int_t start, Int_t stop, Int_t step, Bool_t nx); - -protected: - - void FillPerfStatPerfPlots(TTree* t, Int_t nactive); - - Int_t SetParameters(); - Int_t DeleteParameters(); - -public: - - TProofBenchRunCPU(TPBHistType *histtype = 0, - Int_t nhists=16, TDirectory* dirproofbench=0, - TProof* proof=0, TProofNodes* nodes=0, - Long64_t nevents=1000000, Int_t ntries=2, Int_t start=1, - Int_t stop=-1, Int_t step=1, Int_t draw=0, Int_t debug=0); - - ~TProofBenchRunCPU() override; - - void Run(Long64_t nevents, Int_t start, Int_t stop, Int_t step, Int_t ntries, - Int_t debug, Int_t draw) override; - void Run(const char *, Int_t, Int_t, Int_t, Int_t, Int_t, Int_t) override { } - - void DrawPerfPlots(); - - void Print(Option_t* option="") const override; - - void SetHistType(TPBHistType *histtype); - void SetNHists(Int_t nhists) { fNHists = nhists; } - void SetNEvents(Long64_t nevents) { fNEvents = nevents; } - void SetNTries(Int_t ntries) { fNTries = ntries; } - void SetStart(Int_t start) { fStart = start; } - void SetStop(Int_t stop) { fStop = stop; } - void SetStep(Int_t step) { fStep = step; } - void SetDraw(Int_t draw) { fDraw = draw; } - void SetDebug(Int_t debug) { fDebug = debug; } - - void SetDirProofBench(TDirectory* dir) { fDirProofBench = dir; } - - TPBHistType *GetHistType() const { return fHistType; } - Int_t GetNHists() const { return fNHists; } - Long64_t GetNEvents() const { return fNEvents; } - Int_t GetNTries() const { return fNTries; } - Int_t GetStart() const { return fStart; } - Int_t GetStop() const { return fStop; } - Int_t GetStep() const { return fStep; } - Int_t GetDraw() const { return fDraw; } - Int_t GetDebug() const { return fDebug; } - TDirectory* GetDirProofBench() const { return fDirProofBench; } - TList* GetListPerfPlots() const { return fListPerfPlots; } - TCanvas* GetCanvas() const { return fCanvas; } - const char* GetName() const override { return fName; } - - TString GetNameStem() const; - - ClassDefOverride(TProofBenchRunCPU,0) //CPU-intensive PROOF benchmark -}; - -#endif diff --git a/proof/proofbench/inc/TProofBenchRunDataRead.h b/proof/proofbench/inc/TProofBenchRunDataRead.h deleted file mode 100644 index 3bd1f97f0aab7..0000000000000 --- a/proof/proofbench/inc/TProofBenchRunDataRead.h +++ /dev/null @@ -1,144 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofBenchRunDataRead -#define ROOT_TProofBenchRunDataRead - -////////////////////////////////////////////////////////////////////////// -// // -// TProofBenchRunDataRead // -// // -// I/O-intensive PROOF benchmark test reads in event files distributed // -// on the cluster. Number of events processed per second and size of // -// events processed per second are plotted against number of active // -// workers. Performance rate for unit packets and performance rate // -// for query are plotted. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TString.h" - -#include "TProofBenchRun.h" - -class TProof; -class TCanvas; -class TH2; -class TProfile; -class TLegend; -class TTree; -class TFileCollection; - -class TProofBenchMode; -class TProofBenchDataSet; -class TProofNodes; -class TPBReadType; - -class TProofBenchRunDataRead : public TProofBenchRun { - -private: - TProof* fProof; //pointer to proof - - TPBReadType *fReadType; //read type - TProofBenchDataSet *fDS; //dataset operations handler - - Long64_t fNEvents; //number of events per file - Int_t fNTries; //number of tries - Int_t fStart; //start number of workers - Int_t fStop; //stop number of workers - Int_t fStep; //test to be performed every fStep workers - Int_t fDebug; //debug switch - Int_t fFilesPerWrk; //# of files to be processed per worker - Bool_t fReleaseCache; // Release cache for data reads between runs - - TDirectory *fDirProofBench; //directory for proof outputs - - TProofNodes *fNodes; //list of nodes information - - TList *fListPerfPlots; //list of performance plots - TProfile *fProfile_perfstat_event; - TH2 *fHist_perfstat_event; - TProfile *fProfile_perfstat_evtmax; - TProfile *fNorm_perfstat_evtmax; - TProfile *fProfile_queryresult_event; - TProfile *fNorm_queryresult_event; - TProfile *fProfile_perfstat_IO; - TH2 *fHist_perfstat_IO; - TProfile *fProfile_perfstat_IOmax; - TProfile *fNorm_perfstat_IOmax; - TProfile *fProfile_queryresult_IO; - TProfile *fNorm_queryresult_IO; - TProfile *fProfile_cpu_eff; - - TLegend *fProfLegend_evt; // Legend for profiles evts - TLegend *fNormLegend_evt; // Legend for norms evts - TLegend *fProfLegend_mb; // Legend for profiles mbs - TLegend *fNormLegend_mb; // Legend for norms mbs - - TCanvas *fCPerfProfiles; //canvas for performance profile histograms - - TString fName; //name of this run - - void BuildHistos(Int_t start, Int_t stop, Int_t step, Bool_t nx); - -protected: - - void FillPerfStatProfiles(TTree* t, Int_t nactive); - - Int_t SetParameters(); - Int_t DeleteParameters(); - -public: - - TProofBenchRunDataRead(TProofBenchDataSet *pbds, TPBReadType *readtype = 0, - TDirectory* dirproofbench=0, TProof* proof=0, TProofNodes* nodes=0, - Long64_t nevents=-1, Int_t ntries=2, Int_t start=1, Int_t stop=-1, - Int_t step=1, Int_t debug=0); - - ~TProofBenchRunDataRead() override; - - void Run(Long64_t, Int_t, Int_t, Int_t, Int_t, Int_t, Int_t) override { } - void Run(const char *dset, Int_t start, Int_t stop, Int_t step, Int_t ntries, - Int_t debug, Int_t) override; - - TFileCollection *GetDataSet(const char *dset, Int_t nact, Bool_t nx); - - void DrawPerfProfiles(); - - void Print(Option_t* option="") const override; - - void SetReadType(TPBReadType *readtype) { fReadType = readtype; } - void SetNEvents(Long64_t nevents) { fNEvents = nevents; } - void SetNTries(Int_t ntries) { fNTries = ntries; } - void SetStart(Int_t start) { fStart = start; } - void SetStop(Int_t stop) { fStop = stop; } - void SetStep(Int_t step) { fStep = step; } - void SetDebug(Int_t debug) { fDebug = debug; } - void SetDirProofBench(TDirectory* dir) { fDirProofBench = dir; } - void SetFilesPerWrk(Int_t fpw) { fFilesPerWrk = fpw; } - void SetReleaseCache(Bool_t on = kTRUE) { fReleaseCache = on; } - - TPBReadType *GetReadType() const { return fReadType; } - Long64_t GetNEvents() const { return fNEvents; } - Int_t GetNTries() const { return fNTries; } - Int_t GetStart() const { return fStart; } - Int_t GetStop() const { return fStop; } - Int_t GetStep() const { return fStep; } - Int_t GetDebug() const { return fDebug; } - TDirectory* GetDirProofBench() const { return fDirProofBench; } - TCanvas* GetCPerfProfiles() const { return fCPerfProfiles; } - const char* GetName() const override { return fName; } - - TString GetNameStem() const; - - ClassDefOverride(TProofBenchRunDataRead,0) //IO-intensive PROOF benchmark -}; - -#endif diff --git a/proof/proofbench/inc/TProofBenchTypes.h b/proof/proofbench/inc/TProofBenchTypes.h deleted file mode 100644 index 9fee8977c3ef7..0000000000000 --- a/proof/proofbench/inc/TProofBenchTypes.h +++ /dev/null @@ -1,108 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G. Ganis Oct 2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofBenchTypes -#define ROOT_TProofBenchTypes - -////////////////////////////////////////////////////////////////////////// -// // -// ProofBenchTypes // -// Const and types used by ProofBench and its selectors // -// // -////////////////////////////////////////////////////////////////////////// -#include "TObject.h" -#include "TString.h" - -const char* const kPROOF_BenchCPUSelPar = "ProofBenchCPUSel"; // PAR with bench CPU selectors -const char* const kPROOF_BenchDataSelPar = "ProofBenchDataSel"; // PAR with bench data selectors -const char* const kPROOF_BenchParDir = "proof/proofbench/"; // Dir with ProofBench Par files (under 'etc') -const char* const kPROOF_BenchSrcDir = "proof/proofbench/src/"; // Dir with ProofBench Src files -const char* const kPROOF_BenchIncDir = "proof/proofbench/inc/"; // Dir with ProofBench Inc files -const char* const kPROOF_BenchSelCPUDef = "TSelHist"; // default CPU selector -const char* const kPROOF_BenchSelDataDef = "TSelEvent"; // default Data Read selector -const char* const kPROOF_BenchSelDataGenDef = "TSelEventGen"; // default Data generator selector - -class TPBReadType : public TObject { -public: - enum EReadType { - kReadNotSpecified = 0, //read type not specified - kReadFull = 1, //read in a full event - kReadOpt = 2, //read in part of an event - kReadNo = 4 //do not read in event - }; -private: - EReadType fType; - TString fName; -public: - TPBReadType(EReadType type = kReadOpt) : fType(type), fName("PROOF_Benchmark_ReadType") { } - ~TPBReadType() override { } - - EReadType GetType() const { return fType; } - Bool_t IsReadFull() const { return (fType == kReadFull) ? kTRUE : kFALSE; } - Bool_t IsReadOpt() const { return (fType == kReadOpt) ? kTRUE : kFALSE; } - Bool_t IsReadNo() const { return (fType == kReadNo) ? kTRUE : kFALSE; } - const char *GetName() const override { return fName; } - - ClassDefOverride(TPBReadType, 1) // Streamable PBReadType -}; - -class TPBHistType : public TObject { -public: - enum EHistType { - kHistNotSpecified = 0, //histogram type not specified - kHist1D = 1, //1-D histogram - kHist2D = 2, //2-D histogram - kHist3D = 4, //3-D histogram - kHistAll = kHist1D | kHist2D | kHist3D //1-D, 2-D and 3-D histograms - }; -private: - EHistType fType; - TString fName; -public: - TPBHistType(EHistType type = kHist1D) : fType(type), fName("PROOF_Benchmark_HistType") { } - ~TPBHistType() override { } - - EHistType GetType() const { return fType; } - Bool_t IsHist1D() const { return (fType == kHist1D) ? kTRUE : kFALSE; } - Bool_t IsHist2D() const { return (fType == kHist2D) ? kTRUE : kFALSE; } - Bool_t IsHist3D() const { return (fType == kHist3D) ? kTRUE : kFALSE; } - Bool_t IsHistAll() const { return (fType == kHistAll) ? kTRUE : kFALSE; } - const char *GetName() const override { return fName; } - - ClassDefOverride(TPBHistType, 1) // Streamable PBHistType -}; - -class TPBHandleDSType : public TObject { -public: - enum EHandleDSType { - kReleaseCache = 0, // Release memory cache for the given file - kCheckCache = 1, // Check memory cache for the given files - kRemoveFiles = 2, // Remove (physically) the given files - kCopyFiles = 3 // Copy the given files to a destination dir - }; -private: - EHandleDSType fType; - TString fName; -public: - TPBHandleDSType(EHandleDSType type = kReleaseCache) : fType(type), fName("PROOF_Benchmark_HandleDSType") { } - ~TPBHandleDSType() override { } - - EHandleDSType GetType() const { return fType; } - Bool_t IsReleaseCache() const { return (fType == kReleaseCache) ? kTRUE : kFALSE; } - Bool_t IsCheckCache() const { return (fType == kCheckCache) ? kTRUE : kFALSE; } - Bool_t IsRemoveFiles() const { return (fType == kRemoveFiles) ? kTRUE : kFALSE; } - Bool_t IsCopyFiles() const { return (fType == kCopyFiles) ? kTRUE : kFALSE; } - const char *GetName() const override { return fName; } - - ClassDefOverride(TPBHandleDSType, 1) // Streamable PBHandleDSType -}; - -#endif diff --git a/proof/proofbench/inc/TProofNodes.h b/proof/proofbench/inc/TProofNodes.h deleted file mode 100644 index 4988ff88ee84f..0000000000000 --- a/proof/proofbench/inc/TProofNodes.h +++ /dev/null @@ -1,60 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofNodes -#define ROOT_TProofNodes - -////////////////////////////////////////////////////////////////////////// -// // -// TProofNodes // -// // -// PROOF worker nodes information // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TObject.h" - -class TProof; -class TMap; - -class TProofNodes: public TObject { -private: - TProof *fProof; - TMap *fNodes; // Map of node infos - TMap *fActiveNodes; // Map of active node infos - Int_t fMaxWrksNode; // Max number of workers per node - Int_t fMinWrksNode; // Min number of workers per node - Int_t fNNodes; // Number of nodes - Int_t fNWrks; // Number of workers - Int_t fNActiveWrks; // Number of active workers - Int_t fNCores; // Number of total cores - - void Build(); -public: - TProofNodes(TProof* proof); - - ~TProofNodes() override; - Int_t ActivateWorkers(Int_t nwrks); - Int_t ActivateWorkers(const char *workers); - Int_t GetMaxWrksPerNode() const { return fMaxWrksNode; } - Int_t GetNWorkersCluster() const { return fNWrks; } - Int_t GetNNodes() const { return fNNodes; } - Int_t GetNCores() const { return fNCores; } - Int_t GetMinWrksPerNode() const { return fMinWrksNode; } - Int_t GetNActives() const { return fNActiveWrks; } - TMap* GetMapOfNodes() const { return fNodes; } - TMap* GetMapOfActiveNodes() const { return fActiveNodes; } - void Print(Option_t* option="") const override; - - ClassDefOverride(TProofNodes, 0) //Node and worker information -}; - -#endif diff --git a/proof/proofbench/inc/TProofPerfAnalysis.h b/proof/proofbench/inc/TProofPerfAnalysis.h deleted file mode 100644 index be27b332afe81..0000000000000 --- a/proof/proofbench/inc/TProofPerfAnalysis.h +++ /dev/null @@ -1,121 +0,0 @@ -// @(#)root/proofx:$Id$ -// Author: G.Ganis Nov 2011 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofPerfAnalysis -#define ROOT_TProofPerfAnalysis - -////////////////////////////////////////////////////////////////////////// -// // -// TProofPerfAnalysis // -// // -// Set of tools to analyse the performance tree // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TNamed.h" -#include "TSortedList.h" -#include "TString.h" - -class TFile; -class TH1F; -class TH2F; -class TList; -class TTree; -class TProofPerfAnalysis : public TNamed { - -public: // public because of Sun CC bug - class TFileInfo; - class TPackInfo; - class TWrkEntry; - class TWrkInfo; - class TWrkInfoFile; - -private: - TFile *fFile; // The open performance file - TString fDirName; // The name of the subdir with the perfomance tree - TString fTreeName; // The name of the performance tree - TTree *fTree; // The performance tree - TSortedList fWrksInfo; // Sorted list of workers info - TSortedList fFilesInfo; // Sorted list of files info - Float_t fInitTime; // End of initialization time for this query - Float_t fMergeTime; // Begin of merging time for this query - Float_t fMaxTime; // Max time for this query (slowest worker) - TH1F *fEvents; // Event distribution per worker - TH1F *fPackets; // Packet distribution per worker - Double_t fEvtRateMax; // Max event processing rate per packet - Double_t fMBRateMax; // Max MB processing rate per packet - Double_t fLatencyMax; // Max retrieval latency per packet - TH1F *fEvtRate; // Event processing rate vs query time - TH1F *fEvtRateRun; // Event processing rate running avg vs query time - TH1F *fMBRate; // Byte processing rate vs query time - TH1F *fMBRateRun; // Byte processing rate running avg vs query time - Double_t fEvtRateAvgMax; // Max running event processing rate - Double_t fMBRateAvgMax; // Max running MB processing rate - Double_t fEvtRateAvg; // Average event processing rate - Double_t fMBRateAvg; // Average MB processing rate - TString fFileResult; // File where to save basics of a run when requested - Bool_t fSaveResult; // Whether to save the results of a run - - Int_t fDebug; // Local verbosity level - - static Bool_t fgDebug; // Global verbosity on/off - - Int_t CompareOrd(const char *ord1, const char *ord2); - void FillFileDist(TH1F *hf, TH1F *hb, TH2F *hx, Bool_t wdet = kFALSE); - void FillFileDistOneSrv(TH1F *hx, Bool_t wdet = kFALSE); - void FillWrkInfo(Bool_t force = kFALSE); - void FillFileInfo(Bool_t force = kFALSE); - TString GetCanvasTitle(const char *t); - void GetWrkFileList(TList *wl, TList *sl); - void LoadTree(TDirectory *dir); - void DoDraw(TObject *o, Option_t *opt = "", const char *name = 0); - -public: - - TProofPerfAnalysis(const char *perffile, const char *title = "", - const char *treename = "PROOF_PerfStats"); - TProofPerfAnalysis(TTree *tree, const char *title = ""); - ~TProofPerfAnalysis() override; - - Bool_t IsValid() const { return (fFile && fTree) ? kTRUE : kFALSE; } - Bool_t WrkInfoOK() const { return (fWrksInfo.GetSize() > 0) ? kTRUE : kFALSE; } - - void EventDist(); // Analyse event and packet distribution - void FileDist(Bool_t writedet = kFALSE); // Analyse the file distribution - void LatencyPlot(const char *wrks = 0); // Packet latency distribution vs time - void RatePlot(const char *wrks = 0); // Rate distribution vs time - void WorkerActivity(); // Analyse the worker activity - void PrintWrkInfo(Int_t showlast = 10); // Print workers info - void PrintWrkInfo(const char *wrk); // Print worker info by name - - void PrintFileInfo(Int_t showlast = 10, const char *opt = "", const char *out = 0); // Print file info - void PrintFileInfo(const char *fn, const char *opt = "P", const char *out = 0); // Print file info by name - void FileProcPlot(const char *fn, const char *out = 0); // Plot info about file processing - void FileRatePlot(const char *fns = 0); // Plot info about file processing rates - - Double_t GetEvtRateAvgMax() const { return fEvtRateAvgMax; } // Max running event processing rate - Double_t GetMBRateAvgMax() const { return fMBRateAvgMax; } // Max running MB processing rate - Double_t GetEvtRateAvg() const { return fEvtRateAvg; } // Average event processing rate - Double_t GetMBRateAvg() const { return fMBRateAvg; } // Average MB processing rate - void GetAverages(Double_t &evtmax, Double_t &mbmax, Double_t &evt, Double_t &mb) const - { evtmax = fEvtRateAvgMax; mbmax = fMBRateAvgMax; evt = fEvtRateAvg; mb = fMBRateAvg; return; } - - void Summary(Option_t *opt = "", const char *out = ""); - - Int_t SetSaveResult(const char *file = "results.root", Option_t *mode = "RECREATE"); - - void SetDebug(Int_t d = 0); // Setter for the verbosity level - static void SetgDebug(Bool_t on = kTRUE); // Overall verbosity level - - ClassDefOverride(TProofPerfAnalysis, 0) // Set of tools to analyse the performance tree -}; - -#endif diff --git a/proof/proofbench/inc/TSelEvent.h b/proof/proofbench/inc/TSelEvent.h deleted file mode 100644 index 6f3cec984ff8e..0000000000000 --- a/proof/proofbench/inc/TSelEvent.h +++ /dev/null @@ -1,205 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TSelEvent -#define ROOT_TSelEvent - -////////////////////////////////////////////////////////////////////////// -// // -// TSelEvent // -// // -// PROOF selector for I/O benchmark test. // -// For I/O benchmark, event files are read in and histograms are filled.// -// For memory clean-up, dedicated files large enough to clean up memory // -// cache on the machine are read in. Or memory clean-up can be // -// accompolished by system call on Linux machine inside SlaveBegin(..) // -// which should be much faster the reading in large files. // -// // -////////////////////////////////////////////////////////////////////////// - -#include -#include -#include -#include -#include - -class TROOT; -class TH1F; -class TCanvas; -class THashList; -class TClonesArray; -class TRefArray; -class TPBReadType; - -class TSelEvent : public TSelector { - -private: - - TPBReadType *fReadType; //read type - Bool_t fDebug; //debug switch - TCanvas* fCHist; //canvas to display histograms - - //Output hists - TH1F* fPtHist; - TH1F* fNTracksHist; - -public : - - TTree *fChain; //!pointer to the analyzed TTree or TChain - - // Declaration of leave types -// Event *event; - Char_t fType[20]; - Char_t *fEventName; - Int_t fNtrack; - Int_t fNseg; - Int_t fNvertex; - UInt_t fFlag; - Double32_t fTemperature; - Int_t fMeasures[10]; - Double32_t fMatrix[4][4]; - Double32_t fClosestDistance[21]; //[fNvertex] - EventHeader fEvtHdr; - TClonesArray *fTracks; - TRefArray *fHighPt; - TRefArray *fMuons; - TRef fLastTrack; - TRef fWebHistogram; - TH1F *fH; - TBits fTriggerBits; - Bool_t fIsValid; - - // List of branches - TBranch *b_event_fType; //! - TBranch *b_fEventName; //! - TBranch *b_event_fNtrack; //! - TBranch *b_event_fNseg; //! - TBranch *b_event_fNvertex; //! - TBranch *b_event_fFlag; //! - TBranch *b_event_fTemperature; //! - TBranch *b_event_fMeasures; //! - TBranch *b_event_fMatrix; //! - TBranch *b_fClosestDistance; //! - TBranch *b_event_fEvtHdr; //! - TBranch *b_fTracks; //! - TBranch *b_fHighPt; //! - TBranch *b_fMuons; //! - TBranch *b_event_fLastTrack; //! - TBranch *b_event_fWebHistogram; //! - TBranch *b_fH; //! - TBranch *b_event_fTriggerBits; //! - TBranch *b_event_fIsValid; //! - - TSelEvent(TTree *); - TSelEvent(); - //virtual ~TSelEvent() {if (fCHist) delete fCHist;} - virtual ~TSelEvent() {} - virtual Int_t Version() const {return 1;} - virtual void Begin(TTree *); - virtual void SlaveBegin(TTree *tree); - virtual void Init(TTree *tree); - virtual Bool_t Notify(); - virtual Bool_t Process(Long64_t entry); - virtual void SetOption(const char *option) { fOption = option; } - virtual void SetObject(TObject *obj) { fObject = obj; } - virtual void SetInputList(TList *input) {fInput = input;} - virtual TList *GetOutputList() const { return fOutput; } - virtual void SlaveTerminate(); - virtual void Terminate(); - - TPBReadType *GetReadType() {return fReadType;} - Bool_t GetDebug(){return fDebug;} - TCanvas* GetCHist(){return fCHist;} - TH1F* GetPtHist(){return fPtHist;} - TH1F* GetNTracksHist(){return fNTracksHist;} - - ClassDefOverride(TSelEvent,0) //PROOF selector for I/O-intensive benchmark test -}; - -#endif - -#ifdef TSelEvent_cxx -void TSelEvent::Init(TTree *tree) -{ - // The Init() function is called when the selector needs to initialize - // a new tree or chain. Typically here the branch addresses of the tree - // will be set. It is normaly not necessary to make changes to the - // generated code, but the routine can be extended by the user if needed. - // Init() will be called many times when running with PROOF. - - // Set branch addresses - fEventName=0; - fTracks=0; - fHighPt=0; - fMuons=0; - fH=0; - - if (tree == 0) return; - fChain = tree; - fChain->SetMakeClass(1); - - fChain->SetBranchAddress("fType[20]",fType); - fChain->SetBranchAddress("fEventName",fEventName); - fChain->SetBranchAddress("fNtrack",&fNtrack); - fChain->SetBranchAddress("fNseg",&fNseg); - fChain->SetBranchAddress("fNvertex",&fNvertex); - fChain->SetBranchAddress("fFlag",&fFlag); - fChain->SetBranchAddress("fTemperature",&fTemperature); - fChain->SetBranchAddress("fMeasures[10]",fMeasures); - fChain->SetBranchAddress("fMatrix[4][4]",fMatrix); - fChain->SetBranchAddress("fClosestDistance",fClosestDistance); - fChain->SetBranchAddress("fEvtHdr",&fEvtHdr); - fChain->SetBranchAddress("fTracks",&fTracks); - fChain->SetBranchAddress("fHighPt",&fHighPt); - fChain->SetBranchAddress("fMuons",&fMuons); - fChain->SetBranchAddress("fLastTrack",&fLastTrack); - fChain->SetBranchAddress("fWebHistogram",&fWebHistogram); - fChain->SetBranchAddress("fH",&fH); - fChain->SetBranchAddress("fTriggerBits",&fTriggerBits); - fChain->SetBranchAddress("fIsValid",&fIsValid); -} - -Bool_t TSelEvent::Notify() -{ - // The Notify() function is called when a new file is opened. This - // can be either for a new TTree in a TChain or when when a new TTree - // is started when using PROOF. Typically here the branch pointers - // will be retrieved. It is normaly not necessary to make changes - // to the generated code, but the routine can be extended by the - // user if needed. - - Info("Notify","processing file: %s", fChain->GetCurrentFile()->GetName()); - - // Get branch pointers - b_event_fType = fChain->GetBranch("fType[20]"); - b_fEventName = fChain->GetBranch("fEventName"); - b_event_fNtrack = fChain->GetBranch("fNtrack"); - b_event_fNseg = fChain->GetBranch("fNseg"); - b_event_fNvertex = fChain->GetBranch("fNvertex"); - b_event_fFlag = fChain->GetBranch("fFlag"); - b_event_fTemperature = fChain->GetBranch("fTemperature"); - b_event_fMeasures = fChain->GetBranch("fMeasures[10]"); - b_event_fMatrix = fChain->GetBranch("fMatrix[4][4]"); - b_fClosestDistance = fChain->GetBranch("fClosestDistance"); - b_event_fEvtHdr = fChain->GetBranch("fEvtHdr"); - b_fTracks = fChain->GetBranch("fTracks"); - b_fHighPt = fChain->GetBranch("fHighPt"); - b_fMuons = fChain->GetBranch("fMuons"); - b_event_fLastTrack = fChain->GetBranch("fLastTrack"); - b_event_fWebHistogram = fChain->GetBranch("fWebHistogram"); - b_fH = fChain->GetBranch("fH"); - b_event_fTriggerBits = fChain->GetBranch("fTriggerBits"); - b_event_fIsValid = fChain->GetBranch("fIsValid"); - - return kTRUE; -} - -#endif // #ifdef TSelEvent_cxx diff --git a/proof/proofbench/inc/TSelEventGen.h b/proof/proofbench/inc/TSelEventGen.h deleted file mode 100644 index 2045a0604f96c..0000000000000 --- a/proof/proofbench/inc/TSelEventGen.h +++ /dev/null @@ -1,106 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TSelEventGen -#define ROOT_TSelEventGen - -////////////////////////////////////////////////////////////////////////// -// // -// TSelEventGen // -// // -// PROOF selector for event file generation. // -// List of files to be generated for each node is provided by client. // -// And list of files generated is sent back. // -// Existing files are reused if not forced to be regenerated. // -// // -////////////////////////////////////////////////////////////////////////// - -#include -#include -#include - -class TList; -class TMacro; - -class TSelEventGen : public TSelector { - -private: - - TString fBaseDir; // URL where the files will be written - //Int_t fMaxNWorkers; - Long64_t fNEvents; //number of events in a file - Int_t fNTracks; //avg or min-avg number of tracks in an event - Int_t fNTracksMax; //max-avg number of tracks in an event - Int_t fRegenerate; //force generation of cleanup files - - TObject* fTotalGen; //events generated on this worker - TList* fFilesGenerated; //list of files generated - - TMacro *fGenerateFun; //Macro with the function to generate the files - -protected: - - Long64_t GenerateFiles(const char *filename, Long64_t sizenevents); - -public : - - TTree *fChain; //!pointer to the analyzed TTree or TChain - -// TSelEventGen(TTree *); - TSelEventGen(); - ~TSelEventGen() override { } - Int_t Version() const override {return 1;} - void Begin(TTree *) override; - void SlaveBegin(TTree *tree) override; - void Init(TTree *tree) override; - Bool_t Notify() override; - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override {fInput = input;} - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - void Print(Option_t *option="") const override; - - ClassDefOverride(TSelEventGen,0) //PROOF selector for event file generation -}; - -#endif - -#ifdef TSelEventGen_cxx - -void TSelEventGen::Init(TTree *tree) -{ - // The Init() function is called when the selector needs to initialize - // a new tree or chain. Typically here the branch addresses of the tree - // will be set. It is normaly not necessary to make changes to the - // generated code, but the routine can be extended by the user if needed. - // Init() will be called many times when running with PROOF. - - if (tree == 0) return; - fChain = tree; - fChain->SetMakeClass(1); -} - -Bool_t TSelEventGen::Notify() -{ - // The Notify() function is called when a new file is opened. This - // can be either for a new TTree in a TChain or when when a new TTree - // is started when using PROOF. Typically here the branch pointers - // will be retrieved. It is normaly not necessary to make changes - // to the generated code, but the routine can be extended by the - // user if needed. - - return kTRUE; -} - -#endif // #ifdef TSelEventGen_cxx diff --git a/proof/proofbench/inc/TSelHandleDataSet.h b/proof/proofbench/inc/TSelHandleDataSet.h deleted file mode 100644 index a8ffe380a2a8d..0000000000000 --- a/proof/proofbench/inc/TSelHandleDataSet.h +++ /dev/null @@ -1,61 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TSelHandleDataSet -#define ROOT_TSelHandleDataSet - -////////////////////////////////////////////////////////////////////////// -// // -// TSelHandleDataSet // -// // -// PROOF selector for file cache release. // -// List of files to be cleaned for each node is provided by client. // -// // -////////////////////////////////////////////////////////////////////////// - -#include -#include - -class TList; -class TPBHandleDSType; - -class TSelHandleDataSet : public TSelector { - -private: - TPBHandleDSType *fType; - TString fDestDir; - void ReleaseCache(const char *fn); - void CheckCache(const char *fn); - void RemoveFile(const char *fn); - void CopyFile(const char *fn); - -public : - - TSelHandleDataSet() : fType(0) { } - ~TSelHandleDataSet() override { } - Int_t Version() const override {return 2;} - void Begin(TTree *) override { } - void SlaveBegin(TTree *) override; - void Init(TTree *) override { } - Bool_t Notify() override { return kTRUE; } - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override {fInput = input;} - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override { } - void Terminate() override { } - - ClassDefOverride(TSelHandleDataSet,0) //PROOF selector for event file generation -}; - -#endif - diff --git a/proof/proofbench/inc/TSelHist.h b/proof/proofbench/inc/TSelHist.h deleted file mode 100644 index 31c4f76288b82..0000000000000 --- a/proof/proofbench/inc/TSelHist.h +++ /dev/null @@ -1,63 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TSelHist -#define ROOT_TSelHist - -////////////////////////////////////////////////////////////////////////// -// // -// TSelHist // -// PROOF selector for CPU-intensive benchmark test. // -// Events are generated and 1-D, 2-D, and/or 3-D histograms are filled. // -// // -////////////////////////////////////////////////////////////////////////// - -#include - -class TH1F; -class TH2F; -class TH3F; -class TRandom3; -class TCanvas; -class TPBHistType; - -class TSelHist : public TSelector { -public : - - // Specific members - TPBHistType *fHistType; - Int_t fNHists; - Bool_t fDraw; - TH1F **fHist1D;//[fNHists] - TH2F **fHist2D;//[fNHists] - TH3F **fHist3D;//[fNHists] - TRandom3 *fRandom; - TCanvas *fCHist1D; - TCanvas *fCHist2D; - TCanvas *fCHist3D; - - TSelHist(); - ~TSelHist() override; - Int_t Version() const override { return 2; } - void Begin(TTree *tree) override; - void SlaveBegin(TTree *tree) override; - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override { fInput = input; } - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(TSelHist,0) //PROOF selector for CPU-intensive benchmark test -}; - -#endif diff --git a/proof/proofbench/src/TProofBench.cxx b/proof/proofbench/src/TProofBench.cxx deleted file mode 100644 index bbbd620f881ce..0000000000000 --- a/proof/proofbench/src/TProofBench.cxx +++ /dev/null @@ -1,1605 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: G.Ganis, S.Ryu Feb 2011 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ -/** - \defgroup proofbench PROOF benchmark utilities - \ingroup proof - - Set of utilities to benchmark a PROOF facility. - -*/ - -/** \class TProofBench -\ingroup proofbench - - Steering class for PROOF benchmarks - -*/ - -#include "RConfigure.h" - -#include "TProofBench.h" -#include "Getline.h" -#include "TProofBenchRunCPU.h" -#include "TProofBenchRunDataRead.h" -#include "TProofBenchDataSet.h" -#include "TProofNodes.h" -#include "TClass.h" -#include "TFile.h" -#include "TFileCollection.h" -#include "TFileInfo.h" -#include "THashList.h" -#include "TKey.h" -#include "TMap.h" -#include "TObjString.h" -#include "TProof.h" -#include "TROOT.h" -#include "TSortedList.h" -#include "TTimeStamp.h" -#include "TUrl.h" - -#include "TCanvas.h" -#include "TF1.h" -#include "TGraphErrors.h" -#include "TH1F.h" -#include "TMath.h" -#include "TProfile.h" -#include "TStyle.h" -#include "TLegend.h" -#ifdef WIN32 -#include -#endif - -ClassImp(TProofBench); - -// Functions for fitting - -TF1 *TProofBench::fgFp1 = 0; -TF1 *TProofBench::fgFp1n = 0; -TF1 *TProofBench::fgFp2 = 0; -TF1 *TProofBench::fgFp2n = 0; -TF1 *TProofBench::fgFp3 = 0; -TF1 *TProofBench::fgFp3n = 0; -TF1 *TProofBench::fgFio = 0; -TF1 *TProofBench::fgFioV = 0; -static Int_t gFioVn0 = -1; // Number of real cores for fgFioV -static Int_t gFioVn1 = -1; // Number of real+hyper cores for fgFioV - -TList *TProofBench::fgGraphs = new TList; - -//////////////////////////////////////////////////////////////////////////////// -/// Simple polynomial 1st degree - -Double_t funp1(Double_t *xx, Double_t *par) -{ - Double_t res = par[0] + par[1] * xx[0]; - return res; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Simple polynomial 2nd degree - -Double_t funp2(Double_t *xx, Double_t *par) -{ - Double_t res = par[0] + par[1] * xx[0] + par[2] * xx[0] * xx[0]; - return res; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Normalized 1st degree - -Double_t funp1n(Double_t *xx, Double_t *par) -{ - Double_t res = par[0] / xx[0] + par[1]; - return res; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Normalized 2nd degree - -Double_t funp2n(Double_t *xx, Double_t *par) -{ - Double_t res = par[0] / xx[0] + par[1] + par[2] * xx[0]; - return res; -} - -//////////////////////////////////////////////////////////////////////////////// -/// I/O saturated rate function - -Double_t funio(Double_t *xx, Double_t *par) -{ - Double_t sat = par[0] / par[1] * (xx[0] * par[1] / par[2] - 1.); - if (xx[0] < par[2] / par[1]) sat = 0.; - Double_t res = par[0] * xx[0] / (1. + sat); - return res; -} - -//////////////////////////////////////////////////////////////////////////////// -/// I/O saturated rate function with varying Rcpu - -Double_t funiov(Double_t *xx, Double_t *par) -{ - // par[0] = rio - // par[1] = b1 - // par[2] = b2 - // par[3] = nc - // par[4] = ri - - Double_t rio = par[0] / par[3] * xx[0]; - if (xx[0] > par[3]) rio = par[0]; - - Double_t rcpu = par[1] * xx[0]; - if (xx[0] > gFioVn0) rcpu = par[1]*gFioVn0 + par[2]*(xx[0] - gFioVn0); - if (xx[0] > gFioVn1) rcpu = par[1]*gFioVn0 + par[2]*(gFioVn1 - gFioVn0); - - Double_t res = 1. / (1./par[4] + 1./rio + 1./rcpu); - - return res; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Function with varying Rcpu - -Double_t funcpuv(Double_t *xx, Double_t *par) -{ - // par[0] = offset - // par[1] = rate contribution from real cores - // par[2] = rate contribution from hyper cores - - Double_t n = (xx[0] - par[0]); - Double_t rcpu = par[1] * n; - if (xx[0] > gFioVn0) rcpu = par[1]*gFioVn0 + par[2]*(n - gFioVn0); - if (xx[0] > gFioVn1) rcpu = par[1]*gFioVn0 + par[2]*(gFioVn1 - gFioVn0); - - return rcpu; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Function with varying Rcpu normalized - -Double_t funcpuvn(Double_t *xx, Double_t *par) -{ - // par[0] = offset - // par[1] = rate contribution from real cores - // par[2] = rate contribution from hyper cores - - Double_t n = (xx[0] - par[0]); - Double_t rcpu = par[1] * n; - if (xx[0] > gFioVn0) rcpu = par[1]*gFioVn0 + par[2]*(n - gFioVn0); - if (xx[0] > gFioVn1) rcpu = par[1]*gFioVn0 + par[2]*(gFioVn1 - gFioVn0); - - return rcpu / xx[0]; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor: check PROOF and load selectors PAR - -TProofBench::TProofBench(const char *url, const char *outfile, const char *proofopt) - : fUnlinkOutfile(kFALSE), fProofDS(0), fOutFile(0), - fNtries(4), fHistType(0), fNHist(16), fReadType(0), - fDataSet("BenchDataSet"), fNFilesWrk(2), fReleaseCache(kTRUE), - fDataGenSel(kPROOF_BenchSelDataGenDef), - fRunCPU(0), fRunDS(0), fDS(0), fDebug(kFALSE), fDescription(0) -{ - SetBit(kInvalidObject); - if (!url) { - Error("TProofBench", "specifying a PROOF master url is mandatory - cannot continue"); - return; - } - if (!(fProof = TProof::Open(url, proofopt)) || (fProof && !fProof->IsValid())) { - Error("TProofBench", "could not open a valid PROOF session - cannot continue"); - return; - } - // Get the size of the cluster - fNumWrkMax = fProof->GetParallel(); - if (fProof->UseDynamicStartup() && TProof::GetEnvVars()) { - // It must be passed as PROOF option 'workers=N' and recorded in the envs vars - TNamed *n = (TNamed *) TProof::GetEnvVars()->FindObject("PROOF_NWORKERS"); - if (!n) { - Error("TProofBench", "dynamic mode: you must specify the max number of workers"); - fProof->Close(); - SafeDelete(fProof); - return; - } - TString sn(n->GetTitle()); - if (sn.IsDigit()) fNumWrkMax = sn.Atoi(); - if (!sn.IsDigit()) { - Error("TProofBench", "dynamic mode: wrong specification of the max number of" - " workers ('%s')", n->GetTitle()); - fProof->Close(); - SafeDelete(fProof); - return; - } - } - if (fNumWrkMax <= 0) { - Error("TProofBench", "wrong max number of workers ('%d')", fNumWrkMax); - fProof->Close(); - SafeDelete(fProof); - return; - } - // By default we use the same instance for dataset actions - fProofDS = fProof; - // The object is now valid - ResetBit(kInvalidObject); - // Identifying string - TUrl u(url); - TString host(TString::Format("PROOF at %s", u.GetHost())); - if (!strcmp(u.GetProtocol(), "lite")) host.Form("PROOF-Lite on %s", gSystem->HostName()); - fDescription = new TNamed("PB_description", - TString::Format("%s, %d workers", host.Data(), fNumWrkMax).Data()); - Printf(" Run description: %s", fDescription->GetTitle()); - // Set output file - if (SetOutFile(outfile, kFALSE) != 0) - Warning("TProofBench", "problems opening '%s' - ignoring: use SetOutFile to try" - " again or with another file", outfile); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TProofBench::~TProofBench() -{ - CloseOutFile(); - if (fUnlinkOutfile) gSystem->Unlink(fOutFileName); - SafeDelete(fReadType); - SafeDelete(fRunCPU); - SafeDelete(fRunDS); - SafeDelete(fDescription); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the otuput file -/// Return 0 on success, -1 on error - -Int_t TProofBench::OpenOutFile(Bool_t wrt, Bool_t verbose) -{ - // Remove any bad file - if (fOutFile && fOutFile->IsZombie()) SafeDelete(fOutFile); - - Int_t rc = 0; - if (!fOutFile && fOutFileName.Length() > 0) { - const char *mode = 0; - if (wrt) - mode = gSystem->AccessPathName(fOutFileName) ? "RECREATE" : "UPDATE"; - else - mode = "READ"; - if (!(fOutFile = TFile::Open(fOutFileName, mode)) || (fOutFile && fOutFile->IsZombie())) { - if (verbose) - Warning("OpenOutFile", "problems opening '%s' - ignoring: use SetOutFile to try" - " again or with another file", fOutFileName.Data()); - rc = -1; - } - if (fOutFile) { - gROOT->GetListOfFiles()->Remove(fOutFile); - if (!strcmp(mode, "RECREATE")) { - // Save the description string - fOutFile->cd(); - fDescription->Write(); - } - } - } - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the output file -/// Return 0 on success, -1 on error - -Int_t TProofBench::SetOutFile(const char *outfile, Bool_t verbose) -{ - Int_t rc = 0; - // Close existing file, if any - if (fOutFile) { - if (!fOutFile->IsZombie()) fOutFile->Close(); - SafeDelete(fOutFile); - } - - fOutFileName = outfile; - if (fOutFileName == "") { - // Default output file: proofbench--.root - TDatime dat; - const char *lite = (fProof->IsLite()) ? "-lite" : ""; - fOutFileName.Form("proofbench-%s%s-%dw-%d-%.2d%.2d.root", - fProof->GetMaster(), lite, fNumWrkMax, - dat.GetDate(), dat.GetHour(), dat.GetMinute()); - Info("SetOutFile", "using default output file: '%s'", fOutFileName.Data()); - fUnlinkOutfile = kTRUE; - } - if (!fOutFileName.IsNull()) { - if ((rc = OpenOutFile(kTRUE, kFALSE)) != 0 && verbose) - Warning("SetOutFile", "problems opening '%s' - ignoring: use SetOutFile to try" - " again or with another file", outfile); - } - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Close output file - -void TProofBench::CloseOutFile() -{ - if (SetOutFile(0) != 0) - Warning("CloseOutFile", "problems closing '%s'", fOutFileName.Data()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Perform the CPU run -/// Return 0 on success, -1 on error - -Int_t TProofBench::RunCPU(Long64_t nevents, Int_t start, Int_t stop, Int_t step) -{ - // Open the file for the results - if (OpenOutFile(kTRUE) != 0) { - Error("RunCPU", "problems opening '%s' to save the result", fOutFileName.Data()); - return -1; - } - fUnlinkOutfile = kFALSE; - - SafeDelete(fRunCPU); - TPBHistType *htype = new TPBHistType(TPBHistType::kHist1D); // Owned by the input list - fRunCPU = new TProofBenchRunCPU(htype, fNHist, fOutFile); - if (!fCPUSel.IsNull()) fRunCPU->SetSelName(fCPUSel); - if (!fSelOption.IsNull()) fRunDS->SetSelOption(fSelOption); - if (!fCPUPar.IsNull()) fRunCPU->SetParList(fCPUPar); - fRunCPU->Run(nevents, start, stop, step, fNtries, fDebug, -1); - - // Close the file - if (SetOutFile(0) != 0) - Warning("RunCPU", "problems closing '%s'", fOutFileName.Data()); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Perform the CPU run scanning over the number of workers per node -/// Return 0 on success, -1 on error - -Int_t TProofBench::RunCPUx(Long64_t nevents, Int_t start, Int_t stop) -{ - // Open the file for the results - if (OpenOutFile(kTRUE) != 0) { - Error("RunCPUx", "problems opening '%s' to save the result", fOutFileName.Data()); - return -1; - } - fUnlinkOutfile = kFALSE; - - SafeDelete(fRunCPU); - TPBHistType *htype = new TPBHistType(TPBHistType::kHist1D); // Owned by the input list - fRunCPU = new TProofBenchRunCPU(htype, fNHist, fOutFile); - if (!fCPUSel.IsNull()) fRunCPU->SetSelName(fCPUSel); - if (!fSelOption.IsNull()) fRunDS->SetSelOption(fSelOption); - if (!fCPUPar.IsNull()) fRunCPU->SetParList(fCPUPar); - fRunCPU->Run(nevents, start, stop, -2, fNtries, fDebug, -1); - - // Close the file - if (SetOutFile(0) != 0) - Warning("RunCPUx", "problems closing '%s'", fOutFileName.Data()); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw the CPU speedup plot. -/// opt = 'typewhat', e.g. 'std:max:' -/// type = 'std:' draw standard evt/s plot -/// 'stdx:' draw standard evt/s plot, 1 worker per node -/// 'norm:' draw normalized plot -/// 'normx:' draw normalized plot, 1 worker per node -/// what = 'max:' draw max rate -/// 'avg:' draw average rate -/// 'all:' draw max and average rate on same plot (default) -/// dofit = 0 no fit -/// 1 fit with the relevant '1st degree related' function -/// 2 fit with the relevant '2nd degree related' function -/// 3 fit with varying rcpu function -/// n0 = for dofit == 3, number of real cores -/// n1 = for dofit == 3, number of total cores (real + hyperthreaded) -/// - -void TProofBench::DrawCPU(const char *outfile, const char *opt, Bool_t verbose, - Int_t dofit, Int_t n0, Int_t n1) -{ - // Get the TProfile an create the graphs - TFile *fout = TFile::Open(outfile, "READ"); - if (!fout || (fout && fout->IsZombie())) { - ::Error("DrawCPU", "could not open file '%s' ...", outfile); - return; - } - - // Get description - TString description(""); - TNamed *nmdesc = (TNamed *) fout->Get("PB_description"); - if (nmdesc) description = nmdesc->GetTitle(); - - // Parse option - TString oo(opt); - Bool_t isNorm = (oo.Contains("norm")) ? kTRUE : kFALSE; - Bool_t isX = (oo.Contains("stdx:") || oo.Contains("normx:")) ? kTRUE : kFALSE; - Bool_t doAvg = kTRUE, doMax = kTRUE; - if (oo.Contains("avg:")) doMax = kFALSE; - if (oo.Contains("max:")) doAvg = kFALSE; - - const char *dirn = (isX) ? "RunCPUx" : "RunCPU"; - TDirectory *d = (TDirectory *) fout->Get(dirn); - if (!d) { - ::Error("DrawCPU", "could not find directory '%s' ...", dirn); - fout->Close(); - delete fout; - return; - } - d->cd(); - - TString hprofn, hmaxn; - const char *lx = (isX) ? "_x" : ""; - const char *ln = (isNorm) ? "Norm" : "Prof"; - hprofn.Form("%s%s_CPU_QR_Evts", ln, lx); - hmaxn.Form("%s%s_CPU_PS_MaxEvts", ln, lx); - - Double_t xmin = -1., xmax = -1.; - Double_t ami = -1., amx = -1., mmi = -1., mmx = -1.; - Int_t kamx = -1, kmmx = -1, nbins = -1; - Double_t ymx = -1., ymi = -1.; - - TProfile *pf = 0; - Int_t kmx = -1; - - TProfile *pfav = 0; - TGraphErrors *grav = 0; - if (doAvg) { - if (!(grav = GetGraph(d, hprofn, nbins, xmin, xmax, ami, amx, kamx, pfav))) { - ::Error("DrawCPU", "could not find '%s' ...", hprofn.Data()); - fout->Close(); - delete fout; - return; - } - ymx = amx; - ymi = ami; - pf = pfav; - kmx = kamx; - } - TProfile *pfmx = 0; - TGraphErrors *grmx = 0; - if (doMax) { - if (!(grmx = GetGraph(d, hmaxn, nbins, xmin, xmax, mmi, mmx, kmmx, pfmx))) { - ::Warning("DrawCPU", "could not find '%s': feature added in 5.34/11", hmaxn.Data()); - if (!grav) { - // Nothing to do if not asked for the average - fout->Close(); - delete fout; - return; - } - doMax = kFALSE; - } - if (mmx > ymx) ymx = mmx; - if ((ymi > 0 && mmi < ymi) || (ymi < 0.)) ymi = mmi; - pf = pfmx; - kmx = kmmx; - } - - // Create the canvas - TCanvas *cpu = new TCanvas("cpu", "Rate vs wrks",204,69,1050,502); - cpu->Range(-3.106332,0.7490716,28.1362,1.249867); - - TH1F *hgr = new TH1F("Graph-CPU"," CPU speed-up", nbins*4, xmin, xmax); - hgr->SetMaximum(ymx + (ymx-ymi)*0.2); - hgr->SetMinimum(0); - if (isNorm) hgr->SetMaximum(ymx*1.2); - hgr->SetDirectory(0); - hgr->SetStats(0); - hgr->GetXaxis()->SetTitle(pf->GetXaxis()->GetTitle()); - hgr->GetXaxis()->CenterTitle(true); - hgr->GetXaxis()->SetLabelSize(0.05); - hgr->GetXaxis()->SetTitleSize(0.06); - hgr->GetXaxis()->SetTitleOffset(0.62); - hgr->GetYaxis()->SetTitleSize(0.08); - hgr->GetYaxis()->SetTitleOffset(0.52); - hgr->GetYaxis()->SetTitle("Rate (events/s)"); - - TLegend *leg = 0; - if (isNorm) { - leg = new TLegend(0.7, 0.8, 0.9, 0.9); - } else { - leg = new TLegend(0.1, 0.8, 0.3, 0.9); - } - - gStyle->SetOptTitle(0); - TGraphErrors *gr = 0; - if (doAvg) { - grav->SetFillColor(1); - grav->SetLineColor(13); - grav->SetMarkerColor(4); - grav->SetMarkerStyle(21); - grav->SetMarkerSize(1.2); - grav->SetHistogram(hgr); - - if (verbose) grav->Print(); - grav->Draw("alp"); - leg->AddEntry(grav, "Average", "P"); - gr = grav; - } - if (doMax) { - grmx->SetFillColor(1); - grmx->SetLineColor(13); - grmx->SetMarkerColor(2); - grmx->SetMarkerStyle(29); - grmx->SetMarkerSize(1.8); - grmx->SetHistogram(hgr); - - if (verbose) grmx->Print(); - if (doAvg) { - grmx->Draw("lpSAME"); - } else { - grmx->Draw("alp"); - } - leg->AddEntry(grmx, "Maximum", "P"); - gr = grmx; - } - leg->Draw(); - gPad->Update(); - - if (dofit > 0) { - // Make sure the fitting functions are defined - Double_t xmi = 0.9; - if (nbins > 5) xmi = 1.5; - AssertFittingFun(xmi, nbins + .1); - - // Starting point for the parameters and fit - Double_t normrate = -1.; - if (dofit == 1) { - if (isNorm) { - fgFp1n->SetParameter(0, pf->GetBinContent(1)); - fgFp1n->SetParameter(1, pf->GetBinContent(nbins-1)); - gr->Fit(fgFp1n); - if (verbose) fgFp1n->Print(); - normrate = fgFp1n->GetParameter(1); - } else { - fgFp1->SetParameter(0, 0.); - fgFp1->SetParameter(1, pf->GetBinContent(1)); - gr->Fit(fgFp1); - if (verbose) fgFp1->Print(); - normrate = fgFp1->Derivative(1.); - } - } else if (dofit == 2) { - if (isNorm) { - fgFp2n->SetParameter(0, pf->GetBinContent(1)); - fgFp2n->SetParameter(1, pf->GetBinContent(nbins-1)); - fgFp2n->SetParameter(2, 0.); - gr->Fit(fgFp2n); - if (verbose) fgFp2n->Print(); - normrate = fgFp2n->GetParameter(1); - } else { - fgFp2->SetParameter(0, 0.); - fgFp2->SetParameter(1, pf->GetBinContent(1)); - fgFp2->SetParameter(2, 0.); - gr->Fit(fgFp2); - if (verbose) fgFp2->Print(); - normrate = fgFp2->Derivative(1.); - } - } else { - // Starting point for the parameters and fit - gFioVn0 = (n0 > 0) ? n0 : (Int_t) (nbins + .1)/2.; - gFioVn1 = (n1 > 0) ? n1 : (Int_t) (nbins + .1); - if (isNorm) { - fgFp3n->SetParameter(0, 0.); - fgFp3n->SetParameter(1, pf->GetBinContent(1)); - fgFp3n->SetParameter(2, pf->GetBinContent(nbins-1)); - gr->Fit(fgFp3n); - if (verbose) fgFp3n->Print(); - normrate = pf->GetBinContent(1); - } else { - fgFp3->SetParameter(0, 0.); - fgFp3->SetParameter(1, 0.); - fgFp3->SetParameter(2, pf->GetBinContent(1)); - gr->Fit(fgFp3); - if (verbose) fgFp3->Print(); - normrate = fgFp3->Derivative(1.); - } - } - - // Notify the cluster performance parameters - if (!isNorm) { - printf("* ************************************************************ *\n"); - printf("* *\r"); - printf("* Cluster: %s\n", description.Data()); - printf("* Performance measurement from scalability plot: *\n"); - printf("* *\r"); - printf("* rate max: %.3f\tmegaRNGPS (@ %d workers)\n", ymx/1000000, kmx); - printf("* *\r"); - printf("* per-worker rate: %.3f\tmegaRNGPS \n", normrate/1000000); - printf("* ************************************************************ *\n"); - } else { - printf("* ************************************************************ *\n"); - printf("* *\r"); - printf("* Cluster: %s\n", description.Data()); - printf("* *\r"); - printf("* Per-worker rate from normalized plot: %.3f\tmegaRNGPS\n", normrate/1000000); - printf("* ************************************************************ *\n"); - } - } - // Close the file - fout->Close(); - if (grav) fgGraphs->Add(grav); - if (grmx) fgGraphs->Add(grmx); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get from TDirectory 'd' the TProfile named 'pfn' and create the graph. -/// Return also the max y in mx. - -TGraphErrors *TProofBench::GetGraph(TDirectory *d, const char *pfn, Int_t &nb, - Double_t &xmi, Double_t &xmx, - Double_t &ymi, Double_t &ymx, Int_t &kmx, TProfile *&pf) -{ - // Sanity checks - if (!d || !pfn || (pfn && strlen(pfn) <= 0)) { - ::Error("TProofBench::GetGraph", "directory or name not defined!"); - return (TGraphErrors *)0; - } - - TList *keylist = d->GetListOfKeys(); - TKey *key = 0; - TIter nxk(keylist); - while ((key = (TKey *) nxk())) { - if (TString(key->GetName()).BeginsWith(pfn)) { - pf = (TProfile *) d->Get(key->GetName()); - break; - } - } - // Sanity checks - if (!pf) { - ::Error("TProofBench::GetGraph", "TProfile for '%s' not found in directory '%s'", pfn, d->GetName()); - return (TGraphErrors *)0; - } - - nb = pf->GetNbinsX(); - TGraphErrors *gr = new TGraphErrors(nb); - gr->SetName(TString::Format("Graph_%s", pfn)); - Double_t xx, ex, yy, ey; - ymi = pf->GetBinContent(1); - ymx = ymi; - xmi = pf->GetBinCenter(1) - pf->GetBinWidth(1)/2. ; - xmx = pf->GetBinCenter(nb) + pf->GetBinWidth(nb)/2. ; - kmx = -1; - for (Int_t k = 1;k <= nb; k++) { - xx = pf->GetBinCenter(k); - ex = pf->GetBinWidth(k) * .001; - yy = pf->GetBinContent(k); - ey = pf->GetBinError(k); - if (k == 1) { - ymi = yy; - ymx = yy; - kmx = k; - } else { - if (yy < ymi) ymi = yy; - if (yy > ymx) { ymx = yy; kmx = k; } - } - gr->SetPoint(k-1, xx, yy); - gr->SetPointError(k-1, ex, ey); - } - - // Done - return gr; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that the fitting functions are defined - -void TProofBench::AssertFittingFun(Double_t mi, Double_t mx) -{ - if (!fgFp1) { - fgFp1 = new TF1("funp1", funp1, mi, mx, 2); - fgFp1->SetParNames("offset", "slope"); - } - - if (!fgFp1n) { - fgFp1n = new TF1("funp1n", funp1n, mi, mx, 2); - fgFp1n->SetParNames("decay", "norm rate"); - } - - if (!fgFp2) { - fgFp2 = new TF1("funp2", funp2, mi, mx, 3); - fgFp2->SetParNames("offset", "slope", "deviation"); - } - - if (!fgFp2n) { - fgFp2n = new TF1("funp2n", funp2n, mi, mx, 3); - fgFp2n->SetParNames("decay", "norm rate", "deviation"); - } - - if (!fgFp3) { - fgFp3 = new TF1("funcpuv", funcpuv, mi, mx, 3); - fgFp3->SetParNames("offset", "slope real", "slope hyper"); - } - - if (!fgFp3n) { - fgFp3n = new TF1("funcpuvn", funcpuvn, mi, mx, 3); - fgFp3n->SetParNames("offset", "slope real", "slope hyper"); - } - - if (!fgFio) { - fgFio = new TF1("funio", funio, mi, mx, 3); - fgFio->SetParNames("R1", "RIO", "TotIO"); - } - if (!fgFioV) { - fgFioV = new TF1("funiov", funiov, mi, mx, 5); - fgFioV->SetParNames("rio", "b1", "b2", "nc", "ri"); - } - -} - -//////////////////////////////////////////////////////////////////////////////// - -class fileDesc : public TNamed { -public: - Long_t fMtime; // Modification time - TString fDesc; // Test description string, if any - fileDesc(const char *n, const char *o, - Long_t t, const char *d) : TNamed(n, o), fMtime(t), fDesc(d) { } - Int_t Compare(const TObject *o) const override { - const fileDesc *fd = static_cast(o); - if (!fd || (fd && fd->fMtime == fMtime)) return 0; - if (fMtime < fd->fMtime) return -1; - return 1; - } -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Get performance specs. Check file 'path', or files in directory 'path' -/// (default current directory). -/// The degree of the polynomial used for the fit is 'degfit' (default 1). - -void TProofBench::GetPerfSpecs(const char *path, Int_t degfit) -{ - // Locate the file (ask if many) - TString pp(path), fn, oo; - if (pp.IsNull()) pp = gSystem->WorkingDirectory(); - FileStat_t st; - if (gSystem->GetPathInfo(pp.Data(), st) != 0) { - ::Error("TProofBench::GetPerfSpecs", "path '%s' could not be stat'ed - abort", pp.Data()); - return; - } - TSortedList filels; - if (R_ISDIR(st.fMode)) { - // Scan the directory - void *dirp = gSystem->OpenDirectory(pp.Data()); - if (!dirp) { - ::Error("TProofBench::GetPerfSpecs", "directory path '%s' could nto be open - abort", pp.Data()); - return; - } - const char *ent = 0; - while ((ent = gSystem->GetDirEntry(dirp))) { - if (!strcmp(ent, ".") || !strcmp(ent, "..")) continue; - fn.Form("%s/%s", pp.Data(), ent); - if (gSystem->GetPathInfo(fn.Data(), st) != 0) continue; - if (!R_ISREG(st.fMode)) continue; - fn += "?filetype=raw"; - TFile *f = TFile::Open(fn); - if (!f) continue; - char rr[5] = {0}; - if (!f->ReadBuffer(rr, 4)) { - if (!strncmp(rr, "root", 4)) { - SafeDelete(f); - fn.ReplaceAll("?filetype=raw", ""); - if ((f = TFile::Open(fn))) { - TString desc(""); - TNamed *nmdesc = (TNamed *) f->Get("PB_description"); - if (nmdesc) desc = nmdesc->GetTitle(); - if (f->GetListOfKeys()->FindObject("RunCPU")) - filels.Add(new fileDesc(fn, "std:", st.fMtime, desc.Data())); - if (f->GetListOfKeys()->FindObject("RunCPUx")) - filels.Add(new fileDesc(fn, "stdx:", st.fMtime, desc.Data())); - } else { - ::Warning("TProofBench::GetPerfSpecs", "problems opening '%s'", fn.Data()); - } - } - } - SafeDelete(f); - } - gSystem->FreeDirectory(dirp); - } else if (!R_ISREG(st.fMode)) { - ::Error("TProofBench::GetPerfSpecs", - "path '%s' not a regular file nor a directory - abort", pp.Data()); - return; - } else { - // This is the file - fn = pp; - // Check it - TString emsg; - Bool_t isOk = kFALSE; - if (gSystem->GetPathInfo(fn.Data(), st) == 0) { - fn += "?filetype=raw"; - TFile *f = TFile::Open(fn); - if (f) { - char rr[5] = {0}; - if (!(f->ReadBuffer(rr, 4))) { - if (!strncmp(rr, "root", 4)) { - fn.ReplaceAll("?filetype=raw", ""); - if ((f = TFile::Open(fn))) { - if (f->GetListOfKeys()->FindObject("RunCPU")) oo = "std:"; - if (f->GetListOfKeys()->FindObject("RunCPUx")) oo = "stdx:"; - SafeDelete(f); - if (!oo.IsNull()) { - isOk = kTRUE; - } else { - emsg.Form("path '%s' does not contain the relevant dirs - abort", fn.Data()); - } - } else { - emsg.Form("path '%s' cannot be open - abort", fn.Data()); - } - } else { - emsg.Form("'%s' is not a ROOT file - abort", fn.Data()); - } - } else { - emsg.Form("could not read first 4 bytes from '%s' - abort", fn.Data()); - } - SafeDelete(f); - } else { - emsg.Form("path '%s' cannot be open in raw mode - abort", fn.Data()); - } - } else { - emsg.Form("path '%s' cannot be stated - abort", fn.Data()); - } - if (!isOk) { - ::Error("TProofBench::GetPerfSpecs", "%s", emsg.Data()); - return; - } - } - - fileDesc *nm = 0; - // Ask the user, if more then 1 - if (filels.GetSize() == 1) { - nm = (fileDesc *) filels.First(); - fn = nm->GetName(); - oo = nm->GetTitle(); - } else if (filels.GetSize() > 1) { - TIter nxf(&filels); - Int_t idx = 0; - Printf("Several possible files found:"); - while ((nm = (fileDesc *) nxf())) { - Printf(" %d\t%s\t%s\t%s (file: %s)", idx++, nm->GetTitle(), - TTimeStamp(nm->fMtime).AsString("s"), nm->fDesc.Data(), nm->GetName()); - } - TString a(Getline(TString::Format("Make your choice [%d] ", idx-1))); - if (a.IsNull() || a[0] == '\n') a.Form("%d", idx-1); - idx = a.Atoi(); - if ((nm = (fileDesc *) filels.At(idx))) { - fn = nm->GetName(); - oo = nm->GetTitle(); - } else { - ::Error("TProofBench::GetPerfSpecs", "chosen index '%d' does not exist - abort", idx); - return; - } - } else { - if (fn.IsNull()) { - ::Error("TProofBench::GetPerfSpecs", - "path '%s' is a directory but no ROOT file found in it - abort", pp.Data()); - return; - } - } - - // Now get the specs - TProofBench::DrawCPU(fn.Data(), oo.Data(), kFALSE, degfit); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Perform a test using dataset 'dset' -/// Return 0 on success, -1 on error -/// Open the file for the results - -Int_t TProofBench::RunDataSet(const char *dset, - Int_t start, Int_t stop, Int_t step) -{ - if (OpenOutFile(kTRUE) != 0) { - Error("RunDataSet", "problems opening '%s' to save the result", fOutFileName.Data()); - return -1; - } - fUnlinkOutfile = kFALSE; - - if (fReleaseCache) ReleaseCache(dset); - SafeDelete(fRunDS); - TPBReadType *readType = fReadType; - if (!readType) readType = new TPBReadType(TPBReadType::kReadOpt); - fRunDS = new TProofBenchRunDataRead(fDS, readType, fOutFile); - if (!fDataSel.IsNull()) fRunDS->SetSelName(fDataSel); - if (!fSelOption.IsNull()) fRunDS->SetSelOption(fSelOption); - if (!fDataPar.IsNull()) fRunDS->SetParList(fDataPar); - fRunDS->SetReleaseCache(fReleaseCache); - fRunDS->Run(dset, start, stop, step, fNtries, fDebug, -1); - SafeDelete(readType); - - // Close the file - if (SetOutFile(0) != 0) - Warning("RunDataSet", "problems closing '%s'", fOutFileName.Data()); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Perform a test using dataset 'dset' scanning over the number of workers -/// per node. -/// Return 0 on success, -1 on error -/// Open the file for the results - -Int_t TProofBench::RunDataSetx(const char *dset, Int_t start, Int_t stop) -{ - if (OpenOutFile(kTRUE) != 0) { - Error("RunDataSetx", "problems opening '%s' to save the result", fOutFileName.Data()); - return -1; - } - fUnlinkOutfile = kFALSE; - - ReleaseCache(dset); - SafeDelete(fRunDS); - TPBReadType *readType = fReadType; - if (!readType) readType = new TPBReadType(TPBReadType::kReadOpt); - fRunDS = new TProofBenchRunDataRead(fDS, readType, fOutFile); - if (!fDataSel.IsNull()) fRunDS->SetSelName(fDataSel); - if (!fSelOption.IsNull()) fRunDS->SetSelOption(fSelOption); - if (!fDataPar.IsNull()) fRunDS->SetParList(fDataPar); - fRunDS->Run(dset, start, stop, -2, fNtries, fDebug, -1); - SafeDelete(readType); - - // Close the file - if (SetOutFile(0) != 0) - Warning("RunDataSetx", "problems closing '%s'", fOutFileName.Data()); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw the CPU speedup plot. -/// opt = 'typewhat', e.g. 'std:max:' -/// type = 'std:' draw standard plot -/// 'stdx:' draw standard plot, 1 worker per node -/// 'norm:' draw normalized plot -/// 'normx:' draw normalized plot, 1 worker per node -/// what = 'max:' draw max rate -/// 'avg:' draw average rate -/// 'all:' draw max and average rate on same plot (default) -/// type = 'mbs' MB/s scaling plots (default) -/// 'evts' Event/s scaling plots -/// dofit = 0 no fit -/// 1 fit with default 3 parameter saturated I/O formula -/// 2 fit with 4 parameter saturated I/O formula (varying Rcpu) -/// n0 = for dofit == 2, number of real cores -/// n1 = for dofit == 2, number of total cores (real + hyperthreaded) -/// - -void TProofBench::DrawDataSet(const char *outfile, - const char *opt, const char *type, Bool_t verbose, - Int_t dofit, Int_t n0, Int_t n1) -{ - // Get the TProfile an create the graphs - TFile *fout = TFile::Open(outfile, "READ"); - if (!fout || (fout && fout->IsZombie())) { - ::Error("DrawDataSet", "could not open file '%s' ...", outfile); - return; - } - - // Get description - TString description(""); - TNamed *nmdesc = (TNamed *) fout->Get("PB_description"); - if (nmdesc) description = nmdesc->GetTitle(); - - // Parse option - TString oo(opt); - Bool_t isNorm = (oo.Contains("norm")) ? kTRUE : kFALSE; - Bool_t isX = (oo.Contains("stdx:") || oo.Contains("normx:")) ? kTRUE : kFALSE; - Bool_t doAvg = (oo.Contains("all:") || oo.Contains("avg:")) ? kTRUE : kFALSE; - Bool_t doMax = (oo.Contains("all:") || oo.Contains("max:")) ? kTRUE : kFALSE; - - const char *dirn = (isX) ? "RunDataReadx" : "RunDataRead"; - TDirectory *d = (TDirectory *) fout->Get(dirn); - if (!d) { - ::Error("DrawCPU", "could not find directory '%s' ...", dirn); - fout->Close(); - delete fout; - return; - } - d->cd(); - - TString hprofn, hmaxn; - const char *lx = (isX) ? "_x" : ""; - const char *ln = (isNorm) ? "Norm" : "Prof"; - Bool_t isIO = kTRUE; - if (type && !strcmp(type, "evts")) { - hprofn.Form("%s%s_DataRead_QR_Evts", ln, lx); - hmaxn.Form("%s%s_DataRead_PS_MaxEvts", ln, lx); - isIO = kFALSE; - } else { - hprofn.Form("%s%s_DataRead_QR_IO", ln, lx); - hmaxn.Form("%s%s_DataRead_PS_MaxIO", ln, lx); - } - - Double_t xmin = -1., xmax = -1.; - Double_t ami = -1., amx = -1., mmi = -1., mmx = -1.; - Int_t kamx = -1, kmmx = -1, nbins = -1; - Double_t ymx = -1., ymi = -1.; - - TProfile *pf = 0; - Int_t kmx = -1; - - TProfile *pfav = 0; - TGraphErrors *grav = 0; - if (doAvg) { - if (!(grav = GetGraph(d, hprofn, nbins, xmin, xmax, ami, amx, kamx, pfav))) { - ::Error("DrawCPU", "could not find '%s' ...", hprofn.Data()); - fout->Close(); - delete fout; - return; - } - ymx = amx; - ymi = ami; - pf = pfav; - kmx = kamx; - } - - TProfile *pfmx = 0; - TGraphErrors *grmx = 0; - if (doMax) { - if (!(grmx = GetGraph(d, hmaxn, nbins, xmin, xmax, mmi, mmx, kmmx, pfmx))) { - ::Warning("DrawCPU", "could not find '%s': feature added in 5.34/11", hmaxn.Data()); - if (!grav) { - // Nothing to do if not asked for the average - fout->Close(); - delete fout; - return; - } - doMax = kFALSE; - } - if (mmx > ymx) ymx = mmx; - if ((ymi > 0 && mmi < ymi) || (ymi < 0.)) ymi = mmi; - pf = pfmx; - kmx = kmmx; - } - - // Create the canvas - TCanvas *cpu = new TCanvas("dataset", "Rate vs wrks",204,69,1050,502); - cpu->Range(-3.106332,0.7490716,28.1362,1.249867); - - TH1F *hgr = new TH1F("Graph-DataSet"," Data Read speed-up", nbins*4, xmin, xmax); - hgr->SetMaximum(ymx + (ymx-ymi)*0.2); - hgr->SetMinimum(0); - if (isNorm) hgr->SetMaximum(ymx*1.2); - hgr->SetDirectory(0); - hgr->SetStats(0); - hgr->GetXaxis()->SetTitle(pf->GetXaxis()->GetTitle()); - hgr->GetXaxis()->CenterTitle(true); - hgr->GetXaxis()->SetLabelSize(0.05); - hgr->GetXaxis()->SetTitleSize(0.06); - hgr->GetXaxis()->SetTitleOffset(0.62); - hgr->GetYaxis()->SetLabelSize(0.06); - hgr->GetYaxis()->SetTitleSize(0.08); - hgr->GetYaxis()->SetTitleOffset(0.52); - if (isIO) { - hgr->GetYaxis()->SetTitle("Rate (MB/s)"); - } else { - hgr->GetYaxis()->SetTitle("Rate (events/s)"); - } - - TLegend *leg = 0; - if (isNorm) { - leg = new TLegend(0.7, 0.8, 0.9, 0.9); - } else { - leg = new TLegend(0.1, 0.8, 0.3, 0.9); - } - - TGraphErrors *gr = 0; - if (doAvg) { - grav->SetFillColor(1); - grav->SetLineColor(13); - grav->SetMarkerColor(4); - grav->SetMarkerStyle(21); - grav->SetMarkerSize(1.2); - grav->SetHistogram(hgr); - - if (verbose) grav->Print(); - grav->Draw("alp"); - leg->AddEntry(grav, "Average", "P"); - gr = grav; - } - if (doMax) { - grmx->SetFillColor(1); - grmx->SetLineColor(13); - grmx->SetMarkerColor(2); - grmx->SetMarkerStyle(29); - grmx->SetMarkerSize(1.8); - grmx->SetHistogram(hgr); - - if (verbose) grmx->Print(); - if (doAvg) { - grmx->Draw("lpSAME"); - } else { - grmx->Draw("alp"); - } - leg->AddEntry(grmx, "Maximum", "P"); - gr = grmx; - } - leg->Draw(); - gPad->Update(); - - Double_t normrate = -1.; - if (dofit > 0) { - // Make sure the fitting functions are defined - Double_t xmi = 0.9; - if (nbins > 5) xmi = 1.5; - AssertFittingFun(xmi, nbins + .1); - - if (dofit == 1) { - // Starting point for the parameters and fit - fgFio->SetParameter(0, pf->GetBinContent(1)); - fgFio->SetParameter(1, pf->GetBinContent(nbins-1)); - fgFio->SetParameter(2, pf->GetBinContent(nbins-1)); - gr->Fit(fgFio); - if (verbose) fgFio->Print(); - normrate = fgFio->Derivative(1.); - } else if (dofit > 1) { - // Starting point for the parameters and fit - gFioVn0 = (n0 > 0) ? n0 : (Int_t) (nbins + .1)/2.; - gFioVn1 = (n1 > 0) ? n1 : (Int_t) (nbins + .1); - fgFioV->SetParameter(0, 20.); - fgFioV->SetParameter(1, pf->GetBinContent(1)); - fgFioV->SetParameter(2, pf->GetBinContent(1)); - fgFioV->SetParameter(3, 4.); - fgFioV->SetParameter(4, 1000.); - - gr->Fit(fgFioV); - if (verbose) fgFio->Print(); - normrate = fgFioV->Derivative(1.); - } - } - - // Notify the cluster performance parameters - if (!isNorm) { - printf("* ************************************************************ *\n"); - printf("* *\r"); - printf("* Cluster: %s\n", description.Data()); - printf("* Performance measurement from scalability plot: *\n"); - printf("* *\r"); - if (isIO) { - printf("* rate max: %.3f\tMB/s (@ %d workers)\n", ymx, kmx); - printf("* *\r"); - printf("* per-worker rate: %.3f\tMB/s \n", normrate); - } else { - printf("* rate max: %.3f\tevts/s (@ %d workers)\n", ymx, kmx); - } - printf("* ************************************************************ *\n"); - } - // Close the file - fout->Close(); - if (grav) fgGraphs->Add(grav); - if (grmx) fgGraphs->Add(grmx); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw the efficiency plot. -/// opt = 'cpu' or 'data' (default the first found) -/// - -void TProofBench::DrawEfficiency(const char *outfile, - const char *opt, Bool_t verbose) -{ - // Get the TProfile an create the graphs - TFile *fout = TFile::Open(outfile, "READ"); - if (!fout || (fout && fout->IsZombie())) { - ::Error("DrawEfficiency", "could not open file '%s' ...", outfile); - return; - } - - // Get description - TString description(""); - TNamed *nmdesc = (TNamed *) fout->Get("PB_description"); - if (nmdesc) description = nmdesc->GetTitle(); - - // Parse option - TString oo(opt), ln("CPU"); - const char *dirs[4] = { "RunCPU", "RunCPUx", "RunDataRead", "RunDataReadx"}; - const char *labs[4] = { "CPU", "CPU", "DataRead", "DataRead"}; - Int_t fst = 0, lst = 3; - if (oo == "cpu") { - lst = 0; - } else if (oo == "cpux") { - fst = 1; - lst = 1; - } else if (oo.BeginsWith("data")) { - if (oo.EndsWith("x")) { - fst = 3; - lst = 3; - } else { - fst = 2; - lst = 2; - } - } - const char *dirn = 0; - TDirectory *d = 0; - for (Int_t i = fst; i <= lst; i++) { - if ((d = (TDirectory *) fout->Get(dirs[i]))) { - dirn = dirs[i]; - ln = labs[i]; - break; - } - } - if (!d && !dirn) { - ::Error("DrawEfficiency", "could not find directory ..."); - fout->Close(); - delete fout; - return; - } - d->cd(); - - TString hprof; - hprof.Form("Prof_%s_CPU_eff", ln.Data()); - - Double_t xmin = -1., xmax = -1.; - Int_t kmx = -1, nbins = -1; - Double_t ymx = -1., ymi = -1.; - - TProfile *pf = 0; - TGraphErrors *gr = 0; - if (!(gr = GetGraph(d, hprof, nbins, xmin, xmax, ymi, ymx, kmx, pf))) { - ::Error("DrawEfficiency", "could not find '%s' ...", hprof.Data()); - fout->Close(); - delete fout; - return; - } - - // Create the canvas - TCanvas *cpu = new TCanvas("efficiency", "efficiency vs wrks",204,69,1050,502); - cpu->Range(-3.106332,0.7490716,28.1362,1.249867); - - TH1F *hgr = new TH1F("Graph-Efficiency","CPU effectiveness", nbins*4, xmin, xmax); - hgr->SetMaximum(1.2); - hgr->SetMinimum(0); - hgr->SetDirectory(0); - hgr->SetStats(0); - hgr->GetXaxis()->SetTitle(pf->GetXaxis()->GetTitle()); - hgr->GetXaxis()->CenterTitle(true); - hgr->GetXaxis()->SetLabelSize(0.05); - hgr->GetXaxis()->SetTitleSize(0.06); - hgr->GetXaxis()->SetTitleOffset(0.62); - hgr->GetYaxis()->SetLabelSize(0.06); - hgr->GetYaxis()->SetTitleSize(0.08); - hgr->GetYaxis()->SetTitleOffset(0.52); - hgr->GetYaxis()->SetTitle("CPU effectiveness"); - - gr->SetFillColor(1); - gr->SetLineColor(13); - gr->SetMarkerColor(4); - gr->SetMarkerStyle(21); - gr->SetMarkerSize(1.2); - gr->SetHistogram(hgr); - - if (verbose) gr->Print(); - gr->Draw("alp"); - - // Notify the cluster performance parameters - printf("* ************************************************************ *\n"); - printf("* *\r"); - printf("* Cluster: %s\n", description.Data()); - printf("* CPU effectiveness measurement: *\n"); - printf("* *\r"); - printf("* effectiveness max: %.3f (@ %d workers)\n", ymx, kmx); - printf("* *\r"); - printf("* ************************************************************ *\n"); - // Close the file - fout->Close(); - if (gr) fgGraphs->Add(gr); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Release memory cache for dataset 'dset' -/// Return 0 on success, -1 on error - -Int_t TProofBench::ReleaseCache(const char *dset) -{ - // Do it via the dataset handler - if (!fDS) fDS = new TProofBenchDataSet(fProofDS); - return fDS ? fDS->ReleaseCache(dset) : -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Physically remove the dataset 'dset', i.e. remove the dataset and the files -/// it describes -/// Return 0 on success, -1 on error - -Int_t TProofBench::RemoveDataSet(const char *dset) -{ - // Do it via the dataset handler - if (!fDS) fDS = new TProofBenchDataSet(fProofDS); - return fDS ? fDS->RemoveFiles(dset) : -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create the largest dataset for the run. -/// Defaults for -/// dataset name, filename root -/// are -/// "BenchDataSet", "event" -/// respectively. -/// These can be changed via dset and fnroot, respectively. -/// The string 'fnroot' defines the location of the files, interpreted as an URL. -/// Examples: -/// fnroot files -/// 'event' ``/event_``_`<#>`.root -/// '/mss/event' /mss/event_``_`<#>`.root -/// 'root://srv//mss/event?remote=1' -/// root://srv//mss/event_``_`<#>`?remote=1.root -/// Default selector is TSelEventGen. Use SetDataGenSel and SetDataGenPar to change it -/// and to pass the list of PARs defining the alternative selector. -/// The argument 'nevt' controls the number of events per file (-1 for the default, -/// which is 30000). -/// Return 0 on success, -1 on error - -Int_t TProofBench::MakeDataSet(const char *dset, Long64_t nevt, const char *fnroot, - Bool_t regenerate) -{ - if (dset && strlen(dset) > 0) fDataSet = dset; - - // Load the selector, if needed - if (!TClass::GetClass(fDataGenSel)) { - // Is it the default selector? - if (fDataGenSel == kPROOF_BenchSelDataGenDef) { - // Load the parfile - TString par = TString::Format("%s/%s%s.par", TROOT::GetEtcDir().Data(), kPROOF_BenchParDir, kPROOF_BenchDataSelPar); - Info("MakeDataSet", "uploading '%s' ...", par.Data()); - if (fProof->UploadPackage(par) != 0) { - Error("MakeDataSet", "problems uploading '%s' - cannot continue", par.Data()); - return -1; - } - Info("MakeDataSet", "enabling '%s' ...", kPROOF_BenchDataSelPar); - if (fProof->EnablePackage(kPROOF_BenchDataSelPar) != 0) { - Error("MakeDataSet", "problems enabling '%s' - cannot continue", kPROOF_BenchDataSelPar); - return -1; - } - } else { - if (fDataGenPar.IsNull()) { - Error("MakeDataSet", "you should load the class '%s' before running the benchmark", fDataGenSel.Data()); - return -1; - } - } - // Load additional PAR files, if any or required by the alternative selector - TString par; - Int_t from = 0; - while (fDataGenPar.Tokenize(par, from, ",")) { - Info("MakeDataSet", "Uploading '%s' ...", par.Data()); - if (fProof->UploadPackage(par) != 0) { - Error("MakeDataSet", "problems uploading '%s' - cannot continue", par.Data()); - return -1; - } - Info("MakeDataSet", "Enabling '%s' ...", par.Data()); - if (fProof->EnablePackage(par) != 0) { - Error("MakeDataSet", "problems enabling '%s' - cannot continue", par.Data()); - return -1; - } - } - // Check - if (!TClass::GetClass(fDataGenSel)) { - Error("MakeDataSet", "failed to load '%s'", fDataGenSel.Data()); - return -1; - } - } - - // For files, 30000 evst each (about 600 MB total) per worker - TString fn, fnr("event"); - Bool_t remote = kFALSE; - if (fnroot && strlen(fnroot) > 0) { - TUrl ur(fnroot, kTRUE); - if (!strcmp(ur.GetProtocol(), "file") && - !gSystem->IsAbsoluteFileName(ur.GetFile())) { - fnr = fnroot; - } else { - fnr = gSystem->BaseName(ur.GetFile()); - // We need to set the basedir - TString bdir = gSystem->GetDirName(fnroot); - bdir += "/"; - fProof->SetParameter("PROOF_BenchmarkBaseDir", bdir.Data()); - // Flag as remote, if so - if (strcmp(ur.GetProtocol(), "file")) remote = kTRUE; - } - } - TProofNodes pn(fProof); - TMap *filesmap = new TMap; - TMap *nodesmap = pn.GetMapOfNodes(); - TIter nxnd(nodesmap); - TList *wli = 0; - TObject *obj = 0; - Int_t kf = 1; - while ((obj = nxnd()) != 0) { - if ((wli = dynamic_cast(nodesmap->GetValue(obj)))) { - THashList *fli = new THashList; - Int_t nf = wli->GetSize() * fNFilesWrk; - TSlaveInfo *wi = (TSlaveInfo *) wli->First(); - while (nf--) { - fn.Form("%s-%s-%d.root", fnr.Data(), wi->GetName(), kf++); - // Add to the node list for generation - fli->Add(new TObjString(fn)); - } - filesmap->Add(new TObjString(obj->GetName()), fli); - } - } - filesmap->Print(); - // Prepare for file generation ... add map in the input list - filesmap->SetName("PROOF_FilesToProcess"); - fProof->AddInput(filesmap); - - // Set parameters for processing - TString oldpack; - if (TProof::GetParameter(fProof->GetInputList(), "PROOF_Packetizer", oldpack) != 0) oldpack = ""; - fProof->SetParameter("PROOF_Packetizer", "TPacketizerFile"); - Int_t oldnotass = -1; - if (TProof::GetParameter(fProof->GetInputList(), "PROOF_ProcessNotAssigned", oldnotass) != 0) oldnotass = -1; - fProof->SetParameter("PROOF_ProcessNotAssigned", (Int_t)0); - - // Process - Long64_t ne = (nevt > 0) ? nevt : 30000; - fProof->SetParameter("PROOF_BenchmarkNEvents", ne); - fProof->SetParameter("PROOF_BenchmarkRegenerate", Int_t(regenerate)); - fProof->Process(fDataGenSel, (Long64_t) 1); - fProof->DeleteParameters("PROOF_BenchmarkNEvents"); - fProof->DeleteParameters("PROOF_BenchmarkRegenerate"); - fProof->DeleteParameters("PROOF_BenchmarkBaseDir"); - - // Restore parameters - if (!oldpack.IsNull()) - fProof->SetParameter("PROOF_Packetizer", oldpack); - else - fProof->DeleteParameters("PROOF_Packetizer"); - if (oldnotass != -1) - fProof->SetParameter("PROOF_ProcessNotAssigned", oldnotass); - else - fProof->DeleteParameters("PROOF_ProcessNotAssigned"); - - // Cleanup - if (fProof->GetInputList()) fProof->GetInputList()->Remove(filesmap); - filesmap->SetOwner(kTRUE); - delete filesmap; - - // The dataset to be registered in the end with proper port - TFileCollection *fc = new TFileCollection("dum", "dum"); - - if (fProof->GetOutputList()) { - fProof->GetOutputList()->Print(); - TIter nxout(fProof->GetOutputList()); - while ((obj = nxout())) { - TList *fli = dynamic_cast(obj); - if (fli && TString(fli->GetName()).BeginsWith("PROOF_FilesGenerated_")) { - TIter nxfg(fli); - TFileInfo *fi = 0; - while ((fi = (TFileInfo *) nxfg())) - fc->Add(fi); - fli->SetOwner(kFALSE); - } - } - // Register the new dataset, overwriting any existing dataset wth the same name - // trusting the existing information - fc->Update(); - if (fc->GetNFiles() > 0) { - if (remote) fc->SetBit(TFileCollection::kRemoteCollection); - if (!(fProof->RegisterDataSet(fDataSet, fc, "OT"))) - Warning("MakeDataSet", "problems registering '%s'", dset); - } else { - Warning("MakeDataSet", "dataset '%s' is empty!", dset); - } - } else { - Warning("MakeDataSet", "PROOF output list is empty!"); - } - - SafeDelete(fc); - - // Get updated information - fc = fProof->GetDataSet(fDataSet); - if (fc) { - fc->Print("F"); - } else { - Warning("MakeDataSet", "dataset '%s' was not generated!", fDataSet.Data()); - } - - SafeDelete(fc); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Copy the files of dataset 'dset' to 'destdir' and create a new dataset named 'dsetdst' -/// decribing them. -/// Return 0 on success, -1 on error - -Int_t TProofBench::CopyDataSet(const char *dset, const char *dsetdst, const char *destdir) -{ - // Make some checks - if (!fProof) { - Error("CopyDataSet", "no PROOF found - cannot continue"); - return -1; - } - if (!dset || (dset && !fProof->ExistsDataSet(dset))) { - Error("CopyDataSet", "dataset '%s' does not exist", dset); - return -1; - } - if (!dsetdst || (dsetdst && fProof->ExistsDataSet(dsetdst))) { - if (isatty(0) != 0 && isatty(1) != 0) { - Printf("Target dataset '%s' exists already:" - " do you want to remove it first?", dsetdst); - const char *a = Getline("[Y,n] "); - Printf("a: %s", a); - if (a[0] == 'Y' || a[0] == 'y' || a[0] == '\n') { - Info("CopyDataSet", "removing dataset '%s' ...", dsetdst); - RemoveDataSet(dsetdst); - } else { - return -1; - } - } else { - Error("CopyDataSet", "destination dataset '%s' does already exist: remove it first", dsetdst); - return -1; - } - } - - // The TFileCollection object for the new dataset - TFileCollection *fc = fProof->GetDataSet(dset); - if (!fc) { - Error("CopyDataSet", "problems retrieving TFileCollection for dataset '%s'", dset); - return -1; - } - TFileCollection *fcn = new TFileCollection(dsetdst, ""); - TString fn; - TFileInfo *fi = 0; - TIter nxfi(fc->GetList()); - while ((fi = (TFileInfo *) nxfi())) { - fn.Form("%s/%s", destdir, gSystem->BaseName(fi->GetCurrentUrl()->GetFile())); - Info("CopyDataSet", "adding info for file '%s'", fn.Data()); - fcn->Add(new TFileInfo(fn)); - } - delete fc; - - // Do it via the dataset handler - if (!fDS) fDS = new TProofBenchDataSet(fProofDS); - if (fDS->CopyFiles(dset, destdir) != 0) { - Error("CopyDataSet", "problems copying files of dataset '%s' to dest dir '%s'", dset, destdir); - delete fcn; - return -1; - } - - // Register the new dataset, overwriting any existing dataset wth the same name - // trusting the existing information - Int_t rc = 0; - if (!(fProof->RegisterDataSet(dsetdst, fcn, "OT"))) { - Error("CopyDataSet", "problems registering and verifying '%s'", dsetdst); - rc = -1; - } - delete fcn; - - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the PROOF instance to be used for dataset operations, like releasing -/// cache ... -/// Use SetProofDS(0) to reset and using the default PROOF - -void TProofBench::SetProofDS(TProof *pds) -{ - if (pds && !pds->IsValid()) { - Error("SetProofDS", "trying to set an invalid PROOF instance"); - return; - } - fProofDS = pds ? pds : fProof; - if (fProofDS) { - SafeDelete(fDS); - fDS = new TProofBenchDataSet(fProofDS); - } - // Done - return; -} - diff --git a/proof/proofbench/src/TProofBenchDataSet.cxx b/proof/proofbench/src/TProofBenchDataSet.cxx deleted file mode 100644 index 3a499dd0f96dd..0000000000000 --- a/proof/proofbench/src/TProofBenchDataSet.cxx +++ /dev/null @@ -1,189 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofBenchDataSet -\ingroup proofbench - -Handle operations on datasets used by ProofBench - -*/ - -#include "RConfigure.h" - -#include "TProofBenchDataSet.h" -#include "TClass.h" -#include "TFileCollection.h" -#include "TList.h" -#include "TMap.h" -#include "TProof.h" -#include "TProofBenchTypes.h" -#include "TROOT.h" - - -ClassImp(TProofBenchDataSet); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TProofBenchDataSet::TProofBenchDataSet(TProof *proof) -{ - fProof = proof ? proof : gProof; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Release memory cache for dataset 'dset' -/// Return 0 on success, -1 on error - -Int_t TProofBenchDataSet::ReleaseCache(const char *dset) -{ - // Clear the cache - TPBHandleDSType type(TPBHandleDSType::kReleaseCache); - if (Handle(dset, &type) != 0) { - Error("ReleaseCache", "problems clearing cache for '%s'", dset); - return -1; - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Physically remove the dataset 'dset', i.e. remove the dataset and the files -/// it describes -/// Return 0 on success, -1 on error - -Int_t TProofBenchDataSet::RemoveFiles(const char *dset) -{ - // Phyically remove the files - TPBHandleDSType type(TPBHandleDSType::kRemoveFiles); - if (Handle(dset, &type) != 0) { - Error("RemoveFiles", "problems removing files for '%s'", dset); - return -1; - } - // Remove the meta information - if (!fProof || (fProof && fProof->RemoveDataSet(dset) != 0)) { - Error("RemoveFiles", "problems removing meta-information for dataset '%s'", dset); - return -1; - } - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Copy the files of dataset 'dset' to another directory -/// Return 0 on success, -1 on error - -Int_t TProofBenchDataSet::CopyFiles(const char *dset, const char *destdir) -{ - // Check input - if (!destdir || (destdir && strlen(destdir) <= 0)) { - Error("CopyFiles", "specifying a destination dir is mandatory!"); - return -1; - } - - // Set the destination dir - if (fProof) fProof->SetParameter("PROOF_Benchmark_DestDir", destdir); - - // Copy the files - TPBHandleDSType type(TPBHandleDSType::kCopyFiles); - if (Handle(dset, &type) != 0) { - Error("CopyFiles", "problems copying files for '%s'", dset); - return -1; - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Physically remove the dataset 'dset', i.e. remove the dataset and the files -/// it describes -/// Return 0 on success, -1 on error - -Int_t TProofBenchDataSet::Handle(const char *dset, TObject *type) -{ - // Check input - if (!dset || (dset && strlen(dset) <= 0)) { - Error("Handle", "a valid dataset name is mandatory"); - return -1; - } - - // The dataset must exist - if (!fProof || (fProof && !fProof->ExistsDataSet(dset))) { - Error("Handle", "dataset '%s' does not exist", dset); - return -1; - } - - // Get the dataset - TFileCollection *fc = fProof->GetDataSet(dset); - if (!fc) { - Error("Handle", "TFileCollection object for dataset '%s' could not be retrieved", dset); - return -1; - } - - // Get information per server - TMap *fcmap = fc->GetFilesPerServer(fProof->GetMaster(), kTRUE); - if (!fcmap) { - Error("Handle", "could not create map with per-server info for dataset '%s'", dset); - return -1; - } - fcmap->Print(); - - // Load the selector, if needed - TString selName("TSelHandleDataSet"); - if (!TClass::GetClass(selName)) { - // Load the parfile - TString par = TString::Format("%s/%s%s.par", TROOT::GetEtcDir().Data(), kPROOF_BenchParDir, kPROOF_BenchDataSelPar); - Info("Handle", "Uploading '%s' ...", par.Data()); - if (fProof->UploadPackage(par) != 0) { - Error("Handle", "problems uploading '%s' - cannot continue", par.Data()); - return -1; - } - Info("Handle", "Enabling '%s' ...", kPROOF_BenchDataSelPar); - if (fProof->EnablePackage(kPROOF_BenchDataSelPar) != 0) { - Error("Handle", "problems enabling '%s' - cannot continue", kPROOF_BenchDataSelPar); - return -1; - } - // Check - if (!TClass::GetClass(selName)) { - Error("Handle", "failed to load '%s'", selName.Data()); - return -1; - } - } - - // Add map in the input list - fcmap->SetName("PROOF_FilesToProcess"); - fProof->AddInput(fcmap); - - // Set parameters for processing - TString oldpack; - if (TProof::GetParameter(fProof->GetInputList(), "PROOF_Packetizer", oldpack) != 0) oldpack = ""; - fProof->SetParameter("PROOF_Packetizer", "TPacketizerFile"); - - // Process - fProof->AddInput(type); - fProof->Process(selName, (Long64_t) fc->GetNFiles()); - if (fProof->GetInputList()) fProof->GetInputList()->Remove(type); - - // Restore parameters - if (!oldpack.IsNull()) - fProof->SetParameter("PROOF_Packetizer", oldpack); - else - fProof->DeleteParameters("PROOF_Packetizer"); - - // Cleanup - fProof->GetInputList()->Remove(fcmap); - delete fcmap; - delete fc; - - // Done - return 0; -} diff --git a/proof/proofbench/src/TProofBenchRun.cxx b/proof/proofbench/src/TProofBenchRun.cxx deleted file mode 100644 index 85649f5e094f2..0000000000000 --- a/proof/proofbench/src/TProofBenchRun.cxx +++ /dev/null @@ -1,46 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofBenchRun -\ingroup proofbench - -Abstract base class for PROOF benchmark runs - -*/ - -#include "TProofBenchRun.h" -#include "TList.h" -#include "TProof.h" - -ClassImp(TProofBenchRun); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor: check PROOF and load selectors PAR - -TProofBenchRun::TProofBenchRun(TProof *proof, const char *sel) : fSelName(sel) -{ - fProof = proof; - if (!fProof){ - fProof = gProof; - } - ResetBit(kInvalidObject); - if (!fProof || (fProof && !fProof->IsValid())) { - SetBit(kInvalidObject); - return; - } -} - -//////////////////////////////////////////////////////////////////////////////// -///destructor - -TProofBenchRun::~TProofBenchRun() -{ -} diff --git a/proof/proofbench/src/TProofBenchRunCPU.cxx b/proof/proofbench/src/TProofBenchRunCPU.cxx deleted file mode 100644 index fff63893e5087..0000000000000 --- a/proof/proofbench/src/TProofBenchRunCPU.cxx +++ /dev/null @@ -1,718 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofBenchRunCPU -\ingroup proofbench - -CPU-intensive PROOF benchmark test generates events and fill 1, 2, or 3-D histograms. -No I/O activity is involved. - -*/ - -#include "RConfigure.h" - -#include "TProofBenchRunCPU.h" -#include "TProofNodes.h" -#include "TProofPerfAnalysis.h" -#include "TFileCollection.h" -#include "TFileInfo.h" -#include "TProof.h" -#include "TString.h" -#include "Riostream.h" -#include "TMap.h" -#include "TEnv.h" -#include "TTree.h" -#include "TLeaf.h" -#include "TCanvas.h" -#include "TROOT.h" -#include "TH2.h" -#include "TF1.h" -#include "TProfile.h" -#include "TLegend.h" -#include "TKey.h" -#include "TRegexp.h" -#include "TPerfStats.h" -#include "TQueryResult.h" -#include "TMath.h" -#include "TStyle.h" -#include "TGraphErrors.h" - -ClassImp(TProofBenchRunCPU); - -//////////////////////////////////////////////////////////////////////////////// -/// Default constructor - -TProofBenchRunCPU::TProofBenchRunCPU(TPBHistType *histtype, Int_t nhists, - TDirectory* dirproofbench, TProof* proof, - TProofNodes* nodes, Long64_t nevents, Int_t ntries, - Int_t start, Int_t stop, Int_t step, Int_t draw, - Int_t debug) - : TProofBenchRun(proof, kPROOF_BenchSelCPUDef), - fHistType(histtype), fNHists(nhists), - fNEvents(nevents), fNTries(ntries), fStart(start), fStop(stop), - fStep(step), fDraw(draw), fDebug(debug), fDirProofBench(dirproofbench), - fNodes(nodes), fListPerfPlots(0), - fCanvas(0), fProfile_perfstat_event(0), fHist_perfstat_event(0), - fProfile_perfstat_evtmax(0), fNorm_perfstat_evtmax(0), - fProfile_queryresult_event(0), fNorm_queryresult_event(0), fProfile_cpu_eff(0), - fProfLegend(0), fNormLegend(0), fName(0) -{ - if (TestBit(kInvalidObject)) { - Error("TProofBenchRunCPU", "problems validating PROOF session or enabling selector PAR"); - return; - } - - fName = "CPU"; - - if (!fNodes) fNodes = new TProofNodes(fProof); - - if (stop == -1) fStop = fNodes->GetNWorkersCluster(); - - fListPerfPlots = new TList; - - gEnv->SetValue("Proof.StatsTrace",1); - gStyle->SetOptStat(0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TProofBenchRunCPU::~TProofBenchRunCPU() -{ - fProof=0; - fDirProofBench=0; - SafeDelete(fListPerfPlots); - SafeDelete(fCanvas); - SafeDelete(fNodes); - SafeDelete(fProfLegend); - SafeDelete(fNormLegend); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build histograms, profiles and graphs needed for this run - -void TProofBenchRunCPU::BuildHistos(Int_t start, Int_t stop, Int_t step, Bool_t nx) -{ - TObject *o = 0; - Int_t quotient = (stop - start) / step; - Int_t ndiv = quotient + 1; - Double_t ns_min = start - step/2.; - Double_t ns_max = quotient*step + start + step/2.; - - fProfLegend = new TLegend(0.1, 0.8, 0.3, 0.9); - fNormLegend = new TLegend(0.7, 0.8, 0.9, 0.9); - - TString axtitle("Active Workers"), namelab(GetName()), sellab(GetSelName()); - if (nx) { - axtitle = "Active Workers/Node"; - namelab.Form("x_%s", GetName()); - } - if (fSelName == kPROOF_BenchSelCPUDef) - sellab.Form("%s_%s", GetSelName(), GetNameStem().Data()); - - TString name, title; - - // Book perfstat profile (max evts) - name.Form("Prof_%s_PS_MaxEvts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s PerfStat Event - %s", namelab.Data(), sellab.Data()); - fProfile_perfstat_evtmax = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_perfstat_evtmax->SetDirectory(fDirProofBench); - fProfile_perfstat_evtmax->GetYaxis()->SetTitle("Events/sec"); - fProfile_perfstat_evtmax->GetXaxis()->SetTitle(axtitle); - fProfile_perfstat_evtmax->SetMarkerStyle(23); - fProfile_perfstat_evtmax->SetMarkerColor(2); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_perfstat_evtmax); - fProfLegend->AddEntry(fProfile_perfstat_evtmax, "Maximum"); - - // Book perfstat profile - name.Form("Prof_%s_PS_Evts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s PerfStat Event - %s", namelab.Data(), sellab.Data()); - fProfile_perfstat_event = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_perfstat_event->SetDirectory(fDirProofBench); - fProfile_perfstat_event->GetYaxis()->SetTitle("Events/sec"); - fProfile_perfstat_event->GetXaxis()->SetTitle(axtitle); - fProfile_perfstat_event->SetMarkerStyle(21); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_perfstat_event); - fProfLegend->AddEntry(fProfile_perfstat_event, "Average"); - - // Book perfstat histogram - name.Form("Hist_%s_PS_Evts_%s", namelab.Data(), sellab.Data()); - title.Form("Histogram %s PerfStat Event - %s", namelab.Data(), sellab.Data()); - fHist_perfstat_event = new TH2D(name, title, ndiv, ns_min, ns_max, 100, 0, 0); - fHist_perfstat_event->SetDirectory(fDirProofBench); - fHist_perfstat_event->GetYaxis()->SetTitle("Events/sec"); - fHist_perfstat_event->GetXaxis()->SetTitle(axtitle); - fHist_perfstat_event->SetMarkerStyle(7); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fHist_perfstat_event); - - // Book normalized perfstat profile (max evts) - name.Form("Norm_%s_PS_MaxEvts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s Normalized PerfStat Event - %s", namelab.Data(), sellab.Data()); - fNorm_perfstat_evtmax = new TProfile(name, title, ndiv, ns_min, ns_max); - fNorm_perfstat_evtmax->SetDirectory(fDirProofBench); - fNorm_perfstat_evtmax->GetYaxis()->SetTitle("Events/sec"); - fNorm_perfstat_evtmax->GetXaxis()->SetTitle(axtitle); - fNorm_perfstat_evtmax->SetMarkerStyle(23); - fNorm_perfstat_evtmax->SetMarkerColor(2); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fNorm_perfstat_evtmax); - fNormLegend->AddEntry(fNorm_perfstat_evtmax, "Maximum"); - - // Book queryresult profile - name.Form("Prof_%s_QR_Evts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s QueryResult Event - %s", namelab.Data(), sellab.Data()); - fProfile_queryresult_event = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_queryresult_event->SetDirectory(fDirProofBench); - fProfile_queryresult_event->GetYaxis()->SetTitle("Events/sec"); - fProfile_queryresult_event->GetXaxis()->SetTitle(axtitle); - fProfile_queryresult_event->SetMarkerStyle(22); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_queryresult_event); - - // Book normalized queryresult profile - name.Form("Norm_%s_QR_Evts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s Normalized QueryResult Event - %s", namelab.Data(), sellab.Data()); - fNorm_queryresult_event = new TProfile(name, title, ndiv, ns_min, ns_max); - fNorm_queryresult_event->SetDirectory(fDirProofBench); - fNorm_queryresult_event->GetYaxis()->SetTitle("Events/sec"); - fNorm_queryresult_event->GetXaxis()->SetTitle(axtitle); - fNorm_queryresult_event->SetMarkerStyle(22); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fNorm_queryresult_event); - fNormLegend->AddEntry(fNorm_queryresult_event, "Average"); - - // Book CPU efficiency profile - name.Form("Prof_%s_CPU_eff_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s CPU efficiency - %s", namelab.Data(), sellab.Data()); - fProfile_cpu_eff = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_cpu_eff->SetDirectory(fDirProofBench); - fProfile_cpu_eff->GetYaxis()->SetTitle("Efficiency"); - fProfile_cpu_eff->GetXaxis()->SetTitle(axtitle); - fProfile_cpu_eff->SetMarkerStyle(22); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_cpu_eff); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Run benchmark -/// Input parameters -/// nevents: Number of events to run per file. When it is -1, use data member fNEvents. -/// start: Start scan with 'start' workers. When it is -1, use data member fStart. -/// When 0, the same number of workers are activated on all nodes. -/// stop: Stop scan at 'stop' workers. When it is -1 , use data member fStop. -/// step: Scan every 'step' workers. When it is -1, use data member fStep. -/// ntries: Number of repetitions. When it is -1, use data member fNTries. -/// debug: debug switch. When it is -1, use data member fDebug. -/// draw: draw switch. When it is -1, use data member fDraw. -/// Returns -/// Nothing - -void TProofBenchRunCPU::Run(Long64_t nevents, Int_t start, Int_t stop, - Int_t step, Int_t ntries, Int_t debug, Int_t draw) -{ - if (!fProof){ - Error("Run", "Proof not set"); - return; - } - - nevents = (nevents == -1) ? fNEvents : nevents; - start = (start == -1) ? fStart : start; - stop = (stop == -1) ? fStop : stop; - step = (step == -1) ? fStep : step; - ntries = (ntries == -1) ? fNTries : ntries; - debug = (debug == -1) ? fDebug : debug; - draw = (draw == -1) ? fDraw : draw; - - Bool_t nx = kFALSE; - if (step == -2){ - nx = kTRUE; - start = fStart; - step = 1; - } - - if (nx){ - Int_t minnworkersanode = fNodes->GetMinWrksPerNode(); - if (stop > minnworkersanode) stop = minnworkersanode; - } - - // Load the selector, if needed - if (!TClass::GetClass(fSelName)) { - // Is it the default selector? - if (fSelName == kPROOF_BenchSelCPUDef) { - // Load the parfile - TString par = TString::Format("%s/%s%s.par", TROOT::GetEtcDir().Data(), kPROOF_BenchParDir, kPROOF_BenchCPUSelPar); - Info("Run", "Uploading '%s' ...", par.Data()); - if (fProof->UploadPackage(par) != 0) { - Error("Run", "problems uploading '%s' - cannot continue", par.Data()); - return; - } - Info("Run", "Enabling '%s' ...", kPROOF_BenchCPUSelPar); - if (fProof->EnablePackage(kPROOF_BenchCPUSelPar) != 0) { - Error("Run", "problems enabling '%s' - cannot continue", kPROOF_BenchCPUSelPar); - return; - } - } else { - if (fParList.IsNull()) { - Error("Run", "you should load the class '%s' before running the benchmark", fSelName.Data()); - return; - } else { - TString par; - Int_t from = 0; - while (fParList.Tokenize(par, from, ",")) { - Info("Run", "Uploading '%s' ...", par.Data()); - if (fProof->UploadPackage(par) != 0) { - Error("Run", "problems uploading '%s' - cannot continue", par.Data()); - return; - } - Info("Run", "Enabling '%s' ...", par.Data()); - if (fProof->EnablePackage(par) != 0) { - Error("Run", "problems enabling '%s' - cannot continue", par.Data()); - return; - } - } - } - } - // Check - if (!TClass::GetClass(fSelName)) { - Error("Run", "failed to load '%s'", fSelName.Data()); - return; - } - } - - // Build histograms, profiles and graphs needed for this run - BuildHistos(start, stop, step, nx); - - // Get pad - if (!fCanvas) fCanvas = new TCanvas("Canvas"); - // Cleanup up the canvas - fCanvas->Clear(); - - // Divide the canvas as many as the number of profiles in the list - fCanvas->Divide(2,1); - - TString perfstats_name = "PROOF_PerfStats"; - - SetParameters(); - - if (nx){ - Info("Run", "Running CPU-bound tests; %d ~ %d active worker(s)/node," - " every %d worker(s)/node.", start, stop, step); - } else { - Info("Run", "Running CPU-bound tests; %d ~ %d active worker(s)," - " every %d worker(s).", start, stop, step); - } - - Int_t npad = 1; //pad number - - Int_t nnodes = fNodes->GetNNodes(); // Number of machines - Int_t ncores = fNodes->GetNCores(); // Number of cores - - Double_t ymi = -1., ymx = -1., emx = -1.; - for (Int_t nactive = start; nactive <= stop; nactive += step) { - - // For CPU effectiveness (ok for lite; should do it properly for standard clusters) - Int_t ncoren = (nactive < ncores) ? nactive : ncores; - - // Actvate the wanted workers - Int_t nw = -1; - if (nx) { - TString workers; - workers.Form("%dx", nactive); - nw = fNodes->ActivateWorkers(workers); - } else { - nw = fNodes->ActivateWorkers(nactive); - } - if (nw < 0){ - Error("Run", "could not activate the requested number of" - " workers/node on the cluster; skipping the test point" - " (%d workers/node)", nactive); - continue; - } - - for (Int_t j = 0; j < ntries; j++) { - - if (nx){ - Info("Run", "Running CPU-bound tests with %d active worker(s)/node;" - " trial %d/%d", nactive, j + 1, ntries); - } else { - Info("Run", "Running CPU-bound tests with %d active worker(s);" - " trial %d/%d", nactive, j + 1, ntries); - } - - Int_t nevents_all=0; - if (nx){ - nevents_all=nevents*nactive*nnodes; - } else { - nevents_all=nevents*nactive; - } - - // Process - fProof->Process(fSelName, nevents_all, fSelOption); - - TList *l = fProof->GetOutputList(); - - // Save perfstats - TTree *t = 0; - if (l) t = dynamic_cast(l->FindObject(perfstats_name.Data())); - if (t) { - - //FillPerfStatPerfPlots(t, profile_perfstat_event, nactive); - FillPerfStatPerfPlots(t, nactive); - - TProofPerfAnalysis pfa(t); - Double_t pf_eventrate = pfa.GetEvtRateAvgMax(); -// if (pf_eventrate > emx) emx = pf_eventrate; - fProfile_perfstat_evtmax->Fill(nactive, pf_eventrate); - fCanvas->cd(npad); - fProfile_perfstat_evtmax->SetMaximum(emx*1.6); - fProfile_perfstat_evtmax->SetMinimum(0.); - fProfile_perfstat_evtmax->Draw("L"); - fProfLegend->Draw(); - gPad->Update(); - // The normalised histos - // Use the first bin to set the Y range for the histo - Double_t nert = nx ? pf_eventrate/nactive/nnodes : pf_eventrate/nactive; - fNorm_perfstat_evtmax->Fill(nactive, nert); - Double_t y1 = fNorm_perfstat_evtmax->GetBinContent(1); - Double_t e1 = fNorm_perfstat_evtmax->GetBinError(1); - Double_t dy = 5 * e1; - if (dy / y1 < 0.2) dy = y1 * 0.2; - if (dy > y1) dy = y1*.999999; - if (ymi < 0.) ymi = y1 - dy; - if (fNorm_perfstat_evtmax->GetBinContent(nactive) < ymi) - ymi = fNorm_perfstat_evtmax->GetBinContent(nactive) / 2.; - if (ymx < 0.) ymx = y1 + dy; - if (fNorm_perfstat_evtmax->GetBinContent(nactive) > ymx) - ymx = fNorm_perfstat_evtmax->GetBinContent(nactive) * 1.5; - fNorm_perfstat_evtmax->SetMaximum(ymx); - fNorm_perfstat_evtmax->SetMinimum(ymi); - fCanvas->cd(npad + 1); - fNorm_perfstat_evtmax->Draw("L"); - fNormLegend->Draw(); - gPad->Update(); - - // Build up new name - TString newname = TString::Format("%s_%s_%dwrks%dthtry", t->GetName(), GetName(), nactive, j); - t->SetName(newname); - - if (debug && fDirProofBench->IsWritable()){ - TDirectory *curdir = gDirectory; - TString dirn = nx ? "RunCPUx" : "RunCPU"; - if (!fDirProofBench->GetDirectory(dirn)) - fDirProofBench->mkdir(dirn, "RunCPU results"); - if (fDirProofBench->cd(dirn)) { - t->SetDirectory(fDirProofBench); - t->Write(); - l->Remove(t); - } else { - Warning("Run", "cannot cd to subdirectory '%s' to store the results!", dirn.Data()); - } - curdir->cd(); - } - - } else { - if (l) - Warning("Run", "%s: tree not found", perfstats_name.Data()); - else - Error("Run", "PROOF output list is empty!"); - } - - // Performance measures from TQueryResult - - const char *drawopt = t ? "LSAME" : "L"; - TQueryResult *queryresult = fProof->GetQueryResult(); - if (queryresult) { - queryresult->Print("F"); - TDatime qr_start = queryresult->GetStartTime(); - TDatime qr_end = queryresult->GetEndTime(); - Float_t qr_proc = queryresult->GetProcTime(); - - Long64_t qr_entries = queryresult->GetEntries(); - - // Calculate event rate - Double_t qr_eventrate = qr_entries / Double_t(qr_proc); - if (qr_eventrate > emx) emx = qr_eventrate; - - // Calculate and fill CPU efficiency - Float_t qr_cpu_eff = -1.; - if (qr_proc > 0.) { - qr_cpu_eff = queryresult->GetUsedCPU() / ncoren / qr_proc ; - fProfile_cpu_eff->Fill(nactive, qr_cpu_eff); - Printf("cpu_eff: %f", qr_cpu_eff); - } - - // Fill and draw - fProfile_queryresult_event->Fill(nactive, qr_eventrate); - fCanvas->cd(npad); - fProfile_queryresult_event->Draw(drawopt); - fProfLegend->Draw(); - gPad->Update(); - // The normalised histo - Double_t nert = nx ? qr_eventrate/nactive/nnodes : qr_eventrate/nactive; - fNorm_queryresult_event->Fill(nactive, nert); - // Use the first bin to set the Y range for the histo - Double_t y1 = fNorm_queryresult_event->GetBinContent(1); - Double_t e1 = fNorm_queryresult_event->GetBinError(1); - Double_t dy = 5 * e1; - if (dy / y1 < 0.2) dy = y1 * 0.2; - if (dy > y1) dy = y1*.999999; - if (ymi < 0.) ymi = y1 - dy; - if (fNorm_queryresult_event->GetBinContent(nactive) < ymi) - ymi = fNorm_queryresult_event->GetBinContent(nactive) / 2.; - if (ymx < 0.) ymx = y1 + dy; - if (fNorm_queryresult_event->GetBinContent(nactive) > ymx) - ymx = fNorm_queryresult_event->GetBinContent(nactive) * 1.5; - fNorm_queryresult_event->SetMaximum(ymx); -// fNorm_queryresult_event->SetMinimum(ymi); - fNorm_queryresult_event->SetMinimum(0.); - fCanvas->cd(npad+1); - fNorm_queryresult_event->Draw(drawopt); - fNormLegend->Draw(); - } else { - Warning("Run", "TQueryResult not found!"); - } - gPad->Update(); - - } // for iterations - } // for number of workers - - // Make the result persistent - fCanvas->cd(npad); - fProfile_queryresult_event->SetMaximum(1.6*emx); - fProfile_queryresult_event->DrawCopy("L"); - fProfile_perfstat_evtmax->DrawCopy("LSAME"); - fProfLegend->Draw(); - fCanvas->cd(npad + 1); - fNorm_queryresult_event->DrawCopy("L"); - fNorm_perfstat_evtmax->DrawCopy("LSAME"); - fNormLegend->Draw(); - gPad->Update(); - - //save performance profiles to file - if (fDirProofBench && fDirProofBench->IsWritable()){ - TDirectory *curdir = gDirectory; - TString dirn = nx ? "RunCPUx" : "RunCPU"; - if (!fDirProofBench->GetDirectory(dirn)) - fDirProofBench->mkdir(dirn, "RunCPU results"); - if (fDirProofBench->cd(dirn)) { - fListPerfPlots->Write(0, kOverwrite); - fListPerfPlots->SetOwner(kFALSE); - fListPerfPlots->Clear(); - } else { - Warning("Run", "cannot cd to subdirectory '%s' to store the results!", dirn.Data()); - } - curdir->cd(); - } -} - -//////////////////////////////////////////////////////////////////////////////// - -void TProofBenchRunCPU::FillPerfStatPerfPlots(TTree* t, Int_t nactive) -{ - // Fill performance profiles using tree 't'(PROOF_PerfStats). - // Input parameters - // t: Proof output tree (PROOF_PerfStat) containing performance statistics. - // profile: Profile to be filled up with information from tree 't'. - // nactive: Number of active workers processed the query. - // Return - // Nothing - - // find perfstat profile - if (!fProfile_perfstat_event){ - Error("FillPerfStatPerfPlots", "no perfstat profile found"); - return; - } - - // find perfstat histogram - if (!fHist_perfstat_event){ - Error("FillPerfStatPerfPlots", "no perfstat histogram found"); - return; - } - - // extract timing information - TPerfEvent pe; - TPerfEvent* pep = &pe; - t->SetBranchAddress("PerfEvents",&pep); - Long64_t entries = t->GetEntries(); - - Double_t event_rate_packet = 0; - - for (Long64_t k=0; kGetEntry(k); - - // Skip information from workers - if (pe.fEvtNode.Contains(".")) continue; - - if (pe.fType == TVirtualPerfStats::kPacket){ - if (pe.fProcTime != 0.0) { - event_rate_packet = pe.fEventsProcessed / pe.fProcTime; - fHist_perfstat_event->Fill(Double_t(nactive), event_rate_packet); - } - } - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show settings - -void TProofBenchRunCPU::Print(Option_t* option) const -{ - Printf("+++ TProofBenchRunCPU +++++++++++++++++++++++++++++++++++++++++"); - Printf("Name = %s", fName.Data()); - if (fProof) fProof->Print(option); - Printf("fHistType = k%s", GetNameStem().Data()); - Printf("fNHists = %d", fNHists); - Printf("fNEvents = %lld", fNEvents); - Printf("fNTries = %d", fNTries); - Printf("fStart = %d", fStart); - Printf("fStop = %d", fStop); - Printf("fStep = %d", fStep); - Printf("fDraw = %d", fDraw); - Printf("fDebug = %d", fDebug); - if (fDirProofBench) - Printf("fDirProofBench = %s", fDirProofBench->GetPath()); - if (fNodes) fNodes->Print(option); - if (fListPerfPlots) fListPerfPlots->Print(option); - if (fCanvas) - Printf("Performance Canvas: Name = %s Title = %s", - fCanvas->GetName(), fCanvas->GetTitle()); - Printf("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw Performance plots - -void TProofBenchRunCPU::DrawPerfPlots() -{ - // Get canvas - if (!fCanvas) fCanvas = new TCanvas("Canvas"); - - fCanvas->Clear(); - - // Divide the canvas as many as the number of profiles in the list - Int_t nprofiles = fListPerfPlots->GetSize(); - if (nprofiles <= 2){ - fCanvas->Divide(1,nprofiles); - } else { - Int_t nside = (Int_t)TMath::Sqrt((Float_t)nprofiles); - nside = (nside*nsideDivide(nside,nside); - } - - Int_t npad=1; - TIter nxt(fListPerfPlots); - TProfile* profile=0; - while ((profile=(TProfile*)(nxt()))){ - fCanvas->cd(npad++); - profile->Draw(); - gPad->Update(); - } - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set histogram type - -void TProofBenchRunCPU::SetHistType(TPBHistType *histtype) -{ - fHistType = histtype; - fName.Form("%sCPU", GetNameStem().Data()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get name for this run - -TString TProofBenchRunCPU::GetNameStem() const -{ - TString namestem("+++undef+++"); - if (fHistType) { - switch (fHistType->GetType()) { - case TPBHistType::kHist1D: - namestem = "Hist1D"; - break; - case TPBHistType::kHist2D: - namestem = "Hist2D"; - break; - case TPBHistType::kHist3D: - namestem = "Hist3D"; - break; - case TPBHistType::kHistAll: - namestem = "HistAll"; - break; - default: - break; - } - } - return namestem; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set parameters - -Int_t TProofBenchRunCPU::SetParameters() -{ - if (!fProof) { - Error("SetParameters", "proof not set; Doing nothing"); - return 1; - } - - if (!fHistType) fHistType = new TPBHistType(TPBHistType::kHist1D); - fProof->AddInput(fHistType); - fProof->SetParameter("PROOF_BenchmarkNHists", fNHists); - fProof->SetParameter("PROOF_BenchmarkDraw", Int_t(fDraw)); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete parameters set for this run - -Int_t TProofBenchRunCPU::DeleteParameters() -{ - if (!fProof){ - Error("DeleteParameters", "proof not set; Doing nothing"); - return 1; - } - if (fProof->GetInputList()) { - TObject *type = fProof->GetInputList()->FindObject("PROOF_Benchmark_HistType"); - if (type) fProof->GetInputList()->Remove(type); - } - fProof->DeleteParameters("PROOF_BenchmarkNHists"); - fProof->DeleteParameters("PROOF_BenchmarkDraw"); - return 0; -} - diff --git a/proof/proofbench/src/TProofBenchRunDataRead.cxx b/proof/proofbench/src/TProofBenchRunDataRead.cxx deleted file mode 100644 index b24f26d9fa730..0000000000000 --- a/proof/proofbench/src/TProofBenchRunDataRead.cxx +++ /dev/null @@ -1,1000 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofBenchRunDataRead -\ingroup proofbench - -I/O-intensive PROOF benchmark test reads in event files distributed -on the cluster. Number of events processed per second and size of -events processed per second are plotted against number of active -workers. Performance rate for unit packets and performance rate -for query are plotted. - -*/ - -#include "RConfigure.h" - -#include "TProofBenchRunDataRead.h" -#include "TProofBenchDataSet.h" -#include "TProofPerfAnalysis.h" -#include "TProofNodes.h" -#include "TFileCollection.h" -#include "TFileInfo.h" -#include "TProof.h" -#include "TString.h" -#include "Riostream.h" -#include "TMap.h" -#include "TTree.h" -#include "TH1.h" -#include "TH2D.h" -#include "TCanvas.h" -#include "TProfile.h" -#include "TKey.h" -#include "TRegexp.h" -#include "TPerfStats.h" -#include "THashList.h" -#include "TSortedList.h" -#include "TVirtualPad.h" -#include "TEnv.h" -#include "TLeaf.h" -#include "TQueryResult.h" -#include "TMath.h" -#include "TStyle.h" -#include "TLegend.h" -#include "TROOT.h" - -ClassImp(TProofBenchRunDataRead); - -//////////////////////////////////////////////////////////////////////////////// - -TProofBenchRunDataRead::TProofBenchRunDataRead(TProofBenchDataSet *pbds, TPBReadType *readtype, - TDirectory* dirproofbench, TProof* proof, - TProofNodes* nodes, Long64_t nevents, Int_t ntries, - Int_t start, Int_t stop, Int_t step, Int_t debug) - : TProofBenchRun(proof, kPROOF_BenchSelDataDef), fProof(proof), - fReadType(readtype), fDS(pbds), - fNEvents(nevents), fNTries(ntries), fStart(start), fStop(stop), fStep(step), - fDebug(debug), fFilesPerWrk(2), fReleaseCache(kTRUE), - fDirProofBench(dirproofbench), fNodes(nodes), - fListPerfPlots(0), fProfile_perfstat_event(0), fHist_perfstat_event(0), - fProfile_perfstat_evtmax(0), fNorm_perfstat_evtmax(0), - fProfile_queryresult_event(0), fNorm_queryresult_event(0), - fProfile_perfstat_IO(0), fHist_perfstat_IO(0), - fProfile_perfstat_IOmax(0), fNorm_perfstat_IOmax(0), - fProfile_queryresult_IO(0), fNorm_queryresult_IO(0), fProfile_cpu_eff(0), - fProfLegend_evt(0), fNormLegend_evt(0), fProfLegend_mb(0), fNormLegend_mb(0), - fCPerfProfiles(0), fName(0) -{ - // Default constructor - - if (!fProof) fProof = gProof; - if (!fDS) fDS = new TProofBenchDataSet(fProof); - - // Set name - fName = "DataRead"; - - if (!fNodes) fNodes = new TProofNodes(fProof); - fNodes->GetMapOfActiveNodes()->Print(); - - if (stop == -1) fStop = fNodes->GetNWorkersCluster(); - - fListPerfPlots = new TList; - - gEnv->SetValue("Proof.StatsTrace",1); - gStyle->SetOptStat(0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TProofBenchRunDataRead::~TProofBenchRunDataRead() -{ - fProof=0; - fDirProofBench=0; - SafeDelete(fListPerfPlots); - if (fCPerfProfiles) delete fCPerfProfiles; - SafeDelete(fProfLegend_evt); - SafeDelete(fProfLegend_mb); - SafeDelete(fNormLegend_evt); - SafeDelete(fNormLegend_mb); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Run benchmark -/// Input parameters -/// dset: Dataset on which to run -/// start: Start scan with 'start' workers. -/// stop: Stop scan at 'stop workers. -/// step: Scan every 'step' workers. -/// ntries: Number of tries. When it is -1, data member fNTries is used. -/// debug: debug switch. -/// Int_t: Ignored -/// Returns -/// Nothing - -void TProofBenchRunDataRead::Run(const char *dset, Int_t start, Int_t stop, - Int_t step, Int_t ntries, Int_t debug, Int_t) -{ - if (!fProof){ - Error("Run", "Proof not set"); - return; - } - if (!dset || (dset && strlen(dset) <= 0)){ - Error("Run", "dataset name not set"); - return; - } - // Check if the dataset exists - if (!fProof->ExistsDataSet(dset)) { - Error("Run", "no such data set found; %s", dset); - return; - } - - start = (start == -1) ? fStart : start; - stop = (stop == -1) ? fStop : stop; - step = (step == -1) ? fStep : step; - ntries = (ntries == -1) ? fNTries : ntries; - debug = (debug == -1) ? fDebug : debug; - - Int_t fDebug_sav = fDebug; - fDebug = debug; - - Bool_t nx = kFALSE; - if (step == -2){ - nx = kTRUE; - step = 1; - } - - if (nx){ - Int_t minnworkersanode = fNodes->GetMinWrksPerNode(); - if (stop > minnworkersanode) stop = minnworkersanode; - } - - // Load the selector, if needed - if (!TClass::GetClass(fSelName) || !fDS->IsProof(fProof)) { - // Is it the default selector? - if (fSelName == kPROOF_BenchSelDataDef) { - // Load the parfile - TString par = TString::Format("%s/%s%s.par", TROOT::GetEtcDir().Data(), kPROOF_BenchParDir, kPROOF_BenchDataSelPar); - Info("Run", "Uploading '%s' ...", par.Data()); - if (fProof->UploadPackage(par) != 0) { - Error("Run", "problems uploading '%s' - cannot continue", par.Data()); - return; - } - Info("Run", "Enabling '%s' ...", kPROOF_BenchDataSelPar); - if (fProof->EnablePackage(kPROOF_BenchDataSelPar) != 0) { - Error("Run", "problems enabling '%s' - cannot continue", kPROOF_BenchDataSelPar); - return; - } - } else { - if (fParList.IsNull()) { - Error("Run", "you should load the class '%s' before running the benchmark", fSelName.Data()); - return; - } else { - TString par; - Int_t from = 0; - while (fParList.Tokenize(par, from, ",")) { - Info("Run", "Uploading '%s' ...", par.Data()); - if (fProof->UploadPackage(par) != 0) { - Error("Run", "problems uploading '%s' - cannot continue", par.Data()); - return; - } - Info("Run", "Enabling '%s' ...", par.Data()); - if (fProof->EnablePackage(par) != 0) { - Error("Run", "problems enabling '%s' - cannot continue", par.Data()); - return; - } - } - } - } - // Check - if (!TClass::GetClass(fSelName)) { - Error("Run", "failed to load '%s'", fSelName.Data()); - return; - } - } - - // Build histograms, profiles and graphs needed for this run - BuildHistos(start, stop, step, nx); - - TString dsname(dset); - TString dsbasename = gSystem->BaseName(dset); - - // Get pad - if (!fCPerfProfiles){ - TString canvasname = TString::Format("Performance Profiles %s", GetName()); - fCPerfProfiles = new TCanvas(canvasname.Data(), canvasname.Data()); - } - - // Cleanup up the canvas - fCPerfProfiles->Clear(); - - fCPerfProfiles->Divide(2,2); - - Info("Run", "Running IO-bound tests on dataset '%s'; %d ~ %d active worker(s)," - " every %d worker(s).", dset, start, stop, step); - - Int_t npad = 1; //pad number - - Int_t nnodes = fNodes->GetNNodes(); // Number of machines - Int_t ncores = fNodes->GetNCores(); // Number of cores - - Bool_t drawpf = kFALSE; - Double_t ymi = -1., ymx = -1., emx =- 1, ymiio = -1., ymxio = -1., mbmx = -1.; - for (Int_t nactive = start; nactive <= stop; nactive += step) { - - // For CPU effectiveness (ok for lite; should do it properly for standard clusters) - Int_t ncoren = (nactive < ncores) ? nactive : ncores; - - // Actvate the wanted workers - Int_t nw = -1; - if (nx) { - TString workers; - workers.Form("%dx", nactive); - nw = fNodes->ActivateWorkers(workers); - } else { - nw = fNodes->ActivateWorkers(nactive); - } - if (nw < 0){ - Error("Run", "could not activate the requested number of" - " workers/node on the cluster; skipping the test point" - " (%d workers/node)", nactive); - continue; - } - - // Prepare the dataset for this run. possibly a subsample of - // the total one - TFileCollection *fc = GetDataSet(dsname, nactive, nx); - if (!fc) { - Error("Run", "could not retrieve dataset '%s'", dsname.Data()); - continue; - } - fc->Print("F"); - TString dsn = TString::Format("%s_%d_%d", dsbasename.Data(), nactive, (Int_t)nx); - fProof->RegisterDataSet(dsn, fc, "OT"); - fProof->ShowDataSet(dsn, "F"); - - for (Int_t j=0; jIsProof(fProof)) ? dsn.Data() : dsname.Data(); - if (fReleaseCache) fDS->ReleaseCache(dsnr); - - DeleteParameters(); - SetParameters(); - - Info("Run", "Processing data set %s with" - " %d active worker(s).", dsn.Data(), nactive); - - TTime starttime = gSystem->Now(); - fProof->Process(dsn, fSelName, fSelOption); - - DeleteParameters(); - - TTime endtime = gSystem->Now(); - - TList *l = fProof->GetOutputList(); - - //save perfstats - TString perfstats_name = "PROOF_PerfStats"; - TTree *t = 0; - if (l) t = dynamic_cast(l->FindObject(perfstats_name.Data())); - if (t) { - drawpf = kTRUE; - TTree* tnew=(TTree*)t->Clone("tnew"); - - FillPerfStatProfiles(tnew, nactive); - - TProofPerfAnalysis pfa(tnew); - Double_t pf_eventrate = pfa.GetEvtRateAvgMax(); - Double_t pf_IOrate = pfa.GetMBRateAvgMax(); - fProfile_perfstat_evtmax->Fill(nactive, pf_eventrate); - fCPerfProfiles->cd(npad); - fProfile_perfstat_evtmax->SetMaximum(1.6*emx); - fProfile_perfstat_evtmax->SetMinimum(0.); - fProfile_perfstat_evtmax->Draw(); - fProfLegend_evt->Draw(); - gPad->Update(); - fProfile_perfstat_IOmax->Fill(nactive, pf_IOrate); - fCPerfProfiles->cd(npad + 2); - fProfile_perfstat_IOmax->SetMaximum(1.6*mbmx); - fProfile_perfstat_IOmax->SetMinimum(0.); - fProfile_perfstat_IOmax->Draw(); - fProfLegend_mb->Draw(); - gPad->Update(); - // The normalised histos - // Use the first bin to set the Y range for the histo - Double_t nert = nx ? pf_eventrate/nactive/nnodes : pf_eventrate/nactive; - fNorm_perfstat_evtmax->Fill(nactive, nert); - Double_t y1 = fNorm_perfstat_evtmax->GetBinContent(1); - Double_t e1 = fNorm_perfstat_evtmax->GetBinError(1); - Double_t dy = 5 * e1; - if (dy / y1 < 0.2) dy = y1 * 0.2; - if (dy > y1) dy = y1*.999999; - if (ymi < 0.) ymi = y1 - dy; - if (fNorm_perfstat_evtmax->GetBinContent(nactive) < ymi) - ymi = fNorm_perfstat_evtmax->GetBinContent(nactive) / 2.; - if (ymx < 0.) ymx = y1 + dy; - if (fNorm_perfstat_evtmax->GetBinContent(nactive) > ymx) - ymx = fNorm_perfstat_evtmax->GetBinContent(nactive) * 1.5; - fNorm_perfstat_evtmax->SetMaximum(ymx); - fNorm_perfstat_evtmax->SetMinimum(ymi); - fCPerfProfiles->cd(npad + 1); - fNorm_perfstat_evtmax->Draw(); - gPad->Update(); - // - Double_t niort = nx ? pf_IOrate/nactive/nnodes : pf_IOrate/nactive; - fNorm_perfstat_IOmax->Fill(nactive, niort); - y1 = fNorm_perfstat_IOmax->GetBinContent(1); - e1 = fNorm_perfstat_IOmax->GetBinError(1); - dy = 5 * e1; - if (dy / y1 < 0.2) dy = y1 * 0.2; - if (dy > y1) dy = y1*.999999; - if (ymiio < 0.) ymiio = y1 - dy; - if (fNorm_perfstat_IOmax->GetBinContent(nactive) < ymiio) - ymiio = fNorm_perfstat_IOmax->GetBinContent(nactive) / 2.; - if (ymxio < 0.) ymxio = y1 + dy; - if (fNorm_perfstat_IOmax->GetBinContent(nactive) > ymxio) - ymxio = fNorm_perfstat_IOmax->GetBinContent(nactive) * 1.5; - fNorm_perfstat_IOmax->SetMaximum(ymxio); - fNorm_perfstat_IOmax->SetMinimum(ymiio); - fCPerfProfiles->cd(npad + 3); - fNorm_perfstat_IOmax->Draw(); - gPad->Update(); - - //change the name - TString newname = TString::Format("%s_%s_%dwrks%dthtry", t->GetName(), GetName(), nactive, j); - tnew->SetName(newname); - - if (debug && fDirProofBench->IsWritable()){ - TDirectory *curdir = gDirectory; - TString dirn = nx ? "RunDataReadx" : "RunDataRead"; - if (!fDirProofBench->GetDirectory(dirn)) - fDirProofBench->mkdir(dirn, "RunDataRead results"); - if (fDirProofBench->cd(dirn)) { - tnew->SetDirectory(fDirProofBench); - tnew->Write(); - l->Remove(tnew); - } else { - Warning("Run", "cannot cd to subdirectory '%s' to store the results!", dirn.Data()); - } - curdir->cd(); - } - } else { - if (l) - Warning("Run", "%s: tree not found", perfstats_name.Data()); - else - Error("Run", "PROOF output list is empty!"); - } - - // - const char *drawopt = t ? "SAME" : ""; - // Performance measures from TQueryResult - TQueryResult *queryresult = fProof->GetQueryResult(); - if (queryresult) { - TDatime qr_start = queryresult->GetStartTime(); - TDatime qr_end = queryresult->GetEndTime(); - Float_t qr_proc = queryresult->GetProcTime(); - Long64_t qr_bytes = queryresult->GetBytes(); - - Long64_t qr_entries = queryresult->GetEntries(); - - // Calculate and fill CPU efficiency - Float_t qr_cpu_eff = -1.; - if (qr_proc > 0.) { - qr_cpu_eff = queryresult->GetUsedCPU() / ncoren / qr_proc ; - fProfile_cpu_eff->Fill(nactive, qr_cpu_eff); - Printf("cpu_eff: %f", qr_cpu_eff); - } - - // Calculate event rate, fill and draw - Double_t qr_eventrate=0; - - qr_eventrate = qr_entries / Double_t(qr_proc); - if (qr_eventrate > emx) emx = qr_eventrate; - - fProfile_queryresult_event->Fill(nactive, qr_eventrate); - fCPerfProfiles->cd(npad); - fProfile_queryresult_event->SetMinimum(0.); - fProfile_queryresult_event->Draw(drawopt); - fProfLegend_evt->Draw(); - gPad->Update(); - - // Calculate IO rate, fill and draw - Double_t qr_IOrate = 0; - - const Double_t Dmegabytes = 1024*1024; - - qr_IOrate = qr_bytes / Dmegabytes / Double_t(qr_proc); - if (qr_IOrate > mbmx) mbmx = qr_IOrate; - - fProfile_queryresult_IO->Fill(nactive, qr_IOrate); - fCPerfProfiles->cd(npad + 2); - fProfile_queryresult_IO->SetMinimum(0.); - fProfile_queryresult_IO->Draw(drawopt); - fProfLegend_mb->Draw(); - gPad->Update(); - - // The normalised histos - // Use the first bin to set the Y range for the histo - Double_t nert = nx ? qr_eventrate/nactive/nnodes : qr_eventrate/nactive; - fNorm_queryresult_event->Fill(nactive, nert); - Double_t y1 = fNorm_queryresult_event->GetBinContent(1); - Double_t e1 = fNorm_queryresult_event->GetBinError(1); - Double_t dy = 5 * e1; - if (dy / y1 < 0.2) dy = y1 * 0.2; - if (dy > y1) dy = y1*.999999; - if (ymi < 0.) ymi = y1 - dy; - if (fNorm_queryresult_event->GetBinContent(nactive) < ymi) - ymi = fNorm_queryresult_event->GetBinContent(nactive) / 2.; - if (ymx < 0.) ymx = y1 + dy; - if (fNorm_queryresult_event->GetBinContent(nactive) > ymx) - ymx = fNorm_queryresult_event->GetBinContent(nactive) * 1.5; -// fNorm_queryresult_event->SetMaximum(ymx); - fNorm_queryresult_event->SetMinimum(ymi); - fCPerfProfiles->cd(npad + 1); - fNorm_queryresult_event->Draw(drawopt); - fNormLegend_evt->Draw(); - gPad->Update(); - // - Double_t niort = nx ? qr_IOrate/nactive/nnodes : qr_IOrate/nactive; - fNorm_queryresult_IO->Fill(nactive, niort); - y1 = fNorm_queryresult_IO->GetBinContent(1); - e1 = fNorm_queryresult_IO->GetBinError(1); - dy = 5 * e1; - if (dy / y1 < 0.2) dy = y1 * 0.2; - if (dy > y1) dy = y1*.999999; - if (ymiio < 0.) ymiio = y1 - dy; - if (fNorm_queryresult_IO->GetBinContent(nactive) < ymiio) - ymiio = fNorm_queryresult_IO->GetBinContent(nactive) / 2.; - if (ymxio < 0.) ymxio = y1 + dy; - if (fNorm_queryresult_IO->GetBinContent(nactive) > ymxio) - ymxio = fNorm_queryresult_IO->GetBinContent(nactive) * 1.5; -// fNorm_queryresult_IO->SetMaximum(ymxio); - fNorm_queryresult_IO->SetMinimum(ymiio); - fCPerfProfiles->cd(npad + 3); - fNorm_queryresult_IO->Draw(drawopt); - fNormLegend_mb->Draw(); - gPad->Update(); - } - fCPerfProfiles->cd(0); - } - // Remove temporary dataset - fProof->RemoveDataSet(dsn); - SafeDelete(fc); - } - - // Make the result persistent - fCPerfProfiles->cd(npad); - fProfile_queryresult_event->SetMaximum(1.6*emx); - fProfile_queryresult_event->DrawCopy(); - if (drawpf) fProfile_perfstat_evtmax->DrawCopy("SAME"); - fProfLegend_evt->Draw(); - fCPerfProfiles->cd(npad + 2); - fProfile_queryresult_IO->SetMaximum(1.6*mbmx); - fProfile_queryresult_IO->DrawCopy(); - if (drawpf) fProfile_perfstat_IOmax->DrawCopy("SAME"); - fProfLegend_mb->Draw(); - fCPerfProfiles->cd(npad + 1); - fNorm_queryresult_event->DrawCopy(); - if (drawpf) fNorm_perfstat_evtmax->DrawCopy("SAME"); - fNormLegend_evt->Draw(); - fCPerfProfiles->cd(npad + 3); - fNorm_queryresult_IO->DrawCopy(); - if (drawpf) fNorm_perfstat_IOmax->DrawCopy("SAME"); - fProfLegend_mb->Draw(); - gPad->Update(); - - //save performance profiles to file - if (fDirProofBench->IsWritable()){ - TDirectory *curdir = gDirectory; - TString dirn = nx ? "RunDataReadx" : "RunDataRead"; - if (!fDirProofBench->GetDirectory(dirn)) - fDirProofBench->mkdir(dirn, "RunDataRead results"); - if (fDirProofBench->cd(dirn)) { - fListPerfPlots->Write(0, kOverwrite); - fListPerfPlots->SetOwner(kFALSE); - fListPerfPlots->Clear(); - } else { - Warning("Run", "cannot cd to subdirectory '%s' to store the results!", dirn.Data()); - } - curdir->cd(); - } - // Restore member data - fDebug = fDebug_sav; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get a subsample of dsname suited to run with 'nact' and option 'nx'. - -TFileCollection *TProofBenchRunDataRead::GetDataSet(const char *dset, - Int_t nact, Bool_t nx) -{ - TFileCollection *fcsub = 0; - - // Dataset must exists - if (!fProof || (fProof && !fProof->ExistsDataSet(dset))) { - Error("GetDataSet", "dataset '%s' does not exist", dset); - return fcsub; - } - - // Get the full collection - TFileCollection *fcref = fProof->GetDataSet(dset); - if (!fcref) { - Error("GetDataSet", "dataset '%s' could not be retrieved", dset); - return fcsub; - } - // Is it remote ? - Bool_t remote = (fcref->TestBit(TFileCollection::kRemoteCollection)) ? kTRUE : kFALSE; - - // Separate info per server - TMap *mpref = fcref->GetFilesPerServer(fProof->GetMaster(), kTRUE); - if (!mpref) { - SafeDelete(fcref); - Error("GetDataSet", "problems classifying info on per-server base"); - return fcsub; - } - mpref->Print(); - - // Get Active node information - TMap *mpnodes = fNodes->GetMapOfActiveNodes(); - if (!mpnodes) { - SafeDelete(fcref); - SafeDelete(mpref); - Error("GetDataSet", "problems getting map of active nodes"); - return fcsub; - } - mpnodes->Print(); - - // Number of files: fFilesPerWrk per active worker - Int_t nf = fNodes->GetNActives() * fFilesPerWrk; - Printf(" number of files needed (ideally): %d (%d per worker)", nf, fFilesPerWrk); - - // The output dataset - fcsub = new TFileCollection(TString::Format("%s_%d_%d", fcref->GetName(), nact, nx), - fcref->GetTitle()); - - // Order reference sub-collections - TIter nxnd(mpnodes); - TObject *key = 0; - TFileInfo *fi = 0; - TFileCollection *xfc = 0; - TList *lswrks = 0; - while ((key = nxnd())) { - TIter nxsrv(mpref); - TObject *ksrv = 0; - while ((ksrv = nxsrv())) { - TUrl urlsrv(ksrv->GetName()); - if (TString(urlsrv.GetHostFQDN()).IsNull()) - urlsrv.SetHost(TUrl(gProof->GetMaster()).GetHostFQDN()); - if (remote || - !strcmp(urlsrv.GetHostFQDN(), TUrl(key->GetName()).GetHostFQDN())) { - if ((xfc = dynamic_cast(mpref->GetValue(ksrv)))) { - if ((lswrks = dynamic_cast(mpnodes->GetValue(key)))) { - Int_t nfnd = fFilesPerWrk * lswrks->GetSize(); - while (nfnd-- && xfc->GetList()->GetSize() > 0) { - if ((fi = (TFileInfo *) xfc->GetList()->First())) { - xfc->GetList()->Remove(fi); - fcsub->Add(fi); - } - } - } else { - Warning("GetDataSet", "could not attach to worker list for node '%s'", - key->GetName()); - } - } else { - Warning("GetDataSet", "could not attach to file collection for server '%s'", - ksrv->GetName()); - } - } - } - } - - // Update counters - fcsub->Update(); - fcsub->Print(); - - // Make sure that the tree name is the one of the original dataset - if (fcref) { - TString dflt(fcref->GetDefaultTreeName()); - if (!dflt.IsNull()) fcsub->SetDefaultTreeName(dflt); - } - - // Cleanup - SafeDelete(fcref); - SafeDelete(mpref); - // Done - return fcsub; -} - -//////////////////////////////////////////////////////////////////////////////// - -void TProofBenchRunDataRead::FillPerfStatProfiles(TTree *t, Int_t nactive) -{ - // Fill performance profiles using tree 't'(PROOF_PerfStats). - // Input parameters - // t: Proof output tree (PROOF_PerfStat) containing performance - // statistics. - // nactive: Number of active workers processed the query. - // Return - // Nothing - - // extract timing information - TPerfEvent pe; - TPerfEvent* pep = &pe; - t->SetBranchAddress("PerfEvents",&pep); - Long64_t entries = t->GetEntries(); - - const Double_t Dmegabytes = 1024.*1024.; - Double_t event_rate_packet = 0; - Double_t IO_rate_packet = 0; - - for (Long64_t k=0; kGetEntry(k); - - // Skip information from workers - if (pe.fEvtNode.Contains(".")) continue; - - if (pe.fType==TVirtualPerfStats::kPacket){ - if (pe.fProcTime != 0.0) { - event_rate_packet = pe.fEventsProcessed / pe.fProcTime; - fHist_perfstat_event->Fill(Double_t(nactive), event_rate_packet); - IO_rate_packet = pe.fBytesRead / Dmegabytes / pe.fProcTime; - fHist_perfstat_IO->Fill(Double_t(nactive), IO_rate_packet); - } - } - } - - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print the content of this object - -void TProofBenchRunDataRead::Print(Option_t* option) const -{ - Printf("Name = %s", fName.Data()); - if (fProof) fProof->Print(option); - Printf("fReadType = %s%s", "k", GetNameStem().Data()); - Printf("fNEvents = %lld", fNEvents); - Printf("fNTries = %d", fNTries); - Printf("fStart = %d", fStart); - Printf("fStop = %d", fStop); - Printf("fStep = %d", fStep); - Printf("fDebug = %d", fDebug); - if (fDirProofBench) - Printf("fDirProofBench = %s", fDirProofBench->GetPath()); - if (fNodes) fNodes->Print(option); - if (fListPerfPlots) fListPerfPlots->Print(option); - - if (fCPerfProfiles) - Printf("Performance Profiles Canvas: Name = %s Title = %s", - fCPerfProfiles->GetName(), fCPerfProfiles->GetTitle()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get canvas - -void TProofBenchRunDataRead::DrawPerfProfiles() -{ - if (!fCPerfProfiles){ - TString canvasname = TString::Format("Performance Profiles %s", GetName()); - fCPerfProfiles = new TCanvas(canvasname.Data(), canvasname.Data()); - } - - fCPerfProfiles->Clear(); - - // Divide the canvas as many as the number of profiles in the list - Int_t nprofiles = fListPerfPlots->GetSize(); - if (nprofiles <= 2){ - fCPerfProfiles->Divide(nprofiles); - } else { - Int_t nside = (Int_t)TMath::Sqrt((Float_t)nprofiles); - nside = (nside*nside < nprofiles) ? nside + 1 : nside; - fCPerfProfiles->Divide(nside,nside); - } - - Int_t npad=1; - TIter nxt(fListPerfPlots); - TProfile* profile=0; - while ((profile=(TProfile*)(nxt()))){ - fCPerfProfiles->cd(npad++); - profile->Draw(); - gPad->Update(); - } - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get name for this run - -TString TProofBenchRunDataRead::GetNameStem()const -{ - TString namestem("+++undef+++"); - if (fReadType) { - switch (fReadType->GetType()) { - case TPBReadType::kReadFull: - namestem="Full"; - break; - case TPBReadType::kReadOpt: - namestem="Opt"; - break; - case TPBReadType::kReadNo: - namestem="No"; - break; - default: - break; - } - } - return namestem; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set parameters - -Int_t TProofBenchRunDataRead::SetParameters() -{ - if (!fProof){ - Error("SetParameters", "Proof not set; Doing nothing"); - return 1; - } - if (!fReadType) fReadType = new TPBReadType(TPBReadType::kReadOpt); - fProof->AddInput(fReadType); - fProof->SetParameter("PROOF_BenchmarkDebug", Int_t(fDebug)); - // For Mac Os X only: do not OS cache the files read - fProof->SetParameter("PROOF_DontCacheFiles", Int_t(1)); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete parameters set for this run - -Int_t TProofBenchRunDataRead::DeleteParameters() -{ - if (!fProof){ - Error("DeleteParameters", "Proof not set; Doing nothing"); - return 1; - } - if (fProof->GetInputList()) { - TObject *type = fProof->GetInputList()->FindObject("PROOF_Benchmark_ReadType"); - if (type) fProof->GetInputList()->Remove(type); - } - fProof->DeleteParameters("PROOF_BenchmarkDebug"); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Build histograms, profiles and graphs needed for this run - -void TProofBenchRunDataRead::BuildHistos(Int_t start, Int_t stop, Int_t step, Bool_t nx) -{ - TObject *o = 0; - Int_t quotient = (stop - start) / step; - Int_t ndiv = quotient + 1; - Double_t ns_min = start - step/2.; - Double_t ns_max = quotient*step + start + step/2.; - - fProfLegend_evt = new TLegend(0.1, 0.8, 0.3, 0.9); - fNormLegend_evt = new TLegend(0.7, 0.8, 0.9, 0.9); - fProfLegend_mb = new TLegend(0.1, 0.8, 0.3, 0.9); - fNormLegend_mb = new TLegend(0.7, 0.8, 0.9, 0.9); - - TString axtitle("Active Workers"), namelab(GetName()), sellab(GetSelName()); - if (nx) { - axtitle = "Active Workers/Node"; - namelab.Form("x_%s", GetName()); - } - if (fSelName == kPROOF_BenchSelDataDef) - sellab.Form("%s_%s", GetSelName(), GetNameStem().Data()); - - TString name, title; - - // Book perfstat profile (max evts) - name.Form("Prof_%s_PS_MaxEvts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s PerfStat Event- %s", namelab.Data(), sellab.Data()); - fProfile_perfstat_evtmax = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_perfstat_evtmax->SetDirectory(fDirProofBench); - fProfile_perfstat_evtmax->GetYaxis()->SetTitle("Events/sec"); - fProfile_perfstat_evtmax->GetXaxis()->SetTitle(axtitle); - fProfile_perfstat_evtmax->SetMarkerStyle(23); - fProfile_perfstat_evtmax->SetMarkerStyle(2); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_perfstat_evtmax); - fProfLegend_evt->AddEntry(fProfile_perfstat_evtmax, "Maximum"); - - // Book perfstat profile (evts) - name.Form("Prof_%s_PS_Evts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s PerfStat Event - %s", namelab.Data(), sellab.Data()); - fProfile_perfstat_event = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_perfstat_event->SetDirectory(fDirProofBench); - fProfile_perfstat_event->GetYaxis()->SetTitle("Events/sec"); - fProfile_perfstat_event->GetXaxis()->SetTitle(axtitle); - fProfile_perfstat_event->SetMarkerStyle(21); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_perfstat_event); - - // Book perfstat histogram (evts) - name.Form("Hist_%s_PS_Evts_%s", namelab.Data(), sellab.Data()); - title.Form("Histogram %s PerfStat Event - %s", namelab.Data(), sellab.Data()); - fHist_perfstat_event = new TH2D(name, title, ndiv, ns_min, ns_max, 100, 0, 0); - fHist_perfstat_event->SetDirectory(fDirProofBench); - fHist_perfstat_event->GetYaxis()->SetTitle("Events/sec"); - fHist_perfstat_event->GetXaxis()->SetTitle(axtitle); - fHist_perfstat_event->SetMarkerStyle(7); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fHist_perfstat_event); - - // Book normalized perfstat profile (max evts) - name.Form("Norm_%s_PF_MaxEvts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s Normalized QueryResult Event - %s", namelab.Data(), sellab.Data()); - fNorm_perfstat_evtmax = new TProfile(name, title, ndiv, ns_min, ns_max); - fNorm_perfstat_evtmax->SetDirectory(fDirProofBench); - fNorm_perfstat_evtmax->GetYaxis()->SetTitle("Events/sec"); - fNorm_perfstat_evtmax->GetXaxis()->SetTitle(axtitle); - fNorm_perfstat_evtmax->SetMarkerStyle(23); - fNorm_perfstat_evtmax->SetMarkerColor(2); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fNorm_perfstat_evtmax); - fNormLegend_evt->AddEntry(fNorm_perfstat_evtmax, "Maximum"); - - // Book queryresult profile (evts) - name.Form("Prof_%s_QR_Evts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s QueryResult Event - %s", namelab.Data(), sellab.Data()); - fProfile_queryresult_event = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_queryresult_event->SetDirectory(fDirProofBench); - fProfile_queryresult_event->GetYaxis()->SetTitle("Events/sec"); - fProfile_queryresult_event->GetXaxis()->SetTitle(axtitle); - fProfile_queryresult_event->SetMarkerStyle(22); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_queryresult_event); - fProfLegend_evt->AddEntry(fProfile_queryresult_event, "Average"); - - // Book normalized queryresult profile (evts) - name.Form("Norm_%s_QR_Evts_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s Normalized QueryResult Event - %s", namelab.Data(), sellab.Data()); - fNorm_queryresult_event = new TProfile(name, title, ndiv, ns_min, ns_max); - fNorm_queryresult_event->SetDirectory(fDirProofBench); - fNorm_queryresult_event->GetYaxis()->SetTitle("Events/sec"); - fNorm_queryresult_event->GetXaxis()->SetTitle(axtitle); - fNorm_queryresult_event->SetMarkerStyle(22); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fNorm_queryresult_event); - fNormLegend_evt->AddEntry(fNorm_queryresult_event, "Average"); - - // Book perfstat profile (mbs) - name.Form("Prof_%s_PS_IO_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s PerfStat I/O %s", namelab.Data(), sellab.Data()); - fProfile_perfstat_IO = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_perfstat_IO->SetDirectory(fDirProofBench); - fProfile_perfstat_IO->GetYaxis()->SetTitle("MB/sec"); - fProfile_perfstat_IO->GetXaxis()->SetTitle(axtitle); - fProfile_perfstat_IO->SetMarkerStyle(21); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_perfstat_IO); - - // Book perfstat histogram (mbs) - name.Form("Hist_%s_PS_IO_%s", namelab.Data(), sellab.Data()); - title.Form("Histogram %s PerfStat I/O - %s", namelab.Data(), sellab.Data()); - fHist_perfstat_IO = new TH2D(name, title, ndiv, ns_min, ns_max, 100, 0, 0); - fHist_perfstat_IO->SetDirectory(fDirProofBench); - fHist_perfstat_IO->GetYaxis()->SetTitle("MB/sec"); - fHist_perfstat_IO->GetXaxis()->SetTitle(axtitle); - fHist_perfstat_IO->SetMarkerStyle(7); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fHist_perfstat_IO); - - // Book perfstat profile (max mbs) - name.Form("Prof_%s_PS_MaxIO_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s PerfStat I/O - %s", namelab.Data(), sellab.Data()); - fProfile_perfstat_IOmax = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_perfstat_IOmax->SetDirectory(fDirProofBench); - fProfile_perfstat_IOmax->GetYaxis()->SetTitle("MB/sec"); - fProfile_perfstat_IOmax->GetXaxis()->SetTitle(axtitle); - fProfile_perfstat_IOmax->SetMarkerStyle(21); - fProfile_perfstat_IOmax->SetMarkerColor(2); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_perfstat_IOmax); - fProfLegend_mb->AddEntry(fProfile_perfstat_IOmax, "Maximum"); - - // Book normalized perfstat profile (max mbs) - name.Form("Norm_%s_PS_MaxIO_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s Normalized PerfStat I/O - %s", namelab.Data(), sellab.Data()); - fNorm_perfstat_IOmax = new TProfile(name, title, ndiv, ns_min, ns_max); - fNorm_perfstat_IOmax->SetDirectory(fDirProofBench); - fNorm_perfstat_IOmax->GetYaxis()->SetTitle("MB/sec"); - fNorm_perfstat_IOmax->GetXaxis()->SetTitle(axtitle); - fNorm_perfstat_IOmax->SetMarkerStyle(23); - fNorm_perfstat_IOmax->SetMarkerColor(2); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fNorm_perfstat_IOmax); - fNormLegend_mb->AddEntry(fNorm_perfstat_IOmax, "Maximum"); - - // Book queryresult profile (mbs) - name.Form("Prof_%s_QR_IO_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s QueryResult I/O - %s", namelab.Data(), sellab.Data()); - fProfile_queryresult_IO = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_queryresult_IO->SetDirectory(fDirProofBench); - fProfile_queryresult_IO->GetYaxis()->SetTitle("MB/sec"); - fProfile_queryresult_IO->GetXaxis()->SetTitle(axtitle); - fProfile_queryresult_IO->SetMarkerStyle(22); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_queryresult_IO); - fProfLegend_mb->AddEntry(fProfile_queryresult_IO, "Average"); - - // Book normalized queryresult profile (mbs) - name.Form("Norm_%s_QR_IO_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s Normalized QueryResult I/O - %s", namelab.Data(), sellab.Data()); - fNorm_queryresult_IO = new TProfile(name, title, ndiv, ns_min, ns_max); - fNorm_queryresult_IO->SetDirectory(fDirProofBench); - fNorm_queryresult_IO->GetYaxis()->SetTitle("MB/sec"); - fNorm_queryresult_IO->GetXaxis()->SetTitle(axtitle); - fNorm_queryresult_IO->SetMarkerStyle(22); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fNorm_queryresult_IO); - fNormLegend_mb->AddEntry(fNorm_queryresult_IO, "Average"); - - // Book CPU efficiency profile - name.Form("Prof_%s_CPU_eff_%s", namelab.Data(), sellab.Data()); - title.Form("Profile %s CPU efficiency - %s", namelab.Data(), sellab.Data()); - fProfile_cpu_eff = new TProfile(name, title, ndiv, ns_min, ns_max); - fProfile_cpu_eff->SetDirectory(fDirProofBench); - fProfile_cpu_eff->GetYaxis()->SetTitle("Efficiency"); - fProfile_cpu_eff->GetXaxis()->SetTitle(axtitle); - fProfile_cpu_eff->SetMarkerStyle(22); - if ((o = fListPerfPlots->FindObject(name))) { - fListPerfPlots->Remove(o); - delete o; - } - fListPerfPlots->Add(fProfile_cpu_eff); -} diff --git a/proof/proofbench/src/TProofNodes.cxx b/proof/proofbench/src/TProofNodes.cxx deleted file mode 100644 index d7a16cf70c41c..0000000000000 --- a/proof/proofbench/src/TProofNodes.cxx +++ /dev/null @@ -1,283 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofNodes -\ingroup proofbench - -PROOF worker node information - -*/ - -#include "TProofNodes.h" -#include "TProof.h" -#include "TList.h" -#include "TMap.h" -#include "TObjString.h" - -ClassImp(TProofNodes); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TProofNodes::TProofNodes(TProof* proof) - : fProof(proof), fNodes(0), fActiveNodes(0), - fMaxWrksNode(-1), fMinWrksNode(-1), - fNNodes(0), fNWrks(0), fNActiveWrks(0), fNCores(0) -{ - Build(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TProofNodes::~TProofNodes() -{ - if (fNodes) { - fNodes->SetOwner(kTRUE); - SafeDelete(fNodes); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Desctiption: Build the node list, which is a list of nodes whose members -/// in turn are lists of workers on the node. -/// Input: Nothing -/// Return: Nothing - -void TProofNodes::Build() -{ - if (!fProof || !fProof->IsValid()) { - Warning("Build", "the PROOF instance is undefined or invalid! Cannot continue"); - return; - } - - if (fNodes){ - fNodes->SetOwner(kTRUE); - SafeDelete(fNodes); - } - fNodes = new TMap; - fNodes->SetOwner(kTRUE); - - TList *slaves = fProof->GetListOfSlaveInfos(); - TIter nxtslave(slaves); - TSlaveInfo *si = 0; - TList *node = 0; - TPair *pair = 0; - while ((si = (TSlaveInfo *)(nxtslave()))) { - TSlaveInfo *si_copy = (TSlaveInfo *)(si->Clone()); - if (!(pair = (TPair *) fNodes->FindObject(si->GetName()))) { - node = new TList; - //si's are owned by the member fSlaveInfo of fProof - node->SetOwner(kTRUE); - node->SetName(si_copy->GetName()); - node->Add(si_copy); - fNodes->Add(new TObjString(si->GetName()), node); - } else { - node = (TList *) pair->Value(); - node->Add(si_copy); - } - } - // Update counters and created active nodes map - if (fActiveNodes){ - fActiveNodes->SetOwner(kTRUE); - SafeDelete(fActiveNodes); - } - fActiveNodes = new TMap; - fActiveNodes->SetOwner(kTRUE); - TList *actnode = 0; - fMaxWrksNode = -1; - fMinWrksNode = -1; - fNNodes = 0; - fNWrks = 0; - fNActiveWrks = 0; - TIter nxk(fNodes); - TObject *key = 0; - while ((key = nxk()) != 0) { - node = dynamic_cast(fNodes->GetValue(key)); - if (node) { - fNNodes++; - // Number of cores - si = (TSlaveInfo *) node->First(); - fNCores += si->fSysInfo.fCpus; - // Number of workers - fNWrks += node->GetSize(); - if (fMinWrksNode == -1 || (node->GetSize() < fMinWrksNode)) { - fMinWrksNode = node->GetSize(); - } - if (fMaxWrksNode == -1 || (node->GetSize() > fMaxWrksNode)) { - fMaxWrksNode = node->GetSize(); - } - TIter nxw(node); - while ((si = (TSlaveInfo *) nxw())) { - if (si->fStatus == TSlaveInfo::kActive) { - fNActiveWrks++; - TSlaveInfo *si_copy = (TSlaveInfo *)(si->Clone()); - actnode = dynamic_cast(fActiveNodes->GetValue(key)); - if (actnode) { - actnode->Add(si_copy); - } else { - actnode = new TList; - actnode->SetOwner(kTRUE); - actnode->SetName(si_copy->GetName()); - actnode->Add(si_copy); - fActiveNodes->Add(new TObjString(si->GetName()), actnode); - } - } - } - } else { - Warning("Build", "could not get list for node '%s'", key->GetName()); - } - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Description: Activate 'nwrks' workers; calls TProof::SetParallel and -/// rebuild the internal lists -/// Input: number of workers -/// Return: 0 is successful, <0 otherwise. - -Int_t TProofNodes::ActivateWorkers(Int_t nwrks) -{ - Int_t nw = fProof->SetParallel(nwrks); - if (nw > 0) { - if (nw != nwrks) - Warning("ActivateWorkers", "requested %d got %d", nwrks, nw); - Build(); - } - return nw; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Description: Activate the same number of workers on all nodes. -/// Input: workers: string of the form "nx" where non-negative integer n -/// is the number of worker on each node to be activated. -/// Return: The number of active workers per node when the operation is -/// successful. -/// <0 otherwise. - -Int_t TProofNodes::ActivateWorkers(const char *workers) -{ - TString toactivate; - TString todeactivate; - - // The TProof::ActivateWorker/TProof::DeactivateWorker functions were fixed / - // improved starting with protocol version 33 - Bool_t protocol33 = kTRUE; - if (fProof->GetRemoteProtocol() < 33 || fProof->GetClientProtocol() < 33) { - protocol33 = kFALSE; - // This resets the lists to avoid the problem fixed in protocol 33 - fProof->SetParallel(0); - } - - //Make sure worker list is up-to-date - Build(); - - TString sworkers = TString(workers).Strip(TString::kTrailing, 'x'); - if (!sworkers.IsDigit()) { - Error("ActivateWorkers", "wrongly formatted argument: %s - cannot continue", workers); - return -1; - } - Int_t nworkersnode = sworkers.Atoi(); - Int_t ret = nworkersnode; - TSlaveInfo *si = 0; - TList *node = 0; - TObject *key = 0; - - TIter nxk(fNodes); - while ((key = nxk()) != 0) { - if ((node = dynamic_cast(fNodes->GetValue(key)))) { - TIter nxtworker(node); - Int_t nactiveworkers = 0; - while ((si = (TSlaveInfo *)(nxtworker()))) { - if (nactiveworkers < nworkersnode) { - if (si->fStatus == TSlaveInfo::kNotActive) { - if (protocol33) { - toactivate += TString::Format("%s,", si->GetOrdinal()); - } else { - fProof->ActivateWorker(si->GetOrdinal()); - } - } - nactiveworkers++; - } else { - if (si->fStatus == TSlaveInfo::kActive) { - if (protocol33) { - todeactivate += TString::Format("%s,", si->GetOrdinal()); - } else { - fProof->DeactivateWorker(si->GetOrdinal()); - } - } - } - } - } else { - Warning("ActivateWorkers", "could not get list for node '%s'", key->GetName()); - } - } - - if (!todeactivate.IsNull()) { - todeactivate.Remove(TString::kTrailing, ','); - if (fProof->DeactivateWorker(todeactivate) < 0) ret = -1; - } - if (!toactivate.IsNull()) { - toactivate.Remove(TString::kTrailing, ','); - if (fProof->ActivateWorker(toactivate) < 0) ret = -1; - } - if (ret < 0) { - Warning("ActivateWorkers", "could not get the requested number of workers per node (%d)", - nworkersnode); - return ret; - } - - // Rebuild - Build(); - - // Build() destroyes fNodes so we need to re-create the iterator, resetting is not enough ... - TIter nxkn(fNodes); - while ((key = nxkn()) != 0) { - if ((node = dynamic_cast(fNodes->GetValue(key)))) { - TIter nxtworker(node); - Int_t nactiveworkers = 0; - while ((si = (TSlaveInfo *)(nxtworker()))) { - if (si->fStatus == TSlaveInfo::kActive) nactiveworkers++; - } - if (nactiveworkers != nworkersnode) { - Warning("ActivateWorkers", "only %d (out of %d requested) workers " - "were activated on node %s", - nactiveworkers, nworkersnode, node->GetName()); - ret = -1; - } - } else { - Warning("ActivateWorkers", "could not get list for node '%s'", key->GetName()); - } - } - - // Done - return ret; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Description: Print node information. - -void TProofNodes::Print(Option_t* option) const -{ - TIter nxk(fNodes); - TObject *key = 0; - while ((key = nxk()) != 0) { - TList *node = dynamic_cast(fNodes->GetValue(key)); - if (node) { - node->Print(option); - } else { - Warning("Print", "could not get list for node '%s'", key->GetName()); - } - } -} diff --git a/proof/proofbench/src/TProofPerfAnalysis.cxx b/proof/proofbench/src/TProofPerfAnalysis.cxx deleted file mode 100644 index 61cc5b9b9c3b9..0000000000000 --- a/proof/proofbench/src/TProofPerfAnalysis.cxx +++ /dev/null @@ -1,1864 +0,0 @@ -// @(#)root/proofx:$Id$ -// Author: G.Ganis Nov 2011 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -/** \class TProofPerfAnalysis -\ingroup proofbench - -Set of tools to analyse the performance tree - -*/ - -#include - -#include "TProofPerfAnalysis.h" -#include "TCanvas.h" -#include "TFile.h" -#include "TGraph.h" -#include "TH1F.h" -#include "TH2F.h" -#include "THashList.h" -#include "TKey.h" -#include "TList.h" -#include "TSortedList.h" -#include "TObjString.h" -#include "TPerfStats.h" -#include "TRegexp.h" -#include "TStyle.h" -#include "TSystem.h" -#include "TTree.h" -#include "TMath.h" - -// -// Auxilliary internal classes -//////////////////////////////////////////////////////////////////////////////// - -class TProofPerfAnalysis::TWrkInfo : public TNamed { -public: - TWrkInfo(const char *ord, const char *name) : - TNamed(ord, name), fPackets(0), fRemotePackets(0), fEventsProcessed(0), - fBytesRead(0), fLatency(0), fProcTime(0), fCpuTime(0), fStart(0), fStop(-1), - fRateT(0), fRateRemoteT(0), fMBRateT(0), fMBRateRemoteT(0), fLatencyT(0) { } - ~TWrkInfo() override { SafeDelete(fRateT); SafeDelete(fRateRemoteT); - SafeDelete(fMBRateT); SafeDelete(fMBRateRemoteT); - SafeDelete(fLatencyT); } - - Int_t fPackets; // Number of packets processed - Int_t fRemotePackets; // Number of processed packet from non-local files - Long64_t fEventsProcessed; // Tot events processed - Long64_t fBytesRead; // Tot bytes read - Double_t fLatency; // Tot latency - Double_t fProcTime; // Tot proc time - Double_t fCpuTime; // Tot CPU time - - Float_t fStart; // Start time - Float_t fStop; // Stop time - - TGraph *fRateT; // Event processing rate vs time - TGraph *fRateRemoteT; // Event processing rate of remote packets vs time - TGraph *fMBRateT; // Byte processing rate vs time - TGraph *fMBRateRemoteT; // Byte processing rate of remote packets vs time - TGraph *fLatencyT; // Packet latency vs time - - Double_t AvgRate() { if (fProcTime > 0) return (fEventsProcessed/fProcTime); return -1.; } - Double_t AvgIO() { if (fProcTime > 0) return (fBytesRead/fProcTime); return -1.; } - - void Print(Option_t * = "") const override { - Printf(" +++ TWrkInfo ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ "); - Printf(" +++ Worker: %s, %s", GetName(), GetTitle()); - Printf(" +++ Activity interval: %f -> %f", fStart, fStop); - Printf(" +++ Amounts processed: %d packets (%d remote), %lld evts, %lld bytes", - fPackets, fRemotePackets, fEventsProcessed, fBytesRead); - if (fProcTime) { - Printf(" +++ Processing time: %f s (CPU: %f s)", fProcTime, fCpuTime); - Printf(" +++ Averages: %f evts/s, %f MB/s", (Double_t)fEventsProcessed / fProcTime, (Double_t)fBytesRead /1024./1024./fProcTime); - } - Printf(" +++ Total latency: %f", fLatency); - Printf(" +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ "); - } - - Int_t Compare(const TObject *o) const override { TWrkInfo *wi = (TWrkInfo *)o; - if (wi) { - if (fStop < wi->fStop) { - return -1; - } else if (fStop == wi->fStop) { - return 0; - } - } - return 1; } -}; - - -class TProofPerfAnalysis::TPackInfo : public TNamed { -public: - TPackInfo(const char *ord, const char *host) : TNamed(ord, host), fStart(0), fStop(-1), fSize(0), fMBRate(0.) { } - TPackInfo(const char *ord, const char *host, Float_t start, Float_t stop, Long64_t sz, Double_t mbr) - : TNamed(ord, host), fStart(start), fStop(stop), fSize(sz), fMBRate(mbr) { } - Float_t fStart; // When the packet has been assigned - Float_t fStop; // When the packet has been finished - Long64_t fSize; // Packet size - Double_t fMBRate; // Processing rate MB/s - void Print(Option_t *opt= "") const override { - if (!strcmp(opt, "S")) { - Printf(" \t%10lld evts, \t%12.2f MB/s, \t%12.3f -> %12.3f s", fSize, fMBRate, fStart, fStop); - } else { - Printf(" %s:\t%s \t%10lld evts, \t%12.2f MB/s, \t%12.3f -> %12.3f s", GetTitle(), GetName(), fSize, fMBRate, fStart, fStop); - } - } -}; - -class TProofPerfAnalysis::TWrkInfoFile : public TNamed { -public: - TWrkInfoFile(const char *ord, const char *name) : TNamed(ord, name) { } - ~TWrkInfoFile() override {fPackets.SetOwner(kFALSE); fPackets.Clear("nodelete");} - TList fPackets; // Packest from this file processed by this worker - void Print(Option_t *opt= "") const override { - if (!strcmp(opt, "R")) { - Printf(" Worker: %s,\tpacket(s): %d", GetName(), fPackets.GetSize()); - } else { - Printf(" Worker: %s,\t%d packet(s) from file: %s", GetName(), fPackets.GetSize(), GetTitle()); - } - TIter nxp(&fPackets); - TObject *o = 0; - while ((o = nxp())) { o->Print("S"); } - } -}; - -class TProofPerfAnalysis::TWrkEntry : public TObject { -public: - TWrkEntry(Double_t xx, Double_t er, Double_t mbr, Double_t pt) : fXx(xx), fEvtRate(er), fMBRate(mbr), fProcTime(pt) { } - Double_t fXx; // Bin center - Double_t fEvtRate; // Event processing rate from this worker for this packet - Double_t fMBRate; // I/O processing rate from this worker for this packet - Double_t fProcTime; // Processing time - void Print(Option_t * = "") const override { Printf("%.4f \t%.3f evt/s \t%.3f MB/s \t%.3f s ", fXx, fEvtRate, fMBRate, fProcTime); } -}; - -//////////////////////////////////////////////////////////////////////////////// - -class TProofPerfAnalysis::TFileInfo : public TNamed { -public: - TFileInfo(const char *name, const char *srv) : - TNamed(name, srv), fPackets(0), fRPackets(0), fStart(0), fStop(-1), - fSizeAvg(0), fSizeMax(-1.), fSizeMin(-1.), - fMBRateAvg(0), fMBRateMax(-1.), fMBRateMin(-1.), fSizeP(0), - fRateP(0), fRatePRemote(0), fMBRateP(0), fMBRatePRemote(0) { } - ~TFileInfo() override {SafeDelete(fSizeP); - SafeDelete(fRateP); SafeDelete(fRatePRemote); - SafeDelete(fMBRateP); SafeDelete(fMBRatePRemote); - fPackList.SetOwner(kTRUE); fPackList.Clear(); - fWrkList.SetOwner(kTRUE); fWrkList.Clear(); - fRWrkList.SetOwner(kTRUE); fRWrkList.Clear();} - - Int_t fPackets; // Number of packets from this file - Int_t fRPackets; // Number of different remote workers processing this file - - TList fPackList; // List of packet info - TList fWrkList; // List of worker names processing this packet - TList fRWrkList; // List of remote worker names processing this packet - - Float_t fStart; // When the first packet has been assigned - Float_t fStop; // When the last packet has been finished - - Long64_t fSizeAvg; // Average Packet size - Long64_t fSizeMax; // Max packet size - Long64_t fSizeMin; // Min packet size - - Double_t fMBRateAvg; // Average MB rate - Double_t fMBRateMax; // Max MB rate - Double_t fMBRateMin; // Min MB rate - - TGraph *fSizeP; // Packet size vs packet (all) - TGraph *fRateP; // Event processing rate vs packet (all) - TGraph *fRatePRemote; // Event processing rate vs packet (remote workers) - TGraph *fMBRateP; // Byte processing rate vs packet (all) - TGraph *fMBRatePRemote; // Byte processing rate vs packet (remote workers) - - void Print(Option_t *opt = "") const override { - Printf(" +++ TFileInfo ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ "); - Printf(" +++ Server: %s", GetTitle()); - Printf(" +++ File: %s", GetName()); - Printf(" +++ Processing interval: %f -> %f", fStart, fStop); - Printf(" +++ Packets: %d (%d remote)", fPackets, fRPackets); - Printf(" +++ Processing wrks: %d (%d remote)", fWrkList.GetSize(), fRWrkList.GetSize()); - if (!strcmp(opt, "P")) fPackList.Print(); - if (!strcmp(opt, "WP")) fWrkList.Print("R"); - if (fPackets > 0) { - Printf(" +++ MB rates: %f MB/s (avg), %f MB/s (min), %f MB/s (max)", - fMBRateAvg / fPackets, fMBRateMin, fMBRateMax); - Printf(" +++ Sizes: %lld (avg), %lld (min), %lld (max)", - fSizeAvg / fPackets, fSizeMin, fSizeMax); - } - Printf(" +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ "); - } - - Int_t Compare(const TObject *o) const override { TFileInfo *wi = (TFileInfo *)o; - if (wi) { - if (fStop < wi->fStop) { - return -1; - } else if (fStop == wi->fStop) { - return 0; - } - } - return 1; } -}; - -Bool_t TProofPerfAnalysis::fgDebug = kTRUE; -//////////////////////////////////////////////////////////////////////////////// -/// Constructor: open the file and attach to the tree - -TProofPerfAnalysis::TProofPerfAnalysis(const char *perffile, - const char *title, const char *treename) - : TNamed(perffile, title), fFile(0), fTreeName(treename), fTree(0), - fInitTime(-1.), fMergeTime(-1.), fMaxTime(-1.), - fEvents(0), fPackets(0), - fEvtRateMax(-1.), fMBRateMax(-1.), fLatencyMax(-1.), - fEvtRate(0), fEvtRateRun(0), fMBRate(0), fMBRateRun(0), - fEvtRateAvgMax(-1.), fMBRateAvgMax(-1.), - fEvtRateAvg(-1.), fMBRateAvg(0), - fFileResult(""), fSaveResult(kFALSE), - fDebug(0) -{ - // Use default title, if not specified - if (!title) SetTitle("PROOF Performance Analysis"); - - fTree = 0; - fFile = TFile::Open(perffile); - if (!fFile || (fFile && fFile->IsZombie())) { - SafeDelete(fFile); - Error("TProofPerfAnalysis", "problems opening file '%s'", - perffile ? perffile : ""); - SetBit(TObject::kInvalidObject); - return; - } - - // Set the subdirectory name, if any - if (fTreeName.Contains("/")) { - fDirName = gSystem->GetDirName(fTreeName); - fTreeName = gSystem->BaseName(fTreeName); - } - - // Adjust the name, if requested - if (fTreeName.BeginsWith("+")) - fTreeName.Replace(0, 1, "PROOF_PerfStats"); - - // Point to the right TDirectory - TDirectory *dir = fFile; - if (!fDirName.IsNull()) { - if (!(dir = dynamic_cast(fFile->Get(fDirName)))) { - Error("TProofPerfAnalysis", "directory '%s' not found or not loadable", fDirName.Data()); - fFile->Close(); - SafeDelete(fFile); - SetBit(TObject::kInvalidObject); - return; - } - } - - // Load the performance tree - LoadTree(dir); - if (!fTree) { - Error("TProofPerfAnalysis", "tree '%s' not found or not loadable", fTreeName.Data()); - fFile->Close(); - SafeDelete(fFile); - SetBit(TObject::kInvalidObject); - return; - } - if (fgDebug) - Printf(" +++ TTree '%s' has %lld entries", fTreeName.Data(), fTree->GetEntries()); - - // Init worker information - FillWrkInfo(); - - // Init file information - FillFileInfo(); - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor: open the file and attach to the tree - -TProofPerfAnalysis::TProofPerfAnalysis(TTree *tree, const char *title) - : TNamed("", title), fFile(0), fTree(0), - fInitTime(-1.), fMergeTime(-1.), fMaxTime(-1.), - fEvents(0), fPackets(0), - fEvtRateMax(-1.), fMBRateMax(-1.), fLatencyMax(-1.), - fEvtRate(0), fEvtRateRun(0), fMBRate(0), fMBRateRun(0), - fEvtRateAvgMax(-1.), fMBRateAvgMax(-1.), - fEvtRateAvg(-1.), fMBRateAvg(0), - fDebug(0) -{ - // The tree must be defined - if (!tree) { - SetBit(TObject::kInvalidObject); - return; - } - - // Use default title, if not specified - if (!title) SetTitle("PROOF Performance Analysis"); - - fTree = tree; - fTreeName = fTree->GetName(); - SetName(TString::Format("heap_%s", fTreeName.Data())); - - // Adjust the name, if requested - if (fTreeName.BeginsWith("+")) - fTreeName.Replace(0, 1, "PROOF_PerfStats"); - - if (fgDebug) - Printf(" +++ TTree '%s' has %lld entries", fTreeName.Data(), fTree->GetEntries()); - - // Init worker information - FillWrkInfo(); - - // Init file information - FillFileInfo(); - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor: detach the tree and close the file - -TProofPerfAnalysis::~TProofPerfAnalysis() -{ - SafeDelete(fEvents); - SafeDelete(fPackets); - if (fFile) fFile->Close(); - SafeDelete(fFile); -} - -//////////////////////////////////////////////////////////////////////////////// -/// If defined, add '- ``' to the canvas title 't' - -TString TProofPerfAnalysis::GetCanvasTitle(const char *t) -{ - if (fTitle.IsNull()) return TString(t); - - TString newt; - if (t && strlen(t) > 0) { - newt.Form("%s - %s", t, GetTitle()); - } else { - newt = GetTitle(); - } - // Done - return newt; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load tree fTreeName from directory 'dir'. If not found, look for the -/// first TTree in the directory (and sub-directories) with the name containing -/// fTreeName. -/// The tree pointer is saved in fTree. - -void TProofPerfAnalysis::LoadTree(TDirectory *dir) -{ - fTree = 0; - if (!dir) return; - - // Try first the full name in the top directory - if ((fTree = dynamic_cast(dir->Get(fTreeName)))) return; - - TRegexp re(fTreeName); - // Now look inside: iter on the list of keys first - TIter nxk(dir->GetListOfKeys()); - TKey *k = 0; - while ((k = (TKey *) nxk())) { - if (!strcmp(k->GetClassName(), "TDirectoryFile")) { - TDirectory *kdir = (TDirectory *) dir->Get(k->GetName()); - LoadTree(kdir); - if (fTree) return; - } else if (!strcmp(k->GetClassName(), "TTree")) { - TString tn(k->GetName()); - if (tn.Index(re) != kNPOS) { - if ((fTree = dynamic_cast(dir->Get(tn)))) { - fTreeName = tn; - if (fgDebug) Printf(" +++ Found and loaded TTree '%s'", tn.Data()); - return; - } - } - } - } - - // Nothing found - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Analyse the file distribution. If writedet, underling details are -/// written out to a text file. - -void TProofPerfAnalysis::FileDist(Bool_t writedet) -{ - if (!IsValid()) { - Error("FileDist","not a valid instance - do nothing"); - return; - } - - // Fill file info - TList *wrkList = new TList; - TList *srvList = new TList; - GetWrkFileList(wrkList, srvList); - Info("FileDist", "%d workers were active during this query", wrkList->GetSize()); - Info("FileDist", "%d servers were active during this query", srvList->GetSize()); - - // Fill the worker-data server mapping - TIter nxs(srvList); - TIter nxw(wrkList); - TNamed *sn = 0, *wn = 0; - while ((sn = (TNamed *)nxs())) { - nxw.Reset(); - while ((wn = (TNamed *) nxw())) { - if (!strcmp(TUrl(sn->GetName()).GetHostFQDN(), wn->GetTitle())) { - sn->SetTitle(wn->GetName()); - } - } - } - - // Reorder the lists following the title - TList *nwl = new TList; - TList *nsl = new TList; - nxw.Reset(); - while ((wn = (TNamed *) nxw())) { - TIter nnxw(nwl); - TNamed *nwn = 0; - while ((nwn = (TNamed *) nnxw())) { - if (CompareOrd(wn->GetName(), nwn->GetName()) < 0) { - nwl->AddBefore(nwn, wn); - break; - } - } - if (!nwn) nwl->Add(wn); - // Find the server name, if any - nxs.Reset(); - while ((sn = (TNamed *)nxs())) { - if (!strcmp(sn->GetTitle(), wn->GetName())) { - TIter nnxs(nsl); - TNamed *nsn = 0; - while ((nsn = (TNamed *) nnxs())) { - if (CompareOrd(sn->GetTitle(), nsn->GetTitle()) < 0) { - nsl->AddBefore(nsn, sn); - break; - } - } - if (!nsn) nsl->Add(sn); - break; - } - } - if (sn) srvList->Remove(sn); - } - // Add remaining servers at the end - nxs.Reset(); - while ((sn = (TNamed *)nxs())) { - nsl->Add(sn); - } - // Clean the orginal lists - wrkList->SetOwner(kFALSE); - srvList->SetOwner(kFALSE); - delete wrkList; - delete srvList; - wrkList = nwl; - srvList = nsl; - - // Notify - wrkList->ls(); - srvList->ls(); - - // Separate out the case with only one file server - if (srvList->GetSize() == 1) { - - Printf("\n +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ "); - Printf(" + Only one data server found: full analysis meaningful + "); - Printf(" + only when there are more file servers + "); - Printf(" +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ \n"); - - - // Create a 1D histo for cross packets - TH1F *hxpak = new TH1F("hxpak", "MBytes / Worker", - wrkList->GetSize(), 0., (Double_t)wrkList->GetSize()); - hxpak->SetDirectory(0); - hxpak->SetMinimum(0.); - hxpak->GetXaxis()->SetTitle("Worker"); - - // Set the labels - Int_t j = 1; - TIter nxsw(wrkList); - while ((wn = (TNamed *)nxsw())) { - hxpak->GetXaxis()->SetBinLabel(j++, wn->GetName()); - } - - // Fill the histograms - FillFileDistOneSrv(hxpak, writedet); - - // Display histos - gStyle->SetOptStat(0); - - TCanvas *c2 = new TCanvas("cv-hxpak", GetCanvasTitle(hxpak->GetTitle()), 800,350,700,700); - c2->cd(); - DoDraw(hxpak); - c2->Update(); - - } else { - // Create a 1D histo for file distribution - TH1F *hfdis = new TH1F("hfdis", "Packet dist", - srvList->GetSize(), 0., (Double_t)srvList->GetSize()); - hfdis->SetDirectory(0); - hfdis->SetMinimum(0); - hfdis->GetXaxis()->SetTitle("Server"); - TH1F *hbdis = new TH1F("hbdis", "MBytes dist", - srvList->GetSize(), 0., (Double_t)srvList->GetSize()); - hbdis->SetDirectory(0); - hbdis->SetMinimum(0); - hbdis->GetXaxis()->SetTitle("Server"); - // Create a 2D histo for cross packets - TH2F *hxpak = new TH2F("hxpak", "MBytes / {Worker,Server}", - srvList->GetSize(), 0., (Double_t)srvList->GetSize(), - wrkList->GetSize(), 0., (Double_t)wrkList->GetSize()); - hxpak->SetDirectory(0); - hxpak->GetYaxis()->SetTitle("Worker"); - hxpak->GetXaxis()->SetTitle("Server"); - hxpak->GetXaxis()->SetTitleOffset(1.4); - hxpak->GetYaxis()->SetTitleOffset(1.7); - - // Set the labels - Int_t j = 1; - TIter nxsw(wrkList); - while ((wn = (TNamed *)nxsw())) { - hxpak->GetYaxis()->SetBinLabel(j++, wn->GetName()); - } - j = 1; - TIter nxss(srvList); - while ((sn = (TNamed *)nxss())) { - hfdis->GetXaxis()->SetBinLabel(j, sn->GetName()); - hbdis->GetXaxis()->SetBinLabel(j, sn->GetName()); - hxpak->GetXaxis()->SetBinLabel(j++, sn->GetName()); - } - - // Fill the histograms - FillFileDist(hfdis, hbdis, hxpak, writedet); - - j = 1; - nxss.Reset(); - while ((sn = (TNamed *)nxss())) { - TString lab(sn->GetName()); - lab = TUrl(sn->GetName()).GetHost(); - if (strcmp(sn->GetTitle(), "remote") && lab.Index(".") != kNPOS) lab.Remove(lab.Index(".")); - hfdis->GetXaxis()->SetBinLabel(j, lab); - hbdis->GetXaxis()->SetBinLabel(j, lab); - hxpak->GetXaxis()->SetBinLabel(j++, lab); - } - - // Display histos - gStyle->SetOptStat(0); - - TCanvas *c1 = new TCanvas("cv-hfdis", GetCanvasTitle(hfdis->GetTitle()), 800,50,700,700); - c1->Divide(1,2); - TPad *pad1 = (TPad *) c1->GetPad(1); - TPad *pad2 = (TPad *) c1->GetPad(2); - pad1->cd(); - DoDraw(hfdis); - pad2->cd(); - DoDraw(hbdis); - c1->Update(); - - TCanvas *c2 = new TCanvas("cv-hxpak", GetCanvasTitle(hxpak->GetTitle()), 500,350,700,700); - c2->cd(); - DoDraw(hxpak, "lego"); - c2->Update(); - } - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fill file info - -void TProofPerfAnalysis::GetWrkFileList(TList *wl, TList *sl) -{ - if (!wl || !sl) return; - - // Extract information - TPerfEvent pe; - TPerfEvent* pep = &pe; - fTree->SetBranchAddress("PerfEvents", &pep); - Long64_t entries = fTree->GetEntries(); - TNamed *wn = 0, *sn = 0; - for (Long64_t k=0; kGetEntry(k); - // Analyse only packets - if (pe.fType != TVirtualPerfStats::kPacket) continue; - // Find out the worker instance - TString wrk(TUrl(pe.fSlaveName.Data()).GetHostFQDN()); - wn = (TNamed *) wl->FindObject(pe.fSlave.Data()); - if (!wn) { - wn = new TNamed(pe.fSlave.Data(), wrk.Data()); - wl->Add(wn); - } - // Find out the file server instance - TUrl uf(pe.fFileName); - TString srv(uf.GetUrl()); - Int_t ifn = srv.Index(uf.GetFile()); - if (ifn != kNPOS) srv.Remove(ifn); - sn = (TNamed *) sl->FindObject(srv.Data()); - if (!sn) { - sn = new TNamed(srv.Data(), "remote"); - sl->Add(sn); - } - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return -1 if ord1 comes before ord2, 0 i they are equal, -/// 1 if ord1 comes after ord2 - -Int_t TProofPerfAnalysis::CompareOrd(const char *ord1, const char *ord2) -{ - TString o1(ord1), o2(ord2), p1, p2; - Int_t o1d = 0, o2d = 0; - if ((o1d = o1.CountChar('.')) > (o2d = o2.CountChar('.'))) { - return 1; - } else if (o1d < o2d) { - return -1; - } else { - o1.ReplaceAll(".", " "); - o2.ReplaceAll(".", " "); - Bool_t b1 = o1.Tokenize(p1, o1d, " "); - Bool_t b2 = o2.Tokenize(p2, o2d, " "); - while (b1 && b2) { - if (p1.Atoi() > p2.Atoi()) { - return 1; - } else if (p1.Atoi() < p2.Atoi()) { - return -1; - } else { - b1 = o1.Tokenize(p1, o1d, " "); - b2 = o2.Tokenize(p2, o2d, " "); - } - } - if (b1 && !b2) { - return 1; - } else if (b2 && !b1) { - return -1; - } else { - return 0; - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fill file info - -void TProofPerfAnalysis::FillFileDist(TH1F *hf, TH1F *hb, TH2F *hx, Bool_t wdet) -{ - if (!hf || !hb || !hx) return; - - TString fnout; - FILE *fout = 0; - if (wdet) { - fnout.Form("%s-FileDist-Details.txt", GetName()); - if (!(fout = fopen(fnout.Data(), "w"))) { - Warning("FillFileDist", "asked to save details in '%s' but file could" - " not be open (errno: %d)", fnout.Data(), (int)errno); - } else { - Info("FillFileDist", "saving details to '%s'", fnout.Data()); - } - } - // Extract information - TPerfEvent pe; - TPerfEvent* pep = &pe; - fTree->SetBranchAddress("PerfEvents",&pep); - Long64_t entries = fTree->GetEntries(); - for (Long64_t k=0; kGetEntry(k); - // Analyse only packets - if (pe.fType != TVirtualPerfStats::kPacket) continue; - // Find out the labels ... - TString wrk(pe.fSlave.Data()); - TUrl uf(pe.fFileName); - TString srv(uf.GetUrl()); - Int_t ifn = srv.Index(uf.GetFile()); - if (ifn != kNPOS) srv.Remove(ifn); - // ... and the bins - Double_t xhf = hf->GetXaxis()->GetBinCenter(hf->GetXaxis()->FindBin(srv.Data())); - Double_t xhx = hx->GetXaxis()->GetBinCenter(hx->GetXaxis()->FindBin(srv.Data())); - Double_t yhx = hx->GetYaxis()->GetBinCenter(hx->GetYaxis()->FindBin(wrk.Data())); - // Save details, if asked - if (fout) - fprintf(fout, "%s,%s -> %f,%f (%f)\n", - srv.Data(), wrk.Data(), xhx, yhx, pe.fBytesRead / 1024.); - // Fill now - hf->Fill(xhf); - hb->Fill(xhf, pe.fBytesRead / 1024. / 1024.); - hx->Fill(xhx, yhx, pe.fBytesRead / 1024. / 1024.); - } - if (fout) fclose(fout); - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fill file info when there is only one file server - -void TProofPerfAnalysis::FillFileDistOneSrv(TH1F *hx, Bool_t wdet) -{ - if (!hx) return; - - TString fnout; - FILE *fout = 0; - if (wdet) { - fnout.Form("%s-FileDist-Details.txt", GetName()); - if (!(fout = fopen(fnout.Data(), "w"))) { - Warning("FillFileDistOneSrv", "asked to save details in '%s' but file could" - " not be open (errno: %d)", fnout.Data(), (int)errno); - } else { - Info("FillFileDistOneSrv", "saving details to '%s'", fnout.Data()); - } - } - // Extract information - TPerfEvent pe; - TPerfEvent* pep = &pe; - fTree->SetBranchAddress("PerfEvents",&pep); - Long64_t entries = fTree->GetEntries(); - for (Long64_t k=0; kGetEntry(k); - // Analyse only packets - if (pe.fType != TVirtualPerfStats::kPacket) continue; - // Find out the labels ... - TString wrk(pe.fSlave.Data()); - TUrl uf(pe.fFileName); - TString srv(uf.GetUrl()); - Int_t ifn = srv.Index(uf.GetFile()); - if (ifn != kNPOS) srv.Remove(ifn); - // ... and the bins - Double_t xhx = hx->GetXaxis()->GetBinCenter(hx->GetXaxis()->FindBin(wrk.Data())); - // Save details, if asked - if (fout) - fprintf(fout, "%s,%s -> %f (%f)\n", - srv.Data(), wrk.Data(), xhx, pe.fBytesRead / 1024.); - // Fill now - hx->Fill(xhx, pe.fBytesRead / 1024. / 1024.); - } - if (fout) fclose(fout); - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Measure the worker activity - -void TProofPerfAnalysis::WorkerActivity() -{ - if (!IsValid()) { - Error("WorkerActivity","not a valid instance - do nothing"); - return; - } - - // Fill basic worker info - if (!WrkInfoOK()) FillWrkInfo(); - if (!WrkInfoOK()) { - Error("WorkerActivity", "workers information not available - do nothing"); - return; - } - - TObject *o = 0; - // Create the histograms with activity vs time - if ((o = gDirectory->FindObject("act10"))) delete o; - Float_t t0 = fMergeTime - 2.* (fMaxTime - fMergeTime); - Float_t t1 = 2.*fInitTime; - if (t1 > t0) t1 = t0; - TH1F *hact10 = new TH1F("act10", "Worker activity start (seconds)", 50, 0., t1); - hact10->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("act11"))) delete o; - TH1F *hact11 = new TH1F("act11", "Worker activity stop (seconds)", 50, t0, fMaxTime); - hact11->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("act2"))) delete o; - TH1F *hact2 = new TH1F("act2", "End of activity (seconds)", 50, t0, fMaxTime); - hact2->GetXaxis()->SetTitle("Query Processing Time (s)"); - - // Fine-tune stat printing - Int_t curoptstat = gStyle->GetOptStat(); - gStyle->SetOptStat(1100); - - // Create the sorted list - TIter nxw(&fWrksInfo); - TWrkInfo *wi = 0; - while ((wi = (TWrkInfo *)nxw())) { - Int_t j = 0; - for (j = 1; j < hact10->GetNbinsX()+1 ; j++) { - if (wi->fStart < hact10->GetBinLowEdge(j)) - hact10->Fill(hact10->GetBinCenter(j)); - } - for (j = 1; j < hact11->GetNbinsX()+1 ; j++) { - if (wi->fStop > hact11->GetBinLowEdge(j)) - hact11->Fill(hact11->GetBinCenter(j)); - } - hact2->Fill(wi->fStop); - } - - // Display histos - TCanvas *c1 = new TCanvas("perf", GetCanvasTitle("Activity histos"), 800,10,700,780); - c1->Divide(1,2); - TPad *pad1 = (TPad *) c1->GetPad(1); - pad1->Divide(2,1); - TPad *pad10 = (TPad *) pad1->GetPad(1); - TPad *pad11 = (TPad *) pad1->GetPad(2); - pad10->cd(); - DoDraw(hact10); - pad11->cd(); - DoDraw(hact11); - TPad *pad2 = (TPad *) c1->GetPad(2); - pad2->cd(); - DoDraw(hact2); - c1->cd(); - c1->Update(); - - // Restore stat options - gStyle->SetOptStat(curoptstat); - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print information for all or the slowest showlast workers. -/// Use showlast < 0 to print all - -void TProofPerfAnalysis::PrintWrkInfo(Int_t showlast) -{ - // Create the sorted list - Int_t k = fWrksInfo.GetSize(); - TIter nxw(&fWrksInfo); - TWrkInfo *wi = 0; - while ((wi = (TWrkInfo *)nxw())) { - // Print info about slowest workers - k--; - if (showlast < 0 || k < showlast) wi->Print(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print information for worker 'wn' (ordinal) or on the machine whose -/// ordinal or fqdn matches 'wn'. Multiple specifications separated by ',' -/// or ' ' are supported, as well as wildcards '*', e.g. '0.2*,lxb10* lxf2323.doma.in" - -void TProofPerfAnalysis::PrintWrkInfo(const char *wn) -{ - if (!wn || (wn && strlen(wn) <= 0)) { - Error("PrintWrkInfo", "worker name or host must be defined!"); - return; - } - - // Check exact name - TWrkInfo *wi = (TWrkInfo *) fWrksInfo.FindObject(wn); - if (wi) { - wi->Print(); - } else { - // Check matching - TString ww(wn), w; - TIter nxw(&fWrksInfo); - while ((wi = (TWrkInfo *)nxw())) { - TString n(wi->GetName()), t(wi->GetTitle()); - Ssiz_t from = 0; - while (ww.Tokenize(w, from, "[, ]")) { - TRegexp re(w, kTRUE); - if (n.Index(re) != kNPOS || t.Index(re) != kNPOS) wi->Print(); - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print information for all or the slowest showlast workers. -/// Use showlast < 0 to print all - -void TProofPerfAnalysis::PrintFileInfo(Int_t showlast, const char *opt, const char *out) -{ - RedirectHandle_t rh; - if (out && strlen(out) > 0) gSystem->RedirectOutput(out, "w", &rh); - - // Create the sorted list - Int_t k = fFilesInfo.GetSize(); - TIter nxf(&fFilesInfo); - TFileInfo *fi = 0; - while ((fi = (TFileInfo *)nxf())) { - // Print info about files processed last - k--; - if (showlast < 0 || k < showlast) fi->Print(opt); - } - - if (out && strlen(out) > 0) gSystem->RedirectOutput(0, 0, &rh); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print information for file 'fn' (path including directory) or server 'fn'. -/// Multiple specifications separated by ',' -/// or ' ' are supported, as well as wildcards '*', e.g. 'pippo.root, h4mu*,lxb10*" - -void TProofPerfAnalysis::PrintFileInfo(const char *fn, const char *opt, const char *out) -{ - if (!fn || (fn && strlen(fn) <= 0)) { - Error("PrintFileInfo", "file path must be defined!"); - return; - } - - RedirectHandle_t rh; - if (out && strlen(out) > 0) gSystem->RedirectOutput(out, "w", &rh); - - // Check exact name - TFileInfo *fi = (TFileInfo *) fFilesInfo.FindObject(fn); - if (fi) { - fi->Print(opt); - } else { - // Check matching - TString fw(fn), f; - TIter nxf(&fFilesInfo); - while ((fi = (TFileInfo *)nxf())) { - TString n(fi->GetName()), s(fi->GetTitle()); - Ssiz_t from = 0; - while (fw.Tokenize(f, from, "[, ]")) { - TRegexp re(f, kTRUE); - if (n.Index(re) != kNPOS || s.Index(re) != kNPOS) fi->Print(opt); - } - } - } - - if (out && strlen(out) > 0) gSystem->RedirectOutput(0, 0, &rh); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fill basic worker info; if 'force' rescan the TTree even already done - -void TProofPerfAnalysis::FillWrkInfo(Bool_t force) -{ - // Nothing to do if already called - if (fWrksInfo.GetSize() > 0 && !force) return; - - // Cleanup existing information - fWrksInfo.SetOwner(kTRUE); - fWrksInfo.Clear(); - fInitTime = -1.; - fMergeTime = -1.; - fMaxTime = -1.; - fEvtRateMax = -1.; - fMBRateMax = -1.; - fLatencyMax = -1.; - - TList *wl = new TList; - // Extract worker information - TPerfEvent pe; - TPerfEvent* pep = &pe; - fTree->SetBranchAddress("PerfEvents",&pep); - Long64_t entries = fTree->GetEntries(); - - // First determine binning for global rates - Int_t nraw = entries * 2, jj = 0, kk = 0; - Double_t *xraw = new Double_t[nraw]; - for (Long64_t k=0; kGetEntry(k); - // Analyse only packets - if (pe.fType == TVirtualPerfStats::kPacket) { - Float_t stop = pe.fTimeStamp.GetSec() + 1e-9*pe.fTimeStamp.GetNanoSec(); - Float_t start = stop - pe.fProcTime; - // Bins - xraw[jj++] = start; - xraw[jj++] = stop; - } - } - Int_t nbins = jj; - Int_t *jidx = new Int_t[nbins]; - memset(jidx, 0, nbins * sizeof(Int_t)); - TMath::Sort(nbins, xraw, jidx, kFALSE); - Double_t *xbins = new Double_t[nbins]; - jj = 0; - for (kk = 0; kk < nbins; kk++) { - Double_t xtmp = xraw[jidx[kk]]; - if (jj == 0 || xtmp > xbins[jj - 1] + .5) { - xbins[jj] = xtmp; - jj++; - } - } - nbins = jj; - delete [] xraw; - delete [] jidx; - - // Create the global histograms - Int_t nbin = nbins - 1; - TObject *o = 0; - if ((o = gDirectory->FindObject("gEvtRate"))) delete o; - fEvtRate = new TH1F("gEvtRate", "Total event processing rate (evt/s)", nbin, xbins); - fEvtRate->SetMinimum(0.); - fEvtRate->SetStats(kFALSE); - fEvtRate->SetFillColor(kCyan-8); - fEvtRate->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("gEvtRateAvg"))) delete o; - fEvtRateRun = new TH1F("gEvtRateAvg", "Event processing rate running average (evt/s)", nbin, xbins); - fEvtRateRun->SetMinimum(0.); - fEvtRateRun->SetStats(kFALSE); - fEvtRateRun->SetLineColor(kBlue); - fEvtRateRun->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("gMBRate"))) delete o; - fMBRate = new TH1F("gMBRate", "Total processing rate (MB/s)", nbin, xbins); - fMBRate->SetMinimum(0.); - fMBRate->SetStats(kFALSE); - fMBRate->SetFillColor(kCyan-8); - fMBRate->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("gMBRateAvg"))) delete o; - fMBRateRun = new TH1F("gMBRateAvg", "Processing rate running average (MB/s)", nbin, xbins); - fMBRateRun->SetMinimum(0.); - fMBRateRun->SetStats(kFALSE); - fMBRateRun->SetLineColor(kBlue); - fMBRateRun->GetXaxis()->SetTitle("Query Processing Time (s)"); - // Not needed any longer - delete [] xbins; - - THashList gBins; - TList *gwl = 0, *gbl = 0; - - // Extract the worker info now - TWrkInfo *wi = 0; - for (Long64_t k=0; kGetEntry(k); - // Analyse only packets - if (pe.fType == TVirtualPerfStats::kPacket) { - // Find out the worker instance - if (!(wi = (TWrkInfo *) wl->FindObject(pe.fSlave.Data()))) { - wi = new TWrkInfo(pe.fSlave.Data(), pe.fSlaveName.Data()); - wl->Add(wi); - wi->fRateT = new TGraph(100); - wi->fRateRemoteT = new TGraph(100); - wi->fMBRateT = new TGraph(100); - wi->fMBRateRemoteT = new TGraph(100); - wi->fLatencyT = new TGraph(100); - } - // Add Info now - Float_t stop = pe.fTimeStamp.GetSec() + 1e-9*pe.fTimeStamp.GetNanoSec(); - Float_t start = stop - pe.fProcTime; - if (wi->fPackets <= 0) { - wi->fStart = start; - } else { - wi->fStop = stop; - } - TUrl uf(pe.fFileName), uw(pe.fSlaveName); - fMaxTime = stop; - wi->fEventsProcessed += pe.fEventsProcessed; - wi->fBytesRead += pe.fBytesRead; - wi->fLatency += pe.fLatency; - wi->fProcTime += pe.fProcTime; - wi->fCpuTime += pe.fCpuTime; - // Fill graphs - Double_t tt = stop; - Double_t ert = pe.fEventsProcessed / pe.fProcTime ; - Double_t brt = pe.fBytesRead / pe.fProcTime / 1024. / 1024. ; - wi->fRateT->SetPoint(wi->fPackets, tt, ert); - if (brt > 0.) wi->fMBRateT->SetPoint(wi->fPackets, tt, brt); - wi->fLatencyT->SetPoint(wi->fPackets, tt, pe.fLatency); - if (!pe.fFileName.IsNull() && strcmp(uf.GetHostFQDN(), uw.GetHostFQDN())) { - wi->fRateRemoteT->SetPoint(wi->fRemotePackets, tt, ert); - wi->fMBRateRemoteT->SetPoint(wi->fRemotePackets, tt, brt); - wi->fRemotePackets++; - } - wi->fPackets++; - if (ert > fEvtRateMax) fEvtRateMax = ert; - if (brt > fMBRateMax) fMBRateMax = brt; - if (pe.fLatency > fLatencyMax) fLatencyMax = pe.fLatency; - - // Fill global rate histos - for (kk = 1; kk <= nbins; kk++) { - Double_t mi = fEvtRate->GetBinLowEdge(kk); - if (mi > stop) break; - Double_t wd = fEvtRate->GetBinWidth(kk); - Double_t mx = mi + wd; - Double_t xx = fEvtRate->GetBinCenter(kk); - // Overlap length - Double_t olap = stop - mi; - if (start > mi) olap = mx - start; - if (olap >= 0) { - TString sb = TString::Format("%d", kk); - if (!(gbl = (TList *) gBins.FindObject(sb))) { - gbl = new TList; - gbl->SetName(sb); - gBins.Add(gbl); - } - if (!(gwl = (TList *) gbl->FindObject(pe.fSlave))) { - gwl = new TList; - gwl->SetName(pe.fSlave); - gbl->Add(gwl); - } - gwl->Add(new TWrkEntry(xx, ert, brt, pe.fProcTime)); - } - } - - // Notify - if (fDebug > 1) { - if (pe.fProcTime > 0.) { - Printf(" +++ %s #:%d at:%fs lat:%fs proc:%fs evts:%lld bytes:%lld (rates:%f evt/s, %f MB/s)", - wi->GetName(), wi->fPackets, fMaxTime - pe.fProcTime, - pe.fLatency, pe.fProcTime, pe.fEventsProcessed, pe.fBytesRead, - ert, brt); - } else { - Printf(" +++ %s #:%d at:%fs lat:%fs proc:%fs rate:-- evt/s (-- bytes/s)", - wi->GetName(), wi->fPackets, fMaxTime, pe.fLatency, pe.fProcTime); - } - } - } else if (pe.fType == TVirtualPerfStats::kStart) { - Float_t start = pe.fTimeStamp.GetSec() + 1e-9*pe.fTimeStamp.GetNanoSec(); - if (fDebug > 1) Printf(" +++ %s Start: %f s", pe.fEvtNode.Data(), start); - } else if (pe.fType == TVirtualPerfStats::kStop) { - Float_t stop = pe.fTimeStamp.GetSec() + 1e-9*pe.fTimeStamp.GetNanoSec(); - if (fDebug > 1) Printf(" +++ %s Stop: %f s", pe.fEvtNode.Data(), stop); - } else { - if (fDebug > 2) Printf(" +++ Event type: %d", pe.fType); - } - } - - TIter nxb(&gBins); - gbl = 0; - while ((gbl = (TList *) nxb())) { - gwl = 0; - TIter nxw(gbl); - while ((gwl = (TList *) nxw())) { - Double_t er = 0, br = 0, pt = 0, xx = 0; - TIter nxp(gwl); - TWrkEntry *we = 0; - while ((we = (TWrkEntry *) nxp())) { - if (we->fProcTime > 0) { - er += we->fEvtRate * we->fProcTime; - br += we->fMBRate * we->fProcTime; - pt += we->fProcTime; - } - xx = we->fXx; - } - if (pt > 0.) { - er /= pt; - br /= pt; - fEvtRate->Fill(xx, er); - if (br > 0.) fMBRate->Fill(xx, br); - } - } - } - - // Running averages - Double_t er = 0, br = 0, pt = 0; - for (kk = 1; kk < nbins; kk++) { - Double_t wd = fEvtRate->GetBinWidth(kk); - Double_t wx = fEvtRate->GetBinCenter(kk); - Double_t wer = fEvtRate->GetBinContent(kk); - Double_t wbr = fMBRate->GetBinContent(kk); - - if (kk == 1) { - er = wer; - br = wbr; - pt = wd; - } else { - er *= pt; - br *= pt; - pt += wd; - er += wer * wd; - br += wbr * wd; - er /= pt; - br /= pt; - } - if (er > fEvtRateAvgMax) fEvtRateAvgMax = er; - if (br > fMBRateAvgMax) fMBRateAvgMax = br; - fEvtRateAvg = er; - fMBRateAvg = br; - // Fill - fEvtRateRun->Fill(wx, er); - fMBRateRun->Fill(wx, br); - } - - - // Final analysis to find relevant times - TIter nxw(wl); - while ((wi = (TWrkInfo *) nxw())) { - fWrksInfo.Add(wi); - if (wi->fStart > fInitTime) fInitTime = wi->fStart; - // Resize the graphs - wi->fRateT->Set(wi->fPackets); - wi->fRateRemoteT->Set(wi->fRemotePackets); - wi->fLatencyT->Set(wi->fPackets); - wi->fMBRateT->Set(wi->fPackets); - wi->fMBRateRemoteT->Set(wi->fRemotePackets); - } - wl->SetOwner(kFALSE); - delete wl; - - // Final analysis to find relevant times - fMergeTime = fMaxTime; - Int_t rsw = (fWrksInfo.GetSize() > 1) ? 2 : 1, ksw = 0; - TIter nxsw(&fWrksInfo); - while ((wi = (TWrkInfo *) nxsw())) { - if (wi->fStop > 0.) ksw++; - if (ksw == rsw) break; - } - if (wi) fMergeTime = wi->fStop; - - // (Re-)create the event and packet distribution histograms - SafeDelete(fEvents); - SafeDelete(fPackets); - fEvents = new TH1F("hevents", "Events per worker", fWrksInfo.GetSize(), -.5, fWrksInfo.GetSize()-.5); - fEvents->SetDirectory(0); - fPackets = new TH1F("hpackets", "Packets per worker", fWrksInfo.GetSize(), -.5, fWrksInfo.GetSize()-.5); - fPackets->SetDirectory(0); - Int_t j = 0; - TIter nxwi(&fWrksInfo); - while ((wi = (TWrkInfo *)nxwi())) { - fEvents->GetXaxis()->SetBinLabel(j+1, wi->GetName()); - fEvents->Fill(j, wi->fEventsProcessed); - fPackets->GetXaxis()->SetBinLabel(j+1, wi->GetName()); - fPackets->Fill(j++, wi->fPackets); - } - fEvents->SetMinimum(0.); - fPackets->SetMinimum(0.); - fEvents->SetFillColor(38); - fPackets->SetFillColor(38); - fEvents->GetYaxis()->SetTitle("Events"); - fEvents->GetXaxis()->SetTitle("Worker"); - fPackets->GetYaxis()->SetTitle("Packets"); - fPackets->GetXaxis()->SetTitle("Worker"); - - // Print summary - if (fgDebug) Summary(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print summary of query. Use opt = 'S' for compact version. -/// Output to 'out' or to screen. - -void TProofPerfAnalysis::Summary(Option_t *opt, const char *out) -{ - TString o(out); - RedirectHandle_t rh; - if (!o.IsNull()) { - const char *m = (o.BeginsWith("+")) ? "a" : "w"; - o.Remove(TString::kLeading, '+'); - gSystem->RedirectOutput(o, m, &rh); - } - - // Print summary - if (!strcmp(opt, "S")) { - // Short version - Printf("%d %f %f %f %f %f %f %f", - fWrksInfo.GetSize(), fMaxTime, fInitTime, fMaxTime - fMergeTime, - fEvtRateAvg, fEvtRateAvgMax, fMBRateAvg, fMBRateAvgMax); - } else { - // Long version - Printf(" +++ %d workers were active during this query", fWrksInfo.GetSize()); - Printf(" +++ Total query time: %f secs (init: %f secs, merge: %f secs)", - fMaxTime, fInitTime, fMaxTime - fMergeTime); - Printf(" +++ Avg processing rates: %.4f evts/s, %.4f MB/s", fEvtRateAvg, fMBRateAvg); - Printf(" +++ Max processing rates: %.4f evts/s, %.4f MB/s", fEvtRateAvgMax, fMBRateAvgMax); - } - - if (!o.IsNull()) gSystem->RedirectOutput(0, 0, &rh); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fill basic worker info; if 'force' rescan the TTree even already done - -void TProofPerfAnalysis::FillFileInfo(Bool_t force) -{ - // Nothing to do if already called - if (fFilesInfo.GetSize() > 0 && !force) return; - - // Cleanup existing information - fFilesInfo.SetOwner(kTRUE); - fFilesInfo.Clear(); - - TList *fl = new TList; - // Extract worker information - TPerfEvent pe; - TPerfEvent* pep = &pe; - fTree->SetBranchAddress("PerfEvents",&pep); - Long64_t entries = fTree->GetEntries(); - TFileInfo *fi = 0; - for (Long64_t k=0; kGetEntry(k); - // Analyse only packets - if (pe.fType == TVirtualPerfStats::kPacket) { - TUrl uf(pe.fFileName); - TString srv(uf.GetUrl()); - Int_t ifn = srv.Index(uf.GetFile()); - if (ifn != kNPOS) srv.Remove(ifn); - // Find out the file instance - fi = (TFileInfo *) fl->FindObject(uf.GetFile()); - if (!fi) { - fi = new TFileInfo(uf.GetFile(), srv.Data()); - fl->Add(fi); - fi->fSizeP = new TGraph(10); - fi->fRateP = new TGraph(10); - fi->fRatePRemote = new TGraph(10); - fi->fMBRateP = new TGraph(10); - fi->fMBRatePRemote = new TGraph(10); - } - // Add Info now - Float_t stop = pe.fTimeStamp.GetSec() + 1e-9*pe.fTimeStamp.GetNanoSec(); - Float_t start = stop - pe.fProcTime; - if (fi->fPackets <= 0) { - fi->fStart = start; - } else { - fi->fStop = stop; - } - TUrl uw(pe.fSlaveName); - - // Fill size graphs - fi->fSizeP->SetPoint(fi->fPackets, (Double_t) fi->fPackets, (Double_t) pe.fEventsProcessed); - fi->fSizeAvg += pe.fEventsProcessed; - if (pe.fEventsProcessed > fi->fSizeMax || fi->fSizeMax < 0.) fi->fSizeMax = pe.fEventsProcessed; - if (pe.fEventsProcessed < fi->fSizeMin || fi->fSizeMin < 0.) fi->fSizeMin = pe.fEventsProcessed; - - // Fill rate graphs - Double_t tt = pe.fTimeStamp.GetSec() + 1e-9*pe.fTimeStamp.GetNanoSec(); - Double_t ert = pe.fEventsProcessed / pe.fProcTime ; - Double_t brt = pe.fBytesRead / pe.fProcTime / 1024. / 1024. ; - fi->fRateP->SetPoint(fi->fPackets, tt, ert); - if (brt > 0.) fi->fMBRateP->SetPoint(fi->fPackets, tt, brt); - if (!pe.fFileName.IsNull() && strcmp(uf.GetHostFQDN(), uw.GetHostFQDN())) { - if (!(fi->fRWrkList.FindObject(pe.fSlave))) fi->fRWrkList.Add(new TNamed(pe.fSlave, pe.fSlaveName)); - fi->fRatePRemote->SetPoint(fi->fRPackets, tt, ert); - fi->fMBRatePRemote->SetPoint(fi->fRPackets, tt, brt); - fi->fRPackets++; - } - fi->fPackets++; - if (brt > 0) { - fi->fMBRateAvg += brt; - if (brt > fi->fMBRateMax || fi->fMBRateMax < 0.) fi->fMBRateMax = brt; - if (brt < fi->fMBRateMin || fi->fMBRateMin < 0.) fi->fMBRateMin = brt; - } - - // Packet info - TPackInfo *pi = new TPackInfo(pe.fSlave, pe.fSlaveName, start, stop, pe.fEventsProcessed, brt); - fi->fPackList.Add(pi); - TWrkInfoFile *wif = 0; - if (!(wif = (TWrkInfoFile *) fi->fWrkList.FindObject(pe.fSlave))) { - wif = new TWrkInfoFile(pe.fSlave, uf.GetFile()); - fi->fWrkList.Add(wif); - } - wif->fPackets.Add(pi); - - // Notify - if (fDebug > 1) { - if (pe.fProcTime > 0.) { - Printf(" +++ %s #:%d at:%fs lat:%fs proc:%fs evts:%lld bytes:%lld (rates:%f evt/s, %f MB/s)", - fi->GetName(), fi->fPackets, fMaxTime - pe.fProcTime, - pe.fLatency, pe.fProcTime, pe.fEventsProcessed, pe.fBytesRead, - ert, brt); - } else { - Printf(" +++ %s #:%d at:%fs lat:%fs proc:%fs rate:-- evt/s (-- bytes/s)", - fi->GetName(), fi->fPackets, fMaxTime, pe.fLatency, pe.fProcTime); - } - } - } else if (pe.fType == TVirtualPerfStats::kStart) { - Float_t start = pe.fTimeStamp.GetSec() + 1e-9*pe.fTimeStamp.GetNanoSec(); - if (fDebug > 1) Printf(" +++ %s Start: %f s", pe.fEvtNode.Data(), start); - } else if (pe.fType == TVirtualPerfStats::kStop) { - Float_t stop = pe.fTimeStamp.GetSec() + 1e-9*pe.fTimeStamp.GetNanoSec(); - if (fDebug > 1) Printf(" +++ %s Stop: %f s", pe.fEvtNode.Data(), stop); - } else { - if (fDebug > 2) Printf(" +++ Event type: %d", pe.fType); - } - } - // Final analysis to find relevant times - TIter nxf(fl); - while ((fi = (TFileInfo *) nxf())) { - fFilesInfo.Add(fi); - // Resize the graphs - fi->fRateP->Set(fi->fPackets); - fi->fRatePRemote->Set(fi->fRPackets); - fi->fMBRateP->Set(fi->fPackets); - fi->fMBRatePRemote->Set(fi->fRPackets); - } - fl->SetOwner(kFALSE); - delete fl; - - // Print summary - if (fgDebug) - Printf(" +++ %d files were processed during this query", fFilesInfo.GetSize()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static setter for the verbosity level - -void TProofPerfAnalysis::SetDebug(Int_t d) -{ - fDebug = d; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw object 'o' with options 'opt' -/// Save it with 'name' if in saving mode (see SetSaveResult) - -void TProofPerfAnalysis::DoDraw(TObject *o, Option_t *opt, const char *name) -{ - // Draw - o->Draw(opt); - - // Save the result - if (fSaveResult) { - // Preparation is done in SetSaveResult, here we just update - TDirectory *curdir = gDirectory; - TFile *f = TFile::Open(fFileResult, "UPDATE"); - if (f && !f->IsZombie()) { - const char *n = (name && strlen(name) > 0) ? name : 0; - o->Write(n); - f->Close(); - } - if (f) delete f; - gDirectory = curdir; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set save result mode and validate 'file' according to 'mode'. -/// Return 0 on success, -1 if any problem with the file is encountered -/// (save result mode is not enabled in such a case). -/// If 'file' is null saving is disabled. - -Int_t TProofPerfAnalysis::SetSaveResult(const char *file, Option_t *mode) -{ - // A null 'file' indicates the will to disable - if (!file) { - fFileResult = ""; - fSaveResult = kFALSE; - // Notify - Printf("Drawn objects saving disabled"); - return 0; - } - - // Check if there is a change - if (!fFileResult.IsNull() && fFileResult == file) { - // No change - fSaveResult = kTRUE; - return 0; - } - // New or changed file: validate - fFileResult = ""; - fSaveResult = kFALSE; - TDirectory *curdir = gDirectory; - TFile *f = TFile::Open(file, mode); - if (!f || f->IsZombie()) { - if (f) delete f; - fFileResult = ""; - Error("SetSaveResult", "could not open file '%s' in mode '%s'", file, mode); - gDirectory = curdir; - return -1; - } - f->Close(); - delete f; - gDirectory = curdir; - // Ok - fFileResult = file; - fSaveResult = kTRUE; - // Notify - Printf("Drawn objects will be saved in file '%s'", file); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Static setter for the verbosity level - -void TProofPerfAnalysis::SetgDebug(Bool_t on) -{ - fgDebug = on; -} -//////////////////////////////////////////////////////////////////////////////// -/// Display event and packet distribution - -void TProofPerfAnalysis::EventDist() -{ - if (!fEvents || !fPackets) { - Error("EventDist", "distributions not initialized - do nothing"); - } - - // Display histos - TCanvas *c1 = new TCanvas("evtdist", GetCanvasTitle("Event distributions"),800,10,700,780); - c1->Divide(1,2); - TPad *pad1 = (TPad *) c1->GetPad(1); - pad1->cd(); - fEvents->SetStats(kFALSE); - DoDraw(fEvents); - TPad *pad2 = (TPad *) c1->GetPad(2); - pad2->cd(); - fPackets->SetStats(kFALSE); - DoDraw(fPackets); - c1->cd(); - c1->Update(); - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show event processing or MB processing rate plot vs time - -void TProofPerfAnalysis::RatePlot(const char *wrks) -{ - Bool_t global = (wrks && !strcmp(wrks, "global")) ? kTRUE : kFALSE; - - TH1F *hrt1 = 0, *hrt2 = 0; - if (global) { - hrt1 = fEvtRate; - hrt2 = fMBRate; - } else { - // Create the histograms - TObject *o = 0; - if ((o = gDirectory->FindObject("rt1"))) delete o; - hrt1 = new TH1F("rt1", "Evt processing rate (evt/s)", 100, 0., fMaxTime); - hrt1->SetMinimum(0.); - hrt1->SetMaximum(1.05*fEvtRateMax); - hrt1->SetStats(kFALSE); - hrt1->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("rt2"))) delete o; - hrt2 = new TH1F("rt2", "MB processing rate (MB/s)", 100, 0., fMaxTime); - hrt2->SetMinimum(0.); - hrt2->SetMaximum(1.05*fMBRateMax); - hrt2->SetStats(kFALSE); - hrt2->GetXaxis()->SetTitle("Query Processing Time (s)"); - } - - // Display histo frames - TCanvas *c1 = new TCanvas("rates", GetCanvasTitle("Processing rates"), 800,10,700,780); - c1->Divide(1,2); - TPad *pad1 = (TPad *) c1->GetPad(1); - pad1->cd(); - hrt1->Draw(); - if (global) DoDraw(fEvtRateRun, "SAME", "EvtRateRun"); - TPad *pad2 = (TPad *) c1->GetPad(2); - pad2->cd(); - hrt2->Draw(); - if (global) DoDraw(fMBRateRun, "SAME", "MBRateRun"); - c1->cd(); - c1->Update(); - - // Done if global - if (global) return; - - // Which workers? - THashList *wl = 0; - TString ww(wrks); - if (!ww.IsNull() && ww != "*" && ww != "all") { - TString w; - Ssiz_t from = 0; - while ((ww.Tokenize(w, from, ","))) { - if (!wl) wl = new THashList(); - wl->Add(new TObjString(w.Data())); - } - } - - // Now plot the graphs per worker - Int_t ci = 40, cir = 30, ic = 0; - TIter nxw(&fWrksInfo); - TWrkInfo *wi = 0; - while ((wi = (TWrkInfo *) nxw())) { - if (wl && !wl->FindObject(wi->GetName())) continue; - if (wi->fRateT && wi->fRateT->GetN() > 0) { - wi->fRateT->SetNameTitle(wi->GetName(), wi->GetTitle()); - pad1->cd(); - wi->fRateT->SetLineColor(ci); - DoDraw(wi->fRateT, "L", TString::Format("RateT-%s", wi->fRateT->GetName())); - } - if (wi->fRateRemoteT && wi->fRateRemoteT->GetN() > 0) { - wi->fRateRemoteT->SetNameTitle(wi->GetName(), wi->GetTitle()); - pad1->cd(); - wi->fRateRemoteT->SetLineColor(cir); - DoDraw(wi->fRateRemoteT, "L", TString::Format("RateRemoteT-%s", wi->fRateRemoteT->GetName())); - } - if (wi->fMBRateT && wi->fMBRateT->GetN() > 0) { - wi->fMBRateT->SetNameTitle(wi->GetName(), wi->GetTitle()); - pad2->cd(); - wi->fMBRateT->SetLineColor(ci); - DoDraw(wi->fMBRateT, "L", TString::Format("MBRateT-%s", wi->fMBRateT->GetName())); - } - if (wi->fMBRateRemoteT && wi->fMBRateRemoteT->GetN() > 0) { - wi->fMBRateRemoteT->SetNameTitle(wi->GetName(), wi->GetTitle()); - pad2->cd(); - wi->fMBRateRemoteT->SetLineColor(cir); - DoDraw(wi->fMBRateRemoteT, "L", TString::Format("MBRateRemoteT-%s", wi->fMBRateRemoteT->GetName())); - } - ic++; - ci = ic%10 + 40; - cir = ic%10 + 30; - c1->cd(); - c1->Update(); - } - - // Cleanup - if (wl) { - wl->SetOwner(kTRUE); - delete wl; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show event processing or MB processing rate plot vs time -/// Create the histograms - -void TProofPerfAnalysis::LatencyPlot(const char *wrks) -{ - TObject *o = 0; - if ((o = gDirectory->FindObject("lt1"))) delete o; - TH1F *hlt1 = new TH1F("lt1", "Packet retrieval latency", 100, 0., fMaxTime); - hlt1->SetMinimum(0.); - hlt1->SetMaximum(1.05*fLatencyMax); - hlt1->SetStats(kFALSE); - hlt1->GetXaxis()->SetTitle("Query Processing Time (s)"); - hlt1->GetYaxis()->SetTitle("Latency (s)"); - - // Display histo frames - TCanvas *c1 = new TCanvas("latency", GetCanvasTitle("Packet Retrieval Latency"), 800,10,700,780); - hlt1->Draw(); - c1->cd(); - c1->Update(); - - // Which workers? - THashList *wl = 0; - TString ww(wrks); - if (!ww.IsNull() && ww != "*" && ww != "all") { - TString w; - Ssiz_t from = 0; - while ((ww.Tokenize(w, from, ","))) { - if (!wl) wl = new THashList(); - wl->Add(new TObjString(w.Data())); - } - } - - // Now plot the graphs per worker - Int_t ci = 40, ic = 0; - TIter nxw(&fWrksInfo); - TWrkInfo *wi = 0; - while ((wi = (TWrkInfo *) nxw())) { - if (wl && !wl->FindObject(wi->GetName())) continue; - if (wi->fLatencyT) { - wi->fLatencyT->SetNameTitle(wi->GetName(), wi->GetTitle()); - wi->fLatencyT->SetLineColor(ci); - DoDraw(wi->fLatencyT, "L", TString::Format("LatencyT-%s", wi->fLatencyT->GetName())); - } - ic++; - ci = ic%10 + 40; - c1->cd(); - c1->Update(); - } - - // Cleanup - if (wl) { - wl->SetOwner(kTRUE); - delete wl; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show event processing or MB processing rate plot vs time - -void TProofPerfAnalysis::FileProcPlot(const char *fn, const char *out) -{ - if (!fn || strlen(fn) <= 0) { - Error("FileRatePlot", "file name is mandatory!"); - return; - } - // Get the file info object - TFileInfo *fi = (TFileInfo *) fFilesInfo.FindObject(fn); - if (!fi) { - Error("FileRatePlot", "TFileInfo object for '%s' not found!", fn); - return; - } - - // Output text file, if required - FILE *fo = stdout; - if (out && strlen(out) > 0) { - if (!(fo = fopen(out, "w"))) { - Warning("FileRatePlot", "problems creating '%s': logging to stdout", out); - fo = stdout; - } else { - Printf(" Details logged to %s", out); - } - } - - // Get bins - Int_t nbins = fi->fPackList.GetSize() * 2; - Double_t *xraw = new Double_t[nbins]; - Int_t jj = 0; - TPackInfo *pi = 0; - TIter nxp(&(fi->fPackList)); - while ((pi = (TPackInfo *) nxp())) { - // Bins - xraw[jj++] = pi->fStart; - xraw[jj++] = pi->fStop; - } - Int_t *jidx = new Int_t[nbins]; - memset(jidx, 0, nbins * sizeof(Int_t)); - TMath::Sort(nbins, xraw, jidx, kFALSE); - Double_t *xbins = new Double_t[nbins]; - Int_t kk =0; - for (kk = 0; kk < nbins; kk++) { - xbins[kk] = xraw[jidx[kk]]; - } - delete [] xraw; - delete [] jidx; - - // Create the histograms - Int_t nbin = nbins - 1; - TObject *o = 0; - if ((o = gDirectory->FindObject("rt1"))) delete o; - TH1F *hrt1 = new TH1F("rt1", "Total processing rate (MB/s)", nbins - 1, xbins); - hrt1->SetMinimum(0.); - hrt1->SetStats(kFALSE); - hrt1->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("rt2"))) delete o; - TH1F *hrt2 = new TH1F("rt2", "Number of processing workers", nbins - 1, xbins); - hrt2->SetMinimum(0.); - hrt2->SetMaximum(1.2*fWrksInfo.GetSize()); - hrt2->SetStats(kFALSE); - hrt2->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("rt3"))) delete o; - TH1F *hrt3 = new TH1F("rt3", "Total processing events", nbins - 1, xbins); - hrt3->SetMinimum(0.); - hrt3->SetStats(kFALSE); - hrt3->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("rt4"))) delete o; - TH1F *hrt4 = new TH1F("rt4", "Weighted processing rate (MB/s)", nbins - 1, xbins); - hrt4->SetMinimum(0.); - hrt4->SetStats(kFALSE); - hrt4->GetXaxis()->SetTitle("Query Processing Time (s)"); - // Not needed any longer - delete [] xbins; - - // Fill histos now - Int_t ii = 0; - for (ii = 1; ii <= nbin; ii++) { - Double_t mi = hrt1->GetBinLowEdge(ii); - Double_t wd = hrt1->GetBinWidth(ii); - Double_t mx = mi + wd; - Double_t xx = hrt1->GetBinCenter(ii); - fprintf(fo, " Bin: %d/%d [%f, %f]\n", ii, nbin, mi, mx); - pi = 0; - kk = 0; - nxp.Reset(); - while ((pi = (TPackInfo *) nxp())) { - // Overlap length - Double_t olap = pi->fStop - mi; - if (pi->fStart > mi) olap = mx - pi->fStart; - if (olap >= 0) { - hrt1->Fill(xx, pi->fMBRate); - hrt2->Fill(xx, 1.); - hrt3->Fill(xx, pi->fSize); - hrt4->Fill(xx, pi->fMBRate * pi->fSize); - fprintf(fo, " %d: %s \t%lld \tevts \t%f \tMB/s\n", kk++, pi->GetName(), pi->fSize, pi->fMBRate); - } - } - } - if (fo != stdout) fclose(fo); - - // Display histo frames - TCanvas *c1 = new TCanvas("rates", GetCanvasTitle("File processing info"), 800,10,700,780); - c1->Divide(1,3); - TPad *pad1 = (TPad *) c1->GetPad(1); - pad1->cd(); - DoDraw(hrt1); - TPad *pad2 = (TPad *) c1->GetPad(2); - pad2->cd(); - DoDraw(hrt2); - TPad *pad4 = (TPad *) c1->GetPad(3); - pad4->cd(); - hrt4->Divide(hrt3); - DoDraw(hrt4); - c1->cd(); - c1->Update(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Show MB processing rate plot per file vs time - -void TProofPerfAnalysis::FileRatePlot(const char *fns) -{ - // Create the histograms - TObject *o = 0; - if ((o = gDirectory->FindObject("rt1"))) delete o; - TH1F *hrt1 = new TH1F("rt1", "Event processing rate per packet (evt/s)", 100, 0., fMaxTime); - hrt1->SetMinimum(0.); - hrt1->SetMaximum(1.05*fEvtRateMax); - hrt1->SetStats(kFALSE); - hrt1->GetXaxis()->SetTitle("Query Processing Time (s)"); - if ((o = gDirectory->FindObject("rt2"))) delete o; - TH1F *hrt2 = new TH1F("rt2", "I/O processing rate per packet (MB/s)", 100, 0., fMaxTime); - hrt2->SetMinimum(0.); - hrt2->SetMaximum(1.05*fMBRateMax); - hrt2->SetStats(kFALSE); - hrt2->GetXaxis()->SetTitle("Query Processing Time (s)"); - - // Display histo frames - TCanvas *c1 = new TCanvas("rates", GetCanvasTitle("Processing rates"), 800,10,700,780); - c1->Divide(1,2); - TPad *pad1 = (TPad *) c1->GetPad(1); - pad1->cd(); - hrt1->Draw(); - TPad *pad2 = (TPad *) c1->GetPad(2); - pad2->cd(); - hrt2->Draw(); - c1->cd(); - c1->Update(); - - // Which workers? - THashList *fl = 0; - TString fw(fns); - if (!fw.IsNull() && fw != "*" && fw != "all") { - TString w; - Ssiz_t from = 0; - while ((fw.Tokenize(w, from, ","))) { - if (!fl) fl = new THashList(); - fl->Add(new TObjString(w.Data())); - } - } - - // Now plot the graphs per worker - Int_t ci = 40, cir = 30, ic = 0; - TIter nxf(&fFilesInfo); - TFileInfo *fi = 0; - while ((fi = (TFileInfo *) nxf())) { - if (fl && !fl->FindObject(fi->GetName())) continue; - if (fi->fRateP && fi->fRateP->GetN() > 0) { - fi->fRateP->SetNameTitle(fi->GetName(), fi->GetTitle()); - pad1->cd(); - fi->fRateP->SetLineColor(ci); - DoDraw(fi->fRateP, "L", TString::Format("RateP-%d", ic)); - } - if (fi->fRatePRemote && fi->fRatePRemote->GetN() > 0) { - fi->fRatePRemote->SetNameTitle(fi->GetName(), fi->GetTitle()); - pad1->cd(); - fi->fRatePRemote->SetLineColor(cir); - DoDraw(fi->fRatePRemote, "L", TString::Format("RatePRemote-%d", ic)); - } - if (fi->fMBRateP && fi->fMBRateP->GetN() > 0) { - fi->fMBRateP->SetNameTitle(fi->GetName(), fi->GetTitle()); - pad2->cd(); - fi->fMBRateP->SetLineColor(ci); - DoDraw(fi->fMBRateP, "L", TString::Format("MBRateP-%d", ic)); - } - if (fi->fMBRatePRemote && fi->fMBRatePRemote->GetN() > 0) { - fi->fMBRatePRemote->SetNameTitle(fi->GetName(), fi->GetTitle()); - pad2->cd(); - fi->fMBRatePRemote->SetLineColor(cir); - DoDraw(fi->fMBRatePRemote, "L", TString::Format("MBRatePRemote-%d", ic)); - } - ic++; - ci = ic%10 + 40; - cir = ic%10 + 30; - c1->cd(); - c1->Update(); - } - - // Cleanup - if (fl) { - fl->SetOwner(kTRUE); - delete fl; - } -} diff --git a/proof/proofbench/src/TSelEvent.cxx b/proof/proofbench/src/TSelEvent.cxx deleted file mode 100644 index bfb2ff0384e24..0000000000000 --- a/proof/proofbench/src/TSelEvent.cxx +++ /dev/null @@ -1,265 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TSelEvent -\ingroup proofbench - -Selector for PROOF I/O benchmark test. -For the I/O benchmark, event files are read in and histograms are filled. -For memory clean-up, dedicated files large enough to clean up memory -cache on the machine are read in. Or memory clean-up can be -accompolished by system call on Linux machine inside SlaveBegin(..) -which should be much faster the reading in large files. - -*/ - -#define TSelEvent_cxx - -#include "TSelEvent.h" -#include -#include -#include "TParameter.h" -#include "TProofBenchTypes.h" -#include "TTree.h" -#include "TCanvas.h" -#include "TFileInfo.h" -#include "THashList.h" -#include "TClonesArray.h" -#include "TRefArray.h" -#include -#include -#include -#include "TSystem.h" -#include "TROOT.h" - -ClassImp(TSelEvent); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TSelEvent::TSelEvent(TTree *) - : fReadType(0), fDebug(kFALSE), fCHist(0), fPtHist(0), - fNTracksHist(0), fEventName(0), fTracks(0), fHighPt(0), fMuons(0), - fH(0), b_event_fType(0), b_fEventName(0), b_event_fNtrack(0), b_event_fNseg(0), - b_event_fNvertex(0), b_event_fFlag(0), b_event_fTemperature(0), - b_event_fMeasures(0), b_event_fMatrix(0), b_fClosestDistance(0), - b_event_fEvtHdr(0), b_fTracks(0), b_fHighPt(0), b_fMuons(0), - b_event_fLastTrack(0), b_event_fWebHistogram(0), b_fH(0), - b_event_fTriggerBits(0), b_event_fIsValid(0) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TSelEvent::TSelEvent() - : fReadType(0), fDebug(kFALSE), fCHist(0), fPtHist(0), - fNTracksHist(0), fEventName(0), fTracks(0), fHighPt(0), fMuons(0), - fH(0), b_event_fType(0), b_fEventName(0), b_event_fNtrack(0), b_event_fNseg(0), - b_event_fNvertex(0), b_event_fFlag(0), b_event_fTemperature(0), - b_event_fMeasures(0), b_event_fMatrix(0), b_fClosestDistance(0), - b_event_fEvtHdr(0), b_fTracks(0), b_fHighPt(0), b_fMuons(0), - b_event_fLastTrack(0), b_event_fWebHistogram(0), b_fH(0), - b_event_fTriggerBits(0), b_event_fIsValid(0) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Begin() function is called at the start of the query. -/// When running with PROOF Begin() is only called on the client. -/// The tree argument is deprecated (on PROOF 0 is passed). - -void TSelEvent::Begin(TTree *) -{ - TString option = GetOption(); - - //get parameters - - Bool_t found_readtype=kFALSE; - Bool_t found_debug=kFALSE; - - TIter nxt(fInput); - TString sinput; - TObject *obj; - while ((obj = nxt())){ - sinput=obj->GetName(); - //Info("Begin", "name=%s", sinput.Data()); - if (sinput.Contains("PROOF_Benchmark_ReadType")){ - if ((fReadType = dynamic_cast(obj))) found_readtype = kTRUE; - continue; - } - if (sinput.Contains("PROOF_BenchmarkDebug")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fDebug= a->GetVal(); - found_debug=kTRUE; - //Info("Begin", "PROOF_BenchmarkDebug=%d", fDebug); - } - else{ - Error("Begin", "PROOF_BenchmarkDebug not type TParameter*"); - } - continue; - } - } - - if (!found_readtype){ - fReadType = new TPBReadType(TPBReadType::kReadOpt); - Warning("Begin", "PROOF_Benchmark_ReadType not found; using default: %d", - fReadType->GetType()); - } - if (!found_debug){ - Warning("Begin", "PROOF_BenchmarkDebug not found; using default: %d", - fDebug); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// The SlaveBegin() function is called after the Begin() function. -/// When running with PROOF SlaveBegin() is called on each slave server. -/// The tree argument is deprecated (on PROOF 0 is passed). - -void TSelEvent::SlaveBegin(TTree *tree) -{ - Init(tree); - - TString option = GetOption(); - - Bool_t found_readtype=kFALSE; - Bool_t found_debug=kFALSE; - - //fInput->Print("A"); - TIter nxt(fInput); - TString sinput; - TObject *obj; - while ((obj = nxt())){ - sinput=obj->GetName(); - //Info("SlaveBegin", "name=%s", sinput.Data()); - if (sinput.Contains("PROOF_Benchmark_ReadType")){ - if ((fReadType = dynamic_cast(obj))) found_readtype = kTRUE; - continue; - } - if (sinput.Contains("PROOF_BenchmarkDebug")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fDebug= a->GetVal(); - found_debug=kTRUE; - //Info("SlaveBegin", "PROOF_BenchmarkDebug=%d", fDebug); - } - else{ - Error("SlaveBegin", "PROOF_BenchmarkDebug not type TParameter" - "*"); - } - continue; - } - } - - if (!found_readtype){ - fReadType = new TPBReadType(TPBReadType::kReadOpt); - Warning("SlaveBegin", "PROOF_Benchmark_ReadType not found; using default: %d", - fReadType->GetType()); - } - if (!found_debug){ - Warning("SlaveBegin", "PROOF_BenchmarkDebug not found; using default: %d", - fDebug); - } - - fPtHist = new TH1F("pt_dist","p_{T} Distribution", 100, 0, 5); - fPtHist->SetDirectory(0); - fPtHist->GetXaxis()->SetTitle("p_{T}"); - fPtHist->GetYaxis()->SetTitle("dN/p_{T}dp_{T}"); - - fNTracksHist = new TH1F("ntracks_dist","N_{Tracks} per Event" - " Distribution", 100, 50, 150); - //enable rebinning - fNTracksHist->SetCanExtend(TH1::kAllAxes); - fNTracksHist->SetDirectory(0); - fNTracksHist->GetXaxis()->SetTitle("N_{Tracks}"); - fNTracksHist->GetYaxis()->SetTitle("N_{Events}"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Process() function is called for each entry in the tree (or possibly -/// keyed object in the case of PROOF) to be processed. The entry argument -/// specifies which entry in the currently loaded tree is to be processed. -/// It can be passed to either TTree::GetEntry() or TBranch::GetEntry() -/// to read either all or the required parts of the data. When processing -/// keyed objects with PROOF, the object is already loaded and is available -/// via the fObject pointer. -/// -/// This function should contain the "body" of the analysis. It can contain -/// simple or elaborate selection criteria, run algorithms on the data -/// of the event and typically fill histograms. - -Bool_t TSelEvent::Process(Long64_t entry) -{ - // WARNING when a selector is used with a TChain, you must use - // the pointer to the current TTree to call GetEntry(entry). - // The entry is always the local entry number in the current tree. - // Assuming that fChain is the pointer to the TChain being processed, - // use fChain->GetTree()->GetEntry(entry). - - if (fReadType->GetType() != TPBReadType::kReadNotSpecified){ - switch (fReadType->GetType()){ - case TPBReadType::kReadFull: - // Full read - fChain->GetTree()->GetEntry(entry); - fNTracksHist->Fill(fNtrack); - - for(Int_t j=0;jGetEntries();j++){ - Track* curtrack = dynamic_cast(fTracks->At(j)); - fPtHist->Fill(curtrack->GetPt(),1./curtrack->GetPt()); - } - fTracks->Clear("C"); - break; - case TPBReadType::kReadOpt: - // Partial read - b_event_fNtrack->GetEntry(entry); - - fNTracksHist->Fill(fNtrack); - - if (fNtrack>0) { - b_fTracks->GetEntry(entry); - for(Int_t j=0;jGetEntries();j++){ - Track* curtrack = dynamic_cast(fTracks->At(j)); - fPtHist->Fill(curtrack->GetPt(),1./curtrack->GetPt()); - } - fTracks->Clear("C"); - } - break; - case TPBReadType::kReadNo: - // No read - break; - default: - Error("Process", "Read type not supported; %d", fReadType->GetType()); - return kFALSE; - break; - } - } - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// The SlaveTerminate() function is called after all entries or objects -/// have been processed. When running with PROOF SlaveTerminate() is called -/// on each slave server. - -void TSelEvent::SlaveTerminate() -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Terminate() function is the last function to be called during -/// a query. It always runs on the client, it can be used to present -/// the results graphically or save the results to file. - -void TSelEvent::Terminate() -{ -} diff --git a/proof/proofbench/src/TSelEventGen.cxx b/proof/proofbench/src/TSelEventGen.cxx deleted file mode 100644 index c671942eb28d8..0000000000000 --- a/proof/proofbench/src/TSelEventGen.cxx +++ /dev/null @@ -1,493 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TSelEventGen -\ingroup proofbench - -Selector for event file generation. -List of files to be generated for each node is provided by client. -And list of files generated is sent back. -Existing files are reused if not forced to be regenerated. - -*/ - -#define TSelEventGen_cxx - -#include "TSelEventGen.h" -#include "TParameter.h" -#include "TProofNodeInfo.h" -#include "TProofBenchTypes.h" -#include "TProof.h" -#include "TMap.h" -#include "TDSet.h" -#include "TFileInfo.h" -#include "TFile.h" -#include "TSortedList.h" -#include "TRandom.h" -#include "Event.h" -#include "TProofServ.h" -#include "TMacro.h" - -ClassImp(TSelEventGen); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TSelEventGen::TSelEventGen() - : fBaseDir(""), fNEvents(100000), fNTracks(100), fNTracksMax(-1), - fRegenerate(kFALSE), fTotalGen(0), fFilesGenerated(0), - fGenerateFun(0), fChain(0) -{ - if (gProofServ){ - fBaseDir=gProofServ->GetDataDir(); - // Two directories up - fBaseDir.Remove(fBaseDir.Last('/')); - fBaseDir.Remove(fBaseDir.Last('/')); - } - else{ - fBaseDir=""; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Begin() function is called at the start of the query. -/// When running with PROOF Begin() is only called on the client. -/// The tree argument is deprecated (on PROOF 0 is passed). - -void TSelEventGen::Begin(TTree *) -{ - TString option = GetOption(); - // Determine the test type - TMap *filemap = dynamic_cast - (fInput->FindObject("PROOF_FilesToProcess")); - if (filemap) { - //Info("Begin", "dumping the file map:"); - //filemap->Print(); - } else { - if (fInput->FindObject("PROOF_FilesToProcess")) { - Error("Begin", "object 'PROOF_FilesToProcess' found but not a map" - " (%s)", fInput->FindObject("PROOF_FilesToProcess")->ClassName()); - } else { - Error("Begin", "object 'PROOF_FilesToProcess' not found"); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// The SlaveBegin() function is called after the Begin() function. -/// When running with PROOF SlaveBegin() is called on each slave server. -/// The tree argument is deprecated (on PROOF 0 is passed). - -void TSelEventGen::SlaveBegin(TTree *tree) -{ - Init(tree); - - TString option = GetOption(); - - //get parameters - - Bool_t found_basedir=kFALSE; - Bool_t found_nevents=kFALSE; - Bool_t found_ntracks=kFALSE; - Bool_t found_ntrkmax=kFALSE; - Bool_t found_regenerate=kFALSE; - - TIter nxt(fInput); - TString sinput; - TObject *obj; - - while ((obj = nxt())){ - - sinput=obj->GetName(); - //Info("SlaveBegin", "Input list: %s", sinput.Data()); - - if (sinput.Contains("PROOF_BenchmarkBaseDir")){ - TNamed* a = dynamic_cast(obj); - if (a){ - TString bdir = a->GetTitle(); - if (!bdir.IsNull()){ - TUrl u(bdir, kTRUE); - Bool_t isLocal = !strcmp(u.GetProtocol(), "file") ? kTRUE : kFALSE; - if (isLocal && !gSystem->IsAbsoluteFileName(u.GetFile())) - u.SetFile(TString::Format("%s/%s", fBaseDir.Data(), u.GetFile())); - if (isLocal) { - if ((gSystem->AccessPathName(u.GetFile()) && - gSystem->mkdir(u.GetFile(), kTRUE) == 0) || - !gSystem->AccessPathName(u.GetFile(), kWritePermission)) { - // Directory is writable - fBaseDir = u.GetFile(); - Info("SlaveBegin", "using base directory \"%s\"", fBaseDir.Data()); - } else{ - // Directory is not writable or not available, use default directory - Warning("SlaveBegin", "\"%s\" directory is not writable or not existing," - " using default directory: %s", - bdir.Data(), fBaseDir.Data()); - } - } else { - // We assume the user knows what it does - fBaseDir = bdir; - Info("SlaveBegin", "using non local base directory \"%s\"", fBaseDir.Data()); - } - } else{ - Info("SlaveBegin", "using default directory: %s", - fBaseDir.Data()); - } - found_basedir=kTRUE; - } - else{ - Error("SlaveBegin", "PROOF_BenchmarkBaseDir not type TNamed"); - } - continue; - } - if (sinput.Contains("PROOF_BenchmarkNEvents")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fNEvents= a->GetVal(); - found_nevents=kTRUE; - } - else{ - Error("SlaveBegin", "PROOF_BenchmarkEvents not type TParameter" - "*"); - } - continue; - } - if (sinput.Contains("PROOF_BenchmarkNTracks")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fNTracks=a->GetVal(); - found_ntracks=kTRUE; - } - else{ - Error("SlaveBegin", "PROOF_BenchmarkNTracks not type TParameter" - "*"); - } - continue; - } - if (sinput.Contains("PROOF_BenchmarkNTracksMax")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fNTracksMax=a->GetVal(); - found_ntrkmax=kTRUE; - } - else{ - Error("SlaveBegin", "PROOF_BenchmarkNTracksMax not type TParameter" - "*"); - } - continue; - } - if (sinput.Contains("PROOF_BenchmarkRegenerate")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fRegenerate=a->GetVal(); - found_regenerate=kTRUE; - } - else{ - Error("SlaveBegin", "PROOF_BenchmarkRegenerate not type TParameter" - "*"); - } - continue; - } - if (sinput.Contains("PROOF_GenerateFun")){ - TNamed *a = dynamic_cast(obj); - if (!(fGenerateFun = dynamic_cast(fInput->FindObject(a->GetTitle())))) { - Error("SlaveBegin", "PROOF_GenerateFun requires the TMacro object in the input list"); - } - continue; - } - } - - if (!found_basedir){ - Warning("SlaveBegin", "PROOF_BenchmarkBaseDir not found; using default:" - " %s", fBaseDir.Data()); - } - if (!found_nevents){ - Warning("SlaveBegin", "PROOF_BenchmarkNEvents not found; using default:" - " %lld", fNEvents); - } - if (!found_ntracks){ - Warning("SlaveBegin", "PROOF_BenchmarkNTracks not found; using default:" - " %d", fNTracks); - } - if (!found_ntrkmax){ - Warning("SlaveBegin", "PROOF_BenchmarkNTracksMax not found; using default:" - " %d", fNTracksMax); - } else if (fNTracksMax <= fNTracks) { - Warning("SlaveBegin", "PROOF_BenchmarkNTracksMax must be larger then" - " fNTracks=%d ; ignoring", fNTracks); - fNTracksMax = -1; - found_ntrkmax = kFALSE; - } - if (!found_regenerate){ - Warning("SlaveBegin", "PROOF_BenchmarkRegenerate not found; using" - " default: %d", fRegenerate); - } - - fFilesGenerated = new TList(); - - TString hostname(TUrl(gSystem->HostName()).GetHostFQDN()); - TString thisordinal = gProofServ ? gProofServ->GetOrdinal() : "n.d"; - TString sfilegenerated = - TString::Format("PROOF_FilesGenerated_%s_%s", hostname.Data(), thisordinal.Data()); - fFilesGenerated->SetName(sfilegenerated); - fFilesGenerated->SetOwner(kTRUE); -} - -//////////////////////////////////////////////////////////////////////////////// -///Generate files for IO-bound run -///Input parameters -/// filename: The name of the file to be generated -/// sizenevents: Either the number of events to generate when -/// filetype==kPBFileBenchmark -/// or the size of the file to generate when -/// filetype==kPBFileCleanup -///Returns -/// Either Number of entries in the file when -/// filetype==kPBFileBenchmark -/// or bytes written when filetype==kPBFileCleanup -///return 0 in case error - -Long64_t TSelEventGen::GenerateFiles(const char *filename, Long64_t sizenevents) -{ - Long64_t nentries=0; - TDirectory* savedir = gDirectory; - //printf("current dir=%s\n", gDirectory->GetPath()); - - TFile *f = TFile::Open(filename, "RECREATE"); - - savedir->cd(); - - if (!f || f->IsZombie()) return 0; - - Event *event=new Event(); - Event *ep = event; - TTree* eventtree= new TTree("EventTree", "Event Tree"); - eventtree->SetDirectory(f); - - const Int_t buffersize=32000; - eventtree->Branch("event", "Event", &ep, buffersize, 1); - eventtree->AutoSave(); - - Long64_t i=0; - Long64_t size_generated=0; - -// f->SetCompressionLevel(0); //no compression - Int_t ntrks = fNTracks; - - Info("GenerateFiles", "Generating %s", filename); - while (sizenevents--){ - //event->Build(i++,fNTracksBench,0); - if (fNTracksMax > fNTracks) { - // Required to smear the number of tracks between [min,max] - ntrks = fNTracks + gRandom->Integer(fNTracksMax - fNTracks); - } - event->Build(i++, ntrks, 0); - size_generated+=eventtree->Fill(); - } - nentries=eventtree->GetEntries(); - Info("GenerateFiles", "%s generated with %lld entries", filename, nentries); - savedir = gDirectory; - - f = eventtree->GetCurrentFile(); - f->cd(); - eventtree->Write(); - eventtree->SetDirectory(0); - - f->Close(); - delete f; - f = 0; - eventtree->Delete(); - event->Delete(); - savedir->cd(); - - return nentries; -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Process() function is called for each entry in the tree (or possibly -/// keyed object in the case of PROOF) to be processed. The entry argument -/// specifies which entry in the currently loaded tree is to be processed. -/// It can be passed to either TTree::GetEntry() or TBranch::GetEntry() -/// to read either all or the required parts of the data. When processing -/// keyed objects with PROOF, the object is already loaded and is available -/// via the fObject pointer. -/// -/// This function should contain the "body" of the analysis. It can contain -/// simple or elaborate selection criteria, run algorithms on the data -/// of the event and typically fill histograms. - -Bool_t TSelEventGen::Process(Long64_t entry) -{ - // WARNING when a selector is used with a TChain, you must use - // the pointer to the current TTree to call GetEntry(entry). - // The entry is always the local entry number in the current tree. - // Assuming that fChain is the pointer to the TChain being processed, - // use fChain->GetTree()->GetEntry(entry). - - TDSetElement *fCurrent = 0; - TPair *elemPair = 0; - if (fInput && (elemPair = dynamic_cast - (fInput->FindObject("PROOF_CurrentElement")))) { - if ((fCurrent = dynamic_cast(elemPair->Value()))) { - Info("Process", "entry %lld: file: '%s'", entry, fCurrent->GetName()); - } else { - Error("Process", "entry %lld: no file specified!", entry); - return kFALSE; - } - } - - // Generate - TString filename(fCurrent->GetName()); - if (!fBaseDir.IsNull()) { - if (fBaseDir.Contains("")) { - filename = fBaseDir; - filename.ReplaceAll("", fCurrent->GetName()); - } else { - filename.Form("%s/%s", fBaseDir.Data(), fCurrent->GetName()); - } - } - TString fndset(filename); - - // Set the Url for remote access - TString seed = TString::Format("%s/%s", gSystem->HostName(), filename.Data()), dsrv; - TUrl basedirurl(filename, kTRUE); - if (!strcmp(basedirurl.GetProtocol(), "file")) { - TProofServ::GetLocalServer(dsrv); - TProofServ::FilterLocalroot(fndset, dsrv); - } - - //generate files - Long64_t neventstogenerate = fNEvents; - - Long64_t entries_file=0; - Long64_t filesize=0; - Bool_t filefound=kFALSE; - FileStat_t filestat; - TUUID uuid; - if (!fRegenerate && !gSystem->GetPathInfo(filename, filestat)) { //stat'ed - TFile *f = TFile::Open(filename); - if (f && !f->IsZombie()){ - TTree* t = (TTree *) f->Get("EventTree"); - if (t) { - entries_file = t->GetEntries(); - if (entries_file == neventstogenerate) { - // File size seems to be correct, skip generation - Info("Process", "bench file (%s, entries=%lld) exists:" - " skipping generation.", filename.Data(), entries_file); - filesize = f->GetSize(); - uuid = f->GetUUID(); - filefound = kTRUE; - } - } - f->Close(); - } - SafeDelete(f); - } - - // Make sure there is enough space left of the device, if local - TString bdir(fBaseDir); - bdir.ReplaceAll("", ""); - if (!gSystem->AccessPathName(bdir)) { - Long_t devid, devbsz, devbtot, devbfree; - gSystem->GetFsInfo(bdir, &devid, &devbsz, &devbtot, &devbfree); - // Must be more than 10% of space and at least 1 GB - Long_t szneed = 1024 * 1024 * 1024, tomb = 1024 * 1024; - if (devbfree * devbsz < szneed || devbfree < 0.1 * devbtot) { - Error("Process", "not enough free space on device (%ld MB < {%ld, %ld} MB):" - " skipping generation of: %s", - (devbfree * devbsz) / tomb, - szneed / tomb, (Long_t) (0.1 * devbtot * devbsz / tomb), - filename.Data()); - fStatus = TSelector::kAbortFile; - } - } - - if (!filefound) { // Generate - gRandom->SetSeed(static_cast(TMath::Hash(seed))); - if (fGenerateFun) { - TString fargs = TString::Format("\"%s\",%lld", filename.Data(), neventstogenerate); - entries_file = (Long64_t) fGenerateFun->Exec(fargs); - } else { - entries_file = GenerateFiles(filename, neventstogenerate); - } - - TFile *f = TFile::Open(filename); - if (f && !f->IsZombie()) { - filesize = f->GetSize(); - uuid = f->GetUUID(); - f->Close(); - } else { - Error("Process", "can not open generated file: %s", filename.Data()); - fStatus = TSelector::kAbortFile; - return kFALSE; - } - - SafeDelete(f); - } - - // Add meta data to the file info - TFileInfoMeta* fimeta = new TFileInfoMeta("/EventTree", "TTree", entries_file); - TMD5* md5 = 0; - if (!strcmp(TUrl(filename,kTRUE).GetProtocol(), "file")) - md5 = TMD5::FileChecksum(filename); - TString md5s = (md5) ? md5->AsString() : ""; - TFileInfo *fi = new TFileInfo(TString::Format("%s%s", dsrv.Data(), fndset.Data()), - filesize, uuid.AsString(), md5s.Data(), fimeta); - SafeDelete(md5); - - // Mark it as staged - fi->SetBit(TFileInfo::kStaged); - - // Add the fileinfo to the list - if (fFilesGenerated) fFilesGenerated->Add(fi); - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// The SlaveTerminate() function is called after all entries or objects -/// have been processed. When running with PROOF SlaveTerminate() is called -/// on each slave server - -void TSelEventGen::SlaveTerminate() -{ - if (fFilesGenerated && fFilesGenerated->GetSize() > 0) { - fOutput->Add(fFilesGenerated); - Info("SlaveTerminate", - "list '%s' of files generated by this worker added to the output list", - fFilesGenerated->GetName()); - } else { - if (!fFilesGenerated) { - Warning("SlaveTerminate", "no list of generated files defined!"); - } else { - Warning("SlaveTerminate", "list of generated files is empty!"); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Terminate() function is the last function to be called during -/// a query. It always runs on the client, it can be used to present -/// the results graphically or save the results to file. - -void TSelEventGen::Terminate() -{ -} - -//////////////////////////////////////////////////////////////////////////////// - -void TSelEventGen::Print(Option_t *) const -{ - Printf("fNEvents=%lld", fNEvents); - Printf("fBaseDir=%s", fBaseDir.Data()); - Printf("fNTracks=%d", fNTracks); - Printf("fRegenerate=%d", fRegenerate); -} - diff --git a/proof/proofbench/src/TSelHandleDataSet.cxx b/proof/proofbench/src/TSelHandleDataSet.cxx deleted file mode 100644 index fb2dea4250139..0000000000000 --- a/proof/proofbench/src/TSelHandleDataSet.cxx +++ /dev/null @@ -1,227 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TSelHandleDataSet -\ingroup proofbench - -PROOF selector for file cache release. -List of files to be cleaned for each node is provided by client. - -*/ - -#define TSelHandleDataSet_cxx - -#include "TSelHandleDataSet.h" - -#include "TDSet.h" -#include "TEnv.h" -#include "TFile.h" -#include "TMap.h" -#include "TParameter.h" -#include "TProofBenchTypes.h" -#include "TSystem.h" -#include "TUrl.h" -#include "errno.h" -#include -#include - - -ClassImp(TSelHandleDataSet); - -//////////////////////////////////////////////////////////////////////////////// -/// Init the type from the input parameters - -void TSelHandleDataSet::SlaveBegin(TTree *) -{ - - TObject *o = fInput->FindObject("PROOF_Benchmark_HandleDSType"); - if (o) fType = dynamic_cast(o); - - TNamed *n = dynamic_cast(fInput->FindObject("PROOF_Benchmark_DestDir")); - if (n) { - fDestDir = n->GetTitle(); - if (gSystem->AccessPathName(fDestDir)) { - // Create the directory - if (gSystem->mkdir(fDestDir, kTRUE) != 0) { - fDestDir = ""; - Error("SlaveBegin", "could not create dir '%s'", fDestDir.Data()); - } else { - if (gSystem->AccessPathName(fDestDir, kWritePermission)) { - fDestDir = ""; - Error("SlaveBegin", "dir '%s' is not writable by this process", fDestDir.Data()); - } else { - Info("SlaveBegin", "dir '%s' successfully created", fDestDir.Data()); - } - } - } - } - - // Use default if nothing found in the input list - if (!fType) fType = new TPBHandleDSType(TPBHandleDSType::kReleaseCache); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Release the memory cache associated with file 'fn'. - -void TSelHandleDataSet::ReleaseCache(const char *fn) -{ -#if defined(R__LINUX) - TString filename(fn); - Int_t fd; - fd = open(filename.Data(), O_RDONLY); - if (fd > -1) { - fdatasync(fd); - posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED); - close(fd); - Info("ReleaseCache", "file cache for file '%s' cleaned ...", filename.Data()); - } else { - Error("ReleaseCache", "cannot open file '%s' for cache clean up; errno=%d", - filename.Data(), errno); - } -#else - Info("ReleaseCache", "dummy function: file '%s' untouched ...", fn); -#endif - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check the memory cache associated with file 'fn'. - -void TSelHandleDataSet::CheckCache(const char * /*fn*/) -{ - Warning("CheckCache", "cache checker not implemented yet"); - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Physically remove the file 'fn'. - -void TSelHandleDataSet::RemoveFile(const char *fn) -{ - if (!gSystem->AccessPathName(fn, kWritePermission)) { - if (gSystem->Unlink(fn) != 0) { - Error("RemoveFile", "problems removing file '%s' ...", fn); - } else { - if (!gSystem->AccessPathName(fn)) - Warning("RemoveFile", "'unlink' returned success but the file still exists ('%s')", fn); - } - } else { - if (!gSystem->AccessPathName(fn)) { - Error("RemoveFile", "file '%s' cannot removed by this process", fn); - } else { - Error("RemoveFile", "file '%s' does not exists", fn); - } - } - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Copy file 'fn' to fDestDir - -void TSelHandleDataSet::CopyFile(const char *fn) -{ - // Check if we have a destination dir - if (fDestDir.IsNull()) { - Error("CopyFile", "destination dir undefined: file '%s' not copied", fn); - return; - } - - TString basefn = gSystem->BaseName(TUrl(fn, kTRUE).GetFile()); - TString dst = TString::Format("%s/%s", fDestDir.Data(), basefn.Data()); - if (!TFile::Cp(fn, dst.Data())) { - Error("CopyFile", "problems copying file '%s' to '%s'", fn, dst.Data()); - return; - } - Info("CopyFile", "file '%s' created ...", dst.Data()); - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Process() function is called for each entry in the tree (or possibly -/// keyed object in the case of PROOF) to be processed. The entry argument -/// specifies which entry in the currently loaded tree is to be processed. -/// It can be passed to either TTree::GetEntry() or TBranch::GetEntry() -/// to read either all or the required parts of the data. When processing -/// keyed objects with PROOF, the object is already loaded and is available -/// via the fObject pointer. -/// -/// This function should contain the "body" of the analysis. It can contain -/// simple or elaborate selection criteria, run algorithms on the data -/// of the event and typically fill histograms. - -Bool_t TSelHandleDataSet::Process(Long64_t entry) -{ - // WARNING when a selector is used with a TChain, you must use - // the pointer to the current TTree to call GetEntry(entry). - // The entry is always the local entry number in the current tree. - // Assuming that fChain is the pointer to the TChain being processed, - // use fChain->GetTree()->GetEntry(entry). - - TDSetElement *fCurrent = 0; - TPair *elemPair = 0; - if (fInput && (elemPair = dynamic_cast - (fInput->FindObject("PROOF_CurrentElement")))) { - if ((fCurrent = dynamic_cast(elemPair->Value()))) { - Info("Process", "entry %lld: file: '%s'", entry, fCurrent->GetName()); - } else { - Error("Process", "entry %lld: no file specified!", entry); - return kFALSE; - } - } - - // Resolve the file type; this also adjusts names for Xrd based systems - TUrl url(fCurrent->GetName()); - url.SetAnchor(0); - TString lfname = gEnv->GetValue("Path.Localroot", ""); - TFile::EFileType type = TFile::GetType(url.GetUrl(), "", &lfname); - if (type == TFile::kLocal && - strcmp(url.GetProtocol(),"root") && strcmp(url.GetProtocol(),"xrd")) - lfname = url.GetFileAndOptions(); - - if (fType->GetType() == TPBHandleDSType::kReleaseCache) { - // Release the file cache - if (type == TFile::kLocal) { - ReleaseCache(lfname); - } else if (type == TFile::kFile) { - ReleaseCache(url.GetFile()); - } else { - Error("Process", - "attempt to call ReleaseCache for a non-local file: '%s'", url.GetUrl()); - } - } else if (fType->GetType() == TPBHandleDSType::kCheckCache) { - // Check the file cache - if (type == TFile::kLocal) { - CheckCache(lfname); - } else if (type == TFile::kFile) { - CheckCache(url.GetFile()); - } else { - Error("Process", - "attempt to call CheckCache for a non-local file: '%s'", url.GetUrl()); - } - } else if (fType->GetType() == TPBHandleDSType::kRemoveFiles) { - // Remove the file - RemoveFile(url.GetFileAndOptions()); - } else if (fType->GetType() == TPBHandleDSType::kCopyFiles) { - // Copy file - CopyFile(url.GetFileAndOptions()); - } else { - // Type unknown - Warning("Process", "type: %d is unknown", fType->GetType()); - } - - return kTRUE; -} - diff --git a/proof/proofbench/src/TSelHist.cxx b/proof/proofbench/src/TSelHist.cxx deleted file mode 100644 index d7f0b3ed0ce1e..0000000000000 --- a/proof/proofbench/src/TSelHist.cxx +++ /dev/null @@ -1,388 +0,0 @@ -// @(#)root/proof:$Id$ -// Author: Sangsu Ryu 22/06/2010 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -/** \class TSelHist -\ingroup proofbench - -PROOF selector for CPU-intensive benchmark test. -Events are generated and 1-D, 2-D, and/or 3-D histograms are filled. - -*/ - -#define TSelHist_cxx - -#include "TSelHist.h" -#include "TProofBenchTypes.h" -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -ClassImp(TSelHist); - -//////////////////////////////////////////////////////////////////////////////// -///Constructor - -TSelHist::TSelHist() - : fHistType(0), fNHists(16), fDraw(0), fHist1D(0), fHist2D(0), fHist3D(0), - fRandom(0), fCHist1D(0), fCHist2D(0), fCHist3D(0) -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TSelHist::~TSelHist() -{ - //if (fRandom) delete fRandom; - SafeDelete(fRandom); - - // Info("TSelHist","destroying ..."); - - if (!fDraw){ - for (Int_t i=0; i < fNHists; i++) { - if (fHist1D && fHist1D[i] && !fOutput->FindObject(fHist1D[i])) { - SafeDelete(fHist1D[i]); - } - if (fHist2D && fHist2D[i] && !fOutput->FindObject(fHist2D[i])) { - SafeDelete(fHist2D[i]); - } - if (fHist3D && fHist3D[i] && !fOutput->FindObject(fHist3D[i])) { - SafeDelete(fHist3D[i]); - } - } - } - SafeDelete(fHist1D); - SafeDelete(fHist2D); - SafeDelete(fHist3D); -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Begin() function is called at the start of the query. -/// When running with PROOF Begin() is only called on the client. -/// The tree argument is deprecated (on PROOF 0 is passed). - -void TSelHist::Begin(TTree * /*tree*/) -{ - TString option = GetOption(); - - Bool_t found_histtype=kFALSE; - Bool_t found_nhists=kFALSE; - Bool_t found_draw=kFALSE; - - TIter nxt(fInput); - TString sinput; - TObject *obj; - - - while ((obj = nxt())){ - sinput=obj->GetName(); - //Info("Begin", "object name=%s", sinput.Data()); - if (sinput.Contains("PROOF_Benchmark_HistType")){ - if ((fHistType = dynamic_cast(obj))) found_histtype = kTRUE; - continue; - } - if (sinput.Contains("PROOF_BenchmarkNHists")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fNHists= a->GetVal(); - found_nhists=kTRUE; - //Info("Begin", "PROOF_BenchmarkNHists=%d", fNHists); - } - else{ - Error("Begin", "PROOF_BenchmarkNHists not type TParameter*"); - } - continue; - } - if (sinput.Contains("PROOF_BenchmarkDraw")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fDraw= a->GetVal(); - found_draw=kTRUE; - //Info("Begin", "PROOF_BenchmarkDraw=%d", fDraw); - } - else{ - Error("Begin", "PROOF_BenchmarkDraw not type TParameter*"); - } - continue; - } - } - - if (!found_histtype){ - fHistType = new TPBHistType(TPBHistType::kHist1D); - Warning("Begin", "PROOF_Benchmark_HistType not found; using default: %d", - (Int_t) fHistType->GetType()); - } - if (!found_nhists){ - Warning("Begin", "PROOF_BenchmarkNHists not found; using default: %d", - fNHists); - } - if (!found_draw){ - Warning("Begin", "PROOF_BenchmarkDraw not found; using default: %d", - fDraw); - } - - if (fDraw) { - if (fHistType->GetType() & TPBHistType::kHist1D) fHist1D = new TH1F*[fNHists]; - if (fHistType->GetType() & TPBHistType::kHist2D) fHist2D = new TH2F*[fNHists]; - if (fHistType->GetType() & TPBHistType::kHist3D) fHist3D = new TH3F*[fNHists]; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// The SlaveBegin() function is called after the Begin() function. -/// When running with PROOF SlaveBegin() is called on each slave server. -/// The tree argument is deprecated (on PROOF 0 is passed). - -void TSelHist::SlaveBegin(TTree * /*tree*/) -{ - TString option = GetOption(); - - Bool_t found_histtype=kFALSE; - Bool_t found_nhists=kFALSE; - Bool_t found_draw=kFALSE; - - TIter nxt(fInput); - TString sinput; - TObject *obj; - - while ((obj = nxt())){ - sinput=obj->GetName(); - //Info("SlaveBegin", "object name=%s", sinput.Data()); - if (sinput.Contains("PROOF_Benchmark_HistType")){ - if ((fHistType = dynamic_cast(obj))) found_histtype = kTRUE; - continue; - } - if (sinput.Contains("PROOF_BenchmarkNHists")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fNHists=a->GetVal(); - found_nhists=kTRUE; - //Info("SlaveBegin", "PROOF_BenchmarkNHists=%d", fNHists); - } - else{ - Error("SlaveBegin", "PROOF_BenchmarkNHists not type TParameter" - "*"); - } - continue; - } - if (sinput.Contains("PROOF_BenchmarkDraw")){ - TParameter* a=dynamic_cast*>(obj); - if (a){ - fDraw=a->GetVal(); - found_draw=kTRUE; - //Info("SlaveBegin", "PROOF_BenchmarkDraw=%d", fDraw); - } - else{ - Error("SlaveBegin", "PROOF_BenchmarkDraw not type TParameter" - "*"); - } - continue; - } - } - - if (!found_histtype){ - fHistType = new TPBHistType(TPBHistType::kHist1D); - Warning("SlaveBegin", "PROOF_Benchmark_HistType not found; using default: %d", - fHistType->GetType()); - } - if (!found_nhists){ - Warning("SlaveBegin", "PROOF_BenchmarkNHists not found; using default: %d", - fNHists); - } - if (!found_draw){ - Warning("SlaveBegin", "PROOF_BenchmarkDraw not found; using default: %d", - fDraw); - } - - // Create the histogram - if (fHistType->GetType() & TPBHistType::kHist1D){ - fHist1D = new TH1F*[fNHists]; - for (Int_t i=0; i < fNHists; i++) { - fHist1D[i] = new TH1F(Form("h1d_%d",i), Form("h1d_%d",i), 100, -3., 3.); - fHist1D[i]->SetFillColor(kRed); - if (fDraw) fOutput->Add(fHist1D[i]); - } - } - if (fHistType->GetType() & TPBHistType::kHist2D){ - fHist2D = new TH2F*[fNHists]; - for (Int_t i=0; i < fNHists; i++) { - fHist2D[i] = new TH2F(Form("h2d_%d",i), Form("h2d_%d",i), 100, -3., 3., - 100, -3., 3.); - fHist2D[i]->SetFillColor(kRed); - if (fDraw) fOutput->Add(fHist2D[i]); - } - } - if (fHistType->GetType() & TPBHistType::kHist3D){ - fHist3D = new TH3F*[fNHists]; - for (Int_t i=0; i < fNHists; i++) { - fHist3D[i] = new TH3F(Form("h3d_%d",i), Form("h3d_%d",i), 100, -3., 3., - 100, -3., 3., 100, -3., 3.); - fHist3D[i]->SetFillColor(kRed); - if (fDraw) fOutput->Add(fHist3D[i]); - } - } - // Set random seed - fRandom = new TRandom3(0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Process() function is called for each entry in the tree (or possibly -/// keyed object in the case of PROOF) to be processed. The entry argument -/// specifies which entry in the currently loaded tree is to be processed. -/// It can be passed to either TSelHist::GetEntry() or TBranch::GetEntry() -/// to read either all or the required parts of the data. When processing -/// keyed objects with PROOF, the object is already loaded and is available -/// via the fObject pointer. -/// -/// This function should contain the "body" of the analysis. It can contain -/// simple or elaborate selection criteria, run algorithms on the data -/// of the event and typically fill histograms. -/// -/// The processing can be stopped by calling Abort(). -/// -/// Use fStatus to set the return value of TTree::Process(). -/// -/// The return value is currently not used. - -Bool_t TSelHist::Process(Long64_t) -{ - Double_t x, y, z; - if (fHistType->GetType() & TPBHistType::kHist1D){ - for (Int_t i=0; i < fNHists; i++) { - if (fRandom && fHist1D[i]) { - x = fRandom->Gaus(0.,1.); - fHist1D[i]->Fill(x); - } - } - } - if (fHistType->GetType() & TPBHistType::kHist2D){ - for (Int_t i=0; i < fNHists; i++) { - if (fRandom && fHist2D[i]) { - x = fRandom->Gaus(0.,1.); - y = fRandom->Gaus(0.,1.); - fHist2D[i]->Fill(x, y); - } - } - } - if (fHistType->GetType() & TPBHistType::kHist3D){ - for (Int_t i=0; i < fNHists; i++) { - if (fRandom && fHist3D[i]) { - x = fRandom->Gaus(0.,1.); - y = fRandom->Gaus(0.,1.); - z = fRandom->Gaus(0.,1.); - fHist3D[i]->Fill(x, y, z); - } - } - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// The SlaveTerminate() function is called after all entries or objects -/// have been processed. When running with PROOF SlaveTerminate() is called -/// on each slave server. - -void TSelHist::SlaveTerminate() -{ -} - -//////////////////////////////////////////////////////////////////////////////// -/// The Terminate() function is the last function to be called during -/// a query. It always runs on the client, it can be used to present -/// the results graphically or save the results to file. - -void TSelHist::Terminate() -{ - // - // Create a canvas, with 100 pads - // - - if (!fDraw || gROOT->IsBatch()){ - return; - } - - if (fHistType->GetType() & TPBHistType::kHist1D){ - fCHist1D=dynamic_cast(gROOT->FindObject("CHist1D")); - if (!fCHist1D){ - fCHist1D = new TCanvas("CHist1D","Proof TSelHist Canvas (1D)", 200, 10, - 700,700); - Int_t nside = (Int_t)TMath::Sqrt((Float_t)fNHists); - nside = (nside*nside < fNHists) ? nside+1 : nside; - fCHist1D->Divide(nside,nside,0,0); - } - - for (Int_t i=0; i < fNHists; i++) { - fHist1D[i] = dynamic_cast - (fOutput->FindObject(Form("h1d_%d",i))); - fCHist1D->cd(i+1); - if (fHist1D[i]) fHist1D[i]->Draw(); - } - // Final update - fCHist1D->cd(); - fCHist1D->Update(); - } - if (fHistType->GetType() & TPBHistType::kHist2D){ - fCHist2D=dynamic_cast(gROOT->FindObject("CHist2D")); - if (!fCHist2D){ - fCHist2D = new TCanvas("CHist2D","Proof TSelHist Canvas (2D)", 200, 10, - 700,700); - Int_t nside = (Int_t)TMath::Sqrt((Float_t)fNHists); - nside = (nside*nside < fNHists) ? nside+1 : nside; - fCHist2D->Divide(nside,nside,0,0); - } - for (Int_t i=0; i < fNHists; i++) { - fHist2D[i] = dynamic_cast - (fOutput->FindObject(Form("h2d_%d",i))); - fCHist2D->cd(i+1); - if (fHist2D[i]) fHist2D[i]->Draw("SURF"); - } - // Final update - fCHist2D->cd(); - fCHist2D->Update(); - } - - if (fHistType->GetType() & TPBHistType::kHist3D){ - fCHist3D=dynamic_cast(gROOT->FindObject("CHist3D")); - if (!fCHist3D){ - fCHist3D = new TCanvas("CHist3D","Proof TSelHist Canvas (3D)", 200, 10, - 700,700); - Int_t nside = (Int_t)TMath::Sqrt((Float_t)fNHists); - nside = (nside*nside < fNHists) ? nside+1 : nside; - fCHist3D->Divide(nside,nside,0,0); - } - - fOutput->Print("a"); - for (Int_t i=0; i < fNHists; i++) { - fHist3D[i] = dynamic_cast - (fOutput->FindObject(Form("h3d_%d",i))); - fCHist3D->cd(i+1); - if (fHist3D[i]) printf("fHist3D[%d] found\n", i); - if (fHist3D[i]) fHist3D[i]->Draw(); - } - // Final update - fCHist3D->cd(); - fCHist3D->Update(); - } - -} diff --git a/proof/proofplayer/CMakeLists.txt b/proof/proofplayer/CMakeLists.txt deleted file mode 100644 index eb47af4dad8ed..0000000000000 --- a/proof/proofplayer/CMakeLists.txt +++ /dev/null @@ -1,72 +0,0 @@ -# Copyright (C) 1995-2019, Rene Brun and Fons Rademakers. -# All rights reserved. -# -# For the licensing terms see $ROOTSYS/LICENSE. -# For the list of contributors see $ROOTSYS/README/CREDITS. - -############################################################################ -# CMakeLists.txt file for building ROOT proof/proofplayer package -# @author Pere Mato, CERN -############################################################################ - -ROOT_STANDARD_LIBRARY_PACKAGE(ProofPlayer - NO_INSTALL_HEADERS - HEADERS - TDrawFeedback.h - TEventIter.h - TOutputListSelectorDataMap.h - TPacketizerAdaptive.h - TPacketizerFile.h - TPacketizer.h - TPacketizerMulti.h - TPacketizerUnit.h - TPerfStats.h - TProofLimitsFinder.h - TProofMonSender.h - TProofMonSenderSQL.h - TProofPlayer.h - TProofPlayerLite.h - TStatsFeedback.h - TStatus.h - SOURCES - TDrawFeedback.cxx - TEventIter.cxx - TOutputListSelectorDataMap.cxx - TPacketizerAdaptive.cxx - TPacketizer.cxx - TPacketizerFile.cxx - TPacketizerMulti.cxx - TPacketizerUnit.cxx - TPerfStats.cxx - TProofLimitsFinder.cxx - TProofMonSender.cxx - TProofMonSenderSQL.cxx - TProofPlayer.cxx - TProofPlayerLite.cxx - TStatsFeedback.cxx - TStatus.cxx - DEPENDENCIES - Hist - Net - MathCore - Proof - RIO - Thread - Tree - TreePlayer -) - -ROOT_STANDARD_LIBRARY_PACKAGE(ProofDraw - NO_INSTALL_HEADERS - HEADERS - TProofDraw.h - SOURCES - TProofDraw.cxx - LINKDEF - LinkDefDraw.h - DEPENDENCIES - ProofPlayer - TreePlayer -) - -ROOT_INSTALL_HEADERS() diff --git a/proof/proofplayer/doc/index.txt b/proof/proofplayer/doc/index.txt deleted file mode 100644 index d855ca2a2c2b4..0000000000000 --- a/proof/proofplayer/doc/index.txt +++ /dev/null @@ -1,10 +0,0 @@ -This directory contains the internal classes used for PROOF processing. -BEGIN_HTML - See: - -END_HTML diff --git a/proof/proofplayer/inc/LinkDef.h b/proof/proofplayer/inc/LinkDef.h deleted file mode 100644 index 0815cda1f5786..0000000000000 --- a/proof/proofplayer/inc/LinkDef.h +++ /dev/null @@ -1,50 +0,0 @@ -/* @(#)root/proofplayer:$Id$ */ - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifdef __CLING__ - -#pragma link off all globals; -#pragma link off all classes; -#pragma link off all functions; - -#pragma link C++ nestedtypedefs; -#pragma link C++ nestedclasses; - -#pragma link C++ class TProofPlayer+; -#pragma link C++ class TProofPlayerLite+; -#pragma link C++ class TProofPlayerLocal+; -#pragma link C++ class TProofPlayerRemote+; -#pragma link C++ class TProofPlayerSlave+; -#pragma link C++ class TProofPlayerSuperMaster+; - -#pragma link C++ class TPacketizer+; -#pragma link C++ class TPacketizerUnit+; -#pragma link C++ class TPacketizerAdaptive+; -#pragma link C++ class TPacketizerMulti+; -#pragma link C++ class TPacketizerFile+; - -#pragma link C++ class TEventIter+; -#pragma link C++ class TEventIterUnit+; -#pragma link C++ class TEventIterObj+; -#pragma link C++ class TEventIterTree+; - -#pragma link C++ class TPerfStats; -#pragma link C++ class TPerfEvent+; -#pragma link C++ class TProofMonSender+; -#pragma link C++ class TProofMonSenderSQL+; - -#pragma link C++ class TProofLimitsFinder; -#pragma link C++ class TDrawFeedback+; -#pragma link C++ class TStatsFeedback+; -#pragma link C++ class TStatus-; - -#pragma link C++ class TOutputListSelectorDataMap+; - -#endif diff --git a/proof/proofplayer/inc/LinkDefDraw.h b/proof/proofplayer/inc/LinkDefDraw.h deleted file mode 100644 index e717fd06d0171..0000000000000 --- a/proof/proofplayer/inc/LinkDefDraw.h +++ /dev/null @@ -1,37 +0,0 @@ -/* @(#)root/proofplayer:$Id$ */ - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifdef __CLING__ - -#pragma link off all globals; -#pragma link off all classes; -#pragma link off all functions; - -#pragma link C++ nestedtypedefs; -#pragma link C++ nestedclasses; - -#pragma link C++ class TProofDraw+; -#pragma link C++ class TProofDrawEntryList+; -#pragma link C++ class TProofDrawEventList+; -#pragma link C++ class TProofDrawHist+; -#pragma link C++ class TProofDrawProfile+; -#pragma link C++ class TProofDrawProfile2D+; -#pragma link C++ class TProofDrawGraph+; -#pragma link C++ class TProofDrawPolyMarker3D+; -#pragma link C++ class TProofDrawListOfGraphs+; -#pragma link C++ class TProofDrawListOfPolyMarkers3D+; -#pragma link C++ class TProofDrawListOfGraphs::Point3D_t+; -#pragma link C++ class TProofDrawListOfPolyMarkers3D::Point4D_t+; -#pragma link C++ class std::vector+; -#pragma link C++ class std::vector+; -#pragma link C++ class TProofVectorContainer+; -#pragma link C++ class TProofVectorContainer+; - -#endif diff --git a/proof/proofplayer/inc/TDrawFeedback.h b/proof/proofplayer/inc/TDrawFeedback.h deleted file mode 100644 index 9a5232a3a9695..0000000000000 --- a/proof/proofplayer/inc/TDrawFeedback.h +++ /dev/null @@ -1,57 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 28/10/2003 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TDrawFeedback -#define ROOT_TDrawFeedback - - -////////////////////////////////////////////////////////////////////////// -// // -// TDrawFeedback // -// // -// Utility class to draw objects in the feedback list during queries. // -// Draws histograms in separated canvases and user-defined objects via // -// Draw(). Users requiring advanced treatment should implement their // -// own version following this example. See also TStatsFeedback. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TQObject.h" -#include "TObject.h" - -class TProof; -class TProof; -class THashList; -class TSeqCollection; - -class TDrawFeedback : public TObject, public TQObject { -private: - TString fName; // Identfier for this object - Bool_t fAll; //draw all or selected objects - THashList *fNames; //selected objects - -protected: - Option_t *fOption; //draw option - TProof *fProof; //handle to PROOF session - -public: - TDrawFeedback(TProof *proof = 0, TSeqCollection *names = 0); - ~TDrawFeedback() override; - - void Feedback(TList *objs); - const char *GetName() const override { return fName.Data(); } - ULong_t Hash() const override { return fName.Hash(); } - void SetOption(Option_t *option) { fOption = option; } - - ClassDefOverride(TDrawFeedback,0) // Present PROOF query feedback -}; - -#endif diff --git a/proof/proofplayer/inc/TEventIter.h b/proof/proofplayer/inc/TEventIter.h deleted file mode 100644 index 3ca777b28af13..0000000000000 --- a/proof/proofplayer/inc/TEventIter.h +++ /dev/null @@ -1,191 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 07/01/02 -// Modified: Long Tran-Thanh 04/09/07 (Addition of TEventIterUnit) - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TEventIter -#define ROOT_TEventIter - -////////////////////////////////////////////////////////////////////////// -// // -// TEventIter // -// // -// Special iterator class used in TProofPlayer to iterate over events // -// or objects in the packets. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TNamed.h" -#include "TString.h" - -class TDSet; -class TDSetElement; -class TFile; -class TDirectory; -class TSelector; -class TList; -class TIter; -class TTree; -class TTreeCache; -class TEventList; -class TEntryList; - -//------------------------------------------------------------------------ - -class TEventIter : public TObject { - -public: - enum EIterType { kData = 15}; // True if iterating over data - -protected: - TDSet *fDSet; // data set over which to iterate - - TDSetElement *fElem; // Current Element - - TString fFilename; // Name of the current file - TFile *fFile; // Current file - Long64_t fOldBytesRead; // last reported number of bytes read - TString fPath; // Path to current TDirectory - TDirectory *fDir; // directory containing the objects or the TTree - Long64_t fElemFirst; // first entry to process for this element - Long64_t fElemNum; // number of entries to process for this element - Long64_t fElemCur; // current entry for this element - - TSelector *fSel; // selector to be used - Long64_t fFirst; // first entry to process - Long64_t fNum; // number of entries to process - Long64_t fCur; // current entry - Bool_t fStop; // termination of run requested - TEventList *fEventList; //! eventList for processing - Int_t fEventListPos; //! current position in the eventList - TEntryList *fEntryList; //! entry list for processing - Long64_t fEntryListPos; //! current position in the entrylist - - TList *fPackets; // list of packets processed packets - - Int_t LoadDir(); // Load the directory pointed to by fElem - virtual void PreProcessEvent(Long64_t) { } - -public: - TEventIter(); - TEventIter(TDSet *dset, TSelector *sel, Long64_t first, Long64_t num); - ~TEventIter() override; - - virtual Long64_t GetCacheSize() = 0; - virtual Int_t GetLearnEntries() = 0; - virtual Long64_t GetNextEvent() = 0; - virtual Int_t GetNextPacket(Long64_t &first, Long64_t &num) = 0; - virtual void InvalidatePacket(); - virtual Long64_t GetEntryNumber(Long64_t); - virtual void StopProcess(Bool_t abort); - - TList *GetPackets() { return fPackets; } - - static TEventIter *Create(TDSet *dset, TSelector *sel, Long64_t first, Long64_t num); - - ClassDefOverride(TEventIter,0) // Event iterator used by TProofPlayer's -}; - - -//------------------------------------------------------------------------ - -class TEventIterUnit : public TEventIter { - -private: - Long64_t fNum; - Long64_t fCurrent; - - -public: - TEventIterUnit(); - TEventIterUnit(TDSet *dset, TSelector *sel, Long64_t num); - ~TEventIterUnit() override { } - - Long64_t GetCacheSize() override {return -1;} - Int_t GetLearnEntries() override {return -1;} - Long64_t GetNextEvent() override; - Int_t GetNextPacket(Long64_t &first, Long64_t &num) override; - - ClassDefOverride(TEventIterUnit,0) // Event iterator for objects -}; - - -//------------------------------------------------------------------------ - -class TEventIterObj : public TEventIter { - -private: - TString fClassName; // class name of objects to iterate over - TList *fKeys; // list of keys - TIter *fNextKey; // next key in directory - TObject *fObj; // object found - -protected: - void PreProcessEvent(Long64_t) override; - -public: - TEventIterObj(); - TEventIterObj(TDSet *dset, TSelector *sel, Long64_t first, Long64_t num); - ~TEventIterObj() override; - - Long64_t GetCacheSize() override {return -1;} - Int_t GetLearnEntries() override {return -1;} - Long64_t GetNextEvent() override; - Int_t GetNextPacket(Long64_t &first, Long64_t &num) override; - - ClassDefOverride(TEventIterObj,0) // Event iterator for objects -}; - - -//------------------------------------------------------------------------ -class TEventIterTree : public TEventIter { - -private: - TString fTreeName; // name of the tree object to iterate over - TTree *fTree; // tree we are iterating over - TTreeCache *fTreeCache; // instance of the tree cache for the tree - Bool_t fTreeCacheIsLearning; // Whether cache is in learning phase - Bool_t fUseTreeCache; // Control usage of the tree cache - Long64_t fCacheSize; // Cache size - Bool_t fUseParallelUnzip; // Control usage of parallel unzip - Bool_t fDontCacheFiles; // Control OS caching of read files (Mac Os X only) - TList *fFileTrees; // Files && Trees currently open - - // Auxilliary class to keep track open files and loaded trees - class TFileTree : public TNamed { - public: - Bool_t fUsed; - Bool_t fIsLocal; - TFile *fFile; - TList *fTrees; - TFileTree(const char *name, TFile *f, Bool_t islocal); - ~TFileTree() override; - }; - - TTree* Load(TDSetElement *elem, Bool_t &localfile, const char *objname = 0); - TTree* GetTrees(TDSetElement *elem); - -protected: - void PreProcessEvent(Long64_t ent) override; - -public: - TEventIterTree(); - TEventIterTree(TDSet *dset, TSelector *sel, Long64_t first, Long64_t num); - ~TEventIterTree() override; - - Long64_t GetCacheSize() override; - Int_t GetLearnEntries() override; - Long64_t GetNextEvent() override; - Int_t GetNextPacket(Long64_t &first, Long64_t &num) override; - - ClassDefOverride(TEventIterTree,0) // Event iterator for Trees -}; - -#endif diff --git a/proof/proofplayer/inc/TOutputListSelectorDataMap.h b/proof/proofplayer/inc/TOutputListSelectorDataMap.h deleted file mode 100644 index 462bfedfb32c0..0000000000000 --- a/proof/proofplayer/inc/TOutputListSelectorDataMap.h +++ /dev/null @@ -1,52 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Axel Naumann 2010-06-09 - -/************************************************************************* - * Copyright (C) 1995-2010, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TOutputListSelectorDataMap -#define ROOT_TOutputListSelectorDataMap - -////////////////////////////////////////////////////////////////////////// -// // -// TOutputListSelectorDataMap // -// // -// Set the selector's data members to the corresponding elements of the // -// output list. // -// // -////////////////////////////////////////////////////////////////////////// - - -#include "TObject.h" - -class TSelector; -class TCollection; - -class TOutputListSelectorDataMap: public TObject { -public: - - TOutputListSelectorDataMap(TSelector* sel = 0); - ~TOutputListSelectorDataMap() override {} - - static TOutputListSelectorDataMap* FindInList(TCollection* coll); - - const char* GetName() const override; - - Bool_t Init(TSelector* sel); - Bool_t SetDataMembers(TSelector* sel) const; - Bool_t Merge(TObject* obj); - - TCollection* GetMap() const { return fMap; } - -private: - TCollection* fMap; - ClassDefOverride(TOutputListSelectorDataMap, 1) // Converter from output list to TSelector data members -}; - - -#endif diff --git a/proof/proofplayer/inc/TPacketizer.h b/proof/proofplayer/inc/TPacketizer.h deleted file mode 100644 index da5b8fe51b6ee..0000000000000 --- a/proof/proofplayer/inc/TPacketizer.h +++ /dev/null @@ -1,103 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 18/03/02 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TPacketizer -#define ROOT_TPacketizer - -////////////////////////////////////////////////////////////////////////// -// // -// TPacketizer // -// // -// This class generates packets to be processed on PROOF slave servers. // -// A packet is an event range (begin entry and number of entries) or // -// object range (first object and number of objects) in a TTree // -// (entries) or a directory (objects) in a file. // -// Packets are generated taking into account the performance of the // -// remote machine, the time it took to process a previous packet on // -// the remote machine, the locality of the database files, etc. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TVirtualPacketizer.h" - - -class TMessage; -class TTimer; -class TTree; -class TMap; -class TProofProgressStatus; -class TProofStats; - -class TPacketizer : public TVirtualPacketizer { - -public: // public because of Sun CC bug - class TFileNode; - class TFileStat; - class TSlaveStat; - -private: - TList *fPackets; // all processed packets - - TList *fFileNodes; // nodes with files - TList *fUnAllocated; // nodes with unallocated files - TList *fActive; // nodes with unfinished files - - Long64_t fPacketSize; // global base packet size - // It can be set with PROOF_PacketSize - // parameter, in the input list. - Int_t fMaxPerfIdx; // maximum of our slaves' performance index - - Long_t fMaxSlaveCnt; // maximum number of workers per filenode (Long_t to avoid - // warnings from backward compatibility support) - Int_t fPacketAsAFraction; // used to calculate the packet size - // fPacketSize = fTotalEntries / (fPacketAsAFraction * nslaves) - // fPacketAsAFraction can be interpreted as follows: - // assuming all slaves have equal processing rate, packet size - // is (#events processed by 1 slave) / fPacketSizeAsAFraction. - // It can be set with PROOF_PacketAsAFraction in input list. - - // Add workers controls - Bool_t fHeuristicPSiz; // Whether the packet size is calculated heuristically - Bool_t fDefMaxWrkNode; // Whether the default is used for the max workers per node - - TPacketizer(); - TPacketizer(const TPacketizer&); // no implementation, will generate - void operator=(const TPacketizer&); // error on accidental usage - - TFileNode *NextUnAllocNode(); - void RemoveUnAllocNode(TFileNode *); - - TFileNode *NextActiveNode(); - void RemoveActiveNode(TFileNode *); - - TFileStat *GetNextUnAlloc(TFileNode *node = 0); - TFileStat *GetNextActive(); - void RemoveActive(TFileStat *file); - - void Reset(); - void ValidateFiles(TDSet *dset, TList *slaves, Long64_t maxent = -1, Bool_t byfile = kFALSE); - -public: - TPacketizer(TDSet *dset, TList *slaves, Long64_t first, Long64_t num, - TList *input, TProofProgressStatus *st); - ~TPacketizer() override; - - Int_t AddWorkers(TList *workers) override; - TDSetElement *GetNextPacket(TSlave *sl, TMessage *r) override; - Long64_t GetEntriesProcessed(TSlave *sl) const; - - Float_t GetCurrentRate(Bool_t &all) override; - Int_t GetActiveWorkers() override; - - ClassDefOverride(TPacketizer,0) //Generate work packets for parallel processing -}; - -#endif diff --git a/proof/proofplayer/inc/TPacketizerAdaptive.h b/proof/proofplayer/inc/TPacketizerAdaptive.h deleted file mode 100644 index b1f8e16c9d86a..0000000000000 --- a/proof/proofplayer/inc/TPacketizerAdaptive.h +++ /dev/null @@ -1,123 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Jan Iwaszkiewicz 11/12/06 - -/************************************************************************* - * Copyright (C) 1995-2006, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TPacketizerAdaptive -#define ROOT_TPacketizerAdaptive - -////////////////////////////////////////////////////////////////////////// -// // -// TPacketizerAdaptive // -// // -// This packetizer is based on TPacketizer but uses different // -// load-balancing algorithms and data structures. // -// Two main improvements in the load-balancing strategy: // -// - First one was to change the order in which the files are assigned // -// to the computing nodes in such a way that network transfers are // -// evenly distributed in the query time. Transfer of the remote files // -// was often becoming a bottleneck at the end of a query. // -// - The other improvement is the use of time-based packet size. We // -// measure the processing rate of all the nodes and calculate the // -// packet size, so that it takes certain amount of time. In this way // -// packetizer prevents the situation where the query can't finish // -// because of one slow node. // -// // -// The data structures: TFileStat, TFileNode and TSlaveStat are // -// enriched + changed and TFileNode::Compare method is changed. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TVirtualPacketizer.h" - - -class TMessage; -class TTree; -class TMap; -class TNtupleD; -class TProofStats; -class TRandom; -class TSortedList; - -class TPacketizerAdaptive : public TVirtualPacketizer { - -public: // public because of Sun CC bug - class TFileNode; - class TFileStat; - class TSlaveStat; - -private: - TList *fFileNodes; // nodes with files - TList *fUnAllocated; // nodes with unallocated files - TList *fActive; // nodes with unfinished files - Int_t fMaxPerfIdx; // maximum of our slaves' performance index - TList *fPartitions; // list of partitions on nodes - - TSortedList *fFilesToProcess; // Global list of files (TFileStat) to be processed - - Bool_t fCachePacketSync; // control synchronization of cache and packet sizes - Double_t fMaxEntriesRatio; // max file entries to avg allowed ratio for cache-to-packet sync - - Float_t fFractionOfRemoteFiles; // fraction of TDSetElements that are on non-workers - Long64_t fNEventsOnRemLoc; // number of events in currently - // unalloc files on non-worker loc. - Float_t fBaseLocalPreference; // indicates how much more likely the nodes will be - // to open their local files (1 means indifferent) - Bool_t fForceLocal; // if 1 - eliminate the remote processing - - Long_t fMaxSlaveCnt; // maximum number of workers per filenode (Long_t to avoid - // warnings from backward compatibility support) - Int_t fPacketAsAFraction; // used to calculate the packet size - // fPacketSize = fTotalEntries / (fPacketAsAFraction * nslaves) - // fPacketAsAFraction can be interpreted as follows: - // assuming all slaves have equal processing rate, packet size - // is (#events processed by 1 slave) / fPacketSizeAsAFraction. - // It can be set with PROOF_PacketAsAFraction in input list. - Int_t fStrategy; // 0 means the classic and 1 (default) - the adaptive strategy - Int_t fTryReassign; // Controls attempts to reassign packets (0 == no reassignment) - - TPacketizerAdaptive(); - TPacketizerAdaptive(const TPacketizerAdaptive&); // no implementation, will generate - void InitStats(); // initialise the stats - void operator=(const TPacketizerAdaptive&); // error on accidental usage - - TFileNode *NextNode(); - void RemoveUnAllocNode(TFileNode *); - - TFileNode *NextActiveNode(); - void RemoveActiveNode(TFileNode *); - - TFileStat *GetNextUnAlloc(TFileNode *node = 0, const char *nodeHostName = 0); - TFileStat *GetNextActive(); - void RemoveActive(TFileStat *file); - - void Reset(); - void ValidateFiles(TDSet *dset, TList *slaves, Long64_t maxent = -1, Bool_t byfile = kFALSE); - Int_t ReassignPacket(TDSetElement *e, TList **listOfMissingFiles); - void SplitPerHost(TList *elements, TList **listOfMissingFiles); - -public: - TPacketizerAdaptive(TDSet *dset, TList *slaves, Long64_t first, Long64_t num, - TList *input, TProofProgressStatus *st); - ~TPacketizerAdaptive() override; - - Int_t AddProcessed(TSlave *sl, TProofProgressStatus *st, - Double_t latency, TList **listOfMissingFiles = 0) override; - Int_t GetEstEntriesProcessed(Float_t, Long64_t &ent, Long64_t &bytes, Long64_t &calls) override; - Float_t GetCurrentRate(Bool_t &all) override; - Int_t CalculatePacketSize(TObject *slstat, Long64_t cachesz, Int_t learnent); - TDSetElement *GetNextPacket(TSlave *sl, TMessage *r) override; - void MarkBad(TSlave *s, TProofProgressStatus *status, TList **missingFiles) override; - - Int_t GetActiveWorkers() override; - - ClassDefOverride(TPacketizerAdaptive,0) //Generate work packets for parallel processing -}; - -#endif diff --git a/proof/proofplayer/inc/TPacketizerFile.h b/proof/proofplayer/inc/TPacketizerFile.h deleted file mode 100644 index face353f91fa2..0000000000000 --- a/proof/proofplayer/inc/TPacketizerFile.h +++ /dev/null @@ -1,71 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G. Ganis 2009 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TPacketizerFile -#define ROOT_TPacketizerFile - -////////////////////////////////////////////////////////////////////////// -// // -// TPacketizerFile // -// // -// This packetizer generates packets which conatin a single file path // -// to be used in process. Used for tasks generating files, like in // -// PROOF bench. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TVirtualPacketizer.h" -#include "TMap.h" - - -class TMessage; -class TList; -class TStopwatch; - -class TPacketizerFile : public TVirtualPacketizer { - -public: // This is always needed - class TSlaveStat; - class TIterObj; - -private: - TMap *fFiles; // Files to be produced/processed per node - TList *fNotAssigned; // List of files not assigned to a specific node - TList *fIters; // Iterators on the file lists per node - Long64_t fAssigned; // No.files processed or being processed. - Bool_t fProcNotAssigned; // Whether to process files not asdigned to a worker - Bool_t fAddFileInfo; // Whether to add the TFileInfo object in the packet - - TStopwatch *fStopwatch; // For measuring the start time of each packet - - TPacketizerFile(); - // : fFiles(0), fNotAssigned(0), fIters(0), fAssigned(0), - // fProcNotAssigned(kTRUE), fAddFileInfo(kFALSE), fStopwatch(0) { } - TPacketizerFile(const TPacketizerFile&); // no implementation, will generate - void operator=(const TPacketizerFile&); // error on accidental usage - -public: - TPacketizerFile(TList *workers, Long64_t, TList *input, TProofProgressStatus *st = 0); - ~TPacketizerFile() override; - - TDSetElement *GetNextPacket(TSlave *wrk, TMessage *r) override; - - Double_t GetCurrentTime(); - - Float_t GetCurrentRate(Bool_t &all) override; - Int_t GetActiveWorkers() override { return -1; } - - ClassDefOverride(TPacketizerFile,0) //Generate work packets for parallel processing -}; - -//------------------------------------------------------------------------------- - -#endif diff --git a/proof/proofplayer/inc/TPacketizerMulti.h b/proof/proofplayer/inc/TPacketizerMulti.h deleted file mode 100644 index ef82bf7479fcc..0000000000000 --- a/proof/proofplayer/inc/TPacketizerMulti.h +++ /dev/null @@ -1,80 +0,0 @@ -// $Id$ -// Author: G. Ganis Jan 2010 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TPacketizerMulti -#define ROOT_TPacketizerMulti - -////////////////////////////////////////////////////////////////////////// -// // -// TPacketizerMulti // -// // -// This class allows to do multiple runs in the same query; each run // -// can be a, for example, different dataset or the same dataset with // -// entry list. // -// The multiple packetizer conatins a list of packetizers which are // -// processed in turn. // -// The bit TSelector::kNewRun is set in the TSelector object when a new // -// packetizer is used. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TVirtualPacketizer.h" - - -class TIter; -class TList; -class TMap; -class TMessage; -class TProofProgressStatus; -class TSlave; - -class TPacketizerMulti : public TVirtualPacketizer { - -private: - TList *fPacketizers; // Packetizers to be processed - TIter *fPacketizersIter; // Iterator on fPacketizers - TVirtualPacketizer *fCurrent; // Packetizer being currently processed - TMap *fAssignedPack; // Map {worker,packetizer} of lat assignement - - TPacketizerMulti(); - TPacketizerMulti(const TPacketizerMulti&); // no implementation, will generate - void operator=(const TPacketizerMulti&); // error on accidental usage - - TVirtualPacketizer *CreatePacketizer(TDSet *dset, TList *wrks, Long64_t first, Long64_t num, - TList *input, TProofProgressStatus *st); - -public: - TPacketizerMulti(TDSet *dset, TList *slaves, Long64_t first, Long64_t num, - TList *input, TProofProgressStatus *st); - ~TPacketizerMulti() override; - - TDSetElement *GetNextPacket(TSlave *wrk, TMessage *r) override; - - Int_t GetEstEntriesProcessed(Float_t f, Long64_t &ent, Long64_t &bytes, Long64_t &calls) override - { if (fCurrent) return fCurrent->GetEstEntriesProcessed(f,ent,bytes,calls); - return 1; } - Float_t GetCurrentRate(Bool_t &all) override { all = kTRUE; - return (fCurrent? fCurrent->GetCurrentRate(all) : 0.); } - void StopProcess(Bool_t abort, Bool_t stoptimer = kFALSE) override { - if (fCurrent) fCurrent->StopProcess(abort, stoptimer); - TVirtualPacketizer::StopProcess(abort, stoptimer); } - void MarkBad(TSlave *wrk, TProofProgressStatus *st, TList **missing) override - { if (fCurrent) fCurrent->MarkBad(wrk, st, missing); return; } - Int_t AddProcessed(TSlave *wrk, TProofProgressStatus *st, Double_t lat, TList **missing) override - { if (fCurrent) return fCurrent->AddProcessed(wrk, st, lat, missing); - return -1; } - - Int_t GetActiveWorkers() override { if (fCurrent) return fCurrent->GetActiveWorkers(); return 0; } - - ClassDefOverride(TPacketizerMulti,0) //Generate work packets for parallel processing -}; - -#endif diff --git a/proof/proofplayer/inc/TPacketizerUnit.h b/proof/proofplayer/inc/TPacketizerUnit.h deleted file mode 100644 index aee43282a0de6..0000000000000 --- a/proof/proofplayer/inc/TPacketizerUnit.h +++ /dev/null @@ -1,81 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Long Tran-Thanh 22/07/07 -// Revised: G. Ganis, May 2011 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TPacketizerUnit -#define ROOT_TPacketizerUnit - -////////////////////////////////////////////////////////////////////////// -// // -// TPacketizerUnit // -// // -// This packetizer generates packets of generic units, representing the // -// number of times an operation cycle has to be repeated by the worker // -// node, e.g. the number of Monte carlo events to be generated. // -// Packets sizes are generated taking into account the performance of // -// worker nodes, based on the time needed to process previous packets, // -// with the goal of having all workers ending at the same time. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TVirtualPacketizer.h" -#include "TMap.h" - - -class TMessage; -class TTimer; -class TTree; -class TProofStats; -class TStopwatch; - - -class TPacketizerUnit : public TVirtualPacketizer { - -public: // public because of Sun CC bug - class TSlaveStat; - -private: - TList *fPackets; // All processed packets - TMap *fWrkStats; // Worker status, keyed by correspondig TSlave - TList *fWrkExcluded; // List of nodes excluded from distribution - // (submasters with no active workers) - TStopwatch *fStopwatch; // For measuring the start time of each packet - Long64_t fProcessing; // Event being processed - Long64_t fAssigned; // Entries processed or being processed. - Double_t fCalibFrac; // Size of the calibrating packet as fraction of Ntot/Nwrk - Long64_t fNumPerWorker; // Number of cycles per worker, if this option - // is chosen - Bool_t fFixedNum; // Whether we must assign a fixed number of cycles per worker - - Long64_t fPacketSeq; // Sequential number of the last packet assigned - - TPacketizerUnit(); - TPacketizerUnit(const TPacketizerUnit&); // no implementation, will generate - void operator=(const TPacketizerUnit&); // error on accidental usage - -public: - TPacketizerUnit(TList *slaves, Long64_t num, TList *input, TProofProgressStatus *st = 0); - ~TPacketizerUnit() override; - - Int_t AssignWork(TDSet* /*dset*/, Long64_t /*first*/, Long64_t num) override; - TDSetElement *GetNextPacket(TSlave *sl, TMessage *r) override; - - Double_t GetCurrentTime(); - - Float_t GetCurrentRate(Bool_t &all) override; - Int_t GetActiveWorkers() override { return fWrkStats->GetSize(); } - - Int_t AddWorkers(TList *workers) override; - - ClassDefOverride(TPacketizerUnit,0) //Generate work packets for parallel processing -}; - -#endif diff --git a/proof/proofplayer/inc/TPerfStats.h b/proof/proofplayer/inc/TPerfStats.h deleted file mode 100644 index d947511976685..0000000000000 --- a/proof/proofplayer/inc/TPerfStats.h +++ /dev/null @@ -1,157 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Kristjan Gulbrandsen 11/05/04 - -/************************************************************************* - * Copyright (C) 1995-2004, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TPerfStats -#define ROOT_TPerfStats - -////////////////////////////////////////////////////////////////////////// -// // -// TPerfStats // -// // -// Provides the interface for the PROOF internal performance measurment // -// and event tracing. // -// // -////////////////////////////////////////////////////////////////////////// - - -#include "TObject.h" -#include "TObjArray.h" -#include "TTimeStamp.h" -#include "TString.h" -#include "TVirtualPerfStats.h" - - -class TDSet; -class TH1D; -class TH1I; -class TH2D; -class TList; -class TTree; - -class TPerfEvent : public TObject { - -public: - TString fEvtNode; // node on which the event was generated - TTimeStamp fTimeStamp; // time offset from start of run - TVirtualPerfStats::EEventType fType; - TString fSlaveName; - TString fNodeName; - TString fFileName; - TString fFileClass; - TString fSlave; - Long64_t fEventsProcessed; - Long64_t fBytesRead; - Long64_t fLen; - Double_t fLatency; - Double_t fProcTime; - Double_t fCpuTime; - Bool_t fIsStart; - Bool_t fIsOk; - - TPerfEvent(TTimeStamp *offset = 0); - ~TPerfEvent() override {} - - Bool_t IsSortable() const override { return kTRUE; } - Int_t Compare(const TObject *obj) const override; - void Print(Option_t *option="") const override; - - ClassDefOverride(TPerfEvent,3) // Class holding TProof Event Info -}; - - -class TPerfStats : public TVirtualPerfStats { - -friend class TProofMonSender; - -private: - TTree *fTrace; //!TTree with trace events - TTimeStamp fTzero; //!start time of this run - TPerfEvent *fPerfEvent; //!TPerfEvent used to fill tree - TH1D *fPacketsHist; //!histogram of packets processed per slave - TH1I *fProcPcktHist; //!histogram of packets being processed per slave - TH1D *fEventsHist; //!histogram of events processed per slave - TH1D *fNodeHist; //!histogram of slaves per file serving node - TH2D *fLatencyHist; //!histogram of latency due to packet requests - TH2D *fProcTimeHist; //!histogram of real time spent processing packets - TH2D *fCpuTimeHist; //!histogram of cpu time spent processing packets - Long64_t fBytesRead; //!track bytes read of main file - Double_t fTotCpuTime; //!total cpu time of all slaves - Long64_t fTotBytesRead; //!total bytes read on all slaves - Long64_t fTotEvents; //!total number of events processed - Long64_t fNumEvents; //!total number of events to be processed - Int_t fSlaves; //!number of active slaves - - Bool_t fDoHist; //!Fill histos - Bool_t fDoTrace; //!Trace details in master - Bool_t fDoTraceRate; //!Trace processing rate in master - Bool_t fDoSlaveTrace; //!Full tracing in workers - Bool_t fDoQuota; //!Save stats on SQL server for quota management - - Bool_t fMonitorPerPacket; //!Whether to send the full entry per each packet - - TObjArray fMonSenders; //!Monitoring engines - - TString fDataSet; //!Dataset string - Int_t fDataSetLen; //!Maximum size of the dataset string fDataSet - Int_t fDataSetSize; //!# of files in the dataset - TDSet *fDSet; //!Saved pointer to the TDSet object - TList *fOutput; //!Saved pointer to the output list - - static Long_t fgVirtMemMax; //! Max virtual memory used by this process - static Long_t fgResMemMax; //! Max resident memory used by this process - - TPerfStats(TList *input, TList *output); - void WriteQueryLog(); - - void SetFile(TFile *) override {} - -public: - ~TPerfStats() override; - - void SimpleEvent(EEventType type) override; - void PacketEvent(const char *slave, const char *slavename, const char *filename, - Long64_t eventsprocessed, Double_t latency, - Double_t proctime, Double_t cputime, Long64_t bytesRead) override; - void FileEvent(const char *slave, const char *slavename, const char *nodename, const char *filename, - Bool_t isStart) override; - - void FileOpenEvent(TFile *file, const char *filename, Double_t start) override; - void FileReadEvent(TFile *file, Int_t len, Double_t start) override; - void UnzipEvent(TObject *tree, Long64_t pos, Double_t start, Int_t complen, Int_t objlen) override; - void RateEvent(Double_t proctime, Double_t deltatime, - Long64_t eventsprocessed, Long64_t bytesRead) override; - void SetBytesRead(Long64_t num) override; - Long64_t GetBytesRead() const override; - void SetNumEvents(Long64_t num) override { fNumEvents = num; } - Long64_t GetNumEvents() const override { return fNumEvents; } - - void PrintBasketInfo(Option_t * = "") const override {} - void SetLoaded(TBranch *, size_t) override {} - void SetLoaded(size_t, size_t) override {} - void SetLoadedMiss(TBranch *, size_t) override {} - void SetLoadedMiss(size_t, size_t) override {} - void SetMissed(TBranch *, size_t) override {} - void SetMissed(size_t, size_t) override {} - void SetUsed(TBranch *, size_t) override {} - void SetUsed(size_t, size_t) override {} - void UpdateBranchIndices(TObjArray *) override {} - - static void Start(TList *input, TList *output); - static void Stop(); - static void Setup(TList *input); - static void SetMemValues(); - static void GetMemValues(Long_t &vmax, Long_t &rmax); - - ClassDefOverride(TPerfStats,0) // Class for collecting PROOF statistics -}; - - -#endif diff --git a/proof/proofplayer/inc/TProofDraw.h b/proof/proofplayer/inc/TProofDraw.h deleted file mode 100644 index f996d7b262454..0000000000000 --- a/proof/proofplayer/inc/TProofDraw.h +++ /dev/null @@ -1,326 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 24/09/2003 - -/************************************************************************* - * Copyright (C) 1995-2003, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofDraw -#define ROOT_TProofDraw - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofDraw // -// // -// Implement Tree drawing using PROOF. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TSelector.h" - -#include "TString.h" - -#include "TTreeDrawArgsParser.h" - -#include "TNamed.h" - -#include - - -class TTree; -class TTreeFormulaManager; -class TTreeFormula; -class TStatus; -class TH1; -class TEventList; -class TEntryList; -class TProfile; -class TProfile2D; -class TGraph; -class TPolyMarker3D; -class TCollection; - - -class TProofDraw : public TSelector { - -friend class TProofPlayer; - -protected: - TTreeDrawArgsParser fTreeDrawArgsParser; - TStatus *fStatus; - TString fSelection; - TString fInitialExp; - TTreeFormulaManager *fManager; - TTree *fTree; - TTreeFormula *fVar[4]; // Pointer to variable formula - TTreeFormula *fSelect; // Pointer to selection formula - Int_t fMultiplicity; // Indicator of the variability of the size of entries - Bool_t fObjEval; // true if fVar1 returns an object (or pointer to). - Int_t fDimension; // Dimension of the current expression - Double_t fWeight; // Global weight for fill actions - - void FillWeight(); - void SetCanvas(const char *objname); - void SetDrawAtt(TObject *o); - void SetError(const char *sub, const char *mesg); - -protected: - enum { kWarn = BIT(12) }; - - virtual Bool_t CompileVariables(); - virtual void ClearFormula(); - virtual Bool_t ProcessSingle(Long64_t /*entry*/, Int_t /*i*/); - virtual void DoFill(Long64_t entry, Double_t w, const Double_t *v) = 0; - virtual void DefVar() = 0; - -public: - TProofDraw(); - ~TProofDraw() override; - int Version() const override { return 1; } - void Init(TTree *) override; - void Begin(TTree *) override; - void SlaveBegin(TTree *) override; - Bool_t Notify() override; - Bool_t Process(Long64_t /*entry*/) override; - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(TProofDraw,0) //Tree drawing selector for PROOF -}; - - -class TProofDrawHist : public TProofDraw { - -private: - void DefVar1D(); - void DefVar2D(); - void DefVar3D(); - -protected: - TH1 *fHistogram; - - virtual void Begin1D(TTree *t); - virtual void Begin2D(TTree *t); - virtual void Begin3D(TTree *t); - void DoFill(Long64_t entry, Double_t w, const Double_t *v) override; - void DefVar() override; - -public: - TProofDrawHist() : fHistogram(0) { } - void Begin(TTree *t) override; - void Init(TTree *) override; - void SlaveBegin(TTree *) override; - void Terminate() override; - - ClassDefOverride(TProofDrawHist,0) //Tree drawing selector for PROOF -}; - - -class TProofDrawEventList : public TProofDraw { - -protected: - TEventList* fElist; // event list - TList* fEventLists; // a list of EventLists - - void DoFill(Long64_t entry, Double_t w, const Double_t *v) override; - void DefVar() override { } - -public: - TProofDrawEventList() : fElist(0), fEventLists(0) {} - ~TProofDrawEventList() override {} - - void Init(TTree *) override; - void SlaveBegin(TTree *) override; - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(TProofDrawEventList,0) //Tree drawing selector for PROOF -}; - -class TProofDrawEntryList : public TProofDraw { - protected: - TEntryList *fElist; - - void DoFill(Long64_t entry, Double_t w, const Double_t *v) override; - void DefVar() override {} - - public: - TProofDrawEntryList() : fElist(0) {} - ~TProofDrawEntryList() override {} - - void Init(TTree *) override; - void SlaveBegin(TTree *) override; - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(TProofDrawEntryList, 0) //A Selectoor to fill a TEntryList from TTree::Draw -}; - - -class TProofDrawProfile : public TProofDraw { - -protected: - TProfile *fProfile; - - void DoFill(Long64_t entry, Double_t w, const Double_t *v) override; - void DefVar() override; - -public: - TProofDrawProfile() : fProfile(0) { } - void Init(TTree *) override; - void Begin(TTree *t) override; - void SlaveBegin(TTree *) override; - void Terminate() override; - - ClassDefOverride(TProofDrawProfile,0) //Tree drawing selector for PROOF -}; - - -class TProofDrawProfile2D : public TProofDraw { - -protected: - TProfile2D *fProfile; - - void DoFill(Long64_t entry, Double_t w, const Double_t *v) override; - void DefVar() override; - -public: - TProofDrawProfile2D() : fProfile(0) { } - void Init(TTree *) override; - void Begin(TTree *t) override; - void SlaveBegin(TTree *) override; - void Terminate() override; - - ClassDefOverride(TProofDrawProfile2D,0) //Tree drawing selector for PROOF -}; - - -class TProofDrawGraph : public TProofDraw { - -protected: - TGraph *fGraph; - - void DoFill(Long64_t entry, Double_t w, const Double_t *v) override; - void DefVar() override { } - -public: - TProofDrawGraph() : fGraph(0) { } - void Init(TTree *tree) override; - void SlaveBegin(TTree *) override; - void Terminate() override; - - ClassDefOverride(TProofDrawGraph,0) //Tree drawing selector for PROOF -}; - - -class TProofDrawPolyMarker3D : public TProofDraw { - -protected: - TPolyMarker3D *fPolyMarker3D; - - void DoFill(Long64_t entry, Double_t w, const Double_t *v) override; - void DefVar() override { } - -public: - TProofDrawPolyMarker3D() : fPolyMarker3D(0) { } - void Init(TTree *tree) override; - void SlaveBegin(TTree *) override; - void Terminate() override; - - ClassDefOverride(TProofDrawPolyMarker3D,0) //Tree drawing selector for PROOF -}; - -template -class TProofVectorContainer : public TNamed { - // Owns an std::vector. - // Implements Merge(TCollection*) which merges vectors holded - // by all the TProofVectorContainers in the collection. -protected: - std::vector *fVector; // vector - -public: - TProofVectorContainer(std::vector* anVector) : fVector(anVector) { } - TProofVectorContainer() : fVector(0) { } - ~TProofVectorContainer() override { delete fVector; } - - std::vector *GetVector() const { return fVector; } - Long64_t Merge(TCollection* list); - - ClassDefOverride(TProofVectorContainer,1) //Class describing a vector container -}; - -class TProofDrawListOfGraphs : public TProofDraw { - -public: - struct Point3D_t { - public: - Double_t fX, fY, fZ; - Point3D_t(Double_t x, Double_t y, Double_t z) : fX(x), fY(y), fZ(z) { } - Point3D_t() : fX(0), fY(0), fZ(0) { } - }; - -protected: - TProofVectorContainer *fPoints; - void DoFill(Long64_t entry, Double_t w, const Double_t *v) override; - void DefVar() override { } - -public: - TProofDrawListOfGraphs() : fPoints(0) { } - void SlaveBegin(TTree *) override; - void Terminate() override; - - ClassDefOverride(TProofDrawListOfGraphs,0) //Tree drawing selector for PROOF -}; - - -class TProofDrawListOfPolyMarkers3D : public TProofDraw { - -public: - struct Point4D_t { - public: - Double_t fX, fY, fZ, fT; - Point4D_t(Double_t x, Double_t y, Double_t z, Double_t t) : fX(x), fY(y), fZ(z), fT(t) { } - Point4D_t() : fX(0), fY(0), fZ(0), fT(0) { } - }; - -protected: - TProofVectorContainer *fPoints; - void DoFill(Long64_t entry, Double_t w, const Double_t *v) override; - void DefVar() override { } - -public: - TProofDrawListOfPolyMarkers3D() : fPoints(0) { } - void SlaveBegin(TTree *) override; - void Terminate() override; - - ClassDefOverride(TProofDrawListOfPolyMarkers3D,0) //Tree drawing selector for PROOF -}; - -template -Long64_t TProofVectorContainer::Merge(TCollection* li) -{ - // Adds all vectors holded by all TProofVectorContainers in the collection - // the vector holded by this TProofVectorContainer. - // Returns the total number of poins in the result or -1 in case of an error. - - TIter next(li); - - std::back_insert_iterator > ii(*fVector); - while (TObject* o = next()) { - TProofVectorContainer *vh = dynamic_cast*> (o); - if (!vh) { - Error("Merge", - "Cannot merge - an object which doesn't inherit from TProofVectorContainer found in the list"); - return -1; - } - std::copy(vh->GetVector()->begin(), vh->GetVector()->end(), ii); - } - return fVector->size(); -} - -#endif diff --git a/proof/proofplayer/inc/TProofLimitsFinder.h b/proof/proofplayer/inc/TProofLimitsFinder.h deleted file mode 100644 index 5a71c2fa4bc89..0000000000000 --- a/proof/proofplayer/inc/TProofLimitsFinder.h +++ /dev/null @@ -1,45 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 19/04/2002 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofLimitsFinder -#define ROOT_TProofLimitsFinder - -////////////////////////////////////////////////////////////////////////// -// // -// TProofLimitsFinder // -// // -// Class to find nice axis limits and synchronize them between workers // -// // -////////////////////////////////////////////////////////////////////////// - -#include "THLimitsFinder.h" - -class TH1; -class TString; - -class TProofLimitsFinder : public THLimitsFinder { - -public: - TProofLimitsFinder() { } - ~TProofLimitsFinder() override { } - Int_t FindGoodLimits(TH1 *h, Axis_t xmin, Axis_t xmax) override; - Int_t FindGoodLimits(TH1 *h, Axis_t xmin, Axis_t xmax, Axis_t ymin, Axis_t ymax) override; - Int_t FindGoodLimits(TH1 *h, Axis_t xmin, Axis_t xmax, Axis_t ymin, Axis_t ymax, Axis_t zmin, Axis_t zmax) override; - - static void AutoBinFunc(TString& key, - Double_t& xmin, Double_t& xmax, - Double_t& ymin, Double_t& ymax, - Double_t& zmin, Double_t& zmax); - - ClassDefOverride(TProofLimitsFinder,0) //Find and communicate best axis limits -}; - -#endif diff --git a/proof/proofplayer/inc/TProofMonSender.h b/proof/proofplayer/inc/TProofMonSender.h deleted file mode 100644 index fc4c7e131b246..0000000000000 --- a/proof/proofplayer/inc/TProofMonSender.h +++ /dev/null @@ -1,82 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G.Ganis July 2011 - -/************************************************************************* - * Copyright (C) 1995-2004, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofMonSender -#define ROOT_TProofMonSender - -////////////////////////////////////////////////////////////////////////// -// // -// TProofMonSender // -// // -// Provides the interface for PROOF monitoring to different writers. // -// Allows to decouple the information sent from the backend. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TNamed.h" - -class TDSet; -class TList; -class TPerfStat; - -class TProofMonSender : public TNamed { - -protected: - Int_t fSummaryVrs; // Version of the summary 'table' - Int_t fDataSetInfoVrs; // Version of the dataset info 'table' - Int_t fFileInfoVrs; // Version of the file info 'table' - enum EConfigBits { // TProofMonSender status/config bits - kSendSummary = BIT(15), // Toggle sending of summary - kSendDataSetInfo = BIT(16), // Toggle sending of dataset info - kSendFileInfo = BIT(17) // Toggle sending of files info - }; - - // Auxilliary class describing dataset multiplets - class TDSetPlet : public TNamed { - public: - Int_t fFiles; - Int_t fMissing; - TDSet *fDSet; - TDSetPlet(const char *name, TDSet *ds = 0) : - TNamed(name, ""), fFiles(0), fMissing(0), fDSet(ds) { } - ~TDSetPlet() override { } - }; - -public: - - TProofMonSender(const char *n = "Abstract", - const char *t = "ProofMonSender") : TNamed(n,t), - fSummaryVrs(2), fDataSetInfoVrs(1), fFileInfoVrs(1) - { SetBit(TObject::kInvalidObject); - SetBit(kSendSummary); - SetBit(kSendDataSetInfo); - ResetBit(kSendFileInfo); } - ~TProofMonSender() override { } - - // This changes the send control options - Int_t SetSendOptions(const char *); - - // Object validity - Bool_t IsValid() const { return (TestBit(TObject::kInvalidObject)) ? kFALSE : kTRUE; } - - // Summary record - virtual Int_t SendSummary(TList *, const char *) = 0; - - // Information about the dataset(s) processed - virtual Int_t SendDataSetInfo(TDSet *, TList *, const char *, const char *) = 0; - - // Detailed information about files - virtual Int_t SendFileInfo(TDSet *, TList *, const char *, const char *) = 0; - - ClassDefOverride(TProofMonSender,0); // Interface for PROOF monitoring -}; - -#endif diff --git a/proof/proofplayer/inc/TProofMonSenderSQL.h b/proof/proofplayer/inc/TProofMonSenderSQL.h deleted file mode 100644 index f9ead02595942..0000000000000 --- a/proof/proofplayer/inc/TProofMonSenderSQL.h +++ /dev/null @@ -1,56 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G.Ganis July 2011 - -/************************************************************************* - * Copyright (C) 1995-2004, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofMonSenderSQL -#define ROOT_TProofMonSenderSQL - -////////////////////////////////////////////////////////////////////////// -// // -// TProofMonSenderSQL // -// // -// TProofMonSender implementation for SQL writers. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProofMonSender.h" - -class TDSet; -class TList; -class TPerfStat; -class TVirtualMonitoringWriter; - -class TProofMonSenderSQL : public TProofMonSender { - -private: - TVirtualMonitoringWriter *fWriter; // Writer instance connect to backend - TString fDSetSendOpts; // Opts for posting dataset table - TString fFilesSendOpts; // Opts for posting files table - -public: - - TProofMonSenderSQL(const char *serv, const char *user, const char *pass, - const char *table = "proof.proofquerylog", - const char *dstab = 0, const char *filestab = 0); - ~TProofMonSenderSQL() override; - - // Summary record - Int_t SendSummary(TList *, const char *) override; - - // Information about the dataset(s) processed - Int_t SendDataSetInfo(TDSet *, TList *, const char *, const char *) override; - - // Detailed information about files - Int_t SendFileInfo(TDSet *, TList *, const char *, const char *) override; - - ClassDefOverride(TProofMonSenderSQL, 0); // Interface for PROOF monitoring -}; - -#endif diff --git a/proof/proofplayer/inc/TProofPlayer.h b/proof/proofplayer/inc/TProofPlayer.h deleted file mode 100644 index 2eaea08bc3914..0000000000000 --- a/proof/proofplayer/inc/TProofPlayer.h +++ /dev/null @@ -1,445 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 07/01/02 - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofPlayer -#define ROOT_TProofPlayer - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofPlayer // -// // -// This internal class and its subclasses steer the processing in PROOF.// -// Instances of the TProofPlayer class are created on the worker nodes // -// per session and do the processing. // -// Instances of its subclass - TProofPlayerRemote are created per each // -// query on the master(s) and on the client. On the master(s), // -// TProofPlayerRemote coordinate processing, check the dataset, create // -// the packetizer and take care of merging the results of the workers. // -// The instance on the client collects information on the input // -// (dataset and selector), it invokes the Begin() method and finalizes // -// the query by calling Terminate(). // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TVirtualProofPlayer.h" -#include "TArrayL64.h" -#include "TArrayF.h" -#include "TArrayI.h" -#include "TList.h" -#include "TSystem.h" -#include "TQueryResult.h" -#include "TProofProgressStatus.h" -#include "TError.h" - -#include - -class TSelector; -class TSocket; -class TVirtualPacketizer; -class TSlave; -class TEventIter; -class TProofStats; -class TStatus; -class TTimer; -class THashList; -class TH1; -class TFile; -class TStopwatch; - -//------------------------------------------------------------------------ - -class TProofPlayer : public TVirtualProofPlayer { - -private: - TList *fAutoBins; // Map of min/max values by name for slaves - -protected: - TList *fInput; //-> list with input objects - THashList *fOutput; // list with output objects - TSelector *fSelector; //! the latest selector - Bool_t fCreateSelObj; //! kTRUE when fSelector has been created locally - TClass *fSelectorClass; //! class of the latest selector - TTimer *fFeedbackTimer; //! timer for sending intermediate results - Long_t fFeedbackPeriod; //! period (ms) for sending intermediate results - TEventIter *fEvIter; //! iterator on events or objects - TStatus *fSelStatus; //! status of query in progress - EExitStatus fExitStatus; // exit status - Long64_t fTotalEvents; // number of events requested - TProofProgressStatus *fProgressStatus; // the progress status object; - - Long64_t fReadBytesRun; //! Bytes read in this run - Long64_t fReadCallsRun; //! Read calls in this run - Long64_t fProcessedRun; //! Events processed in this run - - TList *fQueryResults; //List of TQueryResult - TQueryResult *fQuery; //Instance of TQueryResult currently processed - TQueryResult *fPreviousQuery; //Previous instance of TQueryResult processed - Int_t fDrawQueries; //Number of Draw queries in the list - Int_t fMaxDrawQueries; //Max number of Draw queries kept - - TTimer *fStopTimer; //Timer associated with a stop request - std::mutex fStopTimerMtx; //To protect the stop timer - - TTimer *fDispatchTimer; //Dispatch pending events while processing - - TTimer *fProcTimeTimer; //Notifies reaching of allowed max proc time - TStopwatch *fProcTime; //Packet proc time - - TString fOutputFilePath; //Path to file with (partial) results of the query - TFile *fOutputFile; //TFile object attached to fOutputFilePath - Long_t fSaveMemThreshold; //Threshold for saving output to file - Bool_t fSavePartialResults; //Whether to save the partial results - Bool_t fSaveResultsPerPacket; //Whether to save partial results after each packet - - static THashList *fgDrawInputPars; // List of input parameters to be kept on drawing actions - - void *GetSender() override { return this; } //used to set gTQSender - - virtual Int_t DrawCanvas(TObject *obj); // Canvas drawing via libProofDraw - - virtual void SetupFeedback(); // specialized setup - - void MergeOutput(Bool_t savememvalues = kFALSE) override; - -public: // fix for broken compilers so TCleanup can call StopFeedback() - virtual void StopFeedback(); // specialized teardown - -protected: - class TCleanup { - private: - TProofPlayer *fPlayer; - public: - TCleanup(TProofPlayer *p) : fPlayer(p) { } - ~TCleanup() { fPlayer->StopFeedback(); } - }; - - Int_t AssertSelector(const char *selector_file); - Bool_t CheckMemUsage(Long64_t &mfreq, Bool_t &w80r, Bool_t &w80v, TString &wmsg); - - void MapOutputListToDataMembers() const; - -public: - enum EStatusBits { kDispatchOneEvent = BIT(15), kIsProcessing = BIT(16), - kMaxProcTimeReached = BIT(17), kMaxProcTimeExtended = BIT(18) }; - - TProofPlayer(TProof *proof = 0); - ~TProofPlayer() override; - - Long64_t Process(TDSet *set, - const char *selector, Option_t *option = "", - Long64_t nentries = -1, Long64_t firstentry = 0) override; - Long64_t Process(TDSet *set, - TSelector *selector, Option_t *option = "", - Long64_t nentries = -1, Long64_t firstentry = 0) override; - Bool_t JoinProcess(TList *workers) override; - TVirtualPacketizer *GetPacketizer() const override { return 0; } - Long64_t Finalize(Bool_t force = kFALSE, Bool_t sync = kFALSE) override; - Long64_t Finalize(TQueryResult *qr) override; - Long64_t DrawSelect(TDSet *set, const char *varexp, - const char *selection, Option_t *option = "", - Long64_t nentries = -1, Long64_t firstentry = 0) override; - Int_t GetDrawArgs(const char *var, const char *sel, Option_t *opt, - TString &selector, TString &objname) override; - void HandleGetTreeHeader(TMessage *mess) override; - void HandleRecvHisto(TMessage *mess) override; - void FeedBackCanvas(const char *name, Bool_t create); - - void StopProcess(Bool_t abort, Int_t timeout = -1) override; - void AddInput(TObject *inp) override; - void ClearInput() override; - TObject *GetOutput(const char *name) const override; - TList *GetOutputList() const override; - TList *GetInputList() const override { return fInput; } - TList *GetListOfResults() const override { return fQueryResults; } - void AddQueryResult(TQueryResult *q) override; - TQueryResult *GetCurrentQuery() const override { return fQuery; } - TQueryResult *GetQueryResult(const char *ref) override; - void RemoveQueryResult(const char *ref) override; - void SetCurrentQuery(TQueryResult *q) override; - void SetMaxDrawQueries(Int_t max) override { fMaxDrawQueries = max; } - void RestorePreviousQuery() override { fQuery = fPreviousQuery; } - Int_t AddOutputObject(TObject *obj) override; - void AddOutput(TList *out) override; // Incorporate a list - void StoreOutput(TList *out) override; // Adopts the list - void StoreFeedback(TObject *slave, TList *out) override; // Adopts the list - void Progress(Long64_t total, Long64_t processed) override; // *SIGNAL* - void Progress(TSlave *, Long64_t total, Long64_t processed) override - { Progress(total, processed); } - void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) override; // *SIGNAL* - void Progress(TSlave *, Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) override - { Progress(total, processed, bytesread, initTime, procTime, - evtrti, mbrti); } // *SIGNAL* - void Progress(TProofProgressInfo *pi) override; // *SIGNAL* - void Progress(TSlave *, TProofProgressInfo *pi) override { Progress(pi); } // *SIGNAL* - void Feedback(TList *objs) override; // *SIGNAL* - - TDrawFeedback *CreateDrawFeedback(TProof *p) override; - void SetDrawFeedbackOption(TDrawFeedback *f, Option_t *opt) override; - void DeleteDrawFeedback(TDrawFeedback *f) override; - - TDSetElement *GetNextPacket(TSlave *slave, TMessage *r) override; - - Int_t ReinitSelector(TQueryResult *qr) override; - - void UpdateAutoBin(const char *name, - Double_t& xmin, Double_t& xmax, - Double_t& ymin, Double_t& ymax, - Double_t& zmin, Double_t& zmax) override; - - Bool_t IsClient() const override { return kFALSE; } - - void SetExitStatus(EExitStatus st) override { fExitStatus = st; } - EExitStatus GetExitStatus() const override { return fExitStatus; } - Long64_t GetEventsProcessed() const override { return fProgressStatus->GetEntries(); } - void AddEventsProcessed(Long64_t ev) override { fProgressStatus->IncEntries(ev); } - - void SetDispatchTimer(Bool_t on = kTRUE) override; - void SetStopTimer(Bool_t on = kTRUE, - Bool_t abort = kFALSE, Int_t timeout = 0) override; - - void SetInitTime() override { } - - void SetMerging(Bool_t = kTRUE) override { } - - Long64_t GetCacheSize() override; - Int_t GetLearnEntries() override; - - void SetOutputFilePath(const char *fp) override { fOutputFilePath = fp; } - Int_t SavePartialResults(Bool_t queryend = kFALSE, Bool_t force = kFALSE) override; - - void SetProcessing(Bool_t on = kTRUE); - TProofProgressStatus *GetProgressStatus() const override { return fProgressStatus; } - - void UpdateProgressInfo() override; - - ClassDefOverride(TProofPlayer,0) // Basic PROOF player -}; - - -//------------------------------------------------------------------------ - -class TProofPlayerLocal : public TProofPlayer { - -private: - Bool_t fIsClient; - -protected: - void SetupFeedback() override { } - void StopFeedback() override { } - -public: - TProofPlayerLocal(Bool_t client = kTRUE) : fIsClient(client) { } - ~TProofPlayerLocal() override { } - - Bool_t IsClient() const override { return fIsClient; } - Long64_t Process(const char *selector, Long64_t nentries = -1, Option_t *option = ""); - Long64_t Process(TSelector *selector, Long64_t nentries = -1, Option_t *option = ""); - Long64_t Process(TDSet *set, - const char *selector, Option_t *option = "", - Long64_t nentries = -1, Long64_t firstentry = 0) override { - return TProofPlayer::Process(set, selector, option, nentries, firstentry); } - Long64_t Process(TDSet *set, - TSelector *selector, Option_t *option = "", - Long64_t nentries = -1, Long64_t firstentry = 0) override { - return TProofPlayer::Process(set, selector, option, nentries, firstentry); } - ClassDefOverride(TProofPlayerLocal,0) // PROOF player running on client -}; - - -//------------------------------------------------------------------------ - -////////////////////////////////////////////////////////////////////////// -// // -// TProofPlayerRemote // -// // -// Instances of TProofPlayerRemote are created per each query on the // -// master(s) and on the client. On the master(s), TProofPlayerRemote // -// coordinate processing, check the dataset, create the packetizer // -// and take care of merging the results of the workers. // -// The instance on the client collects information on the input // -// (dataset and selector), it invokes the Begin() method and finalizes // -// the query by calling Terminate(). // -// // -////////////////////////////////////////////////////////////////////////// - - -class TProofPlayerRemote : public TProofPlayer { - -protected: - TProof *fProof; // link to associated PROOF session - TList *fOutputLists; // results returned by slaves - TList *fFeedback; // reference for use on master - TList *fFeedbackLists; // intermediate results - TVirtualPacketizer *fPacketizer; // transform TDSet into packets for slaves - Bool_t fMergeFiles; // is True when merging output files centrally is needed - TDSet *fDSet; //!tdset for current processing - ErrorHandlerFunc_t fErrorHandler; // Store previous handler when redirecting output - Bool_t fMergeTH1OneByOne; // If kTRUE forces TH1 merge one-by-one [kTRUE] - TH1 *fProcPackets; //!Histogram with packets being processed (owned by TPerfStats) - TMessage *fProcessMessage; // Process message to replay when adding new workers dynamically - TString fSelectorFileName; // Current Selector's name, set by Process() - - TStopwatch *fMergeSTW; // Merging stop watch - Int_t fNumMergers; // Number of submergers - - Bool_t HandleTimer(TTimer *timer) override; - Int_t InitPacketizer(TDSet *dset, Long64_t nentries, - Long64_t first, const char *defpackunit, - const char *defpackdata); - TList *MergeFeedback(); - Bool_t MergeOutputFiles(); - void NotifyMemory(TObject *obj); - void SetLastMergingMsg(TObject *obj); - virtual Bool_t SendSelector(const char *selector_file); //send selector to slaves - TProof *GetProof() const { return fProof; } - void SetupFeedback() override; // specialized setup - void StopFeedback() override; // specialized teardown - void SetSelectorDataMembersFromOutputList(); - -public: - TProofPlayerRemote(TProof *proof = 0) : fProof(proof), fOutputLists(0), fFeedback(0), - fFeedbackLists(0), fPacketizer(0), - fMergeFiles(kFALSE), fDSet(0), fErrorHandler(0), - fMergeTH1OneByOne(kTRUE), fProcPackets(0), - fProcessMessage(0), fMergeSTW(0), fNumMergers(0) - { fProgressStatus = new TProofProgressStatus(); } - ~TProofPlayerRemote() override; // Owns the fOutput list - Long64_t Process(TDSet *set, const char *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0) override; - Long64_t Process(TDSet *set, TSelector *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0) override; - Bool_t JoinProcess(TList *workers) override; - Long64_t Finalize(Bool_t force = kFALSE, Bool_t sync = kFALSE) override; - Long64_t Finalize(TQueryResult *qr) override; - Long64_t DrawSelect(TDSet *set, const char *varexp, - const char *selection, Option_t *option = "", - Long64_t nentries = -1, Long64_t firstentry = 0) override; - - void RedirectOutput(Bool_t on = kTRUE); - void StopProcess(Bool_t abort, Int_t timeout = -1) override; - void StoreOutput(TList *out) override; // Adopts the list - void StoreFeedback(TObject *slave, TList *out) override; // Adopts the list - Int_t Incorporate(TObject *obj, TList *out, Bool_t &merged); - TObject *HandleHistogram(TObject *obj, Bool_t &merged); - Bool_t HistoSameAxis(TH1 *h0, TH1 *h1); - Int_t AddOutputObject(TObject *obj) override; - void AddOutput(TList *out) override; // Incorporate a list - void MergeOutput(Bool_t savememvalues = kFALSE) override; - void Progress(Long64_t total, Long64_t processed) override; // *SIGNAL* - void Progress(TSlave*, Long64_t total, Long64_t processed) override - { Progress(total, processed); } - void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) override; // *SIGNAL* - void Progress(TSlave *, Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) override - { Progress(total, processed, bytesread, initTime, procTime, - evtrti, mbrti); } // *SIGNAL* - void Progress(TProofProgressInfo *pi) override; // *SIGNAL* - void Progress(TSlave *, TProofProgressInfo *pi) override { Progress(pi); } // *SIGNAL* - void Feedback(TList *objs) override; // *SIGNAL* - TDSetElement *GetNextPacket(TSlave *slave, TMessage *r) override; - TVirtualPacketizer *GetPacketizer() const override { return fPacketizer; } - - Bool_t IsClient() const override; - - void SetInitTime() override; - - void SetMerging(Bool_t on = kTRUE) override; - - ClassDefOverride(TProofPlayerRemote,0) // PROOF player running on master server -}; - - -//------------------------------------------------------------------------ - -class TProofPlayerSlave : public TProofPlayer { - -private: - TSocket *fSocket; - TList *fFeedback; // List of objects to send updates of - - Bool_t HandleTimer(TTimer *timer) override; - -protected: - void SetupFeedback() override; - void StopFeedback() override; - -public: - TProofPlayerSlave(TSocket *socket = 0) : fSocket(socket), fFeedback(0) { } - - void HandleGetTreeHeader(TMessage *mess) override; - - ClassDefOverride(TProofPlayerSlave,0) // PROOF player running on slave server -}; - - -//------------------------------------------------------------------------ - -class TProofPlayerSuperMaster : public TProofPlayerRemote { - -private: - TArrayL64 fSlaveProgress; - TArrayL64 fSlaveTotals; - TArrayL64 fSlaveBytesRead; - TArrayF fSlaveInitTime; - TArrayF fSlaveProcTime; - TArrayF fSlaveEvtRti; - TArrayF fSlaveMBRti; - TArrayI fSlaveActW; - TArrayI fSlaveTotS; - TArrayF fSlaveEffS; - TList fSlaves; - Bool_t fReturnFeedback; - -protected: - Bool_t HandleTimer(TTimer *timer) override; - void SetupFeedback() override; - -public: - TProofPlayerSuperMaster(TProof *proof = 0) : - TProofPlayerRemote(proof), fReturnFeedback(kFALSE) { } - ~TProofPlayerSuperMaster() override { } - - Long64_t Process(TDSet *set, const char *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0) override; - Long64_t Process(TDSet *set, TSelector *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0) override - { return TProofPlayerRemote::Process(set, selector, option, - nentries, firstentry); } - void Progress(Long64_t total, Long64_t processed) override - { TProofPlayerRemote::Progress(total, processed); } - void Progress(Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) override - { TProofPlayerRemote::Progress(total, processed, bytesread, - initTime, procTime, evtrti, mbrti); } - void Progress(TProofProgressInfo *pi) override { TProofPlayerRemote::Progress(pi); } - void Progress(TSlave *sl, Long64_t total, Long64_t processed) override; - void Progress(TSlave *sl, Long64_t total, Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) override; - void Progress(TSlave *sl, TProofProgressInfo *pi) override; - - ClassDefOverride(TProofPlayerSuperMaster,0) // PROOF player running on super master -}; - -#endif diff --git a/proof/proofplayer/inc/TProofPlayerLite.h b/proof/proofplayer/inc/TProofPlayerLite.h deleted file mode 100644 index 2faa7ad6dac85..0000000000000 --- a/proof/proofplayer/inc/TProofPlayerLite.h +++ /dev/null @@ -1,56 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G. Ganis Mar 2008 - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TProofPlayerLite -#define ROOT_TProofPlayerLite - - -////////////////////////////////////////////////////////////////////////// -// // -// TProofPlayerLite // -// // -// This version of TProofPlayerRemote merges the functionality needed // -// by clients and masters. It is used in optmized local sessions. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TProofPlayer.h" - - -class TProofPlayerLite : public TProofPlayerRemote { - -protected: - Bool_t HandleTimer(TTimer *timer) override; - - Int_t MakeSelector(const char *selfile); - void SetupFeedback() override; - -public: - TProofPlayerLite(TProof *proof = 0) : TProofPlayerRemote(proof) { } - - ~TProofPlayerLite() override { } // Owns the fOutput list - - Long64_t Process(TDSet *set, const char *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0) override; - Long64_t Process(TDSet *set, TSelector *selector, - Option_t *option = "", Long64_t nentries = -1, - Long64_t firstentry = 0) override; - Long64_t Finalize(Bool_t force = kFALSE, Bool_t sync = kFALSE) override; - Long64_t Finalize(TQueryResult *qr) override - { return TProofPlayerRemote::Finalize(qr); } - - void StoreFeedback(TObject *slave, TList *out) override; // Adopts the list - - ClassDefOverride(TProofPlayerLite,0) // PROOF player running in PROOF-Lite -}; - -#endif diff --git a/proof/proofplayer/inc/TStatsFeedback.h b/proof/proofplayer/inc/TStatsFeedback.h deleted file mode 100644 index 50996a0840d78..0000000000000 --- a/proof/proofplayer/inc/TStatsFeedback.h +++ /dev/null @@ -1,47 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G. Ganis May 2012 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TStatsFeedback -#define ROOT_TStatsFeedback - - -////////////////////////////////////////////////////////////////////////// -// // -// TStatsFeedback // -// // -// Utility class to display PROOF stats feedback histos during queries. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TQObject.h" -#include "TObject.h" - -class TProof; -class TProof; -class THashList; -class TSeqCollection; - -class TStatsFeedback : public TObject, public TQObject { -protected: - TString fName; // Identfier for this object - TProof *fProof; //handle to PROOF session -public: - TStatsFeedback(TProof *proof = 0); - ~TStatsFeedback() override; - - void Feedback(TList *objs); - const char *GetName() const override { return fName.Data(); } - ULong_t Hash() const override { return fName.Hash(); } - - ClassDefOverride(TStatsFeedback,0) // Present PROOF query feedback -}; - -#endif diff --git a/proof/proofplayer/inc/TStatus.h b/proof/proofplayer/inc/TStatus.h deleted file mode 100644 index ba7ebbd88cc7c..0000000000000 --- a/proof/proofplayer/inc/TStatus.h +++ /dev/null @@ -1,72 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 12/03/2004 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TStatus -#define ROOT_TStatus - -////////////////////////////////////////////////////////////////////////// -// // -// TStatus // -// // -// This class holds the status of a ongoing operation and collects // -// error messages. It provides a Merge() operation allowing it to // -// be used in PROOF to monitor status in the slaves. // -// No messages indicates success. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TNamed.h" -#include "THashList.h" - -#include -#include - -class TStatus : public TNamed { - -public: - enum EProcStatus { - kNotOk = BIT(15) // True if status of things are not OK - }; - -private: - TList fMsgs; // list of error messages - TIter fIter; //!iterator in messages - THashList fInfoMsgs; // list of info messages - - Int_t fExitStatus; // Query exit status ((Int_t)TVirtualProofPlayer::EExitStatus or -1); - Long_t fVirtMemMax; // Max virtual memory used by the worker - Long_t fResMemMax; // Max resident memory used by the worker - Long_t fVirtMaxMst; // Max virtual memory used by the master - Long_t fResMaxMst; // Max resident memory used by the master - -public: - TStatus(); - ~TStatus() override { } - - inline Bool_t IsOk() const { return TestBit(kNotOk) ? kFALSE : kTRUE; } - void Add(const char *mesg); - void AddInfo(const char *mesg); - virtual Int_t Merge(TCollection *list); - void Print(Option_t *option="") const override; - void Reset(); - const char *NextMesg(); - - Int_t GetExitStatus() const { return fExitStatus; } - Long_t GetResMemMax(Bool_t master = kFALSE) const { return ((master) ? fResMaxMst : fResMemMax); } - Long_t GetVirtMemMax(Bool_t master = kFALSE) const { return ((master) ? fVirtMaxMst : fVirtMemMax); } - - void SetExitStatus(Int_t est) { fExitStatus = est; } - void SetMemValues(Long_t vmem = -1, Long_t rmem = -1, Bool_t master = kFALSE); - - ClassDefOverride(TStatus,5); // Status class -}; - -#endif diff --git a/proof/proofplayer/src/TDrawFeedback.cxx b/proof/proofplayer/src/TDrawFeedback.cxx deleted file mode 100644 index 97cc17bfb6a56..0000000000000 --- a/proof/proofplayer/src/TDrawFeedback.cxx +++ /dev/null @@ -1,143 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 28/10/2003 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TDrawFeedback -\ingroup proofkernel - -Utility class to draw objects in the feedback list during queries. -Draws histograms in separated canvases and user-defined objects via -Draw(). Users requiring advanced treatment should implement their -own version following this example. See also TStatsFeedback. - -*/ - -#include "TDrawFeedback.h" - -#include "THashList.h" -#include "TObjString.h" -#include "TProof.h" -#include "TROOT.h" -#include "TH1.h" -#include "TH2.h" -#include "TError.h" -#include "TSeqCollection.h" -#include "TVirtualPad.h" -#include "TProofDebug.h" - -ClassImp(TDrawFeedback); - - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TDrawFeedback::TDrawFeedback(TProof *proof, TSeqCollection *names) - : fAll(kFALSE) -{ - fNames = new THashList; - fNames->SetOwner(); - - if (proof == 0) proof = gProof; - - TProof *p = dynamic_cast(proof); - if (p == 0) { - Error("TDrawFeedback","no valid proof session found"); - return; - } - fProof = p; - fName = fProof->GetSessionTag(); - - Bool_t ok = proof->Connect("Feedback(TList*)", "TDrawFeedback", - this, "Feedback(TList*)"); - - if ( !ok ) { - Error("TDrawFeedback","Connect() failed"); - return; - } - - if (names != 0) { - TIter next(names); - TObjString *name; - while((name = dynamic_cast(next())) != 0) { - fNames->Add(new TNamed(name->GetName(),"")); - } - } else { - fAll = kTRUE; - } - fOption = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TDrawFeedback::~TDrawFeedback() -{ - delete fNames; - - // Required since we overload TObject::Hash. - ROOT::CallRecursiveRemoveIfNeeded(*this); - - fProof->Disconnect("Feedback(TList*)", this, "Feedback(TList*"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display feedback - -void TDrawFeedback::Feedback(TList *objs) -{ - TSeqCollection *canvases = gROOT->GetListOfCanvases(); - TVirtualPad *save = gPad; - - PDB(kFeedback,1) Info("Feedback","%d Objects", objs->GetSize()); - - TIter next(objs); - TObject *o; - while( (o = next()) ) - { - TString name = o->GetName(); - if (fAll || fNames->FindObject(name.Data())) { - - if (TH1 *h = dynamic_cast(o)) { - - // Basic service provided fro histograms, each one drawn in - // a separate canvas named '_canvas' - - name += "_canvas"; - - TVirtualPad *p = (TVirtualPad*) canvases->FindObject(name.Data()); - - if ( p == 0 ) { - gROOT->MakeDefCanvas(); - gPad->SetName(name); - PDB(kFeedback,2) Info("Feedback","Created canvas %s", name.Data()); - } else { - p->cd(); - PDB(kFeedback,2) Info("Feedback","Used canvas %s", name.Data()); - } - - h->DrawCopy(fOption); - gPad->Update(); - - } else { - - // Call the Draw method of the object; this is intended for user-defined - // objects handling their canvas needs inside Draw() as needed - o->Draw(); - } - - } - } - - if (save != 0) { - save->cd(); - } else { - gPad = 0; - } -} diff --git a/proof/proofplayer/src/TEventIter.cxx b/proof/proofplayer/src/TEventIter.cxx deleted file mode 100644 index a97291e244da7..0000000000000 --- a/proof/proofplayer/src/TEventIter.cxx +++ /dev/null @@ -1,1339 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 07/01/02 -// Modified: Long Tran-Thanh 04/09/07 (Addition of TEventIterUnit) - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TEventIter -\ingroup proofkernel - -Special iterator class used in TProofPlayer to iterate over events -or objects in the packets. - -*/ - -#include "TEnv.h" -#include "TEventIter.h" -#include "TFriendElement.h" -#include "TCollection.h" -#include "TDSet.h" -#include "TFile.h" -#include "TKey.h" -#include "TProofDebug.h" -#include "TSelector.h" -#include "TTimeStamp.h" -#include "TTree.h" -#include "TTreeCache.h" -#include "TTreeCacheUnzip.h" -#include "TVirtualPerfStats.h" -#include "TEventList.h" -#include "TEntryList.h" -#include "TList.h" -#include "TMap.h" -#include "TRegexp.h" -#include "TProofServ.h" -#include "TSystem.h" - -#include "TError.h" - -#if defined(R__MACOSX) -#include "fcntl.h" -#endif - -ClassImp(TEventIter); - -//////////////////////////////////////////////////////////////////////////////// -/// Default constructor - -TEventIter::TEventIter() -{ - fDSet = 0; - fElem = 0; - fFile = 0; - fDir = 0; - fSel = 0; - fFirst = 0; - fCur = -1; - fNum = 0; - fStop = kFALSE; - fOldBytesRead = 0; - fEventList = 0; - fEventListPos = 0; - fEntryList = 0; - fEntryListPos = 0; - fElemFirst = 0; - fElemNum = 0; - fElemCur = -1; - ResetBit(TEventIter::kData); - - if ((fPackets = new TList)) { - TString n("ProcessedPackets_"); - if (gProofServ) n += gProofServ->GetOrdinal(); - fPackets->SetName(n); - Info("TEventIter", "fPackets list '%s' created", n.Data()); - } else { - Warning("TEventIter", "fPackets list could not be created"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TEventIter::TEventIter(TDSet *dset, TSelector *sel, Long64_t first, Long64_t num) - : fDSet(dset), fSel(sel) -{ - fElem = 0; - fFile = 0; - fDir = 0; - fFirst = first; - fCur = -1; - fNum = num; - fStop = kFALSE; - fEventList = 0; - fEventListPos = 0; - fEntryList = 0; - fEntryListPos = 0; - fOldBytesRead = 0; - fElemFirst = 0; - fElemNum = 0; - fElemCur = -1; - ResetBit(TEventIter::kData); - - if ((fPackets = new TList)) { - TString n("ProcessedPackets_"); - if (gProofServ) n += gProofServ->GetOrdinal(); - fPackets->SetName(n); - Info("TEventIter", "fPackets list '%s' created", n.Data()); - } else { - Warning("TEventIter", "fPackets list could not be created"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TEventIter::~TEventIter() -{ - if (fPackets) { - fPackets->SetOwner(kTRUE); - SafeDelete(fPackets); - } - delete fFile; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Invalidated the current packet (if any) by setting the TDSetElement::kCorrupted bit - -void TEventIter::InvalidatePacket() -{ - if (fElem) fElem->SetBit(TDSetElement::kCorrupted); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set flag to stop the process - -void TEventIter::StopProcess(Bool_t /*abort*/) -{ - fStop = kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create and instance of the appropriate iterator - -TEventIter *TEventIter::Create(TDSet *dset, TSelector *sel, Long64_t first, Long64_t num) -{ - if (dset->TestBit(TDSet::kEmpty)) { - return new TEventIterUnit(dset, sel, num); - } else if (dset->IsTree()) { - return new TEventIterTree(dset, sel, first, num); - } else { - return new TEventIterObj(dset, sel, first, num); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load directory - -Int_t TEventIter::LoadDir() -{ - Int_t ret = 0; - - // Check Filename - if ( fFile == 0 || fFilename != fElem->GetFileName() ) { - fDir = 0; - delete fFile; fFile = 0; - - fFilename = fElem->GetFileName(); - - TDirectory *dirsave = gDirectory; - - Double_t start = 0; - if (gPerfStats) start = TTimeStamp(); - - // Take into acoount possible prefixes - TFile::EFileType typ = TFile::kDefault; - TString fname = gEnv->GetValue("Path.Localroot",""); - if (!fname.IsNull()) - typ = TFile::GetType(fFilename, "", &fname); - if (typ != TFile::kLocal) - fname = fFilename; - fFile = TFile::Open(fname); - - if (gPerfStats) { - gPerfStats->FileOpenEvent(fFile, fFilename, start); - fOldBytesRead = 0; - } - - if (dirsave) dirsave->cd(); - - if (!fFile || fFile->IsZombie() ) { - if (fFile) - Error("Process","Cannot open file: %s (%s)", - fFilename.Data(), strerror(fFile->GetErrno()) ); - else - Error("Process","Cannot open file: %s (errno unavailable)", - fFilename.Data()); - // cleanup ? - return -1; - } - PDB(kLoop,2) Info("LoadDir","Opening file: %s", fFilename.Data() ); - ret = 1; - } - - // Check Directory - if ( fDir == 0 || fPath != fElem->GetDirectory() ) { - TDirectory *dirsave = gDirectory; - - fPath = fElem->GetDirectory(); - if ( !fFile->cd(fPath) ) { - Error("Process","Cannot cd to: %s", - fPath.Data() ); - return -1; - } - PDB(kLoop,2) Info("Process","Cd to: %s", fPath.Data() ); - fDir = gDirectory; - if (dirsave) dirsave->cd(); - ret = 1; - } - - return ret; -} - -//______________________________________________________________________________ -Long64_t TEventIter::GetEntryNumber(Long64_t next) -{ - // Get the entry number, taking into account event/entry lists - - Long64_t entry = next; - // Set entry number; if data iteration we may need to test the entry or event lists - if (TestBit(TEventIter::kData)) { - if (fEntryList){ - entry = fEntryList->GetEntry(next); - } else if (fEventList) { - entry = fEventList->GetEntry(next); - } - } - // Pre-event processing - PreProcessEvent(entry); - // Done - return entry; -} - -//------------------------------------------------------------------------ - - -ClassImp(TEventIterUnit); - -//////////////////////////////////////////////////////////////////////////////// -/// Default constructor - -TEventIterUnit::TEventIterUnit() -{ - fDSet = 0; - fElem = 0; - fSel = 0; - fNum = 0; - fCurrent = 0; - fStop = kFALSE; - fOldBytesRead = 0; // Measures the bytes written -} - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor - -TEventIterUnit::TEventIterUnit(TDSet* dset, TSelector *sel, Long64_t num) -{ - fDSet = dset; - fElem = 0; - fSel = sel; - fNum = num; - fCurrent = 0; - fStop = kFALSE; - fOldBytesRead = 0; // Measures the bytes written -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get loop range - -Int_t TEventIterUnit::GetNextPacket(Long64_t &fst, Long64_t &num) -{ - if (gPerfStats) { - Long64_t totBytesWritten = TFile::GetFileBytesWritten(); - Long64_t bytesWritten = totBytesWritten - fOldBytesRead; - PDB(kLoop, 2) Info("GetNextPacket", "bytes written: %lld", bytesWritten); - gPerfStats->SetBytesRead(bytesWritten); - fOldBytesRead = totBytesWritten; - } - - if (fDSet->TestBit(TDSet::kIsLocal)) { - if (fElem) { - if (fPackets) { - fPackets->Add(fElem); - PDB(kLoop, 2) - Info("GetNextEvent", "packet added to list (sz: %d)", fPackets->GetSize()); - fElem = 0; - } else { - SafeDelete(fElem); - } - return -1; - } else { - fElem = new TDSetElement("", "", "", 0, fNum); - fElem->SetBit(TDSetElement::kEmpty); - } - } else { - if (fPackets && fElem) { - fPackets->Add(fElem); - PDB(kLoop, 2) - Info("GetNextEvent", "packet added to list (sz: %d)", fPackets->GetSize()); - fElem = 0; - } else { - SafeDelete(fElem); - } - if (!(fElem = fDSet->Next())) - return -1; - } - fElem->SetBit(TDSetElement::kNewPacket); - - if (!fElem->TestBit(TDSetElement::kEmpty)) { - Error("GetNextPacket", "data element must be set to kEmtpy"); - return -1; - } - - // Set output - num = fElem->GetNum(); - if (num == 0) return -1; - fst = fElem->GetFirst(); - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next event - -Long64_t TEventIterUnit::GetNextEvent() -{ - if (fStop || fNum == 0) - return -1; - - if (fElem) fElem->ResetBit(TDSetElement::kNewPacket); - - while (fElem == 0 || fCurrent == 0) { - - if (gPerfStats) { - Long64_t totBytesWritten = TFile::GetFileBytesWritten(); - Long64_t bytesWritten = totBytesWritten - fOldBytesRead; - PDB(kLoop, 2) Info("GetNextEvent", "bytes written: %lld", bytesWritten); - gPerfStats->SetBytesRead(bytesWritten); - fOldBytesRead = totBytesWritten; - } - - if (fDSet->TestBit(TDSet::kIsLocal)) { - if (fElem) { - if (fPackets) { - fPackets->Add(fElem); - PDB(kLoop, 2) - Info("GetNextEvent", "packet added to list (sz: %d)", fPackets->GetSize()); - fElem = 0; - } else { - SafeDelete(fElem); - } - return -1; - } else { - fElem = new TDSetElement("", "", "", 0, fNum); - fElem->SetBit(TDSetElement::kEmpty); - } - } else { - if (fPackets && fElem) { - fPackets->Add(fElem); - PDB(kLoop, 2) - Info("GetNextEvent", "packet added to list (sz: %d)", fPackets->GetSize()); - fElem = 0; - } else { - SafeDelete(fElem); - } - if (!(fElem = fDSet->Next())) - return -1; - } - fElem->SetBit(TDSetElement::kNewPacket); - - if (!fElem->TestBit(TDSetElement::kEmpty)) { - Error("GetNextEvent", "data element must be set to kEmtpy"); - return -1; - } - - fNum = fElem->GetNum(); - if (!(fCurrent = fNum)) { - fNum = 0; - return -1; - } - fFirst = fElem->GetFirst(); - } - Long64_t event = fNum - fCurrent + fFirst ; - --fCurrent; - return event; -} - -//------------------------------------------------------------------------ - - -ClassImp(TEventIterObj); - -//////////////////////////////////////////////////////////////////////////////// -/// Default ctor. - -TEventIterObj::TEventIterObj() -{ - fKeys = 0; - fNextKey = 0; - fObj = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TEventIterObj::TEventIterObj(TDSet *dset, TSelector *sel, Long64_t first, Long64_t num) - : TEventIter(dset,sel,first,num) -{ - fClassName = dset->GetType(); - fKeys = 0; - fNextKey = 0; - fObj = 0; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TEventIterObj::~TEventIterObj() -{ - // delete fKeys ? - delete fNextKey; - delete fObj; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get loop range - -Int_t TEventIterObj::GetNextPacket(Long64_t &first, Long64_t &num) -{ - SafeDelete(fElem); - - if (fStop || fNum == 0) return -1; - - while (fElem == 0 || fCur < fFirst-1) { - - if (gPerfStats && fFile) { - Long64_t bytesRead = fFile->GetBytesRead(); - gPerfStats->SetBytesRead(bytesRead - fOldBytesRead); - fOldBytesRead = bytesRead; - } - - if (fElem) { - // Save it to the list of processed packets - if (fPackets) { - fPackets->Add(fElem); - fElem = 0; - } else { - SafeDelete(fElem); - } - } - fElem = fDSet->Next(fKeys->GetSize()); - if (fElem && fElem->GetEntryList()) { - Error("GetNextPacket", "entry- or event-list not available"); - return -1; - } - - if ( fElem == 0 ) { - fNum = 0; - return -1; - } - fElem->SetBit(TDSetElement::kNewPacket); - - Int_t r = LoadDir(); - - if ( r == -1 ) { - - // Error has been reported - fNum = 0; - return -1; - - } else if ( r == 1 ) { - - // New file and/or directory - fKeys = fDir->GetListOfKeys(); - fNextKey = new TIter(fKeys); - } - - // Validate values for this element - fElemFirst = fElem->GetFirst(); - fElemNum = fElem->GetNum(); - if (fElem->GetEntryList()) { - if (!(fEntryList = dynamic_cast(fElem->GetEntryList()))) - fEventList = dynamic_cast(fElem->GetEntryList()); - } - fEventListPos = 0; - if (fEntryList) - fElemNum = fEntryList->GetEntriesToProcess(); - else if (fEventList) - fElemNum = fEventList->GetN(); - - Long64_t tnum = fKeys->GetSize(); - - if ( fElemFirst > tnum ) { - Error("GetNextPacket","First (%lld) higher then number of keys (%lld) in %s", - fElemFirst, tnum, fElem->GetName()); - fNum = 0; - return -1; - } - - if ( fElemNum == -1 ) { - fElemNum = tnum - fElemFirst; - } else if ( fElemFirst+fElemNum > tnum ) { - Error("GetNextPacket","Num (%lld) + First (%lld) larger then number of keys (%lld) in %s", - fElemNum, fElemFirst, tnum, fElem->GetDirectory()); - fElemNum = tnum - fElemFirst; - } - - // Skip this element completely? - if ( fCur + fElemNum < fFirst ) { - fCur += fElemNum; - continue; - } - - // Position within this element - fNextKey->Reset(); - for(fElemCur = -1; fElemCur < fElemFirst-1 ; fElemCur++, fNextKey->Next()) { } - } - - first = ++fElemCur; - num = fElemNum; - - // Done - return 0; -} -//////////////////////////////////////////////////////////////////////////////// -/// To be executed before by TProofPlayer calling TSelector::Process - -void TEventIterObj::PreProcessEvent(Long64_t) -{ - --fNum; - ++fCur; - TKey *key = (TKey*) fNextKey->Next(); - TDirectory *dirsave = gDirectory; - fDir->cd(); - fObj = key->ReadObj(); - if (dirsave) dirsave->cd(); - fSel->SetObject(fObj); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next event - -Long64_t TEventIterObj::GetNextEvent() -{ - if (fStop || fNum == 0) return -1; - - if (fElem) fElem->ResetBit(TDSetElement::kNewPacket); - - while ( fElem == 0 || fElemNum == 0 || fCur < fFirst-1 ) { - - if (gPerfStats && fFile) { - Long64_t bytesRead = fFile->GetBytesRead(); - gPerfStats->SetBytesRead(bytesRead - fOldBytesRead); - fOldBytesRead = bytesRead; - } - - if (fElem) { - // Save it to the list of processed packets - if (fPackets) { - fPackets->Add(fElem); - fElem = 0; - } else { - SafeDelete(fElem); - } - } - fElem = fDSet->Next(fKeys->GetSize()); - if (fElem && fElem->GetEntryList()) { - Error("GetNextEvent", "Entry- or event-list not available"); - return -1; - } - - if ( fElem == 0 ) { - fNum = 0; - return -1; - } - fElem->SetBit(TDSetElement::kNewPacket); - - Int_t r = LoadDir(); - - if ( r == -1 ) { - - // Error has been reported - fNum = 0; - return -1; - - } else if ( r == 1 ) { - - // New file and/or directory - fKeys = fDir->GetListOfKeys(); - fNextKey = new TIter(fKeys); - } - - // Validate values for this element - fElemFirst = fElem->GetFirst(); - fElemNum = fElem->GetNum(); - if (fElem->GetEntryList()) { - if (!(fEntryList = dynamic_cast(fElem->GetEntryList()))) - fEventList = dynamic_cast(fElem->GetEntryList()); - } - fEventListPos = 0; - if (fEntryList) - fElemNum = fEntryList->GetEntriesToProcess(); - else if (fEventList) - fElemNum = fEventList->GetN(); - - Long64_t num = fKeys->GetSize(); - - if ( fElemFirst > num ) { - Error("GetNextEvent","First (%lld) higher then number of keys (%lld) in %s", - fElemFirst, num, fElem->GetName()); - fNum = 0; - return -1; - } - - if ( fElemNum == -1 ) { - fElemNum = num - fElemFirst; - } else if ( fElemFirst+fElemNum > num ) { - Error("GetNextEvent","Num (%lld) + First (%lld) larger then number of keys (%lld) in %s", - fElemNum, fElemFirst, num, fElem->GetDirectory()); - fElemNum = num - fElemFirst; - } - - // Skip this element completely? - if ( fCur + fElemNum < fFirst ) { - fCur += fElemNum; - continue; - } - - // Position within this element. TODO: more efficient? - fNextKey->Reset(); - for(fElemCur = -1; fElemCur < fElemFirst-1 ; fElemCur++, fNextKey->Next()) { } - } - - --fElemNum; - ++fElemCur; - - // Pre-event processing - PreProcessEvent(fElemCur); - - return fElemCur; -} - -//------------------------------------------------------------------------ - -//////////////////////////////////////////////////////////////////////////////// -/// Default ctor. - -TEventIterTree::TFileTree::TFileTree(const char *name, TFile *f, Bool_t islocal) - : TNamed(name, ""), fUsed(kFALSE), fIsLocal(islocal), fFile(f) -{ - fTrees = new TList; - fTrees->SetOwner(); -} -//////////////////////////////////////////////////////////////////////////////// -/// Default dtor. - -TEventIterTree::TFileTree::~TFileTree() -{ - // Avoid destroying the cache; must be placed before deleting the trees - TTree *tree = (TTree *)fTrees->First(); - while (tree) { - fFile->SetCacheRead(0, tree); - tree = (TTree *)fTrees->After(tree); - } - SafeDelete(fTrees); - SafeDelete(fFile); -} - -ClassImp(TEventIterTree); - -//////////////////////////////////////////////////////////////////////////////// -/// Default ctor. - -TEventIterTree::TEventIterTree() -{ - fTree = 0; - fTreeCache = 0; - fUseTreeCache = 1; - fCacheSize = -1; - fTreeCacheIsLearning = kTRUE; - fFileTrees = 0; - fUseParallelUnzip = 0; - fDontCacheFiles = kFALSE; - SetBit(TEventIter::kData); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TEventIterTree::TEventIterTree(TDSet *dset, TSelector *sel, Long64_t first, Long64_t num) - : TEventIter(dset,sel,first,num) -{ - fTreeName = dset->GetObjName(); - fTree = 0; - fTreeCache = 0; - fTreeCacheIsLearning = kTRUE; - fFileTrees = new TList; - fFileTrees->SetOwner(); - fUseTreeCache = gEnv->GetValue("ProofPlayer.UseTreeCache", 1); - fCacheSize = gEnv->GetValue("ProofPlayer.CacheSize", -1); - fUseParallelUnzip = gEnv->GetValue("ProofPlayer.UseParallelUnzip", 0); - if (fUseParallelUnzip) { - TTreeCacheUnzip::SetParallelUnzip(TTreeCacheUnzip::kEnable); - } else { - TTreeCacheUnzip::SetParallelUnzip(TTreeCacheUnzip::kDisable); - } - fDontCacheFiles = gEnv->GetValue("ProofPlayer.DontCacheFiles", 0); - SetBit(TEventIter::kData); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TEventIterTree::~TEventIterTree() -{ - // Delete the tree cache ... - SafeDelete(fTreeCache); - // ... and the remaining open files - SafeDelete(fFileTrees); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the size in bytes of the cache, if any -/// Return -1 if not used - -Long64_t TEventIterTree::GetCacheSize() -{ - if (fUseTreeCache) return fCacheSize; - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the number of entries in the learning phase - -Int_t TEventIterTree::GetLearnEntries() -{ - return TTreeCache::GetLearnEntries(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a Tree for the main TDSetElement and for all the friends. -/// Returns the main tree or 0 in case of an error. - -TTree* TEventIterTree::GetTrees(TDSetElement *elem) -{ - // Reset used flags - TIter nxft(fFileTrees); - TFileTree *ft = 0; - while ((ft = (TFileTree *)nxft())) - ft->fUsed = kFALSE; - - Bool_t localfile = kFALSE; - TTree* main = Load(elem, localfile); - - if (main && main != fTree) { - // Set the file cache - if (fUseTreeCache) { - TFile *curfile = main->GetCurrentFile(); - if (curfile) { - if (!fTreeCache) { - main->SetCacheSize(fCacheSize); - fTreeCache = (TTreeCache *)curfile->GetCacheRead(main); - if (fCacheSize < 0) fCacheSize = main->GetCacheSize(); - } else { - fTreeCache->ResetCache(); - curfile->SetCacheRead(fTreeCache, main); - fTreeCache->UpdateBranches(main); - } - if (fTreeCache) { - fTreeCacheIsLearning = fTreeCache->IsLearning(); - if (fTreeCacheIsLearning) - Info("GetTrees","the tree cache is in learning phase"); - } - } else { - Warning("GetTrees", "default tree does nto have a file attached: corruption? Tree cache untouched"); - } - } else { - // Disable the cache - main->SetCacheSize(0); - } - } - Bool_t loc = kFALSE; - // Also the friends - TList *friends = elem->GetListOfFriends(); - if (friends) { - TIter nxf(friends); - TDSetElement *dse = 0; - while ((dse = (TDSetElement *) nxf())) { - // The alias, if any, is in the element name options ('friend_alias=|') - TUrl uf(dse->GetName()); - TString uo(uf.GetOptions()), alias; - Int_t from = kNPOS; - if ((from = uo.Index("friend_alias=")) != kNPOS) { - from += strlen("friend_alias="); - if (!uo.Tokenize(alias, from, "|")) - Warning("GetTrees", "empty 'friend_alias' found for tree friend"); - // The options may be used for other things, so remove the internal strings once decoded - uo.ReplaceAll(TString::Format("friend_alias=%s|", alias.Data()), ""); - uf.SetOptions(uo); - dse->SetName(uf.GetUrl()); - } - TTree *friendTree = Load(dse, loc, dse->GetObjName()); - if (friendTree && main) { - // Make sure it has not yet been added - Bool_t addfriend = kTRUE; - TList *frnds = main->GetListOfFriends(); - if (frnds) { - TIter xnxf(frnds); - TFriendElement *fe = 0; - while ((fe = (TFriendElement *) xnxf())) { - if (fe->GetTree() == friendTree) { - addfriend = kFALSE; - break; - } - } - } - if (addfriend) { - if (alias.IsNull()) - main->AddFriend(friendTree); - else - main->AddFriend(friendTree, alias); - } - } else { - return 0; - } - } - } - - // Remove instances not used - nxft.Reset(); - while ((ft = (TFileTree *)nxft())) { - if (!(ft->fUsed)) { - fFileTrees->Remove(ft); - delete ft; - } - } - - // Done, successfully or not - return main; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Load a tree from s TDSetElement - -TTree* TEventIterTree::Load(TDSetElement *e, Bool_t &localfile, const char *objname) -{ - if (!e) { - Error("Load", "undefined element"); - return (TTree *)0; - } - - const char *fn = e->GetFileName(); - const char *dn = e->GetDirectory(); - const char *tn = 0; - if (objname && strlen(objname) > 0) { - tn = objname; - } else { - tn = (fDSet->GetObjName() && strlen(fDSet->GetObjName()) > 0) - ? fDSet->GetObjName() : e->GetObjName(); - if (!tn || (tn && strlen(tn) <= 0)) tn = "*"; - } - PDB(kLoop,2) - Info("Load","loading: fn:'%s' dn:'%s' tn:'%s'", fn, dn, tn); - - TFile *f = 0; - - // Check if the file is already open - TString names(fn); - TString name; - Ssiz_t from = 0; - TFileTree *ft = 0; - while (names.Tokenize(name,from,"|")) { - TString key(TUrl(name).GetFileAndOptions()); - if ((ft = (TFileTree *) fFileTrees->FindObject(key.Data()))) { - f = ft->fFile; - break; - } - } - - // Open the file, if needed - if (!f) { - TFile::EFileType typ = TFile::kDefault; - TString fname = gEnv->GetValue("Path.Localroot",""); - if (!fname.IsNull()) - typ = TFile::GetType(fn, "", &fname); - if (typ != TFile::kLocal) { - fname = fn; - } else { - localfile = kTRUE; - } - - // Open the file - f = TFile::Open(fname); - if (!f) { - Error("Load","file '%s' ('%s') could not be open", fn, fname.Data()); - return (TTree *)0; - } - -#if defined(R__MACOSX) - // If requested set the no cache mode - if (fDontCacheFiles && localfile) { - fcntl(f->GetFd(), F_NOCACHE, 1); - } -#endif - - // Create TFileTree instance in the list - ft = new TFileTree(TUrl(f->GetName()).GetFileAndOptions(), f, localfile); - fFileTrees->Add(ft); - } else { - // Fill locality boolean - localfile = ft->fIsLocal; - PDB(kLoop,2) - Info("Load","file '%s' already open (local:%d)", fn, localfile); - } - - // Check if the tree is already loaded - if (ft && ft->fTrees->GetSize() > 0) { - TTree *t = 0; - if (!strcmp(tn, "*")) - t = (TTree *) ft->fTrees->First(); - else - t = (TTree *) ft->fTrees->FindObject(tn); - if (t) { - ft->fUsed = kTRUE; - return t; - } - } - - TDirectory *dd = f; - // Change dir, if required - if (dn && !(dd = f->GetDirectory(dn))) { - Error("Load","Cannot get to: %s", dn); - return (TTree *)0; - } - PDB(kLoop,2) - Info("Load","got directory: %s", dn); - - // If a wild card we will use the first object of the type - // requested compatible with the reg expression we got - TString on(tn); - TString sreg(tn); - if (sreg.Length() <= 0 || sreg == "" || sreg.Contains("*")) { - if (sreg.Contains("*")) - sreg.ReplaceAll("*", ".*"); - else - sreg = ".*"; - TRegexp re(sreg); - if (dd->GetListOfKeys()) { - TIter nxk(dd->GetListOfKeys()); - TKey *k = 0; - while ((k = (TKey *) nxk())) { - if (!strcmp(k->GetClassName(), "TTree")) { - TString kn(k->GetName()); - if (kn.Index(re) != kNPOS) { - on = kn; - break; - } - } - } - } - } - - // Point to the key - TKey *key = dd->GetKey(gSystem->BaseName(on)); - if (key == 0) { - Error("Load", "Cannot find tree \"%s\" in %s", tn, fn); - return (TTree*)0; - } - - PDB(kLoop,2) Info("Load", "Reading: %s", tn); - - auto tree = key->ReadObject(); - dd->cd(); - - if (tree == 0) { - Error("Load", "Cannot obj to tree \"%s\"", tn); - return (TTree*)0; - } - - // Add track in the cache - ft->fTrees->Add(tree); - ft->fUsed = kTRUE; - PDB(kLoop,2) - Info("Load","TFileTree for '%s' flagged as 'in-use' ...", ft->GetName()); - - // Done - return tree; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get loop range - -Int_t TEventIterTree::GetNextPacket(Long64_t &first, Long64_t &num) -{ - if (first > -1) fEntryListPos = first; - - if (fStop || fNum == 0) return -1; - - Bool_t attach = kFALSE; - - // When files are aborted during processing (via TSelector::kAbortFile) the player - // invalidates the element by settign this bit. We need to ask for a new packet - Bool_t corrupted = kFALSE; - Long64_t rest = -1; - if (fElem) { - corrupted = (fElem->TestBit(TDSetElement::kCorrupted)) ? kTRUE : kFALSE; - rest = fElem->GetNum(); - if (fElemCur >= 0) rest -= (fElemCur + 1 - fElemFirst); - } - - SafeDelete(fElem); - - while (fElem == 0 || fElemNum == 0 || fCur < fFirst-1) { - - if (gPerfStats && fTree) { - Long64_t totBytesRead = fTree->GetCurrentFile()->GetBytesRead(); - Long64_t bytesRead = totBytesRead - fOldBytesRead; - gPerfStats->SetBytesRead(bytesRead); - fOldBytesRead = totBytesRead; - } - - if (fElem) { - // Save it to the list of processed packets - if (fPackets) { - fPackets->Add(fElem); - fElem = 0; - } else { - SafeDelete(fElem); - } - } - while (!fElem) { - // For a corrupted/invalid file the request for a new packet is with totalEntries = -1 - // (the default) so that the packetizer invalidates the element - if (corrupted) { - fElem = fDSet->Next(rest); - } else if (fTree) { - fElem = fDSet->Next(fTree->GetEntries()); - } else { - fElem = fDSet->Next(); - } - - if (!fElem) { - // End of processing - fNum = 0; - return -1; - } - corrupted = kFALSE; - fElem->SetBit(TDSetElement::kNewPacket); - fElem->ResetBit(TDSetElement::kCorrupted); - - TTree *newTree = GetTrees(fElem); - if (newTree) { - if (newTree != fTree) { - // The old tree is owned by TFileTree and will be deleted there - fTree = newTree; - attach = kTRUE; - fOldBytesRead = (fTree->GetCurrentFile()) ? fTree->GetCurrentFile()->GetBytesRead() : 0; - } - // Set range to be analysed - if (fTreeCache) - fTreeCache->SetEntryRange(fElem->GetFirst(), - fElem->GetFirst() + fElem->GetNum() - 1); - } else { - // Could not open this element: ask for another one - SafeDelete(fElem); - // The current tree, if any, is not valid anymore - fTree = 0; - } - } - - // Validate values for this element - fElemFirst = fElem->GetFirst(); - fElemNum = fElem->GetNum(); - fEntryList = 0; - fEventList = 0; - if (fElem->GetEntryList()) { - if (!(fEntryList = dynamic_cast(fElem->GetEntryList()))) - fEventList = dynamic_cast(fElem->GetEntryList()); - } - fEntryListPos = fElemFirst; - fEventListPos = 0; - if (fEntryList) - fElemNum = fEntryList->GetEntriesToProcess(); - else if (fEventList) - fElemNum = fEventList->GetN(); - - Long64_t tnum = (Long64_t) fTree->GetEntries(); - - if (!fEntryList && !fEventList) { - if ( fElemFirst > tnum ) { - Error("GetNextPacket", "first (%lld) higher then number of entries (%lld) in %s", - fElemFirst, tnum, fElem->GetObjName()); - fNum = 0; - return -1; - } - if ( fElemNum == -1 ) { - fElemNum = tnum - fElemFirst; - } else if ( fElemFirst+fElemNum > tnum ) { - Error("GetNextPacket", "num (%lld) + first (%lld) larger then number of entries (%lld) in %s", - fElemNum, fElemFirst, tnum, fElem->GetName()); - fElemNum = tnum - fElemFirst; - } - - // Skip this element completely? - if ( fCur + fElemNum < fFirst ) { - fCur += fElemNum; - continue; - } - // Position within this element - fElemCur = fElemFirst-1; - } - } - - if (attach) { - PDB(kLoop,1) Info("GetNextPacket", "call Init(%p) and Notify()",fTree); - fSel->Init(fTree); - fSel->Notify(); - TIter next(fSel->GetOutputList()); - TEntryList *elist=0; - while ((elist=(TEntryList*)next())){ - if (elist->InheritsFrom(TEntryList::Class())) - elist->SetTree(fTree->GetName(), fElem->GetFileName()); - } - if (fSel->GetAbort() == TSelector::kAbortProcess) { - // the error has been reported - return -1; - } - attach = kFALSE; - } - - // Fill the output now - num = fElemNum; - if (fEntryList) { - first = fEntryListPos; - } else if (fEventList){ - first = fEventListPos; - } else { - first = fElemFirst; - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Actions to be done just before processing entry 'entry'. -/// Called by TProofPlayer. - -void TEventIterTree::PreProcessEvent(Long64_t entry) -{ - if (!(fEntryList || fEventList)) { - --fNum; - ++fCur; - } - - // Signal ending of learning phase - if (fTreeCache && fTreeCacheIsLearning) { - if (!(fTreeCache->IsLearning())) { - fTreeCacheIsLearning = kFALSE; - if (gProofServ) gProofServ->RestartComputeTime(); - } - } - - // For prefetching - if (fTree->LoadTree(entry) < 0) { - Warning("PreEventProcess", "problems setting entry in TTree"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next event - -Long64_t TEventIterTree::GetNextEvent() -{ - if (fStop || fNum == 0) return -1; - - Bool_t attach = kFALSE; - - // When files are aborted during processing (via TSelector::kAbortFile) the player - // invalidates the element by settign this bit. We need to ask for a new packet - Bool_t corrupted = (fElem && fElem->TestBit(TDSetElement::kCorrupted)) ? kTRUE : kFALSE; - - if (fElem) fElem->ResetBit(TDSetElement::kNewPacket); - - while ( fElem == 0 || fElemNum == 0 || fCur < fFirst-1 || corrupted) { - - if (gPerfStats && fTree) { - Long64_t totBytesRead = fTree->GetCurrentFile()->GetBytesRead(); - Long64_t bytesRead = totBytesRead - fOldBytesRead; - gPerfStats->SetBytesRead(bytesRead); - fOldBytesRead = totBytesRead; - } - - Long64_t rest = -1; - if (fElem) { - rest = fElem->GetNum(); - if (fElemCur >= 0) rest -= (fElemCur + 1 - fElemFirst); - // Save it to the list of processed packets - if (fPackets) { - fPackets->Add(fElem); - fElem = 0; - } else { - SafeDelete(fElem); - } - } - - while (!fElem) { - // For a corrupted/invalid file the request for a new packet is with totalEntries = -1 - // (the default) so that the packetizer invalidates the element - if (corrupted) { - fElem = fDSet->Next(rest); - } else if (fTree) { - fElem = fDSet->Next(fTree->GetEntries()); - } else { - fElem = fDSet->Next(); - } - - if (!fElem) { - // End of processing - fNum = 0; - return -1; - } - corrupted = kFALSE; - fElem->SetBit(TDSetElement::kNewPacket); - fElem->ResetBit(TDSetElement::kCorrupted); - - TTree *newTree = GetTrees(fElem); - if (newTree) { - if (newTree != fTree) { - // The old tree is owned by TFileTree and will be deleted there - fTree = newTree; - attach = kTRUE; - fOldBytesRead = (fTree->GetCurrentFile()) ? fTree->GetCurrentFile()->GetBytesRead() : 0; - } - // Set range to be analysed - if (fTreeCache) - fTreeCache->SetEntryRange(fElem->GetFirst(), - fElem->GetFirst() + fElem->GetNum() - 1); - } else { - // Could not open this element: ask for another one - SafeDelete(fElem); - // The current tree, if any, is not valid anymore - fTree = 0; - } - } - - // Validate values for this element - fElemFirst = fElem->GetFirst(); - fElemNum = fElem->GetNum(); - fEntryList = dynamic_cast(fElem->GetEntryList()); - fEventList = (fEntryList) ? (TEventList *)0 - : dynamic_cast(fElem->GetEntryList()); - fEntryListPos = fElemFirst; - fEventListPos = 0; - if (fEntryList) - fElemNum = fEntryList->GetEntriesToProcess(); - else if (fEventList) - fElemNum = fEventList->GetN(); - - Long64_t num = (Long64_t) fTree->GetEntries(); - - if (!fEntryList && !fEventList) { - if ( fElemFirst > num ) { - Error("GetNextEvent", "first (%lld) higher then number of entries (%lld) in %s", - fElemFirst, num, fElem->GetObjName()); - fNum = 0; - return -1; - } - if ( fElemNum == -1 ) { - fElemNum = num - fElemFirst; - } else if ( fElemFirst+fElemNum > num ) { - Error("GetNextEvent", "num (%lld) + first (%lld) larger then number of entries (%lld) in %s", - fElemNum, fElemFirst, num, fElem->GetName()); - fElemNum = num - fElemFirst; - } - - // Skip this element completely? - if ( fCur + fElemNum < fFirst ) { - fCur += fElemNum; - continue; - } - // Position within this element. TODO: more efficient? - fElemCur = fElemFirst-1; - } - } - - if ( attach ) { - PDB(kLoop,1) Info("GetNextEvent", "call Init(%p) and Notify()",fTree); - fSel->Init(fTree); - fSel->Notify(); - TIter next(fSel->GetOutputList()); - TEntryList *elist=0; - while ((elist=(TEntryList*)next())){ - if (elist->InheritsFrom(TEntryList::Class())) - elist->SetTree(fTree->GetName(), fElem->GetFileName()); - } - if (fSel->GetAbort() == TSelector::kAbortProcess) { - // the error has been reported - return -1; - } - attach = kFALSE; - } - Long64_t rv; - - if (fEntryList){ - --fElemNum; - rv = fEntryList->GetEntry(fEntryListPos); - fEntryListPos++; - } else if (fEventList) { - --fElemNum; - rv = fEventList->GetEntry(fEventListPos); - fEventListPos++; - } else { - --fElemNum; - ++fElemCur; - rv = fElemCur; - } - - // Pre-event processing - PreProcessEvent(rv); - - return rv; -} diff --git a/proof/proofplayer/src/TOutputListSelectorDataMap.cxx b/proof/proofplayer/src/TOutputListSelectorDataMap.cxx deleted file mode 100644 index ea1bfd800a3ee..0000000000000 --- a/proof/proofplayer/src/TOutputListSelectorDataMap.cxx +++ /dev/null @@ -1,330 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Axel Naumann, 2010-06-09 - -/************************************************************************* - * Copyright (C) 1995-2010, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TOutputListSelectorDataMap -\ingroup proofkernel - -Set the selector's data members to the corresponding elements of the -output list. - -*/ - -#include "TOutputListSelectorDataMap.h" - -#include "TClass.h" -#include "TDataMember.h" -#include "TExMap.h" -#include "THashTable.h" -#include "TList.h" -#include "TMemberInspector.h" -#include "TProofDebug.h" -#include "TSelector.h" - -#include - -namespace { - - static TClass* IsSettableDataMember(TDataMember* dm) { - if (!dm || !dm->IsaPointer() || dm->IsBasic()) return 0; - TString dtTypeName = dm->GetFullTypeName(); - if (!dtTypeName.EndsWith("*")) return 0; - dtTypeName.Remove(dtTypeName.Length()-1); - return TClass::GetClass(dtTypeName); - } - - class TSetSelDataMembers: public TMemberInspector { - public: - TSetSelDataMembers(const TOutputListSelectorDataMap& owner, TCollection* dmInfo, TList* output); - using TMemberInspector::Inspect; - void Inspect(TClass *cl, const char *parent, const char *name, const void *addr, Bool_t isTransient) override; - Ssiz_t GetNumSet() const { return fNumSet; } - private: - TCollection* fDMInfo; // output list object name / member name pairs for output list entries - TList* fOutputList; // merged output list - Ssiz_t fNumSet; // number of initialized data members - const TOutputListSelectorDataMap& fOwner; // owner, used for messaging - }; -} - -//////////////////////////////////////////////////////////////////////////////// - -TSetSelDataMembers::TSetSelDataMembers(const TOutputListSelectorDataMap& owner, - TCollection* dmInfo, TList* output): - fDMInfo(dmInfo), fOutputList(output), fNumSet(0), fOwner(owner) -{} - -//////////////////////////////////////////////////////////////////////////////// -/// This method is called by the ShowMembers() method for each -/// data member to recursively collect all base classes' members. -/// -/// cl is the pointer to the current class -/// parent is the parent name (in case of composed objects) -/// name is the data member name -/// addr is the data member address - -void TSetSelDataMembers::Inspect(TClass *cl, const char* parent, const char *name, const void *addr, Bool_t /* isTransient */) -{ - while (name[0] == '*') ++name; - - TObject* mapping = fDMInfo->FindObject(name); - if (!mapping) return; - - PDB(kOutput,1) fOwner.Info("SetDataMembers()", - "data member `%s%s::%s' maps to output list object `%s'", - cl->GetName(), parent, name, mapping->GetTitle()); - - TObject* outputObj = fOutputList->FindObject(mapping->GetTitle()); - if (!outputObj) { - PDB(kOutput,1) fOwner.Warning("SetDataMembers()", - "object `%s' not found in output list!", - mapping->GetTitle()); - return; - } - - // Check data member type - TDataMember *dm = cl->GetDataMember(name); - TClass* cldt = IsSettableDataMember(dm); - if (!cldt) { - PDB(kOutput,1) fOwner.Warning("SetDataMembers()", - "unusable data member `%s' should have been detected by TCollectDataMembers!", - name); - return; - } - - char *pointer = (char*)addr; - char **ppointer = (char**)(pointer); - if (*ppointer) { - // member points to something - replace instead of delete to not crash on deleting uninitialized values. - fOwner.Warning("SetDataMembers()", "potential memory leak: replacing data member `%s' != 0. " - "Please initialize %s to 0 in constructor %s::%s()", - name, name, cl->GetName(), cl->GetName()); - } - *ppointer = (char*)outputObj; - ++fNumSet; -} - - -namespace { - class TCollectDataMembers: public TMemberInspector { - public: - TCollectDataMembers(const TOutputListSelectorDataMap& owner): fOwner(owner) { } - ~TCollectDataMembers() override; - using TMemberInspector::Inspect; - void Inspect(TClass *cl, const char *parent, const char *name, const void *addr, Bool_t isTransient) override; - TExMap& GetMemberPointers() { return fMap; } - private: - TExMap fMap; //map of data member's value to TDataMember - const TOutputListSelectorDataMap& fOwner; //owner, used for messaging - }; -} - -//////////////////////////////////////////////////////////////////////////////// -/// This method is called by the ShowMembers() method for each -/// data member to recursively collect all base classes' members. -/// -/// cl is the pointer to the current class -/// parent is the parent name (in case of composed objects) -/// name is the data member name -/// addr is the data member address - -void TCollectDataMembers::Inspect(TClass *cl, const char* /*parent*/, const char *name, const void *addr, Bool_t /* isTransient */) -{ - TDataMember *dm = cl->GetDataMember(name); - if (!IsSettableDataMember(dm)) return; - - char *pointer = (char*)addr; - char **ppointer = (char**)(pointer); - char **p3pointer = (char**)(*ppointer); - if (p3pointer) { - // The data member points to something. - // Handle multiple pointers to the same output list object: - TObject* prev = (TObject*) (ptrdiff_t)fMap.GetValue((Long64_t)(ptrdiff_t)p3pointer); - if (prev) { - // We have a previous entry - is it a data member or already a TList (of data members)? - if (prev->InheritsFrom(TDataMember::Class())) { - fMap.Remove((Long64_t)(ptrdiff_t)p3pointer); - TList* dmList = new TList; - dmList->Add(prev); - dmList->Add(dm); - fMap.Add((Long64_t)(ptrdiff_t)p3pointer, (Long64_t)(ptrdiff_t)dmList); - } else { - TList* prevList = (TList*) prev; - prevList->Add(dm); - } - } else { - fMap.Add((Long64_t)(ptrdiff_t)p3pointer, (Long64_t)(ptrdiff_t)dm); - } - if (name[0] == '*') ++name; - PDB(kOutput,1) fOwner.Info("Init()", "considering data member `%s'", name); - } -} - -TCollectDataMembers::~TCollectDataMembers() { - // Destructor - - // Clean up collection of TDataMembers in fMap - TExMapIter iMembers(&fMap); - Long64_t key; - Long64_t value; - while (iMembers.Next(key, value)) { - TObject* obj = (TObject*) (ptrdiff_t) value; - if (obj->InheritsFrom(TList::Class())) { - delete obj; - } - } -} - -ClassImp(TOutputListSelectorDataMap); - -//////////////////////////////////////////////////////////////////////////////// -/// Create a mapper between output list items and TSelector data members. - -TOutputListSelectorDataMap::TOutputListSelectorDataMap(TSelector* sel /*= 0*/): - fMap(0) -{ - if (sel) Init(sel); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return static name for TOutputListSelectorDataMap objects. - -const char* TOutputListSelectorDataMap::GetName() const -{ - return "PROOF_TOutputListSelectorDataMap_object"; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Initialize the data member <-> output list mapping from a selector. - -Bool_t TOutputListSelectorDataMap::Init(TSelector* sel) -{ - if (!sel) { - PDB(kOutput,1) Warning("Init","Leave (no selector!)"); - return kFALSE; - } - TCollection* outList = sel->GetOutputList(); - if (!outList) { - PDB(kOutput,1) Info("Init()","Leave (no output)"); - return kFALSE; - } - - if (outList->FindObject(GetName())) { - // mapping already exists?! - PDB(kOutput,1) Warning("Init","Mapping already exists!"); - return kFALSE; - } - - if (fMap) delete fMap; - fMap = new THashTable; - fMap->SetOwner(); - - TCollectDataMembers cdm(*this); - if (!sel->IsA()->CallShowMembers(sel, cdm)) { - // failed to map - PDB(kOutput,1) Warning("Init","Failed to determine mapping!"); - return kFALSE; - } - PDB(kOutput,1) Info("Init()","Found %d data members.", - cdm.GetMemberPointers().GetSize()); - - // Iterate over output list entries and find data members pointing to the - // same value. Store that mapping (or a miss). - TIter iOutput(outList); - TObject* output; - TList oneDM; - while ((output = iOutput())) { - TObject* obj = (TObject*) (ptrdiff_t)cdm.GetMemberPointers().GetValue((Long64_t)(ptrdiff_t)output); - if (!obj) continue; - - TList* addAllDM = 0; - if (obj->InheritsFrom(TDataMember::Class())) { - oneDM.Add(obj); - addAllDM = &oneDM; - } else { - addAllDM = (TList*) obj; - } - TIter iDM(addAllDM); - TDataMember* dm = 0; - while ((dm = (TDataMember*) iDM())) { - fMap->Add(new TNamed(dm->GetName(), output->GetName())); - PDB(kOutput,1) Info("Init()","Data member `%s' corresponds to output `%s'", - dm->GetName(), output->GetName()); - } - oneDM.Clear(); - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Given an output list, set the data members of a TSelector. - -Bool_t TOutputListSelectorDataMap::SetDataMembers(TSelector* sel) const -{ - TList* output = sel->GetOutputList(); - if (!output || output->IsEmpty()) return kTRUE; - - // Set fSelector's data members - TSetSelDataMembers ssdm(*this, fMap, output); - Bool_t res = sel->IsA()->CallShowMembers(sel, ssdm); - PDB(kOutput,1) Info("SetDataMembers()","%s, set %d data members.", - (res ? "success" : "failure"), ssdm.GetNumSet()); - return res; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Merge another TOutputListSelectorDataMap object, check -/// consistency. - -Bool_t TOutputListSelectorDataMap::Merge(TObject* obj) -{ - TOutputListSelectorDataMap* other = dynamic_cast(obj); - if (!other) return kFALSE; - - // check for consistency - TIter iMapping(other->GetMap()); - TNamed* mapping = 0; - while ((mapping = (TNamed*)iMapping())) { - TObject* oldMap = fMap->FindObject(mapping->GetName()); - if (!oldMap) { - fMap->Add(new TNamed(*mapping)); - } else { - if (strcmp(oldMap->GetTitle(), mapping->GetTitle())) { - // ouch, contradicting maps! - PDB(kOutput,1) - Warning("Merge()", - "contradicting mapping for data member `%s' (output list entry `%s' vs. `%s'). " - "Cancelling automatic TSelector data member setting!", - mapping->GetName(), oldMap->GetTitle(), mapping->GetTitle()); - fMap->Clear(); - return kFALSE; - } - } - } - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Find a TOutputListSelectorDataMap in a collection - -TOutputListSelectorDataMap* TOutputListSelectorDataMap::FindInList(TCollection* coll) -{ - TIter iOutput(coll); - TObject* out = 0; - TOutputListSelectorDataMap* olsdm = 0; - while ((out = iOutput())) { - if (out->InheritsFrom(TOutputListSelectorDataMap::Class())) { - olsdm = dynamic_cast(out); - if (olsdm) break; - } - } - return olsdm; -} diff --git a/proof/proofplayer/src/TPacketizer.cxx b/proof/proofplayer/src/TPacketizer.cxx deleted file mode 100644 index 9478980a44d3b..0000000000000 --- a/proof/proofplayer/src/TPacketizer.cxx +++ /dev/null @@ -1,1324 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 18/03/02 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TPacketizer -\ingroup proofkernel - -This class generates packets to be processed on PROOF worker servers. -A packet is an event range (begin entry and number of entries) or -object range (first object and number of objects) in a TTree -(entries) or a directory (objects) in a file. -Packets are generated taking into account the performance of the -remote machine, the time it took to process a previous packet on -the remote machine, the locality of the database files, etc. - -*/ - -#include "TPacketizer.h" - -#include "Riostream.h" -#include "TDSet.h" -#include "TEnv.h" -#include "TError.h" -#include "TEventList.h" -#include "TEntryList.h" -#include "TMap.h" -#include "TMessage.h" -#include "TMonitor.h" -#include "TNtupleD.h" -#include "TObject.h" -#include "TParameter.h" -#include "TPerfStats.h" -#include "TProofDebug.h" -#include "TProof.h" -#include "TProofPlayer.h" -#include "TProofServ.h" -#include "TSlave.h" -#include "TSocket.h" -#include "TTimer.h" -#include "TUrl.h" -#include "TClass.h" -#include "TMath.h" - -// -// The following three utility classes manage the state of the -// work to be performed and the slaves involved in the process. -// A list of TFileNode(s) describes the hosts with files, each -// has a list of TFileStat(s) keeping the state for each TDSet -// element (file). -// -// The list of TSlaveStat(s) keep track of the work (being) done -// by each slave -// - - -//------------------------------------------------------------------------------ - -class TPacketizer::TFileStat : public TObject { - -private: - Bool_t fIsDone; // is this element processed - TFileNode *fNode; // my FileNode - TDSetElement *fElement; // location of the file and its range - Long64_t fNextEntry; // cursor in the range, -1 when done - -public: - TFileStat(TFileNode *node, TDSetElement *elem); - - Bool_t IsDone() const {return fIsDone;} - void SetDone() {fIsDone = kTRUE;} - TFileNode *GetNode() const {return fNode;} - TDSetElement *GetElement() const {return fElement;} - Long64_t GetNextEntry() const {return fNextEntry;} - void MoveNextEntry(Long64_t step) {fNextEntry += step;} -}; - - -TPacketizer::TFileStat::TFileStat(TFileNode *node, TDSetElement *elem) - : fIsDone(kFALSE), fNode(node), fElement(elem), fNextEntry(elem->GetFirst()) -{ -} - - -//------------------------------------------------------------------------------ - -class TPacketizer::TFileNode : public TObject { - -private: - TString fNodeName; // FQDN of the node - TList *fFiles; // TDSetElements (files) stored on this node - TObject *fUnAllocFileNext; // cursor in fFiles - TList *fActFiles; // files with work remaining - TObject *fActFileNext; // cursor in fActFiles - Int_t fMySlaveCnt; // number of slaves running on this node - Int_t fSlaveCnt; // number of external slaves processing files on this node - -public: - TFileNode(const char *name); - ~TFileNode() override { delete fFiles; delete fActFiles; } - - void IncMySlaveCnt() { fMySlaveCnt++; } - void IncSlaveCnt(const char *slave) { if (fNodeName != slave) fSlaveCnt++; } - void DecSlaveCnt(const char *slave) { if (fNodeName != slave) fSlaveCnt--; R__ASSERT(fSlaveCnt >= 0); } - Int_t GetSlaveCnt() const {return fMySlaveCnt + fSlaveCnt;} - Int_t GetNumberOfActiveFiles() const { return fActFiles->GetSize(); } - Bool_t IsSortable() const override { return kTRUE; } - - const char *GetName() const override { return fNodeName.Data(); } - - void Add(TDSetElement *elem) - { - TFileStat *f = new TFileStat(this,elem); - fFiles->Add(f); - if (fUnAllocFileNext == 0) fUnAllocFileNext = fFiles->First(); - } - - TFileStat *GetNextUnAlloc() - { - TObject *next = fUnAllocFileNext; - - if (next != 0) { - // make file active - fActFiles->Add(next); - if (fActFileNext == 0) fActFileNext = fActFiles->First(); - - // move cursor - fUnAllocFileNext = fFiles->After(fUnAllocFileNext); - } - - return (TFileStat *) next; - } - - TFileStat *GetNextActive() - { - TObject *next = fActFileNext; - - if (fActFileNext != 0) { - fActFileNext = fActFiles->After(fActFileNext); - if (fActFileNext == 0) fActFileNext = fActFiles->First(); - } - - return (TFileStat *) next; - } - - void RemoveActive(TFileStat *file) - { - if (fActFileNext == file) fActFileNext = fActFiles->After(file); - fActFiles->Remove(file); - if (fActFileNext == 0) fActFileNext = fActFiles->First(); - } - - Int_t Compare(const TObject *other) const override - { - // Must return -1 if this is smaller than obj, 0 if objects are equal - // and 1 if this is larger than obj. - const TFileNode *obj = dynamic_cast(other); - if (!obj) { - Error("Compare", "input is not a TPacketizer::TFileNode object"); - return 0; - } - - Int_t myVal = GetSlaveCnt(); - Int_t otherVal = obj->GetSlaveCnt(); - if (myVal < otherVal) { - return -1; - } else if (myVal > otherVal) { - return 1; - } else { - return 0; - } - } - - void Print(Option_t *) const override - { - std::cout << "OBJ: " << IsA()->GetName() << "\t" << fNodeName - << "\tMySlaveCount " << fMySlaveCnt - << "\tSlaveCount " << fSlaveCnt << std::endl; - } - - void Reset() - { - fUnAllocFileNext = fFiles->First(); - fActFiles->Clear(); - fActFileNext = 0; - fSlaveCnt = 0; - fMySlaveCnt = 0; - } -}; - - -TPacketizer::TFileNode::TFileNode(const char *name) - : fNodeName(name), fFiles(new TList), fUnAllocFileNext(0),fActFiles(new TList), - fActFileNext(0), fMySlaveCnt(0), fSlaveCnt(0) -{ - // Constructor - - fFiles->SetOwner(); - fActFiles->SetOwner(kFALSE); -} - - -//------------------------------------------------------------------------------ - -class TPacketizer::TSlaveStat : public TVirtualPacketizer::TVirtualSlaveStat { - -friend class TPacketizer; - -private: - TFileNode *fFileNode; // corresponding node or 0 - TFileStat *fCurFile; // file currently being processed - TDSetElement *fCurElem; // TDSetElement currently being processed - TProofProgressStatus *AddProcessed(TProofProgressStatus *st) override; -public: - TSlaveStat(TSlave *slave); - ~TSlaveStat() override; - - TFileNode *GetFileNode() const { return fFileNode; } - - void SetFileNode(TFileNode *node) { fFileNode = node; } -}; - - -TPacketizer::TSlaveStat::TSlaveStat(TSlave *slave) - : fFileNode(0), fCurFile(0), fCurElem(0) -{ - fSlave = slave; - fStatus = new TProofProgressStatus(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup - -TPacketizer::TSlaveStat::~TSlaveStat() -{ - SafeDelete(fStatus); -} - -TProofProgressStatus *TPacketizer::TSlaveStat::AddProcessed(TProofProgressStatus *st) -{ - // Update the status info to the 'st'. - // return the difference (*st - *fStatus) - - if (st) { - // The entriesis not correct in 'st' - Long64_t lastEntries = st->GetEntries() - fStatus->GetEntries(); - // The last proc time should not be added - fStatus->SetLastProcTime(0.); - // Get the diff - TProofProgressStatus *diff = new TProofProgressStatus(*st - *fStatus); - *fStatus += *diff; - // Set the correct value - fStatus->SetLastEntries(lastEntries); - return diff; - } else { - Error("AddProcessed", "status arg undefined"); - return 0; - } -} - -//------------------------------------------------------------------------------ - -ClassImp(TPacketizer); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TPacketizer::TPacketizer(TDSet *dset, TList *slaves, Long64_t first, - Long64_t num, TList *input, TProofProgressStatus *st) - : TVirtualPacketizer(input, st) -{ - PDB(kPacketizer,1) Info("TPacketizer", "Enter (first %lld, num %lld)", first, num); - - // Init pointer members - fPackets = 0; - fUnAllocated = 0; - fActive = 0; - fFileNodes = 0; - fMaxPerfIdx = 1; - fMaxSlaveCnt = 0; - fHeuristicPSiz = kFALSE; - fDefMaxWrkNode = kTRUE; - - if (!fProgressStatus) { - Error("TPacketizer", "No progress status"); - return; - } - - Long_t maxSlaveCnt = 0; - if (TProof::GetParameter(input, "PROOF_MaxSlavesPerNode", maxSlaveCnt) == 0) { - if (maxSlaveCnt < 0) { - Warning("TPacketizer", "PROOF_MaxSlavesPerNode must be positive"); - maxSlaveCnt = 0; - } - if (maxSlaveCnt > 0) fDefMaxWrkNode = kFALSE; - } else { - // Try also with Int_t (recently supported in TProof::SetParameter) - Int_t mxslcnt = -1; - if (TProof::GetParameter(input, "PROOF_MaxSlavesPerNode", mxslcnt) == 0) { - if (mxslcnt < 0) { - Warning("TPacketizer", "PROOF_MaxSlavesPerNode must be positive"); - mxslcnt = 0; - } - maxSlaveCnt = (Long_t) mxslcnt; - if (maxSlaveCnt > 0) fDefMaxWrkNode = kFALSE; - } - } - if (!maxSlaveCnt) { - maxSlaveCnt = gEnv->GetValue("Packetizer.MaxWorkersPerNode", slaves->GetSize()); - if (maxSlaveCnt != slaves->GetSize()) fDefMaxWrkNode = kFALSE; - } - if (maxSlaveCnt > 0) { - fMaxSlaveCnt = maxSlaveCnt; - PDB(kPacketizer,1) - Info("TPacketizer", "setting max number of workers per node to %ld", fMaxSlaveCnt); - } - - fPackets = new TList; - fPackets->SetOwner(); - - fFileNodes = new TList; - fFileNodes->SetOwner(); - fUnAllocated = new TList; - fUnAllocated->SetOwner(kFALSE); - fActive = new TList; - fActive->SetOwner(kFALSE); - - - fValid = kTRUE; - - // Resolve end-point urls to optmize distribution - // dset->Lookup(); // moved to TProofPlayerRemote::Process - - // Split into per host entries - dset->Reset(); - TDSetElement *e; - while ((e = (TDSetElement*)dset->Next())) { - if (e->GetValid()) continue; - - TUrl url = e->GetFileName(); - - // Map non URL filenames to dummy host - TString host; - if ( !url.IsValid() || - (strncmp(url.GetProtocol(),"root", 4) && - strncmp(url.GetProtocol(),"file", 4)) ) { - host = "no-host"; - } else if ( url.IsValid() && !strncmp(url.GetProtocol(),"file", 4)) { - host = "localhost"; - url.SetProtocol("root"); - } else { - host = url.GetHost(); - } - // Get full name for local hosts - if (host.Contains("localhost") || host == "127.0.0.1") { - url.SetHost(gSystem->HostName()); - host = url.GetHostFQDN(); - } - - TFileNode *node = (TFileNode*) fFileNodes->FindObject( host ); - - if (node == 0) { - node = new TFileNode(host); - fFileNodes->Add(node); - } - - node->Add( e ); - } - - fSlaveStats = new TMap; - fSlaveStats->SetOwner(kFALSE); - - // Record initial available workers - Int_t nwrks = AddWorkers(slaves); - Info("TPacketizer", "Initial number of workers: %d", nwrks); - - // Setup file & filenode structure - Reset(); - // Optimize the number of files to be open when running on subsample - Int_t validateMode = 0; - Int_t gprc = TProof::GetParameter(input, "PROOF_ValidateByFile", validateMode); - Bool_t byfile = (gprc == 0 && validateMode > 0 && num > -1) ? kTRUE : kFALSE; - if (num > -1) - PDB(kPacketizer,2) - Info("TPacketizer", - "processing subset of entries: validating by file? %s", byfile ? "yes": "no"); - ValidateFiles(dset, slaves, num, byfile); - - if (!fValid) return; - - // apply global range (first,num) to dset and rebuild structure - // ommitting TDSet elements that are not needed - - Int_t files = 0; - fTotalEntries = 0; - fUnAllocated->Clear(); // avoid dangling pointers - fActive->Clear(); - fFileNodes->Clear(); // then delete all objects - PDB(kPacketizer,2) Info("TPacketizer", "processing range: first %lld, num %lld", first, num); - - dset->Reset(); - Long64_t cur = 0; - while (( e = (TDSetElement*)dset->Next())) { - - // Skip invalid or missing file; It will be moved - // from the dset to the 'MissingFiles' list in the player. - if (!e->GetValid()) continue; - - // The dataset name, if any - if (fDataSet.IsNull() && e->GetDataSet() && strlen(e->GetDataSet())) - fDataSet = e->GetDataSet(); - - TUrl url = e->GetFileName(); - Long64_t eFirst = e->GetFirst(); - Long64_t eNum = e->GetNum(); - PDB(kPacketizer,2) - Info("TPacketizer", " --> '%s'", e->GetFileName()); - PDB(kPacketizer,2) - Info("TPacketizer", " --> first %lld, num %lld (cur %lld)", eFirst, eNum, cur); - - if (!e->GetEntryList()){ - // this element is before the start of the global range, skip it - if (cur + eNum < first) { - cur += eNum; - PDB(kPacketizer,2) - Info("TPacketizer", " --> skip element cur %lld", cur); - continue; - } - - // this element is after the end of the global range, skip it - if (num != -1 && (first+num <= cur)) { - cur += eNum; - PDB(kPacketizer,2) - Info("TPacketizer", " --> drop element cur %lld", cur); - continue; // break ?? - } - - Bool_t inRange = kFALSE; - if (cur <= first || (num != -1 && (first+num <= cur+eNum))) { - - if (cur <= first) { - // If this element contains the start of the global range - // adjust its start and number of entries - e->SetFirst( eFirst + (first - cur) ); - e->SetNum( e->GetNum() - (first - cur) ); - PDB(kPacketizer,2) - Info("TPacketizer", " --> adjust start %lld and end %lld", - eFirst + (first - cur), first + num - cur); - inRange = kTRUE; - } - if (num != -1 && (first+num <= cur+eNum)) { - // If this element contains the end of the global range - // adjust its number of entries - e->SetNum( first + num - e->GetFirst() - cur ); - PDB(kPacketizer,2) - Info("TPacketizer", " --> adjust end %lld", first + num - cur); - inRange = kTRUE; - } - - } else { - // Increment the counter ... - PDB(kPacketizer,2) - Info("TPacketizer", " --> increment 'cur' by %lld", eNum); - cur += eNum; - } - // Re-adjust eNum and cur, if needed - if (inRange) { - cur += eNum; - eNum = e->GetNum(); - } - - } else { - TEntryList *enl = dynamic_cast(e->GetEntryList()); - if (enl) { - eNum = enl->GetN(); - } else { - TEventList *evl = dynamic_cast(e->GetEntryList()); - eNum = evl ? evl->GetN() : eNum; - } - if (!eNum) - continue; - } - PDB(kPacketizer,2) - Info("TPacketizer", " --> next cur %lld", cur); - - // Map non URL filenames to dummy host - TString host; - if ( !url.IsValid() || - (strncmp(url.GetProtocol(),"root", 4) && - strncmp(url.GetProtocol(),"file", 4)) ) { - host = "no-host"; - } else if ( url.IsValid() && !strncmp(url.GetProtocol(),"file", 4)) { - host = "localhost"; - url.SetProtocol("root"); - } else { - host = url.GetHostFQDN(); - } - // Get full name for local hosts - if (host.Contains("localhost") || host == "127.0.0.1") { - url.SetHost(gSystem->HostName()); - host = url.GetHostFQDN(); - } - - TFileNode *node = (TFileNode*) fFileNodes->FindObject( host ); - - if ( node == 0 ) { - node = new TFileNode( host ); - fFileNodes->Add( node ); - } - - ++files; - fTotalEntries += eNum; - node->Add(e); - PDB(kPacketizer,2) e->Print("a"); - } - - PDB(kPacketizer,1) - Info("TPacketizer", "processing %lld entries in %d files on %d hosts", - fTotalEntries, files, fFileNodes->GetSize()); - - // Set the total number for monitoring - if (gPerfStats) - gPerfStats->SetNumEvents(fTotalEntries); - - Reset(); - - if (fFileNodes->GetSize() == 0) { - Info("TPacketizer", "no valid or non-empty file found: setting invalid"); - // No valid files: set invalid and return - fValid = kFALSE; - return; - } - - // Below we provide a possibility to change the way packet size is - // calculated or define the packet size directly. - // fPacketAsAFraction can be interpreted as follows: - // assuming all slaves have equal processing rate, - // packet size is (#events processed by 1 slave) / fPacketSizeAsAFraction. - // It substitutes 20 in the old formula to calculate the fPacketSize: - // fPacketSize = fTotalEntries / (20 * nslaves) - Long_t packetAsAFraction = 20; - if (TProof::GetParameter(input, "PROOF_PacketAsAFraction", packetAsAFraction) == 0) - Info("Process", "using alternate fraction of query time as a packet Size: %ld", - packetAsAFraction); - fPacketAsAFraction = (Int_t)packetAsAFraction; - - fPacketSize = 1; - if (TProof::GetParameter(input, "PROOF_PacketSize", fPacketSize) == 0) { - Info("Process","using alternate packet size: %lld", fPacketSize); - } else { - // Heuristic for starting packet size - fHeuristicPSiz = kTRUE; - Int_t nslaves = fSlaveStats->GetSize(); - if (nslaves > 0) { - fPacketSize = fTotalEntries / (fPacketAsAFraction * nslaves); - if (fPacketSize < 1) fPacketSize = 1; - } else { - fPacketSize = 1; - } - } - - PDB(kPacketizer,1) Info("TPacketizer", "Base Packetsize = %lld", fPacketSize); - - if (!fValid) - SafeDelete(fProgress); - - PDB(kPacketizer,1) Info("TPacketizer", "Return"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TPacketizer::~TPacketizer() -{ - if (fSlaveStats) { - fSlaveStats->DeleteValues(); - } - - SafeDelete(fPackets); - SafeDelete(fSlaveStats); - SafeDelete(fUnAllocated); - SafeDelete(fActive); - SafeDelete(fFileNodes); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Adds new workers. Returns the number of workers added, or -1 on failure. - -Int_t TPacketizer::AddWorkers(TList *workers) -{ - if (!workers) { - Error("AddWorkers", "Null list of new workers!"); - return -1; - } - - Int_t curNumOfWrks = fSlaveStats->GetEntries(); - - TSlave *sl; - TIter next(workers); - while (( sl = dynamic_cast(next()) )) - if (!fSlaveStats->FindObject(sl)) { - fSlaveStats->Add(sl, new TSlaveStat(sl)); - fMaxPerfIdx = sl->GetPerfIdx() > fMaxPerfIdx ? sl->GetPerfIdx() : fMaxPerfIdx; - } - - // If heuristic (and new workers) set the packet size - Int_t nwrks = fSlaveStats->GetSize(); - if (fHeuristicPSiz && nwrks > curNumOfWrks) { - if (nwrks > 0) { - fPacketSize = fTotalEntries / (fPacketAsAFraction * nwrks); - if (fPacketSize < 1) fPacketSize = 1; - } else { - fPacketSize = 1; - } - } - - // Update the max number that can access one file node if the default is used - if (fDefMaxWrkNode && nwrks > fMaxSlaveCnt) fMaxSlaveCnt = nwrks; - - // Done - return nwrks; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next unallocated file. - -TPacketizer::TFileStat *TPacketizer::GetNextUnAlloc(TFileNode *node) -{ - TFileStat *file = 0; - - if (node != 0) { - file = node->GetNextUnAlloc(); - if (file == 0) RemoveUnAllocNode(node); - } else { - while (file == 0 && ((node = NextUnAllocNode()) != 0)) { - file = node->GetNextUnAlloc(); - if (file == 0) RemoveUnAllocNode(node); - } - } - - if (file != 0) { - // if needed make node active - if (fActive->FindObject(node) == 0) { - fActive->Add(node); - } - } - - return file; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next unallocated node. - -TPacketizer::TFileNode *TPacketizer::NextUnAllocNode() -{ - fUnAllocated->Sort(); - PDB(kPacketizer,2) { - std::cout << "TPacketizer::NextUnAllocNode()" << std::endl; - fUnAllocated->Print(); - } - - TFileNode *fn = (TFileNode*) fUnAllocated->First(); - if (fn != 0 && fMaxSlaveCnt > 0 && fn->GetSlaveCnt() >= fMaxSlaveCnt) { - PDB(kPacketizer,1) Info("NextUnAllocNode", "reached workers per node limit (%ld)", - fMaxSlaveCnt); - fn = 0; - } - - return fn; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove unallocated node. - -void TPacketizer::RemoveUnAllocNode(TFileNode * node) -{ - fUnAllocated->Remove(node); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next active file. - -TPacketizer::TFileStat *TPacketizer::GetNextActive() -{ - TFileNode *node; - TFileStat *file = 0; - - while (file == 0 && ((node = NextActiveNode()) != 0)) { - file = node->GetNextActive(); - if (file == 0) RemoveActiveNode(node); - } - - return file; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next active node. - -TPacketizer::TFileNode *TPacketizer::NextActiveNode() -{ - fActive->Sort(); - PDB(kPacketizer,2) { - Printf("TPacketizer::NextActiveNode : ----------------------"); - fActive->Print(); - } - - TFileNode *fn = (TFileNode*) fActive->First(); - if (fn != 0 && fMaxSlaveCnt > 0 && fn->GetSlaveCnt() >= fMaxSlaveCnt) { - PDB(kPacketizer,1) - Info("NextActiveNode", "reached workers per node limit (%ld)", fMaxSlaveCnt); - fn = 0; - } - - return fn; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove file from the list of actives. - -void TPacketizer::RemoveActive(TFileStat *file) -{ - TFileNode *node = file->GetNode(); - - node->RemoveActive(file); - if (node->GetNumberOfActiveFiles() == 0) RemoveActiveNode(node); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove node from the list of actives. - -void TPacketizer::RemoveActiveNode(TFileNode *node) -{ - fActive->Remove(node); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset the internal datastructure for packet distribution. - -void TPacketizer::Reset() -{ - fUnAllocated->Clear(); - fUnAllocated->AddAll(fFileNodes); - - fActive->Clear(); - - TIter files(fFileNodes); - TFileNode *fn; - while ((fn = (TFileNode*) files.Next()) != 0) { - fn->Reset(); - } - - TIter slaves(fSlaveStats); - TObject *key; - while ((key = slaves.Next()) != 0) { - TSlaveStat *slstat = (TSlaveStat*) fSlaveStats->GetValue(key); - if (slstat) { - fn = (TFileNode*) fFileNodes->FindObject(slstat->GetName()); - if (fn != 0 ) { - slstat->SetFileNode(fn); - fn->IncMySlaveCnt(); - } - slstat->fCurFile = 0; - } else { - Warning("Reset", "TSlaveStat associated to key '%s' is NULL", key->GetName()); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check existence of file/dir/tree an get number of entries. -/// Assumes the files have been setup. - -void TPacketizer::ValidateFiles(TDSet *dset, TList *slaves, Long64_t maxent, Bool_t byfile) -{ - TMap slaves_by_sock; - TMonitor mon; - TList workers; - - - // Setup the communication infrastructure - - workers.AddAll(slaves); - TIter si(slaves); - TSlave *slm = 0; - while ((slm = (TSlave*)si.Next()) != 0) { - PDB(kPacketizer,3) - Info("ValidateFiles","socket added to monitor: %p (%s)", - slm->GetSocket(), slm->GetName()); - mon.Add(slm->GetSocket()); - slaves_by_sock.Add(slm->GetSocket(), slm); - PDB(kPacketizer,1) - Info("ValidateFiles", - "mon: %p, wrk: %p, sck: %p", &mon, slm, slm->GetSocket()); - } - - mon.DeActivateAll(); - - ((TProof*)gProof)->DeActivateAsyncInput(); - - // Some monitoring systems (TXSocketHandler) need to know this - ((TProof*)gProof)->fCurrentMonitor = &mon; - - // Preparing for client notification - TString msg("Validating files"); - UInt_t n = 0; - UInt_t tot = dset->GetListOfElements()->GetSize(); - Bool_t st = kTRUE; - - Long64_t totent = 0, nopenf = 0; - while (kTRUE) { - - // send work - while( TSlave *s = (TSlave*)workers.First() ) { - - workers.Remove(s); - - // find a file - - TSlaveStat *slstat = (TSlaveStat*)fSlaveStats->GetValue(s); - if (!slstat) { - Error("ValidateFiles", "TSlaveStat associated to slave '%s' is NULL", s->GetName()); - continue; - } - TFileNode *node = 0; - TFileStat *file = 0; - - // try its own node first - if ( (node = slstat->GetFileNode()) != 0 ) { - file = GetNextUnAlloc(node); - if ( file == 0 ) { - slstat->SetFileNode(0); - } - } - - // look for a file on any other node if necessary - if (file == 0) { - file = GetNextUnAlloc(); - } - - if ( file != 0 ) { - // files are done right away - RemoveActive(file); - - slstat->fCurFile = file; - TDSetElement *elem = file->GetElement(); - Long64_t entries = elem->GetEntries(kTRUE, kFALSE); - if (entries < 0 || strlen(elem->GetTitle()) <= 0) { - // This is decremented when we get the reply - file->GetNode()->IncSlaveCnt(slstat->GetName()); - TMessage m(kPROOF_GETENTRIES); - m << dset->IsTree() - << TString(elem->GetFileName()) - << TString(elem->GetDirectory()) - << TString(elem->GetObjName()); - - s->GetSocket()->Send( m ); - mon.Activate(s->GetSocket()); - PDB(kPacketizer,2) - Info("ValidateFiles", - "sent to worker-%s (%s) via %p GETENTRIES on %s %s %s %s", - s->GetOrdinal(), s->GetName(), s->GetSocket(), - dset->IsTree() ? "tree" : "objects", elem->GetFileName(), - elem->GetDirectory(), elem->GetObjName()); - } else { - // Fill the info - elem->SetTDSetOffset(entries); - if (entries > 0) { - // Most likely valid - elem->SetValid(); - if (!elem->GetEntryList()) { - if (elem->GetFirst() > entries) { - Error("ValidateFiles", - "first (%lld) higher then number of entries (%lld) in %s", - elem->GetFirst(), entries, elem->GetFileName()); - // disable element - slstat->fCurFile->SetDone(); - elem->Invalidate(); - dset->SetBit(TDSet::kSomeInvalid); - } - if (elem->GetNum() == -1) { - elem->SetNum(entries - elem->GetFirst()); - } else if (elem->GetFirst() + elem->GetNum() > entries) { - Warning("ValidateFiles", "num (%lld) + first (%lld) larger then number of" - " keys/entries (%lld) in %s", elem->GetNum(), elem->GetFirst(), - entries, elem->GetFileName()); - elem->SetNum(entries - elem->GetFirst()); - } - PDB(kPacketizer,2) - Info("ValidateFiles", - "found elem '%s' with %lld entries", elem->GetFileName(), entries); - } - } - // Notify the client - n++; - gProof->SendDataSetStatus(msg, n, tot, st); - - // This worker is ready for the next validation - workers.Add(s); - } - } - } - - // Check if there is anything to wait for - if (mon.GetActive() == 0) { - if (byfile && maxent > 0 && totent > 0) { - // How many files do we still need ? - Long64_t nrestf = (maxent - totent) * nopenf / totent ; - if (nrestf <= 0 && maxent > totent) nrestf = 1; - if (nrestf > 0) { - PDB(kPacketizer,3) - Info("ValidateFiles", "{%lld, %lld, %lld): needs to validate %lld more files", - maxent, totent, nopenf, nrestf); - si.Reset(); - while ((slm = (TSlave *) si.Next()) && nrestf--) { - workers.Add(slm); - } - continue; - } else { - PDB(kPacketizer,3) - Info("ValidateFiles", "no need to validate more files"); - break; - } - } else { - break; - } - } - - PDB(kPacketizer,3) { - Info("ValidateFiles", "waiting for %d workers:", mon.GetActive()); - TList *act = mon.GetListOfActives(); - TIter next(act); - TSocket *s = 0; - while ((s = (TSocket*) next())) { - Info("ValidateFiles", "found sck: %p", s); - TSlave *sl = (TSlave *) slaves_by_sock.GetValue(s); - if (sl) - Info("ValidateFiles", " worker-%s (%s)", sl->GetOrdinal(), sl->GetName()); - } - delete act; - } - - TSocket *sock = mon.Select(); - // If we have been interrupted break - if (!sock) { - Error("ValidateFiles", "selection has been interrupted - STOP"); - mon.DeActivateAll(); - fValid = kFALSE; - break; - } - mon.DeActivate(sock); - - PDB(kPacketizer,3) Info("ValidateFiles", "select returned: %p", sock); - - TSlave *slave = (TSlave *) slaves_by_sock.GetValue( sock ); - if (!sock->IsValid()) { - // A socket got invalid during validation - Error("ValidateFiles", "worker-%s (%s) got invalid - STOP", - slave->GetOrdinal(), slave->GetName()); - ((TProof*)gProof)->MarkBad(slave); - fValid = kFALSE; - break; - } - - TMessage *reply; - - if ( sock->Recv(reply) <= 0 ) { - // Help! lost a slave? - ((TProof*)gProof)->MarkBad(slave); - fValid = kFALSE; - Error("ValidateFiles", "Recv failed! for worker-%s (%s)", - slave->GetOrdinal(), slave->GetName()); - continue; - } - - if (reply->What() != kPROOF_GETENTRIES) { - // Not what we want: handover processing to the central machinery - Int_t what = reply->What(); - ((TProof*)gProof)->HandleInputMessage(slave, reply); - if (what == kPROOF_FATAL) { - Error("ValidateFiles", "kPROOF_FATAL from worker-%s (%s)", - slave->GetOrdinal(), slave->GetName()); - fValid = kFALSE; - } else { - // Reactivate the socket - mon.Activate(sock); - } - // Get next message - continue; - } - - TSlaveStat *slavestat = (TSlaveStat*) fSlaveStats->GetValue( slave ); - TDSetElement *e = slavestat->fCurFile->GetElement(); - slavestat->fCurFile->GetNode()->DecSlaveCnt(slavestat->GetName()); - Long64_t entries; - - (*reply) >> entries; - - // Extract object name, if there - if ((reply->BufferSize() > reply->Length())) { - TString objname; - (*reply) >> objname; - e->SetTitle(objname); - } - - e->SetTDSetOffset(entries); - if ( entries > 0 ) { - - // This dataset element is most likely valid - e->SetValid(); - - //if (!e->GetEventList()) { - if (!e->GetEntryList()){ - if ( e->GetFirst() > entries ) { - Error("ValidateFiles", "first (%lld) higher then number of entries (%lld) in %s", - e->GetFirst(), entries, e->GetFileName()); - - // Invalidate the element - slavestat->fCurFile->SetDone(); - e->Invalidate(); - dset->SetBit(TDSet::kSomeInvalid); - } - - if ( e->GetNum() == -1 ) { - e->SetNum( entries - e->GetFirst() ); - } else if ( e->GetFirst() + e->GetNum() > entries ) { - Error("ValidateFiles", - "num (%lld) + first (%lld) larger then number of keys/entries (%lld) in %s", - e->GetNum(), e->GetFirst(), entries, e->GetFileName()); - e->SetNum(entries - e->GetFirst()); - } - } - - // Count - totent += entries; - nopenf++; - - // Notify the client - n++; - gProof->SendDataSetStatus(msg, n, tot, st); - - } else { - - Error("ValidateFiles", "cannot get entries for %s (", e->GetFileName() ); - // - // Need to fix this with a user option to allow incomplete file sets (rdm) - // - //fValid = kFALSE; // all element must be readable! - if (gProofServ) { - TMessage m(kPROOF_MESSAGE); - m << TString(Form("Cannot get entries for file: %s - skipping", e->GetFileName())); - gProofServ->GetSocket()->Send(m); - } - - // Invalidate the element - e->Invalidate(); - dset->SetBit(TDSet::kSomeInvalid); - } - PDB(kPacketizer,3) Info("ValidateFiles", " %lld events validated", totent); - - // Ready for the next job, unless we have enough files - if (maxent < 0 || ((totent < maxent) && !byfile)) - workers.Add(slave); - } - - // report std. output from slaves?? - - ((TProof*)gProof)->ActivateAsyncInput(); - - // This needs to be reset - ((TProof*)gProof)->fCurrentMonitor = 0; - - // No reason to continue if invalid - if (!fValid) - return; - - - // compute the offset for each file element - Long64_t offset = 0; - Long64_t newOffset = 0; - TIter next(dset->GetListOfElements()); - TDSetElement *el; - while ( (el = dynamic_cast (next())) ) { - newOffset = offset + el->GetTDSetOffset(); - el->SetTDSetOffset(offset); - offset = newOffset; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get entries processed by the specified slave. - -Long64_t TPacketizer::GetEntriesProcessed(TSlave *slave) const -{ - if ( fSlaveStats == 0 ) return 0; - - TSlaveStat *slstat = (TSlaveStat*) fSlaveStats->GetValue( slave ); - - if ( slstat == 0 ) return 0; - - return slstat->GetEntriesProcessed(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get Estimation of the current rate; just summing the current rates of -/// the active workers - -Float_t TPacketizer::GetCurrentRate(Bool_t &all) -{ - all = kTRUE; - // Loop over the workers - Float_t currate = 0.; - if (fSlaveStats && fSlaveStats->GetSize() > 0) { - TIter nxw(fSlaveStats); - TObject *key; - while ((key = nxw()) != 0) { - TSlaveStat *slstat = (TSlaveStat *) fSlaveStats->GetValue(key); - if (slstat && slstat->GetProgressStatus() && slstat->GetEntriesProcessed() > 0) { - // Sum-up the current rates - currate += slstat->GetProgressStatus()->GetCurrentRate(); - } else { - all = kFALSE; - } - } - } - // Done - return currate; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next packet - -TDSetElement *TPacketizer::GetNextPacket(TSlave *sl, TMessage *r) -{ - if ( !fValid ) { - return 0; - } - - // Find worker - - TSlaveStat *slstat = (TSlaveStat*) fSlaveStats->GetValue( sl ); - - R__ASSERT( slstat != 0 ); - - PDB(kPacketizer,1) - Info("GetNextPacket","worker-%s (%s)", sl->GetOrdinal(), sl->GetName()); - // update stats & free old element - - Bool_t firstPacket = kFALSE; - if ( slstat->fCurElem != 0 ) { - Double_t latency = 0., proctime = 0., proccpu = 0.; - Long64_t bytesRead = -1; - Long64_t totalEntries = -1; - Long64_t totev = 0; - Long64_t numev = slstat->fCurElem->GetNum(); - - fPackets->Add(slstat->fCurElem); - - if (sl->GetProtocol() > 18) { - TProofProgressStatus *status = 0; - (*r) >> latency; - (*r) >> status; - - // Calculate the progress made in the last packet - TProofProgressStatus *progress = 0; - if (status) { - // upadte the worker status - numev = status->GetEntries() - slstat->GetEntriesProcessed(); - progress = slstat->AddProcessed(status); - if (progress) { - // (*fProgressStatus) += *progress; - proctime = progress->GetProcTime(); - proccpu = progress->GetCPUTime(); - totev = status->GetEntries(); // for backward compatibility - bytesRead = progress->GetBytesRead(); - delete progress; - } - delete status; - } else - Error("GetNextPacket", "no status came in the kPROOF_GETPACKET message"); - } else { - - (*r) >> latency >> proctime >> proccpu; - - // only read new info if available - if (r->BufferSize() > r->Length()) (*r) >> bytesRead; - if (r->BufferSize() > r->Length()) (*r) >> totalEntries; - if (r->BufferSize() > r->Length()) (*r) >> totev; - - numev = totev - slstat->GetEntriesProcessed(); - if (numev > 0) slstat->GetProgressStatus()->IncEntries(numev); - if (bytesRead > 0) slstat->GetProgressStatus()->IncBytesRead(bytesRead); - if (numev > 0 || bytesRead > 0) slstat->GetProgressStatus()->SetLastUpdate(); - } - - if (fProgressStatus) { - if (numev > 0) fProgressStatus->IncEntries(numev); - if (bytesRead > 0) fProgressStatus->IncBytesRead(bytesRead); - if (numev > 0 || bytesRead > 0) fProgressStatus->SetLastUpdate(); - } - PDB(kPacketizer,2) - Info("GetNextPacket","worker-%s (%s): %lld %7.3lf %7.3lf %7.3lf %lld", - sl->GetOrdinal(), sl->GetName(), - numev, latency, proctime, proccpu, bytesRead); - - if (gPerfStats) - gPerfStats->PacketEvent(sl->GetOrdinal(), sl->GetName(), slstat->fCurElem->GetFileName(), - numev, latency, proctime, proccpu, bytesRead); - - slstat->fCurElem = 0; - if (fProgressStatus && fProgressStatus->GetEntries() == fTotalEntries) { - HandleTimer(0); // Send last timer message - delete fProgress; fProgress = 0; - } - } else { - firstPacket = kTRUE; - } - - if ( fStop ) { - HandleTimer(0); - return 0; - } - - // get a file if needed - - TFileStat *file = slstat->fCurFile; - - if ( file != 0 && file->IsDone() ) { - file->GetNode()->DecSlaveCnt(slstat->GetName()); - if (gPerfStats) - gPerfStats->FileEvent(sl->GetOrdinal(), sl->GetName(), file->GetNode()->GetName(), - file->GetElement()->GetFileName(), kFALSE); - file = 0; - } - // Reset the current file field - slstat->fCurFile = file; - - if (!file) { - - // Try its own node first - if (slstat->GetFileNode() != 0) { - file = GetNextUnAlloc(slstat->GetFileNode()); - if (!file) { - slstat->SetFileNode(0); - } - } - - // try to find an unused filenode first - if (!file) { - file = GetNextUnAlloc(); - } - - // then look at the active filenodes - if (!file) { - file = GetNextActive(); - } - - if (!file) return 0; - - slstat->fCurFile = file; - file->GetNode()->IncSlaveCnt(slstat->GetName()); - if (gPerfStats) - gPerfStats->FileEvent(sl->GetOrdinal(), sl->GetName(), - file->GetNode()->GetName(), - file->GetElement()->GetFileName(), kTRUE); - } - - // get a packet - - TDSetElement *base = file->GetElement(); - Long64_t num = Long64_t(fPacketSize*(Float_t)slstat->fSlave->GetPerfIdx()/fMaxPerfIdx); - if (num < 1) num = 1; - - Long64_t first = file->GetNextEntry(); - Long64_t last = base->GetFirst() + base->GetNum(); - - if ( first + num >= last ) { - num = last - first; - file->SetDone(); // done - - // delete file from active list (unalloc list is single pass, no delete needed) - RemoveActive(file); - - } else { - file->MoveNextEntry(num); - } - - - slstat->fCurElem = CreateNewPacket(base, first, num); - if (base->GetEntryList()) - slstat->fCurElem->SetEntryList(base->GetEntryList(), first, num); - - // Flag the first packet of a new run (dataset) - if (firstPacket) - slstat->fCurElem->SetBit(TDSetElement::kNewRun); - else - slstat->fCurElem->ResetBit(TDSetElement::kNewRun); - - PDB(kPacketizer,2) - Info("GetNextPacket","%s: %s %lld %lld", sl->GetOrdinal(), base->GetFileName(), first, num); - - return slstat->fCurElem; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the number of workers still processing - -Int_t TPacketizer::GetActiveWorkers() -{ - Int_t actw = 0; - TIter nxw(fSlaveStats); - TObject *key; - while ((key = nxw())) { - TSlaveStat *wrkstat = (TSlaveStat *) fSlaveStats->GetValue(key); - if (wrkstat && wrkstat->fCurFile) actw++; - } - // Done - return actw; -} diff --git a/proof/proofplayer/src/TPacketizerAdaptive.cxx b/proof/proofplayer/src/TPacketizerAdaptive.cxx deleted file mode 100644 index 79a0699134438..0000000000000 --- a/proof/proofplayer/src/TPacketizerAdaptive.cxx +++ /dev/null @@ -1,2193 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Jan Iwaszkiewicz 11/12/06 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * -*************************************************************************/ - -/** \class TPacketizerAdaptive -\ingroup proofkernel - -This packetizer is based on TPacketizer but uses different -load-balancing algorithms and data structures. -Two main improvements in the load-balancing strategy: - - First one was to change the order in which the files are assigned - to the computing nodes in such a way that network transfers are - evenly distributed in the query time. Transfer of the remote files - was often becoming a bottleneck at the end of a query. - - The other improvement is the use of time-based packet size. We - measure the processing rate of all the nodes and calculate the - packet size, so that it takes certain amount of time. In this way - packetizer prevents the situation where the query can't finish - because of one slow node. - -The data structures: TFileStat, TFileNode and TSlaveStat are -enriched + changed and TFileNode::Compare method is changed. - -*/ - -#include "TPacketizerAdaptive.h" - -#include "Riostream.h" -#include "TDSet.h" -#include "TError.h" -#include "TEnv.h" -#include "TEntryList.h" -#include "TEventList.h" -#include "TMap.h" -#include "TMessage.h" -#include "TMonitor.h" -#include "TNtupleD.h" -#include "TObject.h" -#include "TParameter.h" -#include "TPerfStats.h" -#include "TProofDebug.h" -#include "TProof.h" -#include "TProofServ.h" -#include "TSlave.h" -#include "TSocket.h" -#include "TSortedList.h" -#include "TUrl.h" -#include "TClass.h" -#include "TRandom.h" -#include "TMath.h" -#include "TObjString.h" -#include "TList.h" - -// -// The following three utility classes manage the state of the -// work to be performed and the slaves involved in the process. -// A list of TFileNode(s) describes the hosts with files, each -// has a list of TFileStat(s) keeping the state for each TDSet -// element (file). -// -// The list of TSlaveStat(s) keep track of the work (being) done -// by each slave -// - - -//------------------------------------------------------------------------------ - -class TPacketizerAdaptive::TFileStat : public TObject { - -private: - Bool_t fIsDone; // is this element processed - TFileNode *fNode; // my FileNode - TDSetElement *fElement; // location of the file and its range - Long64_t fNextEntry; // cursor in the range, -1 when done // needs changing - -public: - TFileStat(TFileNode *node, TDSetElement *elem, TList *file); - - Bool_t IsDone() const {return fIsDone;} - Bool_t IsSortable() const override { return kTRUE; } - void SetDone() {fIsDone = kTRUE;} - TFileNode *GetNode() const {return fNode;} - TDSetElement *GetElement() const {return fElement;} - Long64_t GetNextEntry() const {return fNextEntry;} - void MoveNextEntry(Long64_t step) {fNextEntry += step;} - - // This method is used to keep a sorted list of remaining files to be processed - Int_t Compare(const TObject* obj) const override - { - // Return -1 if elem.entries < obj.elem.entries, 0 if elem.entries equal - // and 1 if elem.entries < obj.elem.entries. - const TFileStat *fst = dynamic_cast(obj); - if (fst && GetElement() && fst->GetElement()) { - Long64_t ent = GetElement()->GetNum(); - Long64_t entfst = fst->GetElement()->GetNum(); - if (ent > 0 && entfst > 0) { - if (ent > entfst) { - return 1; - } else if (ent < entfst) { - return -1; - } else { - return 0; - } - } - } - // No info: assume equal (no change in order) - return 0; - } - void Print(Option_t * = 0) const override - { // Notify file name and entries - Printf("TFileStat: %s %lld", fElement ? fElement->GetName() : "---", - fElement ? fElement->GetNum() : -1); - } -}; - -TPacketizerAdaptive::TFileStat::TFileStat(TFileNode *node, TDSetElement *elem, TList *files) - : fIsDone(kFALSE), fNode(node), fElement(elem), fNextEntry(elem->GetFirst()) -{ - // Constructor: add to the global list - if (files) files->Add(this); -} - -//------------------------------------------------------------------------------ - -// a class describing a file node as a part of a session -class TPacketizerAdaptive::TFileNode : public TObject { - -private: - TString fNodeName; // FQDN of the node - TList *fFiles; // TDSetElements (files) stored on this node - TObject *fUnAllocFileNext; // cursor in fFiles - TList *fActFiles; // files with work remaining - TObject *fActFileNext; // cursor in fActFiles - Int_t fMySlaveCnt; // number of slaves running on this node - // (which can process remote files) - Int_t fExtSlaveCnt; // number of external slaves processing - // files on this node - Int_t fRunSlaveCnt; // total number of slaves processing files - // on this node - Long64_t fProcessed; // number of events processed on this node - Long64_t fEvents; // number of entries in files on this node - - Int_t fStrategy; // 0 means the classic and 1 (default) - the adaptive strategy - - TSortedList *fFilesToProcess; // Global list of files (TFileStat) to be processed (owned by TPacketizer) - -public: - TFileNode(const char *name, Int_t strategy, TSortedList *files); - ~TFileNode() override { delete fFiles; delete fActFiles; } - - void IncMySlaveCnt() { fMySlaveCnt++; } - Int_t GetMySlaveCnt() const { return fMySlaveCnt; } - void IncExtSlaveCnt(const char *slave) { if (fNodeName != slave) fExtSlaveCnt++; } - void DecExtSlaveCnt(const char *slave) { if (fNodeName != slave) fExtSlaveCnt--; R__ASSERT(fExtSlaveCnt >= 0); } - Int_t GetSlaveCnt() const { return fMySlaveCnt + fExtSlaveCnt; } - void IncRunSlaveCnt() { fRunSlaveCnt++; } - void DecRunSlaveCnt() { fRunSlaveCnt--; R__ASSERT(fRunSlaveCnt >= 0); } - Int_t GetRunSlaveCnt() const { return fRunSlaveCnt; } - Int_t GetExtSlaveCnt() const { return fExtSlaveCnt; } - Int_t GetNumberOfActiveFiles() const { return fActFiles->GetSize(); } - Bool_t IsSortable() const override { return kTRUE; } - Int_t GetNumberOfFiles() { return fFiles->GetSize(); } - void IncProcessed(Long64_t nEvents) - { fProcessed += nEvents; } - Long64_t GetProcessed() const { return fProcessed; } - void DecreaseProcessed(Long64_t nEvents) { fProcessed -= nEvents; } - // this method is used by Compare() it adds 1, so it returns a number that - // would be true if one more slave is added. - Long64_t GetEventsLeftPerSlave() const - { return ((fEvents - fProcessed)/(fRunSlaveCnt + 1)); } - void IncEvents(Long64_t nEvents) { fEvents += nEvents; } - const char *GetName() const override { return fNodeName.Data(); } - Long64_t GetNEvents() const { return fEvents; } - - void Print(Option_t * = 0) const override - { - TFileStat *fs = 0; - TDSetElement *e = 0; - Int_t nn = 0; - Printf("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"); - Printf("+++ TFileNode: %s +++", fNodeName.Data()); - Printf("+++ Evts: %lld (total: %lld) ", fProcessed, fEvents); - Printf("+++ Worker count: int:%d, ext: %d, tot:%d ", fMySlaveCnt, fExtSlaveCnt, fRunSlaveCnt); - Printf("+++ Files: %d ", fFiles ? fFiles->GetSize() : 0); - if (fFiles && fFiles->GetSize() > 0) { - TIter nxf(fFiles); - while ((fs = (TFileStat *) nxf())) { - if ((e = fs->GetElement())) { - Printf("+++ #%d: %s %lld - %lld (%lld) - next: %lld ", ++nn, e->GetName(), - e->GetFirst(), e->GetFirst() + e->GetNum() - 1, e->GetNum(), fs->GetNextEntry()); - } else { - Printf("+++ #%d: no element! ", ++nn); - } - } - } - Printf("+++ Active files: %d ", fActFiles ? fActFiles->GetSize() : 0); - if (fActFiles && fActFiles->GetSize() > 0) { - TIter nxaf(fActFiles); - while ((fs = (TFileStat *) nxaf())) { - if ((e = fs->GetElement())) { - Printf("+++ #%d: %s %lld - %lld (%lld) - next: %lld", ++nn, e->GetName(), - e->GetFirst(), e->GetFirst() + e->GetNum() - 1, e->GetNum(), fs->GetNextEntry()); - } else { - Printf("+++ #%d: no element! ", ++nn); - } - } - } - Printf("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"); - } - - void Add(TDSetElement *elem, Bool_t tolist) - { - TList *files = tolist ? (TList *)fFilesToProcess : (TList *)0; - TFileStat *f = new TFileStat(this, elem, files); - fFiles->Add(f); - if (fUnAllocFileNext == 0) fUnAllocFileNext = fFiles->First(); - } - - TFileStat *GetNextUnAlloc() - { - TObject *next = fUnAllocFileNext; - - if (next != 0) { - // make file active - fActFiles->Add(next); - if (fActFileNext == 0) fActFileNext = fActFiles->First(); - - // move cursor - fUnAllocFileNext = fFiles->After(fUnAllocFileNext); - } - return (TFileStat *) next; - } - - TFileStat *GetNextActive() - { - TObject *next = fActFileNext; - - if (fActFileNext != 0) { - fActFileNext = fActFiles->After(fActFileNext); - if (fActFileNext == 0) fActFileNext = fActFiles->First(); - } - - return (TFileStat *) next; - } - - void RemoveActive(TFileStat *file) - { - if (fActFileNext == file) fActFileNext = fActFiles->After(file); - fActFiles->Remove(file); - if (fFilesToProcess) fFilesToProcess->Remove(file); - if (fActFileNext == 0) fActFileNext = fActFiles->First(); - } - - Int_t Compare(const TObject *other) const override - { - // Must return -1 if this is smaller than obj, 0 if objects are equal - // and 1 if this is larger than obj. - // smaller means more needing a new worker. - // Two cases are considered depending on - // relation between harddrive speed and network bandwidth. - - const TFileNode *obj = dynamic_cast(other); - if (!obj) { - Error("Compare", "input is not a TPacketizer::TFileNode object"); - return 0; - } - - // how many more events it has than obj - - if (fStrategy == 1) { - // The default adaptive strategy. - Int_t myVal = GetRunSlaveCnt(); - Int_t otherVal = obj->GetRunSlaveCnt(); - if (myVal < otherVal) { - return -1; - } else if (myVal > otherVal) { - return 1; - } else { - // if this has more events to process than obj - if ((fEvents - fProcessed) > - (obj->GetNEvents() - obj->GetProcessed())) { - return -1; - } else { - return 1; - } - } - } else { - Int_t myVal = GetSlaveCnt(); - Int_t otherVal = obj->GetSlaveCnt(); - if (myVal < otherVal) { - return -1; - } else if (myVal > otherVal) { - return 1; - } else { - return 0; - } - } - } - - void Reset() - { - fUnAllocFileNext = fFiles->First(); - fActFiles->Clear(); - fActFileNext = 0; - fExtSlaveCnt = 0; - fMySlaveCnt = 0; - fRunSlaveCnt = 0; - } -}; - - -TPacketizerAdaptive::TFileNode::TFileNode(const char *name, Int_t strategy, TSortedList *files) - : fNodeName(name), fFiles(new TList), fUnAllocFileNext(0), - fActFiles(new TList), fActFileNext(0), fMySlaveCnt(0), - fExtSlaveCnt(0), fRunSlaveCnt(0), fProcessed(0), fEvents(0), - fStrategy(strategy), fFilesToProcess(files) -{ - // Constructor - - fFiles->SetOwner(); - fActFiles->SetOwner(kFALSE); -} - -//------------------------------------------------------------------------------ - -class TPacketizerAdaptive::TSlaveStat : public TVirtualPacketizer::TVirtualSlaveStat { - -friend class TPacketizerAdaptive; - -private: - TFileNode *fFileNode; // corresponding node or 0 - TFileStat *fCurFile; // file currently being processed - TDSetElement *fCurElem; // TDSetElement currently being processed - Long64_t fCurProcessed; // events processed in the current file - Float_t fCurProcTime; // proc time spent on the current file - TList *fDSubSet; // packets processed by this worker - -public: - TSlaveStat(TSlave *slave); - ~TSlaveStat() override; - TFileNode *GetFileNode() const { return fFileNode; } - Long64_t GetEntriesProcessed() const { return fStatus?fStatus->GetEntries():-1; } - Double_t GetProcTime() const { return fStatus?fStatus->GetProcTime():-1; } - TFileStat *GetCurFile() { return fCurFile; } - void SetFileNode(TFileNode *node) { fFileNode = node; } - void UpdateRates(TProofProgressStatus *st); - Float_t GetAvgRate() { return fStatus->GetRate(); } - Float_t GetCurRate() { - return (fCurProcTime?fCurProcessed/fCurProcTime:0); } - Int_t GetLocalEventsLeft() { - return fFileNode?(fFileNode->GetEventsLeftPerSlave()):0; } - TList *GetProcessedSubSet() { return fDSubSet; } - TProofProgressStatus *GetProgressStatus() { return fStatus; } - TProofProgressStatus *AddProcessed(TProofProgressStatus *st = 0) override; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TPacketizerAdaptive::TSlaveStat::TSlaveStat(TSlave *slave) - : fFileNode(0), fCurFile(0), fCurElem(0), - fCurProcessed(0), fCurProcTime(0) -{ - fDSubSet = new TList(); - fDSubSet->SetOwner(); - fSlave = slave; - fStatus = new TProofProgressStatus(); - // The slave name is a special one in PROOF-Lite: avoid blocking on the DNS - // for non existing names - fWrkFQDN = slave->GetName(); - if (strcmp(slave->ClassName(), "TSlaveLite")) { - fWrkFQDN = TUrl(fWrkFQDN).GetHostFQDN(); - // Get full name for local hosts - if (fWrkFQDN.Contains("localhost") || fWrkFQDN == "127.0.0.1") - fWrkFQDN = TUrl(gSystem->HostName()).GetHostFQDN(); - } - PDB(kPacketizer, 2) - Info("TSlaveStat", "wrk FQDN: %s", fWrkFQDN.Data()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Cleanup - -TPacketizerAdaptive::TSlaveStat::~TSlaveStat() -{ - SafeDelete(fDSubSet); - SafeDelete(fStatus); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update packetizer rates - -void TPacketizerAdaptive::TSlaveStat::UpdateRates(TProofProgressStatus *st) -{ - if (!st) { - Error("UpdateRates", "no status object!"); - return; - } - if (fCurFile->IsDone()) { - fCurProcTime = 0; - fCurProcessed = 0; - } else { - fCurProcTime += st->GetProcTime() - GetProcTime(); - fCurProcessed += st->GetEntries() - GetEntriesProcessed(); - } - fCurFile->GetNode()->IncProcessed(st->GetEntries() - GetEntriesProcessed()); - st->SetLastEntries(st->GetEntries() - fStatus->GetEntries()); - SafeDelete(fStatus); - fStatus = st; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add the current element to the fDSubSet (subset processed by this worker) -/// and if the status arg is given, then change the size of the packet. -/// return the difference (*st - *fStatus) - -TProofProgressStatus *TPacketizerAdaptive::TSlaveStat::AddProcessed(TProofProgressStatus *st) -{ - if (st && fDSubSet && fCurElem) { - if (fCurElem->GetNum() != st->GetEntries() - GetEntriesProcessed()) - fCurElem->SetNum(st->GetEntries() - GetEntriesProcessed()); - fDSubSet->Add(fCurElem); - TProofProgressStatus *diff = new TProofProgressStatus(*st - *fStatus); - return diff; - } else { - Error("AddProcessed", "processed subset of current elem undefined"); - return 0; - } -} - -//------------------------------------------------------------------------------ - -ClassImp(TPacketizerAdaptive); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TPacketizerAdaptive::TPacketizerAdaptive(TDSet *dset, TList *slaves, - Long64_t first, Long64_t num, - TList *input, TProofProgressStatus *st) - : TVirtualPacketizer(input, st) -{ - PDB(kPacketizer,1) Info("TPacketizerAdaptive", - "enter (first %lld, num %lld)", first, num); - - // Init pointer members - fSlaveStats = 0; - fUnAllocated = 0; - fActive = 0; - fFileNodes = 0; - fMaxPerfIdx = 1; - fCachePacketSync = kTRUE; - fMaxEntriesRatio = 2.; - - fMaxSlaveCnt = -1; - fPacketAsAFraction = 4; - fStrategy = 1; - fFilesToProcess = new TSortedList; - fFilesToProcess->SetOwner(kFALSE); - - if (!fProgressStatus) { - Error("TPacketizerAdaptive", "No progress status"); - return; - } - - // Attempt to synchronize the packet size with the tree cache size - Int_t cpsync = -1; - if (TProof::GetParameter(input, "PROOF_PacketizerCachePacketSync", cpsync) != 0) { - // Check if there is a global cache-packet sync setting - cpsync = gEnv->GetValue("Packetizer.CachePacketSync", 1); - } - if (cpsync >= 0) fCachePacketSync = (cpsync > 0) ? kTRUE : kFALSE; - - // Max file entries to avg allowed ratio for cache-to-packet synchronization - // (applies only if fCachePacketSync is true; -1. disables the bound) - if (TProof::GetParameter(input, "PROOF_PacketizerMaxEntriesRatio", fMaxEntriesRatio) != 0) { - // Check if there is a global ratio setting - fMaxEntriesRatio = gEnv->GetValue("Packetizer.MaxEntriesRatio", 2.); - } - - // The possibility to change packetizer strategy to the basic TPacketizer's - // one (in which workers always process their local data first). - Int_t strategy = -1; - if (TProof::GetParameter(input, "PROOF_PacketizerStrategy", strategy) != 0) { - // Check if there is a global strategy setting - strategy = gEnv->GetValue("Packetizer.Strategy", 1); - } - if (strategy == 0) { - fStrategy = 0; - Info("TPacketizerAdaptive", "using the basic strategy of TPacketizer"); - } else if (strategy != 1) { - Warning("TPacketizerAdaptive", "unsupported strategy index (%d): ignore", strategy); - } - - Long_t maxSlaveCnt = 0; - if (TProof::GetParameter(input, "PROOF_MaxSlavesPerNode", maxSlaveCnt) == 0) { - if (maxSlaveCnt < 0) { - Info("TPacketizerAdaptive", - "The value of PROOF_MaxSlavesPerNode must be positive"); - maxSlaveCnt = 0; - } - } else { - // Try also with Int_t (recently supported in TProof::SetParameter) - Int_t mxslcnt = -1; - if (TProof::GetParameter(input, "PROOF_MaxSlavesPerNode", mxslcnt) == 0) { - if (mxslcnt < 0) { - Info("TPacketizerAdaptive", - "The value of PROOF_MaxSlavesPerNode must be positive"); - mxslcnt = 0; - } - maxSlaveCnt = (Long_t) mxslcnt; - } - } - - if (!maxSlaveCnt) - maxSlaveCnt = gEnv->GetValue("Packetizer.MaxWorkersPerNode", 0); - if (maxSlaveCnt > 0) { - fMaxSlaveCnt = maxSlaveCnt; - Info("TPacketizerAdaptive", "Setting max number of workers per node to %ld", - fMaxSlaveCnt); - } - - // if forceLocal parameter is set to 1 then eliminate the cross-worker - // processing; - // This minimizes the network usage on the PROOF cluser at the expense of - // longer jobs processing times. - // To process successfully the session must have workers with all the data! - fForceLocal = kFALSE; - Int_t forceLocal = 0; - if (TProof::GetParameter(input, "PROOF_ForceLocal", forceLocal) == 0) { - if (forceLocal == 1) - fForceLocal = kTRUE; - else - Info("TPacketizerAdaptive", - "The only accepted value of PROOF_ForceLocal parameter is 1 !"); - } - - // Below we provide a possibility to change the way packet size is - // calculated or define the packet time directly. - // fPacketAsAFraction can be interpreted as follows: - // packet time is (expected job proc. time) / fPacketSizeAsAFraction. - // It substitutes 20 in the old formula to calculate the fPacketSize: - // fPacketSize = fTotalEntries / (20 * nslaves) - Int_t packetAsAFraction = 0; - if (TProof::GetParameter(input, "PROOF_PacketAsAFraction", packetAsAFraction) == 0) { - if (packetAsAFraction > 0) { - fPacketAsAFraction = packetAsAFraction; - Info("TPacketizerAdaptive", - "using alternate fraction of query time as a packet size: %d", - packetAsAFraction); - } else - Info("TPacketizerAdaptive", "packetAsAFraction parameter must be higher than 0"); - } - - // Packet re-assignement - fTryReassign = 0; - Int_t tryReassign = 0; - if (TProof::GetParameter(input, "PROOF_TryReassign", tryReassign) != 0) - tryReassign = gEnv->GetValue("Packetizer.TryReassign", 0); - fTryReassign = tryReassign; - if (fTryReassign != 0) - Info("TPacketizerAdaptive", "failed packets will be re-assigned"); - - // Save the config parameters in the dedicated list so that they will be saved - // in the outputlist and therefore in the relevant TQueryResult - fConfigParams->Add(new TParameter("PROOF_PacketizerCachePacketSync", (Int_t)fCachePacketSync)); - fConfigParams->Add(new TParameter("PROOF_PacketizerMaxEntriesRatio", fMaxEntriesRatio)); - fConfigParams->Add(new TParameter("PROOF_PacketizerStrategy", fStrategy)); - fConfigParams->Add(new TParameter("PROOF_MaxWorkersPerNode", (Int_t)fMaxSlaveCnt)); - fConfigParams->Add(new TParameter("PROOF_ForceLocal", (Int_t)fForceLocal)); - fConfigParams->Add(new TParameter("PROOF_PacketAsAFraction", fPacketAsAFraction)); - - Double_t baseLocalPreference = 1.2; - fBaseLocalPreference = (Float_t)baseLocalPreference; - if (TProof::GetParameter(input, "PROOF_BaseLocalPreference", baseLocalPreference) == 0) - fBaseLocalPreference = (Float_t)baseLocalPreference; - - fFileNodes = new TList; - fFileNodes->SetOwner(); - fUnAllocated = new TList; - fUnAllocated->SetOwner(kFALSE); - fActive = new TList; - fActive->SetOwner(kFALSE); - - fValid = kTRUE; - - // Resolve end-point urls to optmize distribution - // dset->Lookup(); // moved to TProofPlayerRemote::Process - - // Read list of mounted disks - TObjArray *partitions = 0; - TString partitionsStr; - if (TProof::GetParameter(input, "PROOF_PacketizerPartitions", partitionsStr) != 0) - partitionsStr = gEnv->GetValue("Packetizer.Partitions", ""); - if (!partitionsStr.IsNull()) { - Info("TPacketizerAdaptive", "Partitions: %s", partitionsStr.Data()); - partitions = partitionsStr.Tokenize(","); - } - - // Split into per host and disk entries - dset->Reset(); - TDSetElement *e; - while ((e = (TDSetElement*)dset->Next())) { - - if (e->GetValid()) continue; - - // The dataset name, if any - if (fDataSet.IsNull() && e->GetDataSet() && strlen(e->GetDataSet())) - fDataSet = e->GetDataSet(); - - TUrl url = e->GetFileName(); - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", "element name: %s (url: %s)", e->GetFileName(), url.GetUrl()); - - // Map non URL filenames to dummy host - TString host; - if ( !url.IsValid() || - (strncmp(url.GetProtocol(),"root", 4) && - strncmp(url.GetProtocol(),"file", 4)) ) { - host = "no-host"; - } else if ( url.IsValid() && !strncmp(url.GetProtocol(),"file", 4)) { - host = "localhost"; - url.SetProtocol("root"); - } else { - host = url.GetHostFQDN(); - } - // Get full name for local hosts - if (host.Contains("localhost") || host == "127.0.0.1") { - url.SetHost(gSystem->HostName()); - host = url.GetHostFQDN(); - } - - // Find on which disk is the file, if any - TString disk; - if (partitions) { - TIter iString(partitions); - TObjString* os = 0; - while ((os = (TObjString *)iString())) { - // Compare begining of the url with disk mountpoint - if (strncmp(url.GetFile(), os->GetName(), os->GetString().Length()) == 0) { - disk = os->GetName(); - break; - } - } - } - // Node's url - TString nodeStr; - if (disk.IsNull()) - nodeStr.Form("%s://%s", url.GetProtocol(), host.Data()); - else - nodeStr.Form("%s://%s/%s", url.GetProtocol(), host.Data(), disk.Data()); - TFileNode *node = (TFileNode *) fFileNodes->FindObject(nodeStr); - - if (node == 0) { - node = new TFileNode(nodeStr, fStrategy, fFilesToProcess); - fFileNodes->Add(node); - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", "creating new node '%s' or the element", nodeStr.Data()); - } else { - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", "adding element to existing node '%s'", nodeStr.Data()); - } - - node->Add(e, kFALSE); - } - - fSlaveStats = new TMap; - fSlaveStats->SetOwner(kFALSE); - - TSlave *slave; - TIter si(slaves); - while ((slave = (TSlave*) si.Next())) { - fSlaveStats->Add( slave, new TSlaveStat(slave) ); - fMaxPerfIdx = slave->GetPerfIdx() > fMaxPerfIdx ? - slave->GetPerfIdx() : fMaxPerfIdx; - } - - // Setup file & filenode structure - Reset(); - // Optimize the number of files to be open when running on subsample - Int_t validateMode = 0; - Int_t gprc = TProof::GetParameter(input, "PROOF_ValidateByFile", validateMode); - Bool_t byfile = (gprc == 0 && validateMode > 0 && num > -1) ? kTRUE : kFALSE; - if (num > -1) - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", - "processing subset of entries: validating by file? %s", byfile ? "yes": "no"); - ValidateFiles(dset, slaves, num, byfile); - - - if (!fValid) return; - - // apply global range (first,num) to dset and rebuild structure - // ommitting TDSet elements that are not needed - - Int_t files = 0; - fTotalEntries = 0; - fUnAllocated->Clear(); // avoid dangling pointers - fActive->Clear(); - fFileNodes->Clear(); // then delete all objects - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", - "processing range: first %lld, num %lld", first, num); - - dset->Reset(); - Long64_t cur = 0; - while (( e = (TDSetElement*)dset->Next())) { - - // Skip invalid or missing file; It will be moved - // from the dset to the 'MissingFiles' list in the player. - if (!e->GetValid()) continue; - - TUrl url = e->GetFileName(); - Long64_t eFirst = e->GetFirst(); - Long64_t eNum = e->GetNum(); - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", "processing element '%s'", e->GetFileName()); - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", - " --> first %lld, elenum %lld (cur %lld) (entrylist: %p)", eFirst, eNum, cur, e->GetEntryList()); - - if (!e->GetEntryList()) { - // This element is before the start of the global range, skip it - if (cur + eNum < first) { - cur += eNum; - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", " --> skip element cur %lld", cur); - continue; - } - - // This element is after the end of the global range, skip it - if (num != -1 && (first+num <= cur)) { - cur += eNum; - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", " --> drop element cur %lld", cur); - continue; // break ?? - } - - Bool_t inRange = kFALSE; - if (cur <= first || (num != -1 && (first+num <= cur+eNum))) { - - if (cur <= first) { - // If this element contains the start of the global range - // adjust its start and number of entries - e->SetFirst( eFirst + (first - cur) ); - e->SetNum( e->GetNum() - (first - cur) ); - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", " --> adjust start %lld and end %lld", - eFirst + (first - cur), first + num - cur); - inRange = kTRUE; - } - if (num != -1 && (first+num <= cur+eNum)) { - // If this element contains the end of the global range - // adjust its number of entries - e->SetNum( first + num - e->GetFirst() - cur ); - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", " --> adjust end %lld", first + num - cur); - inRange = kTRUE; - } - - } else { - // Increment the counter ... - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", " --> increment 'cur' by %lld", eNum); - cur += eNum; - } - // Re-adjust eNum and cur, if needed - if (inRange) { - cur += eNum; - eNum = e->GetNum(); - } - - } else { - TEntryList *enl = dynamic_cast(e->GetEntryList()); - if (enl) { - eNum = enl->GetN(); - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", " --> entry-list element: %lld entries", eNum); - } else { - TEventList *evl = dynamic_cast(e->GetEntryList()); - eNum = evl ? evl->GetN() : eNum; - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", " --> event-list element: %lld entries (evl:%p)", eNum, evl); - } - if (!eNum) { - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", " --> empty entry- or event-list element!"); - continue; - } - } - PDB(kPacketizer,2) - Info("TPacketizerAdaptive", " --> next cur %lld", cur); - - // Map non URL filenames to dummy host - TString host; - if ( !url.IsValid() || - (strncmp(url.GetProtocol(),"root", 4) && - strncmp(url.GetProtocol(),"file", 4)) ) { - host = "no-host"; - } else if ( url.IsValid() && !strncmp(url.GetProtocol(),"file", 4)) { - host = "localhost"; - url.SetProtocol("root"); - } else { - host = url.GetHostFQDN(); - } - // Get full name for local hosts - if (host.Contains("localhost") || host == "127.0.0.1") { - url.SetHost(gSystem->HostName()); - host = url.GetHostFQDN(); - } - - // Find, on which disk is the file - TString disk; - if (partitions) { - TIter iString(partitions); - TObjString* os = 0; - while ((os = (TObjString *)iString())) { - // Compare begining of the url with disk mountpoint - if (strncmp(url.GetFile(), os->GetName(), os->GetString().Length()) == 0) { - disk = os->GetName(); - break; - } - } - } - // Node's url - TString nodeStr; - if (disk.IsNull()) - nodeStr.Form("%s://%s", url.GetProtocol(), host.Data()); - else - nodeStr.Form("%s://%s/%s", url.GetProtocol(), host.Data(), disk.Data()); - TFileNode *node = (TFileNode*) fFileNodes->FindObject(nodeStr); - - - if (node == 0) { - node = new TFileNode(nodeStr, fStrategy, fFilesToProcess); - fFileNodes->Add( node ); - PDB(kPacketizer, 2) - Info("TPacketizerAdaptive", " --> creating new node '%s' for element", nodeStr.Data()); - } else { - PDB(kPacketizer, 2) - Info("TPacketizerAdaptive", " --> adding element to exiting node '%s'", nodeStr.Data()); - } - - ++files; - fTotalEntries += eNum; - node->Add(e, kTRUE); - node->IncEvents(eNum); - PDB(kPacketizer,2) e->Print("a"); - } - PDB(kPacketizer,1) - Info("TPacketizerAdaptive", "processing %lld entries in %d files on %d hosts", - fTotalEntries, files, fFileNodes->GetSize()); - - // Set the total number for monitoring - if (gPerfStats) - gPerfStats->SetNumEvents(fTotalEntries); - - Reset(); - - InitStats(); - - if (!fValid) - SafeDelete(fProgress); - - PDB(kPacketizer,1) Info("TPacketizerAdaptive", "return"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TPacketizerAdaptive::~TPacketizerAdaptive() -{ - if (fSlaveStats) { - fSlaveStats->DeleteValues(); - } - - SafeDelete(fSlaveStats); - SafeDelete(fUnAllocated); - SafeDelete(fActive); - SafeDelete(fFileNodes); - SafeDelete(fFilesToProcess); -} - -//////////////////////////////////////////////////////////////////////////////// -/// (re)initialise the statistics -/// called at the begining or after a worker dies. - -void TPacketizerAdaptive::InitStats() -{ - // calculating how many files from TDSet are not cached on - // any slave - Int_t noRemoteFiles = 0; - fNEventsOnRemLoc = 0; - Int_t totalNumberOfFiles = 0; - TIter next(fFileNodes); - while (TFileNode *fn = (TFileNode*)next()) { - totalNumberOfFiles += fn->GetNumberOfFiles(); - if (fn->GetMySlaveCnt() == 0) { - noRemoteFiles += fn->GetNumberOfFiles(); - fNEventsOnRemLoc += (fn->GetNEvents() - fn->GetProcessed()); - } - } - - if (totalNumberOfFiles == 0) { - Info("InitStats", "no valid or non-empty file found: setting invalid"); - // No valid files: set invalid and return - fValid = kFALSE; - return; - } - - fFractionOfRemoteFiles = (1.0 * noRemoteFiles) / totalNumberOfFiles; - Info("InitStats", - "fraction of remote files %f", fFractionOfRemoteFiles); - - if (!fValid) - SafeDelete(fProgress); - - PDB(kPacketizer,1) Info("InitStats", "return"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next unallocated file from 'node' or other nodes: -/// First try 'node'. If there is no more files, keep trying to -/// find an unallocated file on other nodes. - -TPacketizerAdaptive::TFileStat *TPacketizerAdaptive::GetNextUnAlloc(TFileNode *node, const char *nodeHostName) -{ - TFileStat *file = 0; - - if (node != 0) { - PDB(kPacketizer, 2) - Info("GetNextUnAlloc", "looking for file on node %s", node->GetName()); - file = node->GetNextUnAlloc(); - if (file == 0) RemoveUnAllocNode(node); - } else { - if (nodeHostName && strlen(nodeHostName) > 0) { - - TFileNode *fn; - // Make sure that they are in the corrected order - fUnAllocated->Sort(); - PDB(kPacketizer,2) fUnAllocated->Print(); - - // Loop over unallocated fileNode list - for (int i = 0; i < fUnAllocated->GetSize(); i++) { - - if ((fn = (TFileNode *) fUnAllocated->At(i))) { - TUrl uu(fn->GetName()); - PDB(kPacketizer, 2) - Info("GetNextUnAlloc", "comparing %s with %s...", nodeHostName, uu.GetHost()); - - // Check, whether node's hostname is matching with current fileNode (fn) - if (!strcmp(nodeHostName, uu.GetHost())) { - node = fn; - - // Fetch next unallocated file from this node - if ((file = node->GetNextUnAlloc()) == 0) { - RemoveUnAllocNode(node); - node = 0; - } else { - PDB(kPacketizer, 2) - Info("GetNextUnAlloc", "found! (host: %s)", uu.GetHost()); - break; - } - } - } else { - Warning("GetNextUnAlloc", "unallocate entry %d is empty!", i); - } - } - - if (node != 0 && fMaxSlaveCnt > 0 && node->GetExtSlaveCnt() >= fMaxSlaveCnt) { - // Unlike in TPacketizer we look at the number of ext slaves only. - PDB(kPacketizer,1) - Info("GetNextUnAlloc", "reached Workers-per-Node Limit (%ld)", fMaxSlaveCnt); - node = 0; - } - } - - if (node == 0) { - while (file == 0 && ((node = NextNode()) != 0)) { - PDB(kPacketizer, 2) - Info("GetNextUnAlloc", "looking for file on node %s", node->GetName()); - if ((file = node->GetNextUnAlloc()) == 0) RemoveUnAllocNode(node); - } - } - } - - if (file != 0) { - // if needed make node active - if (fActive->FindObject(node) == 0) { - fActive->Add(node); - } - } - - PDB(kPacketizer, 2) { - if (!file) { - Info("GetNextUnAlloc", "no file found!"); - } else { - file->Print(); - } - } - - return file; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next node which has unallocated files. -/// the order is determined by TFileNode::Compare - -TPacketizerAdaptive::TFileNode *TPacketizerAdaptive::NextNode() -{ - fUnAllocated->Sort(); - PDB(kPacketizer,2) { - fUnAllocated->Print(); - } - - TFileNode *fn = (TFileNode*) fUnAllocated->First(); - if (fn != 0 && fMaxSlaveCnt > 0 && fn->GetExtSlaveCnt() >= fMaxSlaveCnt) { - // unlike in TPacketizer we look at the number of ext slaves only. - PDB(kPacketizer,1) - Info("NextNode", "reached Workers-per-Node Limit (%ld)", fMaxSlaveCnt); - fn = 0; - } - - return fn; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove unallocated node. - -void TPacketizerAdaptive::RemoveUnAllocNode(TFileNode * node) -{ - fUnAllocated->Remove(node); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next active file. - -TPacketizerAdaptive::TFileStat *TPacketizerAdaptive::GetNextActive() -{ - TFileNode *node; - TFileStat *file = 0; - - while (file == 0 && ((node = NextActiveNode()) != 0)) { - file = node->GetNextActive(); - if (file == 0) RemoveActiveNode(node); - } - - return file; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Get next active node. - -TPacketizerAdaptive::TFileNode *TPacketizerAdaptive::NextActiveNode() -{ - fActive->Sort(); - PDB(kPacketizer,2) { - Info("NextActiveNode", "enter"); - fActive->Print(); - } - - TFileNode *fn = (TFileNode*) fActive->First(); - // look at only ext slaves - if (fn != 0 && fMaxSlaveCnt > 0 && fn->GetExtSlaveCnt() >= fMaxSlaveCnt) { - PDB(kPacketizer,1) - Info("NextActiveNode","reached Workers-per-Node limit (%ld)", fMaxSlaveCnt); - fn = 0; - } - - return fn; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove file from the list of actives. - -void TPacketizerAdaptive::RemoveActive(TFileStat *file) -{ - TFileNode *node = file->GetNode(); - - node->RemoveActive(file); - if (node->GetNumberOfActiveFiles() == 0) RemoveActiveNode(node); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove node from the list of actives. - -void TPacketizerAdaptive::RemoveActiveNode(TFileNode *node) -{ - fActive->Remove(node); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset the internal data structure for packet distribution. - -void TPacketizerAdaptive::Reset() -{ - fUnAllocated->Clear(); - fUnAllocated->AddAll(fFileNodes); - - fActive->Clear(); - - TIter files(fFileNodes); - TFileNode *fn; - while ((fn = (TFileNode*) files.Next()) != 0) { - fn->Reset(); - } - - TIter slaves(fSlaveStats); - TObject *key; - while ((key = slaves.Next()) != 0) { - TSlaveStat *slstat = (TSlaveStat*) fSlaveStats->GetValue(key); - if (!slstat) { - Warning("Reset", "TSlaveStat associated to key '%s' is NULL", key->GetName()); - continue; - } - // Find out which file nodes are on the worker machine and assign the - // one with less workers assigned - TFileNode *fnmin = 0; - Int_t fncnt = fSlaveStats->GetSize(); - files.Reset(); - while ((fn = (TFileNode*) files.Next()) != 0) { - if (!strcmp(slstat->GetName(), TUrl(fn->GetName()).GetHost())) { - if (fn->GetMySlaveCnt() < fncnt) { - fnmin = fn; - fncnt = fn->GetMySlaveCnt(); - } - } - } - if (fnmin != 0 ) { - slstat->SetFileNode(fnmin); - fnmin->IncMySlaveCnt(); - PDB(kPacketizer, 2) - Info("Reset","assigning node '%s' to '%s' (cnt: %d)", - fnmin->GetName(), slstat->GetName(), fnmin->GetMySlaveCnt()); - } - slstat->fCurFile = 0; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check existence of file/dir/tree an get number of entries. -/// Assumes the files have been setup. - -void TPacketizerAdaptive::ValidateFiles(TDSet *dset, TList *slaves, - Long64_t maxent, Bool_t byfile) -{ - TMap slaves_by_sock; - TMonitor mon; - TList workers; - - - // Setup the communication infrastructure - - workers.AddAll(slaves); - TIter si(slaves); - TSlave *slm; - while ((slm = (TSlave*)si.Next()) != 0) { - PDB(kPacketizer,3) - Info("ValidateFiles","socket added to monitor: %p (%s)", - slm->GetSocket(), slm->GetName()); - mon.Add(slm->GetSocket()); - slaves_by_sock.Add(slm->GetSocket(), slm); - } - - mon.DeActivateAll(); - - ((TProof*)gProof)->DeActivateAsyncInput(); - - // Some monitoring systems (TXSocketHandler) need to know this - ((TProof*)gProof)->fCurrentMonitor = &mon; - - // Identify the type - if (!strcmp(dset->GetType(), "TTree")) SetBit(TVirtualPacketizer::kIsTree); - - // Preparing for client notification - TString msg("Validating files"); - UInt_t n = 0; - UInt_t tot = dset->GetListOfElements()->GetSize(); - Bool_t st = kTRUE; - - Long64_t totent = 0, nopenf = 0; - while (kTRUE) { - - // send work - while (TSlave *s = (TSlave *)workers.First()) { - - workers.Remove(s); - - // find a file - - TSlaveStat *slstat = (TSlaveStat*)fSlaveStats->GetValue(s); - if (!slstat) { - Error("ValidateFiles", "TSlaveStat associated to slave '%s' is NULL", s->GetName()); - continue; - } - - TFileNode *node = 0; - TFileStat *file = 0; - - // try its own node first - if ((node = slstat->GetFileNode()) != 0) { - PDB(kPacketizer,3) node->Print(); - file = GetNextUnAlloc(node); - if (file == 0) - slstat->SetFileNode(0); - } - - // look for a file on any other node if necessary - if (file == 0) - file = GetNextUnAlloc(); - - if (file != 0) { - // files are done right away - RemoveActive(file); - - slstat->fCurFile = file; - TDSetElement *elem = file->GetElement(); - Long64_t entries = elem->GetEntries(kTRUE, kFALSE); - if (entries < 0 || strlen(elem->GetTitle()) <= 0) { - // This is decremented when we get the reply - file->GetNode()->IncExtSlaveCnt(slstat->GetName()); - TMessage m(kPROOF_GETENTRIES); - m << dset->IsTree() - << TString(elem->GetFileName()) - << TString(elem->GetDirectory()) - << TString(elem->GetObjName()); - - s->GetSocket()->Send( m ); - mon.Activate(s->GetSocket()); - PDB(kPacketizer,2) - Info("ValidateFiles", - "sent to worker-%s (%s) via %p GETENTRIES on %s %s %s %s", - s->GetOrdinal(), s->GetName(), s->GetSocket(), - dset->IsTree() ? "tree" : "objects", elem->GetFileName(), - elem->GetDirectory(), elem->GetObjName()); - } else { - // Fill the info - elem->SetTDSetOffset(entries); - if (entries > 0) { - // Most likely valid - elem->SetValid(); - if (!elem->GetEntryList()) { - if (elem->GetFirst() > entries) { - Error("ValidateFiles", - "first (%lld) higher then number of entries (%lld) in %s", - elem->GetFirst(), entries, elem->GetFileName()); - // disable element - slstat->fCurFile->SetDone(); - elem->Invalidate(); - dset->SetBit(TDSet::kSomeInvalid); - } - if (elem->GetNum() == -1) { - elem->SetNum(entries - elem->GetFirst()); - } else if (elem->GetFirst() + elem->GetNum() > entries) { - Warning("ValidateFiles", "num (%lld) + first (%lld) larger then number of" - " keys/entries (%lld) in %s", elem->GetNum(), elem->GetFirst(), - entries, elem->GetFileName()); - elem->SetNum(entries - elem->GetFirst()); - } - PDB(kPacketizer,2) - Info("ValidateFiles", - "found elem '%s' with %lld entries", elem->GetFileName(), entries); - } - } - // Count - totent += entries; - nopenf++; - // Notify the client - n++; - gProof->SendDataSetStatus(msg, n, tot, st); - - // This worker is ready for the next validation - workers.Add(s); - } - } - } - - // Check if there is anything to wait for - if (mon.GetActive() == 0) { - if (byfile && maxent > 0) { - // How many files do we still need ? - Long64_t nrestf = (maxent - totent) * nopenf / totent ; - if (nrestf <= 0 && maxent > totent) nrestf = 1; - if (nrestf > 0) { - PDB(kPacketizer,3) - Info("ValidateFiles", "{%lld, %lld, %lld}: needs to validate %lld more files", - maxent, totent, nopenf, nrestf); - si.Reset(); - while ((slm = (TSlave *) si.Next()) && nrestf--) { - workers.Add(slm); - } - continue; - } else { - PDB(kPacketizer,3) - Info("ValidateFiles", "no need to validate more files"); - break; - } - } else { - break; - } - } - - PDB(kPacketizer,3) { - Info("ValidateFiles", "waiting for %d slaves:", mon.GetActive()); - TList *act = mon.GetListOfActives(); - TIter next(act); - while (TSocket *s = (TSocket*) next()) { - TSlave *sl = (TSlave *) slaves_by_sock.GetValue(s); - if (sl) - Info("ValidateFiles", " worker-%s (%s)", - sl->GetOrdinal(), sl->GetName()); - } - delete act; - } - - TSocket *sock = mon.Select(); - // If we have been interrupted break - if (!sock) { - Error("ValidateFiles", "selection has been interrupted - STOP"); - mon.DeActivateAll(); - fValid = kFALSE; - break; - } - mon.DeActivate(sock); - - PDB(kPacketizer,3) Info("ValidateFiles", "select returned: %p", sock); - - TSlave *slave = (TSlave *) slaves_by_sock.GetValue( sock ); - if (!sock->IsValid()) { - // A socket got invalid during validation - Error("ValidateFiles", "worker-%s (%s) got invalid - STOP", - slave->GetOrdinal(), slave->GetName()); - ((TProof*)gProof)->MarkBad(slave, "socket got invalid during validation"); - fValid = kFALSE; - break; - } - - TMessage *reply; - - if (sock->Recv(reply) <= 0) { - // Notify - Error("ValidateFiles", "Recv failed! for worker-%s (%s)", - slave->GetOrdinal(), slave->GetName()); - // Help! lost a slave? ('slave' is deleted inside here ...) - ((TProof*)gProof)->MarkBad(slave, "receive failed during validation"); - fValid = kFALSE; - continue; - } - - if (reply->What() != kPROOF_GETENTRIES) { - // Not what we want: handover processing to the central machinery - Int_t what = reply->What(); - ((TProof*)gProof)->HandleInputMessage(slave, reply); - if (what == kPROOF_FATAL) { - Error("ValidateFiles", "kPROOF_FATAL from worker-%s (%s)", - slave->GetOrdinal(), slave->GetName()); - fValid = kFALSE; - } else { - // Reactivate the socket - mon.Activate(sock); - } - // Get next message - continue; - } - - TSlaveStat *slavestat = (TSlaveStat*) fSlaveStats->GetValue( slave ); - TDSetElement *e = slavestat->fCurFile->GetElement(); - slavestat->fCurFile->GetNode()->DecExtSlaveCnt(slavestat->GetName()); - Long64_t entries; - - (*reply) >> entries; - - // Extract object name, if there - if ((reply->BufferSize() > reply->Length())) { - TString objname; - (*reply) >> objname; - e->SetTitle(objname); - } - - e->SetTDSetOffset(entries); - if (entries > 0) { - - // This dataset element is most likely valid - e->SetValid(); - - if (!e->GetEntryList()) { - if (e->GetFirst() > entries) { - Error("ValidateFiles", - "first (%lld) higher then number of entries (%lld) in %s", - e->GetFirst(), entries, e->GetFileName()); - - // Invalidate the element - slavestat->fCurFile->SetDone(); - e->Invalidate(); - dset->SetBit(TDSet::kSomeInvalid); - } - - if (e->GetNum() == -1) { - e->SetNum(entries - e->GetFirst()); - } else if (e->GetFirst() + e->GetNum() > entries) { - Error("ValidateFiles", - "num (%lld) + first (%lld) larger then number of keys/entries (%lld) in %s", - e->GetNum(), e->GetFirst(), entries, e->GetFileName()); - e->SetNum(entries - e->GetFirst()); - } - } - - // Count - totent += entries; - nopenf++; - - // Notify the client - n++; - gProof->SendDataSetStatus(msg, n, tot, st); - - } else { - - Error("ValidateFiles", "cannot get entries for file: %s - skipping", e->GetFileName() ); - // - // Need to fix this with a user option to allow incomplete file sets (rdm) - // - //fValid = kFALSE; // all element must be readable! - if (gProofServ) { - TMessage m(kPROOF_MESSAGE); - m << TString(Form("Cannot get entries for file: %s - skipping", - e->GetFileName())); - gProofServ->GetSocket()->Send(m); - } - - // invalidate element - e->Invalidate(); - dset->SetBit(TDSet::kSomeInvalid); - } - PDB(kPacketizer,3) Info("ValidateFiles", " %lld events validated", totent); - - // Ready for the next job, unless we have enough files - if (maxent < 0 || ((totent < maxent) && !byfile)) - workers.Add(slave); - } - - // report std. output from slaves?? - - ((TProof*)gProof)->ActivateAsyncInput(); - - // This needs to be reset - ((TProof*)gProof)->fCurrentMonitor = 0; - - // No reason to continue if invalid - if (!fValid) - return; - - // compute the offset for each file element - Long64_t offset = 0; - Long64_t newOffset = 0; - TIter next(dset->GetListOfElements()); - TDSetElement *el; - while ( (el = dynamic_cast (next())) ) { - if (el->GetValid()) { - newOffset = offset + el->GetTDSetOffset(); - el->SetTDSetOffset(offset); - offset = newOffset; - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// The result depends on the fStrategy - -Int_t TPacketizerAdaptive::CalculatePacketSize(TObject *slStatPtr, Long64_t cachesz, Int_t learnent) -{ - Long64_t num; - if (fStrategy == 0) { - // TPacketizer's heuristic for starting packet size - // Constant packet size; - Int_t nslaves = fSlaveStats->GetSize(); - if (nslaves > 0) { - num = fTotalEntries / (fPacketAsAFraction * nslaves); - } else { - num = 1; - } - } else { - // The dynamic heuristic for setting the packet size (default) - // Calculates the packet size based on performance of this slave - // and estimated time left until the end of the query. - TSlaveStat* slstat = (TSlaveStat*)slStatPtr; - Float_t rate = slstat->GetCurRate(); - if (!rate) - rate = slstat->GetAvgRate(); - if (rate) { - - // Global average rate - Float_t avgProcRate = (GetEntriesProcessed()/(GetCumProcTime() / fSlaveStats->GetSize())); - Float_t packetTime = ((fTotalEntries - GetEntriesProcessed())/avgProcRate)/fPacketAsAFraction; - - // Bytes-to-Event conversion - Float_t bevt = (GetEntriesProcessed() > 0) ? GetBytesRead() / GetEntriesProcessed() : -1.; - - // Make sure it is not smaller then the cache, if the info is available and the size - // synchronization is required. But apply the cache-packet size synchronization only if there - // are enough left files to process and the files are all of similar sizes. Otherwise we risk - // to not exploit optimally all potentially active workers. - Bool_t cpsync = fCachePacketSync; - if (fMaxEntriesRatio > 0. && cpsync) { - if (fFilesToProcess && fFilesToProcess->GetSize() <= fSlaveStats->GetSize()) { - Long64_t remEntries = fTotalEntries - GetEntriesProcessed(); - Long64_t maxEntries = -1; - if (fFilesToProcess->Last()) { - TDSetElement *elem = (TDSetElement *) ((TPacketizerAdaptive::TFileStat *) fFilesToProcess->Last())->GetElement(); - if (elem) maxEntries = elem->GetNum(); - } - if (maxEntries > remEntries / fSlaveStats->GetSize() * fMaxEntriesRatio) { - PDB(kPacketizer,3) { - Info("CalculatePacketSize", "%s: switching off synchronization of packet and cache sizes:", slstat->GetOrdinal()); - Info("CalculatePacketSize", "%s: few files (%d) remaining of very different sizes (max/avg = %.2f > %.2f)", - slstat->GetOrdinal(), fFilesToProcess->GetSize(), - (Double_t)maxEntries / remEntries * fSlaveStats->GetSize(), fMaxEntriesRatio); - } - cpsync = kFALSE; - } - } - } - if (bevt > 0. && cachesz > 0 && cpsync) { - if ((Long64_t)(rate * packetTime * bevt) < cachesz) - packetTime = cachesz / bevt / rate; - } - - // Apply min-max again, if required - if (fMaxPacketTime > 0. && packetTime > fMaxPacketTime) packetTime = fMaxPacketTime; - if (fMinPacketTime > 0. && packetTime < fMinPacketTime) packetTime = fMinPacketTime; - - // Translate the packet length in number of entries - num = (Long64_t)(rate * packetTime); - - // Notify - PDB(kPacketizer,2) - Info("CalculatePacketSize","%s: avgr: %f, rate: %f, left: %lld, pacT: %f, sz: %f (csz: %f), num: %lld", - slstat->GetOrdinal(), avgProcRate, rate, fTotalEntries - GetEntriesProcessed(), - packetTime, ((bevt > 0) ? num*bevt/1048576. : -1.), cachesz/1048576., num); - - } else { - // First packet for this worker in this query - // Twice the learning phase - num = (learnent > 0) ? 5 * learnent : 1000; - - // Notify - PDB(kPacketizer,2) - Info("CalculatePacketSize","%s: num: %lld", slstat->GetOrdinal(), num); - } - } - if (num < 1) num = 1; - return num; -} - -//////////////////////////////////////////////////////////////////////////////// -/// To be used by GetNextPacket but also in reaction to kPROOF_STOPPROCESS -/// message (when the worker was asked to stop processing during a packet). -/// returns the #%entries intended in the last packet - #%processed entries - -Int_t TPacketizerAdaptive::AddProcessed(TSlave *sl, - TProofProgressStatus *status, - Double_t latency, - TList **listOfMissingFiles) -{ - // find slave - TSlaveStat *slstat = (TSlaveStat*) fSlaveStats->GetValue( sl ); - if (!slstat) { - Error("AddProcessed", "%s: TSlaveStat instance for worker %s not found!", - (sl ? sl->GetOrdinal() : "x.x"), - (sl ? sl->GetName() : "**undef**")); - return -1; - } - - // update stats & free old element - - if ( slstat->fCurElem != 0 ) { - Long64_t expectedNumEv = slstat->fCurElem->GetNum(); - // Calculate the number of events processed in the last packet - Long64_t numev; - if (status && status->GetEntries() > 0) - numev = status->GetEntries() - slstat->GetEntriesProcessed(); - else - numev = 0; - - // Calculate the progress made in the last packet - TProofProgressStatus *progress = 0; - if (numev > 0) { - // This also moves the pointer in the corrsponding TFileInfo - progress = slstat->AddProcessed(status); - if (progress) { - (*fProgressStatus) += *progress; - // update processing rate - slstat->UpdateRates(status); - } - } else { - progress = new TProofProgressStatus(); - } - if (progress) { - PDB(kPacketizer,2) - Info("AddProcessed", "%s: %s: %lld %7.3lf %7.3lf %7.3lf %lld", - sl->GetOrdinal(), sl->GetName(), progress->GetEntries(), latency, - progress->GetProcTime(), progress->GetCPUTime(), progress->GetBytesRead()); - - if (gPerfStats) - gPerfStats->PacketEvent(sl->GetOrdinal(), sl->GetName(), - slstat->fCurElem->GetFileName(), - progress->GetEntries(), - latency, - progress->GetProcTime(), - progress->GetCPUTime(), - progress->GetBytesRead()); - delete progress; - } - if (numev != expectedNumEv) { - // The last packet was not fully processed - // and will be split in two: - // - The completed part was marked as done. - // - Create a new packet with the part to be resubmitted. - TDSetElement *newPacket = new TDSetElement(*(slstat->fCurElem)); - if (newPacket && numev < expectedNumEv) { - Long64_t first = newPacket->GetFirst(); - newPacket->SetFirst(first + numev); - if (ReassignPacket(newPacket, listOfMissingFiles) == -1) - SafeDelete(newPacket); - } else - Error("AddProcessed", "%s: processed too much? (%lld, %lld)", - sl->GetOrdinal(), numev, expectedNumEv); - - // TODO: a signal handler which will send info from the worker - // after a packet fails. - /* Add it to the failed packets list. - if (!fFailedPackets) { - fFailedPackets = new TList(); - } - fFailedPackets->Add(slstat->fCurElem); - */ - } - - slstat->fCurElem = 0; - return (expectedNumEv - numev); - } else { - // the kPROOF_STOPPRPOCESS message is send after the worker receives zero - // as the reply to kPROOF_GETNEXTPACKET - return -1; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next packet; -/// A meaningfull difference to TPacketizer is the fact that this -/// packetizer, for each worker, tries to predict whether the worker -/// will finish processing it's local files before the end of the query. -/// If yes, it allocates, to those workers, files from non-slave filenodes -/// or from slaves that are overloaded. The check is done every time a new -/// file needs to be assigned. - -TDSetElement *TPacketizerAdaptive::GetNextPacket(TSlave *sl, TMessage *r) -{ - if ( !fValid ) { - return 0; - } - - // find slave - - TSlaveStat *slstat = (TSlaveStat*) fSlaveStats->GetValue( sl ); - if (!slstat) { - Error("GetNextPacket", "TSlaveStat instance for worker %s not found!", - (sl ? sl->GetName() : "**undef**")); - return 0; - } - - // Attach to current file - TFileStat *file = slstat->fCurFile; - - // Update stats & free old element - - Bool_t firstPacket = kFALSE; - Long64_t cachesz = -1; - Int_t learnent = -1; - if ( slstat->fCurElem != 0 ) { - - Long64_t restEntries = 0; - Double_t latency, proctime, proccpu; - TProofProgressStatus *status = 0; - Bool_t fileNotOpen = kFALSE, fileCorrupted = kFALSE; - - if (sl->GetProtocol() > 18) { - - (*r) >> latency; - (*r) >> status; - - if (sl->GetProtocol() > 25) { - (*r) >> cachesz >> learnent; - if (r->BufferSize() > r->Length()) (*r) >> restEntries; - } - fileNotOpen = status->TestBit(TProofProgressStatus::kFileNotOpen) ? kTRUE : kFALSE; - fileCorrupted = status->TestBit(TProofProgressStatus::kFileCorrupted) ? kTRUE : kFALSE; - - } else { - - Long64_t bytesRead = -1; - - (*r) >> latency >> proctime >> proccpu; - // only read new info if available - if (r->BufferSize() > r->Length()) (*r) >> bytesRead; - if (r->BufferSize() > r->Length()) (*r) >> restEntries; - Long64_t totev = 0; - if (r->BufferSize() > r->Length()) (*r) >> totev; - - status = new TProofProgressStatus(totev, bytesRead, -1, proctime, proccpu); - fileNotOpen = (restEntries < 0) ? kTRUE : kFALSE; - } - - if (!fileNotOpen && !fileCorrupted) { - if (AddProcessed(sl, status, latency) != 0) - Error("GetNextPacket", "%s: the worker processed a different # of entries", sl->GetOrdinal()); - if (fProgressStatus->GetEntries() >= fTotalEntries) { - if (fProgressStatus->GetEntries() > fTotalEntries) - Error("GetNextPacket", "%s: processed too many entries! (%lld, %lld)", - sl->GetOrdinal(), fProgressStatus->GetEntries(), fTotalEntries); - // Send last timer message and stop the timer - HandleTimer(0); - SafeDelete(fProgress); - } - } else { - if (file) { - if (file->GetElement()) { - if (fileCorrupted) { - Info("GetNextPacket", "%s: file '%s' turned corrupted: invalidating file (%lld)", - sl->GetOrdinal(), file->GetElement()->GetName(), restEntries); - Int_t nunproc = AddProcessed(sl, status, latency); - PDB(kPacketizer,1) - Info("GetNextPacket", "%s: %d entries un-processed", sl->GetOrdinal(), nunproc); - // Remaining to be processed - Long64_t num = 0; - if (file->GetElement()->TestBit(TDSetElement::kCorrupted)) { - // Add the remainign entries in the packet to the ones already registered - num = file->GetElement()->GetEntries() + restEntries; - } else { - // First call: add the remaining entries in the packet and those of the file - // not yet assigned - Long64_t rest = file->GetElement()->GetEntries() - file->GetNextEntry(); - num = restEntries + rest; - } - file->GetElement()->SetEntries(num); - PDB(kPacketizer,1) - Info("GetNextPacket", "%s: removed file: %s, entries left: %lld", sl->GetOrdinal(), - file->GetElement()->GetName(), file->GetElement()->GetEntries()); - // Flag as corrupted - file->GetElement()->SetBit(TDSetElement::kCorrupted); - } else { - Info("GetNextPacket", "%s: file '%s' could not be open: invalidating related element", - sl->GetOrdinal(), file->GetElement()->GetName()); - } - // Invalidate the element - file->GetElement()->Invalidate(); - // Add it to the failed packets list - if (!fFailedPackets) fFailedPackets = new TList(); - if (!fFailedPackets->FindObject(file->GetElement())) - fFailedPackets->Add(file->GetElement()); - } - // Deactivate this TFileStat - file->SetDone(); - RemoveActive(file); - } else { - Info("GetNextPacket", "%s: error raised by worker, but TFileStat object invalid:" - " protocol error?", sl->GetOrdinal()); - } - } - } else { - firstPacket = kTRUE; - } - - if ( fStop ) { - HandleTimer(0); - return 0; - } - - TString nodeName; - if (file != 0) nodeName = file->GetNode()->GetName(); - TString nodeHostName(slstat->GetName()); - - PDB(kPacketizer,3) - Info("GetNextPacket", "%s: entries processed: %lld - looking for a packet from node '%s'", - sl->GetOrdinal(), fProgressStatus->GetEntries(), nodeName.Data()); - - // If current file is just finished - if ( file != 0 && file->IsDone() ) { - file->GetNode()->DecExtSlaveCnt(slstat->GetName()); - file->GetNode()->DecRunSlaveCnt(); - if (gPerfStats) - gPerfStats->FileEvent(sl->GetOrdinal(), sl->GetName(), file->GetNode()->GetName(), - file->GetElement()->GetFileName(), kFALSE); - file = 0; - } - // Reset the current file field - slstat->fCurFile = file; - - Long64_t avgEventsLeftPerSlave = - (fTotalEntries - fProgressStatus->GetEntries()) / fSlaveStats->GetSize(); - if (fTotalEntries == fProgressStatus->GetEntries()) - return 0; - // Get a file if needed - if ( file == 0) { - // Needs a new file - Bool_t openLocal; - // Aiming for localPreference == 1 when #local == #remote events left - Float_t localPreference = fBaseLocalPreference - (fNEventsOnRemLoc / - (0.4 *(fTotalEntries - fProgressStatus->GetEntries()))); - if ( slstat->GetFileNode() != 0 ) { - // Local file node exists and has more events to process. - fUnAllocated->Sort(); - TFileNode* firstNonLocalNode = (TFileNode*)fUnAllocated->First(); - Bool_t nonLocalNodePossible; - if (fForceLocal) - nonLocalNodePossible = 0; - else - nonLocalNodePossible = firstNonLocalNode ? - (fMaxSlaveCnt < 0 || (fMaxSlaveCnt > 0 && firstNonLocalNode->GetExtSlaveCnt() < fMaxSlaveCnt)) - : 0; - openLocal = !nonLocalNodePossible; - Float_t slaveRate = slstat->GetAvgRate(); - if ( nonLocalNodePossible && fStrategy == 1) { - // OpenLocal is set to kFALSE - if ( slstat->GetFileNode()->GetRunSlaveCnt() > - slstat->GetFileNode()->GetMySlaveCnt() - 1 ) - // External slaves help slstat -> don't open nonlocal files - // -1 because, at this point slstat is not running - openLocal = kTRUE; - else if ( slaveRate == 0 ) { // first file for this slave - // GetLocalEventsLeft() counts the potential slave - // as running on its fileNode. - if ( slstat->GetLocalEventsLeft() * localPreference - > (avgEventsLeftPerSlave)) - openLocal = kTRUE; - else if ( (firstNonLocalNode->GetEventsLeftPerSlave()) - < slstat->GetLocalEventsLeft() * localPreference ) - openLocal = kTRUE; - else if ( firstNonLocalNode->GetExtSlaveCnt() > 1 ) - openLocal = kTRUE; - else if ( firstNonLocalNode->GetRunSlaveCnt() == 0 ) - openLocal = kTRUE; - } else { - // At this point slstat has a non zero avg rate > 0 - Float_t slaveTime = slstat->GetLocalEventsLeft()/slaveRate; - // And thus fCumProcTime, fProcessed > 0 - Float_t avgTime = avgEventsLeftPerSlave - /(fProgressStatus->GetEntries()/GetCumProcTime()); - if (slaveTime * localPreference > avgTime) - openLocal = kTRUE; - else if ((firstNonLocalNode->GetEventsLeftPerSlave()) - < slstat->GetLocalEventsLeft() * localPreference) - openLocal = kTRUE; - } - } - if (openLocal || fStrategy == 0) { - // Try its own node - file = slstat->GetFileNode()->GetNextUnAlloc(); - if (!file) - file = slstat->GetFileNode()->GetNextActive(); - if ( file == 0 ) { - // No more files on this worker - slstat->SetFileNode(0); - } - } - } - - // Try to find an unused filenode first - if(file == 0 && !fForceLocal) - file = GetNextUnAlloc(0, nodeHostName); - - // Then look at the active filenodes - if(file == 0 && !fForceLocal) - file = GetNextActive(); - - if (file == 0) return 0; - - PDB(kPacketizer,3) if (fFilesToProcess) fFilesToProcess->Print(); - - slstat->fCurFile = file; - // if remote and unallocated file - if (file->GetNode()->GetMySlaveCnt() == 0 && - file->GetElement()->GetFirst() == file->GetNextEntry()) { - fNEventsOnRemLoc -= file->GetElement()->GetNum(); - if (fNEventsOnRemLoc < 0) { - Error("GetNextPacket", - "inconsistent value for fNEventsOnRemLoc (%lld): stop delivering packets!", - fNEventsOnRemLoc); - return 0; - } - } - file->GetNode()->IncExtSlaveCnt(slstat->GetName()); - file->GetNode()->IncRunSlaveCnt(); - if (gPerfStats) - gPerfStats->FileEvent(sl->GetOrdinal(), sl->GetName(), - file->GetNode()->GetName(), - file->GetElement()->GetFileName(), kTRUE); - } - - Long64_t num = CalculatePacketSize(slstat, cachesz, learnent); - - // Get a packet - - TDSetElement *base = file->GetElement(); - Long64_t first = file->GetNextEntry(); - Long64_t last = base->GetFirst() + base->GetNum(); - - // If the remaining part is smaller than the (packetsize * 1.5) - // then increase the packetsize - - if ( first + num * 1.5 >= last ) { - num = last - first; - file->SetDone(); // done - // Delete file from active list (unalloc list is single pass, no delete needed) - RemoveActive(file); - } - - // Update NextEntry in the file object - file->MoveNextEntry(num); - - slstat->fCurElem = CreateNewPacket(base, first, num); - if (base->GetEntryList()) - slstat->fCurElem->SetEntryList(base->GetEntryList(), first, num); - - // Flag the first packet of a new run (dataset) - if (firstPacket) - slstat->fCurElem->SetBit(TDSetElement::kNewRun); - else - slstat->fCurElem->ResetBit(TDSetElement::kNewRun); - - PDB(kPacketizer,2) - Info("GetNextPacket","%s: %s %lld %lld (%lld)", sl->GetOrdinal(), base->GetFileName(), first, first + num - 1, num); - - return slstat->fCurElem; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the number of workers still processing - -Int_t TPacketizerAdaptive::GetActiveWorkers() -{ - Int_t actw = 0; - TIter nxw(fSlaveStats); - TObject *key; - while ((key = nxw())) { - TSlaveStat *wrkstat = (TSlaveStat *) fSlaveStats->GetValue(key); - if (wrkstat && wrkstat->fCurFile) actw++; - } - // Done - return actw; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get Estimation of the current rate; just summing the current rates of -/// the active workers - -Float_t TPacketizerAdaptive::GetCurrentRate(Bool_t &all) -{ - all = kTRUE; - // Loop over the workers - Float_t currate = 0.; - if (fSlaveStats && fSlaveStats->GetSize() > 0) { - TIter nxw(fSlaveStats); - TObject *key; - while ((key = nxw()) != 0) { - TSlaveStat *slstat = (TSlaveStat *) fSlaveStats->GetValue(key); - if (slstat && slstat->GetProgressStatus() && slstat->GetEntriesProcessed() > 0) { - // Sum-up the current rates - currate += slstat->GetProgressStatus()->GetCurrentRate(); - } else { - all = kFALSE; - } - } - } - // Done - return currate; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get estimation for the number of processed entries and bytes read at time t, -/// based on the numbers already processed and the latests worker measured speeds. -/// If t <= 0 the current time is used. -/// Only the estimation for the entries is currently implemented. -/// This is needed to smooth the instantaneous rate plot. - -Int_t TPacketizerAdaptive::GetEstEntriesProcessed(Float_t t, Long64_t &ent, - Long64_t &bytes, Long64_t &calls) -{ - // Default value - ent = GetEntriesProcessed(); - bytes = GetBytesRead(); - calls = GetReadCalls(); - - // Parse option - if (fUseEstOpt == kEstOff) - // Do not use estimation - return 0; - Bool_t current = (fUseEstOpt == kEstCurrent) ? kTRUE : kFALSE; - - TTime tnow = gSystem->Now(); - Double_t now = (t > 0) ? (Double_t)t : Long64_t(tnow) / (Double_t)1000.; - Double_t dt = -1; - - // Loop over the workers - Bool_t all = kTRUE; - Float_t trate = 0.; - if (fSlaveStats && fSlaveStats->GetSize() > 0) { - ent = 0; - TIter nxw(fSlaveStats); - TObject *key; - while ((key = nxw()) != 0) { - TSlaveStat *slstat = (TSlaveStat *) fSlaveStats->GetValue(key); - if (slstat) { - // Those surely processed - Long64_t e = slstat->GetEntriesProcessed(); - if (e <= 0) all = kFALSE; - // Time elapsed since last update - dt = now - slstat->GetProgressStatus()->GetLastUpdate(); - // Add estimated entries processed since last update - Float_t rate = (current && slstat->GetCurRate() > 0) ? slstat->GetCurRate() - : slstat->GetAvgRate(); - trate += rate; - // Add estimated entries processed since last update - e += (Long64_t) (dt * rate); - // Add to the total - ent += e; - // Notify - PDB(kPacketizer,3) - Info("GetEstEntriesProcessed","%s: e:%lld rate:%f dt:%f e:%lld", - slstat->fSlave->GetOrdinal(), - slstat->GetEntriesProcessed(), rate, dt, e); - } - } - } - // Notify - dt = now - fProgressStatus->GetLastUpdate(); - PDB(kPacketizer,2) - Info("GetEstEntriesProcessed", - "dt: %f, estimated entries: %lld (%lld), bytes read: %lld rate: %f (all: %d)", - dt, ent, GetEntriesProcessed(), bytes, trate, all); - - // Check values - ent = (ent > 0) ? ent : fProgressStatus->GetEntries(); - ent = (ent <= fTotalEntries) ? ent : fTotalEntries; - bytes = (bytes > 0) ? bytes : fProgressStatus->GetBytesRead(); - - // Done - return ((all) ? 0 : 1); -} - -//////////////////////////////////////////////////////////////////////////////// -/// This method can be called at any time during processing -/// as an effect of handling kPROOF_STOPPROCESS -/// If the output list from this worker is going to be sent back to the master, -/// the 'status' includes the number of entries processed by the slave. -/// From this we calculate the remaining part of the packet. -/// 0 indicates that the results from that worker were lost completely. -/// Assume that the filenodes for which we have a TFileNode object -/// are still up and running. - -void TPacketizerAdaptive::MarkBad(TSlave *s, TProofProgressStatus *status, - TList **listOfMissingFiles) -{ - TSlaveStat *slaveStat = (TSlaveStat *)(fSlaveStats->GetValue(s)); - if (!slaveStat) { - Error("MarkBad", "Worker does not exist"); - return; - } - // Update worker counters - if (slaveStat->fCurFile && slaveStat->fCurFile->GetNode()) { - slaveStat->fCurFile->GetNode()->DecExtSlaveCnt(slaveStat->GetName()); - slaveStat->fCurFile->GetNode()->DecRunSlaveCnt(); - } - - // If status is defined, the remaining part of the last packet is - // reassigned in AddProcessed called from handling kPROOF_STOPPROCESS - if (!status) { - // Get the subset processed by the bad worker. - TList *subSet = slaveStat->GetProcessedSubSet(); - if (subSet) { - // Take care of the current packet - if (slaveStat->fCurElem) { - subSet->Add(slaveStat->fCurElem); - } - // Merge overlapping or subsequent elements - Int_t nmg = 0, ntries = 100; - TDSetElement *e = 0, *enxt = 0; - do { - nmg = 0; - e = (TDSetElement *) subSet->First(); - while ((enxt = (TDSetElement *) subSet->After(e))) { - if (e->MergeElement(enxt) >= 0) { - nmg++; - subSet->Remove(enxt); - delete enxt; - } else { - e = enxt; - } - } - } while (nmg > 0 && --ntries > 0); - // reassign the packets assigned to the bad slave and save the size; - SplitPerHost(subSet, listOfMissingFiles); - // the elements were reassigned so should not be deleted - subSet->SetOwner(0); - } else { - Warning("MarkBad", "subset processed by bad worker not found!"); - } - (*fProgressStatus) -= *(slaveStat->GetProgressStatus()); - } - // remove slavestat from the map - fSlaveStats->Remove(s); - delete slaveStat; - // recalculate fNEventsOnRemLoc and others - InitStats(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// The file in the listOfMissingFiles can appear several times; -/// in order to fix that, a TDSetElement::Merge method is needed. - -Int_t TPacketizerAdaptive::ReassignPacket(TDSetElement *e, - TList **listOfMissingFiles) -{ - if (!e) { - Error("ReassignPacket", "empty packet!"); - return -1; - } - // Check the old filenode - TUrl url = e->GetFileName(); - // Check the host from which 'e' was previously read. - // Map non URL filenames to dummy host - TString host; - if (!url.IsValid() || strncmp(url.GetProtocol(),"root", 4)) { - host = "no-host"; - } else { - host = url.GetHost(); - } - - // If accessible add it back to the old node - // and do DecProcessed - TFileNode *node = (TFileNode*) fFileNodes->FindObject( host ); - if (node && fTryReassign) { - // The packet 'e' was processing data from this node. - node->DecreaseProcessed(e->GetNum()); - // The file should be already in fFilesToProcess ... - node->Add(e, kFALSE); - if (!fUnAllocated->FindObject(node)) - fUnAllocated->Add(node); - return 0; - } else { - // Add to the list of missing files - TFileInfo *fi = e->GetFileInfo(); - if (listOfMissingFiles && *listOfMissingFiles) - (*listOfMissingFiles)->Add((TObject *)fi); - return -1; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Split into per host entries -/// The files in the listOfMissingFiles can appear several times; -/// in order to fix that, a TDSetElement::Merge method is needed. - -void TPacketizerAdaptive::SplitPerHost(TList *elements, - TList **listOfMissingFiles) -{ - if (!elements) { - Error("SplitPerHost", "Empty list of packets!"); - return; - } - if (elements->GetSize() <= 0) { - Error("SplitPerHost", "The input list contains no elements"); - return; - } - TIter subSetIter(elements); - TDSetElement *e; - while ((e = (TDSetElement*) subSetIter.Next())) { - if (ReassignPacket(e, listOfMissingFiles) == -1) { - // Remove from the list in order to delete it. - if (elements->Remove(e)) - Error("SplitPerHost", "Error removing a missing file"); - delete e; - } - - } -} diff --git a/proof/proofplayer/src/TPacketizerFile.cxx b/proof/proofplayer/src/TPacketizerFile.cxx deleted file mode 100644 index d437801f12c71..0000000000000 --- a/proof/proofplayer/src/TPacketizerFile.cxx +++ /dev/null @@ -1,500 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G. Ganis 2009 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TPacketizerFile -\ingroup proofkernel - -This packetizer generates packets which contain a single file path -to be used in process. Used for tasks generating files, like in -PROOF bench. - -*/ - -#include "TPacketizerFile.h" - -#include "Riostream.h" -#include "TDSet.h" -#include "TError.h" -#include "TEventList.h" -#include "TMap.h" -#include "TMessage.h" -#include "TMonitor.h" -#include "TNtupleD.h" -#include "TObject.h" -#include "TParameter.h" -#include "TPerfStats.h" -#include "TProofDebug.h" -#include "TProof.h" -#include "TProofPlayer.h" -#include "TProofServ.h" -#include "TSlave.h" -#include "TSocket.h" -#include "TStopwatch.h" -#include "TTimer.h" -#include "TUrl.h" -#include "TClass.h" -#include "TMath.h" -#include "TObjString.h" -#include "TFileInfo.h" -#include "TFileCollection.h" -#include "THashList.h" - -//------------------------------------------------------------------------------ - -class TPacketizerFile::TSlaveStat : public TVirtualPacketizer::TVirtualSlaveStat { - -friend class TPacketizerFile; - -private: - Long64_t fLastProcessed; // number of processed entries of the last packet - Double_t fSpeed; // estimated current average speed of the processing slave - Double_t fTimeInstant; // stores the time instant when the current packet started - TNtupleD *fCircNtp; // Keeps circular info for speed calculations - Long_t fCircLvl; // Circularity level - -public: - TSlaveStat(TSlave *sl, TList *input); - ~TSlaveStat() override; - - void GetCurrentTime(); - - void UpdatePerformance(Double_t time); - TProofProgressStatus *AddProcessed(TProofProgressStatus *st) override; -}; - -// Iterator wrapper -class TPacketizerFile::TIterObj : public TObject { - -private: - TString fName; // Name of reference - TIter *fIter; // Iterator - -public: - TIterObj(const char *n, TIter *iter) : fName(n), fIter(iter) { } - ~TIterObj() override { if (fIter) delete fIter; } - - const char *GetName() const override {return fName;} - TIter *GetIter() const {return fIter;} - void Print(Option_t* option = "") const override; -}; - -ClassImp(TPacketizerFile); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TPacketizerFile::TPacketizerFile(TList *workers, Long64_t, TList *input, - TProofProgressStatus *st) - : TVirtualPacketizer(input, st) -{ - PDB(kPacketizer,1) Info("TPacketizerFile", "enter"); - ResetBit(TObject::kInvalidObject); - fValid = kFALSE; - fAssigned = 0; - fProcNotAssigned = kTRUE; - fAddFileInfo = kFALSE; - - if (!input || (input && input->GetSize() <= 0)) { - Error("TPacketizerFile", "input file is undefined or empty!"); - SetBit(TObject::kInvalidObject); - return; - } - - // Check if the files not explicitly assigned have to be processed - Int_t procnotass = 1; - if (TProof::GetParameter(input, "PROOF_ProcessNotAssigned", procnotass) == 0) { - if (procnotass == 0) { - Info("TPacketizerFile", "files not assigned to workers will not be processed"); - fProcNotAssigned = kFALSE; - } - } - - // Check if the TFileInfo object has to be added to the packet - Int_t addfileinfo = 0; - if (TProof::GetParameter(input, "PROOF_IncludeFileInfoInPacket", addfileinfo) == 0) { - if (addfileinfo == 1) { - Info("TPacketizerFile", - "TFileInfo object will be included in the packet as associated object"); - fAddFileInfo = kTRUE; - } - } - - // These are the file to be created/processed per node; the information - if (!(fFiles = dynamic_cast(input->FindObject("PROOF_FilesToProcess")))) { - Error("TPacketizerFile", "map of files to be processed/created not found"); - SetBit(TObject::kInvalidObject); - return; - } - - // The worker stats - fSlaveStats = new TMap; - fSlaveStats->SetOwner(kFALSE); - - TList nodes; - nodes.SetOwner(kTRUE); - TSlave *wrk; - TIter si(workers); - while ((wrk = (TSlave *) si.Next())) { - fSlaveStats->Add(wrk, new TSlaveStat(wrk, input)); - TString wrkname = TUrl(wrk->GetName()).GetHostFQDN(); - Info("TPacketizerFile", "worker: %s", wrkname.Data()); - if (!nodes.FindObject(wrkname)) nodes.Add(new TObjString(wrkname)); - } - - // The list of iterators - fIters = new TList; - fIters->SetOwner(kTRUE); - - // There must be something in - fTotalEntries = 0; - fNotAssigned = new TList; - fNotAssigned->SetName("*"); - TIter nxl(fFiles); - TObject *key, *o = 0; - while ((key = nxl()) != 0) { - THashList *wrklist = dynamic_cast(fFiles->GetValue(key)); - if (!wrklist) { - TFileCollection *fc = dynamic_cast(fFiles->GetValue(key)); - if (fc) wrklist = fc->GetList(); - } - if (wrklist) { - TString hname = TUrl(key->GetName()).GetHostFQDN(); - if ((o = nodes.FindObject(hname))) { - fTotalEntries += wrklist->GetSize(); - fIters->Add(new TIterObj(hname, new TIter(wrklist))); - // Notify - PDB(kPacketizer,2) - Info("TPacketizerFile", "%d files of '%s' (fqdn: '%s') assigned to '%s'", - wrklist->GetSize(), key->GetName(), hname.Data(), o->GetName()); - } else { - // We add all to the not assigned list so that they will be distributed - // according to the load - TIter nxf(wrklist); - while ((o = nxf())) - fNotAssigned->Add(o); - // Notify - PDB(kPacketizer,2) - Info("TPacketizerFile", "%d files of '%s' (fqdn: '%s') not assigned", - wrklist->GetSize(), key->GetName(), hname.Data()); - } - } - } - if (fNotAssigned && fNotAssigned->GetSize() > 0) { - fTotalEntries += fNotAssigned->GetSize(); - fIters->Add(new TIterObj("*", new TIter(fNotAssigned))); - Info("TPacketizerFile", "non-assigned files: %d", fNotAssigned->GetSize()); - fNotAssigned->Print(); - } - if (fTotalEntries <= 0) { - Error("TPacketizerFile", "no file path in the map!"); - SetBit(TObject::kInvalidObject); - SafeDelete(fIters); - return; - } else { - Info("TPacketizerFile", "processing %lld files", fTotalEntries); - fIters->Print(); - } - - fStopwatch = new TStopwatch(); - fStopwatch->Start(); - fValid = kTRUE; - PDB(kPacketizer,1) Info("TPacketizerFile", "return"); - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TPacketizerFile::~TPacketizerFile() -{ - if (fNotAssigned) fNotAssigned->SetOwner(kFALSE); - SafeDelete(fNotAssigned); - if (fIters) fIters->SetOwner(kTRUE); - SafeDelete(fIters); - SafeDelete(fStopwatch); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get current time - -Double_t TPacketizerFile::GetCurrentTime() -{ - Double_t retValue = fStopwatch->RealTime(); - fStopwatch->Continue(); - return retValue; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get Estimation of the current rate; just summing the current rates of -/// the active workers - -Float_t TPacketizerFile::GetCurrentRate(Bool_t &all) -{ - all = kTRUE; - // Loop over the workers - Float_t currate = 0.; - if (fSlaveStats && fSlaveStats->GetSize() > 0) { - TIter nxw(fSlaveStats); - TObject *key; - while ((key = nxw()) != 0) { - TSlaveStat *wrkstat = (TSlaveStat *) fSlaveStats->GetValue(key); - if (wrkstat && wrkstat->GetProgressStatus() && wrkstat->GetEntriesProcessed() > 0) { - // Sum-up the current rates - currate += wrkstat->GetProgressStatus()->GetCurrentRate(); - } else { - all = kFALSE; - } - } - } - // Done - return currate; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next packet - -TDSetElement *TPacketizerFile::GetNextPacket(TSlave *wrk, TMessage *r) -{ - TDSetElement *elem = 0; - if (!fValid) return elem; - - // Find slave - TSlaveStat *wrkstat = (TSlaveStat *) fSlaveStats->GetValue(wrk); - if (!wrkstat) { - Error("GetNextPacket", "could not find stat object for worker '%s'!", wrk->GetName()); - return elem; - } - - PDB(kPacketizer,2) - Info("GetNextPacket","worker-%s: fAssigned %lld / %lld", wrk->GetOrdinal(), fAssigned, fTotalEntries); - - // Update stats & free old element - Double_t latency = 0., proctime = 0., proccpu = 0.; - Long64_t bytesRead = -1; - Long64_t totalEntries = -1; // used only to read an old message type - Long64_t totev = 0; - Long64_t numev = -1; - - TProofProgressStatus *status = 0; - if (wrk->GetProtocol() > 18) { - (*r) >> latency; - (*r) >> status; - - // Calculate the progress made in the last packet - TProofProgressStatus *progress = 0; - if (status) { - // upadte the worker status - numev = status->GetEntries() - wrkstat->GetEntriesProcessed(); - progress = wrkstat->AddProcessed(status); - if (progress) { - // (*fProgressStatus) += *progress; - proctime = progress->GetProcTime(); - proccpu = progress->GetCPUTime(); - totev = status->GetEntries(); // for backward compatibility - bytesRead = progress->GetBytesRead(); - delete progress; - } - delete status; - } else - Error("GetNextPacket", "no status came in the kPROOF_GETPACKET message"); - } else { - - (*r) >> latency >> proctime >> proccpu; - - // only read new info if available - if (r->BufferSize() > r->Length()) (*r) >> bytesRead; - if (r->BufferSize() > r->Length()) (*r) >> totalEntries; - if (r->BufferSize() > r->Length()) (*r) >> totev; - - numev = totev - wrkstat->GetEntriesProcessed(); - wrkstat->GetProgressStatus()->IncEntries(numev); - wrkstat->GetProgressStatus()->SetLastUpdate(); - } - - fProgressStatus->IncEntries(numev); - fProgressStatus->SetLastUpdate(); - - PDB(kPacketizer,2) - Info("GetNextPacket","worker-%s (%s): %lld %7.3lf %7.3lf %7.3lf %lld", - wrk->GetOrdinal(), wrk->GetName(), - numev, latency, proctime, proccpu, bytesRead); - - if (gPerfStats != 0) { - gPerfStats->PacketEvent(wrk->GetOrdinal(), wrk->GetName(), "", numev, - latency, proctime, proccpu, bytesRead); - } - - if (fAssigned == fTotalEntries) { - // Send last timer message - HandleTimer(0); - return 0; - } - - if (fStop) { - // Send last timer message - HandleTimer(0); - return 0; - } - - PDB(kPacketizer,2) - Info("GetNextPacket", "worker-%s (%s): getting next files ... ", wrk->GetOrdinal(), - wrk->GetName()); - - // Get next file now - TObject *nextfile = 0; - - // Find iterator associated to the worker - TString wrkname = TUrl(wrk->GetName()).GetHostFQDN(); - TIterObj *io = dynamic_cast(fIters->FindObject(wrkname)); - if (io) { - // Get next file to process in the list of the worker - if (io->GetIter()) - nextfile = io->GetIter()->Next(); - } - - // If not found or all files already processed, check if a generic iterator - // has still some files to process - if (!nextfile && fProcNotAssigned) { - if ((io = dynamic_cast(fIters->FindObject("*")))) { - // Get next file to process in the list of the worker - if (io->GetIter()) - nextfile = io->GetIter()->Next(); - } - } - - // Return if nothing to process - if (!nextfile) return elem; - - // The file name: we support TObjString or TFileInfo - TString filename; - TObjString *os = 0; - TFileInfo *fi = 0; - if ((os = dynamic_cast(nextfile))) { - filename = os->GetName(); - } else { - if ((fi = dynamic_cast(nextfile))) - filename = fi->GetCurrentUrl()->GetUrl(); - } - // Nothing to process - if (filename.IsNull()) { - Warning("GetNextPacket", "found unsupported object of type '%s' in list: it must" - " be 'TObjString' or 'TFileInfo'", nextfile->ClassName()); - return elem; - } - // Prepare the packet - PDB(kPacketizer,2) - Info("GetNextPacket", "worker-%s: assigning: '%s' (remaining %lld files)", - wrk->GetOrdinal(), filename.Data(), (fTotalEntries - fAssigned)); - elem = new TDSetElement(filename, "", "", 0, 1); - elem->SetBit(TDSetElement::kEmpty); - - // Add the element, if required - if (fAddFileInfo && fi) { - elem->AddAssocObj(fi); - PDB(kPacketizer,2) fi->Print("L"); - } - - // Update the total counter - fAssigned += 1; - - return elem; -} - -//------------------------------------------------------------------------------ - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor - -TPacketizerFile::TSlaveStat::TSlaveStat(TSlave *slave, TList *input) - : fLastProcessed(0), - fSpeed(0), fTimeInstant(0), fCircLvl(5) -{ - // Initialize the circularity ntple for speed calculations - fCircNtp = new TNtupleD("Speed Circ Ntp", "Circular process info","tm:ev"); - TProof::GetParameter(input, "PROOF_TPacketizerFileCircularity", fCircLvl); - fCircLvl = (fCircLvl > 0) ? fCircLvl : 5; - fCircNtp->SetCircular(fCircLvl); - fSlave = slave; - fStatus = new TProofProgressStatus(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TPacketizerFile::TSlaveStat::~TSlaveStat() -{ - SafeDelete(fCircNtp); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update the circular ntple - -void TPacketizerFile::TSlaveStat::UpdatePerformance(Double_t time) -{ - Double_t ttot = time; - Double_t *ar = fCircNtp->GetArgs(); - Int_t ne = fCircNtp->GetEntries(); - if (ne <= 0) { - // First call: just fill one ref entry and return - fCircNtp->Fill(0., 0); - fSpeed = 0.; - return; - } - // Fill the entry - fCircNtp->GetEntry(ne-1); - ttot = ar[0] + time; - fCircNtp->Fill(ttot, GetEntriesProcessed()); - - // Calculate the speed - fCircNtp->GetEntry(0); - Double_t dtime = (ttot > ar[0]) ? ttot - ar[0] : ne+1 ; - Long64_t nevts = GetEntriesProcessed() - (Long64_t)ar[1]; - fSpeed = nevts / dtime; - PDB(kPacketizer,2) - Info("UpdatePerformance", "time:%f, dtime:%f, nevts:%lld, speed: %f", - time, dtime, nevts, fSpeed); - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update the status info to the 'st'. -/// return the difference (*st - *fStatus) - -TProofProgressStatus *TPacketizerFile::TSlaveStat::AddProcessed(TProofProgressStatus *st) -{ - if (st) { - // The entriesis not correct in 'st' - Long64_t lastEntries = st->GetEntries() - fStatus->GetEntries(); - // The last proc time should not be added - fStatus->SetLastProcTime(0.); - // Get the diff - TProofProgressStatus *diff = new TProofProgressStatus(*st - *fStatus); - *fStatus += *diff; - // Set the correct value - fStatus->SetLastEntries(lastEntries); - return diff; - } else { - Error("AddProcessed", "status arg undefined"); - return 0; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Printf info - -void TPacketizerFile::TIterObj::Print(Option_t *) const -{ - Printf("Iterator '%s' controls %d units", GetName(), - ((GetIter() && GetIter()->GetCollection()) ? GetIter()->GetCollection()->GetSize() - : -1)); -} diff --git a/proof/proofplayer/src/TPacketizerMulti.cxx b/proof/proofplayer/src/TPacketizerMulti.cxx deleted file mode 100644 index 00bff11db7e70..0000000000000 --- a/proof/proofplayer/src/TPacketizerMulti.cxx +++ /dev/null @@ -1,340 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G. Ganis Jan 2010 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * -*************************************************************************/ - -/** \class TPacketizerMulti -\ingroup proofkernel - -This class allows to do multiple runs in the same query; each run -can be a, for example, different dataset or the same dataset with -entry list. -The multiple packetizer contains a list of packetizers which are -processed in turn. -The bit TSelector::kNewRun is set in the TSelector object when a new -packetizer is used. - -*/ - - -#include "TPacketizerMulti.h" - -#include "TClass.h" -#include "TDSet.h" -#include "TError.h" -#include "TFileInfo.h" -#include "TList.h" -#include "TMap.h" -#include "TMethodCall.h" -#include "TProof.h" -#include "TProofDebug.h" - -ClassImp(TPacketizerMulti); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TPacketizerMulti::TPacketizerMulti(TDSet *dset, TList *wrks, - Long64_t first, Long64_t num, - TList *input, TProofProgressStatus *st) - : TVirtualPacketizer(input, st) -{ - PDB(kPacketizer,1) Info("TPacketizerMulti", - "enter (first %lld, num %lld)", first, num); - fValid = kFALSE; - fPacketizersIter = 0; - fCurrent = 0; - fAssignedPack = 0; - - // Check inputs - if (!dset || !wrks || !input || !st) { - Error("TPacketizerMulti", "invalid inputs: dset:%p wrks:%p input:%p st:%p", - dset, wrks, input, st); - return; - } - // Create the list - fPacketizers = new TList; - - // We do not want progress timers from the single packetizers - TNamed *progTimerFlag = new TNamed("PROOF_StartProgressTimer", "no"); - input->Add(progTimerFlag); - - fTotalEntries = 0; - TVirtualPacketizer *packetizer = 0; - // Simple or multi? - if (!(dset->TestBit(TDSet::kMultiDSet))) { - if ((packetizer = CreatePacketizer(dset, wrks, first, num, input, st))) { - fPacketizers->Add(packetizer); - fTotalEntries = packetizer->GetTotalEntries(); - } else { - Error("TPacketizerMulti", "problems initializing packetizer for single dataset"); - input->Remove(progTimerFlag); - delete progTimerFlag; - return; - } - } else { - // Iterate on the datasets - TIter nxds(dset->GetListOfElements()); - TDSet *ds = 0; - while ((ds = (TDSet *)nxds())) { - if ((packetizer = CreatePacketizer(ds, wrks, first, num, input, st))) { - fPacketizers->Add(packetizer); - fTotalEntries += packetizer->GetTotalEntries(); - } else { - Error("TPacketizerMulti", "problems initializing packetizer for dataset '%s'", ds->GetName()); - } - } - } - // Cleanup temporary additions to the input list - input->Remove(progTimerFlag); - delete progTimerFlag; - - // If no valid packetizer could be initialized we fail - if (fPacketizers->GetSize() <= 0) { - Error("TPacketizerMulti", "no valid packetizer could be initialized - aborting"); - SafeDelete(fPacketizers); - return; - } else { - Info("TPacketizerMulti", "%d packetizer(s) have been successfully initialized (%lld events in total)", - fPacketizers->GetSize(), fTotalEntries); - // To avoid problems with counters we must set the total entries in each packetizer - TIter nxp(fPacketizers); - while ((packetizer = (TVirtualPacketizer *) nxp())) - packetizer->SetTotalEntries(fTotalEntries); - } - - // Create the interator - fPacketizersIter = new TIter(fPacketizers); - - // Set the current the first - if (!(fCurrent = (TVirtualPacketizer *) fPacketizersIter->Next())) { - // Weird - Error("TPacketizerMulti", "could not point to the first valid packetizer"); - fPacketizers->SetOwner(kTRUE); - SafeDelete(fPacketizers); - SafeDelete(fPacketizersIter); - return; - } - - // Create map - fAssignedPack = new TMap; - - // Ok, everything went fine - fValid = kTRUE; - - PDB(kPacketizer,1) Info("TPacketizerMulti", "done"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TPacketizerMulti::~TPacketizerMulti() -{ - if (fPacketizers) { - fPacketizers->SetOwner(kTRUE); - SafeDelete(fPacketizers); - } - SafeDelete(fPacketizers); - fCurrent = 0; - if (fAssignedPack) { - fAssignedPack->SetOwner(kFALSE); - SafeDelete(fAssignedPack); - } - SafeDelete(fPacketizersIter); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next packet from the current packetizer. -/// If the current packetizer is done, move to next. -/// Retun null when all packetizers are done. - -TDSetElement *TPacketizerMulti::GetNextPacket(TSlave *wrk, TMessage *r) -{ - TDSetElement *elem = 0; - - // Must be valid - if (!fValid) return elem; - - // Point to the packetizer last used for thsi worker - TVirtualPacketizer *lastPacketizer = dynamic_cast(fAssignedPack->GetValue(wrk)); - if (lastPacketizer && lastPacketizer != fCurrent) { - PDB(kPacketizer,2) - Info("GetNextPacket", "%s: asking old packetizer %p ... ", wrk->GetOrdinal(), lastPacketizer); - if ((elem = lastPacketizer->GetNextPacket(wrk, r))) return elem; - if (fCurrent) { - // Transfer the status info - TVirtualSlaveStat *oldstat = dynamic_cast(lastPacketizer->GetSlaveStats()->GetValue(wrk)); - TVirtualSlaveStat *curstat = dynamic_cast(fCurrent->GetSlaveStats()->GetValue(wrk)); - if (oldstat && curstat) - *(curstat->GetProgressStatus()) += *(oldstat->GetProgressStatus()); - } - } - - // Need something to be processed - if (!fCurrent) { - HandleTimer(0); // Send last timer message - return elem; - } - - // Get the next packet from the current packetizer - PDB(kPacketizer,2) - Info("GetNextPacket", "%s: asking current packetizer %p ... ", wrk->GetOrdinal(), fCurrent); - if (!(elem = fCurrent->GetNextPacket(wrk, r))) { - // We need to transfer the status info if we change packetizer now - TMap *oldStats = (lastPacketizer && lastPacketizer == fCurrent) ? lastPacketizer->GetSlaveStats() : 0; - // If the packetizer is done, move to next - fCurrent = (TVirtualPacketizer *) fPacketizersIter->Next(); - if (fCurrent) { - // Transfer the status info - if (oldStats) { - TVirtualSlaveStat *oldstat = dynamic_cast(oldStats->GetValue(wrk)); - TVirtualSlaveStat *curstat = dynamic_cast(fCurrent->GetSlaveStats()->GetValue(wrk)); - if (oldstat && curstat) - *(curstat->GetProgressStatus()) += *(oldstat->GetProgressStatus()); - } - PDB(kPacketizer,2) - Info("GetNextPacket", "%s: asking new packetizer %p ... ", wrk->GetOrdinal(), fCurrent); - elem = fCurrent->GetNextPacket(wrk, r); - } - } - if (fCurrent) { - // Save the packetizer - TPair *pair = dynamic_cast(fAssignedPack->FindObject(wrk)); - if (pair) { - pair->SetValue(fCurrent); - } else { - fAssignedPack->Add(wrk, fCurrent); - } - PDB(kPacketizer,2) - Info("GetNextPacket", "assigned packetizer %p to %s (check: %p)", - fCurrent, wrk->GetOrdinal(), fAssignedPack->GetValue(wrk)); - } - - // Check the total number of entries - if (fProgressStatus->GetEntries() >= fTotalEntries) { - if (fProgressStatus->GetEntries() > fTotalEntries) - Error("GetNextPacket", "Processed too many entries!"); - HandleTimer(0); // Send last timer message - SafeDelete(fProgress); - } - - // Done - return elem; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create a packetizer for dataset 'dset' -/// Return null on failure. - -TVirtualPacketizer *TPacketizerMulti::CreatePacketizer(TDSet *dset, TList *wrks, - Long64_t first, Long64_t num, - TList *input, TProofProgressStatus *st) -{ - TVirtualPacketizer *packetizer = 0; - - // Check inputs - if (!dset || !wrks || !input || !st) { - Error("CreatePacketizer", "invalid inputs: dset:%p wrks:%p input:%p st:%p", - dset, wrks, input, st); - return packetizer; - } - - // This is for data-driven runs - if (dset->TestBit(TDSet::kEmpty)) { - Error("CreatePacketizer", "dataset is empty: protocol error?"); - return packetizer; - } - - TString packetizername; - TList *listOfMissingFiles = 0; - - TMethodCall callEnv; - TClass *cl; - - // Lookup - resolve the end-point urls to optmize the distribution. - // The lookup was previously called in the packetizer's constructor. - // A list for the missing files may already have been added to the - // output list; otherwise, if needed it will be created inside - if (!(listOfMissingFiles = (TList *) input->FindObject("MissingFiles"))) { - // Create it - listOfMissingFiles = new TList; - // and add it to the input list; it will be later moved to the output list - input->Add(listOfMissingFiles); - } - dset->Lookup(kTRUE, &listOfMissingFiles); - - if (!(dset->GetListOfElements()) || - !(dset->GetListOfElements()->GetSize())) { - Error("CreatePacketizer", "no files from the data set were found - skipping"); - return packetizer; - } - - if (TProof::GetParameter(input, "PROOF_Packetizer", packetizername) != 0) { - // Using standard packetizer TPacketizer - packetizername = "TPacketizer"; - } else { - Info("CreatePacketizer", "using alternate packetizer: %s", packetizername.Data()); - } - - // Get linked to the related class - cl = TClass::GetClass(packetizername); - if (cl == 0) { - Error("CreatePacketizer", "class '%s' not found", packetizername.Data()); - return packetizer; - } - - // Init the constructor - callEnv.InitWithPrototype(cl, cl->GetName(),"TDSet*,TList*,Long64_t,Long64_t,TList*,TProofProgressStatus*"); - if (!callEnv.IsValid()) { - Error("CreatePacketizer", "cannot find correct constructor for '%s'", cl->GetName()); - return packetizer; - } - callEnv.ResetParam(); - callEnv.SetParam((Longptr_t) dset); - callEnv.SetParam((Longptr_t) wrks); - callEnv.SetParam((Long64_t) first); - callEnv.SetParam((Long64_t) num); - callEnv.SetParam((Longptr_t) input); - callEnv.SetParam((Longptr_t) st); - - // We are going to test validity during the packetizer initialization - dset->SetBit(TDSet::kValidityChecked); - dset->ResetBit(TDSet::kSomeInvalid); - - // Get an instance of the packetizer - Longptr_t ret = 0; - callEnv.Execute(ret); - if ((packetizer = (TVirtualPacketizer *)ret) == 0) { - Error("CreatePacketizer", "cannot construct '%s'", cl->GetName()); - return packetizer; - } - - if (!packetizer->IsValid()) { - Error("CreatePacketizer", - "instantiated packetizer object '%s' is invalid", cl->GetName()); - SafeDelete(packetizer); - } - - // Add invalid elements to the list of missing elements - TDSetElement *elem = 0; - if (dset->TestBit(TDSet::kSomeInvalid)) { - TIter nxe(dset->GetListOfElements()); - while ((elem = (TDSetElement *)nxe())) { - if (!elem->GetValid()) { - listOfMissingFiles->Add(elem->GetFileInfo(dset->GetType())); - dset->Remove(elem, kFALSE); - } - } - // The invalid elements have been removed - dset->ResetBit(TDSet::kSomeInvalid); - } - - // Done - return packetizer; -} diff --git a/proof/proofplayer/src/TPacketizerUnit.cxx b/proof/proofplayer/src/TPacketizerUnit.cxx deleted file mode 100644 index 099faa83d1d2a..0000000000000 --- a/proof/proofplayer/src/TPacketizerUnit.cxx +++ /dev/null @@ -1,609 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Long Tran-Thanh 22/07/07 -// Revised: G. Ganis, May 2011 - -/************************************************************************* - * Copyright (C) 1995-2002, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TPacketizerUnit -\ingroup proofkernel - -This packetizer generates packets of generic units, representing the -number of times an operation cycle has to be repeated by the worker -node, e.g. the number of Monte carlo events to be generated. -Packets sizes are generated taking into account the performance of -worker nodes, based on the time needed to process previous packets, -with the goal of having all workers ending at the same time. - -*/ - - -#include "TPacketizerUnit.h" - -#include "Riostream.h" -#include "TDSet.h" -#include "TError.h" -#include "TEventList.h" -#include "TMap.h" -#include "TMessage.h" -#include "TMonitor.h" -#include "TNtupleD.h" -#include "TObject.h" -#include "TParameter.h" -#include "TPerfStats.h" -#include "TProofDebug.h" -#include "TProof.h" -#include "TProofPlayer.h" -#include "TProofServ.h" -#include "TSlave.h" -#include "TSocket.h" -#include "TStopwatch.h" -#include "TTimer.h" -#include "TUrl.h" -#include "TClass.h" -#include "TMath.h" - - -using namespace TMath; -// -// The following utility class manage the state of the -// work to be performed and the slaves involved in the process. -// -// The list of TSlaveStat(s) keep track of the work (being) done -// by each slave -// - -//------------------------------------------------------------------------------ - -class TPacketizerUnit::TSlaveStat : public TVirtualPacketizer::TVirtualSlaveStat { - -friend class TPacketizerUnit; - -private: - Long64_t fLastProcessed; // Number of processed entries of the last packet - Double_t fRate; // Estimated processing rate averaged over circularity - Double_t fTimeInstant; // Starting time of the current packet - TNtupleD *fCircNtp; // Keeps circular info for speed calculations - Long_t fCircLvl; // Circularity level - -public: - TSlaveStat(TSlave *sl, TList *input); - ~TSlaveStat() override; - -// void GetCurrentTime(); - - void UpdatePerformance(Double_t time); - TProofProgressStatus *AddProcessed(TProofProgressStatus *st) override; - -// ClassDefOverride(TPacketizerUnit::TSlaveStat, 0); -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor - -TPacketizerUnit::TSlaveStat::TSlaveStat(TSlave *slave, TList *input) - : fLastProcessed(0), - fRate(0), fTimeInstant(0), fCircLvl(5) -{ - // Initialize the circularity ntple for speed calculations - fCircNtp = new TNtupleD("Speed Circ Ntp", "Circular process info","tm:ev"); - fCircNtp->SetDirectory(0); - TProof::GetParameter(input, "PROOF_TPacketizerUnitCircularity", fCircLvl); - fCircLvl = (fCircLvl > 0) ? fCircLvl : 5; - fCircNtp->SetCircular(fCircLvl); - fSlave = slave; - fStatus = new TProofProgressStatus(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TPacketizerUnit::TSlaveStat::~TSlaveStat() -{ - SafeDelete(fCircNtp); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update the circular ntple - -void TPacketizerUnit::TSlaveStat::UpdatePerformance(Double_t time) -{ - Double_t ttot = time; - Double_t *ar = fCircNtp->GetArgs(); - Int_t ne = fCircNtp->GetEntries(); - if (ne <= 0) { - // First call: just fill one ref entry and return - fCircNtp->Fill(0., 0); - fRate = 0.; - return; - } - // Fill the entry - fCircNtp->GetEntry(ne-1); - ttot = ar[0] + time; - fCircNtp->Fill(ttot, GetEntriesProcessed()); - - // Calculate the speed - fCircNtp->GetEntry(0); - Double_t dtime = (ttot > ar[0]) ? ttot - ar[0] : ne+1 ; - Long64_t nevts = GetEntriesProcessed() - (Long64_t)ar[1]; - fRate = nevts / dtime; - PDB(kPacketizer,2) - Info("UpdatePerformance", "time:%f, dtime:%f, nevts:%lld, speed: %f", - time, dtime, nevts, fRate); - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update the status info to the 'st'. -/// return the difference (*st - *fStatus) - -TProofProgressStatus *TPacketizerUnit::TSlaveStat::AddProcessed(TProofProgressStatus *st) -{ - if (st) { - // The entriesis not correct in 'st' - Long64_t lastEntries = st->GetEntries() - fStatus->GetEntries(); - // The last proc time should not be added - fStatus->SetLastProcTime(0.); - // Get the diff - TProofProgressStatus *diff = new TProofProgressStatus(*st - *fStatus); - *fStatus += *diff; - // Set the correct value - fStatus->SetLastEntries(lastEntries); - return diff; - } else { - Error("AddProcessed", "status arg undefined"); - return 0; - } -} - -//------------------------------------------------------------------------------ - -ClassImp(TPacketizerUnit); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TPacketizerUnit::TPacketizerUnit(TList *slaves, Long64_t num, TList *input, - TProofProgressStatus *st) - : TVirtualPacketizer(input, st) -{ - PDB(kPacketizer,1) Info("TPacketizerUnit", "enter (num %lld)", num); - - // Init pointer members - fWrkStats = 0; - fPackets = 0; - fInput = input; - - fFixedNum = kFALSE; - Int_t fixednum = -1; - if (TProof::GetParameter(input, "PROOF_PacketizerFixedNum", fixednum) != 0 || fixednum <= 0) { - fFixedNum = kFALSE; - } - else { - Info("TPacketizerUnit", "forcing the same cycles on each worker"); - fFixedNum = kTRUE; - } - - fCalibFrac = 0.01; - if (TProof::GetParameter(input, "PROOF_PacketizerCalibFrac", fCalibFrac) != 0 || fCalibFrac <= 0) - fCalibFrac = 0.01; - PDB(kPacketizer,1) - Info("TPacketizerUnit", "size of the calibration packets: %.2f %% of average number per worker", fCalibFrac); - - fMaxPacketTime = 3.; - Double_t timeLimit = -1; - if (TProof::GetParameter(input, "PROOF_PacketizerTimeLimit", timeLimit) == 0) { - fMaxPacketTime = timeLimit; - Warning("TPacketizerUnit", "PROOF_PacketizerTimeLimit is deprecated: use PROOF_MaxPacketTime instead"); - } - PDB(kPacketizer,1) - Info("TPacketizerUnit", "time limit is %lf", fMaxPacketTime); - - // Different default for min packet time - fMinPacketTime = 1; - Double_t minPacketTime = 0; - if (TProof::GetParameter(input, "PROOF_MinPacketTime", minPacketTime) == 0) fMinPacketTime = minPacketTime; - TParameter *mpt = (TParameter *) fConfigParams->FindObject("PROOF_MinPacketTime"); - if (mpt) { - mpt->SetVal(fMinPacketTime); - } else { - fConfigParams->Add(new TParameter("PROOF_MinPacketTime", fMinPacketTime)); - } - - fProcessing = 0; - fAssigned = 0; - fPacketSeq = 0; - - fStopwatch = new TStopwatch(); - - fPackets = new TList; - fPackets->SetOwner(); - - fWrkStats = new TMap; - fWrkStats->SetOwner(kFALSE); - fWrkExcluded = 0; - - TSlave *slave; - TIter si(slaves); - while ((slave = (TSlave*) si.Next())) { - if (slave->GetParallel() > 0) { - fWrkStats->Add(slave, new TSlaveStat(slave, input)); - } else { - if (!fWrkExcluded) { - fWrkExcluded = new TList; - fWrkExcluded->SetOwner(kFALSE); - } - PDB(kPacketizer,2) - Info("TPacketizerUnit", "node '%s' has NO active worker: excluded from work distribution", slave->GetOrdinal()); - fWrkExcluded->Add(slave); - } - } - - fTotalEntries = 0; - fNumPerWorker = -1; - if (num > 0 && AssignWork(0,0,num) != 0) - Warning("TPacketizerUnit", "some problems assigning work"); - - // Save the config parameters in the dedicated list so that they will be saved - // in the outputlist and therefore in the relevant TQueryResult - fConfigParams->Add(new TParameter("PROOF_PacketizerCalibFrac", fCalibFrac)); - - fStopwatch->Start(); - PDB(kPacketizer,1) Info("TPacketizerUnit", "return"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Assign work to be done to this packetizer - -Int_t TPacketizerUnit::AssignWork(TDSet *, Long64_t, Long64_t num) -{ - if (num < 0) { - Error("AssignWork", "assigned a negative number (%lld) of cycles - protocol error?", num); - return -1; - } - - fTotalEntries += num; - PDB(kPacketizer,1) - Info("AssignWork", "assigned %lld additional cycles (new total: %lld)", num, fTotalEntries); - - // Update fixed number counter - if (fFixedNum && fWrkStats->GetSize() > 0) { - // Approximate number: the exact number is determined in GetNextPacket - fNumPerWorker = fTotalEntries / fWrkStats->GetSize(); - if (fNumPerWorker == 0) fNumPerWorker = 1; - } - - // Update/Save the config parameters in the dedicated list so that they will be saved - // in the outputlist and therefore in the relevant TQueryResult - TParameter *fn = - (TParameter *) fConfigParams->FindObject("PROOF_PacketizerFixedNum"); - if (fn) { - fn->SetVal(fNumPerWorker); - } else { - fConfigParams->Add(new TParameter("PROOF_PacketizerFixedNum", fNumPerWorker)); - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TPacketizerUnit::~TPacketizerUnit() -{ - if (fWrkStats) - fWrkStats->DeleteValues(); - SafeDelete(fWrkStats); - SafeDelete(fWrkExcluded); - SafeDelete(fPackets); - SafeDelete(fStopwatch); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get current time - -Double_t TPacketizerUnit::GetCurrentTime() -{ - Double_t retValue = fStopwatch->RealTime(); - fStopwatch->Continue(); - return retValue; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get Estimation of the current rate; just summing the current rates of -/// the active workers - -Float_t TPacketizerUnit::GetCurrentRate(Bool_t &all) -{ - all = kTRUE; - // Loop over the workers - Float_t currate = 0.; - if (fWrkStats && fWrkStats->GetSize() > 0) { - TIter nxw(fWrkStats); - TObject *key; - while ((key = nxw()) != 0) { - TSlaveStat *slstat = (TSlaveStat *) fWrkStats->GetValue(key); - if (slstat && slstat->GetProgressStatus() && slstat->GetEntriesProcessed() > 0) { - // Sum-up the current rates - currate += slstat->GetProgressStatus()->GetCurrentRate(); - } else { - all = kFALSE; - } - } - } - // Done - return currate; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next packet - -TDSetElement *TPacketizerUnit::GetNextPacket(TSlave *sl, TMessage *r) -{ - if (!fValid) - return 0; - - // Find slave - TSlaveStat *slstat = (TSlaveStat*) fWrkStats->GetValue(sl); - if (!slstat) { - Warning("GetNextPacket", "Received a packet request from an unknown slave: %s:%s", - sl->GetName(), sl->GetOrdinal()); - return 0; - } - - PDB(kPacketizer,2) - Info("GetNextPacket","worker-%s: fAssigned %lld\t", sl->GetOrdinal(), fAssigned); - - // Update stats & free old element - Double_t latency = 0., proctime = 0., proccpu = 0.; - Long64_t bytesRead = -1; - Long64_t totalEntries = -1; // used only to read an old message type - Long64_t totev = 0; - Long64_t numev = -1; - - TProofProgressStatus *status = 0; - if (sl->GetProtocol() > 18) { - (*r) >> latency; - (*r) >> status; - - // Calculate the progress made in the last packet - TProofProgressStatus *progress = 0; - if (status) { - // update the worker status - numev = status->GetEntries() - slstat->GetEntriesProcessed(); - progress = slstat->AddProcessed(status); - if (progress) { - // (*fProgressStatus) += *progress; - proctime = progress->GetProcTime(); - proccpu = progress->GetCPUTime(); - totev = status->GetEntries(); // for backward compatibility - bytesRead = progress->GetBytesRead(); - delete progress; - } - delete status; - } else - Error("GetNextPacket", "no status came in the kPROOF_GETPACKET message"); - } else { - - (*r) >> latency >> proctime >> proccpu; - - // only read new info if available - if (r->BufferSize() > r->Length()) (*r) >> bytesRead; - if (r->BufferSize() > r->Length()) (*r) >> totalEntries; - if (r->BufferSize() > r->Length()) (*r) >> totev; - - numev = totev - slstat->GetEntriesProcessed(); - slstat->GetProgressStatus()->IncEntries(numev); - slstat->GetProgressStatus()->SetLastUpdate(); - } - - fProgressStatus->IncEntries(numev); - fProgressStatus->SetLastUpdate(); - - fProcessing = 0; - - PDB(kPacketizer,2) - Info("GetNextPacket","worker-%s (%s): %lld %7.3lf %7.3lf %7.3lf %lld", - sl->GetOrdinal(), sl->GetName(), - numev, latency, proctime, proccpu, bytesRead); - - if (gPerfStats != 0) { - gPerfStats->PacketEvent(sl->GetOrdinal(), sl->GetName(), "", numev, - latency, proctime, proccpu, bytesRead); - } - - if (fNumPerWorker > 0 && slstat->GetEntriesProcessed() >= fNumPerWorker) { - PDB(kPacketizer,2) - Info("GetNextPacket","worker-%s (%s) is done (%lld cycles)", - sl->GetOrdinal(), sl->GetName(), slstat->GetEntriesProcessed()); - return 0; - } - - if (fAssigned == fTotalEntries) { - Bool_t done = kTRUE; - // If we are on a submaster, check if there is something else to do - if (gProofServ && gProofServ->IsMaster() && !gProofServ->IsTopMaster()) { - TDSetElement *nxe = gProofServ->GetNextPacket(); - if (nxe) { - if (AssignWork(0,0,nxe->GetNum()) == 0) { - if (fAssigned < fTotalEntries) done = kFALSE; - } else { - Error("GetNextPacket", "problems assigning additional work: stop"); - } - SafeDelete(nxe); - } - } - if (done) { - // Send last timer message - HandleTimer(0); - return 0; - } - } - - if (fStop) { - // Send last timer message - HandleTimer(0); - return 0; - } - - - Long64_t num; - - // Get the current time - Double_t cTime = GetCurrentTime(); - - if (slstat->fCircNtp->GetEntries() <= 0) { - // The calibration phase - Long64_t avg = fTotalEntries / fWrkStats->GetSize(); - num = (Long64_t) (fCalibFrac * avg); - if (num < 1) num = (avg >= 1) ? avg : 1; - PDB(kPacketizer,2) - Info("GetNextPacket", "calibration: total entries %lld, workers %d, frac: %.1f %%, raw num: %lld", - fTotalEntries, fWrkStats->GetSize(), fCalibFrac * 100., num); - - // Create a reference entry - slstat->UpdatePerformance(0.); - - } else { - - if (fNumPerWorker < 0) { - - // Schedule tasks for workers based on the currently estimated processing speeds - - // Update performances - // slstat->fStatus was updated before; - slstat->UpdatePerformance(proctime); - - // We need to estimate the total instantaneous rate: for the workers not having yet - // one we assume the average of those having a measurement - // The optimal number for worker j is - // - // n_j = r_j / Sum r_i * N_left - // - - Int_t nrm = 0; - Double_t sumRate = 0.; - TIter nxwrk(fWrkStats); - TSlaveStat *wrkStat = 0; - TSlave *tmpWrk = 0; - while ((tmpWrk = (TSlave *)nxwrk())) { - if ((wrkStat = dynamic_cast(fWrkStats->GetValue(tmpWrk)))) { - if (wrkStat->fRate > 0) { - nrm++; - sumRate += wrkStat->fRate; - } - PDB(kPacketizer,3) - Info("GetNextPacket", "%d: worker-%s: rate %lf /s (sum: %lf /s)", - nrm, tmpWrk->GetOrdinal(), wrkStat->fRate, sumRate); - } else { - Warning("GetNextPacket", "dynamic_cast failing on value for '%s (%s)'! Skipping", - tmpWrk->GetName(), tmpWrk->GetOrdinal()); - } - } - - // Check consistency - if (nrm <= 0) { - Error("GetNextPacket", "no worker has consistent information: stop processing!"); - return (TDSetElement *)0; - } - - Double_t avgRate = sumRate / nrm; - // Check if all workers had meaningful rate information - if (nrm < fWrkStats->GetSize()) { - // For some workers the measurement is missing: use the average - sumRate += (fWrkStats->GetSize() - nrm) * avgRate; - } - PDB(kPacketizer,2) - Info("GetNextPacket", "rate: avg: %lf /s/wrk - sum: %lf /s (measurements %d out of %d)", - avgRate, sumRate, nrm, fWrkStats->GetSize()); - - // Packet size for this worker - Double_t wrkRate = (slstat->fRate > 0.) ? slstat->fRate : avgRate ; - num = (Long64_t) ((fTotalEntries - fAssigned) * wrkRate / sumRate); - PDB(kPacketizer,2) - Info("GetNextPacket", "worker-%s (%s): raw packet size: %lld", sl->GetOrdinal(), sl->GetName(), num); - - // Apply time-per-packet limits - Double_t packTime = num / wrkRate; - if (fMaxPacketTime > 0. && packTime > fMaxPacketTime) { - num = (Long64_t) (fMaxPacketTime * wrkRate) ; - packTime = fMaxPacketTime; - PDB(kPacketizer,2) - Info("GetNextPacket", "worker-%s (%s): time-limited packet size: %lld (upper limit: %.2f secs)", - sl->GetOrdinal(), sl->GetName(), num, fMaxPacketTime); - } - if (fMinPacketTime > 0. && packTime < fMinPacketTime) { - num = (Long64_t) (fMinPacketTime * wrkRate); - PDB(kPacketizer,2) - Info("GetNextPacket", "worker-%s (%s): time-limited packet size: %lld (lower limit: %.2f secs)", - sl->GetOrdinal(), sl->GetName(), num, fMinPacketTime); - } - - } else { - // Fixed number of cycles per worker - num = fNumPerWorker - slstat->fLastProcessed; - if (num > 1 && slstat->fRate > 0 && num / slstat->fRate > fMaxPacketTime) { - num = (Long64_t) (slstat->fRate * fMaxPacketTime); - } - } - } - // Minimum packet size - num = (num > 1) ? num : 1; - fProcessing = (num < (fTotalEntries - fAssigned)) ? num - : (fTotalEntries - fAssigned); - - // Set the information of the current slave - slstat->fLastProcessed = fProcessing; - // Set the start time of the current packet - slstat->fTimeInstant = cTime; - - // Update the sequential number - fPacketSeq++; - TString sseq = TString::Format("p%lld", fPacketSeq); - - PDB(kPacketizer,2) - Info("GetNextPacket", "worker-%s: num %lld, processing %lld, remaining %lld",sl->GetOrdinal(), - num, fProcessing, (fTotalEntries - fAssigned - fProcessing)); - TDSetElement *elem = new TDSetElement(sseq, sseq, "", fAssigned, fProcessing); - elem->SetBit(TDSetElement::kEmpty); - - // Update the total counter - fAssigned += slstat->fLastProcessed; - - return elem; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Adds new workers. Returns the number of workers added, or -1 on failure. - -Int_t TPacketizerUnit::AddWorkers(TList *workers) -{ - if (!workers) { - Error("AddWorkers", "Null list of new workers!"); - return -1; - } - - Int_t curNumOfWrks = fWrkStats->GetEntries(); - - TSlave *sl; - TIter next(workers); - while (( sl = dynamic_cast(next()) )) - fWrkStats->Add(sl, new TSlaveStat(sl, fInput)); - - fNumPerWorker = -1; - if (fFixedNum && fWrkStats->GetSize() > 0) { - // Approximate number: the exact number is determined in GetNextPacket - fNumPerWorker = (fNumPerWorker * curNumOfWrks) / fWrkStats->GetSize(); - if (fNumPerWorker == 0) fNumPerWorker = 1; - } - - fConfigParams->Add(new TParameter("PROOF_PacketizerFixedNum", fNumPerWorker)); - - return fWrkStats->GetEntries(); -} diff --git a/proof/proofplayer/src/TPerfStats.cxx b/proof/proofplayer/src/TPerfStats.cxx deleted file mode 100644 index f6ed038b86ade..0000000000000 --- a/proof/proofplayer/src/TPerfStats.cxx +++ /dev/null @@ -1,794 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Kristjan Gulbrandsen 11/05/04 - -/************************************************************************* - * Copyright (C) 1995-2004, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TPerfStats -\ingroup proofkernel - -Provides the interface for the PROOF internal performance measurement -and event tracing. - -*/ - -#include "TPerfStats.h" - -#include "Riostream.h" -#include "TCollection.h" -#include "TEnv.h" -#include "TError.h" -#include "TFile.h" -#include "TH1.h" -#include "TH2.h" -#include "TDSet.h" -#include "TProofDebug.h" -#include "TProof.h" -#include "TProofServ.h" -#include "TSlave.h" -#include "TStatus.h" -#include "TTree.h" -#include "TSQLServer.h" -#include "TSQLResult.h" -#include "TParameter.h" -#include "TPluginManager.h" -#include "TROOT.h" -#include "TTimeStamp.h" -#include "TProofMonSender.h" - -ClassImp(TPerfEvent); -ClassImp(TPerfStats); - - -//------------------------------------------------------------------------------ - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TPerfEvent::TPerfEvent(TTimeStamp *offset) - : fEvtNode("-3"), fType(TVirtualPerfStats::kUnDefined), fSlave(), - fEventsProcessed(0), fBytesRead(0), fLen(0), fLatency(0.0), fProcTime(0.0), fCpuTime(0.0), - fIsStart(kFALSE), fIsOk(kFALSE) -{ - if (gProofServ != 0) { - fEvtNode = gProofServ->GetOrdinal(); - } else { - if (gProof && gProof->IsLite()) - fEvtNode = "0"; - else - fEvtNode = "-2"; // not on a PROOF server - } - - if (offset != 0) { - fTimeStamp = TTimeStamp(fTimeStamp.GetSec() - offset->GetSec(), - fTimeStamp.GetNanoSec() - offset->GetNanoSec()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Compare method. Must return -1 if this is smaller than obj, -/// 0 if objects are equal and 1 if this is larger than obj. - -Int_t TPerfEvent::Compare(const TObject *obj) const -{ - const TPerfEvent *pe = dynamic_cast(obj); - - if (!pe) { - Error("Compare", "input is not a TPerfEvent object"); - return 0; - } - - if (fTimeStamp < pe->fTimeStamp) { - return -1; - } else if (fTimeStamp == pe->fTimeStamp) { - return 0; - } else { - return 1; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Dump content of this instance - -void TPerfEvent::Print(Option_t *) const -{ - TString where; - if (fEvtNode == -2) { - where = "TPerfEvent: StandAlone "; - } else if ( fEvtNode == -1 ) { - where = "TPerfEvent: Master "; - } else { - where.Form("TPerfEvent: Worker %s ", fEvtNode.Data()); - } - Printf("%s %s %f", where.Data(), - TVirtualPerfStats::EventType(fType), double(fTimeStamp)); -} - -Long_t TPerfStats::fgVirtMemMax = -1; -Long_t TPerfStats::fgResMemMax = -1; - -//////////////////////////////////////////////////////////////////////////////// -/// Normal constructor. - -TPerfStats::TPerfStats(TList *input, TList *output) - : fTrace(0), fPerfEvent(0), fPacketsHist(0), fProcPcktHist(0), - fEventsHist(0), fNodeHist(0), fLatencyHist(0), - fProcTimeHist(0), fCpuTimeHist(0), fBytesRead(0), - fTotCpuTime(0.), fTotBytesRead(0), fTotEvents(0), fNumEvents(0), - fSlaves(0), fDoHist(kFALSE), - fDoTrace(kFALSE), fDoTraceRate(kFALSE), fDoSlaveTrace(kFALSE), fDoQuota(kFALSE), - fMonitorPerPacket(kFALSE), fMonSenders(3), - fDataSet("+++none+++"), fDataSetSize(-1), fOutput(output) -{ - TProof *proof = (gProofServ) ? gProofServ->GetProof() : gProof; - - // Master flag - Bool_t isMaster = ((proof && proof->TestBit(TProof::kIsMaster)) || - (gProofServ && gProofServ->IsMaster())) ? kTRUE : kFALSE; - Bool_t isEndMaster = ((gProofServ && gProofServ->IsEndMaster()) || - (proof && proof->IsLite())) ? kTRUE : kFALSE; - - TList *l = 0; - Bool_t deletel = kFALSE; - TParameter *dyns = (TParameter *) input->FindObject("PROOF_DynamicStartup"); - if (dyns) { - // When starring up dynamically the number of slots needs to be guessed from the - // maximum workers request. There is no way to change this later on. - Int_t nwrks = dyns->GetVal(); - if (nwrks > 0) { - l = new TList; - for (Int_t i = 0; i < nwrks; i++) { - TSlaveInfo *wi = new TSlaveInfo(TString::Format("0.%d", i)); - wi->SetStatus(TSlaveInfo::kActive); - l->Add(wi); - } - l->SetOwner(kTRUE); - deletel = kTRUE; - } - } - if (!l) l = proof ? proof->GetListOfSlaveInfos() : 0 ; - - TIter nextslaveinfo(l); - while (TSlaveInfo *si = dynamic_cast(nextslaveinfo())) - if (si->fStatus == TSlaveInfo::kActive) fSlaves++; - - PDB(kMonitoring,1) Info("TPerfStats", "Statistics for %d slave(s)", fSlaves); - - fDoHist = (input->FindObject("PROOF_StatsHist") != 0); - fDoTrace = (input->FindObject("PROOF_StatsTrace") != 0); - fDoTraceRate = (input->FindObject("PROOF_RateTrace") != 0); - fDoSlaveTrace = (input->FindObject("PROOF_SlaveStatsTrace") != 0); - PDB(kMonitoring,1) - Info("TPerfStats", "master:%d hist:%d,trace:%d,rate:%d,wrktrace:%d", - isMaster, fDoHist, fDoTrace, fDoTraceRate, fDoSlaveTrace); - - // Check per packet monitoring - Int_t perpacket = -1; - if (TProof::GetParameter(input, "PROOF_MonitorPerPacket", perpacket) != 0) { - // Check if there is a global monitor-per-packet setting - perpacket = gEnv->GetValue("Proof.MonitorPerPacket", 0); - } - fMonitorPerPacket = (perpacket == 1) ? kTRUE : kFALSE; - if (fMonitorPerPacket) - Info("TPerfStats", "sending full information after each packet"); - - // Extract the name of the dataset - TObject *o = 0; - TIter nxi(input); - while ((o = nxi())) - if (!strncmp(o->ClassName(), "TDSet", strlen("TDSet"))) break; - if (o) { - fDSet = (TDSet *) o; - fDataSetSize = fDSet->GetNumOfFiles(); - if (fDataSetSize > 0) { - fDataSet = ""; - TString grus = (gProofServ) ? TString::Format("/%s/%s/", gProofServ->GetGroup(), - gProofServ->GetUser()) : TString(""); - TString dss = fDSet->GetName(), ds; - Ssiz_t fd = 0, nq = kNPOS; - while (dss.Tokenize(ds, fd, "[,| ]")) { - if ((nq = ds.Index("?")) != kNPOS) ds.Remove(nq); - ds.ReplaceAll(grus, ""); - if (!fDataSet.IsNull()) fDataSet += ","; - fDataSet += ds; - } - } - } - - // Dataset string limited in length: get the authorized size - fDataSetLen = gEnv->GetValue("Proof.Monitor.DataSetLen", 512); - if (fDataSetLen != 512) - Info("TPerfStats", "dataset string length truncated to %d chars", fDataSetLen); - if (fDataSet.Length() > fDataSetLen) fDataSet.Resize(fDataSetLen); - // - PDB(kMonitoring,1) - Info("TPerfStats", "dataset: '%s', # files: %d", fDataSet.Data(), fDataSetSize); - - if ((isMaster && (fDoTrace || fDoTraceRate)) || (!isMaster && fDoSlaveTrace)) { - // Construct tree - gDirectory->RecursiveRemove(gDirectory->FindObject("PROOF_PerfStats")); - fTrace = new TTree("PROOF_PerfStats", "PROOF Statistics"); - fTrace->SetDirectory(0); - fTrace->Bronch("PerfEvents", "TPerfEvent", &fPerfEvent, 64000, 0); - output->Add(fTrace); - PDB(kMonitoring,1) - Info("TPerfStats", "tree '%s' added to the output list", fTrace->GetName()); - } - - if (fDoHist && isEndMaster) { - // Make Histograms - Double_t time_per_bin = 1e-3; // 10ms - Double_t min_time = 0; - Int_t ntime_bins = 1000; - - gDirectory->RecursiveRemove(gDirectory->FindObject("PROOF_PacketsHist")); - fPacketsHist = new TH1D("PROOF_PacketsHist", "Packets processed per Worker", - fSlaves, 0, fSlaves); - fPacketsHist->SetFillColor(kCyan); - fPacketsHist->SetDirectory(0); - fPacketsHist->SetMinimum(0); - output->Add(fPacketsHist); - PDB(kMonitoring,1) - Info("TPerfStats", "histo '%s' added to the output list", fPacketsHist->GetName()); - - gDirectory->RecursiveRemove(gDirectory->FindObject("PROOF_ProcPcktHist")); - fProcPcktHist = new TH1I("PROOF_ProcPcktHist", "Packets being processed per Worker", - fSlaves, 0, fSlaves); - fProcPcktHist->SetFillColor(kRed); - fProcPcktHist->SetDirectory(0); - fProcPcktHist->SetMinimum(0); - output->Add(fProcPcktHist); - PDB(kMonitoring,1) - Info("TPerfStats", "histo '%s' added to the output list", fProcPcktHist->GetName()); - - gDirectory->RecursiveRemove(gDirectory->FindObject("PROOF_EventsHist")); - fEventsHist = new TH1D("PROOF_EventsHist", "Events processed per Worker", - fSlaves, 0, fSlaves); - fEventsHist->SetFillColor(kGreen); - fEventsHist->SetDirectory(0); - fEventsHist->SetMinimum(0); - output->Add(fEventsHist); - PDB(kMonitoring,1) - Info("TPerfStats", "histo '%s' added to the output list", fEventsHist->GetName()); - - gDirectory->RecursiveRemove(gDirectory->FindObject("PROOF_NodeHist")); - fNodeHist = new TH1D("PROOF_NodeHist", "Slaves per Fileserving Node", - fSlaves, 0, fSlaves); - fNodeHist->SetDirectory(0); - fNodeHist->SetMinimum(0); - fNodeHist->SetCanExtend(TH1::kAllAxes); - output->Add(fNodeHist); - PDB(kMonitoring,1) - Info("TPerfStats", "histo '%s' added to the output list", fNodeHist->GetName()); - - gDirectory->RecursiveRemove(gDirectory->FindObject("PROOF_LatencyHist")); - fLatencyHist = new TH2D("PROOF_LatencyHist", "GetPacket Latency per Worker", - fSlaves, 0, fSlaves, - ntime_bins, min_time, time_per_bin); - fLatencyHist->SetDirectory(0); - fLatencyHist->SetMarkerStyle(4); - fLatencyHist->SetCanExtend(TH1::kAllAxes); - output->Add(fLatencyHist); - PDB(kMonitoring,1) - Info("TPerfStats", "histo '%s' added to the output list", fLatencyHist->GetName()); - - gDirectory->RecursiveRemove(gDirectory->FindObject("PROOF_ProcTimeHist")); - fProcTimeHist = new TH2D("PROOF_ProcTimeHist", "Packet Processing Time per Worker", - fSlaves, 0, fSlaves, - ntime_bins, min_time, time_per_bin); - fProcTimeHist->SetDirectory(0); - fProcTimeHist->SetMarkerStyle(4); - fProcTimeHist->SetCanExtend(TH1::kAllAxes); - output->Add(fProcTimeHist); - PDB(kMonitoring,1) - Info("TPerfStats", "histo '%s' added to the output list", fProcTimeHist->GetName()); - - gDirectory->RecursiveRemove(gDirectory->FindObject("PROOF_CpuTimeHist")); - fCpuTimeHist = new TH2D("PROOF_CpuTimeHist", "Packet CPU Time per Worker", - fSlaves, 0, fSlaves, - ntime_bins, min_time, time_per_bin); - fCpuTimeHist->SetDirectory(0); - fCpuTimeHist->SetMarkerStyle(4); - fCpuTimeHist->SetCanExtend(TH1::kAllAxes); - output->Add(fCpuTimeHist); - PDB(kMonitoring,1) - Info("TPerfStats", "histo '%s' added to the output list", fCpuTimeHist->GetName()); - - nextslaveinfo.Reset(); - Int_t slavebin=1; - while (TSlaveInfo *si = dynamic_cast(nextslaveinfo())) { - if (si->fStatus == TSlaveInfo::kActive) { - fPacketsHist->GetXaxis()->SetBinLabel(slavebin, si->GetOrdinal()); - fProcPcktHist->GetXaxis()->SetBinLabel(slavebin, si->GetOrdinal()); - fEventsHist->GetXaxis()->SetBinLabel(slavebin, si->GetOrdinal()); - fNodeHist->GetXaxis()->SetBinLabel(slavebin, si->GetOrdinal()); - fLatencyHist->GetXaxis()->SetBinLabel(slavebin, si->GetOrdinal()); - fProcTimeHist->GetXaxis()->SetBinLabel(slavebin, si->GetOrdinal()); - fCpuTimeHist->GetXaxis()->SetBinLabel(slavebin, si->GetOrdinal()); - slavebin++; - } - } - } - // Cleanup - if (deletel) delete(l); - - if (isMaster) { - - // Monitoring for query performances using monitoring system (e.g. Monalisa, SQL, ...) - // - // We support multiple specifications separated by ',' or '|' or '\' (the latter need - // top be escaped three times in the regular experession), e.g. - // ProofServ.Monitoring: Monalisa bla bla bla, - // +ProofServ.Monitoring: SQL blu blu blu - - TString mons = gEnv->GetValue("ProofServ.Monitoring", ""), mon; - Ssiz_t fmon = 0; - TProofMonSender *monSender = 0; - while (mons.Tokenize(mon, fmon, "[,|\\\\]")) { - if (mon != "") { - // Extract arguments (up to 9 'const char *') - TString a[10]; - Int_t from = 0; - TString tok, sendopts; - Int_t na = 0; - while (mon.Tokenize(tok, from, " ")) { - if (tok.BeginsWith("sendopts:")) { - tok.ReplaceAll("sendopts:", ""); - sendopts = tok; - } else { - a[na++] = tok; - } - } - na--; - // Get monitor object from the plugin manager - TPluginHandler *h = 0; - if ((h = gROOT->GetPluginManager()->FindHandler("TProofMonSender", a[0]))) { - if (h->LoadPlugin() != -1) { - monSender = - (TProofMonSender *) h->ExecPlugin(na, a[1].Data(), a[2].Data(), a[3].Data(), - a[4].Data(), a[5].Data(), a[6].Data(), - a[7].Data(), a[8].Data(), a[9].Data()); - if (monSender && monSender->TestBit(TObject::kInvalidObject)) SafeDelete(monSender); - if (monSender && monSender->SetSendOptions(sendopts) != 0) SafeDelete(monSender); - } - } - } - - if (monSender) { - fMonSenders.Add(monSender); - PDB(kMonitoring,1) - Info("TPerfStats", "created monitoring object: %s - # of active monitors: %d", - mon.Data(), fMonSenders.GetEntries()); - fDoQuota = kTRUE; - } - monSender = 0; - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TPerfStats::~TPerfStats() -{ - // Shutdown the monitor writers, if any - fMonSenders.SetOwner(kTRUE); - fMonSenders.Delete(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Simple event. - -void TPerfStats::SimpleEvent(EEventType type) -{ - if (type == kStop && fPacketsHist != 0) { - fPacketsHist->LabelsDeflate("X"); - fPacketsHist->LabelsOption("auv","X"); - } - - if (type == kStop && fDoQuota) - WriteQueryLog(); - - if (fTrace == 0) return; - - TPerfEvent pe(&fTzero); - pe.fType = type; - - fPerfEvent = &pe; - fTrace->SetBranchAddress("PerfEvents",&fPerfEvent); - fTrace->Fill(); - fPerfEvent = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Packet event. -/// See WriteQueryLog for the descripition of the structure sent for monitoring -/// when fMonitorPerPacket is kTRUE. - -void TPerfStats::PacketEvent(const char *slave, const char* slavename, const char* filename, - Long64_t eventsprocessed, Double_t latency, Double_t proctime, - Double_t cputime, Long64_t bytesRead) -{ - if (fDoTrace && fTrace != 0) { - TPerfEvent pe(&fTzero); - - pe.fType = kPacket; - pe.fSlaveName = slavename; - pe.fFileName = filename; - pe.fSlave = slave; - pe.fEventsProcessed = eventsprocessed; - pe.fBytesRead = bytesRead; - pe.fLatency = latency; - pe.fProcTime = proctime; - pe.fCpuTime = cputime; - - fPerfEvent = &pe; - fTrace->SetBranchAddress("PerfEvents",&fPerfEvent); - fTrace->Fill(); - fPerfEvent = 0; - } - - PDB(kMonitoring,1) - Info("PacketEvent","%s: fDoHist: %d, fPacketsHist: %p, eventsprocessed: %lld", - slave, fDoHist, fPacketsHist, eventsprocessed); - - if (fDoHist && fPacketsHist != 0) { - fPacketsHist->Fill(slave, 1); - fEventsHist->Fill(slave, eventsprocessed); - fLatencyHist->Fill(slave, latency, 1); - fProcTimeHist->Fill(slave, proctime, 1); - fCpuTimeHist->Fill(slave, cputime, 1); - } - - if (fDoQuota) { - fTotCpuTime += cputime; - fTotBytesRead += bytesRead; - fTotEvents += eventsprocessed; - } - - // Write to monitoring system, if requested - if (!fMonSenders.IsEmpty() && fMonitorPerPacket) { - TQueryResult *qr = (gProofServ && gProofServ->GetProof()) ? - gProofServ->GetProof()->GetQueryResult() : 0; - if (!gProofServ || !gProofServ->GetSessionTag() || !gProofServ->GetProof() || !qr) { - Error("PacketEvent", "some required object are undefined (%p %p %p %p)", - gProofServ, (gProofServ ? gProofServ->GetSessionTag() : 0), - (gProofServ ? gProofServ->GetProof() : 0), - ((gProofServ && gProofServ->GetProof()) ? qr : 0)); - return; - } - - TTimeStamp stop; - TString identifier; - identifier.Form("%s-q%d", gProofServ->GetSessionTag(), qr->GetSeqNum()); - - TList values; - values.SetOwner(); - values.Add(new TParameter("id", 0)); - values.Add(new TNamed("user", gProofServ->GetUser())); - values.Add(new TNamed("proofgroup", gProofServ->GetGroup())); - values.Add(new TNamed("begin", fTzero.AsString("s"))); - values.Add(new TNamed("end", stop.AsString("s"))); - values.Add(new TParameter("walltime", stop.GetSec()-fTzero.GetSec())); - values.Add(new TParameter("bytesread", fTotBytesRead)); - values.Add(new TParameter("events", fTotEvents)); - values.Add(new TParameter("totevents", fNumEvents)); - values.Add(new TParameter("workers", fSlaves)); - values.Add(new TNamed("querytag", identifier.Data())); - - // Memory usage on workers - TStatus *pst = (fOutput) ? (TStatus *) fOutput->FindObject("PROOF_Status") : 0; - // This most likely will be always NULL when sending from GetNextPacket ... - Long64_t vmxw = (pst) ? (Long64_t) pst->GetVirtMemMax() : -1; - Long64_t rmxw = (pst) ? (Long64_t) pst->GetResMemMax() : -1; - values.Add(new TParameter("vmemmxw", vmxw)); - values.Add(new TParameter("rmemmxw", rmxw)); - // Memory usage on master - values.Add(new TParameter("vmemmxm", (Long64_t) fgVirtMemMax)); - values.Add(new TParameter("rmemmxm", (Long64_t) fgResMemMax)); - // Dataset information - values.Add(new TNamed("dataset", fDataSet.Data())); - values.Add(new TParameter("numfiles", fDataSetSize)); - // Missing files - TList *mfls = (fOutput) ? (TList *) fOutput->FindObject("MissingFiles") : 0; - Int_t nmiss = (mfls && mfls->GetSize() > 0) ? mfls->GetSize() : 0; - values.Add(new TParameter("missfiles", nmiss)); - // Query status - Int_t est = (pst) ? pst->GetExitStatus() : -1; - values.Add(new TParameter("status", est)); - // Root version - TString rver = TString::Format("%s|%s", gROOT->GetVersion(), gROOT->GetGitCommit()); - values.Add(new TNamed("rootver", rver.Data())); - - for (Int_t i = 0; i < fMonSenders.GetEntries(); i++) { - TProofMonSender *m = (TProofMonSender *) fMonSenders[i]; - if (m) { - // Send query summary - if (m->SendSummary(&values, identifier) != 0) - Error("PacketEvent", "sending of summary info failed (%s)", m->GetName()); - } else { - Warning("PacketEvent", "undefined entry found in monitors array for id: %d", i); - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// File event. - -void TPerfStats::FileEvent(const char *slave, const char *slavename, const char *nodename, - const char *filename, Bool_t isStart) -{ - if (fDoTrace && fTrace != 0) { - TPerfEvent pe(&fTzero); - - pe.fType = kFile; - pe.fSlaveName = slavename; - pe.fNodeName = nodename; - pe.fFileName = filename; - pe.fSlave = slave; - pe.fIsStart = isStart; - - fPerfEvent = &pe; - fTrace->SetBranchAddress("PerfEvents",&fPerfEvent); - fTrace->Fill(); - fPerfEvent = 0; - } - - if (fDoHist && fPacketsHist != 0) { - fNodeHist->Fill(nodename, isStart ? 1 : -1); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Open file event. - -void TPerfStats::FileOpenEvent(TFile *file, const char *filename, Double_t start) -{ - if (fDoTrace && fTrace != 0) { - TPerfEvent pe(&fTzero); - - pe.fType = kFileOpen; - pe.fFileName = filename; - pe.fFileClass = file != 0 ? file->ClassName() : "none"; - pe.fProcTime = double(TTimeStamp())-start; - pe.fIsOk = (file != 0); - - fPerfEvent = &pe; - fTrace->SetBranchAddress("PerfEvents",&fPerfEvent); - fTrace->Fill(); - fPerfEvent = 0; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Read file event. - -void TPerfStats::FileReadEvent(TFile *file, Int_t len, Double_t start) -{ - if (fDoTrace && fTrace != 0) { - TPerfEvent pe(&fTzero); - - pe.fType = kFileRead; - pe.fFileName = file->GetName(); - pe.fFileClass = file->ClassName(); - pe.fLen = len; - pe.fProcTime = double(TTimeStamp())-start; - - fPerfEvent = &pe; - fTrace->SetBranchAddress("PerfEvents",&fPerfEvent); - fTrace->Fill(); - fPerfEvent = 0; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Record TTree file unzip event. -/// start is the TimeStamp before unzip -/// pos is where in the file the compressed buffer came from -/// complen is the length of the compressed buffer -/// objlen is the length of the de-compressed buffer - -void TPerfStats::UnzipEvent(TObject * /* tree */, Long64_t /* pos */, - Double_t /* start */, Int_t /* complen */, - Int_t /* objlen */) -{ - // Do nothing for now. -} - -//////////////////////////////////////////////////////////////////////////////// -/// Rate event. - -void TPerfStats::RateEvent(Double_t proctime, Double_t deltatime, - Long64_t eventsprocessed, Long64_t bytesRead) -{ - if ((fDoTrace || fDoTraceRate) && fTrace != 0) { - TPerfEvent pe(&fTzero); - - pe.fType = kRate; - pe.fEventsProcessed = eventsprocessed; - pe.fBytesRead = bytesRead; - pe.fProcTime = proctime; - pe.fLatency = deltatime; - - fPerfEvent = &pe; - fTrace->SetBranchAddress("PerfEvents",&fPerfEvent); - fTrace->Fill(); - fPerfEvent = 0; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set number of bytes read. - -void TPerfStats::SetBytesRead(Long64_t num) -{ - fBytesRead = num; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get number of bytes read. - -Long64_t TPerfStats::GetBytesRead() const -{ - return fBytesRead; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send to the connected monitoring servers information related to this query. -/// The information is of three types: 'summary', 'dataset' and 'files'. -/// Actual 'table' formatting is done by the relevant sender, implementation of -/// TProofMonSender, where the details are given. - -void TPerfStats::WriteQueryLog() -{ - TTimeStamp stop; - - // Write to monitoring system - if (!fMonSenders.IsEmpty()) { - TQueryResult *qr = (gProofServ && gProofServ->GetProof()) ? - gProofServ->GetProof()->GetQueryResult() : 0; - if (!gProofServ || !gProofServ->GetSessionTag() || !gProofServ->GetProof() || !qr) { - Error("WriteQueryLog", "some required object are undefined (%p %p %p %p)", - gProofServ, (gProofServ ? gProofServ->GetSessionTag() : 0), - (gProofServ ? gProofServ->GetProof() : 0), - ((gProofServ && gProofServ->GetProof()) ? qr : 0)); - return; - } - - TString identifier; - identifier.Form("%s-q%d", gProofServ->GetSessionTag(), qr->GetSeqNum()); - - TList values; - values.SetOwner(); - values.Add(new TParameter("id", 0)); - values.Add(new TNamed("user", gProofServ->GetUser())); - values.Add(new TNamed("proofgroup", gProofServ->GetGroup())); - values.Add(new TNamed("begin", fTzero.AsString("s"))); - values.Add(new TNamed("end", stop.AsString("s"))); - values.Add(new TParameter("walltime", stop.GetSec()-fTzero.GetSec())); - values.Add(new TParameter("cputime", fTotCpuTime)); - values.Add(new TParameter("bytesread", fTotBytesRead)); - values.Add(new TParameter("events", fTotEvents)); - values.Add(new TParameter("totevents", fTotEvents)); - values.Add(new TParameter("workers", fSlaves)); - values.Add(new TNamed("querytag", identifier.Data())); - - TList *mfls = (fOutput) ? (TList *) fOutput->FindObject("MissingFiles") : 0; - // Memory usage on workers - TStatus *pst = (fOutput) ? (TStatus *) fOutput->FindObject("PROOF_Status") : 0; - Long64_t vmxw = (pst) ? (Long64_t) pst->GetVirtMemMax() : -1; - Long64_t rmxw = (pst) ? (Long64_t) pst->GetResMemMax() : -1; - values.Add(new TParameter("vmemmxw", vmxw)); - values.Add(new TParameter("rmemmxw", rmxw)); - // Memory usage on master - values.Add(new TParameter("vmemmxm", (Long64_t) fgVirtMemMax)); - values.Add(new TParameter("rmemmxm", (Long64_t) fgResMemMax)); - // Dataset information - values.Add(new TNamed("dataset", fDataSet.Data())); - values.Add(new TParameter("numfiles", fDataSetSize)); - // Missing files - Int_t nmiss = (mfls && mfls->GetSize() > 0) ? mfls->GetSize() : 0; - values.Add(new TParameter("missfiles", nmiss)); - // Query status - Int_t est = (pst) ? pst->GetExitStatus() : -1; - values.Add(new TParameter("status", est)); - // Root version - TString rver = TString::Format("%s|%s", gROOT->GetVersion(), gROOT->GetGitCommit()); - values.Add(new TNamed("rootver", rver.Data())); - - for (Int_t i = 0; i < fMonSenders.GetEntries(); i++) { - TProofMonSender *m = (TProofMonSender *) fMonSenders[i]; - if (m) { - // Send query summary - if (m->SendSummary(&values, identifier) != 0) - Error("WriteQueryLog", "sending of summary info failed (%s)", m->GetName()); - // Send dataset information - if (m->SendDataSetInfo(fDSet, mfls, fTzero.AsString("s"), identifier) != 0) - Error("WriteQueryLog", "sending of dataset info failed (%s)", m->GetName()); - // Send file information - if (m->SendFileInfo(fDSet, mfls, fTzero.AsString("s"), identifier) != 0) - Error("WriteQueryLog", "sending of files info failed (%s)", m->GetName()); - } else { - Warning("WriteQueryLog", "undefined entry found in monitors array for id: %d", i); - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Setup the PROOF input list with requested statistics and tracing options. - -void TPerfStats::Setup(TList *input) -{ - const Int_t ntags=3; - const char *tags[ntags] = {"StatsHist", "StatsTrace", "SlaveStatsTrace"}; - - TString varname, parname; - for (Int_t i=0; iFindObject(parname)) - if (gEnv->GetValue(varname, 0)) input->Add(new TNamed(parname.Data(),"")); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Initialize PROOF statistics run. - -void TPerfStats::Start(TList *input, TList *output) -{ - if (gPerfStats) - delete gPerfStats; - fgVirtMemMax = -1; - fgResMemMax = -1; - TPerfStats::SetMemValues(); - - gPerfStats = new TPerfStats(input, output); - if (gPerfStats && !gPerfStats->TestBit(TObject::kInvalidObject)) { - // This measures the time taken by the constructor: not negligeable ... - gPerfStats->SimpleEvent(TVirtualPerfStats::kStart); - } else { - SafeDelete(gPerfStats); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Terminate the PROOF statistics run. - -void TPerfStats::Stop() -{ - if (!gPerfStats) return; - - TPerfStats::SetMemValues(); - gPerfStats->SimpleEvent(TVirtualPerfStats::kStop); - - delete gPerfStats; - gPerfStats = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Record memory usage - -void TPerfStats::SetMemValues() -{ - ProcInfo_t pi; - if (!gSystem->GetProcInfo(&pi)){ - if (pi.fMemVirtual > fgVirtMemMax) fgVirtMemMax = pi.fMemVirtual; - if (pi.fMemResident > fgResMemMax) fgResMemMax = pi.fMemResident; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get memory usage - -void TPerfStats::GetMemValues(Long_t &vmax, Long_t &rmax) -{ - vmax = fgVirtMemMax; - rmax = fgResMemMax; -} diff --git a/proof/proofplayer/src/TProofDraw.cxx b/proof/proofplayer/src/TProofDraw.cxx deleted file mode 100644 index 99094c3b825cc..0000000000000 --- a/proof/proofplayer/src/TProofDraw.cxx +++ /dev/null @@ -1,2222 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn, Marek Biskup 24/09/2003 - -/************************************************************************* - * Copyright (C) 1995-2003, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofDraw -\ingroup proofkernel - -Implement Tree drawing using PROOF - -*/ - - -#include "TProofDraw.h" -#include "TAttFill.h" -#include "TAttLine.h" -#include "TAttMarker.h" -#include "TCanvas.h" -#include "TClass.h" -#include "TError.h" -#include "TH1F.h" -#include "TH2F.h" -#include "TH3F.h" -#include "TProof.h" -#include "TProofDebug.h" -#include "TStatus.h" -#include "TTreeDrawArgsParser.h" -#include "TTreeFormula.h" -#include "TTreeFormulaManager.h" -#include "TTree.h" -#include "TEventList.h" -#include "TEntryList.h" -#include "TProfile.h" -#include "TProfile2D.h" -#include "TEnv.h" -#include "TNamed.h" -#include "TGraph.h" -#include "TPolyMarker3D.h" -#include "TVirtualPad.h" -#include "THLimitsFinder.h" -#include "TView.h" -#include "TStyle.h" -#include "TDirectory.h" -#include "TROOT.h" - -#include -using std::vector; - - -// Simple call to draw a canvas on the fly from applications loading -// this plug-in dynamically -extern "C" { - Int_t DrawCanvas(TObject *obj) - { - // Draw the object if deriving from a canvas - - if (TCanvas* c = dynamic_cast (obj)) { - c->Draw(); - return 0; - } - // Not a TCanvas - return 1; - } -} - -// Simple call to parse arguments on the fly from applications loading -// this plug-in dynamically -extern "C" { - Int_t GetDrawArgs(const char *var, const char *sel, Option_t *opt, - TString &selector, TString &objname) - { - // Parse arguments with the help of TTreeDrawArgsParser - - TTreeDrawArgsParser info; - info.Parse(var, sel, opt); - selector = info.GetProofSelectorName(); - objname = info.GetObjectName(); - - // Done - return 0; - } -} - -// Simple call to create destroy a 'named' canvas -extern "C" { - void FeedBackCanvas(const char *name, Bool_t create) - { - // Create or destroy canvas 'name' - - if (create) { - new TCanvas(name, "FeedBack", 800,30,700,500); - } else { - TCanvas *c = (gROOT->GetListOfCanvases()) ? - (TCanvas *) gROOT->GetListOfCanvases()->FindObject(name) : 0; - if (c) delete c; - } - // Done - return; - } -} - -ClassImp(TProofDraw); - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor. - -TProofDraw::TProofDraw() - : fStatus(0), fManager(0), fTree(0) -{ - fVar[0] = 0; - fVar[1] = 0; - fVar[2] = 0; - fVar[3] = 0; - fManager = 0; - fMultiplicity = 0; - fSelect = 0; - fObjEval = kFALSE; - fDimension = 0; - fWeight = 1.; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TProofDraw::~TProofDraw() -{ - ClearFormula(); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Init the tree. - -void TProofDraw::Init(TTree *tree) -{ - PDB(kDraw,1) Info("Init","Enter tree = %p", tree); - fTree = tree; - CompileVariables(); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Called when a new tree is loaded. - -Bool_t TProofDraw::Notify() -{ - PDB(kDraw,1) Info("Notify","Enter"); - if (fStatus == 0) { - if (!fOutput || (fOutput && - !(fStatus = dynamic_cast(fOutput->FindObject("PROOF_Status"))))) - return kFALSE; - } - if (!fStatus->IsOk()) return kFALSE; - if (!fManager) { - fAbort = TSelector::kAbortProcess; - return kFALSE; - } - fManager->UpdateFormulaLeaves(); - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Executed by the client before processing. - -void TProofDraw::Begin(TTree *tree) -{ - PDB(kDraw,1) Info("Begin","Enter tree = %p", tree); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - if (fTreeDrawArgsParser.GetObjectName() == "") - fTreeDrawArgsParser.SetObjectName("htemp"); - } - - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); - fTree = 0; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Executed by each slave before processing. - -void TProofDraw::SlaveBegin(TTree* /*tree*/) -{ - // Get the weight - TProofDraw::FillWeight(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get weight from input list, if any - -void TProofDraw::FillWeight() -{ - Double_t ww; - if (TProof::GetParameter(fInput, "PROOF_ChainWeight", ww) == 0) - fWeight = ww; - PDB(kDraw,1) Info("FillWeight","fWeight= %f", fWeight); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Processes a single variable from an entry. - -Bool_t TProofDraw::ProcessSingle(Long64_t entry, Int_t i) -{ - Double_t w; - Double_t v[4]; //[TTreeDrawArgsParser::fgMaxDimension]; - - if (fSelect) - w = fWeight * fSelect->EvalInstance(i); - else - w = fWeight; - - PDB(kDraw,3) Info("ProcessSingle","w[%d] = %f", i, w); - - if (w != 0.0) { - R__ASSERT(fDimension <= TTreeDrawArgsParser::GetMaxDimension()); - for (int j = 0; j < fDimension; j++) - v[j] = fVar[j]->EvalInstance(i); - if (fDimension >= 1) - PDB(kDraw,4) Info("Process","v[0] = %f", v[0]); - DoFill(entry, w, v); - } - return kTRUE; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Executed for each entry. - -Bool_t TProofDraw::Process(Long64_t entry) -{ - PDB(kDraw,3) Info("Process", "enter entry = %lld", entry); - - fTree->LoadTree(entry); - Int_t ndata = fManager->GetNdata(); - - PDB(kDraw,3) Info("Process","ndata = %d", ndata); - - for (Int_t i=0;i(fOutput->FindObject("PROOF_Status")); - if (fStatus == 0) { - // did not run selector, error messages were already printed - return; - } - } - - if (!fStatus->IsOk()) { - fStatus->Print(); - return; - } -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Delete internal buffers. - -void TProofDraw::ClearFormula() -{ - ResetBit(kWarn); - for (Int_t i = 0; i < 4; i++) - SafeDelete(fVar[i]); - SafeDelete(fSelect); - fManager = 0; // This is intentional. The manager is deleted when the last formula it manages - // is deleted. This is unusual but was usefull for backward compatibility. - fMultiplicity = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Move to a canvas named `_canvas`; create the canvas if not existing. -/// Used to avoid screwing up existing plots when non default names are used -/// for the final objects - -void TProofDraw::SetCanvas(const char *objname) -{ - TString name = objname; - if (!gPad) { - gROOT->MakeDefCanvas(); - gPad->SetName(name); - PDB(kDraw,2) Info("SetCanvas", "created canvas %s", name.Data()); - } else { - PDB(kDraw,2) - Info("SetCanvas", "using canvas %s", gPad->GetName()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the drawing attributes from the input list - -void TProofDraw::SetDrawAtt(TObject *o) -{ - Int_t att = -1; - PDB(kDraw,2) Info("SetDrawAtt", "setting attributes for %s", o->GetName()); - - // Line Attributes - TAttLine *al = dynamic_cast (o); - if (al) { - // Line color - if (TProof::GetParameter(fInput, "PROOF_LineColor", att) == 0) - al->SetLineColor((Color_t)att); - // Line style - if (TProof::GetParameter(fInput, "PROOF_LineStyle", att) == 0) - al->SetLineStyle((Style_t)att); - // Line color - if (TProof::GetParameter(fInput, "PROOF_LineWidth", att) == 0) - al->SetLineWidth((Width_t)att); - PDB(kDraw,2) Info("SetDrawAtt", "line: c:%d, s:%d, wd:%d", - al->GetLineColor(), al->GetLineStyle(), al->GetLineWidth()); - } - - // Marker Attributes - TAttMarker *am = dynamic_cast (o); - if (am) { - // Marker color - if (TProof::GetParameter(fInput, "PROOF_MarkerColor", att) == 0) - am->SetMarkerColor((Color_t)att); - // Marker size - if (TProof::GetParameter(fInput, "PROOF_MarkerSize", att) == 0) { - Info("SetDrawAtt", "att: %d", att); - Float_t msz = (Float_t)att / 1000.; - am->SetMarkerSize((Size_t)msz); - } - // Marker style - if (TProof::GetParameter(fInput, "PROOF_MarkerStyle", att) == 0) - am->SetMarkerStyle((Style_t)att); - PDB(kDraw,2) Info("SetDrawAtt", "marker: c:%d, s:%d, sz:%f", - am->GetMarkerColor(), am->GetMarkerStyle(), am->GetMarkerSize()); - } - - // Area Fill Attributes - TAttFill *af = dynamic_cast (o); - if (af) { - // Area fill color - if (TProof::GetParameter(fInput, "PROOF_FillColor", att) == 0) - af->SetFillColor((Color_t)att); - // Area fill style - if (TProof::GetParameter(fInput, "PROOF_FillStyle", att) == 0) - af->SetFillStyle((Style_t)att); - PDB(kDraw,2) Info("SetDrawAtt", "area: c:%d, s:%d", - af->GetFillColor(), af->GetFillStyle()); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Sets the error status. - -void TProofDraw::SetError(const char *sub, const char *mesg) -{ - if (fStatus == 0) { - if (!(fStatus = dynamic_cast(fOutput->FindObject("PROOF_Status")))) - return; - } - - TString m; - if (IsA()) - m.Form("%s::%s: %s", IsA()->GetName(), sub, mesg); - else - m.Form("TProofDraw::%s: %s", sub, mesg); - fStatus->Add(m); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Compiles each variable from fTreeDrawArgsParser for the tree fTree. -/// Return kFALSE if any of the variable is not compilable. - -Bool_t TProofDraw::CompileVariables() -{ - // Set aliases, if any - TNamed *nms = (TNamed *) fInput->FindObject("PROOF_ListOfAliases"); - if (nms) { - TString names = nms->GetTitle(), n, na; - Ssiz_t from = 0; - while(names.Tokenize(n, from, ",")) { - if (!n.IsNull()) { - na.Form("alias:%s", n.Data()); - TNamed *nm = (TNamed *) fInput->FindObject(na); - if (na) fTree->SetAlias(n.Data(), nm->GetTitle()); - } - } - } - PDB(kDraw,2) - if (fTree->GetListOfAliases()) fTree->GetListOfAliases()->Print(); - - fDimension = fTreeDrawArgsParser.GetDimension(); - fMultiplicity = 0; - fObjEval = kFALSE; - if (strlen(fTreeDrawArgsParser.GetSelection())) { - fSelect = new TTreeFormula("Selection", fTreeDrawArgsParser.GetSelection(), fTree); - fSelect->SetQuickLoad(kTRUE); - if (!fSelect->GetNdim()) {delete fSelect; fSelect = 0; return kFALSE; } - } - - fManager = new TTreeFormulaManager(); - if (fSelect) fManager->Add(fSelect); - fTree->ResetBit(TTree::kForceRead); - - for (int i = 0; i < fDimension; i++) { - fVar[i] = new TTreeFormula(Form("Var%d", i),fTreeDrawArgsParser.GetVarExp(i),fTree); - fVar[i]->SetQuickLoad(kTRUE); - if (!fVar[i]->GetNdim()) { - ClearFormula(); - Error("CompileVariables", "Error compiling expression"); - SetError("CompileVariables", "Error compiling variables"); - - return kFALSE; - } - fManager->Add(fVar[i]); - } - - fManager->Sync(); - if (fManager->GetMultiplicity()==-1) fTree->SetBit(TTree::kForceRead); - if (fManager->GetMultiplicity()>=1) fMultiplicity = fManager->GetMultiplicity(); - - return kTRUE; -#if 0 - // Commenting out to silence Coverity: - // but why was this made inactive? - if (fDimension==1) { - TClass *cl = fVar[0]->EvalClass(); - if (cl) { - fObjEval = kTRUE; - } - } - return kTRUE; -#endif -} - - -ClassImp(TProofDrawHist); - - -//////////////////////////////////////////////////////////////////////////////// -/// Initialization for 1D Histogram. - -void TProofDrawHist::Begin1D(TTree *) -{ - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 1); - TObject* orig = fTreeDrawArgsParser.GetOriginal(); - TH1* hold; - if (fTreeDrawArgsParser.GetNoParameters() == 0 && (hold = dynamic_cast (orig))) { - hold->Reset(); - fInput->Add(hold); - } else { - delete orig; - DefVar1D(); - } -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Initialization for 2D histogram. - -void TProofDrawHist::Begin2D(TTree *) -{ - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 2); - TObject* orig = fTreeDrawArgsParser.GetOriginal(); - TH2* hold; - if (fTreeDrawArgsParser.GetNoParameters() == 0 && (hold = dynamic_cast (orig))) { - hold->Reset(); - fInput->Add(hold); - } else { - delete orig; - DefVar2D(); - } -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Initialization for 3D histogram. - -void TProofDrawHist::Begin3D(TTree *) -{ - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 3); - TObject* orig = fTreeDrawArgsParser.GetOriginal(); - TH3* hold; - if ((hold = dynamic_cast (orig)) && fTreeDrawArgsParser.GetNoParameters() == 0) { - hold->Reset(); - fInput->Add(hold); - } else { - delete orig; - DefVar3D(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Begin(). - -void TProofDrawHist::Begin(TTree *tree) -{ - PDB(kDraw,1) Info("Begin","Enter tree = %p", tree); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - if (fTreeDrawArgsParser.GetObjectName() == "") - fTreeDrawArgsParser.SetObjectName("htemp"); - - switch (fTreeDrawArgsParser.GetDimension()) { - case 1: - Begin1D(tree); - break; - case 2: - Begin2D(tree); - break; - case 3: - Begin3D(tree); - break; - default: - Error("Begin", "Wrong dimension"); - break; - } - } - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); - fTree = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Define vars for 1D Histogram. - -void TProofDrawHist::DefVar1D() -{ - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 1); - - fTreeDrawArgsParser.SetOriginal(0); - TString exp = fTreeDrawArgsParser.GetVarExp(); - exp += ">>"; - double binsx, minx, maxx; - if (fTreeDrawArgsParser.IsSpecified(0)) - gEnv->SetValue("Hist.Binning.1D.x", fTreeDrawArgsParser.GetParameter(0)); - binsx = gEnv->GetValue("Hist.Binning.1D.x",100); - minx = fTreeDrawArgsParser.GetIfSpecified(1, 0); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, 0); - exp += fTreeDrawArgsParser.GetObjectName(); - exp += '('; - exp += binsx; - exp += ','; - exp += minx; - exp += ','; - exp += maxx; - exp += ')'; - - fInitialExp = exp; - TNamed *n = dynamic_cast (fInput->FindObject("varexp")); - if (n) - n->SetTitle(exp); - else - Error("DefVar1D", "Cannot find varexp on the fInput"); - if (fTreeDrawArgsParser.GetNoParameters() != 3) - fInput->Add(new TNamed("PROOF_OPTIONS", "rebin")); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Define variables for 2D histogram. - -void TProofDrawHist::DefVar2D() -{ - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 2); - - fTreeDrawArgsParser.SetOriginal(0); - TString exp = fTreeDrawArgsParser.GetVarExp(); - exp += ">>"; - double binsx, minx, maxx; - double binsy, miny, maxy; - if (fTreeDrawArgsParser.IsSpecified(0)) - gEnv->SetValue("Hist.Binning.2D.x", fTreeDrawArgsParser.GetParameter(0)); - if (fTreeDrawArgsParser.IsSpecified(3)) - gEnv->SetValue("Hist.Binning.2D.y", fTreeDrawArgsParser.GetParameter(3)); - binsx = gEnv->GetValue("Hist.Binning.2D.x",100); - minx = fTreeDrawArgsParser.GetIfSpecified(1, 0); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, 0); - binsy = gEnv->GetValue("Hist.Binning.2D.y",100); - miny = fTreeDrawArgsParser.GetIfSpecified(4, 0); - maxy = fTreeDrawArgsParser.GetIfSpecified(5, 0); - exp += fTreeDrawArgsParser.GetObjectName(); - exp += '('; - exp += binsx; - exp += ','; - exp += minx; - exp += ','; - exp += maxx; - exp += ','; - exp += binsy; - exp += ','; - exp += miny; - exp += ','; - exp += maxy; - exp += ')'; - fInitialExp = exp; - TNamed *n = dynamic_cast (fInput->FindObject("varexp")); - if (n) - n->SetTitle(exp); - else - Error("DefVar2D", "Cannot find varexp on the fInput"); - if (fTreeDrawArgsParser.GetNoParameters() != 6) - fInput->Add(new TNamed("PROOF_OPTIONS", "rebin")); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Define variables for 3D histogram. - -void TProofDrawHist::DefVar3D() -{ - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 3); - - fTreeDrawArgsParser.SetOriginal(0); - TString exp = fTreeDrawArgsParser.GetVarExp(); - exp += ">>"; - double binsx, minx, maxx; - double binsy, miny, maxy; - double binsz, minz, maxz; - if (fTreeDrawArgsParser.IsSpecified(0)) - gEnv->SetValue("Hist.Binning.3D.x", fTreeDrawArgsParser.GetParameter(0)); - if (fTreeDrawArgsParser.IsSpecified(3)) - gEnv->SetValue("Hist.Binning.3D.y", fTreeDrawArgsParser.GetParameter(3)); - if (fTreeDrawArgsParser.IsSpecified(6)) - gEnv->SetValue("Hist.Binning.3D.z", fTreeDrawArgsParser.GetParameter(6)); - binsx = gEnv->GetValue("Hist.Binning.3D.x",100); - minx = fTreeDrawArgsParser.GetIfSpecified(1, 0); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, 0); - binsy = gEnv->GetValue("Hist.Binning.3D.y",100); - miny = fTreeDrawArgsParser.GetIfSpecified(4, 0); - maxy = fTreeDrawArgsParser.GetIfSpecified(5, 0); - binsz = gEnv->GetValue("Hist.Binning.3D.z",100); - minz = fTreeDrawArgsParser.GetIfSpecified(7, 0); - maxz = fTreeDrawArgsParser.GetIfSpecified(8, 0); - exp += fTreeDrawArgsParser.GetObjectName(); - exp += '('; - exp += binsx; - exp += ','; - exp += minx; - exp += ','; - exp += maxx; - exp += ','; - exp += binsy; - exp += ','; - exp += miny; - exp += ','; - exp += maxy; - exp += ','; - exp += binsz; - exp += ','; - exp += minz; - exp += ','; - exp += maxz; - exp += ')'; - fInitialExp = exp; - TNamed *n = dynamic_cast (fInput->FindObject("varexp")); - if (n) - n->SetTitle(exp); - else - Error("DefVar3D", "Cannot find varexp on the fInput"); - if (fTreeDrawArgsParser.GetNoParameters() != 9) - fInput->Add(new TNamed("PROOF_OPTIONS", "rebin")); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Define variables according to arguments. - -void TProofDrawHist::DefVar() -{ - PDB(kDraw,1) Info("DefVar","Enter"); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - if (fTreeDrawArgsParser.GetObjectName() == "") - fTreeDrawArgsParser.SetObjectName("htemp"); - - switch (fTreeDrawArgsParser.GetDimension()) { - case 1: - DefVar1D(); - break; - case 2: - DefVar2D(); - break; - case 3: - DefVar3D(); - break; - default: - Error("DefVar", "Wrong dimension"); - break; - } - } - PDB(kDraw,1) Info("DefVar","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("DefVar","varexp: %s", fInitialExp.Data()); - fTree = 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Init(). - -void TProofDrawHist::Init(TTree *tree) -{ - PDB(kDraw,1) Info("Init","Enter tree = %p", tree); - if (fTree == 0) { - if (!dynamic_cast (fTreeDrawArgsParser.GetOriginal())) { - fHistogram->SetLineColor(tree->GetLineColor()); - fHistogram->SetLineWidth(tree->GetLineWidth()); - fHistogram->SetLineStyle(tree->GetLineStyle()); - fHistogram->SetFillColor(tree->GetFillColor()); - fHistogram->SetFillStyle(tree->GetFillStyle()); - fHistogram->SetMarkerStyle(tree->GetMarkerStyle()); - fHistogram->SetMarkerColor(tree->GetMarkerColor()); - fHistogram->SetMarkerSize(tree->GetMarkerSize()); - } - } - fTree = tree; - CompileVariables(); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveBegin(). - -void TProofDrawHist::SlaveBegin(TTree *tree) -{ - PDB(kDraw,1) Info("SlaveBegin","Enter tree = %p", tree); - - // Get the weight - TProofDraw::FillWeight(); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - SafeDelete(fHistogram); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - fDimension = fTreeDrawArgsParser.GetDimension(); - TString exp = fTreeDrawArgsParser.GetExp(); - TString objname = fTreeDrawArgsParser.GetObjectName(); - if (objname.Length() > 0 && strcmp(objname.Data(), "htemp")) { - TH1 *hist = dynamic_cast (fInput->FindObject(objname.Data())); - if (hist) { - fHistogram = (TH1 *) hist->Clone(); - PDB(kDraw,1) Info("SlaveBegin","original histogram found"); - } else { - PDB(kDraw,1) Info("SlaveBegin", "original object '%s' not found" - " or it is not a histogram", objname.Data()); - } - } - - // Create the histogram if not found in the input list - if (!fHistogram) { - Int_t countx = 100; double minx = 0, maxx = 0; - Int_t county = 100; double miny = 0, maxy = 0; - Int_t countz = 100; double minz = 0, maxz = 0; - if (fTreeDrawArgsParser.GetNoParameters() != 0) { - countx = (Int_t) fTreeDrawArgsParser.GetIfSpecified(0, countx); - county = (Int_t) fTreeDrawArgsParser.GetIfSpecified(3, county); - countz = (Int_t) fTreeDrawArgsParser.GetIfSpecified(6, countz); - minx = fTreeDrawArgsParser.GetIfSpecified(1, minx); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, maxx); - miny = fTreeDrawArgsParser.GetIfSpecified(4, miny); - maxy = fTreeDrawArgsParser.GetIfSpecified(5, maxy); - minz = fTreeDrawArgsParser.GetIfSpecified(7, minz); - maxz = fTreeDrawArgsParser.GetIfSpecified(8, maxz); - } - if (fTreeDrawArgsParser.GetNoParameters() != 3*fDimension) - Error("SlaveBegin", "Impossible - Wrong number of parameters"); - - if (fDimension == 1) - fHistogram = new TH1F(fTreeDrawArgsParser.GetObjectName(), - fTreeDrawArgsParser.GetObjectTitle(), - countx, minx, maxx); - else if (fDimension == 2){ - fHistogram = new TH2F(fTreeDrawArgsParser.GetObjectName(), - fTreeDrawArgsParser.GetObjectTitle(), - countx, minx, maxx, - county, miny, maxy); - } - else if (fDimension == 3) { - fHistogram = new TH3F(fTreeDrawArgsParser.GetObjectName(), - fTreeDrawArgsParser.GetObjectTitle(), - countx, minx, maxx, - county, miny, maxy, - countz, minz, maxz); - } else { - Info("Begin", "Wrong dimension"); - return; // FIXME: end the session - } - if (minx >= maxx) - fHistogram->SetBuffer(TH1::GetDefaultBufferSize()); - if (TNamed *opt = dynamic_cast (fInput->FindObject("PROOF_OPTIONS"))) { - if (strstr(opt->GetTitle(), "rebin")) - fHistogram->SetCanExtend(TH1::kAllAxes); - } - } - fHistogram->SetDirectory(0); // take ownership - fOutput->Add(fHistogram); // release ownership - } - - fTree = 0; - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Fills the histgram with given values. - -void TProofDrawHist::DoFill(Long64_t, Double_t w, const Double_t *v) -{ - if (fDimension == 1) - fHistogram->Fill(v[0], w); - else if (fDimension == 2) - ((TH2F *)fHistogram)->Fill(v[1], v[0], w); - else if (fDimension == 3) - ((TH3F *)fHistogram)->Fill(v[2], v[1], v[0], w); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Terminate(). - -void TProofDrawHist::Terminate(void) -{ - PDB(kDraw,1) Info("Terminate","Enter"); - TProofDraw::Terminate(); - if (!fStatus) - return; - - fHistogram = (TH1F *) fOutput->FindObject(fTreeDrawArgsParser.GetObjectName()); - if (fHistogram) { - SetStatus((Int_t) fHistogram->GetEntries()); - TH1 *h = 0; - if ((h = dynamic_cast (fTreeDrawArgsParser.GetOriginal()))) { - if (!fTreeDrawArgsParser.GetAdd()) - h->Reset(); - TList l; - l.Add(fHistogram); - h->Merge(&l); - l.Remove(fHistogram); - fOutput->Remove(fHistogram); - delete fHistogram; - } else { - // Set the title - fHistogram->SetTitle(fTreeDrawArgsParser.GetObjectTitle()); - h = fHistogram; - } - if (fTreeDrawArgsParser.GetShouldDraw()) { - // Choose the right canvas - SetCanvas(h->GetName()); - // Draw - SetDrawAtt(h); - h->Draw(fOption.Data()); - } - } - fHistogram = 0; -} - -ClassImp(TProofDrawEventList); - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Init(). - -void TProofDrawEventList::Init(TTree *tree) -{ - PDB(kDraw,1) Info("Init","Enter tree = %p", tree); - - if (fTree) { // new tree is being set - if (!fElist) - Error("Init", "Impossible - fElist cannot be 0"); - fEventLists->Add(fElist); - } - fElist = new TEventList(tree->GetDirectory()->GetName(), tree->GetName()); - fTree = tree; - CompileVariables(); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveBegin(). - -void TProofDrawEventList::SlaveBegin(TTree *tree) -{ - PDB(kDraw,1) Info("SlaveBegin","Enter tree = %p", tree); - - // Get the weight - TProofDraw::FillWeight(); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - - SafeDelete(fEventLists); - - fDimension = 0; - fTree = 0; - fEventLists = new TList(); - fEventLists->SetName("PROOF_EventListsList"); - fOutput->Add(fEventLists); - } - - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Fills the eventlist with given values. - -void TProofDrawEventList::DoFill(Long64_t entry, Double_t , const Double_t *) -{ - fElist->Enter(entry); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveTerminate(). - -void TProofDrawEventList::SlaveTerminate(void) -{ - PDB(kDraw,1) Info("SlaveTerminate","Enter"); - fEventLists->Add(fElist); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Terminate(). - -void TProofDrawEventList::Terminate(void) -{ - TProofDraw::Terminate(); // take care of fStatus - if (!fStatus) - return; - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - - TEventList *el = dynamic_cast (fOutput->FindObject("PROOF_EventList")); - if (el) { - el->SetName(fInitialExp.Data()+2); - SetStatus(el->GetN()); - if (TEventList* old = dynamic_cast (fTreeDrawArgsParser.GetOriginal())) { - if (!fTreeDrawArgsParser.GetAdd()) - old->Reset(); - old->Add(el); - fOutput->Remove(el); - delete el; - } - } - else - Error("Terminate", "Cannot find output EventList"); - -} - -ClassImp(TProofDrawEntryList); - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Init(). - -void TProofDrawEntryList::Init(TTree *tree) -{ - PDB(kDraw,1) Info("Init","Enter tree = %p", tree); - - fTree = tree; - CompileVariables(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveBegin(). - -void TProofDrawEntryList::SlaveBegin(TTree *tree) -{ - PDB(kDraw,1) Info("SlaveBegin","Enter tree = %p", tree); - - // Get the weight - TProofDraw::FillWeight(); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - - SafeDelete(fElist); - - fDimension = 0; - fTree = 0; - fElist = new TEntryList("PROOF_EntryList", "PROOF_EntryList"); - fOutput->Add(fElist); - } - - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fills the eventlist with given values. - -void TProofDrawEntryList::DoFill(Long64_t entry, Double_t , const Double_t *) -{ - fElist->Enter(entry); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveTerminate(). - -void TProofDrawEntryList::SlaveTerminate(void) -{ - PDB(kDraw,1) Info("SlaveTerminate","Enter"); - fElist->OptimizeStorage(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Terminate(). - -void TProofDrawEntryList::Terminate(void) -{ - TProofDraw::Terminate(); // take care of fStatus - if (!fStatus) - return; - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - - TEntryList *el = dynamic_cast (fOutput->FindObject("PROOF_EntryList")); - - if (el) { - el->SetName(fInitialExp.Data()+2); - SetStatus(el->GetN()); - if (TEntryList* old = dynamic_cast (fTreeDrawArgsParser.GetOriginal())) { - if (!fTreeDrawArgsParser.GetAdd()) - old->Reset(); - old->Add(el); - fOutput->Remove(el); - delete el; - } - } - else - Error("Terminate", "Cannot find output EventList"); - -} - -ClassImp(TProofDrawProfile); - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Init(). - -void TProofDrawProfile::Init(TTree *tree) -{ - PDB(kDraw,1) Info("Init","Enter tree = %p", tree); - - if (fTree == 0) { - if (!dynamic_cast (fTreeDrawArgsParser.GetOriginal())) { - fProfile->SetLineColor(tree->GetLineColor()); - fProfile->SetLineWidth(tree->GetLineWidth()); - fProfile->SetLineStyle(tree->GetLineStyle()); - fProfile->SetFillColor(tree->GetFillColor()); - fProfile->SetFillStyle(tree->GetFillStyle()); - fProfile->SetMarkerStyle(tree->GetMarkerStyle()); - fProfile->SetMarkerColor(tree->GetMarkerColor()); - fProfile->SetMarkerSize(tree->GetMarkerSize()); - } - } - fTree = tree; - CompileVariables(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Define relevant variables - -void TProofDrawProfile::DefVar() -{ - PDB(kDraw,1) Info("DefVar","Enter"); - - if (fTreeDrawArgsParser.GetDimension() < 0) { - - // Init parser - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = ov->GetTitle(); - fInitialExp = ov->GetTitle(); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - } - } - - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 2); - - fTreeDrawArgsParser.SetOriginal(0); - TString exp = fTreeDrawArgsParser.GetVarExp(); - exp += ">>"; - double binsx, minx, maxx; - if (fTreeDrawArgsParser.IsSpecified(0)) - gEnv->SetValue("Hist.Binning.2D.Prof", fTreeDrawArgsParser.GetParameter(0)); - binsx = gEnv->GetValue("Hist.Binning.2D.Prof",100); - minx = fTreeDrawArgsParser.GetIfSpecified(1, 0); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, 0); - if (fTreeDrawArgsParser.GetObjectName() == "") - fTreeDrawArgsParser.SetObjectName("htemp"); - exp += fTreeDrawArgsParser.GetObjectName(); - exp += '('; - exp += binsx; - exp += ','; - exp += minx; - exp += ','; - exp += maxx; - exp += ')'; - fInitialExp = exp; - TNamed *n = dynamic_cast (fInput->FindObject("varexp")); - if (n) - n->SetTitle(exp); - else - Error("DefVar", "Cannot find varexp on the fInput"); - if (fTreeDrawArgsParser.GetNoParameters() != 3) - fInput->Add(new TNamed("PROOF_OPTIONS", "rebin")); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Begin(). - -void TProofDrawProfile::Begin(TTree *tree) -{ - PDB(kDraw,1) Info("Begin","Enter tree = %p", tree); - - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 2); - - TObject *orig = fTreeDrawArgsParser.GetOriginal(); - TH1* pold; - if ((pold = dynamic_cast (orig)) && fTreeDrawArgsParser.GetNoParameters() == 0) { - TProfile* pnew = (TProfile*) pold->Clone(); - pnew->Reset(); - fInput->Add(pnew); - } else { - delete orig; - DefVar(); - } - } - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); - fTree = 0; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveBegin(). - -void TProofDrawProfile::SlaveBegin(TTree *tree) -{ - PDB(kDraw,1) Info("SlaveBegin","Enter tree = %p", tree); - - // Get the weight - TProofDraw::FillWeight(); - - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - SafeDelete(fProfile); - - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - fDimension = 2; - TString exp = fTreeDrawArgsParser.GetExp(); - - if (fTreeDrawArgsParser.GetOriginal()) { - fProfile = dynamic_cast (fTreeDrawArgsParser.GetOriginal()); - if (fProfile) { - fOutput->Add(fProfile); - PDB(kDraw,1) Info("SlaveBegin","Original profile histogram found"); - return; - } - else - Error("SlaveBegin","Original object found but it is not a histogram"); - } - Int_t countx = 100; double minx = 0, maxx = 0; - if (fTreeDrawArgsParser.GetNoParameters() != 0) { - countx = (Int_t) fTreeDrawArgsParser.GetIfSpecified(0, countx); - minx = fTreeDrawArgsParser.GetIfSpecified(1, minx); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, maxx); - } - if (fTreeDrawArgsParser.GetNoParameters() != 3) - Error("SlaveBegin", "Impossible - Wrong number of parameters"); - TString constructorOptions = ""; - if (fOption.Contains("profs")) - constructorOptions = "s"; - else if (fOption.Contains("profi")) - constructorOptions = "i"; - else if (fOption.Contains("profg")) - constructorOptions = "g"; - - fProfile = new TProfile(fTreeDrawArgsParser.GetObjectName(), - fTreeDrawArgsParser.GetObjectTitle(), - countx, minx, maxx, - constructorOptions); - if (minx >= maxx) - fProfile->SetBuffer(TH1::GetDefaultBufferSize()); - - if (TNamed *opt = dynamic_cast (fInput->FindObject("PROOF_OPTIONS"))) { - if (strstr(opt->GetTitle(), "rebin")) - fProfile->SetCanExtend(TH1::kAllAxes); - } - fProfile->SetDirectory(0); // take ownership - fOutput->Add(fProfile); // release ownership - } - fTree = 0; - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Fills the profile histogram with the given values. - -void TProofDrawProfile::DoFill(Long64_t , Double_t w, const Double_t *v) -{ - fProfile->Fill(v[1], v[0], w); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Terminate(). - -void TProofDrawProfile::Terminate(void) -{ - PDB(kDraw,1) Info("Terminate","Enter"); - TProofDraw::Terminate(); - if (!fStatus) - return; - - fProfile = (TProfile *) fOutput->FindObject(fTreeDrawArgsParser.GetObjectName()); - if (fProfile) { - SetStatus((Int_t) fProfile->GetEntries()); - TProfile *pf = 0; - if ((pf = dynamic_cast (fTreeDrawArgsParser.GetOriginal()))) { - if (!fTreeDrawArgsParser.GetAdd()) - pf->Reset(); - TList l; - l.Add(fProfile); - pf->Merge(&l); - l.Remove(fProfile); - fOutput->Remove(fProfile); - delete fProfile; - } else { - fProfile->SetTitle(fTreeDrawArgsParser.GetObjectTitle()); - pf = fProfile; - } - if (fTreeDrawArgsParser.GetShouldDraw()) { - // Choose the right canvas - SetCanvas(pf->GetName()); - // Draw - SetDrawAtt(pf); - pf->Draw(fOption.Data()); - } - } - fProfile = 0; -} - - -ClassImp(TProofDrawProfile2D); - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Init(). - -void TProofDrawProfile2D::Init(TTree *tree) -{ - PDB(kDraw,1) Info("Init","Enter tree = %p", tree); - if (fTree == 0) { - if (!dynamic_cast (fTreeDrawArgsParser.GetOriginal())) { - fProfile->SetLineColor(tree->GetLineColor()); - fProfile->SetLineWidth(tree->GetLineWidth()); - fProfile->SetLineStyle(tree->GetLineStyle()); - fProfile->SetFillColor(tree->GetFillColor()); - fProfile->SetFillStyle(tree->GetFillStyle()); - fProfile->SetMarkerStyle(tree->GetMarkerStyle()); - fProfile->SetMarkerColor(tree->GetMarkerColor()); - fProfile->SetMarkerSize(tree->GetMarkerSize()); - } - } - - fTree = tree; - CompileVariables(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Define relevant variables - -void TProofDrawProfile2D::DefVar() -{ - PDB(kDraw,1) Info("DefVar","Enter"); - - if (fTreeDrawArgsParser.GetDimension() < 0) { - - // Init parser - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - } - } - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 3); - - fTreeDrawArgsParser.SetOriginal(0); - TString exp = fTreeDrawArgsParser.GetVarExp(); - exp += ">>"; - double binsx, minx, maxx; - double binsy, miny, maxy; - if (fTreeDrawArgsParser.IsSpecified(0)) - gEnv->SetValue("Hist.Binning.3D.Profx", fTreeDrawArgsParser.GetParameter(0)); - if (fTreeDrawArgsParser.IsSpecified(3)) - gEnv->SetValue("Hist.Binning.3D.Profy", fTreeDrawArgsParser.GetParameter(3)); - binsx = gEnv->GetValue("Hist.Binning.3D.Profx",20); - minx = fTreeDrawArgsParser.GetIfSpecified(1, 0); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, 0); - binsy = gEnv->GetValue("Hist.Binning.3D.Profy",20); - miny = fTreeDrawArgsParser.GetIfSpecified(4, 0); - maxy = fTreeDrawArgsParser.GetIfSpecified(5, 0); - if (fTreeDrawArgsParser.GetObjectName() == "") - fTreeDrawArgsParser.SetObjectName("htemp"); - exp += fTreeDrawArgsParser.GetObjectName(); - exp += '('; - exp += binsx; - exp += ','; - exp += minx; - exp += ','; - exp += maxx; - exp += ','; - exp += binsy; - exp += ','; - exp += miny; - exp += ','; - exp += maxy; - exp += ')'; - fInitialExp = exp; - TNamed *n = dynamic_cast (fInput->FindObject("varexp")); - if (n) - n->SetTitle(exp); - else - Error("DefVar", "Cannot find varexp on the fInput"); - if (fTreeDrawArgsParser.GetNoParameters() != 6) - fInput->Add(new TNamed("PROOF_OPTIONS", "rebin")); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Begin(). - -void TProofDrawProfile2D::Begin(TTree *tree) -{ - PDB(kDraw,1) Info("Begin","Enter tree = %p", tree); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 3); - - TObject *orig = fTreeDrawArgsParser.GetOriginal(); - TProfile2D *pold; - if ((pold = dynamic_cast (orig)) && fTreeDrawArgsParser.GetNoParameters() == 0) { - TProfile2D* pnew = (TProfile2D*) pold->Clone(); - pnew->Reset(); - fInput->Add(pnew); - } else { - delete orig; - DefVar(); - } - } - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveBegin(). - -void TProofDrawProfile2D::SlaveBegin(TTree *tree) -{ - PDB(kDraw,1) Info("SlaveBegin","Enter tree = %p", tree); - - // Get the weight - TProofDraw::FillWeight(); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - - SafeDelete(fProfile); - - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - fDimension = 2; - TString exp = fTreeDrawArgsParser.GetExp(); - - if (fTreeDrawArgsParser.GetOriginal()) { - fProfile = dynamic_cast (fTreeDrawArgsParser.GetOriginal()); - if (fProfile) { - fOutput->Add(fProfile); - PDB(kDraw,1) Info("SlaveBegin","Original profile histogram found"); - return; - } else - Error("SlaveBegin","Original object found but it is not a histogram"); - } - Int_t countx = 40; double minx = 0, maxx = 0; - Int_t county = 40; double miny = 0, maxy = 0; - if (fTreeDrawArgsParser.GetNoParameters() != 0) { - countx = (Int_t) fTreeDrawArgsParser.GetIfSpecified(0, countx); - minx = fTreeDrawArgsParser.GetIfSpecified(1, minx); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, maxx); - county = (Int_t) fTreeDrawArgsParser.GetIfSpecified(3, countx); - miny = fTreeDrawArgsParser.GetIfSpecified(4, minx); - maxy = fTreeDrawArgsParser.GetIfSpecified(5, maxx); - } - if (fTreeDrawArgsParser.GetNoParameters() != 6) - Error("SlaveBegin", "Impossible - Wrong number of parameters"); - - TString constructorOptions = ""; - if (fOption.Contains("profs")) - constructorOptions = "s"; - else if (fOption.Contains("profi")) - constructorOptions = "i"; - else if (fOption.Contains("profg")) - constructorOptions = "g"; - - fProfile = new TProfile2D(fTreeDrawArgsParser.GetObjectName(), - fTreeDrawArgsParser.GetObjectTitle(), - countx, minx, maxx, - county, miny, maxy, - constructorOptions); - if (minx >= maxx) - fProfile->SetBuffer(TH1::GetDefaultBufferSize()); - - if (TNamed *opt = dynamic_cast (fInput->FindObject("PROOF_OPTIONS"))) { - if (strstr(opt->GetTitle(), "rebin")) - fProfile->SetCanExtend(TH1::kAllAxes); - } - fProfile->SetDirectory(0); // take ownership - fOutput->Add(fProfile); // release ownership - } - fTree = 0; - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Fills the histogram with the given values. - -void TProofDrawProfile2D::DoFill(Long64_t , Double_t w, const Double_t *v) -{ - fProfile->Fill(v[2], v[1], v[0], w); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Terminate(). - -void TProofDrawProfile2D::Terminate(void) -{ - PDB(kDraw,1) Info("Terminate","Enter"); - TProofDraw::Terminate(); - if (!fStatus) - return; - - fProfile = (TProfile2D *) fOutput->FindObject(fTreeDrawArgsParser.GetObjectName()); - if (fProfile) { - SetStatus((Int_t) fProfile->GetEntries()); - TProfile2D *pf = 0; - if ((pf = dynamic_cast (fTreeDrawArgsParser.GetOriginal()))) { - if (!fTreeDrawArgsParser.GetAdd()) - pf->Reset(); - TList l; - l.Add(fProfile); - pf->Merge(&l); - l.Remove(fProfile); - fOutput->Remove(fProfile); - delete fProfile; - } else { - fProfile->SetTitle(fTreeDrawArgsParser.GetObjectTitle()); - pf = fProfile; - } - if (fTreeDrawArgsParser.GetShouldDraw()) { - // Choose the right canvas - SetCanvas(pf->GetName()); - // Draw - SetDrawAtt(pf); - pf->Draw(fOption.Data()); - } - } - fProfile = 0; -} - - -ClassImp(TProofDrawGraph); - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Init(). - -void TProofDrawGraph::Init(TTree *tree) -{ - PDB(kDraw,1) Info("Init","Enter tree = %p", tree); - - if (fTree == 0) { - R__ASSERT(fGraph); - fGraph->SetMarkerStyle(tree->GetMarkerStyle()); - fGraph->SetMarkerColor(tree->GetMarkerColor()); - fGraph->SetMarkerSize(tree->GetMarkerSize()); - fGraph->SetLineColor(tree->GetLineColor()); - fGraph->SetLineStyle(tree->GetLineStyle()); - fGraph->SetFillColor(tree->GetFillColor()); - fGraph->SetFillStyle(tree->GetFillStyle()); - } - fTree = tree; - CompileVariables(); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveBegin(). - -void TProofDrawGraph::SlaveBegin(TTree *tree) -{ - PDB(kDraw,1) Info("SlaveBegin","Enter tree = %p", tree); - - // Get the weight - TProofDraw::FillWeight(); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - - SafeDelete(fGraph); - fDimension = 2; - - fGraph = new TGraph(); - fGraph->SetName("PROOF_GRAPH"); - fOutput->Add(fGraph); // release ownership - } - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Fills the graph with the given values. - -void TProofDrawGraph::DoFill(Long64_t , Double_t , const Double_t *v) -{ - fGraph->SetPoint(fGraph->GetN(), v[1], v[0]); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Terminate(). - -void TProofDrawGraph::Terminate(void) -{ - PDB(kDraw,1) Info("Terminate","Enter"); - TProofDraw::Terminate(); - if (!fStatus) - return; - - fGraph = dynamic_cast (fOutput->FindObject("PROOF_GRAPH")); - if (fGraph) { - SetStatus((Int_t) fGraph->GetN()); - TH2F* hist; - TObject *orig = fTreeDrawArgsParser.GetOriginal(); - if ( (hist = dynamic_cast (orig)) == 0 ) { - delete orig; - fTreeDrawArgsParser.SetOriginal(0); - double binsx, minx, maxx; - double binsy, miny, maxy; - if (fTreeDrawArgsParser.IsSpecified(0)) - gEnv->SetValue("Hist.Binning.2D.x", fTreeDrawArgsParser.GetParameter(0)); - if (fTreeDrawArgsParser.IsSpecified(3)) - gEnv->SetValue("Hist.Binning.2D.y", fTreeDrawArgsParser.GetParameter(3)); - binsx = gEnv->GetValue("Hist.Binning.2D.x",100); - minx = fTreeDrawArgsParser.GetIfSpecified(1, 0); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, 0); - binsy = gEnv->GetValue("Hist.Binning.2D.y",100); - miny = fTreeDrawArgsParser.GetIfSpecified(4, 0); - maxy = fTreeDrawArgsParser.GetIfSpecified(5, 0); - hist = new TH2F(fTreeDrawArgsParser.GetObjectName(), fTreeDrawArgsParser.GetObjectTitle(), - (Int_t) binsx, minx, maxx, (Int_t) binsy, miny, maxy); - hist->SetBit(TH1::kNoStats); - hist->SetBit(kCanDelete); - if (fTreeDrawArgsParser.GetNoParameters() != 6) - hist->SetCanExtend(TH1::kAllAxes); - else - hist->SetCanExtend(TH1::kNoAxis); -// if (fTreeDrawArgsParser.GetShouldDraw()) // ?? FIXME -// hist->SetDirectory(0); - } else { - if (!fTreeDrawArgsParser.GetAdd()) - hist->Reset(); - } - if (hist->CanExtendAllAxes() && hist->TestBit(kCanDelete)) { - Double_t* xArray = fGraph->GetX(); - Double_t* yArray = fGraph->GetY(); - Double_t xmin = *std::min_element(xArray, xArray+fGraph->GetN()); - Double_t xmax = *std::max_element(xArray, xArray+fGraph->GetN()); - Double_t ymin = *std::min_element(yArray, yArray+fGraph->GetN()); - Double_t ymax = *std::max_element(yArray, yArray+fGraph->GetN()); - THLimitsFinder::GetLimitsFinder()->FindGoodLimits(hist,xmin,xmax,ymin,ymax); - } - if (!hist->TestBit(kCanDelete)) { - TH1 *h2c = hist->DrawCopy(fOption.Data()); - h2c->SetStats(kFALSE); - } else { - SetDrawAtt(hist); - hist->Draw(); - } - gPad->Update(); - - fGraph->SetEditable(kFALSE); - // FIXME set color, marker size, etc. - - if (fTreeDrawArgsParser.GetShouldDraw()) { - SetDrawAtt(fGraph); - if (fOption == "" || strcmp(fOption, "same") == 0) - fGraph->Draw("p"); - else - fGraph->Draw(fOption); - gPad->Update(); - } - if (!hist->TestBit(kCanDelete)) { - for (int i = 0; i < fGraph->GetN(); i++) { - Double_t x = 0, y = 0; - fGraph->GetPoint(i, x, y); - hist->Fill(x, y, 1); - } - } - } - fGraph = 0; -} - - -ClassImp(TProofDrawPolyMarker3D); - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Init(). - -void TProofDrawPolyMarker3D::Init(TTree *tree) -{ - PDB(kDraw,1) Info("Init","Enter tree = %p", tree); - - if (fTree == 0) { - R__ASSERT(fPolyMarker3D); - fPolyMarker3D->SetMarkerStyle(tree->GetMarkerStyle()); - fPolyMarker3D->SetMarkerColor(tree->GetMarkerColor()); - fPolyMarker3D->SetMarkerSize(tree->GetMarkerSize()); - } - fTree = tree; - CompileVariables(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveBegin(). - -void TProofDrawPolyMarker3D::SlaveBegin(TTree *tree) -{ - PDB(kDraw,1) Info("SlaveBegin","Enter tree = %p", tree); - - // Get the weight - TProofDraw::FillWeight(); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 3); - - SafeDelete(fPolyMarker3D); - fDimension = 3; - - fPolyMarker3D = new TPolyMarker3D(); - fOutput->Add(fPolyMarker3D); // release ownership - } - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Fills the scatter plot with the given values. - -void TProofDrawPolyMarker3D::DoFill(Long64_t , Double_t , const Double_t *v) -{ - fPolyMarker3D->SetNextPoint(v[2], v[1], v[0]); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Terminate(). - -void TProofDrawPolyMarker3D::Terminate(void) -{ - PDB(kDraw,1) Info("Terminate","Enter"); - TProofDraw::Terminate(); - if (!fStatus) - return; - - fPolyMarker3D = 0; - TIter next(fOutput); - while (TObject* o = next()) { - if (dynamic_cast (o)) { - fPolyMarker3D = dynamic_cast (o); - break; - } - } - - Bool_t checkPrevious = kFALSE; - if (fPolyMarker3D) { - SetStatus((Int_t) fPolyMarker3D->Size()); - TH3F* hist; - TObject *orig = fTreeDrawArgsParser.GetOriginal(); - if ( (hist = dynamic_cast (orig)) == 0 ) { - delete orig; - fTreeDrawArgsParser.SetOriginal(0); - if (fOption.Contains("same")) { - // Check existing histogram - hist = dynamic_cast (gDirectory->Get(fTreeDrawArgsParser.GetObjectName())); - } - if (!hist) { - double binsx, minx, maxx; - double binsy, miny, maxy; - double binsz, minz, maxz; - if (fTreeDrawArgsParser.IsSpecified(0)) - gEnv->SetValue("Hist.Binning.3D.x", fTreeDrawArgsParser.GetParameter(0)); - if (fTreeDrawArgsParser.IsSpecified(3)) - gEnv->SetValue("Hist.Binning.3D.y", fTreeDrawArgsParser.GetParameter(3)); - if (fTreeDrawArgsParser.IsSpecified(6)) - gEnv->SetValue("Hist.Binning.3D.z", fTreeDrawArgsParser.GetParameter(6)); - binsx = gEnv->GetValue("Hist.Binning.3D.x",100); - minx = fTreeDrawArgsParser.GetIfSpecified(1, 0); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, 0); - binsy = gEnv->GetValue("Hist.Binning.3D.y",100); - miny = fTreeDrawArgsParser.GetIfSpecified(4, 0); - maxy = fTreeDrawArgsParser.GetIfSpecified(5, 0); - binsz = gEnv->GetValue("Hist.Binning.3D.z",100); - minz = fTreeDrawArgsParser.GetIfSpecified(7, 0); - maxz = fTreeDrawArgsParser.GetIfSpecified(8, 0); - hist = new TH3F(fTreeDrawArgsParser.GetObjectName(), fTreeDrawArgsParser.GetObjectTitle(), - (Int_t) binsx, minx, maxx, - (Int_t) binsy, miny, maxy, - (Int_t) binsz, minz, maxz); - hist->SetBit(TH1::kNoStats); - hist->SetBit(kCanDelete); - if (fTreeDrawArgsParser.GetNoParameters() != 9) - hist->SetCanExtend(TH1::kAllAxes); - else - hist->SetCanExtend(TH1::kNoAxis); - } else { - checkPrevious = kTRUE; - PDB(kDraw,2) - Info("Terminate", "found histo '%s' in gDirectory", - fTreeDrawArgsParser.GetObjectName().Data()); - } - } else { - if (!fTreeDrawArgsParser.GetAdd()) - hist->Reset(); - } - - // Set the ranges; take into account previous histos for 'same' runs - Double_t rmin[3], rmax[3]; - if (hist->CanExtendAllAxes() && hist->TestBit(kCanDelete)) { - rmin[0] = rmax[0] = rmin[1] = rmax[1] = rmin[2] = rmax[2] = 0; - if (fPolyMarker3D->Size() > 0) { - fPolyMarker3D->GetPoint(0, rmin[0], rmin[1], rmin[2]); - fPolyMarker3D->GetPoint(0, rmax[0], rmax[1], rmax[2]); - } - for (int i = 1; i < fPolyMarker3D->Size(); i++) { - Double_t v[3] = {0}; - fPolyMarker3D->GetPoint(i, v[0], v[1], v[2]); - for (int ii = 0; ii < 3; ii++) { - if (v[ii] < rmin[ii]) rmin[ii] = v[ii]; - if (v[ii] > rmax[ii]) rmax[ii] = v[ii]; - } - } - // Compare with previous histo, if any - if (checkPrevious) { - rmin[0] = (hist->GetXaxis()->GetXmin() < rmin[0]) ? hist->GetXaxis()->GetXmin() - : rmin[0]; - rmin[1] = (hist->GetYaxis()->GetXmin() < rmin[1]) ? hist->GetYaxis()->GetXmin() - : rmin[1]; - rmin[2] = (hist->GetZaxis()->GetXmin() < rmin[2]) ? hist->GetZaxis()->GetXmin() - : rmin[2]; - rmax[0] = (hist->GetXaxis()->GetXmax() > rmax[0]) ? hist->GetXaxis()->GetXmax() - : rmax[0]; - rmax[1] = (hist->GetYaxis()->GetXmax() > rmax[1]) ? hist->GetYaxis()->GetXmax() - : rmax[1]; - rmax[2] = (hist->GetZaxis()->GetXmax() > rmax[2]) ? hist->GetZaxis()->GetXmax() - : rmax[2]; - } - - THLimitsFinder::GetLimitsFinder()->FindGoodLimits(hist, - rmin[0], rmax[0], rmin[1], rmax[1], rmin[2], rmax[2]); - } - if (fTreeDrawArgsParser.GetShouldDraw()) { - if (!hist->TestBit(kCanDelete)) { - TH1 *histcopy = hist->DrawCopy(fOption.Data()); - histcopy->SetStats(kFALSE); - } else { - SetDrawAtt(hist); - hist->Draw(fOption); // no draw options on purpose - } - gPad->Update(); - } else { - gPad->Clear(); - gPad->Range(-1,-1,1,1); - TView::CreateView(1,rmin,rmax); - } - if (fTreeDrawArgsParser.GetShouldDraw()) { - SetDrawAtt(fPolyMarker3D); - fPolyMarker3D->Draw(fOption); - } - gPad->Update(); - if (!hist->TestBit(kCanDelete)) { - for (int i = 0; i < fPolyMarker3D->Size(); i++) { - Double_t x = 0, y = 0, z = 0; - fPolyMarker3D->GetPoint(i, x, y, z); - hist->Fill(x, y, z, 1); - } - } - } -} - - -ClassImp(TProofDrawListOfGraphs); - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveBegin(). - -void TProofDrawListOfGraphs::SlaveBegin(TTree *tree) -{ - PDB(kDraw,1) Info("SlaveBegin","Enter tree = %p", tree); - - // Get the weight - TProofDraw::FillWeight(); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 3); - - SafeDelete(fPoints); - - fDimension = 3; - - fPoints = new TProofVectorContainer(new std::vector); - fPoints->SetName("PROOF_SCATTERPLOT"); - fOutput->Add(fPoints); // release ownership (? FIXME) - } - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Fills the scatter plot with the given values. - -void TProofDrawListOfGraphs::DoFill(Long64_t , Double_t , const Double_t *v) -{ - fPoints->GetVector()->push_back(Point3D_t(v[2], v[1], v[0])); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Terminate(). - -void TProofDrawListOfGraphs::Terminate(void) -{ - PDB(kDraw,1) Info("Terminate","Enter"); - TProofDraw::Terminate(); - if (!fStatus) - return; - - fPoints = dynamic_cast*> - (fOutput->FindObject("PROOF_SCATTERPLOT")); - if (fPoints) { - std::vector *points = fPoints->GetVector(); - R__ASSERT(points); - SetStatus((Int_t) points->size()); - TH2F* hist; - TObject *orig = fTreeDrawArgsParser.GetOriginal(); - if ( (hist = dynamic_cast (orig)) == 0 ) { - delete orig; - fTreeDrawArgsParser.SetOriginal(0); - double binsx, minx, maxx; - double binsy, miny, maxy; - if (fTreeDrawArgsParser.IsSpecified(0)) - gEnv->SetValue("Hist.Binning.2D.x", fTreeDrawArgsParser.GetParameter(0)); - if (fTreeDrawArgsParser.IsSpecified(3)) - gEnv->SetValue("Hist.Binning.2D.y", fTreeDrawArgsParser.GetParameter(3)); - binsx = gEnv->GetValue("Hist.Binning.2D.x", 40); - minx = fTreeDrawArgsParser.GetIfSpecified(1, 0); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, 0); - binsy = gEnv->GetValue("Hist.Binning.2D.y", 40); - miny = fTreeDrawArgsParser.GetIfSpecified(4, 0); - maxy = fTreeDrawArgsParser.GetIfSpecified(5, 0); - hist = new TH2F(fTreeDrawArgsParser.GetObjectName(), fTreeDrawArgsParser.GetObjectTitle(), - (Int_t) binsx, minx, maxx, (Int_t) binsy, miny, maxy); - hist->SetBit(TH1::kNoStats); - hist->SetBit(kCanDelete); - if (fTreeDrawArgsParser.GetNoParameters() != 6) - hist->SetCanExtend(TH1::kAllAxes); - else - hist->SetCanExtend(TH1::kNoAxis); - -// if (fTreeDrawArgsParser.GetShouldDraw()) // ?? FIXME -// hist->SetDirectory(0); - } - Double_t rmin[3], rmax[3]; - - // FIXME take rmin and rmax from the old histogram - rmin[0] = rmax[0] = rmin[1] = rmax[1] = rmin[2] = rmax[2] = 0; - if (points->size() > 0) { - rmin[0] = rmax[0] = (*points)[0].fX; - rmin[1] = rmax[1] = (*points)[0].fY; - rmin[2] = rmax[2] = (*points)[0].fZ; - - for (vector::const_iterator i = points->begin() + 1; i < points->end(); ++i) { - if (rmax[0] < i->fX) rmax[0] = i->fX; - if (rmax[1] < i->fY) rmax[1] = i->fY; - if (rmax[2] < i->fZ) rmax[2] = i->fZ; - if (rmin[0] > i->fX) rmin[0] = i->fX; - if (rmin[1] > i->fY) rmin[1] = i->fY; - if (rmin[2] > i->fZ) rmin[2] = i->fZ; - } - // in this case we don't care about user-specified limits - if (hist->CanExtendAllAxes() && hist->TestBit(kCanDelete)) { - THLimitsFinder::GetLimitsFinder()->FindGoodLimits(hist, - rmin[1], rmax[1], rmin[2], rmax[2]); - } - } - - Int_t ncolors = gStyle->GetNumberOfColors(); - TObjArray *grs = (TObjArray*)hist->GetListOfFunctions()->FindObject("graphs"); - Int_t col; - TGraph *gr; - if (!grs) { - grs = new TObjArray(ncolors); - grs->SetOwner(); - grs->SetName("graphs"); - hist->GetListOfFunctions()->Add(grs, "P"); - for (col=0;colSetMarkerColor(col); -// gr->SetMarkerStyle(fTree->GetMarkerStyle()); -// gr->SetMarkerSize(fTree->GetMarkerSize()); - grs->AddAt(gr,col); - } - } - // Fill the graphs acording to the color - for (vector::const_iterator i = points->begin(); - i < points->end(); ++i) { - col = Int_t((ncolors-1)*((i->fX-rmin[0])/(rmax[0]-rmin[0]))); - if (col < 0) col = 0; - if (col > ncolors-1) col = ncolors-1; - gr = (TGraph*)grs->UncheckedAt(col); - if (gr) gr->SetPoint(gr->GetN(), i->fY, i->fZ); - } - // Remove potential empty graphs - for (col=0;colAt(col); - if (gr && gr->GetN() <= 0) grs->Remove(gr); - } - if (fTreeDrawArgsParser.GetShouldDraw()) { - SetDrawAtt(hist); - hist->Draw(fOption.Data()); - gPad->Update(); - } - fOutput->Remove(fPoints); - SafeDelete(fPoints); - } -} - - -ClassImp(TProofDrawListOfPolyMarkers3D); - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::SlaveBegin(). - -void TProofDrawListOfPolyMarkers3D::SlaveBegin(TTree *tree) -{ - PDB(kDraw,1) Info("SlaveBegin","Enter tree = %p", tree); - - // Get the weight - TProofDraw::FillWeight(); - - TObject *os = fInput->FindObject("selection"); - TObject *ov = fInput->FindObject("varexp"); - - if (os && ov) { - fSelection = os->GetTitle(); - fInitialExp = ov->GetTitle(); - fTreeDrawArgsParser.Parse(fInitialExp, fSelection, fOption); - R__ASSERT(fTreeDrawArgsParser.GetDimension() == 4); - - SafeDelete(fPoints); - - fDimension = 4; - - fPoints = new TProofVectorContainer(new std::vector); - fPoints->SetName("PROOF_SCATTERPLOT"); - fOutput->Add(fPoints); // release ownership (? FIXME) - } - PDB(kDraw,1) Info("Begin","selection: %s", fSelection.Data()); - PDB(kDraw,1) Info("Begin","varexp: %s", fInitialExp.Data()); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Fills the scatter plot with the given values. - -void TProofDrawListOfPolyMarkers3D::DoFill(Long64_t , Double_t , const Double_t *v) -{ - fPoints->GetVector()->push_back(Point4D_t(v[3], v[2], v[1], v[0])); -} - - - -//////////////////////////////////////////////////////////////////////////////// -/// See TProofDraw::Terminate(). - -void TProofDrawListOfPolyMarkers3D::Terminate(void) -{ - PDB(kDraw,1) Info("Terminate","Enter"); - TProofDraw::Terminate(); - if (!fStatus) - return; - - fPoints = dynamic_cast*> - (fOutput->FindObject("PROOF_SCATTERPLOT")); - if (fPoints) { - std::vector *points = fPoints->GetVector(); - R__ASSERT(points); - SetStatus((Int_t) points->size()); - TH3F* hist; - TObject *orig = fTreeDrawArgsParser.GetOriginal(); - if ( (hist = dynamic_cast (orig)) == 0 || fTreeDrawArgsParser.GetNoParameters() != 0) { - delete orig; - fTreeDrawArgsParser.SetOriginal(0); - double binsx, minx, maxx; - double binsy, miny, maxy; - double binsz, minz, maxz; - if (fTreeDrawArgsParser.IsSpecified(0)) - gEnv->SetValue("Hist.Binning.3D.x", fTreeDrawArgsParser.GetParameter(0)); - if (fTreeDrawArgsParser.IsSpecified(3)) - gEnv->SetValue("Hist.Binning.3D.y", fTreeDrawArgsParser.GetParameter(3)); - if (fTreeDrawArgsParser.IsSpecified(6)) - gEnv->SetValue("Hist.Binning.3D.z", fTreeDrawArgsParser.GetParameter(3)); - binsx = gEnv->GetValue("Hist.Binning.3D.x", 20); - minx = fTreeDrawArgsParser.GetIfSpecified(1, 0); - maxx = fTreeDrawArgsParser.GetIfSpecified(2, 0); - binsy = gEnv->GetValue("Hist.Binning.3D.y", 20); - miny = fTreeDrawArgsParser.GetIfSpecified(4, 0); - maxy = fTreeDrawArgsParser.GetIfSpecified(5, 0); - binsz = gEnv->GetValue("Hist.Binning.3D.z", 20); - minz = fTreeDrawArgsParser.GetIfSpecified(7, 0); - maxz = fTreeDrawArgsParser.GetIfSpecified(8, 0); - hist = new TH3F(fTreeDrawArgsParser.GetObjectName(), fTreeDrawArgsParser.GetObjectTitle(), - (Int_t) binsx, minx, maxx, - (Int_t) binsy, miny, maxy, - (Int_t) binsz, minz, maxz); - hist->SetBit(TH1::kNoStats); - hist->SetBit(kCanDelete); - if (fTreeDrawArgsParser.GetNoParameters() != 9) - hist->SetCanExtend(TH1::kAllAxes); - else - hist->SetCanExtend(TH1::kNoAxis); - -// if (fTreeDrawArgsParser.GetShouldDraw()) // ?? FIXME -// hist->SetDirectory(0); - } - Double_t rmin[4], rmax[4]; - - - // FIXME take rmin and rmax from the old histogram - rmin[0] = rmax[0] = rmin[1] = rmax[1] = rmin[2] = rmax[2] = 0; - if (points->size() > 0) { - rmin[0] = rmax[0] = (*points)[0].fX; - rmin[1] = rmax[1] = (*points)[0].fY; - rmin[2] = rmax[2] = (*points)[0].fZ; - rmin[3] = rmax[3] = (*points)[0].fT; - - for (vector::const_iterator i = points->begin() + 1; i < points->end(); ++i) { - if (rmax[0] < i->fX) rmax[0] = i->fX; - if (rmax[1] < i->fY) rmax[1] = i->fY; - if (rmax[2] < i->fZ) rmax[2] = i->fZ; - if (rmax[3] < i->fT) rmax[3] = i->fT; - if (rmin[0] > i->fX) rmin[0] = i->fX; - if (rmin[1] > i->fY) rmin[1] = i->fY; - if (rmin[2] > i->fZ) rmin[2] = i->fZ; - if (rmin[3] > i->fT) rmin[3] = i->fT; - } - // in this case we don't care about user-specified limits - if (hist->CanExtendAllAxes() && hist->TestBit(kCanDelete)) { - THLimitsFinder::GetLimitsFinder()->FindGoodLimits(hist, - rmin[1], rmax[1], rmin[2], rmax[2], rmin[3], rmax[3]); - } - } - Int_t ncolors = gStyle->GetNumberOfColors(); - TObjArray *pms = (TObjArray*)hist->GetListOfFunctions()->FindObject("polymarkers"); - Int_t col; - TPolyMarker3D *pm3d; - if (!pms) { - pms = new TObjArray(ncolors); - pms->SetOwner(); - pms->SetName("polymarkers"); - hist->GetListOfFunctions()->Add(pms); - for (col=0;colSetMarkerColor(col); -// pm3d->SetMarkerStyle(fTree->GetMarkerStyle()); -// pm3d->SetMarkerSize(fTree->GetMarkerSize()); - pms->AddAt(pm3d,col); - } - } - for (vector::const_iterator i = points->begin(); - i < points->end(); ++i) { - col = Int_t(i->fX); - if (col < 0) col = 0; - if (col > ncolors-1) col = ncolors-1; - pm3d = (TPolyMarker3D*)pms->UncheckedAt(col); - pm3d->SetPoint(pm3d->GetLastPoint()+1, i->fY, i->fZ, i->fT); - } - if (fTreeDrawArgsParser.GetShouldDraw()) { - SetDrawAtt(hist); - hist->Draw(fOption.Data()); - gPad->Update(); - } - fOutput->Remove(fPoints); - SafeDelete(fPoints); - } -} diff --git a/proof/proofplayer/src/TProofLimitsFinder.cxx b/proof/proofplayer/src/TProofLimitsFinder.cxx deleted file mode 100644 index d8ba6000deb0e..0000000000000 --- a/proof/proofplayer/src/TProofLimitsFinder.cxx +++ /dev/null @@ -1,110 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 19/04/2002 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofLimitsFinder -\ingroup proofkernel - -Class to find axis limits and synchronize them between workers - -*/ - -#include "TProofLimitsFinder.h" -#include "TProofServ.h" -#include "TSocket.h" -#include "TH1.h" -#include "TMessage.h" -#include "TProofDebug.h" -#include "TError.h" - -ClassImp(TProofLimitsFinder); - -//////////////////////////////////////////////////////////////////////////////// -/// Get bining information. Send current min and max and receive common -/// min max in return. - -void TProofLimitsFinder::AutoBinFunc(TString& key, - Double_t& xmin, Double_t& xmax, - Double_t& ymin, Double_t& ymax, - Double_t& zmin, Double_t& zmax) -{ - if (!gProofServ) return; - - TSocket *s = gProofServ->GetSocket(); - TMessage mess(kPROOF_AUTOBIN); - - PDB(kGlobal, 2) { - ::Info("TProofLimitsFinder::AutoBinFunc", - "Sending %f, %f, %f, %f, %f, %f", xmin, xmax, ymin, ymax, zmin, zmax); - } - mess << key << xmin << xmax << ymin << ymax << zmin << zmax; - - s->Send(mess); - - Bool_t notdone = kTRUE; - while (notdone) { - TMessage *answ; - if (s->Recv(answ) <= 0 || !answ) - return; - - Int_t what = answ->What(); - if (what == kPROOF_AUTOBIN) { - (*answ) >> key >> xmin >> xmax >> ymin >> ymax >> zmin >> zmax; - notdone = kFALSE; - } else { - Int_t xrc = gProofServ->HandleSocketInput(answ, kFALSE); - if (xrc == -1) { - ::Error("TProofLimitsFinder::AutoBinFunc", "command %d cannot be executed while processing", what); - } else if (xrc == -2) { - ::Error("TProofLimitsFinder::AutoBinFunc", "unknown command %d ! Protocol error?", what); - } - } - delete answ; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Find good limits - -Int_t TProofLimitsFinder::FindGoodLimits(TH1 *h, Axis_t xmin, Axis_t xmax) -{ - Double_t dummy = 0; - - TString key = h->GetName(); - AutoBinFunc(key, xmin, xmax, dummy, dummy, dummy, dummy); - - return THLimitsFinder::FindGoodLimits( h, xmin, xmax); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Find good limits - -Int_t TProofLimitsFinder::FindGoodLimits(TH1 *h, Axis_t xmin, Axis_t xmax, Axis_t ymin, Axis_t ymax) -{ - Double_t dummy = 0; - - TString key = h->GetName(); - AutoBinFunc(key, xmin, xmax, ymin, ymax, dummy, dummy); - - return THLimitsFinder::FindGoodLimits( h, xmin, xmax, ymin, ymax); -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Find good limits - -Int_t TProofLimitsFinder::FindGoodLimits(TH1 *h, Axis_t xmin, Axis_t xmax, Axis_t ymin, Axis_t ymax, Axis_t zmin, Axis_t zmax) -{ - TString key = h->GetName(); - AutoBinFunc(key, xmin, xmax, ymin, ymax, zmin, zmax); - - return THLimitsFinder::FindGoodLimits( h, xmin, xmax, ymin, ymax, zmin, zmax); -} diff --git a/proof/proofplayer/src/TProofMonSender.cxx b/proof/proofplayer/src/TProofMonSender.cxx deleted file mode 100644 index acda83cad8824..0000000000000 --- a/proof/proofplayer/src/TProofMonSender.cxx +++ /dev/null @@ -1,101 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G.Ganis July 2011 - -/************************************************************************* - * Copyright (C) 1995-2004, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -/** \class TProofMonSender -\ingroup proofkernel - -Provides the interface for PROOF monitoring to different writers. -Allows to decouple the information sent from the backend. - -*/ - -#include "TProofDebug.h" -#include "TProofMonSender.h" - -//////////////////////////////////////////////////////////////////////////////// -/// Parse send options from string 'sendopts'. -/// Format is: -/// "[-,+]S[n]|[-,+]D[m]|[-,+]F[j]" -/// where: -/// 1. The big letter refers to the 'table' following -/// -/// S table with summary log -/// D table with dataset info -/// F table files info -/// -/// 2. The '-,+' in front disables/enables the related table; if -/// absent '+' is assumed -/// -/// 3. The number after the letter is the version of the related -/// table -/// -/// Returns -1 if nothing is enabled; 0 otherwise - -Int_t TProofMonSender::SetSendOptions(const char *sendopts) -{ - - // Must have something to parse - if (sendopts && strlen(sendopts) > 0) { - - PDB(kMonitoring,1) Info("SetSendOptions", "sendopts: '%s'", sendopts); - - Bool_t doit = kTRUE; - Char_t t = 0; - Int_t v = -1; - TString oos(sendopts), oo; - Ssiz_t from = 0; - while (oos.Tokenize(oo, from, ":")) { - PDB(kMonitoring,2) Info("SetSendOptions", "oo: '%s'", oo.Data()); - // Parse info - doit = kTRUE; - if (oo.BeginsWith("+")) oo.Remove(0,1); - if (oo.BeginsWith("-")) { doit = kFALSE; oo.Remove(0,1); } - PDB(kMonitoring,2) Info("SetSendOptions", "oo: '%s' doit:%d", oo.Data(), doit); - t = oo[0]; - oo.Remove(0,1); - PDB(kMonitoring,2) Info("SetSendOptions", "oo: '%s' doit:%d t:'%c'", oo.Data(), doit, t); - v = -1; - if (!oo.IsNull() && oo.IsDigit()) v = oo.Atoi(); - PDB(kMonitoring,2) Info("SetSendOptions", "oo: '%s' doit:%d t:'%c' v:%d", oo.Data(), doit, t, v); - // Fill relevant variables - TProofMonSender::EConfigBits cbit = kSendSummary; - if (t == 'D') cbit = kSendDataSetInfo; - if (t == 'F') cbit = kSendFileInfo; - if (doit) - SetBit(cbit); - else - ResetBit(cbit); - if (v > -1) { - if (t == 'S') fSummaryVrs = v; - if (t == 'D') fDataSetInfoVrs = v; - if (t == 'F') fFileInfoVrs = v; - } - } - } - - // Something must be enabled - if (!(TestBit(kSendSummary) || TestBit(kSendDataSetInfo) || TestBit(kSendFileInfo))) { - Warning("SetSendOptions", "all tables are disabled!"); - return -1; - } - - // Notify - TString snot = TString::Format("%s: sending:", GetTitle()); - if (TestBit(kSendSummary)) snot += TString::Format(" 'summary' (v:%d)", fSummaryVrs); - if (TestBit(kSendDataSetInfo)) snot += TString::Format(" 'dataset info' (v:%d)", fDataSetInfoVrs); - if (TestBit(kSendFileInfo)) snot += TString::Format(" 'file info' (v:%d)", fFileInfoVrs); - Info("SetSendOptions", "%s", snot.Data()); - - // Done - return 0; -} - diff --git a/proof/proofplayer/src/TProofMonSenderSQL.cxx b/proof/proofplayer/src/TProofMonSenderSQL.cxx deleted file mode 100644 index 90605f6a79799..0000000000000 --- a/proof/proofplayer/src/TProofMonSenderSQL.cxx +++ /dev/null @@ -1,503 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G.Ganis July 2011 - -/************************************************************************* - * Copyright (C) 1995-2004, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -/** \class TProofMonSenderSQL -\ingroup proofkernel - -TProofMonSender implementation for the SQL writers - -*/ - -#include "TProofMonSenderSQL.h" - -#include "TDSet.h" -#include "TFileInfo.h" -#include "THashList.h" -#include "TList.h" -#include "TUrl.h" -#include "TPluginManager.h" -#include "TProofDebug.h" -#include "TROOT.h" -#include "TSystem.h" -#include "TObjString.h" -#include "TVirtualMonitoring.h" - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor - -TProofMonSenderSQL::TProofMonSenderSQL(const char *serv, const char *user, - const char *pass, const char *table, - const char *dstab, const char *filestab) - : TProofMonSender(serv,"ProofMonSenderSQL"), - fDSetSendOpts("bulk,table=proofquerydsets"), - fFilesSendOpts("bulk,table=proofqueryfiles") -{ - fWriter = 0; - // Init the sender instance using the plugin manager - TPluginHandler *h = 0; - if ((h = gROOT->GetPluginManager()->FindHandler("TVirtualMonitoringWriter", "SQL"))) { - if (h->LoadPlugin() != -1) { - fWriter = (TVirtualMonitoringWriter *) h->ExecPlugin(4, serv, user, pass, table); - if (fWriter && fWriter->IsZombie()) SafeDelete(fWriter); - } - } - // Flag this instance as valid if the writer initialization succeeded - if (fWriter) ResetBit(TObject::kInvalidObject); - - // Set default send control options - SetBit(TProofMonSender::kSendSummary); - SetBit(TProofMonSender::kSendDataSetInfo); - SetBit(TProofMonSender::kSendFileInfo); - fSummaryVrs = 2; - fDataSetInfoVrs = 1; - fFileInfoVrs = 1; - - // Transfer verbosity requirements - PDB(kMonitoring,1) if (fWriter) fWriter->Verbose(kTRUE); - - // Reformat the send options strings, if needed - if (dstab && strlen(dstab) > 0) fDSetSendOpts.Form("bulk,table=%s", dstab); - if (filestab && strlen(filestab) > 0) fFilesSendOpts.Form("bulk,table=%s", filestab); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TProofMonSenderSQL::~TProofMonSenderSQL() -{ - SafeDelete(fWriter); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send 'summary' record for the table 'proofquerylog'. -/// -/// There are three versions of this record, corresponding the evolution -/// in time of the monitoring requirements. -/// -/// The default version 2 corresponds to the table created with the following command: -/// -/// CREATE TABLE proofquerylog ( -/// id int(11) NOT NULL auto_increment, -/// proofuser varchar(32) NOT NULL, -/// proofgroup varchar(32) default NULL, -/// querybegin datetime default NULL, -/// queryend datetime default NULL, -/// walltime int(11) default NULL, -/// cputime float default NULL, -/// bytesread bigint(20) default NULL, -/// events bigint(20) default NULL, -/// totevents bigint(20) default NULL, -/// workers int(11) default NULL, -/// querytag varchar(64) NOT NULL, -/// vmemmxw bigint(20) default NULL, -/// rmemmxw bigint(20) default NULL, -/// vmemmxm bigint(20) default NULL, -/// rmemmxm bigint(20) default NULL, -/// numfiles int(11) default NULL, -/// missfiles int(11) default NULL, -/// status int(11) default NULL, -/// rootver varchar(32) NOT NULL, -/// PRIMARY KEY (id) ); -/// -/// Version 1 corresponds to the table created with the following command: -/// ('user','begin','end' instead of 'proofuser', 'querybegin', 'queryend'; -/// no 'status', 'missfiles', 'rootver'; 'dataset' field with name(s) of -/// processed dataset(s)) -/// -/// CREATE TABLE proofquerylog ( -/// id int(11) NOT NULL auto_increment, -/// user varchar(32) NOT NULL, -/// proofgroup varchar(32) default NULL, -/// begin datetime default NULL, -/// end datetime default NULL, -/// walltime int(11) default NULL, -/// cputime float default NULL, -/// bytesread bigint(20) default NULL, -/// events bigint(20) default NULL, -/// totevents bigint(20) default NULL, -/// workers int(11) default NULL, -/// querytag varchar(64) NOT NULL, -/// vmemmxw bigint(20) default NULL, -/// rmemmxw bigint(20) default NULL, -/// vmemmxm bigint(20) default NULL, -/// rmemmxm bigint(20) default NULL, -/// numfiles int(11) default NULL, -/// dataset varchar(512) NOT NULL, -/// PRIMARY KEY (id) ); - -Int_t TProofMonSenderSQL::SendSummary(TList *recs, const char *dumid) -{ - // - // Version 0 corresponds to the table created with the following command: - // ('group' instead of 'proofgroup'; no 'querytag', 'vmemmxw', - // 'rmemmxw', 'vmemmxm', 'rmemmxm', 'numfiles', 'dataset') - // - // CREATE TABLE proofquerylog ( - // id int(11) NOT NULL auto_increment, - // user varchar(32) NOT NULL, - // group varchar(32) default NULL, - // begin datetime default NULL, - // end datetime default NULL, - // walltime int(11) default NULL, - // cputime float default NULL, - // bytesread bigint(20) default NULL, - // events bigint(20) default NULL, - // totevents bigint(20) default NULL, - // workers int(11) default NULL, - // PRIMARY KEY (id) ); - // - // Return 0 on success, -1 on any failure. - - if (!IsValid()) { - Error("SendSummary", "invalid instance: do nothing!"); - return -1; - } - - // Are we requested to send this info? - if (!TestBit(TProofMonSender::kSendSummary)) return 0; - - PDB(kMonitoring,1) Info("SendSummary", "preparing (qid: '%s')", dumid); - - // Make sure we have something to send - if (!recs || (recs && recs->GetSize() <= 0)) { - Error("SendSummary", "records list undefined or empty!"); - return -1; - } - TList *xrecs = recs; - - TObject *dsn = 0; - TNamed *nm = 0; - // We may need to correct some variable names first - if (fSummaryVrs > 1) { - if ((nm = (TNamed *) recs->FindObject("user"))) nm->SetName("proofuser"); - if ((nm = (TNamed *) recs->FindObject("begin"))) nm->SetName("querybegin"); - if ((nm = (TNamed *) recs->FindObject("end"))) nm->SetName("queryend"); - if ((dsn = recs->FindObject("dataset"))) recs->Remove(dsn); - } else if (fSummaryVrs == 0) { - // Only the first records - xrecs = new TList; - xrecs->SetOwner(kFALSE); - TIter nxr(recs); - TObject *o = 0; - while ((o = nxr())) { - if (!strcmp(o->GetName(), "vmemmxw")) break; - xrecs->Add(o); - } - } - - PDB(kMonitoring,1) Info("SendSummary", "sending (%d entries)", xrecs->GetSize()); - - // Now we are ready to send - Bool_t rc = fWriter->SendParameters(xrecs, dumid); - - // Restore the "dataset" entry in the list - if (fSummaryVrs > 1 && dsn && xrecs == recs) { - TObject *num = recs->FindObject("numfiles"); - if (num) - recs->AddBefore(num, dsn); - else - recs->Add(dsn); - } - if (xrecs != recs) SafeDelete(xrecs); - - // Done - return (rc ? 0 : -1); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Post information about the processed dataset(s). The information is taken -/// from the TDSet object 'dset' and integrated with the missing files -/// information in the list 'missing'. The string 'qid' is the uninque -/// ID of the query; 'begin' the starting time. -/// -/// The record is formatted for the table 'proofquerydsets'. -/// -/// There are two versions of this record, with or without the starting time. -/// The starting time could be looked up from the summary record, if available. -/// -/// The default version 1 corresponds to the table created with the following command: -/// -/// CREATE TABLE proofquerydsets ( -/// id int(11) NOT NULL auto_increment, -/// dsn varchar(512) NOT NULL, -/// querytag varchar(64) NOT NULL, -/// querybegin datetime default NULL, -/// numfiles int(11) default NULL, -/// missfiles int(11) default NULL, -/// PRIMARY KEY (id), -/// KEY ix_querytag (querytag) ); -/// -/// Version 0 corresponds to the table created with the following command: -/// (no 'querybegin') -/// -/// CREATE TABLE proofquerydsets ( -/// id int(11) NOT NULL auto_increment, -/// dsn varchar(512) NOT NULL, -/// querytag varchar(64) NOT NULL, -/// numfiles int(11) default NULL, -/// missfiles int(11) default NULL, -/// PRIMARY KEY (id), -/// KEY ix_querytag (querytag) ); -/// -/// The information is posted with a bulk insert. -/// -/// Returns 0 on success, -1 on failure. - -Int_t TProofMonSenderSQL::SendDataSetInfo(TDSet *dset, TList *missing, - const char *begin, const char *qid) -{ - if (!IsValid()) { - Error("SendDataSetInfo", "invalid instance: do nothing!"); - return -1; - } - - // Are we requested to send this info? - if (!TestBit(TProofMonSender::kSendDataSetInfo)) return 0; - - // The query id (tag) must be given - if (!qid || (qid && strlen(qid) <= 0)) { - Error("SendDataSetInfo", "query id (tag) undefined!"); - return -1; - } - // The dataset must be given - if (!dset) { - Error("SendDataSetInfo", "TDSet object undefined! (qid: '%s')", qid); - return -1; - } - - PDB(kMonitoring,1) Info("SendDataSetInfo", "preparing (qid: '%s')", qid); - - TList plets; - // Extract the information and save it into the relevant multiplets - TString dss(dset->GetName()), ds; - Ssiz_t from = 0; - while ((dss.Tokenize(ds, from , "[,| ]"))) { - // Create a new TDSetPlet and add it to the list - plets.Add(new TDSetPlet(ds.Data(), dset)); - } - - // Now try to count the files - TDSetPlet *plet = 0; - TIter nxpl(&plets); - TObject *o = 0; - TDSetElement *e = 0, *ee = 0; - TDSet *dsete = 0; - TIter nxe(dset->GetListOfElements()); - TString dse; - while ((o = nxe())) { - if ((e = dynamic_cast(o))) { - dse = e->GetDataSet(); - if (!dse.IsNull()) { - nxpl.Reset(); - while ((plet = (TDSetPlet *) nxpl())) { - if (dse == plet->GetName()) { - plet->fFiles += 1; - break; - } - } - } - } else if ((dsete = dynamic_cast(o))) { - PDB(kMonitoring,1) - Info("SendDataSetInfo", "dset '%s' (%d files)", - o->GetName(), dsete->GetListOfElements()->GetSize()); - TIter nxee(dsete->GetListOfElements()); - while ((ee = (TDSetElement *) nxee())) { - dse = ee->GetDataSet(); - if (!dse.IsNull()) { - nxpl.Reset(); - while ((plet = (TDSetPlet *) nxpl())) { - if (dse == plet->GetName()) { - plet->fFiles += 1; - plet->fDSet = dsete; - break; - } - } - } - } - } else { - Warning("SendDataSetInfo", "ignoring unknown element type: '%s'", o->ClassName()); - } - } - - // Now try to include the missing files info - if (missing) { - TFileInfo *fi = 0; - TIter nxm(missing); - TString dsfi, fn; - while ((fi = (TFileInfo *) nxm())) { - dsfi = fi->GetTitle(); - if (!dsfi.IsNull() && dsfi != "TFileInfo") { - nxpl.Reset(); - while ((plet = (TDSetPlet *) nxpl())) { - if (dsfi == plet->GetName()) { - fn = fi->GetCurrentUrl()->GetUrl(); - if (plet->fDSet && plet->fDSet->GetListOfElements() && - !(plet->fDSet->GetListOfElements()->FindObject(fn))) plet->fFiles += 1; - plet->fMissing += 1; - break; - } - } - } - } - } - - // Now we can prepare the input for SendParameters - TList values; - TString ent("dsn,querytag,querybegin,numfiles,missfiles"); - if (fDataSetInfoVrs == 0) ent = "dsn,querytag,numfiles,missfiles"; - values.Add(new TObjString(ent.Data())); - nxpl.Reset(); - while ((plet = (TDSetPlet *) nxpl())) { - if (fDataSetInfoVrs == 0) - ent.Form("'%s','%s',%d,%d", plet->GetName(), qid, plet->fFiles, plet->fMissing); - else - ent.Form("'%s','%s','%s',%d,%d", plet->GetName(), qid, begin, plet->fFiles, plet->fMissing); - values.Add(new TObjString(ent.Data())); - } - - PDB(kMonitoring,1) - Info("SendDataSetInfo", "sending (%d entries)", values.GetSize()); - - // Now we are ready to send - Bool_t rc = fWriter->SendParameters(&values, fDSetSendOpts); - - // Done - return (rc ? 0 : -1); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Post information about the requested files. The information is taken -/// from the TDSet object 'dset' and integrated with the missing files -/// information in the list 'missing'. The string 'qid' is the unique -/// ID of the query; 'begin' the starting time. -/// -/// The record is formatted for the table 'proofqueryfiles'. -/// -/// There are two versions of this record, with or without the starting time. -/// The starting time could be looked up from the summary record, if available. -/// -/// The default version 1 corresponds to the table created with the following command: -/// -/// CREATE TABLE proofqueryfiles ( -/// id int(11) NOT NULL auto_increment, -/// lfn varchar(255) NOT NULL, -/// path varchar(2048) NOT NULL, -/// querytag varchar(64) NOT NULL, -/// querybegin datetime default NULL, -/// status enum('Ok','Failed') NOT NULL default 'Ok', -/// PRIMARY KEY (id), -/// KEY ix_querytag (querytag) ); -/// -/// Version 0 corresponds to the table created with the following command: -/// (no 'querybegin') -/// -/// CREATE TABLE proofqueryfiles ( -/// id int(11) NOT NULL auto_increment, -/// lfn varchar(255) NOT NULL, -/// path varchar(2048) NOT NULL, -/// querytag varchar(64) NOT NULL, -/// status enum('Ok','Failed') NOT NULL default 'Ok', -/// PRIMARY KEY (id), -/// KEY ix_querytag (querytag) ); -/// -/// The information is posted with a bulk insert. -/// -/// Returns 0 on success, -1 on failure. - -Int_t TProofMonSenderSQL::SendFileInfo(TDSet *dset, TList *missing, - const char *begin, const char *qid) -{ - if (!IsValid()) { - Error("SendFileInfo", "invalid instance: do nothing!"); - return -1; - } - - // Are we requested to send this info? - if (!TestBit(TProofMonSender::kSendFileInfo)) return 0; - - // The query id (tag) must be given - if (!qid || (qid && strlen(qid) <= 0)) { - Error("SendFileInfo", "query id (tag) undefined!"); - return -1; - } - // The dataset must be given - if (!dset) { - Error("SendFileInfo", "TDSet object undefined! (qid: '%s')", qid); - return -1; - } - - PDB(kMonitoring,1) Info("SendFileInfo", "preparing (qid: '%s')", qid); - THashList hmiss; - if (missing) { - TIter nxfm(missing); - TFileInfo *fi = 0; - while ((fi = (TFileInfo *)nxfm())) { - hmiss.Add(new TObjString(fi->GetCurrentUrl()->GetUrl())); - } - PDB(kMonitoring,2) hmiss.Print(); - } - - TList values; - TString ent("lfn,path,querytag,querybegin,status"); - if (fFileInfoVrs == 0) ent = "lfn,path,querytag,status"; - values.Add(new TObjString(ent.Data())); - - // Create the file-plets - TObject *o = 0; - TDSetElement *e = 0, *ee = 0; - TDSet *dsete = 0; - TIter nxe(dset->GetListOfElements()); - TString fne, status; - while ((o = nxe())) { - if ((e = dynamic_cast(o))) { - fne = e->GetName(); - // Try to determine the status - status = "Ok"; - if (hmiss.FindObject(fne)) status = "Failed"; - if (fFileInfoVrs == 0) - ent.Form("'%s','%s','%s','%s'", gSystem->BaseName(fne), gSystem->GetDirName(fne).Data(), - qid, status.Data()); - else - ent.Form("'%s','%s','%s','%s','%s'", gSystem->BaseName(fne), gSystem->GetDirName(fne).Data(), - qid, begin, status.Data()); - values.Add(new TObjString(ent.Data())); - } else if ((dsete = dynamic_cast(o))) { - PDB(kMonitoring,1) - Info("SendFileInfo", "dset '%s' (%d files)", - o->GetName(), dsete->GetListOfElements()->GetSize()); - TIter nxee(dsete->GetListOfElements()); - while ((ee = (TDSetElement *) nxee())) { - fne = ee->GetName(); - // Try to determine the status - status = "Ok"; - if (hmiss.FindObject(fne)) status = "Failed"; - if (fFileInfoVrs == 0) - ent.Form("'%s','%s','%s','%s'", gSystem->BaseName(fne), gSystem->GetDirName(fne).Data(), - qid, status.Data()); - else - ent.Form("'%s','%s','%s','%s','%s'", gSystem->BaseName(fne), gSystem->GetDirName(fne).Data(), - qid, begin, status.Data()); - values.Add(new TObjString(ent.Data())); - } - } else { - Warning("SendFileInfo", "ignoring unknown element type: '%s'", o->ClassName()); - } - } - - PDB(kMonitoring,1) Info("SendFileInfo", "sending (%d entries)", values.GetSize()); - - // Now we are ready to send - Bool_t rc = fWriter->SendParameters(&values, fFilesSendOpts); - - // Done - return (rc ? 0 : -1); -} diff --git a/proof/proofplayer/src/TProofPlayer.cxx b/proof/proofplayer/src/TProofPlayer.cxx deleted file mode 100644 index 3d835db628514..0000000000000 --- a/proof/proofplayer/src/TProofPlayer.cxx +++ /dev/null @@ -1,4912 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 07/01/02 - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofPlayer -\ingroup proofkernel - -Internal class steering processing in PROOF. -Instances of the TProofPlayer class are created on the worker nodes -per session and do the processing. -Instances of its subclass - TProofPlayerRemote are created per each -query on the master(s) and on the client. On the master(s), -TProofPlayerRemote coordinate processing, check the dataset, create -the packetizer and take care of merging the results of the workers. -The instance on the client collects information on the input -(dataset and selector), it invokes the Begin() method and finalizes -the query by calling Terminate(). - -*/ - -#include "TProofDraw.h" -#include "TProofPlayer.h" -#include "THashList.h" -#include "TEnv.h" -#include "TEventIter.h" -#include "TVirtualPacketizer.h" -#include "TSelector.h" -#include "TSocket.h" -#include "TProofServ.h" -#include "TProof.h" -#include "TProofOutputFile.h" -#include "TProofSuperMaster.h" -#include "TSlave.h" -#include "TClass.h" -#include "TROOT.h" -#include "TError.h" -#include "TException.h" -#include "MessageTypes.h" -#include "TMessage.h" -#include "TDSetProxy.h" -#include "TString.h" -#include "TSystem.h" -#include "TFile.h" -#include "TFileCollection.h" -#include "TFileInfo.h" -#include "TFileMerger.h" -#include "TProofDebug.h" -#include "TTimer.h" -#include "TMap.h" -#include "TPerfStats.h" -#include "TStatus.h" -#include "TEventList.h" -#include "TProofLimitsFinder.h" -#include "TSortedList.h" -#include "TTree.h" -#include "TEntryList.h" -#include "TDSet.h" -#include "TDrawFeedback.h" -#include "TNamed.h" -#include "TObjString.h" -#include "TQueryResult.h" -#include "TMD5.h" -#include "TMethodCall.h" -#include "TObjArray.h" -#include "TH1.h" -#include "TVirtualMonitoring.h" -#include "TParameter.h" -#include "TOutputListSelectorDataMap.h" -#include "TStopwatch.h" - -// Timeout exception -#define kPEX_STOPPED 1001 -#define kPEX_ABORTED 1002 - -// To flag an abort condition: use a local static variable to avoid -// warnings about problems with longjumps -static Bool_t gAbort = kFALSE; - -class TAutoBinVal : public TNamed { -private: - Double_t fXmin, fXmax, fYmin, fYmax, fZmin, fZmax; - -public: - TAutoBinVal(const char *name, Double_t xmin, Double_t xmax, Double_t ymin, - Double_t ymax, Double_t zmin, Double_t zmax) : TNamed(name,"") - { - fXmin = xmin; fXmax = xmax; - fYmin = ymin; fYmax = ymax; - fZmin = zmin; fZmax = zmax; - } - void GetAll(Double_t& xmin, Double_t& xmax, Double_t& ymin, - Double_t& ymax, Double_t& zmin, Double_t& zmax) - { - xmin = fXmin; xmax = fXmax; - ymin = fYmin; ymax = fYmax; - zmin = fZmin; zmax = fZmax; - } - -}; - -// -// Special timer to dispatch pending events while processing -//////////////////////////////////////////////////////////////////////////////// - -class TDispatchTimer : public TTimer { -private: - TProofPlayer *fPlayer; - -public: - TDispatchTimer(TProofPlayer *p) : TTimer(1000, kFALSE), fPlayer(p) { } - - Bool_t Notify() override; -}; -//////////////////////////////////////////////////////////////////////////////// -/// Handle expiration of the timer associated with dispatching pending -/// events while processing. We must act as fast as possible here, so -/// we just set a flag submitting a request for dispatching pending events - -Bool_t TDispatchTimer::Notify() -{ - if (gDebug > 0) printf("TDispatchTimer::Notify: called!\n"); - - fPlayer->SetBit(TProofPlayer::kDispatchOneEvent); - - // Needed for the next shot - Reset(); - return kTRUE; -} - -// -// Special timer to notify reach of max packet proc time -//////////////////////////////////////////////////////////////////////////////// - -class TProctimeTimer : public TTimer { -private: - TProofPlayer *fPlayer; - -public: - TProctimeTimer(TProofPlayer *p, Long_t to) : TTimer(to, kFALSE), fPlayer(p) { } - - Bool_t Notify() override; -}; -//////////////////////////////////////////////////////////////////////////////// -/// Handle expiration of the timer associated with dispatching pending -/// events while processing. We must act as fast as possible here, so -/// we just set a flag submitting a request for dispatching pending events - -Bool_t TProctimeTimer::Notify() -{ - if (gDebug > 0) printf("TProctimeTimer::Notify: called!\n"); - - fPlayer->SetBit(TProofPlayer::kMaxProcTimeReached); - - // One shot only - return kTRUE; -} - -// -// Special timer to handle stop/abort request via exception raising -//////////////////////////////////////////////////////////////////////////////// - -class TStopTimer : public TTimer { -private: - Bool_t fAbort; - TProofPlayer *fPlayer; - -public: - TStopTimer(TProofPlayer *p, Bool_t abort, Int_t to); - - Bool_t Notify() override; -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor for the timer to stop/abort processing. -/// The 'timeout' is in seconds. -/// Make sure that 'to' make sense, i.e. not larger than 10 days; -/// the minimum value is 10 ms (0 does not seem to start the timer ...). - -TStopTimer::TStopTimer(TProofPlayer *p, Bool_t abort, Int_t to) - : TTimer(((to <= 0 || to > 864000) ? 10 : to * 1000), kFALSE) -{ - if (gDebug > 0) - Info ("TStopTimer","enter: %d, timeout: %d", abort, to); - - fPlayer = p; - fAbort = abort; - - if (gDebug > 1) - Info ("TStopTimer","timeout set to %s ms", fTime.AsString()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle the signal coming from the expiration of the timer -/// associated with an abort or stop request. -/// We raise an exception which will be processed in the -/// event loop. - -Bool_t TStopTimer::Notify() -{ - if (gDebug > 0) printf("TStopTimer::Notify: called!\n"); - - if (fAbort) - Throw(kPEX_ABORTED); - else - Throw(kPEX_STOPPED); - - return kTRUE; -} - -//------------------------------------------------------------------------------ - -ClassImp(TProofPlayer); - -THashList *TProofPlayer::fgDrawInputPars = 0; - -//////////////////////////////////////////////////////////////////////////////// -/// Default ctor. - -TProofPlayer::TProofPlayer(TProof *) - : fAutoBins(0), fOutput(0), fSelector(0), fCreateSelObj(kTRUE), fSelectorClass(0), - fFeedbackTimer(0), fFeedbackPeriod(2000), - fEvIter(0), fSelStatus(0), - fTotalEvents(0), fReadBytesRun(0), fReadCallsRun(0), fProcessedRun(0), - fQueryResults(0), fQuery(0), fPreviousQuery(0), fDrawQueries(0), - fMaxDrawQueries(1), fStopTimer(0), fDispatchTimer(0), - fProcTimeTimer(0), fProcTime(0), - fOutputFile(0), - fSaveMemThreshold(-1), fSavePartialResults(kFALSE), fSaveResultsPerPacket(kFALSE) -{ - fInput = new TList; - fExitStatus = kFinished; - fProgressStatus = new TProofProgressStatus(); - ResetBit(TProofPlayer::kDispatchOneEvent); - ResetBit(TProofPlayer::kIsProcessing); - ResetBit(TProofPlayer::kMaxProcTimeReached); - ResetBit(TProofPlayer::kMaxProcTimeExtended); - - static Bool_t initLimitsFinder = kFALSE; - if (!initLimitsFinder && gProofServ && !gProofServ->IsMaster()) { - THLimitsFinder::SetLimitsFinder(new TProofLimitsFinder); - initLimitsFinder = kTRUE; - } - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TProofPlayer::~TProofPlayer() -{ - fInput->Clear("nodelete"); - SafeDelete(fInput); - // The output list is owned by fSelector and destroyed in there - SafeDelete(fSelector); - SafeDelete(fFeedbackTimer); - SafeDelete(fEvIter); - SafeDelete(fQueryResults); - SafeDelete(fDispatchTimer); - SafeDelete(fProcTimeTimer); - SafeDelete(fProcTime); - SafeDelete(fStopTimer); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set processing bit according to 'on' - -void TProofPlayer::SetProcessing(Bool_t on) -{ - if (on) - SetBit(TProofPlayer::kIsProcessing); - else - ResetBit(TProofPlayer::kIsProcessing); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Stop the process after this event. If timeout is positive, start -/// a timer firing after timeout seconds to hard-stop time-expensive -/// events. - -void TProofPlayer::StopProcess(Bool_t abort, Int_t timeout) -{ - if (gDebug > 0) - Info ("StopProcess","abort: %d, timeout: %d", abort, timeout); - - if (fEvIter != 0) - fEvIter->StopProcess(abort); - Long_t to = 1; - if (abort == kTRUE) { - fExitStatus = kAborted; - } else { - fExitStatus = kStopped; - to = timeout; - } - // Start countdown, if needed - if (to > 0) - SetStopTimer(kTRUE, abort, to); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable/disable the timer to dispatch pening events while processing. - -void TProofPlayer::SetDispatchTimer(Bool_t on) -{ - SafeDelete(fDispatchTimer); - ResetBit(TProofPlayer::kDispatchOneEvent); - if (on) { - fDispatchTimer = new TDispatchTimer(this); - fDispatchTimer->Start(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Enable/disable the timer to stop/abort processing. -/// The 'timeout' is in seconds. - -void TProofPlayer::SetStopTimer(Bool_t on, Bool_t abort, Int_t timeout) -{ - std::lock_guard lock(fStopTimerMtx); - - // Clean-up the timer - SafeDelete(fStopTimer); - if (on) { - // create timer - fStopTimer = new TStopTimer(this, abort, timeout); - // Start the countdown - fStopTimer->Start(); - if (gDebug > 0) - Info ("SetStopTimer", "%s timer STARTED (timeout: %d)", - (abort ? "ABORT" : "STOP"), timeout); - } else { - if (gDebug > 0) - Info ("SetStopTimer", "timer STOPPED"); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add query result to the list, making sure that there are no -/// duplicates. - -void TProofPlayer::AddQueryResult(TQueryResult *q) -{ - if (!q) { - Warning("AddQueryResult","query undefined - do nothing"); - return; - } - - // Treat differently normal and draw queries - if (!(q->IsDraw())) { - if (!fQueryResults) { - fQueryResults = new TList; - fQueryResults->Add(q); - } else { - TIter nxr(fQueryResults); - TQueryResult *qr = 0; - TQueryResult *qp = 0; - while ((qr = (TQueryResult *) nxr())) { - // If same query, remove old version and break - if (*qr == *q) { - fQueryResults->Remove(qr); - delete qr; - break; - } - // Record position according to start time - if (qr->GetStartTime().Convert() <= q->GetStartTime().Convert()) - qp = qr; - } - - if (!qp) { - fQueryResults->AddFirst(q); - } else { - fQueryResults->AddAfter(qp, q); - } - } - } else if (IsClient()) { - // If max reached, eliminate first the oldest one - if (fDrawQueries == fMaxDrawQueries && fMaxDrawQueries > 0) { - TIter nxr(fQueryResults); - TQueryResult *qr = 0; - while ((qr = (TQueryResult *) nxr())) { - // If same query, remove old version and break - if (qr->IsDraw()) { - fDrawQueries--; - fQueryResults->Remove(qr); - delete qr; - break; - } - } - } - // Add new draw query - if (fDrawQueries >= 0 && fDrawQueries < fMaxDrawQueries) { - fDrawQueries++; - if (!fQueryResults) - fQueryResults = new TList; - fQueryResults->Add(q); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Remove all query result instances referenced 'ref' from -/// the list of results. - -void TProofPlayer::RemoveQueryResult(const char *ref) -{ - if (fQueryResults) { - TIter nxq(fQueryResults); - TQueryResult *qr = 0; - while ((qr = (TQueryResult *) nxq())) { - if (qr->Matches(ref)) { - fQueryResults->Remove(qr); - delete qr; - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get query result instances referenced 'ref' from -/// the list of results. - -TQueryResult *TProofPlayer::GetQueryResult(const char *ref) -{ - if (fQueryResults) { - if (ref && strlen(ref) > 0) { - TIter nxq(fQueryResults); - TQueryResult *qr = 0; - while ((qr = (TQueryResult *) nxq())) { - if (qr->Matches(ref)) - return qr; - } - } else { - // Get last - return (TQueryResult *) fQueryResults->Last(); - } - } - - // Nothing found - return (TQueryResult *)0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set current query and save previous value. - -void TProofPlayer::SetCurrentQuery(TQueryResult *q) -{ - fPreviousQuery = fQuery; - fQuery = q; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add object to input list. - -void TProofPlayer::AddInput(TObject *inp) -{ - fInput->Add(inp); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Clear input list. - -void TProofPlayer::ClearInput() -{ - fInput->Clear(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get output object by name. - -TObject *TProofPlayer::GetOutput(const char *name) const -{ - if (fOutput) - return fOutput->FindObject(name); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get output list. - -TList *TProofPlayer::GetOutputList() const -{ - TList *ol = fOutput; - if (!ol && fQuery) - ol = fQuery->GetOutputList(); - return ol; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reinitialize fSelector using the selector files in the query result. -/// Needed when Finalize is called after a Process execution for the same -/// selector name. - -Int_t TProofPlayer::ReinitSelector(TQueryResult *qr) -{ - Int_t rc = 0; - - // Make sure we have a query - if (!qr) { - Info("ReinitSelector", "query undefined - do nothing"); - return -1; - } - - // Selector name - TString selec = qr->GetSelecImp()->GetName(); - if (selec.Length() <= 0) { - Info("ReinitSelector", "selector name undefined - do nothing"); - return -1; - } - - // Find out if this is a standard selection used for Draw actions - Bool_t stdselec = TSelector::IsStandardDraw(selec); - - // Find out if this is a precompiled selector: in such a case we do not - // have the code in TMacros, so we must rely on local libraries - Bool_t compselec = (selec.Contains(".") || stdselec) ? kFALSE : kTRUE; - - // If not, find out if it needs to be expanded - TString ipathold; - if (!stdselec && !compselec) { - // Check checksums for the versions of the selector files - Bool_t expandselec = kTRUE; - TString dir, ipath; - char *selc = gSystem->Which(TROOT::GetMacroPath(), selec, kReadPermission); - if (selc) { - // Check checksums - TMD5 *md5icur = 0, *md5iold = 0, *md5hcur = 0, *md5hold = 0; - // Implementation files - md5icur = TMD5::FileChecksum(selc); - md5iold = qr->GetSelecImp()->Checksum(); - // Header files - TString selh(selc); - Int_t dot = selh.Last('.'); - if (dot != kNPOS) selh.Remove(dot); - selh += ".h"; - if (!gSystem->AccessPathName(selh, kReadPermission)) - md5hcur = TMD5::FileChecksum(selh); - md5hold = qr->GetSelecHdr()->Checksum(); - - // If nothing has changed nothing to do - if (md5hcur && md5hold && md5icur && md5iold) - if (*md5hcur == *md5hold && *md5icur == *md5iold) - expandselec = kFALSE; - - SafeDelete(md5icur); - SafeDelete(md5hcur); - SafeDelete(md5iold); - SafeDelete(md5hold); - if (selc) delete [] selc; - } - - Bool_t ok = kTRUE; - // Expand selector files, if needed - if (expandselec) { - - ok = kFALSE; - // Expand files in a temporary directory - TUUID u; - dir = Form("%s/%s",gSystem->TempDirectory(),u.AsString()); - if (!(gSystem->MakeDirectory(dir))) { - - // Export implementation file - selec = Form("%s/%s",dir.Data(),selec.Data()); - qr->GetSelecImp()->SaveSource(selec); - - // Export header file - TString seleh = Form("%s/%s",dir.Data(),qr->GetSelecHdr()->GetName()); - qr->GetSelecHdr()->SaveSource(seleh); - - // Adjust include path - ipathold = gSystem->GetIncludePath(); - ipath = Form("-I%s %s", dir.Data(), gSystem->GetIncludePath()); - gSystem->SetIncludePath(ipath.Data()); - - ok = kTRUE; - } - } - TString opt(qr->GetOptions()); - Ssiz_t id = opt.Last('#'); - if (id != kNPOS && id < opt.Length() - 1) - selec += opt(id + 1, opt.Length()); - - if (!ok) { - Info("ReinitSelector", "problems locating or exporting selector files"); - return -1; - } - } - - // Cleanup previous stuff - SafeDelete(fSelector); - fSelectorClass = 0; - - // Init the selector now - Int_t iglevelsave = gErrorIgnoreLevel; - if (compselec) - // Silent error printout on first attempt - gErrorIgnoreLevel = kBreak; - - if ((fSelector = TSelector::GetSelector(selec))) { - if (compselec) - gErrorIgnoreLevel = iglevelsave; // restore ignore level - fSelectorClass = fSelector->IsA(); - fSelector->SetOption(qr->GetOptions()); - - } else { - if (compselec) { - gErrorIgnoreLevel = iglevelsave; // restore ignore level - // Retry by loading first the libraries listed in TQueryResult, if any - if (strlen(qr->GetLibList()) > 0) { - TString sl(qr->GetLibList()); - TObjArray *oa = sl.Tokenize(" "); - if (oa) { - Bool_t retry = kFALSE; - TIter nxl(oa); - TObjString *os = 0; - while ((os = (TObjString *) nxl())) { - TString lib = gSystem->BaseName(os->GetName()); - if (lib != "lib") { - lib.ReplaceAll("-l", "lib"); - if (gSystem->Load(lib) == 0) - retry = kTRUE; - } - } - // Retry now, if the case - if (retry) - fSelector = TSelector::GetSelector(selec); - } - } - } - if (!fSelector) { - if (compselec) - Info("ReinitSelector", "compiled selector re-init failed:" - " automatic reload unsuccessful:" - " please load manually the correct library"); - rc = -1; - } - } - if (fSelector) { - // Draw needs to reinit temp histos - fSelector->SetInputList(qr->GetInputList()); - if (stdselec) { - ((TProofDraw *)fSelector)->DefVar(); - } else { - // variables may have been initialized in Begin() - fSelector->Begin(0); - } - } - - // Restore original include path, if needed - if (ipathold.Length() > 0) - gSystem->SetIncludePath(ipathold.Data()); - - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Incorporate output object (may not be used in this class). - -Int_t TProofPlayer::AddOutputObject(TObject *) -{ - MayNotUse("AddOutputObject"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Incorporate output list (may not be used in this class). - -void TProofPlayer::AddOutput(TList *) -{ - MayNotUse("AddOutput"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Store output list (may not be used in this class). - -void TProofPlayer::StoreOutput(TList *) -{ - MayNotUse("StoreOutput"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Store feedback list (may not be used in this class). - -void TProofPlayer::StoreFeedback(TObject *, TList *) -{ - MayNotUse("StoreFeedback"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Report progress (may not be used in this class). - -void TProofPlayer::Progress(Long64_t /*total*/, Long64_t /*processed*/) -{ - MayNotUse("Progress"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Report progress (may not be used in this class). - -void TProofPlayer::Progress(Long64_t /*total*/, Long64_t /*processed*/, - Long64_t /*bytesread*/, - Float_t /*evtRate*/, Float_t /*mbRate*/, - Float_t /*evtrti*/, Float_t /*mbrti*/) -{ - MayNotUse("Progress"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Report progress (may not be used in this class). - -void TProofPlayer::Progress(TProofProgressInfo * /*pi*/) -{ - MayNotUse("Progress"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set feedback list (may not be used in this class). - -void TProofPlayer::Feedback(TList *) -{ - MayNotUse("Feedback"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw feedback creation proxy. When accessed via TProof avoids -/// link dependency on libProofPlayer. - -TDrawFeedback *TProofPlayer::CreateDrawFeedback(TProof *p) -{ - return new TDrawFeedback(p); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set draw feedback option. - -void TProofPlayer::SetDrawFeedbackOption(TDrawFeedback *f, Option_t *opt) -{ - if (f) - f->SetOption(opt); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Delete draw feedback object. - -void TProofPlayer::DeleteDrawFeedback(TDrawFeedback *f) -{ - delete f; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save the partial results of this query to a dedicated file under the user -/// data directory. The file name has the form -/// ``.q``.root -/// The file pat and the file are created if not existing already. -/// Only objects in the outputlist not being TProofOutputFile are saved. -/// The packets list 'packets' is saved if given. -/// Trees not attached to any file are attached to the open file. -/// If 'queryend' is kTRUE evrything is written out (TTrees included). -/// The actual saving action is controlled by 'force' and by fSavePartialResults / -/// fSaveResultsPerPacket: -/// -/// fSavePartialResults = kFALSE/kTRUE no-saving/saving -/// fSaveResultsPerPacket = kFALSE/kTRUE save-per-query/save-per-packet -/// -/// The function CheckMemUsage sets fSavePartialResults = 1 if fSaveMemThreshold > 0 and -/// ProcInfo_t::fMemResident >= fSaveMemThreshold: from that point on partial results -/// are always saved and expensive calls to TSystem::GetProcInfo saved. -/// The switch fSaveResultsPerPacket is instead controlled by the user or admin -/// who can also force saving in all cases; parameter PROOF_SavePartialResults or -/// RC env ProofPlayer.SavePartialResults . -/// However, if 'force' is kTRUE, fSavePartialResults and fSaveResultsPerPacket -/// are ignored. -/// Return -1 in case of problems, 0 otherwise. - -Int_t TProofPlayer::SavePartialResults(Bool_t queryend, Bool_t force) -{ - Bool_t save = (force || (fSavePartialResults && - (queryend || fSaveResultsPerPacket))) ? kTRUE : kFALSE; - if (!save) { - PDB(kOutput, 2) - Info("SavePartialResults", "partial result saving disabled"); - return 0; - } - - // Sanity check - if (!gProofServ) { - Error("SavePartialResults", "gProofServ undefined: something really wrong going on!!!"); - return -1; - } - if (!fOutput) { - Error("SavePartialResults", "fOutput undefined: something really wrong going on!!!"); - return -1; - } - - PDB(kOutput, 1) - Info("SavePartialResults", "start saving partial results {%d,%d,%d,%d}", - queryend, force, fSavePartialResults, fSaveResultsPerPacket); - - // Get list of processed packets from the iterator - PDB(kOutput, 2) Info("SavePartialResults", "fEvIter: %p", fEvIter); - - TList *packets = (fEvIter) ? fEvIter->GetPackets() : 0; - PDB(kOutput, 2) Info("SavePartialResults", "list of packets: %p, sz: %d", - packets, (packets ? packets->GetSize(): -1)); - - // Open the file - const char *oopt = "UPDATE"; - // Check if the file has already been defined - TString baseName(fOutputFilePath); - if (fOutputFilePath.IsNull()) { - baseName.Form("output-%s.q%d.root", gProofServ->GetTopSessionTag(), gProofServ->GetQuerySeqNum()); - if (gProofServ->GetDataDirOpts() && strlen(gProofServ->GetDataDirOpts()) > 0) { - fOutputFilePath.Form("%s/%s?%s", gProofServ->GetDataDir(), baseName.Data(), - gProofServ->GetDataDirOpts()); - } else { - fOutputFilePath.Form("%s/%s", gProofServ->GetDataDir(), baseName.Data()); - } - Info("SavePartialResults", "file with (partial) output: '%s'", fOutputFilePath.Data()); - oopt = "RECREATE"; - } - // Open the file in write mode - if (!(fOutputFile = TFile::Open(fOutputFilePath, oopt)) || - (fOutputFile && fOutputFile->IsZombie())) { - Error("SavePartialResults", "cannot open '%s' for writing", fOutputFilePath.Data()); - SafeDelete(fOutputFile); - return -1; - } - - // Save current directory - TDirectory *curdir = gDirectory; - fOutputFile->cd(); - - // Write first the packets list, if required - if (packets) { - TDirectory *packetsDir = fOutputFile->mkdir("packets"); - if (packetsDir) packetsDir->cd(); - packets->Write(0, TObject::kSingleKey | TObject::kOverwrite); - fOutputFile->cd(); - } - - Bool_t notempty = kFALSE; - // Write out the output list - TList torm; - TIter nxo(fOutput); - TObject *o = 0; - while ((o = nxo())) { - // Skip output file drivers - if (o->InheritsFrom(TProofOutputFile::Class())) continue; - // Skip control objets - if (!strncmp(o->GetName(), "PROOF_", 6)) continue; - // Skip data members mapping - if (o->InheritsFrom(TOutputListSelectorDataMap::Class())) continue; - // Skip missing file info - if (!strcmp(o->GetName(), "MissingFiles")) continue; - // Trees need a special treatment - if (o->InheritsFrom("TTree")) { - TTree *t = (TTree *) o; - TDirectory *d = t->GetDirectory(); - // If the tree is not attached to any file ... - if (!d || (d && !d->InheritsFrom("TFile"))) { - // ... we attach it - t->SetDirectory(fOutputFile); - } - if (t->GetDirectory() == fOutputFile) { - if (queryend) { - // ... we write it out - o->Write(0, TObject::kOverwrite); - // At least something in the file - notempty = kTRUE; - // Flag for removal from the outputlist - torm.Add(o); - // Prevent double-deletion attempts - t->SetDirectory(0); - } else { - // ... or we set in automatic flush mode - t->SetAutoFlush(); - } - } - } else if (queryend || fSaveResultsPerPacket) { - // Save overwriting what's already there - o->Write(0, TObject::kOverwrite); - // At least something in the file - notempty = kTRUE; - // Flag for removal from the outputlist - if (queryend) torm.Add(o); - } - } - - // Restore previous directory - gDirectory = curdir; - - // Close the file if required - if (notempty) { - if (!fOutput->FindObject(baseName)) { - TProofOutputFile *po = 0; - // Get directions - TNamed *nm = (TNamed *) fInput->FindObject("PROOF_DefaultOutputOption"); - TString oname = (nm) ? nm->GetTitle() : fOutputFilePath.Data(); - if (nm && oname.BeginsWith("ds:")) { - oname.Replace(0, 3, ""); - TString qtag = - TString::Format("%s_q%d", gProofServ->GetTopSessionTag(), gProofServ->GetQuerySeqNum()); - oname.ReplaceAll("", qtag); - // Create the TProofOutputFile for dataset creation - po = new TProofOutputFile(baseName, "DRO", oname.Data()); - } else { - Bool_t hasddir = kFALSE; - // Create the TProofOutputFile for automatic merging - po = new TProofOutputFile(baseName, "M"); - if (oname.BeginsWith("of:")) oname.Replace(0, 3, ""); - if (gProofServ->IsTopMaster()) { - if (!strcmp(TUrl(oname, kTRUE).GetProtocol(), "file")) { - TString dsrv; - TProofServ::GetLocalServer(dsrv); - TProofServ::FilterLocalroot(oname, dsrv); - oname.Insert(0, dsrv); - } - } else { - if (nm) { - // The name has been sent by the client: resolve local place holders - oname.ReplaceAll("", baseName); - } else { - // We did not get any indication; the final file will be in the datadir on - // the top master and it will be resolved there - oname.Form("/%s", baseName.Data()); - hasddir = kTRUE; - } - } - po->SetOutputFileName(oname.Data()); - if (hasddir) - // Reset the bit, so that has a chance to be resolved in AddOutputObject - po->ResetBit(TProofOutputFile::kOutputFileNameSet); - po->SetName(gSystem->BaseName(oname.Data())); - } - po->AdoptFile(fOutputFile); - fOutput->Add(po); - // Flag the nature of this file - po->SetBit(TProofOutputFile::kSwapFile); - } - } - fOutputFile->Close(); - SafeDelete(fOutputFile); - - // If last call, cleanup the output list from objects saved to file - if (queryend && torm.GetSize() > 0) { - TIter nxrm(&torm); - while ((o = nxrm())) { fOutput->Remove(o); } - } - torm.SetOwner(kFALSE); - - PDB(kOutput, 1) - Info("SavePartialResults", "partial results saved to file"); - // We are done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that a valid selector object -/// Return -1 in case of problems, 0 otherwise - -Int_t TProofPlayer::AssertSelector(const char *selector_file) -{ - if (selector_file && strlen(selector_file)) { - SafeDelete(fSelector); - - // Get selector files from cache - TString ocwd = gSystem->WorkingDirectory(); - if (gProofServ) { - gProofServ->GetCacheLock()->Lock(); - gSystem->ChangeDirectory(gProofServ->GetCacheDir()); - } - - fSelector = TSelector::GetSelector(selector_file); - - if (gProofServ) { - gSystem->ChangeDirectory(ocwd); - gProofServ->GetCacheLock()->Unlock(); - } - - if (!fSelector) { - Error("AssertSelector", "cannot load: %s", selector_file ); - return -1; - } - - fCreateSelObj = kTRUE; - Info("AssertSelector", "Processing via filename (%s)", selector_file); - } else if (!fSelector) { - Error("AssertSelector", "no TSelector object define : cannot continue!"); - return -1; - } else { - Info("AssertSelector", "Processing via TSelector object"); - } - // Done - return 0; -} -//////////////////////////////////////////////////////////////////////////////// -/// Update fProgressStatus - -void TProofPlayer::UpdateProgressInfo() -{ - if (fProgressStatus) { - fProgressStatus->IncEntries(fProcessedRun); - fProgressStatus->SetBytesRead(TFile::GetFileBytesRead()-fReadBytesRun); - fProgressStatus->SetReadCalls(TFile::GetFileReadCalls()-fReadCallsRun); - fProgressStatus->SetLastUpdate(); - if (gMonitoringWriter) - gMonitoringWriter->SendProcessingProgress(fProgressStatus->GetEntries(), - fReadBytesRun, kFALSE); - fProcessedRun = 0; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process specified TDSet on PROOF worker. -/// The return value is -1 in case of error and TSelector::GetStatus() -/// in case of success. - -Long64_t TProofPlayer::Process(TDSet *dset, const char *selector_file, - Option_t *option, Long64_t nentries, - Long64_t first) -{ - PDB(kGlobal,1) Info("Process","Enter"); - - fExitStatus = kFinished; - fOutput = 0; - - TCleanup clean(this); - - fSelectorClass = 0; - TString wmsg; - TRY { - if (AssertSelector(selector_file) != 0 || !fSelector) { - Error("Process", "cannot assert the selector object"); - return -1; - } - - fSelectorClass = fSelector->IsA(); - Int_t version = fSelector->Version(); - if (version == 0 && IsClient()) fSelector->GetOutputList()->Clear(); - - fOutput = (THashList *) fSelector->GetOutputList(); - - if (gProofServ) - TPerfStats::Start(fInput, fOutput); - - fSelStatus = new TStatus; - fOutput->Add(fSelStatus); - - fSelector->SetOption(option); - fSelector->SetInputList(fInput); - - // If in sequential (0-PROOF) mode validate the data set to get - // the number of entries - fTotalEvents = nentries; - if (fTotalEvents < 0 && gProofServ && - gProofServ->IsMaster() && !gProofServ->IsParallel()) { - dset->Validate(); - dset->Reset(); - TDSetElement *e = 0; - while ((e = dset->Next())) { - fTotalEvents += e->GetNum(); - } - } - - dset->Reset(); - - // Set parameters controlling the iterator behaviour - Int_t useTreeCache = 1; - if (TProof::GetParameter(fInput, "PROOF_UseTreeCache", useTreeCache) == 0) { - if (useTreeCache > -1 && useTreeCache < 2) - gEnv->SetValue("ProofPlayer.UseTreeCache", useTreeCache); - } - Long64_t cacheSize = -1; - if (TProof::GetParameter(fInput, "PROOF_CacheSize", cacheSize) == 0) { - TString sz = TString::Format("%lld", cacheSize); - gEnv->SetValue("ProofPlayer.CacheSize", sz.Data()); - } - // Parallel unzipping - Int_t useParallelUnzip = 0; - if (TProof::GetParameter(fInput, "PROOF_UseParallelUnzip", useParallelUnzip) == 0) { - if (useParallelUnzip > -1 && useParallelUnzip < 2) - gEnv->SetValue("ProofPlayer.UseParallelUnzip", useParallelUnzip); - } - // OS file caching (Mac Os X only) - Int_t dontCacheFiles = 0; - if (TProof::GetParameter(fInput, "PROOF_DontCacheFiles", dontCacheFiles) == 0) { - if (dontCacheFiles == 1) - gEnv->SetValue("ProofPlayer.DontCacheFiles", 1); - } - fEvIter = TEventIter::Create(dset, fSelector, first, nentries); - - // Control file object swap - // *10 + - // = 0 end of run - // 1 after each packet - // = 0 no, swap only if memory threshold is reached - // 1 swap in all cases, accordingly to - Int_t opt = 0; - if (TProof::GetParameter(fInput, "PROOF_SavePartialResults", opt) != 0) { - opt = gEnv->GetValue("ProofPlayer.SavePartialResults", 0); - } - fSaveResultsPerPacket = (opt >= 10) ? kTRUE : kFALSE; - fSavePartialResults = (opt%10 > 0) ? kTRUE : kFALSE; - Info("Process", "save partial results? %d per-packet? %d", fSavePartialResults, fSaveResultsPerPacket); - - // Memory threshold for file object swap - Float_t memfrac = gEnv->GetValue("ProofPlayer.SaveMemThreshold", -1.); - if (memfrac > 0.) { - // The threshold is per core - SysInfo_t si; - if (gSystem->GetSysInfo(&si) == 0) { - fSaveMemThreshold = (Long_t) ((memfrac * si.fPhysRam * 1024.) / si.fCpus); - Info("Process", "memory threshold for saving objects to file set to %ld kB", - fSaveMemThreshold); - } else { - Error("Process", "cannot get SysInfo_t (!)"); - } - } - - if (version == 0) { - PDB(kLoop,1) Info("Process","Call Begin(0)"); - fSelector->Begin(0); - } else { - if (IsClient()) { - // on client (for local run) - PDB(kLoop,1) Info("Process","Call Begin(0)"); - fSelector->Begin(0); - } - if (!fSelStatus->TestBit(TStatus::kNotOk)) { - PDB(kLoop,1) Info("Process","Call SlaveBegin(0)"); - fSelector->SlaveBegin(0); // Init is called explicitly - // from GetNextEvent() - } - } - - } CATCH(excode) { - ResetBit(TProofPlayer::kIsProcessing); - Error("Process","exception %d caught", excode); - gProofServ->GetCacheLock()->Unlock(); - return -1; - } ENDTRY; - - // Save the results, if needed, closing the file - if (SavePartialResults(kFALSE) < 0) - Warning("Process", "problems seetting up file-object swapping"); - - // Create feedback lists, if required - SetupFeedback(); - - if (gMonitoringWriter) - gMonitoringWriter->SendProcessingStatus("STARTED",kTRUE); - - PDB(kLoop,1) - Info("Process","Looping over Process()"); - - // get the byte read counter at the beginning of processing - fReadBytesRun = TFile::GetFileBytesRead(); - fReadCallsRun = TFile::GetFileReadCalls(); - fProcessedRun = 0; - // force the first monitoring info - if (gMonitoringWriter) - gMonitoringWriter->SendProcessingProgress(0,0,kTRUE); - - // Start asynchronous timer to dispatch pending events - SetDispatchTimer(kTRUE); - - // Loop over range - gAbort = kFALSE; - Long64_t entry; - fProgressStatus->Reset(); - if (gProofServ) gProofServ->ResetBit(TProofServ::kHighMemory); - - TRY { - - Int_t mrc = -1; - // Get the frequency for checking memory consumption and logging information - Long64_t memlogfreq = -1; - if (((mrc = TProof::GetParameter(fInput, "PROOF_MemLogFreq", memlogfreq))) != 0) memlogfreq = -1; - Long64_t singleshot = 1; - Bool_t warnHWMres = kTRUE, warnHWMvir = kTRUE; - TString lastMsg("(unfortunately no detailed info is available about current packet)"); - - // Initial memory footprint - if (!CheckMemUsage(singleshot, warnHWMres, warnHWMvir, wmsg)) { - Error("Process", "%s", wmsg.Data()); - wmsg.Insert(0, TString::Format("ERROR:%s, after SlaveBegin(), ", gProofServ ? gProofServ->GetOrdinal() : "gProofServ is nullptr")); - fSelStatus->Add(wmsg.Data()); - if (gProofServ) { - gProofServ->SendAsynMessage(wmsg.Data()); - gProofServ->SetBit(TProofServ::kHighMemory); - } - fExitStatus = kStopped; - ResetBit(TProofPlayer::kIsProcessing); - } else if (!wmsg.IsNull()) { - Warning("Process", "%s", wmsg.Data()); - } - - TPair *currentElem = 0; - // The event loop on the worker - Long64_t fst = -1, num; - Long_t maxproctime = -1; - Bool_t newrun = kFALSE; - while ((fEvIter->GetNextPacket(fst, num) != -1) && - !fSelStatus->TestBit(TStatus::kNotOk) && - fSelector->GetAbort() == TSelector::kContinue) { - // This is needed by the inflate infrastructure to calculate - // sleeping times - SetBit(TProofPlayer::kIsProcessing); - - // Give the possibility to the selector to access additional info in the - // incoming packet - if (dset->Current()) { - if (!currentElem) { - currentElem = new TPair(new TObjString("PROOF_CurrentElement"), dset->Current()); - fInput->Add(currentElem); - } else { - if (currentElem->Value() != dset->Current()) { - currentElem->SetValue(dset->Current()); - } else if (dset->Current()->TestBit(TDSetElement::kNewRun)) { - dset->Current()->ResetBit(TDSetElement::kNewRun); - } - } - if (dset->Current()->TestBit(TDSetElement::kNewPacket)) { - if (dset->TestBit(TDSet::kEmpty)) { - lastMsg = "check logs for possible stacktrace - last cycle:"; - } else { - TDSetElement *elem = dynamic_cast(currentElem->Value()); - TString fn = (elem) ? elem->GetFileName() : ""; - lastMsg.Form("while processing dset:'%s', file:'%s'" - " - check logs for possible stacktrace - last event:", dset->GetName(), fn.Data()); - } - TProofServ::SetLastMsg(lastMsg); - } - // Set the max proc time, if any - if (dset->Current()->GetMaxProcTime() >= 0.) - maxproctime = (Long_t) (1000 * dset->Current()->GetMaxProcTime()); - newrun = (dset->Current()->TestBit(TDSetElement::kNewPacket)) ? kTRUE : kFALSE; - } - - ResetBit(TProofPlayer::kMaxProcTimeReached); - ResetBit(TProofPlayer::kMaxProcTimeExtended); - // Setup packet proc time measurement - if (maxproctime > 0) { - if (!fProcTimeTimer) fProcTimeTimer = new TProctimeTimer(this, maxproctime); - fProcTimeTimer->Start(maxproctime, kTRUE); // One shot - if (!fProcTime) fProcTime = new TStopwatch(); - fProcTime->Reset(); // Reset counters - } - Long64_t refnum = num; - if (refnum < 0 && maxproctime <= 0) { - wmsg.Form("neither entries nor max proc time specified:" - " risk of infinite loop: processing aborted"); - Error("Process", "%s", wmsg.Data()); - if (gProofServ) { - wmsg.Insert(0, TString::Format("ERROR:%s, entry:%lld, ", - gProofServ->GetOrdinal(), fProcessedRun)); - gProofServ->SendAsynMessage(wmsg.Data()); - } - fExitStatus = kAborted; - ResetBit(TProofPlayer::kIsProcessing); - break; - } - while (refnum < 0 || num--) { - - // Did we use all our time? - if (TestBit(TProofPlayer::kMaxProcTimeReached)) { - fProcTime->Stop(); - if (!newrun && !TestBit(TProofPlayer::kMaxProcTimeExtended) && refnum > 0) { - // How much are we left with? - Float_t xleft = (refnum > num) ? (Float_t) num / (Float_t) (refnum) : 1.; - if (xleft < 0.2) { - // Give another try, 1.5 times the remaining measured expected time - Long_t mpt = (Long_t) (1500 * num / ((Double_t)(refnum - num) / fProcTime->RealTime())); - SetBit(TProofPlayer::kMaxProcTimeExtended); - fProcTimeTimer->Start(mpt, kTRUE); // One shot - ResetBit(TProofPlayer::kMaxProcTimeReached); - } - } - if (TestBit(TProofPlayer::kMaxProcTimeReached)) { - Info("Process", "max proc time reached (%ld msecs): packet processing stopped:\n%s", - maxproctime, lastMsg.Data()); - - break; - } - } - - if (!(!fSelStatus->TestBit(TStatus::kNotOk) && - fSelector->GetAbort() == TSelector::kContinue)) break; - - // Get the netry number, taking into account entry or event lists - entry = fEvIter->GetEntryNumber(fst); - fst++; - - // Set the last entry - TProofServ::SetLastEntry(entry); - - if (fSelector->Version() == 0) { - PDB(kLoop,3) - Info("Process","Call ProcessCut(%lld)", entry); - if (fSelector->ProcessCut(entry)) { - PDB(kLoop,3) - Info("Process","Call ProcessFill(%lld)", entry); - fSelector->ProcessFill(entry); - } - } else { - PDB(kLoop,3) - Info("Process","Call Process(%lld)", entry); - fSelector->Process(entry); - if (fSelector->GetAbort() == TSelector::kAbortProcess) { - ResetBit(TProofPlayer::kIsProcessing); - break; - } else if (fSelector->GetAbort() == TSelector::kAbortFile) { - Info("Process", "packet processing aborted following the" - " selector settings:\n%s", lastMsg.Data()); - fEvIter->InvalidatePacket(); - fProgressStatus->SetBit(TProofProgressStatus::kFileCorrupted); - } - } - if (!fSelStatus->TestBit(TStatus::kNotOk)) fProcessedRun++; - - // Check the memory footprint, if required - if (memlogfreq > 0 && (GetEventsProcessed() + fProcessedRun)%memlogfreq == 0) { - if (!CheckMemUsage(memlogfreq, warnHWMres, warnHWMvir, wmsg)) { - Error("Process", "%s", wmsg.Data()); - if (gProofServ) { - wmsg.Insert(0, TString::Format("ERROR:%s, entry:%lld, ", - gProofServ->GetOrdinal(), entry)); - gProofServ->SendAsynMessage(wmsg.Data()); - } - fExitStatus = kStopped; - ResetBit(TProofPlayer::kIsProcessing); - if (gProofServ) gProofServ->SetBit(TProofServ::kHighMemory); - break; - } else { - if (!wmsg.IsNull()) { - Warning("Process", "%s", wmsg.Data()); - if (gProofServ) { - wmsg.Insert(0, TString::Format("WARNING:%s, entry:%lld, ", - gProofServ->GetOrdinal(), entry)); - gProofServ->SendAsynMessage(wmsg.Data()); - } - } - } - } - if (TestBit(TProofPlayer::kDispatchOneEvent)) { - gSystem->DispatchOneEvent(kTRUE); - ResetBit(TProofPlayer::kDispatchOneEvent); - } - ResetBit(TProofPlayer::kIsProcessing); - if (fSelStatus->TestBit(TStatus::kNotOk) || gROOT->IsInterrupted()) break; - - // Make sure that the selector abort status is reset - if (fSelector->GetAbort() == TSelector::kAbortFile) - fSelector->Abort("status reset", TSelector::kContinue); - } - } - - } CATCH(excode) { - if (excode == kPEX_STOPPED) { - Info("Process","received stop-process signal"); - fExitStatus = kStopped; - } else if (excode == kPEX_ABORTED) { - gAbort = kTRUE; - Info("Process","received abort-process signal"); - fExitStatus = kAborted; - } else { - Error("Process","exception %d caught", excode); - // Perhaps we need a dedicated status code here ... - gAbort = kTRUE; - fExitStatus = kAborted; - } - ResetBit(TProofPlayer::kIsProcessing); - } ENDTRY; - - // Clean-up the envelop for the current element - TPair *currentElem = 0; - if ((currentElem = (TPair *) fInput->FindObject("PROOF_CurrentElement"))) { - if ((currentElem = (TPair *) fInput->Remove(currentElem))) { - delete currentElem->Key(); - delete currentElem; - } - } - - // Final memory footprint - Long64_t singleshot = 1; - Bool_t warnHWMres = kTRUE, warnHWMvir = kTRUE; - Bool_t shrc = CheckMemUsage(singleshot, warnHWMres, warnHWMvir, wmsg); - if (!wmsg.IsNull()) Warning("Process", "%s (%s)", wmsg.Data(), shrc ? "warn" : "hwm"); - - PDB(kGlobal,2) - Info("Process","%lld events processed", fProgressStatus->GetEntries()); - - if (gMonitoringWriter) { - gMonitoringWriter->SendProcessingProgress(fProgressStatus->GetEntries(), - TFile::GetFileBytesRead()-fReadBytesRun, kFALSE); - gMonitoringWriter->SendProcessingStatus("DONE"); - } - - // Stop active timers - SetDispatchTimer(kFALSE); - if (fStopTimer != 0) - SetStopTimer(kFALSE, gAbort); - if (fFeedbackTimer != 0) - HandleTimer(0); - - StopFeedback(); - - // Save the results, if needed, closing the file - if (SavePartialResults(kTRUE) < 0) - Warning("Process", "problems saving the results to file"); - - SafeDelete(fEvIter); - - // Finalize - - if (fExitStatus != kAborted) { - - TIter nxo(GetOutputList()); - TObject *o = 0; - while ((o = nxo())) { - // Special treatment for files - if (o->IsA() == TProofOutputFile::Class()) { - TProofOutputFile *of = (TProofOutputFile *)o; - of->Print(); - of->SetWorkerOrdinal(gProofServ->GetOrdinal()); - const char *dir = of->GetDir(); - if (!dir || (dir && strlen(dir) <= 0)) { - of->SetDir(gProofServ->GetSessionDir()); - } else if (dir && strlen(dir) > 0) { - TUrl u(dir); - if (!strcmp(u.GetHost(), "localhost") || !strcmp(u.GetHost(), "127.0.0.1") || - !strcmp(u.GetHost(), "localhost.localdomain")) { - u.SetHost(TUrl(gSystem->HostName()).GetHostFQDN()); - of->SetDir(u.GetUrl(kTRUE)); - } - of->Print(); - } - } - } - - MapOutputListToDataMembers(); - - if (!fSelStatus->TestBit(TStatus::kNotOk)) { - if (fSelector->Version() == 0) { - PDB(kLoop,1) Info("Process","Call Terminate()"); - fSelector->Terminate(); - } else { - PDB(kLoop,1) Info("Process","Call SlaveTerminate()"); - fSelector->SlaveTerminate(); - if (IsClient() && !fSelStatus->TestBit(TStatus::kNotOk)) { - PDB(kLoop,1) Info("Process","Call Terminate()"); - fSelector->Terminate(); - } - } - } - - // Add Selector status in the output list so it can be returned to the client as done - // by Tree::Process (see ROOT-748). The status from the various workers will be added. - fOutput->Add(new TParameter("PROOF_SelectorStatus", (Long64_t) fSelector->GetStatus())); - - if (gProofServ && !gProofServ->IsParallel()) { // put all the canvases onto the output list - TIter nxc(gROOT->GetListOfCanvases()); - while (TObject *c = nxc()) - fOutput->Add(c); - } - } - - if (gProofServ) - TPerfStats::Stop(); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process specified TDSet on PROOF worker with TSelector object -/// The return value is -1 in case of error and TSelector::GetStatus() -/// in case of success. - -Long64_t TProofPlayer::Process(TDSet *dset, TSelector *selector, - Option_t *option, Long64_t nentries, - Long64_t first) -{ - if (!selector) { - Error("Process", "selector object undefiend!"); - return -1; - } - - SafeDelete(fSelector); - fSelector = selector; - fCreateSelObj = kFALSE; - return Process(dset, (const char *)0, option, nentries, first); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Not implemented: meaningful only in the remote player. Returns kFALSE. - -Bool_t TProofPlayer::JoinProcess(TList *) -{ - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check the memory usage, if requested. -/// Return kTRUE if OK, kFALSE if above 95% of at least one between virtual or -/// resident limits are depassed. - -Bool_t TProofPlayer::CheckMemUsage(Long64_t &mfreq, Bool_t &w80r, - Bool_t &w80v, TString &wmsg) -{ - Long64_t processed = GetEventsProcessed() + fProcessedRun; - if (mfreq > 0 && processed%mfreq == 0) { - // Record the memory information - ProcInfo_t pi; - if (!gSystem->GetProcInfo(&pi)){ - wmsg = ""; - if (gProofServ) - Info("CheckMemUsage|Svc", "Memory %ld virtual %ld resident event %lld", - pi.fMemVirtual, pi.fMemResident, processed); - // Save info in TStatus - fSelStatus->SetMemValues(pi.fMemVirtual, pi.fMemResident); - // Apply limit on virtual memory, if any: warn if above 80%, stop if above 95% of max - if (TProofServ::GetVirtMemMax() > 0) { - if (pi.fMemVirtual > TProofServ::GetMemStop() * TProofServ::GetVirtMemMax()) { - wmsg.Form("using more than %d%% of allowed virtual memory (%ld kB)" - " - STOP processing", (Int_t) (TProofServ::GetMemStop() * 100), pi.fMemVirtual); - return kFALSE; - } else if (pi.fMemVirtual > TProofServ::GetMemHWM() * TProofServ::GetVirtMemMax() && w80v) { - // Refine monitoring - mfreq = 1; - wmsg.Form("using more than %d%% of allowed virtual memory (%ld kB)", - (Int_t) (TProofServ::GetMemHWM() * 100), pi.fMemVirtual); - w80v = kFALSE; - } - } - // Apply limit on resident memory, if any: warn if above 80%, stop if above 95% of max - if (TProofServ::GetResMemMax() > 0) { - if (pi.fMemResident > TProofServ::GetMemStop() * TProofServ::GetResMemMax()) { - wmsg.Form("using more than %d%% of allowed resident memory (%ld kB)" - " - STOP processing", (Int_t) (TProofServ::GetMemStop() * 100), pi.fMemResident); - return kFALSE; - } else if (pi.fMemResident > TProofServ::GetMemHWM() * TProofServ::GetResMemMax() && w80r) { - // Refine monitoring - mfreq = 1; - if (wmsg.Length() > 0) { - wmsg.Form("using more than %d%% of allowed both virtual and resident memory ({%ld,%ld} kB)", - (Int_t) (TProofServ::GetMemHWM() * 100), pi.fMemVirtual, pi.fMemResident); - } else { - wmsg.Form("using more than %d%% of allowed resident memory (%ld kB)", - (Int_t) (TProofServ::GetMemHWM() * 100), pi.fMemResident); - } - w80r = kFALSE; - } - } - // In saving-partial-results mode flag the saving regime when reached to save expensive calls - // to TSystem::GetProcInfo in SavePartialResults - if (fSaveMemThreshold > 0 && pi.fMemResident >= fSaveMemThreshold) fSavePartialResults = kTRUE; - } - } - // Done - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize query (may not be used in this class). - -Long64_t TProofPlayer::Finalize(Bool_t, Bool_t) -{ - MayNotUse("Finalize"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize query (may not be used in this class). - -Long64_t TProofPlayer::Finalize(TQueryResult *) -{ - MayNotUse("Finalize"); - return -1; -} -//////////////////////////////////////////////////////////////////////////////// -/// Merge output (may not be used in this class). - -void TProofPlayer::MergeOutput(Bool_t) -{ - MayNotUse("MergeOutput"); - return; -} - -//////////////////////////////////////////////////////////////////////////////// - -void TProofPlayer::MapOutputListToDataMembers() const -{ - TOutputListSelectorDataMap* olsdm = new TOutputListSelectorDataMap(fSelector); - fOutput->Add(olsdm); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Update automatic binning parameters for given object "name". - -void TProofPlayer::UpdateAutoBin(const char *name, - Double_t& xmin, Double_t& xmax, - Double_t& ymin, Double_t& ymax, - Double_t& zmin, Double_t& zmax) -{ - if ( fAutoBins == 0 ) { - fAutoBins = new THashList; - } - - TAutoBinVal *val = (TAutoBinVal*) fAutoBins->FindObject(name); - - if ( val == 0 ) { - //look for info in higher master - if (gProofServ && !gProofServ->IsTopMaster()) { - TString key = name; - TProofLimitsFinder::AutoBinFunc(key,xmin,xmax,ymin,ymax,zmin,zmax); - } - - val = new TAutoBinVal(name,xmin,xmax,ymin,ymax,zmin,zmax); - fAutoBins->Add(val); - } else { - val->GetAll(xmin,xmax,ymin,ymax,zmin,zmax); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next packet (may not be used in this class). - -TDSetElement *TProofPlayer::GetNextPacket(TSlave *, TMessage *) -{ - MayNotUse("GetNextPacket"); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set up feedback (may not be used in this class). - -void TProofPlayer::SetupFeedback() -{ - MayNotUse("SetupFeedback"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Stop feedback (may not be used in this class). - -void TProofPlayer::StopFeedback() -{ - MayNotUse("StopFeedback"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw (may not be used in this class). - -Long64_t TProofPlayer::DrawSelect(TDSet * /*set*/, const char * /*varexp*/, - const char * /*selection*/, Option_t * /*option*/, - Long64_t /*nentries*/, Long64_t /*firstentry*/) -{ - MayNotUse("DrawSelect"); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle tree header request. - -void TProofPlayer::HandleGetTreeHeader(TMessage *) -{ - MayNotUse("HandleGetTreeHeader|"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Receive histo from slave. - -void TProofPlayer::HandleRecvHisto(TMessage *mess) -{ - TObject *obj = mess->ReadObject(mess->GetClass()); - if (obj->InheritsFrom(TH1::Class())) { - TH1 *h = (TH1*)obj; - h->SetDirectory(0); - TH1 *horg = (TH1*)gDirectory->GetList()->FindObject(h->GetName()); - if (horg) - horg->Add(h); - else - h->SetDirectory(gDirectory); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw the object if it is a canvas. -/// Return 0 in case of success, 1 if it is not a canvas or libProofDraw -/// is not available. - -Int_t TProofPlayer::DrawCanvas(TObject *obj) -{ - static Int_t (*gDrawCanvasHook)(TObject *) = 0; - - // Load the library the first time - if (!gDrawCanvasHook) { - // Load library needed for graphics ... - TString drawlib = "libProofDraw"; - char *p = 0; - if ((p = gSystem->DynamicPathName(drawlib, kTRUE))) { - delete[] p; - if (gSystem->Load(drawlib) != -1) { - // Locate DrawCanvas - Func_t f = 0; - if ((f = gSystem->DynFindSymbol(drawlib,"DrawCanvas"))) - gDrawCanvasHook = (Int_t (*)(TObject *))(f); - else - Warning("DrawCanvas", "can't find DrawCanvas"); - } else - Warning("DrawCanvas", "can't load %s", drawlib.Data()); - } else - Warning("DrawCanvas", "can't locate %s", drawlib.Data()); - } - if (gDrawCanvasHook && obj) - return (*gDrawCanvasHook)(obj); - // No drawing hook or object undefined - return 1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Parse the arguments from var, sel and opt and fill the selector and -/// object name accordingly. -/// Return 0 in case of success, 1 if libProofDraw is not available. - -Int_t TProofPlayer::GetDrawArgs(const char *var, const char *sel, Option_t *opt, - TString &selector, TString &objname) -{ - static Int_t (*gGetDrawArgsHook)(const char *, const char *, Option_t *, - TString &, TString &) = 0; - - // Load the library the first time - if (!gGetDrawArgsHook) { - // Load library needed for graphics ... - TString drawlib = "libProofDraw"; - char *p = 0; - if ((p = gSystem->DynamicPathName(drawlib, kTRUE))) { - delete[] p; - if (gSystem->Load(drawlib) != -1) { - // Locate GetDrawArgs - Func_t f = 0; - if ((f = gSystem->DynFindSymbol(drawlib,"GetDrawArgs"))) - gGetDrawArgsHook = (Int_t (*)(const char *, const char *, Option_t *, - TString &, TString &))(f); - else - Warning("GetDrawArgs", "can't find GetDrawArgs"); - } else - Warning("GetDrawArgs", "can't load %s", drawlib.Data()); - } else - Warning("GetDrawArgs", "can't locate %s", drawlib.Data()); - } - if (gGetDrawArgsHook) - return (*gGetDrawArgsHook)(var, sel, opt, selector, objname); - // No parser hook or object undefined - return 1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Create/destroy a named canvas for feedback - -void TProofPlayer::FeedBackCanvas(const char *name, Bool_t create) -{ - static void (*gFeedBackCanvasHook)(const char *, Bool_t) = 0; - - // Load the library the first time - if (!gFeedBackCanvasHook) { - // Load library needed for graphics ... - TString drawlib = "libProofDraw"; - char *p = 0; - if ((p = gSystem->DynamicPathName(drawlib, kTRUE))) { - delete[] p; - if (gSystem->Load(drawlib) != -1) { - // Locate FeedBackCanvas - Func_t f = 0; - if ((f = gSystem->DynFindSymbol(drawlib,"FeedBackCanvas"))) - gFeedBackCanvasHook = (void (*)(const char *, Bool_t))(f); - else - Warning("FeedBackCanvas", "can't find FeedBackCanvas"); - } else - Warning("FeedBackCanvas", "can't load %s", drawlib.Data()); - } else - Warning("FeedBackCanvas", "can't locate %s", drawlib.Data()); - } - if (gFeedBackCanvasHook) (*gFeedBackCanvasHook)(name, create); - // No parser hook or object undefined - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the size in bytes of the cache - -Long64_t TProofPlayer::GetCacheSize() -{ - if (fEvIter) return fEvIter->GetCacheSize(); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the number of entries in the learning phase - -Int_t TProofPlayer::GetLearnEntries() -{ - if (fEvIter) return fEvIter->GetLearnEntries(); - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Switch on/off merge timer - -void TProofPlayerRemote::SetMerging(Bool_t on) -{ - if (on) { - if (!fMergeSTW) fMergeSTW = new TStopwatch(); - PDB(kGlobal,1) - Info("SetMerging", "ON: mergers: %d", fProof->fMergersCount); - if (fNumMergers <= 0 && fProof->fMergersCount > 0) - fNumMergers = fProof->fMergersCount; - } else if (fMergeSTW) { - fMergeSTW->Stop(); - Float_t rt = fMergeSTW->RealTime(); - PDB(kGlobal,1) - Info("SetMerging", "OFF: rt: %f, mergers: %d", rt, fNumMergers); - if (fQuery) { - if (!fProof->TestBit(TProof::kIsClient) || fProof->IsLite()) { - // On the master (or in Lite()) we set the merging time and the numebr of mergers - fQuery->SetMergeTime(rt); - fQuery->SetNumMergers(fNumMergers); - } else { - // In a standard client we save the transfer-to-client time - fQuery->SetRecvTime(rt); - } - PDB(kGlobal,2) fQuery->Print("F"); - } - } -} - -//------------------------------------------------------------------------------ - -ClassImp(TProofPlayerLocal); - -//////////////////////////////////////////////////////////////////////////////// -/// Process the specified TSelector object 'nentries' times. -/// Used to test the PROOF interator mechanism for cycle-driven selectors in a -/// local session. -/// The return value is -1 in case of error and TSelector::GetStatus() -/// in case of success. - -Long64_t TProofPlayerLocal::Process(TSelector *selector, - Long64_t nentries, Option_t *option) -{ - if (!selector) { - Error("Process", "selector object undefiend!"); - return -1; - } - - TDSetProxy *set = new TDSetProxy("", "", ""); - set->SetBit(TDSet::kEmpty); - set->SetBit(TDSet::kIsLocal); - Long64_t rc = Process(set, selector, option, nentries); - SafeDelete(set); - - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process the specified TSelector file 'nentries' times. -/// Used to test the PROOF interator mechanism for cycle-driven selectors in a -/// local session. -/// Process specified TDSet on PROOF worker with TSelector object -/// The return value is -1 in case of error and TSelector::GetStatus() -/// in case of success. - -Long64_t TProofPlayerLocal::Process(const char *selector, - Long64_t nentries, Option_t *option) -{ - TDSetProxy *set = new TDSetProxy("", "", ""); - set->SetBit(TDSet::kEmpty); - set->SetBit(TDSet::kIsLocal); - Long64_t rc = Process(set, selector, option, nentries); - SafeDelete(set); - - // Done - return rc; -} - - -//------------------------------------------------------------------------------ - -ClassImp(TProofPlayerRemote); - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TProofPlayerRemote::~TProofPlayerRemote() -{ - SafeDelete(fOutput); // owns the output list - SafeDelete(fOutputLists); - - // Objects stored in maps are already deleted when merging the feedback - SafeDelete(fFeedbackLists); - SafeDelete(fPacketizer); - - SafeDelete(fProcessMessage); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Init the packetizer -/// Return 0 on success (fPacketizer is correctly initialized), -1 on failure. - -Int_t TProofPlayerRemote::InitPacketizer(TDSet *dset, Long64_t nentries, - Long64_t first, const char *defpackunit, - const char *defpackdata) -{ - SafeDelete(fPacketizer); - PDB(kGlobal,1) Info("Process","Enter"); - fDSet = dset; - fExitStatus = kFinished; - - // This is done here to pickup on the fly changes - Int_t honebyone = 1; - if (TProof::GetParameter(fInput, "PROOF_MergeTH1OneByOne", honebyone) != 0) - honebyone = gEnv->GetValue("ProofPlayer.MergeTH1OneByOne", 1); - fMergeTH1OneByOne = (honebyone == 1) ? kTRUE : kFALSE; - - Bool_t noData = dset->TestBit(TDSet::kEmpty) ? kTRUE : kFALSE; - - TString packetizer; - TList *listOfMissingFiles = 0; - - TMethodCall callEnv; - TClass *cl; - noData = dset->TestBit(TDSet::kEmpty) ? kTRUE : kFALSE; - - if (noData) { - - if (TProof::GetParameter(fInput, "PROOF_Packetizer", packetizer) != 0) - packetizer = defpackunit; - else - Info("InitPacketizer", "using alternate packetizer: %s", packetizer.Data()); - - // Get linked to the related class - cl = TClass::GetClass(packetizer); - if (cl == 0) { - Error("InitPacketizer", "class '%s' not found", packetizer.Data()); - fExitStatus = kAborted; - return -1; - } - - // Init the constructor - callEnv.InitWithPrototype(cl, cl->GetName(),"TList*,Long64_t,TList*,TProofProgressStatus*"); - if (!callEnv.IsValid()) { - Error("InitPacketizer", - "cannot find correct constructor for '%s'", cl->GetName()); - fExitStatus = kAborted; - return -1; - } - callEnv.ResetParam(); - callEnv.SetParam((Longptr_t) fProof->GetListOfActiveSlaves()); - callEnv.SetParam((Long64_t) nentries); - callEnv.SetParam((Longptr_t) fInput); - callEnv.SetParam((Longptr_t) fProgressStatus); - - } else if (dset->TestBit(TDSet::kMultiDSet)) { - - // We have to process many datasets in one go, keeping them separate - if (fProof->GetRunStatus() != TProof::kRunning) { - // We have been asked to stop - Error("InitPacketizer", "received stop/abort request"); - fExitStatus = kAborted; - return -1; - } - - // The multi packetizer - packetizer = "TPacketizerMulti"; - - // Get linked to the related class - cl = TClass::GetClass(packetizer); - if (cl == 0) { - Error("InitPacketizer", "class '%s' not found", packetizer.Data()); - fExitStatus = kAborted; - return -1; - } - - // Init the constructor - callEnv.InitWithPrototype(cl, cl->GetName(),"TDSet*,TList*,Long64_t,Long64_t,TList*,TProofProgressStatus*"); - if (!callEnv.IsValid()) { - Error("InitPacketizer", "cannot find correct constructor for '%s'", cl->GetName()); - fExitStatus = kAborted; - return -1; - } - callEnv.ResetParam(); - callEnv.SetParam((Longptr_t) dset); - callEnv.SetParam((Longptr_t) fProof->GetListOfActiveSlaves()); - callEnv.SetParam((Long64_t) first); - callEnv.SetParam((Long64_t) nentries); - callEnv.SetParam((Longptr_t) fInput); - callEnv.SetParam((Longptr_t) fProgressStatus); - - // We are going to test validity during the packetizer initialization - dset->SetBit(TDSet::kValidityChecked); - dset->ResetBit(TDSet::kSomeInvalid); - - } else { - - // Lookup - resolve the end-point urls to optmize the distribution. - // The lookup was previously called in the packetizer's constructor. - // A list for the missing files may already have been added to the - // output list; otherwise, if needed it will be created inside - if ((listOfMissingFiles = (TList *)fInput->FindObject("MissingFiles"))) { - // Move it to the output list - fInput->Remove(listOfMissingFiles); - } else { - listOfMissingFiles = new TList; - } - // Do the lookup; we only skip it if explicitly requested so. - TString lkopt; - if (TProof::GetParameter(fInput, "PROOF_LookupOpt", lkopt) != 0 || lkopt != "none") - dset->Lookup(kTRUE, &listOfMissingFiles); - - if (fProof->GetRunStatus() != TProof::kRunning) { - // We have been asked to stop - Error("InitPacketizer", "received stop/abort request"); - fExitStatus = kAborted; - return -1; - } - - if (!(dset->GetListOfElements()) || - !(dset->GetListOfElements()->GetSize())) { - if (gProofServ) - gProofServ->SendAsynMessage("InitPacketizer: No files from the data set were found - Aborting"); - Error("InitPacketizer", "No files from the data set were found - Aborting"); - fExitStatus = kAborted; - if (listOfMissingFiles) { - listOfMissingFiles->SetOwner(); - fOutput->Remove(listOfMissingFiles); - SafeDelete(listOfMissingFiles); - } - return -1; - } - - if (TProof::GetParameter(fInput, "PROOF_Packetizer", packetizer) != 0) - // Using standard packetizer TAdaptivePacketizer - packetizer = defpackdata; - else - Info("InitPacketizer", "using alternate packetizer: %s", packetizer.Data()); - - // Get linked to the related class - cl = TClass::GetClass(packetizer); - if (cl == 0) { - Error("InitPacketizer", "class '%s' not found", packetizer.Data()); - fExitStatus = kAborted; - return -1; - } - - // Init the constructor - callEnv.InitWithPrototype(cl, cl->GetName(),"TDSet*,TList*,Long64_t,Long64_t,TList*,TProofProgressStatus*"); - if (!callEnv.IsValid()) { - Error("InitPacketizer", "cannot find correct constructor for '%s'", cl->GetName()); - fExitStatus = kAborted; - return -1; - } - callEnv.ResetParam(); - callEnv.SetParam((Longptr_t) dset); - callEnv.SetParam((Longptr_t) fProof->GetListOfActiveSlaves()); - callEnv.SetParam((Long64_t) first); - callEnv.SetParam((Long64_t) nentries); - callEnv.SetParam((Longptr_t) fInput); - callEnv.SetParam((Longptr_t) fProgressStatus); - - // We are going to test validity during the packetizer initialization - dset->SetBit(TDSet::kValidityChecked); - dset->ResetBit(TDSet::kSomeInvalid); - } - - // Get an instance of the packetizer - Longptr_t ret = 0; - callEnv.Execute(ret); - if ((fPacketizer = (TVirtualPacketizer *)ret) == 0) { - Error("InitPacketizer", "cannot construct '%s'", cl->GetName()); - fExitStatus = kAborted; - return -1; - } - - if (!fPacketizer->IsValid()) { - Error("InitPacketizer", - "instantiated packetizer object '%s' is invalid", cl->GetName()); - fExitStatus = kAborted; - SafeDelete(fPacketizer); - return -1; - } - - // In multi mode retrieve the list of missing files - if (!noData && dset->TestBit(TDSet::kMultiDSet)) { - if ((listOfMissingFiles = (TList *) fInput->FindObject("MissingFiles"))) { - // Remove it; it will be added to the output list - fInput->Remove(listOfMissingFiles); - } - } - - if (!noData) { - // Add invalid elements to the list of missing elements - TDSetElement *elem = 0; - if (dset->TestBit(TDSet::kSomeInvalid)) { - TIter nxe(dset->GetListOfElements()); - while ((elem = (TDSetElement *)nxe())) { - if (!elem->GetValid()) { - if (!listOfMissingFiles) - listOfMissingFiles = new TList; - listOfMissingFiles->Add(elem->GetFileInfo(dset->GetType())); - dset->Remove(elem, kFALSE); - } - } - // The invalid elements have been removed - dset->ResetBit(TDSet::kSomeInvalid); - } - - // Record the list of missing or invalid elements in the output list - if (listOfMissingFiles && listOfMissingFiles->GetSize() > 0) { - TIter missingFiles(listOfMissingFiles); - TString msg; - if (gDebug > 0) { - TFileInfo *fi = 0; - while ((fi = (TFileInfo *) missingFiles.Next())) { - if (fi->GetCurrentUrl()) { - msg = Form("File not found: %s - skipping!", - fi->GetCurrentUrl()->GetUrl()); - } else { - msg = Form("File not found: %s - skipping!", fi->GetName()); - } - if (gProofServ) gProofServ->SendAsynMessage(msg.Data()); - } - } - // Make sure it will be sent back - if (!GetOutput("MissingFiles")) { - listOfMissingFiles->SetName("MissingFiles"); - AddOutputObject(listOfMissingFiles); - } - TStatus *tmpStatus = (TStatus *)GetOutput("PROOF_Status"); - if (!tmpStatus) AddOutputObject((tmpStatus = new TStatus())); - - // Estimate how much data are missing - Int_t ngood = dset->GetListOfElements()->GetSize(); - Int_t nbad = listOfMissingFiles->GetSize(); - Double_t xb = Double_t(nbad) / Double_t(ngood + nbad); - msg = Form(" About %.2f %c of the requested files (%d out of %d) were missing or unusable; details in" - " the 'missingFiles' list", xb * 100., '%', nbad, nbad + ngood); - tmpStatus->Add(msg.Data()); - msg = Form(" +++\n" - " +++ About %.2f %c of the requested files (%d out of %d) are missing or unusable; details in" - " the 'MissingFiles' list\n" - " +++", xb * 100., '%', nbad, nbad + ngood); - if (gProofServ) gProofServ->SendAsynMessage(msg.Data()); - } else { - // Cleanup - SafeDelete(listOfMissingFiles); - } - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process specified TDSet on PROOF. -/// This method is called on client and on the PROOF master. -/// The return value is -1 in case of an error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProofPlayerRemote::Process(TDSet *dset, const char *selector_file, - Option_t *option, Long64_t nentries, - Long64_t first) -{ - PDB(kGlobal,1) Info("Process", "Enter"); - - fDSet = dset; - fExitStatus = kFinished; - - if (!fProgressStatus) { - Error("Process", "No progress status"); - return -1; - } - fProgressStatus->Reset(); - - // delete fOutput; - if (!fOutput) - fOutput = new THashList; - else - fOutput->Clear(); - - SafeDelete(fFeedbackLists); - - if (fProof->IsMaster()){ - TPerfStats::Start(fInput, fOutput); - } else { - TPerfStats::Setup(fInput); - } - - TStopwatch elapsed; - - // Define filename - TString fn; - fSelectorFileName = selector_file; - - if (fCreateSelObj) { - if(!SendSelector(selector_file)) return -1; - fn = gSystem->BaseName(selector_file); - } else { - fn = selector_file; - } - - TMessage mesg(kPROOF_PROCESS); - - // Parse option - Bool_t sync = (fProof->GetQueryMode(option) == TProof::kSync); - - TList *inputtmp = 0; // List of temporary input objects - TDSet *set = dset; - if (fProof->IsMaster()) { - - PDB(kPacketizer,1) Info("Process","Create Proxy TDSet"); - set = new TDSetProxy( dset->GetType(), dset->GetObjName(), - dset->GetDirectory() ); - if (dset->TestBit(TDSet::kEmpty)) - set->SetBit(TDSet::kEmpty); - - if (InitPacketizer(dset, nentries, first, "TPacketizerUnit", "TPacketizer") != 0) { - Error("Process", "cannot init the packetizer"); - fExitStatus = kAborted; - return -1; - } - - // Reset start, this is now managed by the packetizer - first = 0; - - // Negative memlogfreq disable checks. - // If 0 is passed we try to have 100 messages about memory - // Otherwise we use the frequency passed. - Int_t mrc = -1; - Long64_t memlogfreq = -1, mlf; - if (gSystem->Getenv("PROOF_MEMLOGFREQ")) { - TString clf(gSystem->Getenv("PROOF_MEMLOGFREQ")); - if (clf.IsDigit()) { memlogfreq = clf.Atoi(); mrc = 0; } - } - if ((mrc = TProof::GetParameter(fProof->GetInputList(), "PROOF_MemLogFreq", mlf)) == 0) memlogfreq = mlf; - if (memlogfreq == 0) { - memlogfreq = fPacketizer->GetTotalEntries()/(fProof->GetParallel()*100); - if (memlogfreq <= 0) memlogfreq = 1; - } - if (mrc == 0) fProof->SetParameter("PROOF_MemLogFreq", memlogfreq); - - - // Send input data, if any - TString emsg; - if (TProof::SendInputData(fQuery, fProof, emsg) != 0) - Warning("Process", "could not forward input data: %s", emsg.Data()); - - // Attach to the transient histogram with the assigned packets, if required - if (fInput->FindObject("PROOF_StatsHist") != 0) { - if (!(fProcPackets = (TH1I *) fOutput->FindObject("PROOF_ProcPcktHist"))) { - Warning("Process", "could not attach to histogram 'PROOF_ProcPcktHist'"); - } else { - PDB(kLoop,1) - Info("Process", "attached to histogram 'PROOF_ProcPcktHist' to record" - " packets being processed"); - } - } - - } else { - - // Check whether we have to enforce the use of submergers - if (gEnv->Lookup("Proof.UseMergers") && !fInput->FindObject("PROOF_UseMergers")) { - Int_t smg = gEnv->GetValue("Proof.UseMergers",-1); - if (smg >= 0) { - fInput->Add(new TParameter("PROOF_UseMergers", smg)); - if (gEnv->Lookup("Proof.MergersByHost")) { - Int_t mbh = gEnv->GetValue("Proof.MergersByHost",0); - if (mbh != 0) { - // Administrator settings have the priority - TObject *o = 0; - if ((o = fInput->FindObject("PROOF_MergersByHost"))) { fInput->Remove(o); delete o; } - fInput->Add(new TParameter("PROOF_MergersByHost", mbh)); - } - } - } - } - - // For a new query clients should make sure that the temporary - // output list is empty - if (fOutputLists) { - fOutputLists->Delete(); - delete fOutputLists; - fOutputLists = 0; - } - - if (!sync) { - gSystem->RedirectOutput(fProof->fLogFileName); - Printf(" "); - Info("Process","starting new query"); - } - - // Define fSelector in Client if processing with filename - if (fCreateSelObj) { - SafeDelete(fSelector); - if (!(fSelector = TSelector::GetSelector(selector_file))) { - if (!sync) - gSystem->RedirectOutput(0); - return -1; - } - } - - fSelectorClass = 0; - fSelectorClass = fSelector->IsA(); - - // Add fSelector to inputlist if processing with object - if (!fCreateSelObj) { - // In any input list was set into the selector move it to the PROOF - // input list, because we do not want to stream the selector one - if (fSelector->GetInputList() && fSelector->GetInputList()->GetSize() > 0) { - TIter nxi(fSelector->GetInputList()); - TObject *o = 0; - while ((o = nxi())) { - if (!fInput->FindObject(o)) { - fInput->Add(o); - if (!inputtmp) { - inputtmp = new TList; - inputtmp->SetOwner(kFALSE); - } - inputtmp->Add(o); - } - } - } - fInput->Add(fSelector); - } - // Set the input list for initialization - fSelector->SetInputList(fInput); - fSelector->SetOption(option); - if (fSelector->GetOutputList()) fSelector->GetOutputList()->Clear(); - - PDB(kLoop,1) Info("Process","Call Begin(0)"); - fSelector->Begin(0); - - // Reset the input list to avoid double streaming and related problems (saving - // the TQueryResult) - if (!fCreateSelObj) fSelector->SetInputList(0); - - // Send large input data objects, if any - fProof->SendInputDataFile(); - - if (!sync) - gSystem->RedirectOutput(0); - } - - TCleanup clean(this); - SetupFeedback(); - - TString opt = option; - - // Old servers need a dedicated streamer - if (fProof->fProtocol < 13) - dset->SetWriteV3(kTRUE); - - // Workers will get the entry ranges from the packetizer - Long64_t num = (gProofServ && gProofServ->IsMaster() && gProofServ->IsParallel()) ? -1 : nentries; - Long64_t fst = (gProofServ && gProofServ->IsMaster() && gProofServ->IsParallel()) ? -1 : first; - - // Entry- or Event- list ? - TEntryList *enl = (!fProof->IsMaster()) ? dynamic_cast(set->GetEntryList()) - : (TEntryList *)0; - TEventList *evl = (!fProof->IsMaster() && !enl) ? dynamic_cast(set->GetEntryList()) - : (TEventList *)0; - if (fProof->fProtocol > 14) { - if (fProcessMessage) delete fProcessMessage; - fProcessMessage = new TMessage(kPROOF_PROCESS); - mesg << set << fn << fInput << opt << num << fst << evl << sync << enl; - (*fProcessMessage) << set << fn << fInput << opt << num << fst << evl << sync << enl; - } else { - mesg << set << fn << fInput << opt << num << fst << evl << sync; - if (enl) - // Not supported remotely - Warning("Process","entry lists not supported by the server"); - } - - // Reset the merging progress information - fProof->ResetMergePrg(); - - Int_t nb = fProof->Broadcast(mesg); - PDB(kGlobal,1) Info("Process", "Broadcast called: %d workers notified", nb); - if (fProof->IsLite()) fProof->fNotIdle += nb; - - // Reset streamer choice - if (fProof->fProtocol < 13) - dset->SetWriteV3(kFALSE); - - // Redirect logs from master to special log frame - if (IsClient()) - fProof->fRedirLog = kTRUE; - - if (!IsClient()){ - // Signal the start of finalize for the memory log grepping - Info("Process|Svc", "Start merging Memory information"); - } - - if (!sync) { - if (IsClient()) { - // Asynchronous query: just make sure that asynchronous input - // is enabled and return the prompt - PDB(kGlobal,1) Info("Process","Asynchronous processing:" - " activating CollectInputFrom"); - fProof->Activate(); - - // Receive the acknowledgement and query sequential number - fProof->Collect(); - - return fProof->fSeqNum; - - } else { - PDB(kGlobal,1) Info("Process","Calling Collect"); - fProof->Collect(); - - HandleTimer(0); // force an update of final result - // This forces a last call to TPacketizer::HandleTimer via the second argument - // (the first is ignored). This is needed when some events were skipped so that - // the total number of entries is not the one requested. The packetizer has no - // way in such a case to understand that processing is finished: it must be told. - if (fPacketizer) { - fPacketizer->StopProcess(kFALSE, kTRUE); - // The progress timer will now stop itself at the next call - fPacketizer->SetBit(TVirtualPacketizer::kIsDone); - // Store process info - elapsed.Stop(); - if (fQuery) - fQuery->SetProcessInfo(0, 0., fPacketizer->GetBytesRead(), - fPacketizer->GetInitTime(), - elapsed.RealTime()); - } - StopFeedback(); - - return Finalize(kFALSE,sync); - } - } else { - - PDB(kGlobal,1) Info("Process","Synchronous processing: calling Collect"); - fProof->Collect(); - if (!(fProof->IsSync())) { - // The server required to switch to asynchronous mode - Info("Process", "switching to the asynchronous mode ..."); - return fProof->fSeqNum; - } - - // Restore prompt logging, for clients (Collect leaves things as they were - // at the time it was called) - if (IsClient()) - fProof->fRedirLog = kFALSE; - - if (!IsClient()) { - // Force an update of final result - HandleTimer(0); - // This forces a last call to TPacketizer::HandleTimer via the second argument - // (the first is ignored). This is needed when some events were skipped so that - // the total number of entries is not the one requested. The packetizer has no - // way in such a case to understand that processing is finished: it must be told. - if (fPacketizer) { - fPacketizer->StopProcess(kFALSE, kTRUE); - // The progress timer will now stop itself at the next call - fPacketizer->SetBit(TVirtualPacketizer::kIsDone); - // Store process info - if (fQuery) - fQuery->SetProcessInfo(0, 0., fPacketizer->GetBytesRead(), - fPacketizer->GetInitTime(), - fPacketizer->GetProcTime()); - } - } else { - // Set the input list: maybe required at termination - if (!fCreateSelObj) fSelector->SetInputList(fInput); - } - StopFeedback(); - - Long64_t rc = -1; - if (!IsClient() || GetExitStatus() != TProofPlayer::kAborted) - rc = Finalize(kFALSE,sync); - - // Remove temporary input objects, if any - if (inputtmp) { - TIter nxi(inputtmp); - TObject *o = 0; - while ((o = nxi())) fInput->Remove(o); - SafeDelete(inputtmp); - } - - // Done - return rc; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process specified TDSet on PROOF. -/// This method is called on client and on the PROOF master. -/// The return value is -1 in case of an error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProofPlayerRemote::Process(TDSet *dset, TSelector *selector, - Option_t *option, Long64_t nentries, - Long64_t first) -{ - if (!selector) { - Error("Process", "selector object undefined"); - return -1; - } - - // Define fSelector in Client - if (IsClient() && (selector != fSelector)) { - SafeDelete(fSelector); - fSelector = selector; - } - - fCreateSelObj = kFALSE; - Long64_t rc = Process(dset, selector->ClassName(), option, nentries, first); - fCreateSelObj = kTRUE; - - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Prepares the given list of new workers to join a progressing process. -/// Returns kTRUE on success, kFALSE otherwise. - -Bool_t TProofPlayerRemote::JoinProcess(TList *workers) -{ - if (!fProcessMessage || !fProof || !fPacketizer) { - Error("Process", "Should not happen: fProcessMessage=%p fProof=%p fPacketizer=%p", - fProcessMessage, fProof, fPacketizer); - return kFALSE; - } - - if (!workers || !fProof->IsMaster()) { - Error("Process", "Invalid call"); - return kFALSE; - } - - PDB(kGlobal, 1) - Info("Process", "Preparing %d new worker(s) to process", workers->GetEntries()); - - // Sends the file associated to the TSelector, if necessary - if (fCreateSelObj) { - PDB(kGlobal, 2) - Info("Process", "Sending selector file %s", fSelectorFileName.Data()); - if(!SendSelector(fSelectorFileName.Data())) { - Error("Process", "Problems in sending selector file %s", fSelectorFileName.Data()); - return kFALSE; - } - } - - if (fProof->IsLite()) fProof->fNotIdle += workers->GetSize(); - - PDB(kGlobal, 2) - Info("Process", "Adding new workers to the packetizer"); - if (fPacketizer->AddWorkers(workers) == -1) { - Error("Process", "Cannot add new workers to the packetizer!"); - return kFALSE; // TODO: make new wrks inactive - } - - PDB(kGlobal, 2) - Info("Process", "Broadcasting process message to new workers"); - fProof->Broadcast(*fProcessMessage, workers); - - // Don't call Collect(): we came here from a global Collect() already which - // will take care of new workers as well - - return kTRUE; - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Merge output in files - -Bool_t TProofPlayerRemote::MergeOutputFiles() -{ - PDB(kOutput,1) Info("MergeOutputFiles", "enter: fOutput size: %d", fOutput->GetSize()); - PDB(kOutput,2) fOutput->ls(); - - TList *rmList = 0; - if (fMergeFiles) { - TIter nxo(fOutput); - TObject *o = 0; - TProofOutputFile *pf = 0; - while ((o = nxo())) { - if ((pf = dynamic_cast(o))) { - - PDB(kOutput,2) pf->Print(); - - if (pf->IsMerge()) { - - // Point to the merger - Bool_t localMerge = (pf->GetTypeOpt() == TProofOutputFile::kLocal) ? kTRUE : kFALSE; - TFileMerger *filemerger = pf->GetFileMerger(localMerge); - if (!filemerger) { - Error("MergeOutputFiles", "file merger is null in TProofOutputFile! Protocol error?"); - pf->Print(); - continue; - } - // If only one instance the list in the merger is not yet created: do it now - if (!pf->IsMerged()) { - PDB(kOutput,2) pf->Print(); - TString fileLoc = TString::Format("%s/%s", pf->GetDir(), pf->GetFileName()); - filemerger->AddFile(fileLoc); - } - // Datadir - TString ddir, ddopts; - if (gProofServ) { - ddir.Form("%s/", gProofServ->GetDataDir()); - if (gProofServ->GetDataDirOpts()) ddopts= gProofServ->GetDataDirOpts(); - } - // Set the output file - TString outfile(pf->GetOutputFileName()); - if (outfile.Contains("/")) { - outfile.ReplaceAll("/", ddir.Data()); - if (!ddopts.IsNull()) - outfile += TString::Format("?%s", ddopts.Data()); - pf->SetOutputFileName(outfile); - } - if ((gProofServ && gProofServ->IsTopMaster()) || (fProof && fProof->IsLite())) { - TFile::EFileType ftyp = TFile::kLocal; - TString srv; - TProofServ::GetLocalServer(srv); - TUrl usrv(srv); - Bool_t localFile = kFALSE; - if (pf->IsRetrieve()) { - // This file will be retrieved by the client: we created it in the data dir - // and save the file URL on the client in the title - if (outfile.BeginsWith("client:")) outfile.Replace(0, 7, ""); - TString bn = gSystem->BaseName(TUrl(outfile.Data(), kTRUE).GetFile()); - // The output file path on the master - outfile.Form("%s%s", ddir.Data(), bn.Data()); - // Save the client path in the title if not defined yet - if (strlen(pf->GetTitle()) <= 0) pf->SetTitle(bn); - // The file is local - localFile = kTRUE; - } else { - // Check if the file is on the master or elsewhere - if (outfile.BeginsWith("master:")) outfile.Replace(0, 7, ""); - // Check locality - TUrl uof(outfile.Data(), kTRUE); - TString lfn; - ftyp = TFile::GetType(uof.GetUrl(), "RECREATE", &lfn); - if (ftyp == TFile::kLocal && !srv.IsNull()) { - // Check if is a different server - if (uof.GetPort() > 0 && usrv.GetPort() > 0 && - usrv.GetPort() != uof.GetPort()) ftyp = TFile::kNet; - } - // If it is really local set the file name - if (ftyp == TFile::kLocal) outfile = lfn; - // The file maybe local - if (ftyp == TFile::kLocal || ftyp == TFile::kFile) localFile = kTRUE; - } - // The remote output file name (the one to be used by the client) - TString outfilerem(outfile); - // For local files we add the local server - if (localFile) { - // Remove prefix, if any, if included and if Xrootd - TProofServ::FilterLocalroot(outfilerem, srv); - outfilerem.Insert(0, srv); - } - // Save the new remote output filename - pf->SetOutputFileName(outfilerem); - // Align the filename - pf->SetFileName(gSystem->BaseName(outfilerem)); - } - if (!filemerger->OutputFile(outfile)) { - Error("MergeOutputFiles", "cannot open the output file"); - continue; - } - // Merge - PDB(kSubmerger,2) filemerger->PrintFiles(""); - if (!filemerger->Merge()) { - Error("MergeOutputFiles", "cannot merge the output files"); - continue; - } - // Remove the files - TList *fileList = filemerger->GetMergeList(); - if (fileList) { - TIter next(fileList); - TObjString *url = 0; - while((url = (TObjString*)next())) { - TUrl u(url->GetName()); - if (!strcmp(u.GetProtocol(), "file")) { - gSystem->Unlink(u.GetFile()); - } else { - gSystem->Unlink(url->GetName()); - } - } - } - // Reset the merger - filemerger->Reset(); - - } else { - - // If not yet merged (for example when having only 1 active worker, - // we need to create the dataset by calling Merge on an effectively empty list - if (!pf->IsMerged()) { - TList dumlist; - dumlist.Add(new TNamed("dum", "dum")); - dumlist.SetOwner(kTRUE); - pf->Merge(&dumlist); - } - // Point to the dataset - TFileCollection *fc = pf->GetFileCollection(); - if (!fc) { - Error("MergeOutputFiles", "file collection is null in TProofOutputFile! Protocol error?"); - pf->Print(); - continue; - } - // Add the collection to the output list for registration and/or to be returned - // to the client - fOutput->Add(fc); - // Do not cleanup at destruction - pf->ResetFileCollection(); - // Tell the main thread to register this dataset, if needed - if (pf->IsRegister()) { - TString opt; - if ((pf->GetTypeOpt() & TProofOutputFile::kOverwrite)) opt += "O"; - if ((pf->GetTypeOpt() & TProofOutputFile::kVerify)) opt += "V"; - if (!fOutput->FindObject("PROOFSERV_RegisterDataSet")) - fOutput->Add(new TNamed("PROOFSERV_RegisterDataSet", "")); - TString tag = TString::Format("DATASET_%s", pf->GetTitle()); - fOutput->Add(new TNamed(tag, opt)); - } - // Remove this object from the output list and schedule it for distruction - fOutput->Remove(pf); - if (!rmList) rmList = new TList; - rmList->Add(pf); - PDB(kOutput,2) fOutput->Print(); - } - } - } - } - - // Remove objects scheduled for removal - if (rmList && rmList->GetSize() > 0) { - TIter nxo(rmList); - TObject *o = 0; - while((o = nxo())) { - fOutput->Remove(o); - } - rmList->SetOwner(kTRUE); - delete rmList; - } - - PDB(kOutput,1) Info("MergeOutputFiles", "done!"); - - // Done - return kTRUE; -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Set the selector's data members: -/// find the mapping of data members to otuput list entries in the output list -/// and apply it. - -void TProofPlayerRemote::SetSelectorDataMembersFromOutputList() -{ - TOutputListSelectorDataMap* olsdm - = TOutputListSelectorDataMap::FindInList(fOutput); - if (!olsdm) { - PDB(kOutput,1) Warning("SetSelectorDataMembersFromOutputList", - "failed to find map object in output list!"); - return; - } - - olsdm->SetDataMembers(fSelector); -} - -//////////////////////////////////////////////////////////////////////////////// - -Long64_t TProofPlayerRemote::Finalize(Bool_t force, Bool_t sync) -{ - // Finalize a query. - // Returns -1 in case of an error, 0 otherwise. - - if (IsClient()) { - if (fOutputLists == 0) { - if (force) - if (fQuery) - return fProof->Finalize(Form("%s:%s", fQuery->GetTitle(), - fQuery->GetName()), force); - } else { - // Make sure the all objects are in the output list - PDB(kGlobal,1) Info("Finalize","Calling Merge Output to finalize the output list"); - MergeOutput(); - } - } - - Long64_t rv = 0; - if (fProof->IsMaster()) { - - // Fill information for monitoring and stop it - TStatus *status = (TStatus *) fOutput->FindObject("PROOF_Status"); - if (!status) { - // The query was aborted: let's add some info in the output list - status = new TStatus(); - fOutput->Add(status); - TString emsg = TString::Format("Query aborted after %lld entries", GetEventsProcessed()); - status->Add(emsg); - } - status->SetExitStatus((Int_t) GetExitStatus()); - - PDB(kOutput,1) Info("Finalize","Calling Merge Output"); - // Some objects (e.g. histos in autobin) may not have been merged yet - // do it now - MergeOutput(); - - fOutput->SetOwner(); - - // Add the active-wrks-vs-proctime info from the packetizer - if (fPacketizer) { - TObject *pperf = (TObject *) fPacketizer->GetProgressPerf(kTRUE); - if (pperf) fOutput->Add(pperf); - TList *parms = fPacketizer->GetConfigParams(kTRUE); - if (parms) { - TIter nxo(parms); - TObject *o = 0; - while ((o = nxo())) fOutput->Add(o); - } - - // If other invalid elements were found during processing, add them to the - // list of missing elements - TDSetElement *elem = 0; - if (fPacketizer->GetFailedPackets()) { - TString type = (fPacketizer->TestBit(TVirtualPacketizer::kIsTree)) ? "TTree" : ""; - TList *listOfMissingFiles = (TList *) fOutput->FindObject("MissingFiles"); - if (!listOfMissingFiles) { - listOfMissingFiles = new TList; - listOfMissingFiles->SetName("MissingFiles"); - } - TIter nxe(fPacketizer->GetFailedPackets()); - while ((elem = (TDSetElement *)nxe())) - listOfMissingFiles->Add(elem->GetFileInfo(type)); - if (!fOutput->FindObject(listOfMissingFiles)) fOutput->Add(listOfMissingFiles); - } - } - - TPerfStats::Stop(); - // Save memory usage on master - Long_t vmaxmst, rmaxmst; - TPerfStats::GetMemValues(vmaxmst, rmaxmst); - status->SetMemValues(vmaxmst, rmaxmst, kTRUE); - - SafeDelete(fSelector); - - } else { - if (fExitStatus != kAborted) { - - if (!sync) { - // Reinit selector (with multi-sessioning we must do this until - // TSelector::GetSelector() is optimized to i) avoid reloading of an - // unchanged selector and ii) invalidate existing instances of - // reloaded selector) - if (ReinitSelector(fQuery) == -1) { - Info("Finalize", "problems reinitializing selector \"%s\"", - fQuery->GetSelecImp()->GetName()); - return -1; - } - } - - if (fPacketizer) - if (TList *failedPackets = fPacketizer->GetFailedPackets()) { - fPacketizer->SetFailedPackets(0); - failedPackets->SetName("FailedPackets"); - AddOutputObject(failedPackets); - - TStatus *status = (TStatus *)GetOutput("PROOF_Status"); - if (!status) AddOutputObject((status = new TStatus())); - status->Add("Some packets were not processed! Check the the" - " 'FailedPackets' list in the output list"); - } - - // Some input parameters may be needed in Terminate - fSelector->SetInputList(fInput); - - TList *output = fSelector->GetOutputList(); - if (output) { - TIter next(fOutput); - while(TObject* obj = next()) { - if (fProof->IsParallel() || DrawCanvas(obj) == 1) - // Either parallel or not a canvas or not able to display it: - // just add to the list - output->Add(obj); - } - } else { - Warning("Finalize", "undefined output list in the selector! Protocol error?"); - } - - // We need to do this because the output list can be modified in TSelector::Terminate - // in a way to invalidate existing objects; so we clean the links when still valid and - // we re-copy back later - fOutput->SetOwner(kFALSE); - fOutput->Clear("nodelete"); - - // Map output objects to selector members - SetSelectorDataMembersFromOutputList(); - - PDB(kLoop,1) Info("Finalize","Call Terminate()"); - // This is the end of merging - SetMerging(kFALSE); - // We measure the merge time - fProof->fQuerySTW.Reset(); - // Call Terminate now - fSelector->Terminate(); - - rv = fSelector->GetStatus(); - - // Copy the output list back and clean the selector's list - TIter it(output); - while(TObject* o = it()) { - fOutput->Add(o); - } - - // Save the output list in the current query, if any - if (fQuery) { - fQuery->SetOutputList(fOutput); - // Set in finalized state (cannot be done twice) - fQuery->SetFinalized(); - } else { - Warning("Finalize","current TQueryResult object is undefined!"); - } - - if (!fCreateSelObj) { - fInput->Remove(fSelector); - fOutput->Remove(fSelector); - if (output) output->Remove(fSelector); - fSelector = 0; - } - - // We have transferred copy of the output objects in TQueryResult, - // so now we can cleanup the selector, making sure that we do not - // touch the output objects - if (output) { output->SetOwner(kFALSE); output->Clear("nodelete"); } - SafeDelete(fSelector); - - // Delete fOutput (not needed anymore, cannot be finalized twice), - // making sure that the objects saved in TQueryResult are not deleted - fOutput->SetOwner(kFALSE); - fOutput->Clear("nodelete"); - SafeDelete(fOutput); - - } else { - - // Cleanup - fOutput->SetOwner(); - SafeDelete(fSelector); - if (!fCreateSelObj) fSelector = 0; - } - } - PDB(kGlobal,1) Info("Process","exit"); - - if (!IsClient()) { - Info("Finalize", "finalization on %s finished", gProofServ->GetPrefix()); - } - fProof->FinalizationDone(); - - return rv; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize the results of a query already processed. - -Long64_t TProofPlayerRemote::Finalize(TQueryResult *qr) -{ - PDB(kGlobal,1) Info("Finalize(TQueryResult *)","Enter"); - - if (!IsClient()) { - Info("Finalize(TQueryResult *)", - "method to be executed only on the clients"); - return -1; - } - - if (!qr) { - Info("Finalize(TQueryResult *)", "query undefined"); - return -1; - } - - if (qr->IsFinalized()) { - Info("Finalize(TQueryResult *)", "query already finalized"); - return -1; - } - - // Reset the list - if (!fOutput) - fOutput = new THashList; - else - fOutput->Clear(); - - // Make sure that the temporary output list is empty - if (fOutputLists) { - fOutputLists->Delete(); - delete fOutputLists; - fOutputLists = 0; - } - - // Re-init the selector - gSystem->RedirectOutput(fProof->fLogFileName); - - // Import the output list - TList *tmp = (TList *) qr->GetOutputList(); - if (!tmp) { - gSystem->RedirectOutput(0); - Info("Finalize(TQueryResult *)", "outputlist is empty"); - return -1; - } - TList *out = fOutput; - if (fProof->fProtocol < 11) - out = new TList; - TIter nxo(tmp); - TObject *o = 0; - while ((o = nxo())) - out->Add(o->Clone()); - - // Adopts the list - if (fProof->fProtocol < 11) { - out->SetOwner(); - StoreOutput(out); - } - gSystem->RedirectOutput(0); - - SetSelectorDataMembersFromOutputList(); - - // Finalize it - SetCurrentQuery(qr); - Long64_t rc = Finalize(); - RestorePreviousQuery(); - - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send the selector file(s) to master or worker nodes. - -Bool_t TProofPlayerRemote::SendSelector(const char* selector_file) -{ - // Check input - if (!selector_file) { - Info("SendSelector", "Invalid input: selector (file) name undefined"); - return kFALSE; - } - - if (!strchr(gSystem->BaseName(selector_file), '.')) { - if (gDebug > 1) - Info("SendSelector", "selector name '%s' does not contain a '.':" - " nothing to send, it will be loaded from a library", selector_file); - return kTRUE; - } - - // Extract the fine name first - TString selec = selector_file; - TString aclicMode; - TString arguments; - TString io; - selec = gSystem->SplitAclicMode(selec, aclicMode, arguments, io); - - // Expand possible envs or '~' - gSystem->ExpandPathName(selec); - - // Update the macro path - TString mp(TROOT::GetMacroPath()); - TString np = gSystem->GetDirName(selec); - if (!np.IsNull()) { - np += ":"; - if (!mp.BeginsWith(np) && !mp.Contains(":"+np)) { - Int_t ip = (mp.BeginsWith(".:")) ? 2 : 0; - mp.Insert(ip, np); - TROOT::SetMacroPath(mp); - if (gDebug > 0) - Info("SendSelector", "macro path set to '%s'", TROOT::GetMacroPath()); - } - } - - // Header file - TString header = selec; - header.Remove(header.Last('.')); - header += ".h"; - if (gSystem->AccessPathName(header, kReadPermission)) { - TString h = header; - header.Remove(header.Last('.')); - header += ".hh"; - if (gSystem->AccessPathName(header, kReadPermission)) { - Info("SendSelector", - "header file not found: tried: %s %s", h.Data(), header.Data()); - return kFALSE; - } - } - - // Send files now - if (fProof->SendFile(selec, (TProof::kBinary | TProof::kForward | TProof::kCp | TProof::kCpBin)) == -1) { - Info("SendSelector", "problems sending implementation file %s", selec.Data()); - return kFALSE; - } - if (fProof->SendFile(header, (TProof::kBinary | TProof::kForward | TProof::kCp)) == -1) { - Info("SendSelector", "problems sending header file %s", header.Data()); - return kFALSE; - } - - return kTRUE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Merge objects in output the lists. - -void TProofPlayerRemote::MergeOutput(Bool_t saveMemValues) -{ - PDB(kOutput,1) Info("MergeOutput","Enter"); - - TObject *obj = 0; - if (fOutputLists) { - - TIter next(fOutputLists); - - TList *list; - while ( (list = (TList *) next()) ) { - - if (!(obj = fOutput->FindObject(list->GetName()))) { - obj = list->First(); - list->Remove(obj); - fOutput->Add(obj); - } - - if ( list->IsEmpty() ) continue; - - TMethodCall callEnv; - if (obj->IsA()) - callEnv.InitWithPrototype(obj->IsA(), "Merge", "TCollection*"); - if (callEnv.IsValid()) { - callEnv.SetParam((Longptr_t) list); - callEnv.Execute(obj); - } else { - // No Merge interface, return individual objects - while ( (obj = list->First()) ) { - fOutput->Add(obj); - list->Remove(obj); - } - } - } - SafeDelete(fOutputLists); - - } else { - - PDB(kOutput,1) Info("MergeOutput","fOutputLists empty"); - } - - if (!IsClient() || fProof->IsLite()) { - // Merge the output files created on workers, if any - MergeOutputFiles(); - } - - // If there are TProofOutputFile objects we have to make sure that the internal - // information is consistent for the cases where this object is going to be merged - // again (e.g. when using submergers or in a multi-master setup). This may not be - // the case because the first coming in is taken as reference and it has the - // internal dir and raw dir of the originating worker. - TString key; - TNamed *nm = 0; - TList rmlist; - TIter nxo(fOutput); - while ((obj = nxo())) { - TProofOutputFile *pf = dynamic_cast(obj); - if (pf) { - if (gProofServ) { - PDB(kOutput,2) Info("MergeOutput","found TProofOutputFile '%s'", obj->GetName()); - TString dir(pf->GetOutputFileName()); - PDB(kOutput,2) Info("MergeOutput","outputfilename: '%s'", dir.Data()); - // The dir - if (dir.Last('/') != kNPOS) dir.Remove(dir.Last('/')+1); - PDB(kOutput,2) Info("MergeOutput","dir: '%s'", dir.Data()); - pf->SetDir(dir); - // The raw dir; for xrootd based system we include the 'localroot', if any - TUrl u(dir); - dir = u.GetFile(); - TString pfx = gEnv->GetValue("Path.Localroot",""); - if (!pfx.IsNull() && - (!strcmp(u.GetProtocol(), "root") || !strcmp(u.GetProtocol(), "xrd"))) - dir.Insert(0, pfx); - PDB(kOutput,2) Info("MergeOutput","rawdir: '%s'", dir.Data()); - pf->SetDir(dir, kTRUE); - // The worker ordinal - pf->SetWorkerOrdinal(gProofServ ? gProofServ->GetOrdinal() : "0"); - // The saved output file name, if any - key.Form("PROOF_OutputFileName_%s", pf->GetFileName()); - if ((nm = (TNamed *) fOutput->FindObject(key.Data()))) { - pf->SetOutputFileName(nm->GetTitle()); - rmlist.Add(nm); - } else if (TestBit(TVirtualProofPlayer::kIsSubmerger)) { - pf->SetOutputFileName(0); - pf->ResetBit(TProofOutputFile::kOutputFileNameSet); - } - // The filename (order is important to exclude '.merger' from the key) - dir = pf->GetFileName(); - if (TestBit(TVirtualProofPlayer::kIsSubmerger)) { - dir += ".merger"; - pf->SetMerged(kFALSE); - } else { - if (dir.EndsWith(".merger")) dir.Remove(dir.Last('.')); - } - pf->SetFileName(dir); - } else if (fProof->IsLite()) { - // The ordinal - pf->SetWorkerOrdinal("0"); - // The dir - pf->SetDir(gSystem->GetDirName(pf->GetOutputFileName())); - // The filename and raw dir - TUrl u(pf->GetOutputFileName(), kTRUE); - pf->SetFileName(gSystem->BaseName(u.GetFile())); - pf->SetDir(gSystem->GetDirName(u.GetFile()), kTRUE); - // Notify the output path - Printf("\nOutput file: %s", pf->GetOutputFileName()); - } - } else { - PDB(kOutput,2) Info("MergeOutput","output object '%s' is not a TProofOutputFile", obj->GetName()); - } - } - - // Remove temporary objects from fOutput - if (rmlist.GetSize() > 0) { - TIter nxrm(&rmlist); - while ((obj = nxrm())) - fOutput->Remove(obj); - rmlist.SetOwner(kTRUE); - } - - // If requested (typically in case of submerger to count possible side-effects in that process) - // save the measured memory usage - if (saveMemValues) { - TPerfStats::Stop(); - // Save memory usage on master - Long_t vmaxmst, rmaxmst; - TPerfStats::GetMemValues(vmaxmst, rmaxmst); - TStatus *status = (TStatus *) fOutput->FindObject("PROOF_Status"); - if (status) status->SetMemValues(vmaxmst, rmaxmst, kFALSE); - } - - PDB(kOutput,1) fOutput->Print(); - PDB(kOutput,1) Info("MergeOutput","leave (%d object(s))", fOutput->GetSize()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Progress signal. - -void TProofPlayerRemote::Progress(Long64_t total, Long64_t processed) -{ - if (IsClient()) { - fProof->Progress(total, processed); - } else { - // Send to the previous tier - TMessage m(kPROOF_PROGRESS); - m << total << processed; - gProofServ->GetSocket()->Send(m); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Progress signal. - -void TProofPlayerRemote::Progress(Long64_t total, Long64_t processed, - Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) -{ - PDB(kGlobal,1) - Info("Progress","%lld %lld %lld %f %f %f %f", total, processed, bytesread, - initTime, procTime, evtrti, mbrti); - - if (IsClient()) { - fProof->Progress(total, processed, bytesread, initTime, procTime, evtrti, mbrti); - } else { - // Send to the previous tier - TMessage m(kPROOF_PROGRESS); - m << total << processed << bytesread << initTime << procTime << evtrti << mbrti; - gProofServ->GetSocket()->Send(m); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Progress signal. - -void TProofPlayerRemote::Progress(TProofProgressInfo *pi) -{ - if (pi) { - PDB(kGlobal,1) - Info("Progress","%lld %lld %lld %f %f %f %f %d %f", pi->fTotal, pi->fProcessed, pi->fBytesRead, - pi->fInitTime, pi->fProcTime, pi->fEvtRateI, pi->fMBRateI, - pi->fActWorkers, pi->fEffSessions); - - if (IsClient()) { - fProof->Progress(pi->fTotal, pi->fProcessed, pi->fBytesRead, - pi->fInitTime, pi->fProcTime, - pi->fEvtRateI, pi->fMBRateI, - pi->fActWorkers, pi->fTotSessions, pi->fEffSessions); - } else { - // Send to the previous tier - TMessage m(kPROOF_PROGRESS); - m << pi; - gProofServ->GetSocket()->Send(m); - } - } else { - Warning("Progress","TProofProgressInfo object undefined!"); - } -} - - -//////////////////////////////////////////////////////////////////////////////// -/// Feedback signal. - -void TProofPlayerRemote::Feedback(TList *objs) -{ - fProof->Feedback(objs); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Stop process after this event. - -void TProofPlayerRemote::StopProcess(Bool_t abort, Int_t) -{ - if (fPacketizer != 0) - fPacketizer->StopProcess(abort, kFALSE); - if (abort == kTRUE) - fExitStatus = kAborted; - else - fExitStatus = kStopped; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Incorporate the received object 'obj' into the output list fOutput. -/// The latter is created if not existing. -/// This method short cuts 'StoreOutput + MergeOutput' optimizing the memory -/// consumption. -/// Returns -1 in case of error, 1 if the object has been merged into another -/// one (so that its ownership has not been taken and can be deleted), and 0 -/// otherwise. - -Int_t TProofPlayerRemote::AddOutputObject(TObject *obj) -{ - PDB(kOutput,1) - Info("AddOutputObject","Enter: %p (%s)", obj, obj ? obj->ClassName() : "undef"); - - // We must something to process - if (!obj) { - PDB(kOutput,1) Info("AddOutputObject","Invalid input (obj == 0x0)"); - return -1; - } - - // Create the output list, if not yet done - if (!fOutput) - fOutput = new THashList; - - // Flag about merging - Bool_t merged = kTRUE; - - // Process event lists first - TList *elists = dynamic_cast (obj); - if (elists && !strcmp(elists->GetName(), "PROOF_EventListsList")) { - - // Create a global event list, result of merging the event lists - // coresponding to the various data set elements - TEventList *evlist = new TEventList("PROOF_EventList"); - - // Iterate the list of event list segments - TIter nxevl(elists); - TEventList *evl = 0; - while ((evl = dynamic_cast (nxevl()))) { - - // Find the file offset (fDSet is the current TDSet instance) - // locating the element by name - TIter nxelem(fDSet->GetListOfElements()); - TDSetElement *elem = 0; - while ((elem = dynamic_cast (nxelem()))) { - if (!strcmp(elem->GetFileName(), evl->GetName())) - break; - } - if (!elem) { - Error("AddOutputObject", "Found an event list for %s, but no object with" - " the same name in the TDSet", evl->GetName()); - continue; - } - Long64_t offset = elem->GetTDSetOffset(); - - // Shift the list by the number of first event in that file - Long64_t *arr = evl->GetList(); - Int_t num = evl->GetN(); - if (arr && offset > 0) - for (Int_t i = 0; i < num; i++) - arr[i] += offset; - - // Add to the global event list - evlist->Add(evl); - } - - // Incorporate the resulting global list in fOutput - SetLastMergingMsg(evlist); - Incorporate(evlist, fOutput, merged); - NotifyMemory(evlist); - - // Delete the global list if merged - if (merged) - SafeDelete(evlist); - - // The original object has been transformed in something else; we do - // not have ownership on it - return 1; - } - - // Check if we need to merge files - TProofOutputFile *pf = dynamic_cast(obj); - if (pf) { - fMergeFiles = kTRUE; - if (!IsClient() || fProof->IsLite()) { - if (pf->IsMerge()) { - Bool_t hasfout = (pf->GetOutputFileName() && - strlen(pf->GetOutputFileName()) > 0 && - pf->TestBit(TProofOutputFile::kOutputFileNameSet)) ? kTRUE : kFALSE; - Bool_t setfout = (!hasfout || TestBit(TVirtualProofPlayer::kIsSubmerger)) ? kTRUE : kFALSE; - if (setfout) { - - TString ddir, ddopts; - if (gProofServ) { - ddir.Form("%s/", gProofServ->GetDataDir()); - if (gProofServ->GetDataDirOpts()) ddopts = gProofServ->GetDataDirOpts(); - } - // Set the output file - TString outfile(pf->GetOutputFileName()); - outfile.ReplaceAll("/", ddir.Data()); - if (!ddopts.IsNull()) outfile += TString::Format("?%s", ddopts.Data()); - pf->SetOutputFileName(outfile); - - if (gProofServ) { - // If submerger, save first the existing filename, if any - if (TestBit(TVirtualProofPlayer::kIsSubmerger) && hasfout) { - TString key = TString::Format("PROOF_OutputFileName_%s", pf->GetFileName()); - if (!fOutput->FindObject(key.Data())) - fOutput->Add(new TNamed(key.Data(), pf->GetOutputFileName())); - } - TString of; - TProofServ::GetLocalServer(of); - if (of.IsNull()) { - // Assume an xroot server running on the machine - of.Form("root://%s/", gSystem->HostName()); - if (gSystem->Getenv("XRDPORT")) { - TString sp(gSystem->Getenv("XRDPORT")); - if (sp.IsDigit()) - of.Form("root://%s:%s/", gSystem->HostName(), sp.Data()); - } - } - TString sessionPath(gProofServ->GetSessionDir()); - TProofServ::FilterLocalroot(sessionPath, of); - of += TString::Format("%s/%s", sessionPath.Data(), pf->GetFileName()); - if (TestBit(TVirtualProofPlayer::kIsSubmerger)) { - if (!of.EndsWith(".merger")) of += ".merger"; - } else { - if (of.EndsWith(".merger")) of.Remove(of.Last('.')); - } - pf->SetOutputFileName(of); - } - } - // Notify - PDB(kOutput, 1) pf->Print(); - } - } else { - // On clients notify the output path - Printf("Output file: %s", pf->GetOutputFileName()); - } - } - - // For other objects we just run the incorporation procedure - SetLastMergingMsg(obj); - Incorporate(obj, fOutput, merged); - NotifyMemory(obj); - - // We are done - return (merged ? 1 : 0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Control output redirection to TProof::fLogFileW - -void TProofPlayerRemote::RedirectOutput(Bool_t on) -{ - if (on && fProof && fProof->fLogFileW) { - TProofServ::SetErrorHandlerFile(fProof->fLogFileW); - fErrorHandler = SetErrorHandler(TProofServ::ErrorHandler); - } else if (!on) { - if (fErrorHandler) { - TProofServ::SetErrorHandlerFile(0); - SetErrorHandler(fErrorHandler); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Incorporate the content of the received output list 'out' into the final -/// output list fOutput. The latter is created if not existing. -/// This method short cuts 'StoreOutput + MergeOutput' limiting the memory -/// consumption. - -void TProofPlayerRemote::AddOutput(TList *out) -{ - PDB(kOutput,1) Info("AddOutput","Enter"); - - // We must something to process - if (!out) { - PDB(kOutput,1) Info("AddOutput","Invalid input (out == 0x0)"); - return; - } - - // Create the output list, if not yet done - if (!fOutput) - fOutput = new THashList; - - // Process event lists first - Bool_t merged = kTRUE; - TList *elists = dynamic_cast (out->FindObject("PROOF_EventListsList")); - if (elists) { - - // Create a global event list, result of merging the event lists - // corresponding to the various data set elements - TEventList *evlist = new TEventList("PROOF_EventList"); - - // Iterate the list of event list segments - TIter nxevl(elists); - TEventList *evl = 0; - while ((evl = dynamic_cast (nxevl()))) { - - // Find the file offset (fDSet is the current TDSet instance) - // locating the element by name - TIter nxelem(fDSet->GetListOfElements()); - TDSetElement *elem = 0; - while ((elem = dynamic_cast (nxelem()))) { - if (!strcmp(elem->GetFileName(), evl->GetName())) - break; - } - if (!elem) { - Error("AddOutput", "Found an event list for %s, but no object with" - " the same name in the TDSet", evl->GetName()); - continue; - } - Long64_t offset = elem->GetTDSetOffset(); - - // Shift the list by the number of first event in that file - Long64_t *arr = evl->GetList(); - Int_t num = evl->GetN(); - if (arr && offset > 0) - for (Int_t i = 0; i < num; i++) - arr[i] += offset; - - // Add to the global event list - evlist->Add(evl); - } - - // Remove and delete the events lists object to avoid spoiling iteration - // during next steps - out->Remove(elists); - delete elists; - - // Incorporate the resulting global list in fOutput - SetLastMergingMsg(evlist); - Incorporate(evlist, fOutput, merged); - NotifyMemory(evlist); - } - - // Iterate on the remaining objects in the received list - TIter nxo(out); - TObject *obj = 0; - while ((obj = nxo())) { - SetLastMergingMsg(obj); - Incorporate(obj, fOutput, merged); - // If not merged, drop from the temporary list, as the ownership - // passes to fOutput - if (!merged) - out->Remove(obj); - NotifyMemory(obj); - } - - // Done - return; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Printout the memory record after merging object 'obj' -/// This record is used by the memory monitor - -void TProofPlayerRemote::NotifyMemory(TObject *obj) -{ - if (fProof && (!IsClient() || fProof->IsLite())){ - ProcInfo_t pi; - if (!gSystem->GetProcInfo(&pi)){ - // For PROOF-Lite we redirect this output to a the open log file so that the - // memory monitor can pick these messages up - RedirectOutput(fProof->IsLite()); - Info("NotifyMemory|Svc", "Memory %ld virtual %ld resident after merging object %s", - pi.fMemVirtual, pi.fMemResident, obj->GetName()); - RedirectOutput(0); - } - // Record also values for monitoring - TPerfStats::SetMemValues(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the message to be notified in case of exception - -void TProofPlayerRemote::SetLastMergingMsg(TObject *obj) -{ - TString lastMsg = TString::Format("while merging object '%s'", obj->GetName()); - TProofServ::SetLastMsg(lastMsg); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Incorporate object 'newobj' in the list 'outlist'. -/// The object is merged with an object of the same name already existing in -/// the list, or just added. -/// The boolean merged is set to kFALSE when the object is just added to 'outlist'; -/// this happens if the Merge() method does not exist or if a object named as 'obj' -/// is not already in the list. If the obj is not 'merged' than it should not be -/// deleted, unless outlist is not owner of its objects. -/// Return 0 on success, -1 on error. - -Int_t TProofPlayerRemote::Incorporate(TObject *newobj, TList *outlist, Bool_t &merged) -{ - merged = kTRUE; - - PDB(kOutput,1) - Info("Incorporate", "enter: obj: %p (%s), list: %p", - newobj, newobj ? newobj->ClassName() : "undef", outlist); - - // The object and list must exist - if (!newobj || !outlist) { - Error("Incorporate","Invalid inputs: obj: %p, list: %p", newobj, outlist); - return -1; - } - - // Special treatment for histograms in autobin mode - Bool_t specialH = - (!fProof || !fProof->TestBit(TProof::kIsClient) || fProof->IsLite()) ? kTRUE : kFALSE; - if (specialH && newobj->InheritsFrom(TH1::Class())) { - if (!HandleHistogram(newobj, merged)) { - if (merged) { - PDB(kOutput,1) Info("Incorporate", "histogram object '%s' merged", newobj->GetName()); - } else { - PDB(kOutput,1) Info("Incorporate", "histogram object '%s' added to the" - " appropriate list for delayed merging", newobj->GetName()); - } - return 0; - } - } - - // Check if an object with the same name exists already - TObject *obj = outlist->FindObject(newobj->GetName()); - - // If no, add the new object and return - if (!obj) { - outlist->Add(newobj); - merged = kFALSE; - // Done - return 0; - } - - // Locate the Merge(TCollection *) method - TMethodCall callEnv; - if (obj->IsA()) - callEnv.InitWithPrototype(obj->IsA(), "Merge", "TCollection*"); - if (callEnv.IsValid()) { - // Found: put the object in a one-element list - static TList *xlist = new TList; - xlist->Add(newobj); - // Call the method - callEnv.SetParam((Longptr_t) xlist); - callEnv.Execute(obj); - // Ready for next call - xlist->Clear(); - } else { - // Not found: return individual objects - outlist->Add(newobj); - merged = kFALSE; - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Low statistic histograms need a special treatment when using autobin - -TObject *TProofPlayerRemote::HandleHistogram(TObject *obj, Bool_t &merged) -{ - TH1 *h = dynamic_cast(obj); - if (!h) { - // Not an histo - return obj; - } - - // This is only used if we return (TObject *)0 and there is only one case - // when we set this to kTRUE - merged = kFALSE; - - // Does is still needs binning ? - Bool_t tobebinned = (h->GetBuffer()) ? kTRUE : kFALSE; - - // Number of entries - Int_t nent = h->GetBufferLength(); - PDB(kOutput,2) Info("HandleHistogram", "h:%s ent:%d, buffer size: %d", - h->GetName(), nent, h->GetBufferSize()); - - // Attach to the list in the outputlists, if any - TList *list = 0; - if (!fOutputLists) { - PDB(kOutput,2) Info("HandleHistogram", "create fOutputLists"); - fOutputLists = new TList; - fOutputLists->SetOwner(); - } - list = (TList *) fOutputLists->FindObject(h->GetName()); - - TH1 *href = 0; - if (tobebinned) { - - // The histogram needs to be projected in a reasonable range: we - // do this at the end with all the histos, so we need to create - // a list here - if (!list) { - // Create the list - list = new TList; - list->SetName(h->GetName()); - list->SetOwner(); - fOutputLists->Add(list); - // Move in it any previously merged object from the output list - if (fOutput && (href = (TH1 *) fOutput->FindObject(h->GetName()))) { - fOutput->Remove(href); - list->Add(href); - } - } - TIter nxh(list); - while ((href = (TH1 *) nxh())) { - if (href->GetBuffer() && href->GetBufferLength() < nent) break; - } - if (href) { - list->AddBefore(href, h); - } else { - list->Add(h); - } - // Done - return (TObject *)0; - - } else { - - if (list) { - TIter nxh(list); - while ((href = (TH1 *) nxh())) { - if (href->GetBuffer() || href->GetEntries() < nent) break; - } - if (href) { - list->AddBefore(href, h); - } else { - list->Add(h); - } - // Done - return (TObject *)0; - - } else { - // Check if we can 'Add' the histogram to an existing one; this is more efficient - // then using Merge - TH1 *hout = (TH1*) fOutput->FindObject(h->GetName()); - if (hout) { - // Remove the existing histo from the output list ... - fOutput->Remove(hout); - // ... and create either the list to merge in one-go at the end - // (more efficient than merging one by one) or, if too big, merge - // these two and start the 'one-by-one' technology - Int_t hsz = h->GetNbinsX() * h->GetNbinsY() * h->GetNbinsZ(); - if (fMergeTH1OneByOne || (gProofServ && hsz > gProofServ->GetMsgSizeHWM())) { - list = new TList; - list->Add(hout); - h->Merge(list); - list->SetOwner(); - delete list; - return h; - } else { - list = new TList; - list->SetName(h->GetName()); - list->SetOwner(); - fOutputLists->Add(list); - // Add the existing and the incoming histos - list->Add(hout); - list->Add(h); - // Done - return (TObject *)0; - } - } else { - // This is the first one; add it to the output list - fOutput->Add(h); - return (TObject *)0; - } - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return kTRUE is the histograms 'h0' and 'h1' have the same binning and ranges -/// on the axis (i.e. if they can be just Add-ed for merging). - -Bool_t TProofPlayerRemote::HistoSameAxis(TH1 *h0, TH1 *h1) -{ - Bool_t rc = kFALSE; - if (!h0 || !h1) return rc; - - TAxis *a0 = 0, *a1 = 0; - - // Check X - a0 = h0->GetXaxis(); - a1 = h1->GetXaxis(); - if (a0->GetNbins() == a1->GetNbins()) - if (TMath::Abs(a0->GetXmax() - a1->GetXmax()) < 1.e-9) - if (TMath::Abs(a0->GetXmin() - a1->GetXmin()) < 1.e-9) rc = kTRUE; - - // Check Y, if needed - if (h0->GetDimension() > 1) { - rc = kFALSE; - a0 = h0->GetYaxis(); - a1 = h1->GetYaxis(); - if (a0->GetNbins() == a1->GetNbins()) - if (TMath::Abs(a0->GetXmax() - a1->GetXmax()) < 1.e-9) - if (TMath::Abs(a0->GetXmin() - a1->GetXmin()) < 1.e-9) rc = kTRUE; - } - - // Check Z, if needed - if (h0->GetDimension() > 2) { - rc = kFALSE; - a0 = h0->GetZaxis(); - a1 = h1->GetZaxis(); - if (a0->GetNbins() == a1->GetNbins()) - if (TMath::Abs(a0->GetXmax() - a1->GetXmax()) < 1.e-9) - if (TMath::Abs(a0->GetXmin() - a1->GetXmin()) < 1.e-9) rc = kTRUE; - } - - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Store received output list. - -void TProofPlayerRemote::StoreOutput(TList *out) -{ - PDB(kOutput,1) Info("StoreOutput","Enter"); - - if ( out == 0 ) { - PDB(kOutput,1) Info("StoreOutput","Leave (empty)"); - return; - } - - TIter next(out); - out->SetOwner(kFALSE); // take ownership of the contents - - if (fOutputLists == 0) { - PDB(kOutput,2) Info("StoreOutput","Create fOutputLists"); - fOutputLists = new TList; - fOutputLists->SetOwner(); - } - // process eventlists first - TList* lists = dynamic_cast (out->FindObject("PROOF_EventListsList")); - if (lists) { - out->Remove(lists); - TEventList *mainList = new TEventList("PROOF_EventList"); - out->Add(mainList); - TIter it(lists); - TEventList *aList; - while ( (aList = dynamic_cast (it())) ) { - // find file offset - TIter nxe(fDSet->GetListOfElements()); - TDSetElement *elem; - while ( (elem = dynamic_cast (nxe())) ) { - if (strcmp(elem->GetFileName(), aList->GetName()) == 0) - break; - } - if (!elem) { - Error("StoreOutput", "found the EventList for %s, but no object with that name " - "in the TDSet", aList->GetName()); - continue; - } - Long64_t offset = elem->GetTDSetOffset(); - - // shift the list by the number of first event in that file - Long64_t *arr = aList->GetList(); - Int_t num = aList->GetN(); - if (arr && offset) - for (int i = 0; i < num; i++) - arr[i] += offset; - - mainList->Add(aList); // add to the main list - } - delete lists; - } - - TObject *obj; - while( (obj = next()) ) { - PDB(kOutput,2) Info("StoreOutput","find list for '%s'", obj->GetName() ); - - TList *list = (TList *) fOutputLists->FindObject( obj->GetName() ); - if ( list == 0 ) { - PDB(kOutput,2) Info("StoreOutput", "list for '%s' not found (creating)", obj->GetName()); - list = new TList; - list->SetName( obj->GetName() ); - list->SetOwner(); - fOutputLists->Add( list ); - } - list->Add( obj ); - } - - delete out; - PDB(kOutput,1) Info("StoreOutput", "leave"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Merge feedback lists. - -TList *TProofPlayerRemote::MergeFeedback() -{ - PDB(kFeedback,1) - Info("MergeFeedback","Enter"); - - if ( fFeedbackLists == 0 ) { - PDB(kFeedback,1) - Info("MergeFeedback","Leave (no output)"); - return 0; - } - - TList *fb = new TList; // collection of feedback objects - fb->SetOwner(); - - TIter next(fFeedbackLists); - - TMap *map; - while ( (map = (TMap*) next()) ) { - - PDB(kFeedback,2) - Info("MergeFeedback", "map %s size: %d", map->GetName(), map->GetSize()); - - // turn map into list ... - - TList *list = new TList; - TIter keys(map); - -#ifndef R__TH1MERGEFIXED - Int_t nbmx = -1; - TObject *oref = 0; -#endif - while ( TObject *key = keys() ) { - TObject *o = map->GetValue(key); - TH1 *h = dynamic_cast(o); -#ifndef R__TH1MERGEFIXED - // Temporary fix for to cope with the problem in TH1::Merge. - // We need to use a reference histo the one with the largest number - // of bins so that the histos from all submasters can be correctly - // fit in - if (h && !strncmp(o->GetName(),"PROOF_",6)) { - if (h->GetNbinsX() > nbmx) { - nbmx= h->GetNbinsX(); - oref = o; - } - } -#endif - if (h) { - TIter nxh(list); - TH1 *href= 0; - while ((href = (TH1 *)nxh())) { - if (h->GetBuffer()) { - if (href->GetBuffer() && href->GetBufferLength() < h->GetBufferLength()) break; - } else { - if (href->GetBuffer() || href->GetEntries() < h->GetEntries()) break; - } - } - if (href) { - list->AddBefore(href, h); - } else { - list->Add(h); - } - } else { - list->Add(o); - } - } - - // clone first object, remove from list -#ifdef R__TH1MERGEFIXED - TObject *obj = list->First(); -#else - TObject *obj = (oref) ? oref : list->First(); -#endif - list->Remove(obj); - obj = obj->Clone(); - fb->Add(obj); - - if ( list->IsEmpty() ) { - delete list; - continue; - } - - // merge list with clone - TMethodCall callEnv; - if (obj->IsA()) - callEnv.InitWithPrototype(obj->IsA(), "Merge", "TCollection*"); - if (callEnv.IsValid()) { - callEnv.SetParam((Longptr_t) list); - callEnv.Execute(obj); - } else { - // No Merge interface, return copy of individual objects - while ( (obj = list->First()) ) { - fb->Add(obj->Clone()); - list->Remove(obj); - } - } - - delete list; - } - - PDB(kFeedback,1) - Info("MergeFeedback","Leave (%d object(s))", fb->GetSize()); - - return fb; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Store feedback results from the specified slave. - -void TProofPlayerRemote::StoreFeedback(TObject *slave, TList *out) -{ - PDB(kFeedback,1) - Info("StoreFeedback","Enter"); - - if ( out == 0 ) { - PDB(kFeedback,1) - Info("StoreFeedback","Leave (empty)"); - return; - } - - if ( IsClient() ) { - // in client - Feedback(out); - delete out; - return; - } - - if (fFeedbackLists == 0) { - PDB(kFeedback,2) Info("StoreFeedback","Create fFeedbackLists"); - fFeedbackLists = new TList; - fFeedbackLists->SetOwner(); - } - - TIter next(out); - out->SetOwner(kFALSE); // take ownership of the contents - - const char *ord = ((TSlave*) slave)->GetOrdinal(); - - TObject *obj; - while( (obj = next()) ) { - PDB(kFeedback,2) - Info("StoreFeedback","%s: Find '%s'", ord, obj->GetName() ); - TMap *map = (TMap*) fFeedbackLists->FindObject(obj->GetName()); - if ( map == 0 ) { - PDB(kFeedback,2) - Info("StoreFeedback", "%s: map for '%s' not found (creating)", ord, obj->GetName()); - // Map must not be owner (ownership is with regards to the keys (only)) - map = new TMap; - map->SetName(obj->GetName()); - fFeedbackLists->Add(map); - } else { - PDB(kFeedback,2) - Info("StoreFeedback","%s: removing previous value", ord); - if (map->GetValue(slave)) - delete map->GetValue(slave); - map->Remove(slave); - } - map->Add(slave, obj); - PDB(kFeedback,2) - Info("StoreFeedback","%s: %s, size: %d", ord, obj->GetName(), map->GetSize()); - } - - delete out; - PDB(kFeedback,1) - Info("StoreFeedback","Leave"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Setup reporting of feedback objects. - -void TProofPlayerRemote::SetupFeedback() -{ - if (IsClient()) return; // Client does not need timer - - fFeedback = (TList*) fInput->FindObject("FeedbackList"); - - PDB(kFeedback,1) Info("SetupFeedback","\"FeedbackList\" %sfound", - fFeedback == 0 ? "NOT ":""); - - if (fFeedback == 0 || fFeedback->GetSize() == 0) return; - - // OK, feedback was requested, setup the timer - SafeDelete(fFeedbackTimer); - fFeedbackPeriod = 2000; - TProof::GetParameter(fInput, "PROOF_FeedbackPeriod", fFeedbackPeriod); - fFeedbackTimer = new TTimer; - fFeedbackTimer->SetObject(this); - fFeedbackTimer->Start(fFeedbackPeriod, kTRUE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Stop reporting of feedback objects. - -void TProofPlayerRemote::StopFeedback() -{ - if (fFeedbackTimer == 0) return; - - PDB(kFeedback,1) Info("StopFeedback","Stop Timer"); - - SafeDelete(fFeedbackTimer); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send feedback objects to client. - -Bool_t TProofPlayerRemote::HandleTimer(TTimer *) -{ - PDB(kFeedback,2) Info("HandleTimer","Entry"); - - if (fFeedbackTimer == 0) return kFALSE; // timer already switched off - - // process local feedback objects - - TList *fb = new TList; - fb->SetOwner(); - - TIter next(fFeedback); - while( TObjString *name = (TObjString*) next() ) { - TObject *o = fOutput->FindObject(name->GetName()); - if (o != 0) { - fb->Add(o->Clone()); - // remove the corresponding entry from the feedback list - TMap *m = 0; - if (fFeedbackLists && - (m = (TMap *) fFeedbackLists->FindObject(name->GetName()))) { - fFeedbackLists->Remove(m); - m->DeleteValues(); - delete m; - } - } - } - - if (fb->GetSize() > 0) { - StoreFeedback(this, fb); // adopts fb - } else { - delete fb; - } - - if (fFeedbackLists == 0) { - fFeedbackTimer->Start(fFeedbackPeriod, kTRUE); // maybe next time - return kFALSE; - } - - fb = MergeFeedback(); - - PDB(kFeedback,2) Info("HandleTimer","Sending %d objects", fb->GetSize()); - - TMessage m(kPROOF_FEEDBACK); - m << fb; - - // send message to client; - gProofServ->GetSocket()->Send(m); - - delete fb; - - fFeedbackTimer->Start(fFeedbackPeriod, kTRUE); - - return kFALSE; // ignored? -} - -//////////////////////////////////////////////////////////////////////////////// -/// Get next packet for specified slave. - -TDSetElement *TProofPlayerRemote::GetNextPacket(TSlave *slave, TMessage *r) -{ - // The first call to this determines the end of initialization - SetInitTime(); - - if (fProcPackets) { - Int_t bin = fProcPackets->GetXaxis()->FindBin(slave->GetOrdinal()); - if (bin >= 0) { - if (fProcPackets->GetBinContent(bin) > 0) - fProcPackets->Fill(slave->GetOrdinal(), -1); - } - } - - TDSetElement *e = fPacketizer->GetNextPacket( slave, r ); - - if (e == 0) { - PDB(kPacketizer,2) - Info("GetNextPacket","%s: done!", slave->GetOrdinal()); - } else if (e == (TDSetElement*) -1) { - PDB(kPacketizer,2) - Info("GetNextPacket","%s: waiting ...", slave->GetOrdinal()); - } else { - PDB(kPacketizer,2) - Info("GetNextPacket","%s (%s): '%s' '%s' '%s' %lld %lld", - slave->GetOrdinal(), slave->GetName(), e->GetFileName(), - e->GetDirectory(), e->GetObjName(), e->GetFirst(), e->GetNum()); - if (fProcPackets) fProcPackets->Fill(slave->GetOrdinal(), 1); - } - - return e; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Is the player running on the client? - -Bool_t TProofPlayerRemote::IsClient() const -{ - return fProof ? fProof->TestBit(TProof::kIsClient) : kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Draw (support for TChain::Draw()). -/// Returns -1 in case of error or number of selected events in case of success. - -Long64_t TProofPlayerRemote::DrawSelect(TDSet *set, const char *varexp, - const char *selection, Option_t *option, - Long64_t nentries, Long64_t firstentry) -{ - if (!fgDrawInputPars) { - fgDrawInputPars = new THashList; - fgDrawInputPars->Add(new TObjString("FeedbackList")); - fgDrawInputPars->Add(new TObjString("PROOF_ChainWeight")); - fgDrawInputPars->Add(new TObjString("PROOF_LineColor")); - fgDrawInputPars->Add(new TObjString("PROOF_LineStyle")); - fgDrawInputPars->Add(new TObjString("PROOF_LineWidth")); - fgDrawInputPars->Add(new TObjString("PROOF_MarkerColor")); - fgDrawInputPars->Add(new TObjString("PROOF_MarkerStyle")); - fgDrawInputPars->Add(new TObjString("PROOF_MarkerSize")); - fgDrawInputPars->Add(new TObjString("PROOF_FillColor")); - fgDrawInputPars->Add(new TObjString("PROOF_FillStyle")); - fgDrawInputPars->Add(new TObjString("PROOF_ListOfAliases")); - } - - TString selector, objname; - if (GetDrawArgs(varexp, selection, option, selector, objname) != 0) { - Error("DrawSelect", "parsing arguments"); - return -1; - } - - TNamed *varexpobj = new TNamed("varexp", varexp); - TNamed *selectionobj = new TNamed("selection", selection); - - // Save the current input list - TObject *o = 0; - TList *savedInput = new TList; - TIter nxi(fInput); - while ((o = nxi())) { - savedInput->Add(o); - TString n(o->GetName()); - if (fgDrawInputPars && - !fgDrawInputPars->FindObject(o->GetName()) && - !n.BeginsWith("alias:")) fInput->Remove(o); - } - - fInput->Add(varexpobj); - fInput->Add(selectionobj); - - // Make sure we have an object name - if (objname == "") objname = "htemp"; - - fProof->AddFeedback(objname); - Long64_t r = Process(set, selector, option, nentries, firstentry); - fProof->RemoveFeedback(objname); - - fInput->Remove(varexpobj); - fInput->Remove(selectionobj); - if (TNamed *opt = dynamic_cast (fInput->FindObject("PROOF_OPTIONS"))) { - fInput->Remove(opt); - delete opt; - } - - delete varexpobj; - delete selectionobj; - - // Restore the input list - fInput->Clear(); - TIter nxsi(savedInput); - while ((o = nxsi())) - fInput->Add(o); - savedInput->SetOwner(kFALSE); - delete savedInput; - - return r; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set init time - -void TProofPlayerRemote::SetInitTime() -{ - if (fPacketizer) - fPacketizer->SetInitTime(); -} - -//------------------------------------------------------------------------------ - - -ClassImp(TProofPlayerSlave); - -//////////////////////////////////////////////////////////////////////////////// -/// Setup feedback. - -void TProofPlayerSlave::SetupFeedback() -{ - TList *fb = (TList*) fInput->FindObject("FeedbackList"); - if (fb) { - PDB(kFeedback,1) - Info("SetupFeedback","\"FeedbackList\" found: %d objects", fb->GetSize()); - } else { - PDB(kFeedback,1) - Info("SetupFeedback","\"FeedbackList\" NOT found"); - } - - if (fb == 0 || fb->GetSize() == 0) return; - - // OK, feedback was requested, setup the timer - - SafeDelete(fFeedbackTimer); - fFeedbackPeriod = 2000; - TProof::GetParameter(fInput, "PROOF_FeedbackPeriod", fFeedbackPeriod); - fFeedbackTimer = new TTimer; - fFeedbackTimer->SetObject(this); - fFeedbackTimer->Start(fFeedbackPeriod, kTRUE); - - fFeedback = fb; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Stop feedback. - -void TProofPlayerSlave::StopFeedback() -{ - if (fFeedbackTimer == 0) return; - - PDB(kFeedback,1) Info("StopFeedback","Stop Timer"); - - SafeDelete(fFeedbackTimer); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle timer event. - -Bool_t TProofPlayerSlave::HandleTimer(TTimer *) -{ - PDB(kFeedback,2) Info("HandleTimer","Entry"); - - // If in sequential (0-slave-PROOF) mode we do not have a packetizer - // so we also send the info to update the progress bar. - if (gProofServ) { - Bool_t sendm = kFALSE; - TMessage m(kPROOF_PROGRESS); - if (gProofServ->IsMaster() && !gProofServ->IsParallel()) { - sendm = kTRUE; - if (gProofServ->GetProtocol() > 25) { - m << GetProgressStatus(); - } else if (gProofServ->GetProtocol() > 11) { - TProofProgressStatus *ps = GetProgressStatus(); - m << fTotalEvents << ps->GetEntries() << ps->GetBytesRead() - << (Float_t) -1. << (Float_t) ps->GetProcTime() - << (Float_t) ps->GetRate() << (Float_t) -1.; - } else { - m << fTotalEvents << GetEventsProcessed(); - } - } - if (sendm) gProofServ->GetSocket()->Send(m); - } - - if (fFeedback == 0) return kFALSE; - - TList *fb = new TList; - fb->SetOwner(kFALSE); - - if (fOutput == 0) { - fOutput = (THashList *) fSelector->GetOutputList(); - } - - if (fOutput) { - TIter next(fFeedback); - while( TObjString *name = (TObjString*) next() ) { - // TODO: find object in memory ... maybe allow only in fOutput ? - TObject *o = fOutput->FindObject(name->GetName()); - if (o != 0) fb->Add(o); - } - } - - PDB(kFeedback,2) Info("HandleTimer","Sending %d objects", fb->GetSize()); - - TMessage m(kPROOF_FEEDBACK); - m << fb; - - // send message to client; - if (gProofServ) gProofServ->GetSocket()->Send(m); - - delete fb; - - fFeedbackTimer->Start(fFeedbackPeriod, kTRUE); - - return kFALSE; // ignored? -} - -//////////////////////////////////////////////////////////////////////////////// -/// Handle tree header request. - -void TProofPlayerSlave::HandleGetTreeHeader(TMessage *mess) -{ - TMessage answ(kPROOF_GETTREEHEADER); - - TDSet *dset; - (*mess) >> dset; - dset->Reset(); - TDSetElement *e = dset->Next(); - Long64_t entries = 0; - TFile *f = 0; - TTree *t = 0; - if (!e) { - PDB(kGlobal, 1) Info("HandleGetTreeHeader", "empty TDSet"); - } else { - f = TFile::Open(e->GetFileName()); - t = 0; - if (f) { - t = (TTree*) f->Get(e->GetObjName()); - if (t) { - t->SetMaxVirtualSize(0); - t->DropBaskets(); - entries = t->GetEntries(); - - // compute #entries in all the files - while ((e = dset->Next()) != 0) { - TFile *f1 = TFile::Open(e->GetFileName()); - if (f1) { - TTree *t1 = (TTree*) f1->Get(e->GetObjName()); - if (t1) { - entries += t1->GetEntries(); - delete t1; - } - delete f1; - } - } - t->SetMaxEntryLoop(entries); // this field will hold the total number of entries ;) - } - } - } - if (t) - answ << TString("Success") << t; - else - answ << TString("Failed") << t; - - fSocket->Send(answ); - - SafeDelete(t); - SafeDelete(f); -} - - -//------------------------------------------------------------------------------ - -ClassImp(TProofPlayerSuperMaster); - -//////////////////////////////////////////////////////////////////////////////// -/// Process specified TDSet on PROOF. Runs on super master. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProofPlayerSuperMaster::Process(TDSet *dset, const char *selector_file, - Option_t *option, Long64_t nentries, - Long64_t first) -{ - fProgressStatus->Reset(); - PDB(kGlobal,1) Info("Process","Enter"); - - TProofSuperMaster *proof = dynamic_cast(GetProof()); - if (!proof) return -1; - - delete fOutput; - fOutput = new THashList; - - TPerfStats::Start(fInput, fOutput); - - if (!SendSelector(selector_file)) { - Error("Process", "sending selector %s", selector_file); - return -1; - } - - TCleanup clean(this); - SetupFeedback(); - - if (proof->IsMaster()) { - - // make sure the DSet is valid - if (!dset->ElementsValid()) { - proof->ValidateDSet(dset); - if (!dset->ElementsValid()) { - Error("Process", "could not validate TDSet"); - return -1; - } - } - - TList msds; - msds.SetOwner(); // This will delete TPairs - - TList keyholder; // List to clean up key part of the pairs - keyholder.SetOwner(); - TList valueholder; // List to clean up value part of the pairs - valueholder.SetOwner(); - - // Construct msd list using the slaves - TIter nextslave(proof->GetListOfActiveSlaves()); - while (TSlave *sl = dynamic_cast(nextslave())) { - TList *submasters = 0; - TPair *msd = dynamic_cast(msds.FindObject(sl->GetMsd())); - if (!msd) { - submasters = new TList; - submasters->SetName(sl->GetMsd()); - keyholder.Add(submasters); - TList *setelements = new TSortedList(kSortDescending); - setelements->SetName(TString(sl->GetMsd())+"_Elements"); - valueholder.Add(setelements); - msds.Add(new TPair(submasters, setelements)); - } else { - submasters = dynamic_cast(msd->Key()); - } - if (submasters) submasters->Add(sl); - } - - // Add TDSetElements to msd list - Long64_t cur = 0; //start of next element - TIter nextelement(dset->GetListOfElements()); - while (TDSetElement *elem = dynamic_cast(nextelement())) { - - if (elem->GetNum()<1) continue; // get rid of empty elements - - if (nentries !=-1 && cur>=first+nentries) { - // we are done - break; - } - - if (cur+elem->GetNum()-1GetNum(); - continue; - } - - if (curSetNum(elem->GetNum()-(first-cur)); - elem->SetFirst(elem->GetFirst()+first-cur); - cur=first; - } - - if (nentries==-1 || cur+elem->GetNum()<=first+nentries) { - cur+=elem->GetNum(); - } else { - //modify element to get proper end - elem->SetNum(first+nentries-cur); - cur=first+nentries; - } - - TPair *msd = dynamic_cast(msds.FindObject(elem->GetMsd())); - if (!msd) { - Error("Process", "data requires mass storage domain '%s'" - " which is not accessible in this proof session", - elem->GetMsd()); - return -1; - } else { - TList *elements = dynamic_cast(msd->Value()); - if (elements) elements->Add(elem); - } - } - - TList usedmasters; - TIter nextmsd(msds.MakeIterator()); - while (TPair *msd = dynamic_cast(nextmsd())) { - TList *submasters = dynamic_cast(msd->Key()); - TList *setelements = dynamic_cast(msd->Value()); - - // distribute elements over the masters - Int_t nmasters = submasters ? submasters->GetSize() : -1; - Int_t nelements = setelements ? setelements->GetSize() : -1; - for (Int_t i=0; iGetType(), dset->GetObjName(), - dset->GetDirectory()); - for (Int_t j = (i*nelements)/nmasters; - j < ((i+1)*nelements)/nmasters; - j++) { - TDSetElement *elem = setelements ? - dynamic_cast(setelements->At(j)) : (TDSetElement *)0; - if (elem) { - set.Add(elem->GetFileName(), elem->GetObjName(), - elem->GetDirectory(), elem->GetFirst(), - elem->GetNum(), elem->GetMsd()); - nent += elem->GetNum(); - } else { - Warning("Process", "not a TDSetElement object"); - } - } - - if (set.GetListOfElements()->GetSize()>0) { - TMessage mesg(kPROOF_PROCESS); - TString fn(gSystem->BaseName(selector_file)); - TString opt = option; - mesg << &set << fn << fInput << opt << Long64_t(-1) << Long64_t(0); - - TSlave *sl = dynamic_cast(submasters->At(i)); - if (sl) { - PDB(kGlobal,1) Info("Process", - "Sending TDSet with %d elements to submaster %s", - set.GetListOfElements()->GetSize(), - sl->GetOrdinal()); - sl->GetSocket()->Send(mesg); - usedmasters.Add(sl); - - // setup progress info - fSlaves.AddLast(sl); - fSlaveProgress.Set(fSlaveProgress.GetSize()+1); - fSlaveProgress[fSlaveProgress.GetSize()-1] = 0; - fSlaveTotals.Set(fSlaveTotals.GetSize()+1); - fSlaveTotals[fSlaveTotals.GetSize()-1] = nent; - fSlaveBytesRead.Set(fSlaveBytesRead.GetSize()+1); - fSlaveBytesRead[fSlaveBytesRead.GetSize()-1] = 0; - fSlaveInitTime.Set(fSlaveInitTime.GetSize()+1); - fSlaveInitTime[fSlaveInitTime.GetSize()-1] = -1.; - fSlaveProcTime.Set(fSlaveProcTime.GetSize()+1); - fSlaveProcTime[fSlaveProcTime.GetSize()-1] = -1.; - fSlaveEvtRti.Set(fSlaveEvtRti.GetSize()+1); - fSlaveEvtRti[fSlaveEvtRti.GetSize()-1] = -1.; - fSlaveMBRti.Set(fSlaveMBRti.GetSize()+1); - fSlaveMBRti[fSlaveMBRti.GetSize()-1] = -1.; - fSlaveActW.Set(fSlaveActW.GetSize()+1); - fSlaveActW[fSlaveActW.GetSize()-1] = 0; - fSlaveTotS.Set(fSlaveTotS.GetSize()+1); - fSlaveTotS[fSlaveTotS.GetSize()-1] = 0; - fSlaveEffS.Set(fSlaveEffS.GetSize()+1); - fSlaveEffS[fSlaveEffS.GetSize()-1] = 0.; - } else { - Warning("Process", "not a TSlave object"); - } - } - } - } - - if ( !IsClient() ) HandleTimer(0); - PDB(kGlobal,1) Info("Process","Calling Collect"); - proof->Collect(&usedmasters); - HandleTimer(0); - - } - - StopFeedback(); - - PDB(kGlobal,1) Info("Process","Calling Merge Output"); - MergeOutput(); - - TPerfStats::Stop(); - - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Report progress. - -void TProofPlayerSuperMaster::Progress(TSlave *sl, Long64_t total, Long64_t processed) -{ - Int_t idx = fSlaves.IndexOf(sl); - fSlaveProgress[idx] = processed; - if (fSlaveTotals[idx] != total) - Warning("Progress", "total events has changed for slave %s", sl->GetName()); - fSlaveTotals[idx] = total; - - Long64_t tot = 0; - Int_t i; - for (i = 0; i < fSlaveTotals.GetSize(); i++) tot += fSlaveTotals[i]; - Long64_t proc = 0; - for (i = 0; i < fSlaveProgress.GetSize(); i++) proc += fSlaveProgress[i]; - - Progress(tot, proc); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Report progress. - -void TProofPlayerSuperMaster::Progress(TSlave *sl, Long64_t total, - Long64_t processed, Long64_t bytesread, - Float_t initTime, Float_t procTime, - Float_t evtrti, Float_t mbrti) -{ - PDB(kGlobal,2) - Info("Progress","%s: %lld %lld %f %f %f %f", sl->GetName(), - processed, bytesread, initTime, procTime, evtrti, mbrti); - - Int_t idx = fSlaves.IndexOf(sl); - if (fSlaveTotals[idx] != total) - Warning("Progress", "total events has changed for slave %s", sl->GetName()); - fSlaveTotals[idx] = total; - fSlaveProgress[idx] = processed; - fSlaveBytesRead[idx] = bytesread; - fSlaveInitTime[idx] = (initTime > -1.) ? initTime : fSlaveInitTime[idx]; - fSlaveProcTime[idx] = (procTime > -1.) ? procTime : fSlaveProcTime[idx]; - fSlaveEvtRti[idx] = (evtrti > -1.) ? evtrti : fSlaveEvtRti[idx]; - fSlaveMBRti[idx] = (mbrti > -1.) ? mbrti : fSlaveMBRti[idx]; - - Int_t i; - Long64_t tot = 0; - Long64_t proc = 0; - Long64_t bytes = 0; - Float_t init = -1.; - Float_t ptime = -1.; - Float_t erti = 0.; - Float_t srti = 0.; - Int_t nerti = 0; - Int_t nsrti = 0; - for (i = 0; i < fSlaveTotals.GetSize(); i++) { - tot += fSlaveTotals[i]; - if (i < fSlaveProgress.GetSize()) - proc += fSlaveProgress[i]; - if (i < fSlaveBytesRead.GetSize()) - bytes += fSlaveBytesRead[i]; - if (i < fSlaveInitTime.GetSize()) - if (fSlaveInitTime[i] > -1. && (init < 0. || fSlaveInitTime[i] < init)) - init = fSlaveInitTime[i]; - if (i < fSlaveProcTime.GetSize()) - if (fSlaveProcTime[i] > -1. && (ptime < 0. || fSlaveProcTime[i] > ptime)) - ptime = fSlaveProcTime[i]; - if (i < fSlaveEvtRti.GetSize()) - if (fSlaveEvtRti[i] > -1.) { - erti += fSlaveEvtRti[i]; - nerti++; - } - if (i < fSlaveMBRti.GetSize()) - if (fSlaveMBRti[i] > -1.) { - srti += fSlaveMBRti[i]; - nsrti++; - } - } - srti = (nsrti > 0) ? srti / nerti : 0.; - - Progress(tot, proc, bytes, init, ptime, erti, srti); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Progress signal. - -void TProofPlayerSuperMaster::Progress(TSlave *wrk, TProofProgressInfo *pi) -{ - if (pi) { - PDB(kGlobal,2) - Info("Progress","%s: %lld %lld %lld %f %f %f %f %d %f", wrk->GetOrdinal(), - pi->fTotal, pi->fProcessed, pi->fBytesRead, - pi->fInitTime, pi->fProcTime, pi->fEvtRateI, pi->fMBRateI, - pi->fActWorkers, pi->fEffSessions); - - Int_t idx = fSlaves.IndexOf(wrk); - if (fSlaveTotals[idx] != pi->fTotal) - Warning("Progress", "total events has changed for worker %s", wrk->GetName()); - fSlaveTotals[idx] = pi->fTotal; - fSlaveProgress[idx] = pi->fProcessed; - fSlaveBytesRead[idx] = pi->fBytesRead; - fSlaveInitTime[idx] = (pi->fInitTime > -1.) ? pi->fInitTime : fSlaveInitTime[idx]; - fSlaveProcTime[idx] = (pi->fProcTime > -1.) ? pi->fProcTime : fSlaveProcTime[idx]; - fSlaveEvtRti[idx] = (pi->fEvtRateI > -1.) ? pi->fEvtRateI : fSlaveEvtRti[idx]; - fSlaveMBRti[idx] = (pi->fMBRateI > -1.) ? pi->fMBRateI : fSlaveMBRti[idx]; - fSlaveActW[idx] = (pi->fActWorkers > -1) ? pi->fActWorkers : fSlaveActW[idx]; - fSlaveTotS[idx] = (pi->fTotSessions > -1) ? pi->fTotSessions : fSlaveTotS[idx]; - fSlaveEffS[idx] = (pi->fEffSessions > -1.) ? pi->fEffSessions : fSlaveEffS[idx]; - - Int_t i; - Int_t nerti = 0; - Int_t nsrti = 0; - TProofProgressInfo pisum(0, 0, 0, -1., -1., 0., 0., 0, 0, 0.); - for (i = 0; i < fSlaveTotals.GetSize(); i++) { - pisum.fTotal += fSlaveTotals[i]; - if (i < fSlaveProgress.GetSize()) - pisum.fProcessed += fSlaveProgress[i]; - if (i < fSlaveBytesRead.GetSize()) - pisum.fBytesRead += fSlaveBytesRead[i]; - if (i < fSlaveInitTime.GetSize()) - if (fSlaveInitTime[i] > -1. && (pisum.fInitTime < 0. || fSlaveInitTime[i] < pisum.fInitTime)) - pisum.fInitTime = fSlaveInitTime[i]; - if (i < fSlaveProcTime.GetSize()) - if (fSlaveProcTime[i] > -1. && (pisum.fProcTime < 0. || fSlaveProcTime[i] > pisum.fProcTime)) - pisum.fProcTime = fSlaveProcTime[i]; - if (i < fSlaveEvtRti.GetSize()) - if (fSlaveEvtRti[i] > -1.) { - pisum.fEvtRateI += fSlaveEvtRti[i]; - nerti++; - } - if (i < fSlaveMBRti.GetSize()) - if (fSlaveMBRti[i] > -1.) { - pisum.fMBRateI += fSlaveMBRti[i]; - nsrti++; - } - if (i < fSlaveActW.GetSize()) - pisum.fActWorkers += fSlaveActW[i]; - if (i < fSlaveTotS.GetSize()) - if (fSlaveTotS[i] > -1 && (pisum.fTotSessions < 0. || fSlaveTotS[i] > pisum.fTotSessions)) - pisum.fTotSessions = fSlaveTotS[i]; - if (i < fSlaveEffS.GetSize()) - if (fSlaveEffS[i] > -1. && (pisum.fEffSessions < 0. || fSlaveEffS[i] > pisum.fEffSessions)) - pisum.fEffSessions = fSlaveEffS[i]; - } - pisum.fMBRateI = (nsrti > 0) ? pisum.fMBRateI / nerti : 0.; - - Progress(&pisum); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send progress and feedback to client. - -Bool_t TProofPlayerSuperMaster::HandleTimer(TTimer *) -{ - if (fFeedbackTimer == 0) return kFALSE; // timer stopped already - - Int_t i; - Long64_t tot = 0; - Long64_t proc = 0; - Long64_t bytes = 0; - Float_t init = -1.; - Float_t ptime = -1.; - Float_t erti = 0.; - Float_t srti = 0.; - Int_t nerti = 0; - Int_t nsrti = 0; - for (i = 0; i < fSlaveTotals.GetSize(); i++) { - tot += fSlaveTotals[i]; - if (i < fSlaveProgress.GetSize()) - proc += fSlaveProgress[i]; - if (i < fSlaveBytesRead.GetSize()) - bytes += fSlaveBytesRead[i]; - if (i < fSlaveInitTime.GetSize()) - if (fSlaveInitTime[i] > -1. && (init < 0. || fSlaveInitTime[i] < init)) - init = fSlaveInitTime[i]; - if (i < fSlaveProcTime.GetSize()) - if (fSlaveProcTime[i] > -1. && (ptime < 0. || fSlaveProcTime[i] > ptime)) - ptime = fSlaveProcTime[i]; - if (i < fSlaveEvtRti.GetSize()) - if (fSlaveEvtRti[i] > -1.) { - erti += fSlaveEvtRti[i]; - nerti++; - } - if (i < fSlaveMBRti.GetSize()) - if (fSlaveMBRti[i] > -1.) { - srti += fSlaveMBRti[i]; - nsrti++; - } - } - erti = (nerti > 0) ? erti / nerti : 0.; - srti = (nsrti > 0) ? srti / nerti : 0.; - - TMessage m(kPROOF_PROGRESS); - if (gProofServ->GetProtocol() > 25) { - // Fill the message now - TProofProgressInfo pi(tot, proc, bytes, init, ptime, - erti, srti, -1, - gProofServ->GetTotSessions(), gProofServ->GetEffSessions()); - m << π - } else { - - m << tot << proc << bytes << init << ptime << erti << srti; - } - - // send message to client; - gProofServ->GetSocket()->Send(m); - - if (fReturnFeedback) - return TProofPlayerRemote::HandleTimer(0); - else - return kFALSE; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Setup reporting of feedback objects and progress messages. - -void TProofPlayerSuperMaster::SetupFeedback() -{ - if (IsClient()) return; // Client does not need timer - - TProofPlayerRemote::SetupFeedback(); - - if (fFeedbackTimer) { - fReturnFeedback = kTRUE; - return; - } else { - fReturnFeedback = kFALSE; - } - - // setup the timer for progress message - SafeDelete(fFeedbackTimer); - fFeedbackPeriod = 2000; - TProof::GetParameter(fInput, "PROOF_FeedbackPeriod", fFeedbackPeriod); - fFeedbackTimer = new TTimer; - fFeedbackTimer->SetObject(this); - fFeedbackTimer->Start(fFeedbackPeriod, kTRUE); -} diff --git a/proof/proofplayer/src/TProofPlayerLite.cxx b/proof/proofplayer/src/TProofPlayerLite.cxx deleted file mode 100644 index 3dbb712df03e7..0000000000000 --- a/proof/proofplayer/src/TProofPlayerLite.cxx +++ /dev/null @@ -1,557 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G. Ganis Mar 2008 - -/************************************************************************* - * Copyright (C) 1995-2001, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TProofPlayerLite -\ingroup proofkernel - -Version of TProofPlayerRemote merges the functionality needed by clients -and masters. It is used in optmized local sessions. - -*/ - -#include "TProofPlayerLite.h" - -#include "MessageTypes.h" -#include "TDSet.h" -#include "TDSetProxy.h" -#include "TEntryList.h" -#include "TEventList.h" -#include "THashList.h" -#include "TMap.h" -#include "TMessage.h" -#include "TObjString.h" -#include "TPerfStats.h" -#include "TProofLite.h" -#include "TProofDebug.h" -#include "TProofServ.h" -#include "TROOT.h" -#include "TSelector.h" -#include "TVirtualPacketizer.h" - -//////////////////////////////////////////////////////////////////////////////// -/// Create the selector object and save the relevant files and binary information -/// in the cache so that the worker can pick it up. -/// Returns 0 and fill fSelector in case of success. Returns -1 and sets -/// fSelector to 0 in case of failure. - -Int_t TProofPlayerLite::MakeSelector(const char *selfile) -{ - fSelectorClass = 0; - SafeDelete(fSelector); - if (!selfile || strlen(selfile) <= 0) { - Error("MakeSelector", "input file path or name undefined"); - return -1; - } - - // If we are just given a name, init the selector and return - if (!strchr(gSystem->BaseName(selfile), '.')) { - if (gDebug > 1) - Info("MakeSelector", "selector name '%s' does not contain a '.':" - " no file to check, it will be loaded from a library", selfile); - if (!(fSelector = TSelector::GetSelector(selfile))) { - Error("MakeSelector", "could not create a %s selector", selfile); - return -1; - } - // Done - return 0; - } - - if (((TProofLite*)fProof)->CopyMacroToCache(selfile, 1, &fSelector, TProof::kCp | TProof::kCpBin) < 0) - return -1; - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process specified TDSet on PROOF. -/// This method is called on client and on the PROOF master. -/// The return value is -1 in case of an error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProofPlayerLite::Process(TDSet *dset, TSelector *selector, - Option_t *option, Long64_t nentries, - Long64_t first) -{ - if (!selector) { - Error("Process", "selector object undefined"); - return -1; - } - - // Define fSelector in Client - if (selector != fSelector) { - SafeDelete(fSelector); - fSelector = selector; - } - - fCreateSelObj = kFALSE; - Long64_t rc = Process(dset, selector->ClassName(), option, nentries, first); - fCreateSelObj = kTRUE; - - // Done - return rc; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Process specified TDSet on PROOF. -/// This method is called on client and on the PROOF master. -/// The return value is -1 in case of error and TSelector::GetStatus() in -/// in case of success. - -Long64_t TProofPlayerLite::Process(TDSet *dset, const char *selector_file, - Option_t *option, Long64_t nentries, - Long64_t first) -{ - PDB(kGlobal,1) Info("Process","Enter"); - fDSet = dset; - fExitStatus = kFinished; - - if (!fProgressStatus) { - Error("Process", "No progress status"); - return -1; - } - fProgressStatus->Reset(); - - // delete fOutput; - if (!fOutput) - fOutput = new THashList; - else - fOutput->Clear(); - - TPerfStats::Setup(fInput); - TPerfStats::Start(fInput, fOutput); - - TStopwatch elapsed; - - TMessage mesg(kPROOF_PROCESS); - TString fn(gSystem->BaseName(selector_file)); - - // Parse option - Bool_t sync = (fProof->GetQueryMode(option) == TProof::kSync); - - // Make sure that the temporary output list is empty - if (fOutputLists) { - fOutputLists->Delete(); - delete fOutputLists; - fOutputLists = 0; - } - - if (!sync) { - gSystem->RedirectOutput(fProof->fLogFileName); - Printf(" "); - Info("Process","starting new query"); - } - - if (fCreateSelObj) { - if (MakeSelector(selector_file) != 0) { - if (!sync) - gSystem->RedirectOutput(0); - return -1; - } - } - - fSelectorClass = fSelector->IsA(); - // Add fSelector to inputlist if processing with object - TList *inputtmp = 0; // List of temporary input objects - if (!fCreateSelObj) { - // In any input list was set into the selector move it to the PROOF - // input list, because we do not want to stream the selector one - if (fSelector->GetInputList() && fSelector->GetInputList()->GetSize() > 0) { - TIter nxi(fSelector->GetInputList()); - TObject *o = 0; - while ((o = nxi())) { - if (!fInput->FindObject(o)) { - fInput->Add(o); - if (!inputtmp) { - inputtmp = new TList; - inputtmp->SetOwner(kFALSE); - } - inputtmp->Add(o); - } - } - } - fInput->Add(fSelector); - } - // Set the input list for initialization - fSelector->SetInputList(fInput); - fSelector->SetOption(option); - if (fSelector->GetOutputList()) fSelector->GetOutputList()->Clear(); - - PDB(kLoop,1) Info("Process","Call Begin(0)"); - fSelector->Begin(0); - - // Send large input data objects, if any - gProof->SendInputDataFile(); - - // Attach to the transient histogram with the assigned packets, if required - if (fInput->FindObject("PROOF_StatsHist") != 0) { - if (!(fProcPackets = (TH1 *) fOutput->FindObject("PROOF_ProcPcktHist"))) { - Warning("Process", "could not attach to histogram 'PROOF_ProcPcktHist'"); - } else { - PDB(kLoop,1) - Info("Process", "attached to histogram 'PROOF_ProcPcktHist' to record" - " packets being processed"); - } - } - - PDB(kPacketizer,1) Info("Process","Create Proxy TDSet"); - TDSet *set = new TDSetProxy(dset->GetType(), dset->GetObjName(), - dset->GetDirectory()); - if (dset->TestBit(TDSet::kEmpty)) - set->SetBit(TDSet::kEmpty); - fProof->SetParameter("PROOF_MaxSlavesPerNode", (Long_t) 0); - if (InitPacketizer(dset, nentries, first, "TPacketizerUnit", "TPacketizer") != 0) { - Error("Process", "cannot init the packetizer"); - fExitStatus = kAborted; - return -1; - } - // reset start, this is now managed by the packetizer - first = 0; - - // Negative memlogfreq disable checks. - // If 0 is passed we try to have 100 messages about memory - // Otherwise we use the frequency passed. - Int_t mrc = -1; - Long64_t memlogfreq = -1, mlf; - if ((mrc = TProof::GetParameter(fProof->GetInputList(), "PROOF_MemLogFreq", mlf)) == 0) memlogfreq = mlf; - if (mrc != 0 && gSystem->Getenv("PROOF_MEMLOGFREQ")) { - TString clf(gSystem->Getenv("PROOF_MEMLOGFREQ")); - if (clf.IsDigit()) { memlogfreq = clf.Atoi(); mrc = 0; } - } - if (memlogfreq == 0) { - memlogfreq = fPacketizer->GetTotalEntries()/(fProof->GetParallel()*100); - if (memlogfreq <= 0) memlogfreq = 1; - } - if (mrc == 0) fProof->SetParameter("PROOF_MemLogFreq", memlogfreq); - - // Add the unique query tag as TNamed object to the input list - // so that it is available in TSelectors for monitoring - fProof->SetParameter("PROOF_QueryTag", fProof->GetName()); - // ... and the sequential number - fProof->SetParameter("PROOF_QuerySeqNum", fProof->fSeqNum); - - if (!sync) - gSystem->RedirectOutput(0); - - TCleanup clean(this); - SetupFeedback(); - - TString opt = option; - - // Workers will get the entry ranges from the packetizer - Long64_t num = (fProof->IsParallel()) ? -1 : nentries; - Long64_t fst = (fProof->IsParallel()) ? -1 : first; - - // Entry- or Event- list ? - TEntryList *enl = (!fProof->IsMaster()) ? dynamic_cast(set->GetEntryList()) - : (TEntryList *)0; - TEventList *evl = (!fProof->IsMaster() && !enl) ? dynamic_cast(set->GetEntryList()) - : (TEventList *)0; - // Reset the merging progress information - fProof->ResetMergePrg(); - - // Broadcast main message - PDB(kGlobal,1) Info("Process","Calling Broadcast"); - if (fProcessMessage) delete fProcessMessage; - fProcessMessage = new TMessage(kPROOF_PROCESS); - mesg << set << fn << fInput << opt << num << fst << evl << sync << enl; - (*fProcessMessage) << set << fn << fInput << opt << num << fst << evl << sync << enl; - Int_t nb = fProof->Broadcast(mesg); - PDB(kGlobal,1) Info("Process", "Broadcast called: %d workers notified", nb); - fProof->fNotIdle += nb; - - // Redirect logs from master to special log frame - fProof->fRedirLog = kTRUE; - - if (!sync) { - - // Asynchronous query: just make sure that asynchronous input - // is enabled and return the prompt - PDB(kGlobal,1) Info("Process","Asynchronous processing:" - " activating CollectInputFrom"); - fProof->Activate(); - - // Return the query sequential number - return fProof->fSeqNum; - - } else { - - // Wait for processing - PDB(kGlobal,1) Info("Process","Synchronous processing: calling Collect"); - fProof->Collect(); - - // Restore prompt logging (Collect leaves things as they were - // at the time it was called) - fProof->fRedirLog = kFALSE; - - if (!TSelector::IsStandardDraw(fn)) - HandleTimer(0); // force an update of final result - if (fPacketizer) { - fPacketizer->StopProcess(kFALSE, kTRUE); - // The progress timer will now stop itself at the next call - fPacketizer->SetBit(TVirtualPacketizer::kIsDone); - // Store process info - elapsed.Stop(); - if (fQuery) - fQuery->SetProcessInfo(0, 0., fPacketizer->GetBytesRead(), - fPacketizer->GetInitTime(), - elapsed.RealTime()); - } - StopFeedback(); - - Long64_t rc = -1; - if (GetExitStatus() != TProofPlayer::kAborted) - rc = Finalize(kFALSE, sync); - - // Remove temporary input objects, if any - if (inputtmp) { - TIter nxi(inputtmp); - TObject *o = 0; - while ((o = nxi())) fInput->Remove(o); - SafeDelete(inputtmp); - } - - // Done - return rc; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Finalize a query. -/// Returns -1 in case error, 0 otherwise. - -Long64_t TProofPlayerLite::Finalize(Bool_t force, Bool_t sync) -{ - if (fOutputLists == 0) { - if (force && fQuery) - return fProof->Finalize(Form("%s:%s", fQuery->GetTitle(), - fQuery->GetName()), force); - } - - Long64_t rv = 0; - - TPerfStats::Stop(); - - if (!fQuery) { - Info("Finalize", "query is undefined!"); - return -1; - } - - // Some objects (e.g. histos in autobin) may not have been merged yet - // do it now - MergeOutput(); - - if (fExitStatus != kAborted) { - - if (!sync) { - // Reinit selector (with multi-sessioning we must do this until - // TSelector::GetSelector() is optimized to i) avoid reloading of an - // unchanged selector and ii) invalidate existing instances of - // reloaded selector) - if (ReinitSelector(fQuery) == -1) { - Info("Finalize", "problems reinitializing selector \"%s\"", - fQuery->GetSelecImp()->GetName()); - return -1; - } - } - - // Some input parameters may be needed in Terminate - fSelector->SetInputList(fInput); - - TList *output = fSelector->GetOutputList(); - if (output) { - TIter next(fOutput); - while(TObject* obj = next()) { - if (fProof->IsParallel() || DrawCanvas(obj) == 1) - // Either parallel or not a canvas or not able to display it: - // just add to the list - output->Add(obj); - } - } else { - Warning("Finalize", "undefined output list in the selector! Protocol error?"); - } - - SetSelectorDataMembersFromOutputList(); - - PDB(kLoop,1) Info("Finalize","Call Terminate()"); - fOutput->Clear("nodelete"); - // This is the end of merging - SetMerging(kFALSE); - // We measure the merge time - fProof->fQuerySTW.Reset(); - // Call Terminate now - fSelector->Terminate(); - - rv = fSelector->GetStatus(); - - // copy the output list back and clean the selector's list - TIter it(output); - while(TObject* o = it()) { - fOutput->Add(o); - } - - // Save the output list in the current query, if any - if (fQuery) { - fQuery->SetOutputList(fOutput); - // Set in finalized state (cannot be done twice) - fQuery->SetFinalized(); - } else { - Warning("Finalize","current TQueryResult object is undefined!"); - } - - if (!fCreateSelObj) { - fInput->Remove(fSelector); - fOutput->Remove(fSelector); - if (output) output->Remove(fSelector); - fSelector = 0; - } - - // We have transferred copy of the output objects in TQueryResult, - // so now we can cleanup the selector, making sure that we do not - // touch the output objects - if (output) output->SetOwner(kFALSE); - SafeDelete(fSelector); - - // Delete fOutput (not needed anymore, cannot be finalized twice), - // making sure that the objects saved in TQueryResult are not deleted - fOutput->SetOwner(kFALSE); - SafeDelete(fOutput); - } else { - - // Cleanup - fOutput->SetOwner(); - SafeDelete(fSelector); - if (!fCreateSelObj) fSelector = 0; - } - - PDB(kGlobal,1) Info("Finalize","exit"); - return rv; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Send feedback objects to client. - -Bool_t TProofPlayerLite::HandleTimer(TTimer *) -{ - PDB(kFeedback,2) - Info("HandleTimer","Entry: %p", fFeedbackTimer); - - if (fFeedbackTimer == 0) return kFALSE; // timer already switched off - - - // process local feedback objects - - TList *fb = new TList; - fb->SetOwner(); - - TIter next(fFeedback); - while( TObjString *name = (TObjString*) next() ) { - TObject *o = fOutput->FindObject(name->GetName()); - if (o != 0) fb->Add(o->Clone()); - } - - if (fb->GetSize() > 0) - StoreFeedback(this, fb); // adopts fb - else - delete fb; - - if (fFeedbackLists == 0) { - fFeedbackTimer->Start(fFeedbackPeriod, kTRUE); // maybe next time - return kFALSE; - } - - fb = MergeFeedback(); - - Feedback(fb); - fb->SetOwner(); - delete fb; - - fFeedbackTimer->Start(fFeedbackPeriod, kTRUE); - - return kFALSE; // ignored? -} - -//////////////////////////////////////////////////////////////////////////////// -/// Setup reporting of feedback objects. - -void TProofPlayerLite::SetupFeedback() -{ - fFeedback = (TList*) fInput->FindObject("FeedbackList"); - - if (fFeedback) { - PDB(kFeedback,1) - Info("SetupFeedback","\"FeedbackList\" found: %d objects", fFeedback->GetSize()); - } else { - PDB(kFeedback,1) - Info("SetupFeedback","\"FeedbackList\" NOT found"); - } - - if (fFeedback == 0 || fFeedback->GetSize() == 0) return; - - // OK, feedback was requested, setup the timer - SafeDelete(fFeedbackTimer); - fFeedbackPeriod = 2000; - TProof::GetParameter(fInput, "PROOF_FeedbackPeriod", fFeedbackPeriod); - fFeedbackTimer = new TTimer; - fFeedbackTimer->SetObject(this); - fFeedbackTimer->Start(fFeedbackPeriod, kTRUE); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Store feedback results from the specified slave. - -void TProofPlayerLite::StoreFeedback(TObject *slave, TList *out) -{ - PDB(kFeedback,1) - Info("StoreFeedback","Enter (%p,%p,%d)", fFeedbackLists, out, (out ? out->GetSize() : -1)); - - if ( out == 0 ) { - PDB(kFeedback,1) - Info("StoreFeedback","Leave (empty)"); - return; - } - - if (fFeedbackLists == 0) { - PDB(kFeedback,2) Info("StoreFeedback","Create fFeedbackLists"); - fFeedbackLists = new TList; - fFeedbackLists->SetOwner(); - } - - TIter next(out); - out->SetOwner(kFALSE); // take ownership of the contents - - TObject *obj; - while( (obj = next()) ) { - PDB(kFeedback,2) - Info("StoreFeedback","Find '%s'", obj->GetName() ); - - TMap *map = (TMap*) fFeedbackLists->FindObject(obj->GetName()); - if ( map == 0 ) { - PDB(kFeedback,2) - Info("StoreFeedback", "map for '%s' not found (creating)", obj->GetName()); - // map must not be owner (ownership is with regards to the keys (only)) - map = new TMap; - map->SetName(obj->GetName()); - fFeedbackLists->Add(map); - } else { - PDB(kFeedback,2) - Info("StoreFeedback","removing previous value"); - if (map->GetValue(slave)) - delete map->GetValue(slave); - map->Remove(slave); - } - map->Add(slave, obj); - } - - delete out; - PDB(kFeedback,1) - Info("StoreFeedback","Leave"); -} diff --git a/proof/proofplayer/src/TStatsFeedback.cxx b/proof/proofplayer/src/TStatsFeedback.cxx deleted file mode 100644 index 2063337790f35..0000000000000 --- a/proof/proofplayer/src/TStatsFeedback.cxx +++ /dev/null @@ -1,151 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: G. Ganis May 2012 - -/************************************************************************* - * Copyright (C) 1995-2000, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - - -/** \class TStatsFeedback -\ingroup proofkernel - -Utility class to display PROOF stats feedback histos during queries - -*/ - -#include "TStatsFeedback.h" - -#include "TError.h" -#include "TH1.h" -#include "TH2.h" -#include "THashList.h" -#include "TProof.h" -#include "TProofDebug.h" -#include "TROOT.h" -#include "TSeqCollection.h" -#include "TStyle.h" -#include "TVirtualPad.h" - -ClassImp(TStatsFeedback); - - -//////////////////////////////////////////////////////////////////////////////// -/// Constructor - -TStatsFeedback::TStatsFeedback(TProof *proof) -{ - if (proof == 0) proof = gProof; - - TProof *p = dynamic_cast(proof); - if (p == 0) { - Error("TStatsFeedback", "no valid proof session found"); - SetBit(TObject::kInvalidObject); - return; - } - fProof = p; - fName = fProof->GetSessionTag(); - - if (!(proof->Connect("Feedback(TList*)", "TStatsFeedback", - this, "Feedback(TList*)"))) { - Error("TStatsFeedback", "Connect() failed"); - SetBit(TObject::kInvalidObject); - return; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor - -TStatsFeedback::~TStatsFeedback() -{ - // Required since we overload TObject::Hash. - ROOT::CallRecursiveRemoveIfNeeded(*this); - - fProof->Disconnect("Feedback(TList*)", this, "Feedback(TList*"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Display feedback - -void TStatsFeedback::Feedback(TList *objs) -{ - TSeqCollection *canvases = gROOT->GetListOfCanvases(); - - PDB(kFeedback,1) Info("Feedback", "%d Objects", objs->GetSize()); - - // Attach to the histograms we want to plot - TH1D *hevt = 0, *hpck = 0; - TH1I *hass = 0; - TIter next(objs); - TObject *o = 0; - while((o = next())) { - if (!strcmp(o->GetName(), "PROOF_EventsHist")) { - hevt = dynamic_cast(o); - } else if (!strcmp(o->GetName(), "PROOF_PacketsHist")) { - hpck = dynamic_cast(o); - } else if (!strcmp(o->GetName(), "PROOF_ProcPcktHist")) { - hass = dynamic_cast(o); - } - if (hevt && hpck && hass) break; - } - if (!hevt && !hpck && !hass) { - Warning("Feedback", "none of the requested histograms has been found!"); - return; - } - - // Number of histograms - Int_t nh = 3; - if (!hass) nh = 2; - // Create or attach to canvas - TString cvnm = TString::Format("Stats: %s", fProof->GetSessionTag()); - TVirtualPad *cv = 0; - if (gROOT->GetListOfCanvases()) - cv = (TVirtualPad *) canvases->FindObject(cvnm.Data()); - if (cv && nh == 3 && !cv->GetPad(3)) SafeDelete(cv); - if (!cv) { - Int_t h = (nh == 3) ? 600 : 400; - TString cvcmd = TString::Format("new TCanvas(\"%s\", \"Feedback Stats\",10,300,600,%d)", - cvnm.Data(), h); - if (!(cv = (TVirtualPad *) gROOT->ProcessLine(cvcmd))) { - Warning("Feedback", "could not create canvas!"); - return; - } - PDB(kFeedback,2) Info("Feedback", "created canvas %s", cvnm.Data()); - // Create pads - cv->Divide(1, nh); - } else { - cv->cd(); - PDB(kFeedback,2) Info("Feedback", "using canvas %s", cvnm.Data()); - } - TVirtualPad *pd1 = (TVirtualPad *) cv->GetPad(1); - TVirtualPad *pd2 = (TVirtualPad *) cv->GetPad(2); - TVirtualPad *pd3 = (nh == 3) ? (TVirtualPad *) cv->GetPad(3) : 0; - - UInt_t optstat = gStyle->GetOptStat(); - gStyle->SetOptStat(11); - // Plot - if (hevt) { - if (pd1) pd1->cd(); - hevt->SetFillColor(kGreen); - hevt->DrawCopy(); - } - if (hpck) { - if (pd2) pd2->cd(); - hpck->SetFillColor(kAzure-5); - hpck->DrawCopy(); - } - if (hass) { - if (pd3) pd3->cd(); - hass->SetFillColor(kGray); - hass->SetMaximum(2); - hass->DrawCopy(); - } - - cv->cd(); - cv->Update(); - gStyle->SetOptStat(optstat); -} diff --git a/proof/proofplayer/src/TStatus.cxx b/proof/proofplayer/src/TStatus.cxx deleted file mode 100644 index 3945342aed8d8..0000000000000 --- a/proof/proofplayer/src/TStatus.cxx +++ /dev/null @@ -1,215 +0,0 @@ -// @(#)root/proofplayer:$Id$ -// Author: Maarten Ballintijn 7/06/2004 - -/************************************************************************* - * Copyright (C) 1995-2004, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TStatus -\ingroup proofkernel - -This class holds the status of an ongoing operation and collects -error messages. It provides a Merge() operation allowing it to -be used in PROOF to monitor status in the slaves. -No messages indicates success. - -*/ - -#include "TStatus.h" -#include "Riostream.h" -#include "TBuffer.h" -#include "TClass.h" -#include "TObjString.h" -#include "TProofDebug.h" - -ClassImp(TStatus); - -//////////////////////////////////////////////////////////////////////////////// -/// Default constructor. - -TStatus::TStatus() : fIter(&fMsgs), fExitStatus(-1), - fVirtMemMax(-1), fResMemMax(-1), - fVirtMaxMst(-1), fResMaxMst(-1) -{ - SetName("PROOF_Status"); - fMsgs.SetOwner(kTRUE); - fInfoMsgs.SetOwner(kTRUE); - ResetBit(TStatus::kNotOk); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add an error message. - -void TStatus::Add(const char *mesg) -{ - fMsgs.Add(new TObjString(mesg)); - SetBit(TStatus::kNotOk); - Reset(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add an info message. - -void TStatus::AddInfo(const char *mesg) -{ - fInfoMsgs.Add(new TObjString(mesg)); -} - -//////////////////////////////////////////////////////////////////////////////// -/// PROOF Merge() function. - -Int_t TStatus::Merge(TCollection *li) -{ - TIter stats(li); - PDB(kOutput,1) - Info("Merge", "start: max virtual memory: %.2f MB \tmax resident memory: %.2f MB ", - GetVirtMemMax()/1024., GetResMemMax()/1024.); - while (TObject *obj = stats()) { - TStatus *s = dynamic_cast(obj); - if (s == 0) continue; - - TObjString *os = 0; - // Errors - TIter nxem(&(s->fMsgs)); - while ((os = (TObjString *) nxem())) { - Add(os->GetName()); - } - - // Infos (no duplications) - TIter nxwm(&(s->fInfoMsgs)); - while ((os = (TObjString *) nxwm())) { - if (!fInfoMsgs.FindObject(os->GetName())) - AddInfo(os->GetName()); - } - - SetMemValues(s->GetVirtMemMax(), s->GetResMemMax()); - // Check the master values (relevantt if merging submaster info) - SetMemValues(s->GetVirtMemMax(kTRUE), s->GetResMemMax(kTRUE), kTRUE); - PDB(kOutput,1) - Info("Merge", "during: max virtual memory: %.2f MB \t" - "max resident memory: %.2f MB ", - GetVirtMemMax()/1024., GetResMemMax()/1024.); - if (GetVirtMemMax(kTRUE) > 0) { - PDB(kOutput,1) - Info("Merge", "during: max master virtual memory: %.2f MB \t" - "max master resident memory: %.2f MB ", - GetVirtMemMax(kTRUE)/1024., GetResMemMax(kTRUE)/1024.); - } - } - - return fMsgs.GetSize(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Standard print function. - -void TStatus::Print(Option_t * /*option*/) const -{ - Printf("OBJ: %s\t%s\t%s", IsA()->GetName(), GetName(), (IsOk() ? "OK" : "ERROR")); - - TObjString *os = 0; - // Errors first - if (fMsgs.GetSize() > 0) { - Printf("\n Errors:"); - TIter nxem(&fMsgs); - while ((os = (TObjString *) nxem())) - Printf("\t%s",os->GetName()); - Printf(" "); - } - - // Infos - if (fInfoMsgs.GetSize() > 0) { - Printf("\n Infos:"); - TIter nxem(&fInfoMsgs); - while ((os = (TObjString *) nxem())) - Printf("\t%s",os->GetName()); - Printf(" "); - } - - Printf(" Max worker virtual memory: %.2f MB \tMax worker resident memory: %.2f MB ", - GetVirtMemMax()/1024., GetResMemMax()/1024.); - Printf(" Max master virtual memory: %.2f MB \tMax master resident memory: %.2f MB ", - GetVirtMemMax(kTRUE)/1024., GetResMemMax(kTRUE)/1024.); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Reset the iterator on the messages. - -void TStatus::Reset() -{ - fIter.Reset(); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return the next message or 0. - -const char *TStatus::NextMesg() -{ - TObjString *os = (TObjString *) fIter(); - if (os) return os->GetName(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set max memory values - -void TStatus::SetMemValues(Long_t vmem, Long_t rmem, Bool_t master) -{ - if (master) { - if (vmem > 0. && (fVirtMaxMst < 0. || vmem > fVirtMaxMst)) fVirtMaxMst = vmem; - if (rmem > 0. && (fResMaxMst < 0. || rmem > fResMaxMst)) fResMaxMst = rmem; - } else { - if (vmem > 0. && (fVirtMemMax < 0. || vmem > fVirtMemMax)) fVirtMemMax = vmem; - if (rmem > 0. && (fResMemMax < 0. || rmem > fResMemMax)) fResMemMax = rmem; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Stream an object of class TStatus. - -void TStatus::Streamer(TBuffer &R__b) -{ - if (R__b.IsReading()) { - UInt_t R__s, R__c; - Version_t R__v = R__b.ReadVersion(&R__s, &R__c); - if (R__v > 4) { - R__b.ReadClassBuffer(TStatus::Class(), this, R__v, R__s, R__c); - } else { - // For version <= 4 masters we need a special streamer - TNamed::Streamer(R__b); - std::set msgs; - TClass *cl = TClass::GetClass("set"); - if (cl) { - UInt_t SS__s = 0, SS__c = 0; - UInt_t SS__v = cl->GetClassVersion(); - R__b.ReadClassBuffer(cl, &msgs, SS__v, SS__s, SS__c); - } else { - Error("Streamer", "no info found for 'set' - skip"); - return; - } - std::set::const_iterator it; - for (it = msgs.begin(); it != msgs.end(); ++it) { - fMsgs.Add(new TObjString((*it).c_str())); - } - if (R__v > 2) { - R__b >> fExitStatus; - } - if (R__v > 1) { - R__b >> fVirtMemMax; - R__b >> fResMemMax; - } - if (R__v > 3) { - R__b >> fVirtMaxMst; - R__b >> fResMaxMst; - } - } - } else { - R__b.WriteClassBuffer(TStatus::Class(),this); - } -} - - diff --git a/roofit/roofitcore/src/RooStudyPackage.cxx b/roofit/roofitcore/src/RooStudyPackage.cxx index 01973ad0d2a3d..95f12ef753257 100644 --- a/roofit/roofitcore/src/RooStudyPackage.cxx +++ b/roofit/roofitcore/src/RooStudyPackage.cxx @@ -179,17 +179,6 @@ Int_t RooStudyPackage::initRandom() //gRandom->SetSeed(0) ; Int_t seed = random.Integer(TMath::Limits::Max()) ; - // get worker number - TString worknumber = gEnv->GetValue("ProofServ.Ordinal","undef"); - int iworker = -1; - if (worknumber != "undef") - iworker = int( worknumber.Atof()*10 + 0.1); - - if (iworker >= 0) { - for (int i = 0; i <= iworker; ++i ) - seed = random.Integer( TMath::Limits::Max() ); - } - RooRandom::randomGenerator()->SetSeed(seed) ; gRandom->SetSeed(seed) ; diff --git a/roofit/roostats/inc/RooStats/SimpleLikelihoodRatioTestStat.h b/roofit/roostats/inc/RooStats/SimpleLikelihoodRatioTestStat.h index 2a779421ebd6b..e12d83148c008 100644 --- a/roofit/roostats/inc/RooStats/SimpleLikelihoodRatioTestStat.h +++ b/roofit/roostats/inc/RooStats/SimpleLikelihoodRatioTestStat.h @@ -24,7 +24,7 @@ namespace RooStats { public: - /// Constructor for proof. Do not use. + /// Constructor for proof, now removed. Do not use. SimpleLikelihoodRatioTestStat() = default; /// Takes null and alternate parameters from PDF. Can be overridden. diff --git a/roottest/root/io/arrayobject/Makefile b/roottest/root/io/arrayobject/Makefile index 2164d6682e559..4e63a1dfdd2b5 100644 --- a/roottest/root/io/arrayobject/Makefile +++ b/roottest/root/io/arrayobject/Makefile @@ -31,7 +31,7 @@ RealArraySave.log: foo_C.$(DllSuf) bar_C.$(DllSuf) main_C.$(DllSuf) $(ROOT_LOC)/ $(CMDECHO) $(CALLROOTEXE) -q -b -l RealArraySave.C > RealArraySave.log #main: main.C fooDict.C barDict.C foo.C bar.C -# g++ -g -o $@ $^ -I. `root-config --cflags --libs` -lTreePlayer -lProof +# g++ -g -o $@ $^ -I. `root-config --cflags --libs` -lTreePlayer #libFoo.so: main.C fooDict.C barDict.C foo.C bar.C # g++ -g -shared -o $@ $^ -I. `root-config --cflags --libs` diff --git a/roottest/root/meta/execTypedefList.C b/roottest/root/meta/execTypedefList.C index b969f685c1ab9..0eb6ee9de5009 100644 --- a/roottest/root/meta/execTypedefList.C +++ b/roottest/root/meta/execTypedefList.C @@ -77,13 +77,6 @@ int check_exist(const char *name) #include #include -bool IsProofEnabled() { - return nullptr != strstr(gROOT->GetConfigFeatures(), "proof"); -} -bool IsProofType(const char *what) { - return nullptr != strstr(what, "TProof") || nullptr != strstr(what, "TDSet"); -} - int check_file(const char *filename, int expected_count) { std::ifstream f(filename); @@ -91,18 +84,9 @@ int check_file(const char *filename, int expected_count) int count = 0; int found = 0; char what[1000]; - const bool isProofEnabled = IsProofEnabled(); while( f.getline(what,1000) ) { ++count; if (what[0]=='#') continue; - // If Proof is disabled and this is a name related to it, skip the analysis - if (isProofEnabled || !IsProofType(what)) { - int lres = check_exist(what); - if (lres) { - fprintf(stderr,"Failed on count == %d in %s\n",count,filename); - res = lres; - } - } ++found; } if (found != expected_count) { @@ -120,7 +104,7 @@ int execTypedefList() { int res; // Just in case we have a small pch. - const char *whatToLoad [] = { "TPainter3dAlgorithms", "TLego", "TAuthenticate", "TProofDraw", "TChainIndex", "TF1", "TGeoBoolNode", "TShape", "TXMLEngine" }; + const char *whatToLoad [] = { "TPainter3dAlgorithms", "TLego", "TAuthenticate", "TChainIndex", "TF1", "TGeoBoolNode", "TShape", "TXMLEngine" }; for(unsigned int i = 0 ; i < sizeof(whatToLoad) / sizeof(const char*); ++i) { gInterpreter->AutoLoad(whatToLoad[i]); gInterpreter->AutoParse(whatToLoad[i]); @@ -151,16 +135,16 @@ int execTypedefList() { #if defined(_MSC_VER) res = check_file("typelist_win32.v5.txt",334); if (res) return res; #if __cplusplus > 201402L - res = check_file("typelist_win32.v6.cxx17.txt",1408); if (res) return res; + res = check_file("typelist_win32.v6.cxx17.txt",1370); if (res) return res; #else - res = check_file("typelist_win32.v6.txt",1420); if (res) return res; + res = check_file("typelist_win32.v6.txt",1382); if (res) return res; #endif #elif defined(R__MACOSX) && __cplusplus > 201402L res = check_file("typelist.v5.txt",334); if (res) return res; - res = check_file("typelist.v6.cxx17.txt",1310); if (res) return res; + res = check_file("typelist.v6.cxx17.txt",1280); if (res) return res; #else res = check_file("typelist.v5.txt",334); if (res) return res; - res = check_file("typelist.v6.txt",1322); if (res) return res; + res = check_file("typelist.v6.txt",1292); if (res) return res; #endif return 0; diff --git a/roottest/root/meta/typelist.v6.cxx17.txt b/roottest/root/meta/typelist.v6.cxx17.txt index 9c143965bc7f4..551e03c22f868 100644 --- a/roottest/root/meta/typelist.v6.cxx17.txt +++ b/roottest/root/meta/typelist.v6.cxx17.txt @@ -37,12 +37,6 @@ allocator >::value_type allocator::difference_type allocator::size_type allocator::value_type -allocator::difference_type -allocator::size_type -allocator::value_type -allocator::difference_type -allocator::size_type -allocator::value_type allocator::difference_type allocator::size_type allocator::value_type @@ -1088,30 +1082,6 @@ vector::reference vector::reverse_iterator vector::size_type vector::value_type -vector::allocator_type -vector::const_iterator -vector::const_pointer -vector::const_reference -vector::const_reverse_iterator -vector::difference_type -vector::iterator -vector::pointer -vector::reference -vector::reverse_iterator -vector::size_type -vector::value_type -vector::allocator_type -vector::const_iterator -vector::const_pointer -vector::const_reference -vector::const_reverse_iterator -vector::difference_type -vector::iterator -vector::pointer -vector::reference -vector::reverse_iterator -vector::size_type -vector::value_type vector::allocator_type vector::const_iterator vector::const_pointer diff --git a/roottest/root/meta/typelist.v6.txt b/roottest/root/meta/typelist.v6.txt index 653710e975ff1..310bd68c1140b 100644 --- a/roottest/root/meta/typelist.v6.txt +++ b/roottest/root/meta/typelist.v6.txt @@ -37,12 +37,6 @@ allocator >::value_type allocator::difference_type allocator::size_type allocator::value_type -allocator::difference_type -allocator::size_type -allocator::value_type -allocator::difference_type -allocator::size_type -allocator::value_type allocator::difference_type allocator::size_type allocator::value_type @@ -1100,30 +1094,6 @@ vector::reference vector::reverse_iterator vector::size_type vector::value_type -vector::allocator_type -vector::const_iterator -vector::const_pointer -vector::const_reference -vector::const_reverse_iterator -vector::difference_type -vector::iterator -vector::pointer -vector::reference -vector::reverse_iterator -vector::size_type -vector::value_type -vector::allocator_type -vector::const_iterator -vector::const_pointer -vector::const_reference -vector::const_reverse_iterator -vector::difference_type -vector::iterator -vector::pointer -vector::reference -vector::reverse_iterator -vector::size_type -vector::value_type vector::allocator_type vector::const_iterator vector::const_pointer diff --git a/roottest/root/meta/typelist_win32.v6.cxx17.txt b/roottest/root/meta/typelist_win32.v6.cxx17.txt index 0971c12e829ce..2f6aa667198c8 100644 --- a/roottest/root/meta/typelist_win32.v6.cxx17.txt +++ b/roottest/root/meta/typelist_win32.v6.cxx17.txt @@ -89,20 +89,6 @@ allocator::pointer allocator::reference allocator::size_type allocator::value_type -allocator::const_pointer -allocator::const_reference -allocator::difference_type -allocator::pointer -allocator::reference -allocator::size_type -allocator::value_type -allocator::const_pointer -allocator::const_reference -allocator::difference_type -allocator::pointer -allocator::reference -allocator::size_type -allocator::value_type allocator::const_pointer allocator::const_reference allocator::difference_type @@ -1186,30 +1172,6 @@ vector::reference vector::reverse_iterator vector::size_type vector::value_type -vector::allocator_type -vector::const_iterator -vector::const_pointer -vector::const_reference -vector::const_reverse_iterator -vector::difference_type -vector::iterator -vector::pointer -vector::reference -vector::reverse_iterator -vector::size_type -vector::value_type -vector::allocator_type -vector::const_iterator -vector::const_pointer -vector::const_reference -vector::const_reverse_iterator -vector::difference_type -vector::iterator -vector::pointer -vector::reference -vector::reverse_iterator -vector::size_type -vector::value_type vector::allocator_type vector::const_iterator vector::const_pointer diff --git a/roottest/root/meta/typelist_win32.v6.txt b/roottest/root/meta/typelist_win32.v6.txt index d4a8c354008af..337a6e2bff108 100644 --- a/roottest/root/meta/typelist_win32.v6.txt +++ b/roottest/root/meta/typelist_win32.v6.txt @@ -89,20 +89,6 @@ allocator::pointer allocator::reference allocator::size_type allocator::value_type -allocator::const_pointer -allocator::const_reference -allocator::difference_type -allocator::pointer -allocator::reference -allocator::size_type -allocator::value_type -allocator::const_pointer -allocator::const_reference -allocator::difference_type -allocator::pointer -allocator::reference -allocator::size_type -allocator::value_type allocator::const_pointer allocator::const_reference allocator::difference_type @@ -1198,30 +1184,6 @@ vector::reference vector::reverse_iterator vector::size_type vector::value_type -vector::allocator_type -vector::const_iterator -vector::const_pointer -vector::const_reference -vector::const_reverse_iterator -vector::difference_type -vector::iterator -vector::pointer -vector::reference -vector::reverse_iterator -vector::size_type -vector::value_type -vector::allocator_type -vector::const_iterator -vector::const_pointer -vector::const_reference -vector::const_reverse_iterator -vector::difference_type -vector::iterator -vector::pointer -vector::reference -vector::reverse_iterator -vector::size_type -vector::value_type vector::allocator_type vector::const_iterator vector::const_pointer diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 4d59a9ef7d971..a203af71e33a4 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -294,18 +294,6 @@ ROOT_ADD_TEST(test-stressinterpreter COMMAND stressInterpreter ROOT_EXECUTABLE(stressHepix stressHepix.cxx LIBRARIES Core) #ROOT_ADD_TEST(test-stressHepix COMMAND stressHepix FAILREGEX "FAILED|Error in") -#--stressProof------------------------------------------------------------------------------- -if(proof AND NOT WIN32) - add_custom_target(TestData COMMAND ${CMAKE_COMMAND} -DDST=${CMAKE_SOURCE_DIR}/files -P ${CMAKE_CURRENT_SOURCE_DIR}/rootDownloadData.cmake) - ROOT_EXECUTABLE(stressProof stressProof.cxx LIBRARIES Proof ProofPlayer Hist) - ROOT_ADD_TEST(test-stressproof PRECMD ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target TestData - COMMAND stressProof lite:// - -h1 ${CMAKE_SOURCE_DIR}/files/h1 -event ${CMAKE_SOURCE_DIR}/files/event - -l /tmp/stressProof-%d.log -cleanlog -catlog -noprogress - ENVIRONMENT PATH=${CMAKE_BINARY_DIR}/bin:$ENV{PATH} - FAILREGEX "FAILED|Error in") -endif() - #--testbits---------------------------------------------------------------------------------- ROOT_EXECUTABLE(testbits testbits.cxx LIBRARIES Core) ROOT_ADD_TEST(test-testbits COMMAND testbits) diff --git a/test/CTestCustom.cmake b/test/CTestCustom.cmake index 9fd14bb5649ed..b49f5144f2660 100644 --- a/test/CTestCustom.cmake +++ b/test/CTestCustom.cmake @@ -1,6 +1,6 @@ #---Custom CTest settings--------------------------------------------------- -set(CTEST_CUSTOM_TESTS_IGNORE ${CTEST_CUSTOM_TESTS_IGNORE} test-stressgui test-stressproof) +set(CTEST_CUSTOM_TESTS_IGNORE ${CTEST_CUSTOM_TESTS_IGNORE} test-stressgui) if(WIN32) set(CTEST_CUSTOM_TESTS_IGNORE ${CTEST_CUSTOM_TESTS_IGNORE} test-tcollex) diff --git a/test/Makefile b/test/Makefile deleted file mode 100644 index 4cd8dd79c3eeb..0000000000000 --- a/test/Makefile +++ /dev/null @@ -1,811 +0,0 @@ -# Makefile for the ROOT test programs. -# This Makefile shows nicely how to compile and link applications -# using the ROOT libraries on all supported platforms. -# -# Copyright (c) 2000 Rene Brun and Fons Rademakers -# -# Author: Fons Rademakers, 29/2/2000 - -RC := root-config -ifeq ($(findstring $(MAKECMDTARGET),clean),) -ifeq ($(shell which $(RC) 2>&1 | sed -ne "s@.*/$(RC)@$(RC)@p"),$(RC)) -MKARCH := $(wildcard $(shell $(RC) --etcdir)/Makefile.arch) -RCONFIG := $(wildcard $(shell $(RC) --incdir)/RConfigure.h) -endif -ifneq ($(MKARCH),) -include $(MKARCH) -else -ifeq ($(ROOTSYS),) -ROOTSYS = .. -endif -include $(ROOTSYS)/etc/Makefile.arch -endif -endif -# Tutorials dir needed by stressProof -ifneq ($(RCONFIG),) -TUTDIR := $(wildcard $(shell grep ROOTDOCDIR $(RCONFIG) | sed "s|.*\"\(.*\)\"|\1|")/tutorials) -endif -ifeq ($(TUTDIR),) -ifeq ($(ROOTSYS),) -ROOTSYS = .. -endif -TUTDIR := $(ROOTSYS)/tutorials -endif --include ../MyConfig.mk - -#------------------------------------------------------------------------------ - -EVENTO = Event.$(ObjSuf) EventDict.$(ObjSuf) -EVENTS = Event.$(SrcSuf) EventDict.$(SrcSuf) -EVENTSO = libEvent.$(DllSuf) -EVENT = Event$(ExeSuf) -ifeq ($(PLATFORM),win32) -EVENTLIBS = $(ROOTSYS)/lib/libTreePlayer.lib -EVENTLIB = libEvent.lib -else -EVENTLIBS = -lTreePlayer -EVENTLIB = $(shell pwd)/$(EVENTSO) -endif - -EVENTMTO = EventMT.$(ObjSuf) EventMTDict.$(ObjSuf) -EVENTMTS = EventMT.$(SrcSuf) EventMTDict.$(SrcSuf) -EVENTMTSO = libEventMT.$(DllSuf) -ifeq ($(PLATFORM),win32) -EVENTMTLIB = libEventMT.lib -else -EVENTMTLIB = $(shell pwd)/$(EVENTMTSO) -endif - -MAINEVENTO = MainEvent.$(ObjSuf) -MAINEVENTS = MainEvent.$(SrcSuf) - -HWORLDO = hworld.$(ObjSuf) -HWORLDS = hworld.$(SrcSuf) -HWORLD = hworld$(ExeSuf) - -CTORTUREO = ctorture.$(ObjSuf) -CTORTURES = ctorture.$(SrcSuf) -CTORTURE = ctorture$(ExeSuf) - -HSIMPLEO = hsimple.$(ObjSuf) -HSIMPLES = hsimple.$(SrcSuf) -HSIMPLE = hsimple$(ExeSuf) - -MINEXAMO = minexam.$(ObjSuf) -MINEXAMS = minexam.$(SrcSuf) -MINEXAM = minexam$(ExeSuf) - -TFORMULAO = TFormulaTests.$(ObjSuf) -TFORMULAS = TFormulaTests.$(SrcSuf) -TFORMULA = TFormulaTests$(ExeSuf) - -TSTRINGO = tstring.$(ObjSuf) -TSTRINGS = tstring.$(SrcSuf) -TSTRING = tstring$(ExeSuf) - -TCOLLEXO = tcollex.$(ObjSuf) -TCOLLEXS = tcollex.$(SrcSuf) -TCOLLEX = tcollex$(ExeSuf) - -TCOLLBMO = tcollbm.$(ObjSuf) -TCOLLBMS = tcollbm.$(SrcSuf) -TCOLLBM = tcollbm$(ExeSuf) - -VVECTORO = vvector.$(ObjSuf) -VVECTORS = vvector.$(SrcSuf) -VVECTOR = vvector$(ExeSuf) - -VMATRIXO = vmatrix.$(ObjSuf) -VMATRIXS = vmatrix.$(SrcSuf) -VMATRIX = vmatrix$(ExeSuf) - -STRESSLO = stressLinear.$(ObjSuf) -STRESSLS = stressLinear.$(SrcSuf) -STRESSL = stressLinear$(ExeSuf) - -STRESSGO = stressGraphics.$(ObjSuf) -STRESSGS = stressGraphics.$(SrcSuf) -STRESSG = stressGraphics$(ExeSuf) - -STRESSGUIO = stressGUI.$(ObjSuf) -STRESSGUIS = stressGUI.$(SrcSuf) -STRESSGUI = stressGUI$(ExeSuf) -ifeq ($(PLATFORM),win32) -STRESSGUILIBS = '$(ROOTSYS)/lib/libASImage.lib' '$(ROOTSYS)/lib/libASImageGui.lib' \ - '$(ROOTSYS)/lib/libRecorder.lib' '$(ROOTSYS)/lib/libGuiHtml.lib' -else -STRESSGUILIBS = -lASImage -lASImageGui -lRecorder -lGuiHtml -endif - -ifeq ($(shell $(RC) --has-genvector),yes) -STRESSVECO = stressVector.$(ObjSuf) -STRESSVECS = stressVector.$(SrcSuf) -STRESSVEC = stressVector$(ExeSuf) - -STRESSMATHO = stressMathCore.$(ObjSuf) -STRESSMATHS = stressMathCore.$(SrcSuf) -ifneq ($(USE_REFLEX),) -CXXFLAGS += -DUSE_REFLEX -ifeq ($(PLATFORM),win32) -STRESSMATHLIBS = '$(ROOTSYS)/lib/libGenVector.lib' -else -STRESSMATHLIBS = -lGenVector -endif - -TRACKMATHSRC = TrackMathCoreRflx.$(SrcSuf) -TRACKMATHOBJ = TrackMathCoreRflx.$(ObjSuf) -TRACKMATHLIB = libTrackMathCoreRflx.$(DllSuf) - -else - -ifeq ($(PLATFORM),win32) -STRESSMATHLIBS = '$(ROOTSYS)/lib/libGenVector.lib' -else -STRESSMATHLIBS = -lGenVector -endif - -TRACKMATHSRC = TrackMathCoreDict.$(SrcSuf) -TRACKMATHOBJ = TrackMathCoreDict.$(ObjSuf) -TRACKMATHLIB = libTrackMathCoreDict.$(DllSuf) - -endif - -STRESSMATH = stressMathCore$(ExeSuf) - -endif - -ifeq ($(shell $(RC) --has-mathmore),yes) -STRESSMATHMOREO = stressMathMore.$(ObjSuf) -STRESSMATHMORES = stressMathMore.$(SrcSuf) -ifeq ($(PLATFORM),win32) -STRESSMATHMORELIBS = '$(ROOTSYS)/lib/libMathMore.lib' -else -STRESSMATHMORELIBS = -lMathMore -endif -STRESSMATHMORE = stressMathMore$(ExeSuf) -endif - -ifeq ($(shell $(RC) --has-tmva),yes) -STRESSTMVAO = stressTMVA.$(ObjSuf) -STRESSTMVAS = stressTMVA.$(SrcSuf) -ifeq ($(PLATFORM),win32) -STRESSTMVALIBS = '$(ROOTSYS)/lib/libTMVA.lib' -else -STRESSTMVALIBS = -lTMVA -lMinuit -lXMLIO -lMLP -lTreePlayer -endif -STRESSTMVA = stressTMVA$(ExeSuf) -endif - -VLAZYO = vlazy.$(ObjSuf) -VLAZYS = vlazy.$(SrcSuf) -VLAZY = vlazy$(ExeSuf) - -GUITESTO = guitest.$(ObjSuf) -GUITESTS = guitest.$(SrcSuf) -GUITEST = guitest$(ExeSuf) - -GUIVIEWERO = guiviewer.$(ObjSuf) guiviewerDict.$(ObjSuf) -GUIVIEWERS = guiviewer.$(SrcSuf) guiviewerDict.$(SrcSuf) -GUIVIEWER = guiviewer$(ExeSuf) - -HELLOO = Hello.$(ObjSuf) HelloDict.$(ObjSuf) -HELLOS = Hello.$(SrcSuf) HelloDict.$(SrcSuf) -HELLOSO = Hello.$(DllSuf) - -ACLOCKO = Aclock.$(ObjSuf) AclockDict.$(ObjSuf) -ACLOCKS = Aclock.$(SrcSuf) AclockDict.$(SrcSuf) -ACLOCKSO = Aclock.$(DllSuf) - -TETRISO = Tetris.$(ObjSuf) TetrisDict.$(ObjSuf) -TETRISS = Tetris.$(SrcSuf) TetrisDict.$(SrcSuf) -TETRISSO = Tetris.$(DllSuf) - -STRESSO = stress.$(ObjSuf) -STRESSS = stress.$(SrcSuf) -STRESS = stress$(ExeSuf) - -IOPLUGINSO = stressIOPlugins.$(ObjSuf) -IOPLUGINSS = stressIOPlugins.$(SrcSuf) -IOPLUGINS = stressIOPlugins$(ExeSuf) - -STRESSGEOMETRYO = stressGeometry.$(ObjSuf) -STRESSGEOMETRYS = stressGeometry.$(SrcSuf) -STRESSGEOMETRY = stressGeometry$(ExeSuf) - -STRESSSHAPESO = stressShapes.$(ObjSuf) -STRESSSHAPESS = stressShapes.$(SrcSuf) -STRESSSHAPES = stressShapes$(ExeSuf) - -ifeq ($(shell $(RC) --has-roofit),yes) -STRESSROOFITO = stressRooFit.$(ObjSuf) -STRESSROOFITS = stressRooFit.$(SrcSuf) -STRESSROOFIT = stressRooFit$(ExeSuf) - -STRESSROOSTATSO = stressRooStats.$(ObjSuf) -STRESSROOSTATSS = stressRooStats.$(SrcSuf) -STRESSROOSTATS = stressRooStats$(ExeSuf) - -ifeq ($(shell $(RC) --has-xml),yes) -STRESSHISTFACTORYO = stressHistFactory.$(ObjSuf) -STRESSHISTFACTORYS = stressHistFactory.$(SrcSuf) -STRESSHISTFACTORY = stressHistFactory$(ExeSuf) -endif - -endif - -STRESSFITO = stressFit.$(ObjSuf) -STRESSFITS = stressFit.$(SrcSuf) -STRESSFIT = stressFit$(ExeSuf) - -ifeq ($(shell $(RC) --has-unuran),yes) -ifeq ($(shell $(RC) --has-minuit2),yes) -STRESSHISTOFITO = stressHistoFit.$(ObjSuf) -STRESSHISTOFITS = stressHistoFit.$(SrcSuf) -STRESSHISTOFIT = stressHistoFit$(ExeSuf) -endif -endif - -STRESSENTRYLISTO = stressEntryList.$(ObjSuf) -STRESSENTRYLISTS = stressEntryList.$(SrcSuf) -STRESSENTRYLIST = stressEntryList$(ExeSuf) - -STRESSHEPIXO = stressHepix.$(ObjSuf) -STRESSHEPIXS = stressHepix.$(SrcSuf) -STRESSHEPIX = stressHepix$(ExeSuf) - -STRESSSPO = stressSpectrum.$(ObjSuf) -STRESSSPS = stressSpectrum.$(SrcSuf) -STRESSSP = stressSpectrum$(ExeSuf) - -STRESSPROOFO = stressProof.$(ObjSuf) -STRESSPROOFS = stressProof.$(SrcSuf) -STRESSPROOF = stressProof$(ExeSuf) - -TBENCHO = TBench.$(ObjSuf) TBenchDict.$(ObjSuf) -TBENCHS = TBench.$(SrcSuf) TBenchDict.$(SrcSuf) -TBENCHSO = TBench.$(DllSuf) - -BENCHO = bench.$(ObjSuf) -BENCHS = bench.$(SrcSuf) -BENCH = bench$(ExeSuf) - -TESTBITSO = testbits.$(ObjSuf) -TESTBITSS = testbits.$(SrcSuf) -TESTBITS = testbits$(ExeSuf) - -QPRANDOMO = QpRandomDriver.$(ObjSuf) -QPRANDOMS = QpRandomDriver.$(SrcSuf) -QPRANDOM = QpRandomDriver$(ExeSuf) - -THREADSO = threads.$(ObjSuf) -THREADSS = threads.$(SrcSuf) -THREADS = threads$(ExeSuf) - -STRESSINTERPO = stressInterpreter.$(ObjSuf) -STRESSINTERPS = stressInterpreter.$(SrcSuf) -STRESSINTERP = stressInterpreter$(ExeSuf) - -STRESSITERO = stressIterators.$(ObjSuf) -STRESSITERS = stressIterators.$(SrcSuf) -STRESSITER = stressIterators$(ExeSuf) - -STRESSHISTO = stressHistogram.$(ObjSuf) -STRESSHISTS = stressHistogram.$(SrcSuf) -STRESSHIST = stressHistogram$(ExeSuf) - -ifeq ($(shell $(RC) --has-sqlite),yes) -SQLITETESTO = sqlitetest.$(ObjSuf) -SQLITETESTS = sqlitetest.$(SrcSuf) -SQLITETEST = sqlitetest$(ExeSuf) -endif - - -OBJS = $(EVENTO) $(MAINEVENTO) $(EVENTMTO) $(HWORLDO) $(HSIMPLEO) \ - $(MINEXAMO) $(TFORMULAO) \ - $(TSTRINGO) $(TCOLLEXO) $(VVECTORO) $(VMATRIXO) $(VLAZYO) \ - $(HELLOO) $(ACLOCKO) $(STRESSO) $(TBENCHO) $(BENCHO) \ - $(STRESSSHAPESO) $(TCOLLBMO) $(STRESSGEOMETRYO) $(STRESSLO) \ - $(STRESSGO) $(STRESSSPO) $(TESTBITSO) \ - $(CTORTUREO) $(QPRANDOMO) $(THREADSO) $(STRESSVECO) \ - $(STRESSMATHO) $(STRESSFITO) $(STRESSHISTOFITO) \ - $(STRESSHEPIXO) $(STRESSENTRYLISTO) $(STRESSROOFITO) \ - $(STRESSROOSTATSO) $(STRESSHISTFACTORYO) \ - $(STRESSPROOFO) $(STRESSMATHMOREO) \ - $(STRESSTMVAO) $(STRESSINTERPO) $(STRESSITERO) \ - $(STRESSHISTO) $(STRESSGUIO) $(SQLITETESTO) $(IOPLUGINSO) - -PROGRAMS = $(EVENT) $(EVENTMTSO) $(HWORLD) $(HSIMPLE) $(MINEXAM) $(TFORMULA) \ - $(TSTRING) $(TCOLLEX) $(TCOLLBM) $(VVECTOR) $(VMATRIX) \ - $(VLAZY) $(HELLOSO) $(ACLOCKSO) $(STRESS) $(TBENCHSO) $(BENCH) \ - $(STRESSSHAPES) $(STRESSGEOMETRY) $(STRESSL) $(STRESSG) \ - $(TESTBITS) $(CTORTURE) $(QPRANDOM) $(THREADS) $(STRESSSP) \ - $(STRESSVEC) $(STRESSFIT) $(STRESSHISTOFIT) $(STRESSHEPIX) \ - $(STRESSENTRYLIST) $(STRESSROOFIT) $(STRESSROOSTATS) \ - $(STRESSHISTFACTORY) $(STRESSPROOF) $(STRESSMATH) \ - $(STRESSMATHMORE) $(STRESSTMVA) $(STRESSINTERP) $(STRESSITER) \ - $(STRESSHIST) $(STRESSGUI) $(SQLITETEST) $(IOPLUGINS) - - -OBJS += $(GUITESTO) $(GUIVIEWERO) $(TETRISO) -PROGRAMS += $(GUITEST) $(GUIVIEWER) $(TETRISSO) - -ifeq ($(ARCH),aix5) -MAKESHARED = /usr/vacpp/bin/makeC++SharedLib -endif - -#------------------------------------------------------------------------------ - -.SUFFIXES: .$(SrcSuf) .$(ObjSuf) .$(DllSuf) -.PHONY: Aclock Hello Tetris - -all: $(PROGRAMS) - -$(EVENTSO): $(EVENTO) -ifeq ($(ARCH),aix5) - $(MAKESHARED) $(OutPutOpt) $@ $(LIBS) -p 0 $^ -else -ifeq ($(PLATFORM),macosx) - $(LD) $(SOFLAGS)$@ $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -else -ifeq ($(PLATFORM),win32) - bindexplib $* $^ > $*.def - lib -nologo -MACHINE:IX86 $^ -def:$*.def \ - $(OutPutOpt)$(EVENTLIB) - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $*.exp $(LIBS) \ - $(OutPutOpt)$@ - $(MT_DLL) -else - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -endif -endif -endif - @echo "$@ done" - -$(EVENTMTSO): $(EVENTMTO) -ifeq ($(ARCH),aix5) - $(MAKESHARED) $(OutPutOpt) $@ $(LIBS) -p 0 $^ -else -ifeq ($(PLATFORM),macosx) - $(LD) $(SOFLAGS)$@ $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -else -ifeq ($(PLATFORM),win32) - bindexplib $* $^ > $*.def - lib -nologo -MACHINE:IX86 $^ -def:$*.def \ - $(OutPutOpt)$(EVENTMTLIB) - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $*.exp $(LIBS) \ - $(OutPutOpt)$@ - $(MT_DLL) -else - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -endif -endif -endif - @echo "$@ done" - -$(EVENT): $(EVENTSO) $(MAINEVENTO) - $(LD) $(LDFLAGS) $(MAINEVENTO) $(EVENTO) $(LIBS) $(EVENTLIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(HWORLD): $(HWORLDO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(CTORTURE): $(CTORTUREO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(HSIMPLE): $(HSIMPLEO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(MINEXAM): $(MINEXAMO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(TFORMULA): $(TFORMULAO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(TSTRING): $(TSTRINGO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(TCOLLEX): $(TCOLLEXO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(TCOLLBM): $(TCOLLBMO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(VVECTOR): $(VVECTORO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(VMATRIX): $(VMATRIXO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(VLAZY): $(VLAZYO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESSL): $(STRESSLO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESSG): $(STRESSGO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESSGUI): $(STRESSGUIO) - $(LD) $(LDFLAGS) $^ $(GLIBS) $(STRESSGUILIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESSSP): $(STRESSSPO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libSpectrum.lib' $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lSpectrum $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(STRESSVEC): $(STRESSVECO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libGenVector.lib' $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lGenVector $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(TRACKMATHLIB): $(TRACKMATHOBJ) -ifeq ($(ARCH),aix5) - $(MAKESHARED) $(OutPutOpt) $@ $(LIBS) -p 0 $^ -else -ifeq ($(PLATFORM),macosx) - $(LD) $(SOFLAGS)$@ $(LDFLAGS) $^ $(OutPutOpt) $@ $(LIBS) $(STRESSMATHLIBS) -else - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $(LIBS) $(STRESSMATHLIBS) $(OutPutOpt)$@ - $(MT_DLL) -endif -endif - -$(STRESSMATH): $(STRESSMATHO) $(TRACKMATHLIB) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $(STRESSMATHO) $(LIBS) $(STRESSMATHLIBS) $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $(STRESSMATHO) $(LIBS) $(STRESSMATHLIBS) $(OutPutOpt)$@ -endif - @echo "$@ done" - - -$(STRESSMATHMORE): $(STRESSMATHMOREO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) $(STRESSMATHMORELIBS) $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) $(STRESSMATHMORELIBS) $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(STRESSTMVA): $(STRESSTMVAO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) $(STRESSTMVALIBS) $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) $(STRESSTMVALIBS) $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(TESTBITS): $(TESTBITSO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(THREADS): $(THREADSO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libThread.lib' $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lThread $(OutPutOpt)$@ - @echo "$@ done" -endif - -$(QPRANDOM): $(QPRANDOMO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libQuadp.lib' $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lQuadp $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(GUITEST): $(GUITESTO) - $(LD) $(LDFLAGS) $^ $(GLIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(GUIVIEWER): $(GUIVIEWERO) - $(LD) $(LDFLAGS) $^ $(GLIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESS): $(STRESSO) $(EVENT) - $(LD) $(LDFLAGS) $(STRESSO) $(EVENTO) $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(IOPLUGINS): $(IOPLUGINSO) $(EVENT) - $(LD) $(LDFLAGS) $(IOPLUGINSO) $(EVENTO) $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESSGEOMETRY): $(STRESSGEOMETRYO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libGeom.lib' $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lGeom $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(STRESSSHAPES): $(STRESSSHAPESO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libGeom.lib' $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lGeom $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(STRESSFIT): $(STRESSFITO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESSHISTOFIT): $(STRESSHISTOFITO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libUnuran.lib' $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lUnuran $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(STRESSENTRYLIST): $(STRESSENTRYLISTO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSHEPIX): $(STRESSHEPIXO) $(STRESSGEOMETRY) $(STRESSFIT) $(STRESSL) \ - $(STRESSSP) $(STRESS) - $(LD) $(LDFLAGS) $(STRESSHEPIXO) $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESSROOFITO): stressRooFit_tests.h -$(STRESSROOSTATSO): stressRooStats_tests.h stressRooStats_models.h -$(STRESSHISTFACTORYO): stressHistFactory_tests.cxx stressHistFactory_models.cxx - -ifeq ($(shell $(RC) --has-mathmore),yes) -ifeq ($(PLATFORM),win32) -EXTRAROOFITLIBS = '$(ROOTSYS)/lib/libMathMore.lib' -else -EXTRAROOFITLIBS = -lMathMore -endif -endif - -$(STRESSROOFIT): $(STRESSROOFITO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libRooFit.lib' '$(ROOTSYS)/lib/libRooFitCore.lib' '$(ROOTSYS)/lib/libHtml.lib' '$(ROOTSYS)/lib/libThread.lib' '$(ROOTSYS)/lib/libMinuit.lib' '$(ROOTSYS)/lib/libFoam.lib' '$(ROOTSYS)/lib/libProof.lib' $(EXTRAROOFITLIBS) $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lRooFit -lRooFitCore -lHtml -lThread -lMinuit -lFoam $(EXTRAROOFITLIBS) $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(STRESSROOSTATS): $(STRESSROOSTATSO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libRooStats.lib' '$(ROOTSYS)/lib/libRooFit.lib' '$(ROOTSYS)/lib/libRooFitCore.lib' '$(ROOTSYS)/lib/libHtml.lib' '$(ROOTSYS)/lib/libThread.lib' '$(ROOTSYS)/lib/libMinuit.lib' '$(ROOTSYS)/lib/libFoam.lib' '$(ROOTSYS)/lib/libProof.lib' $(EXTRAROOFITLIBS) $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lRooStats -lRooFit -lRooFitCore -lThread -lMinuit -lFoam $(EXTRAROOFITLIBS) $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(STRESSHISTFACTORY): $(STRESSHISTFACTORYO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libHistFactory.lib' '$(ROOTSYS)/lib/libRooStats.lib' '$(ROOTSYS)/lib/libRooFit.lib' '$(ROOTSYS)/lib/libRooFitCore.lib' '$(ROOTSYS)/\ -lib/libXMLParser.lib' '$(ROOTSYS)/lib/libThread.lib' '$(ROOTSYS)/lib/libMinuit.lib' '$(ROOTSYS)/lib/libFoam.lib' '$(ROOTSYS)/lib/libProof.lib' $(EXTRAROOFITLIBS) $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $^ $(LIBS) -lHistFactory -lRooStats -lRooFit -lRooFitCore -lMinuit -lFoam -lXMLParser $(EXTRAROOFITLIBS) $(OutPutOpt)$@ -endif - @echo "$@ done" - - -$(STRESSPROOF): $(STRESSPROOFO) -ifeq ($(PLATFORM),win32) - $(LD) $(LDFLAGS) $^ $(LIBS) '$(ROOTSYS)/lib/libProof.lib' '$(ROOTSYS)/lib/libThread.lib' $(OutPutOpt)$@ - $(MT_EXE) -else - $(LD) $(LDFLAGS) $(STRESSPROOFO) $(LIBS) -lProof -lProofPlayer -lThread -lSessionViewer $(OutPutOpt)$@ -endif - @echo "$@ done" - -$(BENCH): $(BENCHO) $(TBENCHSO) - $(LD) $(LDFLAGS) $(BENCHO) $(TBENCHO) $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -Hello: $(HELLOSO) -$(HELLOSO): $(HELLOO) -ifeq ($(ARCH),aix5) - $(MAKESHARED) $(OutPutOpt) $@ $(LIBS) -p 0 $^ -else -ifeq ($(PLATFORM),macosx) - $(LD) $(SOFLAGS)$@ $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -else - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $(EXPLLINKLIBS) $(OutPutOpt)$@ - $(MT_DLL) -endif -endif - -Aclock: $(ACLOCKSO) -$(ACLOCKSO): $(ACLOCKO) -ifeq ($(ARCH),aix5) - $(MAKESHARED) $(OutPutOpt) $@ $(LIBS) -p 0 $^ -else -ifeq ($(PLATFORM),macosx) - $(LD) $(SOFLAGS)$@ $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -else - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $(EXPLLINKLIBS) $(OutPutOpt)$@ - $(MT_DLL) -endif -endif - -Tetris: $(TETRISSO) -$(TETRISSO): $(TETRISO) -ifeq ($(ARCH),aix5) - $(MAKESHARED) $(OutPutOpt) $@ $(GLIBS) -p 0 $^ -else -ifeq ($(PLATFORM),macosx) - $(LD) $(SOFLAGS)$@ $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -else - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $(EXPLLINKLIBS) $(OutPutOpt)$@ - $(MT_DLL) -endif -endif - -$(TBENCHSO): $(TBENCHO) -ifeq ($(ARCH),aix5) - $(MAKESHARED) $(OutPutOpt) $@ $(LIBS) -p 0 $^ -else -ifeq ($(PLATFORM),macosx) - $(LD) $(SOFLAGS)$@ $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -else - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $(EXPLLINKLIBS) $(OutPutOpt)$@ - $(MT_DLL) -endif -endif - -$(STRESSINTERP): $(STRESSINTERPO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESSITER): $(STRESSITERO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(STRESSHIST): $(STRESSHISTO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -$(SQLITETEST): $(SQLITETESTO) - $(LD) $(LDFLAGS) $^ $(LIBS) $(OutPutOpt)$@ - $(MT_EXE) - @echo "$@ done" - -clean: - @rm -f $(OBJS) $(TRACKMATHSRC) core *Dict.* cernstaff.root sg* - -distclean: clean - -@(mv -f stressRooStats_ref.root stressRooStats_ref.root- >/dev/null 2>&1;true) - -@(mv -f stressRooFit_ref.root stressRooFit_ref.root- >/dev/null 2>&1;true) - -@(mv -f stressHistFactory_ref.root stressHistFactory_ref.root- >/dev/null 2>&1;true) - @rm -f $(PROGRAMS) $(EVENTSO) $(EVENTLIB) *Dict.* *.def *.exp \ - *.root *.ps *.so *.lib *.dll *.d *.log .def so_locations \ - files/* *.pcm testdb.sqlite - @rm -rf cxx_repository - -@(mv -f stressRooStats_ref.root- stressRooStats_ref.root >/dev/null 2>&1;true) - -@(mv -f stressRooFit_ref.root- stressRooFit_ref.root >/dev/null 2>&1;true) - -@(mv -f stressHistFactory_ref.root- stressHistFactory_ref.root >/dev/null 2>&1;true) - -@cd RootShower && $(MAKE) distclean - -@cd rhtml && $(MAKE) distclean - -@cd RootIDE && $(MAKE) distclean - -@cd periodic && $(MAKE) distclean - -@cd histviewer && $(MAKE) distclean - -.SUFFIXES: .$(SrcSuf) - -### -stressIterators.$(ObjSuf): stressIterators.h - -Event.$(ObjSuf): Event.h -EventMT.$(ObjSuf): EventMT.h -MainEvent.$(ObjSuf): Event.h - -EventDict.$(SrcSuf): Event.h EventLinkDef.h - @echo "Generating dictionary $@..." - $(ROOTCLING) -f $@ -c $^ - -EventMTDict.$(SrcSuf): EventMT.h EventLinkDef.h - @echo "Generating dictionary $@..." - $(ROOTCLING) -f $@ -c $^ - -Hello.$(ObjSuf): Hello.h -HelloDict.$(SrcSuf): Hello.h - @echo "Generating dictionary $@..." - $(ROOTCLING) -f $@ -c $^ - -Aclock.$(ObjSuf): Aclock.h -AclockDict.$(SrcSuf): Aclock.h - @echo "Generating dictionary $@..." - $(ROOTCLING) -f $@ -c $^ - -Tetris.$(ObjSuf): Tetris.h -TetrisDict.$(SrcSuf): Tetris.h - @echo "Generating dictionary $@..." - $(ROOTCLING) -f $@ -c $^ - -TBench.$(ObjSuf): TBench.h -TBenchDict.$(SrcSuf): TBench.h benchLinkDef.h - @echo "Generating dictionary $@..." - $(ROOTCLING) -f $@ -c $^ - -guiviewer.$(ObjSuf): guiviewer.h -guiviewerDict.$(SrcSuf): guiviewer.h guiviewerLinkDef.h - @echo "Generating dictionary $@..." - $(ROOTCLING) -f $@ -c $^ - -stressMathCore.$(ObjSuf): TrackMathCore.h -TrackMathCoreDict.$(SrcSuf): TrackMathCore.h TrackMathCoreLinkDef.h - @echo "Generating dictionary $@ using rootcling ..." - $(ROOTCLING) -f $@ -c $^ - -stressProof.$(ObjSuf): stressProof.$(SrcSuf) - $(CXX) $(CXXFLAGS) -I$(TUTDIR) -c $< - -TrackMathCoreRflx.$(SrcSuf): TrackMathCore.h TrackMathCoreRflx.xml - @echo "Generating dictionary $@ using gccxml ..." - genreflex TrackMathCore.h --selection_file=TrackMathCoreRflx.xml -o TrackMathCoreRflx.cxx -I$(ROOTSYS)/include - -.$(SrcSuf).$(ObjSuf): - $(CXX) $(CXXFLAGS) -c $< - -ifeq ($(GCC_MAJOR),4) -ifeq ($(GCC_MINOR),1) -TBenchDict.o: CXXFLAGS += -Wno-strict-aliasing -endif -endif diff --git a/test/Makefile.win32 b/test/Makefile.win32 deleted file mode 100644 index 0e027feebd681..0000000000000 --- a/test/Makefile.win32 +++ /dev/null @@ -1,609 +0,0 @@ -# Makefile.win32 for root tests on Win32/NMAKE - -!if "$(CFG)" == "" -!if ([findstr /c:"--build=debug" $(ROOTSYS)\bin\root-config > nul ] == 0) -CFG = Win32 Debug -!if ([findstr /c:"--disable-winrtdebug" $(ROOTSYS)\bin\root-config > nul ] == 0) -RUNTIME = Release -!else -RUNTIME = Debug -!endif -!message No configuration specified: Defaulting to Win32 Debug -!message With $(RUNTIME) Runtime DLL (Taken from ROOT config). -!message . -!else -CFG = Win32 Release -RUNTIME = Release -!message No configuration specified: Defaulting to Win32 Release -!message With $(RUNTIME) Runtime DLL (Taken from ROOT config). -!message . -!endif -!else -!if "$(CFG)" == "Win32 Release" -RUNTIME = Release -!elseif "$(CFG)" == "Win32 Debug" -RUNTIME = Debug -!endif -!endif - -!if "$(CFG)" != "Win32 Release" && "$(CFG)" != "Win32 Debug" -!message Invalid configuration "$(CFG)" specified. -!message You can specify a configuration when running NMAKE -!message by defining the macro CFG on the command line. For example: -!message -!message NMAKE /f "Makefile.msc" CFG="Win32 Debug" -!message -!message Possible choices for configuration are: -!message -!message "Win32 Release" (based on "Win32 (x86) Library") -!message "Win32 Debug" (based on "Win32 (x86) Library") -!message -!error An invalid configuration is specified. -!endif - -ObjSuf = obj -SrcSuf = cxx -ExeSuf = .exe -DllSuf = dll -OutPutOpt = -out: - -!IF "$(PLATFORM)" == "x86" -CPU=i386 -DLLENTRY = @12 -MACHINE=IX86 -!endif - -!IF "$(PLATFORM)" == "x64" -CPU=x86_64 -MACHINE=AMD64 -DLLENTRY = -!ENDIF - -# Win32 system with Microsoft Visual C/C++ - -APPVER = 5.01 -cc = cl -link = link -implib = lib -lflags = $(lflags) /INCREMENTAL:NO /NOLOGO -conlflags = $(lflags) -subsystem:console -guilflags = $(lflags) -subsystem:windows -dlllflags = $(lflags) -entry:_DllMainCRTStartup$(DLLENTRY) -dll -CC = $(cc) -CXX = $(cc) -CXXFLAGS = -nologo -EHs -GR -DWIN32 -W3 -D_WIN32 -D_WINDOWS \ - -I$(ROOTSYS)/include -wd4244 -D_CRT_SECURE_NO_DEPRECATE \ - -FIw32pragma.h -LD = $(link) - -!if "$(CFG)" == "Win32 Release" -CXXOPT = -O2 -MD -LDOPT = -opt:ref -!elseif "$(CFG)" == "Win32 Debug" -!if "$(RUNTIME)" == "Debug" -CXXOPT = -Z7 -MDd -!else -CXXOPT = -Z7 -MD -!endif -LDOPT = -debug -!endif - -LDFLAGS = $(LDOPT) $(conlflags) -nologo -SOFLAGS = $(dlllflags:-pdb:none=) -ROOTLIBS = $(ROOTSYS)\lib\libCore.lib $(ROOTSYS)\lib\libHist.lib \ - $(ROOTSYS)\lib\libGraf.lib $(ROOTSYS)\lib\libGraf3d.lib \ - $(ROOTSYS)\lib\libGpad.lib $(ROOTSYS)\lib\libTree.lib \ - $(ROOTSYS)\lib\libRint.lib $(ROOTSYS)\lib\libPostscript.lib \ - $(ROOTSYS)\lib\libMatrix.lib $(ROOTSYS)\lib\libPhysics.lib \ - $(ROOTSYS)\lib\libNet.lib $(ROOTSYS)\lib\libRIO.lib \ - $(ROOTSYS)\lib\libMathCore.lib $(ROOTSYS)\lib\libTreePlayer.lib -LIBS = $(ROOTLIBS) -GLIBS = $(LIBS) $(ROOTSYS)\lib\libGui.lib $(ROOTSYS)\lib\libGraf.lib \ - $(ROOTSYS)\lib\libGpad.lib -LIBSALL = $(ROOTLIBS) -EVENTLIB = libEvent.lib - -# Tutorials dir needed by stressProof -TUTDIR = $(ROOTSYS)/tutorials - -#------------------------------------------------------------------------------ - -EVENTO = Event.$(ObjSuf) EventDict.$(ObjSuf) -EVENTS = Event.$(SrcSuf) EventDict.$(SrcSuf) -EVENTSO = libEvent.$(DllSuf) -EVENT = Event$(ExeSuf) -EVENTLIB = libEvent.lib - -EVENTMTO = EventMT.$(ObjSuf) EventMTDict.$(ObjSuf) -EVENTMTS = EventMT.$(SrcSuf) EventMTDict.$(SrcSuf) -EVENTMTSO = libEventMT.$(DllSuf) -EVENTMTLIB = libEventMT.lib - -MAINEVENTO = MainEvent.$(ObjSuf) -MAINEVENTS = MainEvent.$(SrcSuf) - -HWORLDO = hworld.$(ObjSuf) -HWORLDS = hworld.$(SrcSuf) -HWORLD = hworld$(ExeSuf) - -CTORTUREO = ctorture.$(ObjSuf) -CTORTURES = ctorture.$(SrcSuf) -CTORTURE = ctorture$(ExeSuf) - -HSIMPLEO = hsimple.$(ObjSuf) -HSIMPLES = hsimple.$(SrcSuf) -HSIMPLE = hsimple$(ExeSuf) - -MINEXAMO = minexam.$(ObjSuf) -MINEXAMS = minexam.$(SrcSuf) -MINEXAM = minexam$(ExeSuf) - -TSTRINGO = tstring.$(ObjSuf) -TSTRINGS = tstring.$(SrcSuf) -TSTRING = tstring$(ExeSuf) - -TCOLLEXO = tcollex.$(ObjSuf) -TCOLLEXS = tcollex.$(SrcSuf) -TCOLLEX = tcollex$(ExeSuf) - -TCOLLBMO = tcollbm.$(ObjSuf) -TCOLLBMS = tcollbm.$(SrcSuf) -TCOLLBM = tcollbm$(ExeSuf) - -VVECTORO = vvector.$(ObjSuf) -VVECTORS = vvector.$(SrcSuf) -VVECTOR = vvector$(ExeSuf) - -VMATRIXO = vmatrix.$(ObjSuf) -VMATRIXS = vmatrix.$(SrcSuf) -VMATRIX = vmatrix$(ExeSuf) - -STRESSLO = stressLinear.$(ObjSuf) -STRESSLS = stressLinear.$(SrcSuf) -STRESSL = stressLinear$(ExeSuf) - -STRESSGO = stressGraphics.$(ObjSuf) -STRESSGS = stressGraphics.$(SrcSuf) -STRESSG = stressGraphics$(ExeSuf) - -STRESSGUIO = stressGUI.$(ObjSuf) -STRESSGUIS = stressGUI.$(SrcSuf) -STRESSGUI = stressGUI$(ExeSuf) -STRESSGUILIBS = $(ROOTSYS)\lib\libASImage.lib $(ROOTSYS)\lib\libASImageGui.lib \ - $(ROOTSYS)\lib\libRecorder.lib $(ROOTSYS)\lib\libGuiHtml.lib - -!if exist("$(ROOTSYS)\lib\libGenVector.lib") -STRESSVECO = stressVector.$(ObjSuf) -STRESSVECS = stressVector.$(SrcSuf) -STRESSVEC = stressVector$(ExeSuf) - -STRESSMATHO = stressMathCore.$(ObjSuf) -STRESSMATHS = stressMathCore.$(SrcSuf) -STRESSMATHLIBS = $(ROOTSYS)\lib\libGenVector.lib - -TRACKMATHSRC = TrackMathCoreDict.$(SrcSuf) -TRACKMATHOBJ = TrackMathCoreDict.$(ObjSuf) -TRACKMATHLIB = libTrackMathCoreDict.$(DllSuf) - -STRESSMATH = stressMathCore$(ExeSuf) -!endif - -!if exist("$(ROOTSYS)\lib\libMathMore.lib") -STRESSMATHMOREO = stressMathMore.$(ObjSuf) -STRESSMATHMORES = stressMathMore.$(SrcSuf) -STRESSMATHMORELIBS = $(ROOTSYS)\lib\libMathMore.lib -STRESSMATHMORE = stressMathMore$(ExeSuf) -!endif - -!if exist("$(ROOTSYS)\lib\libTMVA.lib") -STRESSTMVAO = stressTMVA.$(ObjSuf) -STRESSTMVAS = stressTMVA.$(SrcSuf) -STRESSTMVALIBS = $(ROOTSYS)\lib\libTMVA.lib -STRESSTMVA = stressTMVA$(ExeSuf) -!endif - -VLAZYO = vlazy.$(ObjSuf) -VLAZYS = vlazy.$(SrcSuf) -VLAZY = vlazy$(ExeSuf) - -GUITESTO = guitest.$(ObjSuf) -GUITESTS = guitest.$(SrcSuf) -GUITEST = guitest$(ExeSuf) - -GUIVIEWERO = guiviewer.$(ObjSuf) guiviewerDict.$(ObjSuf) -GUIVIEWERS = guiviewer.$(SrcSuf) guiviewerDict.$(SrcSuf) -GUIVIEWER = guiviewer$(ExeSuf) - -HELLOO = Hello.$(ObjSuf) HelloDict.$(ObjSuf) -HELLOS = Hello.$(SrcSuf) HelloDict.$(SrcSuf) -HELLOSO = Hello.$(DllSuf) - -ACLOCKO = Aclock.$(ObjSuf) AclockDict.$(ObjSuf) -ACLOCKS = Aclock.$(SrcSuf) AclockDict.$(SrcSuf) -ACLOCKSO = Aclock.$(DllSuf) - -TETRISO = Tetris.$(ObjSuf) TetrisDict.$(ObjSuf) -TETRISS = Tetris.$(SrcSuf) TetrisDict.$(SrcSuf) -TETRISSO = Tetris.$(DllSuf) - -STRESSO = stress.$(ObjSuf) -STRESSS = stress.$(SrcSuf) -STRESS = stress$(ExeSuf) - -STRESSGEOMETRYO = stressGeometry.$(ObjSuf) -STRESSGEOMETRYS = stressGeometry.$(SrcSuf) -STRESSGEOMETRY = stressGeometry$(ExeSuf) - -STRESSSHAPESO = stressShapes.$(ObjSuf) -STRESSSHAPESS = stressShapes.$(SrcSuf) -STRESSSHAPES = stressShapes$(ExeSuf) - -!if exist("$(ROOTSYS)\lib\libRooFit.lib") -STRESSROOFITO = stressRooFit.$(ObjSuf) -STRESSROOFITS = stressRooFit.$(SrcSuf) -STRESSROOFIT = stressRooFit$(ExeSuf) -!endif - -!if exist("$(ROOTSYS)\lib\libRooStats.lib") -STRESSROOSTATSO = stressRooStats.$(ObjSuf) -STRESSROOSTATSS = stressRooStats.$(SrcSuf) -STRESSROOSTATS = stressRooStats$(ExeSuf) -!endif - -!if exist("$(ROOTSYS)\lib\libHistFactory.lib") -STRESSHISTFACTORYO = stressHistFactory.$(ObjSuf) -STRESSHISTFACTORYS = stressHistFactory.$(SrcSuf) -STRESSHISTFACTORY = stressHistFactory$(ExeSuf) -!endif - -STRESSFITO = stressFit.$(ObjSuf) -STRESSFITS = stressFit.$(SrcSuf) -STRESSFIT = stressFit$(ExeSuf) - -!if exist("$(ROOTSYS)\lib\libUnuran.lib") -!if exist("$(ROOTSYS)\lib\libminuit2.lib") -STRESSHISTOFITO = stressHistoFit.$(ObjSuf) -STRESSHISTOFITS = stressHistoFit.$(SrcSuf) -STRESSHISTOFIT = stressHistoFit$(ExeSuf) -!endif -!endif - -STRESSENTRYLISTO = stressEntryList.$(ObjSuf) -STRESSENTRYLISTS = stressEntryList.$(SrcSuf) -STRESSENTRYLIST = stressEntryList$(ExeSuf) - -STRESSHEPIXO = stressHepix.$(ObjSuf) -STRESSHEPIXS = stressHepix.$(SrcSuf) -STRESSHEPIX = stressHepix$(ExeSuf) - -STRESSSPO = stressSpectrum.$(ObjSuf) -STRESSSPS = stressSpectrum.$(SrcSuf) -STRESSSP = stressSpectrum$(ExeSuf) - -STRESSPROOFO = stressProof.$(ObjSuf) -STRESSPROOFS = stressProof.$(SrcSuf) -STRESSPROOF = stressProof$(ExeSuf) - -TBENCHO = TBench.$(ObjSuf) TBenchDict.$(ObjSuf) -TBENCHS = TBench.$(SrcSuf) TBenchDict.$(SrcSuf) -TBENCHSO = TBench.$(DllSuf) - -BENCHO = bench.$(ObjSuf) -BENCHS = bench.$(SrcSuf) -BENCH = bench$(ExeSuf) - -TESTBITSO = testbits.$(ObjSuf) -TESTBITSS = testbits.$(SrcSuf) -TESTBITS = testbits$(ExeSuf) - -QPRANDOMO = QpRandomDriver.$(ObjSuf) -QPRANDOMS = QpRandomDriver.$(SrcSuf) -QPRANDOM = QpRandomDriver$(ExeSuf) - -THREADSO = threads.$(ObjSuf) -THREADSS = threads.$(SrcSuf) -THREADS = threads$(ExeSuf) - -STRESSINTERPO = stressInterpreter.$(ObjSuf) -STRESSINTERPS = stressInterpreter.$(SrcSuf) -STRESSINTERP = stressInterpreter$(ExeSuf) - -STRESSITERO = stressIterators.$(ObjSuf) -STRESSITERS = stressIterators.$(SrcSuf) -STRESSITER = stressIterators$(ExeSuf) - -STRESSHISTO = stressHistogram.$(ObjSuf) -STRESSHISTS = stressHistogram.$(SrcSuf) -STRESSHIST = stressHistogram$(ExeSuf) - - -OBJS = $(EVENTO) $(MAINEVENTO) $(EVENTMTO) $(HWORLDO) $(HSIMPLEO) $(MINEXAMO) \ - $(TSTRINGO) $(TCOLLEXO) $(VVECTORO) $(VMATRIXO) $(VLAZYO) \ - $(HELLOO) $(ACLOCKO) $(STRESSO) $(TBENCHO) $(BENCHO) \ - $(STRESSSHAPESO) $(TCOLLBMO) $(STRESSGEOMETRYO) $(STRESSLO) \ - $(STRESSGO) $(STRESSSPO) $(TESTBITSO) \ - $(CTORTUREO) $(QPRANDOMO) $(THREADSO) $(STRESSVECO) \ - $(STRESSMATHO) $(STRESSFITO) $(STRESSHISTOFITO) $(STRESSHEPIXO) \ - $(STRESSENTRYLISTO) $(STRESSROOFITO) $(STRESSROOSTATSO) \ - $(STRESSHISTFACTORYO) $(STRESSPROOFO) \ - $(STRESSMATHMOREO) $(STRESSTMVAO) $(STRESSINTERPO) $(STRESSITERO) \ - $(STRESSHISTO) $(STRESSGUIO) $(GUITESTO) $(GUIVIEWERO) $(TETRISO) \ - -PROGRAMS = $(EVENT) $(EVENTMTSO) $(HWORLD) $(HSIMPLE) $(MINEXAM) $(TSTRING) \ - $(TCOLLEX) $(TCOLLBM) $(VVECTOR) $(VMATRIX) $(VLAZY) \ - $(HELLOSO) $(ACLOCKSO) $(STRESS) $(TBENCHSO) $(BENCH) \ - $(STRESSSHAPES) $(STRESSGEOMETRY) $(STRESSL) $(STRESSG) \ - $(TESTBITS) $(CTORTURE) $(QPRANDOM) $(THREADS) $(STRESSSP) \ - $(STRESSVEC) $(STRESSFIT) $(STRESSHISTOFIT) $(STRESSHEPIX) \ - $(STRESSENTRYLIST) $(STRESSROOFIT) $(STRESSROOSTATS) \ - $(STRESSHISTFACTORY) $(STRESSPROOF) $(STRESSMATH) \ - $(STRESSMATHMORE) $(STRESSTMVA) $(STRESSINTERP) $(STRESSITER) \ - $(STRESSHIST) $(STRESSGUI) $(GUITEST) $(GUIVIEWER) $(TETRISSO) \ - - -all: $(PROGRAMS) - -$(EVENTSO): $(EVENTO) - BINDEXPLIB $* $(EVENTO) > $*.def - lib -nologo -MACHINE:$(MACHINE) $(EVENTO) -def:$*.def $(OutPutOpt)$(EVENTLIB) - $(LD) $(SOFLAGS) $(LDFLAGS) $(EVENTO) $*.exp $(LIBS) $(OutPutOpt)$(EVENTSO) - @echo "$(EVENTSO) done" - -$(EVENTMTSO): $(EVENTMTO) - BINDEXPLIB $* $(EVENTMTO) > $*.def - lib -nologo -MACHINE:$(MACHINE) $(EVENTMTO) -def:$*.def $(OutPutOpt)$(EVENTMTLIB) - $(LD) $(SOFLAGS) $(LDFLAGS) $(EVENTMTO) $*.exp $(LIBS) $(OutPutOpt)$(EVENTMTSO) - @echo "$(EVENTMTSO) done" - -$(EVENT): $(EVENTSO) $(MAINEVENTO) - $(LD) $(LDFLAGS) $(MAINEVENTO) $(EVENTLIB) $(LIBS) $(OutPutOpt)$(EVENT) - @echo "$(EVENT) done" - -$(HWORLD): $(HWORLDO) - $(LD) $(LDFLAGS) $(HWORLDO) $(GLIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(CTORTURE): $(CTORTUREO) - $(LD) $(LDFLAGS) $(CTORTUREO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(HSIMPLE): $(HSIMPLEO) - $(LD) $(LDFLAGS) $(HSIMPLEO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(MINEXAM): $(MINEXAMO) - $(LD) $(LDFLAGS) $(MINEXAMO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(TSTRING): $(TSTRINGO) - $(LD) $(LDFLAGS) $(TSTRINGO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(TCOLLEX): $(TCOLLEXO) - $(LD) $(LDFLAGS) $(TCOLLEXO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(TCOLLBM): $(TCOLLBMO) - $(LD) $(LDFLAGS) $(TCOLLBMO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(VVECTOR): $(VVECTORO) - $(LD) $(LDFLAGS) $(VVECTORO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(VMATRIX): $(VMATRIXO) - $(LD) $(LDFLAGS) $(VMATRIXO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(VLAZY): $(VLAZYO) - $(LD) $(LDFLAGS) $(VLAZYO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSL): $(STRESSLO) - $(LD) $(LDFLAGS) $(STRESSLO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSG): $(STRESSGO) - $(LD) $(LDFLAGS) $(STRESSGO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSGUI): $(STRESSGUIO) - $(LD) $(LDFLAGS) $(STRESSGUIO) $(GLIBS) $(STRESSGUILIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSSP): $(STRESSSPO) - $(LD) $(LDFLAGS) $(STRESSSPO) $(LIBS) $(ROOTSYS)\lib\libSpectrum.lib $(OutPutOpt)$@ - @echo "$@ done" - -!if exist("$(ROOTSYS)\lib\libGenVector.lib") -$(STRESSVEC): $(STRESSVECO) - $(LD) $(LDFLAGS) $(STRESSVECO) $(LIBS) $(ROOTSYS)\lib\libGenVector.lib $(OutPutOpt)$@ - @echo "$@ done" - -$(TRACKMATHLIB): $(TRACKMATHOBJ) - $(LD) $(SOFLAGS) $(LDFLAGS) $(TRACKMATHOBJ) $(LIBS) $(ROOTSYS)\lib\libGenVector.lib $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSMATH): $(STRESSMATHO) $(TRACKMATHLIB) - $(LD) $(LDFLAGS) $(STRESSMATHO) $(LIBS) $(ROOTSYS)\lib\libGenVector.lib $(OutPutOpt)$@ - @echo "$@ done" -!endif - -!if exist("$(ROOTSYS)\lib\libMathMore.lib") -$(STRESSMATHMORE): $(STRESSMATHMOREO) - $(LD) $(LDFLAGS) $(STRESSMATHMOREO) $(LIBS) $(ROOTSYS)\lib\libMathMore.lib $(OutPutOpt)$@ - @echo "$@ done" -!endif - -!if exist("$(ROOTSYS)\lib\libTMVA.lib") -$(STRESSTMVA): $(STRESSTMVAO) - $(LD) $(LDFLAGS) $(STRESSTMVAO) $(LIBS) $(ROOTSYS)\lib\libTMVA.lib $(OutPutOpt)$@ - @echo "$@ done" -!endif - -$(TESTBITS): $(TESTBITSO) - $(LD) $(LDFLAGS) $(TESTBITSO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(THREADS): $(THREADSO) - $(LD) $(LDFLAGS) $(THREADSO) $(LIBS) $(ROOTSYS)\lib\libThread.lib $(OutPutOpt)$@ - @echo "$@ done" - -$(QPRANDOM): $(QPRANDOMO) - $(LD) $(LDFLAGS) $(QPRANDOMO) $(LIBS) $(ROOTSYS)\lib\libQuadp.lib $(OutPutOpt)$@ - @echo "$@ done" - -$(GUITEST): $(GUITESTO) - $(LD) $(LDFLAGS) $(GUITESTO) $(GLIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(GUIVIEWER): $(GUIVIEWERO) - $(LD) $(LDFLAGS) $(GUIVIEWERO) $(GLIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESS): $(STRESSO) $(EVENT) - $(LD) $(LDFLAGS) $(STRESSO) $(EVENTLIB) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSGEOMETRY): $(STRESSGEOMETRYO) - $(LD) $(LDFLAGS) $(STRESSGEOMETRYO) $(LIBS) $(ROOTSYS)\lib\libGeom.lib $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSSHAPES): $(STRESSSHAPESO) - $(LD) $(LDFLAGS) $(STRESSSHAPESO) $(LIBS) $(ROOTSYS)\lib\libGeom.lib $(OutPutOpt)$@ - @echo "$@ done" - -!if exist("$(ROOTSYS)\lib\libRooFit.lib") -$(STRESSROOFIT): $(STRESSROOFITO) - $(LD) $(LDFLAGS) $(STRESSROOFITO) $(LIBS) $(ROOTSYS)\lib\libRooFit.lib $(ROOTSYS)\lib\libRooFitCore.lib $(ROOTSYS)\lib\libHtml.lib $(ROOTSYS)\lib\libThread.lib $(ROOTSYS)\lib\libMinuit.lib $(OutPutOpt)$@ - @echo "$@ done" -!endif - -!if exist("$(ROOTSYS)\lib\libRooStats.lib") -$(STRESSROOSTATS): $(STRESSROOSTATSO) - $(LD) $(LDFLAGS) $(STRESSROOSTATSO) $(LIBS) $(ROOTSYS)\lib\libRooStats.lib $(ROOTSYS)\lib\libRooFit.lib $(ROOTSYS)\lib\libRooFitCore.lib $(OutPutOpt)$@ - @echo "$@ done" -!endif - -!if exist("$(ROOTSYS)\lib\libHistFactory.lib") -$(STRESSHISTFACTORY): $(STRESSHISTFACTORYO) - $(LD) $(LDFLAGS) $(STRESSHISTFACTORYO) $(LIBS) $(ROOTSYS)\lib\libHistFactory.lib $(ROOTSYS)\lib\libRooStats.lib $(ROOTSYS)\lib\libRooFit.lib $(ROOTSYS)\lib\libRooFitCore.lib $(ROOTSYS)\lib\libHtml.lib $(ROOTSYS)\lib\libThread.lib $(ROOTSYS)\lib\libMinuit.lib $(ROOTSYS)\lib\libFoam.lib $(ROOTSYS)\lib\libProof.lib $(EXTRAROOFITLIBS) $(OutPutOpt)$@ - @echo "$@ done" -!endif - -$(STRESSFIT): $(STRESSFITO) - $(LD) $(LDFLAGS) $(STRESSFITO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -!if exist("$(ROOTSYS)\lib\libUnuran.lib") -!if exist("$(ROOTSYS)\lib\libminuit2.lib") -$(STRESSHISTOFIT): $(STRESSHISTOFITO) - $(LD) $(LDFLAGS) $(STRESSHISTOFITO) $(LIBS) $(ROOTSYS)\lib\libUnuran.lib $(OutPutOpt)$@ - @echo "$@ done" -!endif -!endif - -$(STRESSENTRYLIST): $(STRESSENTRYLISTO) - $(LD) $(LDFLAGS) $(STRESSENTRYLISTO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSHEPIX): $(STRESSHEPIXO) $(STRESSGEOMETRY) $(STRESSFIT) $(STRESSL) \ - $(STRESSSP) $(STRESS) - $(LD) $(LDFLAGS) $(STRESSHEPIXO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSPROOF): $(STRESSPROOFO) - $(LD) $(LDFLAGS) $(STRESSPROOFO) $(LIBS) $(ROOTSYS)\lib\libProof.lib $(ROOTSYS)\lib\libThread.lib $(OutPutOpt)$@ - @echo "$@ done" - -$(BENCH): $(BENCHO) $(TBENCHSO) - $(LD) $(LDFLAGS) $(BENCHO) $(TBENCHO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -Hello: $(HELLOSO) -$(HELLOSO): $(HELLOO) - BINDEXPLIB $* $(HELLOO) > $*.def - lib -nologo -MACHINE:$(MACHINE) $(HELLOO) -def:$*.def $(OutPutOpt)Hello.lib - $(LD) $(SOFLAGS) $(LDFLAGS) $(HELLOO) $*.exp $(GLIBS) $(OutPutOpt)$@ - -Aclock: $(ACLOCKSO) -$(ACLOCKSO): $(ACLOCKO) - BINDEXPLIB $* $(ACLOCKO) > $*.def - lib -nologo -MACHINE:$(MACHINE) $(ACLOCKO) -def:$*.def $(OutPutOpt)Aclock.lib - $(LD) $(SOFLAGS) $(LDFLAGS) $(ACLOCKO) $*.exp $(GLIBS) $(OutPutOpt)$@ - -Tetris: $(TETRISSO) -$(TETRISSO): $(TETRISO) - BINDEXPLIB $* $(TETRISO) > $*.def - lib -nologo -MACHINE:$(MACHINE) $(TETRISO) -def:$*.def $(OutPutOpt)Tetris.lib - $(LD) $(SOFLAGS) $(LDFLAGS) $(TETRISO) $*.exp $(GLIBS) $(OutPutOpt)$@ - -$(TBENCHSO): $(TBENCHO) - $(LD) $(SOFLAGS) $(LDFLAGS) $(TBENCHO) $(LIBS) $(OutPutOpt)$@ - -$(STRESSINTERP): $(STRESSINTERPO) - $(LD) $(LDFLAGS) $(STRESSINTERPO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSITER): $(STRESSITERO) - $(LD) $(LDFLAGS) $(STRESSITERO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -$(STRESSHIST): $(STRESSHISTO) - $(LD) $(LDFLAGS) $(STRESSHISTO) $(LIBS) $(OutPutOpt)$@ - @echo "$@ done" - -clean: - @del *.obj *Dict.* *.def *.exp *.d *.log .def *.pdb *.ilk *.manifest >nul 2>&1 - -distclean: clean - @del /q *.exe *.root *.ps *.lib *.dll >nul 2>&1 - -### -Event.$(ObjSuf): Event.h -MainEvent.$(ObjSuf): Event.h - -EventDict.$(SrcSuf): Event.h EventLinkDef.h - @echo "Generating dictionary $@..." - @rootcint -f $@ -c Event.h EventLinkDef.h - -EventMTDict.$(SrcSuf): EventMT.h EventLinkDef.h - @echo "Generating dictionary $@..." - @rootcint -f $@ -c EventMT.h EventLinkDef.h - -Hello.$(ObjSuf): Hello.h -HelloDict.$(SrcSuf): Hello.h - @echo "Generating dictionary $@..." - @rootcint -f $@ -c Hello.h - -Aclock.$(ObjSuf): Aclock.h -AclockDict.$(SrcSuf): Aclock.h - @echo "Generating dictionary $@..." - @rootcint -f $@ -c Aclock.h - -Tetris.$(ObjSuf): Tetris.h -TetrisDict.$(SrcSuf): Tetris.h - @echo "Generating dictionary $@..." - @rootcint -f $@ -c Tetris.h - -TBench.$(ObjSuf): TBench.h -TBenchDict.$(SrcSuf): TBench.h benchLinkDef.h - @echo "Generating dictionary $@..." - @rootcint -f $@ -c TBench.h benchLinkDef.h - -guiviewer.$(ObjSuf): guiviewer.h -guiviewerDict.$(SrcSuf): guiviewer.h guiviewerLinkDef.h - @echo "Generating dictionary $@..." - @rootcint -f $@ -c guiviewer.h guiviewerLinkDef.h - -stressMathCore.$(ObjSuf): TrackMathCore.h -TrackMathCoreDict.$(SrcSuf): TrackMathCore.h TrackMathCoreLinkDef.h - @echo "Generating dictionary $@ using rootcint ..." - @rootcint -f $@ -c TrackMathCore.h TrackMathCoreLinkDef.h - -stressProof.$(ObjSuf): stressProof.$(SrcSuf) - $(CXX) $(CXXFLAGS) $(CXXOPT) -I$(TUTDIR) -c $** - -.$(SrcSuf).$(ObjSuf): - $(CXX) $(CXXFLAGS) $(CXXOPT) -c $< - diff --git a/test/ProofBench/Draw_PerfProfiles.C b/test/ProofBench/Draw_PerfProfiles.C deleted file mode 100644 index 4926daaf5744f..0000000000000 --- a/test/ProofBench/Draw_PerfProfiles.C +++ /dev/null @@ -1,220 +0,0 @@ -void Build_Timing_Tree(TFile * f, const Char_t *pattern, Int_t& max_slaves); - -void Draw_PerfProfiles(const Char_t* filename) { - // Plots total processing time as a function of number of slaves - // using each of the 3 selectors. - - // filename: name of file that Run_Node_Tests.C wrote its output into - - gROOT->SetStyle("Plain"); - gStyle->SetOptStat(0); - gStyle->SetNdivisions(505); - gStyle->SetTitleFontSize(0.07); - - if(!TString(gSystem->GetLibraries()).Contains("Proof")) - gSystem->Load("libProof.so"); - - TFile f(filename); - if (f.IsZombie()) { - cout << "file " << filename << " cannot be opened" << endl; - return; - } - - TString perfstats_name = "PROOF_PerfStats"; - - Int_t ns_holder; - Int_t run_holder; - Float_t time_holder; - - Int_t procmax_slaves = 0; - TTree* tt_proc = Build_Timing_Tree(&f, perfstats_name+"_Proc_", - procmax_slaves); - tt_proc->SetMarkerStyle(4); - //set branch addresses - tt_proc->GetBranch("perfproctime")->GetLeaf("nslaves")->SetAddress(&ns_holder); - tt_proc->GetBranch("perfproctime")->GetLeaf("run")->SetAddress(&run_holder); - tt_proc->GetBranch("perfproctime")->GetLeaf("time")->SetAddress(&time_holder); - - Int_t procoptmax_slaves = 0; - TTree* tt_procopt = Build_Timing_Tree(&f, perfstats_name+"_ProcOpt_", - procoptmax_slaves); - tt_procopt->SetMarkerStyle(5); - //set branch addresses - tt_procopt->GetBranch("perfproctime")->GetLeaf("nslaves")->SetAddress(&ns_holder); - tt_procopt->GetBranch("perfproctime")->GetLeaf("run")->SetAddress(&run_holder); - tt_procopt->GetBranch("perfproctime")->GetLeaf("time")->SetAddress(&time_holder); - - Int_t noprocmax_slaves = 0; - TTree* tt_noproc = Build_Timing_Tree(&f, perfstats_name+"_NoProc_", - noprocmax_slaves); - tt_noproc->SetMarkerStyle(6); - //set branch addresses - tt_noproc->GetBranch("perfproctime")->GetLeaf("nslaves")->SetAddress(&ns_holder); - tt_noproc->GetBranch("perfproctime")->GetLeaf("run")->SetAddress(&run_holder); - tt_noproc->GetBranch("perfproctime")->GetLeaf("time")->SetAddress(&time_holder); - - f.Close(); - - Int_t nslaves = procmax_slaves>procoptmax_slaves?procmax_slaves:procoptmax_slaves; - if (nslavesSetMarkerStyle(26); - tt_proc->Draw("time:nslaves>>procprof"); - procprof->GetXaxis()->SetTitle("Number of Slaves"); - procprof->GetYaxis()->SetTitle("Processing Time [s]"); - - TProfile* procoptprof = new TProfile("procoptprof", "Total Processing Time", - nslaves+1, 0, nslaves+1); - procoptprof->SetMarkerStyle(25); - tt_procopt->Draw("time:nslaves>>procoptprof","","same"); - - TProfile* noprocprof = new TProfile("noprocprof", "Total Processing Time", - nslaves+1, 0, nslaves+1); - noprocprof->SetMarkerStyle(24); - tt_noproc->Draw("time:nslaves>>noprocprof","","same"); - - Float_t lm = gPad->GetLeftMargin(); - Float_t rm = gPad->GetRightMargin(); - Float_t tm = gPad->GetTopMargin(); - Float_t bm = gPad->GetBottomMargin(); - - Float_t legxoffset = 0.1; - Float_t legwidth = 0.2; - Float_t legyoffset = 0.02; - Float_t legheight = 0.15; - - TLegend* leg = new TLegend(lm+legxoffset*(1.0-lm-rm), - 1.0-tm-(legyoffset+legheight)*(1.0-tm-bm), - lm+(legxoffset+legwidth)*(1.0-lm-rm), - 1.0-tm-legyoffset*(1.0-tm-bm)); - leg->SetBorderSize(1); - leg->SetFillColor(0); - leg->AddEntry(procprof,"Full Event","p"); - leg->AddEntry(procoptprof,"Partial Event","p"); - leg->AddEntry(noprocprof,"No Data","p"); - leg->Draw(); - - gPad->Update(); - TPaveText* titlepave = dynamic_cast(gPad->GetListOfPrimitives()->FindObject("title")); - if (titlepave) { - Double_t x1ndc = titlepave->GetX1NDC(); - Double_t x2ndc = titlepave->GetX2NDC(); - titlepave->SetX1NDC((1.0-x2ndc+x1ndc)/2.); - titlepave->SetX2NDC((1.0+x2ndc-x1ndc)/2.); - titlepave->SetBorderSize(0); - gPad->Update(); - } - gPad->Modified(); -} - -TTree* Build_Timing_Tree(TFile * f, const Char_t *pattern, Int_t& max_slaves) { - - TTree* timing_tree = new TTree("Timing Tree", "Timing Tree"); - timing_tree->SetDirectory(0); - Int_t ns_holder; - Int_t run_holder; - Float_t time_holder; - TBranch* br = timing_tree->Branch("perfproctime", &ns_holder, - "nslaves/I:run/I:time/F"); - br->GetLeaf("nslaves")->SetAddress(&ns_holder); - br->GetLeaf("run")->SetAddress(&run_holder); - br->GetLeaf("time")->SetAddress(&time_holder); - - // extract timing info - max_slaves = 0; - TIter NextKey(f->GetListOfKeys()); - TKey* key = 0; - while (key = dynamic_cast(NextKey())) { - if(!TString(key->GetName()).Contains(TRegexp(pattern))) - continue; - - TObject* obj = key->ReadObj(); - TTree* t = dynamic_cast(obj); - if (!t) { - delete obj; - continue; - } - - //parse name to get number of slaves and run - Int_t Index = 0; - const Char_t *name = t->GetName(); - while (Index='0' && name[Index]<='9') - break; - Index++; - } - - if (Index == strlen(name)) { - delete t; - continue; - } else { - // this should be the number of slaves - ns_holder = atoi(name+Index); - } - - // get past number of slaves - while (Index'9') - break; - Index++; - } - - if (Index == strlen(name)) { - delete t; - continue; - } - - while (Index='0' && name[Index]<='9') - break; - Index++; - } - - if (Index == strlen(name)) { - delete t; - continue; - } else { - // this should be the run number - run_holder = atoi(name+Index); - } - - if(!t->FindBranch("PerfEvents")) { - delete t; - continue; - } - - // extract timing information - TPerfEvent pe; - TPerfEvent* pep = &pe; - t->SetBranchAddress("PerfEvents",&pep); - Long64_t entries = t->GetEntries(); - Double_t start, end; - Bool_t started=kFALSE; - for (Long64_t k=0; kGetEntry(k); - if (!started) { - if (pe.fType==TVirtualPerfStats::kPacket) { - start = pe.fTimeStamp.GetSec() - + 1e-9*pe.fTimeStamp.GetNanoSec() - - pe.fProcTime; - started=kTRUE; - } - } else { - if (pe.fType==TVirtualPerfStats::kPacket) { - end = pe.fTimeStamp.GetSec() - + 1e-9*pe.fTimeStamp.GetNanoSec(); - } - } - } - - time_holder = end-start; - timing_tree->Fill(); - if (max_slavesGetLibraries()).Contains("Proof")) - gSystem->Load("libProof.so"); - - TFile f(filename); - if (f.IsZombie()) { - cout << "Could not open file " << filename << endl; - return; - } - - TTree* perfstats = dynamic_cast(f.Get(perfstatsname)); - if (perfstats) { - Draw_Slave_Access(perfstats); - } else { - cout << "No Tree named " << perfstatsname - << " found in file " << filename << endl; - } - - delete perfstats; - f.Close(); -} - -void Draw_Slave_Access(TTree *perfstats) { - // Draws a graph of the number of slaves - // accessing files as a function of time - // from a tree made from TPerfStats - - gROOT->SetStyle("Plain"); - gStyle->SetNdivisions(505, "X"); - gStyle->SetNdivisions(505, "Y"); - gStyle->SetTitleFontSize(0.07); - - if(!TString(gSystem->GetLibraries()).Contains("Proof")) - gSystem->Load("libProof.so"); - - if (!perfstats) { - cout << "Input tree invalid" << endl; - return; - } - - //make sure PerfEvents branch exists - if (!perfstats->FindBranch("PerfEvents")) { - cout << "Input tree does not have a PerfEvents branch" << endl; - return; - } - - Int_t nentries = perfstats->GetEntries(); - TPerfEvent pe; - TPerfEvent* pep = &pe; - perfstats->SetBranchAddress("PerfEvents",&pep); - - //make graph - TGraph* graph = new TGraph(1); - graph->SetName("all"); - graph->SetTitle("Global File Access"); - //Set first point to 0 - graph->SetPoint(0,0,0); - - for(Int_t entry=0;entryGetEntry(entry); - - //Check if it is a file event - if(pe.fType==TVirtualPerfStats::kFile){ - - Double_t time = (pe.fTimeStamp.GetSec())+ - (pe.fTimeStamp.GetNanoSec())*1e-9; - Int_t npoints=graph->GetN(); - Double_t y = (graph->GetY())[npoints-1]; - graph->SetPoint(npoints,time,y); - if(pe.fIsStart==kTRUE) y++; - else y--; - graph->SetPoint(npoints+1,time,y); - - } - } - - //reset branch address to 0 since address will be invalid after leaving macro - perfstats->SetBranchAddress("PerfEvents", 0); - - // Draw Canvas - TCanvas* canvas = new TCanvas("slave_access", - "Number of Slaves Accessing Nodes vs Time"); - canvas->cd(); - graph->GetXaxis()->SetTitle("Time [s]"); - graph->GetYaxis()->SetTitle("Number of Slaves Accessing Files"); - graph->Draw("AL"); - gPad->Update(); - - // center title - TPaveText* titlepave = - dynamic_cast(gPad->GetListOfPrimitives()->FindObject("title")); - if (titlepave) { - Double_t x1ndc = titlepave->GetX1NDC(); - Double_t x2ndc = titlepave->GetX2NDC(); - titlepave->SetX1NDC((1.0-x2ndc+x1ndc)/2.); - titlepave->SetX2NDC((1.0+x2ndc-x1ndc)/2.); - titlepave->SetBorderSize(0); - gPad->Update(); - } - - gPad->Modified(); - -} diff --git a/test/ProofBench/Draw_Time_Hists.C b/test/ProofBench/Draw_Time_Hists.C deleted file mode 100644 index 615f00d95095e..0000000000000 --- a/test/ProofBench/Draw_Time_Hists.C +++ /dev/null @@ -1,107 +0,0 @@ -void Draw_Time_Hists(TTree* t); - -void Draw_Time_Hists(const Char_t* filename, const Char_t* perfstatsname) { - // Open the file called filename and get - // the perftstats tree (name perfstatsname) - // from the file - - if(!TString(gSystem->GetLibraries()).Contains("Proof")) - gSystem->Load("libProof.so"); - - TFile f(filename); - if (f.IsZombie()) { - cout << "Could not open file " << filename << endl; - return; - } - - TTree* perfstats = dynamic_cast(f.Get(perfstatsname)); - if (perfstats) { - Draw_Time_Hists(perfstats); - } else { - cout << "No Tree named " << perfstatsname - << " found in file " << filename << endl; - } - - delete perfstats; - f.Close(); -} - -void Draw_Time_Hists(TTree* t) { - // Draw processing time, CPU time, and latency per packet - // distributions from the input tree - - if (!t) { - cout << "Invalid input tree" << endl; - return; - } - - gROOT->SetStyle("Plain"); - gStyle->SetOptStat(0); - gStyle->SetNdivisions(505); - gStyle->SetTitleFontSize(0.1); - - if(!TString(gSystem->GetLibraries()).Contains("Proof")) - gSystem->Load("libProof.so"); - - TCanvas* canvas = new TCanvas("ProfHists","Profile Histograms",800,600); - canvas->Divide(3,1); - - canvas->cd(1); - gPad->SetLogy(); - t->Draw("fProcTime"); - TH1* h = dynamic_cast(gROOT->FindObject("htemp")); - h->SetTitle("Processing Time per Packet"); - h->GetXaxis()->SetTitle("Processing Time [s]"); - - gPad->Update(); - TPaveText* titlepave = 0; - titlepave = dynamic_cast(gPad->GetListOfPrimitives()->FindObject("title")); - if (titlepave) { - Double_t x1ndc = titlepave->GetX1NDC(); - Double_t x2ndc = titlepave->GetX2NDC(); - titlepave->SetX1NDC((1.0-x2ndc+x1ndc)/2.); - titlepave->SetX2NDC((1.0+x2ndc-x1ndc)/2.); - titlepave->SetBorderSize(0); - gPad->Update(); - } - gPad->Modified(); - - canvas->cd(2); - gPad->SetLogy(); - t->Draw("fCpuTime"); - h = dynamic_cast(gROOT->FindObject("htemp")); - h->SetTitle("CPU Time per Packet"); - h->GetXaxis()->SetTitle("CPU Time [s]"); - - gPad->Update(); - titlepave = dynamic_cast(gPad->GetListOfPrimitives()->FindObject("title")); - if (titlepave) { - Double_t x1ndc = titlepave->GetX1NDC(); - Double_t x2ndc = titlepave->GetX2NDC(); - titlepave->SetX1NDC((1.0-x2ndc+x1ndc)/2.); - titlepave->SetX2NDC((1.0+x2ndc-x1ndc)/2.); - titlepave->SetBorderSize(0); - gPad->Update(); - } - gPad->Modified(); - - canvas->cd(3); - gPad->SetLogy(); - t->Draw("fLatency"); - h = dynamic_cast(gROOT->FindObject("htemp")); - h->SetTitle("Request Packet Latency"); - h->GetXaxis()->SetTitle("Latency [s]"); - - gPad->Update(); - titlepave = dynamic_cast(gPad->GetListOfPrimitives()->FindObject("title")); - if (titlepave) { - Double_t x1ndc = titlepave->GetX1NDC(); - Double_t x2ndc = titlepave->GetX2NDC(); - titlepave->SetX1NDC((1.0-x2ndc+x1ndc)/2.); - titlepave->SetX2NDC((1.0+x2ndc-x1ndc)/2.); - titlepave->SetBorderSize(0); - gPad->Update(); - } - gPad->Modified(); - -} diff --git a/test/ProofBench/EventTree_NoProc.C b/test/ProofBench/EventTree_NoProc.C deleted file mode 100644 index d3dd08b14b2d7..0000000000000 --- a/test/ProofBench/EventTree_NoProc.C +++ /dev/null @@ -1,112 +0,0 @@ -#define EventTree_NoProc_cxx -// The class definition in EventTree_NoProc.h has been generated automatically -// by the ROOT utility TTree::MakeSelector(). This class is derived -// from the ROOT class TSelector. For more information on the TSelector -// framework see $ROOTSYS/README/README.SELECTOR or the ROOT User Manual. - -// The following methods are defined in this file: -// Begin(): called everytime a loop on the tree starts, -// a convenient place to create your histograms. -// SlaveBegin(): called after Begin(), when on PROOF called only on the -// slave servers. -// Process(): called for each event, in this function you decide what -// to read and fill your histograms. -// SlaveTerminate: called at the end of the loop on the tree, when on PROOF -// called only on the slave servers. -// Terminate(): called at the end of the loop on the tree, -// a convenient place to draw/fit your histograms. -// -// To use this file, try the following session on your Tree T: -// -// Root > T->Process("EventTree_NoProc.C") -// Root > T->Process("EventTree_NoProc.C","some options") -// Root > T->Process("EventTree_NoProc.C+") -// - -#include "EventTree_NoProc.h" -#include -#include -#include "TCanvas.h" - - -void EventTree_NoProc::Begin(TTree *) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - -} - -void EventTree_NoProc::SlaveBegin(TTree *tree) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - Init(tree); - - TString option = GetOption(); - - fPtHist = new TH1F("pt_dist","p_{T} Distribution",100,0,5); - fPtHist->SetDirectory(0); - fPtHist->GetXaxis()->SetTitle("p_{T}"); - fPtHist->GetYaxis()->SetTitle("dN/p_{T}dp_{T}"); - - fOutput->Add(fPtHist); - - fNTracksHist = new TH1I("ntracks_dist","N_{Tracks} per Event Distribution",5,0,5); - fNTracksHist->SetDirectory(0); - fNTracksHist->GetXaxis()->SetTitle("N_{Tracks}"); - fNTracksHist->GetYaxis()->SetTitle("N_{Events}"); - - fOutput->Add(fNTracksHist); - -} - -Bool_t EventTree_NoProc::Process(Long64_t) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either TTree::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - - // WARNING when a selector is used with a TChain, you must use - // the pointer to the current TTree to call GetEntry(entry). - // The entry is always the local entry number in the current tree. - // Assuming that fChain is the pointer to the TChain being processed, - // use fChain->GetTree()->GetEntry(entry). - - return kTRUE; -} - -void EventTree_NoProc::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - -} - -void EventTree_NoProc::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - TCanvas* canvas = new TCanvas("can","can",800,600); - canvas->SetBorderMode(0); - //canvas->SetLogy(); - TH1F* h = dynamic_cast(fOutput->FindObject("pt_dist")); - if (h) h->DrawCopy(); - else Warning("Terminate", "no pt dist found"); - -} diff --git a/test/ProofBench/EventTree_NoProc.h b/test/ProofBench/EventTree_NoProc.h deleted file mode 100644 index ba47305b2c9fc..0000000000000 --- a/test/ProofBench/EventTree_NoProc.h +++ /dev/null @@ -1,162 +0,0 @@ -////////////////////////////////////////////////////////// -// This class has been automatically generated on -// Fri May 7 07:24:11 2004 by ROOT version 4.00/04 -// from TTree EventTree/Event Tree -// found on file: event_tree_darkstar.root -////////////////////////////////////////////////////////// - -#ifndef EventTree_NoProc_h -#define EventTree_NoProc_h - -#include -#include -#include -#include -#include "Event.h" -#include "TH1.h" - -class EventTree_NoProc : public TSelector { -public : - TTree *fChain; //!pointer to the analyzed TTree or TChain - - // Declaration of leave types - Event *event; - Char_t fType[20]; - Char_t *fEventName; - Int_t fNtrack; - Int_t fNseg; - Int_t fNvertex; - UInt_t fFlag; - Double32_t fTemperature; - Int_t fMeasures[10]; - Double32_t fMatrix[4][4]; - Double32_t fClosestDistance[21]; //[fNvertex] - EventHeader fEvtHdr; - TClonesArray* fTracks; - TRefArray *fHighPt; - TRefArray *fMuons; - TRef fLastTrack; - TRef fWebHistogram; - TH1F *fH; - TBits fTriggerBits; - Bool_t fIsValid; - - //Output hist - TH1F* fPtHist; - TH1I* fNTracksHist; - - // List of branches - TBranch *b_event_fType; //! - TBranch *b_fEventName; //! - TBranch *b_event_fNtrack; //! - TBranch *b_event_fNseg; //! - TBranch *b_event_fNvertex; //! - TBranch *b_event_fFlag; //! - TBranch *b_event_fTemperature; //! - TBranch *b_event_fMeasures; //! - TBranch *b_event_fMatrix; //! - TBranch *b_fClosestDistance; //! - TBranch *b_event_fEvtHdr; //! - TBranch *b_fTracks; //! - TBranch *b_fHighPt; //! - TBranch *b_fMuons; //! - TBranch *b_event_fLastTrack; //! - TBranch *b_event_fWebHistogram; //! - TBranch *b_fH; //! - TBranch *b_event_fTriggerBits; //! - TBranch *b_event_fIsValid; //! - - EventTree_NoProc(TTree *) { } - EventTree_NoProc() { } - virtual ~EventTree_NoProc() { } - virtual Int_t Version() const {return 1;} - virtual void Begin(TTree *); - virtual void SlaveBegin(TTree *tree); - virtual void Init(TTree *tree); - virtual Bool_t Notify(); - virtual Bool_t Process(Long64_t); - virtual void SetOption(const char *option) { fOption = option; } - virtual void SetObject(TObject *obj) { fObject = obj; } - virtual void SetInputList(TList *input) {fInput = input;} - virtual TList *GetOutputList() const { return fOutput; } - virtual void SlaveTerminate(); - virtual void Terminate(); - - ClassDefOverride(EventTree_NoProc,0); -}; - -#endif - -#ifdef EventTree_NoProc_cxx -void EventTree_NoProc::Init(TTree *tree) -{ - // The Init() function is called when the selector needs to initialize - // a new tree or chain. Typically here the branch addresses of the tree - // will be set. It is normaly not necessary to make changes to the - // generated code, but the routine can be extended by the user if needed. - // Init() will be called many times when running with PROOF. - - // Set branch addresses - fEventName=0; - fTracks=0; - fHighPt=0; - fMuons=0; - if (tree == 0) return; - fChain = tree; - fChain->SetMakeClass(1); - - fChain->SetBranchAddress("fType[20]",fType); - fChain->SetBranchAddress("fEventName",fEventName); - fChain->SetBranchAddress("fNtrack",&fNtrack); - fChain->SetBranchAddress("fNseg",&fNseg); - fChain->SetBranchAddress("fNvertex",&fNvertex); - fChain->SetBranchAddress("fFlag",&fFlag); - fChain->SetBranchAddress("fTemperature",&fTemperature); - fChain->SetBranchAddress("fMeasures[10]",fMeasures); - fChain->SetBranchAddress("fMatrix[4][4]",fMatrix); - fChain->SetBranchAddress("fClosestDistance",fClosestDistance); - fChain->SetBranchAddress("fEvtHdr",&fEvtHdr); - fChain->SetBranchAddress("fTracks",&fTracks); - fChain->SetBranchAddress("fHighPt",&fHighPt); - fChain->SetBranchAddress("fMuons",&fMuons); - fChain->SetBranchAddress("fLastTrack",&fLastTrack); - fChain->SetBranchAddress("fWebHistogram",&fWebHistogram); - fChain->SetBranchAddress("fH",&fH); - fChain->SetBranchAddress("fTriggerBits",&fTriggerBits); - fChain->SetBranchAddress("fIsValid",&fIsValid); -} - -Bool_t EventTree_NoProc::Notify() -{ - // The Notify() function is called when a new file is opened. This - // can be either for a new TTree in a TChain or when when a new TTree - // is started when using PROOF. Typically here the branch pointers - // will be retrieved. It is normaly not necessary to make changes - // to the generated code, but the routine can be extended by the - // user if needed. - - // Get branch pointers - b_event_fType = fChain->GetBranch("fType[20]"); - b_fEventName = fChain->GetBranch("fEventName"); - b_event_fNtrack = fChain->GetBranch("fNtrack"); - b_event_fNseg = fChain->GetBranch("fNseg"); - b_event_fNvertex = fChain->GetBranch("fNvertex"); - b_event_fFlag = fChain->GetBranch("fFlag"); - b_event_fTemperature = fChain->GetBranch("fTemperature"); - b_event_fMeasures = fChain->GetBranch("fMeasures[10]"); - b_event_fMatrix = fChain->GetBranch("fMatrix[4][4]"); - b_fClosestDistance = fChain->GetBranch("fClosestDistance"); - b_event_fEvtHdr = fChain->GetBranch("fEvtHdr"); - b_fTracks = fChain->GetBranch("fTracks"); - b_fHighPt = fChain->GetBranch("fHighPt"); - b_fMuons = fChain->GetBranch("fMuons"); - b_event_fLastTrack = fChain->GetBranch("fLastTrack"); - b_event_fWebHistogram = fChain->GetBranch("fWebHistogram"); - b_fH = fChain->GetBranch("fH"); - b_event_fTriggerBits = fChain->GetBranch("fTriggerBits"); - b_event_fIsValid = fChain->GetBranch("fIsValid"); - - return kTRUE; -} - -#endif // #ifdef EventTree_NoProc_cxx diff --git a/test/ProofBench/EventTree_Proc.C b/test/ProofBench/EventTree_Proc.C deleted file mode 100644 index b141d10709298..0000000000000 --- a/test/ProofBench/EventTree_Proc.C +++ /dev/null @@ -1,122 +0,0 @@ -#define EventTree_Proc_cxx -// The class definition in EventTree_Proc.h has been generated automatically -// by the ROOT utility TTree::MakeSelector(). This class is derived -// from the ROOT class TSelector. For more information on the TSelector -// framework see $ROOTSYS/README/README.SELECTOR or the ROOT User Manual. - -// The following methods are defined in this file: -// Begin(): called everytime a loop on the tree starts, -// a convenient place to create your histograms. -// SlaveBegin(): called after Begin(), when on PROOF called only on the -// slave servers. -// Process(): called for each event, in this function you decide what -// to read and fill your histograms. -// SlaveTerminate: called at the end of the loop on the tree, when on PROOF -// called only on the slave servers. -// Terminate(): called at the end of the loop on the tree, -// a convenient place to draw/fit your histograms. -// -// To use this file, try the following session on your Tree T: -// -// Root > T->Process("EventTree_Proc.C") -// Root > T->Process("EventTree_Proc.C","some options") -// Root > T->Process("EventTree_Proc.C+") -// - -#include "EventTree_Proc.h" -#include -#include -#include "TCanvas.h" - - -void EventTree_Proc::Begin(TTree *) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - -} - -void EventTree_Proc::SlaveBegin(TTree *tree) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - Init(tree); - - TString option = GetOption(); - - fPtHist = new TH1F("pt_dist","p_{T} Distribution",100,0,5); - fPtHist->SetDirectory(0); - fPtHist->GetXaxis()->SetTitle("p_{T}"); - fPtHist->GetYaxis()->SetTitle("dN/p_{T}dp_{T}"); - - fOutput->Add(fPtHist); - - fNTracksHist = new TH1I("ntracks_dist","N_{Tracks} per Event Distribution",5,0,5); - fNTracksHist->SetDirectory(0); - fNTracksHist->GetXaxis()->SetTitle("N_{Tracks}"); - fNTracksHist->GetYaxis()->SetTitle("N_{Events}"); - - fOutput->Add(fNTracksHist); - -} - -Bool_t EventTree_Proc::Process(Long64_t entry) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either TTree::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - - // WARNING when a selector is used with a TChain, you must use - // the pointer to the current TTree to call GetEntry(entry). - // The entry is always the local entry number in the current tree. - // Assuming that fChain is the pointer to the TChain being processed, - // use fChain->GetTree()->GetEntry(entry). - - fChain->GetTree()->GetEntry(entry); - - fNTracksHist->Fill(fNtrack); - - for(Int_t j=0;jGetEntries();j++){ - Track* curtrack = dynamic_cast(fTracks->At(j)); - fPtHist->Fill(curtrack->GetPt(),1./curtrack->GetPt()); - } - fTracks->Clear("C"); - - return kTRUE; -} - -void EventTree_Proc::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - -} - -void EventTree_Proc::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - TCanvas* canvas = new TCanvas("can","can",800,600); - canvas->SetBorderMode(0); - canvas->SetLogy(); - TH1F* h = dynamic_cast(fOutput->FindObject("pt_dist")); - if (h) h->DrawCopy(); - else Warning("Terminate", "no pt dist found"); - -} diff --git a/test/ProofBench/EventTree_Proc.h b/test/ProofBench/EventTree_Proc.h deleted file mode 100644 index 1659cb69665a4..0000000000000 --- a/test/ProofBench/EventTree_Proc.h +++ /dev/null @@ -1,163 +0,0 @@ -////////////////////////////////////////////////////////// -// This class has been automatically generated on -// Fri May 7 07:24:11 2004 by ROOT version 4.00/04 -// from TTree EventTree/Event Tree -// found on file: event_tree_darkstar.root -////////////////////////////////////////////////////////// - -#ifndef EventTree_Proc_h -#define EventTree_Proc_h - -#include -#include -#include -#include -#include "Event.h" -#include "TH1.h" - -class EventTree_Proc : public TSelector { -public : - TTree *fChain; //!pointer to the analyzed TTree or TChain - - // Declaration of leave types - Event *event; - Char_t fType[20]; - Char_t *fEventName; - Int_t fNtrack; - Int_t fNseg; - Int_t fNvertex; - UInt_t fFlag; - Double32_t fTemperature; - Int_t fMeasures[10]; - Double32_t fMatrix[4][4]; - Double32_t fClosestDistance[21]; //[fNvertex] - EventHeader fEvtHdr; - TClonesArray *fTracks; - TRefArray *fHighPt; - TRefArray *fMuons; - TRef fLastTrack; - TRef fWebHistogram; - TH1F *fH; - TBits fTriggerBits; - Bool_t fIsValid; - - //Output hist - TH1F* fPtHist; - TH1I* fNTracksHist; - - // List of branches - TBranch *b_event_fType; //! - TBranch *b_fEventName; //! - TBranch *b_event_fNtrack; //! - TBranch *b_event_fNseg; //! - TBranch *b_event_fNvertex; //! - TBranch *b_event_fFlag; //! - TBranch *b_event_fTemperature; //! - TBranch *b_event_fMeasures; //! - TBranch *b_event_fMatrix; //! - TBranch *b_fClosestDistance; //! - TBranch *b_event_fEvtHdr; //! - TBranch *b_fTracks; //! - TBranch *b_fHighPt; //! - TBranch *b_fMuons; //! - TBranch *b_event_fLastTrack; //! - TBranch *b_event_fWebHistogram; //! - TBranch *b_fH; //! - TBranch *b_event_fTriggerBits; //! - TBranch *b_event_fIsValid; //! - - EventTree_Proc(TTree *) { } - EventTree_Proc() { } - virtual ~EventTree_Proc() { } - virtual Int_t Version() const {return 1;} - virtual void Begin(TTree *); - virtual void SlaveBegin(TTree *tree); - virtual void Init(TTree *tree); - virtual Bool_t Notify(); - virtual Bool_t Process(Long64_t entry); - virtual void SetOption(const char *option) { fOption = option; } - virtual void SetObject(TObject *obj) { fObject = obj; } - virtual void SetInputList(TList *input) {fInput = input;} - virtual TList *GetOutputList() const { return fOutput; } - virtual void SlaveTerminate(); - virtual void Terminate(); - - ClassDefOverride(EventTree_Proc,0); -}; - -#endif - -#ifdef EventTree_Proc_cxx -void EventTree_Proc::Init(TTree *tree) -{ - // The Init() function is called when the selector needs to initialize - // a new tree or chain. Typically here the branch addresses of the tree - // will be set. It is normaly not necessary to make changes to the - // generated code, but the routine can be extended by the user if needed. - // Init() will be called many times when running with PROOF. - - // Set branch addresses - fEventName=0; - fTracks=0; - fHighPt=0; - fMuons=0; - fH=0; - if (tree == 0) return; - fChain = tree; - fChain->SetMakeClass(1); - - fChain->SetBranchAddress("fType[20]",fType); - fChain->SetBranchAddress("fEventName",fEventName); - fChain->SetBranchAddress("fNtrack",&fNtrack); - fChain->SetBranchAddress("fNseg",&fNseg); - fChain->SetBranchAddress("fNvertex",&fNvertex); - fChain->SetBranchAddress("fFlag",&fFlag); - fChain->SetBranchAddress("fTemperature",&fTemperature); - fChain->SetBranchAddress("fMeasures[10]",fMeasures); - fChain->SetBranchAddress("fMatrix[4][4]",fMatrix); - fChain->SetBranchAddress("fClosestDistance",fClosestDistance); - fChain->SetBranchAddress("fEvtHdr",&fEvtHdr); - fChain->SetBranchAddress("fTracks",&fTracks); - fChain->SetBranchAddress("fHighPt",&fHighPt); - fChain->SetBranchAddress("fMuons",&fMuons); - fChain->SetBranchAddress("fLastTrack",&fLastTrack); - fChain->SetBranchAddress("fWebHistogram",&fWebHistogram); - fChain->SetBranchAddress("fH",&fH); - fChain->SetBranchAddress("fTriggerBits",&fTriggerBits); - fChain->SetBranchAddress("fIsValid",&fIsValid); -} - -Bool_t EventTree_Proc::Notify() -{ - // The Notify() function is called when a new file is opened. This - // can be either for a new TTree in a TChain or when when a new TTree - // is started when using PROOF. Typically here the branch pointers - // will be retrieved. It is normaly not necessary to make changes - // to the generated code, but the routine can be extended by the - // user if needed. - - // Get branch pointers - b_event_fType = fChain->GetBranch("fType[20]"); - b_fEventName = fChain->GetBranch("fEventName"); - b_event_fNtrack = fChain->GetBranch("fNtrack"); - b_event_fNseg = fChain->GetBranch("fNseg"); - b_event_fNvertex = fChain->GetBranch("fNvertex"); - b_event_fFlag = fChain->GetBranch("fFlag"); - b_event_fTemperature = fChain->GetBranch("fTemperature"); - b_event_fMeasures = fChain->GetBranch("fMeasures[10]"); - b_event_fMatrix = fChain->GetBranch("fMatrix[4][4]"); - b_fClosestDistance = fChain->GetBranch("fClosestDistance"); - b_event_fEvtHdr = fChain->GetBranch("fEvtHdr"); - b_fTracks = fChain->GetBranch("fTracks"); - b_fHighPt = fChain->GetBranch("fHighPt"); - b_fMuons = fChain->GetBranch("fMuons"); - b_event_fLastTrack = fChain->GetBranch("fLastTrack"); - b_event_fWebHistogram = fChain->GetBranch("fWebHistogram"); - b_fH = fChain->GetBranch("fH"); - b_event_fTriggerBits = fChain->GetBranch("fTriggerBits"); - b_event_fIsValid = fChain->GetBranch("fIsValid"); - - return kTRUE; -} - -#endif // #ifdef EventTree_Proc_cxx diff --git a/test/ProofBench/EventTree_ProcOpt.C b/test/ProofBench/EventTree_ProcOpt.C deleted file mode 100644 index 84b64fbf749d6..0000000000000 --- a/test/ProofBench/EventTree_ProcOpt.C +++ /dev/null @@ -1,124 +0,0 @@ -#define EventTree_ProcOpt_cxx -// The class definition in EventTree_ProcOpt.h has been generated automatically -// by the ROOT utility TTree::MakeSelector(). This class is derived -// from the ROOT class TSelector. For more information on the TSelector -// framework see $ROOTSYS/README/README.SELECTOR or the ROOT User Manual. - -// The following methods are defined in this file: -// Begin(): called everytime a loop on the tree starts, -// a convenient place to create your histograms. -// SlaveBegin(): called after Begin(), when on PROOF called only on the -// slave servers. -// Process(): called for each event, in this function you decide what -// to read and fill your histograms. -// SlaveTerminate: called at the end of the loop on the tree, when on PROOF -// called only on the slave servers. -// Terminate(): called at the end of the loop on the tree, -// a convenient place to draw/fit your histograms. -// -// To use this file, try the following session on your Tree T: -// -// Root > T->Process("EventTree_ProcOpt.C") -// Root > T->Process("EventTree_ProcOpt.C","some options") -// Root > T->Process("EventTree_ProcOpt.C+") -// - -#include "EventTree_ProcOpt.h" -#include -#include -#include "TCanvas.h" - - -void EventTree_ProcOpt::Begin(TTree *) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - -} - -void EventTree_ProcOpt::SlaveBegin(TTree *tree) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - Init(tree); - - TString option = GetOption(); - - fPtHist = new TH1F("pt_dist","p_{T} Distribution",100,0,5); - fPtHist->SetDirectory(0); - fPtHist->GetXaxis()->SetTitle("p_{T}"); - fPtHist->GetYaxis()->SetTitle("dN/p_{T}dp_{T}"); - - fOutput->Add(fPtHist); - - fNTracksHist = new TH1I("ntracks_dist","N_{Tracks} per Event Distribution",5,0,5); - fNTracksHist->SetDirectory(0); - fNTracksHist->GetXaxis()->SetTitle("N_{Tracks}"); - fNTracksHist->GetYaxis()->SetTitle("N_{Events}"); - - fOutput->Add(fNTracksHist); -} - -Bool_t EventTree_ProcOpt::Process(Long64_t entry) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either TTree::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - - // WARNING when a selector is used with a TChain, you must use - // the pointer to the current TTree to call GetEntry(entry). - // The entry is always the local entry number in the current tree. - // Assuming that fChain is the pointer to the TChain being processed, - // use fChain->GetTree()->GetEntry(entry). - - b_event_fNtrack->GetEntry(entry); - - fNTracksHist->Fill(fNtrack); - - if (fNtrack>0) { - b_fTracks->GetEntry(entry); - for(Int_t j=0;jGetEntries();j++){ - Track* curtrack = dynamic_cast(fTracks->At(j)); - fPtHist->Fill(curtrack->GetPt(),1./curtrack->GetPt()); - } - fTracks->Clear("C"); - } - - return kTRUE; -} - -void EventTree_ProcOpt::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - -} - -void EventTree_ProcOpt::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - TCanvas* canvas = new TCanvas("can","can",800,600); - canvas->SetBorderMode(0); - canvas->SetLogy(); - TH1F* h = dynamic_cast(fOutput->FindObject("pt_dist")); - if (h) h->DrawCopy(); - else Warning("Terminate", "no pt dist found"); - -} diff --git a/test/ProofBench/EventTree_ProcOpt.h b/test/ProofBench/EventTree_ProcOpt.h deleted file mode 100644 index dd8169c4912f3..0000000000000 --- a/test/ProofBench/EventTree_ProcOpt.h +++ /dev/null @@ -1,163 +0,0 @@ -////////////////////////////////////////////////////////// -// This class has been automatically generated on -// Fri May 7 07:24:11 2004 by ROOT version 4.00/04 -// from TTree EventTree/Event Tree -// found on file: event_tree_darkstar.root -////////////////////////////////////////////////////////// - -#ifndef EventTree_ProcOpt_h -#define EventTree_ProcOpt_h - -#include -#include -#include -#include -#include "Event.h" -#include "TH1.h" - -class EventTree_ProcOpt : public TSelector { -public : - TTree *fChain; //!pointer to the analyzed TTree or TChain - - // Declaration of leave types - Event *event; - Char_t fType[20]; - Char_t *fEventName; - Int_t fNtrack; - Int_t fNseg; - Int_t fNvertex; - UInt_t fFlag; - Double32_t fTemperature; - Int_t fMeasures[10]; - Double32_t fMatrix[4][4]; - Double32_t fClosestDistance[21]; //[fNvertex] - EventHeader fEvtHdr; - TClonesArray *fTracks; - TRefArray *fHighPt; - TRefArray *fMuons; - TRef fLastTrack; - TRef fWebHistogram; - TH1F *fH; - TBits fTriggerBits; - Bool_t fIsValid; - - //Output hist - TH1F* fPtHist; - TH1I* fNTracksHist; - - // List of branches - TBranch *b_event_fType; //! - TBranch *b_fEventName; //! - TBranch *b_event_fNtrack; //! - TBranch *b_event_fNseg; //! - TBranch *b_event_fNvertex; //! - TBranch *b_event_fFlag; //! - TBranch *b_event_fTemperature; //! - TBranch *b_event_fMeasures; //! - TBranch *b_event_fMatrix; //! - TBranch *b_fClosestDistance; //! - TBranch *b_event_fEvtHdr; //! - TBranch *b_fTracks; //! - TBranch *b_fHighPt; //! - TBranch *b_fMuons; //! - TBranch *b_event_fLastTrack; //! - TBranch *b_event_fWebHistogram; //! - TBranch *b_fH; //! - TBranch *b_event_fTriggerBits; //! - TBranch *b_event_fIsValid; //! - - EventTree_ProcOpt(TTree *) { } - EventTree_ProcOpt() { } - ~EventTree_ProcOpt() { } - virtual Int_t Version() const {return 1;} - virtual void Begin(TTree *); - virtual void SlaveBegin(TTree *tree); - virtual void Init(TTree *tree); - virtual Bool_t Notify(); - virtual Bool_t Process(Long64_t entry); - virtual void SetOption(const char *option) { fOption = option; } - virtual void SetObject(TObject *obj) { fObject = obj; } - virtual void SetInputList(TList *input) {fInput = input;} - virtual TList *GetOutputList() const { return fOutput; } - virtual void SlaveTerminate(); - virtual void Terminate(); - - ClassDefOverride(EventTree_ProcOpt,0); -}; - -#endif - -#ifdef EventTree_ProcOpt_cxx -void EventTree_ProcOpt::Init(TTree *tree) -{ - // The Init() function is called when the selector needs to initialize - // a new tree or chain. Typically here the branch addresses of the tree - // will be set. It is normaly not necessary to make changes to the - // generated code, but the routine can be extended by the user if needed. - // Init() will be called many times when running with PROOF. - - // Set branch addresses - fEventName=0; - fTracks=0; - fHighPt=0; - fMuons=0; - if (tree == 0) return; - fChain = tree; - fChain->SetMakeClass(1); - - fChain->SetBranchAddress("fType[20]",fType); - fChain->SetBranchAddress("fEventName",fEventName); - fChain->SetBranchAddress("fNtrack",&fNtrack); - fChain->SetBranchAddress("fNseg",&fNseg); - fChain->SetBranchAddress("fNvertex",&fNvertex); - fChain->SetBranchAddress("fFlag",&fFlag); - fChain->SetBranchAddress("fTemperature",&fTemperature); - fChain->SetBranchAddress("fMeasures[10]",fMeasures); - fChain->SetBranchAddress("fMatrix[4][4]",fMatrix); - fChain->SetBranchAddress("fClosestDistance",fClosestDistance); - fChain->SetBranchAddress("fEvtHdr",&fEvtHdr); - fChain->SetBranchAddress("fTracks",&fTracks); - fChain->SetBranchAddress("fHighPt",&fHighPt); - fChain->SetBranchAddress("fMuons",&fMuons); - fChain->SetBranchAddress("fLastTrack",&fLastTrack); - fChain->SetBranchAddress("fWebHistogram",&fWebHistogram); - fChain->SetBranchAddress("fH",&fH); - fChain->SetBranchAddress("fTriggerBits",&fTriggerBits); - fChain->SetBranchAddress("fIsValid",&fIsValid); - -} - -Bool_t EventTree_ProcOpt::Notify() -{ - // The Notify() function is called when a new file is opened. This - // can be either for a new TTree in a TChain or when when a new TTree - // is started when using PROOF. Typically here the branch pointers - // will be retrieved. It is normaly not necessary to make changes - // to the generated code, but the routine can be extended by the - // user if needed. - - // Get branch pointers - b_event_fType = fChain->GetBranch("fType[20]"); - b_fEventName = fChain->GetBranch("fEventName"); - b_event_fNtrack = fChain->GetBranch("fNtrack"); - b_event_fNseg = fChain->GetBranch("fNseg"); - b_event_fNvertex = fChain->GetBranch("fNvertex"); - b_event_fFlag = fChain->GetBranch("fFlag"); - b_event_fTemperature = fChain->GetBranch("fTemperature"); - b_event_fMeasures = fChain->GetBranch("fMeasures[10]"); - b_event_fMatrix = fChain->GetBranch("fMatrix[4][4]"); - b_fClosestDistance = fChain->GetBranch("fClosestDistance"); - b_event_fEvtHdr = fChain->GetBranch("fEvtHdr"); - b_fTracks = fChain->GetBranch("fTracks"); - b_fHighPt = fChain->GetBranch("fHighPt"); - b_fMuons = fChain->GetBranch("fMuons"); - b_event_fLastTrack = fChain->GetBranch("fLastTrack"); - b_event_fWebHistogram = fChain->GetBranch("fWebHistogram"); - b_fH = fChain->GetBranch("fH"); - b_event_fTriggerBits = fChain->GetBranch("fTriggerBits"); - b_event_fIsValid = fChain->GetBranch("fIsValid"); - - return kTRUE; -} - -#endif // #ifdef EventTree_ProcOpt_cxx diff --git a/test/ProofBench/Makefile_event b/test/ProofBench/Makefile_event deleted file mode 100644 index 7b2baa0adca4c..0000000000000 --- a/test/ProofBench/Makefile_event +++ /dev/null @@ -1,72 +0,0 @@ -# Makefile for the ROOT test programs. -# This Makefile shows nicely how to compile and link applications -# using the ROOT libraries on all supported platforms. -# -# Copyright (c) 2000 Rene Brun and Fons Rademakers -# -# Author: Fons Rademakers, 29/2/2000 - -include Makefile.arch - -#------------------------------------------------------------------------------ - -EVENTO = Event.$(ObjSuf) EventDict.$(ObjSuf) -EVENTS = Event.$(SrcSuf) EventDict.$(SrcSuf) -EVENTSO = libEvent.$(DllSuf) -ifeq ($(PLATFORM),win32) -EVENTLIB = libEvent.lib -else -EVENTLIB = $(EVENTSO) -endif - -MAINEVENTO = MainEvent.$(ObjSuf) -MAINEVENTS = MainEvent.$(SrcSuf) - -OBJS = $(EVENTO) $(MAINEVENTO) - -#------------------------------------------------------------------------------ - -.SUFFIXES: .$(SrcSuf) .$(ObjSuf) .$(DllSuf) - -all: $(EVENTLIB) - -$(EVENTSO): $(EVENTO) -ifeq ($(ARCH),aix5) - $(MAKESHARED) $(OutPutOpt) $@ $(LIBS) -p 0 $^ -else -ifeq ($(PLATFORM),macosx) - $(LD) $(SOFLAGS)$@ $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -else -ifeq ($(PLATFORM),win32) - bindexplib $* $^ > $*.def - lib -nologo -MACHINE:IX86 $^ -def:$*.def \ - $(OutPutOpt)$(EVENTLIB) - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $*.exp $(LIBS) \ - $(OutPutOpt)$@ - $(MT_DLL) -else - $(LD) $(SOFLAGS) $(LDFLAGS) $^ $(OutPutOpt) $@ $(EXPLLINKLIBS) -endif -endif -endif - @echo "$@ done" - -clean: - @rm -f $(OBJS) core - -distclean: clean - @rm -f $(EVENTSO) $(EVENTLIB) *Dict.* *.pcm *.def *.exp \ - *.so *.lib *.dll *.d *.log .def so_locations - @rm -rf cxx_repository - -# Dependencies - -Event.$(ObjSuf): Event.h -MainEvent.$(ObjSuf): Event.h - -EventDict.$(SrcSuf): Event.h EventLinkDef.h - @echo "Generating dictionary $@..." - $(ROOTCLING) -f $@ -c $^ - -.$(SrcSuf).$(ObjSuf): - $(CXX) $(CXXFLAGS) -c $< diff --git a/test/ProofBench/README b/test/ProofBench/README deleted file mode 100644 index 7918b61ec59a1..0000000000000 --- a/test/ProofBench/README +++ /dev/null @@ -1,149 +0,0 @@ -Setup for a benchmark -===================== - -See $ROOTSYS/README/README.PROOF for instructions to install -and configure PROOF - -Create the PAR file - -% ./make_event_par.sh - -Start root - -% root - -Start PROOF - -root[] TProof::Open("") - -Create the files on the nodes in the cluster, the first -argument is the directory the files will reside in. The -second argument is the number of events in each file. -The last argument is the number of files on each node. -(not the number of files per slave!) - -root[] .x make_event_trees.C("/data1/tmp", 100000, 4) - -Create the TDSet for these files: - -root[] .L make_tdset.C -root[] TDSet *d = make_tdset("/data1/tmp",4) -root[] d->Print("a") -OBJ: TDSet type TTree EventTree in / elements 2 -TDSetElement file='root://gluon.local//data1/tmp/event_tree_gluon.local_1.root' dir='' obj='' first=0 num=-1 -TDSetElement file='root://gluon.local//data1/tmp/event_tree_gluon.local_3.root' dir='' obj='' first=0 num=-1 - -Test the system with a simple command - -root[] d->Draw("fTemperature","1") - -You are now ready to run the benchmark! - - -Performance Monitoring -====================== - -Select which performance information should be gathered. -To enable the filling of performance histograms in the master do: - -root[] gEnv->SetValue("Proof.StatsHist",1); - -or add a line to your ".rootrc" or "system.rootrc" file. -The histograms will be returned to the client in the -output list. The histograms are: - -Name Type Description ----- ---- ----------- - -PROOF_PacketsHist TH1D "Packets processed per Slave" - -PROOF_EventsHist TH1D "Events processed per Slave" - -PROOF_NodeHist TH1D "Slaves per Fileserving Node" - dynamically updated, use in Feedback. - -PROOF_LatencyHist TH2D "GetPacket Latency per Slave" - -PROOF_ProcTimeHist TH2D "Packet Processing Time per Slave" - -PROOF_CpuTimeHist TH2D "Packet CPU Time per Slave" - - -To enable the creation of the trace tree and store -performance events in the master server do: - -root[] gEnv->SetValue("Proof.StatsTrace",1); - -to also record the detailed performance events in the slaves -call in addition: - -root[] gEnv->SetValue("Proof.SlaveStatsTrace",1); - -or add them to either your ".rootrc" or "system.rootrc" -A single Tree named "PROOF_PerfStats" will be returned to the client -in the output list. - -The script SavePerfInfo.C can be used to save the -current results in a .root file, e.g. - -root[] .X SavePerfInfo.C("perf_data.root") - -Run the Benchmark -================= - -The benchmark provides 3 selectors, each reading a different amount -of data: - -EventTree_NoProc.C - Reads no data -EventTree_ProcOpt.C - Reads 25% of the data -EventTree_Proc.C - Reads all the data - -First make sure the PAR file is up to date and enabled - -root[] gProof->UploadPackage("event.par") -root[] gProof->EnablePackage("event") - -Request dynamic feedback of some of the monitoring histograms - -root[] gProof->AddFeedback("PROOF_ProcTimeHist") -root[] gProof->AddFeedback("PROOF_LatencyHist") -root[] gProof->AddFeedback("PROOF_EventsHist") - -Create a TDrawFeedback object to automatically draw these histograms - -root[] TDrawFeedback fb(gProof) - -And request the timing of each command - -root[] gROOT->Time() - -Running one of the provided selectors is straight forward, -using the TDSet that was created earlier: - -root[] .L make_tdset.C -root[] TDSet *d = make_tdset("/data1/tmp",4) -root[] gProof->Load("EventTree_Proc.C+") -root[] d->Process("EventTree_Proc","") - -The monitoring histograms should appear shortly after the -processing starts. The resulting histogram from the selector -will also be drawn at the end. - -Loading the selector before processing is not strictly needed but it allows to -circumvent a problem with file distribution that was present in some versions, -including 5.22/00 and 5.22/00a. - -The above set of commands are included in the script Run_Simple_Test.C . - - -Extra Scripts Included -====================== - -The script Draw_Time_Hists.C can be used to create the timing histograms -from the trace tree and draw them on screen. - -The script Run_Node_Tests.C can be used to run a full sequence of -tests. The results can be presented graphically using Draw_PerfProfiles.C. - -The script Draw_Slave_Access.C will draw a graph depicting the number -of slaves accessing a file serving node as a function of time. diff --git a/test/ProofBench/Run_Node_Tests.C b/test/ProofBench/Run_Node_Tests.C deleted file mode 100644 index 322be9e07d0df..0000000000000 --- a/test/ProofBench/Run_Node_Tests.C +++ /dev/null @@ -1,251 +0,0 @@ -void run_selector_tests(TString selector_name, Int_t fps, Int_t niter, - TFile* outputfile, TString name_stem, - const Char_t *basedir, Int_t stepsize, Int_t start); - -void Run_Node_Tests(Int_t fps, Int_t niter, const Char_t *basedir, - TFile *outputfile, Int_t stepsize=5, Int_t start=0) -{ - // This script collects performance statistics (to be saved in outputfile) - // for running a PROOF cluster with an increasing number of slaves. The - // script runs multiple times for each number of slaves to gather - // statistics for each number of slaves. - - // fps: number of files to be processed per slave - // niter: number of iterations to run each test at each number of nodes - // basedir: base directory where files are located on slave - // (same input as given to make_event_trees.C) - // outputfile: outputfile to write output info to - // stepsize: number of slaves to increase by when performing tests - // start: number of slaves of slaves to start test with - // (0 means start with stepsize slaves) - - if (!gProof) { - cout << "PROOF must be started before running this script" << endl; - return; - } - - if (fps<1) { - cout << "Must be testing at least 1 file per slave" << endl; - return; - } - - if (niter<1) { - cout << "Atleast 1 iteration per step is required" << endl; - return; - } - - if (stepsize<1) { - cout << "Must increase subsequent tests by atleast 1 slave" << endl; - return; - } - - if (start<0) { - cout << "starting number of nodes must be atleast 1" << endl - << " (with 0 making start = stepsize)" << endl; - return; - } - - if (start==0) start = stepsize; - -#if 0 - // load Event library - if (!TString(gSystem->GetLibraries()).Contains("Event")) { - if(gSystem->Load("$ROOTSYS/test/libEvent.so")) { - cout << "Could not load libEvent.so" << endl; - return; - } - } - - // add $ROOTSYS/test to include path - if (!TString(gSystem->GetIncludePath()).Contains("-I$ROOTSYS/test")) - gSystem->AddIncludePath("-I$ROOTSYS/test"); -#endif - - if (gProof->UploadPackage("event")) { - cout << "Could not upload Event par file to slaves" << endl; - return; - } - - if (gProof->EnablePackage("event")) { - cout << "Could not enable Event library on PROOF slaves" << endl; - return; - } - - if (gROOT->LoadMacro("make_tdset.C")) { - cout << "Could not load make_tdset macro" << endl; - return; - } - -#if 0 - //use new packetizer - gProof->AddInput(new TNamed("PROOF_NewPacketizer","")); -#else - gProof->SetParameter("PROOF_ForceLocal", 1); - -#endif - - //run the tests - run_selector_tests("EventTree_Proc.C++", fps, niter, outputfile, "_Proc", basedir, stepsize, start); - run_selector_tests("EventTree_ProcOpt.C++", fps, niter, outputfile, "_ProcOpt", basedir, stepsize, start); - run_selector_tests("EventTree_NoProc.C++", fps, niter, outputfile, "_NoProc", basedir, stepsize, start); - -} - -void run_selector_tests(TString selector_file, Int_t fps, Int_t niter, - TFile* outputfile, TString name_stem, - const Char_t *basedir, Int_t stepsize, Int_t start) -{ - - TString selector_name(selector_file); - selector_name.Remove(selector_file.Last('.')); - - gProof->SetParallel(9999); - Int_t nslaves = gProof->GetParallel(); - if (start > nslaves) { - cout << "starting number of nodes must be atleast 1" << endl - << " (with 0 making start = stepsize)" << endl; - return; - } - - //run once on all nodes with no logging - gEnv->SetValue("Proof.StatsHist",0); - gEnv->SetValue("Proof.StatsTrace",0); - gEnv->SetValue("Proof.SlaveStatsTrace",0); - TDSet* dset = make_tdset(basedir, fps); - cout << "Running on all nodes first" << endl; - gProof->Load(selector_file); - dset->Process(selector_name); -// delete dset; - cout << "done" << endl; - - // switch logging on (no slave logging) - gEnv->SetValue("Proof.StatsHist",0); - gEnv->SetValue("Proof.StatsTrace",1); - gEnv->SetValue("Proof.SlaveStatsTrace",0); - - TString perfstats_name = "PROOF_PerfStats"; - - // set up timing tree info - TString time_tree_name = perfstats_name; - time_tree_name+=name_stem; - time_tree_name+="_timing_tree"; - TTree timing_tree(time_tree_name,"Timing Tree"); - Int_t ns_holder; - Int_t run_holder; - Float_t time_holder; - TBranch* br = timing_tree.Branch("tests",&ns_holder,"nslaves/I:run/I:time/F"); - if (outputfile && !outputfile->IsZombie()) - timing_tree.SetDirectory(outputfile); - br->GetLeaf("nslaves")->SetAddress(&ns_holder); - br->GetLeaf("run")->SetAddress(&run_holder); - br->GetLeaf("time")->SetAddress(&time_holder); - - Bool_t done = kFALSE; - for (Int_t nactive=start; !done; nactive+=stepsize) { - - if (nactive >= nslaves) { - done=kTRUE; - nactive=nslaves; - } - - gProof->SetParallel(nactive); - ns_holder = nactive; - for (Int_t j=0; jNow(); - dset->Process(selector_name); - TTime endtime = gSystem->Now(); - time_holder = Long_t(endtime-starttime)/Float_t(1000); - cout << "Processing time was " << time_holder << " seconds" << endl; - - if (outputfile && !outputfile->IsZombie()) { - TList* l = dset->GetOutputList(); - - //save perfstats - TTree* t = dynamic_cast(l->FindObject(perfstats_name.Data())); - if (t) { - TDirectory* trdir = t->GetDirectory(); - TDirectory* dirsav = gDirectory; - outputfile->cd(); - t->SetDirectory(outputfile); - TString origname = t->GetName(); - TString newname = perfstats_name; - newname+=name_stem; - newname+="_"; - newname+=nactive; - newname+="slaves_run"; - newname+=j; - t->SetName(newname); - t->Write(); - t->SetName(origname); - t->SetDirectory(trdir); - dirsav->cd(); - } else { - cout << perfstats_name.Data() << " tree not found" << endl << flush; - } - - //save outputhistos - TString ptdist_name = "pt_dist"; - TH1* h = dynamic_cast(l->FindObject(ptdist_name.Data())); - if (h) { - TDirectory* hdir = h->GetDirectory(); - TDirectory* dirsav = gDirectory; - outputfile->cd(); - h->SetDirectory(outputfile); - TString origname = h->GetName(); - TString newname = ptdist_name; - newname+=name_stem; - newname+="_"; - newname+=nactive; - newname+="slaves_run"; - newname+=j; - h->SetName(newname); - h->Write(); - h->SetName(origname); - h->SetDirectory(hdir); - dirsav->cd(); - } else { - cout << ptdist_name.Data() << " histogram not found" << endl << flush; - } - - TString tracksdist_name = "ntracks_dist"; - TH1* h2 = dynamic_cast(l->FindObject(tracksdist_name.Data())); - if (h2) { - TDirectory* hdir = h2->GetDirectory(); - TDirectory* dirsav = gDirectory; - outputfile->cd(); - h2->SetDirectory(outputfile); - TString origname = h2->GetName(); - TString newname = tracksdist_name; - newname+=name_stem; - newname+="_"; - newname+=nactive; - newname+="slaves_run"; - newname+=j; - h2->SetName(newname); - h2->Write(); - h2->SetName(origname); - h2->SetDirectory(hdir); - dirsav->cd(); - } else { - cout << tracksdist_name.Data() << " histogram not found" << endl << flush; - } - } - -// delete dset; - timing_tree.Fill(); - } - } - - delete dset; - - if (outputfile && !outputfile->IsZombie()) { - TDirectory* dirsav=gDirectory; - outputfile->cd(); - timing_tree.Write(); - timing_tree.SetDirectory(0); - dirsav->cd(); - } -} - diff --git a/test/ProofBench/Run_Simple_Test.C b/test/ProofBench/Run_Simple_Test.C deleted file mode 100644 index 2723648eb8441..0000000000000 --- a/test/ProofBench/Run_Simple_Test.C +++ /dev/null @@ -1,79 +0,0 @@ -// $Id: Run_Simple_Test.C,v 1.5 2005/02/12 02:14:54 rdm Exp $ -// -// - -// Preparation: -// - configure authentication -// - configure $ROOTSYS/proof/etc/proof.conf (use full domain name) -// - optional: change the data directory below (needs 350 Mb) -// - start xrootd with proof - -#include "TSystem.h" -#include "TEnv.h" -#include "TProof.h" -#include "TDSet.h" -#include "TDrawFeedback.h" -#include "TList.h" - -#include "make_event_trees.C" -#include "make_tdset.C" - -const bool show_feedback = true; -const bool show_stats = true; -const int files_per_node = 10; -const int max_files_per_slave = 1; - -void Run_Simple_Test(char *host, const char *data_dir = "/var/tmp", Int_t dbg = 0) -{ - - if (gSystem->AccessPathName("../libEvent.so")) { - gSystem->Exec("cd $ROOTSYS/test; make libEvent.so"); - gSystem->Exec("./make_event_par.sh"); - gSystem->Exec("ln -s ../Event.h"); - gSystem->Load("../libEvent.so"); - } - - TProof *p = 0; - if (dbg > 0) { - p = TProof::Open(host, "", "", dbg); - } else { - p = TProof::Open(host); - } - if (!p || !p->IsValid()) { - cout << "Could not start the PROOF session - exit "< 0) p->SetLogLevel(dbg); - - p->UploadPackage("event"); - p->EnablePackage("event"); - - make_event_trees(data_dir, 100000, files_per_node); - - TDSet *d = make_tdset(data_dir, files_per_node); - - d->Print("a"); - - if (show_stats) gEnv->SetValue("Proof.StatsHist", 1); - - TDrawFeedback *fb(0); - if (show_feedback) { - if (show_stats) { - p->AddFeedback("PROOF_ProcTimeHist"); - p->AddFeedback("PROOF_LatencyHist"); - p->AddFeedback("PROOF_EventsHist"); - } - - p->AddFeedback("pt_dist"); - - fb = new TDrawFeedback(p); - } - - // We load the selector before execution to avoid a problem affecting some versions - // (in particular 5.22/00 and 5.22/00a); with problem-free versions this is equivalent - // to direct processing via d->Process("EventTree_Proc.C+","") - p->Load("EventTree_Proc.C+"); - d->Process("EventTree_Proc"); - - if (fb) delete fb; -} diff --git a/test/ProofBench/SavePerfInfo.C b/test/ProofBench/SavePerfInfo.C deleted file mode 100644 index b39b993799bdd..0000000000000 --- a/test/ProofBench/SavePerfInfo.C +++ /dev/null @@ -1,44 +0,0 @@ -#include -#include "TFile.h" -#include "TIterator.h" -#include "TRegexp.h" -#include "TProof.h" - - -void SavePerfInfo(const Char_t *filename) -{ - // Save PROOF timing information from TPerfStats to file 'filename' - - if (!gProof) { - cout << "PROOF must be run to save output performance information" << endl; - return; - } - - TFile f(filename, "UPDATE"); - if (f.IsZombie()) { - cout << "Could not open file " << filename << " for writing" << endl; - } else { - f.cd(); - - TIter NextObject(gProof->GetOutputList()); - TObject* obj = 0; - while (obj = NextObject()) { - TString objname = obj->GetName(); - if (objname.Contains(TRegexp("^PROOF_"))) { - // must list the objects since other PROOF_ objects exist - // besides timing objects - if (objname == "PROOF_PerfStats" || - objname == "PROOF_PacketsHist" || - objname == "PROOF_EventsHist" || - objname == "PROOF_NodeHist" || - objname == "PROOF_LatencyHist" || - objname == "PROOF_ProcTimeHist" || - objname == "PROOF_CpuTimeHist") - obj->Write(); - } - } - - f.Close(); - } - -} diff --git a/test/ProofBench/make_event_par.sh b/test/ProofBench/make_event_par.sh deleted file mode 100755 index 6360b6392d51f..0000000000000 --- a/test/ProofBench/make_event_par.sh +++ /dev/null @@ -1,65 +0,0 @@ -#! /bin/sh -# -# Make the event.par file to be used to analyse Event objects with PROOF. -# -# Usage: sh make_event_par.sh -# -# Creates the PAR file "event.par" which can be used in PROOF via the -# package manager like: -# gProof->UploadPackage("event.par") -# gProof->EnablePackage("event") -# -# Command to check that package is active and that libEvent.so is loaded: -# gProof->ShowPackages() -# gProof->ShowEnabledPackages() -# gProof->Exec("gSystem->ListLibraries()") -# - -EDIR=event - -if [ -d $EDIR ]; then - rm -rf $EDIR -fi - -mkdir $EDIR - - -SRC=$ROOTSYS/test -ETC=$ROOTSYS/etc -cp $SRC/Event.cxx $SRC/Event.h $SRC/EventLinkDef.h \ - $ETC/Makefile.arch $EDIR -cp Makefile_event $EDIR/Makefile -mkdir $EDIR/PROOF-INF -cd $EDIR/PROOF-INF - -cat > BUILD.sh < SETUP.C <Load("libEvent") == -1) - return -1; - return 0; -} -EOF - -chmod 755 BUILD.sh - -cd ../.. - -tar zcvf event.par $EDIR - -# don't remove the directory as it is needed locally by PROOF-Lite -#rm -rf $EDIR - -exit 0 diff --git a/test/ProofBench/make_event_trees.C b/test/ProofBench/make_event_trees.C deleted file mode 100644 index d07fef758975b..0000000000000 --- a/test/ProofBench/make_event_trees.C +++ /dev/null @@ -1,138 +0,0 @@ -// $Id: make_event_trees.C,v 1.11 2007/07/31 16:20:23 rdm Exp $ -// -// - -#include "Riostream.h" -#include "TProof.h" -#include "TString.h" - - -Bool_t make_event_trees(const char *basedir, Int_t events_per_file, - Int_t files_per_node) -{ - // This script generates files on PROOF nodes with trees containing - // Event objects. - // The created files can be used for benchmarking and demonstrations. - - if (!gProof) { - cout << "Must Start PROOF before using make_event_trees.C" << endl; - return kFALSE; - } - - if (!basedir) { - cout << "'basedir' must not be empty" << endl; - return kFALSE; - } - - if (events_per_file <= 0) { - cout << "events_per_file must be > 0" << endl; - return kFALSE; - } - - if (files_per_node <= 0) { - cout << "files_per_node must be > 0" << endl; - return kFALSE; - } - - if (gProof->UploadPackage("event.par")) return kFALSE; - if (gProof->EnablePackage("event")) return kFALSE; - - ofstream slavemacro("build_trees.C"); - - slavemacro << "#include \"TSystem.h\"" << endl; - slavemacro << "#include \"TProof.h\"" << endl; - slavemacro << "#include \"TProofServ.h\"" << endl; - slavemacro << "#include \"TRandom.h\"" << endl; - slavemacro << "#include \"TFile.h\"" << endl; - slavemacro << "#include \"TTree.h\"" << endl; - slavemacro << "#include \"Riostream.h\"" << endl; - slavemacro << "#include \"event/Event.h\"" << endl; - slavemacro << "void build_trees(const char *basedir, Int_t nevents, Int_t nfiles) {" << endl; - slavemacro << " Int_t slave_number = -1;" << endl; - slavemacro << " Int_t nslaves = 0;" << endl; - if (!strncmp(gProof->GetMaster(), "localhost", 9)) - slavemacro << " TString hn = \"localhost\";" << endl; - else - slavemacro << " TString hn = gSystem->HostName();" << endl; - slavemacro << " TString ord = gProofServ->GetOrdinal();" << endl; - slavemacro << endl; - TList* l = gProof->GetListOfSlaveInfos(); - for (Int_t i=0; iGetSize(); i++) { - TSlaveInfo* si = dynamic_cast(l->At(i)); - if (si->fStatus != TSlaveInfo::kActive) continue; - slavemacro << " if (hn == \""; - slavemacro << si->fHostName; - slavemacro << "\") { nslaves++; if (ord == \""; - slavemacro << si->fOrdinal; - slavemacro << "\") slave_number = nslaves; }" << endl; - } - - slavemacro << " if (gSystem->AccessPathName(basedir)) {" << endl; - slavemacro << " Printf(\"No such file or directory: %s\", basedir);" << endl; - slavemacro << " return;" << endl; - slavemacro << " }" << endl; - slavemacro << endl; - slavemacro << " if (slave_number >= 0) {" << endl; - slavemacro << " for(Int_t i=slave_number; i<=nfiles; i+=nslaves) {" << endl; - slavemacro << endl; - slavemacro << " TString seed = hn;" << endl; - slavemacro << " seed += \"_\";" << endl; - slavemacro << " seed += i;" << endl; - slavemacro << " gRandom->SetSeed(static_cast(TMath::Hash(seed)));" << endl; - slavemacro << endl; - slavemacro << " TString filename = basedir;" << endl; - slavemacro << " filename += \"/event_tree_\";" << endl; - slavemacro << " filename += seed;" << endl; - slavemacro << " filename += \".root\";" << endl; - slavemacro << " TDirectory* savedir = gDirectory;" << endl; - slavemacro << " TFile *f = TFile::Open(filename, \"RECREATE\");" << endl; - slavemacro << " savedir->cd();" << endl; - slavemacro << endl; - slavemacro << " if (!f || f->IsZombie()) break;" << endl; - slavemacro << " Event event;" << endl; - slavemacro << " Event *ep = &event;" << endl; - slavemacro << " TTree eventtree(\"EventTree\", \"Event Tree\");" << endl; - slavemacro << " eventtree.SetDirectory(f);" << endl; - slavemacro << " eventtree.Branch(\"event\", \"Event\", &ep, 32000, 1);" << endl; - slavemacro << " eventtree.AutoSave();" << endl; - slavemacro << endl; - slavemacro << " for(Int_t j=0; jcd();" << endl; - slavemacro << " eventtree.Write();" << endl; - slavemacro << " eventtree.SetDirectory(0);" << endl; - slavemacro << " f->Close();" << endl; - slavemacro << " delete f;" << endl; - slavemacro << " f = 0;" << endl; - slavemacro << " savedir->cd();" << endl; - slavemacro << endl; - slavemacro << " }" << endl; - slavemacro << " } else {" << endl; - slavemacro << " cout << \"Could not find slave hostname=\";" << endl; - slavemacro << " cout << hn << \", ordinal=\" << ord;" << endl; - slavemacro << " cout << \" in file production list.\" << endl;" << endl; - slavemacro << " cout << \"Make sure the proof.conf contains the \";" << endl; - slavemacro << " cout << \"correct slave hostnames.\" << endl;" << endl; - slavemacro << " }" << endl; - slavemacro << "}" << endl; - - slavemacro.close(); - - gProof->Load("build_trees.C+"); - - TString cmd = "build_trees(\""; - cmd += basedir; - cmd += "\","; - cmd += events_per_file; - cmd += ","; - cmd += files_per_node; - cmd += ")"; - - if (gProof->Exec(cmd)<0) return kFALSE; - - return kTRUE; -} diff --git a/test/ProofBench/make_tdset.C b/test/ProofBench/make_tdset.C deleted file mode 100644 index 4532a6de3ab15..0000000000000 --- a/test/ProofBench/make_tdset.C +++ /dev/null @@ -1,77 +0,0 @@ -// $Id: make_tdset.C,v 1.9 2007/07/16 09:57:39 rdm Exp $ -// -// - -#include "Riostream.h" -#include "TDSet.h" -#include "THashList.h" -#include "TObjString.h" -#include "TMap.h" -#include "TProof.h" - - -TDSet *make_tdset(const char *basedir, Int_t files_per_node) -{ - // This script creates a TDSet object that can be used to process - // the files generated with the make_event_trees.C script. - // Conventions for file names made by that script are assumed. - // - // basedir: location of files local to proof slaves - // files_per_slave: number of files per node - - if (!gProof) { - cout << "Must Start PROOF before using make_tdset.C" << endl; - return 0; - } - - if (!basedir) { - cout << "'basedir' must not be empty" << endl; - return 0; - } - - if (files_per_node <= 0) { - cout << "files_per_node must be > 0" << endl; - return 0; - } - - TList* l = gProof->GetListOfSlaveInfos(); - if (!l) { - cout << "No list of workers received!" << endl; - return 0; - } - TIter nxw(l); - TSlaveInfo *si = 0; - - THashList nodelist; - nodelist.SetOwner(kFALSE); - while ((si = (TSlaveInfo *) nxw())) { - if (!nodelist.FindObject(si->GetName())) nodelist.Add(new TPair(new TObjString(si->GetName()), si)); - } - - TDSet *d = new TDSet("TTree","EventTree"); - TIter nxu(&nodelist); - TPair *p = 0; - si = 0; - while ((p = (TPair *) nxu())) { - si = (TSlaveInfo *) p->Value(); - for (Int_t j = 1; j <= files_per_node ; j++) { - TString filestr; - if (gProof->IsLite()) { - filestr += "file://"; - } else { - filestr += "root://"; - filestr += si->GetName(); - filestr += "/"; - } - filestr += basedir; - filestr += "/event_tree_"; - filestr += si->GetName(); - filestr += "_"; - filestr += j; - filestr += ".root"; - d->Add(filestr); - } - } - - return d; -} diff --git a/test/stressProof.cxx b/test/stressProof.cxx deleted file mode 100644 index 5cac5106c8b7a..0000000000000 --- a/test/stressProof.cxx +++ /dev/null @@ -1,4585 +0,0 @@ -// ************************************************************************* // -// * * // -// * s t r e s s P r o o f * // -// * * // -// * This file contains a set of test of PROOF related functionality. * // -// * The tests can be run as a standalone program or with the interpreter. * // -// * To run as a standalone program: * // -// * * // -// * $ cd $ROOTSYS/test * // -// * $ make stressProof * // -// * * // -// * Run stressProof with '-h' to get the list of supported options * // -// * $ ./stressProof -h * // -// * * // -// * To run interactively: * // -// * $ root * // -// * root[] .include $ROOTSYS/tutorials * // -// * root[] .L stressProof.cxx+ * // -// * root[] stressProof(master, tests, wrks, verbose, logfile, dyn, \ * // -// * dyn, skipds, h1src, eventsrc, dryrun) * // -// * * // -// * The arguments have the same meaning as above except for * // -// * verbose [Int_t] increasing verbosity (0 == minimal) * // -// * dyn [Bool_t] if kTRUE run in dynamic startup mode * // -// * skipds [Bool_t] if kTRUE the dataset related tests are skipped * // -// * * // -// * A certain number of swithces can also be controlled via environment * // -// * variables: check './stressProof -h' * // -// * * // -// * The stressProof function returns 0 on success, 1 on failure. * // -// * * // -// * The successful output looks like this: * // -// * * // -// * ****************************************************************** * // -// * * Starting P R O O F - S T R E S S suite * * // -// * ****************************************************************** * // -// * * Log file: /tmp/ProofStress_XrcwBe * // -// * ****************************************************************** * // -// * Test 1 : Open a session ................................... OK * * // -// * Test 2 : Get session logs ................................. OK * * // -// * Test 3 : Simple random number generation .................. OK * * // -// * Test 4 : Dataset handling with H1 files ................... OK * * // -// * Test 5 : H1: chain processing ............................. OK * * // -// * Test 6 : H1: file collection processing ................... OK * * // -// * Test 7 : H1: file collection, TPacketizerAdaptive ......... OK * * // -// * Test 8 : H1: by-name processing ........................... OK * * // -// * Test 9 : H1: multi dataset processing ..................... OK * * // -// * Test 10 : H1: multi dataset and entry list ................. OK * * // -// * Test 11 : Package management with 'event' .................. OK * * // -// * Test 12 : Package argument passing ......................... OK * * // -// * Test 13 : Simple 'event' generation ........................ OK * * // -// * Test 14 : Input data propagation ........................... OK * * // -// * Test 15 : H1, Simple: async mode :.......................... OK * * // -// * Test 16 : Admin functionality .............................. OK * * // -// * Test 17 : Dynamic sub-mergers functionality ................ OK * * // -// * Test 18 : Event range processing ........................... OK * * // -// * Test 19 : Event range, TPacketizerAdaptive ................. OK * * // -// * Test 20 : File-resident output: merge ...................... OK * * // -// * Test 21 : File-resident output: merge w/ submergers ........ OK * * // -// * Test 22 : File-resident output: create dataset ............. OK * * // -// * Test 23 : File-resident output: multi trees ................ OK * * // -// * Test 24 : TTree friends (and TPacketizerFile) .............. OK * * // -// * Test 25 : TTree friends, same file ......................... OK * * // -// * Test 26 : Handling output via file ......................... OK * * // -// * Test 27 : Simple: selector by object ....................... OK * * // -// * Test 28 : H1 dataset: selector by object ................... OK * * // -// * Test 29 : Chain with TTree in subdirs ...................... OK * * // -// * * All registered tests have been passed :-) * * // -// * ****************************************************************** * // -// * * // -// * The application redirects the processing logs to a log file which is * // -// * normally deleted at the end of a successful run; if the test fails * // -// * the caller is asked if she/he wants to keep the log file; if the * // -// * specifies a log file path of her/his choice, the log file is never * // -// * deleted. * // -// * * // -// * SKIPPED means that the test cannot be run. * // -// * * // -// * New tests can be easily added by providing a function performing the * // -// * test and a name for the test; see examples below. * // -// * * // -// * It is also possible to trigger the automatic PROOF valgrind setup by * // -// * means of the env GETPROOF_VALGRIND. * // -// * E.g. to run the master in valgrind do * // -// * * // -// * $ export GETPROOF_VALGRIND="valgrind=master" * // -// * or * // -// * $ export GETPROOF_VALGRIND="valgrind=workers" * // -// * * // -// * before running stressProof. The syntax is the same as for standard * // -// * PROOF valgrind runs. * // -// * * // -// ************************************************************************* // - -#include -#include -#include -#include -#include -#include -#ifdef WIN32 -#include -#endif - -#include "Getline.h" -#include "TApplication.h" -#include "TChain.h" -#include "TDataMember.h" -#include "TDSet.h" -#include "TFile.h" -#include "TFileCollection.h" -#include "TFileInfo.h" -#include "TH1F.h" -#include "TH2F.h" -#include "THashList.h" -#include "TList.h" -#include "TMacro.h" -#include "TMap.h" -#include "TMath.h" -#include "TMethodCall.h" -#include "TNamed.h" -#include "TNtuple.h" -#include "TParameter.h" -#include "TProof.h" -#include "TProofLog.h" -#include "TProofMgr.h" -#include "TProofOutputFile.h" -#include "TQueryResult.h" -#include "TStopwatch.h" -#include "TString.h" -#include "TSystem.h" -#include "TROOT.h" -#include "TSelector.h" -#include "TProofProgressDialog.h" -#include "TProofProgressLog.h" -#include "TProofProgressMemoryPlot.h" -#include "TObjString.h" - -#include "legacy/proof/getProof.C" - -#define PT_NUMTEST 29 - -static const char *urldef = "proof://localhost:40000"; -static TString gtutdir; -static TString gsandbox; -static Int_t gverbose = 1; -static TString gverbproof("kAll"); -static TString glogfile; -static Int_t gpoints = 0; -static Bool_t guseprogress = kTRUE; -static Int_t totpoints = 53; -static RedirectHandle_t gRH; -static RedirectHandle_t gRHAdmin; -static Double_t gH1Time = 0; -static Double_t gSimpleTime = 0; -static Double_t gEventTime = 0; -static Int_t gH1Cnt = 0; -static Int_t gSimpleCnt = 0; -static Int_t gEventCnt = 0; -static TStopwatch gTimer; -static Bool_t gTimedOut = kFALSE; -static Bool_t gDynamicStartup = kFALSE; -static Bool_t gLocalCluster = kTRUE; -static Bool_t gSkipDataSetTest = kTRUE; -static Bool_t gUseParallelUnzip = kFALSE; -static Bool_t gClearCache = kFALSE; -static TString gh1src("http://root.cern/files/h1"); -static Bool_t gh1ok = kTRUE; -static Bool_t gh1local = kFALSE; -static char gh1sep = '/'; -static const char *gh1file[] = { "dstarmb.root", "dstarp1a.root", "dstarp1b.root", "dstarp2.root" }; -static TString geventsrc("http://root.cern/files/data"); -static Bool_t geventok = kTRUE; -static Bool_t geventlocal = kFALSE; -static Int_t geventnf = 10; -static Long64_t gEventNum = 200000; -static Long64_t gEventFst = 65000; -static Long64_t gEventSiz = 100000; -static TStopwatch gStopwatch; - -// The selectors -static TString gTutDir = "$ROOTSYS/tutorials"; // default path -static TList gSelectors; -static TString gH1Sel("/tree/h1analysis.C"); -static TString gEventSel("/proof/ProofEvent.C"); -static TString gEventProcSel("/proof/ProofEventProc.C"); -static TString gSimpleSel("/proof/ProofSimple.C"); -static TString gTestsSel("/proof/ProofTests.C"); -static TString gNtupleSel("/proof/ProofNtuple.C"); -static TString gFriendsSel("/proof/ProofFriends.C"); -static TString gAuxSel("/proof/ProofAux.C"); - -// Special class -static TString gProcFileElem("/proof/ProcFileElements.C"); - -// Special files -static TString gEmptyInclude("/proof/EmptyInclude.h"); -static TString gNtpRndm("/proof/ntprndm.root"); - -// Special packages -static TString gPackEvent("/proof/event.par"); -static TString gPack1("/proof/packtest1.par"); -static TString gPack2("/proof/packtest2.par"); -static TString gPack3("/proof/packtest3.par"); - -int stressProof(const char *url = 0, - const char *tests = 0, Int_t nwrks = -1, - const char *verbose = "1", const char *logfile = 0, - Bool_t dyn = kFALSE, Bool_t skipds = kTRUE, - const char *h1src = 0, const char *eventsrc = 0, - Bool_t dryrun = kFALSE, Bool_t showcpu = kFALSE, - Bool_t clearcache = kFALSE, Bool_t useprogress = kTRUE, - const char *tutdir = 0, Bool_t cleanlog = kFALSE, - Bool_t keeplog = kTRUE, Bool_t catlog = kFALSE); - -//_____________________________batch only_____________________ -#ifndef __CINT__ -int main(int argc,const char *argv[]) -{ - - // Request for help? - if (argc > 1 && !strcmp(argv[1],"-h")) { - printf(" \n"); - printf(" PROOF test suite\n"); - printf(" \n"); - printf(" Usage:\n"); - printf(" \n"); - printf(" $ ./stressProof [-h] [-n ] [-d [scope:]level] [-l logfile] [-dyn] [-ds]\n"); - printf(" [-t tests] [-h1 h1src] [-event src] [-dryrun] [-g] [-cpu]\n"); - printf(" [-clearcache] [-noprogress] [-tut tutdir] [master]\n"); - printf(" \n"); - printf(" Optional arguments:\n"); - printf(" -h prints this menu\n"); - printf(" master entry point of the cluster where to run the test\n"); - printf(" in the form '[user@]host.domain[:port]'; default 'localhost:40000'\n"); - printf(" or the env STRESSPROOF_URL\n"); - printf(" -n wrks number of workers to be started when running on the local host;\n"); - printf(" default is the nuber of local cores\n"); - printf(" -d [scope:]level\n"); - printf(" verbosity scope and level; default level is 1; scope refers to PROOF debug\n"); - printf(" options (see TProofDebug.h) and is enabled only is level > 1 (default kAll).\n"); - printf(" -l logfile file where to redirect the processing logs; must be writable;\n"); - printf(" default is a temporary file deleted at the end of the test\n"); - printf(" in case of success. Some format specifiers can be used in the file name:\n"); - printf(" - %%p is replaced by the current process ID; \n"); - printf(" - %%tmp is replaced by the temporary directory; \n"); - printf(" - %%d is replaced by date and time at test startup in the form YYYYMMDD-HHMM, \n"); - printf(" e.g. 20120904-1138. \n"); - printf(" The log file path can be also passed via the env STRESSPROOF_LOGFILE.\n"); - printf(" In case of failure, the log files of the nodes (master and workers) are saved into\n"); - printf(" a file called .nodes .\n"); - printf(" -c,-cleanlog delete the logfile specified via '-l' in case of a successful run; by default\n"); - printf(" the file specified by '-l' is kept in all cases (default log files are deleted\n"); - printf(" on success); adding this switch allows to keep a user-defined log file only\n"); - printf(" in case of error.\n"); - printf(" -k,-keeplog keep all logfiles, including the ones from the PROOF nodes (in one single file)\n"); - printf(" The paths are printed on the screen.\n"); - printf(" -catlog prints all the logfiles (also the ones from PROOF nodes) on stdout; useful for\n"); - printf(" presenting a single aggregated output for automatic tests. If specified in\n"); - printf(" conjunction with -cleanlog it will only print the logfiles in case of errors\n"); - printf(" -dyn run the test in dynamicStartup mode\n"); - printf(" -ds force the dataset test if skipped by default\n"); - printf(" -t tests run only tests in the comma-separated list and those from which they\n"); - printf(" depend; ranges can be given with 'first-last' syntax, e.g. '3,6-9,23'\n"); - printf(" to run tests 3, 6, 7, 8, 9 and 23. Can be also given via the env STRESSPROOF_TESTS.\n"); - printf(" -h1 h1src specify a location for the H1 files; use h1src=\"download\" to download\n"); - printf(" to a temporary location (or h1src=\"download=/path/for/local/h1\" to download\n"); - printf(" to /path/for/local/h1; by default the files are read directly from the\n"); - printf(" ROOT http server; however this may give failures if the connection is slow.\n"); - printf(" If h1src ends with '.zip' the program assumes that the 4 files are concatenated\n"); - printf(" in a single zip archive and are accessible with the standard archive syntax.\n"); - printf(" Can be also passed via the env STRESSPROOF_H1SRC.\n"); - printf(" -event src\n"); - printf(" specify a location for the 'event' files; use src=\"download\" to download\n"); - printf(" to a temporary location (or eventsrc=\"download=/path/for/local/event\" to download\n"); - printf(" to /path/for/local/event; by default the files are read directly from the\n"); - printf(" ROOT http server; however this may give failures if the connection is slow\n"); - printf(" Can be also passed via the env STRESSPROOF_EVENT.\n"); - printf(" -punzip use parallel unzipping for data-driven processing.\n"); - printf(" -dryrun only show which tests would be run.\n"); - printf(" -g enable graphics; default is to run in text mode.\n"); - printf(" -cpu show CPU times used by each successful test; used for calibration.\n"); - printf(" -clearcache clear memory cache associated with the files processed when local.\n"); - printf(" -noprogress do not show progress (escaped chars may confuse some wrapper applications)\n"); - printf(" -tut tutdir specify alternative location for the ROOT tutorials; allows to run multiple\n"); - printf(" concurrent instances of stressProof, for tests, for example.\n"); - printf(" Can be also passed via the env STRESSPROOF_TUTORIALDIR.\n"); - printf(" \n"); - gSystem->Exit(0); - } - - // Parse options - const char *url = 0; - Int_t nWrks = -1; - const char *verbose = "1"; - Bool_t enablegraphics = kFALSE; - Bool_t dryrun = kFALSE; - Bool_t showcpu = kFALSE; - Bool_t clearcache = kFALSE; - Bool_t useprogress = kTRUE; - Bool_t cleanlog = kFALSE; - Bool_t keeplog = kFALSE; - Bool_t catlog = kFALSE; - const char *logfile = 0; - const char *h1src = 0; - const char *eventsrc = 0; - const char *tests = 0; - const char *tutdir = 0; - Int_t i = 1; - while (i < argc) { - if (!strcmp(argv[i],"-h")) { - // Ignore if not first argument - i++; - } else if (!strcmp(argv[i],"-n")) { - if (i+1 == argc || argv[i+1][0] == '-') { - printf(" -n should be followed by the number of workers: ignoring \n"); - i++; - } else { - nWrks = atoi(argv[i+1]); - i += 2; - } - } else if (!strcmp(argv[i],"-d")) { - if (i+1 == argc || argv[i+1][0] == '-') { - printf(" -d should be followed by the debug '[scope:]level': ignoring \n"); - i++; - } else { - verbose = argv[i+1]; - i += 2; - } - } else if (!strcmp(argv[i],"-l")) { - if (i+1 == argc || argv[i+1][0] == '-') { - printf(" -l should be followed by a path: ignoring \n"); - i++; - } else { - logfile = argv[i+1]; - i += 2; - } - } else if (!strcmp(argv[i], "-c") || !strcmp(argv[i], "-cleanlog")) { - cleanlog = kTRUE; - i++; - } else if (!strcmp(argv[i], "-k") || !strcmp(argv[i], "-keeplog")) { - keeplog = kTRUE; - i++; - } else if (!strcmp(argv[i], "-catlog")) { - catlog = kTRUE; - i++; - } else if (!strncmp(argv[i],"-v",2)) { - // For backward compatibility - if (!strncmp(argv[i],"-vv",3)) verbose = "2"; - if (!strncmp(argv[i],"-vvv",4)) verbose = "3"; - i++; - } else if (!strncmp(argv[i],"-dyn",4)) { - gDynamicStartup = kTRUE; - i++; - } else if (!strncmp(argv[i],"-ds",3)) { - gSkipDataSetTest = kFALSE; - i++; - } else if (!strncmp(argv[i],"-punzip",7)) { - gUseParallelUnzip = kTRUE; - i++; - } else if (!strcmp(argv[i],"-t")) { - if (i+1 == argc || argv[i+1][0] == '-') { - printf(" -t should be followed by a string or a number: ignoring \n"); - i++; - } else { - tests = argv[i+1]; - i += 2; - } - } else if (!strcmp(argv[i],"-h1")) { - if (i+1 == argc || argv[i+1][0] == '-') { - printf(" -h1 should be followed by a path: ignoring \n"); - i++; - } else { - h1src = argv[i+1]; - i += 2; - } - } else if (!strcmp(argv[i],"-event")) { - if (i+1 == argc || argv[i+1][0] == '-') { - printf(" -event should be followed by a path: ignoring \n"); - i++; - } else { - eventsrc = argv[i+1]; - i += 2; - } - } else if (!strncmp(argv[i],"-g",2)) { - enablegraphics = kTRUE; - i++; - } else if (!strncmp(argv[i],"-dryrun",7)) { - dryrun = kTRUE; - i++; - } else if (!strncmp(argv[i],"-cpu",4)) { - showcpu = kTRUE; - i++; - } else if (!strncmp(argv[i],"-clearcache",11)) { - clearcache = kTRUE; - i++; - } else if (!strncmp(argv[i],"-noprogress",11)) { - useprogress = kFALSE; - i++; - } else if (!strcmp(argv[i],"-tut")) { - if (i+1 == argc || argv[i+1][0] == '-') { - printf(" -tut should be followed by a path: ignoring \n"); - i++; - } else { - tutdir = argv[i+1]; - i += 2; - } - } else { - url = argv[i]; - i++; - } - } - - // Enable graphics if required - if (enablegraphics) { - new TApplication("stressProof", 0, 0); - } else { - gROOT->SetBatch(kTRUE); - } - - int rc = stressProof(url, tests, nWrks, verbose, logfile, gDynamicStartup, gSkipDataSetTest, - h1src, eventsrc, dryrun, showcpu, clearcache, useprogress, tutdir, cleanlog, - keeplog, catlog); - - gSystem->Exit(rc); -} -#endif - -//////////////////////////////////////////////////////////////////////////////// -/// Print one '.' and count it - -Int_t PutPoint() -{ - printf("."); - return ++gpoints; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print some progress information - -void PrintStressProgress(Long64_t total, Long64_t processed, Float_t, Long64_t) -{ - gSystem->RedirectOutput(0, 0, &gRH); - - char pc[2] = { '.', ':'}; - static int lstpc = 1; - - int ns = 0; - if (processed < total) { - ns += 6; - fprintf(stderr, "%c %2.0f %%", pc[(lstpc++)%2], - (total ? ((100.0*processed)/total) : 100.0)); - } - while (ns--) fprintf(stderr, "\b"); - - gSystem->RedirectOutput(glogfile, "a", &gRH); -} -//////////////////////////////////////////////////////////////////////////////// -/// Dummy PrintProgress - -void PrintEmptyProgress(Long64_t, Long64_t, Float_t, Long64_t) -{ - return; -} - -// Guard class -class SwitchProgressGuard { -public: - SwitchProgressGuard(Bool_t force = kFALSE) { - if (guseprogress || force) { - gProof->SetPrintProgress(&PrintStressProgress); - } else { - gProof->SetPrintProgress(&PrintEmptyProgress); - } - } - ~SwitchProgressGuard() { gProof->SetPrintProgress(0); } -}; - -//////////////////////////////////////////////////////////////////////////////// -/// Remove all non source files associated with seletor at path 'selpath' - -void CleanupSelector(const char *selpath) -{ - if (!selpath) return; - - TString dirpath = gSystem->GetDirName(selpath); - if (gSystem->AccessPathName(dirpath)) return; - TString selname(gSystem->BaseName(selpath)); - selname.ReplaceAll(".C", "_C"); - void *dirp = gSystem->OpenDirectory(dirpath); - if (!dirp) return; - TString fn; - const char *e = 0; - while ((e = gSystem->GetDirEntry(dirp))) { - if (!strncmp(e, selname.Data(), selname.Length())) { - // Cleanup this entry - fn.Form("%s/%s", dirpath.Data(), e); - gSystem->Unlink(fn); - } - } - gSystem->FreeDirectory(dirp); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set the parallel unzip option - -void AssertParallelUnzip() -{ - if (gUseParallelUnzip) { - gProof->SetParameter("PROOF_UseParallelUnzip", (Int_t)1); - } else { - gProof->SetParameter("PROOF_UseParallelUnzip", (Int_t)0); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Release the memory cache associated with file 'fn'. - -void ReleaseCache(const char *fn) -{ -#if defined(R__LINUX) - TString filename(fn); - Int_t fd; - fd = open(filename.Data(), O_RDONLY); - if (fd > -1) { - fdatasync(fd); - posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED); - close(fd); - } else { - fprintf(stderr, "cannot open file '%s' for cache clean up; errno=%d \n", - filename.Data(), errno); - } -#else - fprintf(stderr, "ReleaseCache: dummy function: file '%s' untouched ...\n", fn); -#endif - // Done - return; -} - -// -// Auxilliary classes for testing -// -class RunTimes { -public: - Double_t fCpu; - Double_t fReal; - RunTimes(Double_t c = -1., Double_t r = -1.) : fCpu(c), fReal(r) { } - - void Set(Double_t c = -1., Double_t r = -1.) { if (c > -1.) fCpu = c; if (r > -1.) fReal = r; } - void Print(const char *tag = "") { printf("%s real: %f s, cpu: %f s\n", tag, fReal, fCpu); } -}; -RunTimes operator-(const RunTimes &rt1, const RunTimes &rt2) { - RunTimes rt(rt1.fCpu - rt2.fCpu, rt1.fReal - rt2.fReal); - return rt; -} -static RunTimes gProofTimesZero(0.,0.); - -typedef Int_t (*ProofTestFun_t)(void *, RunTimes &); -class ProofTest : public TNamed { -private: - Int_t fSeq; // Sequential number for the test - ProofTestFun_t fFun; // Function to be executed for the test - void *fArgs; // Arguments to be passed to the function - TString fDeps; // Test dependencies, e.g. "1,3" - TString fSels; // Selectors used, e.g. "h1analysis,ProofSimple" - Int_t fDepFrom; // Index for looping over deps - Int_t fSelFrom; // Index for looping over selectors - Bool_t fEnabled; // kTRUE if this test is enabled - Double_t fCpuTime; // CPU time used by the test - Double_t fRealTime; // Real time used by the test - Double_t fRefReal; // Ref Real time used for PROOF marks - Double_t fProofMarks; // PROOF marks - Bool_t fUseForMarks; // Use in the calculation of the average PROOF marks - - static Double_t gRefReal[PT_NUMTEST]; // Reference Cpu times - -public: - ProofTest(const char *n, Int_t seq, ProofTestFun_t f, void *a = 0, - const char *d = "", const char *sel = "", Bool_t useForMarks = kFALSE) - : TNamed(n,""), fSeq(seq), fFun(f), fArgs(a), - fDeps(d), fSels(sel), fDepFrom(0), fSelFrom(0), fEnabled(kTRUE), - fCpuTime(-1.), fRealTime(-1.), fProofMarks(-1.), fUseForMarks(useForMarks) - { fRefReal = gRefReal[fSeq-1]; } - virtual ~ProofTest() { } - - void Disable() { fEnabled = kFALSE; } - void Enable() { fEnabled = kTRUE; } - Bool_t IsEnabled() const { return fEnabled; } - - Int_t NextDep(Bool_t reset = kFALSE); - Int_t NextSel(TString &sel, Bool_t reset = kFALSE); - Int_t Num() const { return fSeq; } - - Int_t Run(Bool_t dryrun = kFALSE, Bool_t showcpu = kFALSE); - - Double_t ProofMarks() const { return fProofMarks; } - Bool_t UseForMarks() const { return fUseForMarks; } -}; - -// Reference time measured on a HP DL580 24 core (4 x Intel(R) Xeon(R) CPU X7460 -// @ 2.132 GHz, 48GB RAM, 1 Gb/s NIC) with 4 workers. -Double_t ProofTest::gRefReal[PT_NUMTEST] = { - 3.047808, // #1: Open a session - 0.021979, // #2: Get session logs - 4.779971, // #3: Simple random number generation - 0.276155, // #4: Dataset handling with H1 files - 5.355514, // #5: H1: chain processing - 2.414207, // #6: H1: file collection processing - 3.381990, // #7: H1: file collection, TPacketizerAdaptive - 3.227942, // #8: H1: by-name processing - 3.944204, // #9: H1: multi dataset processing - 9.146988, // #10: H1: multi dataset and entry list - 2.703881, // #11: Package management with 'event' - 3.814625, // #12: Package argument passing - 9.028315, // #13: Simple 'event' generation - 0.123514, // #14: Input data propagation - 0.129757, // #15: H1, Simple: async mode - 0.349625, // #16: Admin functionality - 0.989456, // #17: Dynamic sub-mergers functionality - 11.23798, // #18: Event range processing - 6.087582, // #19: Event range, TPacketizerAdaptive - 2.489555, // #20: File-resident output: merge - 0.180897, // #21: File-resident output: merge w/ submergers - 1.417233, // #22: File-resident output: create dataset - 0.000000, // #23: File-resident output: multi trees - 7.452465, // #24: TTree friends (and TPacketizerFile) - 0.259239, // #25: TTree friends, same file - 6.868858, // #26: Simple generation: merge-via-file - 6.362017, // #27: Simple random number generation by TSelector object - 5.519631, // #28: H1: by-object processing - 7.452465 // #29: Chain with TTree in subdirs -}; - -// -// Timer to stop asynchronous actions -// -class TTimeOutTimer : public TTimer { -public: - TTimeOutTimer(Long_t ms); - Bool_t Notify(); -}; - -TTimeOutTimer::TTimeOutTimer(Long_t ms) - : TTimer(ms, kTRUE) -{ - //constructor - gSystem->AddTimer(this); -} - -Bool_t TTimeOutTimer::Notify() -{ - //notifier - gTimedOut = kTRUE; - Remove(); // one shot only - return kTRUE; -} -//------------------------------------------------------------------------------ -//_____________________________________________________________________________ -Int_t ProofTest::NextDep(Bool_t reset) -{ - // Return index of next dependency or -1 if none (or no more) - // If reset is kTRUE, reset the internal counter before acting. - - if (reset) fDepFrom = 0; - - TString tkn; - if (fDeps.Tokenize(tkn, fDepFrom, ",")) { - if (tkn.IsDigit()) return tkn.Atoi(); - } - // Not found - return -1; -} -//////////////////////////////////////////////////////////////////////////////// -/// Return index of next dependency or -1 if none (or no more) -/// If reset is kTRUE, reset the internal counter before acting. - -Int_t ProofTest::NextSel(TString &sel, Bool_t reset) -{ - if (reset) fSelFrom = 0; - if (fSels.Tokenize(sel, fSelFrom, ",")) { - if (!sel.IsNull()) return 0; - } - // Not found - return -1; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Generic stress steering function; returns 0 on success, -1 on error - -Int_t ProofTest::Run(Bool_t dryrun, Bool_t showcpu) -{ - gpoints = 0; - printf(" Test %2d : %s ", fSeq, GetName()); - PutPoint(); - Int_t rc = 0; - if (!dryrun) { - gSystem->RedirectOutput(glogfile, "a", &gRH); - RunTimes tt; - gStopwatch.Start(); - rc = (*fFun)(fArgs, tt); - gStopwatch.Stop(); - fCpuTime = tt.fCpu; - fRealTime = tt.fReal; - // Proof marks - if (fRefReal > 0) - fProofMarks = (fRealTime > 0) ? 1000 * fRefReal / fRealTime : -1; - gSystem->RedirectOutput(0, 0, &gRH); - if (rc == 0) { - Int_t np = totpoints - strlen(GetName()) - strlen(" OK *"); - while (np--) { printf("."); } - if (showcpu) { - printf(" OK * (rt: %f s, cpu: %f s, marks: %.2f)\n", - fRealTime, fCpuTime, fProofMarks); - } else { - printf(" OK *\n"); - } - } else if (rc == 1) { - Int_t np = totpoints - strlen(GetName()) - strlen(" SKIPPED *"); - while (np--) { printf("."); } - printf(" SKIPPED *\n"); - } else { - Int_t np = totpoints - strlen(GetName()) - strlen(" FAILED *"); - while (np--) { printf("."); } - printf(" FAILED *\n"); - gSystem->ShowOutput(&gRH); - } - } else { - if (fEnabled) { - Int_t np = totpoints - strlen(GetName()) - strlen(" ENABLED *"); - while (np--) { printf("."); } - printf(" ENABLED *\n"); - } else { - Int_t np = totpoints - strlen(GetName()) - strlen(" DISABLED *"); - while (np--) { printf("."); } - printf(" DISABLED *\n"); - } - } - // Done - return rc; -} - -// Test functions -Int_t PT_Open(void *, RunTimes &); -Int_t PT_GetLogs(void *, RunTimes &); -Int_t PT_Simple(void *, RunTimes &); -Int_t PT_H1Http(void *, RunTimes &); -Int_t PT_H1FileCollection(void *, RunTimes &); -Int_t PT_H1DataSet(void *, RunTimes &); -Int_t PT_H1MultiDataSet(void *, RunTimes &); -Int_t PT_H1MultiDSetEntryList(void *, RunTimes &); -Int_t PT_DataSets(void *, RunTimes &); -Int_t PT_Packages(void *, RunTimes &); -Int_t PT_Event(void *, RunTimes &); -Int_t PT_InputData(void *, RunTimes &); -Int_t PT_H1SimpleAsync(void *arg, RunTimes &); -Int_t PT_AdminFunc(void *arg, RunTimes &); -Int_t PT_PackageArguments(void *, RunTimes &); -Int_t PT_EventRange(void *, RunTimes &); -Int_t PT_POFNtuple(void *, RunTimes &); -Int_t PT_POFDataset(void *, RunTimes &); -Int_t PT_Friends(void *, RunTimes &); -Int_t PT_TreeSubDirs(void *, RunTimes &); -Int_t PT_SimpleByObj(void *, RunTimes &); -Int_t PT_H1ChainByObj(void *, RunTimes &); -Int_t PT_AssertTutorialDir(const char *tutdir); -Int_t PT_MultiTrees(void *, RunTimes &); -Int_t PT_OutputHandlingViaFile(void *, RunTimes &); - -// Auxilliary functions -void PT_GetLastTimes(RunTimes &tt) -{ - // Get in tt the Cpu and Real times used during the run - // The runtimes - gStopwatch.Stop(); - tt.fCpu = gStopwatch.CpuTime(); - tt.fReal = gStopwatch.RealTime(); -} -void PT_GetLastProofTimes(RunTimes &tt) -{ - // Get in tt the Cpu and Real times used by PROOF since last call - if (gProof->IsLite()) { - PT_GetLastTimes(tt); - return; - } - // Update the statistics - gProof->GetStatistics(kFALSE); - // The runtimes - RunTimes proofTimesCurrent(gProof->GetCpuTime(), gProof->GetRealTime()); - tt = proofTimesCurrent - gProofTimesZero; - gProofTimesZero = proofTimesCurrent; -} - -// Arguments structures -typedef struct ptopenargs { // Open - const char *url; - Int_t nwrks; -} PT_Open_Args_t; - -// Packetizer parameters -typedef struct ptpacketizer { - const char *fName; - Int_t fType; -} PT_Packetizer_t; - -// Options -typedef struct ptoption { - Int_t fOne; - Int_t fTwo; -} PT_Option_t; - -static PT_Packetizer_t gStd_Old = { "TPacketizerAdaptive", 1 }; - -//////////////////////////////////////////////////////////////////////////////// - -int stressProof(const char *url, const char *tests, Int_t nwrks, - const char *verbose, const char *logfile, Bool_t dyn, Bool_t skipds, - const char *h1src, const char *eventsrc, - Bool_t dryrun, Bool_t showcpu, Bool_t clearcache, Bool_t useprogress, - const char *tutdir, Bool_t cleanlog, Bool_t keeplog, Bool_t catlog) -{ - printf("******************************************************************\n"); - printf("* Starting P R O O F - S T R E S S suite *\n"); - printf("******************************************************************\n"); - - // Use defaults or environment settings where required - if (!url) { - url = getenv("STRESSPROOF_URL"); - if (!url) url = urldef; - } - // Set dynamic mode - gDynamicStartup = (!strcmp(url,"lite://")) ? kFALSE : dyn; - - // Set verbosity - TString vv(verbose); - Ssiz_t icol = kNPOS; - if ((icol = vv.Index(":")) != kNPOS) { - TString sv = vv(icol+1, vv.Length() - icol -1); - gverbose = sv.Atoi(); - vv.Remove(icol); - gverbproof = vv; - } else { - gverbose = vv.Atoi(); - } - - // No progress bar if not tty or explicitly not requested (i.e. for ctest) - guseprogress = useprogress; - if (isatty(0) == 0 || isatty(1) == 0) guseprogress = kFALSE; - if (!guseprogress) { - if (!useprogress) { - printf("* Progress not shown (explicit request) *\n"); - } else { - printf("* Progress not shown (not tty) *\n"); - } - printf("******************************************************************\n"); - } - - // Notify/warn about the dynamic startup option, if any - TUrl uu(url), udef(urldef); - Bool_t extcluster = ((strcmp(uu.GetHost(), udef.GetHost()) || - (uu.GetPort() != udef.GetPort())) && strcmp(url,"lite://"))? kTRUE : kFALSE; - if (gDynamicStartup && gverbose > 0) { - // Check url - if (extcluster) { - printf("* WARNING: request to run a test with per-job scheduling on *\n"); - printf("* an external cluster: %s .\n", url); - printf("* Make sure the dynamic option is set. *\n"); - printf("* *\n"); - gDynamicStartup = kFALSE; - } else { - printf("* Runnning in dynamic mode (per-job scheduling) *\n"); - } - printf("* Tests #15, #22, #23, #26 skipped in dynamic mode **\n"); - printf("******************************************************************\n"); - } - - if (dryrun) { - printf("* Dry-run: only showing what would be done *\n"); - printf("******************************************************************\n"); - } - - // Cluster locality - gLocalCluster = (!extcluster || !strcmp(url, "lite://")) ? kFALSE : kTRUE; - - // Dataset option - if (!skipds) { - gSkipDataSetTest = kFALSE; - } else { - gSkipDataSetTest = gLocalCluster; - } - - // Clear cache - gClearCache = clearcache; - - // Log file path - Bool_t usedeflog = kTRUE; - FILE *flog = 0; - if (!logfile) logfile = getenv("STRESSPROOF_LOGFILE"); - if (logfile && strlen(logfile) > 0 && !dryrun) { - usedeflog = kFALSE; - glogfile = logfile; - if (glogfile.Contains("%tmp")) - glogfile.ReplaceAll("%tmp", gSystem->TempDirectory()); - if (glogfile.Contains("%p")) { - TString apid = TString::Format("%d", gSystem->GetPid()); - glogfile.ReplaceAll("%p", apid); - } - if (glogfile.Contains("%d")) { - TString d(TDatime().AsSQLString()); - d.Remove(d.Last(':')); - d.ReplaceAll("-", ""); - d.ReplaceAll(":", ""); - d.ReplaceAll(" ", "-"); - glogfile.ReplaceAll("%d", d); - } - if (!gSystem->AccessPathName(glogfile, kFileExists)) { - if (gSystem->AccessPathName(glogfile, kWritePermission)) { - printf(" >>> Cannot write to log file %s - ignore file request\n", logfile); - usedeflog = kTRUE; - } - } else { - // Create the file - if (!(flog = fopen(logfile, "w"))) { - printf(" >>> Cannot create log file %s - ignore file request\n", logfile); - usedeflog = kTRUE; - } - } - } - if (usedeflog && !dryrun) { - glogfile = "ProofStress_"; - if (!(flog = gSystem->TempFileName(glogfile, gSystem->TempDirectory()))) { - printf(" >>> Cannot create a temporary log file on %s - exit\n", gSystem->TempDirectory()); - return 1; - } - fclose(flog); - } - if (gverbose > 0) { - printf("* Log file: %s\n", glogfile.Data()); - if (cleanlog) - printf("* (NB: file will be removed if test is successful) *\n"); - printf("******************************************************************\n"); - } - - if (gSkipDataSetTest && gverbose > 0) { - printf("* Test for dataset handling (#4, #8-10) skipped **\n"); - printf("******************************************************************\n"); - } - if (gUseParallelUnzip && gverbose > 0) { - printf("* Using parallel unzip where relevant **\n"); - printf("******************************************************************\n"); - } - if (!strcmp(url,"lite://") && gverbose > 0) { - printf("* PROOF-Lite session (tests #15 and #16 skipped) **\n"); - printf("******************************************************************\n"); - } - if (!h1src) h1src = getenv("STRESSPROOF_H1SRC"); - if (h1src && strlen(h1src)) { - if (!strcmp(h1src, "download") && extcluster) { - if (gverbose > 0) { - printf("* External clusters: ignoring download request of H1 files\n"); - printf("******************************************************************\n"); - } - } else if (!gh1src.BeginsWith(h1src)) { - if (gverbose > 0) { - printf("* Taking 'h1' files from: %s\n", h1src); - printf("******************************************************************\n"); - } - gh1src = h1src; - gh1ok = kFALSE; - } - } - if (!eventsrc) eventsrc = getenv("STRESSPROOF_EVENT"); - if (eventsrc && strlen(eventsrc)) { - if (!strcmp(eventsrc, "download") && extcluster) { - if (gverbose > 0) { - printf("* External clusters: ignoring download request of 'event' files\n"); - printf("******************************************************************\n"); - } - } else if (!geventsrc.BeginsWith(eventsrc)) { - if (gverbose > 0) { - printf("* Taking 'event' files from: %s\n", eventsrc); - printf("******************************************************************\n"); - } - geventsrc = eventsrc; - geventok = kFALSE; - } - } - if (!tutdir) tutdir = getenv("STRESSPROOF_TUTORIALDIR"); - if (tutdir && strlen(tutdir)) { - if (!(gTutDir == tutdir)) { - if (gverbose > 0) { - printf("* Taking tutorial files from: %s\n", tutdir); - printf("******************************************************************\n"); - } - gTutDir = tutdir; - } - } - if (clearcache && gverbose > 0) { - printf("* Clearing cache associated to files, if possible ... **\n"); - printf("******************************************************************\n"); - } - if (keeplog && gverbose > 0) { - printf("* Keeping logfiles (paths specified at the end) **\n"); - printf("******************************************************************\n"); - } - // - // Reset dataset settings - gEnv->SetValue("Proof.DataSetManager",""); - - // - // Register tests - // - TList *testList = new TList; - // Simple open - PT_Open_Args_t PToa = { url, nwrks }; - testList->Add(new ProofTest("Open a session", 1, &PT_Open, (void *)&PToa)); - // Get logs - testList->Add(new ProofTest("Get session logs", 2, &PT_GetLogs, (void *)&PToa, "1")); - // Simple histogram generation - testList->Add(new ProofTest("Simple random number generation", 3, &PT_Simple, 0, "1", "ProofSimple", kTRUE)); - // Test of data set handling with the H1 http files - testList->Add(new ProofTest("Dataset handling with H1 files", 4, &PT_DataSets, 0, "1")); - // H1 analysis over HTTP (chain) - testList->Add(new ProofTest("H1: chain processing", 5, &PT_H1Http, 0, "1", "h1analysis", kTRUE)); - // H1 analysis over HTTP (file collection) - testList->Add(new ProofTest("H1: file collection processing", 6, &PT_H1FileCollection, 0, "1", "h1analysis", kTRUE)); - // H1 analysis over HTTP: adaptive packetizer - testList->Add(new ProofTest("H1: file collection, TPacketizerAdaptive", 7, - &PT_H1FileCollection, (void *)&gStd_Old, "1", "h1analysis", kTRUE)); - // H1 analysis over HTTP by dataset name - testList->Add(new ProofTest("H1: by-name processing", 8, &PT_H1DataSet, 0, "1,4", "h1analysis", kTRUE)); - // H1 analysis over HTTP by dataset name split in two - testList->Add(new ProofTest("H1: multi dataset processing", 9, &PT_H1MultiDataSet, 0, "1,4", "h1analysis", kTRUE)); - // H1 analysis over HTTP by dataset name - testList->Add(new ProofTest("H1: multi dataset and entry list", 10, &PT_H1MultiDSetEntryList, 0, "1,4", "h1analysis", kTRUE)); - // Test package management with 'event' - testList->Add(new ProofTest("Package management with 'event'", 11, &PT_Packages, 0, "1")); - // Test package argument passing - testList->Add(new ProofTest("Package argument passing", 12, &PT_PackageArguments, 0, "1", "ProofTests")); - // Simple event analysis - testList->Add(new ProofTest("Simple 'event' generation", 13, &PT_Event, 0, "1,11", "ProofEvent", kTRUE)); - // Test input data propagation (it only works in the static startup mode) - testList->Add(new ProofTest("Input data propagation", 14, &PT_InputData, 0, "1", "ProofTests")); - // Test asynchronous running - testList->Add(new ProofTest("H1, Simple: async mode", 15, &PT_H1SimpleAsync, 0, "1,3,5", "h1analysis,ProofSimple", kTRUE)); - // Test admin functionality - testList->Add(new ProofTest("Admin functionality", 16, &PT_AdminFunc, 0, "1")); - // Test merging via submergers - PT_Option_t pfoptm = {1, 0}; - testList->Add(new ProofTest("Dynamic sub-mergers functionality", 17, - &PT_Simple, (void *)&pfoptm, "1", "ProofSimple", kTRUE)); - // Test range chain and dataset processing EventProc - testList->Add(new ProofTest("Event range processing", 18, - &PT_EventRange, 0, "1,11", "ProofEventProc,ProcFileElements", kTRUE)); - // Test range chain and dataset processing EventProc with TPacketizerAdaptive - testList->Add(new ProofTest("Event range, TPacketizerAdaptive", 19, - &PT_EventRange, (void *)&gStd_Old, "1,11", "ProofEventProc,ProcFileElements", kTRUE)); - // Test TProofOutputFile technology for ntuple creation - testList->Add(new ProofTest("File-resident output: merge", 20, &PT_POFNtuple, 0, "1", "ProofNtuple", kTRUE)); - // Test TProofOutputFile technology for ntuple creation using submergers - testList->Add(new ProofTest("File-resident output: merge w/ submergers", 21, - &PT_POFNtuple, (void *)&pfoptm, "1", "ProofNtuple", kTRUE)); - // Test TProofOutputFile technology for dataset creation (tests TProofDraw too) - testList->Add(new ProofTest("File-resident output: create dataset", 22, &PT_POFDataset, 0, "1", "ProofNtuple", kTRUE)); - // Test selecting different TTrees in same files - testList->Add(new ProofTest("File-resident output: multi trees", 23, &PT_MultiTrees, 0, "1,22", "ProofNtuple", kTRUE)); - // Test TPacketizerFile and TTree friends in separate files - testList->Add(new ProofTest("TTree friends (and TPacketizerFile)", 24, &PT_Friends, 0, "1", "ProofFriends,ProofAux", kTRUE)); - // Test TPacketizerFile and TTree friends in same file - Bool_t sameFile = kTRUE; - testList->Add(new ProofTest("TTree friends, same file", 25, - &PT_Friends, (void *)&sameFile, "1", "ProofFriends,ProofAux", kTRUE)); - - // Test handling output via file - testList->Add(new ProofTest("Handling output via file", 26, - &PT_OutputHandlingViaFile, 0, "1", "ProofSimple", kTRUE)); - // Simple histogram generation by TSelector object - testList->Add(new ProofTest("Simple: selector by object", 27, &PT_SimpleByObj, 0, "1", "ProofSimple", kTRUE)); - // H1 analysis over HTTP by TSeletor object - testList->Add(new ProofTest("H1 chain: selector by object", 28, &PT_H1ChainByObj, 0, "1", "h1analysis", kTRUE)); - // Test TPacketizerFile and TTree friends in separate files - testList->Add(new ProofTest("Chain with TTree in subdirs", 29, &PT_TreeSubDirs, 0, "1", "ProofFriends,ProofAux", kTRUE)); - // The selectors - if (PT_AssertTutorialDir(gTutDir) != 0) { - printf("* Some of the tutorial files are missing! Stop\n"); - return 1; - } - gSelectors.Add(new TNamed("h1analysis", gH1Sel.Data())); - gSelectors.Add(new TNamed("ProofEvent", gEventSel.Data())); - gSelectors.Add(new TNamed("ProofEventProc", gEventProcSel.Data())); - gSelectors.Add(new TNamed("ProofSimple", gSimpleSel.Data())); - gSelectors.Add(new TNamed("ProofTests", gTestsSel.Data())); - gSelectors.Add(new TNamed("ProcFileElements", gProcFileElem.Data())); - gSelectors.Add(new TNamed("ProofNtuple", gNtupleSel.Data())); - gSelectors.Add(new TNamed("ProofFriends", gFriendsSel.Data())); - gSelectors.Add(new TNamed("ProofAux", gAuxSel.Data())); - if (gverbose > 0) { - if (!dryrun) { - printf("* Cleaning-up all non-source files associated to:\n"); - } else { - printf("* Would clean-up all non-source files associated to:\n"); - } - } - - // Check what to run - ProofTest *t = 0, *treq = 0; - TIter nxt(testList); - Bool_t all = kTRUE; - if (!tests) tests = getenv("STRESSPROOF_TESTS"); - if (tests && strlen(tests)) { - TString tts(tests), tsg, ts, ten; - Ssiz_t from = 0; - while (tts.Tokenize(tsg, from, ",")) { - if (tsg.CountChar('-') > 1) { - printf("* **\r"); - printf("* Wrong syntax for test range specification: %s\n", tsg.Data()); - continue; - } - Ssiz_t fromg = 0; - Int_t test = -1, last = -1; - while (tsg.Tokenize(ts, fromg, "-")) { - if (!ts.IsDigit()) { - printf("* **\r"); - printf("* Test string is not a digit: %s\n", ts.Data()); - continue; - } - if (test < 0) { - test = ts.Atoi(); - } else { - last = ts.Atoi(); - } - } - if (test <= 0) { - printf("* **\r"); - printf("* Non-positive test number: %d\n", test); - continue; - } - const int tmx = PT_NUMTEST; - if (test > tmx) { - printf("* **\r"); - printf("* Unknown test number: %d\n", test); - continue; - } - if (last > tmx) { - printf("* **\r"); - printf("* Upper test range too large: %d - rescaling\n", last); - last = tmx; - } else if (last <= 0) { - last = test; - } - // For final notification - if (ten != "") ten += ","; - ten += tsg; - // Ok, now we can enable - if (all) { - all = kFALSE; - // Disable first all the tests - while ((t = (ProofTest *)nxt())) { t->Disable(); } - } - // Process one by one all enabled tests - TString cleaned; - for (Int_t xt = test; xt <= last; xt++) { - // Locate the ProofTest instance - nxt.Reset(); - while ((t = (ProofTest *)nxt())) { - if (t->Num() == xt) treq = t; - } - if (!treq) { - printf("* **\r"); - printf("* Test %2d not found among the registered tests - exiting\n", xt); - printf("******************************************************************\n"); - return 1; - } - // Enable the required tests - Int_t tn = -1; - while ((tn = treq->NextDep()) > 0) { - nxt.Reset(); - while ((t = (ProofTest *)nxt())) { - if (t->Num() == tn) { - t->Enable(); - break; - } - } - } - // Reset associated selectors - TString sel, tit; - while ((treq->NextSel(sel)) == 0) { - TNamed *nm = (TNamed *) gSelectors.FindObject(sel.Data()); - if (nm && cleaned.Index(nm->GetTitle()) == kNPOS) { - if (!dryrun) CleanupSelector(nm->GetTitle()); - if (gverbose > 0) { - tit = nm->GetName(); tit.Resize(18); - printf("* %s in %s\n", tit.Data(), gSystem->GetDirName(nm->GetTitle()).Data()); - } - cleaned += TString::Format(":%s:", nm->GetTitle()); - } - } - // Enable the required test - treq->Enable(); - } - } - if (!all && !dryrun) { - // Notify the enabled tests - printf("* **\r"); - printf("* Running only test(s) %s (and related)\n", ten.Data()); - printf("******************************************************************\n"); - } - } - if (all) { - // Clean all the selectors - TString tit; - TIter nxs(&gSelectors); - TNamed *nm = 0; - while ((nm = (TNamed *)nxs())) { - if (!dryrun) CleanupSelector(nm->GetTitle()); - if (gverbose > 0) { - tit = nm->GetName(); tit.Resize(18); - printf("* %s in %s\n", tit.Data(), gSystem->GetDirName(nm->GetTitle()).Data()); - } - } - } - if (gverbose > 0) - printf("******************************************************************\n"); - - // If a dry-run show what we would do and exit - if (dryrun) { - nxt.Reset(); - while ((t = (ProofTest *)nxt())) { t->Run(kTRUE); } - printf("******************************************************************\n"); - return 0; - } - - // Add the ACLiC option to the selector strings - gH1Sel += "+"; - gEventSel += "+"; - gEventProcSel += "+"; - gSimpleSel += "+"; - gTestsSel += "+"; - gProcFileElem += "+"; - gNtupleSel += "+"; - gFriendsSel += "+"; - gAuxSel += "+"; - - // - // Run the tests - // - Bool_t failed = kFALSE; - nxt.Reset(); - while ((t = (ProofTest *)nxt())) - if (t->IsEnabled()) { - if (t->Run(kFALSE, showcpu) < 0) { - failed = kTRUE; - break; - } - } - - // Done - Bool_t kept = ((usedeflog || cleanlog) && !keeplog) ? kFALSE : kTRUE; - if (failed) { - kept = kTRUE; - if (usedeflog && !gROOT->IsBatch() && !keeplog) { - const char *answer = Getline(" Some tests failed: would you like to keep the log file (N,Y)? [Y] "); - if (answer && (answer[0] == 'N' || answer[0] == 'n')) { - // Remove log file - gSystem->Unlink(glogfile); - kept = kFALSE; - } - } - } else { - printf("* All registered tests have been passed :-) *\n"); - } - - if (kept) { - TString logfiles(glogfile); - // Save also the logs from the workers - TProofMgr *mgr = gProof ? gProof->GetManager() : 0; - if (mgr) { - gSystem->RedirectOutput(glogfile, "a", &gRH); - TProofLog *pl = mgr->GetSessionLogs(); - if (pl) { - logfiles += ".nodes"; - pl->Retrieve("*", TProofLog::kAll, logfiles); - gSystem->RedirectOutput(0, 0, &gRH); - SafeDelete(pl); - } else { - gSystem->RedirectOutput(0, 0, &gRH); - printf("+++ Warning: could not get the session logs\n"); - } - } else { - printf("+++ Warning: could not attach to manager to get the session logs\n"); - } - printf("******************************************************************\n"); - printf(" Main log file kept at %s (Proof logs in %s)\n", glogfile.Data(), logfiles.Data()); - if (catlog) { - - // Display all logfiles directly on this terminal. Useful for getting - // test results without accessing the test machine (i.e. with CDash) - const size_t readbuf_size = 500; - char readbuf[readbuf_size]; - - printf("******************************************************************\n"); - printf("Content of the main log file: %s\n", glogfile.Data()); - printf("******************************************************************\n"); - std::ifstream glogfile_is( glogfile.Data() ); - if (!glogfile_is) { - printf("Cannot open %s", glogfile.Data()); - } - else { - while ( glogfile_is.good() ) { - glogfile_is.getline(readbuf, readbuf_size); - std::cout << readbuf << std::endl; - } - glogfile_is.close(); - } - - printf("******************************************************************\n"); - printf("Content of the PROOF servers log files: %s\n", logfiles.Data()); - printf("******************************************************************\n"); - std::ifstream logfiles_is( logfiles.Data() ); - if (!logfiles_is) { - printf("Cannot open %s", logfiles.Data()); - } - else { - while ( logfiles_is.good() ) { - logfiles_is.getline(readbuf, readbuf_size); - std::cout << readbuf << std::endl; - } - logfiles_is.close(); - } - } - } else { - // Remove log file if not passed by the user - gSystem->Unlink(glogfile); - } - - printf("******************************************************************\n"); - if (gProof) { - // Get average of single PROOF marks - int navg = 0; - double avgmarks = -1.; - nxt.Reset(); - if (gverbose > 1) printf(" PROOFMARKS for single tests (-1 if unmeasured): \n"); - while ((t = (ProofTest *)nxt())) - if (t->IsEnabled()) { - if (gverbose > 1) printf(" %d:\t %.2f \n", t->Num(), t->ProofMarks()); - if (t->UseForMarks() && t->ProofMarks() > 0) { - navg++; - avgmarks += t->ProofMarks(); - } - } - if (navg > 0) avgmarks /= navg; - - gProof->GetStatistics((gverbose > 0)); - // Reference time measured on a HP DL580 24 core (4 x Intel(R) Xeon(R) CPU X7460 - // @ 2.132 GHz, 48GB RAM, 1 Gb/s NIC) with 4 workers. - const double reftime = 70.169; - double glbmarks = (gProof->GetRealTime() > 0) ? 1000 * reftime / gProof->GetRealTime() : -1; - printf(" ROOTMARKS = %.2f (overall: %.2f) ROOT version: %s\t%s@%s\n", - avgmarks, glbmarks, gROOT->GetVersion(), - gROOT->GetGitBranch(), gROOT->GetGitCommit()); - // Average from the single tests - printf("******************************************************************\n"); - } - - // If not PROOF-Lite, stop the daemon used for the test - if (gProof && !gProof->IsLite() && !extcluster && gROOT->IsBatch()) { - // Get the manager - TProofMgr *mgr = gProof->GetManager(); - // Close the instance - gProof->Close("S"); - delete gProof; - // Delete the manager - SafeDelete(mgr); - // The daemon runs on a port shifted by 1 - if (killXrootdAt(uu.GetPort()+1, "xpdtut") != 0) { - printf("+++ Warning: test daemon probably still running!\n"); - } - } - - // Done - return (failed ? 1 : 0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Release memory cache associated with the H1 files at 'h1src', if it -/// makes any sense, i.e. are local ... - -Int_t PT_H1ReleaseCache(const char *h1src) -{ - if (!h1src || strlen(h1src) <= 0) { - printf("\n >>> Test failure: src dir undefined\n"); - return -1; - } - - // If non-local, nothing to do - if (!gh1local) return 0; - - TString src; - if (gh1sep == '/') { - // Loop through the files - for (Int_t i = 0; i < 4; i++) { - src.Form("%s/%s", h1src, gh1file[i]); - ReleaseCache(src.Data()); - } - } else { - // Release the zip file ... - ReleaseCache(h1src); - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that the needed H1 files are available at 'src' -/// If 'src' is "download", the files are download under /h1 - -Int_t PT_H1AssertFiles(const char *h1src) -{ - if (!h1src || strlen(h1src) <= 0) { - printf("\n >>> Test failure: src dir undefined\n"); - return -1; - } - - // Locality - TUrl u(h1src, kTRUE); - gh1local = (!strcmp(u.GetProtocol(), "file")) ? kTRUE : kFALSE; - - gh1sep = '/'; - // Special cases - if (!strncmp(h1src,"download",8)) { - if (strcmp(h1src,"download")) { - gh1src = h1src; - gh1src.ReplaceAll("download=", ""); - } - if (gh1src.IsNull() || gSystem->AccessPathName(gh1src, kWritePermission)) - gh1src = TString::Format("%s/h1", gtutdir.Data()); - if (gSystem->AccessPathName(gh1src)) { - if (gSystem->MakeDirectory(gh1src) != 0) { - printf("\n >>> Test failure: could not create dir %s\n", gh1src.Data()); - return -1; - } - } - // Copy the files now - Int_t i = 0; - for (i = 0; i < 4; i++) { - TString src = TString::Format("http://root.cern/files/h1/%s", gh1file[i]); - TString dst = TString::Format("%s/%s", gh1src.Data(), gh1file[i]); - if (!TFile::Cp(src, dst)) { - printf("\n >>> Test failure: problems retrieving %s\n", src.Data()); - return -1; - } - gSystem->RedirectOutput(0, 0, &gRH); - printf("%d\b", i); - gSystem->RedirectOutput(glogfile, "a", &gRH); - } - gh1local = kTRUE; - // Done - gh1ok = kTRUE; - return 0; - } else if (TString(h1src).EndsWith(".zip")) { - // The files are in a zip archive ... - if (gSystem->AccessPathName(h1src)) { - printf("\n >>> Test failure: file %s does not exist\n", h1src); - return -1; - } - Int_t i = 0; - for (i = 0; i < 4; i++) { - TString src = TString::Format("%s#%s", h1src, gh1file[i]); - TFile *f = TFile::Open(src); - if (!f || (f && f->IsZombie())) { - printf("\n >>> Test failure: file %s not found in archive %s\n", src.Data(), h1src); - return -1; - } - if (guseprogress) { - gSystem->RedirectOutput(0, 0, &gRH); - printf("%d\b", i); - gSystem->RedirectOutput(glogfile, "a", &gRH); - } - } - gh1sep = '#'; - } else { - - // Make sure the files exist at 'src' - Int_t i = 0; - for (i = 0; i < 4; i++) { - TString src = TString::Format("%s/%s", h1src, gh1file[i]); - if (gSystem->AccessPathName(src)) { - printf("\n >>> Test failure: file %s does not exist\n", src.Data()); - return -1; - } - if (guseprogress) { - gSystem->RedirectOutput(0, 0, &gRH); - printf("%d\b", i); - gSystem->RedirectOutput(glogfile, "a", &gRH); - } - } - } - gh1src = h1src; - - // Done - gh1ok = kTRUE; - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Release memory cache associated with the event files at 'eventsrc', if it -/// makes any sense, i.e. are local ... - -Int_t PT_EventReleaseCache(const char *eventsrc, Int_t nf = 10) -{ - if (!eventsrc || strlen(eventsrc) <= 0) { - printf("\n >>> Test failure: src dir undefined\n"); - return -1; - } - - if (nf > 50) { - printf("\n >>> Test failure: max 50 event files can be checked\n"); - return -1; - } - - // If non-local, nothing to do - if (!geventlocal) return 0; - - TString src; - // Loop through the files - for (Int_t i = 0; i < nf; i++) { - src.Form("%s/event_%d.root", eventsrc, i+1); - ReleaseCache(src.Data()); - } - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that the needed 'event' files are available at 'src' -/// If 'src' is "download", the files are download under /event . -/// By default 10 files are checked; maximum is 50 (idx 1->50 not 0->49). - -Int_t PT_EventAssertFiles(const char *eventsrc, Int_t nf = 10) -{ - if (!eventsrc || strlen(eventsrc) <= 0) { - printf("\n >>> Test failure: src dir undefined\n"); - return -1; - } - - if (nf > 50) { - printf("\n >>> Test failure: max 50 event files can be checked\n"); - return -1; - } - - // Locality - TUrl u(eventsrc, kTRUE); - geventlocal = (!strcmp(u.GetProtocol(), "file")) ? kTRUE : kFALSE; - - // Special case - if (!strncmp(eventsrc,"download",8)) { - if (strcmp(eventsrc,"download")) { - geventsrc = eventsrc; - geventsrc.ReplaceAll("download=", ""); - } - if (geventsrc.IsNull() || gSystem->AccessPathName(geventsrc, kWritePermission)) - geventsrc = TString::Format("%s/event", gtutdir.Data()); - if (gSystem->AccessPathName(geventsrc)) { - if (gSystem->MakeDirectory(geventsrc) != 0) { - printf("\n >>> Test failure: could not create dir %s\n", geventsrc.Data()); - return -1; - } - } - // Copy the files now - Int_t i = 0; - for (i = 0; i < nf; i++) { - TString src = TString::Format("http://root.cern/files/data/event_%d.root", i+1); - TString dst = TString::Format("%s/event_%d.root", geventsrc.Data(), i+1); - if (!TFile::Cp(src, dst)) { - printf("\n >>> Test failure: problems retrieving %s\n", src.Data()); - return -1; - } - if (guseprogress) { - gSystem->RedirectOutput(0, 0, &gRH); - printf("%d\b", i); - gSystem->RedirectOutput(glogfile, "a", &gRH); - } - } - geventlocal = kTRUE; - // Done - geventok = kTRUE; - return 0; - } - - // Make sure the files exist at 'src' - Int_t i = 0; - for (i = 0; i < nf; i++) { - TString src = TString::Format("%s/event_%d.root", eventsrc, i+1); - if (gSystem->AccessPathName(src)) { - printf("\n >>> Test failure: file %s does not exist\n", src.Data()); - return -1; - } - if (guseprogress) { - gSystem->RedirectOutput(0, 0, &gRH); - printf("%d\b", i); - gSystem->RedirectOutput(glogfile, "a", &gRH); - } - } - geventsrc = eventsrc; - - // Done - geventok = kTRUE; - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Make sure that the needed files are available under the specified -/// tutorial directory, setting the relevant variables - -Int_t PT_AssertTutorialDir(const char *tutdir) -{ - if (!tutdir || strlen(tutdir) <= 0) { - printf("\n >>> Test failure: dir undefined\n"); - return -1; - } - - TString path; - // Selectors - gH1Sel.Insert(0, tutdir); - gSystem->ExpandPathName(gH1Sel); - if (gSystem->AccessPathName(gH1Sel)) return -1; - // - gEventSel.Insert(0, tutdir); - gSystem->ExpandPathName(gEventSel); - if (gSystem->AccessPathName(gEventSel)) return -1; - // - gEventProcSel.Insert(0, tutdir); - gSystem->ExpandPathName(gEventProcSel); - if (gSystem->AccessPathName(gEventProcSel)) return -1; - // - gSimpleSel.Insert(0, tutdir); - gSystem->ExpandPathName(gSimpleSel); - if (gSystem->AccessPathName(gSimpleSel)) return -1; - // - gTestsSel.Insert(0, tutdir); - gSystem->ExpandPathName(gTestsSel); - if (gSystem->AccessPathName(gTestsSel)) return -1; - // - gNtupleSel.Insert(0, tutdir); - gSystem->ExpandPathName(gNtupleSel); - if (gSystem->AccessPathName(gNtupleSel)) return -1; - // - gFriendsSel.Insert(0, tutdir); - gSystem->ExpandPathName(gFriendsSel); - if (gSystem->AccessPathName(gFriendsSel)) return -1; - // - gAuxSel.Insert(0, tutdir); - gSystem->ExpandPathName(gAuxSel); - if (gSystem->AccessPathName(gAuxSel)) return -1; - - // Special class - gProcFileElem.Insert(0, tutdir); - gSystem->ExpandPathName(gProcFileElem); - if (gSystem->AccessPathName(gProcFileElem)) return -1; - - // Special files - gEmptyInclude.Insert(0, tutdir); - gSystem->ExpandPathName(gEmptyInclude); - if (gSystem->AccessPathName(gEmptyInclude)) return -1; - // - gNtpRndm.Insert(0, tutdir); - gSystem->ExpandPathName(gNtpRndm); - if (gSystem->AccessPathName(gNtpRndm)) return -1; - - // Special packages - gPackEvent.Insert(0, tutdir); - gSystem->ExpandPathName(gPackEvent); - if (gSystem->AccessPathName(gPackEvent)) return -1; - // - gPack1.Insert(0, tutdir); - gSystem->ExpandPathName(gPack1); - if (gSystem->AccessPathName(gPack1)) return -1; - // - gPack2.Insert(0, tutdir); - gSystem->ExpandPathName(gPack2); - if (gSystem->AccessPathName(gPack2)) return -1; - // - gPack3.Insert(0, tutdir); - gSystem->ExpandPathName(gPack3); - if (gSystem->AccessPathName(gPack3)) return -1; - - // Done - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check the result of the ProofSimple analysis - -Int_t PT_CheckSimple(TQueryResult *qr, Long64_t nevt, Int_t nhist) -{ - if (!qr) { - printf("\n >>> Test failure: query result not found\n"); - return -1; - } - - // Make sure the number of processed entries is the one expected - PutPoint(); - if (qr->GetEntries() != nevt) { - printf("\n >>> Test failure: wrong number of entries processed: %lld (expected %lld)\n", - qr->GetEntries(), nevt); - return -1; - } - - // Make sure the output list is there - PutPoint(); - TList *out = qr->GetOutputList(); - if (!out) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - // Get the histos - PutPoint(); - TH1F **hist = new TH1F*[nhist]; - for (Int_t i=0; i < nhist; i++) { - hist[i] = dynamic_cast(TProof::GetOutput(Form("h%d",i), out)); - if (!hist[i]) { - printf("\n >>> Test failure: 'h%d' histo not found\n", i); - delete[] hist; - return -1; - } - } - - // Check the mean values - PutPoint(); - for (Int_t i=0; i < nhist; i++) { - Double_t ave = hist[i]->GetMean(); - Double_t rms = hist[i]->GetRMS(); - if (TMath::Abs(ave) > 5 * rms / TMath::Sqrt(hist[i]->GetEntries())) { - printf("\n >>> Test failure: 'h%d' histo: mean > 5 * RMS/Sqrt(N)\n", i); - delete[] hist; - return -1; - } - } - - // Clean up - delete[] hist; - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check the ntuple created by the ProofSimple analysis - -Int_t PT_CheckSimpleNtuple(TQueryResult *qr, Long64_t nevt, const char *dsname) -{ - if (!qr) { - printf("\n >>> Test failure: query result not found\n"); - return -1; - } - - // Make sure the output list is there - PutPoint(); - TList *out = qr->GetOutputList(); - if (!out) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - // Get the file collection - PutPoint(); - TFileCollection *fc = dynamic_cast(out->FindObject(dsname)); - if (!fc) { - printf("\n >>> Test failure: TFileCollection for dataset '%s' not" - " found in the output list\n", dsname); - return -1; - } - - // Check the default tree name - const char *tname = "/ntuple"; - PutPoint(); - if (!fc->GetDefaultTreeName() || strcmp(fc->GetDefaultTreeName(), tname)) { - printf("\n >>> Test failure: default tree name does not match (%s != %s)\n", - fc->GetDefaultTreeName(), tname); - return -1; - } - - // Check the number of entries - PutPoint(); - if (fc->GetTotalEntries(tname) != nevt) { - printf("\n >>> Test failure: number of entries does not match (%lld != %lld)\n", - fc->GetTotalEntries(tname), nevt); - return -1; - } - - // Check 'pz' histo - TH1F *hpx = new TH1F("PT_px", "PT_px", 20, -5., 5.); - PutPoint(); - gProof->DrawSelect(dsname, "px >> PT_px"); - if (TMath::Abs(hpx->GetMean()) > 5 * hpx->GetRMS() / TMath::Sqrt(hpx->GetEntries())) { - printf("\n >>> Test failure: 'hpx' histo: mean > 5 * RMS/Sqrt(N) (%f,%f)\n", - hpx->GetMean(), hpx->GetRMS()); - return -1; - } - - // Check 'pz' histo - TH1F *hpz = new TH1F("PT_pz", "PT_pz", 20, 0., 20.); - PutPoint(); - gProof->DrawSelect(dsname, "pz >> PT_pz"); - if (TMath::Abs(hpz->GetMean() - 2.) > 5 * 2. / TMath::Sqrt(hpz->GetEntries())) { - printf("\n >>> Test failure: 'hpz' histo: (mean - 2) > 5 * RMS/Sqrt(N) (%f,%f)\n", - hpz->GetMean(), hpz->GetRMS()); - return -1; - } - - // Check 'random' histo - TH1F *hpr = new TH1F("PT_rndm", "PT_rndm", 20, 0., 20.); - PutPoint(); - gProof->DrawSelect(dsname, "random >> PT_rndm"); - if (TMath::Abs(hpr->GetMean() - .5) > 5 * .5 / TMath::Sqrt(hpr->GetEntries())) { - printf("\n >>> Test failure: 'hpr' histo: (mean - .5) > 5 * RMS/Sqrt(N) (%f,%f)\n", - hpr->GetMean(), hpr->GetRMS()); - return -1; - } - - SafeDelete(hpx); - SafeDelete(hpz); - SafeDelete(hpr); - - // Clear dsname - gProof->ClearData(TProof::kDataset |TProof::kForceClear, dsname); - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check the result of the H1 analysis - -Int_t PT_CheckH1(TQueryResult *qr, Int_t irun = 0) -{ - if (!qr) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - // Make sure the number of processed entries is the one expected - PutPoint(); - Long64_t runEntries[2] = {283813, 7525}; - if (qr->GetEntries() != runEntries[irun]) { - printf("\n >>> Test failure: wrong number of entries processed: %lld (expected %lld)\n", qr->GetEntries(), runEntries[irun]); - return -1; - } - - // Make sure the output list is there - PutPoint(); - TList *out = qr->GetOutputList(); - if (!out) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - // Check the 'hdmd' histo - PutPoint(); - TH1F *hdmd = dynamic_cast(out->FindObject("hdmd")); - if (!hdmd) { - printf("\n >>> Test failure: 'hdmd' histo not found\n"); - return -1; - } - if ((Int_t)(hdmd->GetEntries()) != 7525) { - printf("\n >>> Test failure: 'hdmd' histo: wrong number" - " of entries (%d: expected 7525) \n",(Int_t)(hdmd->GetEntries())); - return -1; - } - if (TMath::Abs((hdmd->GetMean() - 0.15512023) / 0.15512023) > 0.001) { - printf("\n >>> Test failure: 'hdmd' histo: wrong mean" - " (%f: expected 0.15512023) \n", hdmd->GetMean()); - return -1; - } - - PutPoint(); - TH2F *h2 = dynamic_cast(out->FindObject("h2")); - if (!h2) { - printf("\n >>> Test failure: 'h2' histo not found\n"); - return -1; - } - if ((Int_t)(h2->GetEntries()) != 7525) { - printf("\n >>> Test failure: 'h2' histo: wrong number" - " of entries (%d: expected 7525) \n",(Int_t)(h2->GetEntries())); - return -1; - } - if (TMath::Abs((h2->GetMean() - 0.15245688) / 0.15245688) > 0.001) { - printf("\n >>> Test failure: 'h2' histo: wrong mean" - " (%f: expected 0.15245688) \n", h2->GetMean()); - return -1; - } - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check the result of the EventProc analysis - -Int_t PT_CheckEvent(TQueryResult *qr, const char *pack = "TPacketizer") -{ - if (!qr) { - printf("\n >>> Test failure: %s: output list not found\n", pack); - return -1; - } - - // Make sure the output list is there - PutPoint(); - TList *out = qr->GetOutputList(); - if (!out) { - printf("\n >>> Test failure: %s: output list not found\n", pack); - return -1; - } - - // Check the 'hdmd' histo - PutPoint(); - TNamed *nout = dynamic_cast(out->FindObject("Range_Check")); - if (!nout) { - printf("\n >>> Test failure: %s: 'Range_Check' named object not found\n", pack); - return -1; - } - if (strcmp(nout->GetTitle(), "OK")) { - printf("\n >>> Test failure: %s: 'Range_Check': wrong result: %s \n", pack, nout->GetTitle()); - return -1; - } - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check the result of the ProofNtuple analysis - -Int_t PT_CheckNtuple(TQueryResult *qr, Long64_t nevt) -{ - if (!qr) { - printf("\n >>> Test failure: query result not found\n"); - return -1; - } - - // Make sure the number of processed entries is the one expected - PutPoint(); - if (qr->GetEntries() != nevt) { - printf("\n >>> Test failure: wrong number of entries processed: %lld (expected %lld)\n", - qr->GetEntries(), nevt); - return -1; - } - - // Make sure the output list is there - PutPoint(); - TList *out = qr->GetOutputList(); - if (!out) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - // Get the ntuple form the file - TProofOutputFile *pof = dynamic_cast(out->FindObject("SimpleNtuple.root")); - if (!pof) { - printf("\n >>> Test failure: TProofOutputFile not found in the output list\n"); - return -1; - } - - // Get the file full path - TString outputFile(pof->GetOutputFileName()); - TString outputName(pof->GetName()); - outputName += ".root"; - - // Read the ntuple from the file - TFile *f = TFile::Open(outputFile); - if (!f || (f && f->IsZombie())) { - printf("\n >>> Test failure: could not open file: %s", outputFile.Data()); - return -1; - } - - // Get the ntuple - PutPoint(); - TNtuple *ntp = dynamic_cast(f->Get("ntuple")); - if (!ntp) { - printf("\n >>> Test failure: 'ntuple' not found\n"); - return -1; - } - - // Check the ntuple content by filling some histos - TH1F *h1s[3] = {0}; - h1s[0] = new TH1F("h1_1", "3*px+2 with px**2+py**2>1", 50, -15., 15.); - h1s[1] = new TH1F("h1_2", "2*px+2 with pz>2", 50, -10., 10.); - h1s[2] = new TH1F("h1_3", "1.3*px+2 with (px^2+py^2>4) && py>0", 50, 0., 8.); - Float_t px, py, pz; - Float_t *pp = ntp->GetArgs(); - Long64_t ent = 0; - while (ent < ntp->GetEntries()) { - ntp->GetEntry(ent); - px = pp[0]; - py = pp[1]; - pz = pp[2]; - // Fill the histos - if (px*px+py*py > 1.) h1s[0]->Fill(3.*px + 2.); - if (pz > 2.) h1s[1]->Fill(2.*px + 2.); - if (px*px+py*py > 4. && py > 0.) h1s[2]->Fill(1.3*px + 2.); - // Go next - ent++; - } - - Int_t rch1s = 0; - TString emsg; - // Check the histogram entries and mean values - Int_t hent[3] = { 620, 383, 72}; - Double_t hmea[3] = { 1.992, 2.062 , 3.126}; - for (Int_t i = 0; i < 3; i++) { - if ((Int_t)(h1s[i]->GetEntries()) != hent[i]) { - emsg.Form("'%s' histo: wrong number of entries (%d: expected %d)", - h1s[i]->GetName(), (Int_t)(h1s[i]->GetEntries()), hent[i]); - rch1s = -1; - break; - } - if (TMath::Abs((h1s[i]->GetMean() - hmea[i]) / hmea[i]) > 0.001) { - emsg.Form("'%s' histo: wrong mean (%f: expected %f)", - h1s[i]->GetName(), h1s[i]->GetMean(), hmea[i]); - rch1s = -1; - break; - } - } - - // Cleanup - for (Int_t i = 0; i < 3; i++) delete h1s[i]; - f->Close(); - delete f; - - // Check the result - if (rch1s != 0) { - printf("\n >>> Test failure: %s\n", emsg.Data()); - return -1; - } - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check the result of the ProofNtuple analysis creating a dataset -/// Uses and check also TProofDraw - -Int_t PT_CheckDataset(TQueryResult *qr, Long64_t nevt) -{ - if (!qr) { - printf("\n >>> Test failure: query result not found\n"); - return -1; - } - - // Make sure the number of processed entries is the one expected - PutPoint(); - if (qr->GetEntries() != nevt) { - printf("\n >>> Test failure: wrong number of entries processed: %lld (expected %lld)\n", - qr->GetEntries(), nevt); - return -1; - } - - const char *dsname = "testNtuple"; - // Make sure that the dataset exists - PutPoint(); - if (!gProof->ExistsDataSet(dsname)) { - gProof->ShowDataSets(); - printf("\n >>> Test failure: dataset '%s' not found in the repository\n", dsname); - return -1; - } - // ... and that the default tree is 'ntuple' - gProof->SetDataSetTreeName(dsname, "ntuple"); - - // Create the histos - TH1F *h1s[3] = {0}; - h1s[0] = new TH1F("h1s0", "3*px+2 with px**2+py**2>1", 50, -15., 15.); - h1s[1] = new TH1F("h1s1", "2*px+2 with pz>2", 50, -10., 10.); - h1s[2] = new TH1F("h1s2", "1.3*px+2 with (px^2+py^2>4) && py>0", 50, 0., 8.); - - // Fill the histos using TProofDraw - PutPoint(); - { SwitchProgressGuard spg; - gProof->DrawSelect(dsname, "3*px+2 >> h1s0","px**2+py**2>1"); - PutPoint(); - gProof->DrawSelect(dsname, "2*px+2 >> h1s1","pz>2"); - PutPoint(); - gProof->DrawSelect(dsname, "1.3*px+2 >> h1s2","(px^2+py^2>4) && py>0"); - } - - Int_t rch1s = 0; - TString emsg; - // Check the histogram entries and mean values - Float_t hent[3] = { .607700, .364900, .065100}; - Double_t hmea[3] = { 2.022, 2.046 , 3.043}; - Double_t prec = 10. / TMath::Sqrt(nevt); // ~10 sigma ... conservative - for (Int_t i = 0; i < 3; i++) { - Double_t ent = h1s[i]->GetEntries(); - if (TMath::Abs(ent - hent[i] * nevt) / ent > prec) { - emsg.Form("'%s' histo: wrong number" - " of entries (%lld: expected %lld)", - h1s[i]->GetName(), (Long64_t) ent, (Long64_t)(hent[i] *nevt)); - rch1s = -1; - break; - } - Double_t mprec = 5 * h1s[i]->GetRMS() / TMath::Sqrt(h1s[i]->GetEntries()) ; - if (TMath::Abs((h1s[i]->GetMean() - hmea[i]) / hmea[i]) > mprec ) { - emsg.Form("'%s' histo: wrong mean (%f: expected %f - RMS: %f)", - h1s[i]->GetName(), h1s[i]->GetMean(), hmea[i], h1s[i]->GetRMS()); - rch1s = -1; - break; - } - } - - // Cleanup - for (Int_t i = 0; i < 3; i++) delete h1s[i]; - - // Check the result - if (rch1s != 0) { - printf("\n >>> Test failure: %s\n", emsg.Data()); - return -1; - } - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Check the result of the ProofFriends analysis - -Int_t PT_CheckFriends(TQueryResult *qr, Long64_t nevt, bool withfriends) -{ - if (!qr) { - printf("\n >>> Test failure: query result not found\n"); - return -1; - } - - // Make sure the number of processed entries is the one expected - PutPoint(); - if (qr->GetEntries() != nevt) { - printf("\n >>> Test failure: wrong number of entries processed: %lld (expected %lld)\n", - qr->GetEntries(), nevt); - return -1; - } - - // Make sure the output list is there - PutPoint(); - TList *out = qr->GetOutputList(); - if (!out) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - TString emsg; - // Check the histogram entries and mean values - Int_t nchk = (withfriends) ? 4 : 2; - Int_t rchs = 0; - const char *hnam[4] = { "histo1", "histo2", "histo3", "histo4" }; - const char *hcls[4] = { "TH2F", "TH1F", "TH1F", "TH2F" }; - Float_t hent[4] = { 1., .227, .0260, .0260}; - TObject *o = 0; - Double_t ent = -1; - Double_t prec = 1. / TMath::Sqrt(nevt); - for (Int_t i = 0; i < nchk; i++) { - if (!(o = out->FindObject(hnam[i]))) { - emsg.Form("object '%s' not found", hnam[i]); - rchs = -1; - break; - } - if (strcmp(o->IsA()->GetName(), hcls[i])) { - emsg.Form("object '%s' is not '%s'", hnam[i], hcls[i]); - rchs = -1; - break; - } - if (!strcmp(hcls[i], "TH1F")) { - ent = ((TH1F *)o)->GetEntries(); - } else { - ent = ((TH2F *)o)->GetEntries(); - } - if (TMath::Abs(ent - hent[i] * nevt) / (Double_t)ent > prec) { - emsg.Form("'%s' histo: wrong number of entries (%lld: expected %lld) \n", - o->GetName(), (Long64_t) ent, (Long64_t) (hent[i] * nevt)); - rchs = -1; - break; - } - } - - if (rchs != 0) { - printf("\n >>> Test failure: %s\n", emsg.Data()); - return -1; - } - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test session opening - -Int_t PT_Open(void *args, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - PT_Open_Args_t *PToa = (PT_Open_Args_t *)args; - if (!PToa) { - printf("\n >>> Test failure: invalid arguments: %p\n", args); - return -1; - } - - // Temp dir for PROOF tutorials - PutPoint(); -#if defined(R__MACOSX) - // Force '/tmp' under macosx, to avoid problems with lengths and symlinks - TString tmpdir("/tmp"), uspid; -#else - TString tmpdir(gSystem->TempDirectory()), uspid; -#endif - UserGroup_t *ug = gSystem->GetUserInfo(gSystem->GetUid()); - if (!ug) { - printf("\n >>> Test failure: could not get user info"); - return -1; - } - if (!tmpdir.EndsWith(ug->fUser.Data())) { - uspid.Form("/%s/%d", ug->fUser.Data(), gSystem->GetPid()); - delete ug; - } else { - uspid.Form("/%d", gSystem->GetPid()); - } - tmpdir += uspid; -#if !defined(R__MACOSX) - gtutdir.Form("%s/.proof-tutorial", tmpdir.Data()); -#else - gtutdir.Form("%s/.proof", tmpdir.Data()); -#endif - if (gSystem->AccessPathName(gtutdir)) { - if (gSystem->mkdir(gtutdir, kTRUE) != 0) { - printf("\n >>> Test failure: could not assert/create the temporary directory" - " for the tutorial (%s)", gtutdir.Data()); - return -1; - } - } - - // String to initialize the dataset manager - TString dsetmgrstr; - dsetmgrstr.Form("file dir:%s/datasets opt:-Cq:As:Sb:", gtutdir.Data()); - gEnv->SetValue("Proof.DataSetManager", dsetmgrstr.Data()); - - // String to initialize the package dir - TString packdir; - packdir.Form("%s/packages", gtutdir.Data()); - gEnv->SetValue("Proof.PackageDir", packdir.Data()); - - // Get the PROOF Session - PutPoint(); - TProof *p = getProof(PToa->url, PToa->nwrks, gtutdir.Data(), "force", gDynamicStartup, kTRUE); - if (!p || !(p->IsValid())) { - printf("\n >>> Test failure: could not start the session\n"); - return -1; - } - - // Re-check locality: if the logged user name is different from the local one, we may - // not have all the rights we need, so we go no-local - if (gLocalCluster) { - UserGroup_t *pw = gSystem->GetUserInfo(); - if (pw) { - if (strcmp(pw->fUser, p->GetUser())) gLocalCluster = kFALSE; - delete pw; - } - } - - // Check if it is in dynamic startup mode - Int_t dyn = 0; - p->GetRC("Proof.DynamicStartup", dyn); - if (dyn != 0) gDynamicStartup = kTRUE; - - // Set debug level, if required - if (gverbose > 1) { - Int_t debugscope = getDebugEnum(gverbproof.Data()); - p->SetLogLevel(gverbose, debugscope); - } - - PutPoint(); - if (PToa->nwrks > 0 && p->GetParallel() != PToa->nwrks) { - printf("\n >>> Test failure: number of workers different from requested\n"); - return -1; - } - - // Clear Cache - p->ClearCache(); - - // Get some useful info about the cluster (the sandbox dir ...) - gSystem->RedirectOutput(0, 0, &gRH); - TString testPrint(TString::Format("%s/testPrint.log", gtutdir.Data())); - gSystem->RedirectOutput(testPrint, "w", &gRHAdmin); - gProof->Print(); - gSystem->RedirectOutput(0, 0, &gRHAdmin); - gSystem->RedirectOutput(glogfile, "a", &gRH); - TMacro macroPrint(testPrint); - TObjString *os = macroPrint.GetLineWith("Working directory:"); - if (!os) { - printf("\n >>> Test failure: problem parsing output from Print()\n"); - return -1; - } - Int_t from = strlen("Working directory:") + 1; - while (os->GetString().Tokenize(gsandbox, from, " ")) { - if (!gsandbox.IsNull()) break; - } - if (gsandbox.IsNull()) { - printf("\n >>> Test failure: no sandbox dir found\n"); - return -1; - } - gsandbox = gSystem->GetDirName(gsandbox); - gsandbox = gSystem->GetDirName(gsandbox); - PutPoint(); - - // Fill times - PT_GetLastTimes(tt); - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test log retrieving - -Int_t PT_GetLogs(void *args, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - PT_Open_Args_t *PToa = (PT_Open_Args_t *)args; - if (!PToa) { - printf("\n >>> Test failure: invalid arguments: %p\n", args); - return -1; - } - - PutPoint(); - TProofLog *pl = TProof::Mgr(PToa->url)->GetSessionLogs(); - if (!pl) { - printf("\n >>> Test failure: could not get the logs from last session\n"); - return -1; - } - - PutPoint(); - if (PToa->nwrks > 0 && pl->GetListOfLogs()->GetSize() != (PToa->nwrks + 1)) { - printf("\n >>> Test failure: number of logs different from workers of workers + 1\n"); - return -1; - } - - // Fill times - PT_GetLastTimes(tt); - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the ProofSimple analysis (see tutorials) - -Int_t PT_Simple(void *opts, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - PT_Option_t *ptopt = (PT_Option_t *) opts; - - // Setup submergers if required - if (ptopt && ptopt->fOne > 0) { - gProof->SetParameter("PROOF_UseMergers", 0); - } - // Setup save-to-file, if required - TString opt = (ptopt && ptopt->fTwo > 0) ? "stf" : "" ; - - // Define the number of events and histos - Long64_t nevt = 1000000; - Int_t nhist = 16; - // The number of histograms is added as parameter in the input list - gProof->SetParameter("ProofSimple_NHist", (Long_t)nhist); - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(gSimpleSel.Data(), nevt, opt); - gTimer.Stop(); - } - - // Count - gSimpleCnt++; - gSimpleTime += gTimer.RealTime(); - - // Remove any setting related to submergers - gProof->DeleteParameters("PROOF_UseMergers"); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckSimple(gProof->GetQueryResult(), nevt, nhist); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test output handling via file using ProofSimple (see tutorials) - -Int_t PT_OutputHandlingViaFile(void *opts, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - // Not yet supported in dynamic mode - if (gDynamicStartup) { - return 1; - } - PutPoint(); - - PT_Option_t *ptopt = (PT_Option_t *) opts; - - // Setup submergers if required - if (ptopt && ptopt->fOne > 0) { - gProof->SetParameter("PROOF_UseMergers", 0); - } - // Setup save-to-file, if required - TString opt = (ptopt && ptopt->fTwo > 0) ? "stf" : "" ; - - // Define the number of events and histos - Long64_t nevt = 1000000 * gProof->GetParallel(); - Int_t nhist = 16; - // The number of histograms is added as parameter in the input list - gProof->SetParameter("ProofSimple_NHist", (Long_t)nhist); - - // Merged file pptions to be tested - const char *testopt[4] = { "stf", "of=proofsimple.root", "of=proofsimple.root;stf", - "of=master:proofsimple.root" }; - - for (Int_t i = 0; i < 4; i++) { - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Save results to file 'proofsimple.root' - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(gSimpleSel.Data(), nevt, testopt[i]); - gTimer.Stop(); - } - if (PT_CheckSimple(gProof->GetQueryResult(), nevt, nhist) != 0) { - printf("\n >>> Test failure: output handling via file: option '%s'\n", testopt[i]); - return -1; - } - // Count - gSimpleCnt++; - gSimpleTime += gTimer.RealTime(); - // Remove file - gSystem->Unlink("proofsimple.root"); - } - - // Test dataset creationg with a ntuple - const char *dsname = "PT_ds_proofsimple"; - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - if (gProof->ExistsDataSet(dsname)) gProof->RemoveDataSet(dsname); - - // We want the ntuple - gProof->SetParameter("ProofSimple_Ntuple", ""); - - // Save results to file 'proofsimple.root' - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(gSimpleSel.Data(), nevt, TString::Format("ds=%s|V", dsname)); - gTimer.Stop(); - } - if (!gProof->ExistsDataSet(dsname)) { - printf("\n >>> Test failure: output handling via file: dataset '%s' not created\n", dsname); - return -1; - } - - // Remove any setting related to submergers - gProof->DeleteParameters("PROOF_UseMergers"); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckSimpleNtuple(gProof->GetQueryResult(), nevt, dsname); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the H1 analysis as a chain reading the data from HTTP - -Int_t PT_H1Http(void *, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - // Set/unset the parallel unzip flag - AssertParallelUnzip(); - - // Create the chain - PutPoint(); - TChain *chain = new TChain("h42"); - - // Assert the files, if needed - if (!gh1ok) { - if (PT_H1AssertFiles(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not assert the H1 files\n"); - return -1; - } - } - Int_t i = 0; - for (i = 0; i < 4; i++) { - chain->Add(TString::Format("%s%c%s", gh1src.Data(), gh1sep, gh1file[i])); - } - - // Clear associated memory cache - if (gClearCache && PT_H1ReleaseCache(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the H1 files\n"); - return -1; - } - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process - PutPoint(); - chain->SetProof(); - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - chain->Process(gH1Sel.Data()); - gTimer.Stop(); - } - gProof->RemoveChain(chain); - - // Count - gH1Cnt++; - gH1Time += gTimer.RealTime(); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckH1(gProof->GetQueryResult()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the H1 analysis as a file collection reading the data from HTTP - -Int_t PT_H1FileCollection(void *arg, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - // Set/unset the parallel unzip flag - AssertParallelUnzip(); - - // Are we asked to change the packetizer strategy? - if (arg) { - PT_Packetizer_t *strategy = (PT_Packetizer_t *)arg; - if (strcmp(strategy->fName, "TPacketizer")) { - gProof->SetParameter("PROOF_Packetizer", strategy->fName); - } else { - if (strategy->fType != 1) - gProof->SetParameter("PROOF_PacketizerStrategy", strategy->fType); - } - } - - // Create the file collection - PutPoint(); - TFileCollection *fc = new TFileCollection("h42"); - - // Assert the files, if needed - if (!gh1ok) { - if (PT_H1AssertFiles(gh1src.Data()) != 0) { - printf("\n >>> Test failure: could not assert the H1 files\n"); - return -1; - } - } - Int_t i = 0; - for (i = 0; i < 4; i++) { - fc->Add(new TFileInfo(TString::Format("%s%c%s", gh1src.Data(), gh1sep, gh1file[i]))); - } - - // Clear associated memory cache - if (gClearCache && PT_H1ReleaseCache(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the H1 files\n"); - return -1; - } - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(fc, gH1Sel.Data()); - gTimer.Stop(); - } - - // Restore settings - gProof->DeleteParameters("PROOF_Packetizer"); - gProof->DeleteParameters("PROOF_PacketizerStrategy"); - - // Count - gH1Cnt++; - gH1Time += gTimer.RealTime(); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckH1(gProof->GetQueryResult()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the H1 analysis as a named dataset reading the data from HTTP - -Int_t PT_H1DataSet(void *, RunTimes &tt) -{ - // Checking arguments - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - // Not yet supported for PROOF-Lite - if (gSkipDataSetTest) { - return 1; - } - PutPoint(); - - // Set/unset the parallel unzip flag - AssertParallelUnzip(); - - // Name for the target dataset - const char *dsname = "h1dset"; - - // Clear associated memory cache - if (gClearCache && PT_H1ReleaseCache(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the H1 files\n"); - return -1; - } - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process the dataset by name - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(dsname, gH1Sel.Data()); - gTimer.Stop(); - } - - // Count - gH1Cnt++; - gH1Time += gTimer.RealTime(); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckH1(gProof->GetQueryResult()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the H1 analysis as a named dataset reading the data from HTTP - -Int_t PT_H1MultiDataSet(void *, RunTimes &tt) -{ - // Checking arguments - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - // Not yet supported for PROOF-Lite - if (gSkipDataSetTest) { - return 1; - } - PutPoint(); - - // Set/unset the parallel unzip flag - AssertParallelUnzip(); - - // Name for the target dataset - const char *dsname = "h1dseta h1dsetb"; - - // Clear associated memory cache - if (gClearCache && PT_H1ReleaseCache(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the H1 files\n"); - return -1; - } - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process the dataset by name - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(dsname, gH1Sel.Data()); - gTimer.Stop(); - } - - // Count - gH1Cnt++; - gH1Time += gTimer.RealTime(); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckH1(gProof->GetQueryResult()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run using the H1 analysis for the multi-dataset functionality and -/// entry-lists - -Int_t PT_H1MultiDSetEntryList(void *, RunTimes &tt) -{ - // Checking arguments - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - // Not yet supported for PROOF-Lite - if (gSkipDataSetTest) { - return 1; - } - PutPoint(); - - // Set/unset the parallel unzip flag - AssertParallelUnzip(); - - // Multiple dataset used to create the entry list - TString dsname("h1dseta|h1dsetb"); - - // Clear associated memory cache - if (gClearCache && PT_H1ReleaseCache(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the H1 files\n"); - return -1; - } - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Entry-list creation run - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(dsname, gH1Sel.Data(), "fillList=elist.root"); - gTimer.Stop(); - } - - // Cleanup entry-list from the input list - TIter nxi(gProof->GetInputList()); - TObject *o = 0; - while ((o = nxi())) { - if (!strncmp(o->GetName(), "elist", 6) || !strcmp(o->GetName(), "fillList")) { - gProof->GetInputList()->Remove(o); - delete o; - } - } - - // Count - gH1Cnt++; - gH1Time += gTimer.RealTime(); - - // Clear associated memory cache - if (gClearCache && PT_H1ReleaseCache(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the H1 files\n"); - return -1; - } - - // Run using the entrylist - dsname = "h1dseta<Process(dsname, gH1Sel.Data()); - gTimer.Stop(); - } - - // Unlink the entry list file - gSystem->Unlink("elist.root"); - - // Cleanup entry-list from the input list - nxi.Reset(); - while ((o = nxi())) { - if (!strncmp(o->GetName(), "elist", 6)) { - gProof->GetInputList()->Remove(o); - delete o; - } - } - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckH1(gProof->GetQueryResult(), 1); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test dataset registration, verification, usage, removal. -/// Use H1 analysis files on HTTP as example - -Int_t PT_DataSets(void *, RunTimes &tt) -{ - // Checking arguments - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - // Not yet supported for PROOF-Lite - if (gSkipDataSetTest) { - return 1; - } - PutPoint(); - - // Cleanup the area - PutPoint(); - TMap *dsm = gProof->GetDataSets(); - if (!dsm) { - printf("\n >>> Test failure: could not retrieve map of datasets (even empty)!\n"); - return -1; - } - if (dsm->GetSize() > 0) { - // Remove the datasets already registered - TIter nxd(dsm); - TObjString *os = 0; - while ((os = (TObjString *)nxd())) { - gProof->RemoveDataSet(os->GetName()); - } - // Check the result - delete dsm; - dsm = gProof->GetDataSets(); - if (!dsm || dsm->GetSize() > 0) { - printf("\n >>> Test failure: could not cleanup the dataset area! (%p)\n", dsm); - delete dsm; - return -1; - } - } - delete dsm; - - // Create the file collection - PutPoint(); - TFileCollection *fc = new TFileCollection(); - TFileCollection *fca = new TFileCollection(); - TFileCollection *fcb = new TFileCollection(); - - // Assert the files, if needed - if (!gh1ok) { - if (PT_H1AssertFiles(gh1src.Data()) != 0) { - printf("\n >>> Test failure: could not assert the H1 files\n"); - return -1; - } - } - Int_t i = 0; - for (i = 0; i < 4; i++) { - fc->Add(new TFileInfo(TString::Format("%s%c%s", gh1src.Data(), gh1sep, gh1file[i]))); - if (i < 2) { - fca->Add(new TFileInfo(TString::Format("%s%c%s", gh1src.Data(), gh1sep, gh1file[i]))); - } else { - fcb->Add(new TFileInfo(TString::Format("%s%c%s", gh1src.Data(), gh1sep, gh1file[i]))); - } - } - fc->Update(); - fca->Update(); - fcb->Update(); - - // Name for this dataset - const char *dsname = "h1dset"; - const char *dsnamea = "h1dseta"; - const char *dsnameb = "h1dsetb"; - - // Register the dataset - PutPoint(); - gProof->RegisterDataSet(dsname, fc); - gProof->RegisterDataSet(dsnamea, fca); - gProof->RegisterDataSet(dsnameb, fcb); - // Check the result - dsm = gProof->GetDataSets(); - if (!dsm || dsm->GetSize() != 3) { - printf("\n >>> Test failure: could not register '%s,%s,%s' (%p)\n", - dsname, dsnamea, dsnameb, dsm); - delete dsm; - return -1; - } - delete dsm; - - // Test removal - PutPoint(); - gProof->RemoveDataSet(dsname); - gProof->RemoveDataSet(dsnamea); - gProof->RemoveDataSet(dsnameb); - // Check the result - dsm = gProof->GetDataSets(); - if (!dsm || dsm->GetSize() != 0) { - printf("\n >>> Test failure: could not cleanup '%s,%s,%s' (%p)\n", - dsname, dsnamea, dsnameb, dsm); - delete dsm; - return -1; - } - delete dsm; - - // Re-register the dataset - PutPoint(); - gProof->RegisterDataSet(dsname, fc); - gProof->RegisterDataSet(dsnamea, fca); - gProof->RegisterDataSet(dsnameb, fcb); - // Check the result - dsm = gProof->GetDataSets(); - if (!dsm || dsm->GetSize() != 3) { - printf("\n >>> Test failure: could not re-register '%s,%s,%s' (%p)\n", - dsname, dsnamea, dsnameb, dsm); - delete dsm; - return -1; - } - delete dsm; - - // Verify the dataset - PutPoint(); - if (gProof->VerifyDataSet(dsname) != 0) { - printf("\n >>> Test failure: could not verify '%s'!\n", dsname); - return -1; - } - if (gProof->VerifyDataSet(dsnamea) != 0) { - printf("\n >>> Test failure: could not verify '%s'!\n", dsnamea); - return -1; - } - if (gProof->VerifyDataSet(dsnameb) != 0) { - printf("\n >>> Test failure: could not verify '%s'!\n", dsnameb); - return -1; - } - gProof->ShowDataSets(); - - // Remove the file collection - delete fc; - delete fca; - delete fcb; - - // The runtimes - PT_GetLastProofTimes(tt); - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test package clearing, uploading, enabling, removal. -/// Use event.par as example. - -Int_t PT_Packages(void *, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - // Cleanup the area - PutPoint(); - TList *packs = gProof->GetListOfPackages(); - if (!packs) { - printf("\n >>> Test failure: could not retrieve list of packages (even empty)!\n"); - return -1; - } - if (packs->GetSize() > 0) { - // Remove the packages already available - gProof->ClearPackages(); - // Check the result - packs = gProof->GetListOfPackages(); - if (!packs || packs->GetSize() > 0) { - printf("\n >>> Test failure: could not cleanup the package area!\n"); - return -1; - } - } - - // Name and location for this package - const char *pack = "event"; - - // Upload the package - PutPoint(); - gProof->UploadPackage(gPackEvent); - // Check the result - packs = gProof->GetListOfPackages(); - if (!packs || packs->GetSize() != 1) { - printf("\n >>> Test failure: could not upload '%s'!\n", gPackEvent.Data()); - return -1; - } - - // Test cleanup - PutPoint(); - gProof->ClearPackage(pack); - // Check the result - packs = gProof->GetListOfPackages(); - if (!packs || packs->GetSize() != 0) { - printf("\n >>> Test failure: could not cleanup '%s'!\n", pack); - return -1; - } - - // Re-upload the package - PutPoint(); - gProof->UploadPackage(gPackEvent); - // Check the result - packs = gProof->GetListOfPackages(); - if (!packs || packs->GetSize() != 1) { - printf("\n >>> Test failure: could not re-upload '%s'!\n", gPackEvent.Data()); - return -1; - } - - // Enable the package - PutPoint(); - gProof->EnablePackage(pack); - // Check the result - packs = gProof->GetListOfEnabledPackages(); - if (!packs || packs->GetSize() != 1) { - printf("\n >>> Test failure: could not enable '%s'!\n", pack); - return -1; - } - - // Fill times - PT_GetLastTimes(tt); - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the ProofEvent analysis (see tutorials) - -Int_t PT_Event(void *, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - // Define the number of events - Long64_t nevt = 100000; - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gProof->Process(gEventSel.Data(), nevt); - } - - // Make sure the query result is there - PutPoint(); - TQueryResult *qr = 0; - if (!(qr = gProof->GetQueryResult())) { - printf("\n >>> Test failure: query result not found\n"); - return -1; - } - - // Make sure the number of processed entries is the one expected - PutPoint(); - if (qr->GetEntries() != nevt) { - printf("\n >>> Test failure: wrong number of entries processed: %lld (expected %lld)\n", - qr->GetEntries(), nevt); - return -1; - } - - // Make sure the output list is there - PutPoint(); - if (!(gProof->GetOutputList())) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - // Check the 'histo' - PutPoint(); - TH1F *histo = dynamic_cast(gProof->GetOutputList()->FindObject("histo")); - if (!histo) { - printf("\n >>> Test failure: 'histo' not found\n"); - return -1; - } - - // Check the mean values - Double_t ave = histo->GetMean(); - Double_t rms = histo->GetRMS(); - if (TMath::Abs(ave - 50) > 10 * rms / TMath::Sqrt(histo->GetEntries())) { - printf("\n >>> Test failure: 'histo': mean > 5 * RMS/Sqrt(N)\n"); - return -1; - } - - // The runtimes - PT_GetLastProofTimes(tt); - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test input data functionality - -Int_t PT_InputData(void *, RunTimes &tt) -{ - // Checking arguments - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - PutPoint(); - - // Create the test information to be send via input and retrieved - TH1F *h1 = new TH1F("h1data","Input data from file",100,-5.,5.); - h1->FillRandom("gaus", 1000); - TList *h1list = new TList; - h1list->SetName("h1list"); - h1list->SetOwner(kTRUE); - h1list->Add(h1); - h1list->Add(new TParameter("h1avg", h1->GetMean())); - h1list->Add(new TParameter("h1rms", h1->GetRMS())); - TString datafile = glogfile; - datafile += ("_h1data.root"); - TFile *f = TFile::Open(datafile, "RECREATE"); - if (!f) { - printf("\n >>> Test failure: could not open file for input data\n"); - return -1; - } - f->cd(); - h1list->Write(0, TObject::kSingleKey, 0); - f->Close(); - gProof->SetInputDataFile(datafile.Data()); - - // Histo to be sent from memory - TH1F *h2 = new TH1F("h2data","Input data from memory",100,-5.,5.); - h2->FillRandom("gaus", 1000); - TList *h2list = new TList; - h2list->SetName("h2list"); - h2list->SetOwner(kTRUE); - h2list->Add(h2); - h2list->Add(new TParameter("h2avg", h2->GetMean())); - h2list->Add(new TParameter("h2rms", h2->GetRMS())); - gProof->AddInputData(h2list); - - // Normal input parameter - gProof->AddInput(new TNamed("InputObject", glogfile.Data())); - - // Type of test - gProof->AddInput(new TNamed("ProofTests_Type", "InputData")); - - // Define the number of events - Long64_t nevt = 1; - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gProof->Process(gTestsSel.Data(), nevt); - } - - // Cleanup - gSystem->Unlink(datafile.Data()); - gProof->ClearInputData(h1list); - gProof->ClearInputData(h2list); - delete h1list; - delete h2list; - - // Make sure the query result is there - PutPoint(); - TQueryResult *qr = 0; - if (!(qr = gProof->GetQueryResult())) { - printf("\n >>> Test failure: query result not found\n"); - return -1; - } - - // Make sure the output list is there - PutPoint(); - if (!(gProof->GetOutputList())) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - // Check the 'histo's - PutPoint(); - TH1I *stat = dynamic_cast(gProof->GetOutputList()->FindObject("TestStat")); - if (!stat) { - printf("\n >>> Test failure: 'TestStat' histo not found\n"); - return -1; - } - - // Test how many workers got everything successfully - Int_t nw = (Int_t) stat->GetBinContent(1); - PutPoint(); - - if (TMath::Abs(stat->GetBinContent(2) - nw) > .1) { - printf("\n >>> Test failure: histo 'h1' not correctly received on all workers (%.0f/%d)\n", - stat->GetBinContent(2), nw); - return -1; - } - if (TMath::Abs(stat->GetBinContent(3) - nw) > .1) { - printf("\n >>> Test failure: histo 'h2' not correctly received on all workers (%.0f/%d)\n", - stat->GetBinContent(3), nw); - return -1; - } - if (TMath::Abs(stat->GetBinContent(4) - nw) > .1) { - printf("\n >>> Test failure: test input object not correctly received on all workers (%.0f/%d)\n", - stat->GetBinContent(4), nw); - return -1; - } - - // Fill times - PT_GetLastTimes(tt); - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Testing passing arguments to packages - -Int_t PT_PackageArguments(void *, RunTimes &tt) -{ - // Checking arguments - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - PutPoint(); - - // Passing a 'const char *': upload packtest1 - const char *pack1 = "packtest1"; - PutPoint(); - gProof->UploadPackage(gPack1); - // Check the result - TList *packs = gProof->GetListOfPackages(); - if (!packs || !packs->FindObject(pack1)) { - printf("\n >>> Test failure: could not upload '%s'!\n", gPack1.Data()); - return -1; - } - // Enable the package now passing a 'const char *' argument - TString arg("ProofTest.ConstChar"); - if (gProof->EnablePackage(pack1, arg) != 0) { - printf("\n >>> Test failure: could not enable '%s' with argument: '%s'!\n", gPack1.Data(), arg.Data()); - return -1; - } - - // Type of test - gProof->SetParameter("ProofTests_Type", "PackTest1"); - - // Define the number of events - Long64_t nevt = 1; - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Variable to check - gProof->SetParameter("testenv", arg.Data()); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gProof->Process(gTestsSel.Data(), nevt); - } - - // Some cleanup - gProof->ClearPackage(pack1); - gProof->DeleteParameters("ProofTests_Type"); - gProof->DeleteParameters("testenv"); - - // Make sure the query result is there - PutPoint(); - TQueryResult *qr = 0; - if (!(qr = gProof->GetQueryResult())) { - printf("\n >>> Test failure: query result not found\n"); - return -1; - } - - // Make sure the output list is there - PutPoint(); - if (!(gProof->GetOutputList())) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - // Check the 'histo's - PutPoint(); - TH1I *stat = dynamic_cast(gProof->GetOutputList()->FindObject("TestStat")); - if (!stat) { - printf("\n >>> Test failure: 'TestStat' histo not found\n"); - return -1; - } - - // Test how many workers got everything successfully - Int_t nw = (Int_t) stat->GetBinContent(1); - PutPoint(); - - if (TMath::Abs(stat->GetBinContent(2) - nw) > .1) { - printf("\n >>> Test failure: var '%s' not correctly set on all workers (%.0f/%d)\n", - arg.Data(), stat->GetBinContent(2), nw); - return -1; - } - - // Passing a 'TList *': upload packtest2 - const char *pack2 = "packtest2"; - PutPoint(); - gProof->UploadPackage(gPack2); - // Check the result - packs = gProof->GetListOfPackages(); - if (!packs || !packs->FindObject(pack2)) { - printf("\n >>> Test failure: could not upload '%s'!\n", gPack2.Data()); - return -1; - } - // Testing recursive enabling via dependencies: upload packtest3 - const char *pack3 = "packtest3"; - PutPoint(); - gProof->UploadPackage(gPack3); - // Check the result - packs = gProof->GetListOfPackages(); - if (!packs || !packs->FindObject(pack3)) { - printf("\n >>> Test failure: could not upload '%s'!\n", gPack3.Data()); - return -1; - } - // Create the argument list - TList *argls = new TList; - argls->Add(new TNamed("ProofTest.ArgOne", "2.")); - argls->Add(new TNamed("ProofTest.ArgTwo", "3.")); - argls->Add(new TNamed("ProofTest.ArgThree", "4.")); - // Enable the package now passing the 'TList *' argument - if (gProof->EnablePackage(pack3, argls) != 0) { - printf("\n >>> Test failure: could not enable '%s' with argument: '%s'!\n", gPack3.Data(), arg.Data()); - return -1; - } - // Check list of enabled packages - TList *enpkg = gProof->GetListOfEnabledPackages(); - if (!enpkg || enpkg->GetSize() < 3) { - printf("\n >>> Test failure: not all requested packages enabled\n"); - if (!enpkg->FindObject("packtest1")) printf("\n >>> Test failure: 'packtest1' not enabled\n"); - if (!enpkg->FindObject("packtest2")) printf("\n >>> Test failure: 'packtest2' not enabled\n"); - if (!enpkg->FindObject("packtest3")) printf("\n >>> Test failure: 'packtest3' not enabled\n"); - return -1; - } - - // Type of test - gProof->SetParameter("ProofTests_Type", "PackTest2"); - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Variable to check - TString envs("ProofTest.ArgOne,ProofTest.ArgTwo,ProofTest.ArgThree"); - gProof->SetParameter("testenv", envs.Data()); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gProof->Process(gTestsSel.Data(), nevt); - } - - // Some cleanup - gProof->ClearPackage(pack2); - gProof->DeleteParameters("ProofTests_Type"); - gProof->DeleteParameters("testenv"); - - // Make sure the query result is there - PutPoint(); - qr = 0; - if (!(qr = gProof->GetQueryResult())) { - printf("\n >>> Test failure: query result not found\n"); - return -1; - } - - // Make sure the output list is there - PutPoint(); - if (!(gProof->GetOutputList())) { - printf("\n >>> Test failure: output list not found\n"); - return -1; - } - - // Check the 'histo's - PutPoint(); - if (!(stat = dynamic_cast(gProof->GetOutputList()->FindObject("TestStat")))) { - printf("\n >>> Test failure: 'TestStat' histo not found\n"); - return -1; - } - - // Test how many workers got everything successfully - nw = (Int_t) stat->GetBinContent(1); - PutPoint(); - - if (TMath::Abs(stat->GetBinContent(2) - nw) > .1) { - printf("\n >>> Test failure: var 'ProofTest.ArgOne' not correctly set on all workers (%.0f/%d)\n", - stat->GetBinContent(2), nw); - return -1; - } - - if (TMath::Abs(stat->GetBinContent(3) - nw) > .1) { - printf("\n >>> Test failure: var 'ProofTest.ArgTwo' not correctly set on all workers (%.0f/%d)\n", - stat->GetBinContent(3), nw); - return -1; - } - - if (TMath::Abs(stat->GetBinContent(4) - nw) > .1) { - printf("\n >>> Test failure: var 'ProofTest.ArgThree' not correctly set on all workers (%.0f/%d)\n", - stat->GetBinContent(4), nw); - return -1; - } - - // Fill times - PT_GetLastTimes(tt); - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the H1 and Simple analysis in asynchronous mode - -Int_t PT_H1SimpleAsync(void *arg, RunTimes &tt) -{ - // Checking arguments - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - // Not yet supported for PROOF-Lite - if (gProof->IsLite()) { - return 1; - } - // Not supported in dynamic mode - if (gDynamicStartup) { - return 1; - } - PutPoint(); - - // Set/unset the parallel unzip flag - AssertParallelUnzip(); - - // Are we asked to change the packetizer strategy? - if (arg) { - PT_Packetizer_t *strategy = (PT_Packetizer_t *)arg; - if (strcmp(strategy->fName, "TPacketizer")) { - gProof->SetParameter("PROOF_Packetizer", strategy->fName); - } else { - if (strategy->fType != 1) - gProof->SetParameter("PROOF_PacketizerStrategy", strategy->fType); - } - } - - // Create the file collection - PutPoint(); - TFileCollection *fc = new TFileCollection("h42"); - - // Assert the files, if needed - if (!gh1ok) { - if (PT_H1AssertFiles(gh1src.Data()) != 0) { - printf("\n >>> Test failure: could not assert the H1 files\n"); - return -1; - } - } - Int_t i = 0; - for (i = 0; i < 4; i++) { - fc->Add(new TFileInfo(TString::Format("%s%c%s", gh1src.Data(), gh1sep, gh1file[i]))); - } - - // Clear associated memory cache - if (gClearCache && PT_H1ReleaseCache(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the H1 files\n"); - return -1; - } - - // Clear the list of query results - PutPoint(); - if (gProof->GetQueryResults()) { - gProof->GetQueryResults()->Clear(); - gProof->Remove("cleanupdir"); - } - - // Define the number of events and histos - Long64_t nevt = 1000000; - Int_t nhist = 16; - // Submit the processing requests - PutPoint(); - { SwitchProgressGuard spg; - gProof->Process(fc, gH1Sel.Data(), "ASYN"); - - // The number of histograms is added as parameter in the input list - gProof->SetParameter("ProofSimple_NHist", (Long_t)nhist); - gProof->Process(gSimpleSel.Data(), nevt, "ASYN"); - - // Wait a bit as a function of previous runnings - Double_t dtw = 10; - if (gH1Cnt > 0 && gSimpleTime > 0) { - dtw = gH1Time / gH1Cnt + gSimpleTime / gSimpleCnt + 1; - if (dtw < 10) dtw = 10; - } - Int_t tw = (Int_t) (5 * dtw); - - gTimedOut = kFALSE; - TTimeOutTimer t(tw*1000); - - // Wait for the processing - while (!gProof->IsIdle() && !gTimedOut) - gSystem->InnerLoop(); - - } - PutPoint(); - - // Restore settings - gProof->DeleteParameters("PROOF_Packetizer"); - gProof->DeleteParameters("PROOF_PacketizerStrategy"); - - // Retrieve the list of available query results - TList *ql = gProof->GetQueryResults(); - if (ql && ql->GetSize() > 0) { - ql->Print(); - } - - TString ref; - TIter nxq(ql, kIterBackward); - Int_t nd = 2; - TQueryResult *qr = 0; - while ((qr = (TQueryResult *)nxq()) && nd > 0) { - ref.Form("%s:%s", qr->GetTitle(), qr->GetName()); - gProof->Retrieve(ref); - qr = gProof->GetQueryResult(ref); - if (qr && qr->GetSelecImp()) { - if (!strcmp(qr->GetSelecImp()->GetTitle(), "h1analysis")) { - PutPoint(); - if (PT_CheckH1(qr) != 0) return -1; - nd--; - } else if (!strcmp(qr->GetSelecImp()->GetTitle(), "ProofSimple")) { - PutPoint(); - if (PT_CheckSimple(qr, nevt, nhist) != 0) return -1; - nd--; - } else { - printf("\n >>> Test failure: query with unexpected selector '%s'\n", qr->GetSelecImp()->GetTitle()); - return -1; - } - } else { - printf("\n >>> Test failure: query undefined (%p) or with empty selector macro ('%s')\n", qr, ref.Data()); - return -1; - } - } - - // The runtimes - PT_GetLastProofTimes(tt); - - // Done - PutPoint(); - return 0; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the admin functionality - -Int_t PT_AdminFunc(void *, RunTimes &tt) -{ - // Checking arguments - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - // Not yet supported for PROOF-Lite - if (gProof->IsLite()) { - return 1; - } - PutPoint(); - // Attach to the manager - TProofMgr *mgr = gProof->GetManager(); - if (!mgr) { - printf("\n >>> Test failure: no PROOF manager found\n"); - return -1; - } - PutPoint(); - // Directory for this test - TString testDir(TString::Format("%s/stressProof-Admin", gtutdir.Data())); - if (gSystem->AccessPathName(testDir)) { - // Create the directory - if (gSystem->MakeDirectory(testDir)) { - printf("\n >>> Test failure: cannot create %s\n", testDir.Data()); - return -1; - } - } - // Create a small test file - TMacro testMacro; - testMacro.AddLine("// Test macro"); - testMacro.AddLine("#include \"TSystem.h\""); - testMacro.AddLine("void testMacro(Int_t opt = 1)"); - testMacro.AddLine("{"); - testMacro.AddLine(" if (opt == 1) {"); - testMacro.AddLine(" Printf(\"Pid: \", gSystem->GetPid());"); - testMacro.AddLine(" }"); - testMacro.AddLine("}"); - // Save the file in the temporary area - TString testFile(TString::Format("%s/testMacro.C", testDir.Data())); - if (!gSystem->AccessPathName(testFile)) { - // The file exists: remove it first - if (gSystem->Unlink(testFile)) { - printf("\n >>> Test failure: cannot unlink %s\n", testFile.Data()); - return -1; - } - } - testMacro.SaveSource(testFile); - FileStat_t stloc; - if (gSystem->GetPathInfo(testFile, stloc) != 0) { - // The file was not created - printf("\n >>> Test failure: file %s was not created\n", testFile.Data()); - return -1; - } - // Reference checksum - std::unique_ptr testMacroMd5(testMacro.Checksum()); - if (!testMacroMd5.get()) { - // MD5 sum not calculated - printf("\n >>> Test failure: could not calculate the md5 sum of the test macro\n"); - return -1; - } - PutPoint(); - - // Send the file to the sandbox - if (mgr->PutFile(testFile, "~/", "force") != 0) { - // The file was not sent over correctly - printf("\n >>> Test failure: problems sending file to master sandbox\n"); - return -1; - } - PutPoint(); - // Retrieve the file - TString testFile1(TString::Format("%s/testMacro.cxx", testDir.Data())); - if (mgr->GetFile("~/testMacro.C", testFile1, "force") != 0) { - // The file was not retrieved correctly - printf("\n >>> Test failure: problems retrieving file from the master sandbox\n"); - return -1; - } - PutPoint(); - - // Test 'ls' - gSystem->RedirectOutput(0, 0, &gRH); - TString testLs(TString::Format("%s/testLs.log", testDir.Data())); - gSystem->RedirectOutput(testLs, "w", &gRHAdmin); - mgr->Ls("~/testMacro.C"); - gSystem->RedirectOutput(0, 0, &gRHAdmin); - gSystem->RedirectOutput(glogfile, "a", &gRH); - TMacro macroLs(testLs); - TString testLsLine = TString::Format("%s/testMacro.C", gsandbox.Data()); - // The first part of maybe sligthly different -#if defined(R__MACOSX) - if (testLsLine.Index(".proof") != kNPOS) - testLsLine.Remove(0, testLsLine.Index(".proof")); -#else - if (testLsLine.Index(".proof-tutorial") != kNPOS) - testLsLine.Remove(0, testLsLine.Index(".proof-tutorial")); -#endif - if (!macroLs.GetLineWith(testLsLine)) { - printf("\n >>> Test failure: Ls: output not consistent (line: '%s')\n", testLsLine.Data()); - printf(" >>> Log file: '%s'\n", testLs.Data()); - printf("+++ BOF +++\n"); - macroLs.Print(); - printf("+++ EOF +++\n"); - return -1; - } - PutPoint(); - - // Test 'more' - gSystem->RedirectOutput(0, 0, &gRH); - TString testMore(TString::Format("%s/testMore.log", testDir.Data())); - gSystem->RedirectOutput(testMore, "w", &gRHAdmin); - mgr->More("~/testMacro.C"); - gSystem->RedirectOutput(0, 0, &gRHAdmin); - gSystem->RedirectOutput(glogfile, "a", &gRH); - TMacro macroMore(testMore); - if (macroMore.GetListOfLines()->GetSize() < 2) { - printf("\n >>> Test failure: More output not too short: %d lines\n", - macroMore.GetListOfLines()->GetSize()); - return -1; - } - TObjString *os = (TObjString *) macroMore.GetListOfLines()->First(); - while (!os->GetString().BeginsWith("// Test macro")) { - macroMore.GetListOfLines()->Remove(macroMore.GetListOfLines()->First()); - os = (TObjString *) macroMore.GetListOfLines()->First(); - } - std::unique_ptr testMoreMd5(macroMore.Checksum()); - if (!testMoreMd5.get()) { - // MD5 sum not calculated - printf("\n >>> Test failure: could not calculate the md5 sum of the 'more' result\n"); - return -1; - } - if (strcmp(testMacroMd5->AsString(), testMoreMd5->AsString())) { - printf("\n >>> Test failure: More: result not consistent with reference: {%s, %s}\n", - testMacroMd5->AsString(), testMoreMd5->AsString()); - return -1; - } - PutPoint(); - - // Test 'stat' - FileStat_t strem; - if (mgr->Stat("~/testMacro.C", strem) != 0) { - // Stat failure - printf("\n >>> Test failure: could not stat remotely the test file\n"); - return -1; - } - if (strem.fSize != stloc.fSize) { - // Stat failure - printf("\n >>> Test failure: stat sizes inconsistent: %lld vs %lld (bytes)\n", strem.fSize, stloc.fSize); - return -1; - } - PutPoint(); - - // Test 'cp' and 'md5sum; - if (mgr->Cp("http://root.cern/files/h1/dstarmb.root", "~/") != 0) { - // Cp failure - printf("\n >>> Test failure: could not retrieve remotely dstarmb.root from the Root Web server\n"); - return -1; - } - TString sum; - if (mgr->Md5sum("~/dstarmb.root", sum) != 0) { - // MD5 - printf("\n >>> Test failure: calculating the md5sum of dstarmb.root\n"); - return -1; - } - if (sum != "0a60055370e16d954f90fb50c2d1a801") { - // MD5 wrong - printf("\n >>> Test failure: wrong value for md5sum of dstarmb.root: %s\n", sum.Data()); - return -1; - } - PutPoint(); - - // Fill times - PT_GetLastTimes(tt); - - // Done - PutPoint(); - return 0; - -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test processing of sub-samples (entries-from-first) from files with the -/// 'event' structures - -Int_t PT_EventRange(void *arg, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - // Set/unset the parallel unzip flag - AssertParallelUnzip(); - - // Are we asked to change the packetizer strategy? - const char *pack = "TPacketizer"; - if (arg) { - PT_Packetizer_t *strategy = (PT_Packetizer_t *)arg; - if (strcmp(strategy->fName, "TPacketizer")) { - gProof->SetParameter("PROOF_Packetizer", strategy->fName); - pack = strategy->fName; - } else { - if (strategy->fType != 1) - gProof->SetParameter("PROOF_PacketizerStrategy", strategy->fType); - } - } - - // Test first with a chain - PutPoint(); - TChain *chain = new TChain("EventTree"); - - // Assert the files, if needed - if (!geventok) { - if (PT_EventAssertFiles(geventsrc.Data(), geventnf) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not assert the event files\n"); - return -1; - } - } - Int_t i = 0; - for (i = 0; i < geventnf; i++) { - chain->AddFile(TString::Format("%s/event_%d.root", geventsrc.Data(), i+1)); - } - - // Clear associated memory cache - if (gClearCache && PT_EventReleaseCache(geventsrc.Data(), geventnf) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the event files\n"); - return -1; - } - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Load special class for event ranges checks - if (gProof->Load(TString::Format("%s,%s", gProcFileElem.Data(), gEmptyInclude.Data())) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not load auxilliary files %s and %s\n", - gProcFileElem.Data(), gEmptyInclude.Data()); - return -1; - } - - // Add some parameters for later checking in Terminate - Int_t ifst = gEventFst / gEventSiz + 1; - Long64_t efst = gEventFst - (ifst - 1) * gEventSiz; - TString ffst = TString::Format("%s/event_%d.root?fst=%lld", geventsrc.Data(), ifst, efst); - gProof->SetParameter("Range_First_File", ffst.Data()); - - Int_t ilst = (gEventFst + gEventNum) / gEventSiz + 1; - Long64_t elst = (gEventFst + gEventNum) - (ilst - 1) * gEventSiz - 1; - TString flst = TString::Format("%s/event_%d.root?lst=%lld", geventsrc.Data(), ilst, elst); - gProof->SetParameter("Range_Last_File", flst.Data()); - gProof->SetParameter("Range_Num_Files", (Int_t) (ilst - ifst + 1)); - - // Process - PutPoint(); - chain->SetProof(); - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - chain->Process(gEventProcSel.Data(), "", gEventNum, gEventFst); - gTimer.Stop(); - } - gProof->RemoveChain(chain); - - // Count - gEventCnt++; - gEventTime += gTimer.RealTime(); - - // Check the result - Int_t rcch = PT_CheckEvent(gProof->GetQueryResult(), pack); - - // We are done if not dataset test possible - if (gSkipDataSetTest) { - // Restore settings - gProof->DeleteParameters("PROOF_Packetizer"); - gProof->DeleteParameters("PROOF_PacketizerStrategy"); - return rcch; - } - - // Create the dataset - TFileCollection *fc = new TFileCollection("dsevent", "", ""); - for (i = 0; i < geventnf; i++) { - fc->Add(new TFileInfo(TString::Format("%s/event_%d.root", geventsrc.Data(), i+1))); - } - - // Register the dataset - PutPoint(); - gProof->RegisterDataSet("dsevent", fc); - - // Check the result - if (!gProof->ExistsDataSet("dsevent")) { - printf("\n >>> Test failure: could not register 'dsevent'\n"); - return -1; - } - - // Verify the dataset - PutPoint(); - if (gProof->VerifyDataSet("dsevent") < 0) { - printf("\n >>> Test failure: could not verify 'dsevent'!\n"); - return -1; - } - - // Clear associated memory cache - if (gClearCache && PT_EventReleaseCache(geventsrc.Data(), geventnf) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the event files\n"); - return -1; - } - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process("dsevent", gEventProcSel.Data(), "", gEventNum, gEventFst); - gTimer.Stop(); - } - gProof->RemoveDataSet("dsevent"); - - // Restore settings - gProof->DeleteParameters("PROOF_Packetizer"); - gProof->DeleteParameters("PROOF_PacketizerStrategy"); - - // Count - gEventCnt++; - gEventTime += gTimer.RealTime(); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckEvent(gProof->GetQueryResult(), pack); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test TProofOutputFile technology to create a ntuple, with or without -/// submergers - -Int_t PT_POFNtuple(void *opts, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - PT_Option_t *ptopt = (PT_Option_t *) opts; - - // Setup submergers if required - if (ptopt && ptopt->fTwo > 0) { - gProof->SetParameter("PROOF_UseMergers", 0); - } - - // Output file - TString fout("/ProofNtuple.root"); - gProof->AddInput(new TNamed("PROOF_OUTPUTFILE", fout.Data())); - - // We use the 'NtpRndm' for a fixed values of randoms; we need to send over the file - gProof->SetInputDataFile(gNtpRndm); - // Set the related parameter - gProof->SetParameter("PROOF_USE_NTP_RNDM","yes"); - - // Define the number of events and histos - Long64_t nevt = 1000; - - // We do not plot the ntuple (we are in batch mode) - gProof->SetParameter("PROOF_NTUPLE_DONT_PLOT", "yes"); - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(gNtupleSel.Data(), nevt); - gTimer.Stop(); - } - - // Remove any setting related to submergers - gProof->DeleteParameters("PROOF_UseMergers"); - gProof->DeleteParameters("PROOF_NTUPLE_DONT_PLOT"); - gProof->DeleteParameters("PROOF_USE_NTP_RNDM"); - gProof->SetInputDataFile(0); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckNtuple(gProof->GetQueryResult(), nevt); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test TProofOutputFile technology to create a dataset - -Int_t PT_POFDataset(void *, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - const char *dsname = "testNtuple"; - // Clean-up any existing dataset with that name - if (gProof->ExistsDataSet(dsname)) gProof->RemoveDataSet(dsname); - - // Ask for registration of the dataset (the default is the TFileCollection is return - // without registration; the name of the TFileCollection is the name of the dataset - gProof->SetParameter("SimpleNtuple.root", dsname); - - // We use the 'NtpRndm' for a fixed values of randoms; we need to send over the file - gProof->SetInputDataFile(gNtpRndm); - // Set the related parameter - gProof->SetParameter("PROOF_USE_NTP_RNDM","yes"); - - // Define the number of events and histos - Long64_t nevt = 1000000; - - // We do not plot the ntuple (we are in batch mode) - gProof->SetParameter("PROOF_NTUPLE_DONT_PLOT", "yes"); - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(gNtupleSel.Data(), nevt); - gTimer.Stop(); - } - - // Remove any setting related to submergers - gProof->DeleteParameters("PROOF_NTUPLE_DONT_PLOT"); - gProof->DeleteParameters("SimpleNtuple.root"); - gProof->DeleteParameters("PROOF_USE_NTP_RNDM"); - gProof->SetInputDataFile(0); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckDataset(gProof->GetQueryResult(), nevt); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test processing of multiple trees in the same files - -Int_t PT_MultiTrees(void *, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - const char *dsname = "testNtuple"; - // There must be a dataset 'testNtuple' already registered and validated - if (!gProof->ExistsDataSet(dsname)) { - printf("\n >>> Test failure: dataset '%s' does not exist\n", dsname); - return -1; - } - - // Get the associated TFileCollection - TFileCollection *fc = gProof->GetDataSet(dsname); - if (!fc) { - printf("\n >>> Test failure: unable to get TFileCollection for dataset '%s'\n", dsname); - return -1; - } - - // Now create a TDSet out of the TFileCollection - TDSet *dset = new TDSet("testntps", "ntuple", "/", "TTree"); - TChain *ch1 = new TChain("ntuple"); - TChain *ch2 = new TChain("ntuple2"); - TIter nxf(fc->GetList()); - TFileInfo *fi = 0; - while ((fi = (TFileInfo *) nxf())) { - dset->Add(fi->GetCurrentUrl()->GetUrl()); - ch1->Add(fi->GetCurrentUrl()->GetUrl()); - ch2->Add(fi->GetCurrentUrl()->GetUrl()); - } - - // Check the ntuple content by filling some histos - TH1F *h1s[2] = {0}; - h1s[0] = new TH1F("h1_1", "3*px+2 with px**2+py**2>1", 50, -15., 15.); - h1s[1] = new TH1F("h1_2", "vx**2+vy**2 with abs(vz)<.1", 50, 0., 10.); - - Int_t rch1s = 0; - TString emsg; - const char *type[3] = { "dsname", "TDSet", "TChain" }; - for (Int_t j = 0; j < 3; j++) { - - PutPoint(); - - if (j == 0) { - // Fill the first histo from the first ntuple - gProof->SetDataSetTreeName(dsname, "ntuple"); - { SwitchProgressGuard spg; - gProof->DrawSelect(dsname, "3*px+2>>h1_1", "px*px+py*py>1"); - } - // Fill the second histo from the second ntuple - gProof->SetDataSetTreeName(dsname, "ntuple2"); - { SwitchProgressGuard spg; - gProof->DrawSelect(dsname, "vx*vx+vy*vy>>h1_2", "vz>-0.1&&vz<0.1"); - } - } else if (j == 1) { - // Fill the first histo from the first ntuple - { SwitchProgressGuard spg; - gProof->DrawSelect(dset, "3*px+2>>h1_1", "px*px+py*py>1"); - } - // Fill the second histo from the second ntuple - dset->SetObjName("ntuple2"); - { SwitchProgressGuard spg; - gProof->DrawSelect(dset, "vx*vx+vy*vy>>h1_2", "vz>-0.1&&vz<0.1"); - } - } else { - // Fill the first histo from the first ntuple - { SwitchProgressGuard spg; - ch1->Draw("3*px+2>>h1_1", "px*px+py*py>1"); - } - // Fill the second histo from the second ntuple - { SwitchProgressGuard spg; - ch2->Draw("vx*vx+vy*vy>>h1_2", "vz>-0.1&&vz<0.1"); - } - } - - rch1s = 0; - // Check the histogram entries and mean values - Int_t hent[2] = { 607700, 96100}; - Double_t hmea[2] = { 2.022, 1.859}; - for (Int_t i = 0; i < 2; i++) { - if ((Int_t)(h1s[i]->GetEntries()) != hent[i]) { - emsg.Form("%s: '%s' histo: wrong number of entries (%d: expected %d)", - type[j], h1s[i]->GetName(), (Int_t)(h1s[i]->GetEntries()), hent[i]); - rch1s = -1; - break; - } - if (TMath::Abs((h1s[i]->GetMean() - hmea[i]) / hmea[i]) > 0.001) { - emsg.Form("%s: '%s' histo: wrong mean (%f: expected %f)", - type[j], h1s[i]->GetName(), h1s[i]->GetMean(), hmea[i]); - rch1s = -1; - break; - } - } - } - - // Cleanup - for (Int_t i = 0; i < 2; i++) delete h1s[i]; - - // Check the result - if (rch1s != 0) { - printf("\n >>> Test failure: %s\n", emsg.Data()); - return -1; - } - - // Clean-up - gProof->RemoveDataSet(dsname); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return rch1s; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test processing of TTree friends in PROOF - -Int_t PT_Friends(void *sf, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - // Not supported in dynamic mode - if (gDynamicStartup) { - return 1; - } - PutPoint(); - - // Separate or same file ? - Bool_t sameFile = (sf) ? kTRUE : kFALSE; - - // File generation: we use TPacketizerFile in here to create two files per node - TList *wrks = gProof->GetListOfSlaveInfos(); - if (!wrks) { - printf("\n >>> Test failure: could not get the list of information about the workers\n"); - return -1; - } - - // Create the map - TString fntree; - TMap *files = new TMap; - files->SetName("PROOF_FilesToProcess"); - TIter nxwi(wrks); - TSlaveInfo *wi = 0; - while ((wi = (TSlaveInfo *) nxwi())) { - fntree.Form("tree_%s.root", wi->GetOrdinal()); - THashList *wrklist = (THashList *) files->GetValue(wi->GetName()); - if (!wrklist) { - wrklist = new THashList; - wrklist->SetName(wi->GetName()); - files->Add(new TObjString(wi->GetName()), wrklist); - } - wrklist->Add(new TObjString(fntree)); - } - Int_t nwrk = wrks->GetSize(); - - // Generate the files - gProof->AddInput(files); - if (sameFile) { - Printf("runProof: friend tree stored in the same file as the main tree"); - gProof->SetParameter("ProofAux_Action", "GenerateTreesSameFile"); - } else { - gProof->SetParameter("ProofAux_Action", "GenerateTrees"); - } - - // File generation: define the number of events per worker - Long64_t nevt = 1000; - gProof->SetParameter("ProofAux_NEvents", (Long64_t)nevt); - // Special Packetizer - gProof->SetParameter("PROOF_Packetizer", "TPacketizerFile"); - // Now process - gProof->Process(gAuxSel.Data(), 1); - // Remove the packetizer specifications - gProof->DeleteParameters("PROOF_Packetizer"); - - // Check that we got some output - if (!gProof->GetOutputList()) { - printf("\n >>> Test failure: output list not found!\n"); - return -1; - } - - // Create the TDSet objects - TDSet *dset = new TDSet("Tmain", "Tmain"); - TDSet *dsetf = new TDSet("Tfrnd", "Tfrnd"); - // Fill them with the information found in the output list - Bool_t foundMain = kFALSE, foundFriend = kFALSE; - TIter nxo(gProof->GetOutputList()); - TObject *o = 0; - TObjString *os = 0; - while ((o = nxo())) { - TList *l = dynamic_cast (o); - if (l && !strncmp(l->GetName(), "MainList-", 9)) { - foundMain = kTRUE; - TIter nxf(l); - while ((os = (TObjString *) nxf())) - dset->Add(os->GetName()); - } - } - nxo.Reset(); - while ((o = nxo())) { - TList *l = dynamic_cast (o); - if (l && !strncmp(l->GetName(), "FriendList-", 11)) { - foundFriend = kTRUE; - TIter nxf(l); - while ((os = (TObjString *) nxf())) - dsetf->Add(os->GetName()); - } - } - - // If we did not found the main or the friend meta info we fail - if (!foundMain || !foundFriend) { - printf("\n >>> Test failure: 'main' or 'friend' meta info missing!\n"); - return -1; - } - - // Connect the two datasets for processing - dset->AddFriend(dsetf, "friend"); - - // We do not plot the ntuple (we are in batch mode) - gProof->SetParameter("PROOF_DONT_PLOT", "yes"); - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - dset->Process(gFriendsSel.Data()); - gTimer.Stop(); - } - - // Remove any setting - gProof->DeleteParameters("PROOF_DONT_PLOT"); - gProof->GetInputList()->Remove(files); - files->SetOwner(kTRUE); - SafeDelete(files); - // Clear the files created by this run - gProof->ClearData(TProof::kUnregistered | TProof::kForceClear); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckFriends(gProof->GetQueryResult(), nevt * nwrk, 1); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test processing of TTree in subdirectories - -Int_t PT_TreeSubDirs(void*, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - // Not supported in dynamic mode - if (gDynamicStartup) { - return 1; - } - PutPoint(); - - // File generation: we use TPacketizerFile in here to create two files per node - TList *wrks = gProof->GetListOfSlaveInfos(); - if (!wrks) { - printf("\n >>> Test failure: could not get the list of information about the workers\n"); - return -1; - } - - // Create the map - TString fntree; - TMap *files = new TMap; - files->SetName("PROOF_FilesToProcess"); - TIter nxwi(wrks); - TSlaveInfo *wi = 0; - while ((wi = (TSlaveInfo *) nxwi())) { - fntree.Form("tree_%s.root", wi->GetOrdinal()); - THashList *wrklist = (THashList *) files->GetValue(wi->GetName()); - if (!wrklist) { - wrklist = new THashList; - wrklist->SetName(wi->GetName()); - files->Add(new TObjString(wi->GetName()), wrklist); - } - wrklist->Add(new TObjString(fntree)); - } - Int_t nwrk = wrks->GetSize(); - - // Generate the files - gProof->AddInput(files); - gProof->SetParameter("ProofAux_Action", "GenerateTrees:dir1/dir2/dir3"); - - // File generation: define the number of events per worker - Long64_t nevt = 1000; - gProof->SetParameter("ProofAux_NEvents", (Long64_t)nevt); - // Special Packetizer - gProof->SetParameter("PROOF_Packetizer", "TPacketizerFile"); - // Now process - gProof->Process(gAuxSel.Data(), 1); - // Remove the packetizer specifications - gProof->DeleteParameters("PROOF_Packetizer"); - - // Check that we got some output - if (!gProof->GetOutputList()) { - printf("\n >>> Test failure: output list not found!\n"); - return -1; - } - - // Create the TChain objects - TChain *dset = new TChain("dir1/dir2/dir3/Tmain"); - // Fill them with the information found in the output list - Bool_t foundMain = kFALSE; - TIter nxo(gProof->GetOutputList()); - TObject *o = 0; - TObjString *os = 0; - while ((o = nxo())) { - TList *l = dynamic_cast (o); - if (l && !strncmp(l->GetName(), "MainList-", 9)) { - foundMain = kTRUE; - TIter nxf(l); - while ((os = (TObjString *) nxf())) - dset->Add(os->GetName()); - } - } - dset->SetProof(); - - // If we did not found the main or the friend meta info we fail - if (!foundMain) { - printf("\n >>> Test failure: 'main' meta info missing!\n"); - return -1; - } - - // We do not plot the ntuple (we are in batch mode) - gProof->SetParameter("PROOF_DONT_PLOT", "yes"); - - // We do use friends - gProof->SetParameter("PROOF_NO_FRIENDS", "yes"); - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - dset->Process(gFriendsSel.Data()); - gTimer.Stop(); - } - - // Remove any setting - gProof->DeleteParameters("PROOF_DONT_PLOT"); - gProof->GetInputList()->Remove(files); - files->SetOwner(kTRUE); - SafeDelete(files); - // Clear the files created by this run - gProof->ClearData(TProof::kUnregistered | TProof::kForceClear); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckFriends(gProof->GetQueryResult(), nevt * nwrk, 0); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the ProofSimple analysis (see tutorials) passing the -/// selector by object - -Int_t PT_SimpleByObj(void *submergers, RunTimes &tt) -{ - // Checking arguments - PutPoint(); - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - - // Setup submergers if required - if (submergers) { - gProof->SetParameter("PROOF_UseMergers", 0); - } - - // Define the number of events and histos - Long64_t nevt = 1000000; - Int_t nhist = 16; - // The number of histograms is set inside the selector object; make sure - // that it is not passed in the input list - gProof->DeleteParameters("ProofSimple_NHist"); - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Define TSelector object. We use reflection to avoid including the header, - // so being able to change the tutorial directory - TString emsg; - gProof->Load(gSimpleSel); - TSelector *sel = TSelector::GetSelector(gSimpleSel); - if (sel) { - TClass *cl = sel->IsA(); - if (cl) { - TDataMember *dm = cl->GetDataMember("fNhist"); - if (dm) { - TMethodCall *setter = dm->SetterMethod(cl); - if (setter) { - setter->Execute(sel, TString::Format("%d", nhist).Data()); - } else { - emsg = "no SetterMethod for fNhist: check version of ProofSimple"; - } - } else { - emsg = "fNhist not found"; - } - } else { - emsg = "IsA() failed"; - } - } else { - emsg = "GetSelector failed"; - } - if (!emsg.IsNull()) { - printf("\n >>> Test failure: initializing ProofSimple selector: %s\n", emsg.Data()); - return -1; - } - - // Process - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - gProof->Process(sel, nevt); - gTimer.Stop(); - } - - // Count - gSimpleCnt++; - gSimpleTime += gTimer.RealTime(); - - // Remove any setting related to submergers - gProof->DeleteParameters("PROOF_UseMergers"); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckSimple(gProof->GetQueryResult(), nevt, nhist); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Test run for the H1 analysis as a chain reading the data from HTTP and -/// passing the selector by object - -Int_t PT_H1ChainByObj(void *, RunTimes &tt) -{ - // Checking arguments - if (!gProof) { - printf("\n >>> Test failure: no PROOF session found\n"); - return -1; - } - // Not yet supported for PROOF-Lite - if (gSkipDataSetTest) { - return 1; - } - PutPoint(); - - // Set/unset the parallel unzip flag - AssertParallelUnzip(); - - // Create the chain - PutPoint(); - TChain *chain = new TChain("h42"); - - // Assert the files, if needed - if (!gh1ok) { - if (PT_H1AssertFiles(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not assert the H1 files\n"); - return -1; - } - } - Int_t i = 0; - for (i = 0; i < 4; i++) { - chain->Add(TString::Format("%s%c%s", gh1src.Data(), gh1sep, gh1file[i])); - } - - // Clear associated memory cache - if (gClearCache && PT_H1ReleaseCache(gh1src.Data()) != 0) { - gProof->SetPrintProgress(0); - printf("\n >>> Test failure: could not clear memory cache for the H1 files\n"); - return -1; - } - - // Clear the list of query results - if (gProof->GetQueryResults()) gProof->GetQueryResults()->Clear(); - - // Load TSelector - gProof->Load(gH1Sel); - TSelector *sel = TSelector::GetSelector(gH1Sel); - - // Process - PutPoint(); - chain->SetProof(); - PutPoint(); - { SwitchProgressGuard spg; - gTimer.Start(); - chain->Process(sel); - gTimer.Stop(); - } - gProof->RemoveChain(chain); - - // Count - gH1Cnt++; - gH1Time += gTimer.RealTime(); - - // The runtimes - PT_GetLastProofTimes(tt); - - // Check the results - PutPoint(); - return PT_CheckH1(gProof->GetQueryResult()); -} diff --git a/tree/tree/CMakeLists.txt b/tree/tree/CMakeLists.txt index e44a398b5aaa1..5fe4978a94ac2 100644 --- a/tree/tree/CMakeLists.txt +++ b/tree/tree/CMakeLists.txt @@ -52,7 +52,6 @@ ROOT_STANDARD_LIBRARY_PACKAGE(Tree TLeafS.h TNtupleD.h TNtuple.h - TQueryResult.h TreeUtils.h TSelector.h TSelectorList.h @@ -109,7 +108,6 @@ ROOT_STANDARD_LIBRARY_PACKAGE(Tree src/TLeafS.cxx src/TNtuple.cxx src/TNtupleD.cxx - src/TQueryResult.cxx src/TreeUtils.cxx src/TSelector.cxx src/TSelectorList.cxx diff --git a/tree/tree/doc/index.txt b/tree/tree/doc/index.txt deleted file mode 100644 index b111b110fb5cc..0000000000000 --- a/tree/tree/doc/index.txt +++ /dev/null @@ -1,14 +0,0 @@ -This directory contains all the basic classes for creating/manipulating Trees. -BEGIN_HTML - Several documents describing the TTree classes are listed below: - -END_HTML diff --git a/tree/tree/inc/TChain.h b/tree/tree/inc/TChain.h index e1f01528fb0a8..72d12df657a39 100644 --- a/tree/tree/inc/TChain.h +++ b/tree/tree/inc/TChain.h @@ -42,7 +42,6 @@ class TChain : public TTree { TFile *fFile; /// List of file names containing the trees (TChainElement, owned) TList *fStatus; ///< -> List of active/inactive branches (TChainElement, owned) - TChain *fProofChain; ///SetMakeClass(make);} void SetName(const char *name) override; virtual void SetPacketSize(Int_t size = 100); - virtual void SetProof(bool on = true, bool refresh = false, bool gettreeheader = false); void SetWeight(Double_t w=1, Option_t *option="") override; virtual void UseCache(Int_t maxCacheSize = 10, Int_t pageSize = 0); diff --git a/tree/tree/inc/TQueryResult.h b/tree/tree/inc/TQueryResult.h deleted file mode 100644 index 627a959fdeb0d..0000000000000 --- a/tree/tree/inc/TQueryResult.h +++ /dev/null @@ -1,162 +0,0 @@ -// @(#)root/tree:$Id$ -// Author: G Ganis Sep 2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -#ifndef ROOT_TQueryResult -#define ROOT_TQueryResult - - -////////////////////////////////////////////////////////////////////////// -// // -// TQueryResult // -// // -// A container class for the results of a query. // -// // -////////////////////////////////////////////////////////////////////////// - -#include "TNamed.h" -#include "TDatime.h" -#include "TString.h" - -#ifdef R__LESS_INCLUDES -class TMacro; -#else -#include "TMacro.h" -#endif - -class TBrowser; -class TTreePlayer; -class TQueryResult; - -bool operator==(const TQueryResult &qr1, const TQueryResult &qr2); - - -class TQueryResult : public TNamed { - -friend class TTreePlayer; -friend class TProofPlayerLite; -friend class TProofPlayerRemote; -friend class TProof; -friend class TProofLite; -friend class TProofServ; -friend class TQueryResultManager; - -public: - enum EQueryStatus { - kAborted = 0, kSubmitted, kRunning, kStopped, kCompleted - }; - -protected: - Int_t fSeqNum; ///< query unique sequential number - bool fDraw; ///< true if draw action query - EQueryStatus fStatus; ///< query status - TDatime fStart; ///< time when processing started - TDatime fEnd; ///< time when processing ended - Float_t fUsedCPU; ///< real CPU time used (seconds) - TString fOptions; ///< processing options + aclic mode (< opt >#< aclic_mode >) - TList *fInputList; ///< input list; contains also data sets, entry list, ... - Long64_t fEntries; ///< number of entries processed - Long64_t fFirst; ///< first entry processed - Long64_t fBytes; ///< number of bytes processed - TMacro *fLogFile; ///< file with log messages from the query - TMacro *fSelecHdr; ///< selector header file - TMacro *fSelecImp; ///< selector implementation file - TString fLibList; ///< blank-separated list of libs loaded at fStart - TString fParList; ///< colon-separated list of PAR loaded at fStart - TList *fOutputList; ///< output list - bool fFinalized; ///< whether Terminate has been run - bool fArchived; ///< whether the query has been archived - TString fResultFile; ///< URL of the file where results have been archived - Float_t fPrepTime; ///< Prepare time (seconds) (millisec precision) - Float_t fInitTime; ///< Initialization time (seconds) (millisec precision) - Float_t fProcTime; ///< Processing time (seconds) (millisec precision) - Float_t fMergeTime; ///< Merging time (seconds) (millisec precision) - Float_t fRecvTime; ///< Transfer-to-client time (seconds) (millisec precision) - Float_t fTermTime; ///< Terminate time (seconds) (millisec precision) - Int_t fNumWrks; ///< Number of workers at start - Int_t fNumMergers; ///< Number of submergers - - TQueryResult(Int_t seqnum, const char *opt, TList *inlist, - Long64_t entries, Long64_t first, - const char *selec); - - void AddInput(TObject *obj); - void AddLogLine(const char *logline); - TQueryResult *CloneInfo(); - virtual void RecordEnd(EQueryStatus status, TList *outlist = nullptr); - void SaveSelector(const char *selec); - void SetArchived(const char *archfile); - virtual void SetFinalized() { fFinalized = true; } - virtual void SetInputList(TList *in, bool adopt = true); - virtual void SetOutputList(TList *out, bool adopt = true); - virtual void SetProcessInfo(Long64_t ent, Float_t cpu = 0., - Long64_t siz = -1, - Float_t inittime = 0., Float_t proctime = 0.); - void SetPrepTime(Float_t preptime) { fPrepTime = preptime; } - void SetMergeTime(Float_t mergetime) { fMergeTime = mergetime; } - void SetRecvTime(Float_t recvtime) { fRecvTime = recvtime; } - void SetTermTime(Float_t termtime) { fTermTime = termtime; } - void SetNumMergers(Int_t nmergers) { fNumMergers = nmergers; } - -public: - TQueryResult() : fSeqNum(-1), fDraw(false), fStatus(kSubmitted), fUsedCPU(0.), - fInputList(nullptr), fEntries(-1), fFirst(-1), fBytes(0), - fLogFile(nullptr), fSelecHdr(nullptr), fSelecImp(nullptr), - fLibList("-"), fOutputList(nullptr), - fFinalized(false), fArchived(false), fPrepTime(0.), - fInitTime(0.), fProcTime(0.), fMergeTime(0.), - fRecvTime(-1), fTermTime(0.), fNumWrks(-1), fNumMergers(-1) { } - ~TQueryResult() override; - - void Browse(TBrowser *b = nullptr) override; - - Int_t GetSeqNum() const { return fSeqNum; } - EQueryStatus GetStatus() const { return fStatus; } - TDatime GetStartTime() const { return fStart; } - TDatime GetEndTime() const { return fEnd; } - const char *GetOptions() const { return fOptions; } - TList *GetInputList() { return fInputList; } - TObject *GetInputObject(const char *classname) const; - Long64_t GetEntries() const { return fEntries; } - Long64_t GetFirst() const { return fFirst; } - Long64_t GetBytes() const { return fBytes; } - Float_t GetUsedCPU() const { return fUsedCPU; } - TMacro *GetLogFile() const { return fLogFile; } - TMacro *GetSelecHdr() const { return fSelecHdr; } - TMacro *GetSelecImp() const { return fSelecImp; } - const char *GetLibList() const { return fLibList; } - const char *GetParList() const { return fParList; } - TList *GetOutputList() { return fOutputList; } - const char *GetResultFile() const { return fResultFile; } - Float_t GetPrepTime() const { return fPrepTime; } - Float_t GetInitTime() const { return fInitTime; } - Float_t GetProcTime() const { return fProcTime; } - Float_t GetMergeTime() const { return fMergeTime; } - Float_t GetRecvTime() const { return fRecvTime; } - Float_t GetTermTime() const { return fTermTime; } - Int_t GetNumWrks() const { return fNumWrks; } - Int_t GetNumMergers() const { return fNumMergers; } - - bool IsArchived() const { return fArchived; } - virtual bool IsDone() const { return (fStatus > kRunning); } - bool IsDraw() const { return fDraw; } - bool IsFinalized() const { return fFinalized; } - - bool Matches(const char *ref); - - void Print(Option_t *opt = "") const override; - - ClassDefOverride(TQueryResult,5) //Class describing a query -}; - -inline bool operator!=(const TQueryResult &qr1, const TQueryResult &qr2) - { return !(qr1 == qr2); } - -#endif diff --git a/tree/tree/inc/TSelectorScalar.h b/tree/tree/inc/TSelectorScalar.h index 6bf6936842256..bb90ff994fab4 100644 --- a/tree/tree/inc/TSelectorScalar.h +++ b/tree/tree/inc/TSelectorScalar.h @@ -18,9 +18,7 @@ // TSelectorScalar // // // // Named scalar type, based on Long64_t, streamable, storable and // -// mergeable. Ideally to be used in tree selectors in the PROOF // -// environment due to its merge functionality which allows a single // -// merged value to be returned to the user. // +// mergeable. // // // ////////////////////////////////////////////////////////////////////////// diff --git a/tree/tree/src/TChain.cxx b/tree/tree/src/TChain.cxx index 764cf36dc10f4..46c8581837d56 100644 --- a/tree/tree/src/TChain.cxx +++ b/tree/tree/src/TChain.cxx @@ -71,7 +71,7 @@ ClassImp(TChain); TChain::TChain(Mode mode) : TTree(), fTreeOffsetLen(100), fNtrees(0), fTreeNumber(-1), fTreeOffset(nullptr), fCanDeleteRefs(false), fTree(nullptr), - fFile(nullptr), fFiles(nullptr), fStatus(nullptr), fProofChain(nullptr), fGlobalRegistration(mode == kWithGlobalRegistration) + fFile(nullptr), fFiles(nullptr), fStatus(nullptr), fGlobalRegistration(mode == kWithGlobalRegistration) { fTreeOffset = new Long64_t[fTreeOffsetLen]; fFiles = new TObjArray(fTreeOffsetLen); @@ -83,10 +83,6 @@ TChain::TChain(Mode mode) fFile = nullptr; fDirectory = nullptr; - // Reset PROOF-related bits - ResetBit(kProofUptodate); - ResetBit(kProofLite); - if (fGlobalRegistration) { // Add to the global list gROOT->GetListOfDataSets()->Add(this); @@ -139,7 +135,7 @@ TChain::TChain(Mode mode) TChain::TChain(const char *name, const char *title, Mode mode) : TTree(name, title, /*splitlevel*/ 99, nullptr), fTreeOffsetLen(100), fNtrees(0), fTreeNumber(-1), fTreeOffset(nullptr), - fCanDeleteRefs(false), fTree(nullptr), fFile(nullptr), fFiles(nullptr), fStatus(nullptr), fProofChain(nullptr), + fCanDeleteRefs(false), fTree(nullptr), fFile(nullptr), fFiles(nullptr), fStatus(nullptr), fGlobalRegistration(mode == kWithGlobalRegistration) { // @@ -151,10 +147,6 @@ TChain::TChain(const char *name, const char *title, Mode mode) fTreeOffset[0] = 0; fFile = nullptr; - // Reset PROOF-related bits - ResetBit(kProofUptodate); - ResetBit(kProofLite); - if (fGlobalRegistration) { R__LOCKGUARD(gROOTMutex); @@ -179,7 +171,6 @@ TChain::~TChain() gROOT->GetListOfCleanups()->Remove(this); } - SafeDelete(fProofChain); fStatus->Delete(); delete fStatus; fStatus = nullptr; @@ -249,9 +240,6 @@ Int_t TChain::Add(TChain* chain) fFiles->Add(newelement); nf++; } - if (fProofChain) - // This updates the proxy chain when we will really use PROOF - ResetBit(kProofUptodate); return nf; } @@ -422,10 +410,6 @@ Int_t TChain::Add(const char *name, Long64_t nentries /* = TTree::kMaxEntries */ } } - if (fProofChain) - // This updates the proxy chain when we will really use PROOF - ResetBit(kProofUptodate); - return nf; } @@ -577,9 +561,6 @@ Int_t TChain::AddFile(const char* name, Long64_t nentries /* = TTree::kMaxEntrie } delete [] filename; - if (fProofChain) - // This updates the proxy chain when we will really use PROOF - ResetBit(kProofUptodate); return 1; } @@ -624,10 +605,6 @@ Int_t TChain::AddFileInfoList(TCollection* filelist, Long64_t nfiles /* = TTree: if (cnt >= nfiles) break; } - if (fProofChain) { - // This updates the proxy chain when we will really use PROOF - ResetBit(kProofUptodate); - } return 1; } @@ -690,10 +667,6 @@ TFriendElement* TChain::AddFriend(const char* chain, const char* dummy /* = "" * fFriends->Add(fe); - if (fProofChain) - // This updates the proxy chain when we will really use PROOF - ResetBit(kProofUptodate); - // We need to invalidate the loading of the current tree because its list // of real friends is now obsolete. It is repairable only from LoadTree. InvalidateCurrentTree(); @@ -717,10 +690,6 @@ TFriendElement* TChain::AddFriend(const char* chain, TFile* dummy) fFriends->Add(fe); - if (fProofChain) - // This updates the proxy chain when we will really use PROOF - ResetBit(kProofUptodate); - // We need to invalidate the loading of the current tree because its list // of real friend is now obsolete. It is repairable only from LoadTree InvalidateCurrentTree(); @@ -744,10 +713,6 @@ TFriendElement* TChain::AddFriend(TTree* chain, const char* alias, bool /* warn fFriends->Add(fe); - if (fProofChain) - // This updates the proxy chain when we will really use PROOF - ResetBit(kProofUptodate); - // We need to invalidate the loading of the current tree because its list // of real friend is now obsolete. It is repairable only from LoadTree InvalidateCurrentTree(); @@ -819,15 +784,6 @@ void TChain::DirectoryAutoAdd(TDirectory * /* dir */) Long64_t TChain::Draw(const char* varexp, const TCut& selection, Option_t* option, Long64_t nentries, Long64_t firstentry) { - if (fProofChain) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - fProofChain->SetEventList(fEventList); - fProofChain->SetEntryList(fEntryList); - return fProofChain->Draw(varexp, selection, option, nentries, firstentry); - } - return TChain::Draw(varexp, selection.GetTitle(), option, nentries, firstentry); } @@ -839,14 +795,6 @@ Long64_t TChain::Draw(const char* varexp, const TCut& selection, Long64_t TChain::Draw(const char* varexp, const char* selection, Option_t* option,Long64_t nentries, Long64_t firstentry) { - if (fProofChain) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - fProofChain->SetEventList(fEventList); - fProofChain->SetEntryList(fEntryList); - return fProofChain->Draw(varexp, selection, option, nentries, firstentry); - } GetPlayer(); if (LoadTree(firstentry) < 0) return 0; return TTree::Draw(varexp,selection,option,nentries,firstentry); @@ -857,12 +805,6 @@ Long64_t TChain::Draw(const char* varexp, const char* selection, TBranch* TChain::FindBranch(const char* branchname) { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - return fProofChain->FindBranch(branchname); - } if (fTree) { return fTree->FindBranch(branchname); } @@ -878,12 +820,6 @@ TBranch* TChain::FindBranch(const char* branchname) TLeaf* TChain::FindLeaf(const char* searchname) { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - return fProofChain->FindLeaf(searchname); - } if (fTree) { return fTree->FindLeaf(searchname); } @@ -918,12 +854,6 @@ const char* TChain::GetAlias(const char* aliasName) const TBranch* TChain::GetBranch(const char* name) { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - return fProofChain->GetBranch(name); - } if (fTree) { return fTree->GetBranch(name); } @@ -939,13 +869,6 @@ TBranch* TChain::GetBranch(const char* name) bool TChain::GetBranchStatus(const char* branchname) const { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - Warning("GetBranchStatus", "PROOF proxy not up-to-date:" - " run TChain::SetProof(true, true) first"); - return fProofChain->GetBranchStatus(branchname); - } return TTree::GetBranchStatus(branchname); } @@ -977,13 +900,6 @@ Long64_t TChain::GetChainEntryNumber(Long64_t entry) const Long64_t TChain::GetEntries() const { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - Warning("GetEntries", "PROOF proxy not up-to-date:" - " run TChain::SetProof(true, true) first"); - return fProofChain->GetEntries(); - } if (fEntries == TTree::kMaxEntries) { // If the following is true, we are within a recursion about friend, // and `LoadTree` will be no-op. @@ -1088,12 +1004,6 @@ TFile* TChain::GetFile() const TLeaf* TChain::GetLeaf(const char* branchname, const char *leafname) { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - return fProofChain->GetLeaf(branchname, leafname); - } if (fTree) { return fTree->GetLeaf(branchname, leafname); } @@ -1109,12 +1019,6 @@ TLeaf* TChain::GetLeaf(const char* branchname, const char *leafname) TLeaf* TChain::GetLeaf(const char* name) { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - return fProofChain->GetLeaf(name); - } if (fTree) { return fTree->GetLeaf(name); } @@ -1135,12 +1039,6 @@ TLeaf* TChain::GetLeaf(const char* name) TObjArray* TChain::GetListOfBranches() { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - return fProofChain->GetListOfBranches(); - } if (fTree) { return fTree->GetListOfBranches(); } @@ -1158,12 +1056,6 @@ TObjArray* TChain::GetListOfBranches() TObjArray* TChain::GetListOfLeaves() { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - return fProofChain->GetListOfLeaves(); - } if (fTree) { return fTree->GetListOfLeaves(); } @@ -1266,13 +1158,6 @@ Int_t TChain::GetNbranches() Long64_t TChain::GetReadEntry() const { - if (fProofChain && !(fProofChain->TestBit(kProofLite))) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - Warning("GetBranchStatus", "PROOF proxy not up-to-date:" - " run TChain::SetProof(true, true) first"); - return fProofChain->GetReadEntry(); - } return TTree::GetReadEntry(); } @@ -2326,15 +2211,6 @@ void TChain::Print(Option_t *option) const Long64_t TChain::Process(const char *filename, Option_t *option, Long64_t nentries, Long64_t firstentry) { - if (fProofChain) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - fProofChain->SetEventList(fEventList); - fProofChain->SetEntryList(fEntryList); - return fProofChain->Process(filename, option, nentries, firstentry); - } - if (LoadTree(firstentry) < 0) { return 0; } @@ -2348,15 +2224,6 @@ Long64_t TChain::Process(const char *filename, Option_t *option, Long64_t nentri Long64_t TChain::Process(TSelector* selector, Option_t* option, Long64_t nentries, Long64_t firstentry) { - if (fProofChain) { - // Make sure the element list is up to date - if (!TestBit(kProofUptodate)) - SetProof(true, true); - fProofChain->SetEventList(fEventList); - fProofChain->SetEntryList(fEntryList); - return fProofChain->Process(selector, option, nentries, firstentry); - } - return TTree::Process(selector, option, nentries, firstentry); } @@ -2394,10 +2261,6 @@ void TChain::RemoveFriend(TTree* oldFriend) TTree::RemoveFriend(oldFriend); - if (fProofChain) - // This updates the proxy chain when we will really use PROOF - ResetBit(kProofUptodate); - // We need to invalidate the loading of the current tree because its list // of real friends is now obsolete. It is repairable only from LoadTree. InvalidateCurrentTree(); @@ -2816,13 +2679,6 @@ void TChain::SetEntryList(TEntryList *elist, Option_t *opt) fEntryList = elist; return; } - if (fProofChain){ - //for processing on proof, event list and entry list can't be - //set at the same time. - fEventList = nullptr; - fEntryList = elist; - return; - } Int_t ne = fFiles->GetEntries(); Int_t listfound=0; @@ -2999,23 +2855,6 @@ void TChain::SetEventList(TEventList *evlist) return; } - if(fProofChain) { - //on proof, fEventList and fEntryList shouldn't be set at the same time - if (fEntryList){ - //check, if the chain is the owner of the previous entry list - //(it happens, if the previous entry list was created from a user-defined - //TEventList in SetEventList() function) - if (fEntryList->TestBit(kCanDelete)){ - TEntryList *tmp = fEntryList; - fEntryList = nullptr; // Avoid problem with RecursiveRemove. - delete tmp; - } else { - fEntryList = nullptr; - } - } - return; - } - char enlistname[100]; snprintf(enlistname,100, "%s_%s", evlist->GetName(), "entrylist"); TEntryList *enlist = new TEntryList(enlistname, evlist->GetTitle()); @@ -3087,45 +2926,6 @@ void TChain::SetPacketSize(Int_t size) } } -//////////////////////////////////////////////////////////////////////////////// -/// Enable/Disable PROOF processing on the current default Proof (gProof). -/// -/// "Draw" and "Processed" commands will be handled by PROOF. -/// The refresh and gettreeheader are meaningful only if on == true. -/// If refresh is true the underlying fProofChain (chain proxy) is always -/// rebuilt (even if already existing). -/// If gettreeheader is true the header of the tree will be read from the -/// PROOF cluster: this is only needed for browsing and should be used with -/// care because it may take a long time to execute. - -void TChain::SetProof(bool on, bool refresh, bool gettreeheader) -{ - if (!on) { - // Disable - SafeDelete(fProofChain); - // Reset related bit - ResetBit(kProofUptodate); - } else { - if (fProofChain && !refresh && - (!gettreeheader || (gettreeheader && fProofChain->GetTree()))) { - return; - } - SafeDelete(fProofChain); - ResetBit(kProofUptodate); - - // Make instance of TChainProof via the plugin manager - TPluginHandler *h; - if ((h = gROOT->GetPluginManager()->FindHandler("TChain", "proof"))) { - if (h->LoadPlugin() == -1) - return; - if (!(fProofChain = reinterpret_cast(h->ExecPlugin(2, this, gettreeheader)))) - Error("SetProof", "creation of TProofChain failed"); - // Set related bits - SetBit(kProofUptodate); - } - } -} - //////////////////////////////////////////////////////////////////////////////// /// Set chain weight. /// diff --git a/tree/tree/src/TQueryResult.cxx b/tree/tree/src/TQueryResult.cxx deleted file mode 100644 index 70d2a77d2fc4b..0000000000000 --- a/tree/tree/src/TQueryResult.cxx +++ /dev/null @@ -1,528 +0,0 @@ -// @(#)root/tree:$Id$ -// Author: G Ganis Sep 2005 - -/************************************************************************* - * Copyright (C) 1995-2005, Rene Brun and Fons Rademakers. * - * All rights reserved. * - * * - * For the licensing terms see $ROOTSYS/LICENSE. * - * For the list of contributors see $ROOTSYS/README/CREDITS. * - *************************************************************************/ - -/** \class TQueryResult -\ingroup tree - -A container class for query results. -*/ - -#include - -#include "strlcpy.h" -#include "TBrowser.h" -#include "TEventList.h" -#include "TQueryResult.h" -#include "TRegexp.h" -#include "TROOT.h" -#include "TMacro.h" -#include "TMath.h" -#include "TSelector.h" -#include "TSystem.h" -#include "TTimeStamp.h" - -ClassImp(TQueryResult); - -//////////////////////////////////////////////////////////////////////////////// -/// Main constructor. - -TQueryResult::TQueryResult(Int_t seqnum, const char *opt, TList *inlist, - Long64_t entries, Long64_t first, const char *selec) - : fSeqNum(seqnum), fStatus(kSubmitted), fUsedCPU(0.), fOptions(opt), - fEntries(entries), fFirst(first), - fBytes(0), fParList("-"), fOutputList(nullptr), - fFinalized(false), fArchived(false), fResultFile("-"), - fPrepTime(0.), fInitTime(0.), fProcTime(0.), fMergeTime(0.), - fRecvTime(-1), fTermTime(-1), fNumWrks(-1), fNumMergers(-1) -{ - // Name and unique title - SetName(TString::Format("q%d", fSeqNum)); - SetTitle(TString::Format("session-localhost-%ld-%d", - (Long_t)TTimeStamp().GetSec(), gSystem->GetPid())); - - // Start time - fStart.Set(); - fEnd.Set(fStart.Convert()-1); - - // Save input list - fInputList = nullptr; - if (inlist) { - fInputList = (TList *) (inlist->Clone()); - fInputList->SetOwner(); - } - - // Log file - fLogFile = new TMacro("LogFile"); - - // Selector files - fDraw = selec ? TSelector::IsStandardDraw(selec) : false; - if (fDraw) { - // The input list should contain info about the variables and - // selection cuts: save them into the macro title - TString varsel; - if (fInputList) { - TIter nxo(fInputList); - TObject *o = nullptr; - while ((o = nxo())) { - if (!strcmp(o->GetName(),"varexp")) { - varsel = o->GetTitle(); - Int_t iht = varsel.Index(">>htemp"); - if (iht > -1) - varsel.Remove(iht); - varsel.Form("\"%s\";", varsel.Data()); - } - if (!strcmp(o->GetName(),"selection")) - varsel += TString::Format("\"%s\"", o->GetTitle()); - } - if (gDebug > 0) - Info("TQueryResult","selec: %s, varsel: %s", selec, varsel.Data()); - // Log notification also in the instance - fLogFile->AddLine(TString::Format("TQueryResult: selec: %s, varsel: %s", - selec, varsel.Data())); - } - // Standard draw action: save only the name - fSelecImp = new TMacro(selec, varsel); - fSelecHdr = nullptr; - } else { - // Save selector file - fSelecHdr = new TMacro; - fSelecImp = new TMacro; - SaveSelector(selec); - } - - // List of libraries loaded at creation - const char *pl = gSystem->GetLibraries(); - fLibList = (pl && (strlen(pl) > 0)) ? pl : "-"; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Destructor. - -TQueryResult::~TQueryResult() -{ - SafeDelete(fInputList); - SafeDelete(fOutputList); - SafeDelete(fLogFile); - SafeDelete(fSelecImp); - SafeDelete(fSelecHdr); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return an instance of TQueryResult containing only the local -/// info fields, i.e. no outputlist, liblist, dset, selectors, etc.. -/// Used for fast retrieve of information about existing queries -/// and their status. - -TQueryResult *TQueryResult::CloneInfo() -{ - // Create instance - TQueryResult *qr = new TQueryResult(fSeqNum, fOptions, nullptr, fEntries, - fFirst, nullptr); - - // Correct fields - qr->fStatus = fStatus; - qr->fStart.Set(fStart.Convert()); - qr->fEnd.Set(fEnd.Convert()); - qr->fUsedCPU = fUsedCPU; - qr->fEntries = fEntries; - qr->fFirst = fFirst; - qr->fBytes = fBytes; - qr->fParList = fParList; - qr->fResultFile = fResultFile; - qr->fArchived = fArchived; - qr->fPrepTime = fPrepTime; - qr->fInitTime = fInitTime; - qr->fProcTime = fProcTime; - qr->fMergeTime = fMergeTime; - qr->fRecvTime = fRecvTime; - qr->fTermTime = fTermTime; - qr->fNumWrks = fNumWrks; - qr->fNumMergers = fNumMergers; - - qr->fSelecHdr = nullptr; - if (GetSelecHdr()) { - qr->fSelecHdr = new TMacro(); - qr->fSelecHdr->SetName(GetSelecHdr()->GetName()); - qr->fSelecHdr->SetTitle(GetSelecHdr()->GetTitle()); - } - qr->fSelecImp = nullptr; - if (GetSelecImp()) { - qr->fSelecImp = new TMacro(); - qr->fSelecImp->SetName(GetSelecImp()->GetName()); - qr->fSelecImp->SetTitle(GetSelecImp()->GetTitle()); - } - - // Name and title - qr->SetName(GetName()); - qr->SetTitle(GetTitle()); - - return qr; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Save the selector header and implementation into the dedicated -/// TMacro instances. The header is searched for in the same directory -/// of the implementation file. - -void TQueryResult::SaveSelector(const char *selector) -{ - if (!selector) - return; - - // Separate out aclic chars - TString selec = selector; - TString aclicMode; - TString arguments; - TString io; - selec = gSystem->SplitAclicMode(selec, aclicMode, arguments, io); - - // Store aclic options, if any - if (aclicMode.Length() > 0) - fOptions += TString::Format("#%s", aclicMode.Data()); - - // If the selector is in a precompiled shared lib (e.g. in a PAR) - // we just save the name - TString selname = gSystem->BaseName(selec); - fSelecImp->SetName(selname); - Int_t idx = selname.Index("."); - if (idx < 0) { - // Notify - if (gDebug > 0) - Info("SaveSelector", "precompiled selector: just save the name"); - fSelecImp->SetTitle(selname); - } else { - // We locate the file and save it in compressed form - if (idx > -1) - selname.Remove(idx); - fSelecImp->SetTitle(selname); - - // Locate the implementation file - char *selc = gSystem->Which(TROOT::GetMacroPath(), selec, kReadPermission); - if (!selc) { - if (gDebug > 0) - Warning("SaveSelector", - "could not locate selector implementation file (%s)", selec.Data()); - return; - } - - // Fill the TMacro instance - fSelecImp->ReadFile(selc); - fSelecImp->SetName(gSystem->BaseName(selc)); - - // Locate the included header file - char *p = (char *) strrchr(selc,'.'); - if (p) { - strlcpy(p+1,"h",strlen(p)); - } else { - if (gDebug > 0) - Warning("SaveSelector", - "bad formatted name (%s): could not build header file name", selc); - } - if (!(gSystem->AccessPathName(selc, kReadPermission))) { - fSelecHdr->ReadFile(selc); - fSelecHdr->SetName(gSystem->BaseName(selc)); - fSelecHdr->SetTitle(selname); - } else { - if (gDebug > 0) - Warning("SaveSelector", - "could not locate selector header file (%s)", selc); - } - - delete[] selc; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// End of query settings. - -void TQueryResult::RecordEnd(EQueryStatus status, TList *outlist) -{ - // End time - fEnd.Set(); - - // Status - fStatus = (status < kAborted || status > kCompleted) ? kAborted : status; - - // Clone the results - if (outlist && fOutputList != outlist) { - if (fOutputList) { - fOutputList->Delete(); - SafeDelete(fOutputList); - } - if ((fOutputList = (TList *) (outlist->Clone()))) { - fOutputList->SetOwner(); - Info("RecordEnd", "output list cloned successfully!"); - } else { - Warning("RecordEnd", "unable to clone output list!!!"); - } - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set processing info. - -void TQueryResult::SetProcessInfo(Long64_t ent, Float_t cpu, Long64_t bytes, - Float_t init, Float_t proc) -{ - fEntries = (ent > 0) ? ent : fEntries; - fUsedCPU = (cpu > 0.) ? cpu : fUsedCPU; - fBytes = (bytes > 0.) ? bytes : fBytes; - fInitTime = (init > 0.) ? init : fInitTime; - fProcTime = (proc > 0.) ? proc : fProcTime; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Fill log file. - -void TQueryResult::AddLogLine(const char *logline) -{ - if (logline) - fLogFile->AddLine(logline); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Add obj to the input list - -void TQueryResult::AddInput(TObject *obj) -{ - if (fInputList && obj) - fInputList->Add(obj); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set (or update) query in archived state. - -void TQueryResult::SetArchived(const char *archfile) -{ - if (IsDone()) { - fArchived = true; - if (archfile && (strlen(archfile) > 0)) - fResultFile = archfile; - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Print query content. Use opt = "F" for a full listing. - -void TQueryResult::Print(Option_t *opt) const -{ - // Attention: the list must match EQueryStatus - const char *qst[] = { - "aborted ", "submitted", "running ", "stopped ", "completed" - }; - - // Status label - Int_t st = (fStatus > 0 && fStatus <= kCompleted) ? fStatus : 0; - - // Range label - Long64_t last = (fEntries > -1) ? fFirst+fEntries-1 : -1; - - // Option - bool full = ((strchr(opt,'F') || strchr(opt,'f'))) ? true : false; - - // Query number to be printed - Int_t qry = fSeqNum; - TString qn = opt; - TRegexp re("N.*N"); - Int_t i1 = qn.Index(re); - if (i1 != kNPOS) { - qn.Remove(0, i1+1); - qn.Remove(qn.Index("N")); - qry = qn.Atoi(); - } - - // Print separator if full dump - if (full) Printf("+++"); - - TString range; - if (!full && (last > -1)) - range.Form("evts:%lld-%lld", fFirst, last); - - // Print header - if (!fDraw) { - const char *fin = fFinalized ? "finalized" : qst[st]; - const char *arc = fArchived ? "(A)" : ""; - Printf("+++ #:%d ref:\"%s:%s\" sel:%s %9s%s %s", - qry, GetTitle(), GetName(), fSelecImp->GetTitle(), fin, arc, - range.Data()); - } else { - Printf("+++ #:%d ref:\"%s:%s\" varsel:%s %s", - qry, GetTitle(), GetName(), fSelecImp->GetTitle(), - range.Data()); - } - - // We are done, if not full dump - if (!full) return; - - // Time information - Float_t elapsed = (fProcTime > 0.) ? fProcTime - : (Float_t)(fEnd.Convert() - fStart.Convert()); - Printf("+++ started: %s", fStart.AsString()); - if (fPrepTime > 0.) - Printf("+++ prepare: %.3f sec", fPrepTime); - Printf("+++ init: %.3f sec", fInitTime); - Printf("+++ process: %.3f sec (CPU time: %.1f sec)", elapsed, fUsedCPU); - if (fNumMergers > 0) { - Printf("+++ merge: %.3f sec (%d mergers)", fMergeTime, fNumMergers); - } else { - Printf("+++ merge: %.3f sec ", fMergeTime); - } - if (fRecvTime > 0.) - Printf("+++ transfer: %.3f sec", fRecvTime); - if (fTermTime > 0.) - Printf("+++ terminate: %.3f sec", fTermTime); - - // Number of events processed, rate, size - Double_t rate = 0.0; - if (fEntries > -1 && elapsed > 0) - rate = fEntries / (Double_t)elapsed ; - Float_t size = ((Float_t)fBytes) / TMath::Power(2.,20.); - Printf("+++ processed: %lld events (size: %.3f MBs)", fEntries, size); - Printf("+++ rate: %.1f evts/sec", rate); - - Printf("+++ # workers: %d ", fNumWrks); - - // Package information - if (fParList.Length() > 1) - Printf("+++ packages: %s", fParList.Data()); - - // Result information - TString res = fResultFile; - if (!fArchived) { - Int_t dq = res.Index("queries"); - if (dq > -1) { - res.Remove(0,res.Index("queries")); - res.Insert(0,"/"); - } - if (res.BeginsWith("-")) { - res = (fStatus == kAborted) ? "not available" : "sent to client"; - } - } - if (res.Length() > 1) - Printf("+++ results: %s", res.Data()); - - if (fOutputList && fOutputList->GetSize() > 0) - Printf("+++ outlist: %d objects", fOutputList->GetSize()); -} - -//////////////////////////////////////////////////////////////////////////////// -/// To support browsing of the results. - -void TQueryResult::Browse(TBrowser *b) -{ - if (fOutputList) - b->Add(fOutputList, fOutputList->Class(), "OutputList"); -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set / change the input list. -/// The flag 'adopt' determines whether the list is adopted (default) -/// or cloned. If adopted, object ownership is transferred to this object. -/// The internal fInputList will always be owner of its objects. - -void TQueryResult::SetInputList(TList *in, bool adopt) -{ - if (!in || in != fInputList) - SafeDelete(fInputList); - - if (in && in != fInputList) { - if (!adopt) { - fInputList = (TList *) (in->Clone()); - } else { - fInputList = new TList; - TIter nxi(in); - TObject *o = nullptr; - while ((o = nxi())) - fInputList->Add(o); - in->SetOwner(false); - } - fInputList->SetOwner(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Set / change the output list. -/// The flag 'adopt' determines whether the list is adopted (default) -/// or cloned. If adopted, object ownership is transferred to this object. -/// The internal fOutputList will always be owner of its objects. - -void TQueryResult::SetOutputList(TList *out, bool adopt) -{ - if (!out) { - SafeDelete(fOutputList); - return; - } - - if (out && out != fOutputList) { - TObject *o = nullptr; - if (fOutputList) { - TIter nxoo(fOutputList); - while ((o = nxoo())) { - if (out->FindObject(o)) fOutputList->Remove(o); - } - SafeDelete(fOutputList); - } - if (!adopt) { - fOutputList = (TList *) (out->Clone()); - } else { - fOutputList = new TList; - TIter nxo(out); - o = nullptr; - while ((o = nxo())) - fOutputList->Add(o); - out->SetOwner(false); - } - fOutputList->SetOwner(); - } -} - -//////////////////////////////////////////////////////////////////////////////// -/// Compare two query result instances for equality. -/// Session name and query number are compared. - -bool operator==(const TQueryResult &qr1, const TQueryResult &qr2) -{ - if (!strcmp(qr1.GetTitle(), qr2.GetTitle())) - if (qr1.GetSeqNum() == qr2.GetSeqNum()) - return true; - return false; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return TRUE if reference ref matches. - -bool TQueryResult::Matches(const char *ref) -{ - TString lref; lref.Form("%s:%s", GetTitle(), GetName()); - - if (lref == ref) - return true; - - return false; -} - -//////////////////////////////////////////////////////////////////////////////// -/// Return first instance of class 'classname' in the input list. -/// Useful to access TDSet, TEventList, ... - -TObject *TQueryResult::GetInputObject(const char *classname) const -{ - TObject *o = nullptr; - if (classname && fInputList) { - TIter nxi(fInputList); - while ((o = nxi())) - if (!strncmp(o->ClassName(), classname, strlen(classname))) - return o; - } - - // Not found - return o; -} diff --git a/tree/tree/src/TSelector.cxx b/tree/tree/src/TSelector.cxx index 6a3187678493b..c5a8e274729b9 100644 --- a/tree/tree/src/TSelector.cxx +++ b/tree/tree/src/TSelector.cxx @@ -20,14 +20,12 @@ It contains the following main methods: attached. - void TSelector::SlaveBegin(). Create e.g. histograms in this method. - This method is called (with or without PROOF) before looping on the - entries in the Tree. When using PROOF, this method is called on - each worker node. + This method is called before looping on the + entries in the Tree. - void TSelector::Begin(). Mostly for backward compatibility; use SlaveBegin() instead. Both methods are called before looping on the - entries in the Tree. When using PROOF, Begin() is called on the - client only. + entries in the Tree. - bool TSelector::Notify(). This method is called at the first entry of a new file in a chain. @@ -41,7 +39,6 @@ It contains the following main methods: next two functions in one, avoiding to have to maintain state in the class to communicate between these two functions. See WARNING below about entry. - This method is used by PROOF. - bool TSelector::ProcessCut(Long64_t entry). This method is called before processing entry. It is the user's responsibility to read @@ -55,14 +52,9 @@ It contains the following main methods: This method is obsolete, use Process(). See WARNING below about entry. -- void TSelector::SlaveTerminate(). This method is called at the end of - the loop on all PROOF worker nodes. In local mode this method is - called on the client too. - - void TSelector::Terminate(). This method is called at the end of - the loop on all entries. When using PROOF Terminate() is call on - the client only. Typically one performs the fits on the produced - histograms or write the histograms to file in this method. + the loop on all entries. Typically one performs the fits on the + produced histograms or write the histograms to file in this method. __WARNING when a selector is used with a TChain:__ @@ -110,7 +102,7 @@ TSelector::~TSelector() /// aborted and the processing will continue with the next file, if there /// is no next file then Process() will be aborted. Abort() can also be /// called from Begin(), SlaveBegin(), Init() and Notify(). After abort -/// the SlaveTerminate() and Terminate() are always called. The abort flag +/// the Terminate() method is always called. The abort flag /// can be checked in these methods using GetAbort(). void TSelector::Abort(const char *why, EAbort what) @@ -232,7 +224,7 @@ TSelector *TSelector::GetSelector(const char *filename) //////////////////////////////////////////////////////////////////////////////// /// Find out if this is a standard selection used for Draw actions -/// (either TSelectorDraw, TProofDraw or deriving from them). +/// (either TSelectorDraw or deriving from it). bool TSelector::IsStandardDraw(const char *selec) { @@ -249,8 +241,7 @@ bool TSelector::IsStandardDraw(const char *selec) stdselec = true; } else { TClass *cl = TClass::GetClass(selec); - if (cl && (cl->InheritsFrom("TProofDraw") || - cl->InheritsFrom("TSelectorDraw"))) + if (cl && (cl->InheritsFrom("TSelectorDraw"))) stdselec = true; } } @@ -333,13 +324,11 @@ void TSelector::ProcessFill(Long64_t /*entry*/) } //////////////////////////////////////////////////////////////////////////////// -/// The Process() function is called for each entry in the tree (or possibly -/// keyed object in the case of PROOF) to be processed. The entry argument -/// specifies which entry in the currently loaded tree is to be processed. +/// The Process() function is called for each entry in the tree to be processed. +/// The entry argument specifies which entry in the currently loaded +/// tree is to be processed. /// It can be passed to either t01::GetEntry() or TBranch::GetEntry() -/// to read either all or the required parts of the data. When processing -/// keyed objects with PROOF, the object is already loaded and is available -/// via the fObject pointer. +/// to read either all or the required parts of the data. /// /// This function should contain the "body" of the analysis. It can contain /// simple or elaborate selection criteria, run algorithms on the data diff --git a/tree/tree/src/TSelectorScalar.cxx b/tree/tree/src/TSelectorScalar.cxx index fe01606b030cb..3eeacd1f4030a 100644 --- a/tree/tree/src/TSelectorScalar.cxx +++ b/tree/tree/src/TSelectorScalar.cxx @@ -13,9 +13,7 @@ \ingroup tree Named scalar type, based on Long64_t, streamable, storable and -mergeable. Ideally to be used in tree selectors in the PROOF -environment due to its merge functionality which allows a single -merged value to be returned to the user. +mergeable. */ #include "TSelectorScalar.h" diff --git a/tree/tree/src/TTree.cxx b/tree/tree/src/TTree.cxx index 65d5dd0d19ecf..f5688887b6da3 100644 --- a/tree/tree/src/TTree.cxx +++ b/tree/tree/src/TTree.cxx @@ -7541,16 +7541,14 @@ void TTree::PrintCacheStats(Option_t* option) const /// filename must contain a valid class implementation derived from TSelector, /// where TSelector has the following member functions: /// -/// - `Begin()`: called every time a loop on the tree starts, -/// a convenient place to create your histograms. -/// - `SlaveBegin()`: called after Begin(), when on PROOF called only on the -/// slave servers. -/// - `Process()`: called for each event, in this function you decide what -/// to read and fill your histograms. -/// - `SlaveTerminate`: called at the end of the loop on the tree, when on PROOF -/// called only on the slave servers. -/// - `Terminate()`: called at the end of the loop on the tree, -/// a convenient place to draw/fit your histograms. +/// - `Begin()`: called every time a loop on the tree starts, +/// a convenient place to create your histograms. +/// - `SlaveBegin()`: called after Begin() +/// - `Process()`: called for each event, in this function you decide what +/// to read and fill your histograms. +/// - `SlaveTerminate()`: called at the end of the loop on the tree +/// - `Terminate()`: called at the end of the loop on the tree, +/// a convenient place to draw/fit your histograms. /// /// If filename is of the form file.C, the file will be interpreted. /// @@ -7620,16 +7618,14 @@ Long64_t TTree::Process(const char* filename, Option_t* option, Long64_t nentrie /// /// The TSelector class has the following member functions: /// -/// - `Begin()`: called every time a loop on the tree starts, -/// a convenient place to create your histograms. -/// - `SlaveBegin()`: called after Begin(), when on PROOF called only on the -/// slave servers. -/// - `Process()`: called for each event, in this function you decide what -/// to read and fill your histograms. -/// - `SlaveTerminate`: called at the end of the loop on the tree, when on PROOF -/// called only on the slave servers. -/// - `Terminate()`: called at the end of the loop on the tree, -/// a convenient place to draw/fit your histograms. +/// - `Begin()`: called every time a loop on the tree starts, +/// a convenient place to create your histograms. +/// - `SlaveBegin()`: called after Begin() +/// - `Process()`: called for each event, in this function you decide what +/// to read and fill your histograms. +/// - `SlaveTerminate()`: called at the end of the loop on the tree +/// - `Terminate()`: called at the end of the loop on the tree, +/// a convenient place to draw/fit your histograms. /// /// If the Tree (Chain) has an associated EventList, the loop is on the nentries /// of the EventList, starting at firstentry, otherwise the loop is on the diff --git a/tree/treeplayer/inc/TTreeDrawArgsParser.h b/tree/treeplayer/inc/TTreeDrawArgsParser.h index 2f75a99399531..317632a07ec83 100644 --- a/tree/treeplayer/inc/TTreeDrawArgsParser.h +++ b/tree/treeplayer/inc/TTreeDrawArgsParser.h @@ -90,7 +90,6 @@ class TTreeDrawArgsParser : public TObject { Double_t GetIfSpecified(Int_t num, Double_t def) const; Int_t GetNoParameters() const { return fNoParameters; } Double_t GetParameter(int num) const; - TString GetProofSelectorName() const; TString const& GetObjectName() const { return fName; } TString GetObjectTitle() const; bool GetOptionSame() const { return fOptionSame; } diff --git a/tree/treeplayer/src/TSelectorEntries.cxx b/tree/treeplayer/src/TSelectorEntries.cxx index 2d3f0852a81fa..4f5b4b0d611c6 100644 --- a/tree/treeplayer/src/TSelectorEntries.cxx +++ b/tree/treeplayer/src/TSelectorEntries.cxx @@ -16,16 +16,14 @@ information on the TSelector framework see The following methods are defined in this file: - - Begin(): called every time a loop on the tree starts, - a convenient place to create your histograms. - - SlaveBegin(): called after Begin(), when on PROOF called only on the - slave servers. - - Process(): called for each event, in this function you decide what - to read and fill your histograms. - - SlaveTerminate: called at the end of the loop on the tree, when on PROOF - called only on the slave servers. - - Terminate(): called at the end of the loop on the tree, - a convenient place to draw/fit your histograms. + - Begin(): called every time a loop on the tree starts, + a convenient place to create your histograms. + - SlaveBegin(): called after Begin() + - Process(): called for each event, in this function you decide what + to read and fill your histograms. + - SlaveTerminate(): called at the end of the loop on the tree + - Terminate(): called at the end of the loop on the tree, + a convenient place to draw/fit your histograms. To use this file, try the following session on your Tree T: ~~~{.cpp} @@ -75,8 +73,6 @@ TSelectorEntries::~TSelectorEntries() //////////////////////////////////////////////////////////////////////////////// /// The Begin() function is called at the start of the query. -/// When running with PROOF Begin() is only called on the client. -/// The tree argument is deprecated (on PROOF 0 is passed). void TSelectorEntries::Begin(TTree *tree) { @@ -86,8 +82,6 @@ void TSelectorEntries::Begin(TTree *tree) //////////////////////////////////////////////////////////////////////////////// /// The SlaveBegin() function is called after the Begin() function. -/// When running with PROOF SlaveBegin() is called on each slave server. -/// The tree argument is deprecated (on PROOF 0 is passed). void TSelectorEntries::SlaveBegin(TTree *tree) { @@ -123,8 +117,6 @@ Int_t TSelectorEntries::GetEntry(Long64_t entry, Int_t getall) /// pointers of the tree will be set. /// It is normally not necessary to make changes to the generated /// code, but the routine can be extended by the user if needed. -/// Init() will be called many times when running on PROOF -/// (once per file to be processed). void TSelectorEntries::Init(TTree * /* tree */) { @@ -140,13 +132,11 @@ bool TSelectorEntries::Notify() } //////////////////////////////////////////////////////////////////////////////// -/// The Process() function is called for each entry in the tree (or possibly -/// keyed object in the case of PROOF) to be processed. The entry argument -/// specifies which entry in the currently loaded tree is to be processed. +/// The Process() function is called for each entry in the tree to be +/// processed. The entry argument specifies which entry in the +/// currently loaded tree is to be processed. /// It can be passed to either TSelectorEntries::GetEntry() or TBranch::GetEntry() -/// to read either all or the required parts of the data. When processing -/// keyed objects with PROOF, the object is already loaded and is available -/// via the fObject pointer. +/// to read either all or the required parts of the data. /// /// This function should contain the "body" of the analysis. It can contain /// simple or elaborate selection criteria, run algorithms on the data @@ -211,8 +201,6 @@ void TSelectorEntries::SetSelection(const char *selection) //////////////////////////////////////////////////////////////////////////////// /// The SlaveTerminate() function is called after all entries or objects -/// have been processed. When running with PROOF SlaveTerminate() is called -/// on each slave server. void TSelectorEntries::SlaveTerminate() { diff --git a/tree/treeplayer/src/TTreeDrawArgsParser.cxx b/tree/treeplayer/src/TTreeDrawArgsParser.cxx index a8424c820060a..fca686d8454cf 100644 --- a/tree/treeplayer/src/TTreeDrawArgsParser.cxx +++ b/tree/treeplayer/src/TTreeDrawArgsParser.cxx @@ -308,40 +308,6 @@ TTreeDrawArgsParser::EOutputType TTreeDrawArgsParser::DefineType() return kUNKNOWN; } -//////////////////////////////////////////////////////////////////////////////// -/// Returns appropriate TSelector class name for proof for the object that is to be drawn -/// assumes that Parse() method has been called before. - -TString TTreeDrawArgsParser::GetProofSelectorName() const -{ - switch (fOutputType) { - case kUNKNOWN: - return ""; - case kEVENTLIST: - return "TProofDrawEventList"; - case kENTRYLIST: - return "TProofDrawEntryList"; - case kPROFILE: - return "TProofDrawProfile"; - case kPROFILE2D: - return "TProofDrawProfile2D"; - case kGRAPH: - return "TProofDrawGraph"; - case kPOLYMARKER3D: - return "TProofDrawPolyMarker3D"; - case kLISTOFGRAPHS: - return "TProofDrawListOfGraphs"; - case kHISTOGRAM1D: - case kHISTOGRAM2D: - case kHISTOGRAM3D: - return "TProofDrawHist"; - case kLISTOFPOLYMARKERS3D: - return "TProofDrawListOfPolyMarkers3D"; - default: - return ""; - } -} - //////////////////////////////////////////////////////////////////////////////// /// returns *num*-th parameter from brackets in the expression /// in case of an error (wrong number) returns 0.0 diff --git a/tree/treeplayer/src/TTreePlayer.cxx b/tree/treeplayer/src/TTreePlayer.cxx index fa3ebbfbf9022..8b56200328259 100644 --- a/tree/treeplayer/src/TTreePlayer.cxx +++ b/tree/treeplayer/src/TTreePlayer.cxx @@ -1254,9 +1254,7 @@ Int_t TTreePlayer::MakeClass(const char *classname, const char *option) " // a new tree or chain. Typically here the branch addresses and branch\n" " // pointers of the tree will be set.\n" " // It is normally not necessary to make changes to the generated\n" - " // code, but the routine can be extended by the user if needed.\n" - " // Init() will be called many times when running on PROOF\n" - " // (once per file to be processed).\n\n"); + " // code, but the routine can be extended by the user if needed.\n\n"); if (mustInit.Last()) { TIter next(&mustInit); TObject *obj; @@ -1364,8 +1362,7 @@ Int_t TTreePlayer::MakeClass(const char *classname, const char *option) fprintf(fp,"bool %s::Notify()\n",cppClassName.Data()); fprintf(fp,"{\n"); fprintf(fp," // The Notify() function is called when a new file is opened. This\n" - " // can be either for a new TTree in a TChain or when when a new TTree\n" - " // is started when using PROOF. It is normally not necessary to make changes\n" + " // can be for a new TTree in a TChain. It is normally not necessary to make changes\n" " // to the generated code, but the routine can be extended by the\n" " // user if needed. The return value is currently not used.\n\n"); fprintf(fp," return true;\n"); @@ -1450,12 +1447,10 @@ Int_t TTreePlayer::MakeClass(const char *classname, const char *option) fprintf(fpc,"// The following methods are defined in this file:\n"); fprintf(fpc,"// Begin(): called every time a loop on the tree starts,\n"); fprintf(fpc,"// a convenient place to create your histograms.\n"); - fprintf(fpc,"// SlaveBegin(): called after Begin(), when on PROOF called only on the\n" - "// slave servers.\n"); + fprintf(fpc,"// SlaveBegin(): called after Begin()\n"); fprintf(fpc,"// Process(): called for each event, in this function you decide what\n"); fprintf(fpc,"// to read and fill your histograms.\n"); - fprintf(fpc,"// SlaveTerminate: called at the end of the loop on the tree, when on PROOF\n" - "// called only on the slave servers.\n"); + fprintf(fpc,"// SlaveTerminate: called at the end of the loop on the tree.\n"); fprintf(fpc,"// Terminate(): called at the end of the loop on the tree,\n"); fprintf(fpc,"// a convenient place to draw/fit your histograms.\n"); fprintf(fpc,"//\n"); @@ -1474,8 +1469,7 @@ Int_t TTreePlayer::MakeClass(const char *classname, const char *option) fprintf(fpc,"void %s::Begin(TTree * /*tree*/)\n",cppClassName.Data()); fprintf(fpc,"{\n"); fprintf(fpc," // The Begin() function is called at the start of the query.\n"); - fprintf(fpc," // When running with PROOF Begin() is only called on the client.\n"); - fprintf(fpc," // The tree argument is deprecated (on PROOF 0 is passed).\n"); + fprintf(fpc," // The tree argument is deprecated.\n"); fprintf(fpc,"\n"); fprintf(fpc," TString option = GetOption();\n"); fprintf(fpc,"\n"); @@ -1485,8 +1479,7 @@ Int_t TTreePlayer::MakeClass(const char *classname, const char *option) fprintf(fpc,"void %s::SlaveBegin(TTree * /*tree*/)\n",cppClassName.Data()); fprintf(fpc,"{\n"); fprintf(fpc," // The SlaveBegin() function is called after the Begin() function.\n"); - fprintf(fpc," // When running with PROOF SlaveBegin() is called on each slave server.\n"); - fprintf(fpc," // The tree argument is deprecated (on PROOF 0 is passed).\n"); + fprintf(fpc," // The tree argument is deprecated.\n"); fprintf(fpc,"\n"); fprintf(fpc," TString option = GetOption();\n"); fprintf(fpc,"\n"); @@ -1495,13 +1488,10 @@ Int_t TTreePlayer::MakeClass(const char *classname, const char *option) fprintf(fpc,"\n"); fprintf(fpc,"bool %s::Process(Long64_t entry)\n",cppClassName.Data()); fprintf(fpc,"{\n"); - fprintf(fpc," // The Process() function is called for each entry in the tree (or possibly\n" - " // keyed object in the case of PROOF) to be processed. The entry argument\n" + fprintf(fpc," // The Process() function is called for each entry in the tree to be processed. The entry argument\n" " // specifies which entry in the currently loaded tree is to be processed.\n" " // It can be passed to either %s::GetEntry() or TBranch::GetEntry()\n" - " // to read either all or the required parts of the data. When processing\n" - " // keyed objects with PROOF, the object is already loaded and is available\n" - " // via the fObject pointer.\n" + " // to read either all or the required parts of the data.\n" " //\n" " // This function should contain the \"body\" of the analysis. It can contain\n" " // simple or elaborate selection criteria, run algorithms on the data\n" @@ -1520,8 +1510,7 @@ Int_t TTreePlayer::MakeClass(const char *classname, const char *option) fprintf(fpc,"void %s::SlaveTerminate()\n",cppClassName.Data()); fprintf(fpc,"{\n"); fprintf(fpc," // The SlaveTerminate() function is called after all entries or objects\n" - " // have been processed. When running with PROOF SlaveTerminate() is called\n" - " // on each slave server."); + " // have been processed."); fprintf(fpc,"\n"); fprintf(fpc,"\n"); fprintf(fpc,"}\n"); @@ -2148,16 +2137,14 @@ TPrincipal *TTreePlayer::Principal(const char *varexp, const char *selection, Op /// filename must contain a valid class implementation derived from TSelector, /// where TSelector has the following member functions: /// -/// - Begin(): called every time a loop on the tree starts, -/// a convenient place to create your histograms. -/// - SlaveBegin(): called after Begin(), when on PROOF called only on the -/// slave servers. -/// - Process(): called for each event, in this function you decide what -/// to read and fill your histograms. -/// - SlaveTerminate: called at the end of the loop on the tree, when on PROOF -/// called only on the slave servers. -/// - Terminate(): called at the end of the loop on the tree, -/// a convenient place to draw/fit your histograms. +/// - Begin(): called every time a loop on the tree starts, +/// a convenient place to create your histograms. +/// - SlaveBegin(): called after Begin(). +/// - Process(): called for each event, in this function you decide what +/// to read and fill your histograms. +/// - SlaveTerminate(): called at the end of the loop on the tree +/// - Terminate(): called at the end of the loop on the tree, +/// a convenient place to draw/fit your histograms. /// /// If filename is of the form file.C, the file will be interpreted. /// If filename is of the form file.C++, the file file.C will be compiled @@ -2229,16 +2216,14 @@ Long64_t TTreePlayer::Process(const char *filename,Option_t *option, Long64_t ne /// /// The TSelector class has the following member functions: /// -/// - Begin(): called every time a loop on the tree starts, -/// a convenient place to create your histograms. -/// - SlaveBegin(): called after Begin(), when on PROOF called only on the -/// slave servers. -/// - Process(): called for each event, in this function you decide what -/// to read and fill your histograms. -/// - SlaveTerminate: called at the end of the loop on the tree, when on PROOF -/// called only on the slave servers. -/// - Terminate(): called at the end of the loop on the tree, -/// a convenient place to draw/fit your histograms. +/// - Begin(): called every time a loop on the tree starts, +/// a convenient place to create your histograms. +/// - SlaveBegin(): called after Begin() +/// - Process(): called for each event, in this function you decide what +/// to read and fill your histograms. +/// - SlaveTerminate(): called at the end of the loop on the tree +/// - Terminate(): called at the end of the loop on the tree, +/// a convenient place to draw/fit your histograms. /// /// If the Tree (Chain) has an associated EventList, the loop is on the nentries /// of the EventList, starting at firstentry, otherwise the loop is on the diff --git a/tree/treeplayer/src/TTreeProxyGenerator.cxx b/tree/treeplayer/src/TTreeProxyGenerator.cxx index 199b8aa7dbd2e..06987cfa00a76 100644 --- a/tree/treeplayer/src/TTreeProxyGenerator.cxx +++ b/tree/treeplayer/src/TTreeProxyGenerator.cxx @@ -1939,8 +1939,7 @@ namespace Internal { fprintf(hf,"inline void %s::Begin(TTree *tree)\n",classname.Data()); fprintf(hf,"{\n"); fprintf(hf," // The Begin() function is called at the start of the query.\n"); - fprintf(hf," // When running with PROOF Begin() is only called on the client.\n"); - fprintf(hf," // The tree argument is deprecated (on PROOF 0 is passed).\n"); + fprintf(hf," // The tree argument is deprecated.\n"); fprintf(hf,"\n"); fprintf(hf," TString option = GetOption();\n"); fprintf(hf," %s_Begin(tree);\n",scriptfunc.Data()); @@ -1952,8 +1951,7 @@ namespace Internal { fprintf(hf,"inline void %s::SlaveBegin(TTree *tree)\n",classname.Data()); fprintf(hf,"{\n"); fprintf(hf," // The SlaveBegin() function is called after the Begin() function.\n"); - fprintf(hf," // When running with PROOF SlaveBegin() is called on each slave server.\n"); - fprintf(hf," // The tree argument is deprecated (on PROOF 0 is passed).\n"); + fprintf(hf," // The tree argument is deprecated.\n"); fprintf(hf,"\n"); fprintf(hf," Init(tree);\n"); fprintf(hf,"\n"); @@ -1966,13 +1964,10 @@ namespace Internal { fprintf(hf,"inline bool %s::Process(Long64_t entry)\n",classname.Data()); fprintf(hf,"{\n"); - fprintf(hf," // The Process() function is called for each entry in the tree (or possibly\n" - " // keyed object in the case of PROOF) to be processed. The entry argument\n" - " // specifies which entry in the currently loaded tree is to be processed.\n" + fprintf(hf," // The Process() function is called for each entry in the tree to be processed.\n" + " // The entry argument specifies which entry in the currently loaded tree is to be processed.\n" " // It can be passed to either TTree::GetEntry() or TBranch::GetEntry()\n" - " // to read either all or the required parts of the data. When processing\n" - " // keyed objects with PROOF, the object is already loaded and is available\n" - " // via the fObject pointer.\n" + " // to read either all or the required parts of the data.\n" " //\n" " // This function should contain the \"body\" of the analysis. It can contain\n" " // simple or elaborate selection criteria, run algorithms on the data\n" @@ -2007,8 +2002,7 @@ namespace Internal { fprintf(hf,"inline void %s::SlaveTerminate()\n",classname.Data()); fprintf(hf,"{\n"); fprintf(hf," // The SlaveTerminate() function is called after all entries or objects\n" - " // have been processed. When running with PROOF SlaveTerminate() is called\n" - " // on each slave server."); + " // have been processed."); fprintf(hf,"\n"); fprintf(hf," %s_SlaveTerminate();\n",scriptfunc.Data()); fprintf(hf,"}\n\n"); diff --git a/tree/treeplayer/src/TTreeReaderGenerator.cxx b/tree/treeplayer/src/TTreeReaderGenerator.cxx index 120889b640b64..15831e4ee02ad 100644 --- a/tree/treeplayer/src/TTreeReaderGenerator.cxx +++ b/tree/treeplayer/src/TTreeReaderGenerator.cxx @@ -917,8 +917,6 @@ void )CODE" << cppClassName << R"CODE(::Init(TTree *tree) // a new tree or chain. Typically here the reader is initialized. // It is normally not necessary to make changes to the generated // code, but the routine can be extended by the user if needed. - // Init() will be called many times when running on PROOF - // (once per file to be processed). fReader.SetTree(tree); } @@ -926,8 +924,8 @@ void )CODE" << cppClassName << R"CODE(::Init(TTree *tree) bool )CODE" << cppClassName << R"CODE(::Notify() { // The Notify() function is called when a new file is opened. This - // can be either for a new TTree in a TChain or when when a new TTree - // is started when using PROOF. It is normally not necessary to make changes + // can be for a new TTree in a TChain. + // It is normally not necessary to make changes // to the generated code, but the routine can be extended by the // user if needed. The return value is currently not used. @@ -957,16 +955,14 @@ R"CODE(#define )CODE" << cppClassName << R"CODE(_cxx // The following methods are defined in this file: -// Begin(): called every time a loop on the tree starts, -// a convenient place to create your histograms. -// SlaveBegin(): called after Begin(), when on PROOF called only on the -// slave servers. -// Process(): called for each event, in this function you decide what -// to read and fill your histograms. -// SlaveTerminate: called at the end of the loop on the tree, when on PROOF -// called only on the slave servers. -// Terminate(): called at the end of the loop on the tree, -// a convenient place to draw/fit your histograms. +// Begin(): called every time a loop on the tree starts, +// a convenient place to create your histograms. +// SlaveBegin(): called after Begin() +// Process(): called for each event, in this function you decide what +// to read and fill your histograms. +// SlaveTerminate(): called at the end of the loop on the tree. +// Terminate(): called at the end of the loop on the tree, +// a convenient place to draw/fit your histograms. // // To use this file, try the following session on your Tree T: // @@ -983,8 +979,7 @@ R"CODE(#define )CODE" << cppClassName << R"CODE(_cxx void )CODE" << cppClassName << R"CODE(::Begin(TTree * /*tree*/) { // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). + // The tree argument is deprecated. TString option = GetOption(); } @@ -992,8 +987,7 @@ void )CODE" << cppClassName << R"CODE(::Begin(TTree * /*tree*/) void )CODE" << cppClassName << R"CODE(::SlaveBegin(TTree * /*tree*/) { // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). + // The tree argument is deprecated. TString option = GetOption(); @@ -1001,11 +995,9 @@ void )CODE" << cppClassName << R"CODE(::SlaveBegin(TTree * /*tree*/) bool )CODE" << cppClassName << R"CODE(::Process(Long64_t entry) { - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument + // The Process() function is called for each entry in the tree + // to be processed. The entry argument // specifies which entry in the currently loaded tree is to be processed. - // When processing keyed objects with PROOF, the object is already loaded - // and is available via the fObject pointer. // // This function should contain the \"body\" of the analysis. It can contain // simple or elaborate selection criteria, run algorithms on the data @@ -1025,8 +1017,7 @@ bool )CODE" << cppClassName << R"CODE(::Process(Long64_t entry) void )CODE" << cppClassName << R"CODE(::SlaveTerminate() { // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. + // have been processed. } diff --git a/tutorials/CMakeLists.txt b/tutorials/CMakeLists.txt index e7269d65770db..27dc7c61ed327 100644 --- a/tutorials/CMakeLists.txt +++ b/tutorials/CMakeLists.txt @@ -59,7 +59,6 @@ file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/rootlogon.C "{ file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/rootalias.C "") file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/rootlogoff.C "{}") file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/.rootrc " -Proof.Sandbox: /tmp/proof Rint.History: .root_hist ACLiC.LinkLibs: 1 ") @@ -140,10 +139,6 @@ if(NOT geom) set(geom_veto visualisation/geom/*.C geom/*.C visualisation/geom/gdml/*.C legacy/g3d/shapes.* legacy/g3d/na49view.*) endif() -if(NOT ROOT_proof_FOUND) - set(proof_veto roofit/roostats/StandardFrequentistDiscovery.C) -endif() - if(NOT ROOT_spectrum_FOUND) set(spectrum_veto legacy/spectrum/*.C) endif() @@ -425,7 +420,6 @@ set(extra_veto legacy/multicore/mp_H1_lambdas.C # not a tutorial; used by mp104_processH1.C et al. html/*.C legacy/net/*.C - legacy/proof/*.C io/sql/*.C io/tree/hsimpleProxy.C # A driver uses this macro which cannot be executed directly io/tree/tree103_tree.C @@ -441,7 +435,8 @@ set(extra_veto analysis/tree/h1chain.C http/*.C visualisation/eve7/*.C - math/r/rootlogon.C) + math/r/rootlogon.C + roofit/roostats/StandardFrequentistDiscovery.C) if(MSVC) # disable run_h1analysis.C because of Endpoint Security HTTP traffic scanning, @@ -502,7 +497,6 @@ set(all_veto hsimple.C ${macm1_veto} ${clad_veto} ${davix_veto} - ${proof_veto} ) file(GLOB_RECURSE tutorials RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} *.C) diff --git a/tutorials/legacy/proof/EmptyInclude.h b/tutorials/legacy/proof/EmptyInclude.h deleted file mode 100644 index 1e826a39b871c..0000000000000 --- a/tutorials/legacy/proof/EmptyInclude.h +++ /dev/null @@ -1,15 +0,0 @@ -/// \file EmptyInclude.h -/// \ingroup tutorial_proof -/// -/// Empty file to test TProof::Load in runProof.C and StressProof.cxx in -/// conjunction with ProcFileElements and ProofEventProc . -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef EmptyInclude_h -#define EmptyInclude_h - -// Empty file to test TProof::Load in runProof.C and StressProof.cxx in -// conjunction with ProcFileElements and ProofEventProc . - -#endif diff --git a/tutorials/legacy/proof/ProcFileElements.C b/tutorials/legacy/proof/ProcFileElements.C deleted file mode 100644 index c7e6aa37a4f63..0000000000000 --- a/tutorials/legacy/proof/ProcFileElements.C +++ /dev/null @@ -1,239 +0,0 @@ -/// \file -/// \ingroup tutorial_ProcFileElements -/// -/// Class to hold information about the processed elements of a file -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#include "ProcFileElements.h" -#include "TCollection.h" - -//_______________________________________________________________________ -Int_t ProcFileElements::ProcFileElement::Compare(const TObject *o) const -{ - // Compare this element with 'e'. - // Return - // -1 this should come first - // 0 same - // 1 e should come first - - const ProcFileElements::ProcFileElement *e = - dynamic_cast(o); - if (!e) return -1; - - if (fFirst == e->fFirst) { - // They start at the same point - return 0; - } else if (fFirst < e->fFirst) { - // This starts first - return -1; - } else { - // e starts first - return 1; - } -} - -//_______________________________________________________________________ -Int_t ProcFileElements::ProcFileElement::Overlapping(ProcFileElement *e) -{ - // Check overlapping status of this element with 'e'. - // Return - // -1 not overlapping - // 0 adjacent - // 1 overlapping - - if (!e) return -1; - - if (fFirst == 0 && fLast == -1) { - // We cover the full range, so we overlap - return 1; - } - - if (fFirst < e->fFirst) { - if (fLast >= 0) { - if (fLast < e->fFirst - 1) { - // Disjoint - return -1; - } else { - // We somehow overlap - if (fLast == e->fFirst - 1) { - // Just adjacent - return 0; - } else { - // Real overlap - return 1; - } - } - } else { - // Always overlapping - return 1; - } - } else if (fFirst == e->fFirst) { - // Overlapping or adjacent - if (fFirst == fLast || e->fFirst == e->fLast) return 0; - return 1; - } else { - // The other way around - if (e->fLast >= 0) { - if (e->fLast < fFirst - 1) { - // Disjoint - return -1; - } else { - // We somehow overlap - if (e->fLast == fFirst - 1) { - // Just adjacent - return 0; - } else { - // Real overlap - return 1; - } - } - } else { - // Always overlapping - return 1; - } - } - - // Should never be here - Warning("Overlapping", "should never be here!"); - return -1; -} - -//_______________________________________________________________________ -Int_t ProcFileElements::ProcFileElement::MergeElement(ProcFileElement *e) -{ - // Merge this element with element 'e'. - // Return -1 if it cannot be done, i.e. thei are disjoint; 0 otherwise - - // Check if it can be done - if (Overlapping(e) < 0) return -1; - - // Ok, we can merge: set the lower bound - if (e->fFirst < fFirst) fFirst = e->fFirst; - - // Set the upper bound - if (fLast == -1 || e->fLast == -1) { - fLast = -1; - } else { - if (fLast < e->fLast) fLast = e->fLast; - } - // Done - return 0; -} - -//_______________________________________________________________________ -void ProcFileElements::ProcFileElement::Print(Option_t *) const -{ - // Print range of this element - - Printf("\tfirst: %lld\t last: %lld", fFirst, fLast); -} - -//_______________________________________________________________________ -Int_t ProcFileElements::Add(Long64_t fst, Long64_t lst) -{ - // Add a new element to the list - // Return 1 if a new element has been added, 0 if it has been merged - // with an existing one, -1 in case of error - - if (!fElements) fElements = new TSortedList; - if (!fElements) { - Error("Add", "could not create internal list!"); - return -1; - } - - // Create (temporary element) - ProcFileElements::ProcFileElement *ne = - new ProcFileElements::ProcFileElement(fst, lst); - - // Check if if it is adjacent or overlapping with an existing one - TIter nxe(fElements); - ProcFileElements::ProcFileElement *e = nullptr; - while ((e = (ProcFileElements::ProcFileElement *)nxe())) { - if (e->MergeElement(ne) == 0) break; - } - - Int_t rc = 0; - // Remove and re-add the merged element to sort correctly its possibly new position - if (e) { - fElements->Remove(e); - fElements->Add(e); - SafeDelete(ne); - } else { - // Add the new element - fElements->Add(ne); - rc = 1; - } - - // Make sure that all what can be merged is merged (because of the order, some elements - // which could be merged are not merged, making the determination of fFirst and fLast below - // to give incorrect values) - - ProcFileElements::ProcFileElement *ep = nullptr, *en = nullptr; - TObjLink *olp = fElements->FirstLink(), *oln = nullptr; - while (olp && (ep = (ProcFileElements::ProcFileElement *) olp->GetObject())) { - oln = olp->Next(); - while (oln) { - if ((en = (ProcFileElements::ProcFileElement *) oln->GetObject())) { - if (ep->MergeElement(en) == 0) { - fElements->Remove(en); - delete en; - } - } - oln = oln->Next(); - } - olp = olp->Next(); - } - - // New overall ranges - if ((e = (ProcFileElements::ProcFileElement *) fElements->First())) fFirst = e->fFirst; - if ((e = (ProcFileElements::ProcFileElement *) fElements->Last())) fLast = e->fLast; - - // Done - return rc; -} - -//_______________________________________________________________________ -void ProcFileElements::Print(Option_t *) const -{ - // Print info about this processed file - - Printf("--- ProcFileElements ----------------------------------------"); - Printf(" File: %s", fName.Data()); - Printf(" # proc elements: %d", fElements ? fElements->GetSize() : 0); - TIter nxe(fElements); - ProcFileElements::ProcFileElement *e = nullptr; - while ((e = (ProcFileElements::ProcFileElement *)nxe())) { e->Print(); } - Printf(" Raw overall range: [%lld, %lld]", fFirst, fLast); - Printf("-------------------------------------------------------------"); -} - -//_______________________________________________________________________ -Int_t ProcFileElements::Merge(TCollection *li) -{ - // Merge this object with those in the list - // Return number of elements added - - if (!li) return -1; - - if (li->GetSize() <= 0) return 0; - - Int_t nadd = 0; - TIter nxo(li); - ProcFileElements *pfe = nullptr; - while ((pfe = (ProcFileElements *) nxo())) { - if (strcmp(GetName(), pfe->GetName())) - Warning("Merge", "merging objects of different name! ('%s' != '%s')", - GetName(), pfe->GetName()); - TIter nxe(pfe->GetListOfElements()); - ProcFileElements::ProcFileElement *e = nullptr; - while ((e = (ProcFileElements::ProcFileElement *)nxe())) { - Int_t rc = Add(e->fFirst, e->fLast); - if (rc == 1) nadd++; - } - } - // Done - return nadd; -} diff --git a/tutorials/legacy/proof/ProcFileElements.h b/tutorials/legacy/proof/ProcFileElements.h deleted file mode 100644 index 3d25457f5d033..0000000000000 --- a/tutorials/legacy/proof/ProcFileElements.h +++ /dev/null @@ -1,69 +0,0 @@ -/// \file -/// \ingroup tutorial_ProcFileElements -/// -/// Class to hold information about the processed elements of a file -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - - -#ifndef ROOT_ProcFileElements -#define ROOT_ProcFileElements - - -#include "TObject.h" -#include "TSortedList.h" - -class ProcFileElements : public TObject { - -public: - class ProcFileElement : public TObject { - public: - Long64_t fFirst; // Lower bound of this range - Long64_t fLast; // Upper bound of this range - ProcFileElement(Long64_t fst = 0, Long64_t lst = -1) : - fFirst(fst), fLast(lst) { } - ~ProcFileElement() override { } - - Int_t Compare(const TObject *obj) const override; - - Bool_t IsSortable() const override { return kTRUE; } - Int_t MergeElement(ProcFileElement *); - - Int_t Overlapping(ProcFileElement *); - void Print(Option_t *option="") const override; - - ClassDefOverride(ProcFileElement, 1); // ProcFileElement class - }; - -private: - TString fName; // File name - TSortedList *fElements; // List of processed elements - - Long64_t fFirst; // Overall lower bound - Long64_t fLast; // Overall Upper bound - -public: - ProcFileElements(const char *fn = "") : fName(fn), fElements(nullptr), - fFirst(0), fLast(-1) { } - ~ProcFileElements() override { if (fElements) { fElements->SetOwner(); - delete fElements; } } - const char * GetName() const override { return fName; } - ULong_t Hash() const override { return fName.Hash(); } - - Int_t Add(Long64_t fst = 0, Long64_t lst = -1); - Int_t Merge(TCollection *list); - - TSortedList *GetListOfElements() const { return fElements; } - Int_t GetNumElements() const { return (fElements ? fElements->GetSize() : 0); } - - Long64_t GetFirst() const { return fFirst; } - Long64_t GetLast() const { return fLast; } - - void Print(Option_t *option="") const override; - - ClassDefOverride(ProcFileElements, 1); // Processed File Elements class -}; - -#endif diff --git a/tutorials/legacy/proof/ProofAux.C b/tutorials/legacy/proof/ProofAux.C deleted file mode 100644 index 8723804f051be..0000000000000 --- a/tutorials/legacy/proof/ProofAux.C +++ /dev/null @@ -1,462 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofAux -/// -/// Selector used for auxiliary actions in the PROOF tutorials -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofAux_cxx - -#include "ProofAux.h" -#include "TDSet.h" -#include "TProofServ.h" -#include "TMap.h" -#include "TString.h" -#include "TSystem.h" -#include "TParameter.h" -#include "TFile.h" -#include "TUrl.h" -#include "TTree.h" -#include "TRandom.h" -#include "TMath.h" - -//_____________________________________________________________________________ -ProofAux::ProofAux() -{ - // Constructor - - fAction = -1; - fNEvents= -1; - fMainList = 0; - fFriendList = 0; - fDir = ""; -} - -//_____________________________________________________________________________ -ProofAux::~ProofAux() -{ - // Destructor - -} - -//_____________________________________________________________________________ -Int_t ProofAux::GetAction(TList *input) -{ - // Get the required action. - // Returns -1 if unknown. - - Int_t action = -1; - // Determine the test type - TNamed *ntype = dynamic_cast(input->FindObject("ProofAux_Action")); - if (ntype) { - TString act(ntype->GetTitle()); - if (act == "GenerateTreesSameFile") { - action = 1; - } else if (act.BeginsWith("GenerateTrees")) { - action = 0; - // Extract directory, if any - Ssiz_t icol = act.Index(":"); - if (icol != kNPOS) { - action = 2; - act.Remove(0, icol+1); - if (!act.IsNull()) fDir = act; - } - } else { - Warning("GetAction", "unknown action: '%s'", ntype->GetTitle()); - } - } - // Done - return action; -} - - -//_____________________________________________________________________________ -void ProofAux::Begin(TTree * /*tree*/) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). -} - -//_____________________________________________________________________________ -void ProofAux::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - // Determine the action type - fAction = GetAction(fInput); - - // Get the number of events - TParameter *a = (TParameter *) fInput->FindObject("ProofAux_NEvents"); - if (a) fNEvents = a->GetVal(); - - // Create lists - fMainList = new TList; - if (gProofServ) fMainList->SetName(TString::Format("MainList-%s", gProofServ->GetOrdinal())); - if (fAction < 2) { - fFriendList = new TList; - if (gProofServ) fFriendList->SetName(TString::Format("FriendList-%s", gProofServ->GetOrdinal())); - } -} - -//_____________________________________________________________________________ -Bool_t ProofAux::Process(Long64_t entry) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either ProofAux::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - // - // The processing can be stopped by calling Abort(). - // - // Use fStatus to set the return value of TTree::Process(). - // - // The return value is currently not used. - - // Nothing to do if the action if not defined - if (fAction < 0) { - Error("Process", "action not specified!"); - return kFALSE; - } - - // Link to current element, if any - TDSetElement *fCurrent = 0; - TPair *elemPair = 0; - if (fInput && (elemPair = dynamic_cast(fInput->FindObject("PROOF_CurrentElement")))) { - if ((fCurrent = dynamic_cast(elemPair->Value()))) { - Info("Process", "entry %lld: file: '%s'", entry, fCurrent->GetName()); - } else { - Error("Process", "entry %lld: no file specified!", entry); - return kFALSE; - } - } - - // Act now - if (fAction == 0) { - TString fnt; - // Generate the TTree and save it in the specified file - if (GenerateTree(fCurrent->GetName(), fNEvents, fnt) != 0) { - Error("Process", "problems generating tree (%lld, %s, %lld)", - entry, fCurrent->GetName(), fNEvents); - return kFALSE; - } - // The output filename - TString fnf(fnt); - TString xf = gSystem->BaseName(fnf); - fnf = gSystem->GetDirName(fnf); - if (xf.Contains("tree")) { - xf.ReplaceAll("tree", "friend"); - } else { - if (xf.EndsWith(".root")) { - xf.ReplaceAll(".root", "_friend.root"); - } else { - xf += "_friend"; - } - } - fnf += TString::Format("/%s", xf.Data()); - // Generate the TTree friend and save it in the specified file - if (GenerateFriend(fnt, fnf) != 0) { - Error("Process", "problems generating friend tree for %s (%s)", - fCurrent->GetName(), fnt.Data()); - return kFALSE; - } - } else if (fAction == 1) { - TString fnt; - // Generate the TTree and save it in the specified file - if (GenerateTree(fCurrent->GetName(), fNEvents, fnt) != 0) { - Error("Process", "problems generating tree (%lld, %s, %lld)", - entry, fCurrent->GetName(), fNEvents); - return kFALSE; - } - // Generate the TTree friend and save it in the specified file - if (GenerateFriend(fnt) != 0) { - Error("Process", "problems generating friend tree for %s (%s)", - fCurrent->GetName(), fnt.Data()); - return kFALSE; - } - } else if (fAction == 2) { - TString fnt; - // Generate the TTree and save it in the specified file - if (GenerateTree(fCurrent->GetName(), fNEvents, fnt) != 0) { - Error("Process", "problems generating tree (%lld, %s, %lld)", - entry, fCurrent->GetName(), fNEvents); - return kFALSE; - } - } else { - // Unknown action - Warning("Process", "do not know how to process action %d - do nothing", fAction); - return kFALSE; - } - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofAux::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - - if (fMainList && fMainList->GetSize() > 0) fOutput->Add(fMainList); - if (fFriendList && fFriendList->GetSize() > 0) fOutput->Add(fFriendList); -} - -//_____________________________________________________________________________ -void ProofAux::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - -} - -//_____________________________________________________________________________ -Int_t ProofAux::GenerateTree(const char *fnt, Long64_t ent, TString &fn) -{ - // Generate the main tree for the 'friends' tutorial; the tree is called - // 'Tmain', has 'ent' entries and is saved to file 'fnt'. - // The full file path is returned in 'fn'. - // Return 0 on success, -1 on error. - - Int_t rc = -1; - - // Check the filename - fn = fnt; - if (fn.IsNull()) { - Error("GenerateTree", "file name undefined!"); - return rc; - } - TUrl uu(fn, kTRUE); - if (!strcmp(uu.GetProtocol(), "file") && !fn.BeginsWith("/")) { - // Local file with relative path: create under the data directory - if (!gProofServ || - !(gProofServ->GetDataDir()) || strlen(gProofServ->GetDataDir()) <= 0) { - Error("GenerateTree", "data directory undefined!"); - return rc; - } - // Insert data directory - fn.Insert(0, TString::Format("%s/", gProofServ->GetDataDir())); - // Make sure the directory exists - TString dir = gSystem->GetDirName(fn); - if (gSystem->AccessPathName(dir, kWritePermission)) { - if (gSystem->mkdir(dir, kTRUE) != 0) { - Error("GenerateTree", "problems creating directory %s to store the file", dir.Data()); - return rc; - } - } - } - - // Create the file - TDirectory *savedir = gDirectory; - TFile *f = new TFile(fn, "RECREATE"); - if (!f || f->IsZombie()) { - Error("GenerateTree", "problems opening file %s", fn.Data()); - return rc; - } - savedir->cd(); - - // Create sub-drirectory, if required - TDirectory *destdir = f; - if (!fDir.IsNull()) { - if (f->mkdir(fDir.Data())) { - Info("GenerateTree", "sub-directory '%s' successfully created", fDir.Data()); - f->cd(fDir.Data()); - destdir = gDirectory; - } else { - Error("GenerateTree", "creating sub-directory '%s'", fDir.Data()); - f->Close(); - delete f; - return rc; - } - } - - rc = 0; - - // Create the tree - TTree *T = new TTree("Tmain","Main tree for tutorial friends"); - T->SetDirectory(destdir); - Int_t Run = 1; - T->Branch("Run",&Run,"Run/I"); - Long64_t Event = 0; - T->Branch("Event",&Event,"Event/L"); - Float_t x = 0., y = 0., z = 0.; - T->Branch("x",&x,"x/F"); - T->Branch("y",&y,"y/F"); - T->Branch("z",&z,"z/F"); - TRandom r; - for (Long64_t i = 0; i < ent; i++) { - if (i > 0 && i%1000 == 0) Run++; - Event = i; - x = r.Gaus(10,1); - y = r.Gaus(20,2); - z = r.Landau(2,1); - T->Fill(); - } - T->Print(); - destdir->cd(); - T->Write(); - T->SetDirectory(0); - f->Close(); - delete f; - delete T; - - // Notify success - Info("GenerateTree", "file '%s' successfully created", fn.Data()); - - // Add to the list - TString fds(fn); - if (!strcmp(uu.GetProtocol(), "file")) { - if (gSystem->Getenv("LOCALDATASERVER")) { - if (strcmp(TUrl(gSystem->Getenv("LOCALDATASERVER"), kTRUE).GetProtocol(), "file")) - fds.Insert(0, TString::Format("%s/", gSystem->Getenv("LOCALDATASERVER"))); - } else { - fds.Insert(0, TString::Format("root://%s/", gSystem->HostName())); - } - } - fMainList->Add(new TObjString(fds)); - - // Done - return rc; -} - -//_____________________________________________________________________________ -Int_t ProofAux::GenerateFriend(const char *fnt, const char *fnf) -{ - // Generate the friend tree for the main tree in the 'friends' tutorial fetched - // from 'fnt'. - // the tree is called 'Tfriend', has the same number of entries as the main - // tree and is saved to file 'fnf'. If 'fnf' is not defined the filename is - // derived from 'fnt' either replacing 'tree' with 'friend', or adding '_friend' - // before the '.root' extension. - // Return 0 on success, -1 on error. - - Int_t rc = -1; - // Check the input filename - TString fin(fnt); - if (fin.IsNull()) { - Error("GenerateFriend", "file name for the main tree undefined!"); - return rc; - } - // Make sure that the file can be read - if (gSystem->AccessPathName(fin, kReadPermission)) { - Error("GenerateFriend", "input file does not exist or cannot be read: %s", fin.Data()); - return rc; - } - - // File handlers - Bool_t sameFile = kTRUE; - const char *openMain = "UPDATE"; - - // The output filename - TString fout(fnf); - if (!fout.IsNull()) { - sameFile = kFALSE; - openMain = "READ"; - // Make sure the directory exists - TString dir = gSystem->GetDirName(fout); - if (gSystem->AccessPathName(dir, kWritePermission)) { - if (gSystem->mkdir(dir, kTRUE) != 0) { - Error("GenerateFriend", "problems creating directory %s to store the file", dir.Data()); - return rc; - } - } - } else { - // We set the same name - fout = fin; - } - - // Get main tree - TFile *fi = TFile::Open(fin, openMain); - if (!fi || fi->IsZombie()) { - Error("GenerateFriend", "problems opening input file %s", fin.Data()); - return rc; - } - TTree *Tin = (TTree *) fi->Get("Tmain"); - if (!Tin) { - Error("GenerateFriend", "problems getting tree 'Tmain' from file %s", fin.Data()); - delete fi; - return rc; - } - // Set branches - Float_t x, y, z; - Tin->SetBranchAddress("x", &x); - Tin->SetBranchAddress("y", &y); - Tin->SetBranchAddress("z", &z); - TBranch *b_x = Tin->GetBranch("x"); - TBranch *b_y = Tin->GetBranch("y"); - TBranch *b_z = Tin->GetBranch("z"); - - TDirectory* savedir = gDirectory; - // Create output file - TFile *fo = 0; - if (!sameFile) { - fo = new TFile(fout, "RECREATE"); - if (!fo || fo->IsZombie()) { - Error("GenerateFriend", "problems opening file %s", fout.Data()); - delete fi; - return rc; - } - savedir->cd(); - } else { - // Same file - fo = fi; - } - rc = 0; - - // Create the tree - TTree *Tfrnd = new TTree("Tfrnd", "Friend tree for tutorial 'friends'"); - Tfrnd->SetDirectory(fo); - Float_t r = 0; - Tfrnd->Branch("r",&r,"r/F"); - Long64_t ent = Tin->GetEntries(); - for (Long64_t i = 0; i < ent; i++) { - b_x->GetEntry(i); - b_y->GetEntry(i); - b_z->GetEntry(i); - r = TMath::Sqrt(x*x + y*y + z*z); - Tfrnd->Fill(); - } - if (!sameFile) { - fi->Close(); - delete fi; - } - Tfrnd->Print(); - fo->cd(); - Tfrnd->Write(); - Tfrnd->SetDirectory(0); - fo->Close(); - delete fo; - delete Tfrnd; - - // Notify success - Info("GenerateFriend", "friend file '%s' successfully created", fout.Data()); - - // Add to the list - TUrl uu(fout); - if (!strcmp(uu.GetProtocol(), "file")) { - if (gSystem->Getenv("LOCALDATASERVER")) { - if (strcmp(TUrl(gSystem->Getenv("LOCALDATASERVER"), kTRUE).GetProtocol(), "file")) - fout.Insert(0, TString::Format("%s/", gSystem->Getenv("LOCALDATASERVER"))); - } else { - fout.Insert(0, TString::Format("root://%s/", gSystem->HostName())); - } - } - fFriendList->Add(new TObjString(fout)); - - // Done - return rc; -} diff --git a/tutorials/legacy/proof/ProofAux.h b/tutorials/legacy/proof/ProofAux.h deleted file mode 100644 index 3b8079f0106f3..0000000000000 --- a/tutorials/legacy/proof/ProofAux.h +++ /dev/null @@ -1,48 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofAux -/// -/// Selector used for auxiliary actions in the PROOF tutorials -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofAux_h -#define ProofAux_h - -#include -#include - -class TList; - -class ProofAux : public TSelector { -private : - Int_t GenerateTree(const char *fnt, Long64_t ent, TString &fn); - Int_t GenerateFriend(const char *fnt, const char *fnf = nullptr); - Int_t GetAction(TList *input); -public : - - // Specific members - Int_t fAction; - Long64_t fNEvents; - TList *fMainList; - TList *fFriendList; - TString fDir; - - ProofAux(); - ~ProofAux() override; - Int_t Version() const override { return 2; } - void Begin(TTree *tree) override; - void SlaveBegin(TTree *tree) override; - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override { fInput = input; } - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(ProofAux,0); -}; - -#endif diff --git a/tutorials/legacy/proof/ProofEvent.C b/tutorials/legacy/proof/ProofEvent.C deleted file mode 100644 index a1e79387ac9a6..0000000000000 --- a/tutorials/legacy/proof/ProofEvent.C +++ /dev/null @@ -1,115 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofEvent -/// -/// Selector for generic processing with Event -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofEvent_cxx - -#include "ProofEvent.h" -#include "Event.h" - -#include -#include -#include - -//_____________________________________________________________________________ -ProofEvent::ProofEvent() -{ - // Constructor - - fEvent = 0; - fNtrack = -1; - fHisto = 0; - fRandom = 0; -} - -//_____________________________________________________________________________ -ProofEvent::~ProofEvent() -{ - // Destructor - - SafeDelete(fRandom); -} - -//_____________________________________________________________________________ -void ProofEvent::Begin(TTree * /*tree*/) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - Info("Begin", "starting a simple exercise with process option: %s", option.Data()); -} - -//_____________________________________________________________________________ -void ProofEvent::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - Info("SalveBegin", "starting on a slave with process option: %s", option.Data()); - - // Create event - fEvent = new Event(); - - // Create the histogram - fHisto = new TH1F("histo", "tracks multiplicity", 20, 0, 100); - fHisto->GetYaxis()->SetTitle("number of events"); - fHisto->GetXaxis()->SetTitle("number of tracks"); - - //adding histo to selector output list - fOutput->Add(fHisto); - - // Set random seed - fRandom = new TRandom3(0); -} - -//_____________________________________________________________________________ -Bool_t ProofEvent::Process(Long64_t ) -{ - - // Start main loop over all events - // get a random parameter for connstructing event - - int i= (int)(100 * (fRandom->Rndm())); - fEvent->Build(i,(1+i), 2); - fNtrack= (fEvent->GetNtrack()); - if ((fNtrack >= 0 )&& (fNtrack <= 100 )) - fHisto->Fill(fNtrack, 1); - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofEvent::SlaveTerminate() -{ - //nothing to be done - -} - -//_____________________________________________________________________________ -void ProofEvent::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - TCanvas *c1 = new TCanvas("c1","Proof ProofEvent canvas",200,10,700,700); - fHisto = dynamic_cast(fOutput->FindObject(Form("histo"))); - if (fHisto) { - fHisto->Draw("h"); - - // Final update - c1->cd(); - c1->Update(); - } else { - Warning("Terminate", "histogram not found"); - } -} diff --git a/tutorials/legacy/proof/ProofEvent.h b/tutorials/legacy/proof/ProofEvent.h deleted file mode 100644 index 6fc57f80426d2..0000000000000 --- a/tutorials/legacy/proof/ProofEvent.h +++ /dev/null @@ -1,44 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofEvent -/// -/// Selector for generic processing with Event -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofEvent_h -#define ProofEvent_h - -#include -#include "Event.h" - -class TH1F; -class TRandom3; - -class ProofEvent : public TSelector { -public : - - // Specific members - TRandom3 *fRandom; - Event *fEvent; - Int_t fNtrack; - TH1F *fHisto; - - ProofEvent(); - virtual ~ProofEvent(); - virtual Int_t Version() const { return 2; } - virtual void Begin(TTree *tree); - virtual void SlaveBegin(TTree *tree); - virtual Bool_t Process(Long64_t entry); - virtual void SetOption(const char *option) { fOption = option; } - virtual void SetObject(TObject *obj) { fObject = obj; } - virtual void SetInputList(TList *input) { fInput = input; } - virtual TList *GetOutputList() const { return fOutput; } - virtual void SlaveTerminate(); - virtual void Terminate(); - - ClassDef(ProofEvent,0); -}; - -#endif diff --git a/tutorials/legacy/proof/ProofEventProc.C b/tutorials/legacy/proof/ProofEventProc.C deleted file mode 100644 index 61e07c20bbad9..0000000000000 --- a/tutorials/legacy/proof/ProofEventProc.C +++ /dev/null @@ -1,329 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofEventProc -/// -/// Selector to process trees containing Event structures -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofEventProc_cxx - -#include "ProofEventProc.h" -#include -#include "TCanvas.h" -#include "TPad.h" -#include "TH1F.h" -#include "TH2F.h" -#include "TParameter.h" -#include "TRandom.h" -#include "TNamed.h" -#include "TROOT.h" -#include "EmptyInclude.h" - - -//_____________________________________________________________________________ -void ProofEventProc::Begin(TTree *) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - Info("Begin", "starting a simple exercise with process option: %s", option.Data()); -} - -//_____________________________________________________________________________ -void ProofEventProc::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - // How much to read - fFullRead = kFALSE; - TNamed *nm = 0; - if (fInput) { - if ((nm = dynamic_cast(fInput->FindObject("ProofEventProc_Read")))) { - if (!strcmp(nm->GetTitle(), "readall")) fFullRead = kTRUE; - } - } - if (!nm) { - // Check option - if (option == "readall") fFullRead = kTRUE; - } - Info("SlaveBegin", "'%s' reading", (fFullRead ? "full" : "optimized")); - - fPtHist = new TH1F("pt_dist","p_{T} Distribution",100,0,5); - fPtHist->SetDirectory(0); - fPtHist->GetXaxis()->SetTitle("p_{T}"); - fPtHist->GetYaxis()->SetTitle("dN/p_{T}dp_{T}"); - - fOutput->Add(fPtHist); - - fPzHist = new TH1F("pz_dist","p_{Z} Distribution",100,0,5.); - fPzHist->SetDirectory(0); - fPzHist->GetXaxis()->SetTitle("p_{Z}"); - fPzHist->GetYaxis()->SetTitle("dN/dp_{Z}"); - - fOutput->Add(fPzHist); - - fPxPyHist = new TH2F("px_py","p_{X} vs p_{Y} Distribution",100,-5.,5.,100,-5.,5.); - fPxPyHist->SetDirectory(0); - fPxPyHist->GetXaxis()->SetTitle("p_{X}"); - fPxPyHist->GetYaxis()->SetTitle("p_{Y}"); - - fOutput->Add(fPxPyHist); - - // Abort test, if any - TParameter *pi = 0; - if (fInput) - pi = dynamic_cast *>(fInput->FindObject("ProofEventProc_TestAbort")); - if (pi) fTestAbort = pi->GetVal(); - if (fTestAbort < -1 || fTestAbort > 1) { - Info("SlaveBegin", "unsupported value for the abort test: %d not in [-1,1] - ignore", fTestAbort); - fTestAbort = -1; - } else if (fTestAbort > -1) { - Info("SlaveBegin", "running abort test: %d", fTestAbort); - } - - if (fTestAbort == 0) - Abort("Test abortion during init", kAbortProcess); -} - -//_____________________________________________________________________________ -Bool_t ProofEventProc::Process(Long64_t entry) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either TTree::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - - // WARNING when a selector is used with a TChain, you must use - // the pointer to the current TTree to call GetEntry(entry). - // The entry is always the local entry number in the current tree. - // Assuming that fChain is the pointer to the TChain being processed, - // use fChain->GetTree()->GetEntry(entry). - - if (fEntMin == -1 || entry < fEntMin) fEntMin = entry; - if (fEntMax == -1 || entry > fEntMax) fEntMax = entry; - - if (fTestAbort == 1) { - Double_t rr = gRandom->Rndm(); - if (rr > 0.999) { - Info("Process", "%lld -> %f", entry, rr); - Abort("Testing file abortion", kAbortFile); - return kTRUE; - } - } - - if (fFullRead) { - fChain->GetTree()->GetEntry(entry); - } else { - b_event_fNtrack->GetEntry(entry); - } - - if (fNtrack > 0) { - if (!fFullRead) b_fTracks->GetEntry(entry); - if (fTracks) { - for (Int_t j=0;jGetEntries();j++){ - Track *curtrack = dynamic_cast(fTracks->At(j)); - if (curtrack) { - fPtHist->Fill(curtrack->GetPt(),1./curtrack->GetPt()); - fPxPyHist->Fill(curtrack->GetPx(),curtrack->GetPy()); - if (j == 0) fPzHist->Fill(curtrack->GetPz()); - } - } - fTracks->Clear("C"); - } - } - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofEventProc::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - - // Save information about previous element, if any - if (fProcElem) fProcElem->Add(fEntMin, fEntMax); - - if (!fProcElems) { - Warning("SlaveTerminate", "no proc elements list found!"); - return; - } - - // Add proc elements to the output list - TIter nxpe(fProcElems); - TObject *o = 0; - while ((o = nxpe())) { fOutput->Add(o); }; -} - -//_____________________________________________________________________________ -void ProofEventProc::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - // Check ranges - CheckRanges(); - - if (gROOT->IsBatch()) return; - - TCanvas* canvas = new TCanvas("event","event",800,10,700,780); - canvas->Divide(2,2); - TPad *pad1 = (TPad *) canvas->GetPad(1); - TPad *pad2 = (TPad *) canvas->GetPad(2); - TPad *pad3 = (TPad *) canvas->GetPad(3); - TPad *pad4 = (TPad *) canvas->GetPad(4); - - // The number of tracks - pad1->cd(); - pad1->SetLogy(); - TH1F *hi = dynamic_cast(fOutput->FindObject("pz_dist")); - if (hi) { - hi->SetFillColor(30); - hi->SetLineColor(9); - hi->SetLineWidth(2); - hi->DrawCopy(); - } else { Warning("Terminate", "no pz dist found"); } - - // The Pt distribution - pad2->cd(); - pad2->SetLogy(); - TH1F *hf = dynamic_cast(fOutput->FindObject("pt_dist")); - if (hf) { - hf->SetFillColor(30); - hf->SetLineColor(9); - hf->SetLineWidth(2); - hf->DrawCopy(); - } else { Warning("Terminate", "no pt dist found"); } - - // The Px,Py distribution, color surface - TH2F *h2f = dynamic_cast(fOutput->FindObject("px_py")); - if (h2f) { - // Color surface - pad3->cd(); - h2f->DrawCopy("SURF1 "); - // Lego - pad4->cd(); - h2f->DrawCopy("CONT2COL"); - } else { - Warning("Terminate", "no px py found"); - } - - // Final update - canvas->cd(); - canvas->Update(); -} - -//_____________________________________________________________________________ -void ProofEventProc::CheckRanges() -{ - // Check the processed event ranges when there is enough information - // The result is added to the output list - - // Must be something in output - if (!fOutput || (fOutput && fOutput->GetSize() <= 0)) return; - - // Create the result object and add it to the list - TNamed *nout = new TNamed("Range_Check", "OK"); - fOutput->Add(nout); - - // Get info to check from the input list - if (!fInput || (fInput && fInput->GetSize() <= 0)) { - nout->SetTitle("No input list"); - return; - } - TNamed *ffst = dynamic_cast(fInput->FindObject("Range_First_File")); - if (!ffst) { - nout->SetTitle("No first file"); - return; - } - TNamed *flst = dynamic_cast(fInput->FindObject("Range_Last_File")); - if (!flst) { - nout->SetTitle("No last file"); - return; - } - TParameter *fnum = - dynamic_cast *>(fInput->FindObject("Range_Num_Files")); - if (!fnum) { - nout->SetTitle("No number of files"); - return; - } - - // Check first file - TString fn(ffst->GetTitle()), sfst(ffst->GetTitle()); - Ssiz_t ifst = fn.Index("?fst="); - if (ifst == kNPOS) { - nout->SetTitle("No first entry information in first file name"); - return; - } - fn.Remove(ifst); - sfst.Remove(0, ifst + sizeof("?fst=") - 1); - if (!sfst.IsDigit()) { - nout->SetTitle("Badly formatted first entry information in first file name"); - return; - } - Long64_t fst = (Long64_t) sfst.Atoi(); - ProcFileElements *pfef = dynamic_cast(fOutput->FindObject(fn)); - if (!pfef) { - nout->SetTitle("ProcFileElements for first file not found in the output list"); - return; - } - if (pfef->GetFirst() != fst) { - TString t = TString::Format("First entry differs {found: %lld, expected: %lld}", pfef->GetFirst(), fst); - nout->SetTitle(t.Data()); - return; - } - - // Check last file - fn = flst->GetTitle(); - TString slst(flst->GetTitle()); - Ssiz_t ilst = fn.Index("?lst="); - if (ilst == kNPOS) { - nout->SetTitle("No last entry information in last file name"); - return; - } - fn.Remove(ilst); - slst.Remove(0, ilst + sizeof("?lst=") - 1); - if (!slst.IsDigit()) { - nout->SetTitle("Badly formatted last entry information in last file name"); - return; - } - Long64_t lst = (Long64_t) slst.Atoi(); - ProcFileElements *pfel = dynamic_cast(fOutput->FindObject(fn)); - if (!pfel) { - nout->SetTitle("ProcFileElements for last file not found in the output list"); - return; - } - if (pfel->GetLast() != lst) { - nout->SetTitle("Last entry differs"); - return; - } - - // Check Number of files - Int_t nproc = 0; - TIter nxo(fOutput); - TObject *o = 0; - while ((o = nxo())) { - if (dynamic_cast(o)) nproc++; - } - if (fnum->GetVal() != nproc) { - nout->SetTitle("Number of processed files differs"); - return; - } -} diff --git a/tutorials/legacy/proof/ProofEventProc.h b/tutorials/legacy/proof/ProofEventProc.h deleted file mode 100644 index 943adcfb6d6d5..0000000000000 --- a/tutorials/legacy/proof/ProofEventProc.h +++ /dev/null @@ -1,211 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofEventProc -/// -/// Selector to process trees containing Event structures -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofEventProc_h -#define ProofEventProc_h - -#include -#include -#include -#include -#include "Event.h" -#include "ProcFileElements.h" - -class TH1F; -class TH2F; - -class ProofEventProc : public TSelector { -public : - TTree *fChain; //!pointer to the analyzed TTree or TChain - - // Declaration of leave types - Event *event; - Char_t fType[20]; - Char_t *fEventName; - Int_t fNtrack; - Int_t fNseg; - Int_t fNvertex; - UInt_t fFlag; - Double32_t fTemperature; - Int_t fMeasures[10]; - Double32_t fMatrix[4][4]; - Double32_t fClosestDistance[21]; //[fNvertex] - EventHeader fEvtHdr; - TClonesArray *fTracks; - TRefArray *fHighPt; - TRefArray *fMuons; - TRef fLastTrack; - TRef fWebHistogram; - TH1F *fH; - TBits fTriggerBits; - Bool_t fIsValid; - void InitEvent() { event = 0 ; fEventName = 0; fTracks = 0; - fHighPt = 0; fMuons = 0; fH = 0; fIsValid = kFALSE; } - - // Read controller - Bool_t fFullRead; - - // Abortion test - Int_t fTestAbort; // -1 none, 0 init, 1 file - - // Event ranges - Long64_t fEntMin; // Min entry num of the current element - Long64_t fEntMax; // Max entry num of the current element - TList *fProcElems; // List of ProcFileElements processed so far - ProcFileElements *fProcElem; // Element currently processed - void CheckRanges(); - - //Output hist - TH1F* fPtHist; - TH1F* fPzHist; - TH2F* fPxPyHist; - - // List of branches - TBranch *b_event_fType; //! - TBranch *b_fEventName; //! - TBranch *b_event_fNtrack; //! - TBranch *b_event_fNseg; //! - TBranch *b_event_fNvertex; //! - TBranch *b_event_fFlag; //! - TBranch *b_event_fTemperature; //! - TBranch *b_event_fMeasures; //! - TBranch *b_event_fMatrix; //! - TBranch *b_fClosestDistance; //! - TBranch *b_event_fEvtHdr; //! - TBranch *b_fTracks; //! - TBranch *b_fHighPt; //! - TBranch *b_fMuons; //! - TBranch *b_event_fLastTrack; //! - TBranch *b_event_fWebHistogram; //! - TBranch *b_fH; //! - TBranch *b_event_fTriggerBits; //! - TBranch *b_event_fIsValid; //! - - ProofEventProc(TTree *) { InitEvent(); fFullRead = kTRUE; fTestAbort = -1; - fPtHist = 0; fPzHist = 0; fPxPyHist = 0; - fEntMin = -1; fEntMax = -1; fProcElems = 0; fProcElem = 0;} - ProofEventProc() { InitEvent(); fFullRead = kTRUE; fTestAbort = -1; - fPtHist = 0; fPzHist = 0; fPxPyHist = 0; - fEntMin = -1; fEntMax = -1; fProcElems = 0; fProcElem = 0;} - virtual ~ProofEventProc() {if (fProcElems) { fProcElems->SetOwner(kFALSE); - delete fProcElems;} } - virtual Int_t Version() const {return 2;} - virtual void Begin(TTree *); - virtual void SlaveBegin(TTree *tree); - virtual void Init(TTree *tree); - virtual Bool_t Notify(); - virtual Bool_t Process(Long64_t entry); - virtual void SetOption(const char *option) { fOption = option; } - virtual void SetObject(TObject *obj) { fObject = obj; } - virtual void SetInputList(TList *input) {fInput = input;} - virtual TList *GetOutputList() const { return fOutput; } - virtual void SlaveTerminate(); - virtual void Terminate(); - - ClassDef(ProofEventProc,0); -}; - -#endif - -#ifdef ProofEventProc_cxx -void ProofEventProc::Init(TTree *tree) -{ - // The Init() function is called when the selector needs to initialize - // a new tree or chain. Typically here the branch addresses of the tree - // will be set. It is normaly not necessary to make changes to the - // generated code, but the routine can be extended by the user if needed. - // Init() will be called many times when running with PROOF. - - // Set branch addresses - fEventName=0; - fTracks=0; - fHighPt=0; - fMuons=0; - fH=0; - if (tree == 0) return; - fChain = tree; - fChain->SetMakeClass(1); - - fChain->SetBranchAddress("fType[20]",fType); - fChain->SetBranchAddress("fEventName",fEventName); - fChain->SetBranchAddress("fNtrack",&fNtrack); - fChain->SetBranchAddress("fNseg",&fNseg); - fChain->SetBranchAddress("fNvertex",&fNvertex); - fChain->SetBranchAddress("fFlag",&fFlag); - fChain->SetBranchAddress("fTemperature",&fTemperature); - fChain->SetBranchAddress("fMeasures[10]",fMeasures); - fChain->SetBranchAddress("fMatrix[4][4]",fMatrix); - fChain->SetBranchAddress("fClosestDistance",fClosestDistance); - fChain->SetBranchAddress("fEvtHdr",&fEvtHdr); - fChain->SetBranchAddress("fTracks",&fTracks); - fChain->SetBranchAddress("fHighPt",&fHighPt); - fChain->SetBranchAddress("fMuons",&fMuons); - fChain->SetBranchAddress("fLastTrack",&fLastTrack); - fChain->SetBranchAddress("fWebHistogram",&fWebHistogram); - fChain->SetBranchAddress("fH",&fH); - fChain->SetBranchAddress("fTriggerBits",&fTriggerBits); - fChain->SetBranchAddress("fIsValid",&fIsValid); -} - -Bool_t ProofEventProc::Notify() -{ - // The Notify() function is called when a new file is opened. This - // can be either for a new TTree in a TChain or when when a new TTree - // is started when using PROOF. Typically here the branch pointers - // will be retrieved. It is normaly not necessary to make changes - // to the generated code, but the routine can be extended by the - // user if needed. - - TString fn; - if (fChain) { - fn = fChain->GetCurrentFile()->GetName(); - Info("Notify", "processing file: %s", fn.Data()); - } - - // Save information about previous element, if any - if (fProcElem) fProcElem->Add(fEntMin, fEntMax); - - // Reset counters - fEntMin = -1; - fEntMax = -1; - - // Point to current element - if (fProcElems) fProcElem = (ProcFileElements *) fProcElems->FindObject(fn); - if (!fProcElem) { - Info("Notify", "assigned new file: create ProcFileElements entry"); - fProcElem = new ProcFileElements(fn.Data()); - if (!fProcElems) fProcElems = new TList; - if (fProcElems) fProcElems->Add(fProcElem); - } - - // Get branch pointers - b_event_fType = fChain->GetBranch("fType[20]"); - b_fEventName = fChain->GetBranch("fEventName"); - b_event_fNtrack = fChain->GetBranch("fNtrack"); - b_event_fNseg = fChain->GetBranch("fNseg"); - b_event_fNvertex = fChain->GetBranch("fNvertex"); - b_event_fFlag = fChain->GetBranch("fFlag"); - b_event_fTemperature = fChain->GetBranch("fTemperature"); - b_event_fMeasures = fChain->GetBranch("fMeasures[10]"); - b_event_fMatrix = fChain->GetBranch("fMatrix[4][4]"); - b_fClosestDistance = fChain->GetBranch("fClosestDistance"); - b_event_fEvtHdr = fChain->GetBranch("fEvtHdr"); - b_fTracks = fChain->GetBranch("fTracks"); - b_fHighPt = fChain->GetBranch("fHighPt"); - b_fMuons = fChain->GetBranch("fMuons"); - b_event_fLastTrack = fChain->GetBranch("fLastTrack"); - b_event_fWebHistogram = fChain->GetBranch("fWebHistogram"); - b_fH = fChain->GetBranch("fH"); - b_event_fTriggerBits = fChain->GetBranch("fTriggerBits"); - b_event_fIsValid = fChain->GetBranch("fIsValid"); - - return kTRUE; -} - -#endif // #ifdef ProofEventProc_cxx diff --git a/tutorials/legacy/proof/ProofFriends.C b/tutorials/legacy/proof/ProofFriends.C deleted file mode 100644 index b2aa12451cf93..0000000000000 --- a/tutorials/legacy/proof/ProofFriends.C +++ /dev/null @@ -1,201 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofFriends -/// -/// Selector to process tree friends -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofFriends_cxx - -#include "ProofFriends.h" -#include "TCanvas.h" -#include "TColor.h" -#include "TH1F.h" -#include "TH2F.h" -#include "TList.h" -#include "TMath.h" -#include "TString.h" -#include "TStyle.h" - -//_____________________________________________________________________________ -ProofFriends::ProofFriends() -{ - // Constructor - - fXY = nullptr; - fZ = nullptr; - fR = nullptr; - fRZ = nullptr; - fPlot = kTRUE; - fDoFriends = kTRUE; -} - -//_____________________________________________________________________________ -void ProofFriends::Begin(TTree * /*tree*/) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - TNamed *out = (TNamed *) fInput->FindObject("PROOF_DONT_PLOT"); - if (out) fPlot = kFALSE; - out = (TNamed *) fInput->FindObject("PROOF_NO_FRIENDS"); - if (out) fDoFriends = kFALSE; -} - -//_____________________________________________________________________________ -void ProofFriends::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - TNamed *out = (TNamed *) fInput->FindObject("PROOF_NO_FRIENDS"); - if (out) fDoFriends = kFALSE; - - // Histograms - fXY = new TH2F("histo1", "y:x", 50, 5., 15., 50, 10., 30.); - fZ = new TH1F("histo2", "z , sqrt(dx*dx+dy*dy) < 1", 50, 0., 5.); - fZ->SetFillColor(kBlue); - fOutput->Add(fXY); - fOutput->Add(fZ); - if (fDoFriends) { - fR = new TH1F("histo3", "Tfrnd.r , sqrt(dx*dx+dy*dy) < 1, z < 1", 50, 5., 15.); - fRZ = new TH2F("histo4", "Tfrnd.r:z , sqrt(dx*dx+dy*dy) < 1, z < 1", 50, 0., 1., 50, 5., 15.); - fR->SetFillColor(kRed); - fOutput->Add(fR); - fOutput->Add(fRZ); - } -} - -//_____________________________________________________________________________ -Bool_t ProofFriends::Process(Long64_t entry) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either ProofFriends::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - // - // The processing can be stopped by calling Abort(). - // - // Use fStatus to set the return value of TTree::Process(). - // - // The return value is currently not used. - - // Read x and y, fill and apply cut - b_x->GetEntry(entry); - b_y->GetEntry(entry); - fXY->Fill(x, y, 1.); - Double_t dx = x-10.; - Double_t dy = y-20.; - Double_t xpy = TMath::Sqrt(dx*dx + dy*dy); - if (xpy > 1.) return kFALSE; - - // Read z, fill and apply cut - b_z->GetEntry(entry); - fZ->Fill(z, 1.); - if (z > 1.) return kFALSE; - - // Read r and fill - if (fDoFriends) { - b_r->GetEntry(entry); - fR->Fill(r, 1.); - fRZ->Fill(z, r, 1.); - } - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofFriends::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - -} - -//_____________________________________________________________________________ -void ProofFriends::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - if (!fPlot) return; - - gStyle->SetOptStat(1110); - // Create canvas - TCanvas *c1 = new TCanvas("c1","Proof ProofFriends canvas",200,10,700,700); - // Overall background - Int_t cb = TColor::GetColor("#ccffff"); - c1->SetFillColor(cb); - c1->SetBorderMode(0); - // 4 pads - c1->Divide(2, 2); - - Int_t cf = TColor::GetColor("#99cccc"); - TPad *p1 = nullptr; - if ((fXY = dynamic_cast(fOutput->FindObject("histo1")))) { - p1 = (TPad *) c1->cd(1); - p1->SetBorderMode(0); - p1->SetFrameFillColor(cf); - fXY->GetXaxis()->SetTitle("x"); - fXY->GetYaxis()->SetTitle("y"); - fXY->Draw(""); - } - - if ((fZ = dynamic_cast(fOutput->FindObject("histo2")))) { - p1 = (TPad *) c1->cd(2); - p1->SetBorderMode(0); - p1->SetFrameFillColor(cf); - fZ->GetXaxis()->SetTitle("z"); - fZ->GetYaxis()->SetTitle("N / 0.1"); - fZ->Draw(""); - } - - if (fDoFriends) { - - if ((fR = dynamic_cast(fOutput->FindObject("histo3")))) { - p1 = (TPad *) c1->cd(3); - p1->SetBorderMode(0); - p1->SetFrameFillColor(cf); - fR->GetXaxis()->SetTitle("Tfrnd.r"); - fR->GetYaxis()->SetTitle("N / 0.2"); - fR->Draw(); - } - - if ((fRZ = dynamic_cast(fOutput->FindObject("histo4")))) { - p1 = (TPad *) c1->cd(4); - p1->SetBorderMode(0); - p1->SetFrameFillColor(cf); - fRZ->GetXaxis()->SetTitle("z"); - fRZ->GetXaxis()->CenterTitle(true); - fRZ->GetXaxis()->SetTitleOffset(1.5); - fRZ->GetYaxis()->SetTitle("Tfrnd.r"); - fRZ->GetYaxis()->CenterTitle(true); - fRZ->GetYaxis()->SetTitleOffset(1.75); - fRZ->GetZaxis()->SetTitle("N / 0.1 / 0.2"); - fRZ->GetZaxis()->SetTitleOffset(1.25); - fRZ->Draw("lego"); - } - - } - - // Final update - c1->cd(); - c1->Update(); -} diff --git a/tutorials/legacy/proof/ProofFriends.h b/tutorials/legacy/proof/ProofFriends.h deleted file mode 100644 index 08d5b2fc65158..0000000000000 --- a/tutorials/legacy/proof/ProofFriends.h +++ /dev/null @@ -1,107 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofFriends -/// -/// Selector to process tree friends -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofFriends_h -#define ProofFriends_h - -#include "TChain.h" -#include "TSelector.h" - -class TH1F; -class TH2F; - -class ProofFriends : public TSelector { -public : - TTree *fChain; //!pointer to the analyzed TTree or TChain - - Bool_t fPlot; // Whether to plot the result - Bool_t fDoFriends; // Whether to use the friend tree - - // Specific members - TH2F *fXY; - TH1F *fZ; - TH1F *fR; - TH2F *fRZ; - - // Declaration of leaf types - Int_t Run; - Long64_t Event; - Float_t x; - Float_t y; - Float_t z; - - Float_t r; // The friend - - // List of branches - TBranch *b_Run; //! - TBranch *b_Event; //! - TBranch *b_x; //! - TBranch *b_y; //! - TBranch *b_z; //! - - TBranch *b_r; //! The friend branch - - ProofFriends(); - ~ProofFriends() override { } - Int_t Version() const override { return 2; } - void Begin(TTree *tree) override; - void SlaveBegin(TTree *tree) override; - void Init(TTree *tree) override; - Bool_t Notify() override; - Bool_t Process(Long64_t entry) override; - Int_t GetEntry(Long64_t entry, Int_t getall = 0) override { return fChain ? fChain->GetTree()->GetEntry(entry, getall) : 0; } - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override { fInput = input; } - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(ProofFriends,0); -}; - -#endif - -#ifdef ProofFriends_cxx -void ProofFriends::Init(TTree *tree) -{ - // The Init() function is called when the selector needs to initialize - // a new tree or chain. Typically here the branch addresses and branch - // pointers of the tree will be set. - // It is normally not necessary to make changes to the generated - // code, but the routine can be extended by the user if needed. - // Init() will be called many times when running on PROOF - // (once per file to be processed). - - // Set branch addresses and branch pointers - if (!tree) return; - fChain = tree; - fChain->SetMakeClass(1); - - fChain->SetBranchAddress("Run", &Run, &b_Run); - fChain->SetBranchAddress("Event", &Event, &b_Event); - fChain->SetBranchAddress("x", &x, &b_x); - fChain->SetBranchAddress("y", &y, &b_y); - fChain->SetBranchAddress("z", &z, &b_z); - - fChain->SetBranchAddress("r", &r, &b_r); // The friend -} - -Bool_t ProofFriends::Notify() -{ - // The Notify() function is called when a new file is opened. This - // can be either for a new TTree in a TChain or when when a new TTree - // is started when using PROOF. It is normally not necessary to make changes - // to the generated code, but the routine can be extended by the - // user if needed. The return value is currently not used. - - return kTRUE; -} - -#endif // #ifdef ProofFriends_cxx diff --git a/tutorials/legacy/proof/ProofNtuple.C b/tutorials/legacy/proof/ProofNtuple.C deleted file mode 100644 index 77587337f18ae..0000000000000 --- a/tutorials/legacy/proof/ProofNtuple.C +++ /dev/null @@ -1,294 +0,0 @@ -/// \file -/// \ingroup tutorial_legacy -/// -/// Selector to fill a simple ntuple -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofNtuple_cxx - -#include "ProofNtuple.h" -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -//_____________________________________________________________________________ -ProofNtuple::~ProofNtuple() -{ - // Destructor - - SafeDelete(fNtp); - SafeDelete(fNtp2); - SafeDelete(fFile); - SafeDelete(fRandom); -} - -//_____________________________________________________________________________ -void ProofNtuple::PlotNtuple(TNtuple *ntp, const char *ntptitle) -{ - // Make some plots from the ntuple 'ntp' - - // - // Create a canvas, with 2 pads - // - TCanvas *c1 = new TCanvas(Form("cv-%s", ntp->GetName()), ntptitle,800,10,700,780); - c1->Divide(1,2); - TPad *pad1 = (TPad *) c1->GetPad(1); - TPad *pad2 = (TPad *) c1->GetPad(2); - // - // Display a function of one ntuple column imposing a condition - // on another column. - pad1->cd(); - pad1->SetGrid(); - pad1->SetLogy(); - pad1->GetFrame()->SetFillColor(15); - ntp->SetLineColor(1); - ntp->SetFillStyle(1001); - ntp->SetFillColor(45); - ntp->Draw("3*px+2","px**2+py**2>1"); - ntp->SetFillColor(38); - ntp->Draw("2*px+2","pz>2","same"); - ntp->SetFillColor(5); - ntp->Draw("1.3*px+2","(px^2+py^2>4) && py>0","same"); - pad1->RedrawAxis(); - - // - // Display a 3-D scatter plot of 3 columns. Superimpose a different selection. - pad2->cd(); - ntp->Draw("pz:py:px","(pz<10 && pz>6)+(pz<4 && pz>3)"); - ntp->SetMarkerColor(4); - ntp->Draw("pz:py:px","pz<6 && pz>4","same"); - ntp->SetMarkerColor(5); - ntp->Draw("pz:py:px","pz<4 && pz>3","same"); - TPaveText *l2 = new TPaveText(0.,0.6,0.9,0.95); - l2->SetFillColor(42); - l2->SetTextAlign(12); - l2->AddText("You can interactively rotate this view in 2 ways:"); - l2->AddText(" - With the RotateCube in clicking in this pad"); - l2->AddText(" - Selecting View with x3d in the View menu"); - l2->Draw(); - - // Final update - c1->cd(); - c1->Update(); -} - -//_____________________________________________________________________________ -void ProofNtuple::Begin(TTree * /*tree*/) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - TNamed *out = (TNamed *) fInput->FindObject("PROOF_NTUPLE_DONT_PLOT"); - if (out) fPlotNtuple = kFALSE; -} - -//_____________________________________________________________________________ -void ProofNtuple::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - // We may be creating a dataset or a merge file: check it - TNamed *nm = dynamic_cast(fInput->FindObject("SimpleNtuple.root")); - if (nm) { - // Just create the object - UInt_t opt = TProofOutputFile::kRegister | TProofOutputFile::kOverwrite | TProofOutputFile::kVerify; - fProofFile = new TProofOutputFile("SimpleNtuple.root", - TProofOutputFile::kDataset, opt, nm->GetTitle()); - } else { - // For the ntuple, we use the automatic file merging facility - // Check if an output URL has been given - TNamed *out = (TNamed *) fInput->FindObject("PROOF_OUTPUTFILE_LOCATION"); - Info("SlaveBegin", "PROOF_OUTPUTFILE_LOCATION: %s", (out ? out->GetTitle() : "undef")); - fProofFile = new TProofOutputFile("SimpleNtuple.root", (out ? out->GetTitle() : "M")); - out = (TNamed *) fInput->FindObject("PROOF_OUTPUTFILE"); - if (out) fProofFile->SetOutputFileName(out->GetTitle()); - } - - // Open the file - fFile = fProofFile->OpenFile("RECREATE"); - if (fFile && fFile->IsZombie()) SafeDelete(fFile); - - // Cannot continue - if (!fFile) { - Info("SlaveBegin", "could not create '%s': instance is invalid!", fProofFile->GetName()); - return; - } - - // Now we create the ntuple - fNtp = new TNtuple("ntuple","Demo ntuple","px:py:pz:random:i"); - // File resident - fNtp->SetDirectory(fFile); - fNtp->AutoSave(); - - // Now we create the second ntuple - fNtp2 = new TNtuple("ntuple2","Demo ntuple2","vx:vy:vz"); - // File resident - fNtp2->SetDirectory(fFile); - fNtp2->AutoSave(); - - // Should we generate the random numbers or take them from the ntuple ? - TNamed *unr = (TNamed *) fInput->FindObject("PROOF_USE_NTP_RNDM"); - if (unr) { - // Get the ntuple from the input list - if (!(fNtpRndm = dynamic_cast(fInput->FindObject("NtpRndm")))) { - Warning("SlaveBegin", - "asked to use rndm ntuple but 'NtpRndm' not found in the" - " input list! Using the random generator"); - fInput->Print(); - } else { - Info("SlaveBegin", "taking randoms from input ntuple 'NtpRndm'"); - } - } - - // Init the random generator, if required - if (!fNtpRndm) fRandom = new TRandom3(0); -} - -//_____________________________________________________________________________ -Bool_t ProofNtuple::Process(Long64_t entry) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either ProofNtuple::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - // - // The processing can be stopped by calling Abort(). - // - // Use fStatus to set the return value of TTree::Process(). - // - // The return value is currently not used. - - if (!fNtp) return kTRUE; - - // Fill ntuple - Float_t px, py, random; - if (fNtpRndm) { - // Get the entry - Float_t *ar = fNtpRndm->GetArgs(); - Long64_t ent = entry % fNtpRndm->GetEntries(); - fNtpRndm->GetEntry(ent); - random = ar[0]; - px = (Float_t) TMath::ErfInverse((Double_t)(ar[1]*2 - 1.)) * TMath::Sqrt(2.); - py = (Float_t) TMath::ErfInverse((Double_t)(ar[2]*2 - 1.)) * TMath::Sqrt(2.); - } else if (fRandom) { - fRandom->Rannor(px,py); - random = fRandom->Rndm(); - } else { - Abort("no way to get random numbers! Stop processing", kAbortProcess); - return kTRUE; - } - Float_t pz = px*px + py*py; - Int_t i = (Int_t) entry; - fNtp->Fill(px,py,pz,random,i); - - if (!fNtp2) return kTRUE; - - // The second ntuple - Float_t vz = random * 2. - 1.; - fNtp2->Fill(px,py,vz); - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofNtuple::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - - // Write the ntuple to the file - if (fFile) { - if (!fNtp) { - Error("SlaveTerminate", "'ntuple' is undefined!"); - return; - } - Bool_t cleanup = kFALSE; - TDirectory *savedir = gDirectory; - if (fNtp->GetEntries() > 0) { - fFile->cd(); - fNtp->Write(nullptr, TObject::kOverwrite); - if (fNtp2 && fNtp2->GetEntries() > 0) fNtp2->Write(nullptr, TObject::kOverwrite); - fProofFile->Print(); - fOutput->Add(fProofFile); - } else { - cleanup = kTRUE; - } - fNtp->SetDirectory(nullptr); - if (fNtp2) fNtp2->SetDirectory(nullptr); - gDirectory = savedir; - fFile->Close(); - // Cleanup, if needed - if (cleanup) { - TUrl uf(*(fFile->GetEndpointUrl())); - SafeDelete(fFile); - gSystem->Unlink(uf.GetFile()); - SafeDelete(fProofFile); - } - } -} - -//_____________________________________________________________________________ -void ProofNtuple::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - // Do nothing is not requested (dataset creation run) - if (!fPlotNtuple) return; - - // Get the ntuple from the file - if ((fProofFile = - dynamic_cast(fOutput->FindObject("SimpleNtuple.root")))) { - - TString outputFile(fProofFile->GetOutputFileName()); - TString outputName(fProofFile->GetName()); - outputName += ".root"; - Printf("outputFile: %s", outputFile.Data()); - - // Read the ntuple from the file - fFile = TFile::Open(outputFile); - if (fFile) { - Printf("Managed to open file: %s", outputFile.Data()); - fNtp = (TNtuple *) fFile->Get("ntuple"); - } else { - Error("Terminate", "could not open file: %s", outputFile.Data()); - } - if (!fFile) return; - - } else { - Error("Terminate", "TProofOutputFile not found"); - return; - } - - // Plot ntuples - if (fNtp) PlotNtuple(fNtp, "proof ntuple"); - -} diff --git a/tutorials/legacy/proof/ProofNtuple.h b/tutorials/legacy/proof/ProofNtuple.h deleted file mode 100644 index ecf548cbcc5e2..0000000000000 --- a/tutorials/legacy/proof/ProofNtuple.h +++ /dev/null @@ -1,50 +0,0 @@ -/// \file -/// \ingroup tutorial_legacy -/// -/// Selector to fill a simple ntuple -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofNtuple_h -#define ProofNtuple_h - -#include - -class TFile; -class TProofOutputFile; -class TNtuple; -class TRandom3; - -class ProofNtuple : public TSelector { -public : - - // Specific members - TFile *fFile; - TProofOutputFile *fProofFile; // For optimized merging of the ntuple - TNtuple *fNtp; - TNtuple *fNtp2; // To test double TTree in the same file - TRandom3 *fRandom; - Bool_t fPlotNtuple; - TNtuple *fNtpRndm; // Ntuple with random numbers - - ProofNtuple() : fFile(nullptr), fProofFile(nullptr), fNtp(nullptr), fNtp2(nullptr), fRandom(nullptr), fPlotNtuple(kTRUE), fNtpRndm(nullptr) { } - ~ProofNtuple() override; - Int_t Version() const override { return 2; } - void Begin(TTree *tree) override; - void SlaveBegin(TTree *tree) override; - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override { fInput = input; } - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - - void PlotNtuple(TNtuple *, const char *); - - ClassDefOverride(ProofNtuple,0); -}; - -#endif diff --git a/tutorials/legacy/proof/ProofPythia.C b/tutorials/legacy/proof/ProofPythia.C deleted file mode 100644 index d8a0ea155de28..0000000000000 --- a/tutorials/legacy/proof/ProofPythia.C +++ /dev/null @@ -1,171 +0,0 @@ -/// \file -/// \ingroup tutorial_legacy -/// -/// Selector to generate Monte Carlo events with Pythia8 -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofPythia_cxx - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include "TClonesArray.h" -#include "TParticle.h" -#include "TDatabasePDG.h" - -#include "ProofPythia.h" -#include "TPythia8.h" - -//_____________________________________________________________________________ -ProofPythia::ProofPythia() -{ - // Constructor - - fHist = 0; - fPt = 0; - fEta = 0; - fPythia = 0; - fP = 0; -} - -//_____________________________________________________________________________ -ProofPythia::~ProofPythia() -{ - // Destructor - - SafeDelete(fPythia); - SafeDelete(fP); -} - -//_____________________________________________________________________________ -void ProofPythia::Begin(TTree * /*tree*/) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - Info("Begin", "starting a simple exercise with process option: %s", option.Data()); -} - -//_____________________________________________________________________________ -void ProofPythia::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - // Histograms - fTot = new TH1F("histo1", "total multiplicity", 25, 0.5, 2500.5); - fHist = new TH1F("histo2", "charged multiplicity", 20, 0.5, 500.5); - fPt = new TH1F("histo3", "particles pT", 100, 0., 10); - fEta = new TH1F("histo4", "particles Eta", 100, -10., 10); - fTot->SetFillColor(kBlue); - fHist->SetFillColor(kRed); - fOutput->Add(fTot); - fOutput->Add(fHist); - fOutput->Add(fPt); - fOutput->Add(fEta); - - fPythia = new TPythia8(); - // Configure - fPythia->SetName("pythia8"); - fPythia->ReadConfigFile("pythia8/main03.cmnd"); - - // Initialize - fPythia->Initialize( 2212, 2212, 14000.); - fP = new TClonesArray("TParticle", 1000); - -} - -//_____________________________________________________________________________ -Bool_t ProofPythia::Process(Long64_t entry) -{ - // Main event loop - - fPythia->GenerateEvent(); - if (entry < 2) - fPythia->EventListing(); - fPythia->ImportParticles(fP, "All"); - Int_t nTot = fPythia->GetN(); - fPythia->ImportParticles(fP, "All"); - Int_t np = fP->GetEntriesFast(); - // Particle loop - Int_t nCharged = 0; - for (Int_t ip = 0; ip < np; ip++) { - TParticle* part = (TParticle*) fP->At(ip); - Int_t ist = part->GetStatusCode(); - Int_t pdg = part->GetPdgCode(); - if (ist != 1) continue; - Float_t charge = TDatabasePDG::Instance()->GetParticle(pdg)->Charge(); - if (charge == 0.) continue; - nCharged++; - Float_t eta = part->Eta(); - Float_t pt = part->Pt(); - if (pt > 0.) fPt->Fill(pt); - if ((eta > -10) && (eta < 10)) fEta->Fill(eta); - } - fHist->Fill(nCharged); - fTot->Fill(nTot); - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofPythia::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. -} - -//_____________________________________________________________________________ -void ProofPythia::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - // - // Create canvas - // - TCanvas *c1 = new TCanvas("c1","Proof ProofPythia canvas",200,10,700,700); - c1->Divide(2, 2); - - if ((fTot = dynamic_cast(fOutput->FindObject("histo1")))) { - c1->cd(1); - fTot->Draw("h"); - } - - if ((fHist = dynamic_cast(fOutput->FindObject("histo2")))) { - c1->cd(2); - fHist->Draw("h"); - } - - if ((fPt = dynamic_cast(fOutput->FindObject("histo3")))) { - c1->cd(3); - fPt->Draw("h"); - } - - if ((fEta = dynamic_cast(fOutput->FindObject("histo4")))) { - c1->cd(4); - fEta->Draw("h"); - } - - // Final update - c1->cd(); - c1->Update(); -} diff --git a/tutorials/legacy/proof/ProofPythia.h b/tutorials/legacy/proof/ProofPythia.h deleted file mode 100644 index 3124a7ad0908e..0000000000000 --- a/tutorials/legacy/proof/ProofPythia.h +++ /dev/null @@ -1,46 +0,0 @@ -/// \file -/// \ingroup tutorial_legacy -/// -/// Selector to generate Monte Carlo events with Pythia8 -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofPythia_h -#define ProofPythia_h - -#include - -class TClonesArray; -class TH1F; -class TPythia8; - -class ProofPythia : public TSelector { -public : - - // Specific members - TH1F *fTot; - TH1F *fHist; - TH1F *fPt; - TH1F *fEta; - TPythia8 *fPythia; - TClonesArray *fP; - - ProofPythia(); - ~ProofPythia() override; - Int_t Version() const override { return 2; } - void Begin(TTree *tree) override; - void SlaveBegin(TTree *tree) override; - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override { fInput = input; } - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(ProofPythia,0); -}; - -#endif diff --git a/tutorials/legacy/proof/ProofSimple.C b/tutorials/legacy/proof/ProofSimple.C deleted file mode 100644 index 5d940fed9466d..0000000000000 --- a/tutorials/legacy/proof/ProofSimple.C +++ /dev/null @@ -1,604 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofSimple -/// -/// Selector to fill a set of histograms -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofSimple_cxx - -#include "ProofSimple.h" -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include "TProof.h" -#include -#include -#include -#include -#include -#include - -//_____________________________________________________________________________ -ProofSimple::ProofSimple() -{ - // Constructor - - fNhist = -1; - fHist = nullptr; - fNhist3 = -1; - fHist3 = nullptr; - fRandom = nullptr; - fHLab = nullptr; - fFile = nullptr; - fProofFile = nullptr; - fNtp = nullptr; - fHasNtuple = 0; - fPlotNtuple = kFALSE; -} - -//_____________________________________________________________________________ -ProofSimple::~ProofSimple() -{ - // Destructor - - if (fFile) { - SafeDelete(fNtp); - SafeDelete(fFile); - } - SafeDelete(fRandom); -} - -//_____________________________________________________________________________ -void ProofSimple::Begin(TTree * /*tree*/) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - Ssiz_t iopt = kNPOS; - - // Histos array - if (fInput->FindObject("ProofSimple_NHist")) { - TParameter *p = - dynamic_cast*>(fInput->FindObject("ProofSimple_NHist")); - fNhist = (p) ? (Int_t) p->GetVal() : fNhist; - } else if ((iopt = option.Index("nhist=")) != kNPOS) { - TString s; - Ssiz_t from = iopt + strlen("nhist="); - if (option.Tokenize(s, from, ";") && s.IsDigit()) fNhist = s.Atoi(); - } - if (fNhist < 1) { - Abort("fNhist must be > 0! Hint: proof->SetParameter(\"ProofSimple_NHist\"," - " (Long_t) )", kAbortProcess); - return; - } - - if (fInput->FindObject("ProofSimple_NHist3")) { - TParameter *p = - dynamic_cast*>(fInput->FindObject("ProofSimple_NHist3")); - fNhist3 = (p) ? (Int_t) p->GetVal() : fNhist3; - } else if ((iopt = option.Index("nhist3=")) != kNPOS) { - TString s; - Ssiz_t from = iopt + strlen("nhist3="); - if (option.Tokenize(s, from, ";") && s.IsDigit()) fNhist3 = s.Atoi(); - } - - // Ntuple - TNamed *nm = dynamic_cast(fInput->FindObject("ProofSimple_Ntuple")); - if (nm) { - - // Title is in the form - // merge merge via file - // | location of the output file if merge - // |retrieve retrieve to client machine - // dataset create a dataset - // | dataset name (default: dataset_ntuple) - // |plot for a final plot - // or other keep in memory - - fHasNtuple = 1; - - TString ontp(nm->GetTitle()); - if (ontp.Contains("|plot") || ontp == "plot") { - fPlotNtuple = kTRUE; - ontp.ReplaceAll("|plot", ""); - if (ontp == "plot") ontp = ""; - } - if (ontp.BeginsWith("dataset")) fHasNtuple = 2; - } -} - -//_____________________________________________________________________________ -void ProofSimple::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - Ssiz_t iopt = kNPOS; - - // Histos array - if (fInput->FindObject("ProofSimple_NHist")) { - TParameter *p = - dynamic_cast*>(fInput->FindObject("ProofSimple_NHist")); - fNhist = (p) ? (Int_t) p->GetVal() : fNhist; - } else if ((iopt = option.Index("nhist=")) != kNPOS) { - TString s; - Ssiz_t from = iopt + strlen("nhist="); - if (option.Tokenize(s, from, ";") && s.IsDigit()) fNhist = s.Atoi(); - } - if (fNhist < 1) { - Abort("fNhist must be > 0! Hint: proof->SetParameter(\"ProofSimple_NHist\"," - " (Long_t) )", kAbortProcess); - return; - } - fHist = new TH1F*[fNhist]; - - TString hn; - // Create the histogram - for (Int_t i=0; i < fNhist; i++) { - hn.Form("h%d",i); - fHist[i] = new TH1F(hn.Data(), hn.Data(), 100, -3., 3.); - fHist[i]->SetFillColor(kRed); - fOutput->Add(fHist[i]); - } - - // 3D Histos array - if (fInput->FindObject("ProofSimple_NHist3")) { - TParameter *p = - dynamic_cast*>(fInput->FindObject("ProofSimple_NHist3")); - fNhist3 = (p) ? (Int_t) p->GetVal() : fNhist3; - } else if ((iopt = option.Index("nhist3=")) != kNPOS) { - TString s; - Ssiz_t from = iopt + strlen("nhist3="); - if (option.Tokenize(s, from, ";") && s.IsDigit()) fNhist3 = s.Atoi(); - } - if (fNhist3 > 0) { - fHist3 = new TH3F*[fNhist3]; - Info("Begin", "%d 3D histograms requested", fNhist3); - // Create the 3D histogram - for (Int_t i=0; i < fNhist3; i++) { - hn.Form("h%d_3d",i); - fHist3[i] = new TH3F(hn.Data(), hn.Data(), - 100, -3., 3., 100, -3., 3., 100, -3., 3.); - fOutput->Add(fHist3[i]); - } - } - - // Histo with labels - if (fInput->FindObject("ProofSimple_TestLabelMerging")) { - fHLab = new TH1F("hlab", "Test merging of histograms with automatic labels", 10, 0., 10.); - fOutput->Add(fHLab); - } - - // Ntuple - TNamed *nm = dynamic_cast(fInput->FindObject("ProofSimple_Ntuple")); - if (nm) { - - // Title is in the form - // merge merge via file - // | location of the output file if merge - // |retrieve retrieve to client machine - // dataset create a dataset - // | dataset name (default: dataset_ntuple) - // |plot for a final plot - // or other keep in memory - - fHasNtuple = 1; - - TString ontp(nm->GetTitle()); - if (ontp.Contains("|plot") || ontp == "plot") { - fPlotNtuple = kTRUE; - ontp.ReplaceAll("|plot", ""); - if (ontp == "plot") ontp = ""; - } - TString locfn("SimpleNtuple.root"); - if (ontp.BeginsWith("merge")) { - ontp.Replace(0,5,""); - fProofFile = new TProofOutputFile(locfn, "M"); - TString fn; - Ssiz_t iret = ontp.Index("|retrieve"); - if (iret != kNPOS) { - fProofFile->SetRetrieve(kTRUE); - TString rettag("|retrieve"); - if ((iret = ontp.Index("|retrieve=")) != kNPOS) { - rettag += "="; - fn = ontp(iret + rettag.Length(), ontp.Length() - iret - rettag.Length()); - if ((iret = fn.Index('|')) != kNPOS) fn.Remove(iret); - rettag += fn; - } - ontp.ReplaceAll(rettag, ""); - } - Ssiz_t iof = ontp.Index('|'); - if (iof != kNPOS) ontp.Remove(0, iof + 1); - if (!ontp.IsNull()) { - fProofFile->SetOutputFileName(ontp.Data()); - if (fn.IsNull()) fn = gSystem->BaseName(TUrl(ontp.Data(), kTRUE).GetFile()); - } - if (fn.IsNull()) fn = locfn; - // This will be the final file on the client, the case there is one - fProofFile->SetTitle(fn); - } else if (ontp.BeginsWith("dataset")) { - ontp.Replace(0,7,""); - Ssiz_t iof = ontp.Index("|"); - if (iof != kNPOS) ontp.Remove(0, iof + 1); - TString dsname = (!ontp.IsNull()) ? ontp.Data() : "dataset_ntuple"; - UInt_t opt = TProofOutputFile::kRegister | TProofOutputFile::kOverwrite | TProofOutputFile::kVerify; - fProofFile = new TProofOutputFile("SimpleNtuple.root", - TProofOutputFile::kDataset, opt, dsname.Data()); - fHasNtuple = 2; - } else if (!ontp.IsNull()) { - Warning("SlaveBegin", "ntuple options unknown: ignored (%s)", ontp.Data()); - } - - // Open the file, if required - if (fProofFile) { - // Open the file - fFile = fProofFile->OpenFile("RECREATE"); - if (fFile && fFile->IsZombie()) SafeDelete(fFile); - - // Cannot continue - if (!fFile) { - Info("SlaveBegin", "could not create '%s': instance is invalid!", fProofFile->GetName()); - return; - } - } - - // Now we create the ntuple - fNtp = new TNtuple("ntuple","Demo ntuple","px:py:pz:random:i"); - // File resident, if required - if (fFile) { - fNtp->SetDirectory(fFile); - fNtp->AutoSave(); - } else { - fOutput->Add(fNtp); - } - } - - // Set random seed - fRandom = new TRandom3(0); -} - -//_____________________________________________________________________________ -Bool_t ProofSimple::Process(Long64_t entry) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either ProofSimple::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - // - // The processing can be stopped by calling Abort(). - // - // Use fStatus to set the return value of TTree::Process(). - // - // The return value is currently not used. - - for (Int_t i=0; i < fNhist; i++) { - if (fRandom && fHist[i]) { - Double_t x = fRandom->Gaus(0.,1.); - fHist[i]->Fill(x); - } - } - for (Int_t i=0; i < fNhist3; i++) { - if (fRandom && fHist3[i]) { - Double_t x = fRandom->Gaus(0.,1.); - fHist3[i]->Fill(x,x,x); - } - } - if (fHLab && fRandom) { - TSortedList sortl; - Float_t rr[10]; - fRandom->RndmArray(10, rr); - for (Int_t i=0; i < 10; i++) { - sortl.Add(new TParameter(TString::Format("%f",rr[i]), i)); - } - TIter nxe(&sortl); - TParameter *pi = nullptr; - while ((pi = (TParameter *) nxe())) { - fHLab->Fill(TString::Format("hl%d", pi->GetVal()), pi->GetVal()); - } - } - if (fNtp) FillNtuple(entry); - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofSimple::FillNtuple(Long64_t entry) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either ProofNtuple::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - // - // The processing can be stopped by calling Abort(). - // - // Use fStatus to set the return value of TTree::Process(). - // - // The return value is currently not used. - - if (!fNtp) return; - - // Fill ntuple - Float_t px, py, random; - if (fRandom) { - fRandom->Rannor(px,py); - random = fRandom->Rndm(); - } else { - Abort("no way to get random numbers! Stop processing", kAbortProcess); - return; - } - Float_t pz = px*px + py*py; - Int_t i = (Int_t) entry; - fNtp->Fill(px,py,pz,random,i); - - return; -} - - -//_____________________________________________________________________________ -void ProofSimple::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - - // Write the ntuple to the file - if (fFile) { - if (!fNtp) { - Error("SlaveTerminate", "'ntuple' is undefined!"); - return; - } - Bool_t cleanup = kFALSE; - TDirectory *savedir = gDirectory; - if (fNtp->GetEntries() > 0) { - fFile->cd(); - fNtp->Write(); - fProofFile->Print(); - fOutput->Add(fProofFile); - } else { - cleanup = kTRUE; - } - fNtp->SetDirectory(nullptr); - gDirectory = savedir; - fFile->Close(); - // Cleanup, if needed - if (cleanup) { - TUrl uf(*(fFile->GetEndpointUrl())); - SafeDelete(fFile); - gSystem->Unlink(uf.GetFile()); - SafeDelete(fProofFile); - } - } -} - -//_____________________________________________________________________________ -void ProofSimple::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - // - // Create a canvas, with 100 pads - // - TCanvas *c1 = (TCanvas *) gDirectory->FindObject("c1"); - if (c1) { - gDirectory->Remove(c1); - delete c1; - } - c1 = new TCanvas("c1","Proof ProofSimple canvas",200,10,700,700); - Int_t nside = (Int_t)TMath::Sqrt((Float_t)fNhist); - nside = (nside*nside < fNhist) ? nside+1 : nside; - c1->Divide(nside,nside,0,0); - - Bool_t tryfc = kFALSE; - TH1F *h = nullptr; - for (Int_t i=0; i < fNhist; i++) { - if (!(h = dynamic_cast(TProof::GetOutput(Form("h%d",i), fOutput)))) { - // Not found: try TFileCollection - tryfc = kTRUE; - break; - } - c1->cd(i+1); - h->DrawCopy(); - } - - // If the histograms are not found they may be in files: is there a file collection? - if (tryfc && GetHistosFromFC(c1) != 0) { - Warning("Terminate", "histograms not found"); - } else { - // Final update - c1->cd(); - c1->Update(); - } - - // Analyse hlab, if there - if (fHLab && !gROOT->IsBatch()) { - // Printout - Int_t nb = fHLab->GetNbinsX(); - if (nb > 0) { - Double_t entb = fHLab->GetEntries() / nb; - if (entb) { - for (Int_t i = 0; i < nb; i++) { - TString lab = TString::Format("hl%d", i); - Int_t ib = fHLab->GetXaxis()->FindBin(lab); - Info("Terminate"," %s [%d]:\t%f", lab.Data(), ib, fHLab->GetBinContent(ib)/entb); - } - } else - Warning("Terminate", "no entries in the hlab histogram!"); - } - } - - // Process the ntuple, if required - if (fHasNtuple != 1 || !fPlotNtuple) return; - - if (!(fNtp = dynamic_cast(TProof::GetOutput("ntuple", fOutput)))) { - // Get the ntuple from the file - if ((fProofFile = - dynamic_cast(fOutput->FindObject("SimpleNtuple.root")))) { - - TString outputFile(fProofFile->GetOutputFileName()); - TString outputName(fProofFile->GetName()); - outputName += ".root"; - Printf("outputFile: %s", outputFile.Data()); - - // Read the ntuple from the file - fFile = TFile::Open(outputFile); - if (fFile) { - Printf("Managed to open file: %s", outputFile.Data()); - fNtp = (TNtuple *) fFile->Get("ntuple"); - } else { - Error("Terminate", "could not open file: %s", outputFile.Data()); - } - if (!fFile) return; - - } else { - Error("Terminate", "TProofOutputFile not found"); - return; - } - } - // Plot ntuples - if (fNtp) PlotNtuple(fNtp, "proof ntuple"); -} - -//_____________________________________________________________________________ -void ProofSimple::PlotNtuple(TNtuple *ntp, const char *ntptitle) -{ - // Make some plots from the ntuple 'ntp' - - // - // Create a canvas, with 2 pads - // - TCanvas *c1 = new TCanvas(Form("cv-%s", ntp->GetName()), ntptitle,800,10,700,780); - c1->Divide(1,2); - TPad *pad1 = (TPad *) c1->GetPad(1); - TPad *pad2 = (TPad *) c1->GetPad(2); - // - // Display a function of one ntuple column imposing a condition - // on another column. - pad1->cd(); - pad1->SetGrid(); - pad1->SetLogy(); - pad1->GetFrame()->SetFillColor(15); - ntp->SetLineColor(1); - ntp->SetFillStyle(1001); - ntp->SetFillColor(45); - ntp->Draw("3*px+2","px**2+py**2>1"); - ntp->SetFillColor(38); - ntp->Draw("2*px+2","pz>2","same"); - ntp->SetFillColor(5); - ntp->Draw("1.3*px+2","(px^2+py^2>4) && py>0","same"); - pad1->RedrawAxis(); - - // - // Display a 3-D scatter plot of 3 columns. Superimpose a different selection. - pad2->cd(); - ntp->Draw("pz:py:px","(pz<10 && pz>6)+(pz<4 && pz>3)"); - ntp->SetMarkerColor(4); - ntp->Draw("pz:py:px","pz<6 && pz>4","same"); - ntp->SetMarkerColor(5); - ntp->Draw("pz:py:px","pz<4 && pz>3","same"); - TPaveText *l2 = new TPaveText(0.,0.6,0.9,0.95); - l2->SetFillColor(42); - l2->SetTextAlign(12); - l2->AddText("You can interactively rotate this view in 2 ways:"); - l2->AddText(" - With the RotateCube in clicking in this pad"); - l2->AddText(" - Selecting View with x3d in the View menu"); - l2->Draw(); - - // Final update - c1->cd(); - c1->Update(); -} - -//_____________________________________________________________________________ -Int_t ProofSimple::GetHistosFromFC(TCanvas *cv) -{ - // Check for the histograms in the files of a possible TFileCollection - - TIter nxo(fOutput); - TFileCollection *fc = nullptr; - Bool_t fc_found = kFALSE, hs_found = kFALSE; - while ((fc = (TFileCollection *) nxo())) { - if (strcmp(fc->ClassName(), "TFileCollection")) continue; - fc_found = kTRUE; - if (!fHist) { - fHist = new TH1F*[fNhist]; - for (Int_t i = 0; i < fNhist; i++) { fHist[i] = nullptr; } - } else { - for (Int_t i = 0; i < fNhist; i++) { SafeDelete(fHist[i]); } - } - // Go through the list of files - TIter nxf(fc->GetList()); - TFileInfo *fi = nullptr; - while ((fi = (TFileInfo *) nxf())) { - TFile *f = TFile::Open(fi->GetCurrentUrl()->GetUrl()); - if (f) { - for (Int_t i = 0; i < fNhist; i++) { - TString hn = TString::Format("h%d", i); - TH1F *h = (TH1F *) f->Get(hn); - if (h) { - hs_found = kTRUE; - if (!fHist[i]) { - fHist[i] = (TH1F *) h->Clone(); - fHist[i]->SetDirectory(nullptr); - } else { - fHist[i]->Add(h); - } - } else { - Error("GetHistosFromFC", "histo '%s' not found in file '%s'", - hn.Data(), fi->GetCurrentUrl()->GetUrl()); - } - } - f->Close(); - } else { - Error("GetHistosFromFC", "file '%s' could not be open", fi->GetCurrentUrl()->GetUrl()); - } - } - if (hs_found) break; - } - if (!fc_found) return -1; - if (!hs_found) return -1; - - for (Int_t i = 0; i < fNhist; i++) { - cv->cd(i+1); - if (fHist[i]) { - fHist[i]->DrawCopy(); - } - } - Info("GetHistosFromFC", "histograms read from %d files in TFileCollection '%s'", - fc->GetList()->GetSize(), fc->GetName()); - // Done - return 0; -} diff --git a/tutorials/legacy/proof/ProofSimple.h b/tutorials/legacy/proof/ProofSimple.h deleted file mode 100644 index e2d12a7072a4c..0000000000000 --- a/tutorials/legacy/proof/ProofSimple.h +++ /dev/null @@ -1,66 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofSimple -/// -/// Selector to fill a set of histograms -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofSimple_h -#define ProofSimple_h - -#include - -class TH1F; -class TH3F; -class TFile; -class TProofOutputFile; -class TNtuple; -class TRandom3; -class TCanvas; - -class ProofSimple : public TSelector { -public : - - // Specific members - Int_t fNhist; - TH1F **fHist;//![fNhist] - Int_t fNhist3; - TH3F **fHist3;//![fNhist3] - TFile *fFile; - TProofOutputFile *fProofFile; // For optimized merging of the ntuple - TNtuple *fNtp; - Bool_t fPlotNtuple; - Int_t fHasNtuple; - TRandom3 *fRandom;//! - - TH1F *fHLab;//! - - ProofSimple(); - ~ProofSimple() override; - Int_t Version() const override { return 2; } - void Begin(TTree *tree) override; - void SlaveBegin(TTree *tree) override; - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override { fInput = input; } - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - - void FillNtuple(Long64_t entry); - void PlotNtuple(TNtuple *, const char *); - Int_t GetHistosFromFC(TCanvas *); - - // Setters and getters (for TDataMember) - Int_t GetNhist() { return fNhist; } - void SetNhist(Int_t nh) { fNhist = nh; } - Int_t GetNhist3() { return fNhist3; } - void SetNhist3(Int_t nh) { fNhist3 = nh; } - - ClassDefOverride(ProofSimple,3); -}; - -#endif diff --git a/tutorials/legacy/proof/ProofSimpleFile.C b/tutorials/legacy/proof/ProofSimpleFile.C deleted file mode 100644 index 83c7087038a50..0000000000000 --- a/tutorials/legacy/proof/ProofSimpleFile.C +++ /dev/null @@ -1,315 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofSimpleFile -/// -/// Selector to fill a set of histograms and merging via file -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofSimpleFile_cxx - -#include "ProofSimpleFile.h" -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -//_____________________________________________________________________________ -ProofSimpleFile::ProofSimpleFile() -{ - // Constructor - - fNhist = 16; - fHistTop = nullptr; - fHistDir = nullptr; - fRandom = nullptr; - fFile = nullptr; - fProofFile = nullptr; - fFileDir = nullptr; -} - -//_____________________________________________________________________________ -ProofSimpleFile::~ProofSimpleFile() -{ - // Destructor - - if (fRandom) delete fRandom; -} - -//_____________________________________________________________________________ -Int_t ProofSimpleFile::CreateHistoArrays() -{ - // Create the histogram arrays - - if (fNhist <= 0) { - Error("CreateHistoArrays", "fNhist must be positive!"); - return -1; - } - // Histos array - fHistTop = new TH1F*[fNhist]; - fHistDir = new TH1F*[fNhist]; - // Done - return 0; -} - -//_____________________________________________________________________________ -void ProofSimpleFile::Begin(TTree * /*tree*/) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - // Number of histograms (needed in terminate) - Ssiz_t iopt = kNPOS; - if (fInput->FindObject("ProofSimpleFile_NHist")) { - TParameter *p = - dynamic_cast*>(fInput->FindObject("ProofSimpleFile_NHist")); - fNhist = (p) ? (Int_t) p->GetVal() : fNhist; - } else if ((iopt = option.Index("nhist=")) != kNPOS) { - TString s; - Ssiz_t from = iopt + strlen("nhist="); - if (option.Tokenize(s, from, ";") && s.IsDigit()) fNhist = s.Atoi(); - } -} - -//_____________________________________________________________________________ -void ProofSimpleFile::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - // Number of histograms (needed in terminate) - Ssiz_t iopt = kNPOS; - if (fInput->FindObject("ProofSimpleFile_NHist")) { - TParameter *p = - dynamic_cast*>(fInput->FindObject("ProofSimpleFile_NHist")); - fNhist = (p) ? (Int_t) p->GetVal() : fNhist; - } else if ((iopt = option.Index("nhist=")) != kNPOS) { - TString s; - Ssiz_t from = iopt + strlen("nhist="); - if (option.Tokenize(s, from, ";") && s.IsDigit()) fNhist = s.Atoi(); - } - - // The file for merging - fProofFile = new TProofOutputFile("SimpleFile.root", "M"); - TNamed *out = (TNamed *) fInput->FindObject("PROOF_OUTPUTFILE"); - if (out) fProofFile->SetOutputFileName(out->GetTitle()); - TDirectory *savedir = gDirectory; - fFile = fProofFile->OpenFile("RECREATE"); - if (fFile && fFile->IsZombie()) SafeDelete(fFile); - savedir->cd(); - - // Cannot continue - if (!fFile) { - TString amsg = TString::Format("ProofSimpleFile::SlaveBegin: could not create '%s':" - " instance is invalid!", fProofFile->GetName()); - Abort(amsg, kAbortProcess); - return; - } - - // Histos arrays - if (CreateHistoArrays() != 0) { - Abort("ProofSimpleFile::SlaveBegin: could not create histograms", kAbortProcess); - return; - } - - // Create directory - if (!(fFileDir = fFile->mkdir("blue"))) { - Abort("ProofSimpleFile::SlaveBegin: could not create directory 'blue' in file!", - kAbortProcess); - return; - } - - // Create the histograms - for (Int_t i=0; i < fNhist; i++) { - fHistTop[i] = new TH1F(Form("ht%d",i), Form("ht%d",i), 100, -3., 3.); - fHistTop[i]->SetFillColor(kRed); - fHistTop[i]->SetDirectory(fFile); - fHistDir[i] = new TH1F(Form("hd%d",i), Form("hd%d",i), 100, -3., 3.); - fHistDir[i]->SetFillColor(kBlue); - fHistDir[i]->SetDirectory(fFileDir); - } - - // Set random seed - fRandom = new TRandom3(0); -} - -//_____________________________________________________________________________ -Bool_t ProofSimpleFile::Process(Long64_t) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either ProofSimpleFile::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - // - // The processing can be stopped by calling Abort(). - // - // Use fStatus to set the return value of TTree::Process(). - // - // The return value is currently not used. - - for (Int_t i=0; i < fNhist; i++) { - if (fRandom && fHistTop[i] && fHistDir[i]) { - fHistTop[i]->Fill(fRandom->Gaus(0.,1.)); - fHistDir[i]->Fill(fRandom->Gaus(0.,1.)); - } - } - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofSimpleFile::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - - // Write histos to file - if (fFile) { - Bool_t cleanup = kTRUE; - TDirectory *savedir = gDirectory; - fFile->cd(); - for (Int_t i=0; i < fNhist; i++) { - if (fHistTop[i] && fHistTop[i]->GetEntries() > 0) { - fHistTop[i]->Write(); - fHistTop[i]->SetDirectory(nullptr); - cleanup = kFALSE; - } - } - // Change to subdirectory - fFileDir->cd(); - for (Int_t i=0; i < fNhist; i++) { - if (fHistDir[i] && fHistDir[i]->GetEntries() > 0) { - fHistDir[i]->Write(); - fHistDir[i]->SetDirectory(nullptr); - cleanup = kFALSE; - } - } - gDirectory = savedir; - fFile->Close(); - // Cleanup or register - if (cleanup) { - Info("SlaveTerminate", "nothing to save: just cleanup everything ..."); - TUrl uf(*(fFile->GetEndpointUrl())); - SafeDelete(fFile); - gSystem->Unlink(uf.GetFile()); - SafeDelete(fProofFile); - } else { - Info("SlaveTerminate", "objects saved into '%s%s': sending related TProofOutputFile ...", - fProofFile->GetFileName(), fProofFile->GetOptionsAnchor()); - fProofFile->Print(); - fOutput->Add(fProofFile); - } - } - -} - -//_____________________________________________________________________________ -void ProofSimpleFile::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - // Get the histos from the file - if ((fProofFile = - dynamic_cast(fOutput->FindObject("SimpleFile.root")))) { - - TString outputFile(fProofFile->GetOutputFileName()); - TString outputName(fProofFile->GetName()); - outputName += ".root"; - Printf("outputFile: %s", outputFile.Data()); - - // Read the ntuple from the file - if (!(fFile = TFile::Open(outputFile))) { - Error("Terminate", "could not open file: %s", outputFile.Data()); - return; - } - - } else { - Error("Terminate", "TProofOutputFile not found"); - return; - } - - // Histos arrays - if (CreateHistoArrays() != 0) { - Error("Terminate", "could not create histograms"); - return; - } - - // Top histos - PlotHistos(0); - // Dir histos - PlotHistos(1); -} - -//_____________________________________________________________________________ -void ProofSimpleFile::PlotHistos(Int_t opt) -{ - // Plot the histograms ina dedicated canvas - - // Create a canvas, with fNhist pads - if (opt == 0) { - TCanvas *c1 = new TCanvas("c1","ProofSimpleFile top dir canvas",200,10,700,700); - Int_t nside = (Int_t)TMath::Sqrt((Float_t)fNhist); - nside = (nside*nside < fNhist) ? nside+1 : nside; - c1->Divide(nside,nside,0,0); - - for (Int_t i=0; i < fNhist; i++) { - fHistTop[i] = (TH1F *) fFile->Get(TString::Format("ht%d",i)); - c1->cd(i+1); - if (fHistTop[i]) - fHistTop[i]->Draw(); - } - - // Final update - c1->cd(); - c1->Update(); - } else if (opt == 1) { - TCanvas *c2 = new TCanvas("c2","ProofSimpleFile 'blue' sub-dir canvas",400,60,700,700); - Int_t nside = (Int_t)TMath::Sqrt((Float_t)fNhist); - nside = (nside*nside < fNhist) ? nside+1 : nside; - c2->Divide(nside,nside,0,0); - - if ((fFileDir = (TDirectory *) fFile->Get("blue"))) { - for (Int_t i=0; i < fNhist; i++) { - fHistDir[i] = (TH1F *) fFileDir->Get(TString::Format("hd%d",i)); - c2->cd(i+1); - if (fHistDir[i]) - fHistDir[i]->Draw(); - } - } else { - Error("PlotHistos", "directory 'blue' not found in output file"); - } - - // Final update - c2->cd(); - c2->Update(); - } else { - Error("PlotHistos", "unknown option: %d", opt); - } -} diff --git a/tutorials/legacy/proof/ProofSimpleFile.h b/tutorials/legacy/proof/ProofSimpleFile.h deleted file mode 100644 index c007fbe0a419c..0000000000000 --- a/tutorials/legacy/proof/ProofSimpleFile.h +++ /dev/null @@ -1,52 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofSimpleFile -/// -/// Selector to fill a set of histograms and merging via file -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofSimpleFile_h -#define ProofSimpleFile_h - -#include - -class TH1F; -class TRandom3; -class TFile; -class TProofOutputFile; -class TDirectory; - -class ProofSimpleFile : public TSelector { -private: - Int_t CreateHistoArrays(); - void PlotHistos(Int_t opt = 0); -public : - - // Specific members - Int_t fNhist; - TH1F **fHistTop;//[fNhist] - TH1F **fHistDir;//[fNhist] - TRandom3 *fRandom; - TFile *fFile; - TProofOutputFile *fProofFile; // For merging via file - TDirectory *fFileDir; // Subdirectory for some histos - - ProofSimpleFile(); - ~ProofSimpleFile() override; - Int_t Version() const override { return 2; } - void Begin(TTree *tree) override; - void SlaveBegin(TTree *tree) override; - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override { fInput = input; } - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(ProofSimpleFile,0); -}; - -#endif diff --git a/tutorials/legacy/proof/ProofStdVect.C b/tutorials/legacy/proof/ProofStdVect.C deleted file mode 100644 index a742e6055f03f..0000000000000 --- a/tutorials/legacy/proof/ProofStdVect.C +++ /dev/null @@ -1,312 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofStdVec -/// -/// Selector for generic processing with stdlib collections -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofStdVect_cxx - -#include "ProofStdVect.h" -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -//_____________________________________________________________________________ -ProofStdVect::ProofStdVect() -{ - // Constructor - - fCreate = kFALSE; - fTree = nullptr; - fFile = nullptr; - fProofFile = nullptr; - fRandom = nullptr; - fHgood = nullptr; - fHbad = nullptr; -} - -//_____________________________________________________________________________ -ProofStdVect::~ProofStdVect() -{ - // Destructor - - SafeDelete(fTree); - SafeDelete(fFile); - SafeDelete(fRandom); -} - -//_____________________________________________________________________________ -void ProofStdVect::Begin(TTree * /*tree*/) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - // Dataset creation run? - if (fInput && fInput->FindObject("ProofStdVect_Create")) { - fCreate = kTRUE; - } else if (option.Contains("create")) { - fCreate = kTRUE; - } -} - -//_____________________________________________________________________________ -void ProofStdVect::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - // Dataset creation run? - if (fInput && fInput->FindObject("ProofStdVect_Create")) { - fCreate = kTRUE; - } else if (option.Contains("create")) { - fCreate = kTRUE; - } - - // If yes, create the output file ... - if (fCreate) { - // Just create the object - UInt_t opt = TProofOutputFile::kRegister | TProofOutputFile::kOverwrite | TProofOutputFile::kVerify; - fProofFile = new TProofOutputFile("ProofStdVect.root", - TProofOutputFile::kDataset, opt, "TestStdVect"); - - // Open the file - fFile = fProofFile->OpenFile("RECREATE"); - if (fFile && fFile->IsZombie()) SafeDelete(fFile); - - // Cannot continue - if (!fFile) { - Info("SlaveBegin", "could not create '%s': instance is invalid!", fProofFile->GetName()); - return; - } - - // Create a TTree - fTree = new TTree("stdvec", "Tree with std vector"); - fTree->Branch("Vb",&fVb); - fTree->Branch("Vfx",&fVfx); - fTree->Branch("Vfy",&fVfy); - // File resident - fTree->SetDirectory(fFile); - fTree->AutoSave(); - - // Init the random generator - fRandom = new TRandom3(0); - - } else { - // Create two histograms - fHgood = new TH1F("Hgood", "Good hits", 100., -2.5, 2.5); - fHbad = new TH1F("Hbad", "Bad hits", 100., -6., 6.); - fOutput->Add(fHgood); - fOutput->Add(fHbad); - } -} - -//_____________________________________________________________________________ -Bool_t ProofStdVect::Process(Long64_t entry) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either ProofStdVect::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - // - // The processing can be stopped by calling Abort(). - // - // Use fStatus to set the return value of TTree::Process(). - // - // The return value is currently not used. - - if (fCreate) { - if (!fTree) return kTRUE; - - // Number of vectors - Int_t nv = (Int_t) (entry % 10); - if (nv < 1) nv = 1; - - // Create vectors - for (Int_t i = 0; i < nv; i++) { - std::vector vb; - std::vector vfx, vfy; - Int_t np = (Int_t) (entry % 100); - if (np < 1) np = 1; - for (Int_t j = 0; j < np; j++) { - float x = (float)j; - float y = 5.*x; - Double_t sy = (Double_t) (0.1*y); - Double_t ym = fRandom->Gaus((Double_t)y, sy); - Double_t c2 = TMath::Abs((ym - y) / sy); - bool xb = (1. - TMath::Erfc(c2/TMath::Sqrt(2.)) > .95) ? false : true; - vb.push_back(xb); - vfx.push_back(x); - vfy.push_back(float(ym)); - } - fVb.push_back(vb); - fVfx.push_back(vfx); - fVfy.push_back(vfy); - } - - // Fill the tree - fTree->Fill(); - - // Clear the vectors - std::vector >::iterator ivb; - for (ivb = fVb.begin(); ivb != fVb.end(); ivb++) { - (*ivb).clear(); - } - fVb.clear(); - std::vector >::iterator ivf; - for (ivf = fVfx.begin(); ivf != fVfx.end(); ivf++) { - (*ivf).clear(); - } - fVfx.clear(); - for (ivf = fVfy.begin(); ivf != fVfy.end(); ivf++) { - (*ivf).clear(); - } - fVfy.clear(); - } else { - // Read the entry - GetEntry(entry); - // Plot normalized values for bad and good hits - for (UInt_t i = 0; i < fVfyr->size(); i++) { - std::vector &vb = fVbr->at(i); - std::vector &vfx = fVfxr->at(i); - std::vector &vfy = fVfyr->at(i); - for (UInt_t j = 0; j < vfy.size(); j++) { - Double_t ny = (vfy.at(j) - 5*vfx.at(j)) / (0.1 * 5 * vfx.at(j)); - if (vb.at(j) < 0.5) - fHbad->Fill(ny); - else - fHgood->Fill(ny); - } - } - } - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofStdVect::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - - // Nothing to do in read mode - if (!fCreate) return; - - // Write the ntuple to the file - if (fFile) { - if (!fTree) { - Error("SlaveTerminate", "'tree' is undefined!"); - return; - } - Bool_t cleanup = kFALSE; - TDirectory::TContext ctxt; - if (fTree->GetEntries() > 0) { - fFile->cd(); - fTree->Write(); - fProofFile->Print(); - fOutput->Add(fProofFile); - } else { - cleanup = kTRUE; - } - fTree->SetDirectory(nullptr); - fFile->Close(); - // Cleanup, if needed - if (cleanup) { - TUrl uf(*(fFile->GetEndpointUrl())); - SafeDelete(fFile); - gSystem->Unlink(uf.GetFile()); - SafeDelete(fProofFile); - } - } -} - -//_____________________________________________________________________________ -void ProofStdVect::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - - // Nothing to do in create mode - if (fCreate) return; - - // Create a canvas, with 2 pads - TCanvas *c1 = new TCanvas("cvstdvec", "Test StdVec", 800,10,700,780); - c1->Divide(1,2); - TPad *pad1 = (TPad *) c1->GetPad(1); - TPad *pad2 = (TPad *) c1->GetPad(2); - pad2->cd(); - if (fHbad) fHbad->Draw(); - pad1->cd(); - if (fHgood) fHgood->Draw(); - c1->cd(); - c1->Update(); -} - -//_____________________________________________________________________________ -void ProofStdVect::Init(TTree *tree) -{ - // The Init() function is called when the selector needs to initialize - // a new tree or chain. Typically here the branch addresses and branch - // pointers of the tree will be set. - // It is normally not necessary to make changes to the generated - // code, but the routine can be extended by the user if needed. - // Init() will be called many times when running on PROOF - // (once per file to be processed). - - // Nothing to do in create mode - if (fCreate) return; - - // Set object pointer - fVbr = nullptr; - fVfxr = nullptr; - fVfyr = nullptr; - // Set branch addresses and branch pointers - if (!tree) return; - fChain = tree; - fChain->SetMakeClass(1); - - fChain->SetBranchAddress("Vb", &fVbr, &b_Vb); - fChain->SetBranchAddress("Vfx", &fVfxr, &b_Vfx); - fChain->SetBranchAddress("Vfy", &fVfyr, &b_Vfy); -} - - -//_____________________________________________________________________________ -Bool_t ProofStdVect::Notify() -{ - // The Notify() function is called when a new file is opened. This - // can be either for a new TTree in a TChain or when when a new TTree - // is started when using PROOF. It is normally not necessary to make changes - // to the generated code, but the routine can be extended by the - // user if needed. The return value is currently not used. - - // Nothing to do in create mode - if (fCreate) return kTRUE; - Info("Notify","processing file: %s",fChain->GetCurrentFile()->GetName()); - - return kTRUE; -} diff --git a/tutorials/legacy/proof/ProofStdVect.h b/tutorials/legacy/proof/ProofStdVect.h deleted file mode 100644 index 3757da0883f94..0000000000000 --- a/tutorials/legacy/proof/ProofStdVect.h +++ /dev/null @@ -1,76 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofStdVec -/// -/// Selector for generic processing with stdlib collections -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofStdVect_h -#define ProofStdVect_h - -#include -#include - -#include -#ifdef __MAKECINT__ -#pragma link C++ class std::vector >+; -#pragma link C++ class std::vector >+; -#endif - -class TFile; -class TProofOutputFile; -class TTree; -class TRandom3; -class TH1F; - -class ProofStdVect : public TSelector { -public : - - // Specific members - Bool_t fCreate; //! True if in create files mode - - // Create mode - TTree *fTree; //! The tree filled in create mode - TFile *fFile; //! Output file in create mode - TProofOutputFile *fProofFile; //! For dataset creation in create mode - TRandom3 *fRandom; //! Random generator in create mode - TH1F *fHgood; //! Histogram with good hits - TH1F *fHbad; //! Histogram with bad hits - // Std vector members - std::vector > fVb; //! Booleans - std::vector > fVfx; //! Floats x - std::vector > fVfy; //! Floats y - - // Read mode - TTree *fChain; //!pointer to the analyzed TTree or TChain - // Declaration of leaf types - std::vector > *fVbr; //! - std::vector > *fVfxr; //! - std::vector > *fVfyr; //! - // List of branches - TBranch *b_Vb; //! - TBranch *b_Vfx; //! - TBranch *b_Vfy; //! - - ProofStdVect(); - ~ProofStdVect() override; - Int_t Version() const override { return 2; } - void Begin(TTree *tree) override; - void SlaveBegin(TTree *tree) override; - void Init(TTree *tree) override; - Bool_t Notify() override; - Bool_t Process(Long64_t entry) override; - Int_t GetEntry(Long64_t entry, Int_t getall = 0) override { return fChain ? fChain->GetTree()->GetEntry(entry, getall) : 0; } - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override { fInput = input; } - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(ProofStdVect,0); -}; - -#endif diff --git a/tutorials/legacy/proof/ProofTests.C b/tutorials/legacy/proof/ProofTests.C deleted file mode 100644 index 8b75d99f15dc2..0000000000000 --- a/tutorials/legacy/proof/ProofTests.C +++ /dev/null @@ -1,217 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofTests -/// -/// Auxilliary selector used to test PROOF functionality -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#define ProofTests_cxx - -#include "ProofTests.h" -#include -#include -#include -#include -#include -#include -#include - -//_____________________________________________________________________________ -ProofTests::ProofTests() -{ - // Constructor - - fTestType = 0; - fStat = nullptr; -} - -//_____________________________________________________________________________ -ProofTests::~ProofTests() -{ - // Destructor - -} - -//_____________________________________________________________________________ -void ProofTests::ParseInput() -{ - // This function sets some control member variables based on the input list - // content. Called by Begin and SlaveBegin. - - // Determine the test type - TNamed *ntype = dynamic_cast(fInput->FindObject("ProofTests_Type")); - if (ntype) { - if (!strcmp(ntype->GetTitle(), "InputData")) { - fTestType = 0; - } else if (!strcmp(ntype->GetTitle(), "PackTest1")) { - fTestType = 1; - } else if (!strcmp(ntype->GetTitle(), "PackTest2")) { - fTestType = 2; - } else { - Warning("ParseInput", "unknown type: '%s'", ntype->GetTitle()); - } - } - Info("ParseInput", "test type: %d (from '%s')", fTestType, ntype ? ntype->GetTitle() : "undef"); -} - -//_____________________________________________________________________________ -void ProofTests::Begin(TTree * /*tree*/) -{ - // The Begin() function is called at the start of the query. - // When running with PROOF Begin() is only called on the client. - // The tree argument is deprecated (on PROOF 0 is passed). -} - -//_____________________________________________________________________________ -void ProofTests::SlaveBegin(TTree * /*tree*/) -{ - // The SlaveBegin() function is called after the Begin() function. - // When running with PROOF SlaveBegin() is called on each slave server. - // The tree argument is deprecated (on PROOF 0 is passed). - - TString option = GetOption(); - - // Fill relevant members - ParseInput(); - - // Output histo - fStat = new TH1I("TestStat", "Test results", 20, .5, 20.5); - fOutput->Add(fStat); - - // We were started - fStat->Fill(1.); - - // Depends on the test - if (fTestType == 0) { - // Retrieve objects from the input list and copy them to the output - // H1 - TList *h1list = dynamic_cast(fInput->FindObject("h1list")); - if (h1list) { - // Retrieve objects from the input list and copy them to the output - TH1F *h1 = dynamic_cast(h1list->FindObject("h1data")); - if (h1) { - TParameter *h1avg = dynamic_cast*>(h1list->FindObject("h1avg")); - TParameter *h1rms = dynamic_cast*>(h1list->FindObject("h1rms")); - if (h1avg && h1rms) { - if (TMath::Abs(h1avg->GetVal() - h1->GetMean()) < 0.0001) { - if (TMath::Abs(h1rms->GetVal() - h1->GetRMS()) < 0.0001) { - fStat->Fill(2.); - } - } - } else { - Warning("SlaveBegin", "%d: info 'h1avg' or 'h1rms' not found!", fTestType); - } - } else { - Warning("SlaveBegin", "%d: input histo 'h1data' not found!", fTestType); - } - } else { - Warning("SlaveBegin", "%d: input list 'h1list' not found!", fTestType); - } - // H2 - TList *h2list = dynamic_cast(fInput->FindObject("h2list")); - if (h2list) { - // Retrieve objects from the input list and copy them to the output - TH1F *h2 = dynamic_cast(h2list->FindObject("h2data")); - if (h2) { - TParameter *h2avg = dynamic_cast*>(h2list->FindObject("h2avg")); - TParameter *h2rms = dynamic_cast*>(h2list->FindObject("h2rms")); - if (h2avg && h2rms) { - if (TMath::Abs(h2avg->GetVal() - h2->GetMean()) < 0.0001) { - if (TMath::Abs(h2rms->GetVal() - h2->GetRMS()) < 0.0001) { - fStat->Fill(3.); - } - } - } else { - Warning("SlaveBegin", "%d: info 'h2avg' or 'h2rms' not found!", fTestType); - } - } else { - Warning("SlaveBegin", "%d: input histo 'h2data' not found!", fTestType); - } - } else { - Warning("SlaveBegin", "%d: input list 'h2list' not found!", fTestType); - } - - TNamed *iob = dynamic_cast(fInput->FindObject("InputObject")); - if (iob) { - fStat->Fill(4.); - } else { - Warning("SlaveBegin", "%d: input histo 'InputObject' not found!", fTestType); - } - } else if (fTestType == 1) { - // We should find in the input list the name of a test variable which should exist - // at this point in the gEnv table - TNamed *nm = dynamic_cast(fInput->FindObject("testenv")); - if (nm) { - if (gEnv->Lookup(nm->GetTitle())) { - fStat->Fill(2.); - } else { - Warning("SlaveBegin", "%d: lookup for '%s' failed!", fTestType, nm->GetTitle()); - gEnv->Print(); - } - } else { - Warning("SlaveBegin", "%d: TNamed with the test env info not found!", fTestType); - } - } else if (fTestType == 2) { - // We should find in the input list the list of names of test variables which should exist - // at this point in the gEnv table - TNamed *nm = dynamic_cast(fInput->FindObject("testenv")); - if (nm) { - TString nms(nm->GetTitle()), nam; - Int_t from = 0; - while (nms.Tokenize(nam, from, ",")) { - if (gEnv->Lookup(nam)) { - Double_t xx = gEnv->GetValue(nam, -1.); - if (xx > 1.) fStat->Fill(xx); - } else { - Warning("SlaveBegin", "RC-env '%s' not found!", nam.Data()); - } - } - } else { - Warning("SlaveBegin", "%d: TNamed with the test env info not found!", fTestType); - } - } -} - -//_____________________________________________________________________________ -Bool_t ProofTests::Process(Long64_t) -{ - // The Process() function is called for each entry in the tree (or possibly - // keyed object in the case of PROOF) to be processed. The entry argument - // specifies which entry in the currently loaded tree is to be processed. - // It can be passed to either ProofTests::GetEntry() or TBranch::GetEntry() - // to read either all or the required parts of the data. When processing - // keyed objects with PROOF, the object is already loaded and is available - // via the fObject pointer. - // - // This function should contain the "body" of the analysis. It can contain - // simple or elaborate selection criteria, run algorithms on the data - // of the event and typically fill histograms. - // - // The processing can be stopped by calling Abort(). - // - // Use fStatus to set the return value of TTree::Process(). - // - // The return value is currently not used. - - return kTRUE; -} - -//_____________________________________________________________________________ -void ProofTests::SlaveTerminate() -{ - // The SlaveTerminate() function is called after all entries or objects - // have been processed. When running with PROOF SlaveTerminate() is called - // on each slave server. - -} - -//_____________________________________________________________________________ -void ProofTests::Terminate() -{ - // The Terminate() function is the last function to be called during - // a query. It always runs on the client, it can be used to present - // the results graphically or save the results to file. - -} diff --git a/tutorials/legacy/proof/ProofTests.h b/tutorials/legacy/proof/ProofTests.h deleted file mode 100644 index aaafd94fd9757..0000000000000 --- a/tutorials/legacy/proof/ProofTests.h +++ /dev/null @@ -1,42 +0,0 @@ -/// \file -/// \ingroup tutorial_ProofTests -/// -/// Auxilliary selector used to test PROOF functionality -/// -/// \macro_code -/// -/// \author Gerardo Ganis (gerardo.ganis@cern.ch) - -#ifndef ProofTests_h -#define ProofTests_h - -#include - -class TH1I; - -class ProofTests : public TSelector { -private: - void ParseInput(); -public : - - // Specific members - Int_t fTestType; - TH1I *fStat; - - ProofTests(); - ~ProofTests() override; - Int_t Version() const override { return 2; } - void Begin(TTree *tree) override; - void SlaveBegin(TTree *tree) override; - Bool_t Process(Long64_t entry) override; - void SetOption(const char *option) override { fOption = option; } - void SetObject(TObject *obj) override { fObject = obj; } - void SetInputList(TList *input) override { fInput = input; } - TList *GetOutputList() const override { return fOutput; } - void SlaveTerminate() override; - void Terminate() override; - - ClassDefOverride(ProofTests,0); -}; - -#endif diff --git a/tutorials/legacy/proof/event.md5sum b/tutorials/legacy/proof/event.md5sum deleted file mode 100644 index d59d27fbd3dab..0000000000000 --- a/tutorials/legacy/proof/event.md5sum +++ /dev/null @@ -1 +0,0 @@ -af81ba046682da100782fa9e7b4d4cec tutorials/proof/event.par diff --git a/tutorials/legacy/proof/event.par b/tutorials/legacy/proof/event.par deleted file mode 100644 index 27b6da8536ffa..0000000000000 Binary files a/tutorials/legacy/proof/event.par and /dev/null differ diff --git a/tutorials/legacy/proof/finalizeProof.C b/tutorials/legacy/proof/finalizeProof.C deleted file mode 100644 index e77d633cada44..0000000000000 --- a/tutorials/legacy/proof/finalizeProof.C +++ /dev/null @@ -1,199 +0,0 @@ -/// \file -/// \ingroup tutorial_proof -/// -/// Macro to finalize queries run with the macro tutorials/runProof . -/// This macro uses an existing PROOF session or starts one at the indicated URL. -/// In the case non existing PROOF session is found and no URL is given, the macro -/// tries to start a local PROOF session. -/// -/// To run the macro: -/// -/// root[] .L proof/finalizeProof.C+ -/// root[] finalizeProof("") -/// -/// See runProof.C for the analysis currently available. -/// -/// The macro looks for the last completed queries for the chosen analysis and -/// asks which one to finalize. If there is only available, it finalizes it -/// without asking. -/// All queries are considered for this, both those run synchronously and those -/// run asynchronously, e.g. runProof("h1(asyn)"). -/// -/// -/// \macro_code -/// -/// \author Gerardo Ganis - -#include "Getline.h" -#include "TChain.h" -#include "TEnv.h" -#include "TProof.h" -#include "TString.h" -#include "TDrawFeedback.h" -#include "TList.h" -#include "TQueryResult.h" -#include "TObjArray.h" - -#include "getProof.C" - -void finalizeProof(const char *what = "simple", - const char *url = "proof://localhost:11093", - Int_t nwrks = -1) -{ - - // Temp dir for PROOF tutorials - TString tutdir = Form("%s/.proof-tutorial", gSystem->TempDirectory()); - if (gSystem->AccessPathName(tutdir)) { - Printf("runProof: creating the temporary directory" - " for the tutorial (%s) ... ", tutdir.Data()); - if (gSystem->mkdir(tutdir, kTRUE) != 0) { - Printf("runProof: could not assert / create the temporary directory" - " for the tutorial (%s)", tutdir.Data()); - return; - } - } - - // Get / Attach-to the PROOF Session - TProof *proof = getProof(url, nwrks, tutdir.Data(), ""); - if (!proof) { - Printf("runProof: could not start/attach a PROOF session"); - return; - } - - // Get the last session run for the tutorial - TObjArray *qt = new TObjArray(); - TString lasttag; - TString proofsessions(Form("%s/sessions",tutdir.Data())); - // Save tag of the used session - FILE *fs = fopen(proofsessions.Data(), "r"); - if (!fs) { - Printf("runProof: could not create files for sessions tags"); - } else { - char line[1024]; - while (fgets(line, sizeof(line), fs)) { - int l = strlen(line); - if (l <= 0) continue; - if (strncmp(line,"session-",strlen("session-"))) continue; - if (line[l-1] == '\n') line[l-1] = 0; - lasttag = line; - qt->Add(new TObjString(lasttag.Data())); - } - fclose(fs); - } - - // Retrieve the list of available query results - TList *ql = proof->GetListOfQueries("A"); - if (!ql || ql->GetSize() <= 0) { - Printf("runProof: no queries to be finalized"); - return; - } - ql->Print(); - - // Where is the code to run - char *rootbin = gSystem->Which(gSystem->Getenv("PATH"), "root.exe", kExecutePermission); - if (!rootbin) { - Printf("runProof: root.exe not found: please check the environment!"); - return; - } - TString rootsys = gSystem->GetDirName(rootbin); - rootsys = gSystem->GetDirName(rootsys); - TString tutorials(Form("%s/tutorials", rootsys.Data())); - delete[] rootbin; - - // Create feedback displayer - TDrawFeedback fb(proof); - - // Parse 'what'; it is in the form 'analysis(arg1,arg2,...)' - TString args(what); - args.ReplaceAll("("," "); - args.ReplaceAll(")"," "); - args.ReplaceAll(","," "); - Ssiz_t from = 0; - TString act, tok; - if (!args.Tokenize(act, from, " ")) { - // Cannot continue - Printf("runProof: action not found: check your arguments (%s)", what); - return; - } - - TObjArray *qa = new TObjArray(); - TString sel; - // Action - if (act == "simple") { - sel = "ProofSimple"; - } else if (act == "h1") { - sel = "h1analysis"; - } else if (act == "pythia8") { - sel = "ProofPythia"; - } else { - // Do not know what to run - Printf("runProof: unknown tutorial: %s", what); - } - - // Get last completed queries for the chosen analysis - TString ref; - Int_t nt = qt->GetEntriesFast(); - while (ref.IsNull() && nt--) { - lasttag = ((TObjString *)(qt->At(nt)))->GetName(); - if (!lasttag.IsNull()) - Printf("runProof: checking session: %s", lasttag.Data()); - TIter nxq(ql); - TQueryResult *qr = 0; - while ((qr = (TQueryResult *)nxq())) { - if (qr->IsDone() && !lasttag.CompareTo(qr->GetTitle()) && - !sel.CompareTo(qr->GetSelecImp()->GetTitle())) { - TString r = Form("%s:%s",qr->GetTitle(),qr->GetName()); - qa->Add(new TObjString(r.Data())); - } - } - if (qa->GetEntriesFast() > 0) { - Int_t qn = 0; - if (qa->GetEntriesFast() > 1) { - // Query the client which query to finalize - Printf("finalizeProof: queries completed for analysis '%s'", act.Data()); - for (Int_t k = 0; k < qa->GetEntriesFast(); k++) { - Printf(" [%d] %s", k, ((TObjString *)(qa->At(k)))->GetName()); - } - Bool_t ask = kTRUE; - while (ask) { - char *answer = Getline("finalizeProof: enter the one you would like to finalize? [0] "); - if (answer) { - if (answer[0] == 'Q' || answer[0] == 'q') { - ask = kFALSE; - return; - } - TString sn(answer); - sn.Remove(sn.Length()-1); - if (sn.IsDigit()) { - qn = sn.Atoi(); - if (qn >= 0 && qn < qa->GetEntriesFast()) { - break; - } else { - Printf("finalizeProof: choice must be in [0,%d] ('Q' to quit)", - qa->GetEntriesFast()-1); - } - } else { - if (sn.IsNull()) { - qn = 0; - break; - } else { - Printf("finalizeProof: choice must be a number in [0,%d] ('Q' to quit) (%s)", - qa->GetEntriesFast()-1, sn.Data()); - } - } - } - } - } - ref = ((TObjString *)(qa->At(qn)))->GetName(); - } - } - if (!ref.IsNull()) { - // Retrieve - proof->Retrieve(ref); - // Finalize - proof->Finalize(ref); - } else { - Printf("runProof: no queries to be finalized for analysis '%s'", act.Data()); - return; - } -} diff --git a/tutorials/legacy/proof/getProof.C b/tutorials/legacy/proof/getProof.C deleted file mode 100644 index 97a5fa657b1ff..0000000000000 --- a/tutorials/legacy/proof/getProof.C +++ /dev/null @@ -1,717 +0,0 @@ -/// \file -/// \ingroup tutorial_proof -/// -/// Attaches to a PROOF session, possibly at the indicated URL. -/// If no existing PROOF session is found and no URL is given, -/// try to start a local PROOF session. -/// -/// Arguments: -/// 'url' URL of the master where to start/attach the PROOF session; -/// this is also the place where to force creation of a new session, -/// if needed (use option 'N', e.g. "proof://mymaster:myport/?N") -/// -/// The following arguments apply to xrootd responding at 'refloc' only: -/// 'nwrks' Number of workers to be started. [] -/// 'dir' Directory to be used for the files and working areas []. When starting a new -/// instance of the daemon this directory is cleaned with 'rm -fr'. If 'dir' -/// is null, the default is used: `/tmp//.getproof` -/// 'opt' Defines what to do if an existing xrootd uses the same ports; possible -/// options are: "ask", ask the user; "force", kill the xrootd and start -/// a new one; if any other string is specified the existing xrootd will be -/// used ["ask"]. -/// NB: for a change in 'nwrks' to be effective you need to specify opt = "force" -/// 'dyn' This flag can be used to switch on dynamic, per-job worker setup scheduling -/// [kFALSE]. -/// 'tutords' This flag can be used to force a dataset dir under the tutorial dir [kFALSE] -/// -/// It is possible to trigger the automatic valgrind setup by defining the env GETPROOF_VALGRIND. -/// E.g. to run the master in valgrind do -/// -/// $ export GETPROOF_VALGRIND="valgrind=master" -/// -/// (or -/// $ export GETPROOF_VALGRIND="valgrind=master valgrind_opts:--leak-check=full" -/// -/// to set some options) before running getProof. Note that 'getProof' is also called by 'stressProof', -/// so this holds for 'stressProof' runs too. -/// -/// -/// \macro_code -/// -/// \author Gerardo Ganis - -#include "Bytes.h" -#include "Getline.h" -#include "TEnv.h" -#include "TProof.h" -#include "TSocket.h" -#include "TString.h" -#include "TSystem.h" - -// Auxilliary functions -int getDebugEnum(const char *what); -Int_t getXrootdPid(Int_t port, const char *subdir = "xpdtut"); -Int_t checkXrootdAt(Int_t port, const char *host = "localhost"); -Int_t checkXproofdAt(Int_t port, const char *host = "localhost"); -Int_t startXrootdAt(Int_t port, const char *exportdirs = nullptr, Bool_t force = kFALSE); -Int_t killXrootdAt(Int_t port, const char *id = nullptr); - -// Auxilliary structures for Xrootd/Xproofd pinging ... -// The client request -typedef struct { - int first; - int second; - int third; - int fourth; - int fifth; -} clnt_HS_t; -// The body received after the first handshake's header -typedef struct { - int msglen; - int protover; - int msgval; -} srv_HS_t; - -// By default we start a cluster on the local machine -const char *refloc = "proof://localhost:40000"; - -TProof *getProof(const char *url = "proof://localhost:40000", Int_t nwrks = -1, const char *dir = nullptr, - const char *opt = "ask", Bool_t dyn = kFALSE, Bool_t tutords = kFALSE) -{ - - - TProof *p = nullptr; - - // Valgrind options, if any - TString vopt, vopts; -#ifndef WIN32 - if (gSystem->Getenv("GETPROOF_VALGRIND")) { - TString s(gSystem->Getenv("GETPROOF_VALGRIND")), t; - Int_t from = 0; - while (s.Tokenize(t, from , " ")) { - if (t.BeginsWith("valgrind_opts:")) - vopts = t; - else - vopt = t; - } - if (vopts.IsNull()) vopts = "valgrind_opts:--leak-check=full --track-origins=yes"; - TProof::AddEnvVar("PROOF_WRAPPERCMD", vopts.Data()); - Printf("getProof: valgrind run: '%s' (opts: '%s')", vopt.Data(), vopts.Data()); - } -#endif - - // If an URL has specified get a session there - TUrl uu(url), uref(refloc); - Bool_t ext = (strcmp(uu.GetHost(), uref.GetHost()) || - (uu.GetPort() != uref.GetPort())) ? kTRUE : kFALSE; - Bool_t lite = kFALSE; - if (ext && url) { - if (!strcmp(url, "lite://") || !url[0]) { - if (!url[0]) uu.SetUrl("lite://"); - if (dir && strlen(dir) > 0) gEnv->SetValue("Proof.Sandbox", dir); - TString swrk(" workers"); - if (nwrks > 0) { - uu.SetOptions(Form("workers=%d", nwrks)); - swrk.Form("%d workers", nwrks); - } - lite = kTRUE; - gEnv->SetValue("Proof.MaxOldSessions", 1); - Printf("getProof: trying to open a PROOF-Lite session with %s", swrk.Data()); - } else { - Printf("getProof: trying to open a session on the external cluster at '%s'", url); - } - p = TProof::Open(uu.GetUrl(), vopt); - if (p && p->IsValid()) { - // Check consistency - if (ext && !lite && nwrks > 0) { - Printf("getProof: WARNING: started/attached a session on external cluster (%s):" - " 'nwrks=%d' ignored", url, nwrks); - } - if (ext && !lite && dir && strlen(dir) > 0) { - Printf("getProof: WARNING: started/attached a session on external cluster (%s):" - " 'dir=\"%s\"' ignored", url, dir); - } - if (ext && !strcmp(opt,"force")) { - Printf("getProof: WARNING: started/attached a session on external cluster (%s):" - " 'opt=\"force\"' ignored", url); - } - if (ext && dyn) { - Printf("getProof: WARNING: started/attached a session on external cluster (%s):" - " 'dyn=kTRUE' ignored", url); - } - } else { - Printf("getProof: could not get/start a valid session at %s", url); - if (p) delete p; - p = nullptr; - } - // Done - return p; - } - -#ifdef WIN32 - // No support for local PROOF on Win32 (yet; the optimized local Proof will work there too) - Printf("getProof: local PROOF not yet supported on Windows, sorry!"); - return p; -#else - - // Temp dir for tutorial daemons - TString tutdir = dir; - if (!tutdir.IsNull()) { - if (gSystem->AccessPathName(tutdir)) { - // Directory does not exist: try to make it - gSystem->mkdir(tutdir.Data(), kTRUE); - if (gSystem->AccessPathName(tutdir, kWritePermission)) { - if (gSystem->AccessPathName(tutdir)) { - Printf("getProof: unable to create the working area at the requested path: '%s'" - " - cannot continue", tutdir.Data()); - } else { - Printf("getProof: working area at the requested path '%s'" - " created but it is not writable - cannot continue", tutdir.Data()); - } - return p; - } - } else { - // Check if it is writable ... - if (gSystem->AccessPathName(dir, kWritePermission)) { - // ... fail if not - Printf("getProof: working area at the requested path '%s'" - " exists but is not writable - cannot continue", tutdir.Data()); - return p; - } - } - } else { - // Notify - Printf("getProof: working area not specified temp "); - // Force "/tmp/" whenever possible to avoid length problems on MacOsX - tutdir="/tmp"; - if (gSystem->AccessPathName(tutdir, kWritePermission)) tutdir = gSystem->TempDirectory(); - TString us; - UserGroup_t *ug = gSystem->GetUserInfo(gSystem->GetUid()); - if (!ug) { - Printf("getProof: could not get user info"); - return p; - } - us.Form("/%s", ug->fUser.Data()); - if (!tutdir.EndsWith(us.Data())) tutdir += us; - // Add our own subdir - tutdir += "/.getproof"; - if (gSystem->AccessPathName(tutdir)) { - gSystem->mkdir(tutdir.Data(), kTRUE); - if (gSystem->AccessPathName(tutdir, kWritePermission)) { - Printf("getProof: unable to get a writable working area (tried: %s)" - " - cannot continue", tutdir.Data()); - return p; - } - } - } - Printf("getProof: working area (tutorial dir): %s", tutdir.Data()); - - // Dataset dir - TString datasetdir; - if (tutords) { - datasetdir = Form("%s/dataset", tutdir.Data()); - if (gSystem->AccessPathName(datasetdir, kWritePermission)) { - gSystem->mkdir(datasetdir, kTRUE); - if (gSystem->AccessPathName(datasetdir, kWritePermission)) { - Printf("getProof: unable to get a writable dataset directory (tried: %s)" - " - cannot continue", datasetdir.Data()); - return p; - } - Printf("getProof: dataset dir: %s", datasetdir.Data()); - } - } - - // Local url (use a special port to try to not disturb running daemons) - TUrl u(refloc); - u.SetProtocol("proof"); - if (!strcmp(uu.GetHost(), uref.GetHost()) && (uu.GetPort() != uref.GetPort())) - u.SetPort(uu.GetPort()); - Int_t lportp = u.GetPort(); - Int_t lportx = lportp + 1; - TString lurl = u.GetUrl(); - - // Prepare to start the daemon - TString workarea = Form("%s/proof", tutdir.Data()); - TString xpdcf(Form("%s/xpd.cf",tutdir.Data())); - TString xpdlog(Form("%s/xpd.log",tutdir.Data())); - TString xpdlogprt(Form("%s/xpdtut/xpd.log",tutdir.Data())); - TString xpdpid(Form("%s/xpd.pid",tutdir.Data())); - TString proofsessions(Form("%s/sessions",tutdir.Data())); - TString cmd; - Int_t rc = 0; - - // Is there something listening already ? - Int_t pid = -1; - Bool_t restart = kTRUE; - if ((rc = checkXproofdAt(lportp)) == 1) { - Printf("getProof: something else the a XProofd service is running on" - " port %d - cannot continue", lportp); - return p; - - } else if (rc == 0) { - - restart = kFALSE; - - pid = getXrootdPid(lportx); - Printf("getProof: daemon found listening on dedicated ports {%d,%d} (pid: %d)", - lportx, lportp, pid); - if (isatty(0) == 0 || isatty(1) == 0) { - // Cannot ask: always restart - restart = kTRUE; - } else { - if (!strcmp(opt,"ask")) { - char *answer = (char *) Getline("getProof: would you like to restart it (N,Y)? [N] "); - if (answer && (answer[0] == 'Y' || answer[0] == 'y')) - restart = kTRUE; - } - } - if (!strcmp(opt,"force")) - // Always restart - restart = kTRUE; - - // Cleanup, if required - if (restart) { - Printf("getProof: cleaning existing instance ..."); - // Cleaning up existing daemon - cmd = Form("kill -9 %d", pid); - if ((rc = gSystem->Exec(cmd)) != 0) - Printf("getProof: problems stopping xrootd process %d (%d)", pid, rc); - // Wait for all previous connections being cleaned - Printf("getProof: wait 5 secs so that previous connections are cleaned ..."); - gSystem->Sleep(5000); - } - } - - if (restart) { - - // Try to start something locally; make sure that everything is there - char *xpd = gSystem->Which(gSystem->Getenv("PATH"), "xproofd", kExecutePermission); - if (!xpd) { - Printf("getProof: xproofd not found: please check the environment!"); - return p; - } - - // Cleanup the working area - cmd = Form("rm -fr %s/xpdtut %s %s %s %s", tutdir.Data(), workarea.Data(), - xpdcf.Data(), xpdpid.Data(), proofsessions.Data()); - gSystem->Exec(cmd); - - // Try to start something locally; create the xproofd config file - FILE *fcf = fopen(xpdcf.Data(), "w"); - if (!fcf) { - Printf("getProof: could not create config file for XPD (%s)", xpdcf.Data()); - return p; - } - fprintf(fcf,"### Use admin path at %s/admin to avoid interferences with other users\n", tutdir.Data()); - fprintf(fcf,"xrd.adminpath %s/admin\n", tutdir.Data()); -#if defined(R__MACOSX) - fprintf(fcf,"### Use dedicated socket path under /tmp to avoid length problems\n"); - fprintf(fcf,"xpd.sockpathdir /tmp/xpd-sock\n"); -#endif - fprintf(fcf,"### Load the XrdProofd protocol on port %d\n", lportp); - fprintf(fcf,"xrd.protocol xproofd libXrdProofd.so\n"); - fprintf(fcf,"xpd.port %d\n", lportp); - if (nwrks > 0) { - fprintf(fcf,"### Force number of local workers\n"); - fprintf(fcf,"xpd.localwrks %d\n", nwrks); - } - fprintf(fcf,"### Root path for working dir\n"); - fprintf(fcf,"xpd.workdir %s\n", workarea.Data()); - fprintf(fcf,"### Allow different users to connect\n"); - fprintf(fcf,"xpd.multiuser 1\n"); - fprintf(fcf,"### Limit the number of query results kept in the master sandbox\n"); - fprintf(fcf,"xpd.putrc ProofServ.UserQuotas: maxquerykept=2\n"); - fprintf(fcf,"### Limit the number of sessions kept in the sandbox\n"); - fprintf(fcf,"xpd.putrc Proof.MaxOldSessions: 1\n"); - if (tutords) { - fprintf(fcf,"### Use dataset directory under the tutorial dir\n"); - fprintf(fcf,"xpd.datasetsrc file url:%s opt:-Cq:Av:As:\n", datasetdir.Data()); - } - if (dyn) { - fprintf(fcf,"### Use dynamic, per-job scheduling\n"); - fprintf(fcf,"xpd.putrc Proof.DynamicStartup 1\n"); - } - fprintf(fcf,"### For internal file serving use the xrootd protocol on the same port\n"); - fprintf(fcf,"xpd.xrootd libXrdXrootd-4.so\n"); - fprintf(fcf,"### Set the local data server for the temporary output files accordingly\n"); - fprintf(fcf,"xpd.putenv LOCALDATASERVER=root://%s:%d\n", gSystem->HostName(), lportp); - fclose(fcf); - Printf("getProof: xproofd config file at %s", xpdcf.Data()); - - // Start xrootd in the background - Printf("getProof: xproofd log file at %s", xpdlogprt.Data()); - cmd = Form("%s -c %s -b -l %s -n xpdtut -p %d", - xpd, xpdcf.Data(), xpdlog.Data(), lportp); - Printf("(NB: any error line from XrdClientSock::RecvRaw and XrdClientMessage::ReadRaw should be ignored)"); - if ((rc = gSystem->Exec(cmd)) != 0) { - Printf("getProof: problems starting xproofd (%d)", rc); - return p; - } - delete[] xpd; - - // Wait a bit - Printf("getProof: waiting for xproofd to start ..."); - gSystem->Sleep(2000); - - pid = getXrootdPid(lportp); - Printf("getProof: xproofd pid: %d", pid); - - // Save it in the PID file - FILE *fpid = fopen(xpdpid.Data(), "w"); - if (!fpid) { - Printf("getProof: could not create pid file for XPD"); - } else { - fprintf(fpid,"%d\n", pid); - fclose(fpid); - } - } - Printf("getProof: start / attach the PROOF session ..."); - - // Start / attach the session now - p = TProof::Open(lurl, vopt.Data()); - if (!p || !(p->IsValid())) { - Printf("getProof: starting local session failed"); - if (p) delete p; - p = nullptr; - return p; - } - - // Return the session - return p; -#endif -} - -Int_t getXrootdPid(Int_t port, const char *subdir) -{ -#ifdef WIN32 - // No support for Xrootd/Proof on Win32 (yet; the optimized local Proof will work there too) - Printf("getXrootdPid: Xrootd/Proof not supported on Windows, sorry!"); - return -1; -#else - // Get the pid of the started xrootd process - Int_t pid = -1; -#if defined(__sun) - const char *com = "-eo pid,comm"; -#elif defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__APPLE__) - const char *com = "ax -w -w"; -#else - const char *com = "-w -w -eo pid,command"; -#endif - TString cmd; - if (subdir && strlen(subdir) > 0) { - cmd.Form("ps %s | grep xrootd | grep \"\\-p %d\" | grep %s", com, port, subdir); - } else { - cmd.Form("ps %s | grep xrootd | grep \"\\-p %d\"", com, port); - } - FILE *fp = gSystem->OpenPipe(cmd.Data(), "r"); - if (fp) { - char line[2048], rest[2048]; - while (fgets(line, sizeof(line), fp)) { - sscanf(line,"%d %s", &pid, rest); - break; - } - gSystem->ClosePipe(fp); - } - // Done - return pid; -#endif -} - -Int_t checkXrootdAt(Int_t port, const char *host) -{ - // Check if a XrdXrootd service is running on 'port' at 'host' - // Return - // 0 if OK - // -1 if nothing is listening on the port (connection cannot be open) - // 1 if something is listening but not XROOTD - - // Open the connection - TSocket s(host, port); - if (!(s.IsValid())) { - if (gDebug > 0) - Printf("checkXrootdAt: could not open connection to %s:%d", host, port); - return -1; - } - // Send the first bytes - clnt_HS_t initHS; - memset(&initHS, 0, sizeof(initHS)); - initHS.fourth = host2net((int)4); - initHS.fifth = host2net((int)2012); - int len = sizeof(initHS); - s.SendRaw(&initHS, len); - // Read first server response - int type; - len = sizeof(type); - int readCount = s.RecvRaw(&type, len); // 4(2+2) bytes - if (readCount != len) { - if (gDebug > 0) - Printf("checkXrootdAt: 1st: wrong number of bytes read: %d (expected: %d)", - readCount, len); - return 1; - } - // to host byte order - type = net2host(type); - // Check if the server is the eXtended proofd - if (type == 0) { - srv_HS_t xbody; - len = sizeof(xbody); - readCount = s.RecvRaw(&xbody, len); // 12(4+4+4) bytes - if (readCount != len) { - if (gDebug > 0) - Printf("checkXrootdAt: 2nd: wrong number of bytes read: %d (expected: %d)", - readCount, len); - return 1; - } - - } else if (type == 8) { - // Standard proofd - if (gDebug > 0) - Printf("checkXrootdAt: server is ROOTD"); - return 1; - } else { - // We don't know the server type - if (gDebug > 0) - Printf("checkXrootdAt: unknown server type: %d", type); - return 1; - } - // Done - return 0; -} - -Int_t checkXproofdAt(Int_t port, const char *host) -{ - // Check if a XrdProofd service is running on 'port' at 'host' - // Return - // 0 if OK - // -1 if nothing is listening on the port (connection cannot be open) - // 1 if something is listening but not XPROOFD - - // Open the connection - TSocket s(host, port); - if (!(s.IsValid())) { - if (gDebug > 0) - Printf("checkXproofdAt: could not open connection to %s:%d", host, port); - return -1; - } - // Send the first bytes - clnt_HS_t initHS; - memset(&initHS, 0, sizeof(initHS)); - initHS.third = (int)host2net((int)1); - int len = sizeof(initHS); - s.SendRaw(&initHS, len); - // These 8 bytes are need by 'proofd' and discarded by XPD - int dum[2]; - dum[0] = (int)host2net((int)4); - dum[1] = (int)host2net((int)2012); - s.SendRaw(&dum[0], sizeof(dum)); - // Read first server response - int type; - len = sizeof(type); - int readCount = s.RecvRaw(&type, len); // 4(2+2) bytes - if (readCount != len) { - if (gDebug > 0) - Printf("checkXproofdAt: 1st: wrong number of bytes read: %d (expected: %d)", - readCount, len); - return 1; - } - // to host byte order - type = net2host(type); - // Check if the server is the eXtended proofd - if (type == 0) { - srv_HS_t xbody; - len = sizeof(xbody); - readCount = s.RecvRaw(&xbody, len); // 12(4+4+4) bytes - if (readCount != len) { - if (gDebug > 0) - Printf("checkXproofdAt: 2nd: wrong number of bytes read: %d (expected: %d)", - readCount, len); - return 1; - } - xbody.protover = net2host(xbody.protover); - xbody.msgval = net2host(xbody.msglen); - xbody.msglen = net2host(xbody.msgval); - - } else if (type == 8) { - // Standard proofd - if (gDebug > 0) - Printf("checkXproofdAt: server is PROOFD"); - return 1; - } else { - // We don't know the server type - if (gDebug > 0) - Printf("checkXproofdAt: unknown server type: %d", type); - return 1; - } - // Done - return 0; -} - -Int_t startXrootdAt(Int_t port, const char *exportdirs, Bool_t force) -{ - // Start a basic xrootd service on 'port' exporting the dirs in 'exportdirs' - // (blank separated list) - -#ifdef WIN32 - // No support for Xrootd on Win32 (yet; the optimized local Proof will work there too) - Printf("startXrootdAt: Xrootd not supported on Windows, sorry!"); - return -1; -#else - Bool_t restart = kTRUE; - - // Already there? - Int_t rc = 0; - if ((rc = checkXrootdAt(port)) == 1) { - - Printf("startXrootdAt: some other service running on port %d - cannot proceed ", port); - return -1; - - } else if (rc == 0) { - - restart = kFALSE; - - if (force) { - // Always restart - restart = kTRUE; - } else { - Printf("startXrootdAt: xrootd service already available on port %d: ", port); - char *answer = (char *) Getline("startXrootdAt: would you like to restart it (N,Y)? [N] "); - if (answer && (answer[0] == 'Y' || answer[0] == 'y')) { - restart = kTRUE; - } - } - - // Cleanup, if required - if (restart) { - Printf("startXrootdAt: cleaning existing instance ..."); - - // Get the Pid - Int_t pid = getXrootdPid(port, "xrd-basic"); - - // Cleanimg up existing daemon - TString cmd = Form("kill -9 %d", pid); - if ((rc = gSystem->Exec(cmd)) != 0) - Printf("startXrootdAt: problems stopping xrootd process %d (%d)", pid, rc); - } - } - - if (restart) { - if (gSystem->AccessPathName("/tmp/xrd-basic")) { - gSystem->mkdir("/tmp/xrd-basic"); - if (gSystem->AccessPathName("/tmp/xrd-basic")) { - Printf("startXrootdAt: could not assert dir for log file"); - return -1; - } - } - TString cmd; - cmd.Form("xrootd -d -p %d -b -l /tmp/xrd-basic/xrootd.log", port); - if (exportdirs && strlen(exportdirs) > 0) { - TString dirs(exportdirs), d; - Int_t from = 0; - while (dirs.Tokenize(d, from, " ")) { - if (!d.IsNull()) { - cmd += " "; - cmd += d; - } - } - } - Printf("cmd: %s", cmd.Data()); - if ((rc = gSystem->Exec(cmd)) != 0) { - Printf("startXrootdAt: problems executing starting command (%d)", rc); - return -1; - } - // Wait a bit - Printf("startXrootdAt: waiting for xrootd to start ..."); - gSystem->Sleep(2000); - // Check the result - if ((rc = checkXrootdAt(port)) != 0) { - Printf("startXrootdAt: xrootd service not available at %d (rc = %d) - startup failed", - port, rc); - return -1; - } - Printf("startXrootdAt: basic xrootd started!"); - } - - // Done - return 0; -#endif -} - -Int_t killXrootdAt(Int_t port, const char *id) -{ - // Kill running xrootd service on 'port' - -#ifdef WIN32 - // No support for Xrootd on Win32 (yet; the optimized local Proof will work there too) - Printf("killXrootdAt: Xrootd not supported on Windows, sorry!"); - return -1; -#else - - Int_t pid = -1, rc= 0; - if ((pid = getXrootdPid(port, id)) > 0) { - - // Cleanimg up existing daemon - TString cmd = Form("kill -9 %d", pid); - if ((rc = gSystem->Exec(cmd)) != 0) - Printf("killXrootdAt: problems stopping xrootd process %d (%d)", pid, rc); - } - - // Done - return rc; -#endif -} - -int getDebugEnum(const char *what) -{ - // Check if 'what' matches one of the TProofDebug enum and return the corresponding - // integer. Relies on a perfect synchronization with the content of TProofDebug.h . - - TString sws(what), sw; - int rcmask = 0; - int from = 0; - while (sws.Tokenize(sw, from , "|")) { - if (sw.BeginsWith("k")) sw.Remove(0,1); - - if (sw == "None") { - rcmask |= TProofDebug::kNone; - } else if (sw == "Packetizer") { - rcmask |= TProofDebug::kPacketizer; - } else if (sw == "Loop") { - rcmask |= TProofDebug::kLoop; - } else if (sw == "Selector") { - rcmask |= TProofDebug::kSelector; - } else if (sw == "Output") { - rcmask |= TProofDebug::kOutput; - } else if (sw == "Input") { - rcmask |= TProofDebug::kInput; - } else if (sw == "Global") { - rcmask |= TProofDebug::kGlobal; - } else if (sw == "Package") { - rcmask |= TProofDebug::kPackage; - } else if (sw == "Feedback") { - rcmask |= TProofDebug::kFeedback; - } else if (sw == "Condor") { - rcmask |= TProofDebug::kCondor; - } else if (sw == "Draw") { - rcmask |= TProofDebug::kDraw; - } else if (sw == "Asyn") { - rcmask |= TProofDebug::kAsyn; - } else if (sw == "Cache") { - rcmask |= TProofDebug::kCache; - } else if (sw == "Collect") { - rcmask |= TProofDebug::kCollect; - } else if (sw == "Dataset") { - rcmask |= TProofDebug::kDataset; - } else if (sw == "Submerger") { - rcmask |= TProofDebug::kSubmerger; - } else if (sw == "Monitoring") { - rcmask |= TProofDebug::kMonitoring; - } else if (sw == "All") { - rcmask |= TProofDebug::kAll; - } else if (!sw.IsNull()) { - Printf("WARNING: requested debug enum name '%s' does not exist: assuming 'All'", sw.Data()); - rcmask |= TProofDebug::kAll; - } - } - // Done - return rcmask; -} diff --git a/tutorials/legacy/proof/index.md b/tutorials/legacy/proof/index.md deleted file mode 100644 index 739d81d36f1fe..0000000000000 --- a/tutorials/legacy/proof/index.md +++ /dev/null @@ -1,90 +0,0 @@ -\defgroup tutorial_proof Proof tutorials -\ingroup tutorial_legacy -\brief These examples aim to illustrate the usage of PROOF - -\defgroup tutorial_ProcFileElements ProcFileElements -\ingroup tutorial_proof -\brief Class to hold information about the processed elements of a file - -Used for testing. - -\defgroup tutorial_ProofAux ProofAux -\ingroup tutorial_proof -\brief Selector used for auxiliary actions in the PROOF tutorials - -\defgroup tutorial_ProofEvent ProofEvent -\ingroup tutorial_proof -\brief Selector for generic processing with Event - -Example of TSelector implementation to do generic -processing with the test 'Event' structure. -See tutorials/proof/runProof.C, option "event", for an -example of how to run this selector. - -\defgroup tutorial_ProofEventProc ProofEventProc -\ingroup tutorial_proof -\brief Selector to process trees containing Event structures - -Example of TSelector implementation to process trees -containing 'Event' structures, e.g. the files under -http://root.cern/files/data . -See tutorials/proof/runProof.C, option "eventproc", for -an example of how to run this selector. - -\defgroup tutorial_ProofFriends ProofFriends -\ingroup tutorial_proof -\brief Selector to process tree friends - -Example of TSelector implementation to process tree friends in PROOF. -See tutorials/proof/runProof.C, option "eventproc", for -an example of how to run this selector. - -\defgroup tutorial_ProofNtuple ProofNtuple -\ingroup tutorial_proof -\brief Selector to fill a simple ntuple - -Example of TSelector implementation to do generic processing -(filling a simple ntuple, in this case). -See tutorials/proof/runProof.C, option "ntuple", for an -example of how to run this selector. - -\defgroup tutorial_ProofPythia ProofPythia -\ingroup tutorial_proof -\brief Selector to generate Monte Carlo events with Pythia8 - -Example of TSelector implementation to do a Monte Carlo -generation using Pythia8. -See tutorials/proof/runProof.C, option "pythia8", for an -example of how to run this selector. - -\defgroup tutorial_ProofSimple ProofSimple -\ingroup tutorial_proof -\brief Selector to fill a set of histograms - -Example of TSelector implementation to do generic processing (filling a -set of histograms in this case). -See tutorials/proof/runProof.C, option "simple", for an -example of how to run this selector. - -\defgroup tutorial_ProofSimpleFile ProofSimpleFile -\ingroup tutorial_proof -\brief Selector to fill a set of histograms and merging via file - -Example of TSelector implementation to do generic processing -(filling a set of histograms in this case) and merging via -a file, with part of the objects saved in a sub-directory. -See tutorials/proof/runProof.C, option "simplefile", for an -example of how to run this selector. - -\defgroup tutorial_ProofStdVec ProofStdVec -\ingroup tutorial_proof -\brief Selector for generic processing with stdlib collections - -Example of TSelector implementation to do generic -processing with stdlib collections. -See tutorials/proof/runProof.C, option "stdlib", for an -example of how to run this selector. - -\defgroup tutorial_ProofTests ProofTests -\ingroup tutorial_proof -\brief Auxilliary selector used to test PROOF functionality diff --git a/tutorials/legacy/proof/ntprndm.root b/tutorials/legacy/proof/ntprndm.root deleted file mode 100644 index 0500546a2dc15..0000000000000 Binary files a/tutorials/legacy/proof/ntprndm.root and /dev/null differ diff --git a/tutorials/legacy/proof/packtest1.md5sum b/tutorials/legacy/proof/packtest1.md5sum deleted file mode 100644 index 09ed80e50b737..0000000000000 --- a/tutorials/legacy/proof/packtest1.md5sum +++ /dev/null @@ -1 +0,0 @@ -6b1e5774530cd370a11095f0e852b652 tutorials/proof/packtest1.par diff --git a/tutorials/legacy/proof/packtest1.par b/tutorials/legacy/proof/packtest1.par deleted file mode 100644 index 8a40cce764263..0000000000000 Binary files a/tutorials/legacy/proof/packtest1.par and /dev/null differ diff --git a/tutorials/legacy/proof/packtest2.md5sum b/tutorials/legacy/proof/packtest2.md5sum deleted file mode 100644 index 74119918f9668..0000000000000 --- a/tutorials/legacy/proof/packtest2.md5sum +++ /dev/null @@ -1 +0,0 @@ -9f3cf6122638c67e538792b401d6ce2c tutorials/proof/packtest2.par diff --git a/tutorials/legacy/proof/packtest2.par b/tutorials/legacy/proof/packtest2.par deleted file mode 100644 index 75eaac7db1f05..0000000000000 Binary files a/tutorials/legacy/proof/packtest2.par and /dev/null differ diff --git a/tutorials/legacy/proof/packtest3.md5sum b/tutorials/legacy/proof/packtest3.md5sum deleted file mode 100644 index 1a20c2e86f37b..0000000000000 --- a/tutorials/legacy/proof/packtest3.md5sum +++ /dev/null @@ -1 +0,0 @@ -648a6fc24a01e1837a95193b4c6eeb27 tutorials/proof/packtest3.par diff --git a/tutorials/legacy/proof/packtest3.par b/tutorials/legacy/proof/packtest3.par deleted file mode 100644 index ce9d421b45843..0000000000000 Binary files a/tutorials/legacy/proof/packtest3.par and /dev/null differ diff --git a/tutorials/legacy/proof/pythia8.par b/tutorials/legacy/proof/pythia8.par deleted file mode 100644 index d7b3266d1bf39..0000000000000 Binary files a/tutorials/legacy/proof/pythia8.par and /dev/null differ diff --git a/tutorials/legacy/proof/runProof.C b/tutorials/legacy/proof/runProof.C deleted file mode 100644 index 1f640cdec06df..0000000000000 --- a/tutorials/legacy/proof/runProof.C +++ /dev/null @@ -1,1450 +0,0 @@ -/// \file -/// \ingroup proof -/// -/// Macro to run examples of analysis on PROOF, corresponding to the TSelector -/// implementations found under `/tutorials/proof`. -/// This macro uses an existing PROOF session or starts one at the indicated URL. -/// In the case non existing PROOF session is found and no URL is given, the -/// macro tries to start a local PROOF session. -/// -/// To run the macro: -/// -/// root[] .L proof/runProof.C+ -/// root[] runProof("") -/// -/// Currently available analysis are (to see how all this really works check -/// the scope for the specified option inside the macro): -/// -/// 1. "simple" -/// -/// Selector: ProofSimple.h.C -/// -/// root[] runProof("simple") -/// -/// This will create a local PROOF session and run an analysis filling -/// 100 histos with 100000 gaussian random numbers, and displaying them -/// in a canvas with 100 pads (10x10). -/// The number of histograms can be passed as argument 'nhist' to 'simple', -/// e.g. to fill 16 histos with 1000000 entries use -/// -/// root[] runProof("simple(nevt=1000000,nhist=16)") -/// -/// The argument nhist3 controls the creation of 3d histos to simulate -/// merging load. By default, no 3D hitogram is created. -/// -/// 2. "h1" -/// -/// Selector: tutorials/io/tree/h1analysis.h.C -/// -/// root[] runProof("h1") -/// -/// This runs the 'famous' H1 analysis from $ROOTSYS/tree/h1analysis.C.h. -/// By default the data are read from the HTTP server at root.cern, -/// the data source can be changed via the argument 'h1src', e.g. -/// -/// root[] runProof("h1,h1src=/data/h1") -/// -/// (the directory specified must contain the 4 H1 files). -/// -/// The 'h1' example is also used to show how to use entry-lists in PROOF. -/// To fill the list for the events used for the final plots add the option -/// 'fillList': -/// -/// root[] runProof("h1,fillList") -/// -/// To use the list previously created for the events used for the -/// final plots add the option 'useList': -/// -/// root[] runProof("h1,useList") -/// -/// 3. "event" -/// -/// Selector: ProofEvent.h,.C -/// -/// This is an example of using PROOF par files. -/// It runs event generation and simple analysis based on the 'Event' -/// class found under test. -/// -/// root[] runProof("event") -/// -/// 4. "eventproc" -/// -/// Selector: ProofEventProc.h.C -/// -/// This is an example of using PROOF par files and process 'event' -/// data from the ROOT HTTP server. It runs the ProofEventProc selector -/// which is derived from the EventTree_Proc one found under -/// test/ProofBench. The following specific arguments are available: -/// - 'readall' to read the whole event, by default only the branches -/// needed by the analysis are read (read 25% more bytes) -/// - 'datasrc=' to read the files from another server, -/// the files must be named 'event_.root' where ''=1,2,... -/// or -/// - 'datasrc=' to take the file content from a text file, -/// specified one file per line; usefull when testing differences -/// between several sources and distributions -/// - 'files=N' to change the number of files to be analysed (default -/// is 10, max is 50 for the HTTP server). -/// - 'uneven' to process uneven entries from files following the scheme -/// {50000,5000,5000,5000,5000} and so on -/// -/// root[] runProof("eventproc") -/// -/// 5. "pythia8" -/// -/// Selector: ProofPythia.h.C -/// -/// This runs Pythia8 generation based on main03.cc example in Pythia 8.1 -/// -/// To run this analysis ROOT must be configured with pythia8. -/// -/// Note that before executing this analysis, the env variable PYTHIA8 -/// must point to the pythia8100 (or newer) directory, in particular, -/// $PYTHIA8/xmldoc must contain the file Index.xml. The tutorial assumes -/// that the Pythia8 directory is the same on all machines, i.e. local -/// and worker ones. -/// -/// root[] runProof("pythia8") -/// -/// 6. "ntuple" -/// -/// Selector: ProofNtuple.h.C -/// -/// This is an example of final merging via files created on the workers, -/// using TProofOutputFile. The final file is called ProofNtuple.root -/// and it is created in the directory where the tutorial is run. If -/// the PROOF cluster is remote, the file is received by a local xrootd -/// daemon started for the purpose. Because of this, this example can be -/// run only on unix clients. -/// -/// root[] runProof("ntuple") -/// -/// By default the random numbers are generate anew. There is the -/// possibility use a file of random numbers (to have reproducible results) -/// by specify the option 'inputrndm', e.g. -/// -/// root[] runProof("ntuple(inputrndm)") -/// -/// By default the output will be saved in the local file SimpleNtuple.root; -/// location and name of the file can be changed via the argument 'outfile', -/// e.g. -/// -/// root[] runProof("simplefile(outfile=/data0/testntuple.root)") -/// root[] runProof("simplefile(outfile=root://aserver//data/testntuple.root)") -/// -/// 7. "dataset" -/// -/// Selector: ProofNtuple.h.C -/// -/// This is an example of automatic creation of a dataset from files -/// created on the workers, using TProofOutputFile. The dataset is -/// called testNtuple and it is automatically registered and verified. -/// The files contain the same ntuple as in previous example/tutorial 6 -/// (the same selector ProofNTuple is used with a slightly different -/// configuration). The dataset is then used to produce the same plot -/// as in 5 but using the DrawSelect methods of PROOF, which also show -/// how to set style, color and other drawing attributes in PROOF. -/// Depending on the relative worker perforance, some of the produced -/// files may result in having no entries. If this happens, the file -/// will be added to the missing (skipped) file list. Increasing the -/// number of events (via nevt=...) typically solves this issue. -/// -/// root[] runProof("dataset") -/// -/// 8. "friends" -/// -/// Selectors: ProofFriends.h(.C), ProofAux.h(.C) -/// -/// This is an example of TTree friend processing in PROOF. It also shows -/// how to use the TPacketizerFile to steer creation of files. -/// -/// root[] runProof("friends") -/// -/// The trees are by default created in separate files; to create -/// them in the same file use option 'samefile', e.g. -/// -/// root[] runProof("friends(samefile)") -/// -/// 9. "simplefile" -/// -/// Selector: ProofSimpleFile.h.C -/// -/// root[] runProof("simplefile") -/// -/// This will create a local PROOF session and run an analysis filling -/// 16+16 histos with 100000 gaussian random numbers. The merging of -/// these histos goes via file; 16 histos are saved in the top directory, -/// the other 16 into a subdirectory called 'blue'. The final display -/// is done in two canvanses, one for each set of histograms and with -/// 16 pads each (4x4). -/// The number of histograms in each set can be passed as argument -/// 'nhist' to 'simplefile', e.g. to fill 25 histos with 1000000 entries use -/// -/// root[] runProof("simplefile(nevt=1000000,nhist=25)") -/// -/// By default the output will be saved in the local file SimpleFile.root; -/// location and name of the file can be changed via the argument 'outfile', -/// e.g. -/// -/// root[] runProof("simplefile(outfile=/data0/testsimple.root)") -/// root[] runProof("simplefile(outfile=root://aserver//data/testsimple.root)") -/// -/// 10. "stdvec" -/// -/// Selector: ProofStdVect.h.C -/// -/// This is an example of using standard vectors (vector > and -/// vector >) in a TSelector. The same selector is run twice: -/// in 'create' mode it creates a dataset with the tree 'stdvec' containing -/// 3 branches, a vector > and two vector >. The -/// tree is saved into a file on each worker and a dataset is created with -/// these files (the dataset is called 'TestStdVect'); in 'read' mode the -/// dataset is read and a couple fo histograms filled and displayed. -/// -/// root[] runProof("stdvec") -/// -/// General arguments -/// ----------------- -/// -/// The following arguments are valid for all examples (the ones specific -/// to each tutorial have been explained above) -/// -/// 1. ACLiC mode -/// -/// By default all processing is done with ACLiC mode '+', i.e. compile -/// if changed. However, this may lead to problems if the available -/// selector libs were compiled in previous sessions with a different -/// set of loaded libraries (this is a general problem in ROOT). When -/// this happens the best solution is to force recompilation (ACLiC -/// mode '++'). To do this just add one or more '+' to the name of the -/// tutorial, e.g. runProof("simple++") -/// -/// 2. debug=[what:]level -/// -/// Controls verbosity; 'level' is an integer number and the optional string -/// 'what' one or more of the enum names in TProofDebug.h . -/// e.g. runProof("eventproc(debug=kPacketizer|kCollect:2)") runs 'eventproc' enabling -/// all printouts matching TProofDebug::kPacketizer and having level -/// equal or larger than 2 . -/// -/// 3. nevt=N and/or first=F -/// -/// Set the number of entries to N, eventually (when it makes sense, i.e. when -/// processing existing files) starting from F -/// e.g. runProof("simple(nevt=1000000000)") runs simple with 1000000000 -/// runProof("eventproc(first=65000)") runs eventproc processing -/// starting with event 65000 -/// runProof("eventproc(nevt=100000,first=65000)") runs eventproc processing -/// 100000 events starting with event 65000 -/// -/// 4. asyn -/// -/// Run in non blocking mode -/// e.g. root[] runProof("h1(asyn)") -/// -/// 5. nwrk=N -/// -/// Set the number of active workers to N, usefull to test performance -/// on a remote cluster where control about the number of workers is -/// not possible, e.g. runProof("event(nwrk=2)") runs 'event' with -/// 2 workers. -/// -/// 6. punzip -/// -/// Use parallel unzipping in reading files where relevant -/// e.g. root[] runProof("eventproc(punzip)") -/// -/// 7. cache=`` (or `K or ``M) -/// -/// Change the size of the tree cache; 0 or <0 disables the cache, -/// value cane be in bytes (no suffix), kilobytes (suffix 'K') or -/// megabytes (suffix 'M'), e.g. root[] runProof("eventproc(cache=0)") -/// -/// 8. submergers[=S] -/// -/// Enabling merging via S submergers or the optimal number if S is -/// not specified, e.g. root[] runProof("simple(hist=1000,submergers)") -/// -/// 9. rateest=average -/// -/// Enable processed entries estimation for constant progress reporting based on -/// the measured average. This may screw up the progress bar in some cases, which -/// is the reason why it is not on by default . -/// e.g. root[] runProof("eventproc(rateest=average)") -/// -/// 10. perftree=perftreefile.root -/// -/// Generate the perfomance tree and save it to file 'perftreefile.root', -/// e.g. root[] runProof("eventproc(perftree=perftreefile.root)") -/// -/// 11. feedback=name1[,name2,name3,...]|off -/// -/// Enable feedback for the specified names or switch it off; by default it is -/// enabled for the 'stats' histograms (events,packest, packets-being processed). -/// -/// In all cases, to run on a remote PROOF cluster, the master URL must -/// be passed as second argument; e.g. -/// -/// root[] runProof("simple","master.do.main") -/// -/// A rough parsing of the URL is done to determine the locality of the cluster. -/// If using a tunnel the URL can start by localhost even for external clusters: -/// in such cases the default locality determination will be wrong, so one has -/// to tell explicity that the cluster is external via the option field, e.g. -/// -/// root[] runProof("simple","localhost:33002/?external") -/// -/// In the case of local running it is possible to specify the number of -/// workers to start as third argument (the default is the number of cores -/// of the machine), e.g. -/// -/// root[] runProof("simple",0,4) -/// -/// will start 4 workers. Note that the real number of workers is changed -/// only the first time you call runProof into a ROOT session. Following -/// calls can reduce the number of active workers, but not increase it. -/// For example, in the same session of the call above starting 4 workers, -/// this -/// -/// root[] runProof("simple",0,8) -/// -/// will still use 4 workers, while this -/// -/// root[] runProof("simple",0,2) -/// -/// will disable 2 workers and use the other 2. -/// -/// Finally, it is possible to pass as 4th argument a list of objects to be added -/// to the input list to further control the PROOF behaviour: -/// -/// root [] TList *ins = new TList -/// root [] ins->Add(new TParameter("MyParm", 3)) -/// root [] runProof("simple",0,4,ins) -/// -/// the content of 'ins' will then be copied to the input list before processing. -/// -/// -/// \macro_code -/// -/// \author Gerardo Ganis - - -#include "TCanvas.h" -#include "TChain.h" -#include "TDSet.h" -#include "TEnv.h" -#include "TEntryList.h" -#include "TFile.h" -#include "TFileCollection.h" -#include "TFrame.h" -#include "THashList.h" -#include "TList.h" -#include "TPad.h" -#include "TPaveText.h" -#include "TProof.h" -#include "TProofDebug.h" -#include "TString.h" - -#include "getProof.C" -void plotNtuple(TProof *p, const char *ds, const char *ntptitle); -void SavePerfTree(TProof *proof, const char *fn); - -// Variable used to locate the Pythia8 directory for the Pythia8 example -const char *pythia8dir = 0; -const char *pythia8data = 0; - -void runProof(const char *what = "simple", - const char *masterurl = "proof://localhost:40000", - Int_t nwrks = -1, TList *ins = 0) -{ - - gEnv->SetValue("Proof.StatsHist",1); - - TString u(masterurl); - // Determine locality of this session - Bool_t isProofLocal = kFALSE; - if (!u.IsNull() && u != "lite://") { - TUrl uu(masterurl); - TString uopts(uu.GetOptions()); - if ((!strcmp(uu.GetHost(), "localhost") && !uopts.Contains("external")) || - !strcmp(uu.GetHostFQDN(), TUrl(gSystem->HostName()).GetHostFQDN())) { - isProofLocal = kTRUE; - } - // Adjust URL - if (!u.BeginsWith(uu.GetProtocol())) uu.SetProtocol("proof"); - uopts.ReplaceAll("external", ""); - uu.SetOptions(uopts.Data()); - u = uu.GetUrl(); - } - const char *url = u.Data(); - - // Temp dir for PROOF tutorials - // Force "/tmp/" whenever possible to avoid length problems on MacOsX - TString tmpdir("/tmp"); - if (gSystem->AccessPathName(tmpdir, kWritePermission)) tmpdir = gSystem->TempDirectory(); - TString us; - UserGroup_t *ug = gSystem->GetUserInfo(gSystem->GetUid()); - if (!ug) { - Printf("runProof: could not get user info"); - return; - } - us.Form("/%s", ug->fUser.Data()); - if (!tmpdir.EndsWith(us.Data())) tmpdir += us; - gSystem->mkdir(tmpdir.Data(), kTRUE); - if (gSystem->AccessPathName(tmpdir, kWritePermission)) { - Printf("runProof: unable to get a writable tutorial directory (tried: %s)" - " - cannot continue", tmpdir.Data()); - return; - } - TString tutdir = Form("%s/.proof-tutorial", tmpdir.Data()); - if (gSystem->AccessPathName(tutdir)) { - Printf("runProof: creating the temporary directory" - " for the tutorial (%s) ... ", tutdir.Data()); - if (gSystem->mkdir(tutdir, kTRUE) != 0) { - Printf("runProof: could not assert / create the temporary directory" - " for the tutorial (%s)", tutdir.Data()); - return; - } - } - - // For the Pythia8 example we need to set some environment variable; - // This must be done BEFORE starting the PROOF session - if (what && !strncmp(what, "pythia8", 7)) { - // We assume that the remote location of Pythia8 is the same as the local one - pythia8dir = gSystem->Getenv("PYTHIA8"); - if (!pythia8dir || strlen(pythia8dir) <= 0) { - Printf("runProof: pythia8: environment variable PYTHIA8 undefined:" - " it must contain the path to pythia81xx root directory (local and remote) !"); - return; - } - pythia8data = gSystem->Getenv("PYTHIA8DATA"); - if (!pythia8data || strlen(pythia8data) <= 0) { - gSystem->Setenv("PYTHIA8DATA", Form("%s/xmldoc", pythia8dir)); - pythia8data = gSystem->Getenv("PYTHIA8DATA"); - if (!pythia8data || strlen(pythia8data) <= 0) { - Printf("runProof: pythia8: environment variable PYTHIA8DATA undefined:" - " it one must contain the path to pythia81xx/xmldoc" - " subdirectory (local and remote) !"); - return; - } - } - TString env = Form("echo export PYTHIA8=%s; export PYTHIA8DATA=%s", - pythia8dir, pythia8data); - TProof::AddEnvVar("PROOF_INITCMD", env.Data()); - } - - Printf("tutorial dir:\t%s", tutdir.Data()); - - // Get the PROOF Session - TProof *proof = getProof(url, nwrks, tutdir.Data(), "ask"); - if (!proof) { - Printf("runProof: could not start/attach a PROOF session"); - return; - } - - // Refine locality (PROOF-Lite always local) - if (proof->IsLite()) isProofLocal = kTRUE; - -#ifdef WIN32 - if (isProofLocal && what && !strcmp(what, "ntuple", 6)) { - // Not support on windows - Printf("runProof: the 'ntuple' example needs to run xrootd to receive the output file, \n" - " but xrootd is not supported on Windows - cannot continue"); - return; - } -#endif - - TString proofsessions(Form("%s/sessions",tutdir.Data())); - // Save tag of the used session - FILE *fs = fopen(proofsessions.Data(), "a"); - if (!fs) { - Printf("runProof: could not create files for sessions tags"); - } else { - fprintf(fs,"session-%s\n", proof->GetSessionTag()); - fclose(fs); - } - if (!proof) { - Printf("runProof: could not start/attach a PROOF session"); - return; - } - - // Set the number of workers (may only reduce the number of active workers - // in the session) - if (nwrks > 0) - proof->SetParallel(nwrks); - - // Where is the code to run - char *rootbin = gSystem->Which(gSystem->Getenv("PATH"), "root.exe", kExecutePermission); - if (!rootbin) { - Printf("runProof: root.exe not found: please check the environment!"); - return; - } - TString rootsys = gSystem->GetDirName(rootbin); - rootsys = gSystem->GetDirName(rootsys); - TString tutorials(Form("%s/tutorials", rootsys.Data())); - delete[] rootbin; - - // Parse 'what'; it is in the form 'analysis(arg1,arg2,...)' - TString args(what); - args.ReplaceAll("("," "); - args.ReplaceAll(")"," "); - args.ReplaceAll(","," "); - Ssiz_t from = 0; - TString act, tok; - if (!args.Tokenize(act, from, " ")) { - // Cannot continue - Printf("runProof: action not found: check your arguments (%s)", what); - return; - } - // Extract ACLiC mode - TString aMode = "+"; - if (act.EndsWith("+")) { - aMode += "+"; - while (act.EndsWith("+")) { act.Remove(TString::kTrailing,'+'); } - } - Printf("runProof: %s: ACLiC mode: '%s'", act.Data(), aMode.Data()); - - // Parse out number of events and 'asyn' option, used almost by every test - TString aNevt, aFirst, aNwrk, opt, sel, punzip("off"), aCache, aOutFile, - aH1Src("http://root.cern/files/h1"), - aDebug, aDebugEnum, aRateEst, aPerfTree("perftree.root"), - aFeedback("fb=stats"); - Long64_t suf = 1; - Int_t aSubMg = -1; - Bool_t useList = kFALSE, makePerfTree = kFALSE; - while (args.Tokenize(tok, from, " ")) { - // Debug controllers - if (tok.BeginsWith("debug=")) { - aDebug = tok; - aDebug.ReplaceAll("debug=",""); - Int_t icol = kNPOS; - if ((icol = aDebug.Index(":")) != kNPOS) { - aDebugEnum = aDebug(0, icol); - aDebug.Remove(0, icol+1); - } - if (!aDebug.IsDigit()) { - Printf("runProof: %s: error parsing the 'debug=' option (%s) - ignoring", act.Data(), tok.Data()); - aDebug = ""; - aDebugEnum = ""; - } - } - // Number of events - if (tok.BeginsWith("nevt=")) { - aNevt = tok; - aNevt.ReplaceAll("nevt=",""); - if (!aNevt.IsDigit()) { - Printf("runProof: %s: error parsing the 'nevt=' option (%s) - ignoring", act.Data(), tok.Data()); - aNevt = ""; - } - } - // First event - if (tok.BeginsWith("first=")) { - aFirst = tok; - aFirst.ReplaceAll("first=",""); - if (!aFirst.IsDigit()) { - Printf("runProof: %s: error parsing the 'first=' option (%s) - ignoring", act.Data(), tok.Data()); - aFirst = ""; - } - } - // Sync or async ? - if (tok.BeginsWith("asyn")) - opt = "ASYN"; - // Number of workers - if (tok.BeginsWith("nwrk=")) { - aNwrk = tok; - aNwrk.ReplaceAll("nwrk=",""); - if (!aNwrk.IsDigit()) { - Printf("runProof: %s: error parsing the 'nwrk=' option (%s) - ignoring", act.Data(), tok.Data()); - aNwrk = ""; - } - } - // Parallel unzipping ? - if (tok.BeginsWith("punzip")) - punzip = "on"; - // Number of workers - if (tok.BeginsWith("cache=")) { - aCache = tok; - aCache.ReplaceAll("cache=",""); - if (aCache.EndsWith("k")) { aCache.Remove(TString::kTrailing, 'k'); suf = 1024; } - if (aCache.EndsWith("K")) { aCache.Remove(TString::kTrailing, 'K'); suf = 1024; } - if (aCache.EndsWith("M")) { aCache.Remove(TString::kTrailing, 'M'); suf = 1024*1024; } - if (!aCache.IsDigit()) { - Printf("runProof: %s: error parsing the 'cache=' option (%s) - ignoring", act.Data(), tok.Data()); - aCache = ""; - } - } - // Use submergers? - if (tok.BeginsWith("submergers")) { - tok.ReplaceAll("submergers",""); - aSubMg = 0; - if (tok.BeginsWith("=")) { - tok.ReplaceAll("=",""); - if (tok.IsDigit()) aSubMg = tok.Atoi(); - } - } - // H1: use entry-lists ? - if (tok.BeginsWith("useList")) { - useList = kTRUE; - } - if (tok.BeginsWith("fillList")) { - opt += "fillList"; - } - // H1: change location of files? - if (tok.BeginsWith("h1src=")) { - tok.ReplaceAll("h1src=",""); - if (!(tok.IsNull())) aH1Src = tok; - Printf("runProof: %s: reading data files from '%s'", act.Data(), aH1Src.Data()); - } - // Rate estimation technique - if (tok.BeginsWith("rateest=")) { - tok.ReplaceAll("rateest=",""); - if (!(tok.IsNull())) aRateEst = tok; - Printf("runProof: %s: progress-bar rate estimation option: '%s'", act.Data(), aRateEst.Data()); - } - // Create and save the preformance tree? - if (tok.BeginsWith("perftree")) { - makePerfTree = kTRUE; - if (tok.BeginsWith("perftree=")) { - tok.ReplaceAll("perftree=",""); - if (!(tok.IsNull())) aPerfTree = tok; - } - Printf("runProof: %s: saving performance tree to '%s'", act.Data(), aPerfTree.Data()); - } - // Location of the output file, if any - if (tok.BeginsWith("outfile")) { - if (tok.BeginsWith("outfile=")) { - tok.ReplaceAll("outfile=",""); - if (!(tok.IsNull())) aOutFile = tok; - } - Printf("runProof: %s: output file: '%s'", act.Data(), aOutFile.Data()); - } - // Feedback - if (tok.BeginsWith("feedback=")) { - tok.ReplaceAll("feedback=",""); - if (tok == "off" || tok == "OFF" || tok == "0") { - aFeedback = ""; - } else if (!(tok.IsNull())) { - if (tok.BeginsWith("+")) { - tok[0] = ','; - aFeedback += tok; - } else { - aFeedback.Form("fb=%s", tok.Data()); - } - } - Printf("runProof: %s: feedback: '%s'", act.Data(), aFeedback.Data()); - } - } - Long64_t nevt = (aNevt.IsNull()) ? -1 : aNevt.Atoi(); - Long64_t first = (aFirst.IsNull()) ? 0 : aFirst.Atoi(); - Long64_t nwrk = (aNwrk.IsNull()) ? -1 : aNwrk.Atoi(); - from = 0; - - // Set number workers - if (nwrk > 0) { - if (proof->GetParallel() < nwrk) { - Printf("runProof: %s: request for a number of workers larger then available - ignored", act.Data()); - } else { - proof->SetParallel(nwrk); - } - } - - // Debug controllers - if (!aDebug.IsNull()) { - Int_t dbg = aDebug.Atoi(); - Int_t scope = TProofDebug::kAll; - if (!aDebugEnum.IsNull()) scope = getDebugEnum(aDebugEnum.Data()); - proof->SetLogLevel(dbg, scope); - Printf("runProof: %s: verbose mode for '%s'; level: %d", act.Data(), aDebugEnum.Data(), dbg); - } - - // Have constant progress reporting based on estimated info - // (NB: may screw up the progress bar in some cases) - if (aRateEst == "average") - proof->SetParameter("PROOF_RateEstimation", aRateEst); - - // Parallel unzip - if (punzip == "on") { - proof->SetParameter("PROOF_UseParallelUnzip", (Int_t)1); - Printf("runProof: %s: parallel unzip enabled", act.Data()); - } else { - proof->SetParameter("PROOF_UseParallelUnzip", (Int_t)0); - } - - // Tree cache - if (!aCache.IsNull()) { - Long64_t cachesz = aCache.Atoi() * suf; - if (cachesz <= 0) { - proof->SetParameter("PROOF_UseTreeCache", (Int_t)0); - Printf("runProof: %s: disabling tree cache", act.Data()); - } else { - proof->SetParameter("PROOF_UseTreeCache", (Int_t)1); - proof->SetParameter("PROOF_CacheSize", cachesz); - Printf("runProof: %s: setting cache size to %lld", act.Data(), cachesz); - } - } else { - // Use defaults - proof->DeleteParameters("PROOF_UseTreeCache"); - proof->DeleteParameters("PROOF_CacheSize"); - } - - // Enable submergers, if required - if (aSubMg >= 0) { - proof->SetParameter("PROOF_UseMergers", aSubMg); - if (aSubMg > 0) { - Printf("runProof: %s: enabling merging via %d sub-mergers", act.Data(), aSubMg); - } else { - Printf("runProof: %s: enabling merging via sub-mergers (optimal number)", act.Data()); - } - } else { - proof->DeleteParameters("PROOF_UseMergers"); - } - - // The performance tree - if (makePerfTree) { - proof->SetParameter("PROOF_StatsHist", ""); - proof->SetParameter("PROOF_StatsTrace", ""); - proof->SetParameter("PROOF_SlaveStatsTrace", ""); - } - - // Additional inputs from the argument 'ins' - if (ins && ins->GetSize() > 0) { - TObject *oin = 0; - TIter nxo(ins); - while ((oin = nxo())) { proof->AddInput(oin); } - } - - // Full lits of inputs so far - proof->GetInputList()->Print(); - - // Action - if (act == "simple") { - // ProofSimple is an example of non-data driven analysis; it - // creates and fills with random numbers a given number of histos - - if (first > 0) - // Meaningless for this tutorial - Printf("runProof: %s: warning concept of 'first' meaningless for this tutorial" - " - ignored", act.Data()); - - // Default 10000 events - nevt = (nevt < 0) ? 100000 : nevt; - // Find out the number of histograms - TString aNhist, aNhist3; - while (args.Tokenize(tok, from, " ")) { - // Number of histos - if (tok.BeginsWith("nhist=")) { - aNhist = tok; - aNhist.ReplaceAll("nhist=",""); - if (!aNhist.IsDigit()) { - Printf("runProof: error parsing the 'nhist=' option (%s) - ignoring", tok.Data()); - aNhist = ""; - } - } else if (tok.BeginsWith("nhist3=")) { - aNhist3 = tok; - aNhist3.ReplaceAll("nhist3=",""); - if (!aNhist3.IsDigit()) { - Printf("runProof: error parsing the 'nhist3=' option (%s) - ignoring", tok.Data()); - aNhist3 = ""; - } - } - } - Int_t nhist = (aNhist.IsNull()) ? 100 : aNhist.Atoi(); - Int_t nhist3 = (aNhist3.IsNull()) ? -1 : aNhist3.Atoi(); - Printf("\nrunProof: running \"simple\" with nhist= %d, nhist3=%d and nevt= %lld\n", nhist, nhist3, nevt); - - // The number of histograms is added as parameter in the input list - proof->SetParameter("ProofSimple_NHist", (Long_t)nhist); - // The number of histograms is added as parameter in the input list - if (nhist3 > 0) proof->SetParameter("ProofSimple_NHist3", (Long_t)nhist3); - // The selector string - sel.Form("%s/proof/ProofSimple.C%s", tutorials.Data(), aMode.Data()); - // - // Run it for nevt times - TString xopt = aFeedback; if (!opt.IsNull()) xopt += TString::Format(" %s", opt.Data()); - proof->Process(sel.Data(), nevt, xopt); - - } else if (act == "h1") { - // This is the famous 'h1' example analysis run on Proof reading the - // data from the ROOT http server. - - // Create the chain - TChain *chain = new TChain("h42"); - chain->Add(TString::Format("%s/dstarmb.root", aH1Src.Data())); - chain->Add(TString::Format("%s/dstarp1a.root", aH1Src.Data())); - chain->Add(TString::Format("%s/dstarp1b.root", aH1Src.Data())); - chain->Add(TString::Format("%s/dstarp2.root", aH1Src.Data())); - chain->ls(); - // We run on Proof - chain->SetProof(); - // Set entrylist, if required - if (useList) { - TString eln("elist"), elfn("elist.root"); - if (gSystem->AccessPathName(elfn)) { - Printf("\nrunProof: asked to use an entry list but '%s' not found or not readable", elfn.Data()); - Printf("\nrunProof: did you forget to run with 'fillList=%s'?\n", elfn.Data()); - } else { - TFile f(elfn); - if (!(f.IsZombie())) { - TEntryList *elist = (TEntryList *)f.Get(eln); - if (elist) { - elist->SetDirectory(0); //otherwise the file destructor will delete elist - chain->SetEntryList(elist); - } else { - Printf("\nrunProof: could not find entry-list '%s' in file '%s': ignoring", - eln.Data(), elfn.Data()); - } - } else { - Printf("\nrunProof: requested entry-list file '%s' not existing (or not readable):" - " ignoring", elfn.Data()); - } - } - } - // The selector - sel.Form("%s/tree/h1analysis.C%s", tutorials.Data(), aMode.Data()); - // Run it - Printf("\nrunProof: running \"h1\"\n"); - TString xopt = aFeedback; if (!opt.IsNull()) xopt += TString::Format(" %s", opt.Data()); - chain->Process(sel.Data(),xopt,nevt,first); - // Cleanup the input list - gProof->ClearInputData("elist"); - gProof->ClearInputData("elist.root"); - TIter nxi(gProof->GetInputList()); - TObject *o = 0; - while ((o = nxi())) { - if (!strncmp(o->GetName(), "elist", 5)) { - gProof->GetInputList()->Remove(o); - delete o; - } - } - - } else if (act == "pythia8") { - - if (first > 0) - Printf("runProof: %s: warning concept of 'first' meaningless for this tutorial" - " - ignored", act.Data()); - - TString path(Form("%s/Index.xml", pythia8data)); - gSystem->ExpandPathName(path); - if (gSystem->AccessPathName(path)) { - Printf("runProof: pythia8: PYTHIA8DATA directory (%s) must" - " contain the Index.xml file !", pythia8data); - return; - } - TString pythia8par = TString::Format("%s/proof/pythia8.par", tutorials.Data()); - if (gSystem->AccessPathName(pythia8par.Data())) { - Printf("runProof: pythia8: par file not found (tried %s)", pythia8par.Data()); - return; - } - proof->UploadPackage(pythia8par); - proof->EnablePackage("pythia8"); - // Show enabled packages - proof->ShowEnabledPackages(); - Printf("runProof: pythia8: check settings:"); - proof->Exec(".!echo hostname = `hostname`; echo \"ls pythia8:\"; ls pythia8"); - // Loading libraries needed - if (gSystem->Load("libEG.so") < 0) { - Printf("runProof: pythia8: libEG not found \n"); - return; - } - if (gSystem->Load("libEGPythia8.so") < 0) { - Printf("runProof: pythia8: libEGPythia8 not found \n"); - return; - } - // Setting the default number of events, if needed - nevt = (nevt < 0) ? 100 : nevt; - Printf("\nrunProof: running \"Pythia01\" nevt= %lld\n", nevt); - // The selector string - sel.Form("%s/proof/ProofPythia.C%s", tutorials.Data(), aMode.Data()); - // Run it for nevt times - TString xopt = aFeedback; if (!opt.IsNull()) xopt += TString::Format(" %s", opt.Data()); - proof->Process(sel.Data(), nevt, xopt); - - } else if (act == "event") { - - if (first > 0) - // Meaningless for this tutorial - Printf("runProof: %s: warning concept of 'first' meaningless for this tutorial" - " - ignored", act.Data()); - - TString eventpar = TString::Format("%s/proof/event.par", tutorials.Data()); - if (gSystem->AccessPathName(eventpar.Data())) { - Printf("runProof: event: par file not found (tried %s)", eventpar.Data()); - return; - } - - proof->UploadPackage(eventpar); - proof->EnablePackage("event"); - Printf("Enabled packages...\n"); - proof->ShowEnabledPackages(); - - // Setting the default number of events, if needed - nevt = (nevt < 0) ? 100 : nevt; - Printf("\nrunProof: running \"event\" nevt= %lld\n", nevt); - // The selector string - sel.Form("%s/proof/ProofEvent.C%s", tutorials.Data(), aMode.Data()); - // Run it for nevt times - TString xopt = aFeedback; if (!opt.IsNull()) xopt += TString::Format(" %s", opt.Data()); - proof->Process(sel.Data(), nevt, xopt); - - } else if (act == "eventproc") { - - TString eventpar = TString::Format("%s/proof/event.par", tutorials.Data()); - gSystem->ExpandPathName(eventpar); - if (gSystem->AccessPathName(eventpar.Data())) { - Printf("runProof: eventproc: par file not found (tried %s)", eventpar.Data()); - return; - } - - proof->UploadPackage(eventpar); - proof->EnablePackage("event"); - Printf("Enabled packages...\n"); - proof->ShowEnabledPackages(); - - // Load ProcFileElements (to check processed ranges) - TString pfelem = TString::Format("%s/proof/ProcFileElements.C", tutorials.Data()); - gSystem->ExpandPathName(pfelem); - if (gSystem->AccessPathName(pfelem.Data())) { - Printf("runProof: eventproc: ProcFileElements.C not found (tried %s)", pfelem.Data()); - return; - } - pfelem += aMode; - // Add include to test trasmission - pfelem += TString::Format(",%s/proof/EmptyInclude.h", tutorials.Data()); - proof->Load(pfelem); - - // Extract the number of files to process, data source and - // other parameters controlling the run ... - Bool_t uneven = kFALSE; - TString aFiles, aDataSrc("http://root.cern/files/data"), aPartitions; - proof->SetParameter("ProofEventProc_Read", "optimized"); - while (args.Tokenize(tok, from, " ")) { - // Number of events - if (tok.BeginsWith("files=")) { - aFiles = tok; - aFiles.ReplaceAll("files=",""); - if (!aFiles.IsDigit()) { - Printf("runProof: error parsing the 'files=' option (%s) - ignoring", tok.Data()); - aFiles = ""; - } - } else if (tok.BeginsWith("datasrc=")) { - tok.ReplaceAll("datasrc=",""); - if (tok.IsDigit()) { - Printf("runProof: error parsing the 'datasrc=' option (%s) - ignoring", tok.Data()); - } else { - aDataSrc = tok; - Printf("runProof: reading files from: %s", aDataSrc.Data()); - } - } else if (tok == "readall") { - proof->SetParameter("ProofEventProc_Read", "readall"); - Printf("runProof: eventproc: reading the full event"); - } else if (tok == "uneven") { - uneven = kTRUE; - } else if (tok.BeginsWith("partitions=")) { - tok.ReplaceAll("partitions=",""); - if (tok.IsDigit()) { - Printf("runProof: error parsing the 'partitions=' option (%s) - ignoring", tok.Data()); - } else { - aPartitions = tok; - Printf("runProof: partitions: %s included in packetizer operations", aPartitions.Data()); - } - } - } - Int_t nFiles = (aFiles.IsNull()) ? 10 : aFiles.Atoi(); - Printf("runProof: found aFiles: '%s', nFiles: %d", aFiles.Data(), nFiles); - if (nFiles > 50) { - Printf("runProof: max number of files is 50 - resizing request"); - nFiles = 50; - } - - // We create the chain now - TChain *c = new TChain("EventTree"); - - FileStat_t fst; - if (gSystem->GetPathInfo(aDataSrc, fst) == 0 && R_ISREG(fst.fMode) && - !gSystem->AccessPathName(aDataSrc, kReadPermission)) { - // It is a local file, we get the TFileCollection and we inject it into the chain - TFileCollection *fc = new TFileCollection("", "", aDataSrc, nFiles); - c->AddFileInfoList(fc->GetList()); - delete fc; - - } else { - - // Tokenize the source: if more than 1 we rotate the assignment. More sources can be specified - // separating them by a '|' - TObjArray *dsrcs = aDataSrc.Tokenize("|"); - Int_t nds = dsrcs->GetEntries(); - - // Fill the chain - Int_t i = 1, k = 0; - TString fn; - for (i = 1; i <= nFiles; i++) { - k = (i - 1) % nds; - TObjString *os = (TObjString *) (*dsrcs)[k]; - if (os) { - fn.Form("%s/event_%d.root", os->GetName(), i); - if (uneven) { - if ((i - 1)%5 == 0) - c->AddFile(fn.Data(), 50000); - else - c->AddFile(fn.Data(), 5000); - } else { - c->AddFile(fn.Data()); - } - } - } - dsrcs->SetOwner(); - delete dsrcs; - } - // Show the chain - c->ls(); - c->SetProof(); - - // Only validate the files really needed for the analysis - proof->SetParameter("PROOF_ValidateByFile", 1); - - // Send over the partition information, if any - if (!aPartitions.IsNull()) { - aPartitions.ReplaceAll("|", ","); - proof->SetParameter("PROOF_PacketizerPartitions", aPartitions); - } - - // The selector - sel.Form("%s/proof/ProofEventProc.C%s", tutorials.Data(), aMode.Data()); - // Run it - Printf("\nrunProof: running \"eventproc\"\n"); - TString xopt = aFeedback; if (!opt.IsNull()) xopt += TString::Format(" %s", opt.Data()); - c->Process(sel.Data(), xopt, nevt, first); - - } else if (act == "ntuple") { - - // ProofNtuple is an example of non-data driven analysis; it - // creates and fills a disk resident ntuple with automatic file merging - - if (first > 0) - // Meaningless for this tutorial - Printf("runProof: %s: warning concept of 'first' meaningless for this tutorial" - " - ignored", act.Data()); - - // Set the default number of events, if needed - nevt = (nevt < 0) ? 1000 : nevt; - Printf("\nrunProof: running \"ntuple\" with nevt= %lld\n", nevt); - - // Which randoms to use - Bool_t usentprndm = kFALSE; - while (args.Tokenize(tok, from, " ")) { - if (tok == "inputrndm") { - usentprndm = kTRUE; - break; - } - } - if (usentprndm) Printf("runProof: taking randoms from input ntuple\n"); - - // Output file - TString fout(aOutFile); - if (fout.IsNull()) { - fout.Form("%s/ProofNtuple.root", gSystem->WorkingDirectory()); - // Cleanup any existing instance of the output file - gSystem->Unlink(fout); - - if (!isProofLocal) { - // Setup a local basic xrootd to receive the file - Bool_t xrdok = kFALSE; - Int_t port = 9000; - while (port < 9010) { - if (checkXrootdAt(port) != 1) { - if (startXrootdAt(port, gSystem->WorkingDirectory(), kTRUE) == 0) { - xrdok = kTRUE; - break; - } - } - port++; - } - if (!xrdok) { - Printf("runProof: could not start basic xrootd on ports 9000-9009 - cannot continue"); - return; - } - fout.Insert(0, TString::Format("root://%s:%d/", TUrl(gSystem->HostName()).GetHostFQDN(), port)); - // Make a copy of the files on the master before merging - proof->AddInput(new TNamed("PROOF_OUTPUTFILE_LOCATION", "LOCAL")); - } - } - proof->AddInput(new TNamed("PROOF_OUTPUTFILE", fout.Data())); - - // If using the 'NtpRndm' for a fixed values of randoms, send over the file - if (usentprndm) { - // The file with 'NtpRndm' - TString fnr = TString::Format("%s/proof/ntprndm.root", tutorials.Data()); - // Set as input data - proof->SetInputDataFile(fnr); - // Set the related parameter - proof->SetParameter("PROOF_USE_NTP_RNDM","yes"); - // Notify - Printf("runProof: taking randoms from '%s'", fnr.Data()); - } - - // The selector string - sel.Form("%s/proof/ProofNtuple.C%s", tutorials.Data(), aMode.Data()); - - // Run it for nevt times - TString xopt = aFeedback; if (!opt.IsNull()) xopt += TString::Format(" %s", opt.Data()); - Printf("runProof: selector file '%s', options: '%s'", sel.Data(), xopt.Data()); - proof->Process(sel.Data(), nevt, xopt); - - // Reset input variables - if (usentprndm) { - proof->DeleteParameters("PROOF_USE_NTP_RNDM"); - proof->SetInputDataFile(0); - } - - } else if (act == "dataset") { - - // This is an example of analysis creating data files on each node which are - // automatically registered as dataset; the newly created dataset is used to create - // the final plots. The data are of the same type as for the 'ntuple' example. - // Selector used: ProofNtuple - - if (first > 0) - // Meaningless for this tutorial - Printf("runProof: %s: warning concept of 'first' meaningless for this tutorial" - " - ignored", act.Data()); - - // Set the default number of events, if needed - nevt = (nevt < 0) ? 1000000 : nevt; - Printf("\nrunProof: running \"dataset\" with nevt= %lld\n", nevt); - - // Ask for registration of the dataset (the default is the TFileCollection is return - // without registration; the name of the TFileCollection is the name of the dataset - proof->SetParameter("SimpleNtuple.root","testNtuple"); - - // Do not plot the ntuple at this level - proof->SetParameter("PROOF_NTUPLE_DONT_PLOT", ""); - - // The selector string - sel.Form("%s/proof/ProofNtuple.C%s", tutorials.Data(), aMode.Data()); - // - // Run it for nevt times - TString xopt = aFeedback; if (!opt.IsNull()) xopt += TString::Format(" %s", opt.Data()); - proof->Process(sel.Data(), nevt, xopt); - - // The TFileCollection must be in the output - if (proof->GetOutputList()->FindObject("testNtuple")) { - - // Plot the ntuple via PROOF (example of drawing PROOF actions) - plotNtuple(proof, "testNtuple", "proof ntuple from dataset"); - - } else { - Printf("runProof: dataset 'testNtuple' not found in the output list"); - } - // Do not plot the ntuple at this level - proof->DeleteParameters("PROOF_NTUPLE_DONT_PLOT"); - proof->DeleteParameters("SimpleNtuple.root"); - - } else if (act == "friends") { - - // This is an example of analysis creating two data files on each node (the main tree - // and its friend) which are then processed as 'friends' to create the final plots. - // Selector used: ProofFriends, ProofAux - - if (first > 0) - // Meaningless for this tutorial - Printf("runProof: %s: warning concept of 'first' meaningless for this tutorial" - " - ignored", act.Data()); - - // Find out whether to use the same file or separate files - Bool_t sameFile = kFALSE; - while (args.Tokenize(tok, from, " ")) { - // Number of histos - if (tok == "samefile") { - sameFile = kTRUE; - break; - } - } - - // File generation: we use TPacketizerFile in here to create two files per node - TList *wrks = proof->GetListOfSlaveInfos(); - if (!wrks) { - Printf("runProof: could not get the list of information about the workers"); - return; - } - // Create the map - TString fntree; - TMap *files = new TMap; - files->SetName("PROOF_FilesToProcess"); - TIter nxwi(wrks); - TSlaveInfo *wi = 0; - while ((wi = (TSlaveInfo *) nxwi())) { - fntree.Form("tree_%s.root", wi->GetOrdinal()); - THashList *wrklist = (THashList *) files->GetValue(wi->GetName()); - if (!wrklist) { - wrklist = new THashList; - wrklist->SetName(wi->GetName()); - files->Add(new TObjString(wi->GetName()), wrklist); - } - wrklist->Add(new TObjString(fntree)); - } - - // Generate the files - proof->AddInput(files); - if (sameFile) { - Printf("runProof: friend tree stored in the same file as the main tree"); - proof->SetParameter("ProofAux_Action", "GenerateTreesSameFile"); - } else { - proof->SetParameter("ProofAux_Action", "GenerateTrees"); - } - // Default 1000 events - nevt = (nevt < 0) ? 10000 : nevt; - proof->SetParameter("ProofAux_NEvents", (Long64_t)nevt); - // Special Packetizer - proof->SetParameter("PROOF_Packetizer", "TPacketizerFile"); - // Now process - sel.Form("%s/proof/ProofAux.C%s", tutorials.Data(), aMode.Data()); - proof->Process(sel.Data(), 1); - // Remove the packetizer specifications - proof->DeleteParameters("PROOF_Packetizer"); - - // Print the lists and create the TDSet objects - TDSet *dset = new TDSet("Tmain", "Tmain"); - TDSet *dsetf = new TDSet("Tfrnd", "Tfrnd"); - if (proof->GetOutputList()) { - TIter nxo(proof->GetOutputList()); - TObject *o = 0; - TObjString *os = 0; - while ((o = nxo())) { - TList *l = dynamic_cast (o); - if (l && !strncmp(l->GetName(), "MainList-", 9)) { - TIter nxf(l); - while ((os = (TObjString *) nxf())) - dset->Add(os->GetName()); - } - } - nxo.Reset(); - while ((o = nxo())) { - TList *l = dynamic_cast (o); - if (l && !strncmp(l->GetName(), "FriendList-", 11)) { - TIter nxf(l); - while ((os = (TObjString *) nxf())) - dsetf->Add(os->GetName()); - } - } - } - // Process with friends - dset->AddFriend(dsetf, "friend"); - sel.Form("%s/proof/ProofFriends.C%s", tutorials.Data(), aMode.Data()); - TString xopt = aFeedback; if (!opt.IsNull()) xopt += TString::Format(" %s", opt.Data()); - dset->Process(sel, xopt); - // Clear the files created by this run - proof->ClearData(TProof::kUnregistered | TProof::kForceClear); - - } else if (act == "simplefile") { - - // ProofSimpleFile is an example of non-data driven analysis with merging - // via file and objcets saved in different directories; it creates and - // fills with random numbers two sets of a given number of histos - - if (first > 0) - // Meaningless for this tutorial - Printf("runProof: %s: warning concept of 'first' meaningless for this tutorial" - " - ignored", act.Data()); - - // Default 100000 events - nevt = (nevt < 0) ? 1000000 : nevt; - // Find out the number of histograms - TString aNhist; - while (args.Tokenize(tok, from, " ")) { - // Number of histos - if (tok.BeginsWith("nhist=")) { - aNhist = tok; - aNhist.ReplaceAll("nhist=",""); - if (!aNhist.IsDigit()) { - Printf("runProof: error parsing the 'nhist=' option (%s) - ignoring", tok.Data()); - aNhist = ""; - } - } - } - Int_t nhist = (aNhist.IsNull()) ? 16 : aNhist.Atoi(); - Printf("\nrunProof: running \"simplefile\" with nhist= %d and nevt= %lld\n", nhist, nevt); - - // The number of histograms is added as parameter in the input list - proof->SetParameter("ProofSimple_NHist", (Long_t)nhist); - - // Output file - TString fout(aOutFile); - if (fout.IsNull()) { - fout.Form("%s/SimpleFile.root", gSystem->WorkingDirectory()); - // Cleanup any existing instance of the output file - gSystem->Unlink(fout); - - if (!isProofLocal) { - // Setup a local basic xrootd to receive the file - Bool_t xrdok = kFALSE; - Int_t port = 9000; - while (port < 9010) { - if (checkXrootdAt(port) != 1) { - if (startXrootdAt(port, gSystem->WorkingDirectory(), kTRUE) == 0) { - xrdok = kTRUE; - break; - } - } - port++; - } - if (!xrdok) { - Printf("runProof: could not start basic xrootd on ports 9000-9009 - cannot continue"); - return; - } - fout.Insert(0, TString::Format("root://%s:%d/", TUrl(gSystem->HostName()).GetHostFQDN(), port)); - // Make a copy of the files on the master before merging - proof->AddInput(new TNamed("PROOF_OUTPUTFILE_LOCATION", "LOCAL")); - } - } - proof->AddInput(new TNamed("PROOF_OUTPUTFILE", fout.Data())); - - // The selector string - sel.Form("%s/proof/ProofSimpleFile.C%s", tutorials.Data(), aMode.Data()); - // - // Run it for nevt times - TString xopt = aFeedback; if (!opt.IsNull()) xopt += TString::Format(" %s", opt.Data()); - proof->Process(sel.Data(), nevt, xopt); - - } else if (act == "stdvec") { - - // This is an example of runnign a TSelector using standard vectors - // Selector used: ProofStdVect - - if (first > 0) - // Meaningless for this tutorial - Printf("runProof: %s: warning concept of 'first' meaningless for this tutorial" - " - ignored", act.Data()); - - // Set the default number of events, if needed - nevt = (nevt < 0) ? 50000 * proof->GetParallel() : nevt; - Printf("\nrunProof: running \"stdvec\" with nevt= %lld\n", nevt); - - // The selector string - sel.Form("%s/proof/ProofStdVect.C%s", tutorials.Data(), aMode.Data()); - - TString xopt; - // Create the dataset 'TestStdVect' with 'nevt' events - xopt.Form("%s %s create", aFeedback.Data(), opt.Data()); - proof->Process(sel.Data(), nevt, xopt); - - // The dataset must have been registered - if (proof->ExistsDataSet("TestStdVect")) { - - // Use dataset 'TestStdVect' - xopt.Form("%s %s", aFeedback.Data(), opt.Data()); - proof->Process("TestStdVect", sel.Data(), xopt); - - } else { - Printf("runProof: dataset 'TestStdVect' not available!"); - } - - } else { - // Do not know what to run - Printf("runProof: unknown tutorial: %s", what); - } - - // Save the performance tree - if (makePerfTree) { - SavePerfTree(proof, aPerfTree.Data()); - // Cleanup parameters - gProof->DeleteParameters("PROOF_StatsHist"); - gProof->DeleteParameters("PROOF_StatsTrace"); - gProof->DeleteParameters("PROOF_SlaveStatsTrace"); - } -} - -//_______________________________________________________________________________________ -void plotNtuple(TProof *p, const char *ds, const char *ntptitle) -{ - // Make some plots from the ntuple 'ntp' via PROOF - - // - // Create a canvas, with 2 pads - // - TCanvas *c1 = new TCanvas(Form("cv-%s", ds), ntptitle,800,10,700,780); - c1->Divide(1,2); - TPad *pad1 = (TPad *) c1->GetPad(1); - TPad *pad2 = (TPad *) c1->GetPad(2); - // - // Display a function of one ntuple column imposing a condition - // on another column. - pad1->cd(); - pad1->SetGrid(); - pad1->SetLogy(); - pad1->GetFrame()->SetFillColor(15); - - p->SetParameter("PROOF_LineColor", (Int_t)1); - p->SetParameter("PROOF_FillStyle", (Int_t)1001); - p->SetParameter("PROOF_FillColor", (Int_t)45); - p->DrawSelect(ds, "3*px+2","px**2+py**2>1"); - p->SetParameter("PROOF_FillColor", (Int_t)38); - p->DrawSelect(ds, "2*px+2","pz>2","same"); - p->SetParameter("PROOF_FillColor", (Int_t)5); - p->DrawSelect(ds, "1.3*px+2","(px^2+py^2>4) && py>0","same"); - pad1->RedrawAxis(); - - // - // Display a 3-D scatter plot of 3 columns. Superimpose a different selection. - pad2->cd(); - p->DrawSelect(ds, "pz:py:px","(pz<10 && pz>6)+(pz<4 && pz>3)"); - p->SetParameter("PROOF_MarkerColor", (Int_t)4); - p->DrawSelect(ds, "pz:py:px","pz<6 && pz>4","same"); - p->SetParameter("PROOF_MarkerColor", (Int_t)5); - p->DrawSelect(ds, "pz:py:px","pz<4 && pz>3","same"); - TPaveText *l2 = new TPaveText(0.,0.6,0.9,0.95); - l2->SetFillColor(42); - l2->SetTextAlign(12); - l2->AddText("You can interactively rotate this view in 2 ways:"); - l2->AddText(" - With the RotateCube in clicking in this pad"); - l2->AddText(" - Selecting View with x3d in the View menu"); - l2->Draw(); - - // Final update - c1->cd(); - c1->Update(); - - // Clear parameters used for the plots - p->DeleteParameters("PROOF_*Color"); - p->DeleteParameters("PROOF_*Style"); -} - -//______________________________________________________________________________ -void SavePerfTree(TProof *proof, const char *fn) -{ - // Save PROOF timing information from TPerfStats to file 'fn' - - if (!proof) { - Printf("PROOF must be run to save output performance information");; - return; - } - if (!proof->GetOutputList() || proof->GetOutputList()->GetSize() <= 0) { - Printf("PROOF outputlist undefined or empty");; - return; - } - - TFile f(fn, "RECREATE"); - if (f.IsZombie()) { - Printf("ERROR: could not open file '%s' for writing", fn);; - } else { - f.cd(); - TIter nxo(proof->GetOutputList()); - TObject* obj = 0; - while ((obj = nxo())) { - TString objname(obj->GetName()); - if (objname.BeginsWith("PROOF_")) { - // Must list the objects since other PROOF_ objects exist - // besides timing objects - if (objname == "PROOF_PerfStats" || - objname == "PROOF_PacketsHist" || - objname == "PROOF_EventsHist" || - objname == "PROOF_NodeHist" || - objname == "PROOF_LatencyHist" || - objname == "PROOF_ProcTimeHist" || - objname == "PROOF_CpuTimeHist") - obj->Write(); - } - } - f.Close(); - } - -} diff --git a/tutorials/roofit/roostats/HybridInstructional.C b/tutorials/roofit/roostats/HybridInstructional.C index 72f7b6d98baf9..3b1e674a2b847 100644 --- a/tutorials/roofit/roostats/HybridInstructional.C +++ b/tutorials/roofit/roostats/HybridInstructional.C @@ -492,7 +492,7 @@ void HybridInstructional(int ntoys = 6000) c->SaveAs("zbi.pdf"); // ----------------------------------------- - // OUTPUT W/O PROOF (2.66 GHz Intel Core i7) + // OUTPUT (2.66 GHz Intel Core i7) // ========================================= // ----------------------------------------- @@ -545,36 +545,6 @@ void HybridInstructional(int ntoys = 6000) // - CL_s: 0.511341 +/- 0.0158183 // Real time 0:05:06, CP time 306.330 - // --------------------------------------------------------- - // OUTPUT w/ PROOF (2.66 GHz Intel Core i7, 4 virtual cores) - // ========================================================= - - // ----------------------------------------- - // Part 5 - // Results HybridCalculator_result: - // - Null p-value = 0.00075 +/- 0.000173124 - // - Significance = 3.17468 sigma - // - Number of S+B toys: 1000 - // - Number of B toys: 20000 - // - Test statistic evaluated on data: 10.8198 - // - CL_b: 0.99925 +/- 0.000193577 - // - CL_s+b: 0.454 +/- 0.0157443 - // - CL_s: 0.454341 +/- 0.0157564 - // Real time 0:00:16, CP time 0.990 - - // ----------------------------------------- - // Part 6 - // Results HybridCalculator_result: - // - Null p-value = 0.0007 +/- 0.000152699 - // - Significance = 3.19465 sigma - // - Number of S+B toys: 1000 - // - Number of B toys: 30000 - // - Test statistic evaluated on data: 5.03388 - // - CL_b: 0.9993 +/- 0.000152699 - // - CL_s+b: 0.518 +/- 0.0158011 - // - CL_s: 0.518363 +/- 0.0158124 - // Real time 0:01:25, CP time 0.580 - // ---------------------------------- // Comparison // ---------------------------------- diff --git a/tutorials/roofit/roostats/HybridInstructional.py b/tutorials/roofit/roostats/HybridInstructional.py index e801e9b1bf796..a22c3f7e548df 100644 --- a/tutorials/roofit/roostats/HybridInstructional.py +++ b/tutorials/roofit/roostats/HybridInstructional.py @@ -492,7 +492,7 @@ class BinCountTestStat : public TestStatistic { c.SaveAs("zbi.pdf") # ----------------------------------------- -# OUTPUT W/O PROOF (2.66 GHz Intel Core i7) +# OUTPUT (2.66 GHz Intel Core i7) # ========================================= # ----------------------------------------- @@ -545,36 +545,6 @@ class BinCountTestStat : public TestStatistic { # - CL_s: 0.511341 +/- 0.0158183 # Real time 0:05:06, CP time 306.330 -# --------------------------------------------------------- -# OUTPUT w/ PROOF (2.66 GHz Intel Core i7, 4 virtual cores) -# ========================================================= - -# ----------------------------------------- -# Part 5 -# Results HybridCalculator_result: -# - Null p-value = 0.00075 +/- 0.000173124 -# - Significance = 3.17468 sigma -# - Number of S+B toys: 1000 -# - Number of B toys: 20000 -# - Test statistic evaluated on data: 10.8198 -# - CL_b: 0.99925 +/- 0.000193577 -# - CL_s+b: 0.454 +/- 0.0157443 -# - CL_s: 0.454341 +/- 0.0157564 -# Real time 0:00:16, CP time 0.990 - -# ----------------------------------------- -# Part 6 -# Results HybridCalculator_result: -# - Null p-value = 0.0007 +/- 0.000152699 -# - Significance = 3.19465 sigma -# - Number of S+B toys: 1000 -# - Number of B toys: 30000 -# - Test statistic evaluated on data: 5.03388 -# - CL_b: 0.9993 +/- 0.000152699 -# - CL_s+b: 0.518 +/- 0.0158011 -# - CL_s: 0.518363 +/- 0.0158124 -# Real time 0:01:25, CP time 0.580 - # ---------------------------------- # Comparison # ---------------------------------- diff --git a/tutorials/roofit/roostats/HybridStandardForm.C b/tutorials/roofit/roostats/HybridStandardForm.C index c6fd9aa002776..7838dbc339435 100644 --- a/tutorials/roofit/roostats/HybridStandardForm.C +++ b/tutorials/roofit/roostats/HybridStandardForm.C @@ -36,7 +36,6 @@ /// - demonstrates usage of different test statistics /// - explains subtle choices in the prior used for hybrid methods /// - demonstrates usage of different priors for the nuisance parameters -/// - demonstrates usage of PROOF /// /// The basic setup here is that a main measurement has observed x events with an /// expectation of s+b. One can choose an ad hoc prior for the uncertainty on b, @@ -154,8 +153,7 @@ void HybridStandardForm(int ntoys = 6000) // 5. RooStats ToyMC with an equivalent test statistic // 6. RooStats ToyMC with simultaneous control & main measurement - // Part 4 takes ~4 min without PROOF. - // Part 5 takes about ~2 min with PROOF on 4 cores. + // Part 4 takes ~4 min. // Of course, everything looks nicer with more toys, which takes longer. TStopwatch t; @@ -389,7 +387,7 @@ void HybridStandardForm(int ntoys = 6000) return; // so standard tutorial runs faster //--------------------------------------------- - // OUTPUT W/O PROOF (2.66 GHz Intel Core i7) + // OUTPUT (2.66 GHz Intel Core i7) // ============================================ // ----------------------------------------- @@ -409,21 +407,6 @@ void HybridStandardForm(int ntoys = 6000) // - CL_s: 0.495512 +/- 0.0158272 // Real time 0:04:43, CP time 283.780 - // With PROOF - // ----------------------------------------- - // Part 5 - - // Results HybridCalculator_result: - // - Null p-value = 0.00105 +/- 0.000206022 - // - Significance = 3.07571 sigma - // - Number of S+B toys: 1000 - // - Number of B toys: 20000 - // - Test statistic evaluated on data: 10.8198 - // - CL_b: 0.99895 +/- 0.000229008 - // - CL_s+b: 0.491 +/- 0.0158088 - // - CL_s: 0.491516 +/- 0.0158258 - // Real time 0:02:22, CP time 0.990 - //------------------------------------------------------- // Comparison //------------------------------------------------------- diff --git a/tutorials/roofit/roostats/HybridStandardForm.py b/tutorials/roofit/roostats/HybridStandardForm.py index 5956ecdd7f6a8..b3c47eebcf5d2 100644 --- a/tutorials/roofit/roostats/HybridStandardForm.py +++ b/tutorials/roofit/roostats/HybridStandardForm.py @@ -366,7 +366,7 @@ class BinCountTestStat : public TestStatistic { quit() # so standard tutorial runs faster # --------------------------------------------- -# OUTPUT W/O PROOF (2.66 GHz Intel Core i7) +# OUTPUT (2.66 GHz Intel Core i7) # ============================================ # ----------------------------------------- @@ -386,21 +386,6 @@ class BinCountTestStat : public TestStatistic { # - CL_s: 0.495512 +/- 0.0158272 # Real time 0:04:43, CP time 283.780 -# With PROOF -# ----------------------------------------- -# Part 5 - -# Results HybridCalculator_result: -# - Null p-value = 0.00105 +/- 0.000206022 -# - Significance = 3.07571 sigma -# - Number of S+B toys: 1000 -# - Number of B toys: 20000 -# - Test statistic evaluated on data: 10.8198 -# - CL_b: 0.99895 +/- 0.000229008 -# - CL_s+b: 0.491 +/- 0.0158088 -# - CL_s: 0.491516 +/- 0.0158258 -# Real time 0:02:22, CP time 0.990 - # ------------------------------------------------------- # Comparison # ------------------------------------------------------- diff --git a/tutorials/roofit/roostats/OneSidedFrequentistUpperLimitWithBands.C b/tutorials/roofit/roostats/OneSidedFrequentistUpperLimitWithBands.C index 8905a46379401..460ff8389799d 100644 --- a/tutorials/roofit/roostats/OneSidedFrequentistUpperLimitWithBands.C +++ b/tutorials/roofit/roostats/OneSidedFrequentistUpperLimitWithBands.C @@ -52,13 +52,6 @@ /// Building the confidence belt can be computationally expensive. Once it is built, /// one could save it to a file and use it in a separate step. /// -/// We can use PROOF to speed things along in parallel, however, -/// the test statistic has to be installed on the workers -/// so either turn off PROOF or include the modified test statistic -/// in your `$ROOTSYS/roofit/roostats/inc` directory, -/// add the additional line to the LinkDef.h file, -/// and recompile root. -/// /// Note, if you have a boundary on the parameter of interest (eg. cross-section) /// the threshold on the one-sided test statistic starts off very small because we /// are only including downward fluctuations. You can see the threshold in these printouts: diff --git a/tutorials/roofit/roostats/TwoSidedFrequentistUpperLimitWithBands.C b/tutorials/roofit/roostats/TwoSidedFrequentistUpperLimitWithBands.C index c2fcb27b28ed7..a31929d5bb1fa 100644 --- a/tutorials/roofit/roostats/TwoSidedFrequentistUpperLimitWithBands.C +++ b/tutorials/roofit/roostats/TwoSidedFrequentistUpperLimitWithBands.C @@ -59,13 +59,6 @@ /// Building the confidence belt can be computationally expensive. /// Once it is built, one could save it to a file and use it in a separate step. /// -/// We can use PROOF to speed things along in parallel, however, -/// the test statistic has to be installed on the workers -/// so either turn off PROOF or include the modified test statistic -/// in your $ROOTSYS/roofit/roostats/inc directory, -/// add the additional line to the LinkDef.h file, -/// and recompile root. -/// /// Note, if you have a boundary on the parameter of interest (eg. cross-section) /// the threshold on the two-sided test statistic starts off at moderate values and plateaus. /// diff --git a/tutorials/roofit/roostats/TwoSidedFrequentistUpperLimitWithBands.py b/tutorials/roofit/roostats/TwoSidedFrequentistUpperLimitWithBands.py index 69fe303abb3f2..990edce714478 100644 --- a/tutorials/roofit/roostats/TwoSidedFrequentistUpperLimitWithBands.py +++ b/tutorials/roofit/roostats/TwoSidedFrequentistUpperLimitWithBands.py @@ -59,13 +59,6 @@ # Building the confidence belt can be computationally expensive. # Once it is built, one could save it to a file and use it in a separate step. # -# We can use PROOF to speed things along in parallel, however, -# the test statistic has to be installed on the workers -# so either turn off PROOF or include the modified test statistic -# in your $ROOTSYS/roofit/roostats/inc directory, -# add the additional line to the LinkDef.h file, -# and recompile root. -# # Note, if you have a boundary on the parameter of interest (eg. cross-section) # the threshold on the two-sided test statistic starts off at moderate values and plateaus. #