crab preparelocal fails after crab submit --dryrun

[belforte]

with current production version v3.210825
But I think this works after the cleanup done in new master.
So this issue is mostly to remind us to check.
It is quite possible that preparelocal after dryrun is broken since the transition to S3 for crab cache and nobody noticed !

belforte@lxplus726/DEBUG> crab submit --dryrun 
Will use CRAB configuration file crabConfig.py
Importing CMSSW configuration demoanalyzer.py
Finished importing CMSSW configuration demoanalyzer.py
Sending the request to the server at cmsweb.cern.ch
Success: Your task has been delivered to the prod CRAB3 server.
Task name: 211118_171853:belforte_crab_20211118_181835
Project dir: ./crab_20211118_181835
Waiting for task to be processed
Checking task status
Task status: NEW
Please wait...
Task status: QUEUED
Please wait...
Task status: QUEUED
Please wait...
Task status: UPLOADED

Created temporary directory for dry run sandbox in /tmp/belforte/tmpXfoBmX
Executing test, please wait...

Using LumiBased splitting
Task consists of 50 jobs to process 50 lumis
The longest job will process 1 lumis, with an estimated processing time of 1 minutes
The average job will process 1 lumis, with an estimated processing time of 1 minutes
The shortest job will process 1 lumis, with an estimated processing time of 1 minutes
The estimated memory requirement is 718 MB

Timing quantities given below are ESTIMATES. Keep in mind that external factors
such as transient file-access delays can reduce estimate reliability.

For ~480 minute jobs, use:
Data.unitsPerJob = 321
You will need to submit a new task

Dry run requested: task paused
To continue processing, use 'crab proceed'

Log file is /afs/cern.ch/work/b/belforte/CRAB3/TC3/DEBUG/crab_20211118_181835/crab.log
belforte@lxplus726/DEBUG> crab status -d crab_20211118_181835
CRAB project directory:		/afs/cern.ch/work/b/belforte/CRAB3/TC3/DEBUG/crab_20211118_181835
Task name:			211118_171853:belforte_crab_20211118_181835
Grid scheduler - Task Worker:	[email protected] - crab-prod-tw01
Status on the CRAB server:	UPLOADED
Task URL to use for HELP:	https://cmsweb.cern.ch/crabserver/ui/task/211118_171853%3Abelforte_crab_20211118_181835
Dashboard monitoring URL:	https://monit-grafana.cern.ch/d/cmsTMDetail/cms-task-monitoring-task-view?orgId=11&var-user=belforte&var-task=211118_171853%3Abelforte_crab_20211118_181835&from=1637252333000&to=now
Log file is /afs/cern.ch/work/b/belforte/CRAB3/TC3/DEBUG/crab_20211118_181835/crab.log
belforte@lxplus726/DEBUG> crab preparelocal -d crab_20211118_181835
Getting input files into tmp dir /tmp/belforte/tmpWtrKJk
ERROR: RuntimeError: Reading https://cmsweb.cern.ch:8443/S3/crabcache_prod/logfile?name=dry-run-sandbox.tar.gz failed with code 404

	Please email [email protected] for support with the crab.log file or crab.log URL.
	Client Version: v3.210825
	Please use 'crab uploadlog' to upload the log file /afs/cern.ch/work/b/belforte/CRAB3/TC3/DEBUG/crab_20211118_181835/crab.log to the CRAB cache.
belforte@lxplus726/DEBUG> crab --version
CRAB client v3.210825
belforte@lxplus726/DEBUG> 

[belforte]

hmmm...
with latest client v3.211112 and preprod server I get

belforte@lxplus726/TC3> crab preparelocal 
Getting input files into tmp dir /tmp/belforte/tmpTWAay_
ERROR: TypeError: 'str' object is not callable

from call to old crabcache !!

Traceback (most recent call last):
  File "/cvmfs/cms-ib.cern.ch/nweek-02707/share/cms/crab-dev/v3.211112.00/bin/crab.py", line 149, in <module>
    client()
  File "/cvmfs/cms-ib.cern.ch/nweek-02707/share/cms/crab-dev/v3.211112.00/bin/crab.py", line 136, in __call__
    self.cmd()
  File "/cvmfs/cms-ib.cern.ch/nweek-02707/share/cms/crab-dev/v3.211112.00/lib/CRABClient/Commands/preparelocal.py", line 38, in __call__
    self.getInputFiles()
  File "/cvmfs/cms-ib.cern.ch/nweek-02707/share/cms/crab-dev/v3.211112.00/lib/CRABClient/Commands/preparelocal.py", line 72, in getInputFiles
    ufc = CRABClient.Emulator.getEmulator('ufc')({'endpoint' : filecacheurl, "pycurl": True})
TypeError: 'str' object is not callable
ERROR 2021-11-18 17:33:52.161 UTC: 	 ERROR: TypeError: 'str' object is not callable
ERROR 2021-11-18 17:33:52.161 UTC: 	 

[belforte]

the error is not so surprising given the code

CRABClient/src/python/CRABClient/Commands/preparelocal.py

Lines 70 to 76 in c0d33e0

	if status == 'UPLOADED':
	filecacheurl = getColumn(crabDBInfo, 'tm_cache_url')
	ufc = CRABClient.Emulator.getEmulator('ufc')({'endpoint' : filecacheurl, "pycurl": True})
	self.logger.debug("Downloading and extracting 'dry-run-sandbox.tar.gz' from %s" % filecacheurl)
	ufc.downloadLog('dry-run-sandbox.tar.gz', output=os.path.join(os.getcwd(), 'dry-run-sandbox.tar.gz'))
	with tarfile.open('dry-run-sandbox.tar.gz') as tf:
	tf.extractall()

odd that we had not noticed yet.
There is clearly a strong relationship with dmwm/CRABServer#6544

Maybe time to tackle that and do thing neatly and properly.
I am inclined to leave it broken until then. I.e. simply remove the code pointing to old cache so there is no odd err, but just a "crab preparelocal currently only works for SUBMITTED tasks".

[belforte]

the above commit will avid the error and print out a clean "does not work" message. After it is merged I will reopen as a longer time issue for a proper solution in the context of dmwm/CRABServer#6544

[belforte]

preparelocal is now currently disabled after submit --dry. Need to code in a proper solution after we stabilize the sandboxes uploading in S3 as per dmwm/CRABServer#6544