Changes required for adoption of the cmsweb-preprod cluster

[amaltaro]

Impact of the new feature
WMCore in general

Is your feature request related to a problem? Please describe.
We have recently decided to have a 3rd cluster for integration tests, called cmsweb-preprod. A short summary is:
0) cmsweb-testX clusters: a developer's playground environment to test, patch, change, and do whatever we need when developing for central services.

1. cmsweb-testbed is a level up from the testX clusters. Meant for initial integration tests with more stable code
2. cmsweb-preprod is yet another level up from testbed cluster. Meant for pre-production code, so code that we are actually preparing to push to production. It is meant to be stable!
3. cmsweb-prod/cmsweb is the production environment.

Having said that, we are moving towards the adoption of cmsweb-preprod in CMS (and WM). So this ticket is meant to track all the relevant changes that we need to either a) revise or b) update the codebase for.

Note that some cmsweb-preprod details are still under discussion, e.g. dbs url to be used and separation of CouchDB (and MongoDB) backends.

Describe the solution you'd like

1. https://github.com/dmwm/WMCore/blob/2748054/src/python/WMCore/WorkQueue/DataLocationMapper.py and

   WMCore/src/python/WMCore/WorkQueue/Policy/Start/StartPolicyInterface.py

   Line 285 in 2748054

   pileUpinstance = '-testbed' if 'cmsweb-testbed' in dbsUrl else '-prod'

   using a code like:

                pileupInstance = '-testbed' if 'cmsweb-testbed' in dbsUrl else '-prod'

2. Create new MongoDB cluster (or database under cmsweb-testbed)

3. Create new CouchDB cluster

4. Considerations for the data stored in the database? I would suggest moving cmsweb-testbed data over to cmsweb-preprod and start with a fresh database in cmsweb-testbed

5. testbed agents will have to move to cmsweb-preprod

In terms of WMCore codebase, I don't think there is anything else to consider. We might want to revisit the workflow json templates though.

Describe alternatives you've considered
None

Additional context
None