Add multi-container support

[zxhe-sean]

Description

Adding a flag --multi-container for supporting multi-container workloads.

With this flag enabled, running workload create will duplicate the container part in the final yaml file so that the resulting workload will be running in two containers running on one host.

Issue

b/459820949

Testing

https://cloudlogging.app.goo.gl/F81QQQ4Ca9Gew7hs8

[github-actions]

🤖 Hi @zxhe-sean, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[scaliby]

@zxhe-sean why do we need this change? Could you cover it with tests?

[jamOne-]

Thanks for the changes and the unit tests!
I left a couple of comments, PTAL.

[jamOne-]

Thanks for the changes, looks good! Left one more comment

[jamOne-]

@Obliviour @FIoannides PTAL

[Obliviour]

So this PR provides us the ability from the workload to enable two container workloads.

There are two other dimensions that need to be covered that I think we need xpk team's help in:

1. In order for it to be numa aware, the gke nodepool topology manager needs to be configured with best effort. The default is none so when the two containers are scheduled, it won't be numa aware. GKE has an improvement on the way to make best effort the default so that this problem is solved. But in the short term, we need to create the nodepools with an additional topology manager set to best effort.

2. When nap is used, we need the nodepool to created with the same topology manager through nap.

[jamOne-]

Thanks for taking a look @Obliviour!

1. For non-NAP case it should be rather simple. I have a clarifying question though: should we set the topology manager for every non-NAP nodepool, or just the ones that have parallel_containers > 1 (tpu7x currently)?
2. NAP case. Do you know how could we set the topology manager for NAP nodepools? IIUC the current code, we don't create nodepools explicitly for NAP, we just set the proper annotations on the JobSet and the nodepools are then created automatically.