reworking toolbox dumping logic

[upodroid]

This PR tweaks the instance dumping logic:

1. Prioritises instances that fail to join the cluster
2. Use the control plane instances as an SSH bastion if a bastion doesn't exist and the control plane instances have a public IP

We need this for troubleshooting infra flakes with kops

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hakman for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hakman]

Stores ssh logs by instance names instead of IPs

Not sure this makes things better or worse. IPs make it obvious that the node has not registered.

[upodroid]

Stores ssh logs by instance names instead of IPs

Not sure this makes things better or worse. IPs make it obvious that the node has not registered.

kops validate logs failures like this:

VALIDATION ERRORS
KIND	NAME																MESSAGE
Machine	https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-027/zones/us-west1-b/instances/nodes-us-west1-b-dksv	machine "https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-027/zones/us-west1-b/instances/nodes-us-west1-b-dksv" has not yet joined cluster

looking by nodes should make it easier

[hakman]

Stores ssh logs by instance names instead of IPs

Not sure this makes things better or worse. IPs make it obvious that the node has not registered.

kops validate logs failures like this:

VALIDATION ERRORS
KIND	NAME																MESSAGE
Machine	https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-027/zones/us-west1-b/instances/nodes-us-west1-b-dksv	machine "https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-027/zones/us-west1-b/instances/nodes-us-west1-b-dksv" has not yet joined cluster

looking by nodes should make it easier

The usual workflow is to just look at nodes why they failed. It is not really useful to know the ID of the failed one if you just know it will be one with IP address.

[upodroid]

I'll change it back to ip

[upodroid]

/test presubmit-kops-gce-small-scale-ipalias-using-cl2

[upodroid]

ready to be merged, it's working correctly

[upodroid]

/retest