CI Failure (key symptom) in ClusterConfigLegacyDefaultTest.test_removal_of_legacy_default_defaulted

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/16436

Module: rptest.tests.cluster_config_test
Class: ClusterConfigLegacyDefaultTest
Method: test_removal_of_legacy_default_defaulted
Arguments: {
    "wipe_cache": false
}

test_id:    ClusterConfigLegacyDefaultTest.test_removal_of_legacy_default_defaulted
status:     FAIL
run time:   73.065 seconds

RemoteCommandError({'ssh_config': {'host': 'ip-172-31-21-157', 'hostname': '172.31.21.157', 'user': 'root', 'port': 22, 'password': None, 'identityfile': '/home/ubuntu/.ssh/id_rsa'}, 'hostname': 'ip-172-31-21-157', 'ssh_hostname': '172.31.21.157', 'user': 'root', 'externally_routable_ip': '52.41.27.153', '_logger': <Logger rptest.tests.cluster_config_test.ClusterConfigLegacyDefaultTest.test_removal_of_legacy_default_defaulted.wipe_cache=False-979 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7c8cc6f67cd0>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7c8cc7189ea0>, '_custom_ssh_exception_checks': None}, 'curl -fsSL https://vectorized-public.s3.us-west-2.amazonaws.com/releases/redpanda/23.2.29/redpanda-23.2.29-amd64.tar.gz --retry 3 --retry-connrefused --retry-delay 2 --create-dir -o /opt/redpanda_installs/v23.2.29/redpanda.tar.gz && gunzip -c /opt/redpanda_installs/v23.2.29/redpanda.tar.gz | tar -xf - -C /opt/redpanda_installs/v23.2.29 && rm /opt/redpanda_installs/v23.2.29/redpanda.tar.gz', 2, b'')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 105, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/cluster_config_test.py", line 2146, in test_removal_of_legacy_default_defaulted
    self._upgrade(wipe_cache, v)
  File "/home/ubuntu/redpanda/tests/rptest/tests/cluster_config_test.py", line 2070, in _upgrade
    self.installer.install(self.redpanda.nodes, version)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 609, in install
    self._install_unlocked(nodes, install_target)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 658, in _install_unlocked
    raise e
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 638, in _install_unlocked
    self.wait_for_async_ssh(self._redpanda.logger,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 165, in wait_for_async_ssh
    for l in ssh_out_per_node[node]:
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 687, in next
    return next(self.iter_obj)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 354, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-21-157: Command 'curl -fsSL https://vectorized-public.s3.us-west-2.amazonaws.com/releases/redpanda/23.2.29/redpanda-23.2.29-amd64.tar.gz --retry 3 --retry-connrefused --retry-delay 2 --create-dir -o /opt/redpanda_installs/v23.2.29/redpanda.tar.gz && gunzip -c /opt/redpanda_installs/v23.2.29/redpanda.tar.gz | tar -xf - -C /opt/redpanda_installs/v23.2.29 && rm /opt/redpanda_installs/v23.2.29/redpanda.tar.gz' returned non-zero exit status 2.

JIRA Link: CORE-6863

[piyushredpanda]

Closing older-bot-filed CI issues as we transition to a more reliable system.