[gitlab] 'Error loading key "/dev/fd/63": invalid format' error

[130s]

Issue

On a private repo I keep getting this error (which has prevented me from switching to industrial_ci on Gitlab)

$ git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
$ .industrial_ci/gitlab.sh ROS_DISTRO=melodic
Agent pid 47
Error loading key "/dev/fd/63": invalid format
could not add ssh key
Cleaning up file based variables 00:02
ERROR: Job failed: exit code 1

Attempted but doesn't seem to be working

• Got a hint stackoverflow.com#55523151. But I get the same result

  --- a/gitlab.sh
  +++ b/gitlab.sh
  @@ -35,7 +35,7 @@ if [ -n "$SSH_PRIVATE_KEY" ]; then
     # shellcheck disable=SC2046
     eval $(ssh-agent -s)
     # add key to agent
  -  ssh-add <(echo "$SSH_PRIVATE_KEY") || { res=$?; echo "could not add ssh key"; exit $res; }
  +  ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d) || { res=$?; echo "could not add ssh key"; exit $res; }
  

Env

• gitlab.com
• SSH_PRIVATE_KEY is supplied by using the repo's feature (Settings --> CI/CD --> Variables)

[mathias-luedtke]

I believe it's the same issue as https://gitlab.com/gitlab-examples/ssh-private-key/-/issues/1#note_48526556
And there they suggest echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add

[130s]

Thanks. I stopped seeing the exact error message above but I still see the following with plusone-robotics/industrial_ci/fix-gitlab-sshkey.

:
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ .industrial_ci/gitlab.sh ROS_DISTRO=melodic
Agent pid 40
Error loading key "(stdin)": invalid format
could not add ssh key
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

[mathias-luedtke]

So there might be as well something with actual key data.
Without a public test cases, it hard to tell what's going wrong.

[130s]

Tested on a private repo on my personal org (which I don't mind giving an acecss), unlike the OP that is on a corporate org, I can't repro this issue. So I assume there's a difference b/w personal org and corporate org that ICI's Gitlab tool isn't yet handling.

Detail

Same as OP, SSH_PRIVATE_KEY is supplied by using the repo's feature (Settings --> CI/CD --> Variables).

:
$ git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
$ .industrial_ci/gitlab.sh ROS_DISTRO=melodic
Agent pid 35
Identity added: /dev/fd/63 (/dev/fd/63)
Running test 'source_tests'
pull_docker_image 00:03
'pull_docker_image' returned with code '0' after 0 min 3 sec
$ docker create --init --env-file /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src/isolation/docker.env --rm -v /builds/130s/sample_private_repo:/builds/130s/sample_private_repo:ro -e TARGET_REPO_PATH=/builds/130s/sample_private_repo -v /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src:/builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src:ro -e ICI_SRC_PATH=/builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src -v /tmp/ssh-XXXXXXoaMFJi/agent.34:/tmp/ssh-XXXXXXoaMFJi/agent.34:rw -e SSH_AUTH_SOCK=/tmp/ssh-XXXXXXoaMFJi/agent.34 -t --entrypoint  -w /builds/130s/sample_private_repo ubuntu:bionic /bin/bash /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src/run.sh source_tests run_source_tests
Copy credentials: /root/.ssh
$ docker start -a b49c0697fe94eb39c3a60f16f1a699791505c1f31d8aceff328d736c151f374b
:
Job succeeded

.gitlab-ci.yml

image: docker:git
services:
  - docker:dind
before_script:
  - apk add --update bash coreutils tar
  - git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
build:
  script: .industrial_ci/gitlab.sh ROS_DISTRO=melodic

Interestingly, btw, I tested deleting SSH_PRIVATE_KEY on Gitlab from the same repo, which I'd expect ICI to fail with the same/similar error in this ticket as ssh key is now unavailable, but the job still went on and passed.

$ git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
$ .industrial_ci/gitlab.sh ROS_DISTRO=melodic
Running test 'source_tests'
pull_docker_image 00:03
'pull_docker_image' returned with code '0' after 0 min 3 sec
$ docker create --init --env-file /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src/isolation/docker.env --rm -v /builds/130s/sample_private_repo:/builds/130s/sample_private_repo:ro -e TARGET_REPO_PATH=/builds/130s/sample_private_repo -v /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src:/builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src:ro -e ICI_SRC_PATH=/builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src -t --entrypoint  -w /builds/130s/sample_private_repo ubuntu:bionic /bin/bash /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src/run.sh source_tests run_source_tests
$ docker start -a 1d1a767849b582e12d0df5cb1534796933b7999d49e10c90b973fd2c89a4d4ef
init
'init' returned with code '0' after 0 min 34 sec
catkin_tools_setup
:

[130s]

Content of SSH_PRIVATE_KEY is questionable. But in our org SSH_PRIVATE_KEY is maintained at the top org level and I don't have an access to it so I'm wondering how to check its format etc.

With ICI removing tr -d plusone-robotics@5c29a28, the err msg changed from #756 (comment) (no longer seeing Error loading key "/dev/fd/63": invalid format).

https://gitlab.com/foo/baa/-/jobs/1798432233#L127

$ .industrial_ci/gitlab.sh ROS_DISTRO=melodic
Agent pid 35
Error loading key "(stdin)": invalid format
could not add ssh key

[130s]

So I assume there's a difference b/w personal org and corporate org that ICI's Gitlab tool isn't yet handling.

Spoke too soon. I can't say that yet. I have to suspect the key's validity as well as Mathias pointed out.

[130s]

This issue is no longer happening on my repos on gitlab.com although I don't think I've done anything specific (I have no idea if admins of our org has done anything to the admin env vars). Closing.

[130s]

Re-opening to continue discussion.

I still see the same issue using master of ICI. In order to get this issue passed I'm using our branch #806. So I'm still missing something.