PEM data was not found in buffer with valid RSA key in v1.31.0

[ratdude07]

Description

The OCI Cloud Controller Manager (CCM) v1.31.0 fails to initialize with "PEM data was not found in buffer" despite a valid RSA private key being provided at the specified path. The key is readable by the CCM process (confirmed via pre-start debug), works with the OCI CLI, and matches the fingerprint in the config. Tried with standard /etc/oci in-container mount and custom /oci-config mount. Tried regular PEM key and converted RSA key, always the same error from the cloud controller container. Tried using multiple different versions of CCM.

Environment

• CCM Version: ghcr.io/oracle/cloud-provider-oci:v1.31.0
• Kubernetes: v1.31.0
• OCI Region: us-ashburn-1
• Date: March 21, 2025

Steps to Reproduce

1. Generate an RSA key:
   bash
   openssl rsa -in /root/.oci/richard.tylka.clean.pem -out /root/.oci/richard.tylka.rsa.pem -traditional
   kubectl create secret generic oci-credential
   --from-file=cloud-provider.yaml
   --from-file=key.pem=/root/.oci/richard.tylka.rsa.pem
   -n kube-system

cloud-provider.yaml:

auth:
region: us-ashburn-1
tenancy: ocid1.tenancy.oc1..aaaaaaaa6bpzdsduwbti65kqm7gvubwvrspi7626cxona2t6fgguxn3ftvxa
user: ocid1.user.oc1..aaaaaaaaahcwxw3yxfayfxm666prjwi4667zhvnh4ddz5sdzclar3tjpww5q
key: /oci-config/key.pem
fingerprint: 76:75:60:81:15:20:95:b6:3a:cb:29:c6:38:ad:83:76
compartment: ocid1.compartment.oc1..aaaaaaaameevqllpkruwsepv2ps3otacm7vgx257uoi2o7sgeischov64hja
vcn: ocid1.vcn.oc1.iad.amaaaaaausti56qaepofn65jytrp37n5o4jopfbvxnquomr2dt46khs24s4a

oci-ccm.yaml:

oci-ccm.txt
kubectl logs -n kube-system -l k8s-app=oci-cloud-controller-manager -c oci-cloud-controller-manager --tail=-1
total 4
-rw-r--r-- 1 root root 1675 Mar 21 18:45 key.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAtu3ghDp8/LUR6Jc2GNHsDvcFRUgjB72g0yNG9HmPxas785Bu
F6DD5p9WjtpyLuwh2/MH87eDPtJzdmVGX2huI9BS130G4f4e0sGqBGIT0XMhlL+a
9zUAANhZavRWOY6+9qNKdB+MhPMsw5W3Pqc1g4MOjlRBlYVUa60lHZk6OEyi2SqI
lUziYbwBBSQUH7xWkk6ObbH+kdSEabaGhBUvmQ6X+2cYdJAZyrF23L1vAHjmS26H
...truncated...
JU6F8NeWiPiqUQ/kAO8FmG/3gYCIA6WDy0WmyOj7OvG2cs9yKw2hzu7lHw9sk+4D
6qvRAoGAG0TXp5Ub+gpzXx1wQj6pvb8IQKyazROtXst4M0bsqYmrJgw/fwUNoNh9
6Ks0N2W5gSLme5C8Irxi1mZPwntd5/jbwpYdbDCUA7svevM3CclHf5xLQwJtdg+i
bruvA3e2A1Gw8fp29l9OCrIwNsIUQPt4yfRryWpX/VjBYJlzZ40=
-----END RSA PRIVATE KEY-----
2025-03-21T18:56:23.336Z INFO oci-cloud-controller-manager/main.go:69 oci-cloud-controller-manager {"version": "411bfeb2", "build": "411bfeb2"}
I0321 18:56:24.870569 1 serving.go:386] Generated self-signed cert in-memory
I0321 18:56:25.538715 1 serving.go:386] Generated self-signed cert in-memory
W0321 18:56:25.538742 1 client_config.go:659] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0321 18:56:26.555771 1 requestheader_controller.go:247] Loaded a new request header values for RequestHeaderAuthRequestController
2025-03-21T18:56:26.560Z INFO metadata/instance_metadata.go:76 Metadata endpoint /opc/v2/instance/ returned response successfully
2025-03-21T18:56:26.571Z INFO metadata/instance_metadata.go:76 Metadata endpoint /opc/v2/instance/ returned response successfully
2025-03-21T18:56:26.571Z INFO client/utils.go:95 OCI using read rate limit configuration: QPS=20, bucket=5 {"component": "cloud-controller-manager"}
2025-03-21T18:56:26.571Z INFO client/utils.go:99 OCI using write rate limit configuration: QPS=20, bucket=5 {"component": "cloud-controller-manager"}
F0321 18:56:26.571981 1 main.go:82] Cloud provider could not be initialized: could not init cloud provider "oci": NewComputeClientWithConfigurationProvider: can not create client, bad configuration: PEM data was not found in buffer

Additional Info
Key works with OCI CLI using the same fingerprint.

Tested with PKCS#8 and RSA formats, symlinks and real files, various paths (/etc/oci/key.pem, /oci-config/key.pem).

Debug container confirms key presence.

Full logs available upon request.

Suspected Cause
Bug in NewComputeClientWithConfigurationProvider failing to parse a valid RSA key.

[ratdude07]

Could someone please take a look at this issue? Thank you.