etcdctl lease keep-alive & timetolive return success (0) even on failure

[chrislalos]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Calling etcdctl lease keep-alive or timetolive with an expired or non-existent lease id returns a status code of 0. A human-readable error message is printed, but the status code of 0 indicates success to a shell script or any other programmatic invocation.

etcdctl --discovery-srv redacted.domain.dev lease keep-alive 0bad1d0; echo $?
lease 0000000000bad1d0 expired or revoked.
0 

etcdctl --discovery-srv redcated.domain.dev lease timetolive 0bad1d0; echo $?
lease 0000000000bad1d0 already expired
0

This is inconsistent withetcdctl lease revoke, which does return a non-zero status code indicating an error

etcdctl --discovery-srv redcated.domain.dev lease revoke 0bad1d0; echo $?
{"level":"warn","ts":"2025-01-22T16:52:15.600772-0600","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x14000112000/etcd0.dylt.dev.:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: requested lease not found"}
Error: failed to revoke lease (etcdserver: requested lease not found)
1

What did you expect to happen?

A non-zero status code was expected, to indicate an error.

How can we reproduce it (as minimally and precisely as possible)?

etcdctl --discovery-srv valid.domain.dev lease keep-alive 0bad1d0; echo $?

Anything else we need to know?

timetolive seems like an odd subcommand, as opposed to a get, show, display etc subcommand that would display all suitable information about the lease, including timetolive, which could then be parsed out of output as desired. Perhaps such a subcommand could be introduced, and the new subcommand could return non-zero status for non-existent or expired lease id. That would avoid backward compatiblity issues with changing behavior of existing subcommands.

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.17
Git SHA: 507c0de87
Go Version: go1.23.3
Go OS/Arch: darwin/arm64

$ etcdctl version
etcdctl version: 3.5.17
API version: 3.5

Etcd configuration (command line flags or environment variables)

/opt/etcd/etcd --name XXX.XXX.XXX \
               --discovery-srv XXX.XXX.XXX \
               --initial-advertise-peer-urls http://X.X.X.X:2380 \
               --initial-cluster-token XXX --initial-cluster-state existing \
               --advertise-client-urls http://XXX.XXX.XXX.XXX:2379 \
               --listen-client-urls http://XXX.XXX.XXX.XXX:2379,http://127.0.0.1:2379 \
               --listen-peer-urls http://XXX.XXX.XXX.XXX:2380
               --data-dir /var/lib/etcd/

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
+------------------+---------+----------------+----------------------------+-----------------------------+------------+
|        ID        | STATUS  |      NAME      |         PEER ADDRS         |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+----------------+----------------------------+-----------------------------+------------+
| 8f1bc71338fcf7b7 | started | etcd0.xxxxxxxx.dev | http://etcd0.xxxxxxxx.dev:2380 |   http://x.x.x.x:2379 |      false |
| ab7cc91583cb4bd9 | started | etcd1.xxxxxxxx.dev | http://etcd1.xxxxxxxx.dev:2380 |    http://x.x.x.x:2379 |      false |
| eb0a61a295ac4f04 | started | etcd2.xxxxxxx.dev | http://etcd2.xxxxxxxx.dev:2380 | http://x.x.x.x:2379 |      false |
+------------------+---------+----------------+----------------------------+-----------------------------+------------+


$ etcdctl --endpoints=<member list> endpoint status -w table
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| etcd0.xxxxxxxx.dev:2379 | 8f1bc71338fcf7b7 |  3.5.16 |   20 MB |     false |      false |        78 |       1065 |               1065 |        |
| etcd1.xxxxxxxx.dev:2379 | ab7cc91583cb4bd9 |  3.5.16 |   21 MB |     false |      false |        78 |       1065 |               1065 |        |
| etcd2.xxxxxxxx.dev:2379 | eb0a61a295ac4f04 |  3.5.16 |   21 MB |      true |      false |        78 |       1065 |               1065 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Relevant log output

[moficodes]

@chrislalos Could you share instruction on how to reproduce? What is a etcd server configuration used?

[chrislalos]

I think I included instructions on how to reproduce above, but it's pretty simple: just do a keep-alive or a timetolive on a non-existent lease id.

Here's a copy paste from above

etcdctl --discovery-srv redacted.domain.dev lease keep-alive 0bad1d0; echo $?
lease 0000000000bad1d0 expired or revoked.
0 

etcdctl --discovery-srv redcated.domain.dev lease timetolive 0bad1d0; echo $?
lease 0000000000bad1d0 already expired
0

Let me know if you wanted something else

[chrislalos]

As for my configuration, I thought I'd filled that out but it looks like I missed it. Here's the command I use to launch the server, with redactions as appropriate

/opt/etcd/etcd --name XXX.XXX.XXX \
                        --discovery-srv XXX.XXX.XXX \
                        --initial-advertise-peer-urls http://X.X.X.X:2380 \
                        --initial-cluster-token XXX --initial-cluster-state existing \
                        --advertise-client-urls http://XXX.XXX.XXX.XXX:2379 \
                        --listen-client-urls http://XXX.XXX.XXX.XXX:2379,http://127.0.0.1:2379 \
                        --listen-peer-urls http://XXX.XXX.XXX.XXX:2380
                        --data-dir /var/lib/etcd/

Nothing special here, unless for some reason the --discovery-srv flag diverts everything to a different code path.