Linkerd Destination pods continuously stop watch and establish watch

[bartpeeters]

We don't know if this is normal behaviour, but our logs are filled by the Linkerd Destination pods:

time="2021-09-14T10:22:54Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:23:14Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:23:14Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:23:14Z" level=info msg="Stopping watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:23:14Z" level=info msg="Stopping watch on profile default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:25:42Z" level=info msg="Establishing watch on service default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:25:42Z" level=info msg="Starting watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:25:42Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:25:42Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:26:02Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:26:02Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:26:02Z" level=info msg="Stopping watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:26:02Z" level=info msg="Stopping watch on profile default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:28:57Z" level=info msg="Establishing watch on service default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:28:57Z" level=info msg="Starting watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:28:57Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:28:57Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:29:27Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:29:27Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:29:27Z" level=info msg="Stopping watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:29:27Z" level=info msg="Stopping watch on profile default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:34:25Z" level=info msg="Establishing watch on service default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:34:25Z" level=info msg="Starting watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:34:25Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:34:25Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:34:45Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:34:45Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:34:45Z" level=info msg="Stopping watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:34:45Z" level=info msg="Stopping watch on profile default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:42:36Z" level=info msg="Establishing watch on service default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:42:36Z" level=info msg="Starting watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:42:36Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:42:36Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:42:56Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:42:56Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:42:56Z" level=info msg="Stopping watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:42:56Z" level=info msg="Stopping watch on profile default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:53:45Z" level=info msg="Establishing watch on service default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T10:53:45Z" level=info msg="Starting watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:53:45Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:53:45Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:54:05Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:54:05Z" level=info msg="Stopping watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T10:54:05Z" level=info msg="Stopping watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T10:54:05Z" level=info msg="Stopping watch on profile default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T11:02:08Z" level=info msg="Establishing watch on service default/some-service" addr=":8086" component=traffic-split-watcher
time="2021-09-14T11:02:08Z" level=info msg="Starting watch on service default/some-service" addr=":8086" component=opaque-ports-watcher
time="2021-09-14T11:02:08Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher
time="2021-09-14T11:02:08Z" level=info msg="Establishing watch on profile default/some-service.default.svc.cluster.local" addr=":8086" component=profile-watcher

These are the logs for a single service, but we have 50+ services so our logs are just filled with these 'Stopping watch' and 'Establishing watch' log messages.
Is it normal that this happens multiple times per hour?

Should we just change the log level to warn?

What causes the destination pod to stop / establish these watches?

[adleong]

This is normal expected behavior because watches will start and stop as necessary depending on traffic patterns. I agree that these logs are a little bit noisy. We could consider changing these log messages to DEBUG level. Please feel free to file a Linkerd issue or Pull Request if you are interested.

[bartpeeters]

Thanks for the quick answer.
Maybe this logging at the info level is useful in some cases? I'll make an issue and see what the reactions are.