ssh exporter #2138

EHSchmitt4395 · 2024-11-21T03:09:23Z

PR Description

ssh exporter for remote hosts

define ip along with password or key for the creation of custom metrics from a remote host via bash interpolation

ex alloy config:

prometheus.exporter.ssh "example" {
  verbose_logging = true

  targets {
    address         = "192.168.1.10"
    port            = 22
    username        = "admin"
    password        = "password"
    command_timeout = 10

    custom_metrics {
      name    = "load_average"
      command = "cat /proc/loadavg | awk '{print $1}'"
      type    = "gauge"
      help    = "Load average over 1 minute"
    }
  }

  targets {
    address         = "192.168.1.11"
    port            = 22
    username        = "monitor"
    key_file        = "/path/to/private.key"
    command_timeout = 15

    custom_metrics {
      name        = "disk_usage"
      command     = "df / | tail -1 | awk '{print $5}'"
      type        = "gauge"
      help        = "Disk usage percentage"
      parse_regex = "(\\d+)%"
    }
  }
}

prometheus.scrape "demo" {
  targets    = prometheus.exporter.ssh.example.targets
  forward_to = [prometheus.remote_write.demo.receiver]
}

prometheus.remote_write "demo" {
  endpoint {
    url = PROMETHEUS_REMOTE_WRITE_URL

    basic_auth {
      username = USERNAME
      password = PASSWORD
    }
  }
}

Notes to the Reviewer

plz let me know if issue or something missing with my pr - cheers :)

PR Checklist

CHANGELOG.md updated
Documentation added
Tests updated

internal/static/integrations/ssh_exporter/ssh_client.go

ptodev · 2024-11-21T10:55:02Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+| Name              | Type     | Description                                        | Default | Required |
+| ----------------- | -------- | -------------------------------------------------- | ------- | -------- |
+| `verbose_logging` | `bool`   | Enable verbose logging for debugging purposes.     | `false` | no       |
+| `targets`         | `block`  | One or more target configurations for SSH metrics. |         | yes      |


ptodev · 2024-11-21T11:00:07Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+| `port`            | `int`                 | SSH port number.                                                       | `22`    | no       |
+| `username`        | `string`              | SSH username for authentication.                                       |         | yes      |
+| `password`        | `secret`              | Password for password-based SSH authentication.                        |         | Required if `key_file` is not provided |
+| `key_file`        | `string`              | Path to the private key file for key-based SSH authentication.         |         | Required if `password` is not provided |


It's better to just have a key attribute which is a secret string. People can still get that string from a file if they use local.file. The benefit is that it's more flexible, because you can also get the secret from somewhere like remote.vault or remote.kubernetes.secret.

ptodev · 2024-11-21T11:01:19Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+| `address`         | `string`              | The IP address or hostname of the target server.                       |         | yes      |
+| `port`            | `int`                 | SSH port number.                                                       | `22`    | no       |
+| `username`        | `string`              | SSH username for authentication.                                       |         | yes      |
+| `password`        | `secret`              | Password for password-based SSH authentication.                        |         | Required if `key_file` is not provided |


ptodev · 2024-11-21T11:08:23Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+#### Metric Types
+
+- `gauge`: Represents a numerical value that can go up or down.
+- `counter`: Represents a cumulative value that only increases.
+
+#### parse_regex


Suggested change

#### Metric Types

- `gauge`: Represents a numerical value that can go up or down.

- `counter`: Represents a cumulative value that only increases.

#### parse_regex

The `type` argument can have either of the following values:

- `gauge`: Represents a numerical value that can go up or down.

- `counter`: Represents a cumulative value that only increases.

We usually don't use sub-headings under the tables, unless there's too much text and there's no other way to structure the information.

ptodev · 2024-11-21T11:13:58Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+The `prometheus.exporter.ssh` component uses the `known_hosts` file to validate host keys and protect against man-in-the-middle (MITM) attacks. Here's how it handles this:
+
+1. **First-Time Setup**:
+   - If the `known_hosts` file does not exist, the component creates it and fetches the host key using `ssh-keyscan`.


Suggested change

- If the `known_hosts` file does not exist, the component creates it and fetches the host key using `ssh-keyscan`.

- If the `~/.ssh/known_hosts` file does not exist, the component creates it and fetches the host key using `ssh-keyscan`.

Would be good to know where the file is located.

ptodev · 2024-11-21T12:17:12Z

internal/converter/internal/staticconvert/internal/build/ssh_exporter.go

We don't need static mode converters, since static mode doesn't support this exporter :)

ptodev · 2024-11-21T12:19:54Z

internal/static/integrations/ssh_exporter/config.go

Do we really need this file? The reason other exporters have it is that they come from Agent Static mode.

ptodev · 2024-11-21T12:22:10Z

internal/static/integrations/ssh_exporter/ssh_client.go

Is the code copied from an already existing prometheus exporter? If so, we should probably check the license to make sure it's ok to copy it. But it might be better to just import the other exporter if possible?

ptodev · 2024-11-21T12:29:43Z

internal/static/integrations/ssh_exporter/ssh_client.go

+        time.Sleep(time.Second)
+    }
+    if len(output) == 0 {
+        return fmt.Errorf("failed to fetch host key for %s after 3 attempts: last error: %w", targetAddress, scanErr)


If a user fixes the issue with this, would they have to restart Alloy for things to start working? Ideally the component should just keep trying periodically until it works.

ptodev · 2024-11-21T12:30:59Z

internal/static/integrations/ssh_exporter/ssh_client.go

+
+
+
+func (c *SSHClient) RunCommand(command string) (string, error) {


Are there any SSH commands that should not be allowed? I realise this may be hard to enforce, but I just think some users might want extra security and I wonder if there are any further checks we can do.

ptodev · 2024-11-21T12:38:15Z

Once the component is merged, it would be useful to have modules in the alloy-modules repo which show how to extract various metrics in Linux.

wildum · 2024-11-21T13:02:28Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+# prometheus.exporter.ssh
+
+The `prometheus.exporter.ssh` component embeds an SSH exporter for collecting metrics from remote servers over SSH and exporting them as Prometheus metrics.
+


Please add the experimental snippet to document the fact that the component is experimental

wildum · 2024-11-21T13:03:17Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+---
+canonical: https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.ssh/
+aliases:
+  - ../prometheus.exporter.ssh/ # /docs/alloy/latest/reference/components/prometheus.exporter.ssh/


aliases are not need for this component because it's brand new (this was used for mapping after the structure of the doc was changed)

ptodev · 2024-11-21T13:24:16Z

internal/static/integrations/ssh_exporter/ssh_collector.go

+            continue
+        }
+
+        level.Debug(c.logger).Log("msg", "executed custom command", "command", cm.Command, "value", value)


We could add debug statements like this to Live Debugging. They can also include the output of the command.

mattdurham · 2024-11-21T14:15:30Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+| `username`        | `string`              | SSH username for authentication.                                       |         | yes      |
+| `password`        | `secret`              | Password for password-based SSH authentication.                        |         | Required if `key_file` is not provided |
+| `key_file`        | `string`              | Path to the private key file for key-based SSH authentication.         |         | Required if `password` is not provided |
+| `command_timeout` | `int`                 | Timeout in seconds for each command execution over SSH.                | `30`    | no       |


Should this be a duration? We rarely use dedicated seconds.

What is the expected behavior if command timeout > scrape timeout?

mattdurham · 2024-11-21T14:19:08Z

internal/static/integrations/ssh_exporter/ssh_client.go

@@ -0,0 +1,194 @@
+package ssh_exporter


Is this windows compatible?

mattdurham · 2024-11-21T14:19:44Z

internal/static/integrations/ssh_exporter/ssh_client.go

+    timeout time.Duration
+}
+var sshKeyscanCommand = func(targetAddress string) ([]byte, error) {
+    cmd := exec.Command("ssh-keyscan", "-H", targetAddress)


Can target address be hijacked for any injection? We should ensure it is an valid URI.

mattdurham · 2024-11-21T14:24:41Z

internal/static/integrations/ssh_exporter/ssh_client.go

+        }
+    }
+
+    file, err := os.OpenFile(knownHostsPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600)


Is this safe? It likely is because its only called from NewSSHClient but maybe a global mutex?

mattdurham · 2024-11-21T14:27:05Z

internal/static/integrations/ssh_exporter/ssh_collector.go

+}
+
+func (c *SSHCollector) executeCustomCommand(cm CustomMetric) (float64, error) {
+    output, err := c.client.RunCommand(cm.Command)


The underlying component context should likely also be passed here.

And checked against.

clayton-cornell · 2024-11-21T19:19:56Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+The following arguments can be used to configure the exporter's behavior.
+All arguments are optional unless specified. Omitted fields take their default values.


Observation: In reviewing this I've noticed that the wording of some sections in the Prometheus topics are not consistent with the rest of the components. For example, Arguments in other sections simply state: The following arguments are supported:

I'll create an Issue to follow up on this. No action or suggestions for this PR.

clayton-cornell · 2024-11-21T19:22:02Z

docs/sources/reference/components/prometheus/prometheus.exporter.ssh.md

+
+The following blocks are supported inside the definition of `prometheus.exporter.ssh`:
+
+| Block          | Description                                                 | Required |


Should we add/include the Hierarchy column in this table?

EHSchmitt4395 added 2 commits November 20, 2024 18:53

initial

757b3c3

remove these unecessary as doing custom

3a39d7a

EHSchmitt4395 requested review from clayton-cornell and a team as code owners November 21, 2024 03:09

github-advanced-security bot found potential problems Nov 21, 2024

View reviewed changes

internal/static/integrations/ssh_exporter/ssh_client.go Fixed Show fixed Hide fixed

EHSchmitt4395 added 3 commits November 20, 2024 20:21

fix for ssh hole

a80b98c

known hosts security fix

d62de87

kick for build that had unrelated failure

82c4e2b

ptodev self-assigned this Nov 21, 2024

ptodev reviewed Nov 21, 2024

View reviewed changes

wildum reviewed Nov 21, 2024

View reviewed changes

ptodev reviewed Nov 21, 2024

View reviewed changes

mattdurham reviewed Nov 21, 2024

View reviewed changes

internal/static/integrations/ssh_exporter/ssh_client.go

@@ -0,0 +1,194 @@

package ssh_exporter

Copy link

Collaborator

mattdurham Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this windows compatible?

mattdurham reviewed Nov 21, 2024

View reviewed changes

clayton-cornell reviewed Nov 21, 2024

View reviewed changes

clayton-cornell mentioned this pull request Nov 21, 2024

The wording of some sections in the Prometheus topics are not consistent with the rest of the components. #2143

Open

clayton-cornell reviewed Nov 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ssh exporter #2138

ssh exporter #2138

EHSchmitt4395 commented Nov 21, 2024 •

edited

Loading

ptodev Nov 21, 2024

ptodev Nov 21, 2024

ptodev Nov 21, 2024

ptodev Nov 21, 2024

ptodev Nov 21, 2024

ptodev Nov 21, 2024

ptodev Nov 21, 2024

ptodev Nov 21, 2024

ptodev Nov 21, 2024

ptodev Nov 21, 2024

ptodev commented Nov 21, 2024

wildum Nov 21, 2024

wildum Nov 21, 2024

ptodev Nov 21, 2024

mattdurham Nov 21, 2024

mattdurham Nov 21, 2024

mattdurham Nov 21, 2024

mattdurham Nov 21, 2024

mattdurham Nov 21, 2024

mattdurham Nov 21, 2024

mattdurham Nov 21, 2024

clayton-cornell Nov 21, 2024

clayton-cornell Nov 21, 2024

	\| `password` \| `secret` \| Password for password-based SSH authentication. \| \| Required if `key_file` is not provided \|
	\| `password` \| `secret` \| Password for password-based SSH authentication. \| \| no

	- If the `known_hosts` file does not exist, the component creates it and fetches the host key using `ssh-keyscan`.
	- If the `~/.ssh/known_hosts` file does not exist, the component creates it and fetches the host key using `ssh-keyscan`.




		func (c *SSHClient) RunCommand(command string) (string, error) {

		# prometheus.exporter.ssh

		The `prometheus.exporter.ssh` component embeds an SSH exporter for collecting metrics from remote servers over SSH and exporting them as Prometheus metrics.

		The following arguments can be used to configure the exporter's behavior.
		All arguments are optional unless specified. Omitted fields take their default values.


		The following blocks are supported inside the definition of `prometheus.exporter.ssh`:

		\| Block \| Description \| Required \|

ssh exporter #2138

Are you sure you want to change the base?

ssh exporter #2138

Conversation

EHSchmitt4395 commented Nov 21, 2024 • edited Loading

PR Description

Notes to the Reviewer

PR Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ptodev commented Nov 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EHSchmitt4395 commented Nov 21, 2024 •

edited

Loading