cloudapi: git sha version api and journal for workers #4484

lzap · 2024-11-20T13:35:00Z

Adds a runtime information similarily as in image builder and a logrus hook that is used to put git sha to every single log message. This way we can tell which version was log created with since composer and workers sometimes are running different versions.

This data is then leveraged in a new simple /version cloud API call that returns the data similarily to IB. My plan is to call this from IB too so IB reports both its and composer version info.

Finally, I noticed that workers create pretty unreadable logs, it is a string-encoded JSON:

{"message":"{\"channel\":\"staging\",\"file\":\"/opt/app-root/src/internal/common/slogger/logrus.go:29\",\"func\":\"github.com/osbuild/osbuild-composer/internal/common/slogger.(*simpleLogrus).log\",\"job_dependencies\":\"[8402b9c0-ac94-4952-8609-1ad4e575a28a e30715cd-0a59-4993-a530-fdb2189ad2bb]\",\"job_id\":\"be17e941-d8f5-4de2-9eb3-e7a1b3ee41d8\",\"job_type\":\"manifest-id-only\",\"level\":\"info\",\"msg\":\"Dequeued job\",\"time\":\"2024-11-20T12:10:37Z\"}\n","ident":"osbuild-composer","host":"composer-5dcd8bfdf9-bhrv6"}

This patch builds on top of native journald capability which I added last year and is used in on-prem mode. Instead using standard output/err worker now detects if there is a systemd-journald running and uses its native API to send logs in structured form.

This means all logs are native now, it can be inspected with journalctl and what is more important they should get into Kibana/Splunk as regular JSON improving our searching capabilities:

journalctl -o verbose
...
    BUILD_TIME=2024-11-20T11:24:49Z
    BUILD_COMMIT=ff2660
    PRIORITY=3
    MESSAGE=Requesting job failed: Post "https://0:8881/api/worker/v1/jobs": dial tcp 0.0.0.0:8881: connect: connection refused
    _PID=254883
    _COMM=__debug_bin3968
    _EXE=/home/lzap/osbuild-composer/cmd/osbuild-worker/__debug_bin396826732
    _CMDLINE=/home/lzap/osbuild-composer/cmd/osbuild-worker/__debug_bin396826732 0:8881
    _SOURCE_REALTIME_TIMESTAMP=1732108805201853
Wed 2024-11-20 14:20:05.201871 CET [s=83cfd176837d4a15b19c452be25c8593;i=1ed8;b=3d1a7f8752454afda1a56b25debf50d9;m=664c2be936;t=62758020be16c;x=299933d4e616281a]
    _SELINUX_CONTEXT=kernel
    _BOOT_ID=3d1a7f8752454afda1a56b25debf50d9
    _MACHINE_ID=90304ac7a1274d6ca3abfee98b7975c1
    _HOSTNAME=dev.home.lan
    _RUNTIME_SCOPE=system
...

I know this is sort of two features in one PR but these are tiny changes, the API endpoint is super small and I wrote a test.

cmd/osbuild-worker/main.go

lzap · 2024-11-20T13:37:32Z

internal/cloudapi/v2/handler.go

+	}
+	return ctx.JSON(http.StatusOK, version)
+}
+


We have the very same code in image-builder too, this is a copy-paste literally.

mvo5

Quick drive-by wondering inline

mvo5 · 2024-11-20T14:19:43Z

internal/common/runtime.go

+import "runtime/debug"
+
+var (
+	// Git SHA commit (first 4 characters)


This comment (4 chars) seems to not match the implementaion AFACIT

mvo5 · 2024-11-20T14:20:47Z

internal/common/runtime.go

+		BuildCommit = "HEAD"
+	}
+
+	BuildGoVersion = bi.GoVersion


Is bi actually defined (i.e. non-nil) if we get a !ok before (if not we would crash here)? I wonder if it wouldn't be cleaner to define the defaults directly in var BuildCommit = "N/A" and just return on if !ok { return }(?)

schuellerf

No major comment additionally to the already mentioned ones.
In image builder we changed the way some paths are handled so they don't need authentication, which might be neat for /version but I'd approve as this might be easier in a followup PR

cmd/osbuild-worker/config.go

schuellerf · 2024-11-27T09:35:15Z

internal/cloudapi/v2/server.go

 	RegisterHandlers(e.Group(path, mws...), &handler)

+	// no auth endpoints
+	e.GET("/status", func(c echo.Context) error {


I like the paths without auth 😎
But should it be

Suggested change

e.GET("/status", func(c echo.Context) error {

e.GET("/version", func(c echo.Context) error {

?

I know right, but this is what is in the image-builder :-) Do you want me to break this pattern? I am fine with that, can also create both maybe?

I think /version exists as it's in the openapi.v2.yml so this might already result in both existing (which is good)
Rather somehow assure /version not to be authenticated

Ok added /version with non-auth middleware and also adding status/ready as fixed paths that simply return 200 without ANY middleware because we do not this to be logged or have metrics as k8s calls this every now and then.

lzap · 2024-11-27T11:10:11Z

.golangci.yml

@@ -1,3 +1,6 @@
+exclude-dirs:
+  - vendor
+


For full transparency, I sneaked this hunk in with the last amend, I realized that go linter lints also vendor directories and that slows it significantly. Can do a separate PR if requested.

schuellerf · 2024-11-27T11:18:40Z

internal/cloudapi/v2/openapi.v2.yml

@@ -17,6 +17,21 @@ servers:
  description: current domain

 paths:
+  /version:


I think this is good, but when thinking about it … that's the version of osbuild-composer not the worker, right? (in contrast to the PR title and commit subject :-/ )

The API is "push" in this case - so the worker would have to "push" it's version, maybe?

osbuild-composer/internal/worker/api/openapi.yml

Line 349 in 64ff0e3

properties:

Oh yeah I started the patch with intention to include sha git in all log messages which was done. Then the version endpoint was added, yeah, this is just for the composer.

We can definitely add workers at some point, out of scope for this patch I would say.

So this could be either split in two commits or at least the description (PR and commit) should denote that this is touching worker and composer?

cmd/osbuild-worker/main.go

lzap · 2024-11-29T10:55:13Z

I have split it into multiple commits, rebased.

schuellerf

lgtm, thanks!

lzap · 2024-11-29T20:15:13Z

The only test failing is edge-farm, rebasing.

lzap · 2024-12-04T14:14:32Z

@croissanne @ezr-ondrej @mvo5 please re-review tests are good I want to get this one merged to increase readability of logs from composer and continue further improving logs with standardized correlation thanks!

lzap requested review from schuellerf and croissanne November 20, 2024 13:35

lzap commented Nov 20, 2024

View reviewed changes

cmd/osbuild-worker/main.go Show resolved Hide resolved

lzap commented Nov 20, 2024

View reviewed changes

lzap force-pushed the version-api branch from 3980d11 to c1e885b Compare November 20, 2024 13:45

mvo5 reviewed Nov 20, 2024

View reviewed changes

schuellerf reviewed Nov 20, 2024

View reviewed changes

cmd/osbuild-worker/config.go Outdated Show resolved Hide resolved

lzap force-pushed the version-api branch 2 times, most recently from c1efe97 to 5df2627 Compare November 27, 2024 08:38

schuellerf reviewed Nov 27, 2024

View reviewed changes

lzap force-pushed the version-api branch 2 times, most recently from 5953c49 to ac5e75d Compare November 27, 2024 11:08

lzap commented Nov 27, 2024

View reviewed changes

schuellerf reviewed Nov 27, 2024

View reviewed changes

lzap force-pushed the version-api branch from ac5e75d to a0c7b5d Compare November 27, 2024 14:47

mvo5 reviewed Nov 27, 2024

View reviewed changes

cmd/osbuild-worker/main.go Outdated Show resolved Hide resolved

lzap force-pushed the version-api branch 2 times, most recently from e229e0d to c4c9682 Compare November 29, 2024 10:54

schuellerf approved these changes Nov 29, 2024

View reviewed changes

lzap force-pushed the version-api branch from c4c9682 to bd11cc4 Compare November 29, 2024 20:15

lzap requested review from ezr-ondrej, schuellerf and mvo5 November 29, 2024 20:15

lzap added 5 commits December 4, 2024 15:13

test: exclude vendor/ directory for linter

29ef833

common: introduce runtime info with git sha

e4644cc

cmd/osbuild-composer: add runtime git info to logs

d202713

cmd/osbuild-worker: add runtime git info to logs

f4a4776

cloudapi: introduce version endpoint

c1f0d26

lzap force-pushed the version-api branch from bd11cc4 to c1f0d26 Compare December 4, 2024 14:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cloudapi: git sha version api and journal for workers #4484

cloudapi: git sha version api and journal for workers #4484

lzap commented Nov 20, 2024

lzap Nov 20, 2024

mvo5 left a comment

mvo5 Nov 20, 2024

mvo5 Nov 20, 2024

schuellerf left a comment

schuellerf Nov 27, 2024

lzap Nov 27, 2024

schuellerf Nov 27, 2024

lzap Nov 27, 2024

lzap Nov 27, 2024

schuellerf Nov 27, 2024

schuellerf Nov 27, 2024

lzap Nov 27, 2024

schuellerf Nov 29, 2024

lzap commented Nov 29, 2024

schuellerf left a comment

lzap commented Nov 29, 2024

lzap commented Dec 4, 2024

	e.GET("/status", func(c echo.Context) error {
	e.GET("/version", func(c echo.Context) error {

cloudapi: git sha version api and journal for workers #4484

Are you sure you want to change the base?

cloudapi: git sha version api and journal for workers #4484

Conversation

lzap commented Nov 20, 2024

Choose a reason for hiding this comment

mvo5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schuellerf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzap commented Nov 29, 2024

schuellerf left a comment

Choose a reason for hiding this comment

lzap commented Nov 29, 2024

lzap commented Dec 4, 2024