Set pipeline name automatically in transform definitions #2973

mrodm · 2025-10-02T11:08:19Z

This PR adds support to set automatically the pipeline name used in transforms adding the current package version as a prefix.

In order to achieve that elasticsearch/transform/*/transform.yml is managed by elastic-package as a template file (Golang html template). And it is allowed to use a function in the template named IngestPipelineName that adds the current version of the package to the pipeline.

Example:

dest:
  index: "logs-ti_anomali_latest.threatstream-2"
  pipeline: '{{ IngestPipelineName "transform" }}'
  aliases:
    - alias: "logs-ti_anomali_latest.threatstream"
      move_on_creation: true

This rendering of the template happens at different stages to ensure always is performed:

Building package: elastic-package build
Installing package: elastic-package install
System tests: elastic-package test system

Author's checklist

Test rendering while building
Test installing package
Test running system tests
Add note in Revert stack version pinning for packages failing with 9.x #2773 for the new test package.

How to test this PR locally

elastic-package stack up -v -d
package_path="test/packages/parallel/ti_anomali_template"

# Build and check that pipeline has been replaced
elastic-package -C "${package_path}" build -v
grep -A 5 "dest:"  build/packages/ti_anomali_template/1.22.1/elasticsearch/transform/latest_ioc/transform.yml 
# Expected output:
# dest:
#   index: "logs-ti_anomali_latest.threatstream-2"
#   pipeline: '1.22.1-transform'
#   aliases:
#     - alias: "logs-ti_anomali_latest.threatstream"
#       move_on_creation: true


# Install package
elastic-package -C "${package_path}" install -v

# check in Kibana that the transform has been installed with the right pipeline
curl -k -s -u elastic:changeme 'https://localhost:9200/_transform/logs-ti_anomali_template.latest_ioc-default-0.3.0' | jq -r '.transforms[] | .dest'
# Expected output:
# {
#   "index": "logs-ti_anomali_latest.threatstream-2",
#   "aliases": [
#     {
#       "alias": "logs-ti_anomali_latest.threatstream",
#       "move_on_creation": true
#     }
#   ],
#   "pipeline": "1.22.1-transform"
# }

# Running system tests for this package should work successfully
# Transform should be detected and validated
elastic-package -C "${package_path}" test system -v

elastic-package stack down -v

Allow to set an IngestPipelineName function in these templates in order to set the right ingest template filename. This filename must contain as a prefix the package version (from the manifest).

mrodm · 2025-10-02T14:44:53Z

/test

mrodm · 2025-10-02T15:55:43Z

/test

mrodm · 2025-10-02T17:03:04Z

/test

mrodm · 2025-10-02T17:42:45Z

/test

mrodm · 2025-10-02T18:17:45Z

/test

mrodm · 2025-10-03T08:14:18Z

/test

mrodm · 2025-10-03T11:09:20Z

/test

jsoriano

LGTM, some nitpicking and wondering if we should use text/template instead of html/template.

jsoriano · 2025-10-07T14:20:36Z

internal/packages/packages.go

 	"encoding/json"
 	"errors"
 	"fmt"
+	"html/template"


Is there any reason to use html/template instead of text/template? The html one escapes some characters for safe HTML rendering, but we probably don't want or need this here.

Suggested change

"html/template"

"text/template"

No specific reason, I'll test with text/template 👍

jsoriano · 2025-10-07T14:22:10Z

internal/packages/packages.go

+
+	t := template.New(filepath.Base(transformPath))
+	t, err = t.Funcs(template.FuncMap{
+		"IngestPipelineName": func(pipelineName string) (string, error) {


Nit. Looking to examples in the doc I think we should use lower case for functions.

Suggested change

"IngestPipelineName": func(pipelineName string) (string, error) {

"ingestPipelineName": func(pipelineName string) (string, error) {

internal/packages/packages.go

jsoriano · 2025-10-07T14:34:24Z

internal/packages/packages.go

+	_, err = os.Stat(filepath.Join(packageRootPath, "elasticsearch", "ingest_pipeline", pipelineFileName))
+	if err != nil {
+		return nil, TransformDefinition{}, fmt.Errorf("destination ingest pipeline file %s not found: %w", pipelineFileName, err)
+	}


Did you test with integrations? Maybe some package fails with this 😬 Though if it does we should fix the package.

Testing to build locally all packages in the integrations repository, there is one package that sets as dest.pipeline the ingest pipeline from the data stream:

dest: index: "aws_billing.cur-v1" pipeline: "metrics-aws_billing.cur-0.1.0-pipeline_extract_metadata" aliases: - alias: "aws_billing.cur_latest" move_on_creation: true

That fails with the current validation 🤔

That package (aws_billing) is failing this validation because of:

It's using a pipeline from a data stream

It set an old version in the destination pipeline, it should be 0.2.0.

That pipeline from cur data stream looks like it is not used:
https://github.com/elastic/integrations/blob/34df8c08914a678dc3310bd000e7908a1af42ef6/packages/aws_billing/data_stream/cur/elasticsearch/ingest_pipeline/pipeline_extract_metadata.yml

At least from the default pipeline there is no pipeline processor to trigger that other pipeline:
https://github.com/elastic/integrations/blob/34df8c08914a678dc3310bd000e7908a1af42ef6/packages/aws_billing/data_stream/cur/elasticsearch/ingest_pipeline/default.yml

Here I've added the support to detect if the ingest pipeline exists on any data stream:
4f2756e

Probably, it could be moved the pipeline in that package to be on the elasticsearch folder in the root, but in any case it could be interesting to keep the support to check if pipelines are located in data streams.

WDYT ?

Co-authored-by: Jaime Soriano Pastor <[email protected]>

mrodm · 2025-10-07T19:18:32Z

test integrations

elastic-vault-github-plugin-prod · 2025-10-07T19:25:16Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15594

elasticmachine · 2025-10-07T19:39:46Z

💚 Build Succeeded

Buildkite Build
Commit: 4f2756e

History

💔 Build #6300 failed a44d6cc
💚 Build #6288 succeeded 351171c
💚 Build #6287 succeeded 351171c
💚 Build #6284 succeeded 351171c
💔 Build #6283 failed 351171c

cc @mrodm

mrodm added 2 commits October 2, 2025 12:26

Set transform manifest as template

07f3e15

Allow to set an IngestPipelineName function in these templates in order to set the right ingest template filename. This filename must contain as a prefix the package version (from the manifest).

Add a copy of ti_anomaly with a dummy ingest pipeline for transform

5961b7e

mrodm self-assigned this Oct 2, 2025

mrodm added 6 commits October 2, 2025 13:29

Skip check if there is no destination pipeline

951defe

Check for empty pipelines

6cf6d08

Add test

9093315

Pin stack version for ti_anomali_template

ff9aa3f

Update stack version used for ti_anomali and ti_anomali_logsdb

bf90e2b

Update index and alias names in tranform

351171c

mrodm marked this pull request as ready for review October 3, 2025 11:46

mrodm requested a review from a team October 6, 2025 08:42

jsoriano reviewed Oct 7, 2025

View reviewed changes

mrodm and others added 5 commits October 7, 2025 19:24

Apply suggestions from code review

789693b

Co-authored-by: Jaime Soriano Pastor <[email protected]>

Set to lowercase first letter in template function

d8f0447

Use text/template

ff0daa5

Re-phrase error message

a44d6cc

Validate if ingest pipeline exists on data stream

4f2756e

elastic-vault-github-plugin-prod bot mentioned this pull request Oct 7, 2025

Test elastic-package#2973 - DO NOT MERGE elastic/integrations#15594

Draft

	"IngestPipelineName": func(pipelineName string) (string, error) {
	"ingestPipelineName": func(pipelineName string) (string, error) {

Set pipeline name automatically in transform definitions #2973

Are you sure you want to change the base?

Set pipeline name automatically in transform definitions #2973

Uh oh!

Conversation

mrodm commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Author's checklist

How to test this PR locally

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 3, 2025

Uh oh!

mrodm commented Oct 3, 2025

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jsoriano Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm commented Oct 7, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 7, 2025

Uh oh!

elasticmachine commented Oct 7, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

mrodm commented Oct 2, 2025 •

edited

Loading