v4.0.0-pre-3

🥳 It's finally (almost) here! Welcome to gomplate v4.0.0-pre-3! This release is a (pre-release of a ) major release, with breaking changes. Read on for more details...

Probably the most important feature that's evolved in gomplate over the years is its support for reading and parsing data from a variety of sources. Using URLs to specify where to find data has proven to be quite flexible, and has allowed for a lot of different data sources to be supported.

In early 2021, I took what I had learned from supporting all of these different data sources, and I created a new module called go-fsimpl, which provides a number of Go filesystem implementations that can be used with Go's filesystem interfaces.

My goal with go-fsimpl was always to use it in gomplate, and now that it supports most of the data sources that gomplate supports, it's time to make the switch! As a result, and because go-fsimpl presents a much more consistent and unified approach to data sources, I've had to drop support for one data source, and in fixing some inconsistencies in how URLs have been handled by some of the data sources, I've had to make some breaking changes there as well.

Please, if you're using gomplate in production, please read these release notes carefully and in full, and review your usage of gomplate to make sure you can adapt to these changes.

Breaking changes

Dropped support for the BoltDB data source

Support for the boltdb URL scheme has been removed. The implementation was never very robust, and I'm not aware of anyone using it. It had some unique quirks that would have made it quite difficult to support in go-fsiml as-is, so I've decided to drop it.

No more slim binaries

The slim binaries were deprecated in v3.11, and will no longer be produced.

Stricter URL handling

Subpaths are now always relative URLs

When using the datasource function with a subpath (second argument), the subpath has previously been intepreted by some data sources as a simple addition to the URL, and by others as a relative URL to be interpreted relative to the data source's base URL (as specified by the --datasource flag).

Now, all data sources will interpret the subpath as a relative URL, and will resolve it relative to the data source's base URL. This means that, given a base URL of git+https://github.com/hairyhenderson/gomplate//random and a subpath of random.go, the resulting URL will be git+https://github.com/hairyhenderson/gomplate//random.go. To correct this, make sure that base URLs end with a trailing slash when the last component is a directory.

Also, if the subpath begins with a slash, it will be interpreted as an absolute path, and will replace the base URL's path. For example, given a base URL of git+file:///tmp/repos//myrepo/ and a subpath of /myfile.txt, the resulting URL will be git+file:///myfile.txt. To correct this, make sure that subpaths don't begin with a slash. Note that for the special // separator indicating the separation between a Git repository and the path within, a subpath should now begin with .// rather than //.

This affects at least the git and file data sources.

Directories in datasource URLs must end with a slash

Previously, some data sources would assume URLs that end without a trailing slash are directories, and relative lookups would be performed within that directory. This was inconsistent with how URLs are normally interpreted, and could lead to unexpected results.

Now, all data sources will always interpret URLs without a trailing slash as files, and relative lookups will be performed within the parent directory of the URL.

For example, given a base URL of vault:///secret and a subpath of foo, the resulting URL will be vault:///foo. To correct this, make sure that base URLs end with a trailing slash when the last component is a directory.

Consistent data values

The aws+smp data source previously returned the AWS output object, with both metadata and data. Users needed to reference the Value field to get the actual data. Now, the aws+smp data source will return the data directly.

The consul data source supports directory semantics, but previously returned all the data as a JSON array of key/value pair objects. Now, the consul data source will return directory listings like all other data sources, as a JSON array of strings (the names of the keys).

Vault app-id auth support removed

Support for the Vault app-id auth method has been removed. This method was deprecated in Vault 0.6.1, and removed in Vault 1.13. If you're still using this method, you must switch to a different auth method.

The suppressEmpty behaviour is now always enabled

In previous versions of gomplate, output that consist only of whitespace would by default still be written to files, with the suppressEmpty flag or GOMPLATE_SUPPRESS_EMPTY environment variable being required to suppress this behaviour. Now, this behaviour is always enabled, and the suppressEmpty flag and GOMPLATE_SUPPRESS_EMPTY environment variable have been removed.

Log message formatting changes

A new logging framework is now in use (the standard library's log/slog package), and so error and debug (--verbose) messages will appear differently (different colours for the console format, and key/value pairs may are sorted differently).

Removed functions

The following functions have been removed:

• conv.Dict, conv.Slice, conv.Has - These have been deprecated since v3.2.0 and have been replaced with coll.Dict, coll.Slice, and coll.Has.

Deprecations

See the deprecation policy.

• A number of functions are deprecated, and will likely be removed in v5:
  • the slice alias for coll.Slice (use coll.Slice instead)
  • The net.Parse* functions which produced values from the inet.af/netaddr module have been deprecated and replaced with functions using the Go standard library's net/netip package:
    • net.ParseIP (use net.ParseAddr instead)
    • net.ParseIPPrefix (use net.ParsePrefix instead)
    • net.ParseIPRange (use net.ParseRange instead)
• The array form of configuring nested templates is deprecated, and a new map-based form is now available. This will be removed in a later v4 minor release (i.e. not in v4.0.0, but before v5).
• Support for the AWS_META_ENDPOINT environment variable for overriding the IMDS endpoint in aws functions has been deprecated in favor of the AWS-native AWS_EC2_METADATA_SERVICE_ENDPOINT. This will be removed in a later v4 minor release (i.e. not in v4.0.0, but before v5).
• A number of functions and types in the Go API are deprecated. Using a linter like staticcheck will help you find these in your code if you're using gomplate as a module. Follow the instructions in the deprecation comments to find out what to use instead.

Thanks

Plenty of people have contributed to gomplate by filing issues, suggesting new features, or submitting pull requests. I'd like to especially thank the following people for submitting pull requests in this release:

• @adriens
• @ahochsteger
• @AndrewSav
• @giorgiga
• @hartzell
• @horvski
• @iblancasa
• @isavcic
• @Juneezee
• @mayurwaghmode
• @mihaitodor
• @stefanb
• @sukhil-suresh
• @tenstad
• @wuhuizuo
• @xxxcoltxxx
• @zregvart
• @lliissoonngg

---

Be sure to also check out the multi-platform Docker images available at hairyhenderson/gomplate.

If you've gained value out of gomplate and want to find a way to encourage development, please consider sponsoring gomplate!

---

v4.0.0-pre-3 (2024-06-09)

Full Changelog (since v3.11)
Changes since v4.0.0-pre-2

Release Notes

New features and changes

• #1336 Use go-fsimpl to read from datasources
• #1949 Add missing-key flag to manage behavior in case of non-existing key - thanks to @xxxcoltxxx!
• #1922 Flip to ltsc2022 image for windows variant
• #1781 Support for CUE
• #1837 Add support for Linux s390x - thanks to @iblancasa!
• #1700 Add tea.xyz package manager - thanks to @adriens!
• #1698 New plugin args option (implements #1552)
• #1668 Remove support for obsolete Vault app-id auth
• #1440 Added ppc64le support - thanks to @mayurwaghmode!
• #1615 Deprecate netaddr-based funcs
• #1538 Deprecate non-pipelineable strings aliases
• #1433 Removed slim (UPX-compressed) binary/image support
• #1397 Remove BoltDB support
• #1977 Move funcs package to internal/funcs
• #1963 Add --exclude-processing option
• #2068 Switch from zerolog to slog
• #1981 Always suppress empty output
• #2069 coll.Pick now supports slice of strings as input
• #2083 Replaced the data.Data type with a datasource registry
• #2089 Update strings.Indent to error on bad input
• #2095 Removed deprecated RunTemplates function and old Config type
• #2094 Exported new Config type and updated Run function and other API functions

New functions

• #1900 Add support for Ed25519 - thanks to @horvski!
• #1829 feat(funcs): add semver functions - thanks to @wuhuizuo!
• #1589 New coll.Index function
• #1585 Add coll.JQ using gojq library - thanks to @ahochsteger!
• #1587 Add strings.SkipLines function
• #1464 Add coll.GoSlice and deprecate slice alias

Bug fixes

• #1918 Clarify config parsing errors
• #1915 Fix the wdFS behaviour due to Go 1.21.4's changes to IsAbs
• #1916 Fix git+file bug with non-current windows drives
• #1851 Remove defers from main() to stop suppressing panics
• #1835 Make strings.Title not lowercase acronyms - thanks to @zregvart!
• #1643 Fix file descriptor leak
• #1480 Fix nested template panic - thanks to @mihaitodor!

Documentation fixes/updates

• #1868 List release versions for each function, add unreleased label
• #1873 Explain that index supports nested keys - thanks to @isavcic!
• #1885 docs: format user template in hello example - thanks to @tenstad!
• #1866 document the workaround for Vault KV secrets engine - version 2 - thanks to @AndrewSav!
• #1745 Fix output-map usage docs
• #1729 Migrate docs site to new theme
• #1714 Clarified documentation of data.CSVBy* functions - thanks to @giorgiga!
• #1588 Clarify documentation to highlight the differences between index and .
• #1622 Fix minor typo: Pic -> Pick - thanks to @hartzell!
• #1518 Fix incorrect coll.GoSlice example (fixes #1517)
• #1506 Clarify git URL docs - stop implying subpaths starting with '//' are a good idea

Misc. internal contributions

• #1940 test: use t.Setenv to set env vars in tests - thanks to @Juneezee!
• #1828 build: bump github.com/Masterminds/semver to github.com/Masterminds/semver/v3 - thanks to @wuhuizuo!
• #1796 Bump go4.org/unsafe/assume-no-moving-gc for compatibility with Go 1.21 - thanks to @stefanb!
• #1739 Update Makefile to use CGO_ENABLED environment variable - thanks to @sukhil-suresh!