Save computed value

[henningWoehr]

Hi there,

I just found out about the feature, that you can use when: false to create a computed value. In my case, I need to calculate it conditionally and save the calculated value. This is because I want to create an encrypted token (at the moment just for renovate) out of a given token (docker, github etc.), which I then want to save, because I can't put it in a seperate file. The conditional calculation is because I want to encrypt the token only if it is given and that would be only at the first copy of the template or if the token expired or similar.

I also thought of jinja_extensions or tasks but these all get executed after the copier-answers file is generated and therfore I can't somehow save my calculated value

[pawamoy]

Sorry, this isn't clear to me. Can you show your copier.yml contents, explain what doesn't work, and what you would expect instead?

[henningWoehr]

Yes of course.

My copier.yaml currently looks like this:

project_type:
  type: str
  help: "PROJECT TYPE: Choose your project type"
  choices:
    Python Poetry: python_poetry
    Docker image extension (extending a base image with additional features): docker_image_extension
    Other: other

schedule:
  type: json
  help: "SCHEDULE: What is your update schedule?"
  default: ["* * * * *"]

docker_username:
  type: str
  help: "DOCKER: Enter your docker username (leave blank if not used)"
  default: schulzbot

docker_token:
  when: "{% if docker_username %} true {% endif %}"
  type: str
  help: "DOCKER: Enter your docker token"
  secret: true
  default: ""

docker_token_encrypted:  # Placeholder
  when: "{% if docker_token %} true {% endif %}"
  default: "{{ docker_token | lower }}"

# ----- misc -----

# _jinja_extensions:
#   - jinja2_workarounds.MultiLineInclude
#   - copier_templates_extensions.TemplateExtensionLoader
#   - copier/checks.py:Checks
#   - copier/context_updater.py:ContextUpdater

_exclude:
  - "*"
  - "!/.github/.copier-answers-renovate-template.yaml"
  - "!renovate.json"

_tasks:
  - "gh label create renovate -c 19D4A8 -d 'This pr got created by renovate' --force"
  - "gh label create major -c 00FFFF -d 'Update of a major dependency' --force"
  - "gh label create dev -c FFFFFF -d 'Update of a dev dependency' --force"

_answers_file: .github/.copier-answers-renovate-template.yaml

My renovate.json:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:recommended"
  ],
  "versioning": "docker",
  "labels": [
    "renovate"
  ],
  "includePaths": [
    "Dockerfile"
  ],
  "packageRules": [
    {
      "matchDatasources": "docker",
      "matchUpdateTypes": [
        "major"
      ],
      "addLabels": [
        "major"
      ],
      "automerge": false
    },
    {
      "matchDatasources": "docker",
      "matchUpdateTypes": [
        "minor",
        "patch",
        "pin",
        "digest"
      ],
      "automerge": true
    }
  ],
  "ignoreTests": true,
  "rebaseWhen": "behind-base-branch",
  "platformAutomerge": true,
  "schedule": ["* * * * *"],
  "hostRules": [
    {
      "hostType": "docker",
      "matchHost": "docker.io",
      "username": "henningwoehr",
      "encrypted": {
        "password": "wcFMA/xDdHCJBTolARAAkwP31qbMKibvaA1ildYWPqR8gwH1ZJ9rX//QJINdCnt/j9mMLzctpb3q7nQVvOhRrmLeBwhO1pzuVSQ9rrxPkRSB3jgr/KPz8c4BFCZli6dwOa7OKQe7QIH9p4HEwna3GVfQccBfAV+IL6qnZHdAZacOWqUvVdjG5tIrPD8WwxPu0h+LXFtRlHSi+px61zEUjH2+Q3JB29eZypXNcw5c5Ocr9LXpl0Gyqs5aDr8pRDVLZLaYIEnvUt2RwG4TKwmLZBUEb/brMkbcgZfBroXJ6N0xOiSU6TkEo/9/jW1imk+NoDVV0BTHuiGCO2BSC5RM3BdN1XG2k+54+IXreRLptQgf+0y5SXC7jDp5y7+PbvvYzMH+2lizQR910My6JioN8CT+PB2hi/Ir7DkZr7anF9WB4uTvRkgfmaXbSYokenatHt7DSAMBE9Ga7QDqjstwIGHvux/Vko8q5gTL8lJxYk2a/0LpkFM+S96x0isuOqur03t06qxD9SHTrJnRGPDis6ZzEUEkZXCtGxQuLv7hrsDMaxSxp+Ksh5QA0rNfgCjBhPqaRWaxM9odTr54+8OvqWJGq/nNBdyLUmYJf9Qhlivck+5xyqY9Rfvef9DLm1yk0FYwyy9+QfvKXNmBeqWmdV1KFMgFWNgawL00T0ue8FYkGjLXXFcxVfPixdGmOKnSUwE8DnsQv0tZ+t8lPw9xILNx5pPN1oM+aFn716QmkUObfDDhNvL3cnlROdzgwrAJZSmZYujcSMNqmb4L/wDVrfzO6zksEkf1Ies2znm3EqDOLmvl"
      }
    }
  ]
}

Now if I enter the docker token, I don't want it in the repository, therefore it is a secret. But in my case, the only way to include the token in the renovate.json is per a encrypted secret, which will be encrypted using a public pgp key either through python or shell. Because the encrypted token is used inside the renovate.json, I need to save the encrypted token in the copier-answers or something else. The docker_token_encrypted question was just a way to get it to work with a computed variable but that doesn't work

[pawamoy]

It's more clear, thank you 🙂 I actually never used secrets in copier.yml so I'm afraid I won't be able to help here, as I'm not sure what would the best way (secure and convenient) to provide the encrypted token to renovatebot.

[henningWoehr]

Security is not really the problem because the docker token gets encrypted so that it can be checked in and only be read by the renovate bot. My problem is that I don't have a way to encrypt (or do something else) the input from the user.

Maybe something like this to change the value of the answer after answering could be helpful if there are more use-cases than just mine:

docker_token:
  when: "{% if docker_username %} true {% endif %}"
  type: str
  help: "DOCKER: Enter your docker token"
  secret: true
  default: ""
  compute: "{{ docker_token | encrypt_with_special_function }}"

Or another idea could be this.

docker_token:
  when: "{% if docker_username %} true {% endif %}"
  type: str
  help: "DOCKER: Enter your docker token"
  secret: true
  default: ""
  
docker_token_encrypted:  # Placeholder
  when: "{% if docker_token %} true {% endif %}"
  save: true
  default: "{{ docker_token | lower }}"

This way you could specify that the value should be saved independent of the outcome of when: ... and only when when: ... is true it get's recalculated.

But if it only made sense for my use case, it might be too much of a hassle.

[pawamoy]

I see you are already using copier-templates-extensions. You could add such a encrypt_with_special_function filter yourself with a custom Jinja extension. Or you could use Copier's tasks somehow.

[henningWoehr]

Yeah I thought of that but if I'm right and use a own filter in the jinja template like {{docker_token | encrypt}} I would need to have the token every time I update. And a workaround with tasks would also be possible if there is nothing more convinient

[henningWoehr]

I found a workaround for my problem:

copier.yaml

project_type:
  type: str
  help: "PROJECT TYPE: Choose your project type"
  choices:
    Python Poetry: python_poetry
    Docker image extension (extending a base image with additional features): docker_image_extension
    Other: other

schedule:
  type: json
  help: "SCHEDULE: What is your update schedule?"
  default: ["* * * * *"]

docker_username:
  type: str
  help: "DOCKER: Enter your docker username (leave blank if not used)"
  default: schulzbot

docker_token:
  when: "{% if docker_username %} true {% endif %}"
  type: str
  help: "DOCKER: Enter your docker token"
  secret: true
  placeholder: "dckr_pat_..."
  validator: "{% if docker_token == '' %} Docker token can't be empty {% endif %}"

# ----- encryption -----
repo:
  when: "{% if docker_token %} true {% endif %}"
  type: str
  help: "What is the owner and the name of this GitHub repo?"
  default: ""
  validator: "{% if not repo | regex_search('\\S+\\/\\S+')%} Repo must be of the format <owner>/<name> {% endif %}"

# ----- misc -----

_jinja_extensions:
  - jinja2_workarounds.MultiLineInclude
  - copier_templates_extensions.TemplateExtensionLoader
  - copier/checks.py:Checks
  - copier/context_updater.py:ContextUpdater

_exclude:
  - "*"
  - "!.github"
  - "!renovate.json"

_tasks:
  - "gh label create renovate -c 19D4A8 -d 'This pr got created by renovate' --force"
  - "gh label create major -c 00FFFF -d 'Update of a major dependency' --force"
  - "gh label create dev -c FFFFFF -d 'Update of a dev dependency' --force"
  - "rm -r copier/ -f"

_answers_file: .copier-answers-renovate-template.yaml

copier/context_updater.py

from copier_templates_extensions import ContextHook

import subprocess
import json
from urllib.request import urlretrieve
import os

import json
from pathlib import Path


DOCKER_TOKEN_ENCRYPTED_FILE = Path("copier/docker_token_encrypted")


def renovate_encrypt(organization: str, repository: str, secret: str):
    data = {
        "o": f"{organization}",
        "r": f"{repository}",
        "v": f"{secret}"
    }

    urlretrieve("https://app.renovatebot.com/renovate.pgp", "renovate.pgp")

    encrypted = subprocess.check_output(
        ["gpg", "--encrypt", "-a", "--recipient-file", "renovate.pgp"],
        text=True,
        input=json.dumps(data))
    encrypted = encrypted.replace("\n", "")
    encrypted = encrypted.removeprefix("-----BEGIN PGP MESSAGE-----")
    encrypted = encrypted.removesuffix("-----END PGP MESSAGE-----")

    os.remove("renovate.pgp")

    return encrypted


class ContextUpdater(ContextHook):
    update = False

    def hook(self, context: dict):
        organization, repository = context.get("repo").split("/")
        docker_token = context["docker_token"]

        if DOCKER_TOKEN_ENCRYPTED_FILE.exists():
            with DOCKER_TOKEN_ENCRYPTED_FILE.open() as f:
                docker_token_encrypted = f.read()
        else:
            docker_token_encrypted = ""

        if context.get("docker_token_encrypted") and context.get("_copier_answers").get("docker_token_encrypted"):
            return

        if docker_token and not docker_token_encrypted:
            docker_token_encrypted = renovate_encrypt(organization, repository, docker_token)
            context["docker_token_encrypted"] = docker_token_encrypted
            DOCKER_TOKEN_ENCRYPTED_FILE.parent.mkdir(parents=True, exist_ok=True)
            with DOCKER_TOKEN_ENCRYPTED_FILE.open("w") as f:
                f.write(docker_token_encrypted)
        elif context.get("docker_token_encrypted") and not docker_token_encrypted:
            docker_token_encrypted = context.get("docker_token_encrypted")

        context["docker_token_encrypted"] = docker_token_encrypted
        context["_copier_answers"]["docker_token_encrypted"] = docker_token_encrypted

In the context updater I do the encrypting and what I found out is that you can add data to context._copier_answers, which will put the encrypted token in the answers file. This will then be available in the context on the next update. One problem is, that the context updater will get called multiple times. For that I will save the encrypted value in a file while rendering and cleaning it up with a task afterwards. That way I only generate the encrypted value only once.

Maybe this discussion is also a bit related to #695 and I could use the solution there in my code or other way around.