Make it possible to run filters without type formatting

[frikky]

Hey,

I was wondering if the detection of raw data and its type could be done automatically?
We're currently doing a lot of things with Liquid, and there's therefore a lot of edgecases showing up.

Here's a basic example of a problem related to strings:

# Does not work
{{ this is a test | slice: 1 }}

# Does work
{{ 'this is a test' | slice: 1 }}

# Does not work
{{ 'this 'is a test | slice: 1 }}

# Does work
{{ 'this \'is a test' | slice: 1 }}

# Does not work
{{ 'this
is a test' | slice: 1 }}

# Does work
{{ 'this\nis a test' | slice: 1 }}

The reason this is a problem is because the data being sent into a filter can change form, based on usecase, or the end-user. Sometimes there are quotes. Sometimes there are newlines. Sometimes it's JSON. Other times HTML. etc.

Do you have some way to find e.g. what is in between {{ -> | automatically? This is especially prominent before the first filter on our side. We have built automatic parsers for this, but I'm just wondering if it's been thought about from your side at all, or if we should try to fork and find a way to do it.

[pwwang]

It's great to see some edge-cases, as solving them will make liquidpy better.

Regarding your request, if I am understanding correctly, you're looking for an implicit transformation of the first item within {{ }}, wherein your transformed data will replace the raw data.

In this regard, I have a few questions:

• Would you also require the same transformation for {% %}?
• If we implement this feature, you will have the ability to use your own transform function, right? For instance, I would not have to worry about whether to convert 1 to an integer or "1".
• What if a variable passed in? For example, {{ var | slice: 1}}. Should we treat var as a variable, or "var" literally?

[pwwang]

This function will not be included in liquidpy itself, but you can write an extension to implement it. Here is an example to make {{a' b | upper}} work, with output "A' B ". But without the extension, we need {{"a' b" | upper}}

import re
from jinja2.ext import Extension
from liquid import Liquid

# Your transform function, here I just quote everything to make `a' b` to `"a' b"`
# You may need more sophisticated transform function to make it valid template syntax
transform = lambda x: f'"{x}"'


class TransformExtension(Extension):
    
    # Since the raw data could be arbitrary, for example `a' b`, which is not a valid jinja token,
    # we can't implement this in the APIs after lexing, for example, filter_stream()
    def preprocess(self, source, name, filename=None) -> str:
        source = re.sub( 
            # Replace anything between `{{` and `|` or `}}` with the transformed data
            # Again, this is just a rough version, you may have to consider cases 
            # such as {{ "a|b" | upper}}, where you don't want to break up `a` and `b`
            r"(?<=\{\{)(.*?)(?=\||\}\})", 
            lambda m: transform(m.group(1)), 
            source,
        )
        return super().preprocess(source, name, filename)


liq = Liquid(
    "{{a' b | upper}}", 
    # Enable the extension
    extensions=[TransformExtension], 
    from_file=False,
)
# Or you can load the extension after instantiation:
# liq = Liquid(...)
# liq.env.add_extension(TransformExtension)
liq.render()

[frikky]

It's great to see some edge-cases, as solving them will make liquidpy better.

Regarding your request, if I am understanding correctly, you're looking for an implicit transformation of the first item within {{ }}, wherein your transformed data will replace the raw data.

In this regard, I have a few questions:

• Would you also require the same transformation for {% %}?
• If we implement this feature, you will have the ability to use your own transform function, right? For instance, I would not have to worry about whether to convert 1 to an integer or "1".
• What if a variable passed in? For example, {{ var | slice: 1}}. Should we treat var as a variable, or "var" literally?

Thanks for the prompt reply!

We have indeed implemented custom filters, custom parsing and such already, and adding a Transformextension as you're showing shouldn't be a any problem at all. I know more edgecases for typing will show up, but our goal is to help do all that automatically for our users, as a lot of them aren't used to dealing with types, and it long-term makes for a much better experience

I'll try this out in the next few days <3