Use grep to extract from file for testing with PyFunceble

[spirillen]

This example is taken from one of my own use-cases where i would like to extract domains from EasyList and then test them against already added (known) and therefore blocked items, this will create a tested output of records that's needs to be added to my own RPZ lists.

The first step is to determine the regex string to be used for extracting the domains from easylist/easylist_thirdparty.txt which is mark by ||domain.name.tld^or ||domain.name.tld^$third-party

This is done with the following regex string

\|\|.*\.cloudfront\.net\^($|\$third-party)

A bit of explaining for the different stuff in the line

1. \|\| as the | by it self means or we need to add the backslash \ to make it a literal pipe |
2. .* is a wildcard in regex as of the dot . is the wildcard and the star * mean any number of occurrences
3. \. This time we tells regex the dot (otherwise wildcard) is literally a dot . as it is used for splitting the domain levels.
4. \^ In regex a ^ means the beginning of or starts with we needs to make it a literal with a backslash since the occurrence of the ^ most be there and it is not a regex function.
   1. ($|\$third-party) is a grouping containing a OR argument (the |)
   2. Since the lines we would like to find can end in two ways we first match any line ending with the ^ that is done by saying ($ (In regex the dollar sign means end of string)
   3. We now add the OR sign |
   4. \$third-party) This is the alternative ending of the lines we like to extract matching literal ^$third-party)

You can see the above code in action on the following two online examples

1. https://regexper.com/#%5C%7C%5C%7C.*%5C.cloudfront.net%5C%5E%28%24%7C%5C%24third-party%29
2. https://regex101.com/r/rU6Roe/1

Now that we have developed out regex string, we will use it in combination with grep
The file from which we would like to extract the cloudfront.net spam domains is:

    $ grep -E '\|\|.*\.cloudfront\.net\^($|\$third-party)' easylist/easylist_thirdparty.txt

The -E in grep mean extended regex...

Now we finally parse the grep + regex to the pyfunceble command

    $ pyfunceble --adblock --database-type mariadb -d $(grep -iE '\|\|.*\.cloudfront.net\^($|\$third-party)' easylist/easylist_thirdparty.txt)

Notice that we now uses the -d (domain) parser of our records and not the -f (file) parser argument as we used grep to extract from the file.

In this example we have bypassed the usage of the --filter option, as it would have given us unwanted results like ||d3rp5jatom3eyn.cloudfront.net^$domain=~my.na

That's it, Have fun and I hoped you found more to use pyfunceble too and alternative ways of usage.

[funilrys]

Okay, so you don't want the --filter to filter the decoded domain but the read line. Interesting.

If we implement this it should be with a new argument.

[spirillen]

Yes I like the --filter however, as shown in the example, then the --filter would also extract domains, which is undesirable. IE. I know @nextdns is using a similar regex to import only the first-party (||example.com^) blocked records, whereas I also imported the ^$thirdparty as unwanted ware is unwanted ware despite how it served

If you uses the --filter option you would end up blocking stuff which should only have been blocked by a URI and not as domain 😃

[funilrys]

@spirillen, Mhh are you saying that the decoder itself should be better?

Is it related to #13 (not a priority right now) ?

[spirillen]

Yes, the example is to exactly avoid what @dnmTX is referring to with his example in #13 (comment)

basically you saying that api.google.com will be blocked?

What if you have let's say: ||yahoo.com/papi/action$popup so the end result will be:
0.0.0.0 yahoo.com in ACTIVE folder.

That why I think this usage example is good, and why I made it. Since the adblocking apps are so different and far more "advanced" than domain / hosts blacklists

[funilrys]

Touches #218.

[funilrys]

Is this still an open thing @spirillen ?

[spirillen]

It's a show and tell 😉 not an issue