www/squid - UT1 blacklist not showing category #4392

gobiursus · 2024-12-10T19:53:00Z

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[X ] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
[X ] I have searched the existing issues, open and closed, and I'm convinced that mine is new.
[X ] The title contains the plugin to which this issue belongs

Describe the bug
A clear and concise description of what the bug is, including last known working version (if any).

I was following this tutorial to set up web filtering on my OPNsense instance: https://docs.opnsense.org/manual/how-tos/proxywebfilter.html
But UT1 blacklist is not showing any category after I successfully download and apply the list.
Please help!

Expected behavior
A clear and concise description of what you expected to happen.
Show the categorizes

Environment
Software version used and hardware type if relevant.

OPNsense 24.7.10_2-amd64
QEMU Virtual CPU version 2.5+ (4 cores, 4 threads)
Network Mellanox ConnectX-3

vnguyen1 · 2024-12-11T05:09:15Z

Yes, I have the same issue here.
And now, when I enable the Remote Blacklist UT1, I can't start the Proxy Service.
Error Code:

Starting squid.
CPU Usage: 4.279 seconds = 4.201 user + 0.079 sys
Maximum Resident Size: 1317024 KB
Page faults with physical i/o: 0
2024/12/11 16:08:15| Processing Configuration File: /usr/local/etc/squid/squid.conf (depth 0)
2024/12/11 16:08:15| Starting Authentication on port 127.0.0.1:3128
2024/12/11 16:08:15| Disabling Authentication on port 127.0.0.1:3128 (interception enabled)
2024/12/11 16:08:15| Starting Authentication on port [::1]:3128
2024/12/11 16:08:15| Disabling Authentication on port [::1]:3128 (interception enabled)
2024/12/11 16:08:15| Starting Authentication on port 127.0.0.1:3129
2024/12/11 16:08:15| Disabling Authentication on port 127.0.0.1:3129 (interception enabled)
2024/12/11 16:08:15| Starting Authentication on port [::1]:3129
2024/12/11 16:08:15| Disabling Authentication on port [::1]:3129 (interception enabled)
2024/12/11 16:08:19| ERROR: '.ai-nude.adult' is a subdomain of '.adult'
2024/12/11 16:08:19| ERROR: You need to remove '.ai-nude.adult' from the ACL named 'remoteblacklist_UT1'
2024/12/11 16:08:19| Not currently OK to rewrite swap log.
2024/12/11 16:08:19| storeDirWriteCleanLogs: Operation aborted.
2024/12/11 16:08:19| FATAL: Bungled /usr/local/etc/squid/squid.conf line 56: acl remoteblacklist_UT1 dstdomain "/usr/local/etc/squid/acl/UT1"
2024/12/11 16:08:19| Squid Cache (Version 6.10): Terminated abnormally.
/usr/local/etc/rc.d/squid: WARNING: failed to start squid

droumanet · 2024-12-20T08:19:45Z

Confirmed here too.
A (weird) workaround is to create multiples lists one by one.

By the way, what are requirement for using these functionalities? My FW become unable to pass any frame after a few moment and RAM seems 90% busy in dashboard.
My hardware is 12GB RAM, 250GB disk and Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (4 cores, 4 threads)

AUSK1LL9 · 2024-12-21T03:54:09Z

Can confirm, I am having the same issue. However I can not get any list to work. Even simplified list like cooking or adult. I have tried FTp HHTTPS. Recycled service, deleted and remade. Removed and re-added os-squid plugin. Same issue.
squid-6.12
OPNsense 24.7.11_2-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15

Logs:

droumanet · 2024-12-27T22:23:38Z

I can use university of Toulouse list... but on OPNSense 24.1 version.
Problem occurs with version 24.7.x
Is there a regression?

AdSchellevis · 2024-12-28T08:31:14Z

Could be the size of the list, easiest way to debug this is by using the script manually (/usr/local/opnsense/scripts/proxy/fetchACLs.py)

By the way, the OPNProxy plugin uses the same list but has a better approach in handling the contents, static lists within squid have strict sorting requirements which is one of the main bottlenecks in this feature.

w9hdg · 2024-12-31T23:19:38Z

I just tried calling the above mentioned script manually, it ran for a minute or two before exiting with no output.

gobiursus · 2025-01-01T00:35:42Z

Please fix this issue in next release!

AdSchellevis · 2025-01-01T09:48:59Z

feel free to open a pull request if there's anything to fix....

droumanet · 2025-01-01T11:26:12Z

/usr/local/opnsense/scripts/proxy/fetchACLs.py

Thanks for the tips. On my firewall, I've got a full page of errors.
`/usr/local/opnsense/scripts/proxy/fetchACLs.py
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 716, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/local/lib/python3.11/http/client.py", line 1303, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.11/http/client.py", line 1349, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output
self.send(msg)
File "/usr/local/lib/python3.11/http/client.py", line 996, in send
self.connect()
File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x241749633450>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 802, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 594, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='dsi.ut-capitole.fr', port=80): Max retries exceeded with url: /blacklists/download/games.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x241749633450>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/opnsense/scripts/proxy/fetchACLs.py", line 381, in
main()
File "/usr/local/opnsense/scripts/proxy/fetchACLs.py", line 325, in main
for filename, basefilename, file_ext, line in acl.download():
File "/usr/local/opnsense/scripts/proxy/fetchACLs.py", line 153, in download
self.fetch()
File "/usr/local/opnsense/scripts/proxy/fetchACLs.py", line 83, in fetch
req = requests.get(**req_opts)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='dsi.ut-capitole.fr', port=80): Max retries exceeded with url: /blacklists/download/games.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x241749633450>: Failed to establish a new connection: [Errno 61] Connection refused'))
`
What should I do to help more?

AdSchellevis · 2025-01-01T13:12:23Z

@droumanet looks like dsi.ut-capitole.fr is unreachable, from here I also can't reach their location.

debtodevnull · 2025-01-02T22:03:41Z

The UT1 blacklist seems reachable again and the issue is still there (using OPNSense 24.7.11_2) :

abonanni · 2025-01-03T12:09:47Z

👍

Categories are generated in /usr/local/etc/squid/acl/UT.index :

{"adult": "adult", "agressif": "agressif", "arjel": "arjel", "astrology": "astrology", "audio-video": "audio-video", "bank": "bank", "bitcoin": "bitcoin", "blog": "blog", "celebrity": "celebrity", "chat": "chat", "child": "child", "cleaning": "cleaning", "cooking": "cooking", "dangerous_material": "dangerous_material", "dating": "dating", "dialer": "dialer", "drogue": "drogue", "educational_games": "educational_games", "filehosting": "filehosting", "financial": "financial", "forums": "forums", "gambling": "gambling", "games": "games", "hacking": "hacking", "jobsearch": "jobsearch", "lingerie": "lingerie", "liste_blanche": "liste_blanche", "liste_bu": "liste_bu", "malware": "malware", "manga": "manga", "marketingware": "marketingware", "mixed_adult": "mixed_adult", "mobile-phone": "mobile-phone", "phishing": "phishing", "press": "press", "publicite": "publicite", "radio": "radio", "reaffected": "reaffected", "redirector": "redirector", "remote-control": "remote-control", "sect": "sect", "sexual_education": "sexual_education", "shopping": "shopping", "social_networks": "social_networks", "sports": "sports", "strict_redirector": "strict_redirector", "strong_redirector": "strong_redirector", "translation": "translation", "tricheur": "tricheur", "warez": "warez", "webmail": "webmail", "special": "special", "download": "download", "ddos": "ddos", "update": "update", "associations_religieuses": "associations_religieuses", "shortener": "shortener", "cryptojacking": "cryptojacking", "exceptions_liste_bu": "exceptions_liste_bu", "vpn": "vpn", "stalkerware": "stalkerware", "doh": "doh", "examen_pix": "examen_pix", "tricheur_pix": "tricheur_pix", "residential-proxies": "residential-proxies", "fakenews": "fakenews"}

But no category is available in web interface.

gobiursus · 2025-01-24T19:20:13Z

👍

Categories are generated in /usr/local/etc/squid/acl/UT.index :

{"adult": "adult", "agressif": "agressif", "arjel": "arjel", "astrology": "astrology", "audio-video": "audio-video", "bank": "bank", "bitcoin": "bitcoin", "blog": "blog", "celebrity": "celebrity", "chat": "chat", "child": "child", "cleaning": "cleaning", "cooking": "cooking", "dangerous_material": "dangerous_material", "dating": "dating", "dialer": "dialer", "drogue": "drogue", "educational_games": "educational_games", "filehosting": "filehosting", "financial": "financial", "forums": "forums", "gambling": "gambling", "games": "games", "hacking": "hacking", "jobsearch": "jobsearch", "lingerie": "lingerie", "liste_blanche": "liste_blanche", "liste_bu": "liste_bu", "malware": "malware", "manga": "manga", "marketingware": "marketingware", "mixed_adult": "mixed_adult", "mobile-phone": "mobile-phone", "phishing": "phishing", "press": "press", "publicite": "publicite", "radio": "radio", "reaffected": "reaffected", "redirector": "redirector", "remote-control": "remote-control", "sect": "sect", "sexual_education": "sexual_education", "shopping": "shopping", "social_networks": "social_networks", "sports": "sports", "strict_redirector": "strict_redirector", "strong_redirector": "strong_redirector", "translation": "translation", "tricheur": "tricheur", "warez": "warez", "webmail": "webmail", "special": "special", "download": "download", "ddos": "ddos", "update": "update", "associations_religieuses": "associations_religieuses", "shortener": "shortener", "cryptojacking": "cryptojacking", "exceptions_liste_bu": "exceptions_liste_bu", "vpn": "vpn", "stalkerware": "stalkerware", "doh": "doh", "examen_pix": "examen_pix", "tricheur_pix": "tricheur_pix", "residential-proxies": "residential-proxies", "fakenews": "fakenews"}

But no category is available in web interface.

You are right, no categorize on web interface.

AdSchellevis · 2025-01-25T08:29:23Z

probably this opnsense/core#8180

w9hdg mentioned this issue Dec 31, 2024

Squid Web Proxy Service won't start when Remote Blacklist is enable. #4398

Open

abonanni added a commit to teicee/plugins that referenced this issue Jan 3, 2025

ref opnsense#4392 - fix squid black list not showing categories

cb18e84

This was referenced Jan 3, 2025

ref #4392 - fix squid black list not showing categories #4440

Open

Regression in src/opnsense/mvc/app/models/OPNsense/Base/FieldTypes/JsonKeyValueStoreField.php opnsense/core#8180

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

www/squid - UT1 blacklist not showing category #4392

www/squid - UT1 blacklist not showing category #4392

gobiursus commented Dec 10, 2024

vnguyen1 commented Dec 11, 2024

droumanet commented Dec 20, 2024

AUSK1LL9 commented Dec 21, 2024 •

edited

Loading

droumanet commented Dec 27, 2024

AdSchellevis commented Dec 28, 2024

w9hdg commented Dec 31, 2024 •

edited

Loading

gobiursus commented Jan 1, 2025

AdSchellevis commented Jan 1, 2025

droumanet commented Jan 1, 2025

AdSchellevis commented Jan 1, 2025

debtodevnull commented Jan 2, 2025

abonanni commented Jan 3, 2025 •

edited

Loading

gobiursus commented Jan 24, 2025

AdSchellevis commented Jan 25, 2025

www/squid - UT1 blacklist not showing category #4392

www/squid - UT1 blacklist not showing category #4392

Comments

gobiursus commented Dec 10, 2024

vnguyen1 commented Dec 11, 2024

droumanet commented Dec 20, 2024

AUSK1LL9 commented Dec 21, 2024 • edited Loading

droumanet commented Dec 27, 2024

AdSchellevis commented Dec 28, 2024

w9hdg commented Dec 31, 2024 • edited Loading

gobiursus commented Jan 1, 2025

AdSchellevis commented Jan 1, 2025

droumanet commented Jan 1, 2025

AdSchellevis commented Jan 1, 2025

debtodevnull commented Jan 2, 2025

abonanni commented Jan 3, 2025 • edited Loading

gobiursus commented Jan 24, 2025

AdSchellevis commented Jan 25, 2025

AUSK1LL9 commented Dec 21, 2024 •

edited

Loading

w9hdg commented Dec 31, 2024 •

edited

Loading

abonanni commented Jan 3, 2025 •

edited

Loading