v1.28.0

New features

• Support for captures with a headed browser (setting allow_headed). Only possible if lacus is running in a graphical environment. (see below for details)
• Complete rewrite of the indexing system, allows pagination (transparent on upgrade, but may take a while, be careful if you have a big instance)
• Index IPs and TLDs
• Link hostname and IPs indexes, expose it on the UI

• Trigger CIRCL Passive DNS wherever possible (on IPs and on Hostnames)

• Allow to share direct links to IP/Hostnames/Favicon/... modals

• Push capture to another Lookyloo instance (typically, a headed capture done locally to a central repository)

• Major overhaul of the categorization, use MISP Taxonomy by default (dark-web)

• Optionally makes modules admin only (avoid using all the tokens available on 3rd party APIs)
• Enable full text search on indexes when relevant
• Optional auto-submit of onion URLs to an AIL instance

Changes

• MISP export improvements
• Refactoring and improvements in the javascripts, massive speedups
• Render indexes with AJAX
• Render hostnode popup much faster by moving indexes to a subsequent page
• Submit any file to Pandora
• Trigger optional auto-report in background script
• Allow user accessible MISP servers
• Remove RiskIQ module (killed by microsoft)
• Support KV Rocks 2.11 (major speed improvements)
• Many improvements in logging, reduce noise, and use WARNING more appropriately
• Improve favicon rendering
• Update URLHaus module as it now requires an auth key

Fixes

• Re-enable submit to pandora buttons
• Deduplicate notifications
• Many, many other bugfixes

Notes for headed captures

The classical use of Lookyloo is to have it running on a server with no graphical interface (no X/Wayland server). The capture with Playwright uses a headless browser, runs some interactions on the page (see PlaywrightCapture for details), and finishes after a certain amount of time and/or no traffic. This method is good enough most of the time, but all the interactions on the page are predefined and cannot be modified by the user triggering the capture.

If you use a dedicated lacus instance, please refer to the lacus release for details first. And you need the following to trigger a headed capture:

• The configuration setting "allow_headed" = True in config/generic.json
• Lookyloo installed on a machine with a graphical interface (or at least lacus, if you're using a remote instance)
• [Web UI] On the capture page, in Capture configuration -> untick Use headless browser (it it's missing, check config/generic.json and restart the app)
• [Web UI] Optionally, on the capture page, in Capture configuration -> set Max capture time to a time, if needed (90s by default)
• [API] Pass headless set to False in the capture settings
• [API] Optionally general_timeout_in_sec set to the amount of time you want to interact with the page (it is set to 90 by default)

The new headed capture mode opens a full browser configured with the settings passed to the capture, and none of the predefined interactions. It lets the user interact with the page for a set amount of time (general_timeout_in_sec in the API, or Max capture time on the web interface), stops the capture, and store the result as usual. It is mostly helpful to manually bypass captchas and other techniques used by websites to detect bots.

mondial.webm

PRs

What's Changed

• change redis to valkey by @sebdraven in #1028

New Contributors

• @sebdraven made their first contribution in #1028

Full Changelog: v1.27.0...v1.28.0

v1.27.0

This release requires some system upgrades:

• Valkey 8.0+
• Kvrocks 2.10+
• Python 3.9+

Changes

• Improve documentation on capture page
• Speedup on-demand stop of archiver script
• Make all remaining indexes paginated
• Use new indexes to speedup rendering of hostnode popups and all views relying on indexes
• Maintenance and bugfixes

For Lacus, LacusCore and PlaywrightCapture changes, see Lacus release notes.

Full Changelog: v1.26.1...v1.27.0

v1.26.1

New features

• Major improvements in indexes, allows to paginate hits based on capture timestamp
• Add index for TLDs
• Transparent migration of urls, hostnames, ressources, HTTP Headers Hashes,and cookies indexes to new format
• Run many more DNS requests (MX, SOA, NS)
• Use new indexes on web interface, speedup rendering
• Optionally disable JavaScript during capture

What's Changed

• build(deps): bump docker/build-push-action from 5 to 6 by @dependabot in #939

Full Changelog: v1.26.0...v1.26.1

v1.26.0

Lacus, LacusCore, and PlaywrightCapture changes

• See Lacus v 1.11.0 release notes for details
• Many bugfixes and improvements
• Use more recent browsers

Har2Tree

Full Changelog: Lookyloo/har2tree@v1.25.0...v1.26.0

• Improve handling of embedded content
• Properly generate docs

Lookyloo

Full Changelog: v1.25.0...v1.26.0

• Improve typing for capture settings (Pydantic)
• Make it possible to categorize captures
• Improve error handling, logging
• Index categorized captures
• Get Captures via UUID in the API
• Add controller to Start/Stop scripts individually
• Add CSP HTTP headers whenever possible

What's Changed

• putting the login message in the right block by @adrima01 in #924
• Categories by @AntoniaBK in #925
• new: get uuids by category via API by @adrima01 in #926
• add new endpoint to remove capture by @jeroengui in #929

New Contributors

• @jeroengui made their first contribution in #929

v1.25.0 - Pass the Salt 2024

New features

• Endpoint to remove capture from the index (safely copies it in a directory)
• Configurable way to gather contact information for takedown of malicious websites
• FuzzImy hash based on HTML structure of the rendered page (algorithm of CERT PL)
• New simple capture page without any of the options
• Optional user config to overwrite capture settings and/or have default parameters
• Download tree as PNG (as long as the tree isn't too big)
• Optional auto-report on capture page (admin only)
• Optionally change the index page to the capture page, useful when the index is very big and takes a long time to load

Changes

• Many improvements when rendering panels on tree
• New menus
• Vast amount of bugfixes and improvements everywhere in the project by @AntoniaBK and @adrima01, see below.
• Support for valkey, new kvrocks, Ubuntu 24.04
• Optionally disable SRI validation while developing
• Partial removal of jQuery
• Many improvements in loading index, uses caching much more efficiently
• Restart webserver more often, avoids memory leaks
• Improve logging, reduces noise

What's Changed

• Lookup of abuse-c by @AntoniaBK in #904
• Remove redundant clause by @AntoniaBK in #906
• adding uwhoisd installation by @adrima01 in #907
• Fix: DataTable rename to treeHashesTable by @AntoniaBK in #909
• adding 3rd party report to mail by @adrima01 in #908
• Menu by @AntoniaBK in #910
• changes so that you can ignore the sri while developing by @adrima01 in #911
• Simple interface by @adrima01 in #912
• fix: removing unnecessary script and jQuery by @adrima01 in #913
• new: downloadable tree as png by @adrima01 in #915
• New: admin-only checkbox for auto-report by @AntoniaBK in #914
• fix: [modules] Gracefully accept no hashlookup fixes #916 by @cvandeplas in #917
• Fixing typo by @adrima01 in #918
• New: upload a capture via the API by @AntoniaBK in #919
• Recent captures by @adrima01 in #921
• Update generic.json.sample by @adrima01 in #923

New Contributors

• @cvandeplas made their first contribution in #917

Full Changelog: v1.24.0...v1.25.0

v1.24.0

New features

• Optionally attempt to allow tracking on capture, see lacus v1.9.0 release for details.
• [Admin Only] Index all captures, not only the public ones (in kvrocks instead of redis)
• Multiple improvements in correlation pages (Favicons, ressources
• Index favicons
• Compute favicons MM3H and like it to Shodan
• Search and index captcha IDs (reCaptcha, hCaptcha and Cloudflare)

What's Changed

• Changed misp_url by @adrima01 in #894
• Proxy with VT module by @DocArmoryTech in #897
• Module response added by @adrima01 in #898
• Made send_email available from the API by @AntoniaBK in #899
• Speedup async capture checks when the backlog in long
• Improve favicon rendering on tree
• Split capture building and indexing in two different scripts
• Reduce memory usage for indexing scripts

New Contributors

• @adrima01 made their first contribution in #894
• @DocArmoryTech made their first contribution in #897
• @AntoniaBK made their first contribution in #899

Full Changelog: v1.23.0...v1.24.0

v1.23.0

What's Changed

• New: autoreport to investigation team from API

• New: Archiving old captures to a S3 bucket (well over 2M on the production instance)

• New CIRCL Passive DNS module

• Many perf improvements

• Refactor indexing module

• Display IDNA domain when possible

• Improve statistics

• Improve lacus monitoring

• Fix docker image

• Python 3.12 support

• Strict typing

• Maintenance, dependencies updates, bugfixes

• PlaywrightCapture changelog

• LacusCore changelog

Full Changelog: v1.22.0...v1.23.0

Hack.lu 2023

If you haven't been upgrading lookyloo since the last tagged release, this release contains a lot of changes.

New features

• Support for HTTP Headers Hashing
• Support for archiving capture on S3FS, this is not completely implemented, but we use it on the public instance. Expect it to be usable for anyone in next release.
• Store HARs dumps in gzip archives
• MISP export when the capture downloads a file
• Handle captures where we have a rendered HTML and a no-click download is triggered in Javascript (TODO: support multiple downloads triggered in a single capture)
• Get downloaded file via the API
• Fetch favicons using default URL, and HTML content
• Support multiple MISP instances for submission and lookup

Changes

• Better rendering of the capture time
• Support re-processing captures that were mistakenly considered broken
• Improve logging
• Improve caching
• Auto-restart webservice to avoid memory leak
• Strip URL to capture (space and new line)
• Update dependencies, new browsers

Bugfixes

• Allow to run multiple backgroung indexing scripts
• Many related to the compression of the HAR and the pickles to reduce disk use
• Various encoding issues with rendered HTML

For more details regarding the captures, see Lacus v1.7.0 release notes.

Pass The Salt 2023

New Features

• Allow to pass a timezone, geolocation coordinates, locale, and color scheme to a capture
• Add a global proxy option in the settings
• Improve SMTP auth for notifications

Changes

• Store the capture settings in order to reuse them later (like for re-capture)
• Avoid failing if Lacus isn't available ant retry a few times

Bugfixes

• Properly handle captures with errors, improve logging accordingly
• Resubmit captures if they were deleted on Lacus without storing a response (generally if something crashed)

Spring release

This release is the outcome of a good two months of work on Lookyloo itself but also Lacus and its dependencies leading to the v1.5.0 release.

It also improves the support for the monitoring interface (still to be considered beta).

New features

• Compare captures via the API
• Submit any for to Pandora (if available)
• Allow automatic reporting via the API
• Can set an email to notify in the monitoring form

Changes

• Improve handling of long running processes,
• Improve logging all over the place
• Changes related to Lacus/LacusCore/PyLacus changes
• Easy way to check if two captures are different or not
• Store capture settings in the capture directory for potential later use
• Show proxy in UI if one was given
• Improve response when comparing captures

Bugfixes

• Avoid issues when the pickle requires too much recursivity
• Cloudflare services was always flagging URLs as their own
• The usual batch of bugfixes all over the place