hashlookup-server v1.3 released

hashlookup-server v1.3 released (2022-05-21)

New

• [server] /children and /parents end-points added. [Alexandre Dulaunoy]

  The two new endpoints children and parents allow to paginate over the large-set of parents or
  children.

  • The first value is the SHA1 value having children or parents.
  • The second value is the number of elements to get (by default is 100 if
    the value is set to 0).
  • The third value is the cursor to paginate over the element (for
    starting the cursor must be set to 0).

  A sample usage:

  adulau@kolmogorov ~ $ curl -s http://127.0.0.1:5000/children/31C43D24d696BC5F5309CCBFA5BDEF65A7170439/10/0  | jq .
  {
    "children": [
      "003587440172055C75130EF1A063C3BB050C3251",
      "007C1E16B3F0F2E48C114E458308397953C7D224",
      "014D1060C674FBBCEAFFD94B85D60AD00618B56B",
      "01A2FACD61D157FC80DD0C5F6B525CC9EDE4B6DE",
      "01D1A98F559966A05923A74EE239C6BBEEB0FDAC",
      "01D381F2FCDD1BDF642AF83C9E96083F2C8D1C03",
      "02B37BA21D1831C120C1C9C1D41893B4DB424EE7",
      "02DED521ADCF17AA8818EA1142F63E05F558E668",
      "0364E0EFE65D9B6502084813189B4D888C117859",
      "05C9A276A0E03F7A5F99DE5CC8911583FD8FD60E"
    ],
    "cursor": "05C9A276A0E03F7A5F99DE5CC8911583FD8FD60E",
    "total": 774
  }
  adulau@kolmogorov ~ $ curl -s http://127.0.0.1:5000/children/31C43D24d696BC5F5309CCBFA5BDEF65A7170439/10/05C9A276A0E03F7A5F99DE5CC8911583FD8FD60E  | jq .
  {
    "children": [
      "063EC5526DA21372D77AFC3C40F694478521829B",
      "0647EA948ED37383F74CC68A94E2DC3CBC2A9E4E",
      "0648AAAC06A76A58CB1E999882447BBDEEA42C57",
      "06A62F10F269824FFD75A917A35ACD3F2461981C",
      "0727FE9E2437B15B3F879C7617973AE11E55BA13",
      "074A0CA7131AE8FD9665CFE68A0C124EB6AD0170",
      "075B11AE383071BDA9BE66E336C916F6E6E1F49C",
      "081A336DE7D636F95F0150B7708C614592CBBDAE",
      "08DF546EE44D4B7546FCE5A7B7E284CA35F1B059",
      "0947CE713B69C2318CA684BBB63912621CC17A6A"
    ],
    "cursor": "0947CE713B69C2318CA684BBB63912621CC17A6A",
    "total": 774
  }
  

Changes

• [doc] add reference to the RDSv3 importer. [Alexandre Dulaunoy]

• [doc] add reference to the generic NDJSON importer. [Alexandre Dulaunoy]

• [doc] updated. [Alexandre Dulaunoy]

• [doc] API - add /children and /parents example. [Alexandre Dulaunoy]

• [kvrocks] update the kvrocks install script. [Alexandre Dulaunoy]

• [requirements] to the hashlookup version of redis-py. [Alexandre Dulaunoy]

  as hashlookup server relies on kvrocks. The scan command works
  a bit different which causes some issues with the original scan.
  Now we use the fork which supports properly the scan and the
  specific cursor name given by kvrocks.

  ref: apache/kvrocks#417

First release of hashlookup-server version 1.2

hashlookup-server v1.2 (2022-01-15)

hashlookup-server is a minimal and fast open source server (ReST/API) to lookup quickly hash value from large dataset.

New

• [server/api] improved children handling - if too many children are returned it's stripped - and a sample is extracted. [Alexandre Dulaunoy]

• [api] hashlookup:trust added in the output. [Alexandre Dulaunoy]

  The trust level is calculated based on the number of parent to the file.
  If the file has been seen on many sources, the trust level increase.

  The scale of the trust level is between 0 and 100. By default, the trust
  level is 50 meaning we don't know the trust. Below 50, the file is
  suspicious. Above 50, we have evidences that the file is more
  legitimate. The calculation is based on the number of parents seen per
  file. If a file is seen more often in various sources, it increases the
  trust level to reach a maximum of 100.

• [api] /lookup/sha256 api endpoint added. [Alexandre Dulaunoy]

• [api] add a hashlookup:parent-total which indicates the cardinality of the parents. [Alexandre Dulaunoy]

  This can be used for the new API endpoint to paginate over large set of
  parents.

• [api:stats/top] Add a new optional entry to point to get the top 100 of most queried hashes (existing and non-existing) [Alexandre Dulaunoy]

• [api:bulk] add support for pub-sub channel of existing and non-existing hashes. [Alexandre Dulaunoy]

• [feature] session handling added. [Alexandre Dulaunoy]

  A user can now create a session, assign lookup results to a session
  and retrieve the lookup session results in one shot.

  This partially implement feature requested in issue #2 to support
  DFIR sessions.

  Thanks to Koen Van Impe for the idea.

• [pub-sub] add a pub-sub functionality for searched hashes in two different channels: - nx: non-existing hash value - exist: existing hash value. [Alexandre Dulaunoy]

• [statistics] add an optional statistic option in the server to have a sorted set of hashes matching and non-matching. [Alexandre Dulaunoy]

• [hashlookup-server] initial import of the code. [Alexandre Dulaunoy]

  • This includes a simple HTTP server for doing bulk and lookup of hashes.
  • A simple DNS server to do lookup via DNS
  • Various import script for NSRL

  This works on a test instance.

  TODO:

  • Automatic script for NSRL download and import
  • Bloomfilter export
  • Improved documentation

Changes

• [server] black -S all the code. [Alexandre Dulaunoy]

• [server] update server description. [Alexandre Dulaunoy]

• [import-poc] never flush. [Alexandre Dulaunoy]

• [api] expose the total keys of a hashlookup server. [Alexandre Dulaunoy]

• [server] add auth header in pub-sub. [Alexandre Dulaunoy]

• [api/info] now return the default stats from the new importer. [Alexandre Dulaunoy]

• [config] don't flush db by default. [Alexandre Dulaunoy]

• [stats/top] remove recently added hashes from previously nx hash. [Alexandre Dulaunoy]

• [api/md5] data sources with MD5 only hashes. [Alexandre Dulaunoy]

• [api] md5 lookup updated to allow MD5 only records. [Alexandre Dulaunoy]

• [doc] added keys related to packages. [Alexandre Dulaunoy]

• [req] added. [Alexandre Dulaunoy]

• [server] add children hashes if these exist. [Alexandre Dulaunoy]

• [api] lookup add parent details. [Alexandre Dulaunoy]

• [doc] children added. [Alexandre Dulaunoy]

• [doc] parent added. [Alexandre Dulaunoy]

• [doc] add the default keys used. [Alexandre Dulaunoy]

• [doc] document basic storage format. [Alexandre Dulaunoy]

• [doc] add the import stat key. [Alexandre Dulaunoy]

• [poc-import] JSON updated to add legacy and JSON file jqified. [Alexandre Dulaunoy]

• [requirements] remove standard lib python modules. [Alexandre Dulaunoy]

• [install] kvrocks default repo and latest release. [Alexandre Dulaunoy]

Fix

• [api/stats] existing hash value from nx removed. [Alexandre Dulaunoy]

• [dns] records are now too large for single hashlookup record. [Alexandre Dulaunoy]

  reduce to some fields and the HTTP interface the fall-back to get more
  info

• [api/lookup/sha1] missing parents bug. [Alexandre Dulaunoy]

• [api:md5/sha1] large set of parents are now limited and give a random selection. [Alexandre Dulaunoy]

• [api/md5] if there is already more data in default SHA1 we use that one and not the fall-back MD5 lookup. [Alexandre Dulaunoy]

• [api:bulk] add proper check of MD5 and SHA1 value before further processing. [Alexandre Dulaunoy]

• [req] moved. [Alexandre Dulaunoy]

• [api] typo fixed. [Alexandre Dulaunoy]

• [api] fix ttl missing bug. [Alexandre Dulaunoy]

Other

• Merge branch 'main' of github.com:adulau/hashlookup-server into main. [Alexandre Dulaunoy]

• Merge branch 'main' of github.com:adulau/hashlookup-server into main. [Alexandre Dulaunoy]

• Update DATABASE.md. [Alexandre Dulaunoy]

• Merge pull request #8 from wllm-rbnt/fix. [Alexandre Dulaunoy]

  Fix jq parse error on special chars

• Fix jq parse error on special chars. [William Robinet]

• Update DATABASE.md. [Alexandre Dulaunoy]

• Merge branch 'main' of github.com:adulau/hashlookup-server into main. [Alexandre Dulaunoy]

• Update README.md. [Alexandre Dulaunoy]

• Typo fixed. [Alexandre Dulaunoy]

• Merge pull request #4 from cudeso/main. [Alexandre Dulaunoy]

  PoC to better streamline the import of NSRL data.

• Support for import of NSRL datasets in ISO and ZIP format. [Koen Van Impe]

• Merge branch 'main' of https://github.com/cudeso/hashlookup-server into main. [Koen Van Impe]

• Merge pull request #1 from cudeso/main. [Alexandre Dulaunoy]

  Update README.md

• PoC for streamlining import. [Koen Van Impe]

  PoC to better streamline the import of NSRL data.
  Still requires some work but basic concept works.
  Currently only tested with Android

• Update README.md. [Koen Van Impe]