HybridAnalysis: switch POST /search/hash to GET and add overview fallback

Author: pranavshukla

.github/pull_request_template.md

@@ -26,8 +26,8 @@ Please delete options that are not relevant.
     - [ ] Check if it could make sense to add that analyzer/connector to other [freely available playbooks](https://intelowlproject.github.io/docs/IntelOwl/usage/#list-of-pre-built-playbooks).
     - [ ] I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
     - [ ] If the plugin interacts with an external service, I have created an attribute called precisely `url` that contains this information. This is required for Health Checks (HEAD HTTP requests). 
-    - [ ] If the plugin requires mocked testing, `_monkeypatch()` was used in its class to apply the necessary decorators.
-    - [ ] I have added that raw JSON sample to the `MockUpResponse` of the `_monkeypatch()` method. This serves us to provide a valid sample for testing.
+    - [ ] If a new analyzer has beed added, I have created a unittest for it in the appropriate dir. I have also mocked all the external calls, so that no real calls are being made while testing.
+    - [ ] I have added that raw JSON sample to the `get_mocker_response()` method of the unittest class. This serves us to provide a valid sample for testing. 
     - [ ] I have created the corresponding `DataModel` for the new analyzer following the [documentation](https://intelowlproject.github.io/docs/IntelOwl/contribute/#how-to-create-a-datamodel)
 - [ ] I have inserted the copyright banner at the start of the file: ```# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl # See the file 'LICENSE' for copying permission.```
 - [ ] Please avoid adding new libraries as requirements whenever it is possible. Use new libraries only if strictly needed to solve the issue you are working for. In case of doubt, ask a maintainer permission to use a specific library.

.github/workflows/codeql-analysis.yml

@@ -65,7 +65,7 @@ jobs:
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v3.28.0
+      uses: github/codeql-action/init@v4.31.0
       with:
         languages: python
         # Override the default behavior so that the action doesn't attempt
@@ -93,4 +93,4 @@ jobs:
     #   make release
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v3.28.0
+      uses: github/codeql-action/analyze@v4.31.0

.github/workflows/pull_request_automation.yml

@@ -115,6 +115,9 @@ jobs:
       - name: Run test
         run: |
           docker exec intelowl_uwsgi coverage run manage.py test --keepdb tests
+      - name: Run async tests
+        run: |
+          docker exec intelowl_uwsgi coverage run manage.py test --keepdb async_tests
 
   frontend-tests:
     runs-on: ubuntu-latest

.github/workflows/scorecard.yml

@@ -59,7 +59,7 @@ jobs:
       # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
       # format to the repository Actions tab.
       - name: "Upload artifact"
-        uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
         with:
           name: SARIF file
           path: results.sarif
@@ -68,6 +68,6 @@ jobs:
 
       # Upload the results to GitHub's code scanning dashboard.
       - name: "Upload to code-scanning"
-        uses: github/codeql-action/upload-sarif@5b6e617dc0241b2d60c2bccea90c56b67eceb797 # v2.22.11
+        uses: github/codeql-action/upload-sarif@8d77149e0c9e2199ac9cfc90c9e15116f5c69c48 # v2.22.11
         with:
           sarif_file: results.sarif

README.md

@@ -64,7 +64,7 @@ You can see the full list of all available analyzers in the [documentation](http
 
 As open source project maintainers, we strongly rely on external support to get the resources and time to work on keeping the project alive, with a constant release of new features, bug fixes and general improvements.
 
-Because of this, we joined [Open Collective](https://opencollective.com/intelowl-project) to obtain non-profit equal level status which allows the organization to receive and manage donations transparently. Please support IntelOwl and all the community by choosing a plan (BRONZE, SILVER, etc).
+Because of this, we joined [Open Collective](https://opencollective.com/intelowl-project) to obtain US and EU non-profit equal level status which allows the organization to receive and manage donations transparently and with tax exemption. Please support IntelOwl and all the community by choosing a plan (BRONZE, SILVER, etc).
 
 <a href="https://opencollective.com/intelowl-project/donate" target="_blank">
   <img src="https://opencollective.com/intelowl-project/donate/[email protected]?color=blue" width=200 />
@@ -78,7 +78,7 @@ Because of this, we joined [Open Collective](https://opencollective.com/intelowl
 
 [Certego](https://certego.net/?utm_source=intelowl) is a MDR (Managed Detection and Response) and Threat Intelligence Provider based in Italy.
 
-IntelOwl was born out of Certego's Threat intelligence R&D division and is constantly maintained and updated thanks to them.
+IntelOwl was born out of Certego's Threat intelligence R&D division and is mostly maintained and updated thanks to them.
 
 #### The Honeynet Project
 

api_app/analyzables_manager/filters.py

@@ -0,0 +1,18 @@
+import rest_framework_filters as filters
+from django_filters.widgets import QueryArrayWidget
+
+from api_app.analyzables_manager.models import Analyzable
+
+
+class CharInFilter(filters.BaseInFilter, filters.CharFilter):
+    pass
+
+
+class AnalyzableFilter(filters.FilterSet):
+    name = CharInFilter(widget=QueryArrayWidget)
+
+    class Meta:
+        model = Analyzable
+        fields = {
+            "discovery_date": ["lte", "gte"],
+        }

api_app/analyzables_manager/models.py

@@ -1,3 +1,4 @@
+import logging
 from typing import Type, Union
 
 from django.core.exceptions import ValidationError
@@ -7,17 +8,14 @@
 
 from api_app.analyzables_manager.queryset import AnalyzableQuerySet
 from api_app.choices import Classification
-from api_app.data_model_manager.models import (
-    BaseDataModel,
-    DomainDataModel,
-    FileDataModel,
-    IPDataModel,
-)
+from api_app.data_model_manager.models import BaseDataModel
 from api_app.data_model_manager.queryset import BaseDataModelQuerySet
 from api_app.defaults import file_directory_path
 from api_app.helpers import calculate_md5, calculate_sha1, calculate_sha256
 from certego_saas.models import User
 
+logger = logging.getLogger(__name__)
+
 
 class Analyzable(models.Model):
     name = models.CharField(max_length=255)
@@ -82,23 +80,11 @@ def get_all_user_events_data_model(
                     ).values_list("data_model__pk", flat=True)
                 )
             query |= query2
+        logger.debug(f"{query=}")
         return self.get_data_model_class().objects.filter(query)
 
     def get_data_model_class(self) -> Type[BaseDataModel]:
-        if self.classification == Classification.IP.value:
-            return IPDataModel
-        elif self.classification in [
-            Classification.URL.value,
-            Classification.DOMAIN.value,
-        ]:
-            return DomainDataModel
-        elif self.classification in [
-            Classification.HASH.value,
-            Classification.FILE.value,
-        ]:
-            return FileDataModel
-        else:
-            raise NotImplementedError()
+        return self.CLASSIFICATIONS.get_data_model_class(self.classification)
 
     def _set_hashes(self, value: Union[str, bytes]):
         if isinstance(value, str):
@@ -111,6 +97,10 @@ def _set_hashes(self, value: Union[str, bytes]):
             self.sha1 = calculate_sha1(value)
 
     def clean(self):
+        if self.file:
+            self.classification = Classification.FILE.value
+        else:
+            self.classification = Classification.calculate_observable(self.name)
         if self.classification == Classification.FILE.value:
             from api_app.analyzers_manager.models import MimeTypes
 

api_app/analyzables_manager/queryset.py

@@ -10,14 +10,21 @@ class AnalyzableQuerySet(QuerySet):
     def visible_for_user(self, user):
 
         from api_app.models import Job
+        from api_app.user_events_manager.models import UserAnalyzableEvent
 
-        analyzables = (
+        analyzables_job = (
             Job.objects.visible_for_user(user)
             .values("analyzable")
             .distinct()
             .values_list("analyzable__pk", flat=True)
         )
-        return self.filter(pk__in=analyzables)
+        analyzables_ue = (
+            UserAnalyzableEvent.objects.visible_for_user(user)
+            .values("analyzable")
+            .distinct()
+            .values_list("analyzable__pk", flat=True)
+        )
+        return self.filter(pk__in=analyzables_job) | self.filter(pk__in=analyzables_ue)
 
     def create(self, *args, **kwargs):
         obj = self.model(**kwargs)

api_app/analyzables_manager/serializers.py

@@ -1,12 +1,64 @@
-from rest_framework.serializers import ModelSerializer
+import logging
+
+from rest_framework import serializers as rfs
 
 from api_app.analyzables_manager.models import Analyzable
+from api_app.choices import Classification
+from api_app.models import Job
 from api_app.serializers.job import JobRelatedField
 
+logger = logging.getLogger(__name__)
+
 
-class AnalyzableSerializer(ModelSerializer):
+class AnalyzableSerializer(rfs.ModelSerializer):
     jobs = JobRelatedField(many=True, read_only=True)
 
     class Meta:
         model = Analyzable
         fields = "__all__"
+        read_only_fields = [
+            "jobs",
+            "discovery_date",
+            "md5",
+            "classification",
+            "sha256",
+            "sha1",
+            "mimetype",
+        ]
+
+    def to_representation(self, instance):
+        logger.debug(f"{instance=}")
+        analyzable = super().to_representation(instance)
+        job = (
+            Job.objects.filter(id__in=analyzable["jobs"])
+            .order_by("-finished_analysis_time")
+            .first()
+        )
+        user_event_data_model = (
+            instance.get_all_user_events_data_model().order_by("-date").first()
+        )
+        if not job and not user_event_data_model:
+            analyzable["last_data_model"] = None
+            return analyzable
+        elif (job and job.data_model) or user_event_data_model:
+            if not job or not job.data_model:
+                last_data_model = user_event_data_model
+            elif not user_event_data_model:
+                last_data_model = job.data_model
+            else:
+                if (job and job.data_model) and (
+                    job.data_model.date > user_event_data_model.date
+                ):
+                    last_data_model = job.data_model
+                else:
+                    last_data_model = user_event_data_model
+
+            serializer_class = Classification.get_data_model_class(
+                classification=analyzable["classification"],
+            ).get_serializer()
+            analyzable["last_data_model"] = serializer_class(last_data_model).data
+        return analyzable
+
+    def create(self, validated_data):
+        instance, _ = self.Meta.model.objects.get_or_create(**validated_data)
+        return instance

api_app/analyzables_manager/views.py

@@ -1,14 +1,76 @@
+import logging
+from http import HTTPStatus
+
 from rest_framework import viewsets
+from rest_framework.decorators import action
+from rest_framework.exceptions import ValidationError
 from rest_framework.permissions import IsAuthenticated
+from rest_framework.response import Response
 
+from api_app.analyzables_manager.filters import AnalyzableFilter
+from api_app.analyzables_manager.models import Analyzable
 from api_app.analyzables_manager.serializers import AnalyzableSerializer
+from api_app.serializers.job import JobAnalyzableHistorySerializer
+from api_app.user_events_manager.serializers import (
+    UserAnalyzableEventSerializer,
+    UserDomainWildCardEventSerializer,
+    UserIPWildCardEventSerializer,
+)
+
+logger = logging.getLogger(__name__)
 
 
 class AnalyzableViewSet(viewsets.ReadOnlyModelViewSet):
 
     serializer_class = AnalyzableSerializer
     permission_classes = [IsAuthenticated]
+    queryset = Analyzable.objects.all()
+    filterset_class = AnalyzableFilter
 
     def get_queryset(self):
         user = self.request.user
         return super().get_queryset().visible_for_user(user)
+
+    @action(detail=True)
+    def history(self, request, pk=None):
+        user = request.user
+        try:
+            analyzable: Analyzable = self.get_queryset().get(pk=pk)
+        except Analyzable.DoesNotExist:
+            raise ValidationError({"detail": "Requested analyzable does not exist."})
+
+        jobs_queryset = analyzable.jobs.visible_for_user(user).order_by(
+            "-finished_analysis_time"
+        )
+        user_events_queryset = analyzable.user_events.visible_for_user(user).order_by(
+            "-date"
+        )
+        user_domain_wildcard_events_queryset = (
+            analyzable.user_domain_wildcard_events.visible_for_user(user).order_by(
+                "-date"
+            )
+        )
+        user_ip_wildcard_events_queryset = (
+            analyzable.user_ip_wildcard_events.visible_for_user(user).order_by("-date")
+        )
+
+        jobs = JobAnalyzableHistorySerializer(jobs_queryset, many=True).data
+        user_events = UserAnalyzableEventSerializer(
+            user_events_queryset, many=True
+        ).data
+        user_domain_wildcard_events = UserDomainWildCardEventSerializer(
+            user_domain_wildcard_events_queryset, many=True
+        ).data
+        user_ip_wildcard_events = UserIPWildCardEventSerializer(
+            user_ip_wildcard_events_queryset, many=True
+        ).data
+
+        return Response(
+            status=HTTPStatus.OK.value,
+            data={
+                "jobs": jobs,
+                "user_events": user_events,
+                "user_domain_wildcard_events": user_domain_wildcard_events,
+                "user_ip_wildcard_events": user_ip_wildcard_events,
+            },
+        )