feat: Add Redis caching for navigation pagination #484

YuriZmytrakov · 2025-10-01T18:46:14Z

Related Issue(s):

Page navigation "previous" rel #439

Description:

Add Redis caching support for navigation pagination to enable proper prev/next links in STAC API responses.

PR Checklist:

Code is formatted and linted (run pre-commit run --all-files)
Tests pass (run make test)
Documentation has been updated to reflect changes, if applicable
Changes are added to the changelog

jonhealy1

Great stuff here. Just a few thoughts, requests.

jonhealy1 · 2025-10-06T05:14:52Z

stac_fastapi/core/setup.py

    "pygeofilter~=0.3.1",
    "jsonschema~=4.0.0",
    "slowapi~=0.1.9",
+    "redis==6.4.0",


Let's move this to extras so people can do this pip install stac-fastapi-core[redis] if they need redis. Some may not want to install the library if they are not using redis.

I have moved redis into extras, thank you!

jonhealy1 · 2025-10-06T05:18:46Z

stac_fastapi/elasticsearch/setup.py

    "elasticsearch[async]~=8.18.0",
    "uvicorn~=0.23.0",
    "starlette>=0.35.0,<0.36.0",
+    "redis==6.4.0",


I think we can remove redis from elasticsearch, opensearch and sfeos_helpers

jonhealy1 · 2025-10-06T05:18:55Z

stac_fastapi/opensearch/setup.py

    "opensearch-py[async]~=2.8.0",
    "uvicorn~=0.23.0",
    "starlette>=0.35.0,<0.36.0",
+    "redis==6.4.0",


I think we can remove redis from elasticsearch, opensearch and sfeos_helpers

jonhealy1 · 2025-10-06T05:19:09Z

stac_fastapi/sfeos_helpers/setup.py


 install_requires = [
    "stac-fastapi.core==6.5.1",
+    "redis==6.4.0",


remove here too

jonhealy1 · 2025-10-06T05:21:04Z

README.md

+
+> [!NOTE]
+> Use either the Sentinel configuration (`REDIS_SENTINEL_HOSTS`, `REDIS_SENTINEL_PORTS`, `REDIS_SENTINEL_MASTER_NAME`) OR the Redis configuration (`REDIS_HOST`, `REDIS_PORT`), but not both.
+


We should add a REDIS_ENABLED global env var to turn on/off redis. The try/except blocks can add latency to the application if someone doesn't want to use Redis and the api keeps trying to connect.

I have introduced REDIS_ENABLED env var that is used to indicate if redis has been configured and enabled.

jonhealy1 · 2025-10-06T05:22:58Z

stac_fastapi/core/stac_fastapi/core/core.py


+        current_url = str(request.url)
+        redis = None
+        try:


before the try we should check to see if REDIS_ENABLED is true.

Fixed, thank you!

jonhealy1 · 2025-10-06T05:23:26Z

stac_fastapi/core/stac_fastapi/core/core.py

            HTTPException: If there is an error with the cql2_json filter.
        """
        base_url = str(request.base_url)
+        try:


here too, check REDIS_ENABLED

Fixed, thank you!

jonhealy1 · 2025-10-06T05:23:53Z

stac_fastapi/core/stac_fastapi/core/core.py

        """
        base_url = str(request.base_url)
+        try:
+            redis = await connect_redis_sentinel()


connect_redis here too I think?

Fixed, thank you!

jonhealy1 · 2025-10-06T05:30:47Z

stac_fastapi/tests/api/test_api.py

+
+    result_missing = await get_prev_link(mock_redis, "dummy_token_2")
+    assert result_missing is None
+    mock_redis.get.assert_awaited_once_with("nav:self:dummy_token_2")


@YuriZmytrakov Should we have integration testing with a live redis server? We would need to set up redis in gh actions too. I think it's a good idea.

Thank you for the recommendation, I have added integration tests in this commits, and its using a live redis server as suggested. Github Actions have been configured too
e04fa57
601a8a4

jonhealy1 · 2025-10-06T05:34:37Z

dockerfiles/Dockerfile.dev.es

 RUN pip install --no-cache-dir -e ./stac_fastapi/core
 RUN pip install --no-cache-dir -e ./stac_fastapi/sfeos_helpers
 RUN pip install --no-cache-dir -e ./stac_fastapi/elasticsearch[dev,server]
+RUN pip install --no-cache-dir redis types-redis


do we need this? we should create a new compose.yml file and add redis to it for local testing I think - compose-redis.yml or something like that.

Here is a compose config for a redis cache

redis: image: redis:alpine ports: - "6379:6379" volumes: - redis_data:/data volumes: redis_data:

This was fixed, thank you @jonhealy1 . Thank you for the suggested code @jamesfisher-geo .

jamesfisher-geo · 2025-10-06T17:44:56Z

stac_fastapi/core/stac_fastapi/core/core.py

        links = await PagingLinks(request=request, next=next_token).get_links()

+        collection_links = []
+        if (


Is this needed? I believe at least the parent link is automatically created by the API with https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/blob/main/stac_fastapi/core/stac_fastapi/core/models/links.py#L125

Yes, unfortunately this was reported as a bug, the parent and collection relations are missing. For this reason, I’m resolving it here. This was the original issue for this ticket, which eventually evolved into the Redis cache for navigation project.

jamesfisher-geo · 2025-10-06T17:47:19Z

stac_fastapi/core/stac_fastapi/core/core.py

+            await save_self_link(redis, next_token, self_link)
+
+            prev_link = await get_prev_link(redis, token_param)
+            if prev_link:


How do you feel about handling the previous paging link additions inside the PagingLinks class? This would be more maintainable from a code perspective.

You could also pass the previous link into get_links with the extra_links parameter?

https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/blob/main/stac_fastapi/core/stac_fastapi/core/models/links.py#L92

Good point, I tried this approach as well. However, I’d need to pass the Redis connection into the PagingLinks class. Also, the token for the previous link is derived from the current token, which isn’t available in PagingLinks it’s only accessible in the current link that I retrieve in the endpoint where caching is enabled. Hope this makes sense.

jamesfisher-geo · 2025-10-06T17:49:08Z

dockerfiles/Dockerfile.dev.es

 RUN pip install --no-cache-dir -e ./stac_fastapi/core
 RUN pip install --no-cache-dir -e ./stac_fastapi/sfeos_helpers
 RUN pip install --no-cache-dir -e ./stac_fastapi/elasticsearch[dev,server]
+RUN pip install --no-cache-dir redis types-redis


Here is a compose config for a redis cache

redis: image: redis:alpine ports: - "6379:6379" volumes: - redis_data:/data volumes: redis_data:

**Related Issue(s):** - stac-utils#461 **Description:** - GET `/collections` collection search fields extension ex. `/collections?fields=id,title`. [stac-utils#465](stac-utils#465) - Improved error messages for sorting on unsortable fields in collection search, including guidance on how to make fields sortable. [stac-utils#465](stac-utils#465) - Added field alias for `temporal` to enable easier sorting by temporal extent, alongside `extent.temporal.interval`. [stac-utils#465](stac-utils#465) **PR Checklist:** - [x] Code is formatted and linted (run `pre-commit run --all-files`) - [x] Tests pass (run `make test`) - [x] Documentation has been updated to reflect changes, if applicable - [x] Changes are added to the changelog

**Related Issue(s):** - stac-utils#468 **Description:** **PR Checklist:** - [x] Code is formatted and linted (run `pre-commit run --all-files`) - [x] Tests pass (run `make test`) - [x] Documentation has been updated to reflect changes, if applicable - [x] Changes are added to the changelog

**Related Issue(s):** - stac-utils#460 **Description:** ex. `/collections?q=Sentinel-2a` **PR Checklist:** - [x] Code is formatted and linted (run `pre-commit run --all-files`) - [x] Tests pass (run `make test`) - [x] Documentation has been updated to reflect changes, if applicable - [x] Changes are added to the changelog

**Related Issue(s):** - None **Description:** - Release v6.4.0 **PR Checklist:** - [x] Code is formatted and linted (run `pre-commit run --all-files`) - [x] Tests pass (run `make test`) - [x] Documentation has been updated to reflect changes, if applicable - [x] Changes are added to the changelog

YuriZmytrakov · 2025-10-08T13:19:16Z

@jonhealy1 @jamesfisher-geo apologies, I have to close this PR and cherry-pick the changes into a new one. This PR has been sitting here for a while, and any change now triggers too many conflicts to resolve.
New PR: #488

dummy

2a8dd32

YuriZmytrakov force-pushed the CAT-1382 branch from 107d343 to 88cee0d Compare October 1, 2025 18:47

YuriZmytrakov changed the title ~~Cat 1382~~ feat: Add Redis caching for navigation pagination Oct 1, 2025

YuriZmytrakov marked this pull request as draft October 1, 2025 19:10

YuriZmytrakov force-pushed the CAT-1382 branch from 88cee0d to d7e2f95 Compare October 1, 2025 19:44

YuriZmytrakov marked this pull request as ready for review October 1, 2025 19:48

jonhealy1 requested changes Oct 6, 2025

View reviewed changes

jonhealy1 requested a review from jamesfisher-geo October 6, 2025 05:38

jamesfisher-geo requested changes Oct 6, 2025

View reviewed changes

YuriZmytrakov force-pushed the CAT-1382 branch 5 times, most recently from 42d6c98 to c056704 Compare October 8, 2025 09:50

jonhealy1 and others added 6 commits October 8, 2025 12:08

docs: Add redis configuration documentation and changelog

a4ae475

fix: apply recommendations

64bf85b

YuriZmytrakov force-pushed the CAT-1382 branch from a9c440e to 64bf85b Compare October 8, 2025 10:13

YuriZmytrakov closed this Oct 8, 2025


		> [!NOTE]
		> Use either the Sentinel configuration (`REDIS_SENTINEL_HOSTS`, `REDIS_SENTINEL_PORTS`, `REDIS_SENTINEL_MASTER_NAME`) OR the Redis configuration (`REDIS_HOST`, `REDIS_PORT`), but not both.

feat: Add Redis caching for navigation pagination #484

feat: Add Redis caching for navigation pagination #484

Uh oh!

Conversation

YuriZmytrakov commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonhealy1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonhealy1 Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YuriZmytrakov commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YuriZmytrakov commented Oct 1, 2025 •

edited

Loading

jonhealy1 Oct 6, 2025 •

edited

Loading

YuriZmytrakov commented Oct 8, 2025 •

edited

Loading