Command `statamic:stache:warm` does not warm everything

[faustbrian]

Bug description

While using php artisan statamic:stache:warm on a fairly large repository of ours I noticed one odd or two odd things. Note that I ran into this because I'm debugging awfully slow rendering/caching times, talking 10 seconds plus.

The first thing is that there are 2740 queries (Debugbar) on the first page visit when nothing is cached. Those each take around 20μs but there also hundreds of select * from entries where id = ? queries that take 200μs to 250μs but no biggie, those will only be executed once and we can use php artisan statamic:stache:warm to get warmed up and avoid that they pop up. That's where we noticed something odd and we basically have 2 scenarios in which we can reproduce this:

1. We visit the page without php artisan statamic:stache:warm and see the 2740 queries with a total render time of 11s. On the second visit we only see 125 queries with a total render time of 1s. This is what we would expect since the 2740 queries moved some things into the stache.
2. We run php artisan statamic:stache:warm and then visit the page in hopes of only encountering the 125 queries. We are still seeing 1154 queries with a render time of 6s. On the second visit we only see 125 queries with a total render time of 1s. This was a bit unexpected since we were hoping that php artisan statamic:stache:warm would eliminate all of those queries.

Is there some obvious reason for the difference in how those queries are cached by visiting the page vs. using php artisan statamic:stache:warm in that case? We are seeing 8-14 seconds caching times per page when deploying and after adding some logging we noticed that php artisan statamic:stache:warm not warming everything is part of the problem.

cd /home/forge/statamic.page
git pull origin $FORGE_SITE_BRANCH

$FORGE_COMPOSER install --no-dev --no-interaction --prefer-dist --optimize-autoloader

( flock -w 10 9 || exit 1
    echo 'Restarting FPM...'; sudo -S service $FORGE_PHP_FPM reload ) 9>/tmp/fpmlock

$FORGE_PHP artisan cache:clear
$FORGE_PHP artisan config:clear
$FORGE_PHP artisan route:clear

npm ci
npm run build

$FORGE_PHP artisan statamic:assets:generate-presets --queue
$FORGE_PHP artisan statamic:static:clear
$FORGE_PHP artisan statamic:static:warm --queue
$FORGE_PHP artisan statamic:stache:refresh
$FORGE_PHP artisan statamic:search:update --all

$FORGE_PHP artisan horizon:terminate

How to reproduce

Have a repository with 6 sites and 1200+ pages. I can't provide a reproduction repository due to it being private but could invite a team member to it.

Logs

No response

Environment

Environment
Application Name: Statamic
Laravel Version: 11.34.2
PHP Version: 8.3.15
Composer Version: 2.8.3
Environment: local
Debug Mode: ENABLED
URL: statamic.test
Maintenance Mode: OFF
Timezone: UTC
Locale: en

Cache
Config: CACHED
Events: CACHED
Routes: CACHED
Views: CACHED

Drivers
Broadcasting: null
Cache: file
Database: sqlite
Logs: stack / single
Mail: log
Queue: sync
Session: file

Livewire
Livewire: v3.5.17

Sentry
Enabled: MISSING DSN
Environment: local
Laravel SDK Version: 4.10.1
PHP SDK Version: 4.10.0
Release: NOT SET
Sample Rate Errors: 100%
Sample Rate Performance Monitoring: NOT SET
Sample Rate Profiling: NOT SET
Send Default PII: DISABLED

Statamic
Addons: 4
Sites: 6
Stache Watcher: Disabled
Static Caching: Disabled
Version: 5.42.0 PRO

Statamic Addons
aerni/advanced-seo: 2.9.3
jonassiewertsen/statamic-livewire: 3.8.1
mitydigital/sitemapamic: 3.2.0
spatie/statamic-responsive-images: 5.2.1

Installation

Fresh statamic/statamic site via CLI

Additional details

No response

[Krzemo]

Isn't it related to issue #11323 ?
Maybe you can check PR #11324 I opened and see if it solves it?

[faustbrian]

That doesn't seem to make a difference.

[daun]

Could this be caused by the order of the console commands?

Running statamic:static:warm before statamic:stache:refresh might lead to the first requests being fired before the stache has been rebuilt. On large sites, rebuilding the stache takes quite long, so there might be some advantage to switching those around.

[faustbrian]

The issue can be reproduced by clearing everything and just running php artisan statamic:stache:warm so for reproduction purposes that shouldn't matter.

[SteJW]

Did you ran stache:doctor? If you visit a couple of pages and run it, it will show you indexes that are used in queries, but are not available in stache:warm. You have to configure them in the stache config so stache:warm will know what indexes to create.

[johncarter-]

This could be something to do with spatie/statamic-responsive-images. Because it loads a placeholder svg which is replaced by js on an load/resize/intersect? event, I'm not sure if the commands trigger the image conversions to occur until you actually load the page in a browser? I have often wondered this myself.