count() issue when using instance_of()

[Karmak23]

Hi, I hit this strange behaviour:

In [22]: k.feeds.instance_of(MailFeed)
Out[22]: [<MailFeed: MailFeed “Tweets envoyés via mail” of user karmak23 (#6667)>]

In [23]: k.feeds.instance_of(MailFeed).count()
Out[23]: 0

In [24]: len(k.feeds.instance_of(MailFeed))
Out[24]: 1

MailFeed is a child of BaseFeed. Obviously, count() should return 1.

The strangest thing is that it doesn't happen with another child class of the same level (RssAtomFeed, to name it) :

In [25]: k.feeds.instance_of(RssAtomFeed).count()
Out[25]: 233

In [26]: len(k.feeds.instance_of(RssAtomFeed))       
Out[26]: 233

Where should I head to find if the issue comes from my code or polymorphic ? Do you have any hint of what could cause this strange behavior ?

[BertrandBordage]

Hi @Karmak23,

First, is the issue recurrent or did it just happened once and never came back again?
If it happened once, it may be due to a concurrent write in the database.
Otherwise, run the following and paste us the output:

from django.db import connection
connection.queries = []
k.feeds.instance_of(MailFeed).count()
len(k.feeds.instance_of(MailFeed))
print(connection.queries)

[Karmak23]

Interesting…
Yesterday it happened consistently during all my testing time (say ~1 hour).
Now, testing again, it don't happen anymore and the result is constantly 1. I can revert my patch, but don't understand why.

The question raised by the concurrent write is “how the heck ?”. The mailfeed has been in the database for days. How can this be a concurrent write make it “disappear” temporarily from count() but not from all() ?? I just wish to better understand whats going on at the database level, in the django layer, or both.

BTW, I ran your code and print(connection.queries) returned []. The machine is not in DEBUG mode.

Thanks a lot,

[BertrandBordage]

Are you using an ORM caching tool like django-cache-machine, django-cacheops, johnny-cache or django-cachalot?

Could you run it again using DEBUG = True?

[Karmak23]

Yes, well guessed, I use cacheops.

Given how long the count() returned 0, it seems very probable cacheops cached the result (it is configured to cache for a week on feeds). The right count was returned simply after the cache expired. The more I think about it, the more it seems obvious cacheops is not aware of polymorphic and it didn't invalidate the cache count in this particular condition. I should obviously do it myself.

I'll will run it with DEBUG as soon as possible today, and most probably close this issue after reporting the result.

Thanks a lot for your hints @BertrandBordage

[BertrandBordage]

Don’t look further: django-cacheops and django-cache-machine rely a lot on timeout. They often return stale results and don’t handle ridiculously common queries, such as COUNTs (for cache-machine) and M2M updates (cacheops). That’s exactly why I developed cachalot…

This issue can be closed in my opinion.

[BertrandBordage]

… or multi-table inheritance for cacheops (Suor/django-cacheops#31), the exact reason why you’re experiencing this problem (since polymorphic is based on multi-table inheritance).

[Karmak23]

Yep.
I didn't know about cachalot and I'm currently studying switching to it.
Thanks.