WIP: Caching spacy doc #93
Conversation
…. We had broken the connection between doc and student with the reorder and were pulling in errors from inactive students. This presents a fix for both things.
…t we had. We had broken the connection between doc and student with the reorder and were pulling in errors from inactive students. This presents a fix for both things." This reverts commit 9f96e25.
Add menu dropdown and status button
Lastdocfix
… contents to writing analysis.
… contents to writing analysis.
Lastdocfix
…tarting to move rosters to use new API
* Fixing the manifest issue for permissions. * Fixing manifest to reflect new version and to remove permissions that we do not make use of.
…erver Dealing with existing conflicts dfuring merge.
* Caching Version 0 Initial Commit * Caching Version 0 Initial Commit With Fixes * Caching Version 0 Initial Commit With Fixes * Caching Version 0 Initial Commit With Fixes * Refactoring Caching Code * Modularize Caching function, add helper functions to util.py * Improve modularization and reducing dirty reads. * Renamed functions & variables, added parameter definitions * Reduce gap between cache read and write * Remove blank lines --------- Co-authored-by: Bradley Erickson <[email protected]>
Merge Master into Caching
| @@ -129,7 +131,7 @@ def process_text(text, options=None): | |||
| return results | |||
|
|
|||
|
|
|||
| async def process_texts_serial(texts, options=None): | |||
| async def process_texts_serial(texts, doc=None, options=None): | |||
There was a problem hiding this comment.
Currently unsued method. This will be removed in the future with the communication protocol.
This can stay as is.
| @@ -169,7 +171,7 @@ def run_in_fork(func): | |||
| os._exit(0) | |||
|
|
|||
|
|
|||
| async def process_texts_parallel(texts, options=None): | |||
| async def process_texts_parallel(texts, doc=None, options=None): | |||
There was a problem hiding this comment.
Currently unsued method. This will be removed in the future with the communication protocol.
This can stay as is.
| @@ -280,18 +286,28 @@ async def process_and_cache_missing_features(unfound_features, found_features, r | |||
| :param writing: The writing data. | |||
| :return: The updated writing data. | |||
| """ | |||
| # Calculate the features required and prepare the temporary cache dict | |||
There was a problem hiding this comment.
compute_nlp_features_needed()
add_to_our_needed_nlp_features_queue()
while await_running_processes():
random_delay
compute_nlp_features_we_need_and_queue_needs()
update_cache_to_tell_it_were_running_and_what_were_running()
run_needed_features()
update_cache()
returnThere was a problem hiding this comment.
prior person wanted: A and D
brad: I want B and C
sarthak: want C and D and E
Brad is running on the system running features B and C.
Then you make your request for C and D and E
compute_nlp_features_needed - return C, E since it found D from the prior
we add C, E to the queue of features needing to be run - add_to_our_needed_nlp_features_queue
Wait for any blocking features (i.e. brad running B and C) . 0-100ms
compute_nlp_features_we_andthe_queue_needs - Only returns E since C is covered from Brad's run
We start running E, telling the cache we are running E and then actually running E.
Update with E and return the items we need,
DrLynch
force-pushed
the
caching_spacy_doc
branch
from
95980a2
to
25b987d
Compare
No description provided.