Feat/workers#1
Merged
Merged
Conversation
BaseResource._list_all aceita prefetch=N (padrao 1). Cada nova pagina ja dispara o download da proxima em background via asyncio.ensure_future — a latencia da API fica sobreposta ao processamento do consumidor. - _list_all com prefetch em BaseResource - Parametro exposto em todos os list_all* (contratos, contratacoes, atas) - README atualizado na secao Paginacao com diagrama e exemplo de prefetch=0
O bug: remaining era decrementado no finally do worker ANTES de _STOP ser posto na fila. O consumidor via remaining=0 e saia do while, ignorando itens ainda na fila. Troca remaining por contagem de _STOP recebidos: cada worker poe _STOP ao finalizar, e o consumidor conta ate num_workers. prefetch agora: - 0: sequencial - 1: preload simples (1 pagina de antecipacao) - >=2: N workers concorrentes com stride e buffer ordenado
Adiciona test_list_all_with_workers (prefetch=2) e test_list_all_sequential (prefetch=0) para cobrir as 3 estrategias do _list_all router. Cobertura de base.py: 46% → 90% Cobertura total: 90.49%
README agora explica as 3 estrategias: - prefetch=0: sequencial - prefetch=1: preload background (padrao) - prefetch=N: N workers concorrentes com stride e diagrama Docstrings dos resources atualizadas para: Nivel de concorrencia: 0=seq, 1=prefetch, N=N workers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new, configurable prefetch and concurrency mechanism for paginated resource iteration, enabling significant performance improvements when retrieving large datasets. The API is extended to support a
prefetchparameter, allowing users to select between sequential, simple prefetch, or multi-worker concurrent fetching. Documentation and docstrings are updated accordingly, and resource classes are streamlined for clarity.Prefetch and Concurrency for Pagination
prefetchparameter to alllist_all*methods inAtasResource,ContratosResource, andContratacoesResource, allowing users to control the level of concurrency: sequential (0), simple prefetch (1, default), or N workers (N≥2). [1] [2] [3] [4] [5] [6] [7]BaseResource._list_all: sequential, prefetch (background fetch of next page), and concurrent workers (multiple pages fetched in parallel with ordered delivery).Documentation and Code Cleanup
prefetchparameter and its behavior. [1] [2] [3] [4] [5] [6]Internal Refactoring
_STOPsentinel for worker coordination and refactored the code to useasynciomore robustly for background and concurrent fetching. [1] [2]These changes enable much faster data collection and scraping scenarios, while maintaining a simple, backward-compatible API for end users.