Skip to content

DocIDServer contains more than configured maxPagesToFetch Url count #413

New issue
New issue
Open
#430
Open
DocIDServer contains more than configured maxPagesToFetch Url count#413
#430
@Mideen

Description

@Mideen
Mideen
opened on Oct 17, 2019 · edited by Mideen

If we configured maxPagesToFetch, There is no need to call the ShouldVisit method and Creation of DocID for that URL if the limit is reached. It takes unwanted memory consumption.

Activity

dgoiko
linked a pull request that will close this issue on Jan 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @Mideen

      Issue actions
        DocIDServer contains more than configured maxPagesToFetch Url count · Issue #413 · yasserg/crawler4j