We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PulsarSession session = PulsarContexts.createSession(); LoadOptions options = session.options("-parse -refresh"); options.getEvent().getBrowseEventHandlers().getOnDocumentActuallyReady().addLast((page, driver, other) -> { FeaturedDocument loadDocument = session.parse(page); CheckStatus checkStatus = CheckStatus.SUCCESS; for (SiteStatusCheckHandle handler: handlers) { checkStatus = handler.process(page, loadDocument); if(CheckStatus.SUCCESS != checkStatus){ break; } } System.out.printf(page.getUrl() + " -> " + checkStatus + "\n"); siteCheckService.handleCheckResultForUrl(page.getUrl(), checkStatus); return loadDocument; }); List<String> urls = this.siteMapper.getAllSite().stream().map(siteModel -> siteModel.getUrl()).toList(); session.loadAllAsync(urls, options); session.getContext().await(); session.close();
服务器内存12G,采集检查的 URL 总共才 1600 条,基本上到1000 条左右,就因为内存不够无法执行; 查了一下应该是浏览器的资源没有释放;
不知道我这种写法是不是有什么问题?没有正确释放资源?
The text was updated successfully, but these errors were encountered:
Use session.submit() instead of session.loadAllAsync().
Sorry, something went wrong.
No branches or pull requests
环境
关键业务代码
问题描述
服务器内存12G,采集检查的 URL 总共才 1600 条,基本上到1000 条左右,就因为内存不够无法执行;
查了一下应该是浏览器的资源没有释放;
不知道我这种写法是不是有什么问题?没有正确释放资源?
The text was updated successfully, but these errors were encountered: