Issues

• Switch to boto3

  datalad/datalad-crawler#142 · Filter by author yarikoptic opened on Jun 6, 2024
  1

• s3: make use of start-after available for list_objects_v2

  datalad/datalad-crawler#132 · Filter by author yarikoptic opened on May 8, 2023

• datalad core testing started to fail

  datalad/datalad-crawler#124 · Filter by author yarikoptic opened on Aug 15, 2022
  2

• save versions .json state file reflecting the actual state, not the "target"

  datalad/datalad-crawler#123 · Filter by author yarikoptic opened on Aug 11, 2022

• add option to periodically cause saveing of the result

  datalad/datalad-crawler#122 · Filter by author yarikoptic opened on Aug 11, 2022

• Equivalent of datalad install with datalad crawler

  question
  datalad/datalad-crawler#114 · Filter by author N-HEDGER opened on Nov 15, 2021
  1

• Update Travis config to use new codecov uploader

  datalad/datalad-crawler#110 · Filter by author jwodder opened on Oct 13, 2021

• xnat: automate query to establish a new provider for non-public project

  datalad/datalad-crawler#101 · Filter by author yarikoptic opened on Apr 20, 2021
  3

• 'CommandError' object has no attribute 'failed'

  datalad/datalad-crawler#91 · Filter by author yarikoptic opened on Mar 4, 2021

• Discontinue use of Repo.add()

  datalad/datalad-crawler#83 · Filter by author mih opened on Oct 13, 2020

• s3: does not detect complete removal of a key

  datalad/datalad-crawler#74 · Filter by author yarikoptic opened on Apr 13, 2020

• s3 crawler -- doesn't handle "gracefully" situation of completely removed keys in a versioned bucket

  datalad/datalad-crawler#73 · Filter by author yarikoptic opened on Mar 24, 2020

• Support (store) ETag if remote (http) resource provides it

  datalad/datalad-crawler#69 · Filter by author yarikoptic opened on Feb 11, 2020

• Add digester to enrich git annex metadata for every key

  enhancement
  datalad/datalad-crawler#65 · Filter by author yarikoptic opened on Nov 15, 2019
  1

• NDA crawler fails with error

  datalad/datalad-crawler#56 · Filter by author loj opened on Sep 24, 2019
  8

• Crawling multiple web pages

  datalad/datalad-crawler#54 · Filter by author joeyzhou98 opened on Aug 28, 2019
  2

• --is-pipeline relative import error

  bug
  datalad/datalad-crawler#52 · Filter by author joeyzhou98 opened on Aug 27, 2019
  3

• datalad crawl: Changing behaviour between HCP900/1200

  datalad/datalad-crawler#48 · Filter by author TobiasKadelka opened on Jul 16, 2019
  1

• Credential lookup failure crashes s3 crawler

  datalad/datalad-crawler#46 · Filter by author TobiasKadelka opened on Jul 11, 2019
  2

• Support split archives

  datalad/datalad-crawler#39 · Filter by author yarikoptic opened on Mar 4, 2019

• using the datalad crawler for LORIS

  datalad/datalad-crawler#34 · Filter by author thomasbeaudry opened on Jan 21, 2019
  3

• Generic framework for crawling data providers with versions

  enhancement
  datalad/datalad-crawler#22 · Filter by author yarikoptic opened on Nov 1, 2018
  3

• "Reproducible" crawled datasets

  datalad/datalad-crawler#21 · Filter by author yarikoptic opened on Nov 1, 2018

• figshare crawler

  datalad/datalad-crawler#20 · Filter by author yarikoptic opened on Nov 1, 2018

• RF: move crawl-init into crawl making crawl accept additional params to be passed into pipeline

  good first issue
  datalad/datalad-crawler#19 · Filter by author mih opened on Oct 28, 2018