Things for ingest: Verify format of zephir_ingested_items.tsv.gz; reconsider handling for groove_full.tsv.gz

[aelkiss]

This is for both groove_full.tsv.gz and zephir_ingested_items.tsv.gz.

The format appears to be:

aeu.ark:/13960/t30305j3b        AEU     AEU     ia      ia.cihm_76481

I think that's an HTID, a campus(?) id, a collection id, a digitization source, and an optional IA ID, but we should double-check that.

I don't recall what the difference between the two items is -- maybe groove_full.tsv.gz is things that Zephir has bib data for, but aren't ingested? @cscollett could confirm.

In any case, it would probably make sense reconsider where we handle some of these things (https://github.com/hathitrust/feed_internal/blob/main/bin/jobs/feed.monthly/zephir_diff.pl#L32) and if/where we do verification.

Especially for groove_full.tsv.gz -- given that post-zephir processing is just passing it along, and ingest already knows how to connect to the zephir ftps (https://github.com/hathitrust/feed_internal/blob/main/bin/jobs/feed.daily/GetZephirFiles.pm). This may have to be rethought when we use S3, but it should be easy enough to add an event-driven workflow in ingest as well for the files it cares about.

[aelkiss]

Clarifications:

groove_full.tsv.gz comes from Zephir (it is groove_export_${zephir_date}.tsv.gz).
zephir_ingested_items.tsv.gz is generated by post-zephir processing as a result of processing the incoming json from Zephir, so we should verify the format there.

[cscollett]

groove full is everything we have without an indication it's ingested yet. groove incremental is new things that are not ingested yet. The fields are: htid, source, collection code, digitizing source, and the AI scan id.

[aelkiss]

Thanks -- I will have to look at what's going on with groove_full.tsv.gz because the one I checked only has 6,777,967 lines, which makes me think it's only the non-ingested stuff. There might be some other processing going on on our end.

[cscollett]

Yes, that's right. We have a field where we record when we get notice an item is in the volume repository. But some people give us their whole catalog, with items that will never be scanned, so it's not surprising there are so many records.

[aelkiss]

Oh, I misread your comment -- you had said "everything we have without an indication it's ingested yet" but I think I inserted an extra "if", i.e. read it as everything you have regardless of whether or not it's ingested. I think we're all good then.