Still some products with non-ingested detections

[wfastrononomer]

Looking at @MikeeRead's email of 18DEC2025.

"
I've run another check on the VVVX which suggests there are some undeprecated deepstacks whose detections are not in a raw detection table though I might not have looked in all the rights tables. I THINK I've looked in every table whose name

WHERE t.name LIKE '%detectionraw%'
AND t.name NOT LIKE '%short%'
AND t.name NOT LIKE '%defa%'

They all seem to come from cueventid=(504312,504322) run 2024-09-24. There are a few thousand of them

So for example in which raw detection table would one expect to find

frameType   utdate      deprecated  deprecated  project     cuEventID
deepstack   2010-04-22 00:00:00.000 0     0     VVVX  504322
deepstack   2010-03-15 00:00:00.000 0     0     VVVX  504312

If they are not in any detection table should they be or should they be deprecated?

Apologies if already known about I got a bit lost in all the various updates/duplicates etc.

Also no-one answered my previous query below though I doubt this affects things.
....

sorry should have given MFIDs with those examples

frameType   utdate      deprecated  deprecated  project     cuEventID   multiframeID
deepstack   2010-03-15 00:00:00.000 0     0     VVVX  504312      1000000090877
deepstack   2010-04-22 00:00:00.000 0     0     VVVX  504322      1000000094316
"

I checked those two and confirmed what Mike says

Done more checks.

Ran DeprecateVVVXDuplicates code in test mode to find more.
The results are in the /disk62/sys/test/njc/duplicates_20260115_4.log

All the line starting " Not detections in detection table" give the multi frames of deep stacks that haven't been fully ingested.

[esutorius]

Sorry, I don't understand, the filenames in your /disk62/sys/test/njc/badProductsMissStack_20260115.txt point all to files run in 20240809_v2 and 20240918_v2. None of those are from 2024-09-24.

[wfastrononomer]

Hi @esutorius, I think the date Mike gives is from ArchiveCurationHistory for cu3 - when the metadata was ingested into the archive, not the dates you have given above which were when the deep stacks were created:

select * from ArchiveCurationHistory where cuEventID=504312
.----------------------------------------------------------------------------------------------------------------------------.------------------------------------------------------------
.| | cuEventID | cuID | logFile | resultsFile | timeStamp | curator | comment .| rolledBack | |
.----------------------------------------------------------------------------------------------------------------------------.------------------------------------------------------------
.|>| 504312 | 3 | /disk78/genOps/vsa/logs/VSAVVV15_cu3id504312.log | NONE | 2024-09-24 19:59:42.47 | etws .| ingested: Running CU0/CU3 20240809_v2 | 0 |<|

So we are missing detections from files created on 20240809 and 20240918 still.

[esutorius]

So the mystery deepens since your line clearly states that the metadata is ingested and Mike says it isn't.
I'm rerunning the ingest checker and see if I can re-re-ingest the data. Might have even more multiframe duplicates ingested, so I guess we have to reingest all those from scratch.

[wfastrononomer]

Mike doesn't say that the metadata isn't ingested - he says the detections aren't ingested. These are all files with multiframeIDs.

[esutorius]

From his Dec email: "so agreement apart from VVVX deepstacks, where a lot more are in VSAVVV15 than VSA." Since there are no VVVX detections in the VSA my assumption was he was talking about metadata. But maybe I'm wrong and we need detections in the VSA as well.

[wfastrononomer]

Hi Eckhard - the difference in the number of deep stacks between the VSA and VSAVVV15 is a separate problem and I looked into that and resolved that on Wednesday - I sent an email about it:

"
I should have looked into this earlier:

VSA> select frameType,project,deprecated,COUNT(*) from Multiframe where multiframeID in (select multiframeID from ramses23.VSAVVV15.dbo.Multiframe where frameType='deepstack' and project='VVVX' and deprecated=0) group by frameType,project,deprecated order by frameType,project,deprecated

| | frameType | project | deprecated | | |

|>| deepstack | VVVX | 0 | 7578 |<|
|>| deepstack | VVVX | 255 | 17970 |<|

Rows returned: 2 (execution time: 0.373834 s)

So it seems all of the missing VVVX deepstacks are in the VSA and correctly marked as VVVX deepstacks, just deprecated 255, so that is good.
I will try and resolve all other remaining issues.
"

In Mike's follow up email he identified deep stacks with missing detections.

Cheers,
Nick