✨ Add events for account and record update/delete/deactivation #2661

foysalit · 2024-07-25T12:14:59Z

This PR adds 4 new events to enable ozone to keep track of account deletion/deactivation and record update/delete. The record update event will primarily be used for profiles and lists for the time being.

Questions:

Should we add logic in the event logger to ensure that duplicate events for account or record delete are discarded?

bnewbold · 2024-08-01T00:39:51Z

thanks for writing these up! having something concrete is always great for getting brain flowing.

I'd be pretty inclined to roll up the account events in to 1-2 event types. Eg, either a single "account status change", or two events, one aligning with the #account firehose event (deletion, deactivation) and one aligning with #identity (handle change, PDS migration). would add fields like accountStatus (active, deleted, etc), handle, and pdsHost to track those values over time.

So either a single event:

#accountStateEvent
  comment: string
  accountStatus: string, one of 'deleted', 'deactivated', 'suspended', 'takedown', 'active'
  handle: string
  pdsHost: string (URL)
  timestamp: datetime (estimate of when the state change actually happened, which might be different from ozone timestamp if there was a delay or backfill) (alt names welcome)

or two:

#identityEvent
  comment: string
  handle: string
  pdsHost: string (URL)
  timestamp: datetime

#accountEvent
  accountStatus: string, one of 'deleted', 'deactivated', 'suspended', 'takedown', 'active'
  timestamp: datetime

For the record events, it feels like we could also merge them in to a single #recordEvent? But i'm not as sure.

#recordEvent
  comment: string
  recordCid: CID of the record itself (for updates; nil means deleted?)
  deleted: bool (if we don't want to rely on recordCid being nil to mean deleted?)
  created: bool (indicates the record is created, not updated)
  record: do we embed the actual record, for updates? I guess that will be stored/cached elsewhere
  timestamp: datetime

In addition to profile records, we would probably have automod track:

when records are deleted or updated, check if they have active labels, and record the event in that case (aka, if there has been a mod intervention on a record, want to record future state for that record)
all updates to labeler declaration records
mod list declarations (as mentioned)

foysalit · 2024-08-08T20:54:22Z

made some adjustments according to your feedback. couple of questions though

not sure why we'd need a created event for record? I wouldn't expect to be proactively tracking record creations from within ozone.
suspended takedown states for accounts would only be helpful for 3p labelers I guess? just wondering if this would feel weird for instances with takedown capabilites?

bnewbold · 2024-08-09T00:59:53Z

for 1: seems good to track if a record event is an update or creation just for completeness? I could imagine us deciding we want to track creation events for some specific record types, like labeler declarations, which are relatively high-stakes and low-volume in the network.

for 2: we'd want to track the status of accounts on independent PDS instances. it does get a bit "loopy" to double-track this in our ozone instance for accounts on our PDS instances, maybe we would skip generating those events? remember that we'll also be splitting out infra takedowns as a distinct event type from current takedowns, as some separate tech debt to clean up.

bnewbold

overall very much like this shape/direction!

bnewbold · 2024-08-09T01:02:57Z

lexicons/tools/ozone/moderation/defs.json

+        }
+      }
+    },
+    "identityEvent": {


for completeness, there is a concept of "tombstoning" an identity. this has been very rare in practice. in theory it is a permanent deletion, though technically it can be reverted in PLC within a time period (72hr), and can be reverted any time with did:web.

it is sort of a corner-case, but would actually be a good thing to track in ozone because otherwise it would be very hard for mods to understand what had happened: a deleted identity makes it pretty hard to learn anything about an account from the live network.

foysalit · 2024-08-09T09:13:25Z

instead of making accountStatus enum we could just make it a string with knownValues so that suspended and takedown etc. can be facilitated but not documented? but yeah, happy to add it to the list too.
for recordEvent, wondering if we should take a similar approach? instead of having separate flags for created updated etc., lifecycle string field with knownValues: ['created', 'updated', 'deleted'] could fit better, right? in fact, maybe this lifecycle/status property is suitable for all 3 events now that you mention that identity events can also have a tombstone?

…ccount-events

bnewbold · 2024-08-09T22:48:52Z

Yeah, I think in other contexts we also have it as knownValues instead of an enum or closed union. This allows future flexibility ("stale"? "maintenance"?) while keeping the active/not-active aspect clear.

I think for identity and record states, we know more clearly that there are just the 3x operations on a record (create, update, delete), and that an identity is active or tombstoned (this is inherited from the W3C DID semantics). I would do just a boolean for identity tombstoned. For records, it could in theory be just a boolean and the cid string: no cid would mean deleted, and created: true would mean creation and created: false (or missing) would mean updated. But a string with knownValues wouldn't be crazy.

…vent

foysalit · 2024-08-11T18:19:00Z

But a string with knownValues wouldn't be crazy.

Sounds like you're not super against it and I like the idea of just storing an op: 'created' | 'updated' | 'deleted' flag instead of the boolean flag approach so I've redesigned to use the op property.

Also added an optional tombstone: boolean flag in identity event. @bnewbold

bnewbold · 2024-08-12T03:42:23Z

for accountStatus, I think we still need takendown and suspended. the use-case for this is when an account is takendown on a self-hosted PDS which we don't moderate: this is the sort of situation which would be very confusing for mod team to understand/debug without this sort of event tracking.

here are the knownValues from com.atproto.sync.subscribeRepos, which I think we should match one-to-one:
https://github.com/bluesky-social/atproto/blob/main/lexicons/com/atproto/sync/subscribeRepos.json#L145

I would also put the active boolean flag in; if it is easier to have an active string value as well that is fine (eg, I could imagine wanting to query with acountStatus=active, or do aggregations on a single value), but I would basically just try to keep the field as one-to-one with the repo stream schema as possible.

record event looks good!

if we can get a quick set of eyes from @devinivy on this it would be great.

devinivy

Just a few small Qs and lexicon suggestions. Looks good!

devinivy · 2024-08-14T03:35:10Z

lexicons/tools/ozone/moderation/defs.json

+          "type": "string",
+          "knownValues": ["create", "update", "delete"]
+        },
+        "cid": { "type": "string" },


Suggested change

"cid": { "type": "string" },

"cid": { "type": "string", "format": "cid" },

devinivy · 2024-08-14T03:35:41Z

lexicons/tools/ozone/moderation/defs.json

+      "required": ["timestamp"],
+      "properties": {
+        "comment": { "type": "string" },
+        "handle": { "type": "string" },


Suggested change

"handle": { "type": "string" },

"handle": { "type": "string", "format": "handle" },

devinivy · 2024-08-14T03:39:48Z

packages/ozone/src/mod-service/index.ts

@@ -317,6 +320,25 @@ export class ModerationService {
      }
    }

+    if (isAccountEvent(event)) {
+      meta.active = !!event.active


If active is missing do we want to interpret it as though the account went inactive? I wonder if we may want to omit it from meta altogether in this case.

I guess the !! is confusing here but this is a rerouted from #account from the firehose and the active flag should be always set https://github.com/bluesky-social/atproto/blob/main/lexicons/com/atproto/sync/subscribeRepos.json#L133

devinivy · 2024-08-14T03:40:28Z

packages/ozone/src/mod-service/views.ts

@@ -184,6 +184,25 @@ export class ModerationViews {
      eventView.event.remove = event.removedTags || []
    }

+    if (event.action === 'tools.ozone.moderation.defs#accountEvent') {
+      eventView.event.active = !!event.meta?.active


Similar question here!

…ccount-events

devinivy · 2024-09-11T01:39:54Z

lexicons/tools/ozone/moderation/defs.json

@@ -395,7 +395,7 @@
    "accountEvent": {
      "type": "object",
      "description": "Logs account status related events on a repo subject. Normally captured by automod from the firehose and emitted to ozone for historical tracking.",
-      "required": ["timestamp", "status"],
+      "required": ["timestamp", "status", "active"],


Adding a required field means that old events missing this field will become invalid. Is that fine in this case (i.e. all pass events in practice already include the field)?

Oh haha nevermind! I was reading diff from recent commits and forgot this event was brand new in the PR 🙃

…ccount-events

* ✨ Store deleted/updated event data in subject_status * 🐛 Fix query for recordDeletedAt and recordUpdatedAt * ✨ Add tombstoned status * ✨ Move from record to hosting term * ✅ Add tests for hosting params * ✨ Update lexicons for hostingStatuses * ✅ Update snapshots * ✅ Update snapshots * ✅ Update snapshots * ✨ Adjust hosting statuses

…ccount-events

✨ Add events for account and record update/delete/deactivation

49c4c5d

foysalit requested review from bnewbold and devinivy July 25, 2024 12:16

✨ Add handle change event

3e820b7

✨ Reduce account events to 2 types and record events to 1

0f15836

foysalit force-pushed the ozone-account-events branch from 9422d76 to 0f15836 Compare August 8, 2024 20:55

bnewbold reviewed Aug 9, 2024

View reviewed changes

✨ Store metadata from account, identity and record events

2b09c10

foysalit added 3 commits August 9, 2024 15:59

✨ Add created event for record

c4db563

Merge branch 'main' of github.com:bluesky-social/atproto into ozone-a…

dc8741c

…ccount-events

✨ Add ndd the new events to allowed types in emitEvent

dbfad25

✨ Use string value for record op and add tombstone flag to identity e…

8bb799a

…vent

foysalit added 2 commits August 12, 2024 18:27

✨ Add active flag on account events

ce9599b

✨ Change accountStatus -> status to match with firehose event

4e61f25

bnewbold approved these changes Aug 14, 2024

View reviewed changes

devinivy approved these changes Aug 14, 2024

View reviewed changes

foysalit added 2 commits August 14, 2024 12:35

Merge branch 'main' of github.com:bluesky-social/atproto into ozone-a…

da7b1ed

…ccount-events

✨ Make active flag required

d58c9a5

foysalit requested a review from devinivy August 15, 2024 13:59

Vincentroland approved these changes Sep 5, 2024

View reviewed changes

Merge branch 'main' of github.com:bluesky-social/atproto into ozone-a…

27131fe

…ccount-events

devinivy reviewed Sep 11, 2024

View reviewed changes

devinivy approved these changes Sep 11, 2024

View reviewed changes

foysalit added 3 commits September 30, 2024 20:48

:rightwards_twisted_arrows: Merge with upstream

0807a5b

🚨 fix prettier style issue

1b522f1

Merge branch 'main' of github.com:bluesky-social/atproto into ozone-a…

149b29f

…ccount-events

foysalit mentioned this pull request Oct 30, 2024

✨ Track record/account delete and update data in subject status #2804

Merged

foysalit added 3 commits October 30, 2024 22:43

Merge branch 'main' of github.com:bluesky-social/atproto into ozone-a…

e3e03cd

…ccount-events

📝 Add changeset

443c81d

foysalit mentioned this pull request Oct 30, 2024

✨ Show account, identity and record events bluesky-social/ozone#223

Open

devinivy approved these changes Nov 7, 2024

View reviewed changes

:rightwards_twisted_arrows: Merge with upstream

03d31f1

foysalit merged commit d6f33b4 into main Nov 7, 2024
10 checks passed

foysalit deleted the ozone-account-events branch November 7, 2024 21:51

github-actions bot mentioned this pull request Nov 7, 2024

Version packages #2932

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Add events for account and record update/delete/deactivation #2661

✨ Add events for account and record update/delete/deactivation #2661

foysalit commented Jul 25, 2024

bnewbold commented Aug 1, 2024 •

edited

Loading

foysalit commented Aug 8, 2024

bnewbold commented Aug 9, 2024

bnewbold left a comment

bnewbold Aug 9, 2024

foysalit commented Aug 9, 2024 •

edited

Loading

bnewbold commented Aug 9, 2024

foysalit commented Aug 11, 2024

bnewbold commented Aug 12, 2024

devinivy left a comment

devinivy Aug 14, 2024

devinivy Aug 14, 2024

devinivy Aug 14, 2024

foysalit Aug 14, 2024

devinivy Aug 14, 2024

devinivy Sep 11, 2024 •

edited

Loading

devinivy Sep 11, 2024

	"cid": { "type": "string" },
	"cid": { "type": "string", "format": "cid" },

	"handle": { "type": "string" },
	"handle": { "type": "string", "format": "handle" },

✨ Add events for account and record update/delete/deactivation #2661

✨ Add events for account and record update/delete/deactivation #2661

Conversation

foysalit commented Jul 25, 2024

bnewbold commented Aug 1, 2024 • edited Loading

foysalit commented Aug 8, 2024

bnewbold commented Aug 9, 2024

bnewbold left a comment

Choose a reason for hiding this comment

bnewbold Aug 9, 2024

Choose a reason for hiding this comment

foysalit commented Aug 9, 2024 • edited Loading

bnewbold commented Aug 9, 2024

foysalit commented Aug 11, 2024

bnewbold commented Aug 12, 2024

devinivy left a comment

Choose a reason for hiding this comment

devinivy Aug 14, 2024

Choose a reason for hiding this comment

devinivy Aug 14, 2024

Choose a reason for hiding this comment

devinivy Aug 14, 2024

Choose a reason for hiding this comment

foysalit Aug 14, 2024

Choose a reason for hiding this comment

devinivy Aug 14, 2024

Choose a reason for hiding this comment

devinivy Sep 11, 2024 • edited Loading

Choose a reason for hiding this comment

devinivy Sep 11, 2024

Choose a reason for hiding this comment

bnewbold commented Aug 1, 2024 •

edited

Loading

foysalit commented Aug 9, 2024 •

edited

Loading

devinivy Sep 11, 2024 •

edited

Loading