🦅 Osprey v0 Feedback Discussion

[julietshen]

👋 Welcome!

Welcome to the Osprey v0 public release! We’d love your feedback as we shape the future of this open-source rules engine. Please use the prompts below to share your thoughts, bugs, or feature requests.

Your Feedback

Please answer any that are relevant:

• Setup Experience:

  • How smooth was it to install and get Osprey running locally? Were the docs clear enough?

• Usage Experience:

  • Did the built-in rules, UDFs, or sample data work as expected? How was performance?

• Developer Experience:

  • Was it easy to understand and modify rules or add new logic? Any friction in the code structure or API?

• Missing Features or Gaps:

  • Anything you expected to see in v0 but didn’t?

• Bugs or Issues:

  • Please describe steps to reproduce, error logs, or screenshots if relevant.

Suggestions or Ideas

What would make Osprey significantly more useful for your use case?

[verdverm]

Rules in Osprey are written in Some Madeup Language (SML) and follow most syntax conventions present in the Osprey Query UI. SML is a subset of Python with additional restrictions to make the rules simpler to craft.

From https://github.com/roostorg/osprey/blob/main/docs/rules.md

Bespoke languages are a red flag for me. Claims they make thing simpler or easier never pan out. Lack of mature tooling and niceties (like basic syntax highlighting) that developers are used to offsets any gains that could have been had. It also requires the team to spend resources on developing and maintaining the bespoke language, documentation, IDE integrations/extensions, etc...

Why not use Python directly? The justification for SML instead of Python is missing.

[EXBreder]

Its a good point! Even internal to Discord, we went back and forth on this a few times for the exact reasons you mentioned (tooling and idiosyncrasies in the language, as well as inability to search for help)

We kept it as is though due to the audience of rule writers we were aiming for - T&S agents that have some technical chops, but restricted enough that we wouldn't worry about stuff like infinite loops or poor performance. Since the rules are meant to be dynamic and fast to deploy, it needed to be assured at runtime that it wouldn't brick the system (to some extent, obviously someone could still just remove all the rules and do the equivalent of bricking the system), and having a restricted language helps with that.

But you are absolutely right that this should be documented as to why this choice was made. Thanks!

[haileyok]

Just adding another +1 for the choice. The target audience for people that work in the T&S space - particularly analysts or similar - is low/no-code, and having a simple language with many guardrails helps a ton.

To note though - as someone who just went through implementing things and has also found themselves maintaining a rulset for some time - I think this particular language is actually extremely logical to pick up, and works very well for the "mindset" of writing moderation rules

• Ideally imo, a rule should comprise the least logic possible so that the widest array of people can read and understand what it does (at Bluesky, we have historically used Golang for automod rules, but people we onboard for T&S roles don't know Go and can't understand what they do!)
• Writing UDFs significantly reduces the amount of - and forces you to take advantage of - reuse of "common tools" which are useful to anyone writing the rules themselves. Again, keeping people on the same page internally.
• Even as someone who writes Go, I would still quite occasionally ship things that were broken, which is something you reallyyyyyy don't want to happen with your moderation automation......

I'd personally have been very wary of moving to an engine that took direct advantage of Python, because as someone who has actually created and maintained a rules engine that was written entirely in the "source language", I've seen and dealt with the problems that come with doing that (you don't want your engineers to be spending time writing the rules for T&S people, you want them engineering things that make the lives of T&S people easier!)

[verdverm]

Thanks for the replies, the reasons for bespoke are the expected ones, especially the target audience and not having engineer time diverted. I'll personally be fine wrapping with CUE which I can turn that into any language and get even earlier protections and reuse.

More documentation is always great, as well as the story around how imports / packages / modules might work for sharing rules across users and companies. Sounds like a trajectory towards dependency management (yikes!) unless we can build on pythons dependency management, since maybe it is python under the hood?

[julietshen]

created #11 to track the documentation task

[verdverm]

CEL is a popular, non-turing complete "mini" language, as something I would reach for, not advocating it here, just trying to add some more context for our future selves

https://github.com/google/cel-spec

[verdverm]

Another question I have, who is the target audience for this project?

Coming from my DevOps background, I look at the docker-compose with Kafka, Druid, and in total more than 12+ services, and it screams full-time devops team, or at least person. Is this more for large operations than it is for the self-hosting PDS, upstart app, or single person labeller?

(context, I'm also coming from the atproto ecosystem)

[EXBreder]

It definitely came from a "we need about 0.5 people on this for devops". However - most of the services included are the "batteries included" if you want to use them, and are mostly optional/easy to replace with other components. Its definitely feasible to deploy osprey with a minimal set of these, especially if all you care about is the worker/rules.

• kafka -> there needs to be some input for the system. Discord uses pubsub here, but this can also be easily plugin'd via stdin or some other grpc/rest API. As part of the v1, we are also pushing out a "coordinator" system that enables sync calls. Kafka is also used for druid data piping, which is only used if the UI is also needed.
• postgres -> General UI pieces.
• druid -> only necessary for the UI. About half of the services in the compose file are just druid pieces

But to answer your question - osprey is originally more aimed at larger operations where you would have a (small) team of analysts, separate from an engineering team (same reason that we use a bespoke lang). I don't think that's the end goal, and I'd like to see it easier to set up, but its current incarnation is as such.

Of the audiences you listed, in the medium term would love to see us gear towards the upstart app, lowering the barrier to entry for safety in social media.

[turt2live]

Hey all, we're really excited to start experimenting with this in the next few months! A slight concern ahead of that trial: does Osprey (both UI and backend) handle arbitrary UTF-8 in places like room names, user IDs, message content, etc relatively safely? For better or worse, we allow arbitrary UTF-8 (including non-printables) in a bunch of our fields, and that's caused problems in our own apps/tools before.

If Osprey doesn't handle UTF-8 safely, that's fine - we just want to be prepared before data takes out a production deploy 😅

[marticodes]

Hello! I will be doing some testing of the UI with regular moderators, so I wanted to share a feedback review on UI issues or small fixes I’ve noticed to help make the experience smoother.

1. When submitting a query, it seems that you don’t necessarily have to click the button for it to be sent. Every time I change the timeframe, either by selecting one of the preset options or setting a custom range, the query gets submitted automatically. This often causes empty queries to be submitted before I’ve written the one I actually want. I’d suggest making the submission behavior more standardized and less sensitive, ideally only submitting when the user clicks the button.
2. I’ve noticed that the time series chart doesn’t always show data if the selected timeframe is too large, and instead it displays “No data available.” For example, if my database only has data from the last 50 minutes but I select a 1-day view, nothing appears. However, when I switch to a 15-minute view, the data shows up correctly.
3. There’s a small issue with the sidebar, as it scrolls together with the main UI. I noticed this when using the UDF documentation, the sidebar scrolls as I move through the content, so I have to scroll all the way back to the top to select another item. Fixing the sidebar in place would make navigation easier.
4. The hover behavior feels very sensitive. Every time I move my cursor around the UI, elements pop up frequently. It might help to reduce how reactive the hover effects are or add a small delay.
5. In the event stream, when viewing it in its raw form (without selecting any features), there are spacing issues and some events overlap. Once I select features, the problem disappears. I also wondered if there’s a limit to how many features can be selected, since when I choose six, only five seem to appear.
6. The naming inside the event stream could be more consistent. Sometimes the interface uses the term “Features” and other times “Filters,” which can be confusing. At first, I thought these options would actually filter the events, but they only control which features are displayed.

I was also curious about moderating multiple content types at once and whether Osprey could make the UI visually distinguish them better in the charts. For example, for a platform like Facebook where both posts and comments are common, if I query overall content text, it would be helpful if charts showed them in different colors, one for posts and another for comments.

Thank you so much for your work!! Please let me know if you have any questions or need clarification on any of the points I’ve raised.