RFC 18 - Add Note Indexes

[kevinslin]

This is a discussion thread for the following rfc

[namjul]

The first question that came up for me is if this will be part of plugin-core or also available through dendron-cli.

[kevinslin]

Both makes sense

[jonathanyeung]

A few questions:

• When we add a new child and re-run the command, a new entry will be created. What if we remove a child?

• After it has run, Dendron will add indexLastUpdated attribute with the timestamp of the last added note. On subsequent runs, Add Index will only add children that have been created after the indexLastUpdated timestamp.

  Is the reason we need the indexLastUpdated frontmatter so that we can use it to compare w/ the Created timestamp on child notes?

• What about auto-running Add Index on subsequent hierarchy updates? Can we just make it a part of the filewatcher - when a note is created, if parent note contains index, then update the index?

• Wondering if there's a more descriptive name than Dendron: Add Index. Reading that command name at face-value makes me think it could potentially do a variety of things. Thinking out loud: Add Index Block to Current Note, Add Index for Child Notes, something else?

• What if a user tries to add an index when there are no children? Error message, or permitted operation?

[hikchoi]

What about auto-running Add Index on subsequent hierarchy updates? Can we just make it a part of the filewatcher - when a note is created, if parent note contains index, then update the index?

I am a bit concerned with attaching it to a watcher, unless we also have an automatic way to resolve any malformed index blocks. With the index block being appended to the end of the note as default, I think it's even more prone to deletion / editing than the frontmatter.

I guess we should be defining a fallback behavior for cases when updating could fail, have a way to gracefully let the user know it has failed, and it should be fine sticking it into the filewatcher so it updates the index when there is change in the children note(s).

I have proposed some changes to the RFC below, and I think it addresses the remaining questions you had. Would love some feedback.

[hikchoi]

I think we need to step back and ask a more broader question with this:

1. Is this metadata or note content?
2. If it is metadata, should it leak out to be part of the content of the note?
3. If it is note content, should the plugin have the ability to change it?
4. What is the end goal of the common use case described in the RFC?

With the proposed RFC, we want to autogenerate an optional block that is a list of wikilinks to direct children, and we want to attach a feature that can automatically detect what has been changed to the direct children of the note, and apply it to the block.

Q1: Is this metadata or note content?

Case 1: It is note content.

If we assume this, we are opening up to potentially having to take into an unbounded amount of different use cases for this feature, or blocking users from using this feature because it doesn't fit their needs.

• On surface level, I can say that this is part of the note's body.

  • It will show up in preview as a list of wikilinks.
  • You will be able to hover over it in the editor and peek it's contents.
  • You will be able to perform Go To Note while you have the cursor on wikilink.
  • The Backlink view will display these as backlinks.
  • The doctor command could potentially mark any stale indices as wild, and act accordingly.

• If the added note index is note content, a corollary to this is that the user is free to change it and it should be allowed.

  • This introduces some problems with the implementation:
    • The autogenerated markers are prone to be edited / removed.
      • The proposed solution to this is to generate again at the bottom, ignoring what was already indexed before indexLastUpdated.
      • I personally think this introduces a lot of surprises and doesn't encourage the subsequent use of this feature.
    • The content of the note index is free to be edited.
      • This will not be too big of an issue if the update behavior is non-destructive, but as @jonathanyeung pointed out, we have to take into account when a child note is removed between now and indexLastUpdate.
      • The natural thing to do is find the link to a deleted child note (which is now a wild link, or potentially implicitly linking to another vaults note with the same name), and remove it.
      • Since we can't guarantee that we are only deleting content that we put there in the first place, we risk removing content that the user expects to be there.

• Another concern here, assuming the consensus that note index is indeed note content, is that the user is free to chose where the note index should be.

  • The created block appears, at surface level, to be a typical markdown list that has a bunch of wikilinks.
  • We cannot reliably assume that all users want their index list to be at the end of the note.
  • As it looks like standard markdown, they are free to re-locate them to the top. or to anywhere in the note.
    • Maybe they will edit / format the list to their liking.
  • It won't cause any problems for them until they try to run update, at which point they are at the mercy of the auto-update behavior to comply to what their needs are with their index list.
  • Without us providing them with a configurable update behavior, this will cause some resistance; the feature that we added so that a common action could be automated, actually introduces a bit of manual clean-up work on each subsequent updates.

• This give us two opposing answers to Q3: should the plugin have the ability to change it?

  • We should, but in a very careful and strict way. We are dealing with an unbounded variety of different state of the existing note index, and risking any destructive change or surprises here would not be worth the utility this feature initially provides.
  • We shouldn't. It is the feature's job to provide as much information about the current state of the children when the users asks for it, in the most simple and malleable form possible so that it can be a starting point for the users to work based on it.

Case 2: It is metadata

• If we assume the added note index is more of a metadata than note content, then it shouldn't be standard markdown and shouldn't leak out to the body of the note. Because then we risk users misinterpreting this as note content and introduces various concerns illustrated above.
• If it's truly meant to be and viewed as read-only metadata, we already have enough metadata in the NoteProps internally to determine the direct children of any particular note, and it make this feature a bit redundant.
  • It would probably make more sense to provide a dedicated view separate from the editor.
    • We are already doing this with the tree view, but we are using this feature as an interim fix to some needs that the tree view cannot provide to the users as-is.
• By exposing metadata to the user to be open for editing, we introduce risk of it being out of sync (as illustrated in the above case), making more problems.

Q4: What is the end goal of the common use case described in the RFC?

• This feature is trying to fill the gap between:

  • the metadata (all children of note) that is not easily exposed to the user, and
  • the user's need to easily insert note content that is mostly generated from metadata, to the editor.

• At one end (publishing), this is done seamlessly. The cli has access to all of the metadata, and a standardized way of presenting them in a static page.

• At the opposite end (editor), we are attempting to take the available metadata and present it in standardized form in an extremely dynamic environment, with a lot of uncertainties that challenges the initial standard that we impose.

• I propose that if the end goal of this is to provide the users an easy way to insert content (list of direct children) that they can edit as they please, we should give as much agency to the user as possible and avoid assuming a standardized automatic behavior.

Proposed changes

Add Dendron: Add Note Index to cursor location

• This command will take NoteProps and generate standard markdown (hereinafter "index block"), and insert it to the cursor location.
• There is no tracking of last updated, or any automatic behavior defined for updating: The user simply selects the old block and delete it / rerun the command to add a snapshot of the current note index at the cursor location.
• By doing so, we eliminate the need to account for the unbounded variety of use cases that naturally comes with users expecting different ways it would behave. We could even add in snippet feature that defines how the list should look like.
• We build this cli-first (default to appending it to the end), so that it is naturally exposed to the cli.
  • This can be extended to an export pod, for use cases where a user would want to have the note index added to their note.
  • this is similar to the issue described here: Auto-embedded backlinks on each note #695
• Of course this comes with a cost: that the user has to manually update this. Which brings me to the next proposal.

Add a special note that displays a note index, and mark it as such in the frontmatter

• We are isolating the specific use case described in the RFC to a note marked as a note that contains it's index.
• We are introducing a best practice when using this special note, giving it a common baseline understanding that we as developers and the users can agree upon.
• This comes with a few parts:
  • 1. Add Dendron: Make note into index note.
    • This will add a property in frontmatter (say, isIndex: true), that will tell us that this note contains an autogenerated index that is expected by the user to be automatically updated as necessary.
    • The index block is added to the bottom (or the top. I think this should be configurable), and is guarded with a marker as the RFC suggested.
    • We are defining a special note, and we are explicitly suggesting to our users that they do not modify the content between the markers as an update will overwrite what has been modified inside it.
      • With a defined way of using this note, we are giving a clear expectation of how it is supposed to work, letting users to freely use Dendron: Add Note Index to cursor location elsewhere, if they wish to deviate from our suggested best practice.
    • enhance: A doctor command to fix any malformed index note could be added as well.
  • 2. Add Dendron: Update index notes.
    • This will only update the note that are explicitly marked as an index note.
    • This command will take all the notes, pick the ones marked, go to the note's body and replace the entire index block with the current snapshot of the index. (which is in fact, just calling the aforementioned Dendron: Add Note Index to cursor location command at the start of the marker and deleting the existing one).
      • This eliminates the need to track the last index update time and mutate what was existing in the index block and resolving it to the current state.

[jonathanyeung]

Wow, great write up :)

My initial thoughts is that I like this proposal

Add a special note that displays a note index, and mark it as such in the frontmatter

I think having the concept of a type system for notes is a natural evolution for Dendron; this could be the first practical example of that. We should maybe think a little more thoroughly on the implications for note-typing and set up a good foundation for it.

With a typing system, we can even create a code warning if the index block gets malformed.

The index block is added to the bottom (or the top. I think this should be configurable), and is guarded with a marker as the RFC suggested.

I feel like maybe configuration is overkill here? We can stick it at the bottom by default, and the user is free to cut and paste it to wherever they want in the note (so long as they preserve the block format). Our parsing can then handle it wherever it is.

Suggestion to not auto-update

I think this is fine, the only concern I have is around users expecting the index to be up to date and forgetting to update it upon creating children. Maybe this is something we can also add a code warning around? If we detect that an index isn't reflecting it's children, we can have some red squiggly warning suggesting the user run the update index command. I think there are multiple ways we can do this intelligently to make it obvious to the user that their index is stale.

Overall looks great!

[kevinslin]

Love the proposal as well as the breakdown of tasks. Lets keep things simple for phase 1 and just add the index at cursor as is. Also like the direction we're going with note types. Makes me think of typings for "journal" notes and "scratch" notes and if special behavior there (eg. journal note titles are not very useful as is). Thoughts for another RFC :)

[hikchoi]

I feel like maybe configuration is overkill here? We can stick it at the bottom by default, and the user is free to cut and paste it to wherever they want in the note (so long as they preserve the block format). Our parsing can then handle it wherever it is.

Yes. I agree. I kinda missed the part where we are wrapping it between markers 😄

[hikchoi]

I think this is fine, the only concern I have is around users expecting the index to be up to date and forgetting to update it upon creating children.

yeah, I think it's best for us to think through how we want to move forward with the note-typing idea before we start watching for changes. That being said, I do think it should be something that happens without the user's prompt to do so.

[hikchoi]

I think having the concept of a type system for notes is a natural evolution for Dendron

Also like the direction we're going with note types. Makes me think of typings for "journal" notes and "scratch" notes and if special behavior there (eg. journal note titles are not very useful as is). Thoughts for another RFC :)

This is great. Glad this has opened up some more ideas 👍🏼

Lets keep things simple for phase 1 and just add the index at cursor as is.

Will do. This would be a gentle introduction to our plans about bigger ideas. Something we can tease in the release announcement when phase 1 gets shipped 😄