Support for external managers of pdf papers and general URL schemes (e.g., allowing bibtex collections of papers)

[alberti42]

Premise: The plugin PDF++ is awesome!

Usage case description: I have a huge collection of PDF documents for 15 GB I integrated over time containing research papers I downloaded from the internet. I also have a large bibtex file .bib which I use to store bibliographic information. This is used to manage references when writing scientific articles in Latex. Each bibtex entry is linked to a PDF file.

Instead of using dummy files containing file:///here is my path/to the doc.pdf, I suggest discussing a format (maybe a json file) to store more general information and allowing handling generic URL schemes.

Currently, PDF++ has hard-coded two types, HTML links and file. I forked the project (see GitHub page) and added a new URL scheme allowing content such as: x-bdsk://Manetsch%3A2024?doc=1

This content refers to a URL scheme for BibDesk, a scriptable macOS app, which I have used for a long time to manage my bibtex entries.

I also wrote a plugin for exposing a method to process bibdesk URL scheme (see GitHub page of my plugin). The plugin is not polished but works nicely for my needs (under MacOS). I can process a library of 10000 PDF publications.

Everything works great for me. The only downside is that these changes will probably never be accepted upstream, so I did not even attempt a pull request. In the meanwhile, maybe someone else can already install my fork and plugin, test them, and use them if they are useful for them.

Here, I would like to discuss and collect ideas what is the best way to move forward that supports in the future different formats.

I am thinking of linking PDF++ to a third-party plugin (to be written?) that manages different URL schemes. Or should we include in PDF++ multiple types of URL schemes?

The URL schemes can be different, but eventually, they need to provide a file path to a PDF document that PDF++ plugin can open and visualize.

It would be great to hear your thoughts and find a long-term solution for different collections and managers of PDF documents.

[alberti42]

Short update: I further developed my plugin BibTex Integration for Obsidian, which allows one to manage an external BibTex library and open the linked PDF files using PDF++.

In the new version of my plugin, I use a Monkey patch of PDF++ plugin. This allows me to do the necessary modifications without requiring to install an ad-hoc customized version of PDF++ in my vault.

Hopefully, other people will find this development useful. I think it is still very important to discuss how to provide support with PDF++ to manage large external libraries of PDF files (e.g., Zotero or BibTex-based libraries).

[RyotaUshio]

Hi @alberti42, I'm so sorry for replying this late.

I was impressed by your idea and how you implemented it! Also your idea can be easily extended to other URL schemes, allowing to use any kind of PDF manager of each user's choice, like Zotero. It will open up a huge possibility,

As you have already explored, there can be several ways to add a new URL scheme for PDF++ to recognize.

One possibility is to modify PDF++'s getExternalPDFUrl function. In this case, additional URL scheme support will be included directly in PDF++'s codebase.
In my opinion, this is not ideal because you wouldn't want the speed of your development to be limited by PDF++'s release cycle.
So I think it should be something like a plugin inside PDF++, in other words, an add-on for PDF++ that you can develop separately from PDF++.

In practice, such a PDF++ add-on would be implemented as an Obsidian plugin.
Letting the plugin to monkey-patch PDF++ (just like you did in your plugin) is a clever idea, but there are some problems. For example, some (possibly time-consuming) code has to be repeated, e.g. await app.vault.read(file).

So it would be best if PDF++ could provide an API function for this.
For example, your plugin could do something like PdfPlusApi.registerDummyPdfProcessor("x-bdsk", async (url) => getAbsoluteFilePathFromUrl(url)).
Then I could update the getExternalPDFUrl function to something like

async getExternalPDFUrl(file: TFile): Promise<string | null> {
        if (file.stat.size > 300) return null;

        const content = (await this.app.vault.read(file)).trim();

        if (content.startsWith('https://') || content.startsWith('http://')) {
            const res = await requestUrl(content);
            if (res.status === 200) {
                const url = URL.createObjectURL(new Blob([res.arrayBuffer], { type: 'application/pdf' }));
                return url;
            }
        } else if (content.startsWith('file:///')) {
            return Platform.resourcePathPrefix + content.substring(8);
        } else {
            // !!!
            for (const { id, processor } of this.plugin.dummyPdfProcessors) {
                if (content.startsWith(id)) {
                    const url = await processor(content)
                    return url;
                }
            }
        }
        return null;
    }

I have been thinking about an add-on system inside PDF++ for a while and this discussion seems to be a perfect application.

---

Also, as you said, it will be make sense to update the current dummy PDF format to something more structured like JSON.

[alberti42]

Great, and thanks for reviewing my suggestions and code. Let me know how I can be helpful if there is anything you want me to test in detail.

In my view, there is no hurry to take any hasty decision. But as soon as you have a thought out a good concept, it would be great for future support of other plugins.

You mention something:

(possibly time-consuming) code has to be repeated, e.g. await app.vault.read(file).

but I was not sure what you refer to. I checked again my code and it is very short and creates no duplication or extra time:

// Patch getExternalPDFUrl function
        const removeMonkeyPatchFnc = around(pdf_plus_plugin.lib, {
            getExternalPDFUrl: function (next: (file: TFile) => Promise<string | null>) {
                return async function (this: PDFPlusLib, file: TFile): Promise<string | null> { 
                    if (file.stat.size > 300) return null;

                    const content = (await this.app.vault.read(file)).trim();

                    if (content.startsWith('x-bdsk://')) {
                        if(self.bibtexManager) {
                            const url = await self.bibtexManager.getPdfUrlFromUrl(content);
                            if(url) {
                                return Platform.resourcePathPrefix + url.substring(8);
                            } else {
                                console.error("Error:", `could not resolve url: ${content}`);
                                return null;
                            }
                        } else {
                            console.error("Error:", "BibTex parser not initialized correctly");
                            return null;    
                        }
                    }

                    // Call the original function using .call to preserve the `this` context
                    const result = await next.call(this, file);

                    // Return the (potentially modified) result
                    return result;
                };
            }
        });

Was your comment more general or did you spot something in particular in my code? Just curious. I am quite happy with the current solution, but if you can provide a guideline how things should be done, this is great, it avoids possible conflicts between addons in the future.

[RyotaUshio]

Sorry, "time-consuming" was not quite right. The file size is at most 300 bytes so it won't take too long.
But in general, copying code snippets is not a good practice, right?
In particular, the criteria file.stat.size > 300 for determining whether a file is dummy might change in the future, in which case your code will become invalid.

Also, as I wrote here,

obsidian-pdf-plus/src/main.ts

Lines 19 to 20 in b7bc9e4

	/** The core internal API. Not intended to be used by other plugins. */
	lib: PDFPlusLib = new PDFPlusLib(this);

I don't assume any external tools would depend on lib, so your patching based method might stop working in the future. For example, there is non-zero possibility that I will rename the method to getExternalPdfUrl.

I will consider what the PDF++ API should look like and will get back to you once it's ready. Any suggestions are welcomed.

[alberti42]

Sounds great. I full agree with everything you said. Let me know if you want to discuss ideas or possibilities.

PS: I was wondering whether there is a way to completely avoid using sort of dummy files. My PDF library contains 10,000 PDF. Granted. Unlikely I will now import all of them in my vault. But after a few years, I expect hundreds of PDF dummy PDF files be created in the vault. I am still giving thoughts what could be a good way to maintain robust links to outside the vault.

In case of my plugin, the answer seems obvious. There is already a library (the .bib file) managed by my pluging. It knows exactly where to find the PDF files. If one integrated PDF+ with Zotero, the same applies, we do not need dummy physical files if we have a good URI scheme. In my view, this would be by far the best. What are the disadvantages?

PS2: I am not sure whether you are a mac user. On mac, you can use Bookmark to store the position of files. These are used for example by BibDesk and are supported by my plugin. The great thing about bookmark is that if the user rename the file or move it to a different folder, the bookmark can still resolve it. I believe Apple stores the physical location on the HD rather than the name on the file system. Having a mechanism of this kind would make any external link much more robust for PDF+.