Support for Copying Mathematical Equations

[kishore-1404]

Hello,

I'm reaching out to request an enhancement in the way this PDF viewer handles the copying of mathematical equations. Currently, when attempting to copy mathematical content from a PDF, the structure and formatting of equations are often lost. This makes it difficult to transfer equations directly into LaTeX or text editors without extensive manual reformatting.

[panangam]

Not a Sioyek-specific answer, but there are paid and open source solutions for LaTeX OCR that can convert a snapshot of a LaTeX equation to its formula.

https://lukas-blecher.github.io/LaTeX-OCR/
https://mathpix.com/

I don't think it's possible at all to recover the LaTeX formulas from a PDF document directly, as the rendering process doesn't keep the formula around.

[curtischong]

Maybe there should be an option to provide your own openai api key so it can send a screenshot of the equation to gpt4o (into latex) and send back a transcription of the equation?

[ahrm]

I don't like adding openai-specific code to sioyek. However, in c96c79c I improved the javascript commands so the user can define a command to do so themselves. Here is the script I wrote:

openai_latex.js (a file next to prefs_user.config:

let selected_rect = JSON.parse(sioyek.select_rect());
let window_rect = sioyek_api.absolute_to_window_rect_json(selected_rect);
let screenshot_file_name = 'openai';
sioyek_api.screenshot_js("e:/sioyek_screenshots/" + screenshot_file_name + ".png", window_rect);

let file_content_base_64 = sioyek_api.read_file("e:/sioyek_screenshots/" + screenshot_file_name + ".png", true);


let url = 'https://api.openai.com/v1/chat/completions';
let headers = {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer sk-proj-whatever'
};

let request = {
    "model": "gpt-4o-mini",
    "messages": [
        {
            "role": "user", "content": [
                {
                    "type": "text",
                    "text": "Give LaTeX code to generate this image. Reply only with LaTeX code. Do not include any other text."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64," + file_content_base_64
                    }
                }
            ]
        }
    ],
    "max_tokens": 500
};


let raw_resp = sioyek_api.perform_network_request_with_headers('POST', url, headers, request);

let resp = JSON.parse(raw_resp);
let completion = resp.choices[0].message.content
console.log(completion);

let rect_string = '' + selected_rect.x0 + ' ' + selected_rect.y0 + ' ' + selected_rect.x1 + ' ' + selected_rect.y1;
sioyek.add_freetext_bookmark(rect_string, completion);

// alternatively we could copy it to clipboard
// sioyek_api.copy_text_to_clipboard(completion);
// sioyek.set_status_string('Copied result to clipboard.');
// sioyek.wait(2000);
// sioyek.clear_status_string();

Now in prefs_user.config:

new_async_js_command _convert_to_latex openai_latex.js

Here is how it looks:

openai_extension.mp4

[curtischong]

This looks so cool! Thank you so much for adding this!