Free "extension" idea: ml5js-like suite for py5

[villares]

This weekend I saw a presentation on ml5js "Easy Machine Learning for the Web", which helps bring Machine Learning tools for people using p5js.

Some (most?) of it seems based on mediapipe which also has Python bindings. I wish someone could come up with a wrapper library making it easy to use it with py5.

[hx2A]

I wish someone could come up with a wrapper library making it easy to use it with py5.

I heard a lot about ml5js when I was at NYU. I didn't know about its relation to mediapipe though. What would a wrapper library do beyond what it can do already? There is python example code and it looks pretty straightforward:

https://ai.google.dev/edge/mediapipe/solutions/vision/face_detector
https://colab.research.google.com/github/googlesamples/mediapipe/blob/main/examples/face_detector/python/face_detector.ipynb

[villares]

The example code that I saw used a context manager around a "main loop". I found that awkward to combine with setup/draw in py5. My guess is you shouldn't restart that context on each frame... I made a clumsy (not good) experiment setting it around the py5.run_sketch() call: https://github.com/villares/sketch-a-day/blob/main/2025/sketch_2025_11_02/sketch_2025_11_02.py

On ML5 you set it up and you are good to go. I suppose one could use exiting() to clean up the things too, but I'd need to see what it is doing. The example you linked might be more helpful for me to understand it.

Oh. Probably the best way is to put detection on a separate thread outside draw, right?

[villares]

Would something in this direction be advisable?

import mediapipe as mp
import cv2
import py5

mp_drawing = mp.solutions.drawing_utils
mp_holistic = mp.solutions.holistic
py5_img = mp_image = None
results = None

def setup():
    py5.size(640, 480)
    py5.launch_thread(process_capture)

def process_capture():
    global mp_image, results, cap
    cap = cv2.VideoCapture(0)
    print(cap.isOpened())
    with mp_holistic.Holistic(min_detection_confidence=0.5, min_tracking_confidence=0.5) as holistic:
        print(holistic)
        while cap.isOpened():
            ret, frame = cap.read()
            mp_image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            results = holistic.process(mp_image)
            
            mp_drawing.draw_landmarks(mp_image, results.face_landmarks, mp_holistic.FACEMESH_CONTOURS,
                                      mp_drawing.DrawingSpec(color=(80, 110, 10), thickness=1, circle_radius=1),
                                      mp_drawing.DrawingSpec(color=(80, 256, 121), thickness=1, circle_radius=1))

            mp_drawing.draw_landmarks(mp_image, results.right_hand_landmarks, mp_holistic.HAND_CONNECTIONS,
                                      mp_drawing.DrawingSpec(color=(80, 22, 10), thickness=2, circle_radius=4),
                                      mp_drawing.DrawingSpec(color=(80, 44, 121), thickness=2, circle_radius=2))

            mp_drawing.draw_landmarks(mp_image, results.left_hand_landmarks, mp_holistic.HAND_CONNECTIONS,
                                      mp_drawing.DrawingSpec(color=(121, 22, 76), thickness=2, circle_radius=4),
                                      mp_drawing.DrawingSpec(color=(121, 44, 250), thickness=2, circle_radius=2))

            mp_drawing.draw_landmarks(mp_image, results.pose_landmarks, mp_holistic.POSE_CONNECTIONS,
                                      mp_drawing.DrawingSpec(color=(245, 117, 66), thickness=2, circle_radius=4),
                                      mp_drawing.DrawingSpec(color=(245, 66, 230), thickness=2, circle_radius=2))
def draw():
    global py5_img
    if mp_image is not None:
        py5_img = py5.create_image_from_numpy(mp_image, 'RGB', dst=py5_img)
        py5.image(py5_img, 0, 0)

def exiting():
    cap.release()
    py5.stop_all_threads(wait=True)
    print('Done.')
    
def key_pressed():
    print(results)
    
py5.run_sketch(block=False)

[hx2A]

Yes this is a good approach! You can probably even move the create_image_from_numpy() call to inside the process_capture() function. Good use of the dst keyword there, that is essential for good performance.

[hx2A]

You can probably even move the create_image_from_numpy() call to inside the process_capture() function.

Although it is a good practice to avoid calling py5 methods in a thread because some much of Processing is not thread safe. In this case it seems to be fine.