resolve segfault triggered by rust_minifb

[wdoconnell]

To date, testing has been done on my 2019 Macbook Pro. It has these relevant specs

• 2.3 GHz 8-core Intel i9 processor
• 16 GB 2667 Mhz DDR memory
• AMD Radeon Pro 5500M 4GB graphics card

On OSX, after the concurrency refactors with crossbeam to send and receive audio/video via channels (e531111, 9807462), I now get approx. 60 FPS and clear audio on OSX.

Ultimately the goal is to optimize this for embedded devices so that it can be used on cheap "connector" hardware that I can keep plugged in. Today I tested it on a raspberry pi zero 2:

• 1GHz quad-core 64-bit Arm Cortex-A53 CPU
• 512MB SDRAM
• VideoCore IV GPU

No good results there. A video opens briefly, shows one frame of black screen, and is followed by a segfault, which appears to be coming from rust_minifb, which we use for windowed framebuffer output.

wdo@raspberrypi: $ valgrind ./target/debug/rustds
. . .
Invalid write of size 4:
image_resize_linear_stride (scalar.c:54)
image_resize_linear_aspect_fill (scalar.c:91)
minifb::os::posix::x11::Window::raw_blit:buffer (x11.rs:911)
minifb::os::posix::x11:: Window::update_with_buffer_stride (x11.rs:585)
minifb::os::posix::Window::update_with_buffer_stride (mod.rs:98)
minifb::Window::update_with_buffer (lib.rs:387)
rustds::serve_video (main.rs:73)

[wdoconnell]

In minifb_rust the following fn, which is used for the X11 window system implementation is wrapped in an unsafe block :

    pub fn update_with_buffer_stride(
        &mut self,
        buffer: &[u32],
        buf_width: usize,
        buf_height: usize,
        buf_stride: usize,
    ) -> Result<()> {
        check_buffer_size(buffer, buf_width, buf_height, buf_stride)?;

        unsafe { self.raw_blit_buffer(buffer, buf_width, buf_height, buf_stride) };

        self.update();

        Ok(())
    }

which calls:

    unsafe fn raw_blit_buffer(
        &mut self,
        buffer: &[u32],
        buf_width: usize,
        buf_height: usize,
        buf_stride: usize,
    ) {
        match self.scale_mode {
            ScaleMode::Stretch => {
                image_resize_linear(
                    self.draw_buffer.as_mut_ptr(),
                    self.width,
                    self.height,
                    buffer.as_ptr(),
                    buf_width as u32,
                    buf_height as u32,
                    buf_stride as u32,
                );
            }

            ScaleMode::AspectRatioStretch => {
                image_resize_linear_aspect_fill(
                    self.draw_buffer.as_mut_ptr(),
                    self.width,
                    self.height,
                    buffer.as_ptr(),
                    buf_width as u32,
                    buf_height as u32,
                    buf_stride as u32,
                    self.bg_color,
                );
            }

            ScaleMode::Center => {
                image_center(
                    self.draw_buffer.as_mut_ptr(),
                    self.width,
                    self.height,
                    buffer.as_ptr(),
                    buf_width as u32,
                    buf_height as u32,
                    buf_stride as u32,
                    self.bg_color,
                );
            }

            ScaleMode::UpperLeft => {
                image_upper_left(
                    self.draw_buffer.as_mut_ptr(),
                    self.width,
                    self.height,
                    buffer.as_ptr(),
                    buf_width as u32,
                    buf_height as u32,
                    buf_stride as u32,
                    self.bg_color,
                );
            }
        }

        (self.d.lib.XPutImage)(
            self.d.display,
            self.handle,
            self.d.gc,
            self.ximage,
            0,
            0,
            0,
            0,
            self.width,
            self.height,
        );
        (self.d.lib.XFlush)(self.d.display);
    }

which uses foreign function interfaces/FFI via the extern keyword to call the C functions image_resize_linear_aspect_fill and image_resize_linear_stride found here.

So ultimately we have some sort of memory out-of-bounds error at the minifb level in these C fns, which isn't being caught due to the use of FFI and having to wrap these fns in an unsafe block.

I got an analogous segfault using when using Wayland over x11, so it does seem to be restricted to this code block.

[wdoconnell]

Since these were triggered by scaling, I was able to get around it on the raspberry pi by changing the window scaling configuration from autoscaling to simply centered representation of a 1:1 window of the 3ds (720x240). Temporarily updated my newType for window options to:

https://doc.rust-lang.org/rust-by-example/generics/new_types.html

impl CustomWindowOptions {
    pub fn new(resize: bool, scale: Scale, scale_mode: ScaleMode) -> Self {
        Self {
            opts: WindowOptions {
                resize,
                scale,
                scale_mode,
                ..WindowOptions::default()
            },
        }
    }

    pub fn inner(&self) -> WindowOptions {
        self.opts
    }
}

    let opts = CustomWindowOptions::new(false, Scale::X1, ScaleMode::Center); 

And now I get ca. 50 fps on the raspberry PI. However, I still get some sort of buffer overrun on the x11 window, as shown below. I also can't resize or scale the window. So the workaround won't help us meaningfully here.

To recap: the OSX implementation of minifb (in OSX this appears to use Metal, which does seem to work, Linux window systems don't seem to allow scaling with the current implementation. At this point I think we'll need to look for another library.

[wdoconnell]

Some other options to look into:

• softbuffer https://github.com/rust-windowing/softbuffer
• pixels https://github.com/parasyte/pixels (has hardware acceleration)
• directly output to the /dev/fb0 device (at least on linux)