null fn check incorrect on release builds

[rillian]

Testing Firefox against rustc 1.17.0-nightly (ccce2c6 2017-03-27) I have a unit test failure where it looks like a nullptr check on a function pointer in a struct passed over FFI doesn't work correctly. The failing test is supposed to check for a null callback function pointer and return null in that case. Instead it returns an allocated context object.

One can verify with ./mach gtest rust.MP4MetadataEmpty with ac_add_options --enable-optimize in mozconfig.

See also https://bugzilla.mozilla.org/show_bug.cgi?id=1351497

[rillian]

Here is a simplified testcase: https://play.rust-lang.org/?gist=2438d07552b441017c9a870e347edef2

This code should print "Ok!" and does on debug builds, and stable release builds, but not on nightly release builds.

type ReadFn = extern fn(*mut u8, usize) -> isize;

#[repr(C)]
#[derive(Clone)]
pub struct mp4parse_io {
    pub read: ReadFn,
}

unsafe extern fn mp4parse_new(io: *const mp4parse_io) -> *mut mp4parse_io {
    if ((*io).read as *mut std::os::raw::c_void).is_null() {
        return std::ptr::null_mut();
    }
    let parser = Box::new( (*io).clone() );

    Box::into_raw(parser)
}

fn boom() {
    let parser = unsafe {
        let io = mp4parse_io {
            read: std::mem::transmute::<*const (), ReadFn>(std::ptr::null()),
        };
        mp4parse_new(&io)
    };
    assert!(parser.is_null());
}

fn main() {
    println!("Testing null read callback... ");
    boom();
    println!("Ok!");
}

Note: I'm willing to believe the code is bad, but I don't understand how, so looking for guidance either way. :)

[steveklabnik]

Tagging and nominating.

[rillian]

I did a bit more experimentation. The function call between creating the struct and the null check is important. The .clone() and #[repr(C)] are not.

[jonas-schievink]

That transmute seems like UB, no?

Anyways, nightly produces this for the null check:

  %4 = load i64 (i8*, i64)*, i64 (i8*, i64)** %3, !dbg !275, !nonnull !49
  %5 = bitcast i64 (i8*, i64)* %4 to i8*, !dbg !275
  %6 = call zeroext i1 @"_ZN4core3ptr31_$LT$impl$u20$$BP$mut$u20$T$GT$7is_null17hdd9ae6e9ddd6f4bcE"(i8* %5), !dbg !275

...and Beta this:

  %4 = load i64 (i8*, i64)*, i64 (i8*, i64)** %3, !dbg !254
  %5 = bitcast i64 (i8*, i64)* %4 to i8*, !dbg !254
  %6 = call zeroext i1 @"_ZN4core3ptr31_$LT$impl$u20$$BP$mut$u20$T$GT$7is_null17hdd9ae6e9ddd6f4bcE"(i8* %5), !dbg !254

So it's pretty clear what's happening :)

[rillian]

The transmute is to represent receiving a null function pointer over FFI from C, just to make the testcase easier to run.

[Diggsey]

@rillian function pointers are not allowed to be null in rust. Your transmute is therefore UB.

If you want to represent nullable function pointers, use Option<ReadFn> instead - it is guaranteed to be layout compatible.

[retep998]

Winapi consistently uses Option<unsafe extern "something" fn(...) -> ...> across the board because it has a compatible representation with C function pointers, gracefully handles null pointers, and the unsafe bit ensures that code knows what it is doing before it invokes a function pointer provided by the system.

Rust function pointers are like Rust references, they can never be null. There is also no raw pointer equivalent of a Rust function pointer, so Option is the only way to handle nullable function pointers.

[arielb1]

I marked function pointer loads as nonnull in one of the commits in #39628 (I thought I would be using that codepath for vtables, where we have nonnull today I wanted to preserve - not that I think it helps anybody - but eventually ended up not using it).

However, null function pointers are UB since, well, forever (see Option<fn()>).