Allow wildcards in content-type

[haBuu]

Currently (Rocket version 0.4.0-dev) if you use a wildcard in content-type you will get a warning like this:

warning: 'image/*' is not a known media type
 --> src/handlers/image.rs:5:33
  |
5 | #[post("/image", format = "image/*", data = "<data>")]
  |                                 ^^^^^^^^^

Using wildcard e.g. for images is needed as you do not know what the actual type of the image will be (png, gif, jpg etc.) and you want to handle all of them in the same route.

[nickdonnelly]

Would love to see a way to suppress these warnings.

[SergioBenitez]

@nickdonnelly That's not currently possible with Rust's procedural macro infrastructure. I'm hoping to add it soon.

First, hopefully the ^^^^ are in the right place! Second: yes, we should handle this. It would require Rocket to know about some top-level types, which seems easy to do. I'd be happy to mentor an implementation of this.

[jwnhy]

I am quite new to rust community and want to find something to try, may I take this issue?
Could your give me some guide?

[jebrosen]

I think as a first step we need to decide what exactly to do and figure out how much work / how difficult the parts are. This is what I found so far:

1. The format= is used for GET requests to match the Accept header (media type wanted by the client), and for POST requests to match the Content-Type header (media type sent by the client). The example seems clearly motivated for allowing POST to use wildcards; I'm not as certain about GET.
2. The code generation (impl FromMeta for MediaType in core/codegen/src/http_codegen.rs) emits a warning on encountering an "unknown" media type (both top- and sub-level must be known as a unit). This should be relaxed at least to allow any known top-level with the * sub-level - */* may be nice to allow for consistency but should be the same as leaving format= off entirely. Currently the warning happens during parsing, but we may need to move that warning logic into the route macro especially if GET vs POST makes a difference.
3. The route matching (formats_collide, formats_match, media_types_collide in core/lib/src/router/collider.rs) appears to already handle wildcards, treating image/png as a match to image/*. It also appears to match the (nonsensical) */png against image/*, but that's a separate issue.

This may not be the easiest thing to work on for a newcomer because of the code generation aspect, but any thoughts or concerns on the above points would be greatly appreciated.

[giantcow]

I think this works? at least it seems to work out of the box for me...

Working example code

#[macro_use]
extern crate rocket;

use rocket::fs::TempFile;

type V1APIResult<T> = Result<T, rocket::response::Debug<anyhow::Error>>;

#[post("/upload", format = "*/*", data = "<file>", rank = 2)]
async fn post_upload(mut file: TempFile<'_>) -> V1APIResult<()> {
    println!("Temp file is {:?}", file);
    Ok(())
}

#[put("/upload", format = "*/*", data = "<file>", rank = 1)]
async fn put_upload(mut file: TempFile<'_>) -> V1APIResult<()> {
    println!("Temp file is {:?}", file);
    Ok(())
}

#[launch]
fn rocket() -> _ {
    rocket::build().mount("/v1", routes![post_upload, put_upload])
}

# Rocket.toml
[debug.limits]
file = "1Gb"

Executing the following:

curl -svvv \
    -T ../bbb_sunflower_1080p_30fps_normal.mp4 \
    -H 'Content-Type: application/mp4' \
    -L http://127.0.0.1:8000/v1/upload

leads to a 200:

PUT /v1/upload application/mp4:
   >> Matched: (put_upload) PUT /v1/upload */*
Temp file is File { file_name: None, content_type: Some(ContentType(MediaType { source: Custom("application/mp4"), top: (0, 11), sub: (12, 15), params: Dynamic([]) })), path: Left("/tmp/.tmpYJX7Zc"), len: 276134947 }
   >> Outcome: Success(200 OK)
   >> Response succeeded.

---

However this doesn't seem to work with the limits config, ie having:

# Rocket.toml
[debug.limits]
"application/*" = "1Gb"

Executing the following:

curl -svvv \
    -T ../bbb_sunflower_1080p_30fps_normal.mp4 \
    -H 'Content-Type: application/mp4' \
    -L http://127.0.0.1:8000/v1/upload

leads to:

PUT /v1/upload application/mp4:
   >> Matched: (put_upload) PUT /v1/upload */*
   >> Data limit reached while reading the request body.
   >> Data guard `TempFile < '_ >` failed: Custom { kind: UnexpectedEof, error: "data limit exceeded" }.
   >> Outcome: Error(400 Bad Request)
   >> No 400 catcher registered. Using Rocket default.
   >> Response succeeded.

[teotwaki]

@giantcow I can confirm this issue is indeed resolved.

Simplified example:

#[macro_use]
extern crate rocket;

use rocket::fs::TempFile;

#[put("/any", format = "*/*", data = "<_f>")]
fn any(_f: TempFile<'_>) {}

#[put("/any_app", format = "application/*", data = "<_f>")]
fn any_app(_f: TempFile<'_>) {}

#[launch]
fn rocket() -> _ {
    rocket::build().mount("/", routes![any, any_app])
}

This appears to work as expected:

• curl -T small.bin -H 'Content-Type: john/doe' http://localhost:8000/any: Works
• curl -T small.bin -H 'Content-Type: john' http://localhost:8000/any: Fails
• curl -T small.bin -H 'Content-Type: application/doe' http://localhost:8000/any_app: Works

By expanding the limits, we can accept files larger than the defaults:

Rocket.toml

[debug.limits]
"file" = "1GB"

• curl -T large.bin -H 'Content-Type: application/doe' http://localhost:8000/any_app: Works

It is true that putting "application/*" as a limit does not work, however I don't believe that was ever the intention. It is possible to set something along the lines of "file/jpg", however this will not be used by a curl -T ... upload as in that scenario curl doesn't provide anyfilename, so the request guard doesn't have anything to filter on. The following, however, does work:

#[macro_use]
extern crate rocket;

use rocket::{form::Form, fs::TempFile};

#[derive(FromForm)]
struct Upload<'a> {
    f: TempFile<'a>,
}

#[post("/upload", format = "multipart/form-data", data = "<_f>")]
fn jpgform(_f: Form<Upload<'_>>) {}

#[launch]
fn rocket() -> _ {
    rocket::build().mount("/", routes![jpgform])
}

[debug.limits]
"data-form" = "1GB"
"file/jpeg" = "1GB"

curl --form "f=@large.bin;filename=foo.jpg" http://localhost:8000/upload

So we need to increase the whole data-form (because it emcompasses the uploaded file), and we can increase the limit for JPEG files specifically. Other file types won't be accepted.

PS: I created the small.bin and large.bin using dd:

• dd if=/dev/zero of=small.bin bs=1000 count=1 ~1KB
• dd if=/dev/zero of=large.bin bs=1000 count=10000 ~10MB

@SergioBenitez I believe this can be closed.