About the use of Radar

[xdcui2020]

I have a few questions about the use, and I hope to get your answers. Thank you.

1. Commercial detection tools such as gptzero will limit the length of the text, for example, there are at least 250 words. What is rader's limit on the length of the text, and how many words are there to get a credible probability? I found that if the text is short, then the text will be recognized as AI.
2. When testing, can the text contain emoticons or other symbols, and what impact will they have on the test results?

[pinyuchen]

Hello @xdcui2020

These are great questions!

1. In practice we recommend to use the detector with at lease 100 tokens. In Fig.6 of https://openreview.net/forum?id=DEJIDCmWOz the authors have compared RADAR with other watermark-based detection mentors with varying token lengths
2. We haven’t done studies like these explicitly. We simply ask LLMs to generate/rewrite text (which may or may not include symbols). Would be interesting to investigate text with other symbols.

[xdcui2020]

Thank you for your reply. I often use Radar to test whether the posts on Instagram are written by AI, but I find that the probability of Vicuna output is very high, which is greater than 0.9 or even 0.99. Maybe the post is too short. I tested it on more than 100 words, and the result is still very poor. 90% of the post content detection results are more than 0.9, and Dolly's results seem to be more reliable. Could you please provide Dolly's Checkpoint and model card, which is very important to me and thank you very much?

[pinyuchen]

Hello @xdcui2020. Thank you for your interest and support in using our tool. The use case you provided is fascinating. You are right that these detectors could have been further calibrated. Right now, we only released the checkpoint trained with Vinuca, which has a non-commercial license. We are planning to provide these detectors as a service soon. Please send me an email if you're interested in using the tools.

[xdcui2020]

My email address is xdcuimath@gmail.com. I don't use radar for commercial purposes. radar is the most reliable and coded model I have found, but it is still far from the commercial model (gptzero, etc.) in terms of accuracy. I look forward to seeing more mature and cheaper solutions. Of course, it is best to publish the checkpoint of other models. Thank you for your reply.

[OverglowWT]

Hello @pinyuchen , I’ve been using the online demo to test various texts and found that RADAR-Camel-5B performs noticeably better than the Vicuna-based version in many of my evaluations. For example, Camel-5B gives more calibrated scores on o3 generated and paraphrasaed texts. Unfortunately, I couldn’t find a checkpoint for the Camel-5B model—only Vicuna is available at the moment. Would it be possible to share the Camel-5B checkpoint for non-commercial, research-only use? I’d like to test it more systematically on a wider set of texts and contribute findings if helpful.

[OverglowWT]

Hello @pinyuchen , I’ve been using the online demo to test various texts and found that RADAR-Camel-5B performs noticeably better than the Vicuna-based version in many of my evaluations. For example, Camel-5B gives more calibrated scores on o3 generated and paraphrasaed texts. Unfortunately, I couldn’t find a checkpoint for the Camel-5B model—only Vicuna is available at the moment. Would it be possible to share the Camel-5B checkpoint for non-commercial, research-only use? I’d like to test it more systematically on a wider set of texts and contribute findings if helpful.

My email address is weehaotang@gmail.com. Thank you for your outstanding work on this project!