关于更多的实验数据

[Belye]

作者你好，在最后的Further Analysis章节显示，您这边应该还有很多实验没有贴在论文上，请问下这些实验，大概什么时候能够更新，我现在比较想知道[MemVerse]在locomo上的表现

[Wings-Of-Disaster]

感谢你的关注和提问. 关于 Further Analysis 中提到但未在论文中完整展示的实验，我们确实还进行了若干补充评测.

其中，MemVerse 在 LoCoMo 上的结果基于社区中常用的 LLM-as-a-judge 评估模板，得到的分数接近满分. 但在后续对评估 prompt 的人工核查中，我们认为该设置在该任务上并不完全合理，可能高估模型的实际能力，因此未在公开版本中详细汇报，仅在提交论文中进行了说明.

此外，我们认为 LoCoMo 对长时间间隔对话的覆盖相对有限，而这类场景在真实应用中较为常见. 基于这一观察，我们正在筹备一个更贴近实际使用场景的 benchmark，相关结果将于后续更新.

[Belye]

用的是mem0的评估模版？ 也就是

ACCURACY_PROMPT = """
Your task is to label an answer to a question as ’CORRECT’ or ’WRONG’. You will be given the following data:
(1) a question (posed by one user to another user),
(2) a ’gold’ (ground truth) answer,
(3) a generated answer
which you will score as CORRECT/WRONG.

The point of the question is to ask about something one user should know about the other user based on their prior conversations.
The gold answer will usually be a concise and short answer that includes the referenced topic, for example:
Question: Do you remember what I got the last time I went to Hawaii?
Gold answer: A shell necklace
The generated answer might be much longer, but you should be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT.

For time related questions, the gold answer will be a specific date, month, year, etc. The generated answer might be much longer or use relative time references (like "last Tuesday" or "next month"), but you should be generous with your grading - as long as it refers to the same date or time period as the gold answer, it should be counted as CORRECT. Even if the format differs (e.g., "May 7th" vs "7 May"), consider it CORRECT if it's the same date.

Now it's time for the real question:
Question: {question}
Gold answer: {gold_answer}
Generated answer: {generated_answer}

First, provide a short (one sentence) explanation of your reasoning, then finish with CORRECT or WRONG.
Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script.

Just return the label CORRECT or WRONG in a json format with the key as "label".
"""

[Wings-Of-Disaster]

用的是mem0的评估模版？ 也就是

ACCURACY_PROMPT = """ Your task is to label an answer to a question as ’CORRECT’ or ’WRONG’. You will be given the following data: (1) a question (posed by one user to another user), (2) a ’gold’ (ground truth) answer, (3) a generated answer which you will score as CORRECT/WRONG.

The point of the question is to ask about something one user should know about the other user based on their prior conversations. The gold answer will usually be a concise and short answer that includes the referenced topic, for example: Question: Do you remember what I got the last time I went to Hawaii? Gold answer: A shell necklace The generated answer might be much longer, but you should be generous with your grading - as long as it touches on the same topic as the gold answer, it should be counted as CORRECT.

For time related questions, the gold answer will be a specific date, month, year, etc. The generated answer might be much longer or use relative time references (like "last Tuesday" or "next month"), but you should be generous with your grading - as long as it refers to the same date or time period as the gold answer, it should be counted as CORRECT. Even if the format differs (e.g., "May 7th" vs "7 May"), consider it CORRECT if it's the same date.

Now it's time for the real question: Question: {question} Gold answer: {gold_answer} Generated answer: {generated_answer}

First, provide a short (one sentence) explanation of your reasoning, then finish with CORRECT or WRONG. Do NOT include both CORRECT and WRONG in your response, or it will break the evaluation script.

Just return the label CORRECT or WRONG in a json format with the key as "label". """

是的，我们还调查了其他report LLM as a Judge分数的方法，都用了同一套评测prompt