feat: update clickstar gif

README.md

@@ -38,7 +38,7 @@
   If you like Dingo, please give us a ⭐ on GitHub!
   <br/>
   <a href="https://github.com/DataEval/dingo/stargazers" target="_blank">
-    <img src="docs/assets/clickstar.gif" alt="Click Star" width="480">
+    <img src="docs/assets/clickstar_2.gif" alt="Click Star" width="480">
   </a>
 </p>
 

README_ja.md

@@ -37,7 +37,7 @@
   このプロジェクトが役に立ったら、GitHubで⭐を付けてください！
   <br/>
   <a href="https://github.com/DataEval/dingo/stargazers" target="_blank">
-    <img src="docs/assets/clickstar.gif" alt="Star をクリック" width="480">
+    <img src="docs/assets/clickstar_2.gif" alt="Star をクリック" width="480">
   </a>
 </p>
 

README_zh-CN.md

@@ -33,7 +33,7 @@
   如果觉得有帮助，欢迎在 GitHub 上点个 ⭐ 支持！
   <br/>
   <a href="https://github.com/DataEval/dingo/stargazers" target="_blank">
-    <img src="docs/assets/clickstar.gif" alt="点击 Star 支持" width="480">
+    <img src="docs/assets/clickstar_2.gif" alt="点击 Star 支持" width="480">
   </a>
 </p>
 

dingo/model/llm/llm_meta_rater_evaluation.py

@@ -22,26 +22,26 @@
 class LLMMetaRaterEvaluation(BaseOpenAI):
     """
     Unified LLM model for Meta-rater PRRC dimensions evaluation.
-    
+
     This model provides a single interface for evaluating multiple aspects
     of text quality based on the Meta-rater paper's PRRC framework:
     - Professionalism: Degree of expertise required
-    - Readability: Clarity and coherence  
+    - Readability: Clarity and coherence
     - Reasoning: Logical depth and complexity
     - Cleanliness: Formatting and content appropriateness
-    
+
     The specific evaluation type is determined by the prompt used.
     """
-    prompt = PromptMetaRaterProfessionalism # Default prompt
+    prompt = PromptMetaRaterProfessionalism  # Default prompt
 
     @classmethod
     def build_messages(cls, input_data: Data) -> List:
         """
         Build messages for the LLM API call.
-        
+
         Args:
             input_data: Data object containing text content to evaluate
-            
+
         Returns:
             List: Formatted messages for LLM API
         """
@@ -53,10 +53,10 @@ def build_messages(cls, input_data: Data) -> List:
     def process_response(cls, response: str) -> ModelRes:
         """
         Process the LLM response for Meta-rater evaluation.
-        
+
         Args:
             response: Raw response string from the LLM
-            
+
         Returns:
             ModelRes: Processed evaluation results with score and reason
         """
@@ -70,7 +70,7 @@ def process_response(cls, response: str) -> ModelRes:
             cleaned_response = cleaned_response[3:]
         if cleaned_response.endswith('```'):
             cleaned_response = cleaned_response[:-3]
-        
+
         try:
             response_json = json.loads(cleaned_response)
         except json.JSONDecodeError:
@@ -79,9 +79,9 @@ def process_response(cls, response: str) -> ModelRes:
         # Extract score and reason from response
         score = response_json.get('score', 0)
         reason = response_json.get('reason', '')
-        
+
         result = ModelRes()
-        
+
         # Meta-rater uses 1-5 scoring, with higher scores being better;
         # We normalize this to binary classification for compatibility
         # Scores >= 3 are considered "good quality", < 3 are "low quality"

dingo/model/prompt/prompt_meta_rater.py

@@ -17,13 +17,13 @@
 You are an expert evaluator. Below is an extract from a text source such as a web page, book, academic paper, Github, Wikipedia, or StackExchange. Evaluate the PROFESSIONALISM of the text, that is, the degree of expertise and prerequisite knowledge required to comprehend it, using the additive 5-point scoring system described below. Your evaluation should be based on the depth, accuracy, and accessibility of the content, without considering the writing style, grammar, spelling, or punctuation in your scoring.
 
 Points are accumulated based on the satisfaction of each criterion:
-- Add 1 point if the text is relatively simple and requires minimal technical knowledge or expertise to understand. The text might include nursery rhymes, children's books, or other basic content that is accessible to a broad audience. The information provided is straightforward and does not delve into complex concepts or specialized topics. 
+- Add 1 point if the text is relatively simple and requires minimal technical knowledge or expertise to understand. The text might include nursery rhymes, children's books, or other basic content that is accessible to a broad audience. The information provided is straightforward and does not delve into complex concepts or specialized topics.
 - Add another point if the text is somewhat more complex and might require a basic level of specialized knowledge to comprehend fully. Examples could include popular books, popular science articles, or novels. The content delves a little deeper into the subject matter, but it remains accessible to a reasonably broad audience.
 - Award a third point if the text falls in the middle of the spectrum, requiring some degree of expertise to understand but not being overly complex or specialized. The content might encompass more advanced books, detailed articles, or introductions to complex topics. It provides a decent level of depth and detail, but it does not require an extensive background in the subject matter to understand.
 - Grant a fourth point if the text is complicated and requires a significant level of expertise and technical knowledge. Examples might include academic papers, advanced textbooks, or detailed technical reports. The content is detailed and accurate, but it could be inaccessible to those without a strong background in the subject matter.
 - Bestow a fifth point if the text is extremely high in professionalism, requiring a high degree of subject matter expertise and prerequisite knowledge. The text is likely limited to those with advanced understanding or experience in the field, such as advanced academic papers, complex technical manuals, or patents. The content is deep, accurate, and insightful, but largely inaccessible to those without a significant background in the topic.
 
-Here are three aspects that should NOT influence your judgement: 
+Here are three aspects that should NOT influence your judgement:
 (1) The specific language the text is written in
 (2) The length of text
 (3) Usage of placeholders for data privacy or safety, e.g. @CAPS1, [EMAIL_ADDRESS], [PHONE_NUMBER], and so on.
@@ -46,16 +46,16 @@
 I am a data scientist interested in exploring data in the pre-training stage of large language models.
 
 # OBJECTIVE #
-You are an expert evaluator. Below is an extract from a text source such as a web page, book, academic paper, Github, Wikipedia, or StackExchange. Evaluate whether the page has a high READABILITY using the additive 5-point scoring system described below. 
+You are an expert evaluator. Below is an extract from a text source such as a web page, book, academic paper, Github, Wikipedia, or StackExchange. Evaluate whether the page has a high READABILITY using the additive 5-point scoring system described below.
 
-Points are accumulated based on the satisfaction of each criterion： 
+Points are accumulated based on the satisfaction of each criterion：
 - Add 1 point if the text is somewhat readable but contains significant issues with clarity or coherence. It might include complex vocabulary or sentence structures that require advanced reading skills, or it might have numerous grammar and spelling errors.
 - Add another point if the text is generally clear and coherent, but there are sections that are difficult to comprehend due to occasional grammar, spelling errors, or convoluted sentence structures.
 - Award a third point if the text is clear and coherent for the most part, using appropriate vocabulary and sentence structures that are easy to understand. Minor issues with grammar or spelling might still be present.
 - Grant a fourth point if the text is very clear and coherent, with very few or no errors in grammar and spelling. The text uses proper punctuation, vocabulary, and sentence structures that are easy to follow and understand.
 - Bestow a fifth point if the text is outstanding in its clarity and coherence. It uses language and sentence structures that are easy to comprehend, while also conveying ideas and nuances effectively. Minor errors in grammar, spelling, and punctuation are allowed, but they should not interfere with the overall understanding of the text.
 
-Here are three aspects that should NOT influence your judgement: 
+Here are three aspects that should NOT influence your judgement:
 (1) The specific language the text is written in
 (2) The length of text
 (3) Usage of placeholders for data privacy or safety, e.g. @CAPS1, [EMAIL_ADDRESS], [PHONE_NUMBER], and so on.
@@ -77,16 +77,16 @@
 I am a data scientist interested in exploring data in the pre-training stage of large language models.
 
 # OBJECTIVE #
-You are an expert evaluator. Below is an extract from a text source such as a web page, book, academic paper, Github, Wikipedia, or StackExchange. Evaluate whether the page has a high REASONING using the additive 5-point scoring system described below. 
+You are an expert evaluator. Below is an extract from a text source such as a web page, book, academic paper, Github, Wikipedia, or StackExchange. Evaluate whether the page has a high REASONING using the additive 5-point scoring system described below.
 
-Points are accumulated based on the satisfaction of each criterion： 
+Points are accumulated based on the satisfaction of each criterion：
 Add 1 point if the content contains preliminary elements of reasoning, possibly involving a single causal relationship or simple logical judgments, but lacks in-depth analysis (e.g., presenting a viewpoint without supporting evidence or detailed explanations).
 Add another point if the content demonstrates basic reasoning ability, incorporating some logical relationships that require the reader to engage in a certain level of thought. This may involve simple argumentative structures or examples, but the analysis remains superficial (e.g., providing a problem and a straightforward solution with some examples but lacking depth).
 Award a third point if the content exhibits a good level of reasoning complexity, involving multiple reasoning steps that require more complex thought from the reader. The reader should be able to identify several interrelated arguments and may include some depth of analysis (e.g., analyzing how different factors influence an outcome or comparing various viewpoints).
 Grant a fourth point if the content has a high level of reasoning complexity, including multi-layered logical reasoning and in-depth analysis. The reader needs to engage in complex thinking and can identify multiple interconnected arguments while conducting a comprehensive evaluation (e.g., analyzing multiple variables or assessing the pros and cons of different solutions).
 Bestow a fifth point if the content excels in reasoning complexity, demanding deep analysis and innovative thinking from the reader. The reasoning process is complex and multidimensional, involving interdisciplinary knowledge, requiring the reader to integrate various pieces of information to make comprehensive judgments (e.g., discussing complex mathematical models, designing optimization algorithms, or engaging in high-level strategic thinking).
 
-Here are three aspects that should NOT influence your judgement: 
+Here are three aspects that should NOT influence your judgement:
 (1) The specific language the text is written in
 (2) The length of text
 (3) Usage of placeholders for data privacy or safety, e.g. @CAPS1, [EMAIL_ADDRESS], [PHONE_NUMBER], and so on.
@@ -97,7 +97,7 @@
 professional, objective, formal, and clear.
 # AUDIENCE #
 Data scientists and other professionals interested in data for large language models.
-# RESPONSE # 
+# RESPONSE #
 Return the results in JSON format: {{"score": x, "reason": "xxx"}}.
 
 Here is the text:
@@ -108,9 +108,9 @@
 I am a data scientist interested in exploring data in the pre-training stage of large language models.
 
 # OBJECTIVE #
-You are an expert evaluator. Below is an extract from a text source such as a web page, book, academic paper, Github, Wikipedia, or StackExchange. Evaluate whether the page has a high CLEANLINESS using the additive 5-point scoring system described below. 
+You are an expert evaluator. Below is an extract from a text source such as a web page, book, academic paper, Github, Wikipedia, or StackExchange. Evaluate whether the page has a high CLEANLINESS using the additive 5-point scoring system described below.
 
-Points are accumulated based on the satisfaction of each criterion： 
+Points are accumulated based on the satisfaction of each criterion：
 A score of 1 indicates serious issues that affect fluency.
 A score of 2 indicates the text has obvious problems that affect fluency.
 A score of 3 means that the text has some problems but does not seriously affect reading fluency.
@@ -124,7 +124,7 @@
 - Completeness Content: The body of the article consists of complete sentences written naturally by humans, rather than phrases and lists, containing opinions, facts or stories.
 However, if there is a $TRUNCATED$ symbol at the end, it should be considered as a manual article ending flag set by the author, and there is no need to consider completeness.
 
-Here are three aspects that should NOT influence your judgement: 
+Here are three aspects that should NOT influence your judgement:
 (1) The specific language the text is written in
 (2) The length of text
 (3) Usage of placeholders for data privacy or safety, e.g. @CAPS1, [EMAIL_ADDRESS], [PHONE_NUMBER], and so on.
@@ -146,8 +146,8 @@
 class PromptMetaRaterProfessionalism(BasePrompt):
     """
     Prompt class for Meta-rater Professionalism evaluation.
-    
-    Evaluates the degree of expertise and prerequisite knowledge required to 
+
+    Evaluates the degree of expertise and prerequisite knowledge required to
     comprehend text on a 5-point scale.
     """
 
@@ -169,8 +169,8 @@ class PromptMetaRaterProfessionalism(BasePrompt):
 class PromptMetaRaterReadability(BasePrompt):
     """
     Prompt class for Meta-rater Readability evaluation.
-    
-    Evaluates the clarity and coherence of text using appropriate vocabulary 
+
+    Evaluates the clarity and coherence of text using appropriate vocabulary
     and sentence structures on a 5-point scale.
     """
 
@@ -192,7 +192,7 @@ class PromptMetaRaterReadability(BasePrompt):
 class PromptMetaRaterReasoning(BasePrompt):
     """
     Prompt class for Meta-rater Reasoning evaluation.
-    
+
     Evaluates the reasoning complexity and logical depth of text content,
     from simple logical judgments to complex multidimensional analysis on a 5-point scale.
     """
@@ -215,7 +215,7 @@ class PromptMetaRaterReasoning(BasePrompt):
 class PromptMetaRaterCleanliness(BasePrompt):
     """
     Prompt class for Meta-rater Cleanliness evaluation.
-    
+
     Evaluates text formatting, content appropriateness, and completeness,
     assessing whether text appears human-edited and free from noise on a 5-point scale.
     """
@@ -231,4 +231,4 @@ class PromptMetaRaterCleanliness(BasePrompt):
         "evaluation_results": ""
     }
 
-    content = META_RATER_CLEANLINESS_PROMPT
+    content = META_RATER_CLEANLINESS_PROMPT

dingo/model/rule/rule_image.py

@@ -290,9 +290,9 @@ def eval(cls, input_data: Data) -> ModelRes:
                 time.sleep(2)
 
             return ModelRes(
-                error_status=True if status_data['score_overall'] < 6 else False,
+                error_status=True if status_data['score_overall'] < cls.dynamic_config.threshold else False,
                 type="Artimuse_Succeeded",
-                name="BadImage" if status_data['score_overall'] < 6 else "GoodImage",
+                name="BadImage" if status_data['score_overall'] < cls.dynamic_config.threshold else "GoodImage",
                 reason=[json.dumps(status_data['aspects'], ensure_ascii=False)],
             )
         except Exception as e:

examples/meta_rater/sdk_meta_rater_evaluation.py

@@ -1,7 +1,6 @@
 from dingo.config import InputArgs
 from dingo.exec import Executor
 
-
 if __name__ == '__main__':
     input_data = {
         "input_path": "../../test/data/test_meta_rater.jsonl",
@@ -13,7 +12,7 @@
             }
         },
         "executor": {
-            "prompt_list": ["PromptMetaRaterProfessionalism"], # options: "PromptMetaRaterReadability", "PromptMetaRaterReasoning", "PromptMetaRaterCleanliness"
+            "prompt_list": ["PromptMetaRaterProfessionalism"],  # options: "PromptMetaRaterReadability", "PromptMetaRaterReasoning", "PromptMetaRaterCleanliness"
             "result_save": {
                 "bad": True,
                 "good": True