AI-powered marking #1248
AI-powered marking #1248
Conversation
There was a problem hiding this comment.
Thanks for this feature! Quite a bit of comments. Please look through, resolve them. I do have some clarification comments as well so please answer those as well.
One question I have is when are these comments used? I couldn't find anytime in which the comments are returned/retrieved to/by the FE.
| Retrieves an AI comment for a specific submission and question. | ||
| Returns `nil` if no comment exists. | ||
| """ | ||
| def get_ai_comments_for_submission(submission_id, question_id) do |
There was a problem hiding this comment.
Naming implies you are getting all AI comments. Also what is the use case for getting only one of the comments?
There was a problem hiding this comment.
It appears this function is not used either
There was a problem hiding this comment.
I think we can remove this function then.
There was a problem hiding this comment.
Will keep this for eventual AI comments retrieval when loading a submission
…add-AI-generated-comments-grading
|
@sentry review |
There was a problem hiding this comment.
Pull Request Overview
This PR introduces an AI-powered marking feature that leverages LLMs to automate feedback generation for code submissions in Source Academy, reducing grading workload for teaching assistants.
Key Changes:
- Added AI comment generation infrastructure with OpenAI API integration and AES-GCM encrypted API key storage at course level
- Created database schema and logging system for AI-generated comments with new endpoints for generating, saving, and managing AI feedback
- Extended existing models with LLM-related fields including course-level, assessment-level, and question-level prompts
Reviewed Changes
Copilot reviewed 30 out of 31 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| lib/cadet_web/controllers/generate_ai_comments.ex | New controller implementing AI comment generation, LLM API interaction, and comment persistence |
| lib/cadet/ai_comments.ex | New module handling CRUD operations for AI comments |
| lib/cadet/ai_comments/ai_comment.ex | Schema definition for ai_comment_logs table |
| lib/cadet/courses/course.ex | Added LLM configuration fields and AES-GCM encryption logic for API keys |
| lib/cadet/assessments/assessments.ex | Extended get_answers_in_submission to support AI comments and added assessment prompt retrieval |
| priv/repo/migrations/* | Database migrations for LLM features and ai_comment_logs table |
| test/cadet_web/controllers/ai_code_analysis_controller_test.exs | Test coverage for AI comment generation endpoints |
| config/test.exs | Added encryption key configuration for testing |
| lib/cadet_web/router.ex | Registered new AI comment endpoints |
| lib/cadet_web/admin_views/admin_grading_view.ex | Updated grading view to include AI comments |
| test/support/seeds.ex | Updated test fixtures with new fields |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
priv/repo/migrations/20240320000001_add_llm_api_key_to_courses.exs
Outdated
Show resolved
Hide resolved
Co-authored-by: Copilot <[email protected]>
…ource-academy/backend into feat/add-AI-generated-comments-grading
|
@sentry review |
![sentry-io[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
6 participants
Description
Created a new feature to automate feedback generation of code submissions in Source Academy, with the help of LLMs. This will help save TAs some time in grading code submissions from students!
generate_ai_commentsendpoint to fetch question details and generate AI-generated comments for submissions.save_chosen_commentsendpoint to save multiple chosen comments for a submission and question for logging purposes.save_final_commentendpoint to save the final comment chosen for a submission for logging purposes.ai_comment_logstable to log various data points from inputs, original student's code, outputs generated by LLM, comments chosen, and final comment.AICommentsmodule to handle creation, retrieval, and updates for AI comments, including saving final and chosen comments.AICodeAnalysisControllerto handle AI comment generation, saving final comments, and saving chosen comments.generate_ai_comments,save_final_comment, andsave_chosen_commentsendpoints inAICodeAnalysisControllerTest.Note: This may require changes to the DB diagram in README.md.
Type of Change
Checklist