Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task-Centric Memory #5227

Open
wants to merge 134 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
442a9d8
initial checkin
rickyloynd-microsoft Nov 29, 2024
f8584cd
support for extensive evaluations
rickyloynd-microsoft Dec 2, 2024
607e7ff
Enhance retrieval with task generalization and insight validation
rickyloynd-microsoft Dec 4, 2024
b045636
Support TRAPI client.
rickyloynd-microsoft Dec 9, 2024
63b28d7
Restoring earlier results, and general cleanup.
rickyloynd-microsoft Dec 24, 2024
b921d83
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Dec 24, 2024
9dfb074
Modify imports after merge from main.
rickyloynd-microsoft Dec 24, 2024
93a5ca4
Log model and token counts.
rickyloynd-microsoft Dec 26, 2024
2cb9344
Only instantiate the client once.
rickyloynd-microsoft Dec 26, 2024
878f458
Fix bug that was duplicating insights across trials.
rickyloynd-microsoft Dec 26, 2024
21562f1
Add the Grader class.
rickyloynd-microsoft Dec 27, 2024
3a40b30
Adjustments for comparison tests.
rickyloynd-microsoft Dec 28, 2024
8622c5e
Test generalization over multiple tasks.
rickyloynd-microsoft Dec 30, 2024
20b26c1
Add teachability and a test for it.
rickyloynd-microsoft Dec 31, 2024
9d47227
Learning from demonstration, in-progress.
rickyloynd-microsoft Jan 1, 2025
52d4e00
In memory retrieval, validate insights separately rather than together.
rickyloynd-microsoft Jan 1, 2025
6b15777
Finish learning from demonstration.
rickyloynd-microsoft Jan 2, 2025
a18674c
Added RecordableChatCompletionClient as a guardrail during refactoring.
rickyloynd-microsoft Jan 3, 2025
52e213e
Ran 3 evals with session recording and replay.
rickyloynd-microsoft Jan 5, 2025
a440b0a
Add results to recorded sessions, including session length.
rickyloynd-microsoft Jan 5, 2025
cab51f1
Use yaml file for eval settings.
rickyloynd-microsoft Jan 7, 2025
d91e58c
Simplify paths and other settings.
rickyloynd-microsoft Jan 7, 2025
f1d7a2f
Renamed the memory classes.
rickyloynd-microsoft Jan 7, 2025
17d4c42
Apprentice.
rickyloynd-microsoft Jan 8, 2025
19654e8
Moved test into the evaluator, and removed eval.py's other util funct…
rickyloynd-microsoft Jan 8, 2025
7aa20c1
renaming
rickyloynd-microsoft Jan 8, 2025
83a7ddc
Rerouted calls to AgenticMemoryController through FastLearner.
rickyloynd-microsoft Jan 9, 2025
3047c1c
Replace task_assignment_callback with AgentWrapper.
rickyloynd-microsoft Jan 9, 2025
1f20b79
Segregate files into subfolders, eval framework vs. implementation, etc.
rickyloynd-microsoft Jan 10, 2025
de4c12b
Rename FastLearner subclass to Apprentice, and import it only as spec…
rickyloynd-microsoft Jan 10, 2025
a9d6108
Refactoring, preparatory to removing eval_framework from the branch a…
rickyloynd-microsoft Jan 11, 2025
d67e2cc
Remove the outdated final_format_instructions parameter.
rickyloynd-microsoft Jan 11, 2025
6470fd8
Move tasks into yaml files.
rickyloynd-microsoft Jan 12, 2025
b025199
Move client support to a subdir.
rickyloynd-microsoft Jan 12, 2025
4f9267c
Move evaluations to a separate dir.
rickyloynd-microsoft Jan 12, 2025
db34844
single line
rickyloynd-microsoft Jan 14, 2025
c780852
Add baseline evaluation for the no-memory case.
rickyloynd-microsoft Jan 16, 2025
fa688f7
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Jan 17, 2025
43bda2f
Support o1 models
rickyloynd-microsoft Jan 18, 2025
be081b3
simplification of client creation code
rickyloynd-microsoft Jan 18, 2025
29d1494
simplify folder structure
rickyloynd-microsoft Jan 18, 2025
8e9a550
Move task data strings out of the eval functions.
rickyloynd-microsoft Jan 20, 2025
b3fe084
simplify page_log
rickyloynd-microsoft Jan 21, 2025
077615f
simplify page_log
rickyloynd-microsoft Jan 21, 2025
8847168
simplify page_log
rickyloynd-microsoft Jan 21, 2025
4091ab3
conventional logging terminology
rickyloynd-microsoft Jan 22, 2025
3865cff
control logger enabling
rickyloynd-microsoft Jan 22, 2025
6c73674
add logging to string map
rickyloynd-microsoft Jan 22, 2025
db5e07b
simplify logging
rickyloynd-microsoft Jan 22, 2025
07cb3f0
simplify logging
rickyloynd-microsoft Jan 22, 2025
e88bd69
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Jan 22, 2025
9b3f77d
merge from main
rickyloynd-microsoft Jan 23, 2025
a0dee67
Changes made by poe check.
rickyloynd-microsoft Jan 23, 2025
7e359e9
docstrings etc.
rickyloynd-microsoft Jan 23, 2025
9466ea8
docstrings etc.
rickyloynd-microsoft Jan 24, 2025
4ec9bff
docstrings etc.
rickyloynd-microsoft Jan 24, 2025
76c16f9
docstrings etc.
rickyloynd-microsoft Jan 24, 2025
a8cd0d7
docstrings etc.
rickyloynd-microsoft Jan 24, 2025
ed7fae1
docstrings etc.
rickyloynd-microsoft Jan 25, 2025
93de858
docstrings etc.
rickyloynd-microsoft Jan 25, 2025
1a309f9
docstrings etc.
rickyloynd-microsoft Jan 25, 2025
8993aa1
docstrings etc.
rickyloynd-microsoft Jan 25, 2025
fa60d5a
Simplify naming
rickyloynd-microsoft Jan 25, 2025
882d578
Simplify tests
rickyloynd-microsoft Jan 26, 2025
00cbb8c
standardize logging levels
rickyloynd-microsoft Jan 27, 2025
88294d2
Remove Evaluator class
rickyloynd-microsoft Jan 27, 2025
7d0ed63
sample code
rickyloynd-microsoft Jan 27, 2025
5b3876f
readme
rickyloynd-microsoft Jan 28, 2025
21220d4
readme fixes
rickyloynd-microsoft Jan 28, 2025
232ed0f
samples readme
rickyloynd-microsoft Jan 28, 2025
87ee27b
readme files
rickyloynd-microsoft Jan 28, 2025
b21d140
readme files
rickyloynd-microsoft Jan 28, 2025
1e88eb6
remove ame
rickyloynd-microsoft Jan 28, 2025
a3addc1
readme
rickyloynd-microsoft Jan 28, 2025
c6ffa43
comment out api_key lines
rickyloynd-microsoft Jan 28, 2025
8f66612
Optional disabling of prefix caching (to decorrelate repeated runs)
rickyloynd-microsoft Jan 28, 2025
491964f
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Jan 28, 2025
2ed08ae
Remove unnecessary instantiation of Grader
rickyloynd-microsoft Jan 29, 2025
f879487
Updated image using git-lfs
rickyloynd-microsoft Jan 30, 2025
60f8ad3
Merge branch 'agentic_memory' of github.com:microsoft/autogen into ag…
rickyloynd-microsoft Jan 30, 2025
ed0a4a6
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Jan 30, 2025
f0eceef
installation fixes
rickyloynd-microsoft Jan 30, 2025
70db202
Refactor to remove AgentWrapper, and use AssistantAgent as a TaskRunn…
rickyloynd-microsoft Jan 31, 2025
5e4ad48
uv fixes
rickyloynd-microsoft Jan 31, 2025
b6c59ae
uv fixes
rickyloynd-microsoft Jan 31, 2025
bef7e5d
uv fixes
rickyloynd-microsoft Feb 1, 2025
1fb5ee4
uv fixes
rickyloynd-microsoft Feb 1, 2025
1d7f4eb
uv fixes
rickyloynd-microsoft Feb 1, 2025
516e689
uv fixes
rickyloynd-microsoft Feb 1, 2025
ffe719a
uv fixes
rickyloynd-microsoft Feb 1, 2025
ba14e78
uv fixes
rickyloynd-microsoft Feb 3, 2025
2eb817e
uv fixes
rickyloynd-microsoft Feb 3, 2025
95b1276
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Feb 3, 2025
6633169
uv fixes
rickyloynd-microsoft Feb 4, 2025
880df13
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Feb 4, 2025
b4ea0ce
uv fixes
rickyloynd-microsoft Feb 5, 2025
53da266
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Feb 5, 2025
6a04851
Add line to autogenstudio section of uv.lock
rickyloynd-microsoft Feb 5, 2025
ad514eb
poe check fixes
rickyloynd-microsoft Feb 5, 2025
18ae4dc
uv
rickyloynd-microsoft Feb 5, 2025
7be995d
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Feb 5, 2025
0e01720
hash output for detecting log changes
rickyloynd-microsoft Feb 6, 2025
0416e8d
Merge branch 'main' of github.com:microsoft/autogen
rickyloynd-microsoft Feb 6, 2025
0298591
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Feb 6, 2025
0bc0500
Make logger and config args optional
rickyloynd-microsoft Feb 7, 2025
66029cf
terminology: settings -> configs
rickyloynd-microsoft Feb 7, 2025
d9ad986
Merge branch 'main' of github.com:microsoft/autogen
rickyloynd-microsoft Feb 7, 2025
2b2cbdb
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Feb 7, 2025
566709b
Simplify the API in preparation for Webby
rickyloynd-microsoft Feb 11, 2025
2ef5e4a
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Feb 11, 2025
9e7d245
uv.lock
rickyloynd-microsoft Feb 11, 2025
1ce4cd9
Restore version for autogenstudio
rickyloynd-microsoft Feb 11, 2025
494d81e
Add a retrieval sample, which accesses the memory controller directly…
rickyloynd-microsoft Feb 12, 2025
54b0faa
Change of terminology: Agentic Memory -> Task-Centric Memory
rickyloynd-microsoft Feb 12, 2025
0b9f042
Move support files into utils subdir.
rickyloynd-microsoft Feb 12, 2025
a720863
Update readme files per reviewer feedback.
rickyloynd-microsoft Feb 14, 2025
b0e72a7
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Feb 14, 2025
dba5b55
Get API Reference documentation to build correctly.
rickyloynd-microsoft Feb 15, 2025
01d8b9d
Add code example provided by @ekzhu
rickyloynd-microsoft Feb 15, 2025
193466b
Added installation instructions and a code snippet to the docstring.
rickyloynd-microsoft Feb 15, 2025
39d460a
Code format fix in the docstring
rickyloynd-microsoft Feb 15, 2025
8f9d066
Use TypedDicts in the nested-config pattern to minimize code changes …
rickyloynd-microsoft Feb 17, 2025
94eab06
Merge branch 'refs/heads/main' into agentic_memory
rickyloynd-microsoft Feb 17, 2025
f892c18
Add clarifying diagrams.
rickyloynd-microsoft Feb 18, 2025
a15bfd1
fix image sizes
rickyloynd-microsoft Feb 18, 2025
3ea8011
Merge branch 'main' into agentic_memory
rickyloynd-microsoft Feb 19, 2025
58ecd7e
Add file docstrings to sample code.
rickyloynd-microsoft Feb 20, 2025
00e27e1
Merge branch 'agentic_memory' of github.com:microsoft/autogen into ag…
rickyloynd-microsoft Feb 20, 2025
4466eee
uv sync --all-extras
rickyloynd-microsoft Feb 20, 2025
64dc3c0
restore previous uv.lock
rickyloynd-microsoft Feb 20, 2025
e15d0eb
changes for webby
rickyloynd-microsoft Feb 21, 2025
af362f6
experimental
rickyloynd-microsoft Feb 21, 2025
4d6c9f4
docs
rickyloynd-microsoft Feb 23, 2025
261fe6f
Add Teachability(Memory)
rickyloynd-microsoft Feb 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Enhance retrieval with task generalization and insight validation
rickyloynd-microsoft committed Dec 4, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 607e7ff1cc9da023ef6dc45ecfceaab6cc775d9b
Original file line number Diff line number Diff line change
@@ -83,15 +83,18 @@ async def add_insight_to_memory(self, task: str, insight: str):
details="",
method_call="AgenticMemory.add_insight_to_memory")

# Generalize the task.
generalized_task = await self.prompter.generalize_task(task)

# Get a combined list of topics from the task and insight.
task_plus_insight = task.strip() + "\n(Hint: " + insight + ")"
task_plus_insight = generalized_task.strip() + "\n(Hint: " + insight + ")"
topics = await self.prompter.find_index_topics(task_plus_insight)
page.add_lines("\nTOPICS EXTRACTED FROM TASK AND INSIGHT:")
page.add_lines("\n".join(topics))
page.add_lines("")

# Add the insight to the archive.
self.archive.add_insight(insight, task, topics)
self.archive.add_insight(insight, generalized_task, topics)

self.page_log.finish_page(page)

@@ -102,8 +105,11 @@ async def retrieve_relevant_insights(self, task: str):
details="",
method_call="AgenticMemory.retrieve_relevant_insights")

# Generalize the task.
generalized_task = await self.prompter.generalize_task(task)

# Get a list of topics from the task.
topics = await self.prompter.find_index_topics(task)
topics = await self.prompter.find_index_topics(generalized_task)
page.add_lines("\nTOPICS EXTRACTED FROM TASK:")
page.add_lines("\n".join(topics))
page.add_lines("")
@@ -114,10 +120,14 @@ async def retrieve_relevant_insights(self, task: str):
page.add_lines("\nUNFILTERED INSIGHTS")
for insight, relevance in unfiltered_insights.items():
page.add_lines(" INSIGHT: {}\n RELEVANCE: {:.3f}".format(insight, relevance))
if relevance > 5.0:
filtered_insights.append(insight)
filtered_insights.append(insight)
page.add_lines("\nFiltered to top {} insights".format(len(filtered_insights)))

if len(filtered_insights) > 0:
# Apply a final filtering stage to keep only the insights that the LLM believes are relevant.
filtered_insights = await self.prompter.validate_insights(filtered_insights, task)
page.add_lines("\n{} insights were validated".format(len(filtered_insights)))

self.page_log.finish_page(page)
return filtered_insights

@@ -184,7 +194,6 @@ async def _iterate_on_task(self, task: str, expected_answer: str, assign_task_to
page.add_lines("----- TRAIN TRIAL {} -----\n".format(trial), flush=True)

# Add any new insights we've accumulated so far.
# memory_section = self.format_memory_section(old_insights + new_insights)
if last_insight is not None:
memory_section = self.format_memory_section(old_insights + [last_insight])
else:
Original file line number Diff line number Diff line change
@@ -50,6 +50,13 @@ def __init__(
self.last_insight_id = len(self.uid_insight_dict)
parent_page.add_lines("\n{} INSIGHTS LOADED".format(len(self.uid_insight_dict)))

def save_archive(self):
self.memo_store.save_memos()
parent_page = self.page_log.last_page()
parent_page.add_lines("\nSAVING INSIGHTS TO DISK {}".format(self.path_to_dict))
with open(self.path_to_dict, "wb") as file:
pickle.dump(self.uid_insight_dict, file)

def add_insight(self, insight_str: str, task_str: Optional[str] = None, topics: Optional[List[str]] = None):
"""Adds an insight to the knowledge archive."""
assert topics is not None, "For now, the topics list must be provided."
@@ -62,13 +69,6 @@ def add_insight(self, insight_str: str, task_str: Optional[str] = None, topics:
self.uid_insight_dict[str(id_str)] = insight
self.save_archive()

def save_archive(self):
self.memo_store.save_memos()
parent_page = self.page_log.last_page()
parent_page.add_lines("\nSAVING INSIGHTS TO DISK {}".format(self.path_to_dict))
with open(self.path_to_dict, "wb") as file:
pickle.dump(self.uid_insight_dict, file)

def get_relevant_insights(self, task_str: Optional[str] = None, topics: Optional[List[str]] = None):
"""Returns any insights from the knowledge archive that are relevant to the given task or topics."""
assert (task_str is not None) or (topics is not None), "Either the task string or the topics list must be provided."
Original file line number Diff line number Diff line change
@@ -181,3 +181,65 @@ async def find_index_topics(self, input_string):
topic_list.append(line)

return topic_list

async def generalize_task(self, task_description):
# Returns a list of topics related to the input string.

sys_message = """You are a helpful and thoughtful assistant."""

user_message = ["We have been given a task description. Our job is not to complete the task, but merely rephrase the task in simpler, more general terms, if possible. Please reach through the following task description, then explain your understanding of the task in detail, as a single, flat list of all the important points."]
user_message.append("\n# Task description")
user_message.append(task_description)

self.clear_history()
response1, page = await self.call_model(
system_message=sys_message,
user_content=user_message,
details="to rephrase the task in a list of important points")

user_message = ["Do you see any parts of this list that are irrelevant to actually solving the task? If so, explain which items are irrelevant."]
response2, page = await self.call_model(
system_message=sys_message,
user_content=user_message,
details="to identify irrelevant points")

user_message = ["Revise your original list to include only the most general terms, those that are critical to solving the task, removing any themes or descriptions that are not essential to the solution. Your final list may be shorter, but do not leave out any part of the task that is needed for solving the task. Do not add any additional commentary either before or after the list."]
generalized_task, page = await self.call_model(
system_message=sys_message,
user_content=user_message,
details="to make a final list of general terms")

return generalized_task

async def validate_insights(self, insights, task_description):
# Returns only the insights that the client verifies are relevant to the task.

sys_message = """You are a helpful and thoughtful assistant."""

user_message = ["""We have been given a list of insights that may or may not be useful for solving the given task.
- First review the following task.
- Then review the list of insights that follow, and discuss which ones could be useful in solving the given task.
- Do not attempt to actually solve the task. That will come later."""]
user_message.append("\n# Task description")
user_message.append(task_description)
user_message.append("\n# Possibly useful insights")
user_message.extend(insights)
self.clear_history()
response1, page = await self.call_model(
system_message=sys_message,
user_content=user_message,
details="to review the task and insights")

user_message = ["""Now output a verbatim copy the insights that you decided are relevant to the task.
- The original list of insights is provided below for reference.
- If an insight is not relevant to the task, simply omit it from your response.
- Do not add any additional commentary either before or after the relevant tasks.
- If none of the tasks are relevant, simply write "None"."""]
user_message.append("\n# Original list of possibly useful insights")
user_message.extend(insights)
validated_insights, page = await self.call_model(
system_message=sys_message,
user_content=user_message,
details="to list the relevant insights")

return [validated_insights] if validated_insights != "None" else []