Deep Research agent workflow project #228

strickvl · 2025-05-27T18:00:16Z

Summary

Please provide a short summary explaining the motivation behind these changes.

Checklist

I have read the contributing guidelines.
I have run the necessary tests and linters.
I have updated relevant documentation where applicable.

Related Issues

Please link to any relevant issues or discussions.

dagshub · 2025-05-27T18:00:19Z

Join the discussion on DagsHub!

Copilot

Pull Request Overview

This PR introduces a new deep research agent workflow project with several new files covering pipeline orchestration, configuration, logging, and custom materializers for report visualization and decision approval. Key changes include the implementation of the main research pipeline interface, multiple YAML configuration files for different research modes, and custom materializers for handling complex output formats.

Reviewed Changes

Copilot reviewed 55 out of 55 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
deep_research/run.py	Implements the main CLI and pipeline configuration with research mode presets
deep_research/pipelines/parallel_research_pipeline.py	Defines the parallelized deep research pipeline using ZenML's pipeline decorator
deep_research/materializers/*	Adds custom materializers for visualization of reflection outputs and approval decisions
deep_research/configs/*.yaml	Provides several configuration files for various research modes
deep_research/logging_config.py	Sets up logging configuration for the application

deep_research/materializers/reflection_output_materializer.py

deep_research/configs/deep_research.yaml

htahir1

Smaller things:

The main README.md needs an update
Have we tried with other models too?

Ive also stopped mid-way because i feel like we need to agree on some patterns moving forward that seem bigger to be fair:

How do we expect prompts to be versioned?
Do we want one big "ResearchState" model to be passed around as "memory" of the whole thing? Maybe we can call this "memory"?

In both cases I feel like the big problem is that downstream steps are so "entangled" with the structure of the schema and assume that these will be in a certain state. Therefore the chances of error are super high even if one small thing changes in the future

deep_research/logging_config.py

deep_research/materializers/approval_decision_materializer.py

deep_research/utils/helper_functions.py

deep_research/utils/llm_utils.py

deep_research/steps/initialize_prompts_step.py

deep_research/steps/query_decomposition_step.py

htahir1 · 2025-05-27T18:33:30Z

deep_research/steps/query_decomposition_step.py

+        decomposed_questions = get_structured_llm_output(
+            prompt=state.main_query,
+            system_prompt=updated_system_prompt,
+            model=llm_model,
+            fallback_response=fallback_questions,
+            project=langfuse_project_name,
+        )


where is the expected schema of the output? Is it embedde in state.main_query? If so thats also super hard to understand

yeah this is actually one bit from what you gave me that I never changed. I didn't want to dump in instructor or something like it to force the structured output as I never needed it so far.

strickvl · 2025-05-27T19:03:59Z

I'm not sure about using the word memory as it's just too overloaded at the moment.

Prompts are being versioned in git and also via ZenML as an artifact with the pydantic model we're passing around.

- Add appropriate tags to all artifacts across pipeline steps - Use add_tags() with artifact parameter matching Annotated names - Tags help categorize and filter artifacts in the ZenML dashboard

* Complete refactoring of ResearchState to artifact-based architecture This commit implements the design document for splitting the monolithic ResearchState into separate, named artifacts with custom visualizations. Major changes include: ## New Artifact Classes - **QueryContext**: Immutable context containing the research query and sub-questions - **SearchData**: All search results and cost tracking information - **SynthesisData**: Synthesized information from searches (including enhanced versions) - **AnalysisData**: Cross-viewpoint analysis and reflection metadata - **FinalReport**: The generated HTML report with metadata ## Custom Materializers Each artifact now has its own materializer with beautiful HTML visualizations: - Interactive charts using Chart.js for search costs - Collapsible sections for better organization - Consistent styling across all artifact views - Metadata tables with key statistics ## Pipeline Updates - All steps refactored to use the new artifact-based approach - Proper dependencies established between parallel steps - Fixed merge step to run after parallel sub-question processing - Updated metadata logging and tagging throughout ## Bug Fixes - Fixed log_metadata calls to include infer_artifact=True parameter - Fixed template variable names in final report generation - Corrected enhanced_info merging logic to preserve original synthesis data - Added proper step dependencies in parallel pipeline ## Test Updates - Updated all tests to use the new artifact-based interface - Tests now create individual artifacts instead of ResearchState - Maintained test coverage for all functionality This refactoring improves modularity, enables better artifact visualization in the ZenML dashboard, and makes the pipeline more maintainable and extensible. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Delete legacy code --------- Co-authored-by: Claude <[email protected]>

- Created shared CSS file at assets/styles.css consolidating all common styles - Added CSS utility functions in utils/css_utils.py for consistent styling - Updated all materializers to use shared CSS instead of inline styles - Updated report template in prompts.py to use shared CSS - Fixed unused 'question' variable in approval_decision_materializer.py - Reduced code duplication across all visualization components This refactoring improves maintainability by centralizing style definitions and provides consistent visual styling across all materializers. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

…yles - Update STATIC_HTML_TEMPLATE in utils/prompts.py to use shared CSS - Replace ~650 lines of inline CSS with reference to shared styles - Maintain report-specific styles only where necessary - Achieve significant code reduction across the codebase

- Move LLM output processing functions to llm_utils.py (remove_reasoning_from_output, clean_json_tags, clean_markdown_tags, safe_json_loads) - Move HTML extraction to css_utils.py (extract_html_from_content) - Create new config_utils.py for configuration management (load_pipeline_config, check_required_env_vars) - Update all imports across the codebase - Delete the now-empty helper_functions.py This improves code organization by grouping related functionality together. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Copilot

Pull Request Overview

Adds a new CLI entry point and supporting ZenML pipeline with rich visualizations and configuration presets.

Introduces run.py for the Deep Research Agent CLI with mode presets
Defines a parallelized ZenML pipeline in pipelines/parallel_research_pipeline.py
Implements several custom materializers for interactive HTML output
Adds YAML configuration files for rapid, balanced, and deep research modes
Updates requirements.txt with necessary dependencies

Reviewed Changes

Copilot reviewed 52 out of 52 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
deep_research/run.py	CLI entry point applying presets, logging, and env checks
deep_research/pipelines/parallel_research_pipeline.py	Defines the parallel research ZenML pipeline
deep_research/materializers/*	Custom Pydantic materializers generating HTML views
deep_research/configs/*.yaml	Preset configurations for different research modes
deep_research/requirements.txt	Adds and pins project dependencies

deep_research/pipelines/parallel_research_pipeline.py

deep_research/run.py

- Use Click context to detect user-provided overrides instead of hardcoded defaults - Set explicit default="tavily" for search-provider to match help text 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

htahir1 · 2025-06-02T14:56:50Z

Merging for now we do follow ups later

strickvl added 2 commits May 27, 2025 19:57

commented files

758a712

initial commit

09e27e2

strickvl requested review from htahir1 and Copilot May 27, 2025 18:00

strickvl added enhancement New feature or request internal labels May 27, 2025

Copilot AI reviewed May 27, 2025

View reviewed changes

deep_research/materializers/reflection_output_materializer.py Outdated Show resolved Hide resolved

deep_research/configs/deep_research.yaml Show resolved Hide resolved

htahir1 requested changes May 27, 2025

View reviewed changes

Refactor PromptBundle into individual prompts

142f2aa

strickvl marked this pull request as draft May 27, 2025 20:15

strickvl and others added 7 commits May 27, 2025 22:16

Fix the tracingmetadata visualization

5d3bd77

Add artifact tags to all pipeline steps

1a80539

- Add appropriate tags to all artifacts across pipeline steps - Use add_tags() with artifact parameter matching Annotated names - Tags help categorize and filter artifacts in the ZenML dashboard

Delete old configs + set gemini flash 2-lite as default base model

3830977

strickvl requested a review from Copilot May 28, 2025 08:20

Copilot AI reviewed May 28, 2025

View reviewed changes

deep_research/pipelines/parallel_research_pipeline.py Show resolved Hide resolved

deep_research/run.py Outdated Show resolved Hide resolved

deep_research/run.py Outdated Show resolved Hide resolved

strickvl requested a review from htahir1 May 28, 2025 15:57

strickvl marked this pull request as ready for review May 28, 2025 16:53

htahir1 merged commit b0b4e11 into main Jun 2, 2025
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Deep Research agent workflow project #228

Deep Research agent workflow project #228

Uh oh!

strickvl commented May 27, 2025

Uh oh!

dagshub bot commented May 27, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

htahir1 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htahir1 May 27, 2025

Uh oh!

strickvl May 27, 2025

Uh oh!

strickvl commented May 27, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htahir1 commented Jun 2, 2025

Uh oh!

Uh oh!

Uh oh!

Deep Research agent workflow project #228

Deep Research agent workflow project #228

Uh oh!

Conversation

strickvl commented May 27, 2025

Summary

Checklist

Related Issues

Uh oh!

dagshub bot commented May 27, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

htahir1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htahir1 May 27, 2025

Choose a reason for hiding this comment

Uh oh!

strickvl May 27, 2025

Choose a reason for hiding this comment

Uh oh!

strickvl commented May 27, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htahir1 commented Jun 2, 2025

Uh oh!

Uh oh!

Uh oh!