Skip to content

Gibbs test | Fix dynamic model test in Gibbs sampler suite #2579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 27, 2025
Merged

Conversation

AoifeHughes
Copy link
Contributor

The existing dynamic model test was problematic:

  • Used a complex Dirichlet Process mixture model that was slow (10,000 samples)
  • Only performed regression testing against "old" values with a TODO indicating it wasn't ideal
  • Failed to properly test the core dynamic model functionality that Gibbs sampling should support

Here I've changed this with:

a new dynamic_bernoulli_normal model that:

  • Has varying dimensions: A Bernoulli variable b controls whether the model has 1 or 2 parameters
  • Has analytical ground truth: Marginal likelihoods and posterior for b can be computed exactly
  • Is fast to sample: Only 1,000 samples vs 10,000, runs in ~7-12 seconds
  • Tests core functionality: Validates that Gibbs sampling correctly handles dimension changes without errors

@AoifeHughes AoifeHughes requested a review from Copilot June 4, 2025 09:23
@AoifeHughes AoifeHughes self-assigned this Jun 4, 2025
Copilot

This comment was marked as outdated.

@AoifeHughes
Copy link
Contributor Author

Leave this as a draft for a teeny bit, I need to properly test and fully understand how julia tests work so I'm not breaking what I think I understand - no need for any reviews yet.

@AoifeHughes
Copy link
Contributor Author

This should address: #2402 when completed.

@mhauru
Copy link
Member

mhauru commented Jun 4, 2025

As discussed on Slack, feel free to change the base branch of this to main. Since it's not a user-interface breaking change no need to make a breaking release for it and make it go through the breaking branch.

@AoifeHughes AoifeHughes changed the base branch from breaking to main June 4, 2025 10:01
@AoifeHughes
Copy link
Contributor Author

Results from my local testing:

Test Summary:                                          | Pass  Total      Time
Turing                                                 | 2613   2613  19m50.9s
  Aqua                                                 |           0      0.1s
  AD                                                   |           0      0.0s
  essential                                            |           0      0.0s
  samplers (without AD)                                |           0      0.0s
  inference with samplers                              | 2613   2613  19m50.8s
    GibbsContext                                       |  868    868     27.7s
      type stability                                   |  868    868     27.7s
    Invalid Gibbs constructor                          |    8      8      0.1s
    Sampler call order                                 |    1      1     53.9s
    Equivalence of RepeatSampler and repeating Sampler |    1      1      4.5s
    Gibbs warmup                                       |    8      8      2.3s
    Testing gibbs.jl                                   | 1727   1727  18m19.3s
      Gibbs constructors                               |   13     13     19.7s
      Gibbs inference                                  |   22     22   6m06.7s
      transitions                                      |           0      2.1s
      dynamic model with analytical posterior          |    5      5      8.4s
      dynamic model with ESS                           |    6      6     33.4s
      dynamic model with dot tilde                     |           0     11.3s
      Demo model                                       | 1564   1564   8m32.2s
      multiple varnames                                |   20     20      1.5s
      non-identity varnames                            |    2      2      9.2s
      submodels                                        |    2      2     11.3s
      CSMC + ESS                                       |   42     42     38.9s
      CSMC + ESS (usage of implicit varname)           |   42     42     51.8s
      externalsampler                                  |    8      8     28.2s
      linking changes dimension                        |    1      1      4.5s
  variational algorithms                               |           0      0.0s
  mode estimation                                      |           0      0.0s
  variational optimisers                               |           0      0.0s
  stdlib                                               |           0      0.0s
  utilities                                            |           0      0.0s
───────────────────────────────────────────────────────────
                                Time          Allocations
                          ───────────────   ───────────────
     Total measured:           1191s             534GiB

Section           ncalls     time    %tot     alloc    %tot
───────────────────────────────────────────────────────────
inference              1    1191s  100.0%    534GiB  100.0%
  mcmc/gibbs.jl        1    1191s  100.0%    534GiB  100.0%
───────────────────────────────────────────────────────────     Testing Turing tests passed

@AoifeHughes AoifeHughes marked this pull request as ready for review June 5, 2025 10:02
Copy link
Contributor

github-actions bot commented Jun 5, 2025

Turing.jl documentation for PR #2579 is available at:
https://TuringLang.github.io/Turing.jl/previews/PR2579/

Copy link

codecov bot commented Jun 5, 2025

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 44.59%. Comparing base (7ebde76) to head (3895f37).
Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
src/mcmc/external_sampler.jl 0.00% 2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (7ebde76) and HEAD (3895f37). Click for more details.

HEAD has 14 uploads less than BASE
Flag BASE (7ebde76) HEAD (3895f37)
28 14
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #2579       +/-   ##
===========================================
- Coverage   85.57%   44.59%   -40.98%     
===========================================
  Files          22       22               
  Lines        1456     1444       -12     
===========================================
- Hits         1246      644      -602     
- Misses        210      800      +590     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coveralls
Copy link

coveralls commented Jun 5, 2025

Pull Request Test Coverage Report for Build 15921714768

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 2 of 2 (100.0%) changed or added relevant lines in 1 file are covered.
  • 14 unchanged lines in 1 file lost coverage.
  • Overall coverage increased (+0.03%) to 85.665%

Files with Coverage Reduction New Missed Lines %
src/mcmc/Inference.jl 14 79.28%
Totals Coverage Status
Change from base Build 15765001949: 0.03%
Covered Lines: 1249
Relevant Lines: 1458

💛 - Coveralls

@AoifeHughes AoifeHughes requested a review from Copilot June 9, 2025 07:50
Copilot

This comment was marked as outdated.

@yebai yebai requested a review from penelopeysm June 20, 2025 10:36
@AoifeHughes AoifeHughes requested a review from Copilot June 20, 2025 11:53
Copilot

This comment was marked as outdated.

@AoifeHughes AoifeHughes requested a review from Copilot June 23, 2025 08:36
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request refactors several model definitions and tests to adopt the new default-type parameter syntax and introduces a new dynamic model (dynamic_bernoulli_normal) with an analytical posterior for faster and more robust testing of the Gibbs sampler. Key changes include:

  • Updating function signatures across multiple tests to use the new parameter syntax with parentheses.
  • Introducing a dynamic model with a Bernoulli variable controlling parameter dimensionality.
  • Simplifying and speeding up tests while ensuring core Gibbs sampling functionality is exercised.

Reviewed Changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated no comments.

File Description
test/mcmc/hmc.jl Updated the default type parameter syntax in model definitions.
test/mcmc/gibbs.jl Updated function signatures and added a new dynamic model test.
test/mcmc/Inference.jl Applied similar parameter signature updates across model tests.
src/mcmc/external_sampler.jl Updated constructor and helper function parameter syntax.
Comments suppressed due to low confidence (2)

test/mcmc/gibbs.jl:564

  • [nitpick] The check for θ[2] values only verifies that one of the conditions is met. Consider splitting the test into two separate assertions to clearly validate that when b=0, θ[2] is missing, and when b=1, θ[2] exists.
            @test n_missing_theta2 > 0 || n_present_theta2 > 0  # One of these should be true

src/mcmc/external_sampler.jl:47

  • [nitpick] Consider adding a comment or type annotations to clarify the expected type and meaning of the 'unconstrained' flag, which would improve maintainability and readability of the code.
function requires_unconstrained_space(

@yebai
Copy link
Member

yebai commented Jun 23, 2025

Thanks @AoifeHughes -- good work!

Copy link
Member

@penelopeysm penelopeysm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise, if @yebai if happy, I am too

AoifeHughes and others added 2 commits June 26, 2025 17:25
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@yebai yebai merged commit 2d3f228 into main Jun 27, 2025
30 checks passed
@yebai yebai deleted the gibbs-test branch June 27, 2025 08:22
@yebai
Copy link
Member

yebai commented Jun 27, 2025

Thanks @AoifeHughes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants