Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core][aDAG] Set buffer size to 1 for regression #47639

Merged
merged 5 commits into from
Sep 16, 2024
Merged

[Core][aDAG] Set buffer size to 1 for regression #47639

merged 5 commits into from
Sep 16, 2024

Conversation

rkooo567
Copy link
Contributor

@rkooo567 rkooo567 commented Sep 12, 2024 •
edited
Loading

Why are these changes needed?

There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@rkooo567 rkooo567 changed the title [wip] regression teset [Core][aDAG] Set buffer size to 1 for regression Sep 14, 2024
@rkooo567 rkooo567 mentioned this pull request Sep 14, 2024
Closed
8 tasks
ruisearch42
ruisearch42 approved these changes Sep 16, 2024
View reviewed changes
python/ray/dag/compiled_dag_node.py
@@ -661,7 +661,7 @@ def __init__(
# We conservatively set num_shm_buffers to _max_inflight_executions.
# It means that the DAG can be underutilized, but it guarantees there's
# no false positive timeouts.
num_shm_buffers=self._max_inflight_executions,
num_shm_buffers=1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you update the comment above and add a TODO?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix this this week, so it may be okay?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, feel free to merge first if urgent.

@rkooo567 rkooo567 enabled auto-merge (squash) September 16, 2024 16:42
@github-actions github-actions bot added the go add ONLY when ready to merge, run all tests label Sep 16, 2024
kevin85421
kevin85421 approved these changes Sep 16, 2024
View reviewed changes
Copy link
Member

@kevin85421 kevin85421 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have an issue to track the regression caused by the buffer PR? If not, would you mind opening an issue to track the progress?

@rkooo567 rkooo567 merged commit 2b69277 into ray-project:master Sep 16, 2024
6 of 7 checks passed
@rkooo567
Copy link
Contributor Author

It is in my sprint task.

simonsays1980 pushed a commit to simonsays1980/ray that referenced this pull request Sep 17, 2024
@rkooo567 @simonsays1980
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
2216f2d 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@rkooo567 @ujjawal-khare
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
683ee12 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@rkooo567 @ujjawal-khare
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
9af682e 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@rkooo567 @ujjawal-khare
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
7f1339d 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@rkooo567 @ujjawal-khare
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
be6ba57 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@rkooo567 @ujjawal-khare
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
ace2c5c 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@rkooo567 @ujjawal-khare
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
f859afa 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@rkooo567 @ujjawal-khare
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
4af8caf 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@rkooo567 @ujjawal-khare
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
2a7679d 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Signed-off-by: ujjawal-khare <[email protected]>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@rkooo567 @ujjawal-khare
[Core][aDAG] Set buffer size to 1 for regression (ray-project#47639)
27c71b6 
There's a regression with buffer size 10. I am going to investigate but I will revert it to buffer size 1 for now until further investigation.
With buffer size 1, regression seems to be gone https://buildkite.com/ray-project/release/builds/22594#0191ed4b-5477-45ff-be9e-6e098b5fbb3c. probably some sort of contention or sth like that

Signed-off-by: ujjawal-khare <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ruisearch42 ruisearch42 ruisearch42 approved these changes

@kevin85421 kevin85421 kevin85421 approved these changes

Assignees

@kevin85421 kevin85421

@ruisearch42 ruisearch42

Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rkooo567 @kevin85421 @ruisearch42