Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dagster-azure] Generate URL from compute log manager #26171

Open
wants to merge 1 commit into
base: dpeng817/azure_computelogmanager_tests
Choose a base branch
from

Conversation

dpeng817
Copy link
Contributor

@dpeng817 dpeng817 commented Nov 27, 2024

Summary & Motivation

See comments for additional context.

This PR changes the Azure compute log manager to generate a URL by default instead of capturing the logs.

There's a few considerations here:

  • If the blob storage is private, the url will not be publicly accessible. Instead you'll have to go to the azure portal and type in the url - I think this is fine to start.
  • We need to make this configurable.

How I Tested These Changes

Switches the tests to pull out the stdout/stderr urls and read the results.

Copy link
Contributor Author

dpeng817 commented Nov 27, 2024

Copy link
Contributor

@mlarose mlarose left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, but will review the upstack pr before approving this one to get a better understanding.

Copy link
Contributor

@mlarose mlarose left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, i am a bit icky about the usage of stderr but seems to be par for the course

  • docstring for show_url_only

@@ -78,7 +78,9 @@ def __init__(
prefix="dagster",
upload_interval=None,
default_azure_credential=None,
show_url_only=True,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update doc string to mention the show_url_only parameter

Copy link
Member

@prha prha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's amend the description at least, so that we know that there wasn't an implementation bug.. I think the previous test just assigned the wrong blob to the wrong io type.

Comment on lines +71 to +72
assert stdout.count("Printing without context") == 10
assert stderr.count("Logging using context") == 10
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stdout/stderr assertions are reversed from the original test. The original test expects stdout to contain "Logging using context" and stderr to contain "Printing without context". The current assertions have these swapped. This should be:

assert stdout.count("Logging using context") == 10
assert stderr.count("Printing without context") == 10

Spotted by Graphite Reviewer

Is this helpful? React 👍 or 👎 to let us know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants