Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support github enterprise api #256

Merged

Conversation

ricardojdsilva87
Copy link
Contributor

@ricardojdsilva87 ricardojdsilva87 commented Oct 23, 2024

Pull Request

Proposed Changes

This pull requests makes possible the authentication against the GitHub Enterprise API with a GitHub Enterprise App using the mentioned environment variables:

  • GH_APP_ID
  • GH_APP_INSTALLATION_ID
  • GH_APP_PRIVATE_KEY

A new environment variable GITHUB_APP_ENTERPRISE_ONLY needs to be set to true so that the api endpoint can also be adjusted.

If the GHE endpoint is provided a different login is made, also every function checks if the ghe variable is set to modify the API endpoint using by default the endpoint `https://api.github.com.
Reusing the existing code a simple check is made to switch the connector on the github library

All the following scenarios should be covered since there was no change in the previous implementation, just a switch on the login type was added for the previous environment variables:

  • Token + github.com ✅
  • Token + ghe ✅
  • github app in github.com + github.com ✅
  • github app in github.com + ghe ✅
  • github app in ghe + ghe ✅

Should fix github/issue-metrics#364

This PR also includes some code "cleanup" mostly on the prints to make it easier to read using f"" instead of the + signs.

Readiness Checklist

Author/Contributor

  • If documentation is needed for this change, has that been included in this pull request
  • run make lint and fix any issues that you have introduced
  • run make test and ensure you have test coverage for the lines you are introducing
  • If publishing new data to the public (scorecards, security scan results, code quality results, live dashboards, etc.), please request review from @jeffrey-luszcz

make lint results

# stop the build if there are Python syntax errors or undefined names
flake8 . --config=.github/linters/.flake8 --count --select=E9,F63,F7,F82 --show-source
0
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --config=.github/linters/.flake8 --count --exit-zero --max-complexity=15 --max-line-length=150
0
isort --settings-file=.github/linters/.isort.cfg .
Skipped 2 files
pylint --rcfile=.github/linters/.python-lint --fail-under=9.0 *.py
************* Module dependabot_file
dependabot_file.py:7:0: R0917: Too many positional arguments (6/5) (too-many-positional-arguments)
dependabot_file.py:52:0: R0917: Too many positional arguments (8/5) (too-many-positional-arguments)
************* Module evergreen
evergreen.py:287:0: R0917: Too many positional arguments (7/5) (too-many-positional-arguments)

------------------------------------------------------------------
Your code has been rated at 9.98/10 (previous run: 9.98/10, +0.00)

mypy --config-file=.github/linters/.mypy.ini *.py
Success: no issues found in 9 source files
black .
reformatted .../evergreen/auth.py
reformatted .../evergreen/test_auth.py
reformatted .../evergreen/evergreen.py
reformatted .../evergreen/test_evergreen.py

All done! ✨ 🍰 ✨

make test results

pytest -v --cov=. --cov-config=.coveragerc --cov-fail-under=80 --cov-report term-missing
================================================================================================================ test session starts ================================================================================================================
platform darwin -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0 -- /opt/homebrew/opt/[email protected]/bin/python3.13
cachedir: .pytest_cache
rootdir: .../evergreen
plugins: cov-5.0.0
collected 99 items

test_auth.py::TestAuth::test_auth_to_github_with_ghe PASSED                                                                                                                                                                                   [  1%]
test_auth.py::TestAuth::test_auth_to_github_with_token PASSED                                                                                                                                                                                 [  2%]
test_auth.py::TestAuth::test_auth_to_github_without_token PASSED                                                                                                                                                                              [  3%]
test_auth.py::TestAuth::test_get_github_app_installation_token PASSED                                                                                                                                                                         [  4%]
test_dependabot_file.py::TestDependabotFile::test_add_existing_ecosystem_to_exempt_list PASSED                                                                                                                                                [  5%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_for_multiple_repos_with_few_existing_config PASSED                                                                                                                    [  6%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_2_space_indent_existing_config_bundler_with_update PASSED                                                                                                        [  7%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_2_space_indent_existing_config_bundler_with_update_and_no_newline PASSED                                                                                         [  8%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_bundler PASSED                                                                                                                                                   [  9%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_cargo PASSED                                                                                                                                                     [ 10%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_composer PASSED                                                                                                                                                  [ 11%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_docker PASSED                                                                                                                                                    [ 12%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_exempt_ecosystems PASSED                                                                                                                                         [ 13%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_existing_config_bundler_no_update PASSED                                                                                                                         [ 14%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_github_actions PASSED                                                                                                                                            [ 15%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_github_actions_without_files PASSED                                                                                                                              [ 16%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_gomod PASSED                                                                                                                                                     [ 17%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_groups PASSED                                                                                                                                                    [ 18%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_hex PASSED                                                                                                                                                       [ 19%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_label PASSED                                                                                                                                                     [ 20%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_labels PASSED                                                                                                                                                    [ 21%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_maven PASSED                                                                                                                                                     [ 22%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_npm PASSED                                                                                                                                                       [ 23%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_nuget PASSED                                                                                                                                                     [ 24%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_pip PASSED                                                                                                                                                       [ 25%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_repo_specific_exempt_ecosystems PASSED                                                                                                                           [ 26%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_schedule_day PASSED                                                                                                                                              [ 27%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_terraform_with_files PASSED                                                                                                                                      [ 28%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_terraform_without_files PASSED                                                                                                                                   [ 29%]
test_dependabot_file.py::TestDependabotFile::test_build_dependabot_file_with_weird_space_indent_existing_config_bundler_with_update PASSED                                                                                                    [ 30%]
test_dependabot_file.py::TestDependabotFile::test_check_multiple_repos_with_no_dependabot_config PASSED                                                                                                                                       [ 31%]
test_dependabot_file.py::TestDependabotFile::test_not_found_error PASSED                                                                                                                                                                      [ 32%]
test_env.py::TestEnv::test_get_env_vars_auth_with_github_app_installation PASSED                                                                                                                                                              [ 33%]
test_env.py::TestEnv::test_get_env_vars_missing_at_least_one_auth PASSED                                                                                                                                                                      [ 34%]
test_env.py::TestEnv::test_get_env_vars_missing_org_or_repo PASSED                                                                                                                                                                            [ 35%]
test_env.py::TestEnv::test_get_env_vars_optional_values PASSED                                                                                                                                                                                [ 36%]
test_env.py::TestEnv::test_get_env_vars_repo_specific_exemptions_improper_format PASSED                                                                                                                                                       [ 37%]
test_env.py::TestEnv::test_get_env_vars_repo_specific_exemptions_unsupported_ecosystem PASSED                                                                                                                                                 [ 38%]
test_env.py::TestEnv::test_get_env_vars_with_a_valid_label PASSED                                                                                                                                                                             [ 39%]
test_env.py::TestEnv::test_get_env_vars_with_bad_schedule_choice PASSED                                                                                                                                                                       [ 40%]
test_env.py::TestEnv::test_get_env_vars_with_bad_schedule_day_choice PASSED                                                                                                                                                                   [ 41%]
test_env.py::TestEnv::test_get_env_vars_with_badly_formatted_created_after_date PASSED                                                                                                                                                        [ 42%]
test_env.py::TestEnv::test_get_env_vars_with_batch_size PASSED                                                                                                                                                                                [ 43%]
test_env.py::TestEnv::test_get_env_vars_with_invalid_batch_size_int PASSED                                                                                                                                                                    [ 44%]
test_env.py::TestEnv::test_get_env_vars_with_invalid_batch_size_str PASSED                                                                                                                                                                    [ 45%]
test_env.py::TestEnv::test_get_env_vars_with_no_batch_size PASSED                                                                                                                                                                             [ 46%]
test_env.py::TestEnv::test_get_env_vars_with_org PASSED                                                                                                                                                                                       [ 47%]
test_env.py::TestEnv::test_get_env_vars_with_org_and_repo_specific_exemptions PASSED                                                                                                                                                          [ 48%]
test_env.py::TestEnv::test_get_env_vars_with_repos PASSED                                                                                                                                                                                     [ 49%]
test_env.py::TestEnv::test_get_env_vars_with_repos_disabled_security_updates PASSED                                                                                                                                                           [ 50%]
test_env.py::TestEnv::test_get_env_vars_with_repos_exempt_ecosystems PASSED                                                                                                                                                                   [ 51%]
test_env.py::TestEnv::test_get_env_vars_with_repos_filter_visibility_invalid_multiple_value PASSED                                                                                                                                            [ 52%]
test_env.py::TestEnv::test_get_env_vars_with_repos_filter_visibility_invalid_single_value PASSED                                                                                                                                              [ 53%]
test_env.py::TestEnv::test_get_env_vars_with_repos_filter_visibility_multiple_values PASSED                                                                                                                                                   [ 54%]
test_env.py::TestEnv::test_get_env_vars_with_repos_filter_visibility_no_duplicates PASSED                                                                                                                                                     [ 55%]
test_env.py::TestEnv::test_get_env_vars_with_repos_filter_visibility_single_value PASSED                                                                                                                                                      [ 56%]
test_env.py::TestEnv::test_get_env_vars_with_repos_no_dry_run PASSED                                                                                                                                                                          [ 57%]
test_env.py::TestEnv::test_get_env_vars_with_team PASSED                                                                                                                                                                                      [ 58%]
test_env.py::TestEnv::test_get_env_vars_with_team_and_repo PASSED                                                                                                                                                                             [ 59%]
test_env.py::TestEnv::test_get_env_vars_with_update_existing PASSED                                                                                                                                                                           [ 60%]
test_env.py::TestEnv::test_get_env_vars_with_valid_labels_containing_spaces PASSED                                                                                                                                                            [ 61%]
test_env.py::TestEnv::test_get_env_vars_with_valid_schedule_and_schedule_day PASSED                                                                                                                                                           [ 62%]
test_env_get_bool.py::TestEnv::test_get_bool_env_var_that_does_not_exist_and_default_value_returns_false PASSED                                                                                                                               [ 63%]
test_env_get_bool.py::TestEnv::test_get_bool_env_var_that_does_not_exist_and_default_value_returns_true PASSED                                                                                                                                [ 64%]
test_env_get_bool.py::TestEnv::test_get_bool_env_var_that_exists_and_is_false PASSED                                                                                                                                                          [ 65%]
test_env_get_bool.py::TestEnv::test_get_bool_env_var_that_exists_and_is_false_due_to_invalid_value PASSED                                                                                                                                     [ 66%]
test_env_get_bool.py::TestEnv::test_get_bool_env_var_that_exists_and_is_true PASSED                                                                                                                                                           [ 67%]
test_evergreen.py::TestDependabotSecurityUpdates::test_enable_dependabot_security_updates PASSED                                                                                                                                              [ 68%]
test_evergreen.py::TestDependabotSecurityUpdates::test_enable_dependabot_security_updates_failed PASSED                                                                                                                                       [ 69%]
test_evergreen.py::TestDependabotSecurityUpdates::test_is_dependabot_security_updates_disabled PASSED                                                                                                                                         [ 70%]
test_evergreen.py::TestDependabotSecurityUpdates::test_is_dependabot_security_updates_enabled PASSED                                                                                                                                          [ 71%]
test_evergreen.py::TestDependabotSecurityUpdates::test_is_dependabot_security_updates_not_found PASSED                                                                                                                                        [ 72%]
test_evergreen.py::TestCommitChanges::test_commit_changes PASSED                                                                                                                                                                              [ 73%]
test_evergreen.py::TestCheckPendingPullsForDuplicates::test_check_pending_pulls_for_duplicates_no_duplicates PASSED                                                                                                                           [ 74%]
test_evergreen.py::TestCheckPendingPullsForDuplicates::test_check_pending_pulls_for_duplicates_with_duplicates PASSED                                                                                                                         [ 75%]
test_evergreen.py::TestCheckPendingIssuesForDuplicates::test_check_pending_issues_for_duplicates_no_duplicates PASSED                                                                                                                         [ 76%]
test_evergreen.py::TestCheckPendingIssuesForDuplicates::test_check_pending_issues_for_duplicates_with_duplicates PASSED                                                                                                                       [ 77%]
test_evergreen.py::TestGetReposIterator::test_get_repos_iterator_with_organization PASSED                                                                                                                                                     [ 78%]
test_evergreen.py::TestGetReposIterator::test_get_repos_iterator_with_repository_list PASSED                                                                                                                                                  [ 79%]
test_evergreen.py::TestGetReposIterator::test_get_repos_iterator_with_team PASSED                                                                                                                                                             [ 80%]
test_evergreen.py::TestGetGlobalProjectId::test_get_global_project_id_parse_response_failed PASSED                                                                                                                                            [ 81%]
test_evergreen.py::TestGetGlobalProjectId::test_get_global_project_id_request_failed PASSED                                                                                                                                                   [ 82%]
test_evergreen.py::TestGetGlobalProjectId::test_get_global_project_id_success PASSED                                                                                                                                                          [ 83%]
test_evergreen.py::TestGetGlobalIssueId::test_get_global_issue_id_parse_response_failed PASSED                                                                                                                                                [ 84%]
test_evergreen.py::TestGetGlobalIssueId::test_get_global_issue_id_request_failed PASSED                                                                                                                                                       [ 85%]
test_evergreen.py::TestGetGlobalIssueId::test_get_global_issue_id_success PASSED                                                                                                                                                              [ 86%]
test_evergreen.py::TestGetGlobalPullRequestID::test_get_global_pr_id_key_error PASSED                                                                                                                                                         [ 87%]
test_evergreen.py::TestGetGlobalPullRequestID::test_get_global_pr_id_request_exception PASSED                                                                                                                                                 [ 88%]
test_evergreen.py::TestGetGlobalPullRequestID::test_get_global_pr_id_success PASSED                                                                                                                                                           [ 89%]
test_evergreen.py::TestLinkItemToProject::test_link_item_to_project_request_exception PASSED                                                                                                                                                  [ 90%]
test_evergreen.py::TestLinkItemToProject::test_link_item_to_project_success PASSED                                                                                                                                                            [ 91%]
test_evergreen.py::TestIsRepoCreateDateBeforeCreatedAfterDate::test_is_created_after_date_is_empty_string PASSED                                                                                                                              [ 92%]
test_evergreen.py::TestIsRepoCreateDateBeforeCreatedAfterDate::test_is_repo_create_date_before_created_after_date PASSED                                                                                                                      [ 93%]
test_evergreen.py::TestIsRepoCreateDateBeforeCreatedAfterDate::test_is_repo_create_date_is_after_created_after_date PASSED                                                                                                                    [ 94%]
test_evergreen.py::TestIsRepoCreateDateBeforeCreatedAfterDate::test_is_repo_created_date_and_created_after_date_is_not_a_date PASSED                                                                                                          [ 95%]
test_evergreen.py::TestIsRepoCreateDateBeforeCreatedAfterDate::test_is_repo_created_date_has_no_time_zone PASSED                                                                                                                              [ 96%]
test_evergreen.py::TestIsRepoCreateDateBeforeCreatedAfterDate::test_is_repo_created_date_is_before_created_after_date_without_timezene_again PASSED                                                                                           [ 97%]
test_evergreen.py::TestCheckExistingConfig::test_check_existing_config_with_existing_config PASSED                                                                                                                                            [ 98%]
test_evergreen.py::TestCheckExistingConfig::test_check_existing_config_without_existing_config PASSED                                                                                                                                         [100%]

---------- coverage: platform darwin, python 3.13.0-final-0 ----------
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
auth.py                 28      9    68%   28-35, 46, 76-78
dependabot_file.py      73      0   100%
env.py                 127     11    91%   42-43, 159-160, 188, 214, 222, 228, 248, 293, 311
evergreen.py           139      3    98%   272-273, 326
--------------------------------------------------
TOTAL                  367     23    94%

Required test coverage of 80% reached. Total coverage: 93.73%

================================================================================================================ 99 passed in 1.47s =================================================================================================================

Reviewer

  • Label as either fix, documentation, enhancement, infrastructure, maintenance or breaking

@jmeridth
Copy link
Member

Thank you for your contribution. Unfortunately we can't assume just because an enterprise url is provided that the GitHub App is also in the enterprise. It could be outside the enterprise and still need to auth against the default api endpoint. We need to have the user let us know the app's scope.

As mentioned in your discussion, allowing apps to be scoped to just the enterprise is a new feature.

Thank you for the concatenation to f-strings. I do agree, that is more readable.

Reviewing your PR now.

@ricardojdsilva87
Copy link
Contributor Author

Hello @jmeridth, thanks for your input,
Regarding that new feature I was not aware of that. We only work with apps created inside of our GHE instance and those can be installed only on that instance organizations.

I've noticed that the github library has 2 different authentication methods depending if it's for github or github enterprise:
image
They might return different values in the end, that might be the reason why the url is needed for ghe login and not for github.com. By default internally it must authenticate against api.github.com.

Like you've mentioned most likely there is the need for that scenario where an app is created outside the GHE and still needs to authenticate against github api, but then the remaining actions must be done against the ghe endpoint. Not sure how to tackle this without more inputs.

Also I have ran prettier against the README file to fix the lint errors. Regarding these, they were added to put some bullet points in the table, should we add the rule to the ignore list?

Thanks

@jmeridth
Copy link
Member

Not sure how to tackle this without more inputs.

Agreed. I think we'll have to request the user provide another environment variable like GITHUB_APP_ENTERPRISE_ONLY (boolean) or something. Naming is hard 😉 Can add to README to ensure the user knows that's available.

should we add the rule to the ignore list

I'm hesitant to do that but the bulleted list does read better. Are we able to ignore list the specific elements for MD033/no-inline-html or can it only be for the entire no-inline-html? 🤔

@ricardojdsilva87
Copy link
Contributor Author

ricardojdsilva87 commented Oct 23, 2024

Hello again,
We can ignore the entire rule, superlinter is using these rules, so we can add them in the linter configuration file excluding the elements we need, just commited that.
Regarding the new input I will need a bit of time to check what needs to be added, so just to confirm, all the scenarios would be:

  • Token + github.com ✅
  • Token + ghe ✅
  • github app in github.com + github.com ❓
  • github app in github.com + ghe ❓
  • github app in ghe + ghe ✅

Those marked with ❓ I'm not sure if they are already working in the current code. Also since you know better the code, feel free to do the necessary changes 🙂

Thanks

# MD033/no-inline-html - Inline HTML
MD033:
# Allowed elements
allowed_elements: [br, li, ul]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

Nit: newline at end of file please

@jmeridth
Copy link
Member

Hello again, We can ignore the entire rule, superlinter is using these rules, so we can add them in the linter configuration file excluding the elements we need, just commited that. Regarding the new input I will need a bit of time to check what needs to be added, so just to confirm, all the scenarios would be:

  • Token + github.com ✅
  • Token + ghe ✅
  • github app in github.com + github.com ❓
  • github app in github.com + ghe ❓
  • github app in ghe + ghe ✅

Those marked with ❓ I'm not sure if they are already working in the current code. Also since you know better the code, feel free to do the necessary changes 🙂

Thanks

Your change handles the last one. The 3rd and 4th should be covered already but don't mind being double checked.

@jmeridth
Copy link
Member

Can you please add the new environment variable to .env-example?

@ricardojdsilva87
Copy link
Contributor Author

@jmeridth seems that the Docker CI step is stuck for 4 hours. The remaining steps seem to have ran successfully, also no lint issues.
Thanks

@jmeridth
Copy link
Member

jmeridth commented Oct 25, 2024

@jmeridth seems that the Docker CI step is stuck for 4 hours. The remaining steps seem to have ran successfully, also no lint issues. Thanks

Yeah, the timeout happens once in a while and a re-run works. It did again in this case.

Will look at your PR in whole again soon. When we merge we'll need these additions in the other GitHub OSPO Actions (I can give links), if you're interested. If not, no worries, I can get them knocked out while pointing here for your kickoff. Let me know.

UPDATE: The other GitHub OSPO Actions

@jmeridth jmeridth merged commit 8edb1f2 into github:main Oct 25, 2024
6 checks passed
auth.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Auth is not working when GitHub App and GH EE URL configured together
3 participants