diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..3a54b26 --- /dev/null +++ b/.github/CODE_OF_CONDUCT.md @@ -0,0 +1,68 @@ +## Pledge + +We pledge to make our community welcoming, safe, and equitable for all. + +We are committed to fostering an environment that respects and promotes the dignity, rights, and contributions of all individuals, regardless of characteristics including race, ethnicity, caste, color, age, physical characteristics, neurodiversity, disability, sex or gender, gender identity or expression, sexual orientation, language, philosophy or religion, national or social origin, socio-economic position, level of education, or other status. The same privileges of participation are extended to everyone who participates in good faith and in accordance with this Covenant. + +### Encouraged Behaviors + +While acknowledging differences in social norms, we all strive to meet our community's expectations for positive behavior. We also understand that our words and actions may be interpreted differently than we intend based on culture, background, or native language. + +With these considerations in mind, we agree to behave mindfully toward each other and act in ways that center our shared values, including: + +- Respecting the purpose of our community, our activities, and our ways of gathering. +- Engaging kindly and honestly with others. +- Respecting different viewpoints and experiences. +- Taking responsibility for our actions and contributions. +- Gracefully giving and accepting constructive feedback. +- Committing to repairing harm when it occurs. +- Behaving in other ways that promote and sustain the well-being of our community. + +## Restricted Behaviors + +We agree to restrict the following behaviors in our community. Instances, threats, and promotion of these behaviors are violations of this Code of Conduct. + +- Harassment. Violating explicitly expressed boundaries or engaging in unnecessary personal attention after any clear request to stop. +- Character attacks. Making insulting, demeaning, or pejorative comments directed at a community member or group of people. +- Stereotyping or discrimination. Characterizing anyone's personality or behavior on the basis of immutable identities or traits. +- Sexualization. Behaving in a way that would generally be considered inappropriately intimate in the context or purpose of the community. +- Violating confidentiality. Sharing or acting on someone's personal or private information without their permission. +- Endangerment. Causing, encouraging, or threatening violence or other harm toward any person or group. +- Behaving in other ways that threaten the well-being of our community. + +### Other Restrictions + +- Misleading identity. Impersonating someone else for any reason, or pretending to be someone else to evade enforcement actions. +- Failing to credit sources. Not properly crediting the sources of content you contribute. +- Irresponsible communication. Failing to responsibly present content which includes, links or describes any other restricted behaviors. + +## Reporting an Issue + +Tensions can occur between community members even when they are trying their best to collaborate. Not every conflict represents a code of conduct violation, and this Code of Conduct reinforces encouraged behaviors and norms that can help avoid conflicts and minimize harm. + +When an incident does occur, it is important to report it promptly. To report a possible violation, contact the project maintainers directly through GitHub issues or discussions. + +Community Moderators take reports of violations seriously and will investigate all reports of code of conduct violations. In order to honor the values of safety and confidentiality, enforcement actions are carried out as transparently as possible while prioritizing the safety and privacy of those involved. + +### Addressing and Repairing Harm + +If an investigation by the Community Moderators finds that this Code of Conduct has been violated, the following enforcement ladder may be used to determine how best to repair harm, based on the incident's impact on the individuals involved and the community as a whole. Depending on the severity of a violation, lower rungs on the ladder may be skipped. + +- **Warning** + - Event: A violation involving a single incident or series of incidents. + - Consequence: A private, written warning from the Community Moderators. + - Repair: Examples of repair include a private written apology, acknowledgement of responsibility, and seeking clarification on expectations. +- **Temporary Suspension** + - Event: A pattern of repeated violation which the Community Moderators have tried to address with warnings, or a single serious violation. + - Consequence: A private written warning with conditions for return from suspension. + - Repair: Examples of repair include respecting the spirit of the suspension, meeting the specified conditions for return, and being thoughtful about how to reintegrate with the community when the suspension is lifted. +- **Permanent Ban** + - Event: A pattern of repeated code of conduct violations that other steps on the ladder have failed to resolve, or a violation so serious that the Community Moderators determine there is no way to keep the community safe with this person as a member. + - Consequence: Access to all community spaces, tools, and communication channels is removed. + - Repair: There is no possible repair in cases of this severity. + +This enforcement ladder is intended as a guideline. It does not limit the ability of Community Managers to use their discretion and judgment, in keeping with the best interests of our community. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public or other spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..cffc411 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,33 @@ +--- +name: Bug Report +about: Report a bug or unexpected behavior +labels: bug +--- + +## Description + + + +## Steps to Reproduce + +1. +2. +3. + +## Expected Behavior + + + +## Actual Behavior + + + +## Environment + +- OS: +- Runtime/Node version: +- Browser (if applicable): + +## Additional Context + + diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..da939ec --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: true +contact_links: + - name: Security Vulnerability + url: https://github.com/entrius/das-github-mirror/security/advisories/new + about: Report security vulnerabilities privately via GitHub Security Advisories diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000..47a71e0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,17 @@ +--- +name: Feature Request +about: Suggest a new feature or improvement +labels: enhancement +--- + +## Summary + + + +## Motivation + + + +## Proposed Solution + + diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..78b710f --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,27 @@ +## Summary + + + +## Related Issues + + + +## Type of Change + +- [ ] Bug fix +- [ ] New feature +- [ ] Refactor +- [ ] Documentation +- [ ] Other (describe below) + +## Testing + + + +## Checklist + +- [ ] I have read the [Contributing Guide](./CONTRIBUTING.md) +- [ ] Code builds without errors +- [ ] New and existing tests pass (if applicable) +- [ ] Documentation updated (if applicable) +- [ ] No unnecessary dependencies added diff --git a/.github/workflows/pr-source-check.yml b/.github/workflows/pr-source-check.yml new file mode 100644 index 0000000..42a6c29 --- /dev/null +++ b/.github/workflows/pr-source-check.yml @@ -0,0 +1,18 @@ +name: pr-source-check + +on: + pull_request: + branches: [main] + +jobs: + pr-source-check: + runs-on: ubuntu-latest + steps: + - name: Enforce source = test for PRs into main + run: | + if [ "${{ github.event.pull_request.head.ref }}" != "test" ]; then + echo "::error::PRs into main must originate from 'test'. Head is '${{ github.event.pull_request.head.ref }}'." + echo "::error::Open your PR against 'test' instead. main only advances via the Release workflow." + exit 1 + fi + echo "PR source is 'test' — allowed." diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..48cf627 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,109 @@ +# das-github-mirror Contributor Guide + +## Getting Started + +Before contributing, please: + +1. Read the [README](./README.md) to understand the project +2. Familiarize yourself with the project structure and tech stack +3. Check existing issues and PRs to avoid duplicate work + +## Local Development + +1. Ensure you have the required runtime installed (check README for version requirements) +2. Clone the repo and install dependencies +3. Copy `.env.example` to `.env` and configure as needed +4. Follow the README to start the development server + +## Creating Issues + +When opening an issue, use the appropriate template: + +- **[Bug Report](.github/ISSUE_TEMPLATE/bug_report.md)** - Report bugs or unexpected behavior. Include steps to reproduce, expected vs actual behavior, and environment details. +- **[Feature Request](.github/ISSUE_TEMPLATE/feature_request.md)** - Suggest new features or improvements. Explain the motivation and proposed solution. +- **Blank Issue** - For issues that don't fit the above templates. + +For security vulnerabilities, **do not create a public issue**. Report them privately via [GitHub Security Advisories](https://github.com/entrius/das-github-mirror/security/advisories/new). + +## Pull Request Process + +### 1. Create Your Branch + +- Branch off of `test` and target it with your PR. PRs that target the wrong base branch will be closed without review. +- Ensure there are no conflicts before submitting + +### 2. Make Your Changes + +- Write clean, well-documented code +- Follow existing code patterns and architecture +- Update documentation if applicable +- Ensure everything builds and runs correctly before submitting + +### 3. Submit Pull Request + +1. Push your branch to the repository +2. Open a PR targeting `test` +3. Fill out the [PR template](.github/PULL_REQUEST_TEMPLATE.md): + - **Summary**: Clear description of changes + - **Related Issues**: Link issues using `Fixes #123` or `Closes #456` + - **Type of Change**: Select bug fix, new feature, refactor, documentation, or other + - **Testing**: Confirm manual testing performed + - **Checklist**: Verify your changes meet the repo's standards + +### 4. Code Review + +- Reviewers will be assigned automatically +- Address review comments promptly + +### Issue Scope + +PRs should focus on the linked issue. Minor incidental changes are fine. PRs dominated by unrelated changes (>50% of the diff) will be asked to scope down. + +### PR Iteration Expectations + +The repository runs an automated maintainer agent that may close PRs in the following cases: + +- Failing CI for 12+ hours with no fix pushed +- Unresolved merge conflicts for 12+ hours with no resolution push +- Requested changes from a maintainer for 12+ hours with no follow-up commits + +## PR Labels + +Apply appropriate labels to help categorize and track your contribution: + +- `bug` - Bug fixes +- `feature` - New feature additions +- `enhancement` - Improvements to existing features +- `refactor` - Code refactoring without functionality changes +- `documentation` - Documentation updates + +## Code Standards + +### Quality Expectations + +- Follow repository conventions (commenting style, variable naming, etc.) +- Use sensible component decomposition to keep files manageable +- Write clean, readable, maintainable code +- Avoid modifying unrelated files +- Avoid adding unnecessary dependencies +- Ensure all build checks pass before submitting + +## Branches + +### `test` + +**Purpose**: Main development and production-ready code + +**Restrictions**: + +- Requires pull request +- Requires all checks to pass +- Requires at least one approval + +## License + +By contributing to das-github-mirror, you agree that your contributions will be licensed under the project's MIT license. + +--- + +Thank you for contributing to das-github-mirror! diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..72f6ea3 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 entrius + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..b4ca1db --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,40 @@ +# Security Policy + +## Reporting a Vulnerability + +If you discover a security vulnerability in das-github-mirror or any of its components, we strongly encourage you to report it responsibly. + +Please **do not publicly disclose** the vulnerability until we have had a reasonable chance to address it. + +### Confidential Reporting + +To report a vulnerability, you can use any of the following methods: + +- Use [GitHub Security Advisories](https://github.com/entrius/das-github-mirror/security/advisories/new) to report privately. + +### What to Include + +When reporting a vulnerability, please provide as much detail as possible: + +- Affected component/files +- Version or commit hash +- Description of the vulnerability +- Steps to reproduce (if possible) +- Impact assessment +- Any potential mitigations or recommendations + +--- + +## Response Process + +1. We will acknowledge your report within **48 hours**. +2. We will investigate and confirm the issue. +3. If confirmed, we will coordinate on a fix and set an embargo period if needed. +4. A fix will be developed, tested, and released as soon as possible. +5. You will be credited (if you wish) in the security section of our release notes. + +--- + +## Thank You + +We appreciate your efforts in keeping das-github-mirror secure and responsible. diff --git a/packages/das/src/api/miners/miners.controller.ts b/packages/das/src/api/miners/miners.controller.ts index befa9f1..61288eb 100644 --- a/packages/das/src/api/miners/miners.controller.ts +++ b/packages/das/src/api/miners/miners.controller.ts @@ -37,21 +37,25 @@ export class MinersController { @ApiOperation({ summary: "Issues authored by a miner", description: - "Returns every issue the miner has authored since the given date, " + - "including current labels with actor attribution and the PR number " + - "(if any) that solved the issue.", + "Returns issues the miner has authored, with current labels (actor " + + "attribution) and any solving PR. When `since` is provided, returns " + + "OPEN issues created on/after that date plus CLOSED issues closed " + + "on/after that date (scoring window). When `since` is omitted, " + + "returns all currently-OPEN issues with no time bound and no CLOSED " + + "history (open-issue load counting).", }) @ApiParam({ name: "githubId", description: "GitHub user ID (numeric)" }) @ApiQuery({ name: "since", required: false, description: - "ISO timestamp. Defaults to 35 days ago (midnight UTC) if omitted.", + "ISO timestamp. When omitted, the response contains all currently-" + + "OPEN issues with no time bound and no CLOSED history.", }) async getIssues( @Param("githubId") githubId: string, @Query("since") since?: string, ): Promise { - return this.miners.getIssues(githubId, MinersService.resolveSince(since)); + return this.miners.getIssues(githubId, since ?? null); } } diff --git a/packages/das/src/api/miners/miners.service.ts b/packages/das/src/api/miners/miners.service.ts index bb93bc0..180953f 100644 --- a/packages/das/src/api/miners/miners.service.ts +++ b/packages/das/src/api/miners/miners.service.ts @@ -20,13 +20,13 @@ export class MinersService { const rows = await this.dataSource.query( ` SELECT - p.repo_full_name, + LOWER(p.repo_full_name) AS repo_full_name, p.pr_number, - p.title, + COALESCE(p.title, '') AS title, p.body, p.state, p.author_github_id, - p.author_login, + COALESCE(p.author_login, '') AS author_login, p.author_association, p.created_at, p.closed_at, @@ -35,14 +35,14 @@ export class MinersService { p.merged_by_login, p.base_ref, p.head_ref, - p.head_repo_full_name, + LOWER(p.head_repo_full_name) AS head_repo_full_name, r.default_branch, p.head_sha, p.base_sha, p.merge_base_sha, - p.additions, - p.deletions, - p.commits_count, + COALESCE(p.additions, 0) AS additions, + COALESCE(p.deletions, 0) AS deletions, + COALESCE(p.commits_count, 0) AS commits_count, p.scoring_data_stored, (p.last_edited_at IS NOT NULL AND p.merged_at IS NOT NULL AND p.last_edited_at > p.merged_at) AS edited_after_merge, @@ -72,7 +72,7 @@ export class MinersService { COALESCE(( SELECT json_agg(json_build_object( 'number', li.issue_number, - 'title', li.issue_title, + 'title', COALESCE(li.issue_title, ''), 'state', li.issue_state, 'state_reason', li.issue_state_reason, 'author_github_id', li.issue_author_github_id, @@ -124,19 +124,19 @@ export class MinersService { async getIssues( githubId: string, - since: string, + since: string | null, ): Promise<{ github_id: string; - since: string; + since: string | null; generated_at: string; issues: unknown[]; }> { const rows = await this.dataSource.query( ` SELECT - i.repo_full_name, + LOWER(i.repo_full_name) AS repo_full_name, i.issue_number, - i.title, + COALESCE(i.title, '') AS title, i.state, i.state_reason, i.author_github_id, @@ -188,8 +188,10 @@ export class MinersService { AND plt.pr_number = sp.pr_number ), '[]'::json), 'review_summary', json_build_object( - 'maintainer_changes_requested_count', - COALESCE(rs.maintainer_changes_requested_count, 0) + 'maintainer_changes_requested_count', COALESCE(rs.maintainer_changes_requested_count, 0), + 'changes_requested_count', COALESCE(rs.changes_requested_count, 0), + 'approved_count', COALESCE(rs.approved_count, 0), + 'commented_count', COALESCE(rs.commented_count, 0) ) ) FROM pull_requests sp @@ -198,11 +200,15 @@ export class MinersService { AND rs.pr_number = sp.pr_number WHERE sp.repo_full_name = i.repo_full_name AND sp.pr_number = i.solved_by_pr + -- Skip null-author solving PRs (no one to credit) + AND sp.author_github_id IS NOT NULL + -- Skip corrupted MERGED-without-merged_at shape + AND NOT (sp.state = 'MERGED' AND sp.merged_at IS NULL) ) AS solving_pr FROM issues i WHERE i.author_github_id = $1 AND ( - (i.state = 'OPEN' AND i.created_at >= $2) + (i.state = 'OPEN' AND ($2::timestamptz IS NULL OR i.created_at >= $2)) OR (i.state = 'CLOSED' AND i.closed_at >= $2) ) ORDER BY i.created_at DESC diff --git a/packages/das/src/api/pulls/pulls.service.ts b/packages/das/src/api/pulls/pulls.service.ts index 42a4d8c..f6084e3 100644 --- a/packages/das/src/api/pulls/pulls.service.ts +++ b/packages/das/src/api/pulls/pulls.service.ts @@ -16,7 +16,7 @@ export class PullsService { const rows = await this.dataSource.query( ` SELECT - p.repo_full_name, + LOWER(p.repo_full_name) AS repo_full_name, p.pr_number, p.head_sha, p.base_sha, @@ -44,8 +44,13 @@ export class PullsService { AND f.pr_number = p.pr_number ), '[]'::json) AS files FROM pull_requests p - WHERE p.repo_full_name = $1 - AND p.pr_number = $2 + -- Look up the canonical-case repo_full_name via the small repos + -- table so the pull_requests PK seek stays index-driven + WHERE p.repo_full_name = ( + SELECT repo_full_name FROM repos + WHERE LOWER(repo_full_name) = LOWER($1) + ) + AND p.pr_number = $2 `, [repoFullName, prNumber], ); diff --git a/packages/das/src/queue/constants.ts b/packages/das/src/queue/constants.ts index 9151658..fd77f4e 100644 --- a/packages/das/src/queue/constants.ts +++ b/packages/das/src/queue/constants.ts @@ -7,3 +7,14 @@ export const FETCH_JOBS = { } as const; export const DEFAULT_BACKFILL_DAYS = 40; + +export function prFilesJobId( + repoFullName: string, + prNumber: number, + headSha: string | null, + baseSha: string | null, +): string { + return `files-${repoFullName}-${prNumber}-${headSha ?? "no-head"}-${ + baseSha ?? "no-base" + }`; +} diff --git a/packages/das/src/queue/fetch.processor.ts b/packages/das/src/queue/fetch.processor.ts index 5449e3b..27860d2 100644 --- a/packages/das/src/queue/fetch.processor.ts +++ b/packages/das/src/queue/fetch.processor.ts @@ -1,11 +1,16 @@ import { Processor, WorkerHost, InjectQueue } from "@nestjs/bullmq"; import { Logger } from "@nestjs/common"; import { InjectRepository } from "@nestjs/typeorm"; -import { Repository } from "typeorm"; +import { IsNull, Repository } from "typeorm"; import { Job, Queue } from "bullmq"; import { Issue, PullRequest } from "../entities"; import { GitHubFetcherService } from "../webhook/github-fetcher.service"; -import { FETCH_QUEUE, FETCH_JOBS, DEFAULT_BACKFILL_DAYS } from "./constants"; +import { + FETCH_QUEUE, + FETCH_JOBS, + DEFAULT_BACKFILL_DAYS, + prFilesJobId, +} from "./constants"; export interface PrMetadataJobData { repoFullName: string; @@ -15,6 +20,8 @@ export interface PrMetadataJobData { export interface PrFilesJobData { repoFullName: string; prNumber: number; + expectedHeadSha?: string | null; + expectedBaseSha?: string | null; } export interface BackfillRepoJobData { @@ -22,6 +29,11 @@ export interface BackfillRepoJobData { days?: number; } +interface PrFilesGeneration { + headSha: string | null; + baseSha: string | null; +} + type JobData = PrMetadataJobData | PrFilesJobData | BackfillRepoJobData; @Processor(FETCH_QUEUE, { concurrency: 5 }) @@ -48,8 +60,7 @@ export class FetchProcessor extends WorkerHost { break; } case FETCH_JOBS.PR_FILES: { - const { repoFullName, prNumber } = job.data as PrFilesJobData; - await this.handlePrFiles(repoFullName, prNumber); + await this.handlePrFiles(job.data as PrFilesJobData); break; } case FETCH_JOBS.BACKFILL_REPO: { @@ -92,18 +103,25 @@ export class FetchProcessor extends WorkerHost { } } - private async handlePrFiles( - repoFullName: string, - prNumber: number, - ): Promise { + private async handlePrFiles(data: PrFilesJobData): Promise { + const { repoFullName, prNumber } = data; this.logger.log(`Fetching PR files for ${repoFullName}#${prNumber}`); + const generation = { + headSha: data.expectedHeadSha ?? null, + baseSha: data.expectedBaseSha ?? null, + }; + await this.fetcher.fetchAndStorePrFiles(repoFullName, prNumber); - await this.prRepo.update( - { repoFullName, prNumber }, + const updateResult = await this.prRepo.update( + this.prGenerationCriteria(repoFullName, prNumber, generation), { scoringDataStored: true }, ); + + if (!updateResult.affected) { + await this.handleStalePrFilesJob(repoFullName, prNumber); + } } private async handleBackfill( @@ -122,7 +140,7 @@ export class FetchProcessor extends WorkerHost { this.logger.log(`Backfilled ${prs.length} PRs from ${repoFullName}`); // Enqueue follow-up jobs (metadata + files for every PR). - for (const { prNumber } of prs) { + for (const { prNumber, headSha, baseSha } of prs) { await this.fetchQueue.add( FETCH_JOBS.PR_METADATA, { repoFullName, prNumber }, @@ -135,16 +153,11 @@ export class FetchProcessor extends WorkerHost { }, ); - await this.fetchQueue.add( - FETCH_JOBS.PR_FILES, - { repoFullName, prNumber }, - { - jobId: `files-${repoFullName}-${prNumber}`, - removeOnComplete: true, - removeOnFail: 50, - attempts: 3, - backoff: { type: "exponential", delay: 5000 }, - }, + await this.enqueuePrFilesJob( + repoFullName, + prNumber, + headSha ?? null, + baseSha ?? null, ); } @@ -152,4 +165,61 @@ export class FetchProcessor extends WorkerHost { await this.fetcher.backfillIssues(repoFullName, sinceDate); this.logger.log(`Backfilled issues from ${repoFullName}`); } + + private async handleStalePrFilesJob( + repoFullName: string, + prNumber: number, + ): Promise { + await this.prRepo.update( + { repoFullName, prNumber }, + { scoringDataStored: false }, + ); + + const pr = await this.prRepo.findOneBy({ repoFullName, prNumber }); + if (!pr) return; + + await this.enqueuePrFilesJob( + repoFullName, + prNumber, + pr.headSha ?? null, + pr.baseSha ?? null, + ); + } + + private async enqueuePrFilesJob( + repoFullName: string, + prNumber: number, + expectedHeadSha: string | null, + expectedBaseSha: string | null, + ): Promise { + await this.fetchQueue.add( + FETCH_JOBS.PR_FILES, + { repoFullName, prNumber, expectedHeadSha, expectedBaseSha }, + { + jobId: prFilesJobId( + repoFullName, + prNumber, + expectedHeadSha, + expectedBaseSha, + ), + removeOnComplete: true, + removeOnFail: 50, + attempts: 3, + backoff: { type: "exponential", delay: 5000 }, + }, + ); + } + + private prGenerationCriteria( + repoFullName: string, + prNumber: number, + generation: PrFilesGeneration, + ): Record { + return { + repoFullName, + prNumber, + headSha: generation.headSha ?? IsNull(), + baseSha: generation.baseSha ?? IsNull(), + }; + } } diff --git a/packages/das/src/webhook/github-fetcher.service.ts b/packages/das/src/webhook/github-fetcher.service.ts index 98c8de4..88ad900 100644 --- a/packages/das/src/webhook/github-fetcher.service.ts +++ b/packages/das/src/webhook/github-fetcher.service.ts @@ -477,15 +477,19 @@ export class GitHubFetcherService implements OnModuleInit { headSha: string, baseSha: string | null, ): Promise { - // Only fetch contents for files that have a meaningful version to fetch - const scored = files.filter((f) => f.status !== "removed"); - if (scored.length === 0) return; + // Added files have only a head blob; removed files have only a base blob. + // Keep removed files when a base SHA is available so deletion scoring has + // the content that existed before the PR. + const contentFiles = files.filter( + (f) => f.status !== "removed" || baseSha !== null, + ); + if (contentFiles.length === 0) return; let batchSize = GRAPHQL_FILES_BATCH_SIZE; const minBatchSize = 5; - for (let i = 0; i < scored.length; ) { - const batch = scored.slice(i, i + batchSize); + for (let i = 0; i < contentFiles.length; ) { + const batch = contentFiles.slice(i, i + batchSize); try { await this.fetchContentBatch( repoFullName, @@ -537,11 +541,14 @@ export class GitHubFetcherService implements OnModuleInit { `base${i}: object(expression: "${baseExpr}") { ... on Blob { text byteSize isBinary } }`, ); } - // Head version (already filtered out removed files at caller) - const headExpr = this.escapeGraphql(`${headSha}:${file.filename}`); - fields.push( - `head${i}: object(expression: "${headExpr}") { ... on Blob { text byteSize isBinary } }`, - ); + // Removed files do not exist at head; store a null headContent while + // still fetching the base blob above. + if (file.status !== "removed") { + const headExpr = this.escapeGraphql(`${headSha}:${file.filename}`); + fields.push( + `head${i}: object(expression: "${headExpr}") { ... on Blob { text byteSize isBinary } }`, + ); + } } const query = ` @@ -636,7 +643,9 @@ export class GitHubFetcherService implements OnModuleInit { async backfillPullRequests( repoFullName: string, sinceDate: Date, - ): Promise<{ prNumber: number }[]> { + ): Promise< + { prNumber: number; headSha: string | null; baseSha: string | null }[] + > { const [owner, repo] = repoFullName.split("/"); const token = await this.getTokenForRepo(repoFullName); @@ -718,7 +727,11 @@ export class GitHubFetcherService implements OnModuleInit { } `; - const prs: { prNumber: number }[] = []; + const prs: { + prNumber: number; + headSha: string | null; + baseSha: string | null; + }[] = []; let cursor: string | null = null; let defaultBranchWritten = false; @@ -823,7 +836,11 @@ export class GitHubFetcherService implements OnModuleInit { pr.timelineItems?.nodes ?? [], ); - prs.push({ prNumber: pr.number }); + prs.push({ + prNumber: pr.number, + headSha: pr.headRefOid ?? null, + baseSha: pr.baseRefOid ?? null, + }); } if (shouldStop || !page.pageInfo.hasNextPage) break; @@ -931,26 +948,29 @@ export class GitHubFetcherService implements OnModuleInit { break; } - await this.issueRepo.upsert( - { - repoFullName, - issueNumber: issue.number, - authorGithubId: String(issue.author?.databaseId ?? ""), - authorLogin: issue.author?.login ?? null, - authorAssociation: issue.authorAssociation ?? null, - title: issue.title, - state: issue.state, // OPEN / CLOSED - stateReason: issue.stateReason ?? null, - createdAt: issue.createdAt, - closedAt: issue.closedAt ?? null, - updatedAt: issue.updatedAt ?? null, - lastEditedAt: issue.lastEditedAt ?? null, - labels: (issue.labels?.nodes ?? []).map( - (l: { name: string }) => l.name, - ), - }, - ["repoFullName", "issueNumber"], - ); + const issueData: Partial = { + repoFullName, + issueNumber: issue.number, + authorGithubId: String(issue.author?.databaseId ?? ""), + authorLogin: issue.author?.login ?? null, + authorAssociation: issue.authorAssociation ?? null, + title: issue.title, + state: issue.state, // OPEN / CLOSED + stateReason: issue.stateReason ?? null, + createdAt: issue.createdAt, + closedAt: issue.closedAt ?? null, + updatedAt: issue.updatedAt ?? null, + lastEditedAt: issue.lastEditedAt ?? null, + labels: (issue.labels?.nodes ?? []).map( + (l: { name: string }) => l.name, + ), + }; + + if (issue.state === "OPEN") { + issueData.solvedByPr = null; + } + + await this.issueRepo.upsert(issueData, ["repoFullName", "issueNumber"]); // Upsert label events for this issue await this.saveLabelTimelineEvents( @@ -969,8 +989,8 @@ export class GitHubFetcherService implements OnModuleInit { /** * Upsert a list of LABELED_EVENT / UNLABELED_EVENT timeline nodes into * the label_events table. Actor role is resolved at read time via - * contributor_repo_roles — GraphQL's actor type doesn't expose - * authorAssociation. + * contributor_repo_roles using stored PR/issue, review, and comment + * association evidence; GraphQL's actor type doesn't expose authorAssociation. */ private async saveLabelTimelineEvents( repoFullName: string, diff --git a/packages/das/src/webhook/handlers/issue.handler.ts b/packages/das/src/webhook/handlers/issue.handler.ts index 98f9090..a94edbc 100644 --- a/packages/das/src/webhook/handlers/issue.handler.ts +++ b/packages/das/src/webhook/handlers/issue.handler.ts @@ -20,6 +20,7 @@ export class IssueHandler { // Skip pull request events delivered as issue events if (issue.pull_request) return; + const issueState = issue.state.toUpperCase(); const data: Partial = { repoFullName, issueNumber: issue.number, @@ -27,7 +28,7 @@ export class IssueHandler { authorLogin: issue.user.login, authorAssociation: issue.author_association, title: issue.title ?? null, - state: issue.state.toUpperCase(), + state: issueState, stateReason: issue.state_reason?.toUpperCase() ?? null, createdAt: issue.created_at, closedAt: issue.closed_at ?? null, @@ -35,6 +36,10 @@ export class IssueHandler { labels: (issue.labels ?? []).map((l: any) => l.name), }; + if (issueState === "OPEN") { + data.solvedByPr = null; + } + // The `edited` action fires specifically for body or title changes. // Use the webhook's updated_at as the precise edit timestamp — for // other actions (labeled, closed, commented, etc.) don't touch diff --git a/packages/das/src/webhook/handlers/label.handler.ts b/packages/das/src/webhook/handlers/label.handler.ts index 4a4eaaf..5304fa1 100644 --- a/packages/das/src/webhook/handlers/label.handler.ts +++ b/packages/das/src/webhook/handlers/label.handler.ts @@ -34,8 +34,8 @@ export class LabelHandler { source === "pr" ? payload.pull_request.number : payload.issue.number; // Append to label_events log. Actor's repo role is resolved at read time - // via contributor_repo_roles (see pr_labels_by_actor view) — neither the - // webhook sender nor GraphQL LabeledEvent.actor expose author_association. + // via contributor_repo_roles using stored PR/issue, review, and comment + // association evidence; label actors themselves don't expose it. await this.labelEventRepo.save({ repoFullName, targetNumber, diff --git a/packages/das/src/webhook/handlers/pull-request.handler.ts b/packages/das/src/webhook/handlers/pull-request.handler.ts index dd54838..7213834 100644 --- a/packages/das/src/webhook/handlers/pull-request.handler.ts +++ b/packages/das/src/webhook/handlers/pull-request.handler.ts @@ -5,7 +5,7 @@ import { InjectQueue } from "@nestjs/bullmq"; import { Repository } from "typeorm"; import { Queue } from "bullmq"; import { PullRequest, Repo } from "../../entities"; -import { FETCH_QUEUE, FETCH_JOBS } from "../../queue/constants"; +import { FETCH_QUEUE, FETCH_JOBS, prFilesJobId } from "../../queue/constants"; @Injectable() export class PullRequestHandler { @@ -33,7 +33,7 @@ export class PullRequestHandler { authorLogin: pr.user.login, authorAssociation: pr.author_association, title: pr.title, - state: pr.merged ? "MERGED" : pr.state.toUpperCase(), + state: pr.merged && pr.merged_at ? "MERGED" : pr.state.toUpperCase(), createdAt: pr.created_at, closedAt: pr.closed_at ?? null, mergedAt: pr.merged_at ?? null, @@ -96,10 +96,17 @@ export class PullRequestHandler { ); } - const jobId = `files-${repoFullName}-${prNumber}`; + const expectedHeadSha = data.headSha ?? null; + const expectedBaseSha = data.baseSha ?? null; + const jobId = prFilesJobId( + repoFullName, + prNumber, + expectedHeadSha, + expectedBaseSha, + ); await this.fetchQueue.add( FETCH_JOBS.PR_FILES, - { repoFullName, prNumber }, + { repoFullName, prNumber, expectedHeadSha, expectedBaseSha }, { jobId, removeOnComplete: true, diff --git a/packages/db/02_pull_requests.sql b/packages/db/02_pull_requests.sql index abbf844..080a41f 100644 --- a/packages/db/02_pull_requests.sql +++ b/packages/db/02_pull_requests.sql @@ -27,7 +27,9 @@ CREATE TABLE IF NOT EXISTS pull_requests ( closing_issue_numbers INTEGER[], scoring_data_stored BOOLEAN NOT NULL DEFAULT FALSE, - PRIMARY KEY (repo_full_name, pr_number) + PRIMARY KEY (repo_full_name, pr_number), + CONSTRAINT pull_requests_merged_has_merged_at + CHECK (state != 'MERGED' OR merged_at IS NOT NULL) ); CREATE INDEX IF NOT EXISTS idx_pull_requests_author ON pull_requests(author_github_id); diff --git a/packages/db/20_view_contributor_repo_roles.sql b/packages/db/20_view_contributor_repo_roles.sql index eb0f13f..1e084df 100644 --- a/packages/db/20_view_contributor_repo_roles.sql +++ b/packages/db/20_view_contributor_repo_roles.sql @@ -1,5 +1,8 @@ -- Latest known association per contributor per repo. --- Unions PRs and issues, takes the most recently created record. +-- Uses every table that stores GitHub's author_association/reviewer_association: +-- PR authors, issue authors, submitted reviews, and issue/PR thread comments. +-- Rows without a stored association are ignored; label views should use the +-- latest known role, not let a missing observation erase earlier evidence. CREATE OR REPLACE VIEW contributor_repo_roles AS SELECT DISTINCT ON (repo_full_name, author_github_id) @@ -8,10 +11,62 @@ SELECT DISTINCT ON (repo_full_name, author_github_id) author_login, author_association FROM ( - SELECT repo_full_name, author_github_id, author_login, author_association, created_at + SELECT + repo_full_name, + author_github_id, + author_login, + author_association, + created_at AS observed_at, + 10 AS source_rank, + 'pr:' || pr_number::text AS source_key FROM pull_requests + WHERE author_github_id IS NOT NULL + AND author_github_id <> '' + AND author_association IS NOT NULL + UNION ALL - SELECT repo_full_name, author_github_id, author_login, author_association, created_at + + SELECT + repo_full_name, + author_github_id, + author_login, + author_association, + created_at AS observed_at, + 10 AS source_rank, + 'issue:' || issue_number::text AS source_key FROM issues + WHERE author_github_id IS NOT NULL + AND author_github_id <> '' + AND author_association IS NOT NULL + + UNION ALL + + SELECT + repo_full_name, + reviewer_github_id AS author_github_id, + reviewer_login AS author_login, + reviewer_association AS author_association, + submitted_at AS observed_at, + 20 AS source_rank, + 'review:' || pr_number::text || ':' || submitted_at::text AS source_key + FROM reviews + WHERE reviewer_github_id IS NOT NULL + AND reviewer_github_id <> '' + AND reviewer_association IS NOT NULL + + UNION ALL + + SELECT + repo_full_name, + author_github_id, + author_login, + author_association, + COALESCE(updated_at, created_at) AS observed_at, + 30 AS source_rank, + 'comment:' || comment_id::text AS source_key + FROM comments + WHERE author_github_id IS NOT NULL + AND author_github_id <> '' + AND author_association IS NOT NULL ) combined -ORDER BY repo_full_name, author_github_id, created_at DESC; +ORDER BY repo_full_name, author_github_id, observed_at DESC, source_rank DESC, source_key DESC; diff --git a/packages/db/24_view_pr_labels_by_actor.sql b/packages/db/24_view_pr_labels_by_actor.sql index d6cbc88..8b10f27 100644 --- a/packages/db/24_view_pr_labels_by_actor.sql +++ b/packages/db/24_view_pr_labels_by_actor.sql @@ -2,8 +2,8 @@ -- Collapses label_events to the latest action per (repo, pr, label); only rows -- where the latest action was "labeled" are included (i.e. label still applied). -- actor_association is resolved from contributor_repo_roles (the actor's most --- recently observed role from PRs/issues they've authored in this repo). --- Actors who've never authored anything return NULL for actor_association. +-- recently observed role from authored PRs/issues, reviews, or comments in +-- this repo). Actors with no stored association evidence return NULL. CREATE OR REPLACE VIEW pr_labels_by_actor AS WITH latest_events AS (