Skip to content

Commit 0e9c0b8

Browse files
ethanyhouethanyhou
authored
test: Add test plans for different Copilot subscription tiers focusing on quota usage. (#222)
1 parent 9055a01 commit 0e9c0b8

7 files changed

Lines changed: 1219 additions & 0 deletions

File tree

‎test-plans/token-based-billing/README.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Copilot Plan / Quota UI Test Plans
2+
3+
## Overview
4+
Test plans for the plan- and quota-aware UI surfaces in GitHub Copilot for
5+
Eclipse. One file per Copilot plan; each plan exercises the same three
6+
surfaces so cross-plan differences stay easy to compare.
7+
8+
All accounts under test are on token-based billing.
9+
10+
## Surfaces Under Test
11+
1. **Status-bar usage menu** — opened by clicking the Copilot icon in the
12+
Eclipse status bar.
13+
2. **Menu-bar usage menu** — opened from `GitHub Copilot` in the Eclipse menu
14+
bar. Must be visually identical to the status-bar menu.
15+
3. **Quota-warning surfaces in the chat view**
16+
- **Static banner** shown above the chat input area when the user crosses
17+
one of the chat-view quota-usage thresholds (see *Quota-Warning
18+
Thresholds* below).
19+
- **Inline warning widget** shown under a chat turn when a chat request
20+
fails because the quota is exhausted.
21+
22+
The chat-view warning surfaces are driven only by the quotas that affect
23+
chat usage: the **Chat Messages** quota on the Free plan and the
24+
**AI-credit (premium interactions)** quota on every paid plan. The Code
25+
Completions quota does not trigger banners or inline warnings in the chat
26+
view, even though it is shown in the status-bar / menu-bar menus.
27+
28+
## Quota-Warning Thresholds
29+
The static banner is pushed by the language server when the signed-in
30+
account's tracked usage crosses one of these two thresholds:
31+
32+
| Usage crossed | Banner severity / icon |
33+
|---------------|------------------------|
34+
| 75% | Info icon |
35+
| 90% | Warning icon |
36+
37+
Each threshold fires at most once per quota reset period. To re-trigger a
38+
banner for the same threshold, the quota usage on the GitHub quota portal
39+
must first be lowered back below the threshold (or the reset date must pass)
40+
so the language server re-arms the notification.
41+
42+
Per-plan cases below are ordered from **rich quota to poor quota**: usage
43+
**under 75%** (no banner), usage **above 75%** (info banner), and usage
44+
**above 90%** (warning banner). To exercise a specific threshold set usage
45+
on the quota portal a few percentage points **above** the target threshold
46+
(for example, **~76%** to cross 75% and **~91%** to cross 90%) and then send
47+
a chat message.
48+
49+
## Plan → File Map
50+
| Copilot plan | Plan label shown in the menu | File |
51+
|--------------------|-----------------------------------|----------------------------|
52+
| Free | Copilot Free Plan | `TBB-test-free.md` |
53+
| Pro | Copilot Pro Plan | `TBB-test-pro.md` |
54+
| Pro+ | Copilot Pro+ Plan | `TBB-test-pro-plus.md` |
55+
| Max | Copilot Max Plan | `TBB-test-max.md` |
56+
| Business | Business Plan | `TBB-test-business.md` |
57+
| Enterprise | Enterprise Plan | `TBB-test-enterprise.md` |
58+
59+
## Common Preconditions
60+
- Eclipse is running with the GitHub Copilot for Eclipse plugin installed.
61+
- The tester has access to the GitHub quota portal at
62+
`https://github.com/github-copilot/quotas/<accountId>` to adjust usage on the
63+
signed-in account. The menus refresh in real time after a quota change; no
64+
Eclipse restart is required.
65+
- The Copilot chat view is open in Agent mode for chat-related cases.
66+
- Each plan starts with a sign-in case (`TC-<plan>-000`) using an account on
67+
that plan with token-based billing enabled.
68+
69+
## Quota Action Summary
70+
This table describes which overage action the chat-view warning surfaces are
71+
expected to show for each plan.
72+
73+
| Plan | Overage action |
74+
|--------------|-------------------------------------------------------|
75+
| Free | — |
76+
| Pro | Enable Additional Usage / Increase Budget (primary) |
77+
| Pro+ | Enable Additional Usage / Increase Budget (primary) |
78+
| Max | Enable Additional Usage / Increase Budget (primary) |
79+
| Business | — |
80+
| Enterprise | — |
81+
82+
Notes:
83+
- The label switches from "Enable Additional Usage" to "Increase Budget" once
84+
additional paid usage has been enabled for the user.
85+
- The **Upgrade Plan** action (row in the menu, link in the static banner,
86+
button in the inline warning) is **not** plan-driven. It must appear if and
87+
only if the language server reports `canUpgradePlan = true` for the
88+
signed-in account; otherwise it must be absent from every surface. Each
89+
per-plan file starts with a step that records the actual `canUpgradePlan`
90+
value from the language-server log, and all later cases reference that
91+
recorded value.
92+
93+
## How to Read `canUpgradePlan` From the Language-Server Log
94+
1. In Eclipse, open `Window → Preferences → Language Servers`.
95+
2. Tick **Log to console** (and optionally **Log to file**) for the GitHub
96+
Copilot language server, then `Apply and Close`.
97+
3. Open `Window → Show View → Other... → General → Console`.
98+
4. From the Console toolbar dropdown (the open-console icon), choose the
99+
`Eclipse Copilot Language Servers Log` console entry that traces traffic for the GitHub
100+
Copilot language server.
101+
5. Sign in to GitHub (or sign out and back in) so a fresh handshake is logged.
102+
6. Locate the most recent `copilot/quotaChange` notification (sent from the
103+
server). Note the value of the `canUpgradePlan` field in its payload — it
104+
is `true`, `false`, or absent.
105+
- `true` → expect Upgrade Plan everywhere it can appear.
106+
- `false` or absent → expect Upgrade Plan to **not** appear.

‎test-plans/token-based-billing/TBB-test-business.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# Copilot Business Plan (Unlimited Org) — Quota UI
2+
3+
## Overview
4+
This file covers the **Business plan with unlimited premium-interaction
5+
usage** configured by the organization. Under this configuration the usage
6+
menus replace the Monthly Limit row with a single informational message, and
7+
the chat-view quota-warning surfaces stay dormant because there is no
8+
monthly cap to cross.
9+
10+
### Surface-by-surface expectations (unlimited org)
11+
12+
| Surface | Expected behaviour |
13+
|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
14+
| Plan label in menu header | `Business Plan` |
15+
| Header row icon | Blue (full) usage icon |
16+
| Monthly Limit row | **Hidden.** Replaced by a disabled message: `You have no monthly limit on AI credits usage set by your organization`. |
17+
| Allowance-reset row | **Hidden.** There is no monthly allowance to reset. |
18+
| Additional-usage status row | **Hidden.** |
19+
| Enable Additional Usage / Increase Budget row | **Hidden.** |
20+
| `Upgrade Plan` row | Present **iff** the language server reports `canUpgradePlan = true` for the signed-in account. Typically `false`. |
21+
| Static banner (chat view) | Not triggered. No 75% / 90% threshold to cross. |
22+
| Inline quota warning under a chat turn | Not triggered by quota exhaustion (no cap). |
23+
24+
### Why the chat-view warning surfaces are not exercised here
25+
With unlimited premium-interaction usage there is no allowance to cross, so
26+
neither the 75% info banner nor the 90% warning banner can fire, and the
27+
quota-exhausted inline warning is unreachable. Banner / inline cases live in
28+
the bounded-org and CFI plans. Any appearance of those surfaces on an
29+
unlimited Business account is a regression.
30+
31+
---
32+
33+
## Test Cases
34+
35+
### TC-CB-U-000: Sign in with an unlimited Business org account, record `canUpgradePlan`
36+
37+
**Type:** `Happy Path`
38+
**Priority:** `P0`
39+
40+
#### Preconditions
41+
- Plugin is installed and Eclipse is signed out.
42+
- A GitHub account that belongs to a **Copilot Business organization
43+
configured with unlimited premium-interaction usage** is available.
44+
- Language-server logging is enabled and the Eclipse Copilot Language Servers
45+
Log console is open (see the README section *How to Read `canUpgradePlan`
46+
From the Language-Server Log*).
47+
48+
#### Steps
49+
1. Open the Copilot status-bar menu and click `Sign in to GitHub`.
50+
2. Complete the device-flow sign-in with the unlimited Business account.
51+
3. Wait for the status-bar icon to switch to the signed-in icon.
52+
4. In the Eclipse Copilot Language Servers Log console, find the most recent
53+
`copilot/quotaChange` notification and:
54+
- **Record `canUpgradePlan`** (`true`, `false`, or absent). Later cases
55+
refer to this as "the recorded `canUpgradePlan` value". For a typical
56+
Business account this is `false`.
57+
- Confirm the payload indicates that the premium-interactions quota is
58+
unlimited. If it is not, the account is not actually on an unlimited
59+
org and the test should be re-run on a correctly provisioned account.
60+
61+
#### Expected Result
62+
- The header row of both menus reads `<username> — Business Plan`.
63+
- The `canUpgradePlan` value and the "unlimited" indication have been
64+
recorded.
65+
66+
#### 📸 Key Screenshots
67+
- [ ] Signed-in state with the `Business Plan` label
68+
- [ ] Language-server log showing `canUpgradePlan` and the unlimited flag
69+
70+
---
71+
72+
### TC-CB-U-001: Status-bar menu — unlimited org informational message
73+
74+
**Type:** `Happy Path`
75+
**Priority:** `P0`
76+
77+
#### Preconditions
78+
- TC-CB-U-000 completed; the signed-in user is on an unlimited Business org.
79+
80+
#### Steps
81+
1. Click the Copilot icon in the Eclipse status bar.
82+
83+
#### Expected Result
84+
1. Header row: `<username> — Business Plan` on top; second row reads
85+
`Copilot Usage` with the **blue/full usage icon**.
86+
2. A **single disabled message row** reading exactly:
87+
`You have no monthly limit on AI credits usage set by your organization`.
88+
3. **No** Monthly Limit row.
89+
4. **No** allowance-reset row (no `Resets today` / `Reset in N days …` text).
90+
5. **No** `Additional usage enabled` / `Additional usage not enabled` status
91+
row, and **no** tooltip on such a row.
92+
6. **No** `Enable Additional Usage` / `Increase Budget` action row.
93+
7. `Upgrade Plan` row — present **iff** the recorded `canUpgradePlan` value
94+
(TC-CB-U-000) is `true`; absent otherwise. For a typical Business account
95+
it must not appear.
96+
8. Standard footer rows (Sign Out, Preferences, etc.) appear as usual below
97+
the usage section.
98+
99+
#### 📸 Key Screenshots
100+
- [ ] Status-bar menu, unlimited Business org (full menu)
101+
102+
---
103+
104+
### TC-CB-U-002: Menu-bar menu mirrors the status-bar menu
105+
106+
**Type:** `Happy Path`
107+
**Priority:** `P0`
108+
109+
#### Steps
110+
1. Open `GitHub Copilot` from the Eclipse menu bar.
111+
2. Compare against the menu captured in TC-CB-U-001.
112+
113+
#### Expected Result
114+
The menu-bar menu shows the same rows in the same order with the same labels,
115+
tooltips, and icons as the status-bar menu. The unlimited-org informational
116+
message must be identical between the two surfaces.
117+
118+
#### 📸 Key Screenshots
119+
- [ ] Menu-bar menu, unlimited Business org
120+
121+
---
122+
123+
### TC-CB-U-003: Chat works without any quota banner or inline warning
124+
125+
**Type:** `Happy Path`
126+
**Priority:** `P0`
127+
128+
#### Preconditions
129+
- TC-CB-U-001 succeeded.
130+
- Copilot chat view is open in Agent mode.
131+
132+
#### Steps
133+
1. Send a normal chat message (e.g. `Hello, what can you do?`) and wait for
134+
the response.
135+
2. Send a second chat message that triggers tool use (e.g. ask to read a file
136+
in the workspace).
137+
3. Inspect the chat view above the input area and under each completed turn.
138+
139+
#### Expected Result
140+
- Both turns complete successfully.
141+
- **No** static banner appears above the chat input (no info-icon banner, no
142+
warning-icon banner, no `Upgrade Plan` link).
143+
- **No** inline warning appears under either turn (no quota-exhausted
144+
message, no `Upgrade Plan` button, no `Enable Additional Usage` button).
145+
146+
#### 📸 Key Screenshots
147+
- [ ] Chat view with two completed turns, no banner and no inline warning
148+
149+
---
150+
151+
## Screenshots Checklist
152+
- [ ] `TC-CB-U-000` Signed-in `Business Plan` label + language-server log payload
153+
- [ ] `TC-CB-U-001` Status-bar menu, unlimited org informational message
154+
- [ ] `TC-CB-U-002` Menu-bar menu mirrors status-bar menu
155+
- [ ] `TC-CB-U-003` Chat view with no banner / no inline warning

0 commit comments

Comments
 (0)