Skip to content

Commit 1dd2054

Browse files
zachblumezachblume
committed
npx works and update readme to reflect that
1 parent adb8623 commit 1dd2054

File tree

5 files changed

+150
-82
lines changed

5 files changed

+150
-82
lines changed

‎.env.example

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
OPENAI_API_KEY=your-openai-api-key-needs-to-get-pasted-here-in-dot-env-file
2-
GOOGLE_GENERATIVE_AI_API_KEY=your-google-generative-ai-api-key-needs-to-get-pasted-here-in-dot-env-file
2+
GOOGLE_GENERATIVE_AI_API_KEY=your-google-generative-ai-api-key-needs-to-get-pasted-here-in-dot-env-file
3+
ANTHROPIC_API_KEY=your-anthropic-api-key-needs-to-get-pasted-here-in-dot-env-file

‎README.md

+62-56
Original file line numberDiff line numberDiff line change
@@ -13,44 +13,52 @@ behavior.
1313
implementation, not just check for regressions.
1414
- It requires no configuration, making it straightforward to use.
1515

16-
### Get started
16+
### Quick start
1717

18-
1. Copy the sample .env file, you'll need to fill in the OPENAI_API_KEY
19-
or GOOGLE_GENERATIVE_AI_API_KEY before running the app:
20-
21-
```bash
22-
mv .env.example .env
23-
```
24-
25-
2. Install dependencies:
18+
Generate and run 10 specs on TodoMVC, a classic example web app:
2619

2720
```
28-
npm install
21+
npx autospecai --url https://todomvc.com/examples/react/dist/ --apikey YOUR_OPENAI_API_KEY
2922
```
3023

31-
3. Generate and run 10 specs on TodoMVC, a classic example web app:
24+
That's it!
3225

33-
```
34-
npx autospec --url https://todomvc.com/examples/react/dist/ --model gpt-4o --spec-limit 10
35-
```
26+
You'll need to say "yes" to install the autospecai package, and the first run
27+
will take a few minutes to download dependencies like browser binaries, etc.
3628

37-
### Learn more about configuration
29+
### Using environment variables instead of passing keys as a flag
3830

31+
Copy the sample .env file, and fill in the OPENAI_API_KEY
32+
or GOOGLE_GENERATIVE_AI_API_KEY before running the app:
33+
34+
```bash
35+
mv .env.example .env # rename the example to .env
36+
nano .env # edit as you like
3937
```
40-
npx autospec --help
41-
Usage: npx autospec --url <url> [--model <model>] [--spec_limit <limit>] [--help | -h]
4238

43-
Required:
44-
--url <url> The target URL to run the autospec tests against.
39+
### Learn more about configuration
4540

46-
Optional:
47-
--help, -h Show this help message.
48-
--spec*limit <limit> The max number of specs to generate. Default 10.
49-
--model <model> The model to use for spec generation
50-
* "gpt-4o" (default)
51-
* "gemini-1.5-flash-latest"
52-
* "claude-3-haiku"
53-
* (note: Gemini flash is free up to rate limits)
41+
```bash
42+
> npx autospec --help
43+
Usage: npx autospec --url <url> [--model <model>] [--spec_limit <limit>] [--help | -h]
44+
45+
Required:
46+
--url <url> The target URL to run the autospec tests against.
47+
48+
Optional:
49+
--help, -h Show this help message.
50+
--spec_limit <limit> The max number of specs to generate. Default 10.
51+
--model <model> The model to use for spec generation
52+
* "gpt-4o" (default)
53+
* "gemini-1.5-flash-latest"
54+
* "claude-3-haiku"
55+
* (note: Gemini flash is free up to rate limits)
56+
--apikey <key> The relevant API key for the chosen model's API.
57+
* If not specified, we'll fall back on the
58+
following environment variables:
59+
* OPENAI_API_KEY
60+
* GOOGLE_GENERATIVE_AI_API_KEY
61+
* ANTHROPIC_API_KEY
5462

5563
```
5664

@@ -60,44 +68,42 @@ Logs are printed to the console like so, as well as to a trajectories/\*\*
6068
folder alongside a video, png and HTML/JSON DOM snapshots:
6169

6270
```
63-
6471
Test Summary:
6572
✔ 1. Check if the page loads successfully
66-
1.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The page loaded successfully with all expected elements visible., planningThoughtAboutTheActionIWillTake: The page has loaded successfully as indicated by the presence of the header, input field, and footer. I will mark the spec as complete.
73+
1.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The page loaded successfully with all expected elements visible., planningThoughtAboutTheActionIWillTake: The page has loaded successfully as indicated by the presence of the header, input field, and footer. I will mark the spec as complete.
6774
✔ 2. Check if the title 'todos' is displayed
68-
2.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The title 'todos' is displayed as an h1 element in the header section., planningThoughtAboutTheActionIWillTake: I will verify if the title 'todos' is displayed by checking the presence of the h1 element with the text 'todos'.
75+
2.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The title 'todos' is displayed as an h1 element in the header section., planningThoughtAboutTheActionIWillTake: I will verify if the title 'todos' is displayed by checking the presence of the h1 element with the text 'todos'.
6976
✔ 3. Check if the input box with placeholder 'What needs to be done?' is present
70-
3.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The input box with the placeholder 'What needs to be done?' is present in the HTML snapshot., planningThoughtAboutTheActionIWillTake: I will check if the input box with the placeholder 'What needs to be done?' is present in the HTML snapshot.
77+
3.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The input box with the placeholder 'What needs to be done?' is present in the HTML snapshot., planningThoughtAboutTheActionIWillTake: I will check if the input box with the placeholder 'What needs to be done?' is present in the HTML snapshot.
7178
✔ 4. Check if a new todo item can be added
72-
4.1) action: clickOn, cssSelector: input.new-todo, nth: 0, planningThoughtAboutTheActionIWillTake: I will move the mouse cursor to the input field to add a new todo item.
73-
4.2) action: keyboardInputString, cssSelector: input.new-todo, nth: 0, string: Buy groceries, planningThoughtAboutTheActionIWillTake: I will type a new todo item into the input field.
74-
4.3) action: keyboardInputSingleKey, cssSelector: input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: I will press the Enter key to add the new todo item.
75-
4.4) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The new todo item 'Buy groceries' was successfully added to the list., planningThoughtAboutTheActionIWillTake: I will mark the spec as complete because the new todo item 'Buy groceries' has been successfully added.
79+
4.1) action: clickOn, cssSelector: input.new-todo, nth: 0, planningThoughtAboutTheActionIWillTake: I will move the mouse cursor to the input field to add a new todo item.
80+
4.2) action: keyboardInputString, cssSelector: input.new-todo, nth: 0, string: Buy groceries, planningThoughtAboutTheActionIWillTake: I will type a new todo item into the input field.
81+
4.3) action: keyboardInputSingleKey, cssSelector: input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: I will press the Enter key to add the new todo item.
82+
4.4) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The new todo item 'Buy groceries' was successfully added to the list., planningThoughtAboutTheActionIWillTake: I will mark the spec as complete because the new todo item 'Buy groceries' has been successfully added.
7683
✔ 5. Check if a todo item can be marked as completed
77-
5.1) action: keyboardInputString, cssSelector: input.new-todo, nth: 0, string: Test Todo Item, planningThoughtAboutTheActionIWillTake: To check if a todo item can be marked as completed, I first need to add a new todo item. I will type a new todo item in the input field and press Enter.
78-
5.2) action: keyboardInputSingleKey, cssSelector: input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: I need to press Enter to add the new todo item to the list.
79-
5.3) action: clickOn, cssSelector: input.toggle, nth: 0, planningThoughtAboutTheActionIWillTake: Now that the todo item is added, I will mark it as completed by clicking the checkbox next to the todo item.
80-
5.4) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The todo item was successfully marked as completed., planningThoughtAboutTheActionIWillTake: The todo item has been marked as completed, as indicated by the strikethrough text and the checkbox being checked. I will mark the spec as complete.
84+
5.1) action: keyboardInputString, cssSelector: input.new-todo, nth: 0, string: Test Todo Item, planningThoughtAboutTheActionIWillTake: To check if a todo item can be marked as completed, I first need to add a new todo item. I will type a new todo item in the input field and press Enter.
85+
5.2) action: keyboardInputSingleKey, cssSelector: input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: I need to press Enter to add the new todo item to the list.
86+
5.3) action: clickOn, cssSelector: input.toggle, nth: 0, planningThoughtAboutTheActionIWillTake: Now that the todo item is added, I will mark it as completed by clicking the checkbox next to the todo item.
87+
5.4) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The todo item was successfully marked as completed., planningThoughtAboutTheActionIWillTake: The todo item has been marked as completed, as indicated by the strikethrough text and the checkbox being checked. I will mark the spec as complete.
8188
✔ 6. Check if a todo item can be edited
82-
6.1) action: keyboardInputString, cssSelector: input.new-todo, nth: 0, string: Test Todo Item, planningThoughtAboutTheActionIWillTake: To check if a todo item can be edited, I first need to add a todo item. I will type a todo item in the input field and press Enter.
83-
6.2) action: keyboardInputSingleKey, cssSelector: input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: I need to press Enter to add the todo item to the list.
84-
6.3) action: doubleClickOn, cssSelector: ul.todo-list li div.view label, nth: 0, planningThoughtAboutTheActionIWillTake: To edit the todo item, I need to double-click on the todo item label.
85-
6.4) action: keyboardInputString, cssSelector: ul.todo-list li div.input-container input.new-todo, nth: 0, string: Edited Todo Item, planningThoughtAboutTheActionIWillTake: Now that the todo item is in edit mode, I will change the text of the todo item.
86-
6.5) action: keyboardInputSingleKey, cssSelector: ul.todo-list li div.input-container input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: To save the edited todo item, I need to press Enter.
87-
6.6) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The todo item was successfully edited and saved as 'Edited Todo Item'., planningThoughtAboutTheActionIWillTake: The todo item has been successfully edited and saved. I will mark the spec as complete.
89+
6.1) action: keyboardInputString, cssSelector: input.new-todo, nth: 0, string: Test Todo Item, planningThoughtAboutTheActionIWillTake: To check if a todo item can be edited, I first need to add a todo item. I will type a todo item in the input field and press Enter.
90+
6.2) action: keyboardInputSingleKey, cssSelector: input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: I need to press Enter to add the todo item to the list.
91+
6.3) action: doubleClickOn, cssSelector: ul.todo-list li div.view label, nth: 0, planningThoughtAboutTheActionIWillTake: To edit the todo item, I need to double-click on the todo item label.
92+
6.4) action: keyboardInputString, cssSelector: ul.todo-list li div.input-container input.new-todo, nth: 0, string: Edited Todo Item, planningThoughtAboutTheActionIWillTake: Now that the todo item is in edit mode, I will change the text of the todo item.
93+
6.5) action: keyboardInputSingleKey, cssSelector: ul.todo-list li div.input-container input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: To save the edited todo item, I need to press Enter.
94+
6.6) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The todo item was successfully edited and saved as 'Edited Todo Item'., planningThoughtAboutTheActionIWillTake: The todo item has been successfully edited and saved. I will mark the spec as complete.
8895
✔ 7. Check if a todo item can be deleted
89-
7.1) action: keyboardInputString, cssSelector: input.new-todo, nth: 0, string: Test Todo Item, planningThoughtAboutTheActionIWillTake: To check if a todo item can be deleted, I first need to add a todo item. I will type a todo item in the input field and press Enter.
90-
7.2) action: keyboardInputSingleKey, cssSelector: input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: I need to press Enter to add the todo item to the list.
91-
7.3) action: clickOn, cssSelector: button.destroy, nth: 0, planningThoughtAboutTheActionIWillTake: Now that the todo item is added, I need to click on the delete button (the button with class 'destroy') to delete the todo item.
92-
7.4) action: hoverOver, cssSelector: li[data-testid='todo-item'], nth: 0, planningThoughtAboutTheActionIWillTake: The delete button is not visible. I need to hover over the todo item to make the delete button visible.
93-
7.5) action: clickOn, cssSelector: button.destroy, nth: 0, planningThoughtAboutTheActionIWillTake: Now that the delete button is visible, I will click on the delete button to delete the todo item.
94-
7.6) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The todo item was successfully deleted and is no longer visible on the page., planningThoughtAboutTheActionIWillTake: The todo item has been successfully deleted as it is no longer visible on the page.
96+
7.1) action: keyboardInputString, cssSelector: input.new-todo, nth: 0, string: Test Todo Item, planningThoughtAboutTheActionIWillTake: To check if a todo item can be deleted, I first need to add a todo item. I will type a todo item in the input field and press Enter.
97+
7.2) action: keyboardInputSingleKey, cssSelector: input.new-todo, nth: 0, key: Enter, planningThoughtAboutTheActionIWillTake: I need to press Enter to add the todo item to the list.
98+
7.3) action: clickOn, cssSelector: button.destroy, nth: 0, planningThoughtAboutTheActionIWillTake: Now that the todo item is added, I need to click on the delete button (the button with class 'destroy') to delete the todo item.
99+
7.4) action: hoverOver, cssSelector: li[data-testid='todo-item'], nth: 0, planningThoughtAboutTheActionIWillTake: The delete button is not visible. I need to hover over the todo item to make the delete button visible.
100+
7.5) action: clickOn, cssSelector: button.destroy, nth: 0, planningThoughtAboutTheActionIWillTake: Now that the delete button is visible, I will click on the delete button to delete the todo item.
101+
7.6) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The todo item was successfully deleted and is no longer visible on the page., planningThoughtAboutTheActionIWillTake: The todo item has been successfully deleted as it is no longer visible on the page.
95102
✔ 8. Check if the footer text 'Double-click to edit a todo' is displayed
96-
8.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The footer text 'Double-click to edit a todo' is displayed in the HTML snapshot., planningThoughtAboutTheActionIWillTake: I will check if the footer text 'Double-click to edit a todo' is displayed by examining the HTML snapshot.
103+
8.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The footer text 'Double-click to edit a todo' is displayed in the HTML snapshot., planningThoughtAboutTheActionIWillTake: I will check if the footer text 'Double-click to edit a todo' is displayed by examining the HTML snapshot.
97104
✔ 9. Check if the footer text 'Created by the TodoMVC Team' is displayed
98-
9.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The footer text 'Created by the TodoMVC Team' is displayed as expected., planningThoughtAboutTheActionIWillTake: I will mark the spec as complete because the footer text 'Created by the TodoMVC Team' is displayed in the screenshot and confirmed in the HTML snapshot.
105+
9.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The footer text 'Created by the TodoMVC Team' is displayed as expected., planningThoughtAboutTheActionIWillTake: I will mark the spec as complete because the footer text 'Created by the TodoMVC Team' is displayed in the screenshot and confirmed in the HTML snapshot.
99106
✔ 10. Check if the footer text 'Part of TodoMVC' is displayed
100-
10.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The footer text 'Part of TodoMVC' is displayed in the HTML snapshot., planningThoughtAboutTheActionIWillTake: I will check the HTML snapshot to see if the footer text 'Part of TodoMVC' is present.
107+
10.1) action: markSpecAsComplete, reason: The spec passed, fullProseExplanationOfReasoning100charmax: The footer text 'Part of TodoMVC' is displayed in the HTML snapshot., planningThoughtAboutTheActionIWillTake: I will check the HTML snapshot to see if the footer text 'Part of TodoMVC' is present.
101108
autospec passed for https://todomvc.com/examples/react/dist/
102-
103-
```
109+
```

‎cli.js

+16-2
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,12 @@ if (args.includes("--help") || args.includes("-h")) {
4747
* "gemini-1.5-flash-latest"
4848
* "claude-3-haiku"
4949
* (note: Gemini flash is free up to rate limits)
50+
--apikey <key> The relevant API key for the chosen model's API.
51+
* If not specified, we'll fall back on the
52+
following environment variables:
53+
* OPENAI_API_KEY
54+
* GOOGLE_GENERATIVE_AI_API_KEY
55+
* ANTHROPIC_API_KEY
5056
`);
5157
process.exit(0);
5258
}
@@ -59,13 +65,21 @@ if (!testUrl) {
5965
process.exit(1);
6066
}
6167

62-
const model = getArgValue("--model", "gpt-4o");
68+
const apiKey = getArgValue("--apikey", null);
69+
if (!apiKey) {
70+
console.warn(
71+
"Warning: No API key provided via CLI flag --apikey. Falling back to environment variables.",
72+
);
73+
}
74+
75+
const modelName = getArgValue("--model", "gpt-4o");
6376
const specLimit = getArgValue("--spec_limit", 10);
6477

6578
main({
6679
testUrl,
67-
model,
80+
modelName,
6881
specLimit,
82+
apiKey,
6983
})
7084
.then(console.log)
7185
.catch(console.error);

0 commit comments

Comments
 (0)