Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor the configure_workers_and_start.py script used internally by Complement. #16803

Closed
wants to merge 20 commits into from
Closed

Refactor the configure_workers_and_start.py script used internally by Complement. #16803

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
b39a50a
Remove obsolete `"app"` from worker templates
reivilibre Nov 16, 2023
a22eb7d
Convert worker templates into dataclass
reivilibre Nov 16, 2023
ba3b6a4
Use a lambda for the worker name rather than search and replace later
reivilibre Nov 16, 2023
67d4fc8
Collapse WORKERS_CONFIG by removing entries with defaults
reivilibre Nov 16, 2023
94a85b3
Convert listener_resources and endpoint_patterns to Set[str]
reivilibre Nov 16, 2023
26073fa
Tweak comments
reivilibre Nov 16, 2023
8b74639
Add `merge_into`
reivilibre Nov 16, 2023
f38297b
Remove special logic for adding stream_writers: just make it part of …
reivilibre Nov 16, 2023
7d8824e
Rename function to add_worker_to_instance_map given reduction of scope
reivilibre Nov 16, 2023
f49dbc7
Add `sharding_allowed` to the WorkerTemplate rather than having a sep…
reivilibre Nov 16, 2023
3bb21a9
Use `merge_into` when adding workers to the shared config
reivilibre Nov 16, 2023
fbafde8
Promote mark_filepath to constant
reivilibre Nov 16, 2023
321d359
Add a --generate-only option
reivilibre Nov 16, 2023
259a808
Docstring on WorkerTemplate
reivilibre Dec 6, 2023
3a46cf0
Fix comment and mutation bug on merge_worker_template_configs
reivilibre Dec 6, 2023
2f1d727
Update comment on `merged`
reivilibre Dec 6, 2023
ad4bb0e
Tweak `instantiate_worker_template`, both in name, description and va…
reivilibre Dec 6, 2023
2ff1de3
Newsfile
reivilibre Jan 10, 2024
29541fd
Move `stream_writers` to their own field in the WorkerTemplate
reivilibre Jan 17, 2024
c91ab4b
Remove `merge_into` and just have `merged` which copies inputs to avo…
reivilibre Jan 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions docker/configure_workers_and_start.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -784,10 +784,9 @@ def generate_worker_files(
{"name": worker_name, "port": str(worker_port), "config_path": config_path}
)

# Update the shared config with any worker_type specific options. The first of a
# given worker_type needs to stay assigned and not be replaced.
worker_config["shared_extra_conf"].update(shared_config)
shared_config = worker_config["shared_extra_conf"]
# Update the shared config with any options needed to enable this worker.
merge_into(shared_config, worker_config["shared_extra_conf"])
erikjohnston marked this conversation as resolved.
Show resolved Hide resolved

if using_unix_sockets:
healthcheck_urls.append(
f"--unix-socket /run/worker.{worker_port} http://localhost/health"
Expand Down