Skip to content

Conversation

@borzunov
Copy link
Collaborator

@borzunov borzunov commented Aug 23, 2023

This PR makes both clients and servers work on macOS. Specifically, it:

  • Follows Use proper p2pd binary on macOS learning-at-home/hivemind#586 to run a macOS-compatible p2pd binary (both x86-64 and ARM64 are supported)
  • Fixes forking issues and tests on macOS, Python 3.10+
  • Introduces basic support for serving model blocks on Apple M1/M2 GPUs (torch.mps)
  • Increases max number of open files by default (it's not enough on Linux and is really small on macOS)

@borzunov borzunov marked this pull request as draft August 24, 2023 22:17
@borzunov borzunov force-pushed the macos-support branch 2 times, most recently from 98ad98f to 70b8418 Compare August 24, 2023 23:10

# Simulate requests coming from ConnectionHandlers
proc = mp.context.ForkProcess(target=_submit_tasks, args=(pools, results_valid))
proc.start()
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving that before Runtime.start() since (a) that's how real server works and (b) so we don't fork after creating active threads (this provably improves test stability)

- { model: 'bigscience/bloom-560m', os: 'ubuntu', python-version: '3.8' }
- { model: 'bigscience/bloom-560m', os: 'ubuntu', python-version: '3.11' }
- { model: 'Maykeye/TinyLLama-v0', os: 'ubuntu', python-version: '3.8' }
- { model: 'Maykeye/TinyLLama-v0', os: 'ubuntu', python-version: '3.11' }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • would be nice to add adapters for llama tests since they are the new main
  • if no time, please create an issue

@borzunov borzunov force-pushed the macos-support branch 2 times, most recently from ce2a348 to f47e922 Compare August 28, 2023 17:55
Copy link
Collaborator

@justheuristic justheuristic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@borzunov borzunov marked this pull request as ready for review August 28, 2023 20:03
@borzunov borzunov force-pushed the macos-support branch 2 times, most recently from 568a1c4 to 8889cab Compare August 29, 2023 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants