Limit parallel Scala.js linking jobs to avoid high memory pressure #6260
Conversation
Fix: com-lihaoyi#6226 I just hardcoded the limit to `2`.
lefou
force-pushed
the
tr-scalajs-linker-parallel
branch
from
72beb07 to
d2aa908
Compare
davesmith00000
left a comment
davesmith00000
left a comment
There was a problem hiding this comment.
LGTM, thanks for picking this up @lefou. 🙏
It's kind of unsatisfying having to hardcore a value here, but I believe that setting a hardcoded sensible value is a better situation than people accidentally getting into an OutOfMemory blackhole by performing a common action, such as running all their tests.
| def scalaJSWorker: Worker[ScalaJSWorker] = Task.Worker { | ||
| new ScalaJSWorker( | ||
| jobs = Task.ctx().jobs, | ||
| linkerJobs = 2 |
There was a problem hiding this comment.
Just a suggestion: Perhaps this value could be exposed on ScalaJSModuleAPI? It can have a nice low default but allow people to tweak it to their needs or based on some environmental heuristic? E.g. They have a massive CI server and can afford to open up the parallelism.
Exposing it on the API also slightly improves the transparency around what's going on here, but perhaps this will need to be documented somehow?
There was a problem hiding this comment.
Yeah, I already thought about how to configure it, but didn't want to overengineer it.
The natural place for a config task would be the ScalaJSWorker, which is currently not designed to be customized, in a way other worker are, for example the JvmWorkerModule. Also, since there are potentially more than one ScalaJSWorker, we would need to introduce a new shared worker, so this route isn't a trivial change.
What would be somewhat easier is accepting an environment variable.
Also, we should converge to a "sensible default". I don't work with Scala.JS often, so I have no "feeling" for what a good value might be. We might also apply some logic based on heuristics, which I don't have.
There was a problem hiding this comment.
I'm not sure what reasonable heuristics you could sensibly apply, and I suspect attempting to do that might be a lot of work for not a lot of reward. 🤷
FWIW, @lolgab was suggesting a concurrency of 1 in a discussion on Discord, and I'm using 2 in CI:
https://github.com/PurpleKingdomGames/indigoengine/blob/main/ci.sh#L9-L10
2 participants
Fix: #6226
I just hardcoded the limit to
2for now.