Support for bytecode files and libraries #2
Conversation
mpickering
force-pushed
the
wip/bytecode-options
branch
from
98c34c3 to
9408a12
Compare
mpickering
changed the title
Implements cabal support for persistent bytecode artifacts (.gbc files) and bytecode libraries alongside native code, enabling faster build times and better tooling integration for development and REPL use cases.
mpickering
force-pushed
the
wip/bytecode-options
branch
from
9408a12 to
d7be9e9
Compare
|
Rendered 👀 |
Thanks, I added the link to the main post. |
|
Small question: is the byte code inherently platform or architecture specific? Or is it portable? Just curious. |
|
I'm curious about that as well: while technically bytecode needn't be architecture-specific, word size and endianness can still affect it. (To be clear, this isn't a Cabal question or directly related to the proposal; it might affect uptake, but I doubt it since it would be the only piece of a registered package that would be portable anyway.) |
There was a problem hiding this comment.
Consider this conditional, since I'll want to also review the GHC ticket/MR. But taken by itself, it shouldn't be much more than a cut-and-paste of the vanilla build way. (Which does make me wonder if we have a sensible architecture for ways in place, but that's not really part of this proposal as such.)
Inherently, I think the bytecode part is quite portable but there are a few things to consider which would make it impractical now.
I think @TerrorJack and @alt-romes could add more |
|
Interesting proposal! I've also been concerned about it being platform specific, however I believe this is completely irrelevant, because cabal only installs these local, compiler specific (and thus platform specific) store? Am I mistaken? I am however reminded about the Fat Interface Files discussion we had a few years back that Josh was working on. If gbc was part of .hi files, and .hi files were containers, we wouldn't need a change to cabal at all, right? And all of this could live exclusively in GHC? |
|
I don't think this would fit in fat interface files (which exist now). And yes, these would be in package dbs, which are already host-specific. As, come to think of it, are the .hi files. |
|
As far as implementation, I think haskell/cabal#9900 could be used as a template? |
Why not? Sounds to me like the bytecode is technically metadata for the compiler (same as the interface files), and if it was in the interface files, the whole change would be transparent to cabal and self-contained in ghc? |
|
You'll have to make a ghc proposal for that, though. The current plan is to treat it as a new object format and compiler way, and that will require cabal (and stack) support. |
Yes, I'm questioning the premise here. I don't see why you do. The hi file format is internal to ghc; I do not believe there are any guarantees around the .hi file format. It's an opaque blob to tooling. I don't think there are any interoperability guarantees for it. Would MicroHS read ghc .hi files? Again, ghc bytecode is internal to GHC, and if it was part of the hi files, it would be transparent to tooling, and automatically supported by cabal and stack (and any other tool dealing with ghc build products, nix, bazel, buck, ...) And again, if bytecode is considered internal to GHC (and I don't think we have a haskell bytecode spec?), then I'm not convinced it would need a ghc proposal. |
stack reads It also seems to me that adding bytecode to it (a) complicates compilation (does that mean bytecode is added even during normal compiles?) and (b) slows compilation in general because |
|
I am raising this on the ghc issue tracker as well. I do raise it here because if it's part of the .hi file. This proposal, and any subsequent change to cabal or stack could be cut short.
Maybe this is a great opportunity to spec a .hi container format if we want to make it publicly readable. Seekable containers are a fairly common pattern? |
|
I have a feeling that if the ghc proposal is advanced enough that they're proposing cabal changes to support it, it's too far advanced to be redirected at this point. And I think mpilgrem proposed a container format for .hi files as well. |
I don't see a ghc proposal. I only see a ghc issue tracking it? Is there a ghc-proposal for this I'm not aware of? |
|
Hm. I would have expected there to be, but indeed there seems to be only the tracking issue and Discourse RFC. |
|
At today’s Cabal-dev meeting there were no objections about this. This proposal is accepted. |
So these proposals will just be voted on, without discussions on the proposal being addressed? Is there a recording of said cabal-dev meeting? PS: I don't mind if the cabal-devs, proposer, or someone else deems raised concerns irrelevant, misguided, or something else entirely, but just outright ignoring them has me questioning the point of this repository and process. |
|
Hello Moritz, Cabal-dev meeting notes are here. As per Cabal Proposal Process the Cabal-dev meeting does not debate on the technical merit of the proposal, but only if rough consensus has been reached in the proposal PR. Evaluating both this thread and the one on Discourse, we felt the proposal has rough consensus. Cabal-proposal process is still new, any feedback is welcome to improve the process. |
|
I found Brandon's assignment above convincing, but in the hindsight we should have waited for @angerman to react. Indeed, this is a new process, and we're still learning. |
|
I don't consider any of my concerns addressed. My fundamental question remains: why does this need to be in cabal and can not be self contained in ghc. The proposal doesn't address this, and the discussion doesn't address this. I've had some discussions with others around this, and there are some reasons for why this path might have been chosen, yet it's not reflected anywhere. If this proposal process is supposed to be meaningful we need to have raised questions addressed and this could perfectly fine be the cabal devs saying they consider these questions to be irrelevant, made in bad faith, unproductive, or something else entirely; not addressing or acknowledging them in any form leaves me wondering what this process is for. Let's hope we can all learn from this! I would still very much like to see a rational for why this logic needs to live in cabal, and can't be contained within GHC itself. Any added complexity is a liability and my understanding is that this process is precisely here to help with this. Maybe I'm wrong. |
|
I will repeat that that is not a Cabal question, it is a GHC question. The GHC team made its decision and launched a Cabal proposal based on it; it is not our place to question the GHC team's decision, only to consider its impact on Cabal. |
That's not a reason to accept the proposal though. Otherwise it seems GHC can dictate whatever they prefer to push down into cabal. I feel this proposal process lacks the very fundamentals of a process: discussing the technical merits. |
|
Also I'd like to note that spreading discussion across two venues seems unhelpful, either there will be duplication or people will need to follow both to know the current status. |
|
I think "ghc-internal-proposal" and cabal-proposal should be together. Since normal Haskell users don't use ghc directly, so the discussion platform for (static/dynamic linking, interface files and other things affecting the build of the program itself) should be in one place? Why not put all proposals affecting the build of the program here? Then, in order for the proposal to be accepted, we need to add to the cabal-proposal committee those who currently approve it in ghc? |
|
Otherwise, a situation is created here where a proposal can be accepted by the cabal committee, but not by the ghc devs? Why should we get into such a situation? I would probably like to have a place like |
|
With all due respect to committees, there is no Cabal Committee. The closest one is CLC and we welcome their engagement in matters such as this one, which span more than one component of the Haskell ecosystem. I don't think there's a problem if a GHC dev want to probe if cabal is in principle ready to accept a given set of changes induced by a GHC feature before ironing out the details of the feature among GHC devs and users. BTW, IIRC the GHC feature in question has been discussed in GHC tickets and on discourse (or at least there was a call for discussion some time ago), which was chronologically before this here cabal proposal. |
Could you please clarify whether Matthew has deemed the discussion complete and requested for the proposal to be decided at the Cabal-dev meeting? I do not see anything to such effect in the thread, but maybe it was requested by other channels? Lines 77 to 78 in 37f681e |
|
The decision was not requested by Matthew.
I wrote the point in the meeting agenda and asked for a decision to be made.
Il 12 settembre 2025 alle 13:57 ˌbodʲɪˈɡrʲim ha scritto:
… Bodigrim left a comment (haskell/cabal-proposals#2)
> As per [Cabal Proposal Process](https://github.com/haskell/cabal-proposals/blob/master/README.md) the Cabal-dev meeting does not debate on the technical merit of the proposal, but only if [rough consensus](https://datatracker.ietf.org/doc/html/rfc7282) has been reached in the proposal PR.
Could you please clarify whether Matthew has deemed the discussion complete and requested for the proposal to be decided at the Cabal-dev meeting? I do not see anything to such effect in the thread, but maybe it was requested by other channels?
https://github.com/haskell/cabal-proposals/blob/37f681e36daaf24dea0ca5be7296f075dab08554/README.md?plain=1#L77-L78
--
Reply to this email directly or view it on GitHub:
#2 (comment)
You are receiving this because you commented.
Message ID: ***@***.***>
|
While I agree with this statement, I would like to make my sentiment loud and clear: the history of mixing up GHC and Cabal concerns has been devastating. The amount of complexity that over the years got into Cabal to alleviate GHC complexity and work around its quirks and its bugs is staggering. I guesstimate that maybe 40% of Cabal is about making sure the user does not accidentally call GHC in a way that can fail. This is off topic wrt to the proposal but, as we briefly discussed the proposal process, I think it is an appropriate remark. PS: I have been on PTO and I didn't look at the proposal in detail. That's why I chose the word "sentiment". |
|
Good morning everyone, a lot seems to have happened whilst I was away rambling. It does seem like @angerman points out a good point in this instance about an opportunity to improve on this process. He had concerns which he didn't feel like had been addressed, and so he hasn't yet been brought into the consensus. I think he summaries that well in this comment.
It seems like a good improvement already that cabal developers wanted to pursue a clear decision quickly rather than waiting for a long while. Just in this particular instance it was a little premature. I'm sure that over time the cadence will be established more naturally once there are more proposals to talk about. So how about in this instance we re-open the proposal whilst there is still ongoing discussion with @angerman? As for the scope of the process, it seems clear to me that cabal proposals should primarily be about issues that cabal developers control (ie, not issues to do with GHC). GHC issues are discussed on the GHC issue tracker. There is a decision making process for GHC and a different one for Cabal. In this particular case:
I can see there could be some nuance with the second point:
Finally, it seems that @angerman and @andreabedini have identified some areas of GHC which could be improved in order to better facilitate working with Cabal, so could you open some GHC and cabal issues which track these potential improvements? I'm sure that everyone will be happy if issues can be identified and fixed. |
|
I think proposal author has the authority to re-open a proposal discussion, so I think @mpickering has just done that and this proposal is open for further discussion and a new decision will be made for an amended (or not) proposal in due time. Are there any objections? @angerman, @andreabedini and others, please kindly continue the discussion, I hope it's clear we all value your feedback, regardless of at which forum it's provided. |
|
Thank you all for the input, I'm trying to understand how this could work better. We definitely have deficiency in cross project coordination for proposals. I don't think I agree with the "hand down" approach to cabal. This seems to me like: "We've decided on X, and we now would like you to agree to Y". What happens if cabal developers disagree? If cabal developers have request for a different approach that would work better with cabal? The assumption seems to be: it's been decided we've just got to rubber stamp it. That seems bad. So my understanding is, I'll need to keep a close eye on gitlab issues outside of the cabal-proposals repo, an on discourse to see what might end up being decided, and then just passed on here? What about stack? Has @mpilgrem, and other stack maintainers, had their chance to provide input? I'm a bit at a loss providing constructive feedback on how this could be done better going forward; but it seems we need some form of cross project proposals system, where everyone comes to the table to discuss items that are not confined to projects. If
Again, right now I'm at a loss what the point (especially what the realistic expectations/actions/influence) of this repo is. I'm sure there are probably good reasons why we don't want to use the fat |
|
A few remarks and clarifications:
Cabal is, fundamentally, a build tool. We didn't get to dictate to existing compilers how they should work in the beginning, and it's not clear to me that it's our job to do so now. (I'm pretty sure we would have had things to say to the
My understanding was that it was supposed to be the first. A number of people commenting on this proposal seem to think it's the second. I would have opposed the (To be honest, I had been wondering about Stack support as well. As far as MicroHS is concerned, I think @augustss has pretty much given up on any process applying to it; in particular, haskell/cabal#9718 removed any ability (if any had remained) for |
|
I agree with @geekosaur, this repo is about changes to cabal. Hence this proposal is about the options which cabal could provide if the compiler support serialising bytecode. I think it would be good if cabal supported this feature, I'm not sure how users could benefit if it didn't. Cabal developers (and anyone else) are invited to participate in the design of the feature by participating on gitlab, attending the GHC developer meetings or commenting on the other discussion threads. That's the mechanism that GHC developers use to decide on changes which don't require GHC proposals. When designing a feature for GHC, considering how it will work with Cabal and other tools is one of the main considerations. If you’re interested in influencing GHC development, it’s important to follow the channels where those discussions happen. Some of the concerns raised seem to stem from uncertainty about whether GHC processes sufficiently capture Cabal’s perspective. It seems that this case has been a good example of gathering feedback from cabal developers during the development process? The proposal has been raised in conjunction with the feature being developed, and otherwise advertised widely. That seems like a good improvement over how other features have been developed in the past, and has been made possibly by this proposals process. |
That's something I've seen GHC expecting others to do for a while now. E.g. GHC has been expecting stakeholders of their releases to follow their own channels, issue trackers, non-advertised meeting calls etc. It took me months of effort to convince GHC HQ to become more "anticipating" about stakeholders, which ultimately resulted in the GHC HQ release management document. I think this document is a signal: no, not everyone has the capacity to follow GHC development closely. Or to put it more simply: I don't think it's proper to just go ahead with a design, because cabal developers have not commented on an issue that (possibly) affects cabal one way or another. The proper form is to CC them and ask for their opinions. We all have busy inboxes and even a discourse post is likely not enough to get the attention of the relevant folks. |
|
I agree with you, Julian, that building consensus among stakeholders is important. That’s why I prepared a document explaining how Cabal could interact with the new GHC feature and sent it to cabal developers for comment, along with links where feedback on the GHC design can be provided. Any input there can be incorporated both into the GHC design and into Cabal’s integration. Do you see another concrete step which would be an improvement? My point about “following the channels” is simply that now those links are available, anyone who wants to participate has a clear path to do so. Perhaps I should have made this more explicit in the proposal document. I had assumed it was a given that feedback on the design would be welcome. |
|
The discussion about the separate files or not is continuing over here: https://gitlab.haskell.org/ghc/ghc/-/issues/26298#note_637273 |
|
I will further add that I think GHC needs a process for this kind of thing. Wider dissemination should be part of it, and the process will give stakeholders a single place to monitor. (Why, for example, would arbitrary GHC users think to look in Cabal's proposals for information about a new GHC feature?) Other things that could benefit:
|
|
@angerman, thanks for looping me in. There is a lot to read up on, but I think that Stack is likely downstream of choices made by Cabal (the library). As an aside on Stack and GHC's |
Yes, that's pretty much my understanding as well. I just wonder if it wouldn't be better if this wasn't cabal-proposals, but these kinds of proposal would be rather in haskell-ecosystem-proposals, where the relevant stakeholders, including stack, would be asked for input. Given the general consensus(?) that this repo is for
This specific proposal seems to be straddling the lines between GHC and Cabal. I think my biggest concern here is that, it's asking for downstream changes due to an upstream change, but it's unclear what the input on this proposal is supposed to be here other than "Ok 🤷♂️". I can see some use out of this repo for conceptual/structural changes to cabal itself, where proposals of restructuring and similar of cabal is concerned. For items like the one under discussion, it seems we'd rather want a ghc+cabal+... proposal to get agreement on altogether (if we want to go down the proposal route). |
|
After writing the last comment, I feel more confused than before. We seem to have two situations:
For (1) I can understand the proposal process can help build consensus. However, even then it's all fairly hypothetical at that point, there might still be a lot of unknown-unknowns surfacing during the implementation, which could end up contradicting assumptions? The benefit would be that general consensus of direction is reached, and buy-in from the relevant stakeholders? |
|
I've been in open source for nearly two decades and have worked on projects bigger than Cabal and GHC, so I'm familiar with different types of processes. But I do not understand this process. |
|
I understand the topics are intertwined, but we need to have a thread/PR about each problem:
This thread is over fifty-messages long, and indeed there is a lot to read up on. If nobody objects, I can split the latter in a new issue inside this repo, but I wouldn’t know where 2. should be (GHC gitlab)? |
|
Thanks for raising your concerns, @hasufell. I think it can be summarised that if you want to make a significant change to cabal then make a proposal to discuss the change. I can answer any specific questions if you have them. The question about "haskell-ecosystem" proposals is a much bigger question than "cabal-proposals" can answer. It seems to me that the hardest part of establishing that such a process would have legitimacy to establish direction with the large number of projects in the community. For instance, I would imagine that projects which may be interested would be GHC, cabal, stack, GHCup, haskell.nix, HLS, Haskell Nixpkgs infrastructure, hackage etc The current community model is decentralised and built upon the principle of trust and cooperation. In response to your (1) vs (2) comment, it seems that we can agree about the utility of (1). For (2), I see your point that much of the discussion will naturally happen on the PR. Still, I think the proposal process does add something here.
In other words, it’s more about capturing agreement and documenting trade-offs at the "design intent" level. I think an example of this process is in the implementation of bytecode objects. I have already prepared the code for that patch, but we are discussing and establishing the specification separately on the issue. I find that process really useful. Thanks for bringing up these questions, the process is still new and it's useful to establish what people should expect when writing a proposal. |
|
We have a variety of proposal processes in the Haskell ecosystem.
I think the variety of proposal processes reflects that we have a variety of self governing projects (Conway's law). We all talk to each other but ultimately decisions are made by these individual groups. For instance, I think it's good for stack devs to comment on cabal and GHC changes. It is not clear to me why we need a separate cross cutting decision process to allow that to happen. (This overlaps a bit with Matt's points. I wrote this before seeing that but couldn't send it due to lack of internet) |
I don't think it achieves that. You're expecting people to monitor an additional repository, read the proposals and know when it matters to them and when their input is required. Anyone doing that is already an engaged expert. To me, this process looks more like a way of creating documentation. I don't consider it a decision making process. And that's why it's super confusing...
And this thread is proof of that. Matt created a proposal, some people commented, apparently were not deemed "experts" and their comments ignored and a decision reached anyway. This has the risk of exacerbating bias. Because both the "experts" and the "decision makers" are undefined. |
|
I think we’ve agreed in this thread, @hasufell, that a mistake was made in accepting this proposal prematurely. The assessment of consensus failed, since it wasn’t evident to everyone in the discussion that all concerns had been addressed. That’s an opportunity to learn from and improve the process for future proposals. People who are interested in Cabal development can follow this repo and offer comments. Where needed, opinion can also be solicited explicitly by mentioning individuals or referring them to the thread. @angerman has done a great job of that in this thread by involving @mpilgrem. There is a section in the proposal to identify who the stakeholders are, which provides the opportunity to reflect on who will be affected by a change. Ultimately, the process relies on trust that Cabal developers will make decisions with the broader ecosystem in mind. At this stage, my understanding is that Cabal developers don’t want a more formal process with defined "experts" and "decision makers" - the added bureaucracy would outweigh the benefits. The aim here is to keep things lightweight, to gather consensus and document rationale without slowing decisions. The alternative I see is to return to the previous model, which often led to very delayed decisions (or no decisions at all). |
I don't see it as about centralization, but synchronization. If the stakeholders aren't on the same page, the ecosystem has a problem; a centralized venue that stakeholders can easily follow for cross-project issues would help everyone impacted by such changes. But instead of centralized decision making, it's where people would discuss changes and iron out differences. |
|
I don't think I have anything more to contribute here. The relevant ticket for discussion seems to be ghc!26298. The consensus seems to be that this repo is cabal-only proposals, and not ecosystem proposals.
This seems to be the primary misunderstanding on my part. This repository is about capturing agreement and documenting trade-offs. Not discussing the implementation. I will treat it as such going forward. It would be great if this proposal would then also document the trade-offs as discussed in ghc!26298? Or is a reference from the proposal sufficient? I've raised my concerns, and they have been largely deemed out of scope. |
That's essentially what we agreed on in the proposal for it. I'm not sure I agree fully, because I'm not sure it actually accomplishes much, but everyone else went that way so that's what was done. |
|
It’s been a few weeks since the last round of comments, and it seems we’re in agreement that this proposal’s scope is limited to adding first-class support in Cabal and cabal-install for producing bytecode. I have responded to all the comments I believe were outstanding. In particular there were questions by Chris and Moritz about particular aspects of the design. Given that consensus, I’d like to resubmit the proposal for consideration at the Cabal developers meeting tonight. |
14 participants
This proposal describes extensions to
Cabalandcabal-installto support building, installing, and managing persistent bytecode artifacts (.gbcfiles) and bytecode libraries alongside native code for faster development workflows.Rendered