Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimal collection for running on Xen/ARM64 #1358

Draft
wants to merge 45 commits into
base: main
Choose a base branch
from
Draft

Conversation

ehem
Copy link
Contributor

@ehem ehem commented Aug 2, 2024

For people who didn't figure out the end goal of all the pull requests, here is a proof of concept with the pieces gathered together. Large amounts of other cleanup was found during implementation, hopefully those get in too.

The presently tested and known working bootloader is the ArmVirtXen build of Tianocore/EDK2. Due to a change in Xen at present one patch is required. Hopefully a similar patch will be integrated soon, but presently that extra is needed.

Configure the VM with kernel = "XEN_EFI.fi" and a FreeBSD mini-memstick build as a disk. Then everything will boot as expected.

Likely a bunch of squashing is needed, this though is a bit tricky to ensure Julien Grall gets appropriate credit. Though the Xen implementation was heavily rewritten.

Issues:

sys/kern/kern_clock.c Outdated Show resolved Hide resolved
@ehem
Copy link
Contributor Author

ehem commented Aug 3, 2024

Commit cecb5a1 was tested and confirmed to work.

sys/kern/kern_clock.c Outdated Show resolved Hide resolved
sys/kern/kern_clock.c Outdated Show resolved Hide resolved
@ehem
Copy link
Contributor Author

ehem commented Aug 5, 2024

Before being rebased, 54bf574 was tested and confirmed to work. The delta is simply removing the stray clock interrupt fix and pulling 2 of the intermediates.

Mostly there are 3 problems in need of fixes. First, since Xen interrupts are very dynamic, release needs to work #1281. Second, some sort of per-processor hook is needed; pic_init_secondary() is basically ideal, but the current implementation is problematic, #1280.

Lastly, INTRNG's interface is appropriate for many PICs. Problem is due to being an unusual bus, this is well suited to a lower-level interface, #1285. Trying to work with the existing interface would be quite expensive.

Then simply becomes an issue of figuring out how much squashing should be done (while @jgrall did the original implementation, this has rather a lot of changes to get it into shape).

@ehem
Copy link
Contributor Author

ehem commented Aug 8, 2024

So instead of #1280, #1363 also functions to provide access to the appropriate hooks. For sys/arm64/include/intr.h use:

+#ifdef XENHVM
+#define INTR_ROOT_XEN	2
+#define INTR_ROOT_NUM	3
+#else
+
 #define INTR_ROOT_NUM	2
+#endif

In sys/dev/xen/intrng/xen_arch_intr.c add:

+	i = intr_pic_claim_root_num(dev, ofw_bus_get_node(dev),
+	    (intr_irq_filter_t *)panic, NULL, INTR_ROOT_XEN);
+	if (i != 0)
+		return (i);

After the intr_pic_register() call. This is somewhat hack-ish as we're barely using INTRNG, but it does seem kind of appropriate as it does amount to a disconnected PIC.

Copy link
Member

@markjdb markjdb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just looked at intrng, not very much at Xen-specific bits yet.

sys/kern/subr_intr.c Show resolved Hide resolved
sys/kern/subr_intr.c Outdated Show resolved Hide resolved
sys/kern/subr_intr.c Outdated Show resolved Hide resolved
sys/kern/subr_intr.c Outdated Show resolved Hide resolved
sys/kern/subr_intr.c Outdated Show resolved Hide resolved
@ehem
Copy link
Contributor Author

ehem commented Sep 23, 2024

I'm expecting the per-processor startup hook to change in light of other recent commits. Two issues with INTRNG remain though:

  1. Since Xen events are rather more dynamic than hardware interrupts, released events need some sort of handling. The obvious approach is intr_isrc_deregister(), problem is this hadn't previously been tested and it leaks the struct intr_event.
  2. Since they're a specialized type of interrupt, not all of INTRNG's services are needed. In particular the lower level interface of intr_add_handler() (rename isrc_add_handler()) and intr_describe() (break intr_describe_irq() apart) is rather superior.

ehem and others added 29 commits January 23, 2025 09:54
This distributes the event channels among processors instead of placing
all of them on vCPU#0.  Normal interrupt sources are balanced once, at
the end of the boot process.  Since new event channels can be created
any time, they need to be dynamically balanced.

Differential Revision: https://reviews.freebsd.org/D31690
The isrc allocation has been moved to architecture code where
the decision to allocate or not can be made.

This means setting of several fields needs to be handled by architecture
code.  Modify the prototype of xen_arch_intr_alloc(), start passing the
variables needed, and finally set them on aarch64.

This also means releasing of isrcs also needs to be moved to
architecture code.

Differential Revision: https://reviews.freebsd.org/D31063
Fallout from abc7a4a.  Apparently nowhere else in FreeBSD was the
PAGE_MASK_4K macro used.  Though perhaps few places were using the
explicitly sized PAGE_* macros.
New implementation of the check.  While Julien Grall's implementation
had the check, it was completely removed during bring-up for expedience.
This has been rebased on top to recognize the original also had the test
even though its action no longer works.
Nominally the function should work for this case too, just it hasn't
been tested in this situation.
With the variables now in xen_common.c, this file no longer needs to
define them.  Nuke this remnant.
This is early enough for the setup.  No barrier is needed, we merely
need invocation on every processor during bring-up.
Some previous builds appeared to potentially need the simplebus binding,
but this has been shown to be unnecessary.  Only use ofwbus since that is
how Xen/ARM passes the hypervisor information.

This is known to work once based on top of INTRNG, does this actually
work when on top of the kernel events?
Initial implementation.  Best to replace the existing file Julien Grall
originally created.  I /think/ this is what these are supposed to do...
Alas, ARM declared xen_ulong_t to be 64-bits long, unlike i386 where
it matches the word size.  As a result, compatibility wrappers are
needed for Xen atomic operations.
The step of enabling the setup in general is better a separate commit.

This was broken off of "xen/arm64: add xen platform".

Submitted by: Elliott Mitchell <[email protected]>
Original implementation: Julien Grall <[email protected]>, 2014-01-13 17:40:58
Differential Revision: https://reviews.freebsd.org/D30950
Presently the probing mechanism for Xen requires device-trees, but no
other portion requires FDT.  As a result marking everything as depending
upon FDT would be wrong, but this requires excluding Xen from LINT-ACPI.

Add a similar section to LINT-FDT since it will likely be needed in the
future.
Initial implementation.  Best to replace the existing file Julien Grall
originally created.  I /think/ this is what these are supposed to do...
Should handle the interface for rebinding interrupts.  May well produce
error returns though.  Based on GICv3.
This /looks/ plausible based on examples.
Seems using the proper interface adds a colon for us.  Adjust to
match what we should look like.
When this is called there /should/ be a source, right?  (hmm, that may
be backwards...)
The interface considers was previously willing to call bind without
the event having been set.  This is now impossible, but keep this as
an assert.

This fixes the previous commit.  Appears subsequent adjustments omitted
handling this.
Several functions have turned back into extern, now onto implementation.
Rather than looking at the event, INTRNG chooses to duplicate this data.
As such need to update the information here too.
The use of these has completely disappeared, so nuke them.
Hand regions provided by Xen for grant-table and other allocations.
This was originally intended for grant-table mappings, but Xen allows
them to be used for anything.  Xen guarantees these regions will be
empty, and the device-tree code prevents them from being used for other
purposes.
Despite the documentation being unclear, this *does* need to be
initialized.
Now that we can add the handler and describe via INTRNG, do so.  This
might also fix balancing during start.
Appears INTRNG wants the PIC to choose the processor, not suggesting
a processor?  Seems odd.  Try this.
Many thanks to Julien Grall who did the initial work in 2014-2015.  Now
we can have FreeBSD on Xen on ARM64 machines.  Want accelerated graphics
on a Raspberry PI?

Differential Revision: https://reviews.freebsd.org/D31956
ARM64 has been completed.  A great deal of the recent work was done by
royger.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants