Failing to discover Benchmarks when package root path is not named Benchmarks.

[vanvoorden]

https://github.com/ordo-one/package-benchmark/blob/main/Plugins/BenchmarkCommandPlugin/BenchmarkCommandPlugin.swift#L310-L321

I seem to be running into situations where a Benchmarks package that should build with valid benchmarks fails to find any. I seem to be running into this code which filters Benchmarks by removing the last file path component.

The trouble seems to be when my Benchmarks.swift is already defined in [Root]/Benchmarks (where [Root] is the root directory of my swift package). At that point, the code in BenchmarkCommandPlugin.swift seems to be removing the leaf Benchmarks and only continuing if the root of the swift package is equal to "Benchmarks".

I am unblocked with a workaround: I can define a subdirectory in Benchmarks. At that point, I have something like [root]/Benchmarks/Benchmarks. The BenchmarkCommandPlugin.swift removes the leaf directory and continues running my benchmarks.

[hassila]

I assume you have a separate Package.swift for the Benchmarks subfolder?

In that case it would be expected, The source for all benchmarks must reside in a directory named Benchmarks in the root of your swift package. "The Swift package" in this context is the sub-package.swift you have - originally benchmarks were expected to be defined in the top level package, but since then most people have ended up having a separate "subproject" folder that happens to be named Benchmarks too.

Add benchmark exectuable targets manually
Optionally if you don’t want the plugin to modify your project for you, you can do those steps manually.
First create an executable target in Package.swift for each benchmark suite you want to measure.
The source for all benchmarks must reside in a directory named Benchmarks in the root of your swift package.
The benchmark plugin uses this directory combined with the executable target information to automatically discover and run your benchmarks.

Is this what's happening, or do you have some repo I can look at?

[vanvoorden]

https://github.com/vanvoorden/2024-08-28

@hassila Here is an example of a directory structure to repro what I was seeing:

2024-08-28
├── Benchmarks
│   └── Benchmarks.swift
└── Package.swift

Here I have a "root" folder (2024-08-28) and a Benchmarks folder with one Swift file. This swift file defines one empty benchmark. The package manifest specifies Benchmarks as the path to my executableTarget. When I run this from the command line:

swift package benchmark list

I see no benchmarks listed. When I then move my Swift file into an arbitrary subdirectory like this (and specify the path of my executableTarget to include the new subdirectory):

2024-08-28
├── Benchmarks
│   └── Benchmarks
│       └── Benchmarks.swift
├── Package.resolved
└── Package.swift

Now I see the Benchmark when I run list.

I am unblocked with this workaround for now. I don't think this needs to be a high-priority fix… but I ran in to this problem a few times when trying to set up benchmarks before I found out the pattern that was causing these benchmarks to go missing.

[vanvoorden]

The source for all benchmarks must reside in a directory named Benchmarks in the root of your swift package.

I think there might be an opportunity here to maybe clarify the documentation. While it is true that all benchmarks must reside in a Benchmarks directory, the package also (currently) makes some assumptions about the directory structure inside that Benchmarks directory and engineers might not be prepared for that.

For example, this version fails to detect any valid benchmarks:

2024-08-28
├── Benchmarks
│   └── Benchmarks.swift
├── Package.resolved
└── Package.swift

After setting the path of my executableTarget to Benchmarks, I am unable to run swift package benchmark list to then detect the one benchmark defined in Benchmarks.swift.

This version does detect valid benchmarks:

2024-08-28
├── Benchmarks
│   └── Root
│       └── Benchmarks.swift
├── Package.resolved
└── Package.swift

After setting the path of my executableTarget to Benchmarks/Root, I can run swift package benchmark list and I detect the one benchmark defined in Benchmarks.swift.

This version does not detect valid benchmarks:

2024-08-28
├── Benchmarks
│   └── Root
│       └── Leaf
│           └── Benchmarks.swift
├── Package.resolved
└── Package.swift

After setting the path of my executableTarget to Benchmarks/Root/Leaf, I am unable to run swift package benchmark list to then detect the one benchmark defined in Benchmarks.swift.

This seems to be an artifact of the algorithm in BenchmarkCommandPlugin.swift. That algorithm removes the last component from the path of the executable target and checks if the next component is equal to Benchmarks.

I am unblocked on this because it is trivial for me to shuffle around the files and directories in my package… but there might be an opportunity here to expand the documentation with more details about this behavior so that new engineers trying to get started with Benchmarks do not get blocked on this situation where benchmarks are "disappearing" and not being detected.