index: add VPP repository

[nplanelcisco]

FD.io's Vector Packet Processor (VPP) is a fast, scalable layer 2-4 multi-platform network stack. It runs in Linux Userspace on multiple architectures including x86, ARM, and Power architectures.

VPP's high performance network stack is quickly becoming the network stack of choice for applications around the world.

[tleb]

Hi and thanks for your contribution @nplanelcisco! We don't use VPP at Bootlin but we already index DPDK that lives in the same space. We aren't against adding VPP if it is helpful to others.

I'd have (a few) questions:

• Have you tested your contribution? I am in particular surprised there is nothing to be done about the project's tags. It is possible it matches the default expected value, but most projects don't. See the projects/ folder where most projects override the functions listing tags.
• VPP has 7.5K lines of Go that won't be indexed. Is it important?
• VPP has 6.2K lines of CMake that won't be indexed. Is is important?
• VPP has 3.7K lines of Makefiles that we could index. Is it a voluntary choice to not index it? See elixir/filters/projects.py.
• Grepping for #include <, I see many matches to files that look like part of VPP. For those you might want to add CppPathIncFilter inside the projects' filters. See elixir/filters/projects.py and elixir/filters/cpppathinc.py.
• Is the code for VPP useful by itself? Some projects are split across repos and many functions used are not located here. I checked that files referenced through #include live in the repo and they all appear to be, so that is a good sign that the project isn't dependent on external code.

Thanks!

[nplanelcisco]

• Have you tested your contribution? I am in particular surprised there is nothing to be done about the project's tags. It is possible it matches the default expected value, but most projects don't. See the projects/ folder where most projects override the functions listing tags.

Yes I a quick test locally, thanks to the index tool
We have the same tags model that DPDK project, so it's works out of the box :)

• VPP has 7.5K lines of Go that won't be indexed. Is it important?

It's not mandatory it's in the extras/ directory

• VPP has 6.2K lines of CMake that won't be indexed. Is is important?

It's not mandatory, would be a plus as our build chain is based on cmake, meson and ninja

• VPP has 3.7K lines of Makefiles that we could index. Is it a voluntary choice to not index it? See elixir/filters/projects.py.

Thanks I will test locally and update this PR

• Grepping for #include <, I see many matches to files that look like part of VPP. For those you might want to add CppPathIncFilter inside the projects' filters. See elixir/filters/projects.py and elixir/filters/cpppathinc.py.

Thanks I will test locally and update this PR

• Is the code for VPP useful by itself? Some projects are split across repos and many functions used are not located here. I checked that files referenced through #include live in the repo and they all appear to be, so that is a good sign that the project isn't dependent on external code.

Most of the VPP code is useful by itself as it introduce vectoring principle with different software layers like vpp infra, vlib, vnet.
Please take a look at the documentation https://s3-docs.fd.io/vpp/25.06/developer/corearchitecture/softwarearchitecture.html

One our plugin (DPDK) have a dependency on DPDK that you host already.
(totally optional) Would be possible to cross link project ? based on a Makefile value ? (build/external/packages/dpdk.mk:dpdk_version ?= 24.11.1)

[tleb]

Thanks for answering exhaustively.

---

One our plugin (DPDK) have a dependency on DPDK that you host already. (totally optional) Would be possible to cross link project ? based on a Makefile value ? (build/external/packages/dpdk.mk:dpdk_version ?= 24.11.1)

My initial thought was "oh no". Actually, if it is really useful, then we could possibly do it. Here is a brain dump of what it would look like:

• Query.get_tokenized_file() (that's part of Refactor query #397, which is what I have checked out locally) should be updated to check either the project's database or fallback projects. That's the code that goes over tokens inside files and checks if they exist as definitions.
• Currently this is pretty messy, the token string is replaced by b'\033[31m' + token + b'\033[0m'. We can keep the current string-all-the-way approach and put more info inbetween the two escape codes. Or we could rework the system to pass more semantic data, eg a list of tokens: raw string, identifier, etc. Then the identifier token would
• Then it is IdentFilter that picks up the escape codes and should be updated to call ctx.get_ident_url() with a project argument.
• Some project-specific code should tell us what the fallback projects' versions are. In the case of VPP it would find the value you highlighted (and check a matching version exists in the DPDK repo). It should go somewhere inside the database to avoid finding that version number on each request.
• Some additional checks are required here and there. For example nothing tells us every Elixir instance hosting VPP are also hosting DPDK. The intuitive answer is "it should fail gracefully". Maybe a warning makes sense?

That's doable, but it is really low in our priority list. If someone wants to invest some of their time working on that feature, we can talk.

[nplanelcisco]

@tleb thanks for the advice, I will take a look in another PR

Let's merge this one if you are ok, as it's do 99% of the jobs with referencing internal includes thanks to cpppathinc filter.
I modified a bit to add some search paths

[nplanelcisco]

@tleb ping

[fstachura]

I tested this patch locally and I don't have much more to add. You may want to replace ctx.query.query('exist' with ctx.query.file_exists( on rebase. @tleb Any other comments?

[nplanelcisco]

I tested this patch locally and I don't have much more to add. You may want to replace ctx.query.query('exist' with ctx.query.file_exists( on rebase. @tleb Any other comments?

@fstachura Query class doesn't have the file_exists() method (even in prod branch) or I missing something ?

[fstachura]

@nplanelcisco Query class refactor was merged some time ago
https://github.com/bootlin/elixir/blob/master/elixir/query.py#L90

[nplanelcisco]

@nplanelcisco Query class refactor was merged some time ago https://github.com/bootlin/elixir/blob/master/elixir/query.py#L90

thanks, sorry for the noise