"too many executable mappings" error

[PietroPasotti]

Describe the bug
I am running locally parca and parca-agent and I see a stack trace in parca and lots of error logs in parca-agent. I can't find traces for the python processes I'm trying to analyze (I searched by pid).
I googled the error messages but couldn't find anything useful. Can you help me figure out what's going on?

To Reproduce
run parca, parca-agent.

# parca.yaml config:
 object_storage:                                      
   bucket:                                            
     type: "FILESYSTEM"                               
     config:                                          
       directory: "./parca_data"                      
                                                                                                            
 scrape_configs:                                      
  - job_name: "parca"                                 
    scrape_interval: "10s"                            
    static_configs:                                   
      - targets: [ '127.0.0.1:7070' ]                 

run any python process (tried: ipython 8.5.0, pytest (local project, python 3.11.8)).

Expected behavior
I'd expect to see profiles for the python programs running on the host.

Screenshots or Profiles (using https://pprof.me)
n/a

Logs
From parca: (repeated multiple times)

recovered stack trace:                                                                                                                                                                         
 goroutine 232 [running]:                                                                                                                                                                      
runtime/debug.Stack()                                                                                                                                                                          
        runtime/debug/stack.go:24 +0x5e                                                                                                                                                        
github.com/parca-dev/parca/pkg/symbol/addr2line.(*DwarfLiner).PCToLines.func1()                                                                                                                
        github.com/parca-dev/parca/pkg/symbol/addr2line/dwarf.go:107 +0x45                                                                                                                     
panic({0x4001d60?, 0xc056084270?})                                                                                                                                                             
        runtime/panic.go:914 +0x21f                                                                                                                                                            
github.com/parca-dev/parca/pkg/symbol/elfutils.(*debugInfoFile).SourceLines(0xc0284ab170, 0x243bc)                                                                                             
        github.com/parca-dev/parca/pkg/symbol/elfutils/debuginfofile.go:120 +0x7a8                                                                                                             
github.com/parca-dev/parca/pkg/symbol/addr2line.(*DwarfLiner).PCToLines(0xc051b00500?, 0xc056027680?)                                                                                          
        github.com/parca-dev/parca/pkg/symbol/addr2line/dwarf.go:112 +0x6c                                                                                                                     
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).pcToLines(0xc0016dfad0, {0x60b65a0, 0xc02a826d40}, {0xc0551d3050, 0x28}, 0x0?)                                                         
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:574 +0x53                                                                                                                      
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).symbolizeLocationsForMapping(0xc0016dfad0, {0x60b5bf0, 0xc00185f890}, 0xc051ee3880, {0xc051ee6de0, 0x3, 0xc051ee3680?})                
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:517 +0x1afb                                                                                                                    
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Symbolize(0xc0016dfad0, {0x60b5bf0, 0xc00185f890}, {0xc000efe600, 0x32f, 0x7?})                                                        
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:297 +0x971                                                                                                                     
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).runSymbolizationCycle(0xc0016dfad0, {0x60b5bf0, 0xc00185f890})                                                                         
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:221 +0x152                                                                                                                     
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run.func1()                                                                                                                            
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:190 +0xf1                                                                                                                      
github.com/parca-dev/parca/pkg/runutil.Repeat(0xc001222040?, 0xc0001822a0, 0xc007e15dd8)                                                                                                       
        github.com/parca-dev/parca/pkg/runutil/runutil.go:38 +0x82                                                                                                                             
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run(0xc0016dfad0, {0x60b5bf0, 0xc00185f890}, 0x3f87f00?)                                                                               
        github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:188 +0x73                                                                                                                      
github.com/parca-dev/parca/pkg/parca.Run.func5.1({0x60b5bf0?, 0xc00185f890?})                                                                                                                  
        github.com/parca-dev/parca/pkg/parca/parca.go:531 +0x3d                                                                                                                                
runtime/pprof.Do({0x60b5c28?, 0xc0012c4e60?}, {{0xc001222040?, 0x0?, 0x0?}}, 0xc00187af40)                                                                                                     
        runtime/pprof/runtime.go:51 +0x9d                                                                                                                                                      
github.com/parca-dev/parca/pkg/parca.Run.func5()                                                                                                                                               
        github.com/parca-dev/parca/pkg/parca/parca.go:530 +0x1ef                                                                                                                               
github.com/oklog/run.(*Group).Run.func1({0xc0013374c0?, 0xc0013374e0?})                                                                                                                        
        github.com/oklog/[email protected]/group.go:38 +0x29                                                                                                                                          
created by github.com/oklog/run.(*Group).Run in goroutine 1                                                                                                                                    
        github.com/oklog/[email protected]/group.go:37 +0x67                                                                                                     

From parca-agent: (repeated many times)

level=warn name=parca-agent ts=2024-04-04T12:48:18.16188908Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"            
level=warn name=parca-agent ts=2024-04-04T12:48:19.081906622Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:21.172132294Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:21.173922614Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:22.5785825Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"             
level=warn name=parca-agent ts=2024-04-04T12:48:22.879245819Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:24.995136311Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=error name=parca-agent ts=2024-04-04T12:48:25.245026903Z caller=maps.go:1279 component=cpu_profiler component=bpf_maps msg="addUnwindTableForProcess failed" err="setUnwindTableForMapping for executable /home/me/lib/python3.10/site-packages/pydantic_core/_pydantic_core.cpython-310-x86_64-linux-gnu.so starting at 0x7a940524b000 failed: not enough profiling rounds with this unwind info"                                      
level=warn name=parca-agent ts=2024-04-04T12:48:25.991695868Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:25.995384796Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:26.493269298Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           
level=warn name=parca-agent ts=2024-04-04T12:48:28.751652154Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"           

Software (please complete the following information):
parca, version 0.21.0 (commit: 089e78068a1c4f8773b1d5a8bac6645c6414fdfd)
parca-agent, version 0.30.0 (commit: da59a7c, date: 2024-03-04T20:55:59Z), arch: amd64

Workload (please complete the following information):
pytest ./some-local-project

Environment (please complete the following information):
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"

• 6.5.0-26-generic
• x86_64

[brancz]

I opened #2675 to improve the error message so we understand better how many mappings would be enough.

I think the best resolution for this would be if we could make the max number of mappings configurable.

Regarding the symbolization, we actually just started revamping the symbolizer.