Recommended server size to run Parca

[nabokihms]

Hello, team! I have a question.

After completing the Kubernetes deployment guide, I now have an installation of Parca. For now, there are no agents, only two profile scrape sources configured manually.

Yet there is a problem that the memory consumption is growing infinitely until the pod is OOMkilled.

https://pprof.me/4aae8e1/

So my question is: is everything OK? Do I need to give Parca more resources? Maybe I missed something.

[brancz]

Great question! Ultimately it really depends on your application, how many distinct stacks there are, how many and how long function names are, and so on, so this may be perfectly ok.

That said, I have a few recommendations:

• Update to v0.20.0 (which we just released an hour ago) which has a ton of memory improvements.
• I recommend dropping all memory profiling data except for memory heap. There are multiple reasons I say this:
  1. Memory in-use objects are rarely useful
  2. Memory-allocated objects and bytes are extremely expensive in their regular form, as they contain all stacks since the beginning of the process that have ever allocated memory, the longer a process is alive the more extreme the amounts of data tend to get.

To keep only memory in-use bytes use this configuration on your scrape config:

      - job_name: parca
        scrape_interval: "10s"
        profiling_config:
          pprof_config:
            memory:
              keep_sample_type:
              - type: inuse_space
                unit: bytes

And last but not least, if possible (and we understand it's not necessarily always possible), use Parca Agent for CPU profiling, and turn off the in-process profilers from scraping using:

      - job_name: parca
        scrape_interval: "10s"
        profiling_config:
          pprof_config:
            process_cpu:
              enabled: false

In addition to that, you can tweak the active in-memory size allowed by the storage using the --storage-active-memory flag, which defaults to 512mb (and in v0.20.0 it's much more closely respected but there are still lots of improvements that it's actually close to identical but it's a knob you can tune down).

[nabokihms]

Thank you for pieces of advice. I applied the first and the second.

• I only managed to find all possible options for the memory profile to filter in the code here. Is it worth to be added to the docs? Are you open to accepting contributions?

• RAM consumption looks better now. However, graphs still have spikes

About CPU profiles, I tried to enable the agent, but it consumed some resources as well. Is there an option to collect only profiles of specific services with the agent?

[nabokihms]

My desire is to have some predictable resource consumption. Using limits in Kubernetes is probably acceptable for this case, yet still, I have a feeling that there is a more resilient way to handle the problem.