English | 中文
Reporting runtime metrics every minutes
import the package into your service code:
import _ "trpc.group/trpc-go/trpc-metrics-runtime"The runtime of Go can be considered as the infrastructure required for Go to run, primarily consisting of memory allocation, garbage collection, corouting scheduling, encapsulation of differences between operating systems and CPUs, pprof support, implementation of built-in types and reflection, etc.
Go runtime package provides functions to inspect the state of the runtime itself (such as runtime.ReadMemStats, runtime.GOMAXPROCS). This library will report this data periodically.
Here is a brief introduction to the meanings of these metrics.
Note that these may not completely solve runtime-related issues. You can also use gctrace or pprof to troubleshoot.
| Metric | Description | Abnormal Judgment |
|---|---|---|
| GoroutineNum | Number of Goroutine | This is generally related to the request count and service consumption time. It is normal for a service with a QPS of several thousand to have 1000 or 2000 goroutines. The number of goroutines will not decrease after it increases. As long as it is not too many (such as over 10,000), there is no need to pay too much attention to it |
| ThreadNum | Number of M in Go runtime(It's roughly the number of threads, exclude the threads start in C code) | Generally, the number of cores in a machine is somewhat related to it, and it is normal to have 10-50 cores. If blocking cgo is used, there may be more cores involved. Based on the load, it should not exceed 100 |
| GOMAXPROCSNum | The maximum parallelism (not concurrency) of the code in Go | Generally, it is equal to the number of machine cores recognized by Go. In containers, the Uber's automaxprocs library should be used to correctly set quotas, otherwise many exceptions may occur |
| CPUCoreNum | The number of logical CPUs usable by the current Go process (Actually, it is setting CPU affinity for processes, see sched_getaffinity) | None |
| PauseNsLt100usTimes | The number of times the pause time <100us in the past 256 GC cycles | This cannot determine whether GC is frequent. It only shows that within the recent 256 GC data, the majority is less than 500us, and occasionally exceeding 1ms is normal. If there are a lot of data exceeding 10ms, attention should be paid. |
| PauseNs100_500usTimes | The number of times the pause time between 100-500us in the past 256 GC cycles | Same as above |
| PauseNs500us_1msTimes | The number of times the pause time between 500us-1ms in the past 256 GC cycles | Same as above |
| PauseNs1_10msTimes | The number of times the pause time between 1-10ms in the past 256 GC cycles | Same as above |
| PauseNs10_50msTimes | The number of times the pause time between 10-50ms in the past 256 GC cycles | Same as above |
| PauseNs50_100msTimes | The number of times the pause time between 50-100ms in the past 256 GC cycles | Same as above |
| PauseNs100_500msTimes | The number of times the pause time between 100-500ms in the past 256 GC cycles | Same as above |
| PauseNs500ms_1sTimes | The number of times the pause time between 500ms-1s in the past 256 GC cycles | Same as above |
| PauseNsBt1sTimes | The number of times the pause time >1s in the past 256 GC cycles | Same as above |
| AllocMem_MB | The number of bytes of objects allocated in the GC heap from the current GC cycle until now | This depends on the progress of memory allocation, and in general, it is not significant |
| SysMem_MB | The memory allocated by Go from system, including heap, stack, and memory used by some structures in runtime | It can be considered as the amount of virtual memory used by the Go process |
| NextGCMem_MB | The target heap size of the next GC cycle | When AllocMem_MB is approximately less than NextGCMem_MB, the next GC process will begin. The GC process is expected to end when it reaches approximately equal to NextGCMem_MB. Generally speaking, this refers to the memory occupied during GC |
| PauseNs_us | Total time of historical GC pause | |
| GCCPUFraction_ppb | The fraction of this program's available CPU time used by the GC since the program started | It is not significant. It may have a very small value as it's averaged out. Normal service rates are 0% or 1%. If it's around 5%, then it should be considered if there's an issue |
| MaxFdNum | Maximum number of allowed file descriptors for a process (Unix only) | |
| CurrentFdNum | The number of file descriptors currently opened by the process (Unix only) | |
| PidNum | The number of processes currently running in the machine/container (Unix only) | |
| TcpNum | Total number of protocol socket descriptors in use (including allocated and pending closure) by the system (Unix only) | Generally, the number of connections in a container is relatively large (such as tens of thousands), which is not significant for most businesses. However, it can be a reference for services that require a massive amount of connections (such as hundreds of thousands) |
| DiskFree | Disk available in machine/root directory (Unix only, in GB) | None |
| DiskUsed | Disk usage of machine/root directory (Unix only, in GB) | None |
| DiskUsedFraction | Disk usage fraction of machine/root directory (Unix only) | None |