GATEWAYS-4306: exporting metrics for conntrack per zone #137

shrouti1995 · 2025-08-13T08:58:12Z

This PR implements a new connection tracking monitoring system that leverages netlink to directly access kernel conntrack data. The implementation provides zone-based monitoring capabilities, allowing for more granular network traffic analysis

Tested with 1 M and 2M conntracks created in s2r7node11.

armando-migliaccio · 2025-08-13T10:11:33Z

_ No description provided. _

Please provide a descriptive commit message.

shrouti1995 · 2025-08-13T10:38:33Z

_ No description provided. _

Please provide a descriptive commit message.

sure! for now updated the description

Copilot

Pull Request Overview

This PR implements a new connection tracking monitoring system that leverages netlink to directly access kernel conntrack data. The system provides zone-based monitoring capabilities for granular network traffic analysis and DDoS detection.

Adds a new ConntrackService that uses netlink to query kernel conntrack entries
Integrates the conntrack service into the main OVS client with proper lifecycle management
Updates dependencies to support the new conntrack functionality

Reviewed Changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 5 comments.

File	Description
ovsnl/conntrack.go	New service implementing conntrack entry retrieval and conversion from kernel data
ovsnl/client.go	Integration of ConntrackService into main client with initialization and cleanup
go.mod	Dependency updates including ti-mo/conntrack library and Go version upgrade

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-08-13T10:40:34Z

ovsnl/conntrack.go

+	"syscall"
+
+	"github.com/digitalocean/go-openvswitch/ovsnl/internal/ovsh"
+	"github.com/ti-mo/conntrack"


The conntrack import is added but there's no corresponding entry in the go.mod require section. The ti-mo/conntrack dependency should be explicitly listed in the require section rather than just indirect.

Copilot · 2025-08-13T10:40:34Z

ovsnl/conntrack.go

+
+// List lists all conntrack entries from the kernel.
+// datapathName is not used in this direct Netlink query, as it's a global dump.
+// List lists all conntrack entries from the kernel.


This comment is duplicated on lines 94 and 95. The duplicate comment about datapathName on line 94 should be removed as it's incomplete and confusing.

Copilot · 2025-08-13T10:40:35Z

ovsnl/conntrack.go

+
+	// Handle TCP state specifically
+	if f.TupleOrig.Proto.Protocol == unix.IPPROTO_TCP {
+		entry.State = parseConntrackStateFlags(uint32(f.ProtoInfo.TCP.State))


The TCP state parsing only handles TCP protocol, but other protocols like UDP and ICMP may also have state information. Consider adding state parsing for other protocols or documenting why only TCP state is handled.

Suggested change

entry.State = parseConntrackStateFlags(uint32(f.ProtoInfo.TCP.State))

// Handle protocol-specific state.

switch f.TupleOrig.Proto.Protocol {

case unix.IPPROTO_TCP:

entry.State = parseConntrackStateFlags(uint32(f.ProtoInfo.TCP.State))

case unix.IPPROTO_UDP:

// UDP state is less detailed, but we can indicate if it's "UNREPLIED" or "REPLIED".

// See https://www.netfilter.org/projects/conntrack-tools/manpage.html

if f.ProtoInfo.UDP.State == 1 {

entry.State = "UNREPLIED"

} else if f.ProtoInfo.UDP.State == 2 {

entry.State = "REPLIED"

} else {

entry.State = "UNKNOWN"

}

case unix.IPPROTO_ICMP, unix.IPPROTO_ICMPV6:

// ICMP state is not tracked in the same way, but we can indicate type/code.

entry.State = fmt.Sprintf("TYPE_%d_CODE_%d", f.TupleOrig.Proto.ICMPType, f.TupleOrig.Proto.ICMPCode)

default:

// For other protocols, state is not tracked or not relevant.

entry.State = "UNTRACKED"

ovsnl/client.go

Copilot · 2025-08-13T10:40:35Z

go.mod

+	github.com/mdlayher/ethtool v0.0.0-20210210192532-2b88debcdd43 // indirect
+	github.com/mdlayher/socket v0.5.1 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
+	github.com/ti-mo/conntrack v0.5.2 // indirect


The ti-mo/conntrack dependency is marked as indirect but it's directly imported in ovsnl/conntrack.go. This should be moved to the direct require section.

Suggested change

github.com/ti-mo/conntrack v0.5.2 // indirect

do-msingh · 2025-08-13T19:18:39Z

ovsnl/conntrack.go

+	// Start dump in goroutine
+	go func() {
+		defer close(flowChan)
+		flows, err := s.client.Dump(nil)


I am curious how well Dump scales, and whether you should be using DumpFilter or DumpExpect instead (or maybe even some new variant that just counts if needed)
How many entries did you scale to , in your test setup?

I did a POC, did not test yet on heavy traffic. Will check on DumpFilter or DumpExpect as well. Let me get back to you

You won't need heavy traffic. Just scale up to like a million or two Conntrack entries and see if it performs well.

updated the code. Checked with 1 million conntrack entries. It is working. The code needs a lot of cleanup. Just a heads up.

That's encouraging. While you are at it, could you try the max conntrack limit as well?

Looking at the snapshot from your scaled run, it does indicate the irq plateau for the duration of the run. I am assuming there were no CPU lockup messages from the kernel during this run, correct?

Did you get a chance to optimize the frequency of metrics collection?

On another note, this collection should be controlled with a config knob and we should slow roll this carefully.

Also cc @jcooperdo for another pair of eyes.

@do-msingh was working on the issue you caught from screenshot. It was not properly doing conntrack count refresh. It looks under control now. As of now I tested seeding conntracks to a specific droplet in a hyperviser. In this screenshot you will see only around 400K jump because for easy testing I kept the timeout 10 mins. But actually created 2.6 Million conntracks only. I will test scenarios like adding timeout for 1 hr, seeding conntracks to multiple droplets (10-20), and see how the system performs. Will keep posted here.

While testing with 1 hr timeout with 2.6M conntracks created against 1 droplet in a single zone. There are some small discrepencies due to processing delay: ( For example we have missed 12k events to count while running sync for 2.6M conntracks )
I can try to fix it later as an improvement task

tested with 3 droplets with three different zones each receiving 2.6 mililon events over 4 hrs.

did not see any oom kill error

in the same way when I tested creating conntracks without my changes in openvswitch_exporter the graph looks like above

shrouti1995 · 2025-09-08T13:52:40Z

The build is failing due to go version in my local machine. In this repository we are using older version. When the code is signed off I will try to install the older version and push it. Kept it like this for now.

do-msingh · 2025-09-08T22:56:52Z

ovsnl/conntrack.go

+
+// NewZoneMarkAggregator creates a new aggregator with its own listening connection.
+func NewZoneMarkAggregator(s *ConntrackService) (*ZoneMarkAggregator, error) {
+	log.Printf("Creating new conntrack zone mark aggregator...")


Could you remove these logs?

do-msingh · 2025-09-08T22:57:03Z

ovsnl/conntrack.go

+		return nil, fmt.Errorf("failed to create listening connection: %w", err)
+	}
+
+	log.Printf("Successfully created conntrack listening connection")


same as above

do-msingh · 2025-09-09T17:32:15Z

ovsnl/conntrack.go

+	// Start dump in goroutine
+	go func() {
+		defer close(flowChan)
+		flows, err := s.client.Dump(nil)


This script would be nice to integrate with chef and maybe export some metrics using node exporter so we can build some dashboards around it. In your tests, could you run at scale for an extended period like a couple of hours and check average CPU util? Do you only see CPU spikes around the time metrics are collected? For how long? Also @jcooper had a suggestion to reduce the frequency of collecting the metrics, or maybe optimizing it to reduce load.
Lastly, can you check dmesg output as well at scale to make sure we are not missing anything?

jcooperdo · 2025-09-29T23:36:59Z

go.mod

 module github.com/digitalocean/go-openvswitch

-go 1.16
+go 1.23.0


The build is failing due to go version in my local machine. In this repository we are using older version. When the code is signed off I will try to install the older version and push it. Kept it like this for now.

It's probably time we bump this. Lets try to get tests to pass with a recent version.

jcooperdo · 2025-09-29T23:59:47Z

ovsnl/client.go

 package ovsnl

 import (
+	"context" // Used in commented aggregator code


nit: I'm not seeing this used anywhere, can it be removed?

jcooperdo · 2025-09-30T13:50:56Z

ovsnl/conntrack.go

+}
+
+// ForceSync performs a manual sync (disabled for large tables)
+func (a *ZoneMarkAggregator) ForceSync() error {


This method doesn't appear to be used, is it needed?

removed it.

jcooperdo · 2025-09-30T13:55:12Z

ovsnl/conntrack.go

+}
+
+// IsHealthy checks if the aggregator is in a healthy state
+func (a *ZoneMarkAggregator) IsHealthy() bool {


This method doesn't appear to be used, is it needed?

removed it. missed cleaning it up. Added this for debugging purpose.

jcooperdo · 2025-09-30T14:14:49Z

ovsnl/conntrack.go

+	CPUs               int
+}
+
+// ConntrackService manages the connection to the kernel's conntrack via Netlink.


How does ConntrackService do what the comment suggests? Its only references I can find are no-op constructors/closers.

ovsnl/conntrack.go

jcooperdo · 2025-09-30T14:26:47Z

ovsnl/conntrack.go

+	// primary counts (zone -> mark -> count)
+	mu     sync.RWMutex
+	counts map[uint16]map[uint32]int


Do we gain any benefit from mapping by zone->mark? Could we instead simplify this by mapping by zmKey?

jcooperdo · 2025-09-30T14:34:18Z

ovsnl/conntrack.go

+	return out
+}
+
+// GetTotalCount returns the total counted entries (best-effort)


Is this used by anything? Would it return the same as nf_conntrack_count?

removed it. missed cleaning it up. Added this for debugging purpose.

Co-authored-by: jcooperdo <[email protected]>

* test 1 * test 2 * test 2 * test 3 * test 4 * test 5 * test 6 * test 7 * test 8 * test 9 * test 10 * test 12 * test 13 * test 14 * test 15 * test 16 * Update go.yml * clean up

GATEWAYS-4306: exporting metrics for conntrack per zone

e306589

shrouti1995 requested a review from Copilot August 13, 2025 10:38

Copilot AI reviewed Aug 13, 2025

View reviewed changes

do-msingh reviewed Aug 13, 2025

View reviewed changes

GATEWAYS-4306: scaling with event driven approach

80916dc

shrouti1995 requested a review from do-msingh August 28, 2025 16:14

code cleanup

8b03af1

shrouti1995 marked this pull request as ready for review September 2, 2025 20:00

shrouti1995 added 3 commits September 8, 2025 13:36

GATEWAYS-4306: updating go.mod

e8eeddb

GATEWAYS-4306: updating test cases

dc73f17

GATEWAYS-4306: editing go.mod again

7b31316

do-msingh reviewed Sep 9, 2025

View reviewed changes

shrouti1995 added 3 commits September 22, 2025 20:43

conntrack destroy rate limiting part 1

02ce3ca

conntrack destroy rate limiting part 2

15e3b1a

configuration update

b1b102c

shrouti1995 requested review from do-msingh and jcooperdo September 26, 2025 11:41

configuration update

9796da5

jcooperdo reviewed Sep 29, 2025

View reviewed changes

jcooperdo reviewed Sep 30, 2025

View reviewed changes

ovsnl/conntrack.go Outdated Show resolved Hide resolved

jcooperdo reviewed Sep 30, 2025

View reviewed changes

review suggestions

df047c7

Co-authored-by: jcooperdo <[email protected]>

shrouti1995 added 14 commits October 6, 2025 15:18

review comments addressed

51dbc5b

fix: adding license

0f97bc6

editing dynamic strings

578db7f

go fmt check edits

33d1c71

go lint check edits

3d1d088

ci cd version upgrades

68f438c

clean go.mod

4809d11

using io instead if ioutil

ff11970

fixing build issue

5ac45d4

fixing build issue

64416ca

adding back toolchain

457550e

test cases issue solve

6b139de

* test 1 * test 2 * test 2 * test 3 * test 4 * test 5 * test 6 * test 7 * test 8 * test 9 * test 10 * test 12 * test 13 * test 14 * test 15 * test 16 * Update go.yml * clean up

simplify by mapping with zmKey

3f97fdb

solve lint issue

6b5e843

-		entry.State = parseConntrackStateFlags(uint32(f.ProtoInfo.TCP.State))
+	// Handle protocol-specific state.
+	switch f.TupleOrig.Proto.Protocol {
+	case unix.IPPROTO_TCP:
+		entry.State = parseConntrackStateFlags(uint32(f.ProtoInfo.TCP.State))
+	case unix.IPPROTO_UDP:
+		// UDP state is less detailed, but we can indicate if it's "UNREPLIED" or "REPLIED".
+		// See https://www.netfilter.org/projects/conntrack-tools/manpage.html
+		if f.ProtoInfo.UDP.State == 1 {
+			entry.State = "UNREPLIED"
+		} else if f.ProtoInfo.UDP.State == 2 {
+			entry.State = "REPLIED"
+		} else {
+			entry.State = "UNKNOWN"
+		}
+	case unix.IPPROTO_ICMP, unix.IPPROTO_ICMPV6:
+		// ICMP state is not tracked in the same way, but we can indicate type/code.
+		entry.State = fmt.Sprintf("TYPE_%d_CODE_%d", f.TupleOrig.Proto.ICMPType, f.TupleOrig.Proto.ICMPCode)
+	default:
+		// For other protocols, state is not tracked or not relevant.
+		entry.State = "UNTRACKED"

GATEWAYS-4306: exporting metrics for conntrack per zone #137

Are you sure you want to change the base?

GATEWAYS-4306: exporting metrics for conntrack per zone #137

Conversation

shrouti1995 commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

armando-migliaccio commented Aug 13, 2025

Uh oh!

shrouti1995 commented Aug 13, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrouti1995 Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrouti1995 Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrouti1995 Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrouti1995 Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrouti1995 commented Sep 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrouti1995 Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

shrouti1995 commented Aug 13, 2025 •

edited

Loading

shrouti1995 Sep 23, 2025 •

edited

Loading

shrouti1995 Sep 23, 2025 •

edited

Loading

shrouti1995 Sep 26, 2025 •

edited

Loading

shrouti1995 Sep 29, 2025 •

edited

Loading

shrouti1995 Oct 3, 2025 •

edited

Loading