Nfpcap / nfdump: Usage of flow timeouts / time until new context is established

[MrMcNiki]

Hello everyone,

I try to use nfpcap and nfdump, to create from a .pcap micro-flows and use them in samples for ML DDoS detection. Since I would like to get as close to real-time data as possible, I would like to rebuild the flow context after the shortest possible time spans.

So It tried to acvieve that with the active and inactive timeouts. It does not what I expected:

nfdump/bin/nfpcapd -e 10,5 -t 10 -r Friday-WorkingHours.pcap -l out/

and nfdump/bin/nfdump -b -O tstart -t '2017/07/07.20:46:00'-'2017/07/07.21:26:00' -o csv -R out/ 'not inet6 and inet' >fridayDump.csv

creates flows, but the duration of the flows are as if I don't touch these paramters.

I got

 Source IP     sp  Destination IP       dp            Timestamp                   Flow Duration         ipkt         opkt  
0       192.168.10.5  55970  184.84.243.199     80.0  2017-07-07 20:46:01       145.022                19.0        17.0 

but I want something like:

 Source IP     sp  Destination IP       dp            Timestamp         Flow Duration  ipkt  opkt  
0       192.168.10.5  55970  184.84.243.199     80.0  2017-07-07 20:46:01       10  -  -
1       192.168.10.5  55970  184.84.243.199     80.0  2017-07-07 20:46:11       10  -  -

So in short: is there a possibility with this tools, to split the flow (entries) into more finely granular entries ?

(If I change the rotation value, it increases the number of packages by a corresponding factor, but still everything in one line).

Thanks for some advice/tips!

[phaag]

Hi,
Nfpcapd maintains a flow cache which is time based. Flows are exported (flushed) either to a file or a host if:

• The flow is terminated (TCP Fin)
• The flow is an ICMP flow.
• The flow is UDP port 53 DNS (single packet)
• The flow is idle for t > inactive timeout. (no new packets added to flow)
• The flows is active for t > active timeout. (continuous flow)

This means, the time set to rotate the file does not affect the flow cache. All flows flushed during the rotate period end up in the same file. Therefore you never will see a duration exactly the same as the rotation period, as each flow has its own timestamps. Even for continuous flows the first/last packet most likely never falls exactly on the rotation period frame.

Therefore, the smaller the active/inactive timeout is, the more granular the resulting flow. However, the smaller these timeouts are, the more load you put on the collector, as at each of these timeouts you need to expire the flow cache. If you only process caps, then this may work, on an active interface with lots of traffic, this is highly not recommended.

If you process pcaps, then you get the most granular flows by splitting the pcap in many individual files, each with a few packets only and then let nfpcapd process all these files. If a EOF of a pcap is reached the entire flow cache is flushed.

[MrMcNiki]

Thank you for the explanation.

I misunderstood the roation time.

To split the pcaps into smaller slices works the way I prefered - however, the function of the active timeout is still somewhat unclear to me:

Therefore, the smaller the active/inactive timeout is, the more granular the resulting flow.
If I change the acitve timeout to smaller values, nothing really happens and I get the same granular flows as with higher granularity (e.g. a flow with duration 145s and a 180s active timeout produce the same one flow entry like with a acitve timeout of 120s).

More fine granular would mean to me, that it would result in 2 flows: one with 120s and 1 with round about 25s duration.

Or does the timeout just releases storage at the node and the consecutive flows are joined?