Reading BMP messages from Kafka stream

[ACodingfreak]

Hi All,

As per my understanding bgpreader provides data_interface option to read from a particular topic in kafka stream and dump output.

While using pybgpstream, I can see data_interface option where I can provide Kafka, but I am not sure how I can share kafka topic and filter for the same? Any ideas ?

[alistairking]

Hey there,
Yeah, you should be able to use set_data_interface_option (https://bgpstream.caida.org/docs/api/pybgpstream/_pybgpstream.html#_pybgpstream.BGPStream.set_data_interface_option)
And pass the topic option (https://github.com/CAIDA/libbgpstream/blob/master/lib/datainterfaces/bsdi_kafka.c#L69-L119)

[ACodingfreak]

Thanks for the reply alistairking that worked.

I have one more question. When working with MRT dumps for BGP updates I can see all the updates from a collector in time sequential order but when I process it on Kafka stream, only bgp updates from a particular peer is Time sequential order but not from a single collector. Is this expected behavior ?

[alistairking]

Yeah. The topics in the RouteViews BMP stream are partitioned based on peer ASN.

[ACodingfreak]

Thanks for the info @alistairking

Where can I get more details regarding the routeviews bmp kafka stream partitions and topics supported
Currently I am only aware of the kafka stream as "stream.routeviews.org:9092" and topics as "routeviews.linx.6830.bmp_raw" taken from an example in BGPKIT.

If there are any other BMP KAFKA streams which can be used for live processing of BGP updates from RRC or routeviews please do share the same.

[alistairking]

the RV topics are all structured as routeviews.<collector>.<peer-asn>.bmp_raw.
you can get a sense of the collectors and peer ASNs here: http://www.routeviews.org/peers/peering-status.html (most though not all are available via kafka).

afaik there is not currently a kafka stream for RIS data.

[ACodingfreak]

Thanks for the info @alistairking