Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Route-map: null when loading config during FRR restart #17552

Open
2 tasks done
karampok opened this issue Dec 3, 2024 · 10 comments
Open
2 tasks done

Route-map: null when loading config during FRR restart #17552

karampok opened this issue Dec 3, 2024 · 10 comments
Labels

Comments

@karampok
Copy link

karampok commented Dec 3, 2024

Description

In MetalLB we use FRR inside a kubernetes pod and in our CI we observe not very often a failure.
Upon a pod restart (and therefore an FRR restart), the FRR process is not sending out prefixes, and test fails.
What we observe is

2024/11/21 11:55:47 BGP: [H5AW4-JFYQC] Route-map: null, prefix: 192.168.2.0/24, result: deny
2024/11/21 11:55:47 BGP: [VERY4-P6JC8] 172.30.0.3(Unknown) [Update:SEND] 192.168.2.0/24 is filtered by route-map '172.30.0.3-out'

while the frr config clearly should allow.

The full logs

2024/11/21 11:55:12 WATCHFRR: [QDG3Y-BY5TN] zebra state -> up : connect succeeded
2024/11/21 11:55:12 WATCHFRR: [QDG3Y-BY5TN] mgmtd state -> up : connect succeeded
2024/11/21 11:55:12 WATCHFRR: [QDG3Y-BY5TN] staticd state -> up : connect succeeded
2024/11/21 11:55:12 WATCHFRR: [QDG3Y-BY5TN] bfdd state -> up : connect succeeded
2024/11/21 11:55:12 MGMTD: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:12 STATIC: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:12 BFD: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:12 ZEBRA: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15 WATCHFRR: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15.188 ZEBRA: [V4EEB-GQBPV] Add route-map 172.30.0.3-out
2024/11/21 11:55:15.188 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.3-out add sequence 1, type: permit
2024/11/21 11:55:15.189 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.3-pl-ipv4 in route-map 172.30.0.3-out
2024/11/21 11:55:15.189 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.3-pl-ipv4: 172.30.0.3-out
2024/11/21 11:55:15.191 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.3-out add sequence 2, type: permit
2024/11/21 11:55:15.193 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.3-pl-ipv4 in route-map 172.30.0.3-out
2024/11/21 11:55:15.193 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.3-pl-ipv4: 172.30.0.3-out
2024/11/21 11:55:15.194 ZEBRA: [RWPWC-9TYFK] Filter 172.30.0.3-pl-ipv4 updated
2024/11/21 11:55:15.194 ZEBRA: [PQ39Y-44VF4] Notifying 172.30.0.3-out of dependency
2024/11/21 11:55:15.200 ZEBRA: [V4EEB-GQBPV] Add route-map 172.30.0.3-in
2024/11/21 11:55:15.200 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.3-in add sequence 3, type: deny
2024/11/21 11:55:15.202 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.3-denied-inpl-ipv4 in route-map 172.30.0.3-in
2024/11/21 11:55:15.202 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.3-denied-inpl-ipv4: 172.30.0.3-in
2024/11/21 11:55:15.204 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.3-in add sequence 4, type: deny
2024/11/21 11:55:15.207 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.3-denied-inpl-ipv4 in route-map 172.30.0.3-in
2024/11/21 11:55:15.207 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.3-denied-inpl-ipv4: 172.30.0.3-in
2024/11/21 11:55:15.216 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.3-in add sequence 5, type: permit
2024/11/21 11:55:15.217 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.3-inpl-ipv4 in route-map 172.30.0.3-in
2024/11/21 11:55:15.217 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.3-inpl-ipv4: 172.30.0.3-in
2024/11/21 11:55:15.218 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.3-in add sequence 6, type: permit
2024/11/21 11:55:15.220 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.3-inpl-ipv4 in route-map 172.30.0.3-in
2024/11/21 11:55:15.220 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.3-inpl-ipv4: 172.30.0.3-in
2024/11/21 11:55:15.224 ZEBRA: [V4EEB-GQBPV] Add route-map 172.30.0.2-out
2024/11/21 11:55:15.224 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.2-out add sequence 1, type: permit
2024/11/21 11:55:15.247 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.2-pl-ipv4 in route-map 172.30.0.2-out
2024/11/21 11:55:15.247 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.2-pl-ipv4: 172.30.0.2-out
2024/11/21 11:55:15.251 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.2-out add sequence 2, type: permit
2024/11/21 11:55:15.252 MGMTD: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15.252 MGMTD: [G6NKK-8C6DV] end_config: VTY:0x7f307d1df2c0, pending SET-CFG: 0
2024/11/21 11:55:15.264 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.2-pl-ipv4 in route-map 172.30.0.2-out
2024/11/21 11:55:15.264 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.2-pl-ipv4: 172.30.0.2-out
2024/11/21 11:55:15.266 ZEBRA: [RWPWC-9TYFK] Filter 172.30.0.2-pl-ipv4 updated
2024/11/21 11:55:15.266 ZEBRA: [PQ39Y-44VF4] Notifying 172.30.0.2-out of dependency
2024/11/21 11:55:15.272 STATIC: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15.272 STATIC: [G6NKK-8C6DV] end_config: VTY:0x7fe730a29360, pending SET-CFG: 0
2024/11/21 11:55:15.273 ZEBRA: [V4EEB-GQBPV] Add route-map 172.30.0.2-in
2024/11/21 11:55:15.273 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.2-in add sequence 3, type: deny
2024/11/21 11:55:15.274 BFD: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15.274 BFD: [G6NKK-8C6DV] end_config: VTY:0x7f9fb1912020, pending SET-CFG: 0
2024/11/21 11:55:15.283 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.2-denied-inpl-ipv4 in route-map 172.30.0.2-in
2024/11/21 11:55:15.283 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.2-denied-inpl-ipv4: 172.30.0.2-in
2024/11/21 11:55:15.285 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.2-in add sequence 4, type: deny
2024/11/21 11:55:15.289 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.2-denied-inpl-ipv4 in route-map 172.30.0.2-in
2024/11/21 11:55:15.289 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.2-denied-inpl-ipv4: 172.30.0.2-in
2024/11/21 11:55:15.300 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.2-in add sequence 5, type: permit
2024/11/21 11:55:15.304 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.2-inpl-ipv4 in route-map 172.30.0.2-in
2024/11/21 11:55:15.304 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.2-inpl-ipv4: 172.30.0.2-in
2024/11/21 11:55:15.306 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.30.0.2-in add sequence 6, type: permit
2024/11/21 11:55:15.310 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.30.0.2-inpl-ipv4 in route-map 172.30.0.2-in
2024/11/21 11:55:15.310 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.30.0.2-inpl-ipv4: 172.30.0.2-in
2024/11/21 11:55:15.322 ZEBRA: [V4EEB-GQBPV] Add route-map 172.18.0.6-out
2024/11/21 11:55:15.322 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.6-out add sequence 1, type: permit
2024/11/21 11:55:15.325 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.6-pl-ipv4 in route-map 172.18.0.6-out
2024/11/21 11:55:15.325 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.6-pl-ipv4: 172.18.0.6-out
2024/11/21 11:55:15.327 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.6-out add sequence 2, type: permit
2024/11/21 11:55:15.331 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.6-pl-ipv4 in route-map 172.18.0.6-out
2024/11/21 11:55:15.331 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.6-pl-ipv4: 172.18.0.6-out
2024/11/21 11:55:15.333 ZEBRA: [RWPWC-9TYFK] Filter 172.18.0.6-pl-ipv4 updated
2024/11/21 11:55:15.333 ZEBRA: [PQ39Y-44VF4] Notifying 172.18.0.6-out of dependency
2024/11/21 11:55:15.340 ZEBRA: [V4EEB-GQBPV] Add route-map 172.18.0.6-in
2024/11/21 11:55:15.340 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.6-in add sequence 3, type: deny
2024/11/21 11:55:15.343 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.6-denied-inpl-ipv4 in route-map 172.18.0.6-in
2024/11/21 11:55:15.343 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.6-denied-inpl-ipv4: 172.18.0.6-in
2024/11/21 11:55:15.345 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.6-in add sequence 4, type: deny
2024/11/21 11:55:15.347 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.6-denied-inpl-ipv4 in route-map 172.18.0.6-in
2024/11/21 11:55:15.347 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.6-denied-inpl-ipv4: 172.18.0.6-in
2024/11/21 11:55:15.348 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.6-in add sequence 5, type: permit
2024/11/21 11:55:15.351 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.6-inpl-ipv4 in route-map 172.18.0.6-in
2024/11/21 11:55:15.351 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.6-inpl-ipv4: 172.18.0.6-in
2024/11/21 11:55:15.353 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.6-in add sequence 6, type: permit
2024/11/21 11:55:15.356 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.6-inpl-ipv4 in route-map 172.18.0.6-in
2024/11/21 11:55:15.356 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.6-inpl-ipv4: 172.18.0.6-in
2024/11/21 11:55:15.361 ZEBRA: [V4EEB-GQBPV] Add route-map 172.18.0.5-out
2024/11/21 11:55:15.361 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.5-out add sequence 1, type: permit
2024/11/21 11:55:15.365 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.5-pl-ipv4 in route-map 172.18.0.5-out
2024/11/21 11:55:15.365 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.5-pl-ipv4: 172.18.0.5-out
2024/11/21 11:55:15.367 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.5-out add sequence 2, type: permit
2024/11/21 11:55:15 WATCHFRR: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15.370 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.5-pl-ipv4 in route-map 172.18.0.5-out
2024/11/21 11:55:15.370 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.5-pl-ipv4: 172.18.0.5-out
2024/11/21 11:55:15.371 ZEBRA: [RWPWC-9TYFK] Filter 172.18.0.5-pl-ipv4 updated
2024/11/21 11:55:15.371 ZEBRA: [PQ39Y-44VF4] Notifying 172.18.0.5-out of dependency
2024/11/21 11:55:15.380 ZEBRA: [V4EEB-GQBPV] Add route-map 172.18.0.5-in
2024/11/21 11:55:15.380 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.5-in add sequence 3, type: deny
2024/11/21 11:55:15.383 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.5-denied-inpl-ipv4 in route-map 172.18.0.5-in
2024/11/21 11:55:15.383 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.5-denied-inpl-ipv4: 172.18.0.5-in
2024/11/21 11:55:15.385 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.5-in add sequence 4, type: deny
2024/11/21 11:55:15.388 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.5-denied-inpl-ipv4 in route-map 172.18.0.5-in
2024/11/21 11:55:15.388 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.5-denied-inpl-ipv4: 172.18.0.5-in
2024/11/21 11:55:15.390 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.5-in add sequence 5, type: permit
2024/11/21 11:55:15.393 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.5-inpl-ipv4 in route-map 172.18.0.5-in
2024/11/21 11:55:15.393 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.5-inpl-ipv4: 172.18.0.5-in
2024/11/21 11:55:15.396 ZEBRA: [Q5QB0-7A1DJ] Route-map 172.18.0.5-in add sequence 6, type: permit
2024/11/21 11:55:15.399 ZEBRA: [MB1AM-D3Z0A] Adding dependency for filter 172.18.0.5-inpl-ipv4 in route-map 172.18.0.5-in
2024/11/21 11:55:15.399 ZEBRA: [KSWFP-ABCEJ] route_map_print_dependency: Dependency for 172.18.0.5-inpl-ipv4: 172.18.0.5-in
2024/11/21 11:55:15.423 ZEBRA: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:01
2024/11/21 11:55:15.423 ZEBRA: [G6NKK-8C6DV] end_config: VTY:0x7f45b3312040, pending SET-CFG: 0
2024/11/21 11:55:15.743 MGMTD: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15.743 MGMTD: [G6NKK-8C6DV] end_config: VTY:0x7f307d1df300, pending SET-CFG: 0
2024/11/21 11:55:15.811 STATIC: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15.811 STATIC: [G6NKK-8C6DV] end_config: VTY:0x7fe730a293a0, pending SET-CFG: 0
2024/11/21 11:55:15.817 BFD: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15.817 BFD: [G6NKK-8C6DV] end_config: VTY:0x7f9fb1912850, pending SET-CFG: 0
2024/11/21 11:55:15.876 ZEBRA: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2024/11/21 11:55:15.876 ZEBRA: [G6NKK-8C6DV] end_config: VTY:0x7f45b3312780, pending SET-CFG: 0
2024/11/21 11:55:17 WATCHFRR: [YFT0P-5Q5YX] Forked background command [pid 276]: /usr/lib/frr/watchfrr.sh restart bgpd
Cannot stop bgpd: pid 166 not running
2024/11/21 11:55:17 WATCHFRR: [QDG3Y-BY5TN] bgpd state -> up : connect succeeded
2024/11/21 11:55:17 WATCHFRR: [KWE5Q-QNGFC] all daemons up, doing startup-complete notify
2

indicate that the config was added (using the frr-reloader) before the bgpd is up, and probably bgpd deamon crashed (Cannot stop bgpd: pid 166 not running or bgpd was stopped). That is not transcient behavior, once it happens it remains like that.

This looks like a bug because config should be inserted only if daemons are up.
What do you think?

Thank you!

Version

quay.io/frrouting/frr:9.1.0

How to reproduce

It happens in our CI, we have not been able to reproduce it.

Expected behavior

The frr-reloader script should not be able to insert config unless daemon is up.

Actual behavior

frr-reloader script not to be able to add any config unless daemon are up.

Additional context

No response

Checklist

  • I have searched the open issues for this bug.
  • I have not included sensitive information in this report.
@karampok karampok added the triage Needs further investigation label Dec 3, 2024
@ton31337
Copy link
Member

ton31337 commented Dec 3, 2024

2024/11/21 11:55:17 WATCHFRR: [QDG3Y-BY5TN] bgpd state -> up : connect succeeded

This means it's connected later than the configuration is processed.

and probably bgpd deamon crashed

How do you know that? Can you get a backtrace or even a coredump?

Also, what is the configuration?

@karampok
Copy link
Author

karampok commented Dec 3, 2024

In the past i have seen this bgpd crashed in startup and I cannot see other reason why pid dissappeared.

umping logs for pod frr-k8s-system-frrk8s-frr-k8s-wffq9-frr
2024/11/18 17:44:55 WATCHFRR: [T83RR-8SM5G] watchfrr 9.1_git starting: vty@0
2024/11/18 17:44:55 WATCHFRR: [ZCJ3S-SPH5S] zebra state -> down : initial connection attempt failed
2024/11/18 17:44:55 WATCHFRR: [ZCJ3S-SPH5S] mgmtd state -> down : initial connection attempt failed
2024/11/18 17:44:55 WATCHFRR: [ZCJ3S-SPH5S] bgpd state -> down : initial connection attempt failed
2024/11/18 17:44:55 WATCHFRR: [ZCJ3S-SPH5S] staticd state -> down : initial connection attempt failed
2024/11/18 17:44:55 WATCHFRR: [ZCJ3S-SPH5S] bfdd state -> down : initial connection attempt failed
2024/11/18 17:44:55 WATCHFRR: [YFT0P-5Q5YX] Forked background command [pid 82]: /usr/lib/frr/watchfrr.sh restart all
2024/11/18 17:44:55 ZEBRA: [NNACN-54BDA][EC 4043309110] Disabling MPLS support (no kernel support)
bgpd crashed in startup, signal 11
Failed to start bgpd!
[183|zebra] sending configuration
[183|zebra] done
Waiting for children to finish applying config...
[197|watchfrr] sending configuration
[199|staticd] sending configuration
[200|bfdd] sending configuration
2024/11/18 17:44:55 WATCHFRR: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
[197|watchfrr] done
[199|staticd] done
[182|mgmtd] done
[200|bfdd] done

I have added a modified branch to catch the coredump but is not happening. As I said not very often.

How can I get a backtrace? Also for completeness is there a documentation how to catch the coredump (or the generic way)?

The config that frr-reloader script loads is

####### Dumping file /etc/frr/frr.conf
frr version 9.1_git
frr defaults traditional
hostname frr-k8s-worker2
log file /etc/frr/frr.log
log timestamp precision 3
service integrated-vtysh-config
!
debug zebra events
debug zebra kernel
debug zebra rib
debug zebra nht
debug zebra nexthop
debug bgp keepalives
debug bgp neighbor-events
debug bgp nht
debug bgp updates in
debug bgp updates out
debug bgp zebra
debug bfd peer
debug bfd zebra
debug bfd network
!
debug route-map
!
router bgp 64512
 no bgp ebgp-requires-policy
 no bgp hard-administrative-reset
 no bgp default ipv4-unicast
 no bgp graceful-restart notification
 bgp graceful-restart preserve-fw-state
 no bgp network import-check
 neighbor 172.18.0.5 remote-as 64512
 neighbor 172.18.0.5 password ibgp-test
 neighbor 172.18.0.5 graceful-restart
 neighbor 172.18.0.6 remote-as 4200000000
 neighbor 172.18.0.6 graceful-restart
 neighbor 172.30.0.2 remote-as 64512
 neighbor 172.30.0.2 password ibgp-test
 neighbor 172.30.0.2 port 180
 neighbor 172.30.0.2 graceful-restart
 neighbor 172.30.0.3 remote-as 4200000000
 neighbor 172.30.0.3 password ebgp-test
 neighbor 172.30.0.3 port 180
 neighbor 172.30.0.3 ebgp-multihop
 neighbor 172.30.0.3 graceful-restart
 !
 address-family ipv4 unicast
  network 192.168.2.0/24
  neighbor 172.18.0.5 activate
  neighbor 172.18.0.5 route-map 172.18.0.5-in in
  neighbor 172.18.0.5 route-map 172.18.0.5-out out
  neighbor 172.18.0.6 activate
  neighbor 172.18.0.6 route-map 172.18.0.6-in in
  neighbor 172.18.0.6 route-map 172.18.0.6-out out
  neighbor 172.30.0.2 activate
  neighbor 172.30.0.2 route-map 172.30.0.2-in in
  neighbor 172.30.0.2 route-map 172.30.0.2-out out
  neighbor 172.30.0.3 activate
  neighbor 172.30.0.3 route-map 172.30.0.3-in in
  neighbor 172.30.0.3 route-map 172.30.0.3-out out
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor 172.18.0.5 activate
  neighbor 172.18.0.5 route-map 172.18.0.5-in in
  neighbor 172.18.0.5 route-map 172.18.0.5-out out
  neighbor 172.18.0.6 activate
  neighbor 172.18.0.6 route-map 172.18.0.6-in in
  neighbor 172.18.0.6 route-map 172.18.0.6-out out
  neighbor 172.30.0.2 activate
  neighbor 172.30.0.2 route-map 172.30.0.2-in in
  neighbor 172.30.0.2 route-map 172.30.0.2-out out
  neighbor 172.30.0.3 activate
  neighbor 172.30.0.3 route-map 172.30.0.3-in in
  neighbor 172.30.0.3 route-map 172.30.0.3-out out
 exit-address-family
exit
!
ip prefix-list 172.30.0.3-pl-ipv4 seq 1 permit 192.168.2.0/24
ip prefix-list 172.30.0.3-inpl-ipv4 seq 1 deny any
ip prefix-list 172.30.0.3-denied-inpl-ipv4 seq 1 permit 192.167.9.0/24 le 32
ip prefix-list 172.30.0.2-pl-ipv4 seq 1 permit 192.168.2.0/24
ip prefix-list 172.30.0.2-inpl-ipv4 seq 1 deny any
ip prefix-list 172.30.0.2-denied-inpl-ipv4 seq 1 permit 192.167.9.0/24 le 32
ip prefix-list 172.18.0.6-pl-ipv4 seq 1 permit 192.168.2.0/24
ip prefix-list 172.18.0.6-inpl-ipv4 seq 1 deny any
ip prefix-list 172.18.0.6-denied-inpl-ipv4 seq 1 permit 192.167.9.0/24 le 32
ip prefix-list 172.18.0.5-pl-ipv4 seq 1 permit 192.168.2.0/24
ip prefix-list 172.18.0.5-inpl-ipv4 seq 1 deny any
ip prefix-list 172.18.0.5-denied-inpl-ipv4 seq 1 permit 192.167.9.0/24 le 32
!
ipv6 prefix-list 172.30.0.3-pl-ipv4 seq 2 deny any
ipv6 prefix-list 172.30.0.3-inpl-ipv4 seq 2 deny any
ipv6 prefix-list 172.30.0.3-denied-inpl-ipv4 seq 2 permit fc00:f553:ccd:e799::/64 le 128
ipv6 prefix-list 172.30.0.2-pl-ipv4 seq 2 deny any
ipv6 prefix-list 172.30.0.2-inpl-ipv4 seq 2 deny any
ipv6 prefix-list 172.30.0.2-denied-inpl-ipv4 seq 2 permit fc00:f553:ccd:e799::/64 le 128
ipv6 prefix-list 172.18.0.6-pl-ipv4 seq 2 deny any
ipv6 prefix-list 172.18.0.6-inpl-ipv4 seq 2 deny any
ipv6 prefix-list 172.18.0.6-denied-inpl-ipv4 seq 2 permit fc00:f553:ccd:e799::/64 le 128
ipv6 prefix-list 172.18.0.5-pl-ipv4 seq 2 deny any
ipv6 prefix-list 172.18.0.5-inpl-ipv4 seq 2 deny any
ipv6 prefix-list 172.18.0.5-denied-inpl-ipv4 seq 2 permit fc00:f553:ccd:e799::/64 le 128
!
route-map 172.30.0.3-out permit 1
 match ip address prefix-list 172.30.0.3-pl-ipv4
exit
!
route-map 172.30.0.3-out permit 2
 match ipv6 address prefix-list 172.30.0.3-pl-ipv4
exit
!
route-map 172.30.0.3-in deny 3
 match ip address prefix-list 172.30.0.3-denied-inpl-ipv4
exit
!
route-map 172.30.0.3-in deny 4
 match ipv6 address prefix-list 172.30.0.3-denied-inpl-ipv4
exit
!
route-map 172.30.0.3-in permit 5
 match ip address prefix-list 172.30.0.3-inpl-ipv4
exit
!
route-map 172.30.0.3-in permit 6
 match ipv6 address prefix-list 172.30.0.3-inpl-ipv4
exit
!
route-map 172.30.0.2-out permit 1
 match ip address prefix-list 172.30.0.2-pl-ipv4
exit
!
route-map 172.30.0.2-out permit 2
 match ipv6 address prefix-list 172.30.0.2-pl-ipv4
exit
!
route-map 172.30.0.2-in deny 3
 match ip address prefix-list 172.30.0.2-denied-inpl-ipv4
exit
!
route-map 172.30.0.2-in deny 4
 match ipv6 address prefix-list 172.30.0.2-denied-inpl-ipv4
exit
!
route-map 172.30.0.2-in permit 5
 match ip address prefix-list 172.30.0.2-inpl-ipv4
exit
!
route-map 172.30.0.2-in permit 6
 match ipv6 address prefix-list 172.30.0.2-inpl-ipv4
exit
!
route-map 172.18.0.6-out permit 1
 match ip address prefix-list 172.18.0.6-pl-ipv4
exit
!
route-map 172.18.0.6-out permit 2
 match ipv6 address prefix-list 172.18.0.6-pl-ipv4
exit
!
route-map 172.18.0.6-in deny 3
 match ip address prefix-list 172.18.0.6-denied-inpl-ipv4
exit
!
route-map 172.18.0.6-in deny 4
 match ipv6 address prefix-list 172.18.0.6-denied-inpl-ipv4
exit
!
route-map 172.18.0.6-in permit 5
 match ip address prefix-list 172.18.0.6-inpl-ipv4
exit
!
route-map 172.18.0.6-in permit 6
 match ipv6 address prefix-list 172.18.0.6-inpl-ipv4
exit
!
route-map 172.18.0.5-out permit 1
 match ip address prefix-list 172.18.0.5-pl-ipv4
exit
!
route-map 172.18.0.5-out permit 2
 match ipv6 address prefix-list 172.18.0.5-pl-ipv4
exit
!
route-map 172.18.0.5-in deny 3
 match ip address prefix-list 172.18.0.5-denied-inpl-ipv4
exit
!
route-map 172.18.0.5-in deny 4
 match ipv6 address prefix-list 172.18.0.5-denied-inpl-ipv4
exit
!
route-map 172.18.0.5-in permit 5
 match ip address prefix-list 172.18.0.5-inpl-ipv4
exit
!
route-map 172.18.0.5-in permit 6
 match ipv6 address prefix-list 172.18.0.5-inpl-ipv4
exit
!

@ton31337
Copy link
Member

ton31337 commented Dec 3, 2024

To be sure, is this happening (a crash with 10.2 or master)? Or what happens if you move prefix-list and route-map definitions before "router bgp"?

@ton31337 ton31337 added the bgp label Dec 3, 2024
@karampok
Copy link
Author

karampok commented Dec 3, 2024

This is happening with quay.io/frrouting/frr:9.1.0 , I cannot switch yet to latest master or 10.2 version.

if you move prefix-list and route-map definitions before "router bgp"?

I might be mistaken but the frr-reloader script seems to re-organizing the input so what order we have is not the final order. Is that statement correct?

@ton31337
Copy link
Member

ton31337 commented Dec 3, 2024

Can't test with quay.io/frrouting/frr:master?

@ton31337
Copy link
Member

ton31337 commented Dec 3, 2024

Is it possible to somehow replicate this deterministically?

@karampok
Copy link
Author

karampok commented Dec 4, 2024

We try but for the time being we cannot reproduce deterministically unfortunately.

This means it's connected later than the configuration is processed.

Can you expand on, I am not sure I understand who is connected, who is processing the config, thanks

@ton31337
Copy link
Member

ton31337 commented Dec 4, 2024

Ignore my previous comment, I just realized the timing is different here:

2024/11/21 11:55:17 WATCHFRR: [QDG3Y-BY5TN] bgpd state -> up : connect succeeded

and later:

2024/11/21 11:55:47 BGP: [H5AW4-JFYQC] Route-map: null, prefix: 192.168.2.0/24, result: deny

@karampok is this somehow related to this issue also #16367?

@fedepaol
Copy link

fedepaol commented Dec 4, 2024

@karampok is this somehow related to this issue also #16367?

I am not entirely sure about it, as #16367 was happening without any frr restart

@ton31337 ton31337 removed the triage Needs further investigation label Dec 5, 2024
@ton31337
Copy link
Member

@karampok could you repeat again and send us the logs with the master tag? In master there are two PRs merged that would provide more useful information what's going on in your case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants