This frame relay troubleshooting hit me big time. I’ve spent about 7.5 hours just to get this up and running, apparently, due to simple mistake.
The network configured as below.
R5 (s0/0/0 - DLCI 504) --- ( DLCI 405 - s0/0/0) R4
R5 (s0/0/0 - DLCI 513) --- ( DLCI 315 - s1/1) R3
Rack13R5#sh run int s0/0/0
interface Serial0/0/0
ip address 183.13.0.5 255.255.255.0
ip pim sparse-dense-mode
encapsulation frame-relay
ip ospf network broadcast
ip ospf dead-interval minimal hello-multiplier 3
ip ospf priority 255
frame-relay map ip 183.13.0.3 513 broadcast
frame-relay map ip 183.13.0.4 504 broadcast
no frame-relay inverse-arp
end
Rack13R4(config-router)#do sh run int s0/0/0
interface Serial0/0/0
ip address 183.13.0.4 255.255.255.0
encapsulation frame-relay
ip ospf network broadcast
ip ospf dead-interval minimal hello-multiplier 3
ip ospf priority 0
no keepalive
frame-relay map ip 183.13.0.5 405 broadcast
frame-relay map ip 183.13.0.3 405
no frame-relay inverse-arp
end
!R3
interface Serial1/1
no shutdown
ip address 183.13.0.3 255.255.255.0
ip pim sparse-dense-mode
encapsulation frame-relay
ip ospf network broadcast
ip ospf dead-interval minimal hello-multiplier 3
ip ospf priority 0
frame-relay map ip 183.13.0.4 315
frame-relay map ip 183.13.0.5 315 broadcast
no frame-relay inverse-arp
end
R5-R3 connection is fine but I cannot ping R4 FR IP Addr 183.13.0.4 from R5 and R3.
Rack13R5#ping 183.13.0.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 183.13.0.4, timeout is 2 seconds:
..
Success rate is 0 percent (0/2)
debug ip packet shown below.
Rack13R5#x
*Feb 5 22:09:48.171: IP: s=183.13.0.5 (local), d=183.13.0.4 (Serial0/0/0), len 100, sending
*Feb 5 22:09:48.171: ICMP type=8, code=0
*Feb 5 22:09:48.171: IP: s=183.13.0.5 (local), d=183.13.0.4 (Serial0/0/0), len 100,
encapsulation failed
*Feb 5 22:09:48.171: ICMP type=8, code=0
*Feb 5 22:09:50.171: IP: s=183.13.0.5 (local), d=183.13.0.4 (Serial0/0/0), len 100, sending
*Feb 5 22:09:50.171: ICMP type=8, code=0
*Feb 5 22:09:50.171: IP: s=183.13.0.5 (local), d=183.13.0.4 (Serial0/0/0), len 100,
encapsulation failed
*Feb 5 22:09:50.171: ICMP type=8, code=0
Apparently the L2 is having problem. Let’s check the DLCI mapping.
Rack13R5#sh frame-relay map
Serial0/0/0 (up): ip 183.13.0.4 dlci 504(0x1F8,0x7C80), static,
broadcast,
CISCO, status defined, inactive
Serial0/0/0 (up): ip 183.13.0.3 dlci 513(0x201,0x8010), static,
broadcast,
CISCO, status defined, active
Rack13R5#sh frame-relay pvc 504
PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)
DLCI = 504, DLCI USAGE = LOCAL, PVC STATUS = INACTIVE, INTERFACE = Serial0/0/0
input pkts 0 output pkts 24 in bytes 0
out bytes 1918 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 26 out bcast bytes 1966
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
pvc create time 00:04:16, last time pvc status changed 00:04:06
Gotcha, the DLCI 504 is inactive. This is why R5 cannot send L3 packets to R4 via DLCI 504 due to the PVC is INACTIVE. This would also mean that R3 won’t be able to ping R4 as R5 is the hub to reach R4.
Rack13R3#ping 183.13.0.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 183.13.0.4, timeout is 2 seconds:
..
Success rate is 0 percent (0/2)
Let’s see whether R4 can send the packets out to R5 via DLCI 405.
Rack13R4#ping 183.13.0.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 183.13.0.5, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Rack13R4#
Feb 5 22:04:55.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb 5 22:04:55.270: ICMP type=8, code=0
Feb 5 22:04:55.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb 5 22:04:55.270: ICMP type=8, code=0
Feb 5 22:04:57.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb 5 22:04:57.270: ICMP type=8, code=0
Feb 5 22:04:57.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb 5 22:04:57.270: ICMP type=8, code=0
Feb 5 22:04:59.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb 5 22:04:59.270: ICMP type=8, code=0
Feb 5 22:04:59.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb 5 22:04:59.270: ICMP type=8, code=0
Feb 5 22:05:01.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb 5 22:05:01.270: ICMP type=8, code=0
Feb 5 22:05:01.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb 5 22:05:01.270: ICMP type=8, code=0
Feb 5 22:05:03.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb 5 22:05:03.270: ICMP type=8, code=0
Feb 5 22:05:03.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb 5 22:05:03.270: ICMP type=8, code=0
Rack13R4#sh frame-relay pvc 405
PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)
DLCI = 405, DLCI USAGE = LOCAL, PVC STATUS = STATIC, INTERFACE = Serial0/0/0
input pkts 2606 output pkts 38365 in bytes 197551
out bytes 3061869 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 37556 out bcast bytes 3011444
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 1000 bits/sec, 3 packets/sec
pvc create time 06:34:40, last time pvc status changed 00:12:34
Looks like R4 is just fine, sending packets out to R5.
INACTIVE status means the PVC is not up. The reason can be the local DLCI mapping issue or the other end is not up. Obviously, the local DLCI mapping is just fine. So, this could be an issue on the other end. (well, there is an FR switch in the middle but I really doubt it causing the problem. I did reload and check the config on the FR switch, all ok)
Rack13R5#sh run int s0/0/0
interface Serial0/0/0
ip address 183.13.0.5 255.255.255.0
ip pim sparse-dense-mode
encapsulation frame-relay
ip ospf network broadcast
ip ospf dead-interval minimal hello-multiplier 3
ip ospf priority 255
frame-relay map ip 183.13.0.3 513 broadcast
frame-relay map ip 183.13.0.4 504 broadcast
no frame-relay inverse-arp
end
If we look the PVC 504 on R5, we could see that the input packets is 0, it means that pvc 504 is not receiving LMI from 405.
Rack13R5#sh frame-relay pvc 504
PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)
DLCI = 504, DLCI USAGE = LOCAL, PVC STATUS = INACTIVE, INTERFACE = Serial0/0/0
input pkts 0 output pkts 24 in bytes 0
out bytes 1918 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 26 out bcast bytes 1966
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
pvc create time 00:04:16, last time pvc status changed 00:04:06
Checking the PVC 405 on R4 again
Rack13R4#sh frame-relay pvc 405
PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)
DLCI = 405, DLCI USAGE = LOCAL, PVC STATUS = STATIC, INTERFACE = Serial0/0/0
input pkts 2606 output pkts 38365 in bytes 197551
out bytes 3061869 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 37556 out bcast bytes 3011444
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 1000 bits/sec, 3 packets/sec
pvc create time 06:34:40, last time pvc status changed 00:12:34
The DLCI 405 is showing STATIC. What STATIC? I know INACTIVE, ACTIVE, and DELETED. But STATIC? Cisco Doco in here and here shows that STATIC means that the PVC is statically created thus no LMI/keepalive being sent.
Checking the config again on R4, you will find the culprit keepalive was disabled.
Rack13R4(config-router)#do sh run int s0/0/0
interface Serial0/0/0
ip address 183.13.0.4 255.255.255.0
encapsulation frame-relay
ip ospf network broadcast
ip ospf dead-interval minimal hello-multiplier 3
ip ospf priority 0
no keepalive <<<===
frame-relay map ip 183.13.0.5 405 broadcast
frame-relay map ip 183.13.0.3 405
no frame-relay inverse-arp
end
Rack13R4(config-if)#do sh int s0/0/0
Serial0/0/0 is up, line protocol is up
Hardware is GT96K Serial
Internet address is 183.13.0.4/24
MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, loopback not set
Keepalive not set <<<===
CRC checking enabled
LMI DLCI 1023 LMI type is CISCO frame relay DTE
FR SVC disabled, LAPF state down
Broadcast queue 0/64, broadcasts sent/dropped 11470/0, interface broadcasts 11470
Last input 01:36:59, output 00:35:57, output hang never
Last clearing of "show interface" counters 01:37:00
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1 packets input, 53 bytes, 0 no buffer
Enabling the keepalive (default 10 seconds) brings back both the PVC 405 and 504 to ACTIVE.
Rack13R4(config-if)#int s0/0/0
Rack13R4(config-if)#keepalive ?
<0-30> Keepalive period (default 10 seconds)
Rack13R4(config-if)#keepalive
Rack13R4(config-if)#
Feb 6 22:52:30.611: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0,
changed state to down
Rack13R4(config-if)#
Feb 6 22:52:50.611: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0,
changed state to up
Rack13R4(config-if)#do sh fram pvc 405
PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)
DLCI = 405, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0
input pkts 0 output pkts 0 in bytes 0
out bytes 0 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 0 out bcast bytes 0
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
pvc create time 00:36:53, last time pvc status changed 00:00:20
Rack13R5(config-if)#do sh fram pvc 504
PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)
DLCI = 504, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0
input pkts 6446 output pkts 6446 in bytes 540482
out bytes 540590 dropped pkts 0 in pkts dropped 0
out pkts dropped 0 out bytes dropped 0
in FECN pkts 0 in BECN pkts 0 out FECN pkts 0
out BECN pkts 0 in DE pkts 0 out DE pkts 0
out bcast pkts 0 out bcast bytes 0
5 minute input rate 2000 bits/sec, 3 packets/sec
5 minute output rate 2000 bits/sec, 3 packets/sec
pvc create time 01:13:54, last time pvc status changed 00:35:42
We can also see that PVC 504 receiving packets.
Initially, I could point the issue that R5 is not receiving LMI. But I guess that’s not good enough. Knowing that PVC STATIC is basically statically assigned DLCI thus won’t send any LMI out would save a lot of my time.
On the other hand, I also need to fix the OSPF issue on this FR by applying non-broadcast network type as shown below.