There are several things to be satisfied before OSPF can establish its full adjacency. These are Area-ID, Stub-Flags, Interface-Type, Timers, Authentication, and MTU. These attributes must match to the other peering.

Let’s say that you have a scenario to create Q-in-Q which requires you to alter the default MTU to 1504 using system mtu 1504. The verification also shows that your system MTU is indeed 1504.

SWITCH-3560#sh system mtu

System MTU size is 1504 bytes
System Jumbo MTU size is 1504 bytes
Routing MTU size is 1500 bytes

Suddenly, your OSPF adjacency went down, re-initialized, and then stuck in EXSTART/EXCHANGE. In this state, both sides will try to exchange the DBD packets. This problem can occurs when the MTU attribute between peers doesn’t match. (Also, you can check for Unicast traffic – if any, ACL, or even NAT)

Well, isn’t that you have your switches MTU’s adjusted to 1504 using command system mtu 1504? Yes, the SYSTEM MTU is 1504, which was configured for Q-in-Q.

OSPF, however, uses ROUTING MTU instead of SYSTEM MTU. Therefore, to change the ROUTING MTU we need to use command system mtu routing 1504.

Another way to fix this issue is to disable the MTU checking during adjacency process. You can use ip ospf mtu-ignore. This can be handy if you don’t have the option to change the MTU value.

SWITCH-3560#sh system mtu

System MTU size is 1504 bytes
System Jumbo MTU size is 1504 bytes
Routing MTU size is 1500 bytes
SWITCH-3560#
*Mar  1 00:03:16.050: OSPF: Rcv DBD from 10.4.4.4 on Port-channel14 seq 0x1995 opt 0x52 flag 0x2 len 52  mtu 1504 state EXSTART
*Mar  1 00:03:16.050: OSPF: Nbr 10.4.4.4 has larger interface MTU
*Mar  1 00:03:20.974: OSPF: Send hello to 224.0.0.5 area 0 on Port-channel14 from 163.17.0.1
*Mar  1 00:03:21.058: OSPF: Send DBD to 10.4.4.4 on Port-channel14 seq 0x1995 opt 0x52 flag 0x7 len 32
*Mar  1 00:03:21.058: OSPF: Retransmitting DBD to 10.4.4.4 on Port-channel14 [20]
*Mar  1 00:03:21.058: OSPF: Rcv DBD from 10.4.4.4 on Port-channel14 seq 0x1995 opt 0x52 flag 0x2 len 52  mtu 1504 state EXSTART
*Mar  1 00:03:21.058: OSPF: Nbr 10.4.4.4 has larger interface MTU
*Mar  1 00:03:21.460: OSPF: Rcv hello from 10.4.4.4 area 0 from Port-channel14 163.17.0.4l
SWITCH-3560#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SWITCH-3560(config)#system mtu ?
  <1500-1998>  MTU size in bytes
  jumbo        Set Jumbo MTU value for GigabitEthernet or TenGigabitEthernet
               interfaces
  routing      Set the Routing MTU for the system

SWITCH-3560(config)#system mtu routing 1504
*Mar  1 00:03:46.492: %SYS-5-CONFIG_I: Configured from console by console
*Mar  1 00:03:51.106: %OSPF-5-ADJCHG: Process 1, Nbr 10.4.4.4 on Port-channel14 from EXSTART to DOWN, Neighbor Down: Too many SWITCH-3560(config)#int port 14
SWITCH-3560(config-if)#shut
SWITCH-3560(config-if)#no shut
SWITCH-3560(config-if)#
*Mar  1 00:04:02.162: %OSPF-5-ADJCHG: Process 1, Nbr 10.4.4.4 on Port-channel14 from DOWN to DOWN, Neighbor Down: Interface down or detached
*Mar  1 00:04:03.135: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel14, changed state to down
SWITCH-3560(config-if)#
SWITCH-3560(config-if)#
*Mar  1 00:04:04.133: %LINK-3-UPDOWN: Interface Port-channel14, changed state to down
SWITCH-3560(config-if)#
*Mar  1 00:04:05.106: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/20, changed state to down
*Mar  1 00:04:05.123: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/21, changed state to down
SWITCH-3560(config-if)#
*Mar  1 00:04:06.574: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/21, changed state to up
*Mar  1 00:04:06.650: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/20, changed state to up
SWITCH-3560(config-if)#^Z
SWITCH-3560#
*Mar  1 00:04:07.572: %LINK-3-UPDOWN: Interface Port-channel14, changed state to up
SWITCH-3560#sh ip os
*Mar  1 00:04:08.369: %SYS-5-CONFIG_I: Configured from console by console
*Mar  1 00:04:08.588: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel14, changed state to up
*Mar  1 00:04:17.563: OSPF: Rcv hello from 10.4.4.4 area 0 from Port-channel14 163.17.0.4
*Mar  1 00:04:17.563: OSPF: 2 Way Communication to 10.4.4.4 on Port-channel14, state 2WAY
*Mar  1 00:04:17.563: OSPF: End of hello processing
*Mar  1 00:04:17.572: OSPF: Send hello to 224.0.0.5 area 0 on Port-channel14 from 163.17.0.1
*Mar  1 00:04:17.572: OSPF: will poll [count 10] interface status for Port-channel14
*Mar  1 00:04:27.563: OSPF: Rcv hello from 10.4.4.4 area 0 from Port-channel14 163.17.0.4
*Mar  1 00:04:27.563: OSPF: End of hello processing
*Mar  1 00:04:27.579: OSPF: Send hello to 224.0.0.5 area 0 on Port-channel14 from 163.17.0.1
*Mar  1 00:04:27.579: OSPF: will poll [count 9] interface status for Port-channel14
*Mar  1 00:04:37.562: OSPF: Rcv hello from 10.4.4.4 area 0 from Port-channel14 163.17.0.4
*Mar  1 00:04:37.562: OSPF: End of hello processing
*Mar  1 00:04:37.579: OSPF: Send hello to 224.0.0.5 area 0 on Port-channel14 from 163.17.0.1
*Mar  1 00:04:37.579: OSPF: will poll [count 8] interface status for Port-channel14
*Mar  1 00:04:47.569: OSPF: Rcv hello from 10.4.4.4 area 0 from Port-channel14 163.17.0.4
*Mar  1 00:04:47.569: OSPF: Backup seen Event before WAIT timer on Port-channel14
*Mar  1 00:04:47.569: OSPF: DR/BDR election on Port-channel14
*Mar  1 00:04:47.569: OSPF: Elect BDR 0.0.0.0
*Mar  1 00:04:47.569: OSPF: Elect DR 10.4.4.4
*Mar  1 00:04:47.569:        DR: 10.4.4.4 (Id)   BDR: none
*Mar  1 00:04:47.569: OSPF: Send DBD to 10.4.4.4 on Port-channel14 seq 0x12F3 opt 0x52 flag 0x7 len 32
*Mar  1 00:04:47.569: OSPF: End of hello processing
*Mar  1 00:04:47.569: OSPF: Rcv DBD from 10.4.4.4 on Port-channel14 seq 0x1041 opt 0x52 flag 0x7 len 32  mtu 1504 state EXSTART
*Mar  1 00:04:47.569: OSPF: First DBD and we are not SLAVE
*Mar  1 00:04:47.569: OSPF: Rcv DBD from 10.4.4.4 on Port-channel14 seq 0x12F3 opt 0x52 flag 0x2 len 92  mtu 1504 state EXSTART
*Mar  1 00:04:47.569: OSPF: NBR Negotiation Done. We are the MASTER
*Mar  1 00:04:47.569: OSPF: Send DBD to 10.4.4.4 on Port-channel14 seq 0x12F4 opt 0x52 flag 0x3 len 52
*Mar  1 00:04:47.578: OSPF: Rcv DBD from 10.4.4.4 on Port-channel14 seq 0x12F4 opt 0x52 flag 0x0 len 32  mtu 1504 state EXCHANGE
*Mar  1 00:04:47.578: OSPF: Send DBD to 10.4.4.4 on Port-channel14 seq 0x12F5 opt 0x52 flag 0x1 len 32
*Mar  1 00:04:47.578: OSPF: Send LS REQ to 10.4.4.4 length 36 LSA count 3
*Mar  1 00:04:47.578: OSPF: Rcv LS REQ from 10.4.4.4 on Port-channel14 length 36 LSA count 1
*Mar  1 00:04:47.578: OSPF: Send UPD to 163.17.0.4 on Port-channel14 length 40 LSA count 1
*Mar  1 00:04:47.578: OSPF: Rcv DBD from 10.4.4.4 on Port-channel14 seq 0x12F5 opt 0x52 flag 0x0 len 32  mtu 1504 state EXCHANGE
*Mar  1 00:04:47.578: OSPF: Exchange Done with 10.4.4.4 on Port-channel14
*Mar  1 00:04:47.578: OSPF: Rcv LS UPD from 10.4.4.4 on Port-channel14 length 156 LSA count 3
*Mar  1 00:04:47.578: OSPF: Synchronized with 10.4.4.4 on Port-channel14, state FULL
*Mar  1 00:04:47.578: %OSPF-5-ADJCHG: Process 1, Nbr 10.4.4.4 on Port-channel14 from LOADING to FULL, Loading Done
SWITCH-3560#sh system mtu

System MTU size is 1504 bytes
System Jumbo MTU size is 1504 bytes
Routing MTU size is 1504 bytes

• Proxy ARP (Cisco Page)
• To check whether an interface is using Proxy ARP or not.

  Rack10R6#sh ip int f0/0.146 | i ARP
    Proxy ARP is enabled
    Local Proxy ARP is disabled
  

• To enable/disable Proxy ARP on the interface

  Rack10R6(config)#int f0/0.146
  Rack10R6(config-subif)#no ip proxy-arp
  Rack10R6(config-subif)#do sh ip int f0/0.146 | i ARP
    Proxy ARP is disabled
    Local Proxy ARP is disabled
  

• To disable Proxy ARP from the global configuration (Proxy ARP is enabled by default)

  Rack10R6(config)#ip arp proxy disable
  Rack10R6(config)#do sh ip int f0/0.146 | i ARP
    Proxy ARP is disabled (Globally)
    Local Proxy ARP is disabled
  

When you configure switchport access vlan 20 on an interface:

• This will make the interface become a layer 2 switchport or non-routeable port.
• There is no 802.1q/ISL tagging involvement in this port as the traffic will be received and sent as untagged.
• Because there is no tagging involved, the port has no clue what VLAN this frame should be. Therefore, any packets passing this port is assumed to belong to the VLAN assigned to this port. In this case vlan 20.
• If you add the configuration with switchport mode access this will make the interface turned to nontrunking permanently and also negotiates or telling the other end of the link to be a nontrunk link. This interface will permanently become nontrunk interface regardless the other end is trunk or nontrunk link. With this in mind, if one port is configured with switchport mode access and other end port happens to be configured with switchport mode dynamic auto or switchport mode dynamic desirable

  sw3#sh run int f0/19
  Building configuration...
  
  Current configuration : 58 bytes
  !
  interface FastEthernet0/19
   switchport mode access
  end
  
  sw3#sh int f0/19 status
  
  Port      Name               Status       Vlan       Duplex  Speed Type
  Fa0/19                       connected    1          a-full  a-100 10/100BaseTX
  sw3#sh spann      
  
  VLAN0001
    Spanning tree enabled protocol ieee
    Root ID    Priority    32769
               Address     a8b1.d45b.3b80
               Cost        19
               Port        21 (FastEthernet0/19)
               Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
  
    Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
               Address     aca0.1644.b180
               Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
               Aging Time  300 sec
  
  Interface           Role Sts Cost      Prio.Nbr Type
  ------------------- ---- --- --------- -------- --------------------------------
  Fa0/19              Root FWD 19        128.21   P2p 
  
  sw3#sh int f0/19 sw
  Name: Fa0/19
  Switchport: Enabled
  Administrative Mode: static access
  Operational Mode: static access
  Administrative Trunking Encapsulation: negotiate
  Operational Trunking Encapsulation: native
  Negotiation of Trunking: Off
  Access Mode VLAN: 1 (default)
  Trunking Native Mode VLAN: 1 (default)
  Administrative Native VLAN tagging: enabled
  Voice VLAN: none
  Administrative private-vlan host-association: none
  Administrative private-vlan mapping: none
  Administrative private-vlan trunk native VLAN: none
  Administrative private-vlan trunk Native VLAN tagging: enabled
  Administrative private-vlan trunk encapsulation: dot1q
  Administrative private-vlan trunk normal VLANs: none
  Administrative private-vlan trunk associations: none
  Administrative private-vlan trunk mappings: none
  Operational private-vlan: none
  Trunking VLANs Enabled: ALL
  Pruning VLANs Enabled: 2-1001
  Capture Mode Disabled
  Capture VLANs Allowed: ALL
  
  Protected: false
  Unknown unicast blocked: disabled
  Unknown multicast blocked: disabled
  Appliance trust: none
  
  sw4#sh run int f0/19
  Building configuration...
  
  Current configuration : 34 bytes
  !
  interface FastEthernet0/19
  end
  
  sw4#sh int f0/19 status
  
  Port      Name               Status       Vlan       Duplex  Speed Type
  Fa0/19                       connected    1          a-full  a-100 10/100BaseTX
  sw4#sh spann 
  
  VLAN0001
    Spanning tree enabled protocol ieee
    Root ID    Priority    32769
               Address     a8b1.d45b.3b80
               This bridge is the root
               Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
  
    Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
               Address     a8b1.d45b.3b80
               Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
               Aging Time  300 sec
  
  Interface           Role Sts Cost      Prio.Nbr Type
  ------------------- ---- --- --------- -------- --------------------------------
  Fa0/19              Desg FWD 19        128.21   P2p 
  
  sw4#sh int f0/19 sw
  sw4#sh int f0/19 switchport
  Name: Fa0/19
  Switchport: Enabled
  Administrative Mode: dynamic auto
  Operational Mode: static access
  Administrative Trunking Encapsulation: negotiate
  Operational Trunking Encapsulation: native
  Negotiation of Trunking: On
  Access Mode VLAN: 1 (default)
  Trunking Native Mode VLAN: 1 (default)
  Administrative Native VLAN tagging: enabled
  Voice VLAN: none
  Administrative private-vlan host-association: none
  Administrative private-vlan mapping: none
  Administrative private-vlan trunk native VLAN: none
  Administrative private-vlan trunk Native VLAN tagging: enabled
  Administrative private-vlan trunk encapsulation: dot1q
  Administrative private-vlan trunk normal VLANs: none
  Administrative private-vlan trunk associations: none
  Administrative private-vlan trunk mappings: none
  Operational private-vlan: none
  Trunking VLANs Enabled: ALL
  Pruning VLANs Enabled: 2-1001
  Capture Mode Disabled
  Capture VLANs Allowed: ALL
  
  Protected: false
  Unknown unicast blocked: disabled
  Unknown multicast blocked: disabled
  Appliance trust: none
  

• You can actually peer this with a trunk port as configured below

  ! SW3
  interface FastEthernet0/19
   switchport access vlan 20
   switchport mode access
   spanning-tree portfast
   spanning-tree bpdufilter enable
  int vlan 10
   ip addr 10.10.10.3 255.255.255.0
  exit
  
  ! SW4
  interface FastEthernet0/19
   switchport trunk encapsulation dot1q
   switchport trunk native vlan 20
   switchport mode trunk
   switchport nonegotiate
   spanning-tree portfast trunk
  int vlan 10
   ip addr 10.10.10.4 255.255.255.0
  exit
  
  sw4#ping 10.10.10.3      
  
  Type escape sequence to abort.
  Sending 5, 100-byte ICMP Echos to 10.10.10.3, timeout is 2 seconds:
  !!!!!
  Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms
  

  Since switchport access vlan 20 will send and receive frame without tagging it and neither the other end (the trunk port), this peering will successfully send and receive frame and assume that it is belong to vlan 20.

Catalyst QOS Port-Based Policing Cannot Co-exist with Vlan-based Classification.

SW1#sh ver | i cisco

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

export@cisco.com.
cisco WS-C3560-24TS (PowerPC405) processor (revision D0) with 122880K/8184K bytes of memory.

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#
SW1(config)#no policy-map PM_QOS
SW1(config)#
SW1(config)#ip access-list ext ACL_IP
SW1(config-ext-nacl)# permit ip any any
SW1(config-ext-nacl)#exit
SW1(config)#
SW1(config)#class-map CM_IP
SW1(config-cmap)# match access-group name ACL_IP
SW1(config-cmap)#exit
SW1(config)#
SW1(config)#no policy-map PM_QOS
SW1(config)#policy-map PM_QOS
SW1(config-pmap)# class class-default
SW1(config-pmap-c)#  police 128000 125000 exceed-action drop
SW1(config-pmap-c)# exit
SW1(config-pmap)#exit
SW1(config)#
SW1(config)#int f0/1
SW1(config-if)# service-policy in PM_QOS
%QoS: policymap is not attachable to vlan-based interface
Service Policy attachment failed
SW1(config-if)#exit
SW1(config)#exit
SW1#
*Mar  1 09:26:17.854: %SYS-5-CONFIG_I: Configured from console by console
SW1#sh run int f0/1
Building configuration...

Current configuration : 81 bytes
!
interface FastEthernet0/1
 switchport access vlan 146
 mls qos vlan-based
end

SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#int f0/1
SW1(config-if)#no mls qos vlan-based
SW1(config-if)#^Z
SW1#
*Mar  1 09:26:35.545: %SYS-5-CONFIG_I: Configured from console by console
SW1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW1(config)#int f0/1
SW1(config-if)# service-policy in PM_QOS
SW1(config-if)#exit
SW1(config)#exit
SW1#
*Mar  1 09:26:47.550: %SYS-5-CONFIG_I: Configured from console by console
SW1#sh poli
SW1#sh policy-map int f0/1
 FastEthernet0/1 

  Service-policy input: PM_QOS

    Class-map: class-default (match-any)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any
        0 packets, 0 bytes
        5 minute rate 0 bps
SW1#

This article explains prefix-list in a very easy way to understand.

Legacy Frame Relay Traffic Shaping

int s0/0/0
 band 1536
 frame-relay traffic-shaping
 frame-relay interface-dlci 513
  class MC_513
 frame-relay interface-dlci 504
  class MC_504
 exit
exit

map-class frame-relay MC_513
 frame-relay cir 128000
 frame-relay bc  6400
 frame-relay be  51200
 frame-relay mincir 96000
 frame-relay adaptive-shaping becn
map-class frame-relay MC_504
 frame-relay cir 512000
 frame-relay bc  25600
 frame-relay be  0
 frame-relay mincir 384000
 frame-relay adaptive-shaping becn
exit

Class-Based Generic Traffic Shaping for Frame Relay

class-map CM_DLCI_513
 match fr-dlci 513
class-map CM_DLCI_504
 match fr-dlci 504
exit

policy-map PM_FRTS
 class CM_DLCI_513
  shape average 128000 6400 0
  shape adaptive 96000
 class CM_DLCI_504
  shape average 512000 25600 51200
  shape adaptive 384000
 exit
exit

int s0/0/0
 bandwidth 1536
 no frame-relay traffic-shaping
 service-policy output PM_FRTS
exit

MQC-based Frame Relay Traffic Shaping

int s0/0/0
 band 1536
 no frame-relay traffic-shaping
 frame-relay interface dlci 513
  class MC_513
 frame-relay interface dlci 504
  class MC_504
 exit
exit

map-class frame-relay MC_513
 service-policy out PM_513
map-class frame-relay MC_504
 service-policy out PM_504
exit

policy-map PM_513
 class class-default
  shape average 128000 6400 0
  shape adaptive 96000 ! automatically adapt to BECN
 exit
exit

policy-map PM_504
 class class-default
  shape average 512000 25600 51200
  shape adaptive 384000 ! automatically adapt to BECN
 exit
exit

Notes.
Bc = CIR (in bits per seconds) * (Tc/1000)
(Bc + Be) / (Tc/1000) = AIR

Example.

Bc = CIR (in bits per seconds) * (Tc/1000)
Bc = 512000 * 50/1000
Bc = 512000 * 0.05
Bc = 25600

(Bc + Be) / (Tc/1000) = AIR
(25600 + Be) / 0.05 = 1536000
Be = (1536000 * 0.05) – 25600
Be = 51200

Note – 30 December 2011.
For MQC-based Frame Relay Traffic Shaping, inside the policy-map, command shape average needs to be applied first.

ROUTER(config)#policy-map PM_513
ROUTER(config-pmap)# class class-default
ROUTER(config-pmap-c)#  shape adaptive 256000
Traffic Shaping must be enabled on this class before the command can be issued
ROUTER(config-pmap-c)# shape average 384000
ROUTER(config-pmap-c)#shape adaptive 256000

R5 (s0/0/0 – DLCI 504) — (DLCI 405 – s0/0/0) R4
R5 (f0/1 – DLCI 504) — (DLCI 405 – f0/0) R4

I’m trying to configure and mpls ldp between R4 and R5. The configuration is quite straightforward and I’m expecting to have two peering between R5-R4 via serial and fastethernet interfaces.

!R4

conf t

mpls ip
mpls label protocol ldp

int f0/0
 mpls ip
int s0/0/0
 mpls ip
exit

exit

!R5

conf t

mpls ip

int f0/1
 mpls ip
int s0/0/0
 mpls ip
exit

exit

Apparently the ldp peering is only happening between fastethernet interfaces. No adjacency between serial interfaces.

Rack13R5#sh mpls ldp nei
    Peer LDP Ident: 150.13.4.4:0; Local LDP Ident 150.13.5.5:0
        TCP connection: 150.13.4.4.646 - 150.13.5.5.18850
        State: Oper; Msgs sent/rcvd: 126/144; Downstream
        Up time: 01:24:48
        LDP discovery sources:
          FastEthernet0/1, Src IP addr: 183.13.45.4
        Addresses bound to peer LDP Ident:
          183.13.45.4     183.13.46.4     183.13.0.4      150.13.4.4    

Rack13R4#sh mpls ldp nei
    Peer LDP Ident: 150.13.5.5:0; Local LDP Ident 150.13.4.4:0
        TCP connection: 150.13.5.5.18850 - 150.13.4.4.646
        State: Oper; Msgs sent/rcvd: 144/126; Downstream
        Up time: 01:25:07
        LDP discovery sources:
          FastEthernet0/0, Src IP addr: 183.13.45.5
        Addresses bound to peer LDP Ident:
          183.13.105.5    183.13.45.5     183.13.0.5      150.13.5.5

Checking the ldp peering discovery resulting as below

Rack13R5#
Feb 10 23:23:11.183: ldp: Send ldp hello; Serial0/0/0, src/dst 183.13.0.5/224.0.0.2, inst_id 0
Feb 10 23:23:11.375: ldp: Send ldp hello; FastEthernet0/1, src/dst 183.13.45.5/224.0.0.2, inst_id 0
Feb 10 23:23:11.819: ldp: Rcvd ldp hello; FastEthernet0/1, from 183.13.45.4 (150.13.4.4:0), intf_id 0, opt 0xC
Rack13R5#
Feb 10 23:23:15.275: ldp: Send ldp hello; FastEthernet0/1, src/dst 183.13.45.5/224.0.0.2, inst_id 0
Feb 10 23:23:15.871: ldp: Rcvd ldp hello; FastEthernet0/1, from 183.13.45.4 (150.13.4.4:0), intf_id 0, opt 0xC
Feb 10 23:23:15.975: ldp: Send ldp hello; Serial0/0/0, src/dst 183.13.0.5/224.0.0.2, inst_id 0
Rack13R5#
Feb 10 23:23:19.999: ldp: Send ldp hello; FastEthernet0/1, src/dst 183.13.45.5/224.0.0.2, inst_id 0
Feb 10 23:23:20.655: ldp: Rcvd ldp hello; FastEthernet0/1, from 183.13.45.4 (150.13.4.4:0), intf_id 0, opt 0xC
Feb 10 23:23:20.911: ldp: Send ldp hello; Serial0/0/0, src/dst 183.13.0.5/224.0.0.2, inst_id 0
Rack13R5#
Feb 10 23:23:23.751: ldp: Data received!
Feb 10 23:23:24.527: ldp: Send ldp hello; FastEthernet0/1, src/dst 183.13.45.5/224.0.0.2, inst_id 0

I can see R5 is sending ldp hello via f0/1 and s0/0/0. However, R5 only receiving ldp hello back from f0/1 but not s0/0/0.

What’s going on with s0/0/0 sending packet to 224.0.0.2?

Let’s check it out!

Rack13R5#sh access-list 100
Extended IP access list 100
    10 permit ip 183.13.0.0 0.0.0.255 host 224.0.0.2 (140 matches)
Rack13R5#debug ip packet 100
IP packet debugging is on for access list 100
Rack13R5#
Feb 10 23:25:37.435: IP: s=183.13.0.5 (local), d=224.0.0.2 (Serial0/0/0), len 62,
sending broad/multicast
Feb 10 23:25:37.435: IP: s=183.13.0.5 (local), d=224.0.0.2 (Serial0/0/0), len 62,
encapsulation failed
Rack13R5#
Feb 10 23:25:41.859: IP: s=183.13.0.5 (local), d=224.0.0.2 (Serial0/0/0), len 62,
sending broad/multicast
Feb 10 23:25:41.859: IP: s=183.13.0.5 (local), d=224.0.0.2 (Serial0/0/0), len 62,
encapsulation failed

Apparently s0/0/0 having problem sending broadcast/multicast packet destined to 224.0.0.2.

The ultimate question is, does R5 s0/0/0 DLCI 504 and R4 s0/0/0 DLCI 405 configured as broadcast or non-broadcast?

Rack13R5#sh frame-relay map 504
Serial0/0/0 (up): ip 183.13.0.4 dlci 504(0x1F8,0x7C80), static,
              CISCO, status defined, active

Rack13R4#sh frame-relay map 405
Serial0/0/0 (up): ip 183.13.0.5 dlci 405(0x195,0x6450), static,
              CISCO, status defined, active

Apparently they’re not. Therefore, the fix should be pretty should be easy

! R4
int s0/0/0
 frame-relay map ip 183.13.0.5 405 broadcast
! R5
int s0/0/0
 frame-relay map ip 183.13.0.4 504 broadcast

Rack13R5#debug mpls ldp transport events
LDP transport events debugging is on
Rack13R5#
Feb 10 23:32:48.795: ldp: Send ldp hello; FastEthernet0/1, src/dst 183.13.45.5/224.0.0.2, inst_id 0
Rack13R5#
Feb 10 23:32:50.699: ldp: Rcvd ldp hello; Serial0/0/0, from 183.13.0.4 (150.13.4.4:0), intf_id 0, opt 0xC
Feb 10 23:32:51.695: ldp: Send ldp hello; Serial0/0/0, src/dst 183.13.0.5/224.0.0.2, inst_id 0
Rack13R5#
Feb 10 23:32:51.847: ldp: Rcvd ldp hello; FastEthernet0/1, from 183.13.45.4 (150.13.4.4:0), intf_id 0, opt 0xC
Feb 10 23:32:52.687: ldp: Send ldp hello; FastEthernet0/1, src/dst 183.13.45.5/224.0.0.2, inst_id 0
Rack13R5#
Feb 10 23:32:55.387: ldp: Rcvd ldp hello; Serial0/0/0, from 183.13.0.4 (150.13.4.4:0), intf_id 0, opt 0xC
Feb 10 23:32:55.663: ldp: Rcvd ldp hello; FastEthernet0/1, from 183.13.45.4 (150.13.4.4:0), intf_id 0, opt 0xC
Feb 10 23:32:55.911: ldp: Data received!
Feb 10 23:32:56.119: ldp: Send ldp hello; Serial0/0/0, src/dst 183.13.0.5/224.0.0.2, inst_id 0
Rack13R5#
Feb 10 23:32:56.887: ldp: Send ldp hello; FastEthernet0/1, src/dst 183.13.45.5/224.0.0.2, inst_id 0
Rack13R5#
Feb 10 23:32:59.443: ldp: Rcvd ldp hello; FastEthernet0/1, from 183.13.45.4 (150.13.4.4:0), intf_id 0, opt 0xC
Feb 10 23:32:59.987: ldp: Rcvd ldp hello; Serial0/0/0, from 183.13.0.4 (150.13.4.4:0), intf_id 0, opt 0xC
Rack13R5#
Feb 10 23:33:00.795: ldp: Send ldp hello; FastEthernet0/1, src/dst 183.13.45.5/224.0.0.2, inst_id 0
Feb 10 23:33:00.971: ldp: Send ldp hello; Serial0/0/0, src/dst 183.13.0.5/224.0.0.2, inst_id 0

Rack13R5#sh mpls ldp nei
    Peer LDP Ident: 150.13.4.4:0; Local LDP Ident 150.13.5.5:0
        TCP connection: 150.13.4.4.646 - 150.13.5.5.18850
        State: Oper; Msgs sent/rcvd: 140/158; Downstream
        Up time: 01:37:04
        LDP discovery sources:
          FastEthernet0/1, Src IP addr: 183.13.45.4
          Serial0/0/0, Src IP addr: 183.13.0.4
        Addresses bound to peer LDP Ident:
          183.13.45.4     183.13.46.4     183.13.0.4      150.13.4.4 

Rack13R4#sh mpls ldp nei
    Peer LDP Ident: 150.13.5.5:0; Local LDP Ident 150.13.4.4:0
        TCP connection: 150.13.5.5.18850 - 150.13.4.4.646
        State: Oper; Msgs sent/rcvd: 158/140; Downstream
        Up time: 01:37:19
        LDP discovery sources:
          FastEthernet0/0, Src IP addr: 183.13.45.5
          Serial0/0/0, Src IP addr: 183.13.0.5
        Addresses bound to peer LDP Ident:
          183.13.105.5    183.13.45.5     183.13.0.5      150.13.5.5

This frame relay troubleshooting hit me big time. I’ve spent about 7.5 hours just to get this up and running, apparently, due to simple mistake.

The network configured as below.

R5 (s0/0/0 - DLCI 504) --- ( DLCI 405 - s0/0/0) R4
R5 (s0/0/0 - DLCI 513) --- ( DLCI 315 - s1/1) R3

Rack13R5#sh run int s0/0/0
interface Serial0/0/0
 ip address 183.13.0.5 255.255.255.0
 ip pim sparse-dense-mode
 encapsulation frame-relay
 ip ospf network broadcast
 ip ospf dead-interval minimal hello-multiplier 3
 ip ospf priority 255
 frame-relay map ip 183.13.0.3 513 broadcast
 frame-relay map ip 183.13.0.4 504 broadcast
 no frame-relay inverse-arp
end

Rack13R4(config-router)#do sh run int s0/0/0
interface Serial0/0/0
 ip address 183.13.0.4 255.255.255.0
 encapsulation frame-relay
 ip ospf network broadcast
 ip ospf dead-interval minimal hello-multiplier 3
 ip ospf priority 0
 no keepalive
 frame-relay map ip 183.13.0.5 405 broadcast
 frame-relay map ip 183.13.0.3 405
 no frame-relay inverse-arp
end

!R3
interface Serial1/1
 no shutdown
 ip address 183.13.0.3 255.255.255.0
 ip pim sparse-dense-mode
 encapsulation frame-relay
 ip ospf network broadcast
 ip ospf dead-interval minimal hello-multiplier 3
 ip ospf priority 0
 frame-relay map ip 183.13.0.4 315
 frame-relay map ip 183.13.0.5 315 broadcast
 no frame-relay inverse-arp
end

R5-R3 connection is fine but I cannot ping R4 FR IP Addr 183.13.0.4 from R5 and R3.

Rack13R5#ping 183.13.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 183.13.0.4, timeout is 2 seconds:
..
Success rate is 0 percent (0/2)

debug ip packet shown below.

Rack13R5#x
*Feb  5 22:09:48.171: IP: s=183.13.0.5 (local), d=183.13.0.4 (Serial0/0/0), len 100, sending
*Feb  5 22:09:48.171:     ICMP type=8, code=0
*Feb  5 22:09:48.171: IP: s=183.13.0.5 (local), d=183.13.0.4 (Serial0/0/0), len 100,
encapsulation failed
*Feb  5 22:09:48.171:     ICMP type=8, code=0
*Feb  5 22:09:50.171: IP: s=183.13.0.5 (local), d=183.13.0.4 (Serial0/0/0), len 100, sending
*Feb  5 22:09:50.171:     ICMP type=8, code=0
*Feb  5 22:09:50.171: IP: s=183.13.0.5 (local), d=183.13.0.4 (Serial0/0/0), len 100,
encapsulation failed
*Feb  5 22:09:50.171:     ICMP type=8, code=0

Apparently the L2 is having problem. Let’s check the DLCI mapping.

Rack13R5#sh frame-relay map
Serial0/0/0 (up): ip 183.13.0.4 dlci 504(0x1F8,0x7C80), static,
              broadcast,
              CISCO, status defined, inactive
Serial0/0/0 (up): ip 183.13.0.3 dlci 513(0x201,0x8010), static,
              broadcast,
              CISCO, status defined, active

Rack13R5#sh frame-relay pvc 504

PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)

DLCI = 504, DLCI USAGE = LOCAL, PVC STATUS = INACTIVE, INTERFACE = Serial0/0/0

  input pkts 0             output pkts 24           in bytes 0
  out bytes 1918           dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 26        out bcast bytes 1966
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 00:04:16, last time pvc status changed 00:04:06

Gotcha, the DLCI 504 is inactive. This is why R5 cannot send L3 packets to R4 via DLCI 504 due to the PVC is INACTIVE. This would also mean that R3 won’t be able to ping R4 as R5 is the hub to reach R4.

Rack13R3#ping 183.13.0.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 183.13.0.4, timeout is 2 seconds:
..
Success rate is 0 percent (0/2)

Let’s see whether R4 can send the packets out to R5 via DLCI 405.

Rack13R4#ping 183.13.0.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 183.13.0.5, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Rack13R4#
Feb  5 22:04:55.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb  5 22:04:55.270:     ICMP type=8, code=0
Feb  5 22:04:55.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb  5 22:04:55.270:     ICMP type=8, code=0
Feb  5 22:04:57.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb  5 22:04:57.270:     ICMP type=8, code=0
Feb  5 22:04:57.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb  5 22:04:57.270:     ICMP type=8, code=0
Feb  5 22:04:59.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb  5 22:04:59.270:     ICMP type=8, code=0
Feb  5 22:04:59.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb  5 22:04:59.270:     ICMP type=8, code=0
Feb  5 22:05:01.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb  5 22:05:01.270:     ICMP type=8, code=0
Feb  5 22:05:01.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb  5 22:05:01.270:     ICMP type=8, code=0
Feb  5 22:05:03.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100, sending
Feb  5 22:05:03.270:     ICMP type=8, code=0
Feb  5 22:05:03.270: IP: s=183.13.0.4 (local), d=183.13.0.5 (Serial0/0/0), len 100,
sending full packet
Feb  5 22:05:03.270:     ICMP type=8, code=0

Rack13R4#sh frame-relay pvc 405

PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)

DLCI = 405, DLCI USAGE = LOCAL, PVC STATUS = STATIC, INTERFACE = Serial0/0/0

  input pkts 2606          output pkts 38365        in bytes 197551
  out bytes 3061869        dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 37556     out bcast bytes 3011444
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 1000 bits/sec, 3 packets/sec
  pvc create time 06:34:40, last time pvc status changed 00:12:34

Looks like R4 is just fine, sending packets out to R5.

INACTIVE status means the PVC is not up. The reason can be the local DLCI mapping issue or the other end is not up. Obviously, the local DLCI mapping is just fine. So, this could be an issue on the other end. (well, there is an FR switch in the middle but I really doubt it causing the problem. I did reload and check the config on the FR switch, all ok)

Rack13R5#sh run int s0/0/0
interface Serial0/0/0
 ip address 183.13.0.5 255.255.255.0
 ip pim sparse-dense-mode
 encapsulation frame-relay
 ip ospf network broadcast
 ip ospf dead-interval minimal hello-multiplier 3
 ip ospf priority 255
 frame-relay map ip 183.13.0.3 513 broadcast
 frame-relay map ip 183.13.0.4 504 broadcast
 no frame-relay inverse-arp
end

If we look the PVC 504 on R5, we could see that the input packets is 0, it means that pvc 504 is not receiving LMI from 405.

Rack13R5#sh frame-relay pvc 504

PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)

DLCI = 504, DLCI USAGE = LOCAL, PVC STATUS = INACTIVE, INTERFACE = Serial0/0/0

  input pkts 0             output pkts 24           in bytes 0
  out bytes 1918           dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 26        out bcast bytes 1966
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 00:04:16, last time pvc status changed 00:04:06

Checking the PVC 405 on R4 again

Rack13R4#sh frame-relay pvc 405

PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)

DLCI = 405, DLCI USAGE = LOCAL, PVC STATUS = STATIC, INTERFACE = Serial0/0/0

  input pkts 2606          output pkts 38365        in bytes 197551
  out bytes 3061869        dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 37556     out bcast bytes 3011444
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 1000 bits/sec, 3 packets/sec
  pvc create time 06:34:40, last time pvc status changed 00:12:34

The DLCI 405 is showing STATIC. What STATIC? I know INACTIVE, ACTIVE, and DELETED. But STATIC? Cisco Doco in here and here shows that STATIC means that the PVC is statically created thus no LMI/keepalive being sent.

Checking the config again on R4, you will find the culprit keepalive was disabled.

Rack13R4(config-router)#do sh run int s0/0/0
interface Serial0/0/0
 ip address 183.13.0.4 255.255.255.0
 encapsulation frame-relay
 ip ospf network broadcast
 ip ospf dead-interval minimal hello-multiplier 3
 ip ospf priority 0
 no keepalive <<<===
 frame-relay map ip 183.13.0.5 405 broadcast
 frame-relay map ip 183.13.0.3 405
 no frame-relay inverse-arp
end

Rack13R4(config-if)#do sh int s0/0/0
Serial0/0/0 is up, line protocol is up
  Hardware is GT96K Serial
  Internet address is 183.13.0.4/24
  MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation FRAME-RELAY, loopback not set
  Keepalive not set <<<===
  CRC checking enabled
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  FR SVC disabled, LAPF state down
  Broadcast queue 0/64, broadcasts sent/dropped 11470/0, interface broadcasts 11470
  Last input 01:36:59, output 00:35:57, output hang never
  Last clearing of "show interface" counters 01:37:00
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1158 kilobits/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     1 packets input, 53 bytes, 0 no buffer

Enabling the keepalive (default 10 seconds) brings back both the PVC 405 and 504 to ACTIVE.

Rack13R4(config-if)#int s0/0/0
Rack13R4(config-if)#keepalive ?
  <0-30>  Keepalive period (default 10 seconds)
  

Rack13R4(config-if)#keepalive
Rack13R4(config-if)#
Feb  6 22:52:30.611: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0,
changed state to down
Rack13R4(config-if)#
Feb  6 22:52:50.611: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0,
changed state to up
Rack13R4(config-if)#do sh fram pvc 405

PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)

DLCI = 405, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0

  input pkts 0             output pkts 0            in bytes 0
  out bytes 0              dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 0         out bcast bytes 0
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  pvc create time 00:36:53, last time pvc status changed 00:00:20

Rack13R5(config-if)#do sh fram pvc 504

PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)

DLCI = 504, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0

  input pkts 6446          output pkts 6446         in bytes 540482
  out bytes 540590         dropped pkts 0           in pkts dropped 0
  out pkts dropped 0                out bytes dropped 0
  in FECN pkts 0           in BECN pkts 0           out FECN pkts 0
  out BECN pkts 0          in DE pkts 0             out DE pkts 0
  out bcast pkts 0         out bcast bytes 0
  5 minute input rate 2000 bits/sec, 3 packets/sec
  5 minute output rate 2000 bits/sec, 3 packets/sec
  pvc create time 01:13:54, last time pvc status changed 00:35:42

We can also see that PVC 504 receiving packets.

Initially, I could point the issue that R5 is not receiving LMI. But I guess that’s not good enough. Knowing that PVC STATIC is basically statically assigned DLCI thus won’t send any LMI out would save a lot of my time.

On the other hand, I also need to fix the OSPF issue on this FR by applying non-broadcast network type as shown below.