I’d been searching everywhere but I guess I didn’t get the right keyword to match. Apparently, multicast ipsec were the keywords as I found it on Cisco Doco.

IPsec Deployment with Point-to-Point GRE

Generic Routing Encapsulation (GRE) is often deployed with IPsec for several reasons, including the following:

• IPsec Direct Encapsulation supports unicast IP only. If network layer protocols other than IP are to be supported, an IP encapsulation method must be chosen so that those protocols can be transported in IP packets.
• IPmc [IP Multicast] is not supported with IPsec Direct Encapsulation. IPsec was created to be a security protocol between two and only two devices, so a service such as multicast is problematic. An IPsec peer encrypts a packet so that only one other IPsec peer can successfully perform the de-encryption. IPmc is not compatible with this mode of operation.

Until the introduction of IPsec Virtual Tunnel Interface (VTI), IPsec tunnels were not logical tunnel interfaces for routing purposes. A point-to-point (p2p) GRE tunnel, on the other hand, is a logical router interface for purposes of forwarding IP (or any other network protocol) traffic. A tunnel interface can appear as a next-hop interface in the routing table.

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/V3PNIPmc.html#wp349808

The max-reserved-bandwidth command is not supported in Cisco IOS Release 12.2SR or in 12.2SX. It is supported in 12.4T, but only up to the 12.4(20)T release in which HQF functionality was integrated.

http://www.cisco.com/en/US/docs/ios/qos/command/reference/qos_m1.html#wp1054626

This investigation by Pavel Bykov explains it all.

Verification

!!! R4

R4#sh ppp multilink
Multilink1
  Bundle name: R5
  Remote Endpoint Discriminator: [1] R5
  Local Endpoint Discriminator: [1] R4
  Bundle up for 00:15:12, total bandwidth 128, load 33/255
  Receive buffer limit 12000 bytes, frag timeout 1000 ms
  Interleaving enabled
    0/0 fragments/bytes in reassembly list
    0 lost fragments, 0 reordered
    0/0 discarded fragments/bytes, 0 lost received
    0x5DE9 received sequence, 0x8244 sent sequence
  Member links: 1 active, 0 inactive (max not set, min not set)
    Se0/1/0, since 00:15:12, 160 weight, 152 frag size
No inactive multilink interfaces

R4#sh int s0/1/0
Serial0/1/0 is up, line protocol is up
  Hardware is GT96K Serial
  MTU 1500 bytes, BW 128 Kbit/sec, DLY 20000 usec,
     reliability 255/255, txload 13/255, rxload 11/255
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Multilink1, loopback not set
  Keepalive set (10 sec)
  CRC checking enabled
  Last input 00:00:09, output 00:00:04, output hang never
  Last clearing of "show interface" counters 01:37:19
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 6000 bits/sec, 11 packets/sec
  30 second output rate 7000 bits/sec, 11 packets/sec
     74352 packets input, 5015177 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     85309 packets output, 7605592 bytes, 0 underruns
     0 output errors, 0 collisions, 9 interface resets
     1 unknown protocol drops
     1 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     19 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

R4#sh int multilink 1
Multilink1 is up, line protocol is up
  Hardware is multilink group interface
  Internet address is 155.13.45.4/24
  MTU 1500 bytes, BW 128 Kbit/sec, DLY 100000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open
  Open: IPCP, CDPCP, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 2 seconds on reset
  Last input 00:00:01, output never, output hang never
  Last clearing of "show interface" counters 01:37:46
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 539
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/539 (size/max total/threshold/drops)
     Conversations  0/3/32 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 96 kilobits/sec
  5 minute input rate 0 bits/sec, 6 packets/sec
  5 minute output rate 1000 bits/sec, 0 packets/sec
     27574 packets input, 1624852 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     23371 packets output, 4200099 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 unknown protocol drops
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

!!! R5

R5#sh ppp multilink
Multilink1
  Bundle name: R4
  Remote Endpoint Discriminator: [1] R4
  Local Endpoint Discriminator: [1] R5
  Bundle up for 00:16:40, total bandwidth 128, load 1/255
  Receive buffer limit 12000 bytes, frag timeout 1000 ms
  Interleaving enabled
    0/0 fragments/bytes in reassembly list
    0 lost fragments, 0 reordered
    0/0 discarded fragments/bytes, 0 lost received
    0x8666 received sequence, 0x61E7 sent sequence
  Member links: 1 active, 0 inactive (max not set, min not set)
    Se0/1/0, since 00:16:40, 160 weight, 152 frag size
No inactive multilink interfaces

R5#sh int s0/1/0
Serial0/1/0 is up, line protocol is up
  Hardware is GT96K Serial
  MTU 1500 bytes, BW 128 Kbit/sec, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 3/255
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Multilink1, loopback not set
  Keepalive set (10 sec)
  CRC checking enabled
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 01:38:33
  Input queue: 0/75/1/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 2000 bits/sec, 5 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
     86437 packets input, 7687360 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     75447 packets output, 5091445 bytes, 0 underruns
     0 output errors, 0 collisions, 6 interface resets
     0 unknown protocol drops
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     14 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

R5#sh int multilink 1
Multilink1 is up, line protocol is up
  Hardware is multilink group interface
  Internet address is 155.13.45.5/24
  MTU 1500 bytes, BW 128 Kbit/sec, DLY 100000 usec,
     reliability 255/255, txload 5/255, rxload 7/255
  Encapsulation PPP, LCP Open, multilink Open
  Open: IPCP, CDPCP, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 2 seconds on reset
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters 01:39:01
  Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/0 (size/max total/threshold/drops)
     Conversations  0/1/32 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 96 kilobits/sec
  5 minute input rate 4000 bits/sec, 3 packets/sec
  5 minute output rate 3000 bits/sec, 3 packets/sec
     25132 packets input, 4064849 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     29325 packets output, 1916796 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 unknown protocol drops
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

Source Using Multilink PPP over Serial Interface Links

Do not use the neighbor next-hop-self command to modify the next hop attribute for a route reflector when this feature is enabled for a route reflector client. Using the neighbor next-hop-self command on the route reflector will modify next hop attributes only for routes that are learned from eBGP peers and not the intended routes that are being reflected from the route reflector clients. To modify the next hop attribute when reflecting a route, use an outbound route map.

There are two ways to make these addresses available throughout the OSPF domain.

The first one is to include this to the OSPF process as below

int lo9
 ip addr 9.9.9.9 255.255.255.255
 ip ospf 1 a 2
int lo99
 ip addr 99.99.99.99 255.255.255.0
 ip ospf 1 a 2
exit

or you can use the good-old network command

int lo9
 ip addr 9.9.9.9 255.255.255.255
int lo99
 ip addr 99.99.99.99 255.255.255.0
exit

router ospf 1
 net 9.9.9.9 0.0.0.0 a 2
 net 99.99.99.99 0.0.0.0 a 2
exit

Interestingly, we’re seeing a network type as LOOPBACK and it is treated as a stub host.

Rack1SW3#sh ip ospf int lo9
Loopback9 is up, line protocol is up
  Internet Address 9.9.9.9/32, Area 2
  Process ID 1, Router ID 150.1.9.9, Network Type LOOPBACK, Cost: 1
  Enabled by interface config, including secondary ip addresses
  Loopback interface is treated as a stub Host
Rack1SW3#sh ip ospf int lo99
Loopback99 is up, line protocol is up
  Internet Address 99.99.99.99/24, Area 2
  Process ID 1, Router ID 150.1.9.9, Network Type LOOPBACK, Cost: 1
  Enabled by interface config, including secondary ip addresses
  Loopback interface is treated as a stub Host

More interestingly, as we know that those addresses are advertised as stub, therefore we should see those addresses were advertised as /32 subnet.

Rack1R5(config-router)#do sh ip route 9.9.9.9
Routing entry for 9.9.9.9/32
  Known via "ospf 1", distance 110, metric 67, type inter area
  Last update from 155.1.0.3 on Serial0/0, 00:00:18 ago
  Routing Descriptor Blocks:
  * 155.1.0.3, from 150.1.3.3, 00:00:18 ago, via Serial0/0
      Route metric is 67, traffic share count is 1

Rack1R5(config-router)#do sh ip route 99.99.99.99
Routing entry for 99.99.99.99/32
  Known via "ospf 1", distance 110, metric 67, type inter area
  Last update from 155.1.0.3 on Serial0/0, 00:03:45 ago
  Routing Descriptor Blocks:
  * 155.1.0.3, from 150.1.3.3, 00:03:45 ago, via Serial0/0
      Route metric is 67, traffic share count is 1

NOTE. We can actually make 99.99.99.99/24 advertised as /24 by changing the OSPF network type as below

int lo99
 ip addr 99.99.99.99 255.255.255.0
 ip ospf network-type point-to-point
exit

Also, from the above result, we can see that the area type is Inter Area, which mean it is advertised as OSPF Inter Area

Rack1R5(config-router)#do sh ip route | i _9.
     99.0.0.0/32 is subnetted, 1 subnets
O IA    99.99.99.99 [110/67] via 155.1.0.3, 00:17:19, Serial0/0
     9.0.0.0/32 is subnetted, 1 subnets
O IA    9.9.9.9 [110/67] via 155.1.0.3, 00:13:57, Serial0/0

From the above examples, we know that:
1. OSPF network type loopback will be advertised as stub (/32), and
2. will have route type as inter area.

The second example to make these addresses available throughout the OSPF domain is to redistribute these addresses into OSPF process.

int lo9
 ip addr 9.9.9.9 255.255.255.255
int lo99
 ip addr 99.99.99.99 255.255.255.0
exit

router ospf 1
 redistribute connected subnets
exit

If we try to see whether these interfaces are in the OSPF process, that’s just not possible. The reason is that these two addresses is not actually included int the OSPF process yet we only redistribute it into OSPF process. Just like any other routing protocol that we redistribute into OSPF domain. Check out the example below.

Rack1SW3#sh ip ospf int lo9
%OSPF: OSPF not enabled on Loopback9
Rack1SW3#sh ip ospf int lo99
%OSPF: OSPF not enabled on Loopback99

However, remember that when we redistribute something into OSPF it will have:
1. LSA type 7, which will be injected by ASBR, and which eventually will be translated to

Rack1SW3#sh ip ospf d self-originate

            OSPF Router with ID (150.1.9.9) (Process ID 1)
<ommitted>…
		Type-7 AS External Link States (Area 2)

Link ID         ADV Router      Age         Seq#       Checksum Tag
9.9.9.9         150.1.9.9       434         0x80000001 0x0063BB 0
99.99.99.0      150.1.9.9       434         0x80000001 0x000910 0
<ommitted>...

2. LSA type 5 by ABR.

Rack1R5#sh ip ospf d          

            OSPF Router with ID (150.1.5.5) (Process ID 1)
<ommitted>...
                Type-5 AS External Link States

Link ID         ADV Router      Age         Seq#       Checksum Tag
9.9.9.9         150.1.6.6       588         0x80000001 0x001F10 0
99.99.99.0      150.1.6.6       599         0x80000001 0x00C464 0
<ommitted>...

3. Route type as External type 2 (E2), and it maintains its subnet information and metric (by default is 20).

Rack1R5#sh ip route 9.9.9.9
Routing entry for 9.9.9.9/32
  Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 67
  Last update from 155.1.0.3 on Serial0/0, 00:14:36 ago
  Routing Descriptor Blocks:
  * 155.1.0.3, from 150.1.6.6, 00:14:36 ago, via Serial0/0
      Route metric is 20, traffic share count is 1

Rack1R5#sh ip route 99.99.99.99
Routing entry for 99.99.99.0/24
  Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 67
  Last update from 155.1.0.3 on Serial0/0, 00:14:42 ago
  Routing Descriptor Blocks:
  * 155.1.0.3, from 150.1.6.6, 00:14:42 ago, via Serial0/0
      Route metric is 20, traffic share count is 1

Note. We can change this default behaviour E2 with command redistribute connected subnets metric-type 1 which, unlike E2, E1 will change its metric information every time it passes L3 device.

Bart Bunting, right, of Australia, and his guide, Nathan Chivers, ski in the men’s giant slalom visually impaired event during the Winter Paralympic Games in Whistler, British Columbia on March 16, 2010. (AP Photo/The Canadian Press, Jonathan Hayward)

More about Bart Bunting.

Picture taken from here

ip access-list ext TEST
 permit ip host 1.1.1.1 host 2.2.2.2
exit

int f0/0
 ip addr 1.1.1.1 255.255.255.0
 ip access-group TEST out

This will allow traffic

FROM 1.1.1.1 going OUT TO 2.2.2.2.

If we change the direction from out to in as the example below,

ip access-list ext TEST
 permit ip host 2.2.2.2 host 1.1.1.1
exit

int f0/0
 ip addr 1.1.1.1 255.255.255.0
 ip access-group TEST in

this will allow traffic

Coming IN FROM 2.2.2.2 TO 1.1.1.1

in SVI or vlan interface, this has a slight different understanding.

ip access-list ext TEST
 permit ip host 1.1.1.1 host 2.2.2.2
exit

int vlan100
 ip addr 1.1.1.1 255.255.255.0
 ip access-group TEST in

The above example will allow traffic

Coming IN FROM 1.1.1.1 TO 2.2.2.2

While the example below

ip access-list ext TEST
 permit ip host 1.1.1.1 host 2.2.2.2
exit

int vlan100
 ip addr 2.2.2.2 255.255.255.0
 ip access-group TEST out

will allow traffic

FROM 1.1.1.1 going OUT TO 2.2.2.2

From the above understanding, we cannot do redistribution within stub area as redistribution will create External LSA (type 5). Again, a stub area is where External LSA (type 5) and ASBR summary LSA (type 4) are not being flooded.

We might all know that we have not-so-stubby-area (NSSA) if we want to use redistribution in stub area. But, allow me to post my experience on how I get this understanding.

I have a stub area. The lab says that:
1. Create a loopback address and redistribute this to OSPF
2. Change the area into NSSA.

Task 1 was damn easy! I rushed to my keyboard and typing madly.

int lo10
 ip addr 10.10.10.10 255.255.255.255
router ospf 1
 redistribute connected subnets

Wouldn’t be that hard, would it? until I saw this.

*Mar  1 02:06:04.183: %OSPF-4-ASBR_WITHOUT_VALID_AREA: Router is currently an 
ASBR while having only one area which is a stub area

Huh, what was that? Googling only gives me this
http://www.cisco.com/en/US/docs/ios/12_2sx/system/messages/sm2sx06.html

Error Message    %OSPF-4-ASBR_WITHOUT_VALID_AREA: Router is currently an ASBR while having only one area which is a stub area
Explanation    An ASBR must be attached to an area which can carry AS external or NSSA LSAs.

Recommended Action    Make the area to which the router is attached into an NSSA or regular area.

I was 100% sure that my routers were not ASBR, I don’t believe that another router / switches leaking its L2 somewhere, and it is a Stub area. It took me about 1 hour just to make sure of this, for several times. I’ve even reloaded the router and switches not knowing where the problem was.

Recommended action advised me to change the router into NSSA or regular area. NO! I don’t want that! I want my stub to be able to redistribute my lo10.

I checked the lab and it says 2. Change the area into NSSA. Then I grabbed my Routing TCP/IP vol1 to check why.

Ok, again, stub area won’t flood LSA Type 5, therefore I could not do my redistribution.

(and No, I cannot use ip ospf 1 are 3 command in my lo10 as the lab clearly says to redistribute).

Changed my area into NSSA fixed the issue. Good lesson learned.

ip access-list ext MORE_THAN_1_PORTS
 permit ip any any eq 1
 permit ip any any eq 2
 permit ip any any eq 3
 permit ip any any eq 4
 permit ip any any eq 5
 permit ip any any eq 6
 permit ip any any eq 7
 permit ip any any eq 8
 permit ip any any eq 9
 permit ip any any eq 10
exit

However, there is a limit on how many ports you can put in one line. It cannot exceed more than 10 ports.

iRack1R1(config-ext-nacl)#$any 10.65.2.0  0.0.0.255 eq 1 2 3 4 5 6 7 8 9 ?
  <0-65535>    Port number
  ack          Match on the ACK bit
  bgp          Border Gateway Protocol (179)
  chargen      Character generator (19)
  cmd          Remote commands (rcmd, 514)
  daytime      Daytime (13)
  discard      Discard (9)
  domain       Domain Name Service (53)
  drip         Dynamic Routing Information Protocol (3949)
  dscp         Match packets with given dscp value
  echo         Echo (7)
  established  Match established connections
  exec         Exec (rsh, 512)
  fin          Match on the FIN bit
  finger       Finger (79)
  ftp          File Transfer Protocol (21)
  ftp-data     FTP data connections (20)
  gopher       Gopher (70)
  hostname     NIC hostname server (101)
  ident        Ident Protocol (113)
  irc          Internet Relay Chat (194)
  klogin       Kerberos login (543)

Rack1R1(config-ext-nacl)#$any 10.65.2.0  0.0.0.255 eq 1 2 3 4 5 6 7 8 9 10 ?
  ack          Match on the ACK bit
  dscp         Match packets with given dscp value
  established  Match established connections
  fin          Match on the FIN bit
  log          Log matches against this entry
  log-input    Log matches against this entry, including input interface
  match-all    Match if all specified flags are present
  match-any    Match if any specified flag is present
  option       Match packets with given IP Options value
  precedence   Match packets with given precedence value
  psh          Match on the PSH bit
  reflect      Create reflexive access list entry
  rst          Match on the RST bit
  syn          Match on the SYN bit
  time-range   Specify a time-range
  tos          Match packets with given TOS value
  ttl          Match packets with given TTL value
  urg          Match on the URG bit
  

Rack1R1(config-ext-nacl)#$any 10.65.2.0  0.0.0.255 eq 1 2 3 4 5 6 7 8 9 10 11
permit tcp any 10.65.2.0  0.0.0.255 eq 1 2 3 4 5 6 7 8 9 10 11
                                                            ^
% Invalid input detected at '^' marker.

It’s good experience to know that my R1 (spoke) is keep informing that its adjacency to R2, R3, and R4 is failing.

Mar 16 01:09:19.286: %OSPF-5-ADJCHG: Process 1, Nbr 150.5.3.3 on Serial0/0 from DOWN to DOWN, Neighbor Down: Ignore timer expired
Mar 16 01:11:39.220: %OSPF-5-ADJCHG: Process 1, Nbr 150.5.3.3 on Serial0/0 from EXSTART to DOWN, Neighbor Down: Too many retransmissions
Mar 16 01:11:49.408: %OSPF-5-ADJCHG: Process 1, Nbr 150.5.2.2 on Serial0/0 from EXSTART to DOWN, Neighbor Down: Too many retransmissions
Mar 16 01:12:05.374: %OSPF-5-ADJCHG: Process 1, Nbr 150.5.4.4 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Dead timer expired

It’s like R1 attempting to make adjacency to R2, R3, and R4 which should not. So the question is, why R1 is even attempting to establish adjacency to R2, R3, and R4?

I did this command below to check the Frame Relay

Rack5R1#sh frame-relay map
Serial0/0 (up): ip 0.0.0.0 dlci 113(0x71,0x1C10)
              broadcast,
              CISCO, status defined, inactive
Serial0/0 (up): ip 0.0.0.0 dlci 104(0x68,0x1880)
              broadcast,
              CISCO, status defined, inactive

Serial0/0 (up): ip 0.0.0.0 dlci 103(0x67,0x1870)
              broadcast,
              CISCO, status defined, active
Serial0/0 (up): ip 0.0.0.0 dlci 102(0x66,0x1860)
              broadcast,
              CISCO, status defined, active
Serial0/0 (up): ip 155.5.0.5 dlci 105(0x69,0x1890), static,
              broadcast,
              CISCO, status defined, active

I know that I’m expecting to see DLCI 105 as my L2 to the hub. However, I didn’t expect to see DLCI 102 and 103 as active.

R1 knows how to reach R2, R3, and R4 and then it’s attempting to establish the adjacency. However, in Point-to-Multipoint network this is not the expected behaviour.

After reloading, yes, only reloading would get rid of the DLCI mapping.

Rack5R1#sh frame-relay map
Serial0/0 (up): ip 155.5.0.5 dlci 105(0x69,0x1890), static,
              broadcast,
              CISCO, status defined, active

This would also let us know to be careful when configuring ospf over Frame Relay as we might need to reload the router during the exam and it will really wasting your time.