Cisco ACI Multi-Pod Configuration Guide – From Scratch

Cisco ACI Multi-Pod Configuration Guide

Introduction

This guide is designed to help junior network engineers understand and configure Cisco ACI Multi-Pod. We'll break down the concepts and steps into easy-to-follow explanations and examples.

What is Cisco ACI Multi-Pod?

Imagine you have a single, very efficient network (a Cisco ACI fabric) in one location. Now, imagine you want to expand that network to another location, maybe for redundancy or increased capacity, but you still want to manage it all as one system. That's where Multi-Pod comes in.

Cisco ACI Multi-Pod allows you to connect multiple ACI fabrics (called "Pods") across an IP network. This lets you:

• Extend your network: Span your ACI fabric across different locations.
• Increase redundancy: If one location has a problem, traffic can flow through another.
• Scale your infrastructure: Add more capacity by adding more Pods.
• Single Management: Manage all pods from a single APIC cluster.

Key Components

Before we dive into configuration, let's define the key components:

• ACI Fabric: A basic ACI setup, including APICs (Application Policy Infrastructure Controllers), Spine switches, and Leaf switches.
• Pod: A single ACI fabric (Spine and Leaf switches) in one location.
• APIC (Application Policy Infrastructure Controller): The brain of the ACI fabric. It's a cluster of servers that manage the entire ACI system. In a Multi-Pod setup, a single APIC cluster manages all the Pods.
• Spine Switch: High-speed switches that form the core of the ACI fabric.
• Leaf Switch: Switches that connect to endpoints like servers and storage.
• IPN (Inter-Pod Network): The IP network that connects the Pods. This is crucial – it's the "highway" that carries traffic between Pods. This network must be able to handle the required bandwidth and latency.
• TEP (Tunnel EndPoint): An IP address on each Leaf and Spine switch used to encapsulate and decapsulate traffic that travels across the IPN.

Visual Representation

Here's a simple diagram to help you visualize the Multi-Pod setup:

IPN Requirements

The IPN is critical. Here are some key requirements:

• IP Connectivity: The IPN must provide basic IP connectivity between all Spine switches in all Pods.
• Bandwidth: Sufficient bandwidth to handle the traffic between Pods. Consider your application needs.
• Latency: Low latency is important for application performance. ACI has specific latency requirements.
• Multicast: The IPN must support multicast, specifically Protocol Independent Multicast (PIM), and in most cases, Bi-Directional PIM (PIM-BIDIR). ACI uses multicast for efficient flooding of some traffic.
• MTU: The Maximum Transmission Unit (MTU) must be large enough to handle the VXLAN encapsulated traffic. A jumbo MTU (around 9000 bytes) is typically recommended.
• No NAT: Network Address Translation (NAT) is generally not supported in the IPN.

Configuration Steps

Here's a high-level overview of the configuration process. We'll break down each step in more detail.

1. Initial ACI Fabric Setup: Make sure your initial ACI fabrics (Pods) are up and running before you connect them with Multi-Pod. This includes configuring the APICs, Spine switches, and Leaf switches in each Pod.
2. IPN Configuration: Configure the IP network that will connect the Pods. This involves configuring routing (OSPF or BGP), multicast (PIM), and ensuring correct MTU settings on the IPN devices.
3. APIC Multi-Pod Configuration: Configure the APIC to recognize the additional Pods and establish connectivity between them.
4. Policy Configuration: Extend your ACI policies (like VRFs, Bridge Domains, and EPGs) across the Pods as needed.
5. Verification: Verify that the Multi-Pod setup is working correctly.

Step 1: Initial ACI Fabric Setup

This step assumes you already have a working ACI fabric in each location (Pod) that you want to connect. If you're new to ACI, you should get familiar with basic ACI configuration before attempting Multi-Pod. This involves:

• APIC Cluster Setup: Forming the APIC cluster. Best practice is to have APICs in different pods for redundancy.
• Fabric Discovery: Discovering the Spine and Leaf switches in each Pod.
• Basic Policies: Configuring basic fabric policies.

Step 2: IPN Configuration

This is a critical step. The IPN provides the connectivity between the Pods.

• Routing Protocol: You'll need a routing protocol to exchange routing information between the Spine switches in different Pods. OSPF is commonly used. BGP is another option, especially in more complex scenarios.
  • OSPF Example:

    router ospf 1
      vrf overlay-1  # Important:  Use the correct VRF
        area 0
         interface <IPN Interface to Spine 1>
         ip ospf network point-to-point
         interface <IPN Interface to Spine 2>
         ip ospf network point-to-point
        !
    
    

  • BGP Example:

    router bgp 65001
      vrf overlay-1
        neighbor <IPN Peer IP 1> remote-as <Peer AS>
        neighbor <IPN Peer IP 1> ebgp-multihop 2
        address-family ipv4 unicast
          neighbor <IPN Peer IP 1> activate
        exit-address-family
       !
    
    

• Multicast (PIM): ACI uses multicast for some traffic, so you need to enable PIM on the IPN routers. PIM Bi-Directional (PIM-BIDIR) is often required.
  • PIM Configuration Example (PIM-BIDIR):

    interface <IPN Interface>
      ip pim sparse-mode
      ip pim bidir
    !
    ip pim rp-address <RP Address> bidir  # Configure a Rendezvous Point (RP)
    !
    
    

• MTU Configuration: Ensure that the MTU is consistent across the IPN and is large enough to accommodate the VXLAN overhead.
  • MTU Example:

    interface <IPN Interface>
      mtu 9000
    !
    
    

• DHCP Relay (Optional but Recommended): If you plan to add new Leaf or Spine switches to a Pod, you'll need DHCP relay on the IPN to forward DHCP requests to the APIC.

Step 3: APIC Multi-Pod Configuration

Once the IPN is ready, you'll configure the APIC to enable Multi-Pod.

• Fabric Membership: In the APIC, you'll add the additional Pods to the fabric. This typically involves specifying the Pod IDs and the Spine switches in each Pod.
• IPN Configuration in APIC: You'll configure the IP addressing and routing information for the IPN within the ACI fabric configuration.

Step 4: Policy Configuration

After the Multi-Pod fabric is established, you can extend your ACI policies across the Pods. This means that you can create:

• Tenants: Logical containers for your applications.
• VRFs (Virtual Routing and Forwarding instances): Logical routing tables.
• Bridge Domains: Layer 2 forwarding domains.
• EPGs (Endpoint Groups): Groups of endpoints (servers, VMs) that share the same policy requirements.
• Contracts: Rules that define how EPGs can communicate with each other.

You can create these policies and then apply them to EPGs that reside in different Pods, allowing seamless communication across the extended fabric.

Step 5: Verification

It's crucial to verify that your Multi-Pod setup is working correctly. Here are some things to check:

• Pod Status: Verify that all Pods are online and healthy in the APIC.
• IPN Connectivity: Verify IP connectivity between Spine switches in different Pods. Use ping, traceroute, and other network tools.
• Routing: Check that the routing protocol (OSPF or BGP) is working correctly and that routes are being exchanged between Pods.
• Multicast: Verify that multicast is working across the IPN. You can use show commands on the IPN devices and the ACI switches.
• TEP Connectivity: Verify that the Spine switches can communicate with the Leaf switches in the remote Pods using the TEP addresses.
• End-to-End Connectivity: Test connectivity between endpoints in different Pods. For example, try pinging a server in Pod 1 from a server in Pod 2.
• Policy Enforcement: Verify that your ACI policies are being enforced correctly across the Pods.

Important Considerations

• Latency: Multi-Pod introduces some latency. Design your IPN to minimize this as much as possible. ACI has maximum latency requirements between Pods.
• Bandwidth: Ensure your IPN has enough bandwidth to handle the traffic between Pods.
• MTU: Consistent MTU is critical. A mismatch can cause connectivity problems.
• Time Synchronization: Synchronize the clocks on all devices (APICs, Spine switches, Leaf switches, and IPN devices) using NTP.
• DNS: Ensure proper DNS resolution for all devices.

Troubleshooting

• Connectivity Issues: If you have connectivity problems, check the IPN configuration, routing, and MTU settings.
• Multicast Issues: If multicast is not working, check the PIM configuration on the IPN devices.
• Policy Issues: If policies are not being enforced correctly, check the APIC configuration and make sure the policies are applied to the correct EPGs.

Conclusion

Cisco ACI Multi-Pod allows you to create a scalable and resilient network across multiple locations. By understanding the key components and following the configuration steps in this guide, you can successfully deploy and manage a Multi-Pod fabric. Remember to plan your IPN carefully and verify your configuration thoroughly.

Additional Resources

• Cisco Application Centric Infrastructure Design Guide
• Cisco ACI Multi-Pod White Papers
• Cisco ACI Troubleshooting Guides