Great to here that cisco announced ACI 3.0 … let me brief my experience with ACI

ACI is fantastic product for green field deployment but for brown field deployment had a bad experience … here are couple of issues.

1) End point fluctuation – if you see more end points moves frequently in fabric, end point process get crashed and eventually, fabric get crashed.

Command to check endpoint move on leaf:

tail -f /var/log/dme/log/epm-trace.txt | grep “EP move” -B 1
tail -f epm-trace.txt | grep “1.1.1.1”

2) VTEP IP assign issues – When we added new leaf in fabric, given used VTEP IP to new leaf due to which my production leaf (which has that IP) went down, not sure how it’s assigned the used IP ??because IP assigning to new leafs through DHCP , seems to be ACI not tracking DCHP assigned IP`s ?

3) When registering the new leafs, suddenly 4-5 others leafs got rebooted and impacted.

4 ) Directly connected route issue – if you are mapped Subnet-1 to VRF-1 and later you decided to move \d subnet-1 to VRF-2 but still your subnet-1 shows as directly connected VRF-1 as well VRF-2 which makes that subnet unreachable

Work around – erase the specific leaf configuration (ensure you have dual home connectivity) and reload the leaf – seems to be fixed.

Being a CCIE guy I am not trying to project ACI a bad product , assuming cisco will ensure ACI is the best product because cisco has lunched this product with high expectations , soon will see stable product

We  wish cisco will  bounce back with new code addressing all the issues in software defined competitive world .

Best of luck !!

I have noted  for myself during troubleshooting, thought to share with all of you. but its not in order, I suggest you all to use APIC controller to get the right information  for endpoint(leaf/VPC ).
firmware path in ACI – cd firmware/fwrepo

show port-chann ext
show vlan ext(will show s/w and h/w vlan, encap vlan)
acidiag fnvread( VTEP information)

acidiag avread (application vector, give APIC controller information(Version/IP/VTEP pool..)
show coop internal info ip-db | grep 172.31.15.70 -A 10 – check endpoint in spine

vsh_lc (To check internal )
show system internal elmn info vlan bri

Type only vsh, gets into NXOS mode

ENd point move checking:

End point move check:

tail -f /var/log/dme/log/epm-trace.txt | grep “EP move” -B 1

tail -f emp.txt | grep -C 10 “MAC ADDRESS”

show coop internal info global – Which spine is primary
show oob – Mgmt IP`s (From APIC)

show system internal epmc endpoint ip 172.31.15.70
show system internal ethpm event-history interface eth1/17

cat /mit/sys/lldp/inst/if-[eth1/1]/summary – Wiring issues
cat /mit/uni/fabric/compcat-default/swhw-*/summary | grep model
cat /mit/sys/summary – OBB ip

iping -V xyz.vrf -c 1000 172.31.15.70

From APIC Controller:

show vlan-domain vlan 1226 –> mapped to which EPG and static bindings
show endpoints vlan 1226 – endpoints of specific VLAN
show oob from APIC controller
show tenant XYZ application database (complete application profile end end points)
show tenant XYZ application database epg phx-e2-tims-db5-2309 endpoints(specific endpoints)
show tenant XYZ epg phx-e1-payb-app1-1213 detail – Static bindings
show tenant XYZ endpoint vlan 1213 – to know the expg
show vpc map to check list of vpc`s configured
show tenant XYZ endpoints | egrep “10.21.206.41|AEPg”
show tenant XYZ ip interface bridge-domain | egrep “x.x.x.x|Interface” – To know the BD
show ip interface bridge-domain | grep 10.0.10 -B 3 -A 3 – Where you don`t have endpoint, got only subnet and you want find BD/EPG(reverse engineering)

show tenant E1-eCP ip interface bridge-domain | grep -A3 -B3 10.20.182

show vlan ext

show port-channel ext

show vpc ext

show vlan id 75,81 ext

OSPF commands
———————

show ip ospf database self-originated vrf common:intra-app-west-vrf
show ip ospf neighbors vrf common:internet-west-vrf
show ip ospf database vrf common:internet-west-vrf

Continues…………………   in next part!!!

It’s pretty straight forward but little tricky while configure route-maps in ACI.

Example : Changing set Metric/type.

Click on L3-out

Click on Route  Maps/Profiles

Create route-map/profile – The name should be “default-import/Default-export “(depends on your requirement) name should not be  given any other names.

Example : If you want to set Metric/Type for all outgoing routes from fabric.

Create a route map with default-export

Select Match Prefix AND Routing policy

Create a route context (leave default # 0) – name (here can be anything )

Set the rule and rule name (can be anything) – Set metric – Value which you want to be and Metric type (type1/Type) also, you can do something in this section(can modify the BGP attributes as well)

I am trying to recollect some more points on ACI which is continues to my previous post.

ACI is a simple modular switch, below depicted diagram suppose to be in part -1  🙂

AVS -> Application Virtual Switching supported only on VMWARE (VEM)

•         Essentially a modified N1K VEM with an Opflex agent (port-groups backed by VxLANs)
•       APIC will also talk to AVS/VEM over OPFLEX and assign it IP address just like any other f    Fabric component

AVS switching modes:

•            Local switching: Intra-EPGs traffic switched on the same host
•           FEX mode: All traffic sent to Leaf for switching
•           Full switching : : Full APIC policy enforcement on server

• X9700 only supported ACI supported line card
• NXOS line cards are different that ACI line cards
• Leaf and Spine communicate over IS-IS (by default) and IBGP (configurable for route leaking)
• Traffic is normalized into eVXLAN (ACI VXLAN) at the spine and communication happens based on source and destination EPG
• If leaf does not know dest mac, traffic is sent to spine
• If even spine does not know, then the frame is dropped by default, however we can configure it to flood such frames
• Leaf identifies a new host as it comes up with any snooping technology and reports the Spine through a communication protocol called COOP
• Old entries on leaf switch will be removed after 5 minutes
• APIC is configurable through CIM-C and KVM
• APIC will further configure the spine and leaf switches starting with IP assignment
• Management IP offered by APIC to fabric are only for management communication and not for any outside access
• APIC will communicate with fabric over a dedicated VRF called Overlay-1
• VM kernel IP address subnet should be different than APIC IP assignment subnet
• VLAN ID is required for infrastructure network 4093
• Kernel of APIC is CENT OS
• Cannot conf t to leaf switches

Almost been a year that i was trained on ACI, hence, would like to refresh my memory before its completely  wiped off 🙂

My first impression, i felt the product was quite interesting because simplification of networking configuration/application  driven policy/Micro segmentation/Multi-tenancy/automation…

My view,  takes little time to understand for core networking folks but the person who is from Core Network + virtualization  cloud able to understand the concepts well and able to integrate ACI with different hypervisors with ease(VMvamre vSwitch/Hyper-v,Xen,Openstack(Neutron Component as well)

What is ACI??? 

Application Centric Infrastructure (ACI) in the data center is a holistic architecture with centralized automation and policy-driven application profiles. ACI delivers software flexibility with the scalability of hardware performance.

Key characteristics of ACI include:

• Simplified automation by an application-driven policy model
• Centralized visibility with real-time, application health monitoring
• Scalable performance and multi-tenancy in hardware

Works on eVxlan, it extension of Vxlan

In simple, what is Vxlan:

VXLAN enables you to create a logical network for your virtual machines across different networks. You can create a layer 2 network on top of your layer 3 networks. This is why VXLAN is called an overlay technology. Normally if you want a virtual machine to “talk” to a virtual machine in a different subnet you need to use a layer 3 router to bridge the gap between networks.

1. Each VXLAN(Virtual extension lan) is LAN extension over L3 and  segment has unique  24-bit Virtual Network Identifier(VNI) enables up to 16 Million  unique virtual LAN segments.
2. VXLAN uses MAC over IP/UDP.
3. VXLAN is first host based overlays means,VXLAN  encap and decap starts from physical server and virtual switch sitting on physical server.
4. Enables VM mobility at layer 2 across layer 3 boundaries

For more information on Layer2 over Layer 3 protocols, please go through Massive data center design book which covers(TRILL,Fabric path, VXLAN,NVGRE)

Overview:

ACI infrastructure essentially has 3 components:

• Spine Switches (SPINE -> 9500 series, Baby Spine -> 9336PQ)
• Leaf Switches (LEAF -> 9396 (2U) & 93128 (3U))
• Application Policy Infrastructure Controller – APIC Cluster (min 3 devices in a cluster)
• As of now 6 Spines and 18 leaves are supported in an ACI fabric. The ratio of Spine to Leaf is 1:3
• Support upto 1000 tenants and 128K End point
• North bound ports on the Spine are always 40 Gig while South bound ports on the Leaf for Access layer are 1/10 Gig
• ACI also comes with a different line card which is other than the conventional Nexus line cards
• Leaf and Spine switches communicate with each other through IS-IS protocol
• Please refer below link for more information
• NFE -> Switching (Broadcom T2), ALE -> Routing (Cisco)

Hardware information:

Partner Ecosystem(stale data): uses oplex protocol(Cisco ) for integration, as per my knowledge, ACI does not support openflow protocol.

L4-L7 Compatibility List: