Connect to Leaf from APIC
bash
attach [node-name]
ssh user@[node-name]
Endpoint, EPG, BD form APIC
<show ip interface bridge-domain [name_bd]>
<show epg [epg-name] [detail]>
Ping from Leaf
iping [-dDFLnqRrv] [-V vrf] [-c count] [-i wait] [-p pattern] [-s packetsize] [-t timeout] [-S source ip/interface] host
VPC/PC from APIC
show vpc map [VPC-Name]
show port-channel map [VPC-Name]
VPC/PC from Leaf
show vpc extended
show port-channel extended
show lacp [counters | interface] ethernet [slot/number]
VMware from APIC
show vmware domain name <VMM-Name>
show vmware vm [name | ip | mac] <vm-name | ip-address | mac-address>
attach-avs <OpflexDevid>
attach-eve <OpflexDevid>
Clustering User Commands
<controller> – shows the current cluster size and state of APICs
<cd /aci/system/controllers/1/cluster> <moset administrative-cluster-size (#)> <moconfig commit>– changes the size of the cluster
<controller -d -t (ID)> – Decommissions the APIC of the given ID
<eraseconfig setup> – Factory resets APIC and after reboot will load into setup script
<reload [controller|switch] (nodeID)> – Reboots the APIC of the given ID
<acidiag rvread> – shows replica which are not healthy
<acidiag rvread (svc) (shard) (replica)> – shows the state of one replica
<avread> – large output which will show cluster size, chassisID, if node is active, and summary of replica health
<acidiag fnvread> – shows fabric node vector
<acidiag avread> – shows appliance vector
<acidiag verifyapic> – verifies APIC hardware
<acidiag reboot> – Reload APIC
<ip link> – shows link status
<cat /proc/net/bonding/(ID)> – shows the status of bond link
<show dhcp internal info client> – shows dhcp client information to confirm dhcp address from APIC
<fabricnode (nodeID) [commission|decommission|wipeout]> – commissions, decommissions, or wipes out given node. wipeout will completely wipeout the node including configuration. Use sparingly.
SSL Troubleshooting
<openssl s_client -connect (IP):12151> – tries to connect ssl between APIC and Node and gives output of SSL information
<zgrep SSL svc_ifc_appliancedirector.bin.log*> -shows logging of DME-logs for node
<zgrep SSL svc_ifc_policyelem.log*> – shows policy-element logs for SSL connectivity
Can also check logs in the /var/log/dme/log directory
Switch Cert Verification
<openssl asn1parse < /securedata/ssl/server.crt> – Next to PRINTABLESTRING, it will list Insieme or Cisco Manufacturing CA. Cisco means new secure certs are installed, Insieme means old unsecure are installed
<openssl x509 -noout -issuer -subject -dates -in /securedata/ssl/server.crt> – Shows start and end dates of certificate. Must be within range for APIC to accept
<act_util key_pair show (#)> – Shows keypairs of specified cert
Switch Diagnostics
<show module internal event-history module (#)> – shows bootup tests and diagnostics of given module
<show diagnostic content module (ID)> – shows ongoing tests of given module
<show diagnostic result module [all|(moduleID)]> – shows diagnostic result of given module or all modules
<show diagnostic result module (moduleID) test (testID) detail> – shows diagnostic result of given test on given module
<show diagnostic internal [diagmgr|diagclient|port_lb]> – show debug information for the diagnostic modules
Debug Commands
<debug platform internal emon [heartbeat|kfsm|stats|traffic]> – shows debug output of given argument
<debug platform internal emon [heartbeat|kfsm|stats|traffic] [enable|disable]> – enables/disables given argument on all modules
<debug platform internal emon [heartbeat|kfsm|stats|traffic] interval get> – gets the interval of given argument
<debug platform internal emon stats get (ID)> – EPC mon statistics
<debug platform internal emon kfsm state get (ID)> – EPC mon statistics
<debug platform internal marvell switch [0|1] status> – EOBC/EPC switch status (0: EOBC, 1: EPC)
<debug platform internal broadcom switch status> – SC card broadcom switch status
Insieme ELTM VRF, VLAN, Interface Commands
<debug system internal [eltm|eltmc] trace output file> – dumps ELTM trace to output file
<show system internal [eltm|eltmc] info trace> – dumps eltm trace to console
<show system internal [eltm|eltmc] info vrf (vrf)> – shows vrf table of given vrf
<show platform internal ns forwarding segments> –
<show platform internal ns forwarding epgs> –
<cat summary> – vrf summary, shows ID, pcTag, scope
<show system internal eltmc info vlan brief> – shows vlan information. Can substitute (brief) for a vlan ID
<show sytem internal eltmc info interface (interface ID)> –
OSPF CLI Commands
<show ip ospf neighbors vrf (vrf|all)> – shows OSPF neighbors of given vrf
<show ip ospf route vrf (vrf|all)> – shows OSPF routes of given vrf
<show ip ospf interface vrf (vrf|all)> – shows ospf interfaces of given vrf
<show ip ospf vrf (vrf|all)> – shows ospf information of given vrf
<show ip ospf traffic vrf (vrf|all)> – shows ospf traffic of given vrf
External Connectivity
<show ip arp vrf (vrf)> – shows arp entries for given vrf
<show ip ospf neighbors vrf (vrf)> – shows ospf neighbors for given vrf
<show bgp sessions vrf (vrf)> – shows bgp sessions/peers for given vrf
<show ip ospf route vrf (vrf)> – shows ospf routes for given vrf
<show bgp ipv4 unicast vrf (vrf)> – shows bgp unicast routes for given vrf
<show ip static-route vrf (vrf)> – shows static routes for given vrf
<show ip route vrf (vrf)> – shows routes for given vrf
<l3 defip show> – shows external LPMs
<l3 egress show> – shows next hops towards NorthStar ASIC or external router
<show platform internal ns table mth_lux_slvd_DHS_HigigDstMapTable_memif_data ingress> – HigigDstMapTable Indexed using DMOD/DPORT coming from T2. Provides a pointer to DstEncapTable.
<show platform internal ns table mth_lux_slvg_DHS_DstEncapTable_memif_data ingress> – DstEncapTable Indexed using the HigigDstMapTable’s result. Gives tunnel forwarding data.
<show platform internal ns table mth_rwx_slva_DHS_RwEncapTable_memif_data ingress> – RwEncapTable Indexed using the HigigDstMapTable’s result. Gives tunnel encap data.
ISIS Fabric Unicast Debugging
<show isis protocol> – shows ISIS statistics
<show isis adjacency [detail] vrf (vrf)> – shows ISIS adjacencies for given vrf. Can also add detail
<show lldp neighbor> – shows lldp neigbor status
<show interface (interface ID)> – shows interface status information and statistics
<show isis database [detail] vrf (vrf)> – shows isis database, can also add detail
<show isis route vrf (vrf)> – shows isis route information
<show isis traffic vrf (vrf)> – shows isis traffic information
<show isis dtep vrf (vrf)> – shows all discovered tunnel end points
<show isis statistics (vrf)> – shows isis statistics of given vrf
<show isis event-history [detail]> – shows isis event history
<show isis internal mem-stats [detail]> – shows isis memory statistics
<show tech-support-service isis> – provides isis tech-support output for TAC
ASIC Platform Commands
<show platform internal [ns|alp] mac asic [0|1]> – shows the MAC port status
<show platform internal [ns|alp] counters mac asic [0|1]> – shows the MAC port counters
<show platform internal [ns|alp] counters asic-block [all|bax|lbx|lux|prx|qsx|rwx|scx|top]> – shows ASIC block counters for given ASIC. Can also add [detail] for more details
<show platform internal [ns|alp] interrupts> – shows interrupts for given ASIC
ASIC Platform Commands – T2 Specific
<show c rpkt> – shows receive counters for T2
<show c tpkt> – shows transmit counters for T2
<show c xe12> – shows per port packet type counters
<g chg ing_event_debug> – shows ingress drop counters
<g chg_egr_drop_vector> – shows egress drop counters
<s RDBGC(#)_SELECT bitmap=(hex)> & <g chg RDBGC(#)_SELECT> – setting register to specific trigger. 9 registers per port (0-8)
<s RDBGC3_SELECT bitmap=0x2000> <g chg RDBGC3_SELECT> – sets 4th register to select RFILDR selector (bit 13)
<cstat xe17> – checking the stats for above command
ASIC Platform Commands – NS Specific
<show platform internal counters port> – shows port counters
<show platform internal counters port internal> – shows internal port counters
<show platform internal counters vlan> – shows vlan counters
<show platform internal counters tep> – shows per-tunnel counters
<show platform internal ns counters asic-block all> – shows ASIC block counters
<show platform internal ns forwarding list> – shows well-defined tables
Fabric Multicast – General
<show isis internal mcast routes ftag> – shows currecnt state of FTAG, cost, root port, OIF list
<show isis database mgroup detail vrf (vrf)> – shows GM-LSP database
<show isis internal mcast routes gipo> – shows GIPO routes, Local/transit, OIF list
<show isis internal mcast statistics> – shows topology and compute stats, MRIB update stats, Sync+Ack packet stats, Object store stats
<show isis event-history mcast> – shows isis multicast event history logs
<show isis event-history mcast-convergence> – more detailed than above command, specifically dealing with forwarding events and forwarding updates
Fabric Multicast Debugging – MFDM
<show forwarding distribution l2 multicast> – flood/OMF/GIPi membership
<show forwarding distribution l2 multicast vlan (vlanID)> per BD
<show forwarding distribution l2 multicast flood> – flood membership
<show forwarding distribution l2 multicast flood vlan (vlan ID)> – per BD
<show forwarding distribution l2 multicast omf> – OMF membership
<show forwarding distribution l2 multicast omf vlan (vlan ID)> – per BD
<show system internal forwarding distribution multicast ipmc> – IPMC membership
<show system internal forwarding distribution multicast ipmc 0x3> – specific IPMC
<show forwarding distribution multicast ipmc-sw>
<show forwarding distribution multicast ipmc-sw (ID)>
Fabric Multicast Debugging – L2 Multicast
<show system internal forwarding l2 multicast> – flood/OMF/GIPi membership
<show system internal forwarding l2 multicast vlan (vlanID)> – per BD
<show system internal forwarding l2 multicast gipi> – GIPi membership
<show system internal forwarding l2 multicast gipi (IP)> – specific
<show system internal forwarding l2 multicast gipi vlan (vlanID)> – per BD
<show system internal forwarding l2 multicast gipi (IP) vlan (vlanID)> – specific per BD
<show system internal forwarding l2 multicast flood> – flood membership
<show system internal forwarding l2 multicast flood bd (bdID)> – per BD
<show system internal forwarding l2 multicast met> – MET membership
<show system internal forwarding l2 multicast met (ID)> – specific MET
<show system internal forwarding l2 multicast met flood> – flood MET
<show system internal forwarding l2 multicast met gipi> – GIPi MET
<show system internal forwarding l2 multicast met gipi bd (bdID)> – per BD
<show system internal forwarding l2 multicast met gipi (IP) bd (bdID)> – specific per BD
<show system internal forwarding l2 multicast ipmc> – IPMC membership
<show system internal forwarding l2 multicast ipmc (ID)> – specific IPMC
Fabric Multicast Debugging – MRIB
<show ip mroute vrf (vrf)> – shows IP multicast routing table for given vrf
Fabric Multicast Debugging – MFIB
<show ip fib mroute ftag> – shows FTAGs
<show forwarding vrf all multicast route> – shows GIPO routes
Fabric Multicast Debugging – IGMP
<show ip igmp snooping groups> – shows multicast route information in IGMP
<show ip igmp snooping mrouter> – shows multicast router information IGMP
<show ip igmp snooping encap-db> – FD to BD vlan mapping. IGMP gets FD and G from Istack. It needs to know the BD to create (BD, G)
<show ip igmp snooping vlan (vlanID)> – verify BD membership of a port in IGMP. Only when ports are part of BD joins are processed
<show ip igmp snooping vtep-if-db> – verify the tunnel to IF mapping in IGMP. IGMP uses this to get the groups on VPC and only sync them.
Fabric Multicast Debugging – MFDM
<show forwarding distribution ip multicast route vrf (vrf)> – shows IPv4 multicast routing table for given vrf
<show forwarding distribution multicast vlan_db> – Verify FD to BD vlan mapping. MFDM gets (FD,port) memberships from vlan_mgr and uses this information go create BD floodlists.
<show forwarding distribution multicast bd_gipo> – BD to GIPO mapping. GIPO is used by Mcast in Fabric
<show forwarding distribution multicast epg_gipo_prime> – FD-vxlan to GIPO mapping
<show forwarding distribution multicast vtep_if_db> – tunnel to phy mapping
Fabric Multicast Debugging – M2rib
<show l2 mroute> – shows multicast route information in M2rib
<show l2 mroute omf> – shows multicast route informatino in M2rib
Fabric Multicast Debugging – PIXM
<show system internal pixm info ltl-range start-ltl 0x0 ltl-cnt 4000> – RID to IPMC mapping. IFIDX is RID and LTL is IPMC
Fabric Multicast Debugging – VNTAG Mgr
<show system internal vntag dvif-allocation> – IPMC to DVIF mapping. LTL is IPMC
EP Announce – Debugging
<show system internal epm announce>
<show system internal epm counters announce>
<show system internal epm vlan (vlanID) detail>
<show system internal epm vrf (vrf) detail>
<show system internal epm periodic>
<show system internal epm endpoint all>
iBash CLI
<show mac address-table>
<show endpoint [summary|address|interface|vlan|vrf]> – show endpoint information
BCM Table Dump
<bcm-shell-hw “l2 show”>
<bcm-shell-hw “l3 l3table show”>
Fabric QoS Debugging – CoPP CLI
<show copp policy>
<show system inernal aclqos brcm coppp entries unit 0> – CoPP statistics (red = dropped, green = allowed)
<show system internal qos classes> – shows QoS classes configured
<show system internal qos vlan all> – shows QoS classes/policices configured per vlan
<show system internal qos ppf [pinst|nodes]> – shows ppf details
<show system internal aclqos qos classes> – shows QoS classes configured in hardware
<show system internal aclqos qos vlan (vlanID)> – shows the QoS DSCP/dot1p policy configured for a vlan in HW
<show system internal aclqos qos policy summary> – shows QoS DSCP/dot1p policy summary
<show system internal aclqos qos policy detail> – shows QoS DSCP/dot1p policy in detail
<show system internal aclqos brcm tcam entries unit 0 group [efp-bpdu|efp-ctrl-pol|efp-mark|ifp-ctrl|ifp-dscp|ifp-elmc-vleaf|ifp-span-port-vlan|ifp-span-port-vlan-egress|ifp-span-vlan-egress|ifp-vni-udf|vfp-vni]> – shows T2 TCAM entries for specified group
<show platform internal counters port (#)> – shows QoS counters on each port
<show platform internal counters port internal (#)> – shows QoS counters on each port (internal)
<show platform internal counters class (#)> – shows QoS counters for each class for all ports
MCP CLI
<show mcp internal info global> – shows the edge port config on the HIF (FEX) ports, the internal VLAN mapping and the STP TCN packet statistics received on the fabric ports
<show mcp internal info interface [all|interfaceID]> – shows mcp information by interface
<show mcp internal info stats interface> – shows stats for all interfaces
<show mcp internal info vlan [all|vlanID]> – shows mcp information per vlan
<show mcp internal stats vlan> – shows stats for all vlans
<show mcp internal info msti [all|(region name) (instance ID)]> – shows mcp information per msti region
<show mcp internal info stats msti> – shows stats for all msti regions
iTraceroute CLI
<itraceroute (destinationIP) (pld-size)> – node traceroute
<itraceroute (destinationIP) vrf (vrf) encap vlan (vlan-encap) payload (pld-size)> – Tenant traceroute for vlan encapped source EP
<itraceroute (destinationIP) vrf (vrf) encap vxlan (vxlan-encap) dst-mac (dst-mac) payload (pld-size)> – Tenant traceroute for vxlan encapped source EP
ELAM Setup and debugging (follow commands in order)
<debug platform internal ns elam asic (#)> – starts ELAM on given ASIC
<trigger init ingress in-select 3 out-select 0> – sets trigger for ELAM
<set outer l2 dst_mac (destination mac) src_mac (source mac)> – sets source and destination mac addresses
<start> – Starts capture
<status> – shows capture status
<report> – shows report of the capture
VMM Troubleshooting
<show vmware controllers> – shows VM controllers and their attributes such as IP/hostname, state, model, serial number
<show vmware domain mininet (name) inventory> – shows hypervisor inventory of given VM controller
<show vmware domain mininet (name) [inventory|policy|status]>
<show vmware domain mininet (name) inventory [hypervisors|portgroups|virtual-machines|virtual-switches]>
TOR Sync Troubleshooting
<netstat -tp | grep epm>
<tcpdump -i kpm_inb>
<show system internal epm vpc>
<show system internal epm counters vpc>
<show system internal epm counter zmq>
<show system internal epm announce>
<show system internal epm counters announce>
<show system internal epm vlan (vlanID) [detail]> – can see which VLAN is learn disable
<show system internal epm vrf (vrf) [detail]> – can see which VLAN is learn disable
<show system internal epm periodic> – see if timer is attached on the VLAN/vrf
<show system internal epm counters all>
<show system internal epmc counters all>
OpFlex Debugging
<vemcmd show openflex> – shows if OpFlex is online (status = 12 means OpFlex is online, remoteIP is anycast IP, intra vlan is vlan used by VTEP, FTEP IP is the iLeaf’s IP)
<vem status> – check if DPA is running
<vemcmd show sod>
<vemcmd show port> – uplinks and vtep should be in forwarding state. PC-LTL of uplink port should be non-zero
<vemcmd show pc> – Check port channel type
<vemcmd show lacp> – if port channel type is LACP, can use this command to see the individual uplink LACP state
<esxcfg-vmknic -l> – verify if the VTEP received a valid DHCP IP address
SPAN Debugging
<vemcmd show span>
BPDU Debugging
<vemcmd show card> – shows if BPDU Guard/Filter is enabled or disabled
<vemcmd show bpdu-stats> – check if the bpdu-drop stats are incrementing on the uplinks/virtual ports
VEM Misc Commands
<vemcmd show openflex> – show channel status
<vemcmd show port> – check port status
<vemcmd show bd> – check per EPG flood lists
<vemcmd show epp multicast> – check vLeaf multicast membership
<vemcmd show stats> – show packet stats
<vemcmd show packets> – show packet counters
<vemlog debug sfport all> – debug vxlan packet path
<vemlog debug sflayer2 all> – debug vxlan packet path
<vemlog show all> – show above logging output
<vempkt capture [egress|pre-ingress]>
<vempkt clear>
<vempkt start>
<vempkt stop>
<vempkt display brief all>
<vempkt display detail entry (#)>
<vempkt cancel capture all>
FEX Troubleshooting
<show fex> – shows all FEXs and their states
<show fex (#) [detail]> – gives detailed stats of given FEX
<show environment fex> – gives environmental stats of FEX
<show fex transceiver>
<show fex version> – shows FEX version
<show interface fex-fabric> – shows FEX fabric interface information
<show logging level fex> – shows logging information for FEX
<show interface transceiver fex-fabric> – shows transceiver information for FEX
<show system reset-reason fex> – show FEX reset reason
<show module fex> – shows FEX module information
<show system internal fex log | grep (anything)> – shows debugging information and you can grep to find what you want
<show system internal fex internal event-history msgs> – use to find out which service is failing the sequence and you can debug that process further
Reference:
