Juniper QFX5100 Series (2015)

Appendix A. Under the Hood

One of the most frustrating things about networking technology is that oftentimes operators are caged into a box called the command-line interface (CLI). Anything behind the curtain doesn’t exist and isn’t supported. Unfortunately, I can’t change the not-supported part, but I can at least show you what’s behind the curtains and how it works.

Any additional information you’re able to pull from a piece of technology ultimately makes your network better, whether it’s better network management, graphing, or troubleshooting.

Big Scary Disclaimer

Everything I’m about to show you in this chapter isn’t supported by Juniper Networks or the Juniper Technical Assistance Center (JTAC). Don’t use these commands in production. Use them at your own risk. Changing any values at a low level will cause instability in the network because the changes will not be synchronized with the control plane.

With that out of the way, let’s get on with having some fun!

The Broadcom Shell

The Broadcom shell, owned and maintained by Broadcom, is the standard CLI that you can use to directly access the Broadcom chipsets. It is a simple tool that you can put to work gathering additional debugging information from the system.

NOTE

Messing around with the Broadcom shell is really powerful but at the same time really dangerous to production systems. Many of the commands are simply not documented for the average user. If you want the full documentation of the commands and command output, it’s required that you be part of the Broadcom NDA. Generally, that’s reserved for vendors such as Juniper Networks and very, very, very large customers who have a business need to get low-level access. The official documentation for the Broadcom tables and registers is well over 9,000 pages in length.

Be warned. With great power comes great responsibility. Try to stick with simple show commands. I highly recommend that you never use commands to write values directly to the Broadcom chip for two simple reasons:

1. You have no idea what you’re doing. You don’t have the documentation. Moreover, even if you did, you wouldn’t be reading this chapter as a reference anyway.

2. Any changes you make to the Broadcom chipset are not synchronized with Junos; they will be out of sync. Things break.

Overview

There are three types of primary data structures in the Broadcom chipset:

Tables

Tables contain a set of views.

Views

Views contain a structured data.

Registers

Registers contains key-value pairs.

I will briefly walk you through each step on how to get data from each of the three types of data stores.

Tables

The first step to learning the Junos µkern is logging in:

dhanks@QFX5100:RE:0% vty fpc0

TOR platform (1500Mhz Pentium processor, 255MB memory, 0KB flash)

TFXPC0(vty)#

The next step is to determine how to list what tables exist within the Broadcom chipset. The good news is that’s easy:

TFXPC0(vty)# set dcbcm bcmshell "listmem"

HW (unit 0)

Flags Name Entry/Copy Description

----bC ALTERNATE_EMIRROR_BITMAP 256 Source Modid based blocking mask table

----bC BCAST_BLOCK_MASK 107 Broadcast Block Mask, FeatureSpeci...

-----C COS_MAP_SEL 107 Select one of four sections of COS...

--A-bC CPU_COS_MAP 128 index by COPYTO_CPU reasons code a...

-----C CPU_COS_MAP_DATA_ONLY 128 CPU_COS_MAP Data SRAM for CPU_COS_...

--A-bC CPU_COS_MAP_ONLY 128 CPU_COS_MAP TCAM only view

-----C CPU_PBM 1 Specifies the port(s) that is (are...

-----C CPU_PBM_2 1 Specifies the port(s) that is (are...

----bC CPU_TS_MAP 256 vlan range match table

----bC DEST_TRUNK_BITMAP 1024 Destination Trunk Bitmap Table.

The bad news is that there are nearly 900 tables. Don’t forget that each table has multiple views.

Views

Now that you know there are nearly 900 tables to play around with, let’s see how many views one of them has. A really good table to look at to see the IPv4 Forwarding Information Base (FIB) is the L3_ENTRY table:

TFXPC0(vty)# set dcbcm bcmshell "listmem L3_ENTRY"

HW (unit 0)

Flags Name Entry/Copy Description

------ L3_ENTRY_HIT_ONLY 36864 L3 Hit bit table

------ L3_ENTRY_HIT_ONLY_X 36864 L3 Hit bit table, FeatureSpecific-...

------ L3_ENTRY_HIT_ONLY_Y 36864 L3 Hit bit table, FeatureSpecific-...

--h--C L3_ENTRY_IPV4_MULTICAST73728 L3 routing table IPV4 MULTICAST view

--h--C L3_ENTRY_IPV4_UNICAST 147456 L3 routing table IPV4 UNICAST view

--h--C L3_ENTRY_IPV6_MULTICAST36864 L3 routing table IPV6 MULTICAST view

--h--C L3_ENTRY_IPV6_UNICAST 73728 L3 routing table IPV6 UNICAST view

------ L3_ENTRY_LP 36864 L3_ENTRY LP Control Table.

--h-b- L3_ENTRY_ONLY 147456 L3 routing table with fb_regs arch...

Flags: (r)eadonly, (d)ebug, (s)orted, (h)ashed

C(A)M, (c)bp, (b)ist-able, (C)achable

The L3_ENTRY table has nine views. Take a glance at the data inside the view L3_ENTRY_IPV4_UNICAST:

TFXPC0(vty)# set dcbcm bcmshell "dump chg L3_ENTRY_IPV4_UNICAST"

HW (unit 0)

L3_ENTRY_IPV4_UNICAST.ipipe0[82656]:

<VRF_ID=1,VALID=1,NEXT_HOP_INDEX=0x10e,KEY=0x0021400000e0,IP_ADDR=0xa000007,IPV4UC:

VRF_ID=1,IPV4UC:NEXT_HOP_INDEX=0x10e,IPV4UC:KEY=0x0021400000e0,IPV4UC:IP_ADDR=0xa00

0007,IPV4UC:HASH_LSB=7,IPV4UC:ECMP_PTR=0x10e,IPV4UC:DATA=0x21c000,HASH_LSB=7,ECMP_P

TR=0x10e,DATA=0x21c000,>

L3_ENTRY_IPV4_UNICAST.ipipe0[84164]:

<VRF_ID=1,VALID=1,NEXT_HOP_INDEX=0x10c,KEY=0x003815000360,IP_ADDR=0xc0a8001b,IPV4UC

:VRF_ID=1,IPV4UC:NEXT_HOP_INDEX=0x10c,IPV4UC:KEY=0x003815000360,IPV4UC:IP_ADDR=0xc0

a8001b,IPV4UC:HASH_LSB=0x1b,IPV4UC:ECMP_PTR=0x10c,IPV4UC:DATA=0x218000,HASH_LSB=0x1

b,ECMP_PTR=0x10c,DATA=0x218000,>

A lot of the data is in hexadecimal and requires conversion to be human readable. For an example, look at the following key-value pair:

IP_ADDR=0xa000007

The IP address would need to be translated from hexadecimal into dotted decimal. In this example, the human-readable IP address would be 10.0.0.7.

Registers

The final place to poke around and find data is in the registers. There are over 3,800 registers; it would certainly require a lot of time to look at each of them and make heads and tails of any of it.

The first step is to simply find out what registers exist:

TFXPC0(vty)# set dcbcm bcmshell "listreg -s *"

HW (unit 0)

g3--- ARB_RAM_DBGCTRL ipipe0 ARB_RAM_CONTROL

p3--- ASF_PORT_CFG mmu0 ASF_PORT_SPEED

g3--- AUX_ARB_CONTROL ipipe0 IP auxilary arbiter control re...

g3--- AUX_ARB_CONTROL_2 ipipe0 IP auxilary arbiter control re...

?3--- AXI_SRAM_MEMC_CONFIG cmic0 AXI SRAM MEMC Configuration - ...

g3--- BFD_RX_ACH_TYPE_CONTROL0 ipipe0 Stores the ACH Channel Type va...

g3--- BFD_RX_ACH_TYPE_CONTROL1 ipipe0 Stores the ACH Channel Type va...

g3--- BFD_RX_ACH_TYPE_MPLSTP ipipe0 Stores the ACH Channel Types f...

g6--- BFD_RX_ACH_TYPE_MPLSTP1 ipipe0 Stores the ACH Channel Types f...

g3--- BFD_RX_UDP_CONTROL ipipe0 UDP destination Port number fo...

g3--- BFD_RX_UDP_CONTROL_1 ipipe0 UDP destination Port number fo...

g6--r BKPMETERINGDISCSTATUS0 mmu0 PORT_BITMAP: Current Back Pres...

g6--r BKPMETERINGDISCSTATUS1 mmu0 PORT_BITMAP: Current Back Pres...

g6--r BKPMETERINGWARNSTATUS0 mmu0 PORT_BITMAP: Current Back Pres...

g6--r BKPMETERINGWARNSTATUS1 mmu0 PORT_BITMAP: Current Back Pres...

g3--- BST_HW_SNAPSHOT_EN mmu0 Enable Buffer Statistics Track...

g3--- BST_SNAPSHOT_ACTION_EN mmu0 Enable reset BST_TRACKING_ENAB...

g3--- BST_TRACKING_ENABLE mmu0 Enable Buffer Statistics Track...

There are no views associated with registers; you can pull the data directly from them. A good register to look at is CPU_CONTROL_1:

TFXPC0(vty)# set dcbcm bcmshell "getreg CPU_CONTROL_1"

HW (unit 0)

CPU_CONTROL_1.ipipe0[1][0x3a000300]=0x8524020: <VXLT_MISS_TOCPU=0,

V6L3ERR_TOCPU=0,V6L3DSTMISS_TOCPU=0,V4L3ERR_TOCPU=1,V4L3DSTMISS_TOCPU=0,

UUCAST_TOCPU=0,URPF_MISS_TOCPU=0,UMC_TOCPU=0,TUNNEL_ERR_TOCPU=0,

STATICMOVE_TOCPU=0,SRCROUTE_TOCPU=0,RESERVED_7=0,RESERVED_21=0,

RESERVED_19=0,RESERVED_16=0,NONSTATICMOVE_TOCPU=0,NIP_L3ERR_TOCPU=0,

MC_INDEX_ERROR_TOCPU=0,MARTIAN_ADDR_TOCPU=1,L3_SLOWPATH_TOCPU=1,

L3_MTU_FAIL_TOCPU=1,L3UC_TTL_ERR_TOCPU=1,L3UC_TTL1_ERR_TOCPU=1,

IPMC_TTL_ERR_TOCPU=0,IPMC_TTL1_ERR_TOCPU=0,IPMCPORTMISS_TOCPU=0,

IPMCERR_TOCPU=0,HG_HDR_TYPE1_TOCPU=0,HG_HDR_ERROR_TOCPU=0,FCOE_DST_MISS_TOCPU=0,

CLASS_BASED_SM_PREVENTED_TOCPU=0>

The CPU_CONTROL_1 register shows you how many host packets were sent to the CPU because of a failure.

Broadcom Shell and cprod

You can also use the cprod command to execute the Broadcom shell commands. If you want to dump a list of the tables and views to the FreeBSD file system, use the following:

root@temp-spine-02:RE:0% time cprod -A fpc0 -c 'set dcbcm bcmshell "listmem"' >

/tmp/listmem

0.000u 0.004s 0:00.50 0.0% 0+0k 0+0io 0pf+0w

How did I know that there are nearly 900 tables? Just use wc:

root@temp-spine-02:RE:0% wc -l /tmp/listmem

892 /tmp/listmem

Same thing, but for the registers:

root@temp-spine-02:RE:0% time cprod -A fpc0 -c 'set dcbcm bcmshell "listreg -s *"'

> /tmp/listreg

0.006u 0.006s 0:02.00 0.0% 0+0k 0+2io 0pf+0w

root@temp-spine-02:RE:0% wc -l /tmp/listreg

3845 /tmp/listreg

If you wanted to get a little bit fancy, you can begin parsing the data we saved from the tables:

root@temp-spine-02:RE:0% cat /tmp/listmem | awk '{print $2}' | tail -5

VLAN_XLATE_LP

VOQ_COS_MAP

VOQ_MOD_MAP

VOQ_PORT_MAP

VRF

XLPORT_WC_UCMEM_DATA

The astute reader could then write a shell or Python script to cycle through all of the tables, find all of the views, and then dump all of the table.view data.

Summary

This chapter introduced the Broadcom shell. Although it was littered with warnings, the overall intent of the chapter was to give you additional tools for retrieving data from the QFX5100 series of switches. Although I made every attempt to show relevant Broadcom Shell commands throughout each chapter, there are cases for which you might need to dig a bit deeper. If you’re a truly fearless network operator, I showed you how to get lost in the Broadcom shell. If you ever hit a scenario for which Junos doesn’t provide you with enough low-level information and you have nothing but time on your hands, you could always find it buried somewhere within the Broadcom shell.