Collaborating and Winning

Every weekday at three in the afternoon, we have a recurring thirty-minute virtual meeting called “coffee klatch”. Yesterday, one of our staff members faced a challenge of provisioning a new high-end router at a customer’s data center and he brought it up during coffee klatch. The SFP needed to build the connection was marked as unknown although the logs indicated the SFP were installed. In the end there was no communication. The customer provided the hardware and one of their engineers was onsite, while our engineer was working remotely from home. Immediately, all the brains on the call started spinning, options were discussed, CLI commands were exchanged, and outputs were shared. The error pattern was inconclusive, however, we suspected it might have something to do with third party optics. The vendor of this router in discussion does support third-party optics and although the 100 GB short-range variant in the same device were working perfectly, the 10 GB long-range one acted erratically.Some suggested we might have one of the original and supported optics in stock and could loan it to the customer. Someone else went through our stock list and found the original part and required QSFP to SFP adapter. One of the engineers on the coffee klatch was in the office, so he prepared a loan agreement, while the other engineer called the customer’s engineer and asked him to pick up the part from our office. Luckily, the customer’s data center was only a five minute-drive from our office. After thirty minutes or so, the optic was plugged-in. It was installed and detected properly but there was still no connection. Another brainstorming and it turned out that aside from the SFP, the RX/TX cables had to be rolled over. Then, the line lit up.

You can test and prepare all you want in your lab, but the customer’s environment might just deviate a tiny bit and when you go on a call with the customer’s engineer, you feel the heat and start sweating. Only by collaborating on an informal call, we were able to solve a problem that seemed insurmountable for one person at the time. As the old saying goes: “Talent wins games, but teamwork and intelligence win championships.”