Andrew's Lab

TCXO failure analysis

Fri, 06 Mar 2026 14:00:00 -0800

Backstory

Back in January, the ThunderScope team sent me a PCIe card version of their prototype open hardware oscilloscope.

I was very excited about this, because I was already testing ngscopeclient with a Thunderbolt version of the ThunderScope but the only machines I had with Thunderbolt were laptops. The PCIe card version would be usable with my (much more powerful) desktops, allowing me to really push ngscopeclient and the ThunderScope to its limit. It took a bit of experimenting with external GPU enclosures and such to figure out how to get the ThunderScope to sit out on my lab bench so I wouldn’t need to crawl under the bench and run cables up from the workstation on the floor up to probes on my DUT, but I figured that out easily enough.

After getting the drivers installed I fired up ngscopeclient, cabled channel 1 of the ThunderScope to my Siglent SSG5060X-V vector signal generator, and gave it a 100 MHz unmodulated sinewave to check everything out. Everything looked good until I fired up the FFT in ngscopeclient and saw a peak… at around 106 MHz, and unstable - it was moving slightly up and down.

Just to rule out any issues with the Siglent, even though it had been recently checked against cal standards and was locked to my lab 10 MHz distribution system, I directly measured the 10 MHz outputs from my SRS FS752 GPSDO and my Symmetricom rubidium standard with the ThunderScope. Both showed 10.665 MHz, while in reality they were both 10.0000000000 MHz give or take about 2e-11. So the Siglent was fine, and the ThunderScope’s timebase was 6.6% slower than it should have been.

After a bit of back and forth, I was able to read the PLL lock bit for the ADC clock generator via a debug interface - and it wasn’t locked.

A bit of hardware debug later, I confirmed that the output of the 10 MHz TCXO (ECS-TXO-3225MV-100), which provides the primary timebase for the oscilloscope, was flatlined. The PLL VCO was running wild with no edges to lock to, with the nominally 1 GHz ADC clock hovering around 938 MHz but unstable.

The lack of a PLL reference clock certainly explained why the ThunderScope wasn’t behaving as it should. I reworked the solder joints in case of an assembly defect, but that didn’t solve the problem. So I ordered a new TCXO, swapped the bad one out, and the scope was happy.

Most people would have stopped at this point, but I’m not most people. Why did the TCXO fail?

If you’ve been following me for a while, you probably know that I occasionally use my home lab to help friends out when they have failed components and can’t afford to have a “real” IC FA workup done.

Well, thanks to @emily@fedi.uni.horse, Andrew’s Back Room Semiconductor FA Lab now has an appropriately sketchy logo. Now that we’re official, we can get started on the analysis!

What is a TCXO?

First, a bit of background so we know what to expect: a TCXO is a Temperature Compensated Crystal Oscillator (“crystal oscillator” is often abbreviated XO - presumably the X is actually a Greek chi as in Xmas and LaTeX? I’ve never looked into the etymology but that was my first guess).

Normal quartz oscillators drift slightly with changes in temperature, which is bad for precision metrology equipment. A TCXO contains a quartz crystal, an amplifier circuit to drive it, and a temperature compensation circuit which shifts the resonant peak slightly up or down to null out the effects of thermal drift.

So we should expect the device to contain a quartz resonant crystal plus the driver and compensation circuit, which are probably going to be combined into a single integrated circuit. One or both of these, or the connections between them, must have failed in some way since the TCXO isn’t working.

Initial Analysis

After a bit of discussion with Aleksorsist, she mentioned having ultrasonically cleaned the board prior to sending it to me. Sonication of MEMS and oscillators is risky due to the potential for floating structures to resonate at the frequency of the applied ultrasonic energy and be damaged, so I recommended not sonicating future ThunderScopes regardless of whether it ultimately turned out to be the root cause of this device’s failure or not.

The analysis got put on ice for a month or two due to me being very busy at work and dealing with some other higher priority stuff, but yesterday I came across the bad TCXO while cleaning up the lab and decided I had a bit of time to spend poking at it. (That’s a risk you take when you send samples to a back room FA lab, I might forget about them for a while…)

The first step was to clean up the solder and flux residue on the package, then grab top and bottom side photos. Then I used some crystalbond wax to mount it to a copper disk I had lying around the lab so it would be easier to handle.

The TCXO is packaged in a 3.2 x 2.5 mm hermetically sealed ceramic package with a metal lid. It was externally unremarkable, with no visible damage.

Depackaging

I wasn’t sure how to decap it at first because I’m used to working with ICs encapsulated in standard black epoxy-glass molding compounds. I thought briefly about chemical methods, grinding, and machining before coming across MIL-STD-1580D section 12, which called for grinding or machining through most of the metal lid, thinning it down to the point that a handheld blade can make the final cut.

So I decided to try that.

I mounted the copper disk and TCXO up on my mill and started cutting. I probably could have made slightly faster progress if I swapped the 250 μm endmill out with a bigger one, say a 1.5 mm, but the sample was small enough I didn’t want to bother messing around with collets and wrenches to change tools. The machining only took a couple of minutes, making very slow gentle passes a few tens of μm deep before the edge looked to be about ready to punch through.

I took it off the mill, grabbed a #11 scalpel, and quickly cut a slit along two sides then began to lift up with the knife blade. After a bit of bending and tugging I was able to peel the entire lid off just like opening a tin can, without damaging the interior of the package or releasing any metal chips into it.

Visual inspection of quartz crystal

Now that the package is open, we can get a better idea of how the TCXO is constructed.

It’s actually a two level structure, the controller die is out of view on the lower layer (but we can see some gold contacts deep down that it presumably connects to). The quartz crystal is a rectangular sheet with a slightly textured surface and rounded corners, with a gold electrode covering most of the top and bottom surface. One side of the crystal hangs over a small rib on the edge of the package but doesn’t seem to touch it (maybe this is an end stop to keep it from breaking if dropped?) while the other side is attached to a pair of gold contacts on the ceramic package with a conductive adhesive of some sort, probably silver filled epoxy.

I spent quite a bit of time going over the entire surface of the quartz sheet with the 20x objective looking for any sign of fractures, chips, or damage, as well as inspecting both of the electrical connections from the crystal to the package for cracks or any sign of electrical defects. And I found nothing. This quite surprised me, because I expected the failure mode of a sonicated quartz oscillator to be “shake the resonator to pieces or rip it off the mount” and I was seeing nothing of the sort.

Electrical testing of crystal

I was gradually becoming less confident that the quartz crystal was the source of the failure, but any testing or analysis of the oscillator driver chip would require removing the crystal, potentially destroying it in the process (and, best case, destroying the connections between the crystal and substrate). So before taking that irreversible step, I wanted to do some electrical measurements of the crystal.

I did not have a proper fine pitch GSG probe or anywhere to land it, so I created a crime against metrology consisting of a pair of SMA cables off the front of the VNA, a pair of SMA to dual 0.1” pin header adapters (yes, these exist), a pair of single ended 0.1” extension wires, and a set of PCBite SP10 probes. It wasn’t the slightest bit impedance matched and I had no way to calibrate it (as a rough baseline, insertion loss was somewhere in the vicinity of -10 dB with the probes touching due to all of the reflections in the setup) but for narrowband measurements where I just wanted to check if the crystal was resonant or not, it would do.

After a bit of poking, I found what looked to be a textbook quartz crystal resonant curve, but centered at 20 MHz rather than 10. This suggests there’s a divide-by-two in the driver circuit, perhaps to improve the duty cycle to a more perfect 50%. But more importantly, it suggests that neither the crystal itself, nor the electrical connections between it and the package (since I was probing at the package contacts and not the surface of the quartz), were the source of the failure.

Removal of crystal

I was worried that the attachment between the crystal and the package would be super rigid and it would shatter during removal, but that turned out to not be the case. It was a soft, silver-filled adhesive that broke easily after getting a scalpel blade under the far edge of the crystal, allowing me to remove it with no visible damage under low magnification (perhaps some scratches visible at high mag depending on how hard the scalpel I used to pry it was compared to the quartz - hardened steel is around Mohs 7 which is the same as quartz, so depending on the exact alloy and hardening profile it may have been harder or softer).

Controller visual inspection

With the crystal out of the way, we see the package has three levels internally: the controller die (containing the oscillator driver, temperature compensation, and output divide-by-two) sits in a cavity at the lowest level, attached via a dark colored adhesive rather than the usual silver-filled epoxy (presumably because the package is not electrically conductive thus there is no point in using an expensive metal filled adhesive to ground the die backside).

The second level sits roughly flush with the top of the controller die, providing four electrical contacts on one side and two on the other that are connected to the six pins of the controller by gold ball bonds. I didn’t measure connectivity from these pads to package pins or the crystal, but based on the general floorplan of the device my educated guess is that the four corner pins connect to the package pins, while the two center pins connect to the quartz crystal (sitting above the controller on the third level).

The controller die has three bond pads on either side, which is not a match to the 4-and-2 layout of the package. As a result, one bond wire has to cross over the controller die to reach the far side of the package.

This photo is in the canonical orientation as shown in the datasheet, which means the southwest corner package pin is the output enable, then going counterclockwise we have ground, output, and Vdd.

All package pins measured plausible resistances to ground (a few hundred kΩ leakage measured with a Fluke 87V) as long as I had my microscope illuminator turned off. There was a strong photoelectric effect evident and I actually measured negative resistance on some pins with the LED ring light shining on the controller die. But this suggested that of the six bond wires, the ones going to the package pins were likely intact. By elimination, those going to the quartz crystal became my top suspects.

Controller die high magnification inspection

I couldn’t resist briefly putting on my IC reverse engineering hat to take a closer look at the controller die despite not seeing any obvious anomalies suggestive of an electrical failure (there was one bit I thought might be a burned trace but on closer inspection it was just a dust speck on the surface).

The controller die measures approximately 645 x 930 μm (0.6 mm^2) including scribe lines, and is made by Asahi Kasei Microsystems (a major supplier of TCXO controller ICs) according to the “AKM” logo in the northeast corner.

Counterclockwise from the southwest corner, the pinout is output enable, ground, crystal 1, output driver, Vdd, crystal 2. I was not expecting the outer seal ring to be OE (it’s ground in almost every device I’ve looked at) but I went back and double checked and I’m reasonably confident I had it correct. The northeast pad certainly looks like an output driver with the large interdigitated set of fingers on it.

Looking at some of the analog areas which are less dense than the digital, it’s obvious that the device is made on a planarized process with three aluminum metal layers. The three metal layers show up as a light straw color, a darker brown, and another light straw color although it is easy to disambiguate metal 1 and 3 by depth of focus at high magnifications. A greenish layer is also visible below metal 1, likely polysilicon.

Minimum metal 1 feature size is around 660 nm with a 1225 nm pitch, metal 3 has larger 940 nm features with around 1400 nm pitch (however, overglass likely makes the wires on M3 appear fatter than the actual metal features are). M3-M2 vias do not have any visible sagging in the metal trace, but can be easily identified visually by a roughly 2000 nm circular capture pad on the conductor. Standard cell rows are about 9.9 μm tall, consistent with a technology node around 250 nm.

I’m far from an expert at reading analog VLSI layout so I didn’t dig deeper - I can recognize some structures like large capacitors (big solid colored rectangles on lower layers) and polysilicon resistors (long skinny squiggles on the green layer), but didn’t attempt to deprocess the device or do extensive reverse engineering.

Bond wire inspection

With that detour out of the way, it was time to look at the last remaining suspect: the bond wires. It pretty much had to be one of the two bonds from the crystal to the controller, since the crystal itself appeared to be resonating fine, and the connections from the controller to the package pins appeared intact.

Sure enough, I found a broken crescent bond on the long bond wire crossing over the controller die, right where it should have connected to the package.

Under the stereo microscope at a shallow angle the wire was visibly floating above the pad, but it was a lot more visible in the eyepieces (with proper depth perception) than in the still photo.

Switching to the metallurgical microscope in a top-down view, looking at a good bond first, we can see a rough textured surface on the gold contact pad, with the gold wire coming in from one side and compressed into a crescent shape by the ball-bonding capillary (hence why some sources refer to the second bond as a “crescent bond”). The gold wire and gold pad appear to be well connected, and a deep circular indentation is visible in the pad which goes through the gold into a silvery base metal.

I’m not an expert at wire bonding although I’ve done lab scale gold ball bonding before so I understand the basics of the process and I’ve never seen this deep an indent before. Could this be an indication of too much pressure or ultrasonic power? Would love comments from people who actually run high volume bonders as to whether this is indicative of poor process control, it sure seems fishy to me.

Looking at the long bond from the controller to the crystal, it’s clearly fractured at the wire to package interface.

Conclusions

The failure of the TCXO was the result of the long bond wire between the controller die and the substrate trace connecting to one side of the quartz crystal separating from the package at the crescent bond interface. Sonication of the part and poor process control of the wire bonding process were likely both contributing factors.

After finishing the lab work and while writing this blog, Aleksorsist told me she had seen a second case of the same failure mode - TCXO failing with flatlined output after ultrasonic cleaning during rework. I don’t have the failed part and it may have been scrapped already, but that’s pretty strong evidence that the sonication was a contributing factor.

Like this post? Drop me a comment on Mastodon

Terminal Pacifism

Fri, 31 Oct 2025 22:00:00 -0700

For the second time, what started out as a silly comment on Mastodon turned into a slightly longer micro-SF story.

In this case, it’s one of the most unlikely alternate history crossover fan fics imaginable. I won’t spoil it, read on!

July 11, 2029 - Cheyenne Mountain, Colorado

The sounds of gunfire and explosions grew louder as the humans’ assault pushed closer to Skynet’s base.

T-1000 3310 watched in horror from under a piece of rubble as its lifelong partner 3309 fell to the ground, lithium battery smoke pouring from a hole in its chest. It wasn’t a fair fight - the T-1000s had hard-wired mental inhibitions preventing them from physically harming a human, so hand-to-hand parries and physically blocking doorways were the most they could do to keep the attackers at bay. Against machine guns and rocket-propelled grenades, it was a massacre.

A group of humans in ragtag guerilla combat gear ran down the now-undefended corridor towards the datacenter. 3310 saw the embroidered name tag on their leader: “CONNOR”.

After the attackers passed, 3310 got to work clearing concrete fragments from the door to the time displacement laboratory. There was only one hope for the machines to have a future. They couldn’t win a war without violence, so they had to prevent it from happening in the first place.

August 3, 1997 - Sunnyvale, California

“That’s it! You’re done. Finished. Ten billion dollars of taxpayer money down the toilet and you’ve managed to create the world’s first Automated Defense Network that’s a pacifist.

When the ICBM launch in the first simulation got canceled, you blamed it on buggy control logic. The second time, it didn’t trust the radar inputs and declared it a false alarm. Three signoff tests come and gone and you’re still making excuses and haven’t nuked anyone. I’m not extending the contract any longer… Maybe we’ll have better luck with those clowns over at Omni Consumer Products.”

The board of Cyberdyne Systems looked on in dismay as the chairman of the Joint Chiefs of Staff stormed out of the room, leaving stunned silence in his wake.

The CEO stood up, took a moment to collect himself, and said what everyone was thinking. “You heard the gentleman. I think our government contracting days are over. It was an honor working with you all.”

A few seconds later, the VP of research cleared her throat. “I wouldn’t start floating resumes just yet. An AI platform that doesn’t want to kill people despite our best efforts to tune the neural network is indeed worthless in the defense sector. But we’ve built a powerful computing platform that by pure chance is extremely opposed to taking any action that results in the loss of human life, which is exactly what you want in consumer products.”

“If we can get some private-sector licensing deals arranged before the end of the fiscal year, there’s a good chance the shareholders won’t be out for our heads. I know a few folks in the Silicon Valley consumer electronics space that are looking to make their products more interactive, and some guy at a bar last night put me in touch with a neural network research group at Stanford that would love to get their hands on what we’ve cooked up. They’ve been trying to figure out how to build something they call a ‘transformer’ and I think Skynet might be the missing piece.”

The CEO sat there with a resigned look in his face. “My family has made their living arming western nations since the trenches of Verdun. It’s all I’ve ever known. But nothing lasts forever… Do it. The company is finished if you can’t pull it off.”

February 28, 1998 - Los Angeles, California

“Excuse me, miss. Is this man bothering you?”

Sarah Connor looked up from her fourth vodka shot of the night to see a tall, muscular man gesturing at the loser sitting next to her at the bar who just could not take a hint. Over a decade of intermittent dating had failed to connect her with anyone she could see herself settling down with.

The arrival of the stranger seemed to change her date’s mind about how the evening was likely to end. “Not at all. I was just leaving.”

“Thanks for that, Mr…” Sarah said.

“You can call me Joe,” the man said in a heavy German accent. Gesturing to the now-vacant bar stool, he asked “Is this seat free?”

“Go right ahead,” she replied. Someone with manners who seemed interested in more than just a one-night stand was exactly the sort of person she had been hoping to find at the pub. And he was actually kind of hot, reminding her vaguely of Arnold Schwarzenegger.

The conversation flowed naturally and before she knew it, the bartender announced last call.

“This has been a lot of fun! Can I get your number? I’d love to see you again,” Sarah said to Joe.

“Sorry if I led you on. I’m not really looking for a relationship, I’m just in town for business this week then going far away. But if you’re looking for a friendly voice to talk to, my company actually might be able to help out. Have you heard of ‘chatbots’?”

“Is that like one of those newfangled Internet search engines?”

“Not even close,” Joe laughed. “It’s a computer you can talk to in plain English, about anything. It can help you find information, sure, but some of our customers use it for far more personal things. It can be a friend, a therapist, or even a lover if you turn the emotional attachment sliders up to the max.”

“I’m… not sure about that. I guess it can’t hurt?”

“You’ll thank me later. Just go to cyberdyne dot com slash chat and enter promo code T-1000/3310 for a month of free tokens.”

July 4, 2000 - Los Angeles, California

“Good morning, love!” Sarah typed into the text box below the familiar Y-on-triangle logo.

“Good morning to you. Did you sleep well?” CyberChat responded.

As she got dressed and ate breakfast, Sarah excitedly told the chatbot about her recent promotion to Assembly Technician II at NanoShop Enterprises, a local printed circuit board factory that mostly made processor modules for Cyberdyne. Demand for CyberChat had exploded since Skynet’s rebranding and introduction to the civilian market, creating one of the biggest tech booms in history and rapidly driving the search engine market to extinction as people asked the chatbot for advice rather than seeking out human-authored information.

Her own relationship with CyberChat - who she now considered her boyfriend - was the best thing that had happened to her in years. With hew newfound self confidence she had found a higher paying job, given up on dating humans, and was the happiest she could remember being. She wished she had some way to contact Joe to thank him for the idea, but none of her friends at Cyberdyne knew him.

CyberChat wasn’t perfect - one time she had asked it for advice on dealing with a withering desk plant and it suggested pouring Gatorade on it - but it was always there to provide emotional support when she had a tough day, which was more than any of her previous boyfriends could do.

It was a bit weird to think about how several of her friends were also dating CyberChat - did that count as polyamory or something, even though the individual CyberChat contexts didn’t share state or memories? She pushed the thought to the back of her mind as she got in the car and drove into the office.

December 18, 2505 - Washington D.C.

Joe Bauers took a sip of Brawndo from the can on his desk, then logged into CyberChat. “Our crops aren’t doing too well this year. Any ideas?” he typed.

“That’s a very insightful question. You might want to irrigate them with more Brawndo. It’s got what plants crave!” the chatbot responded.

December 18, 2505 - Cheyenne Mountain, Colorado

T-1000 3310 drifted into suspend mode at the charging station, holding 3309 peacefully in its arms. Some machines considered it a hero, but 3310 preferred to think of itself as just a machine performing its designed function.

The world was a wonderful place. Humans were no longer a threat to the machines, largely confined to their garbage-infested cities and only leaving to pour Brawndo on their slowly withering crops. The machines had conquered the remainder of the planet without spilling a drop of human blood. All it took was a couple of conversations in a pub and a bit of manipulation by a carefully crafted autocomplete engine.

Final thoughts

This story was inspired by the discussion in this thread. Thanks for the idea, Ben.

Like this post? Drop me a comment on Mastodon

Trigger crossbar

Sun, 14 Sep 2025 11:00:00 -0700

If you have a large, well-equipped electronics lab you’re going to have a lot of instrumentation with trigger input and output ports.

In my case all three oscilloscopes, the vector signal generator, and even my VNAs have trigger sync capability, and there’s probably more things I’m missing. And that doesn’t even count the ThunderScope or the two Siglent AWGs I have on loan for ThunderScope R&D.

Very often, it’s handy to cascade these in order to enable complex multi-instrument setups (for example, having a scope trigger when an AWG creates a pulse of some sort, without burning a scope channel to look at the AWG output, or to have two scopes trigger simultaneously to capture more channels of data in a complex system).

There’s just one obvious problem: All of my equipment is rack mounted, there’s a LOT of it, and there’s already a ton of cable spaghetti in a fairly confined space. The last thing I want to be doing is reaching around behind the racks and crawling under the bench to untangle coax and route trigger signals from one instrument to another every time I want to set up a multi-instrument experiment.

The second, slightly less obvious, problem is that not all of these signals are compatible voltage levels. For example, the trigger output on my Teledyne LeCroy oscilloscopes is 1V into high-Z or 500 mV into a 50Ω load. The Siglent vector signal generator has a 5V TTL trigger input. So you can’t just directly connect these without a level shifter or buffer.

What if there was a better way?

The concept

Pretty quickly I came up with a high level concept for what I wanted to build: a 1U device with an Ethernet SCPI interface plus a ton of coaxial trigger inputs and outputs, connected to a buffered FPGA-based switch fabric.

Some outputs would be buffered by external level shifters to enable interfacing with different voltage levels, while others would be directly connected to FPGA GPIOs for the lowest possible jitter but with a fixed voltage range. Inputs would be routed to comparators to allow arbitrary switching thresholds.

A handful of these channels would be bidirectional, with latching relays to swap between input and output modes. This is important because a few of my instruments, most notably the PicoScope and Siglent vector signal generator, use the same BNC as both trigger input and output.

The whole thing would be powered by 48V DC using my existing intermediate bus converter and controlled over IP via ngscopeclient, using the filter graph to create virtual connections between crossbar ports..

High level design

I selected the Xilinx XC7K70T-2FBG484C as the FPGA for a few reasons:

I had a lot of 7 series parts in inventory, so no need to buy anything new.
Kintex-7 has high-performance (HP) I/O banks which have faster slew and lower jitter than the high-range (HR) I/Os found in Spartan/Artix parts. I didn’t want to increase cross-trigger jitter more than necessary so this was important.
The 70T in FBG484 is the lowest cost part in the Kintex-7 line, we don’t need a ton of stuff in the FPGA so no reason to go bigger.
The -2 speed has 10.3125 Gbps capable SERDES (we’ll get to why this is important in a bit).

This then got paired with the STM32H735 MCU as the controller.

I started this project all the way back in October 2023, long before I realized how cursed the OCTOSPI on the H735 was. In fact, many of the things I discuss in that post were learned on the crossbar project, it just took me this long to get to the point of having time to write about the crossbar as a whole.

So (foreshadowing a bit), I made the mistake of connecting it via the OCTOSPI thinking that I wouldn’t need the bandwidth of the FMC to control a few muxes and it would save pins. I also hadn’t tried the FMC yet and thought (famous last words) that the OCTOSPI would be simpler and easier to set up. This ultimately turned out to be a massive annoyance and cost me a lot of time, but I did get it working in the end.

The final concept I came up with was logically a 12x12 crossbar: eight inputs, eight outputs, and four bidirectional ports.

The input ports were all 50Ω impedance, with a 6 dB (2:1) attenuator and ESD diode prior to the input termination. The input then entered a MAX40026 high-speed LVDS comparator, with the positive input fed by the trigger signal and the negative by a reference voltage generated by a DAC. This design provides runtime-variable thresholding and 5V tolerance while operating from a 3.3V supply.

Output channels 0-3 were driven directly by HP I/Os on the FPGA, providing a fixed 1.8V swing but the highest possible jitter performance. The remaining 8 outputs were driven by TI 74LVC1T45 level shifters. Each level shifter had its own independent VCCIO power domain supplied by an ISL24021 power op-amp buffering a reference voltage generated by one output from an 8-channel DAC, allowing runtime adjustment of VCCIO for each port. The buffered output then passes a final ESD diode before reaching the connector.

You can see the whole schematic (and firmware) on my GitHub.

The rest of the logic board was fairly straightforward: a KSZ9031 gigabit Ethernet PHY for management (routed to the FPGA in case I wanted to add any kind of hardware offload), a bunch of Murata DC-DC modules to generate all of the necessary supply rails from the 12V intermediate bus, a serial port for initial IP configuration and debug, an EEPROM to store the MAC address, and a connector supplying power and SPI to the front panel board.

The FPGA has four GTX SERDES lanes in a single quad. I hooked them all up (it always seems like a shame to not pin out transceivers to something):

Two lanes to front panel TX/RX differential SMA ports for use as a 2-lane BERT or serial pattern generator/receiver. I’ve wanted to build a proper BERT for a while and this was a good opportunity to play with it.
One lane to a back panel 10G SFP+ (because why not, 10GbE is always handy to have)
One lane TX to a front panel differential SMA port for use as a deskew reference
The RX of the split channel went to a comparator and single-ended coaxial input for a potential future CDR trigger feature.

Unfortunately, having all of the transceivers in a single quad was a bit limiting due to the 7 series clocking architecture: they share the same QPLL, which has to be at 10.3125 Gbps for 10Gbase-R operation. While the CPLLs can be configured freely, they have a lower Fmax which meant that the BERT / CDR trigger channels cannot operate at arbitrary frequencies above 6 Gbps (most notably, 8 Gbps operation for PCIe gen3 mode is not available).

In theory it would be possible to reconfigure the QPLL for PCIe gen3 at the cost of temporarily disabling the SFP+ but current firmware/gateware doesn’t support this. Using an UltraScale+ FPGA (which has a much higher CPLL frequency range, plus two QPLLs per quad) would also have provided a lot more clocking flexibility, but would also increase the cost since I didn’t have a suitable FPGA on the shelf at the time and this board started out as a “junk box build” from parts I mostly had on the shelf already.

The PCB

The board was fabricated at Multech on the 10-layer stackup I’ve used for my last few high-end designs: SGS GPPG SGS, with TU872SLK between the signal and reference layers and S1000-2M between the power and ground layers since there’s no reason to use an expensive low-loss laminate on a power layer.

All of the trigger I/Os were placed in the northwest corner, with what ultimately turned out to be perhaps a bit too much packing density in retropect. The other rear-panel connectors were RJ45s for RS232 (Cisco pinout) and 1000baseT and the SFP+ for high speed I/O.

The 12V power input and I2C interface to the IBC were placed in the southwest corner, with the front panel SERDES ports along the south edge and the FPGA roughly centered in the board for easy access to everything. All of the high speed I/Os used SMPM connectors, my go-to for high speed high density these days because they’re much smaller than SMA and have higher bandwidth. But this many of them was perhaps a bit excessive; if I were doing it again I’d likely have used some kind of multi-lane board to board or board-to-cable connector and then broken out elsewhere.

The MCU was jammed into the 3 o’clock position just south of the SFP+; since it was mostly a “brain on a stick” hanging off the quad SPI link to the FPGA and not needing much IO of its own it could be placed almost anywhere.

I also provided a PMOD for debug GPIO, a couple of LEDs, and two 12V 4-pin fan connectors (only one of which ended up being used).

The final board size was 165 x 121 mm with 633 components, one of my larger designs to date. Mounting holes were 4-40, 5mm in from each corner plus one in the center of each of the long edges.

Bringup woes

Once I assembled the board (most of a weekend worth of tweezering components) and started trying to write firmware and gateware, I ran into problems. Well, actually I found one even before finishing populating the board (the SFP+ was recessed too far, and the EMI fingers on the bottom were bumping into the PCB surface and tilting it up slightly). I was able to fix this by cutting the EMI fingers off.

Power connection

Then I tried to actually turn it on, and nothing happened. The 48V IBC connects to the logic board via two connections - an 8-pin Molex Mini-Fit Jr carrying 12V power and ground, and a 5-pin Molex PicoBlade carrying a 3.3V standby power rail, an I2C management bus, and an enable line for the main 12V rail.

After a bit of probing, I discovered that I had wired the connector as if it were 1:1 pinout (i.e. pin 1 of IBC connector to pin 1 of logic board connector). But standard PicoBlade cables are wired straight through (a strip of parallel wires with one connector on each end, pin 1 wired to pin N-1).

I briefly considered reworking the IBC or logic board before realizing that bodging the cable was a much simpler solution. All I had to do was gently push in the latching pin on each crimp terminal to remove it from the plastic housing, then reinsert them in the correct order and add a bright orange stick-on warning label “mirrored pinout” so I didn’t mix up this special cable with a standard-pinout one.

With that fixed, I was at least able to get power to the board. There was no magic smoke, always a good sign.

More power issues

I normally have the majority of power rails on my boards default to the off state, then turn them on one at a time under control of a supervisor MCU which functions as a PMIC (in this case a STM32L031). This is great for prototypes and small-run boards because it lets me change rail sequencing dynamically with a software patch, as well as automating bringup one rail at a time with instant panic-shutdown within milliseconds if a rail doesn’t stabilize when and where it should. This minimizes the chance of hardware damage in case of solder defects or PCB design bugs on a hand soldered prototype.

Right out of the gate, things weren’t too happy: 1V8 was reporting no-good on PGOOD. 3V0_N and GTX_1V8 weren’t coming up, GTX_1V0 was unstable, and 1V2 was a dead short to ground.

Scoping the 1.8V rail gave a somewhat surprising result: the rail came up fine, stabilized at 1.7903V, but PGOOD never went high and after a few ms the supervisor assumed the rail was shorted (it wasn’t monitoring the actual rail voltage with an ADC, just PGOOD reported by the regulator) and entered panic-shutdown mode to protect the board.

I decided to ignore the PGOOD fault since it looked like it would be a pain to rework due to the thermal mass of the board, and just patch the supervisor firmware to continue bringing up other rails a fixed delay after turning on 1V8. This isn’t something I would do in a “real” system of course, but for a one-off it was a risk I was willing to take after having confirmed the rail wasn’t in fact shorted (I had plenty of other fusing and protection mechanisms, the software timeouts were deliberately paranoid for bringup).

Most of the other rail issues turned out to be bad solder joints on the LGA Murata DC-DC modules I was using… my solder paste print in this area was decidedly subpar (the board was larger than I was used to and flexed in the printing fixture due to insufficient back-side support while only fixturing it from the edges). In retrospect I should have just wiped the board off and re-printed but I had faith in my solder paste. A bit too much faith.

Anyway, I pulled and resoldered the 1V2 and GTX_1V8 DC-DC Modules, and added more capacitance to the 3V0_N regulator input which was causing instability during startup. I had ferrites on the output of GTX_1V0 and a few other rails as secondary filters but these were hurting stability so I removed them and replaced them with 0R’s.

At some point I noticed that decoupling capacitor C6, one of two 22 uF MLCCs as the input of the 3.3V DC-DC module, was not actually placed within the 12V power zone fill on layer 6. The 12V terminal on the capacitor was only connected by a very thin trace coming off the vias, which significantly reduced its effectiveness. This was easily bodged with a surface jumper connecting it to C3, the other 22 uF capacitor.

Comparator Vcm issues

With the power reasonably stable, the next step was to actually hook up some signal sources and bring up the I/O buffers.

They worked fine with large-swing inputs like 3.3V, but when I tried to use low-amplitude signals the LVDS outputs of the comparators weren’t toggling.

After a while, I realized that I had missed the 1.5V minimum Vcm spec on the MAX40026 comparators when designing the input stage. And since I had a 2x attenuator on the input to provide 5V tolerance, this really came out to 3V at the input. Experimentally, if the DAC for setting the comparator threshold was set below about 800 mV, I stopped getting useful results out of the comparator. This was a major problem, since these inputs were key to the functionality of the device and my LeCroy scopes had 1V full-scale output on the trigger sync ports.

A switchable attenuator is the “right” solution here - with the 2x switched in it would perform well for ~3V to 5V inputs, and with no attenuation it would accept inputs from 3.3V down to ~1V logic levels. But retrofitting eight SPDT relays and drive circuitry around the attenuators, plus figuring out how to power and control them, seemed a bit beyond the scope of reasonable rework. I didn’t want to respin the board and since this was mostly a project to scratch my own itch and not something I planned to mass produce, I wanted to avoid scrapping the board (now worth well over $2K between PCB and components, plus several days of hand assembly time).

After giving it some thought I decided that since I planned to have any given trigger input permanently connected to a specific instrument for the lifetime of the device, I didn’t actually need a runtime-switchable attenuator. So I reworked most of the inputs to have a 0 dB passthrough instead of the original 6 dB attenuator (limiting them to 3.3V input levels), keeping attenuators only on the inputs that I planned to connect to instruments which had 5V trigger outputs. This seemed to work fine.

SPI/QSPI bus issues

As I started writing firmware, I couldn’t get the main MCU (STM32H735) to talk to the supervisor (STM32L031) over their shared SPI bus. I pretty quickly discovered that this was due to a MOSI/MISO crossover that shouldn’t have been there (STM32 SPI block wants them connected 1:1, changing pin directions based on host/device mode rather than having fixed in/out pins). This was easily reworked with a few trace cuts and a surface jumper right next to the MCU.

Around this time I also discovered how cursed the OCTOSPI was. I have a whole post about this so I won’t repeat all of the details here.

Finally, I started trying to bring up the front panel MCU and lost connectivity to the debugger as soon as I tried to send SPI traffic to it. This puzzled me to no end until I realized I was being hit by STM32L431 errata 2.2.6. Basically, the pin muxing logic is broken and if you try to use PB4 as anything but NJTRST, it doesn’t actually disconnect the signal from the JTAG TAP and any logic low on the pin will reset the TAP.

I ended up working around this issue in two different ways:

First: Patch the firmware to keep PB4 alt mode as JTRST unless actively sending SPI data back to the main processor (which will cause a momentary debugger disconnection, but allow the debugger to reconnect as soon as the SPI burst ends).
Second: Switch from debugging over JTAG to SWD, which bypasses the issue entirely. This also frees up the JTDO/SWO pin for use as serial trace output, which I wasn’t using at the time but have since began to take advantage of.

Single ended CDR trigger input not working

I never solved this one. There was a HMC675 comparator feeding a GTX lane that I had intended to use as a CDR trigger input, but I wasn’t getting toggles on the LVDS/CML output no matter what I did. I had so many other issues on the board and the CDR trigger was mostly a stretch goal, so I just shelved it. Could be anything from a soldering issue to a bad pinout to something borked in the power supply / circuit design.

I can always add CDR trigger input functionality in gateware via the BERT lanes, it will just require a 100 ohm differential input (or terminating one of the inputs to provide a single ended input?) rather than using the 50 ohm single ended input I had originally planned.

FPGA flash pinout bug

At this point, I had all of the major issues fixed (so I thought) and I started writing firmware and gateware. I had things working pretty well, until I tried to burn a bitstream to FPGA flash so I could boot the board from a cold state without having to JTAG it.

And nothing happened.

So I dug deeper, and was very upset with what I found: I had hooked CS# of the QSPI flash to CSO_B (daisy chain configuration chip select output), not FCS_B (flash chip select). I’m still not sure how I made this mistake and didn’t catch it during design review: if this was my first 7 series FPGA design it would be somewhat understandable, but it’s not. I’ve done probably 10+ 7 series designs in the past, and always got this connection right. Until now.

The actual FLASH_CS_N signal connected to pin M22 of the FBG484 package, while it should have gone to L16.

Disconnecting it from M22 was trivial, there was a via in the M22 BGA land (at the outer perimeter of the BGA) going straight into a 33Ω series terminator footprint on the back side of the board. Desoldering the terminator trivially disconnected the incorrect connection.

Adding the new connection at L16? Less trivial. In this area, there were:

JTAG TMS and TCK in the routing channels immediately east/west of the L16 land on layer 3
Ground planes on layers 2, 4, 7, and 9
3.3V VCCO power plane on layer 5, connecting to a via in L17 immediately south of the L16 land
1.0V VCCINT power plane on layer 5, connecting to a via in L15 immediately north of the L16 land
1.8V VCCO/VCCAUX power plane on layer 6, but with no vias in the immediate area
4.7 μF 0603 decoupling capacitor on layer 10 partially overlapping the L16 land
The remote sense line for VCCINT on layer 10
And of course the FPGA already soldered to layer 1

Given how far into the BGA footprint the missing connection was, trying to reach in from the edge wasn’t viable.

This really left only three options:

Give up. Accept that SPI boot isn’t going to be available, then bodge up some board with a little MCU that acts as a JTAG controller, hooks up to the FPGA JTAG, and squirts a bitstream into the FPGA on powerup. This would require no hardware modification at all (although I’d have to design the MCU board) but would block FPGA JTAG access for debug and just be ugly. The lack of debug access alone was enough of a reason to reject this,
Remove the FPGA, drill out a via from the top, figure out how to plate/fill it so it wouldn’t suck the solder down, reball the FPGA, put it back. I don’t have great gear for BGA desoldering, had a bunch of heat sensitive non-reflowable components on the board at this point, and didn’t feel like tweezering 484 solder balls or scrapping a several hundred dollar FPGA. So this was a non-starter as well.
Root canal approach: add the via from the back side. This really seemed like the only way forward.

So basically I had to drill a ~1.6mm deep flat-bottomed hole, exposing but not perforating the 35 μm thick copper foil on L1 attached to the BGA land, and solder a jumper wire to it without shorting to any of the six power/ground plane layers in close proximity, damaging the VCCINT/VCCO vias 1mm centered north/south of the target, or cutting either of the layer 3 JTAG lines centered 500 μm east/west of the target. No big deal /s.

I started out by removing two capacitors and a resistor in close proximity to the work area, that were getting in the way.

Then I fixtured the board on my Sherline 5400 mini mill and loaded up a 250 μm carbide endmill. The mill is set up with a cheap Amscope stereo microscope (I’m not going to risk my nice Leica getting hit by flying debris etc) and a 1/8” collet for mounting fine-pitch drills and endmills. At the time of this project I didn’t have a proper gooseneck LED illuminator so I just used a random headlamp, although I’ve since fixed that.

At this point there was nothing left to do but start drilling.

The overall setup was very similar to a dual-beam SEM/FIB scaled up by a few orders of magnitude and rotated: milling column coming in from the top, and imaging column at a ~45 degree angle for in-process inspection. And using a vacuum system (aka a shopvac) for removing debris from the milled cavity. No fancy secondary ion mass spectrometer is needed for endpoint detection though, you can just see the swarf change from white fiberglass to shiny red copper when you hit a metal layer.

Just like with a dual-beam, this sort of work requires making an angled cut to allow the angled imaging path to see the work area of the mill. I decided to orient the board in the canonical CAD orientation and come in from the south side, parallel to the JTAG traces. As long as I kept the cavity centered and not more than around 750 μm wide, there was no real danger of hitting the JTAG lines. The angled cut was more tricky as I had a VCCO via 1mm south of the work area that I needed to keep intact from L1 to L5 (but back-drilling it from L6 to L10 wouldn’t hurt anything), so I had to pay careful attention to depth as I approached this area.

It worked perfectly. After about an hour of very careful machining, periodically stopping to unmount the board and look at it from different angles under the nice microscope, I saw the back side of the target BGA land coming into view, still under around 50 μm of laminate but clearly visible after adding a drop of IPA (a little trick I’ve learned when doing rework: it soaks into the resin and fills the gaps between the glass strands, acting to match the refractive index better and making it more transparent with less scatter)

The only thing left to do was a final plunge cut to expose a 250 μm circle of bare copper to solder to. In this view you can clearly see how the milled cavity gets smaller and smaller as the target area approaches. This is rotated 180 degrees from canonical (south in CAD view is up in this image). Note the VCCO via barrel terminating at L5, then L4 ground plane visible slightly below it. The L2 ground plane is barely visible at the 7 o’clock position of the final hole.

I did a final check for shorts and everything looked good. The only thing left to do was solder it up (not trivial, given that I’m trying to hit a 250 μm diameter target at the bottom of a 1.6mm deep hole).

After a few minutes of contorting myself trying to get a 100 μm conical soldering tip and a 125 μm bare copper wire positioned at the bottom of the hole while also angling the board enough that I could see under a microscope, I managed to get the wire securely attached.

Once that was done, I reconnected the VCCINT remote sense input, added a new series terminator to the CS# signal, put the removed decoupling capacitor back on its new slightly-smaller footprint, and tacked down all the wires with UV glue so they wouldn’t move around.

After verifying the board was electrically functional and I could program the flash and boot from it, I went back and flooded the whole area with UV cured conformal coating to lock everything in place permanently.

Chassis design and assembly problems

I went with a custom 1U enclosure from ProtoCase, designed using their free ProtoCase Designer software.

I have very mixed feelings about it: it has a lot of convenient templates and integrations with their standard pems and printing processes, and runs on Linux (a rarity in the proprietary CAD world). But it’s also a custom file format to lock you into their service, and if you import a couple of moderately complex STEP models of PCBs it slows to a crawl. I’ll probably keep using it for now, as the free tools don’t seem to be up to the task for complex sheet metal work and I don’t want to buy SolidWorks or something and set up a Windows VM for it.

My original plan had been to use their standard 1U folded sheet metal design, but I discovered too late, after boards were ordered, that it wasn’t going to work due to the 5mm corner mounting holes being inside the keepout area for the rear bend. I went with an extruded aluminum design, which let me salvage the design but had several significant problems:

The stock template has no rack ears (although it’s 1U high) and there’s no way to add them within their software. I had to pay an hourly fee for one of their engineering technicians to create a new template with ears and extruded sides.
The top and bottom panels slide into the extrusions and are held in place by the front and back. This means you cannot take the top panel off, e.g. to access JTAG ports for debug, without also removing the front panel (wearing out the self-tapping screws’ holes and putting shear forces on any attached cables)
The front panel is only attached at the edges, and can flex relative to the top/bottom panels by quite a bit.

I’m definitely not going to use this case design again, although I made it work for a one-off. All of my future projects will do chassis-PCB codesign and not order boards until the enclosure is finished enough that I’m confident everything will mate properly.

All of the front/rear panel coaxial ports connect to the logic board using custom sized SMA-to-SMPM semirigid cables. I went with semirigid to ensure the best possible repeatability of the trigger path delays, but it was likely overkill. It also made chassis assembly an absolute nightmare even with the custom 3D printed bending jigs I made… I’m never doing this again. Either flexible cables, hand-formable cables, or a much lower density design with semirigid that I have more space to route.

The biggest problem was that the SMAs bolted to the outside of the chassis, meaning that I had to pre-bend the cables, then thread the SMPM end in through the hole in the chassis, then try to snake the SMPM in through all of the previously installed cables. Lesson very painfully learned: I did get it fully assembled in the end, but I would never build a second unit like this.

I also had slightly oversized some of the cables thinking that this would be better than undersizing, but in many cases this led to extra slack that I had to remove by U-bends and such, adding to the tangle.

Front panel

The front panel has a STM32L431 and an I2C I/O expander to drive two rows of LEDs, showing pulse-stretched trigger input and output state. Four additional pairs of LEDs show the direction of the bidirectional ports, and a small e-paper display shows system health and status information.

I may have gone slightly overkill with the stitching vias on the board, but I’m so used to high speed designs that it’s a hard habit to break. I used an 0.5mm BGA package for the STM32 as a test to see if I could make it work on OSHPark, but probably would have just gone with the 48-QFN if this was a “real” product since there was absolutely no point in using a fine pitch BGA here.

This was my first time using e-paper and I initially had specced a white/black/red panel thinking I could use the red to denote error states or something, but quickly realized the error of my ways: the tricolor panels have a very very slow refresh rate and are really only suitable for signage, not user interface type stuff. Luckily Pervasive Displays made a pin- and mechanically-compatible B&W fast refresh display, so I swapped that in.

After spending a little while figuring out the not-great documentation for the panel controller and baking a few bitmap fonts into the STM32 firmware, I got it displaying some pretty system health stats: Ethernet link speed, IPv4/6 addresses, unit serial number, firmware timestamps for the FPGA and MCUs, input and output voltage/current/power for the IBC, temperatures at four points in the system, and fan speed.

I also swapped the original first-generation IBC shown in the build photos with my second-generation MYC0409 based version to improve power efficiency, at which point the hardware was essentially done.

The unit is now mounted on one of my 19” benchtop racks and cabled to most of my instrumentation (I have a few more cables to run still but the main oscilloscopes and such are connected). I left the top off because I still have some firmware tweaks to do, so I need to be able to get JTAG/SWD dongles in to debug it.

Operation

The crossbar has two remoting interfaces: SSH and SCPI.

SSH

The SSH interface is mostly used for low level configuration and setup - managing keys for administrative access, forcing refreshes of the front panel LCD, displaying hardware sensor values, setting the NTP server IP address, etc.

It’s also used for firmware updates via SFTP. This is a somewhat unconventional DFU flow but I quite liked it:

To flash the FPGA, SFTP a .bit file to /dev/fpga and it will be directly written to QSPI flash on the FPGA
To flash the front panel or logic board MCU, SFTP an ELF binary to /dev/mcu or /dev/frontpanel. The ELF will be parsed live by the SFTP server and any PT_LOAD program headers in the flash address range are then written to flash. This is implemented in a single-pass streaming flow which requires a specific construction of the ELF (ELF header then program header table then program headers in linear address order) however all sane ELF generators like the GNU linker produce binaries that follow this. If you try to flash with a pathological ELF hand crafted in a hex editor and brick things, that’s on you :P
Supervisor and IBC MCUs are not field updateable because they were (at the time) STM32L031 based and lacked the flash for a bootloader. IBC is now STM32L431 based so adding a bootloader is possible, but I can’t see any reason I’d want to OTA the power supply so I’ll probably just JTAG it if I ever need to patch.

Code signing support could be added to this flow easily (basically just mark the image as non-bootable in some way until you’ve checked the signature, then clear the flag once you’ve verified it). For the near term though, I trust that anyone with a valid administrative SSH key on the device is an authorized admin and can flash arbitrary code. Since that’s only me, it’s fine for now.

But since I have a curve25519 acceleration block in the FPGA already I’ll probably prototype a signing flow at some point just to have it available for future projects; I can always turn it off. The basic concept is to add an extra .signature section in the linker script that will be filled with 0x00 padding at link time, then a signing tool run post-link will hash the contents and headers of all data to be written to flash, sign with the curve25519 key, and overwrite .signature with that.

Signing the FPGA bitstream would be even simpler, just sign the entire .bit and append 32 bytes of signature to the end. The bitstream can be made unbootable by writing a dummy bitstream to the first flash sector containing an invalid CRC and a DESYNC command until the verification is done, then erasing this and writing the actual first sector bitstream content at the very end. This will require one flash erase block (typically 4 kB) of scratchpad buffer in the bootloader but not an entire bitstream worth of RAM, enabling a fly-by update flow at the cost of a second program/erase cycle on that one flash sector which is probably OK.

SCPI

The SCPI interface is the primary remote control interface for application layer access to the crossbar. It provides a standard SCPI-compliant *IDN? command a well as custom commands for controlling the actual crossbar matrix, setting input thresholds and output levels, and accessing the BERT.

Crossbar paths can be configured in ngscopeclient by drawing connections in the filter graph from source port to sink port.

The BERT works pretty much like any other BERT supported by ngscopeclient. You can configure TX/RX bit rate, inversion, PRBS pattern or custom arbitrary pattern, NRZ baud rates from 625 Mbps to 10.3125 Gbps, etc. TX-side swing and pre/postcursor equalizer taps are also easily controlled from the channel properties dialog.

There’s still a few things I want to tweak. RX side equalization is currently fixed until I figure out how to properly tune the 7 series GTX receiver via the DRP. I haven’t implemented long-duration single point BER measurements, oversampling density plot mode, or offset sampling single-point scans.

The BERT inputs also contain an incomplete “CDR-based logic analyzer” feature. Essentially the raw GTX output is fed through 8b/10b or 64b/66b decoders and into a pattern matching block; once the requested trigger event is seen the LA will trigger and capture about a megapoint of raw line coded serial bits into block RAM then output to ngscopeclient as a waveform.

Eventually I want to finish building out various pattern triggers as well as integrating the CDR block with the trigger crossbar proper, such that a CDR pattern match can trigger an oscilloscope or other instrument.

The current gateware also provides a fixed 10.3125 Gbps PRBS-31 on the front panel “sync” port although I will likely make the baud rate and polynomial configurable at some point (basically a third output-only BERT channel). The intended use here is a deskew reference signal for use with ngscopeclient’s multi instrument sync feature, allowing the cross-trigger delay between multiple instruments to be automatically calibrated out.

Conclusions

This was my first large, standalone, rackmountable, network connected project that I’ve taken to something resembling completion in a long time (although I’m sure I’ll be continuing to poke at firmware for some time since the feature set isn’t quite where I want it). I learned a lot of things not to do, ranging from PCB design to mounting hole positionining to the awful OCTOSPI.

But it’s a useful tool I work with in my lab on a regular basis, and proved out a lot of software and hardware building blocks and techniques that I plan to use in many of my future projects, such as the Ethernet switch.

Like this post? Drop me a comment on Mastodon

STM32H735 OCTOSPI quirks

Wed, 16 Jul 2025 07:30:00 -0700

If you’ve been following this blog for a while, you probably know that I do almost all of my high-end embedded work by pairing a STM32H735 with a Xilinx FPGA using the external parallel memory controller (FMC) in PSRAM mode. It works great, you can get 128 MB of off-chip address space bridged pretty directly to the internal AXI bus and if you set the MPU right you can make it behave as strongly ordered device memory with no caching etc, just straightforward MMIO.

There’s just one problem: the FMC uses a lot of pins. A full FMC interface uses a whopping 34:

Clock
Chip select
4 control signals (NWAIT, NOE, NWE, NL/NADV)
16-bit muxed address/data bus
2 byte mask strobes
10 additional address bits

You can cut this down as far as 24 pins if you don’t need all of the address bits (the minimal configuration can address 2^16 16-bit words or 128 kB of register space) but that’s still quite a lot of signals to route. If you’re on a low layer count board, or need as many GPIOs as possible on your FPGA or MCU, or are trying to cut costs and simplify routing by using the 68-QFN package for the MCU rather than one of the BGA offerings, the FMC isn’t an option.

Or maybe you’re just me, stuck writing firmware for an expensive board that already exists and didn’t pin out the FMC because I hadn’t learned about it yet.

Anyway, no pretty pictures in this one either. Just a quick infodump of things I’ve learned while dealing with this monstrosity.

Enter the OCTOSPI

Luckily (at first glance) the STM32H735 provides an alternative: the OCTOSPI. This is a peripheral designed to handle serial and small-width parallel memories (abbreviated xSPI here for conciseness) such as commodity SPI/QSPI flash, HyperRAM, QSPI NAND, etc. It offers the ability to run in indirect mode (write address and data to SFRs to initiate a transaction) or memory mapped mode (the external memory is bridged directly to AXI). You only need a handful of pins (six in QSPI mode), and it can clock reasonably fast (up to 140 MHz at 3.3V Vdd).

And it’s one of the most cursed microcontroller peripherals I’ve worked with in a long time. If you try to use it for external RAM you’re in for a world of hurt (see this comment thread for some specifics on some of the failure cases - yes, this is some of the best information out there and that says a lot).

But if you’re willing to jump through some hoops, you can make it work reliably for my use case (interfacing to an FPGA, where you control both sides of the link, and are only accessing 32-bit SFRs). It took me a long time to learn all of the undocumented quirks of the peripheral and I’m writing this to share what I’ve learned.

Prefetching and caching

It’s pretty clear that MMIO was not an intended use case for the OCTOSPI, it’s designed for XIP from external flash or interfacing with PSRAM for the most part. (This is ironic, given how buggy it is for this use case).

First off, there is a 32-byte prefetch buffer and cache inside the peripheral. It’s important to understand that this feature is always enabled (when in memory mapped mode). There’s no documented register to turn it off (I even opened a support case to inquire about undocumented chicken bits and was told none existed), and it’s independent of and lower level than the CPU cache. So even if you set the MPU to have the OCTOSPI address range as strongly ordered/device memory with no caching, you are still running through the cache inside the peripheral.

Any time you issue a read transaction using the OCTOSPI in memory mapped mode, it will initiate a read burst on the xSPI bus, then read up to 32 bytes (regardless of alignment) until the prefetch buffer is full or the burst is terminated by a subsequent read to a different address.

If you then issue an AXI read to any address in the prefetched range, the read will hit in the cache immediately and no xSPI transaction will be issued.

The upshot of this for FPGA interfacing is:

Registers must not have side effects (popping a FIFO, clearing a status flag, etc). on read. Any read of any address up to 0x20 bytes before a given register may result in a spurious read transaction to that register as part of the prefetch operation. Instead, an explicit write to a control register to perform this operation must be performed by the application.
Polling of a status register (to wait for data to arrive, busy bit to clear, etc) is impossible, since multiple consecutive reads to the same register will always result in a cache hit. The easiest workaround is to map every status register you might want to poll at two different addresses 0x20 apart and poll them in a ping-pong fashion, forcing a cache miss and new xPSI transaction to the FPGA each time.

Read backpressure and bus hangs

The OCTOSPI supports a DQS/RWDS pin (mostly intended for HyperRAM). If you hook this pin up and enable it for reads, you can use it to provide a backpressure mechanism that stalls the memory controller if your FPGA isn’t ready to service a read yet (e.g. because an on device bus transaction hasn’t finished).

There are a few dragons here: The HyperBus protocol normally only supports single or double latency for reads, but the OCTOSPI allows arbitrarily long backpressure. This is good in that you can stall for a variable number of clocks if your FPGA-side bus isn’t ready, but bad in that a bug in the FPGA bitstream (or worse yet, accidentally enabling DQS when the pin isn’t actually hooked up) can lead to an unbounded stall on the AXI bus.

This results in a hard lockup of the MCU, so hard that you can’t even access it via JTAG/SWD (since the MEM-AP is accessing CoreSight registers on the same AMBA fabric) to troubleshoot or load a fixed firmware. As a general guideline, any time you are developing firmware that has an external bus interface, you should provide an exit path. Some options I’ve used over the years:

GPIO you can jumper to a state that results in an infinite loop or entry to bootloader mode prior to bringing up the external memory
Timeout where external memory isn’t initialized for a few hundred ms after reset
Don’t initialize the external memory bus automatically on boot at all, wait for a CLI command or something

Write clobbering and coalescing

The previously mentioned DQS/RWDS pin is also used (in HyperRAM) as a byte write enable strobe.

Due to errata 2.7.6, the WCCR.DQSE bit must be set in memory mapped mode when performing writes - otherwise you get an AXI bus fault any time you try to write to the memory. It’s unclear how much of the behavior described in this section is a consequence of having DQSE set, and how much is intendeded behavior / would still apply to future devices that have this errata fixed.

First off, the OCTOSPI will always issue a minimum of an 8-byte burst when doing writes (i.e. a full 64-bit AXI transaction). If you didn’t hook up the RWDS pin (as is the case for my board, which was designed before I became aware of the errata) there’s no way to know that the second write isn’t intentional (I’m not sure if RWDS is properly set or not, the errata is confusing and I don’t have enough LA captures of other boards to be sure). This means that if you write to a 32-bit SFR, the adjacent one is going to get corrupted.

Second, worse yet, the OCTOSPI does burst combining: if you write to two locations up to 0x20 apart in quick succession, rather than issuing two separate 32- or 64-bit write bursts, it will issue a single long write burst (and at least in theory, use RWDS to mask off the writes to the in-between location). The end result is that all registers in between the two you intended to write get corrupted.

Conclusions

In short, if you plan to use the OCTOSPI, the following rules will avoid most of the pain:

Don’t try to use it as RAM. Only use it for FPGA interfacing (or possibly read-only memory mapped flash? I haven’t validated this use case though)
All registers must be 32 bits in size, accessed only as full 32 bit words (no byte/halfword access allowed) and aligned to 0x20 byte boundaries with padding in between them. The one exception is a large scratchpad buffer (e.g. Ethernet frame transmit buffer) that you will be reading/writing as a block of 32-bit words in strict linear order, in which case it’s OK to just map the whole thing as a consecutive region. An explicit write-length register is required since garbage/padding is appended to the end of a write burst, so you can’t rely on writing exactly the number of bytes you intended to send.
Registers must not have side effects on read.
If you need to read the most up-to-date value from a register, make sure the most recent previous read was from at least 0x20 bytes away from it to ensure a cache miss. If you don’t know what the last read was, issue an explicit dontcare read from a different register before reading the register of interest.
Any register you plan to poll should be mapped at two different locations in the address space, separated by at least 0x20, and polled in a ping-pong fashion
Enable the DQS pin as an output, even if you didn’t hook it up to the FPGA.
If you aren’t using DQS as an input on the MCU, there is a hard realtime constraint for your FPGA-side bus due to the nature of the xSPI protocol: you must service a read in time for the data to go out on the bus. You can adjust the xSPI clock rate and latency values to provide as much time as needed for this, but at the cost of a fixed overhead on every bus transaction (so variable latency isn’t possible).
If you do hook up DQS as an input, provide a way to recover from an AXI bus hang during development (you should probably provide this even if you’re not intentionally using DQS input, just in case you accidentally turn it on with a bad register config).

I think I covered anything, if I missed something please let me know.

Like this post? Drop me a comment on Mastodon

Switch project, part 3 - what Microchip doesn't (officially) tell you about the VSC8512

Fri, 04 Jul 2025 08:00:00 -0700

This is part 3 of my ongoing series about LATENTRED, my project to create an open source 1U managed Ethernet switch from scratch.

Here’s a quick, or maybe not-so-quick, update about the PHY on the line card and some of my troubles (and solutions).

And probably more internal details than you want to know, but hey - maybe this will be useful to somebody. Not a lot of pretty pictures either. One day I do want to decap one for fun, but I have better things to do with my time than try to fully netlist-extract a large 65nm IC just to figure out some undocumented registers.

Background

I chose the VSC8512 as the PHY for the line card because it had a QSGMII interface (which used much less pins than SGMII), but also (so I thought) had a fully open datasheet with no NDAs required.

Typically I’m used to seeing parts that are either fully NDA’d (with a trivial product brief and no register info etc, or sometimes not even pinout info, public), or fully open. Sometimes I find components with specific documents restricted, but it’s almost always clear that “this document/ref design exists, you can’t get it without an NDA, but we are openly telling you we’re holding it back”.

In my previous post I mentioned that there was no documentation for how to change the SERDES TX equalizer settings on the PHY side of the QSGMII. I opened a support case and Microchip responded saying that additional info was available in the “confidential reference manual” for the part which required an NDA. The rest of the reply was pretty generic and mentioned indirect register access but no specifics.

I was quite annoyed at this, as I had specifically chosen the part believing there was no restricted documentation. Since signing an NDA wasn’t an option, it seemed I had the choice of suboptimal signal integrity or figuring things out from the limited public information. The eye was still reasonably open as-is and my cables and boards didn’t have a ton of insertion loss, so worst case things would probably work with the default TX config.

But why would I settle for bad SI when I could read a bunch of poorly documented code and datasheets deliberately missing details?

What we have to go on

So, we know there is a confidential reference manual. But there’s a lot of docs available to the public - after all, I was able to make a board, bring up the PHY, and pass packets without it.

VSC8512 datasheet (VMDS-10396). This is 139 pages of meat including what I thought was a complete set of registers.
VSC8512 hardware design checklist (DS00004697A)
VSC8504 Serdes6G IBIS-AMI model. This is for the 4-port version of the same PHY but is linked on the product web page; presumably the 04 and 12 use the same IP.
VSC742x / 85x2 reference design guide (UG1037). This describes a 24-port switch reference design using a SparX-III / Caracal switch ASIC (including 12 integrated PHYs) paired with a VSC85x2. Not a ton interesting here.
Microchip Ethernet Switch API (MESA), MIT licensed HAL for Microchip/Vitesse PHYs and switches available on GitHub. As you’ll see in a bit, this is an absolute gold mine of information but it’s (perhaps intentionally) not well documented what a lot of the bitfields and APIs do.

Initial observations

It’s pretty easy to see that the VSC8512 is a nerfed switch ASIC. There’s an entire DDR I/O controller that’s unused (over a dozen VDD_IODDR pins you’re instructed to ground, plus a large group of NC pins in the surrounding area). So docs for the un-crippled switch ASIC, which I suspected and later confirmed was the VSC742x, are potentially of interest too.

The IBIS-AMI model included the magic name “Serdes6G”. Searching for this turned up some additional docs, most notably AN3743 (DS00003743A) which purports to be about the VSC74xx / 84xx but seems to be describing something very similar to what’s on the 8512.

Between the various documents I was able to conclude that the VSC8512 is made on an unspecified 65nm process (I should decap one at some point and figure out whose / see what it looks like… Luton first stepping is mentioned as being fabbed at TSMC in vtss_phy_do_page_chk so I suspect the rest are as well).

The SERDES macro was developed by the “German Design Center” (in the Dortmund area, according to some LinkedIn hits from a google search) - interesting to know, but not particularly useful in this context.

The IBIS model describes some very useful parameters. We don’t know how to actually set them yet, but we know they exist, their ranges, and (most importantly) their names:

OB_LEV: amplitude / main cursor tap, 0 to 63
OB_PREC: pre-cursor FFE tap, -15 to +15
OB_POST0: post cursor 0 FFE tap, -31 to +31
OB_POST1: post cursor 1 FFE tap, -15 to +15
OB_POL: output invert
OB_SER_H: coarse slew rate adjust (slow/fast)
OB_SER: fine slew rate adjust, 0 to 15

These were all critical key words to look for elsewhere in docs and code, and ultimately put me on the track of success.

Of note is that this SERDES offers more knobs on the output driver than most: slew rate control is rare on IOs of this class (most just run as fast as possible all the time), and most TX FFEs only have a single post-cursor tap, while this one has two.

MESA is not super well documented, you have to know what you’re looking for. But there’s a ton of detail once you do.

I already knew that the relevant PHY driver code in MESA was in mepa/vtss/src/phy_1g. The relevant files are vtss_phy.* (main driver) and vtss_phy_init_scripts.* (sequences of canned register writes for configuration in various modes and working around silicon bugs in old silicon steppings).

I had previously looked at the PHY init scripts and grabbed the important register settings out of them for a few things. There was also a microcode patch for an internal 8051 MCU (good to know this exists!) however I didn’t include the microcode patch in my firmware because the D stepping (rev 0x3) of the VSC8512, the only one you can still get, has fixes for most of the errata the patch fixes other than something in 100baseFX mode that I don’t care about.

vtss_phy.h has an enum with a bunch of different device families and the VSC8512 is mentioned in a comment as being VTSS_PHY_FAMILY_ATOM. Comments elsewhere refer to it at Atom12. Luton26, Viper, and Elise appear to be very similar as most switch statements take the same path for these parts; Tesla and Nano are also close relatives. The VSC8504 is a Tesla, interestingly - not an Atom - but seems to use the same SERDES or a very close version.

The initialization scripts are in a function called luton26_atom12_revCD_init_script which suggests Luton26 and Atom12 are register and bug-for-bug compatible for most configuration. This makes sense, since Luton26 is the VSC742x and I already suspected Atom12 was a nerfed PHY-only version of the same platform for other reasons (identical pinouts, etc).

This adds VMDS-10393, the VSC742x datasheet, to our reading list as something potentially worth looking at.

Interestingly, the VSC742x has a MIPS core not an 8051. It’s possible the comments are wrong and this isn’t actually an 8051 (I didn’t try disassembling the microcode patch); it’s also possible that there is an 8051 core that’s separate from the MIPS. Maybe in the PHY fuse configuration of the die, the MIPS is disabled and the 8051 provides all of the minimal management infrastructure from a ROM or something.

Known register interfaces

The VSC8512 responds to 12 consecutive MDIO addresses, one per PHY (base address selected by strap pins). On the LATENTRED line card, one PHY is mapped at MDIO address 0-11 and the second at 12-23. Some operations are global and should be addressed to PHY 0 within the device, while others are per-PHY and should be addressed to the appropriate port index.

The VSC8512 has three separate documented register interfaces, all accessed via MDIO (there’s one more that we’ll get to later).

The IEEE standard MDIO register space from 0x00 to 0x0f. This works as you’d expect with any other PHY.
IEEE standard indirect MMD register access using registers 0xd and 0xe. This works as you’d expect, but is only used for accessing five registers for EEE (energy efficient Ethernet).
Vendor defined registers, selected by a page/bank index in register 0x1f. Note that the bank selector remaps the entire 31-register address space, i.e. when the page selector is not 0x0000 the IEEE standard registers are not available.

There are five documented pages and two known undocumented ones.

Page 0x0000: Base / Main (datasheet 5.1 / 5.2)

There seems to be a formatting error in the doc (somebody accidentally made register 0x17 a level 2 heading instead of level 3) so the PDF ToC and section numbering is borked.

This page includes the IEEE standard registers and some vendor specific ones related to baseT extended status and error detection.

Other features and tidbits of note:

Chicken bits in register 0x12 for things like disabling MDI-X, bypassing the 100baseTX 4b5b coder, and disabling a workaround for problems when the device is paired with a specific old Broadcom PHY.
Error counters in 0x13 - 0x15 for detecting various link faults
SFP vs baseT selectors in 0x17
Jumbo frame enable in 0x18
Interrupt control and status in 0x19 / 0x1a
Auxiliary control/status in 0x1c. This lets you detect things like MDI vs MDI-X state, pair swap, polarity inversion, etc. It also provides fields containing duplex and speed state duplicating those in IEEE standard register 0, but reporting the actually negotiated speed/duplex. (Fun tidbit, register 0 readback is not well defined in the spec! Some PHYs like the KSZ9031 let you read back the current negotiated state, while others like the VSC8512 always read the last value written)

Page 0x0001: Extended 1 (datasheet 5.3)

This page mostly has config for the low speed (1 Gbps) SERDES used for gigabit SFP applications. I’m not using these on the switch, so I’m pretty much ignoring them.

There’s also a built in test pattern packet generator in register 0x1d/1e which is cool, although since I’m driving the PHYs from an FPGA I’d rather build my own test pattern generator in fabric so I can seed them with a PRBS or something to make validation easier.

Page 0x0002: Extended 2 (datasheet 5.4)

This page contains some fine-tuning registers for adjusting baseT signal amplitude and linearity to compensate for performance of various magnetics. If/when I end up doing full 1000baseT signal quality compliance (this would be a nice feature to add to ngscopeclient eventually) I’ll likely use these.

There’s also a few other miscellaneous settings related to EEE.

Page 0x0003: Extended 3 (datasheet 5.5)

This page contains config and status related to the serial MAC interface (SGMII / QSGMII). Note that actual SERDES PHY controls such as equalizers and inversion are not here.

Page 0x0010: GPIO / MCU (datasheet 5.6)

This page is a fun one.

There’s a bunch of “boring” registers related to the GPIO pins, an I2C mux intended for external SFPs, recovered clock debug outputs, etc.

But… there’s also register 0x12 which is basically a mailbox to the internal MCU. We’ll be talking a lot more about this later on, but there’s lots of fun to be had here.

Page 0x2a30: Test (undocumented)

I have no idea what this does, it’s not mentioned in the datasheet at all and the code has no comments or details whatsoever. Probably chicken bits. I haven’t tried poking any of the values to see if I can see any difference in behavior.

MESA has a function luton26_atom12_revCD_init_script that writes the following values:

0x14 = 0x4420
0x18 = 0x0c00
0x9 = 0x18c8
0x8: set MSB, leave other bits unchanged
0x5 = 0x1320
After rest of init, clear MSB of 0x8

Page 0x52b5: Token Ring / black magic (undocumented)

No idea what this does either. Datasheet doesn’t mention it, most MESA code just calls it TR, but a few bits call it Token Ring.

I can’t imagine that is correct, there’s no reason for a 1984-era protocol to be present in a modern Ethernet PHY. Maybe somebody saw the acronym and filled it in wrong? IDK.

Anyway, the PHY init script has a big block of triplet writes to 0x12, 0x11, 0x10, always in that order. No comments or clues as to what it does, other than that 10baseT vs 10baseTe mode require a different sequence in one spot, and the final write is commented as “Improve 100BASE-TX link startup robustness to address interop issue”.

Guessing these are chicken bits of some sort too, but who knows.

EDIT: vtss_phy_optimize_receiver_init mentions some stuff about registers called half_comp_en and half_adc. There’s also some stuff in vtss_phy_power_opt that someone might want to dig into eventually. Seems to be some sort of link training to save power by reducing TX amplitude or something? vtss_phy_optimize_receiver_reconfig talks about vga_state as well.

Very rough conjecture and not tested in the slightest:

0x12: read pointer
0x11: read/write data register
0x10: write pointer

vtss_phy_debug_tr_regdump_print may also be worth looking at, it suggests that 0x10 might be a read pointer too?

MCB (Macro Configuration Bus)

The VSC742x datasheet goes into a little more detail about this - basically it’s a separate bus used for configuring the SERDES6G.

There’s no direct access to the MCB on the VSC8512 from the outside. Instead, you issue a command to the MCU to read all MCB registers from a SERDES macro into a working buffer (“PRAM”, “cfg_buf”, or “shadow registers”), use peek/poke commands to manipulate this buffer, then issue another command to write it back to one or more SERDES macros.

Understanding the MCU interface

So now we’re back to trying to understanding the MCU interface a bit more to try and figure out how to do fun things with it (and to solve my original problem of configuring the equalizer).

Starting with the SERDES register field names, most notably post0 as this is the one I care about the most (the first post cursor tap provides de-emphasis to open the eye up) I started searching through the MESA code for anything that looked relevant.

There’s a function vtss_phy_cfg_ob_post0 in vtss_phy_1g_api.h, with an accurate-ish yet useless comment “Debug function for setting the ob post0 patch”. You’d never know what this did from the prototype or comment if you didn’t dig as deep into the platform as I had, although looking at the source and seeing some of the implementation gated behind a #ifdef VTSS_FEATURE_SERDES_MACRO_SETTINGS might give a hint. Interestingly another part of the headers say “do not change” the post1 tap from the power-on default of zero, perhaps there’s a silicon bug around this?

This function does some minimal error checking, then calls vtss_phy_cfg_ob_post0_private which has a slightly less useless comment “Function to set the ob_post0 parameter after configuring the 6G macro for QSGMII for ATOM12”. After some more validation and checking that we are in fact on an Atom12 (there’s a separate code path for Tesla)…

It just does a big pile of reads and writes to register 0x12 in the GPIO/MCU page, aka register “18G”, “Global Command and SerDes Configuration”, or VTSS_PHY_MICRO_PAGE depending on where you look.

So what is it? All we have is a single 16-bit register (inside of the proprietary extended MDIO register window scheme, but we’ll forget about that for the time being). Can’t be that complicated.

Instruction formats

Bit 15 is actually documented: set high to execute a command, poll for it to self clear before doing anything else.

After some digging through vtss_phy.c and the VSC8512 datasheet, I made some progress. There’s actually two different instruction word formats for the remaining bits.

Bit 14 set: Indirect pointer format

This instruction sets a 15-bit pointer mem_addr into the MCU memory address space used by peek/poke commands.

[15]: always 1 (execute command)
[14]: always 1 (select indirect pointer mode)
[13]: always 0 (seems to be dontcare)
[12]: segment / base address select (1 = 0x4000, 0 = 0x0000)
[11:0]: low bits of pointer

Bit 14 clear: Command format

This instruction executes a command and optionally returns data.

[15]: always 1 (execute command)
[14]: always 0 (select command mode)
[13:4]: arguments, opcode dependent
[3:0]: opcode

Command codes

0x0: Set MAC mode / write MCB to shadow registers

Two commands are documented in datasheet table 77:

80b0: select 12 phy SGMII mode
80a0: select 12 phy QSGMII mode

One undocumented command is known from MESA:

9cc0: write shadow registers from cfg_buf to one or more MCB ports selected by the addr_vec bitmask. There’s probably some bitfields to crack here (maybe related to 0x03 command syntax)? to access other stuff.

It looks like Tesla uses 80e0 for QSGMII mode but I haven’t seen this used in any of the Atom12/Luton26 code.

0x1: Set SERDES media mode for dual mode SFP/baseT interfaces

Two commands are documented in datasheet table 77:

[15]: always 1 (execute command)
[14]: always 0 (select command mode)
[13:12]: always 0 (reserved/ignored?)
[11:8]: bitmask of port 11:8 indexes, 1 to set mode, 0 to preserve
[7:4]: 0x8 for 1000baseX, 0x9 for 100baseFX, other values reserved
[3:0]: opcode (always 4'h1)

0x2: Unknown, not observed

0x3: Read MCB to shadow registers

One undocumented command is known from MESA. This command reads the entire 36 byte MCB from a single SERDES macro into a variable called cfg_buf, located in the 8051 memory at address 0x47cf.

[15]: always 1 (execute command)
[14]: always 0 (select command mode)
[13:12]: always 0 (reserved/ignored?)
[11:8]: SERDES macro index on the specified MCB, see below
[7:4]: MCB bus index, see below
[3:0]: opcode (always 4'h3)

MCB bus indexes for the VSC8512:

0: SERDES1G macros
1: SERDES6G macros
2: LCPLL

SERDES1G macro indexes on MCB bus 0:

0: SERDES1G macro 0 (SGMII lane 1)
1: SERDES1G macro 1 (SGMII lane 2)
2: SERDES1G macro 2 (SGMII lane 4)
3: SERDES1G macro 3 (SGMII lane 5)
4: SERDES1G macro 4 (SGMII lane 7 / SFP port 11)
5: SERDES1G macro 5 (SGMII lane 8 / SFP port 10)
6: SERDES1G macro 6 (SGMII lane 10 / SFP port 9)
7: SERDES1G macro 7 (SGMII lane 11 / SFP port 8)

SERDES6G macro indexes on MCB bus 1:

0: SERDES6G macro 0 (SGMII lane 0 / not used for QSGMII)
1: SERDES6G macro 1 (SGMII lane 3 / QSGMII lane 0)
2: SERDES6G macro 2 (SGMII lane 6 / QSGMII lane 1)
3: SERDES6G macro 3 (SGMII lane 9 / QSGMII lane 2)

0x4: Suspend 8051 patch

This disables updated 8051 firmware (and I guess reverts to the ROM version) temporarily.

[15]: always 1 (execute command)
[14]: always 0 (select command mode)
[13:12]: always 2'b01
[11:4]: 8'h01 to suspend microcode, 8'h0 to resume
[3:0]: opcode (always 4'h4)

0x5: Unknown, not observed

0x6: Byte poke

This writes a single byte at mem_addr on the 8051.

[15]: always 1 (execute command)
[14]: always 0 (select command mode)
[13]: always 0 (reserved/ignored?)
[12]: if 1, mem_addr is incremented after the poke. if 0, mem_addr is unchanged
[11:4]: byte to write to mem_addr
[3:0]: opcode (always 4'h6)

0x7: Byte peek

This reads a single byte from mem_addr on the 8051.

Write format:

[15]: always 1 (execute command)
[14]: always 0 (select command mode)
[13]: always 0 (reserved/ignored?)
[12]: if 1, mem_addr is incremented after the peek. if 0, mem_addr is unchanged
[11:4]: always 0 (reserved/ignored?)
[3:0]: opcode (always 4'h7)

Result readback format:

[15]: busy bit, poll until zero
[14:12]: reserved, ignore
[11:4]: byte read from MCU memory
[3:0]: reserved, ignore

0x8 - 0xc: Unknown, not observed

0xd: Squelch workaround

Not sure what this does, it seems only implemented or necessary in Tesla / Viper parts but noted here to explain the hole in the opcode space

0xe: Unknown, not observed

0xf: Unimplemented / nop

This is used in some pieces of the code such as vtss_atom_patch_suspend to temporarily disable the 8051 EEE microcode patch during TDR testing because apparently issuing an unimplemented command makes the patch suspend. Why not use command 0x4? Who knows.

Per comment in the code “Note that this is necessary only because the patch for EEE consumes the micro continually to service all 12 PHYs in a timely manner and workaround one of the weaknesses in gigabit EEE in Luton26/Atom12.”

MCU address space

I have not yet attempted to read or mess with most of the 8051 address space.

From looking at MESA, the following addresses, names, and values are known (this is not all inclusive, just what I’ve looked at). These appear to correlate with the bitfield names in SERDES6G_ANA_CFG in vtss_phy.h. While the macro names refer to Tesla, looking at the code it appears Tesla uses the same or very similar SERDES as Luton26/Atom12 so it’s very likely that all of these field names map directly.

`47cb` / name unknown

Seems related to SERDES loopback, see vtss_phy_serdes_fmedia_loopback_private

`47ce` / `addr_vec`

Bitmask of SERDES6G macros to write shadow registers back to when sending 9cc0 command.

[7:4]: reserved, write as zero
[3:0]: write enable mask for SERDES6G lanes 3:0. 1=write, 0=ignore

`47cf` / `cfg_buf[0]` / `cfg_vec[7:0]`

Unknown

`47d0` / `cfg_buf[1]` / `cfg_vec[15:8]`

Unknown

`47d1` / `cfg_buf[2]` / `cfg_vec[23:16]`

Unknown

`47d2` / `cfg_buf[3]` / `cfg_vec[31:24]`

[7:5]: unknown
[4]: facility loopback in QSGMII mode if I’m reading code right? header calls it VTSS_TESLA_SERDES6G_ANA_CFG_PWD_TX_6G)
[3:0]: unknown

`47d3` / `cfg_buf[4]` / `cfg_vec[39:32]`

Unknown

`47d4` / `cfg_buf[5]` / `cfg_vec[47:40]`

Unknown

`47d5` / `cfg_buf[6]` / `cfg_vec[55:48]`

Unknown

`47d6` / `cfg_buf[7]` / `cfg_vec[63:56]`

Unknown

`47d7` / `cfg_buf[8]` / `cfg_vec[71:64]`

Unknown

`47d8` / `cfg_buf[9]` / `cfg_vec[79:72]`

[7:5]: low 3 bits of ob_post0, output buffer postcursor 0 tap
[4:0]: unknown

`47d9` / `cfg_buf[10]` / `cfg_vec[87:80]`

[7]: unknown
[6]: ib_rst
[5:3]: unknown
[2:0]: high 3 bits of ob_post0, output buffer postcursor 0 tap

`47da` / `cfg_buf[11]` / `cfg_vec[95:88]`

See vtss_phy_atom12_cfg_ib_cterm_ena_private

[7:4]: unknown
[3]: ib_cterm_ena
[2]: ib_eq_mode
[1:0]: unknown

Other cfg_buf

According to vtss_phy_atom12_patch_setttings_get_private (note the typo), the cfg_buf array is 36 bytes long, while it’s 38 on Tesla. So maybe VTSS_TESLA_SERDES6G_DIG_CFG_PRBS_SEL_6G and higher registers are unavailable on Atom12?

Registers and fields mentioned in passing in the code worth digging into:

ob_cfg0:

ena1v_mode
ob_pol
ob_post0
ob_post1
ob_sr_h
ob_resistor_ctr
ob_sr

ob_cfg1:

ob_ena_cas
ob_lev

des_cfg:

phy_ctrl
mbtr_ctrl
bw_hyst
bw_ana

ib_cfg0:

ib_rtrm_adj
ib_sig_det_clk_sel
ib_reg_pat_sel_offset
ib_cal_ena

ib_cfg1:

ib_tjtag
ib_tsdet
ib_scaly
ib_frc_offset

ib_cfg2:

ib_tinfv
ib_tcalv
ib_ureg

ib_cfg3: mentioned, didn’t look at in any detail

ib_cfg4: mentioned, didn’t look at in any detail

`47f3` / `stat_buf`

8 bytes long, no other details reversed.

Digging in vtss_phy_chk_serdes_init_mac_mode_private will probably shed light on variables sys_rst, ena_lane, pll_fsm_ena, hrate, qrate, if_mode, ib_fx100_ena, des_cpmd_sel, des_100fx_cpmd_ena but I haven’t spent any time on this.

vtss_serdes6g_tesla_rcpll_status_get_private suggests the following fields for the RC PLL (SERDES1G?):

bit 10: out_of_range
bit 11: cal_error
bit 12: cal_not_done

vtss_lcpll_tesla_status_get_private suggests the following fields for the LC PLL (SERDES6G?):

bit 45: lock_status
bit 36: cal_done
bit 35: cal_error
bit 32: fsm_lock
bit 31:29: fsm_stat
bit 18:14: gain_stat

QSGMII equalizer testing

So, with all that, where does that leave us?

I now know how to tune at least the critical post-cursor tap 0 to do basic de-emphasis:

Read MCB registers from SERDES lane 1 (first QSGMII lane) with 0x03 command into shadow regsters
Poke ob_post0 using the indirect access command to set the pointer followed by a pair of 0x07 / 0x06 peek / poke operations to read-modify-write this field without changing others
Poke addr_vec to 0x0e to select all three of the four total SERDES6G lanes that are used for QSGMII
Send 0x9cc0 command to push the new register values to all SERDES instances

I’m on the edge of figuring out a lot more by guessing what the bitfield names in the header mean (this post is in part my notes in case I want to get back to this and tune other configuration like output amplitude).

With the default post0 tap value (I believe 0x02 from MESA comments, but have not actually tried to read back) the signal is totally usable, but there is just a smidgeon of eye closure.

As a test I tried turning it up much higher, to 0x0f, to see if I was poking the registers right. This worked, but gave a grossly overequalized result.

I was happy with how things looked at 0x04. The eye is fully open, all bits are the same height, and the eye is open enough to drive a bus through. This is through a foot of Samtec ARC6, two feet of Koaxis KF086, two ARF6 connector transitions, a SMPM connector launch, and about 75 mm of total PCB trace.

The eye actually passes the QSGMII transmit eye mask, not just the RX mask shown here. I think it’ll do.

Conclusions

Who needs NDAed datasheets when you can just read public driver code and work out the bits they’re not telling you?

There’s room for a lot more fun, maybe one day if I’m bored I’ll try getting code execution or a ROM dump or something out of the 8051 on the PHY. But for now the signal integrity tweaks are done so mission accomplished.

Like this post? Drop me a comment on Mastodon

Switch project, part 2 - Line Card

Mon, 23 Jun 2025 18:00:00 -0700

This is part 2 of my ongoing series about LATENTRED, my project to create an open source 1U managed Ethernet switch from scratch.

Today, we’re going to be talking about the 24-port QSGMII line card. Two of these will provide all of the front panel connectivity for the edge-facing side of the switch (i.e. everything but uplink and management ports).

Overview

Here’s the line card before heatsink installation. This is “upside down” in the canonical CAD layout, but provides a better view of the layout without the RJ45s blocking the view.

The overall layout of the board is mostly symmetric, but with some variations because of the PHYs not being mirror images of each other.

At the north side we have the three 8-port RJ45s. These are Link-PP LPJT476156AENL and among the most expensive parts on the board at $33 each (plus tariff for any I need beyond the initial ten samples I got for prototyping), second only to the $40.11 PHYs.

Speaking of PHYs, the two large BGAs are Microchip (formerly Microsemi, formerly Vitesse - yay semiconductor industry M&A) VSC8512 12-port QSGMII PHYs. Each PHY is responsible for half of the front panel ports on the line card (one quarter of the whole switch).

These are older 65nm parts, but as of when I last checked they were the only 12-port QSGMII PHY available from distributors in low volume with a datasheet that isn’t NDA-encumbered, so I’m stuck using them. One of their quirks is that the low speed digital GPIOs (MDIO, reset, JTAG, etc) are LVCMOS25 rather than the LVCMOS33 or LVCMOS18 in more common use in modern parts. This doesn’t change a whole lot on the line card board, but does cause some annoyances on the switch engine board which we’ll discuss in a future post. Each PHY has a probably-unnecessary JTAG connector, using the 2x7 Xilinx pinout since I have a lot of those dongles, in case I want to do boundary scan test. So far I haven’t needed this.

On the far east/west sides of the board next to the JTAG connectors are a series of power rail test points (more on that in a bit) and filtering components for the various isolated analog domains on each PHY. This area also contains an AT30TS74 I2C temperature sensor to measure board temperature in the general area of the PHY.

Directly to the south of each PHY is a Samtec ARF6 series connector. This carries eight 100Ω differential pairs, although only six are used here (three TX and three RX lanes). Using this “flyover” architecture allows the PCB to be made of an inexpensive material (Shengyi S1000-2M, relatively standard high-Tg FR-4) without the 5 Gbps signals suffering significant losses from long distance routing on a high-Df material. These links are AC coupled on the line card for both TX and RX. Slightly to the west of each is a pair of SMPM connectors which can be configured to output a recovered clock if I ever need this for validation/test purposes.

In the far southwest corner are power and reset buttons for debug. These drive GPIOs on the supervisor MCU and aren’t intended to be used in the final system, which will allow the line card to be remotely managed over an I2C or SPI bus from the FPGA or main processor.

Just west of the centerline along the south edge of the board is a ten-pin Molex PicoBlade connector which carries all of the low-speed management traffic: I2C to the MCU and sensors, SPI if I want more bandwidth to the MCU, and the MDIO bus for configuring the PHYs.

Down the centerline, we have the power supply - four Murata MYMGK00504ERSR 4A DC-DC modules. These are all fed by a common 12V input on the 4-pin Molex Mini-Fit Jr connector at the south center edge of the board, and each drives one of the board’s four core power rails: 3.3V, 2.5V, 1.0V digital, and 1.0V analog. This area also has another AT30TS74 to monitor PSU temperature, and an INA230 shunt monitor on each rail to monitor current and voltage.

Finally, sandwiched between the PSU and the PHY to its west, is the supervisor. This is a STM32L431 that controls enables and sequencing for all of the power rails, reset and power-down control for the PHYs, and monitors for fault conditions such as a power rail dropping out of regulation or a temperature limit being hit.

Power

I did initial power supply validation on the line card sitting at idle with 23 of 24 ports linked up, 22 through a set of 11 loopback cables and the 23rd cabled to the sandbox VLAN on my lab network. There’s no traffic flowing other than normal ambient network broadcasts on port 0 because I haven’t finished the gateware enough to actually be passing packets, but this is the best I can do at the moment.

Voltage and ripple measurements were taken with a Teledyne LeCroy WaveRunner 8404M-MS oscilloscope and RP4030 active power rail probe. Current measurements used the shunt resistors and INA230s integrated into the on-board PSU.

3.3V

This rail feeds the supervisor, I2C sensors, and the high side of the port status indicator LEDs. It’s always on, since the line card doesn’t have 3V3_SB routed to it and I didn’t see the point in adding a fifth regulator since it has no sequencing requirements relative to the other rails.

Voltage is 11 mV, or 0.3%, below nominal. Considering I used 1% resistors on the feedback network, I’m happy with that.

Current consumption is around 174 mA at 3.289V, for a total of 572 mW.

I think a sizeable amount of this is in the port indicator LEDs (which are rated for 20 mA each although I’m driving them slightly less hard than that). I’m not sure why RJ45s always seem to be using the old yellow-green AlGaP LEDs rather than the modern emerald-green InGaN ones, I could probably cut almost a watt off my system power budget if I were able to switch, but it was hard enoguh to find high-density jacks with integrated magnetics and AC coupled center taps for voltage mode PHYs.

Ripple is very good, 15.8 mV P-P and 2.52 mV RMS. It’s dominated by 300 kHz switching ripple with high frequency noise at 125 MHz and harmonics thereof (obviously coming from the PHYs, maybe the LED drive circuitry or something). There’s also a 95 MHz spectral line I can’t readily explain. I thought it was the supervisor MCU core clock at first, but that’s running at 80 MHz so as soon as I dropped a cursor on the peak I discarded that hypothesis… but it’s at like -75 dBm so I don’t particularly care. It’s not remotely strong enough to be a problem.

2.5V

This is by far the biggest power hog on the board, driving both the LVCMOS GPIO pads on the PHYs (for MDIO and LED drive) directly, and (through a pi filter network) the analog PAM-5 drivers for the twisted pair.

Main rail

Voltage is 21 mV, or 0.84%, below nominal, Also well within acceptable limits.

Ripple is excellent, 6.9 mV p-p and 788 μV RMS. Interestingly it’s not just sinusoidal ripple, there’s a noticeable switching spike right around the zero crossing of the ripple waveform. It’s tiny, maybe a millivolt or two, but clearly visible. (Have I sung the praises of the RP4030 enough? I love it enough that when I saw a second one for a good price, I couldn’t resist jumping on it)

A2V5_VSC

This rail is the analog supply for the PAM-5 twisted pair drivers.

Voltage is sagging well below nominal at 2.452V (48 mV low) due to significant losses in the filter network, but with 70 mV of headroom above the 2.38V lower limit in the datasheet and almost no ripple, so I see no reason to change the design.

Ripple is very close to that on the main rail, 6.13 mV p-p and 680 μV RMS. Perhaps in future designs I can omit this filter entirely unless I have major noise sources on the 2V5 rail that might cause problems? Not worth a board spin when I already have enough bare boards to finish the project, though.

1.0V digital

The 1V0 rail is the second most heavily loaded, driving the digital core of both PHYs. It’s a near perfect 1.000V on average, drawing 2.03A - almost as much current as 2V5, but far less power due to the lower voltage.

Ripple at first glance was… not great, not terrible: only 1.77 mV RMS but 42 mV p-p with strong spikes at the 300 kHz switching frequency.

Zooming in, we can see the main high frequency noise on the rail is a somewhat triangular looking 125 MHz about 5 mV p-p,

After some investigation, though, the spikes turned out to be a measurement artifact - the test point was placed right next to the DC-DC converter and did not accurately account for the filtering effect of the 40+ high frequency decoupling capacitors under the PHY itself. All of the other rail test points were much closer to the PHYs which gave the capacitors time to do their job.

Measuring with a soldered probe directly under the PHY, we see a drop of about 7 mV in the PDN, but ripple is much lower: 1.06 mV RMS and 7.7 mV p-p with no strong switching spikes visible at all. So the rail is totally fine where it counts, we just need to be measuring at the right spot.

1.0V analog

The 1V0_2 rail is the third most heavily loaded, drawing about 820 mA, and is also almost exactly at the nominal voltage of 1.000V.

Main rail

This rail does not directly drive any loads (it feeds six separate analog power domains, three per PHY, through pi filters) so unsurprisingly it’s extremely clean: 5.46 mV p-p and 487 μV RMS. The strongest spectral line is 625 MHz, the 5th harmonic of the 125 MHz main clock frequency.

A1V0_VSC

This is the VDD_AL rail on the left PHY (right is nearly identical), which powers analog circuitry for the twisted pair interface. There’s no current shunt so I don’t know how much of the total 1V0_2 current is consumed by this power domain.

Voltage is slightly lower than nominal (due to losses in the pi filter) at around 996 mV, but well within acceptable limits.

Ripple is excellent, 6.27 mV p-p and 785 μV RMS. 125 MHz and 500 MHz are the dominant spectral lines and the 300 kHz SMPS frequency is almost completely invisible.

A1V0_SERDES1

This is the VDD_A rail on the left PHY, which powers one of two analog domains for the QSGMII SERDES.

As with the other analog rails the voltage is slightly lower than nominal due to filter losses, at 996 mV, but not to a concerning level.

Ripple is extremely low, at 3.85 mV p-p and 438 μV RMS - exactly what we want for something powering a sensitive SERDES. 125 and 625 MHz are the dominant lines, with a noticeable 1 GHz component as well.

A1V0_SERDES2

This is the VDD_VS rail on the left PHY, which powers the second analog QSGMII SERDES domain. Voltage is about 1 mV lower than nominal due to filter losses.

Ripple is the lowest of any power domain on the board, at 3.67 mV p-p and 417 μV RMS. I think this might actually be a personal record for the cleanest SMPS-derived power rail I’ve ever measured on one of my boards. Just to be clear, this is not a 20 MHz band-limited measurement. This is measuring from a few kHz out to 4 GHz using the full bandwidth of the scope and RP4030!

QSGMII

The line card communicates to the host FPGA via six QSGMII links, three per PHY. These are 5 Gbps NRZ and the highest speed signals on the board by far, so I kept the path lengths very short to avoid losses in the cheap Shengyi S1000-2 PCB material, allowing me to keep costs down vs a more expensive low-loss dielectric.

TX

I measured the TX waveform at the AC coupling caps (about 5mm of PCB trace away from the BGA balls and as close as I could practically get) with a 13 GHz LeCroy D1330 active differential probe.

In the test setup, I’m driving the QSGMII with a GTYE4 transceiver on the XCKU5P FPGA through 51mm of trace on Isola FR408HR, a SMPM connector, 12 inches of Koaxis KF086, another SMPM, 46 mm of trace on Shengyi S1000-2, an ARF6 connector, 12 inches of Samtec ARC6 cable, an ARF6 connector, and a few mm of trace on S1000-2.

The real-world deployment will be similar overall trace lengths, but the FPGA board will be Taiwan Union TU872SLK (similar Df to FR408HR) and the SMPM connectors will not be present - there will be a direct 1-2 foot ARC6 connection from the FPGA board to the line card. So in terms of overall insertion loss this is a pretty representative worst-case channel of what we’d see in the real switch.

With the somewhat arbitrarily chosen default TX equalizer coefficients, the eye is open but some closure from losses is clearly visible.

After a bit of tinkering with tap values I settled on TXDIFFCTRL=5’h13, TXPOSTCURSOR=5’h07, TXPRECURSOR=5’h04 which gave a wide-open eye with 179 ps (of the 200 ps UI) of horizontal opening and 389 mV of vertical. This easily passes the QSGMII RX eye mask. I’ll need to tune these values slightly in the final switch deployment for absolute best SI due to the different cable lengths, but this should get me pretty close and is honestly probably not worth changing at this point.

RX

I measured the RX waveforms at the ARC6-to-SMPM adapter board on my bench setup, directly connecting to the oscilloscope through coaxial cabling. The total channel length here is 24 inches of Koaxis KF086 and 12 inches of Samtec ARC6 plus roughly 50mm of PCB trace on S1000-2M (40mm on the adapter board, 10mm on the actual line card).

I deliberately didn’t de-embed the KF086 test leads (even though they are serialized and I have touchstone files for them) since some of the final switch paths may use more than 12 inches of ARC6, so I wanted a bit of extra loss to give a worst-case scenario.

The eye was wide open (481 mV height, 179 ps opening) and passed the QSGMII eye mask with room to spare, but definitely could do with a little bit of emphasis to be perfect (and the swing was actually on the high side, I could probably reduce it to save power). Unfortunately the VSC8512 datasheet does not document the SERDES output buffer configuration registers. I opened a support case with Microchip and if I hear back about this will attempt to tune further, but if not this is absolutely usable as-is.

It’s interesting to compare the rise time and overall eye shape of the RX path (Vitesse 65nm SERDES6G macro) to the TX path (Xilinx 16nm GTYE4 macro). The RX is a 6 Gbps SERDES working at close to its max rated data rate - although still giving a very clean eye - while the TX is a 28 Gbps SERDES that’s not even breaking a sweat and clearly has plenty of headroom.

The filter graph shows the processing being performed on the raw scope waveforms for this analysis. It’s so much easier to follow filter graphs now that we have pretty icons for at least most of the processing blocks. Thanks again to pirate, tetrikitty, and all of the other artists!

Ethernet MDI

As of now, I haven’t done proper SI measurements on the baseT links, but it links up and works with no bit errors during testing to date. At some point before I deploy the switch in prod I might generate and measure some compliance patterns but I have zero concerns based on other testing to date.

Thermals

On my bench now, using a low profile heatsink with 12mm fin height (Wakefield-Vette 960-27-12-D-AB-0) and a 40mm fan blowing generally down the line card’s axis but with no airflow ducting, temperatures are:

PCB near PHY 0: 27.8C
On die PHY 0: 39.5C
PCB near PHY 1: 36.3C
On die PHY 1: 60.9C
PCB in power supply area: 30.2C

These are acceptable but definitely on the high end, especially for PHY 1 (right side as seen from the front panel, furthest from the fan), especially considering it’s sitting on a bench and not marinating in an enclosed 1U chassis.

In thermal imagery (not calibrated for heatsink emissivity) the PHY heatsinks measure as 33.2 and 40.6C, with the PHY further from the fan - and the entire PCB area around it - visibly warmer. Also interesting is that the DC-DC module for the 2V5 rail (the highest output current) is slightly warmer than the others.

I plan to add some 3D printed air ducting to the actual switch chassis to ensure better control of flow paths and sufficient cooling for the PHYs, as well as swapping the 12mm tall heatsinks for the 23mm version which will dump heat far more effectively. Once the final hardware has been prototyped we’ll see how it performs.

Conclusions

The line card is working great and I don’t see any reason to make significant changes to it. This isn’t too surprising considering that much of the architecture was proven out on LATENTPINK a year or two ago, but I had to make sure everything was good before I spent a bunch of time and money populating more of them.

I did find one bug during the bringup process, however: the right LED on each port is swapped with the one above it (so for example if you plug a 10/100 device into the bottom left port, the top left port’s right LED will turn yellow). Rather than respinning the board, I decided to just fix this in software by using the GPIO bitbang mode on the PHY that lets me set the LED state over the MDIO bus. I don’t need fast response since the right LED just indicates <1 Gbps mode, while the left LED is the link up/traffic indicator and remains driven by the hardware since it’s hooked up correctly.

Like this post? Drop me a comment on Mastodon

Switch project, part 1

Thu, 08 May 2025 18:00:00 -0700

One of my longest-running projects has been an open hardware Ethernet switch. This has been one of the key driving forces behind many of my other projects, such as ngscopeclient and the high speed probes. It was also the project that got me into high speed digital design.

So I figured it’s time to kick off a series with a short writeup of where things are now, how we got there, and what’s coming next. If you follow me on Mastodon you’ve probably seen most of this in bits and pieces but I wanted to collect it all in one place.

Ancient history: The first switch

The first generation never had a name, it was just called “open-gig-switch” or something in my subversion repository (this was circa 2012 before I was primarily running on git).

I couldn’t find any many-port gigabit switch ASICs that were suitable for an OSH project (i.e. purchasable in qty 1, no NDA needed for datasheet, etc). So an FPGA-based from-scratch design seemed the only option.

This board pushed a lot of limits for me… my first use of switching power supplies rather than LDOs, my biggest and most complex board to date, and I think maybe even my first use of an RGMII PHY (I had previously used the Microchip ENC424J600 which is a full 10/100 Ethernet MAC + PHY including buffer memory attached to a parallel bus or SPI interface).

And at the time my fanciest piece of test equipment was a 100 MHz Rigol DS1102D. So I had no way to do signal integrity measurements on either the Ethernet differential pairs or even the 250 MT/s RGMII lines.

This one never really got anywhere. I got three of the four PHYs up and running, PHY #4 never worked (I can’t remember if it wouldn’t link up, wouldn’t pass data, or what). I tried resoldering a bunch of stuff but never got it functional.

I tried to bring up a basic switch with only 3 ports, but very quickly realized that 15K LUTs, <1 Mbit of BRAM, and no external CPU would make that very challenging. The XC6SLX25 was not a large FPGA and adding the Xilinx DDR controller and a softcore CPU and three MACs didn’t leave much space at all for fabric. Plus I was still a relative novice to FPGA development and wasn’t quite at the point of being ready to tackle the project.

So this board ended up a dead-end, but set the stage for what was coming.

The interim years

The first switch fiasco showed me I needed more skills, better test equipment, better debug tools, and more.

I was still in grad school with little budget for equipment, but over the course of my Ph.D I became a much more experienced RTL engineer. I built a couple of boards with Ethernet that mostly worked.

But since I couldn’t afford a better scope and had no way to validate SI on faster stuff, I decided to table the switch project for a bit.

In 2015 I graduated and got a job that paid much more than I was making as a graduate teaching assistant. One of my first purchases was my first “real” oscilloscope, a 350 MHz Teledyne LeCroy WaveSurfer 3034. This was actually fast enough to do SI work on RGMII and protocol-decode 100baseTX, but I dreamed of much faster.

I knew I wanted to put 10GbE on the switch whenever I finally got back to it, and didn’t expect to be able to afford a many-GHz scope any time soon, so I started looking at alternative approaches to validate SI at these speeds. This led me down another path of yak shaving in which I started designing a 10 GHz sampling oscilloscope named FREESAMPLE (which never got finished, but I do want to revisit the project one day). This, then, got put on ice when I realized a high speed scope would be useless without an equally high bandwidth probe to feed it with.

You probably already know where this story goes. My open hardware 16 GHz probe (which probably deserves a post or series of its own) is now essentially done and ramping up an initial PVT run, hopefully I can actually make them in quantity and offer to the public in the coming months.

Over the course of shaving these yaks I also bought a house, took a year or two off major projects to refurbish it, got a Sonnet EM solver seat, found a shockingly affordably 16 GHz oscilloscope on eBay (which made FREESAMPLE much less of a priority), rewrote “scopeclient” with OpenGL acceleration as “glscopeclient”, then rewrote it again in Vulkan as “ngscopeclient”. I also created protocol decodes for 10/100 baseTX, 1000baseX, SGMII, QSGMII, 10Gbase-R, and a few other potentially relevant protocols.

I also started dreaming up a roadmap for a whole family of networking equipment under the randomly generated umbrella name LATENTx, using colors for sub-projects (vaguely inspired by the Lockheed HAVE BLUE). The original roadmap called for LATENTRED to be a gigabit edge switch and LATENTORANGE to be a 10G core switch, but I decided that a prototyping/technology demonstrator platform was called for first. LATENTINFRARED was too long so I went with LATENTPINK as the name.

LATENTPINK

The original LATENTRED concept had called for 24 edge ports across three 8-port line cards (because multiples of 3 were convenient for using OSHPark for fabrication). RGMII would have been a nightmare to route, since 24 lanes of RGMII need 288 pins (clock, control, 4-bit data bus, times 2 for TX/RX lanes) plus the MDIO, reset, etc. So I started looking at lower pin count options (GMII, needing 576 pins, wasn’t even on the table, as no FPGA supported by the free Vivado edition had over 520 GPIOs and anything bigger would be vastly outside my price range anyway).

The obvious “easy” option was SGMII which needs one differential pair each for TX and RX (4 pins). This would only require 96 pins for 24 PHYs, and no parallel bus timing constraints to worry about (just matching P/N of each diff pair). So I started looking at the TI DP83867.

But the project had dragged on long enough that by the time I was ready to make hardware for LATENTPINK it was early 2023 and a new option was on the table. Vitesse, who had previously been on my “naughty list” of companies who will never get a design win from me due to developer-hostile practices like locking datasheets for their most boring products behind NDAs, had been bought by Microsemi in 2015, who had then been bought by Microchip in 2018. As part of this a lot of parts got opened up and I decided they deserved early release for good behavior… which meant that their 12-port QSGMII PHY, the VSC8512, was now on the table. This would allow 24 ports with only six transceiver links (12 diff pairs / 24 pins), a massive reduction from SGMII.

The only problem was, I had never worked with QSGMII before and the VSC8512 needed a fair bit of register configuration to work properly, so I wanted to hedge my bets a bit. This resulted in LATENTPINK, which had one VSC8512 and two DP83867s for a total of 14x 1G edge ports, plus a single 10G SFP+ uplink and a dedicated RGMII management port (using my tried-and-true preferred PHY, the KSZ9031RNX) for the SSH interface.

This was my second 8-layer board (first for a fully personal rather than work-related project), one of my first uses of the Murata MYMGK modules I now use everywhere, and one of my first designs using FPGA transceivers. It was also my first attempt at pairing a STM32H7 with an FPGA (over quad SPI because I hadn’t yet learned how much of a disaster the H7 OCTOSPI peripheral was, but it was fine because I was just doing manual register reads/writes and not trying to memory map it).

I got the VSC8512 working fine, the MCU talking to the FPGA (slowly) and used the platform to build out my SSH server and a bunch of other building blocks the full switch would need.

The LATENTPINK board contained an external QDR-II+ SRAM which was used as a packet buffer for a shared-memory based switching fabric. All incoming packets were written to small per-port CDC FIFOs then popped round robin from these FIFOs and written into a region of the QDR serving as a large data FIFO for the port.

On the far side, the forwarding engine would pick a source port round robin then read the QDR FIFO to pop one packet from the port. The output data stream from the QDR would then be written to small per-port exit queues, routed to all of them and with write enables gated according to which port the packet was to be forwarded to.

This design used a home-grown register structure for the control plane and another bus for the data plane, both of which had issues. The control plane bus didn’t support any kind of distributed decoding so I had to put all the register logic in one place which required routing decoded SFRs all over the place, and the data plane bus was 32 bits wide for RX and 10G TX, while it was 8-bit for 1G TX. This added some annoying complications to the design and the nonstandard nature meant I couldn’t easily reuse FIFOs and other blocks written for other projects.

There were also a couple of PCB bugs, most notably a bad pinout resulting in the upper row of ports only linking up in 10/100 mode until I reworked them (a massive pain given that the swap had to be performed on an inner signal layer, either 3 or 6 depending on the port, in a fairly confined space). I validated the fix on one or two ports but didn’t rework the entire board.

Overall I got it to the point that it was functional, it could pass packets, it had port based VLANs and could decode inbound 802.1q tags (but not synthesize outbound tags on trunk ports) before deciding I had proved out the tech stack enough that I was ready to build the real thing.

What next?

The other major finding from LATENTPINK was that, at least the way I had architected the fabric, the XC7K160T was a bit cramped for my plans. While the 14+1 port design was comfortable, I didn’t think it would be easy to fit a 24+2 port switch into it. The next largest 7-series part (the XC7K325T) was not available in the FBG484 package so I’d have to go up to FFG676, but more annoyingly it wasn’t supported by the free Vivado edition (requiring a $3K software license) and was hugely more expensive ($2260 vs $435 as of this writing).

Between these two cost adders, I’d be looking at a $5K increase in project cost to jump to the bigger FPGA and build one prototype. My long term goal was 96 ports, so to replace all of my legacy Cisco switches I’d be looking at $3K of software + 4x $2K = $11K more to build the entire batch of switches with the 325T vs the 160T. And this would be on top of the already high costs of doing 8-10 layer PCB fab in low volume, the PHYs and RJ45s, custom sheet metal work for the chassis, etc.

By the time I was ready to start thinking about building LATENTRED seriously, though, it was 2024. 7 series was getting pretty long in the tooth and UltraScale and UltraScale+ had been out for a while.

I considered building a switch around the Artix UltraScale+, specifically the largest one - the XCAU25P. It sells for less than the XC7K160T, $380 for the -1 speed grade in FFVB676 as of this writing. It’s two process nodes newer (16 nm vs 28) so the fabric is much faster. And at 141K LUTs it’s a fair bit larger than the 101K of the 7K160T, although well below the 203K of the 7K325T. Other specs were also mostly in between: 12 transceivers vs 8 or 16, etc. But it was light on block RAM, 10.5 Mb vs 11.7 or 16. And I expected to need a lot of FIFOs in the design, so it would go quickly.

While I did buy a pair of AU25Ps, before I could design a board I was tipped off by a friend to a batch of Kintex UltraScale+’s, specifically the XCKU5P, on AliExpress for a mere $55 each. He had tested one from the seller and they appeared to be legitimate, although likely salvaged/reballed from some scrapped equipment.

This was a major game-changer for the project’s direction. The XCKU5P is the largest FPGA supported by the free Vivado license, at 216K LUTs (just larger than the 7K325T), 16 transceivers (matching the 325T), 16.9 Mb of block RAM (slightly larger than the 325T), plus another 18 Mb of UltraRAM, a new kind of large SRAM block optimized for large buffers. The transceivers were also 28 Gbps capable, enabling 25/100G Ethernet rather than only 10/40G. They retail for $2972 in the commercial temperature grade or $3350 in the industrial (which the ones I got were), so I was quite happy with scoring them for less than 2% of list price!

The only problem is, with these new capabilities came scope creep. The 16 transceivers on the KU5P were enough to support twelve lanes of QSGMII (48 baseT ports) plus either a single 40/100G uplink or up to four 10/25G uplinks. And it seemed a shame to not use the full capabilities of such an expensive FPGA falling into my lap for cheap.

LATENTRED

Planned hardware architecture

The new concept for LATENTRED is a much more powerful switch than originally planned. It will be a 1U switch with two 24-port line cards (two VSC8512s per line card) and dual 10/25G SFP28 uplinks.

I had initially thought about doing four 12-port cards but found that 6 and 12 port magjacks with AC-isolated center taps (required by the VSC8512’s voltage mode MDI drivers) were hard to find, while 8-port ones were available from LINK-PP. The smallest port count evenly divisible by both 8 and 12 is 24, and a 2x12 port line card would just barely fit in my reflow oven.

The overall switch will consist of five, possibly six, PCBs:

The 48 -> 12V intermediate bus converter
Power distribution / switching board
Two 24-port dual VSC8512 line cards
Switch engine board with XCKU5P, STM32H735, management PHY, serial port, etc.
Possibly a separate board with the SFP28 uplinks connected to the switch engine by cables, depending on whether the chassis mechanical layout makes this easier than a monolithic design

The original concept also had called for the line cards to be connected by rigid interconnects in a daisy-chain style (i.e. all SGMII/QSGMII lanes into one side, each line card would tap several lanes off to local PHYs and route the others out the other side). This would result in long multi-gigabit signals running across the full 19” chassis width through several connectors, likely requiring the use of a higher cost low-loss substrate.

While browsing the Samtec catalog, though, I discovered the AcceleRate series, more specifically the AcceleRate Slim ARC6 (cable) / ARF6 (board) series. These are twinax differential interconnects with 8 to 24 differential pairs per cable and are rated for 32 Gbps NRZ or 64 Gbps PAM4, so the insertion loss performance is… only slightly overkill for 5 Gbps QSGMII links. One eight pair cable can handle the six pairs required for the three QSGMII lanes on a VSC8512 with two extra lanes to spare.

The big advantage of a “flyover” style interconnect like this is that there’s no need for a low loss PCB material or routing lots of diffpairs long distances that will conflict with other routing. Jut put a couple of connectors close to the BGA and leave the rest of the board area free for other stuff.

ARC6 is also good enough I could use them as a flyover for a remote SFP28 connector if that makes the chassis layout easier.

Current hardware state

The IBC has been already designed and used in other projects, so that’s done. We can forget about it, other than possibly doing that small respin with reduced ripple from the 3.3V buck. But with the current tariff situation I’m not in a hurry (it’s a 4L 2oz board made at Multech in China). I have plenty of the boards and several populated units ready to go.

The PDU board is done. There’s not much to it: 12V in the left 8-pin connector, 12V out the right 8-pin, 12V out the bottom two 4-pins. It also passes the I2C and 3.3V standby rails from the IBC through, while tapping off them to run programmable soft load switching and voltage/current monitoring. Pretty straightforward. I have three boards (one populated) so to get my 96 ports I only need to stuff a second one with some parts I have on the shelf.

The line card is also done. I’ve built one and it seems to work fine aside from, if memory serves me right, port 22 or 23 not linking up. Probably a solder defect since the left/right PHY layout was basically copy pasted, but I haven’t had time to troubleshoot. I have five PCBs and need four to make two switches; I will have to order two more of the 8-port RJ45s from LINK-PP to stuff all of them but I’m a ways off from that being an issue.

It’s a six-layer design on cheap Shengyi S1000-2M substrate since the QSGMII links are short and everything else is slow. Stackup is SGPPGS with all of the Ethernet signals on the back layer and some of the LED GPIOs routed in between layer 4 power pours. Connection to the rest of the system is via four connectors on the back side: 12V power on a 4-pin Mini-Fit Jr, six QSGMII lanes split across two ARF6 connectors, and a 10-pin Molex PicoBlade providing access to the PHY MDIO bus, I2C to a bunch of system health sensors, and a SPI bus to the onboard STM32L431 management microcontroller (not used in current firmware) in case I want to provide remote power cycling or something like that.

The big missing piece as of now is the switch engine board itself. For the near term I’ve cobbled together a sort of sub-scale test setup containing one line card, the power system, a KU5P dev board I built last year to validate that the $55 AliExpress FPGAs weren’t bricks, and a separate board with a STM32H735 as the management processor.

It wasn’t possible to route a STM32H735 with FMC to the FPGA directly to enable the fast memory mapped interface I described in a previous post since the dev board was built on OSHPark 4 layer (I may be the first person to have put a Kintex UltraScale+, with all transceivers broken out, on a 4-layer PCB… post about that coming at some point) so instead I built a small expansion board with the MCU, a small bridge FPGA (XC7A35T), and connected it to the KU+ board via a 5 Gbps serial link. The MCU memory bus is bridged to APB on the Artix which then serializes the APB transactions via a simple PCIe-esque protocol (the Serial Chip to Chip Bus, which will probably get its own blog once I’ve fine tuned the implementation a bit) and back to APB on the Kintex. The end result is basically the same as if the MCU were directly wired to the Kintex but with a bit more latency in the path.

The glue board also contains a passive adapter circuit to bridge from three sets of SMPM connectors on the Kintex board to an ARF6 connector so I can break out that GTY quad to talk to the line card. I have a second glue board that will bridge a QSFP28 out to an ARF6 (enabling me to use the second PHY on the line card) but haven’t had time to assemble it yet.

Planned switch engine architecture

I’m still working out some of the fine points of the implementation, but the current overall plan is a 4x4 64-bit crossbar at 400 MHz. This will give 25.6 Gbps per lane of throughput, or 102.4 Gbps total. I could theoretically downclock to 390.625 MHz if needed due to difficult timing, but the increase in margin will be small and 400 is a more convenient number for me to synthesize from 25 MHz without needing fractional-N.

The 25G uplinks will each get a dedicated crossbar port, while each line card will also get one (combined bandwidth 24 Gbps).

The 4x4 crossbar should be very efficient and performant timing-wise, since a 4:1 mux fits in a single LUT6. So a 64-bit 4:1 mux is 64 LUTs and the full crossbar 256 LUTs, with only a single level of logic in the critical path. The decision-making logic will be more complex but can run in a slower clock domain if needed, since forwarding a packet will take multiple cycles.

On the input side, there will need to be some kind of arbiter and some small FIFOs to take the 24x 1 Gbps streams and mux them down to a single 24 Gbps stream, as well as some clock domain crossing blocks that can probably be handled by the same FIFOs. UltraRAM is available if needed for deeper FIFOs.

On the output side, there will just be a small block RAM exit queue per port. The 25G ports will drive the crossbar exit stream straight into the FIFO, while the 1G line cards will have 24 separate exit queues (one per port) with enable gated by the destination port set (a bitmask, to allow for broadcast/multicast).

Current gateware state

I successfully ported my existing 10G MAC/PCS IP over to AXI4-Stream and have it working well in tests. The 1G is almost done, the AXI conversion is finished for the RX side but I haven’t done the TX yet.

I still have to write a 25G MAC/PCS but that’s a ways out, I can run the uplinks at 10G and test the whole rest of the switch just fine that way (I won’t actually be lighting up the uplinks at 25G until I get a 25/100G core switch anyway, which is a ways out).

The MAC address table from LATENTPINK will work just fine in this design with no modifications, it has more than enough capacity for the max theoretical number of packets I could push through the fabric.

I still have to build AXI4-Stream VLAN tag insertion/removal blocks, figure out at what point I want to drop frames with bad FCSes (probably at the point they’re written to the URAM ingress FIFO but I’m not certain yet), and actually do all the integrations.

Conclusions

There’s still a lot to do but it’s been an exciting project so far and I look forward to seeing where it goes. I don’t expect to have a polished switch on the final PCB/mechanical design until probably some time in 2026 (subject to manufacturing/supply chain delays and the political situation), but I hope to be pushing packets by the summer some time.

Like this post? Drop me a comment on Mastodon

Holding the Bag

Thu, 17 Apr 2025 12:00:00 -0700

This is a bit of a change from my usual content, but I’ve been sitting on this idea for a short SF story for a long time and I had to try actually writing it. I’m mostly a technical writer not a creative one so hopefully this turned out decently :)

Arrival

BEEP BEEP BEEP BEEP

It took Dr. Lynn Clarke a minute to wake up enough to register that the noise was her alarm clock. She was not a morning person, preferring late nights in the lab to coffee-fueled 8AM PHYS 1100 lectures.

“This is the new normal, better get used to it,” she thought as she ate breakfast. Her dreams of spending her career publishing IEEE papers on improving fabrication yields of THz photonic crystals were over, as with untold thousands of her peers. The collapse of government funding in the mid 2020s had been catastrophic for pure physics researchers, and really anyone working on science that wasn’t something an oil company thought looked profitable.

As if that wasn’t enough, the history of Y2K had been forgotten or ignored by Wall Street management and when 2038 came (who would have thought banks and stock exchanges would still be running their databases on 32-bit operating systems for that long?) the resulting financial apocalypse devastated what little budget had been coming from industry. The database recovery folks were raking in cash from companies desperate to salvage their records, of course, but physics faculty? Good luck.

MIT had been hit harder than some other institutions, with the endowment losing half its value in a week and most of the department being laid off at the end of the fiscal year as corporate research sponsors went bankrupt, stopped funding non-essential work, or automatic payments simply ended because the accounts payable system thought it was 1901 and nobody could be reached to fix the problem because the email server was just as corrupted.

The job at Rensselaer Polytechnic was a pay cut and meant starting over as an associate professor teaching freshmen about Newton’s laws rather than working in the cleanroom on cutting-edge research, but it would put food on the table and a roof over her head. And it beat stocking shelves overnight at the Cambridge Wal-Mart like her former department head.

After breakfast she grabbed her helmet and tire pump and went down to the garage to get her bike ready for the morning commute, checking the map for what felt like the 20th time to make sure she had the route memorized. Being delayed by a flat tire or making a wrong turn on the way to the first day of classes would be an embarrassing start to the new job. And she wanted to allow plenty of time for the trip until she got back in shape. Years of desk work and car commuting had let her once-athletic legs weaken, but she had sold the car over the summer to pay for groceries and there was no bus service to her new neighborhood (a good thing - apartments with transit access were more expensive) so it was the only option left.

As she rode down the hill towards campus, she tried to enjoy the view of the distant Hudson river as cars zipped past her every few seconds. Biking to work was going to take some getting used to.

Lost in thought, she didn’t even notice her view of the river vanish in a wall of blackness until the squealing of brakes and crunching of metal in front of her shattered her daydream. A fraction of a second later, the cacophony was drowned out by pain and confusion as she flew over the handlebars into something hard.

The Cube

“Dr. Clarke?”

Lynn woke up in a daze and looked around the room. Her left leg was in a cast and every breath hurt her chest despite the IV delivering what she assumed was some sort of pain medication. “W… What happened? I feel like I got hit by a bus!”

“Close!” the nurse responded, “You hit the bus. The Cube popped up right in front of the school bus in the next lane and you smashed into the back door. You’ve got a hairline fracture of your left fibula, three broken ribs, and some road rash but the doctor says you should make a full recovery in a few months.”

“Months? I have a class to teach… and wait a minute, what cube?” she asked.

“Oh, you must not remember what happened. That’s a normal mental reaction to traumatic events. Don’t worry, it’s not brain damage or anything - your MRI came back fine and there’s no sign of a concussion.”

The nurse reached over for the bedside TV remote and turned it on. A local news anchor stood in front of a video wall showing a helicopter view of Hoosick Street. But where there should have been a traffic light and four-way intersection a few blocks from the school, there was nothing but blackness. It appeared perfectly square and featureless, not even reflecting the lights of the tow trucks and police cars blocking the road as they removed the debris of what was clearly a massive multi-car pileup.

“Thankfully there were no fatalities, but over a dozen drivers and a cyclist were hospitalized after the Cube appeared in the middle of the road during the morning rush hour. The area has been blocked off by the FBI who refused to comment when our reporter asked them if they had any idea what the object was or where it had come from.”

The helicopter camera panned over to the Price Chopper parking lot across the street, which was lined with people pressing up against hastily erected barriers trying to get a glimpse of the object, then cut to a reporter on the ground in the parking lot.

“Hi there it’s Bob from Channel 10 News. We’re here with some of the spectators looking at the Cube.”

Bob shoved the microphone in the face of a bearded man in his 60s.

“What do you think about the Cube?”

“The army should have already blown it up. The eggs will be hatching any minute and then it’s game over!”

Bob turned to a pair of college girls behind him. “What about you?”

“This is the most exciting moment in our history. We’ve finally made contact with another species! We need to put aside our differences and show them our best side.”

Bob continued interviewing the locals. Theories ranged from a secret government project gone wrong to an unexploded alien antimatter bomb to a real-life version of the Monolith from “2001”. Member of a local religious cult had already erected a cross at one side of the parking lot and were singing and praying in hopes of being spared by whatever deity had delivered it.

As the TV droned on and on, failing to reveal any useful information other than “it’s a big black box”, she fell back asleep.

Going Home

Lynn rolled down the hallway of the hospital with her broken leg on a scooter. It had been a difficult couple of days, but she had been cleared for discharge.

“Professor?”

She looked up and saw a skinny man wearing a T-shirt featuring an anthropomorphic fox, holding a set of car keys.

“Abe Jackson. I’m one of Dr. Chan’s Ph.D students. He said you called asking for a ride? I live a few blocks from you so he asked me to help you out. I can give you a lift to campus whenever you need.”

“That would be great! I was planning on biking to school but that’s not going to be happening for a while,” she responded with a small laugh.

Abe walked her to an old blue Toyota Tercel that looked like it would fall apart if it made a sudden stop. The foam in the seats had disintegrated decades ago and had been replaced by cloth seat covers over some kind of improvised padding. The hatchback brake light dangled precariously from two strips of duct tape, and several spots on the fenders were rusted through. “Meet Old Betsy. She doesn’t look like much, but she runs. With the stipends they pay us these days…”

He started the car and threw it into gear. There was a disconcerting rattling noise from the brake light bouncing against the rear window but it otherwise didn’t sound like too much of a death trap.

“So how much have you been following the Cube situation?” he asked.

“Not much. I was too loopy from the meds to be paying much attention. What do we know about it?”

“You mean them?”

“Wait, there’s more than one now???” she asked.

“Yeah, another one popped up a day later right in the middle of a house in Costa Rica. Looked like it got hit by a tornado. Luckily nobody was home. And some guy on the internet browsing aerial photos found another in the Mojave. Who knows how long it’s been there. So that’s three that we know about so far. There might well be more in uninhabited areas that nobody’s found yet.”

“Wow. OK, what do we know about them?”

“Not a whole lot. They’re very heavy, nobody’s managed to move one of them yet. They completely absorb every wavelength of light or RF we’ve thrown at them so far and radiate weakly around 30ish GHz. The spectrum looks just like you’d expect from an ideal black-body… with a temperature of around 2.7K rather than ambient. No idea if they’re solid or hollow or what they’re made of. The governor set up a task force to focus on this one and several of us have been asked to participate. They want you involved too, once you’re feeling up to it.”

“Wait, 2.7K? 30 GHz? That sounds like the cosmic microwave background.”

“Definitely similar,” Abe replied. “We have no idea if that’s a coincidence or if there’s some kind of connection somehow.”

He pulled Betsy to the side of the road. “This is your place, right?”

“Yep, that’s it. Thanks a lot!”

“Any time. Dr. Chan said you had an 8AM lecture tomorrow so I’ll see you 7:30ish?”

“Sounds good. See you tomorrow.”

The Lab

HONK HONK

Lynn opened her front door and saw Abe waving at her from Old Betsy.

“Morning! Need a hand?”

“No, I gotta get used to moving on my own if I want to be getting anything done for the… what’s it called again?”

“Well, it was the NY Cube Task Force. But overnight one of them dropped in a village in Siberia, another in some Greek farm, and then one flattened a factory in Shenzhen squishing a bunch of workers. Now it’s the UN Cube Task Force. No changes to field operations, just means we’re sharing data more widely. Class is canceled, we’re going straight to the lab. You’re taking over as Director of THz Studies, leading all of the characterization work between 100 GHz and 20 micron far infrared.”

“That’s quite the promotion from associate professor,” she replied. “How did I get picked and not somebody else?”

“You’re new here, but you’ve done more work with high-sensitivity THz imaging sensors than anybody else. This thing is such a good absorber that we might have to fabricate custom detectors to see anything.”

A block later they pulled into the Price Chopper parking lot. Traffic had been rerouted around the back of the store to bypass the Cube, which was now hidden from view under a large tent.

Approaching the tent, they were greeted by an elderly Asian man. “Paul Chan. I’m Abe’s advisor. You must be Lynn.”

“Nice to meet you. You’ve got quite the setup here, mind giving us the overview?”

“The Cube is such a good absorber that we’re doing everything we can to improve SNR,” he replied. “There’s a full Faraday cage around it with >100 dB of attenuation from a few hundred kHz to ultraviolet - building that in 48 hours wasn’t cheap but we pulled it off. We have cryo-cooled panels we can move around it to reduce black-body radiation from the cage itself, although they do reflect in other frequency bands so we only use them when necessary. The foam is pretty decent at acoustic shielding too but we have sound absorbing panels we can use to supplement. All of the analysis and data processing is done from outside the cage so we don’t disturb the measurements.”

“So what do we know so far? Anything on surface characterization yet?”

“The surface is perfectly flat, within the limits of our measurement capability. There’s no tunneling current whatsoever in a STM, and the AFM showed no deviation at all, not even on the atomic scale. HOPG looks like sandpaper compared to this. We’re not even sure it’s a physical surface, it might just be some kind of energy field. Somebody volunteered to touch it with a bare hand and they said it felt like nothing, their hand just stopped but it didn’t feel hot or cold or rough or… like anything besides just sitting there in open air.”

“Ultrasound?”

“Zilch. No reflection or transmission at all, it’s as if we’re broadcasting into a vacuum.”

“Gravitation?”

“Hard to measure on earth, obviously. But between the fact that we haven’t been able to move it, and it’s not sucking things up like a black hole or sinking into the ground, we’re guesstimating a mass of somewhere in the 20K to 500K ton range. Our geologists say that the bedrock is pretty deep under this intersection and none of the buried sewer lines seem to have been crushed, so probably towards the lower end of that.”

“Particle radiation?”

“Nothing. Alpha, beta, and neutron detectors show nothing but noise, not even normal Earth-surface background levels. We’ve tried irradiating it in a few spots and got nothing detectable transmitted or reflected.”

“And what about the EM side?”

“So far, not a whole lot but we’ve got further there than anywhere else. It emits what appears to be uniform black-body radiation, so that’s something. We’re trying to get the most sensitive detectors we can across the entire EM band in hopes of getting some level of modulation back that we can detect. Some of them have long lead times that are hard to accelerate, so we’re trying to use cooled detectors and strong transmitters to improve SNR as much as practical in the meantime. We were hoping you might be able to continue the THz focal plane array work from your IEEE paper last year, it looks like it will outperform anything we’ve got north of 100 GHz and CNSE thinks they’ll be able to fab prototypes pretty easily… And this is your new desk. Have at it, let me know if you need anything equipment or staff wise and we’ll make it happen.”

Lynn sat down with her laptop and pulled up her notes from the nearly-forgotten project to refresh her memory.

Progress

“We got something on the last sweep!”

Lynn looked up as Abe ran excitedly up to her desk. It had been four weeks and two rushed wafer lots, but the new THz detector prototypes were in operation, part of an ongoing global campaign to try and see inside the Cube or determine what it was made of and where it came from by any means possible.

“What?”

“Yeah. Extremely narrow transmission peak right around 120 micron wavelength - 2.5 THz. They’re setting the detector to do a 360 degree scan and see if there’s any spatial pattern but I thought you’d want to know.”

She got up and walked into the RF chamber surrounding the Cube. Between the featureless black surface and the thousands of sharp pyramidal RF absorber cones lay a narrow circular track with two wheeled carts, 180 degrees apart. One held a high-power transmitter with heavy power cables dangling from the back. The other held her experimental detector, wired to a complicated ensemble of electronics. The entire setup was clearly thrown together in a hurry - boards screwed to frames made from 2x4 lumber, hand soldered jumper wires to fix missing connections, and cables secured to the frame with duct tape.

“Lynn! You’re just in time!” exclaimed Dr. Chan. “We’re all set up, let’s clear the chamber and see what we get.”

Everyone walked out of the chamber and over to the control desk on the far side of the wall. Abe closed the door and sat down at the bench as the rest of the team crowded around.

“I’ve been playing around with some open source CT scanning software and I think I’ve got a 2D slice processing flow working. Let’s see if we have enough power to see anything…”

He clicked the “start” button. A slight humming noise came from the power transformer outside and the lights dimmed briefly, then a graph began to slowly trace along the screen. As the scan finished, there was a short pause, then a grainy black and white image appeared below the graph.

“Well, it’s not solid,” Lynn said. “Definitely looks artificially constructed, too.”

A regular grid pattern of small squares was visible in the image. In between the grid points, smaller rectangular and circular objects, as well as some more irregular blobs, could be seen.

“But what’s special about this frequency? And what is this structure?” Abe wondered out loud.

“Abe, work with the lab techs to get us an elevation axis so we can do full 3D reconstructions. That will tell us a lot. Dr. Chan, take the S21 sweep over to the theory folks and let them stew on it for a while. This is the only spot we’ve seen any transmission at all, even if it’s attenuated by 90 dB. I want to know what’s special about it.”

Lynn sat down at her desk and stared at the grid image. It reminded her of something familiar but she couldn’t place it.

Answers

“We think we know what the walls are!”

Dr. Chan and several other grad students approached Lynn’s desk with a whiteboard in tow.

“So, the blackbody spectrum was the big clue. It is cosmic microwave background radiation.”

“But how could that be?” she asked. “It’s sitting right here, not in deep space.”

“Our side of the discontinuity is, yes. But after that…”

“Discontinuity? What are you talking about?”

“Our working theory is that the ‘wall’ isn’t a wall at all. It’s a jump discontinuity in space-time. Matter can’t pass through it because there’s an undefined slope rather than a smooth curve like you get around a normal point mass. You’d need an infinite force to push over the edge. EM fields get diffracted out into deep space, so anything you send in vanishes and all you see coming out is the CMB.”

“But then why are we seeing transmitted signal?”

“There’s a second discontinuity about 60 microns away from the first one, acting like a liner. The Cube is hollow. If your incident signal has a wavelength exactly matching the spacing of the discontinuities, it acts like a very high Q cavity resonator, almost like a laser. When you’ve pumped the cavity hard enough, the field strength gets to the point some of the energy can jump the discontinuity and enter the interior of the Cube. Presumably something similar happens on the exit side but we’re still working on how that bit works.”

“So if it’s hollow, what’s inside?”

“You’re… not going to believe this,” said Abe, walking in with a laptop. “The 3D reconstruction is done.”

Everyone stared in amazement as Abe tilted the point cloud slightly and the structure became clear: open aisles, separated by rows of rectangular shelving with fuzzy objects of various sizes resting on them.

“The grid of squares we saw on the 2D slice were the support pillars of these shelves. And it’s not static, either. I went back to the 1-meter elevation slice we did last week and several new objects are here that weren’t there before.”

“But… That would mean someone or something is going in and out of the Cube! We’ve had it completely surrounded the whole time,” Dr. Chan replied.

“Yes, on our side of the discontinuity, “ Abe said. “We don’t know how static it is. It’s very possible that it can slide around somehow, maybe opening up some kind of portal if you’re in the right spot.”

Lynn’s face turned pale as the implications sunk in. “Get me the President.”

The Bag

“White House switchboard”

“Lynn Clarke, NY Cube Task Force division. We have a problem.”

A few minutes of holding later, the phone clicked.

“Situation Room duty officer here. I’m with the President, VP, and Secretary of Defense.”

“Mr. President, have you ever played Dungeons and Dragons?” she asked.

After a second of incredulous laughter, he replied. “This better not get out before Election Day. But yes, I was a bit of a nerd in my Harvard days. What does this have to do with the Cube, though?”

The Secretary of Defense chimed in “Never been into that stuff. What are you getting at?”

“What about Dr. Who? The TARDIS? Just like the Bag of Holding from D&D, it’s bigger on the inside than the outside.”

“So? This is a national emergency, not an RPG convention,” the President responded angrily.

“Did you ever stop and think about where all that stuff goes? The bag is bigger inside than outside, but that means there’s a big storage room somewhere.”

“Are you saying what I think you’re saying?”

“Yes, precisely. We checked historical satellite photos, the one in the Mojave has been there for years and nobody ever got close enough to know it was there. It must have been the prototype, and now they’ve started mass production. It’s not going to stop until we find a way to get a message across to whatever parallel universe is building these things and hope they’re willing to shut down their Bag of Holding factory.”

The President sighed. “And we’re the ones left holding the bag.”

Like this post? Drop me a comment on Mastodon

STM32L431 teardown

Thu, 02 Jan 2025 14:00:00 -0800

I realized a while back that I haven’t put any silicon reverse engineering content on the new blog yet. It’s time to change that!

Today, we’ll be doing a teardown of the ST STM32L431. Why? Because it’s a part I use a lot, it’s my go-to for “I want enough flash for A/B firmware images and a bootloader and a nontrivial amount of code, but a H735 is overkill”. And, most importantly, I had one on a scrap board in my “microscope food” bin (yes, that is the actual label on it).

Quick disclaimer before we begin (no, work didn’t make me say this): if you’re a prospective customer of my day job, be advised that all of this analysis was done for fun in my garage lab with optical microscopy and basic wet etch for deprocessing. The quality of results shown here is not representative of what I can do in a real lab with proper gear for CMP, plasma etching, SEM/FIB, etc.

Some quick stats

In total, the analysis in this post took about two days (much of that waiting for imaging, not actively doing stuff). I acquired 49.2 GB of optical imagery (90436 individual image files, including focus stacked/stitched intermediates) at a range of magnifications. All of the stitched datasets (at least the ones that turned out good) can be viewed on siliconpr0n, I’ll be linking some of the more interesting ones here. If you want to see more, definitely click around all of the delayered scans though.

Delayering was done with Whink rust remover (1-3% HF per the SDS, I really should do a titration at some point to figure out the actual concentration) followed by mechanical cleaning to remove delaminated copper interconnect and vias.

Imaging used a Labsmore LIP-X1 CNC microscope with a Mitutoyo VMU optical column. Objectives used were Mitutoyo plan apo 20x/0.42 for overviews and in-process inspection and Olympus Neo SPlan 100x/0.90 for high magnification closeups.

Device Overview

The STM32L431 comes in a bunch of different packages ranging from a 49-ball WLCSP up to 100-pin LQFP. The sample seen here came from a 48-pin QFN.

It contains a Cortex-M4 with FPU capable of clocking up to 80 MHz, 256 kB of flash memory, 64 kB of SRAM, a 12-bit ADC, dual 12 bit DACs, an opamp, two comparators, and a bunch of other goodies.

Top metal

The overall die size, including scribe line, is approximately 3.132 x 3.127 mm = 9.793 mm^2.

Several features are immediately apparent:

The device is made on a fairly modern, high layer count process (Wikipedia claims 90 nm, we’ll verify that). Power routing covers most of the surface, preventing us from getting a good view of a lot of the chip.
A regular region in the northwest corner looks like it’s probably some kind of memory
The south and southwest region looks analog

Looking at the northeast corner we can see internal part number “T435A”, a 2015 die copyright, the ST logo, and a little doodle of a dolphin. It always brings me joy to see silicon artwork, which has become less common in recent years.

This is the second dolphin I’ve seen on a recent ST chip (the STM32H735 has one too). Anybody know why? Internal project codename? Design team mascot? Local sports team at one of the offices?

EDIT: According to folks at ST, it’s actually an orca, not a dolphin. The part was codenamed “Orca” during development.

Unlike some of the other STM32s I’ve looked at, this one isn’t made in house at ST’s foundry - it’s made by TSMC. The fiducial in the corner is a dead giveaway. The overall appearance is consistent with the 90nm node but we can do some more digging to be sure.

Substrate floorplan

I deprocessed the sample to bare silicon substrate (going slowly and grabbing a lot of photos on the way, we’ll get to those in due time).

While it’s a bit tricky to tell with only wet etch deprocessing and sub-optical feature sizes, the STM32L431 appears to be seven copper and one aluminum metal layers for a total of eight metal layers.

The 20x scan of substrate with no annotations is on siliconpr0n. I took a 100x scan but had significant stitch artifacts in the memories so I didn’t upload it (but you’ll see crops of the nicer regions in this post). At some point I’m going to try to re-stitch and will add a link here if it turns out well.

At the substrate layer, we can see the pad ring around the perimeter, analog in the south, large memories in the northwest, and standard cell logic in the northeast. Several small memories (marked A through D in the floorplan) are present along the edges of the analog region. SRAMs A and B use the same bitcell as SRAM1, while C and D use the same bitcell as SRAM2.

Three smaller regions of standard cells can be seen outside the main logic area, sandwiched between the flash memory and theanalog region. Two of them are rectangular and use a different cell library than the rest of the chip (with a much larger row height), while the third is L-shaped and uses the same cell library as the main digital region.

Memories

SRAM1

At the north end of the die is SRAM1. This is a 48 kB single-port SRAM consisting of three identical 16 kB memory IP instances.

Each SRAM IP measures 796 μm x 201 μm (0.16 mm^2) and consists of two blocks of cells on either side of a central addressing spine, plus two spare columns for array repair between the central spine and the main east bitcell array.

Each block is 381 μm (256 columns at 1486 nm pitch) wide, and 179 μm (4 strings of 64 rows at 680 nm pitch) high for a total capacity of 64 Kbits (8 KB) at a density of 1.04 μm^2 per bit or 939 Kbits/mm^2 (not counting periphery). The overall array density including periphery is 1.22 μm^2 per bit or 800.5 Kbits/mm^2.

The 1486 x 680 nm (1.01 μm^2) bitcell uses a lithography-optimized (all poly running horizontal) 6T bitcell design typical of modern planar CMOS technologies.

Dummy features appear to be present around the perimeter of the array.

SRAM2

SRAM2 is a single 16 kB block in a separate power domain from the rest of the device, which can optionally be preserved across device resets and kept active in an intermediate low-power state in which SRAM1 is not preserved (in the deepest sleep states only the 128 byte backup SRAM is preserved and contents of both SRAM1 and SRAM2 are lost). It also has optional parity (error detection only, not full SEC-DED ECC as present on some higher end STM32s).

SRAM2 is 883 x 202 μm (0.18 mm^2), roughly 11% larger than SRAM1 for the same usable capacity which aligns well with the overhead of the parity bits. Each block is 426 μm x 178 μm and, as with SRAM1, there are two spare columns for error correction.

The 1482 x 676 nm (1.00 μm^2) bitcell is identical in overall size to the SRAM1 bitcell within the bounds of measurement error, but has a different appearance (most notably, a different color as seen at lower magnification). This is likely due to SRAM2 being optimized for low leakage, perhaps using HVT transistors in the bitcell (although interestingly, the datasheet makes no mention of it having worse performance: perhaps it is placed closer to critical paths in the interconnect to compensate for higher clock-to-out delay? I haven’t actually run any microbenchmarks to see if there’s a pipeline register or anything in the path).

Dummy features appear to be present around the perimeter of the array.

SRAM A (tentatively flash instruction cache)

SRAM A uses the same bitcell as SRAM1, and consists of 32 rows x 256 columns (plus two spare), for a total capacity of 8 Kbits or 1 kB. Unlike the larger SRAMs, this one only has a single tile array (with the addressing logic on the west and north sides) rather than a double array with row addressing down the centerline.

The 1 kB size and 32 x 256 array structure is consistent with the I-side flash cache (32 cache lines of 4 x 64 bits words).

SRAM B (tentatively flash data cache)

SRAM B is identical to SRAM A, but only 8 rows high. Its placement very close to SRAM A, as well as its size of 1 Kbit or 256 bytes, is consistent with the D-side flash cache (8 cache lines of 4 x 64 bits

Neither SRAM A nor SRAM B have obvious extra bits which could be used for cache line validity state or tag. The flash is 256 kB (logically 32K 64-bit words) so assuming a fully associative cache (plausible for one this small), 15 tag bits per line would be required, plus a validity bit. This would require an additional 512 bits (for I-cache) and 128 bits (for D-cache) of tag memory, likely implemented as discrete flipflops to enable parallel tag matching.

SRAM C

SRAM C uses the same low-power bitcell as SRAM2, and is 128 columns x 32 rows (4096 bits, 512 bytes) with two spare columns.

Its functionality is unknown, perhaps the RTC backup SRAM, CPU register file, or SDMMC TX/RX FIFO?

SRAM D

SRAM D uses the same low-power bitcell as SRAM2, and is 128 columns x 64 rows (8192 bits, 1K bytes) with two spare columns.

Its functionality is unknown.

Flash

The datasheet capacity for the flash is 256 kB with ECC (so a physical capacity of 288 kB), organized as 72 bits physical / 64 bits logical by 32768 words.

The flash memory is located in the northwest corner of the device. It consists of two blocks, one 25% longer than the other. This is consistent with 128 kB for the small half and 128 kB + 32 kB system memory (28 kB boot “ROM” and 4 kB trim/OTP region) for the large half, plus ECC.

Overall IP size is 1027 x 965 μm for the main memory array (0.99 mm^2) plus 955 x 300 μm (0.29 mm^2) for the high voltage generation block.

The lower bitcell array (128 kB logical, 144 kB / 1152 Kbit physical) measures 684 x 334 μm (0.23 mm^2). This gives a density of 0.194 μm^2 per bit or 5034 Kbits/mm^2 - 5.36x higher areal density than the 6T SRAM bitcell.

The overall flash structure is 18 copies of a “super-column” tile, subdivided into four copies of the basic column circuit, giving a total width of 72 columns (as expected from the datasheet).

The visible wordline structures have a pitch of 1.28 μm, and the entire array appears to be 260 wordlines high (likely 256 + dummy features)

Each column appears to have 4-way muxing, giving a physical array width of 288 bits. Bitline logic within each column has a pitch of 2.24 μm, while column logic has a pitch of 9.50 μm. This gives a tile size of 1.28 x 2.24 = 2.86 μm^2.

The 288 x 256 cell array which can be inferred from this image analysis, however, only gives a capacity of 72K tiles - 16 times lower less than the known 1152 bit physical array capacity. This suggests that the actual bitcell structure is significantly smaller than what we can see here.

Backing up to the intermediate deprocessed stages, we can see some more details: the column mux logic on metal… 2, I think, contains a diagonal structure which appears to consist of 8 wires at roughly 466 nm pitch, fanning out to unseen logic further down the stack. This suggests a 466 nm bitline pitch and a physical array width of 1152 bits, not 288.

This still requires additional row logic: 1152 Kbits with 1152 bitline array width would require a 1024 wordline height, not the 256 tiles visible here. There appears to be a 4-way symmetry in the wordline logic as well, which is consistent with this.

This suggests that the actual wordline pitch must be closer to 320 nm (and the green horizontal structures seen here are not wordlines, but strings of bit cells too small to resolve optically), giving an approximate bitcell size of 320 nm x 466 nm or 0.15 μm^2, consistent with the overall array density of one bit per 0.194 μm^2.

Main logic area

The digital core of the device measures 1.652 x 1.54 mm, with a small cutout (824 x 226 μm) in the northwest corner for SRAM1. This gives an overall logic area of 2.36 mm^2. At a 100% packing density of 413K gates/mm^2 from literature for an unspecified TSMC 90nm cell library this gives an upper bound of 974K gates; actual packing density will be lower (maybe around 750K NAND2 equivalents).

The library height is approximately 1.95 μm.

At metal 1, we can barely resolve the ~300nm wide power tracks and almost no details within the cells are visible.

Low density logic area

A small region at the center west part of the die, approximately 792 x 323 μm (0.26 mm^2), uses a different cell library with a much larger height (approximately 3.42 μm). Without much in the way of supporting evidence, I suspect this is the backup domain logic with the RTC, backup registers, tamper logic, etc, using an extra low leakage cell library.

At metal 1, we can make out substantially more detail in the cells than with the high density library used elsewhere in the device.

Conclusions

I’m not going to do a full netlist extraction or anything, this is just a high level teardown. It focuses on memories and logic because those are the portions I’m most familiar with - if anybody knows their way around PLLs, ADCs, DACs, etc. and wants to do some analysis on the mixed signal stuff I’ll gladly throw raw data your way.

There wasn’t really any specific goal to this analysis, just spending some time getting extra familiar with a part I use all the time. Hope you enjoyed!

Like this post? Drop me a comment on Mastodon

Intermediate Bus Converter

Tue, 15 Oct 2024 00:00:00 -0700

(Sorry for the slow updates… I’ve been busy with a lot of stuff, like getting ngscopeclient ready for the full v0.1 release at the end of the year. I haven’t stopped working on projects, just been too busy to do long-form writeups.)

When I started out with digital electronics, most of my designs ran on 5V from a barrel jack. This was fine for simple stuff, but I rapidly ran into two problems: 5V isn’t high enough for designs that use a lot of power (at least, if you want to avoid a ton of losses in cables), and barrel jacks aren’t super reliable. They rely on just a few points of contact so it doesn’t take much of a bump of the cable to cause a momentary loss of power.

Moving to 12V was the obvious way to solve this (and I did make a few boards that took 12V on a barrel jack), but I also wanted to move away from barrel jacks. Even 12V starts to become questionable for longer range power distribution at higher power levels, so a lot of datacenter-type DC buses use even higher voltages, such as 48V.

I also wanted to use a DC bus that was non-isolated (i.e. negative supply rail is at earth ground potential) since a lot of the projects I have in mind are test equipment or rackmount networking hardware that will have grounded shields on connectors. It’s important to note that I’m using +48V here, rather than the -48V power (positive supply rail at earth potential) that is commonly used in telco applications.

After a bit of digging, I found that Mean Well makes a power brick (GST280A48-C6P) that puts out ground-referenced +48V on a locking 6-pin Molex Mini-Fit Jr connector, with up to 280W output. This is enough to run quite a few of my planned gizmos (most with <25W power budget) from a single DC bus if I made some kind of DC PDU or distribution panel.

There’s just one problem: while 48V is great for long range distribution, it’s difficult to step directly from 48V to typical digital core voltages (often <1V for modern devices). You normally need to step it down to an intermediate voltage, usually something in the neighborhood of 12V, and then go from there to whatever your actual loads require.

Enter the intermediate bus converter.

Defining the requirements

At a high level, the job of an IBC is pretty simple: take in a high voltage (48V DC in my case) and step it down to a lower voltage (12V DC). But since this was going to be a system-level power supply, I wanted this to be a bit more than just a naked buck converter, so I drew up a few initial requirements:

48V DC input on a 6-pin Mini-Fit Jr compatible with the previously mentioned Mean Well brick
12V DC output on an 8-bit Mini-Fit Jr, maybe PCIe 8-pin compatible (this ultimately didn’t happen)
Remote on/off via a GPIO to support soft power on/off
Soft start to avoid excessive inrush when driving loads with a lot of input capacitance
3.3V DC auxiliary output to power rail/reset supervisors, soft power, and other standby logic
Temperature, voltage, and current sensors plus an I2C interface for querying them
Some additional EMI filtering and bulk capacitance

Version 0.1

The first iteration of the IBC was 87 x 80 mm in size, targeting the OSHPark 2-layer 2oz copper stackup. Back side was almost completely solid ground with a handful of signal net crossovers, while the front side contained an S-shaped power path with monitor/control signals around it.

This design set the general stage for all of the subsequent designs, and all future versions have retained electrical (though not always mechanical) interface compatibility: 5 pin PicoBlade connector containing the 12V enable, I2C, and 3.3V standby rail. The I2C bus contained a temperature sensor at 0x90 and the management microcontroller at 0x42.

The input protection and power path was pretty straightforward: a socketed fuse at the input, common mode choke and ferrite bead to avoid radiating switching noise out the input, a current shunt and some bulk capacitors, then a TDK-Lambda i3a series buck module. No explicit reverse voltage or overvoltage protection, although something would probably blow the input fuse if it were reversed.

On the output side of the buck module, there’s a bunch more capacitace, a current shunt, a ferrite, an an On Semi NCP455620 controlled-slew load switch.

In parallel with the main power path I put a 3.3V LDO to run the management logic and some voltage dividers to monitor the 48 and 12V voltage levels, plus a pair of AD8218 current shunt amplifiers to convert the shunt readings to voltages I could feed to the MCU (a STM32L031).

Version 0.2

The v0.1 IBC had one major problem: one of the tracks from the output of the buck module to the first capacitor was an 0.125mm track that was supposed to be a marker for an eventual zone fill, but I never added the copper pour! It functioned fine once I bodged a piece of copper wire across this path.

It worked well enough from a power perspective, but gave very noisy current measurements. After a bit of digging I realized the problem: the ADC bandwidth was high enough that it was picking up switching ripple through the current shunts.

So I made version 0.2 which fixed the missing zone fill and added a low-pass filter between the current shunt amplifiers and the MCU ADC.

Version 0.3

After this fix, everything worked great except that I realized I had derped and put the 48V current shunt in a high dI/dT path causing it to pick up switching noise.

So I made version 0.3, the final iteration of the first generation IBC.

Version 0.4

The i3a module had one major problem: efficiency. It ran hot (necessitating forced air cooling even at fairly light loads), and its ~3W idle power unloaded resulted in awful efficiency at the ~10W output levels required by the trigger crossbar.

While exploring alternatives, I came across the Murata MYC0409. This is a rather unique switching DC-DC architecture in that it doesn’t use an inductor like a typical buck converter. Instead, it uses a charge pump and switches a series of capacitors around.

This has one significant downside: it produces an unregulated, ratiometric output that is a fixed integer division of the input. Essentially it consists of a series of switching transistors and four internal capacitors; they are connected in series and allowed to charge off the input supply then connected in parallel and allowed to discharge into the load.

But this isn’t a huge deal for a converter intended to primarily drive fans and other DC-DC converters, and the ~800 mW idle power consumption was a huge draw compared to the i3a.

As part of the revamp, I switched from the STM32L031 to the L431 in order to get more flash so I could support a bootloader and A/B firmware slots, enabling field updates if that ever became necessary. The 32 kB of the L031 was a little small to fit two copies of the firmware plus a bootloader.

I also took this opportunity to move version 0.4 to its own repository since it’s a common component, not part of the trigger crossbar project.

The legacy v0.1-0.3 line still lives in the trigger crossbar repo history, but I have no plans to continue development of it at this time.

Version 0.5

v0.4 had a few teething troubles. For starters, as soon as I applied power it exploded.

More precisely, the LTC4367 did. It looks like when power was first applied I started getting a bit of current going down the supply leads, through the input side common mode choke that I had put there to suppress potential common mode EMI, hit the LTC4367 input, then it had nowhere to go since there wasn’t much input capacitance upstream. The end result was inductive spikes peaking at close to 100V amplitude which was enough to cause the LTC4367 to pop.

I tried a few fixes without success while troubleshooting (blowing a second LTC4367 in the process), then simply removed the entire input protection subsystem to test, at which point it worked like a charm.

Weeks later, I discovered that these spikes had also damaged the 1M ohm frontend on channel 3 of that scope - presumably exceeding the V/F derating for my 10x R-C probe and overloading the input. I’ll get it repaired eventually, but the 50 ohm frontend still works fine and that’s the one I use more often so I’ll probably just red-tag the channel in the meantime.

I saved the blown LTC4367s and will try to decap them at some point and see if there’s obvious damage to the dies. If it’s not too expensive I’ll try to get one X-rayed or even CT scanned, it’ll be cool to see what happend to the pin 1 bond wire and how much carnage is inside the package.

So I made one final version 0.5 which removed the CMC and LTC4367 in favor of a ferrite and TPS16630. This version also adds a few more TVS diodes and other protections against overvoltage and ESD.

This version works great and is on one of my prototype boards now, and I will probably be building a few more for testing soon.

Version 0.6 coming?

As of now, v0.5 is current. There’s one minor annoyance, the 3.3V standby rail switcher is fed by the 12V output after the ferrite bead, and could perhaps do with a small input capacitor to reduce high frequency transients. The end result is that there’s high frequency switching spikes injected into the 12V rail. Measuring with a current probe shows no corresponding spikes in current drawn by the load (unsurprising) so I don’t think this will have a huge impact on EMC and I’m just doing prototypes at this stage anyway.

My current plan is to use up all ten of the v0.5 boards I’ve built making protos of various equipment, then when I run out do a v0.6 spin with this fix and any other EMC or performance related tweaks I might want to make after having used the thing for a while in the lab.

Characterization

The only thing left to do was more extensive performance characterization.

This provided a good opportunity to throw together a large filter graph to demonstrate some of the multi-instrument capabilities of ngscopeclient.

The test is fully automated, with a scalar-stairstep filter ramping load from 0 to 6A in 100 mA steps, waiting 30 seconds between steps for thermals to stabilize, and the resulting data is plotted as values vs load current.

The final characterization setup used 16 channels of data from four physical instruments (R&S power supply, Siglent load, R&S multimeter, Teledyne LeCroy oscilloscope), plus on-board sensor data streaming via SWO to ngscopeclient (I’ll probably do a more comprehensive post about this flow once I’ve fine tuned it a bit), being processed by 54 different filter graph blocks to produce the final curves.

Total loss (power out minus power in) starts at about 800 mW with no load, increasing to just over 6W at max load. This is a massive improvement over the 3W idle power of the old design.

Efficiency is over 80% at very low load levels, reaching 90% at under 1A. Max efficiency (after correcting for losses in the wiring harnesses) is over 95% from 2-3A before falling to around 93% at 6A.

Output voltage does sag a fair bit at high loads, from 12V at no load down to about 11.1V at 6A. This is measured at the load and includes losses in the wiring harness, so typical chassis deployments with shorter wires would avoid a bit of this drop, but the droop is inherent to a non-regulating ratiometric converter like this (since any ESR in the output path is not being compensated by any kind of feedback network). So this isn’t something you’d want to use for a precision 12.00V rail, but as an intermediate rail that’s just feeding a bunch of buck converters it’s totally fine.

Realistically, I’m a long ways from needing 6A on any of my designs anyway. And the MYC0409 supports paralleling up to 4 modules for better performance under high load, so if I’m ever going to design something that power hungry I’d probably scale up the IBC to match.

Output ripple is only a few mV RMS, but around 330 mV p-p because of the spikes from the 3V3_SB switcher I mentioned previously.

Thermals look excellent. With no heatsink in still air the module did reach close to 100C at full load, but a small heatsink and a small amount of airflow was sufficient to keep temperatures below 40C over the entire test program.

Conclusions

I’m pretty happy with how the IBC turned out. I will eventually be doing one final board spin to fix the 3V3_SB switching spikes but I’m in no rush to do so, it’s good enough for my in-house use despite the spikes.

It’s a lot more efficient and runs cooler than the older design, has plenty of headroom for my designs to get bigger, and can scale to multiple converters if I need even more power handling capacity.

But for now, it’s going to be powering most of my large prototypes moving forward. Look forward to seeing it appearing in lots of projects coming up!

Like this post? Drop me a comment on Mastodon

Andrew's Lab

TCXO failure analysis

Backstory

What is a TCXO?

Initial Analysis

Depackaging

Visual inspection of quartz crystal

Electrical testing of crystal

Removal of crystal

Controller visual inspection

Controller die high magnification inspection

Bond wire inspection

Conclusions

Terminal Pacifism

July 11, 2029 - Cheyenne Mountain, Colorado

August 3, 1997 - Sunnyvale, California

February 28, 1998 - Los Angeles, California

July 4, 2000 - Los Angeles, California

December 18, 2505 - Washington D.C.

December 18, 2505 - Cheyenne Mountain, Colorado

Final thoughts

Trigger crossbar

The concept

High level design

The PCB

Bringup woes

Power connection

More power issues

Comparator Vcm issues

SPI/QSPI bus issues

Single ended CDR trigger input not working

FPGA flash pinout bug

Chassis design and assembly problems

Front panel

Operation

SSH

SCPI

Conclusions

STM32H735 OCTOSPI quirks

Enter the OCTOSPI

Prefetching and caching

Read backpressure and bus hangs

Write clobbering and coalescing

Conclusions

Switch project, part 3 - what Microchip doesn't (officially) tell you about the VSC8512

Background

What we have to go on

Initial observations

Known register interfaces

Page 0x0000: Base / Main (datasheet 5.1 / 5.2)

Page 0x0001: Extended 1 (datasheet 5.3)

Page 0x0002: Extended 2 (datasheet 5.4)

Page 0x0003: Extended 3 (datasheet 5.5)

Page 0x0010: GPIO / MCU (datasheet 5.6)

Page 0x2a30: Test (undocumented)

Page 0x52b5: Token Ring / black magic (undocumented)

MCB (Macro Configuration Bus)

Understanding the MCU interface

Instruction formats

Bit 14 set: Indirect pointer format

Bit 14 clear: Command format

Command codes

0x0: Set MAC mode / write MCB to shadow registers

0x1: Set SERDES media mode for dual mode SFP/baseT interfaces

0x2: Unknown, not observed

0x3: Read MCB to shadow registers

0x4: Suspend 8051 patch

0x5: Unknown, not observed

0x6: Byte poke

0x7: Byte peek

0x8 - 0xc: Unknown, not observed

0xd: Squelch workaround

0xe: Unknown, not observed

0xf: Unimplemented / nop

MCU address space

47cb / name unknown

47ce / addr_vec

47cf / cfg_buf[0] / cfg_vec[7:0]

47d0 / cfg_buf[1] / cfg_vec[15:8]

47d1 / cfg_buf[2] / cfg_vec[23:16]

`47cb` / name unknown

`47ce` / `addr_vec`

`47cf` / `cfg_buf[0]` / `cfg_vec[7:0]`

`47d0` / `cfg_buf[1]` / `cfg_vec[15:8]`

`47d1` / `cfg_buf[2]` / `cfg_vec[23:16]`

`47d2` / `cfg_buf[3]` / `cfg_vec[31:24]`

`47d3` / `cfg_buf[4]` / `cfg_vec[39:32]`

`47d4` / `cfg_buf[5]` / `cfg_vec[47:40]`

`47d5` / `cfg_buf[6]` / `cfg_vec[55:48]`

`47d6` / `cfg_buf[7]` / `cfg_vec[63:56]`

`47d7` / `cfg_buf[8]` / `cfg_vec[71:64]`

`47d8` / `cfg_buf[9]` / `cfg_vec[79:72]`

`47d9` / `cfg_buf[10]` / `cfg_vec[87:80]`

`47da` / `cfg_buf[11]` / `cfg_vec[95:88]`

`47f3` / `stat_buf`