Of FOSDEM Mixers and Men

So, FOSDEM 2025 happened. For the first time, we used some of our own 100% home-grown hardware, and managed to use most of its abilities.

In this (not-so-short) technical write-up, I’ll cover what we made, and what issues (rakes!) we had with it.

First, I really can’t stress enough that this wouldn’t be possible without Martijn Braam and Angel Angelov, who did all the hardware. They actually did a whole lecture about it, which you can watch here.

Check Martijn’s post about all the design decisions and process behind the FOSDEM Audio Board and its hardware.

The final product

The final FOSDEM box (for 2025) includes the following PCBs:

• a custom audio board (Martijn)
• a custom 5-port switch (Martijn)
• a custom power board (Angel)
• some small HAT boards for changing pinouts (Angel)
• an off-the-shelf HDMI capture card
• an off-the-shelf SBC
• a triple-display RPi HAT

We also use some MTA100 cables between the HATs, USB for some inter-board communication, and UTP to the switch. We only had 2 failed cables while assembling the boxes.

VentsiBox (some beta components)	Production Box

The Beginning

Martijn came to the NOC at FOSDEM 2024 and told us “you know, I had time to kill, so I did a thing…”, and the thing was Rev0 of the audio mixer.

So, Martijn did the first FOSDEM-spec revision of the board (with the correct I/O), and Angel helped to finalize Rev2 (which was actually in use).

FOSDEM Audio Interface rev. A

He even made a 3D-printed case for it (see the blog post!)

Having a semi-working, noisy-as-hell audio board, we started writing the firmware required for FOSDEM. We had to be able to:

• change how loud the microphones are
• independently control room and streaming outputs

Around Christmas, we received the audio boards for the boxes, and I took one with me to write and debug the firmware. Also, I didn’t have enough jumper cables, and had to buy more…

Mom: We have a mixer at home // The mixer at home:

The firmware

From plain-text to OSC

What does the Teensy in the board have by default? Serial!

We started writing an interface using plain-text over the tty the Teensy provides. This started becoming a problem when we needed to control it programatically, and needed to use timeouts for synchronization, and that being too slow. This commit replaces the control protocol with OSC over SLIP. We also added one more TTY (this time plain-text) for writing debug messages.

The next issue we had (even with OSC) was that the mixer doesn’t acknowledge changes, so we couldn’t write a generic multiplexer that doesn’t care about the specific commands sent. By sending back the same command we received, we didn’t need to depend on timeouts at all for normal functions.

The unwieldy main.cpp and who reformatted the code

The main.cpp file, containing all our code, got up to 600 lines long, and it was time for refactoring. Very carefully, I started to split everything in different files (see the mixer-refactoring history the month before FOSDEM)… and the board started crashing. Oops, the EEEPROM patches made calls to AudioMixer4::gain() with NaN as a value.

Also, the teensy audio web-generator-thingy really doesn’t like properly formatted code, so we had to exclude it from the autoformats. Now it’s in a seperate teensyaudio_generated.cpp file, with the following very-linter-friendly line on top:

// clang-format off

Sliders, knobs and buttons

The initial firmware (@Martijn, thank you again!) allowed us to speccify the loudness for each input and output, as a 6x6 matrix:

\	OUT1	OUT2	OUT3	OUT4	OUT5	OUT6
IN1	g11	g12	g13	g14	g15	g16
IN2	g21	g22	g23	g24	g25	g26
IN3	g31	g32	g33	g34	g35	g36
IN4	g41	g42	g43	g44	g45	g46
IN5	g51	g52	g53	g54	g55	g56
IN6	g61	g62	g63	g64	g65	g66

This allows us to send $g_{ij}$ percent of the volume from $IN_{i}$ to $OUT_{j}$, with $0.0$ being muted and $1.0$ being unamplified pass-through. This was saved to EEPROM with a sensible default value.

This is still exposed in firmware. However, I decided that this won’t be very helpful to the per-building video teams, and we made something more akin to a traditional mixer, having (pseudo) gain and volume knobs/sliders, as well as allowing muting channels while keeping the last-known values (which the matrix won’t allow us to do).

The final data model, also stored in flash, looks like this:

1. we have the matrix itself (see above): $G = (g_{ij})$
2. pseudo-gain control (input multipliers): $IM = (im_i)$
3. volume control (output multipliers): $OM = (om_j)$
4. another 6x6 matrix of mutes (stored as a packet bitfield): $M = (m_{ij})$

The final value we send to the mixer is $\texttt{RealGain}(i, j) = g_{ij} \cdot im_i \cdot om_j \cdot \overline{m_{ij}}$.

The only issue is all sliders are linear, which we would need to fix in the WebUI (and we didn’t get there in time for the conference). Screenshots below :)

The bugs

Who killed the display

Angel told me that the display doesn’t work after rebooting the board, but having no issues on a power-cycle.

Attaching a LED to the display RST pin showed that we never reset it on reboot:

The LED that did not blink

Turns out the prototypes had the display reset pin hard-wired high, but it was floating on the real board, as no-one told the library to drive it… Now we have a working display!

Audio board audio = audio board crash

This is caused by the 48 kHz sample rate patches, as the Teensy Audio Library natively only supports CD quality (16-bit, 44.1 kHz). Something somewhere dereferences a nullptr, but I’m leaving the debugging to people with more C/++ experience than me :)

No auto-restart on board crash

This actually bit us at FOSDEM, as two boards crashed during the setup on Friday. Adding a debug port on the Teensy, we initially made it wait until read on crash and forgot to remove it.

Exposing everything over the Internet

Python

I’m always vocal about this, but every time I start doing something serious in Python, I am forced to be reminded why I despise it.

CI and Debian packaging

Part of the reason we actually chose Python is letting the OS manage dependencies. The lazy way out (independent of language) was shipping all dependencies in the package (be it vendor, node_modules, or a venv), but that is not The Sysadmin Way (TM) and just the thought makes me shiver.

We packaged one of our pip dependencies missing in Debian, that being pythonosc – the parser for all mixer communications. Then we created our own packages for the control interfaces.

Note: Someone (me) managed to fsck up dependencies in debian/control by not pinning specific versions, and apt-get install of the UI doesn’t upgrade all underlying libraries. We found that out, as always, on Friday, when deploying everything.

The interfaces

We wanted to have two ways to manage the mixers - via ssh (for debug) and web (for the per-building teams to use). This meant we had to write a multiplexer for the connections to the board (first version made by Martijn, then I had to add multiprocessing to it because of issues with pyserial).

The WebUI is a simple FastAPI app, and the CLI is a simple shell/console executable. The smarts are abstracted in a library, which can be used for even wilder control interfaces (MIDI!)

Again, because of Python handling asyncio badly for the serial port comms, we had to add multiprocessing support to the Web UI.

JavaScript!

We managed to create this beautiful UI (for certain values of beautiful):

Mixer UI, courtesy of yours truly

It can be run as pure client-side JavaScript, although we use some PHP for rendering the correct room.

Authentication is also handled in the easiest to configure way possible – directly by the web server.

In theory, running the mixer outside of FOSDEM should be as simple as:

1. Getting an audio board
2. Installing fosdem-mixer-api from our debian repo (or just building it yourself)
3. Exposing the API (all relevant config is in /etc/mixerapi.conf), preferably with authentication
4. Publishing mixer.js, mixer.css, and creating a page with the design from vocto.php from here
5. Pushing some audio, so you can see the VU meters move!

Note: I haven’t merged this yet from my fork, will do it in the next couple of days.

The bugs

Number Theory is hard

Instead of every 10 ms, we received audio levels every 2 minutes…

Polling the audio board from the API had to happen on fixed intervals, and someone (again, me) managed to make an error in the maths required for that.

Huh, we use more CPU than ffmpeg

Reading byte by byte from a file descriptor is always CPU-hungry, but Python makes it even worse.

We tried to fix this on Saturday, but started dropping about half of the larger bundles. Everything still worked, though, just logs looked ugly!

If Influx drops one message, we have no data

We periodically push data to InfluxDB, so we can have near-real-time graphs of audio. We, however, managed to kill the whole InfluxDB instance several times… and mixers stopped reporting data.

The offending line looks simple enough:

requests.post(url, data=data.encode())

Guess what, that means the timeout is actually infinite, and when influxdb dies, the box never starts sending data again…

We didn’t fail!

Well, we actually managed to pull it off! Yes, there were some other minor issues, but FOSDEM happened, and most lectures are succeessfully recorded and streamed.