Wednesday April 15th

Someone had a good idea.

I’ve been waffling about adding another unwanted think piece to the LLM pile, but then I ranted at my coworker about it today and it came out as a pretty good summary of my experience with LLMs I guess! So here it is:

I actually have made serious attempts to “vibe code” as experiments – those experiments are over though. I tried a few small side projects over the last year – scripts to manage my personal finances (I almost couldn’t pay rent the first month because I trusted the LLM too much and mismanaged my bills) and extensions to my personal reminder / note-taking system (after several frustrating and nearly useless months it’s all fully disabled now) and I even tried to use it to prototype an esp32 project, that one had some interesting aspects – there is a lot of boilerplate in embedded programming, so it was somewhat useful just as a “hello world” generator for various components that I could study… but ultimately it made a total mess of the code and I’m starting over completely on the project… the experiments are over, but I did discover legitimate useful cases in the process. if i had to summarize it would be: use it for problems rooted in pattern-matching, always give it obvious and concrete examples (so start by writing code manually always, then let the LLM fill out the boring parts, or show it code you’ve already written and explain how it should differ) and never let a session last longer than a few back-and-forths… it’s actually quite useful for a lot of things in the end, even though I occasionally fall into a trap of thinking I can do more and then waste time cleaning up the inevitable mess. on a short leash so to speak it’s pretty useful… but I wouldn’t actually bother if speed / time wasn’t important for . i don’t plan to use it in any of my personal work, i don’t see the point. it even seems actively harmful.

Oh! the other really important one is you have to ground everything in concrete tool calls and scripts or automated tests that the LLM can use to self-evaluate. Otherwise it just makes random shit up like always. The models themselves are incrementally better, but they fundamentally still have all the same problems. The tool harnesses are what really changed in the last year – the whole “agent” paradigm is basically based around the idea of using little islands of context by calling out to other LLMs and getting useful grounding context (the results of running a series of exploratory tool calls, or a web search or etc) to move forward with. They’re completely useless without doing that.

That’s also the sort of thing that will make tiny local models actually practically useful (for certain tasks of course) – a couple weeks ago I tried training another small model. this time on synthetic documents that showed examples of natural language search queries that get translated into small bash scripts with typical find/grep/etc commands. the experiment this time was to combine that with a (vibe coded, lol, but the script is ~100 lines) bash script which just takes a path and a natural language query as inputs, and feeds them to the LLM which is trained to only produce executable bash outputs – so at that point I can just do static analysis, run it through shellcheck etc and call the LLM in a loop until it produces something valid / executable if needed – then the final part of the script is just: it shows me the bash script and I can say “ok run it” or reject it. that’s the core interaction of something like claude code when it comes down to it, plus tool calling (which is just parsing the output of the LLM and calling tools the old fashioned way – same flow as the bash script generation, but with special symbols trained into the LLM for tool calling.)

I think what was most interesting me through this whole process was discovering certain kinds of errors of accumulation that I hadn’t really encountered like this before. It’s like the experience of technical debt, but it happens on the time scale of days not months or years, and you also don’t get the benefit of deep interaction with the system, so you can come out from supervising every detail of the code, and have correct code on the local scale so to speak, but in the process create architectural problems that aren’t always easy to spot until it’s too late and you have a mess to clean up. what’s interesting is how easy it is to do, but also how it doesn’t really have to do with correctness per se – the code might fail in seriously weird ways and be inscrutable, but will generally be correct in each isolated scope you study, the problems emerge in composition at the large scale. which makes total sense because these things don’t think, and that’s a level beyond where rote pattern matching will yield anything but insanity.

Sunday April 12th

Today I’m going to try to fix the amplifer module in the contraption.

There are two stereo amplifier boards stacked on standoffs behind the module panel. They are each connected to a gain pot, an audio breakout and the lead to the power supply.

The bottom board produces a significantly weaker signal than the top. I’m hoping for something simple like a loose connection or weak solder joint, but if I need to replace the board altogether I’ve got a backup, too.

It’s a good opportunity to practice debugging with the scope and the multimeter, which I have rarely done!

First, I’m going to probe every connection to see if I can find anything loose or broken.

In this case, probing every connection – eight audio lines, six gain pot lines, four power lines – didn’t turn up any problems. Ruling out a bad connection is kind of a bummer, since that would be the easiest thing to fix!

Next I’m going to try sending test tones through the amps while they are powered and connected to a scope, and see if I can reproduce the problem I’ve been having with the lower amp board. Fingers crossed I don’t need to replace it since I made the questionable choice to solder most connections directy to the board itself (rather than use terminals) and it’ll be a bit annoying to redo. :-p

I’ve got one of the battery power supplies I’m using in the contraption in a standalone enclosure (well sandwiched between two boards) that will be handy for testing (if I can rule everything else out, I can try a different power source and see if that fixes it – but I really hope that’s not the problem!) and I’ve connected one channel from each board to the scope probes, so I can compare the amplified signals coming from both in the scope.

As a test tone, the easiest thing is to just put a WAV file on my mp3 player, I think, with a stereo test tone. The amplifer boards have 1/8” TRS audio connectors, which is the way I’m feeding the inputs in the contraption itself, so it’ll be easy to feed the test tones into both through the same cables.

(pictured displaying the test tone in ASCII graphics is harry by mathr)

This makes the test setup, hopefully essentially identical to the way it is when it’s being problematic inside the contraption, except I can probe at it.

It’s hard to communicate just how confusing and fiddly it is to probe for signals on a contraption like this haha. I managed to get things arranged to be able to check both board outputs, but looking at the amplified signal was weird. The boards are D-class, which means they try to be off as often as they can, and actually construct the signal with pulses, so just comparing the two outputs visually on the scope didn’t help me very much.

Maybe if I knew the scope better I’d be able to suss out the normal noise from problematic noise by looking?

I didn’t get anywhere with the scope. :-/

Instead I hooked the amps up to two exciters (which they’re normally hooked up to) to debug by ear. I hear the same weak & crackly signal from the bottom board.

At this point I think it’s worth swapping out the board – the one on the bottom is an older design, which shouldn’t matter at all, but it’s older so maybe I managed to damage it at some point without realizing it.

Yep, I had a bad amplifier board! The replacement is working well, and the contraption is back to four channels again.

My mp3 player died (hopefully not for good?) so I used a CD (just arrived from Commune Disc though nearly 20 years old now!) of Hidenobu Ito’s reimagining of Systemisch as a test tone, instead.

Wednesday April 1st

I just started reading W. Ross Ashby’s An Introduction to Cybernetics last month. I’m amazed to find that the operand, operator, transform and transformation concepts introduced right away in the book seem to encompass functional programming, relational database design and other fundamental concepts embedded deep into programming languages and systems, including machine learning.

I guess it makes sense! My understanding is that information theory and cybernetics drove a lot of early applied theory in computing, but I’m curious to learn more…