While getting on with changes to the Musiland, I was hearing stories about the Hiface sounding so much better than the Musiland. So I got a Hiface and, straight out of the box it sounded nearly as good as my modified Musiland. Back to the drawing board with the Hiface then!
Ok, so let's have a look. This already had two oscillators, check. Now my focus moved to the PS as I'd improved the sound by addressing this on the Musiland. Once again, this was being derived from the USB power. I put a scope on it to see what noise there was - distinct spikes in the waveform. Here's where I got to synthesise two strands of my thinking. I'd recently found out about a new type of battery designed for electric vehicles - a LiFePO4 battery (that's lithium ferrophosphate for chemistry buffs). Its specifications impressed me (particularly its low impedance - lower than most capacitors) and it's 3.3V output was perfect for powering digital circuits. I bought some. Looking at the Hiface circuit I realised I could isolate the power to the clocks and run them from this battery.