Well that's interesting. I would bet perhaps this occurs on the newer hardware builds of the SheevaPlug. Perhaps the same newer revision shares this and the Serial Console bug that newer Linux Kernels Fix?
Well that's two reasons to upgrade the kernel in the next build

Munin will chart ntp clock drift visually as well, so users of a sheevaplug with a working serial console can quickly check if their clock is drifting a great deal or not, feel free to post your munin ntp drift charts here.
(on a side note, I'm likely axing munin - I only included it as a quick way of tracking workload and memory leaks, I'm satisfied enough to not bother cluttering the ram disk in the next release with it. )