I ran into this several weeks ago and initially blew it off as something I was probably not doing right.
I was trying to add my own button to an existing group on one of the Word 2007 tabs.
Nothing I tried would work. I finally gave up and just created a new, 1 button group with my button in it.
Fine.
Today, I ran across this PDF of a presentation by Ken Getz. On one of the slides towards the end, he makes an offhand comment about not being able to customize an existing group in the Ribbon.
Instead, he goes on, you could hide the existing group, and recreate it entirely, then add your own button(s).
He does not recommend doing so, though, because other addin’s might end up wanting to alter the original group and the results would be less that perfect.
So, I guess you just can’t.