I get the same thing. I only use the roady on the rollers which isn’t really the right geo for me but it’s all I got at the mo’. The only tactic I’ve used with any success is to grab the very top of the bar, put your sit bones at the very back of the seat and sit there for a few minutes (still pedaling of course). Or learn to ride no hands. Takes a bit to get used to and you’ve gotta have the rollers set up proper, but it works just fine and your numbnuts will thank you.
If you’re on a level floor your rollers will have your front wheel slightly higher than your back wheel - the subtle change in your position might be contributing to your problem.
I get sore butt, wrists and hands if I ride the rollers for too long. It’s not a problem with the setup of the bike. It’s because I find it hard to shift my position regularly when on the rollers … It throws off my balance. If you stay in one position for too long, it will hurt no matter how perfect the bike setup is. I had the same problem when I rode the hour record … too few micro-adjustments on the saddle compared to normal riding. If your bike setup is good, I suspect this might be the problem.