This is not my area of expertise, but it seems like you might want to check your rudder shaft bearings, and also check under the cockpit for problems with the cables and sheaves. The hard part about checking the bearings is that unless you get in the water and wiggle the rudder around a little, it's hard to feel if there is excess play (rudder shaft looseness). There may be a cover over the top end of the shaft log that would be located in the center aft area of the cockpit. If you remove that cover and turn the rudder back and forth, you may be able to see it wiggle in the bearing. I'm not sure how your rudder tube is installed, though.
I suppose it could be a bent rudder shaft, too. Do you know if the boat has any grounding history? If it's not bent much, it may be hard to detect unless you drop the rudder.
I hope this helps.