I see the problem(s) from a slightly different angle (of heel)...
You mention roundups while reaching in heavy air and also running.
When going downwind, you may have to change down sail area a bit sooner. Your design, like a jillion others from that era, has an IOR-influenced stern section. It likes to roll downwind if driven hard. Since it was never mean't to plane or be stable in surfing mode, you may have to slow down a bit. For comparison, look at the stern section of a J boat, or an Express or Olson. Notice that the stern is flatter and wider. Those boats were built to surf off the wind.
As to reaching, you are basically getting turned to weather by the hull shape as the boat heels. When reaching in big winds, reef sooner, let the main out more, generally reduce area to keep her on her feet. And, flatten the main.
If you are reaching with a full main or with the first reef, you may find that the main is too full -- maybe stretched to much by years and usage.
At 20 knots plus, I would expect to see one good reef in the main, and get that foot pulled out tight...
For a headsail I would want no more than a 110%. Note that if you reduce the headsail by rolling it in, the shape is not optimal, and will contribute to more heeling.
I have a modern 135 reefable genny, and in these conditions (rolled in 3 rolls) it is ok, but the shape is compromised some.
If you really want to change the rudder, do it right. Have the design work done by an NA before you have a new one fabbed up. Perhaps the existing ruddershaft has enough reserve strength for the extra load, but I would not quickly assume such...
I had a conversation about rudders with a local boatbuilder after I got our Olson 34; I had heard of someone putting a larger rudder on a couple of Olsons "for better control". The local builder, Steve Rander at Schooner Creek Boatworks, told me to save my money and learn to trim the sails to solve helm problems! This from the guy who routinely tiller-steers his ultralight 70 foot sloop to Hawaii and back. He often beats the wealthy pros, to boot!
Please Note: I am not casting dispersions on your helming. Really! But over the decades I have often found that the most boat speed on my boats a can be found by adjusting "the nut on the helm".
Please let us know what you decide to do. This is interesting stuff. If anyone wants to see the eliptical rudder and keel on our Olson, I have some new pics I could post, FWIW.
Fair winds,
Loren
Olson 34 #8