I think we need a pic. “Cotter key” is confusing here - are you referring to a Woodruff key or a cotter pin? The aft end of the square Woodruff key should fit entirely inside the prop keyway so the retaining nut is flush against the prop (and the locknut is flush against the retaining nut). If the key sticks out the back of the prop, it can be cut down to size as long as it’s installed far enough forward to properly engage the shaft and prop. if I recall my key, it’s beveled a bit on the forward edge so it rests a bit in the front of the shaft keyway, which is tapered. The key can protrude a bit from the forward edge of the prop.
The cotter pin goes through a hole just aft of the locknut. If there is too much space there you’re missing the locknut or the shaft hasn’t been tapered correctly to fit the prop. In a situation where that pin is called upon to do anything I strongly suspect it will abandon its post with alacrity.