I had that same experience at one time. But owning a wooden Folkboat taught me a lot about varnish. What is screwing things up for you is likely dust. Dust you cannot see or feel. In the Hinckley factory (at least the one time I visited some years ago) they have rooms where they sand and do prep and rooms where they do the finish varnish, and then drying rooms and airlocks in between.
Here is what I have learned:
1. I generally no longer plan to varnish on the same day that I sand. I know I need to not be in a hurry and probably need to be wearing different clothes so that I am not the contaminator. Clean up takes more time than sanding.
2. Cleanliness is much more important than godlliness to varnish results. It may endanger your future in eternity to be swearing while you are varnishing, but what really counts for a beautiful varnish outcome is making sure you have a very clean environment. After sanding each coat (220 on the early coats, lightly 320 on final) here are my steps:
a. vacuum everything with a drywall filter bag--a regular shop vac bag is likely to spread the dust you can't see more evenly all around the area but it will still be there to rain on your varnish. Close up anything that stirs the air and take anything you can off the boat and to a room with no breeze whatsoever. Some pros who work in my yard actually hose down the area when working outside, but this extends drying time and you don't need a lot of moisture around.
b. Then thoroughly wipe down everything with real tack rags. I use the Crystal Premium brand and prefer them above all others. I do this immediately after finishing sanding and vacuuming and then wipe down again the following day with a clean tack rage before I actually varnish.
c. On varnish day, I wear different clothes from those I sand in.
d. I used to use a badger bristle brush but have changed to Jen Manufacturing 2" foam brushes (they are stiff enough to help you "tip" the varnish--the cheap ones at Home Depot are not, and some will fall apart under the solvents) because I have decided that the environmental effect is not any worse than all the toxic solvents I had to use to clean my old bristle brush. A bad brush will leave little nodules of dried varnish in your finish. I think brush dirt might be part of your issue--the new varnish sometimes releases old stuff in the bristles, but I can't tell from the picture. it is either dust or varnish particles.
e. If you want that absolutely smooth finish you are going to have to commit to 5 or 6 coats of gloss (I prefer Captain's or Flagship Varnish over Epiphanes) to fill in the grain. Epiphanes is a deeper color and requires more thinning--probably has better UV inhibitors, but the Petit products are easier to use and seem to level better and dry faster. I recommend more coats of thinner varnish rather than trying to slather it on in fewer coats. I heard you groan-patience is the price of mirror finishes. Thinning is key. I start out with 15% thinner with Captains varnish and end up with full strength. More thinner if you are using Epiphanes. If you want a smooth finish the thinning will vary with the temp and humidity of the day you do it, the brand of varnish, and your technique. There are a lot of good videos on tipping varnish but it is an acquired skill--keep a wet edge and avoid going back to "touch up". You need gloss as an under layer even if you finish with a matt or rubbed effect--it is the only way to fill in the grain and provide UV resistance.
f. I leave the area after spreading and tipping the varnish. Stirring around where there is wet varnish is asking for dust generation. I try to only varnish when the temp is above 50 and below 70. I have found that the first coat generally takes more than a day to dry no matter how much I thin it. Subsequent coats go faster. I do not use the "no sand" varnishes because, IMHO, they really do not level out all that well.
Varnishing technique is probably as controversial as choosing an anchor, but I actually enjoy the zen of the process. And as I have improved my technique over the last 40 years, it is nice to have a visible result to document your effort. Much more satisfying than most of the other boat chores.
