One possible source is the notion that ejaculation reduces your testosterone levels.

Making changes to your diet, substance use, or exercise will not make your penis bigger or smaller. Masturbation won’t affect your penis size one way or another.