Sexual fantasies are a great sexual equalizer. Most people have sexual fantasies and even though some people may not fantasize, or may find it hard to have sexual fantasies, with some work anyone can have a sexual fantasy.
Roughly 95% of men and women say they have sexual fantasies in one situation or another. How common sexual fantasizing is depends in part on whether you are asking people about sexual fantasies during masturbation, during sex with a partner, or at times when they aren’t engaged in any sexual activity (other than the fantasizing). Research indicates that frequency of sexual fantasy declines with age (although the reasons for this have not been explored).
Men tend to fantasize more than women during masturbation and when not engaged in sexual activity, but this is not the case for fantasies during intercourse, where the gender differences are minor (and some studies find women engaging in sexual fantasy more than men).
While there isn’t any research on when we first start to have sexual fantasies, but in surveys that ask adults to recall, most say that sexual fantasies began somewhere between the ages of 11 and 13.
In terms of frequency over time it is interesting to note that there hasn’t been any significant change in the number of people who have sexual fantasies from the 1950s (when Kinsey asked about sexual fantasies) to the present.
Source:
Leitenberg, H. and Henning, K. “Sexual Fantasy” Psychological Bulletin Volume 117, No. 3. (1995): 469-496.

