Your penis gets smaller when it’s cold because it is drawing itself close to the warmth of your body. The temperature of the penis and testicles is tightly regulated to protect the sperm, which are healthiest within a particular temperature range. If it’s hot outside, they will hang away from the warmth of your body, and if it’s cold outside, they will draw in close (and look smaller) to be warm.