To investigate the psychological effects of clothing color on people’s sense of warmth, a questionnaire survey was conducted. Participants were presented with T-shirt-shaped fabrics in different colors. The results demonstrated that people had a higher sense of warmth towards warm colors, such as red and yellow, and also felt a higher sense of warmth towards darker shades within the same color family. The fabrics were subsequently irradiated with light using a reflex lamp and their temperature changes were measured. The results indicated that among the different hues, the highest reach temperature of dark-colored fabrics was higher than that of light-colored fabrics, due to light absorption. However, the temperature increase associated with light irradiation did not necessarily correspond to the psychological sense of warmth. For example, the highest reach temperature for blue fabrics, a cold color, was higher than that for yellow fabrics, a warm color. Furthermore, spectral analysis of transmitted and reflected light was performed on these fabrics. The results showed a strong correlation between the sum of solar transmittance and solar reflectance in a wavelength region, including the near-infrared region, and the highest reach temperature measured by the light irradiation from a reflex lamp. In other words, the higher the proportion of light absorption by the fabric, including near-infrared light, the more easily its temperature increased.