The effectiveness of Carbon Dioxide Removal (CDR) as a climate change mitigation measure has been acknowledged. However, there are few comprehensive studies evaluating CDR, and they often overlook the social constraints that act as barriers to its implementation. This study focuses on CDR directly related to the energy system and aims to elucidate the impact of social constraints on the quantity of CDR introduced and energy supply and demand. The results suggest that the effects on the energy system vary depending on the constraints. For instance, if Carbon Capture and Storage are restricted, a significant transition to low-carbon energy sources will be necessary. Conversely, if energy crops are restricted, there is an increase in the introduction of CDR, and a tendency to continue using fossil fuels in the transportation sector is observed. Therefore, the importance of considering the impact of social constraints on the energy system when deliberating decarbonization policies is highlighted.