This paper proposes a sound source localization method which is robust for multiple sound sources and sound pressure difference. We propose the Main-Lobe Fitting algorithm using the directional pattern of a microphone array to reduce localization error in reverberant environments and noisy conditions such as a robot moving. The localization method is a post process of beam forming and it filters out reflected or interfered signals by selecting spectra which are fitted with the main-lobe model. We implemented the method on our 32ch microphone array mounted on a mobile robot, and evaluated the performance in different conditions. The experimental results show that the system provides accurate and robust sound localization for multiple different sound sources during robot motion.