When modeling real-world biological multi-agents with reinforcement learning, there is a domain gap between the source real-world data and the target reinforcement learning environment. Therefore, the target dynamics are adapted to the unknown source dynamics. In this study, we propose a reinforcement learning method that uses information obtained by adapting source action to target action in a supervised manner as a method for domain adaptation in multi-agent reinforcement learning from real-world demonstrations. In limited situations such as 2vs1 chase-escape, 2vs2 and 4vs8 in soccer, we show that the agent learned to imitate the demonstrations and obtain rewards compared to the baseline.