Radical measures are expected to reform the way of working at construction sites. One effective approach is to quantify the movements within the site and evaluate labor productivity. In this study, we first conducted a hearing survey with on-site engineers and systematically organized the focus points (where engineers are paying attention) in the five elements of construction management, QCDSE, based on the survey. We demonstrated that these focus points can be quantified as Key Performance Indicators (KPIs). Next, we focused on D (Delivery) and S (safety) in QCDSE and developed a system to visualize the operation rate of construction machinery and potential risks, applying it to a dam construction site. By evaluating the operational effectiveness of the system from the perspective of on-site engineers, we confirmed the usefulness of the developed construction management system, as it can streamline management tasks for engineers and extract previously overlooked events.