In order to predict seismic damage of a building, we must identify its major physical parameters, that is, shear and bending stiffness and mass. However, it is difficult to directly estimate these physical parameters. In this study, we try to identify the physical parameters of a full-scale five-storied structure based on the observed microtremor records. First, we estimate resonant frequencies by using Fourier spectral ratios of each floor response with respect to the ground. We then observe the changes of these resonant frequencies due to loading of an added weight with known amount. Finally, using an equivalent lumped mass flexural-shear model, we identify its physical parameters through the Hybrid Heuristic Search method.