For vibration analysis of a rotor-bearing system considering gyroscopic effect and large damping forces due to bearings, a new analytical method to handle various boundary condition has been applied. It can be called a quasi-modal transformation method developed from the modal transformation. A complex eigenvalue solution method based upon the quasi-modal transformation is described and the results of corresponding numerical examples indicate that the solution has a sufficient accuracy. The quasi-modal method is also applied to the response history analysis in nonlinear rotor dynamics. The method is most suitable and powerful for vibration analysis considering nonlinear boundaries. In this analysis a new numerical integration method is introduced. It may be called a hybrid integration which combines the conventional direct integration and the modal one. The numerical examples show that the hybrid integration is numerically stable and practical.