A controller design method is proposed for the communication disturbance observer, which improves the steady-state property by the modeling-error feedback. The proposed method stabilizes the system with a time-varying delay by designing a feedback gain in the observer system and output feedback controller using the Lyapunov-Krasovskii functional-based stability theorem. The controller is calculated by solving problems in the form of LMI by maximizing the allowable bound of the time delay and improving the performance of the system.