We have developed a tracking system to provide a Cartesian robot with shape and pose of the human hand, in order to grip or move an object in an assembly task. Particle filter is applied to estimate hand shape and pose with two kinds of hand models. One hand model is a palm model for controlling a robot axis, and the other is a sideward model for gripping or releasing an object. We carried out two kinds of experiments for hand estimation. Experimental results show that the proposed system can estimate hand position approximately and recognize the difference between motions for gripping and releasing.