This paper proposes an operation support system using images of scanning electron microscope (SEM) for micro assembly, which is called “image driven operation system (IDOS)”. IDOS works as an interface between an operation site and a micro processing site. The authors designed the following three functions in IDOS for micro assembly using a real time image matching and a SEM magnification controlling: (1) indication of a deformed figure for easy operation, (2) image-driven manipulation using a visual pixel feedback, and (3) automatic operation of tasks which hides low magnification images. Through the experimental pick-and-place task using a 100μm-sized “micro torii” under SEM observation, we evaluated that IDOS worked effectively as intended.