The patient was a 46-year-old woman who had received right breast conserving surgery and axillary lymph node dissection (II) for right breast cancer elsewhere 5 years previously. The final pathological diagnosis was invasive lobular carcinoma, pT3N1 (1/22) M0 pStage IIIA. Four years later, she complained of severe constipation and, following a close examination, she was diagnosed with rectal cancer (signet-ring cell carcinoma) cT4bN0M0, cStage II. Infiltration into the uterus and sacral bone was suspected with a CT scan. After colostomy, we started administration of FOLFOXIRI plus bevacizumab (Bev), S-1 plus Bev, followed by Hartmann's procedure. Immunostaining of the rectal lesion showed ER (+), PgR (-), HER2 (1+), and E-cadherin (-). The rectal lesion was similar in morphology to the primary breast cancer. The diagnosis of rectal metastasis of invasive lobular carcinoma was made. Oral letrozole treatment was started, and she has been followed without metastasis for 3 years after the rectal surgery. It is known that invasive lobular carcinoma sometimes causes gastrointestinal metastasis, but its diagnosis is often difficult in clinical practice. The diagnosis and treatment policy for rectal metastasis of breast cancer are discussed.