This paper investigates the labor market effects of family caregiving, focusing on childcare and eldercare. Using large panel data in Japan and an event-study design that accounts for staggered treatment timing, I find sizable and persistent employment penalties for females after childbirth. Mothers' employment decreases by 36 percentage points one year after childbirth and stays lower five years later by approximately 19 percentage points. These effects vary by job characteristics, contract type, and co-residence status, highlighting substantial heterogeneity. In contrast, eldercare has smaller, at most 5 percentage points, and often statistically insignificant average effects on employment. However, eldercare penalties are larger and statistically significant, reaching up to 10 percentage points, for females with certain pre-event characteristics: those in low-teleworkability jobs, those in high physical proximity jobs, those on non-regular contracts, and those employed in small firms.