Background: Quality of life in adenomyosis (AD) patients has been poorly investigated. Previous data suggest that AD has negative impact on the quality of life in these women. Methods: From September 2018 to December 2019, all consecutive female premenopausal patients aged ≥18 years diagnosed with AD by transvaginal ultrasound (TVU) were invited to participate in a comparative cross-sectional study. The Short Form-36-item (SF-36) health questionnaire and the Hospital Anxiety and Depression Scale (HADS) were administered. Work productivity and activity impairment were assessed using the Work Productivity and Activity Impairment Questionnaire: General Health version (WPAI:GH). Data obtained from these patients were compared with women with normal-appearing myometrium in the TVU recruited during routine gynecological visits. The study was approved by the Clinical Research Ethics Committee of the Hospital Clinic (reference HCB/2018/0919). Results: One hundred three patients with AD and 214 without AD were analyzed. Patients with AD compared to those without AD showed significantly lower scores in all domains of the SF-36 questionnaire and mean (SD) higher scores in the HADS questionnaire for anxiety (10.06 [3.04] vs. 6.92 [2.98], p < 0.001) and depression (6.39 [3.89] vs. 2.74 [2.01], p < 0.002). Significant differences (p < 0.001) were also found for the percentages of absenteeism (12.2% vs. 1.1%), presenteeism (31.1% vs. 11.4%), overall work productivity loss (38.2% vs. 12.4%), and activity impairment (55.7% vs. 9.9%). The presence of AD was associated with higher yearly estimated indirect costs of €5161.32 (€7928.0 vs. €2460.8, p < 0.001). Conclusions: AD negatively affects women's health-related quality of life, psychological health, and work productivity, with impairment at work and daily activities, and higher risk for anxiety and depression.