Previously, we showed that caveolin-1 (Cav-1) expression is down-regulated in human breast cancer-associated fibroblasts. However, it remains unknown whether loss of Cav-1 occurs in the breast tumor stroma in vivo. Here, we immunostained a well-annotated breast cancer tissue microarray with antibodies against Cav-1 and scored its stromal expression. An absence of stromal Cav-1 was associated with early disease recurrence, advanced tumor stage, and lymph node metastasis, resulting in a 3.6-fold reduction in progression-free survival. When tamoxifen-treated patients were selected, an absence of stromal Cav-1 was a strong predictor of poor clinical outcome, suggestive of tamoxifen resistance. Interestingly, in lymph node-positive patients, an absence of stromal Cav-1 predicted an 11.5-fold reduction in 5-year progression-free survival. Clinical outcomes among patients positive for HER2, and patients triple-negative for estrogen receptor, progesterone receptor and HER2, were also strictly dependent on stromal Cav-1 levels. When our results were adjusted for tumor and nodal staging, an absence of stromal Cav-1 remained an independent predictor of poor outcome. Thus, stromal Cav-1 expression can be used to stratify human breast cancer patients into low-risk and high-risk groups, and to predict their risk of early disease recurrence at diagnosis. Based on related mechanistic studies, we suggest that breast cancer patients lacking stromal Cav-1 might benefit from anti-angiogenic therapy in addition to standard regimens. We conclude that Cav-1 functions as a tumor suppressor in the stromal microenvironment.