Repeated experiences and activities drive personality development. Leisure activities are among the daily routines that may elicit personality change. Yet despite the important role they play in daily life, little is known about their prospective effects on personality traits and vice versa. The objective of this study was to examine the extent to which within-person changes in leisure activities lead to prospective changes in personality traits, and whether changes in personality elicit prospective changes in leisure activities. We applied random-intercept cross-lagged panel models (RI-CLPM) to four waves of 13-year longitudinal data (2005−2017) from the German Socio-Economic Panel Study (SOEP) for the sample as a whole (N = 55,790) and for three specific age groups (young, middle-aged, and older adults). We examined between-person associations and within-person auto-regressive effects, correlated change and cross-lagged effects for Big Five personality traits (i.e., openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism) with self-reported frequency of leisure activities (i.e., physical activities, socializing, volunteering, political activity, artistic and musical activity, going out) and overall participation in leisure activities. At the between-person level, leisure activities and overall participation were most strongly associated with openness to experience. At the within-person level, we found reciprocal effects of extraversion only with overall participation in leisure activities and socializing. We found unidirectional within-person cross-lagged effects between leisure activities and personality traits and vice versa. Some effects were age-group-specific only. These findings suggest that leisure activities that are associated with certain traits at the between-person level are not necessarily those that trigger change in the respective personality trait. We discuss our findings based on the TESSERA framework for personality development. We conclude that the specificity of an experience or behavior and its corresponding trait is essential for personality development and should be subjected to further research.