Dynamical systems often exhibit the emergence of long-lived coherent sets, which are regions in state space that keep their geometric integrity to a high extent and thus play an important role in transport. In this article, we provide a method for extracting coherent sets from possibly sparse Lagrangian trajectory data. Our method can be seen as an extension of diffusion maps to trajectory space, and it allows us to construct “dynamical coordinates,” which reveal the intrinsic low-dimensional organization of the data with respect to transport. The only a priori knowledge about the dynamics that we require is a locally valid notion of distance, which renders our method highly suitable for automated data analysis. We show convergence of our method to the analytic transfer operator framework of coherence in the infinite data limit and illustrate its potential on several two- and three-dimensional examples as well as real world data. One aspect of the coexistence of regular structures and chaos in many dynamical systems is the emergence of coherent sets: If we place a large number of passive tracers in a coherent set at some initial time, then macroscopically they perform a collective motion and stay close together for a long period of time, while their surrounding can mix chaotically. Natural examples are moving vortices in atmospheric or oceanographic flows. In this article, we propose a method for extracting coherent sets from possibly sparse Lagrangian trajectory data. This is done by constructing a random walk on the data points that captures both the inherent time-ordering of the data and the idea of closeness in space, which is at the heart of coherence. In the rich data limit, we can show equivalence to the well-established functional-analytic framework of coherent sets. One output of our method are “dynamical coordinates,” which reveal the intrinsic low- dimensional transport-based organization of the data.