Is the assessment of motor milestones valid and scaled equivalently for all infants? It is not only important to understand if the way we use gross and fine motor scores are appropriate for monitoring motor milestones but also to determine if these scores are confounded by specific infant characteristics. Therefore, the aim of the study is to investigate the latent structure underlying motor milestone assessment in infancy and measurement invariance across sex, birth weight, and gestational age. For this study, the birth cohort data from the United Kingdom Millennium Cohort Study (MCS) was used, which includes the assessment of eight motor milestone tasks from the Denver Developmental Screening Test in 9-month-old infants (N = 18,531), depicting early motor development of the first children of generation Z. Confirmatory factor analyses showed a better model fit for a two-factor structure (i.e., gross and fine motor development) compared to a one-factor structure (i.e., general motor development), and multiple indicators multiple causes modeling revealed no differential item functioning related to sex, birth weight, and gestational age. The study provides support for the use of gross and fine motor scores when assessing motor milestones in infants—both boys and girls with different birth weights and of varying gestational ages. Further investigation into widely adopted assessment tools is recommended to support the use of valid composite scores in early childhood research and practice.