Background Cortical acetylcholine released from cells in the basal forebrain facilitates cue detection and improves attentional performance. Cholinergic fibres to the cortex originate from the CH4 cell group, sometimes referred to as the Nucleus basalis of Meynert and the Nucleus subputaminalis of Ayala. The aim of this work was to investigate the effects of volumes of cholinergic nuclei on attention and executive function. Methods The volumes of CH4 and CH4p subregions were measured in a subgroup of 38 subjects (33.5 ± 11 years, 20 females) from a population-based cohort study of smokers and never-smokers who have undergone additional MR imaging. To define regions of interest, we applied a DARTEL-based procedure implemented in SPM8 and a validated probabilistic map of the basal forebrain. Attention and executive function were measured with Trail-Making Test (TMT A+B) and Stroop-Task. Results We found a quadratic effect of the left CH4 subregion on performance of the TMT. Extremely small as well as extremely large volumes are associated with poor test performance. Conclusions Our results indicate that a small CH4 volume predisposes for a hypocholinergic state, whereas an extremely large volume predisposes for a hypercholinergic state. Both extremes have detrimental effects on attention. Comparable nonlinear effects have already been reported in pharmacological studies on the effects cholinergic agonists on attention.