Dietary fibre is mainly classified according to its chemical characteristics but structure and particle size of fibre-rich feedstuff can also be decisive for digestion and performance. So far, only few studies investigated this in pigs. This experiment aimed to compare coarse and finely ground dried hemp plants and apple pomace regarding performance and ileal and total tract nutrient digestibility of growing pigs. Coarse or finely ground apple pomace or dried hemp plants were added to the diet of 56 nine weeks old growing pigs (DanBred x Duroc), housed in flat decks with each 2 animals. The growing pigs received the experimental diets for three weeks while performance was recorded. Eight pigs per group were sacrificed and digesta and organ tissue sampled. The stomach health was evaluated by visually scoring of the mucosa integrity. Apparent ileal (AID) and total tract digestibility (ATTD) were calculated using titanium dioxide as marker. Statistical analyses were performed using two-way ANOVA (p < 0.05). The highest feed intake (fibre particle size, p = 0.018) and bodyweight gain (fibre particle size, p = 0.018; fibre source x particle size interaction, p = 0.040), was observed in animals fed finely ground apple pomace, while the feed conversion ratio was 8–12% lower in pigs fed finely ground fibre sources (p = 0.012). No differences in stomach mucosa integrity were detected between the groups. The relative pancreas (p = 0.045), stomach (p < 0.001), and jejunum (p = 0.010) weights were higher in animals fed diets containing apple pomace. In contrast, the relative liver, caecum and colon weights were not affected by fibre source or particle size. The AID of protein and amino acids was not affected, while ATTD was increased by fibre source (hemp vs. apple pomace) reducing faecal nitrogen excretion. The AID of calcium was increased when diets contained apple pomace (p < 0.001), while zinc AID and ATTD were enhanced when diets contained dried hemp (p = 0.016; p = 0.016, respectively). Our results suggest that the structure as well as the chemical characteristics should be considered in a future fibre evaluation system in pigs.