Interlayer excitons (IXs) in van der Waals heterostructures exhibit unique optical properties due to their spatially separated charge carriers. However, the weak oscillator strength and radiative broadening of IXs make them difficult to detect with conventional absorption spectroscopy. Here, we use scattering-type scanning near-field optical microscopy (s-SNOM) to directly probe the dielectric response at the nanoscale. We first validate this approach by measuring the B-exciton in a four-layer MoS2 sample, where ion irradiation introduced defect-induced broadening. Extending this method to a MoSe2/WSe2 heterostructure, we observe a Lorentzian resonance at 1.35 eV, characteristic of interlayer excitons, with broadening dominated by nonradiative decay. These results demonstrate the capability of s-SNOM to image and characterize weak excitonic resonances at the nanoscale, overcoming the limitations of conventional techniques and providing new insights into localized exciton dynamics in 2D heterostructures.