Discovery of protein-binding fragments for precisely defined binding sites is an unmet challenge to date. Herein, formylglycine is investigated as a molecular probe for the sensitive detection of fragments binding to a spatially defined protein site . Formylglycine peptide 3 was derived from a phosphotyrosine-containing peptide substrate of protein tyrosine phosphatase PTP1B by replacing the phosphorylated amino acid with the reactive electrophile. Fragment ligation with formylglycine occurred in situ in aqueous physiological buffer. Structures and kinetics were validated by NMR spectroscopy. Screening and hit validation revealed fluorinated and non-fluorinated hit fragments being able to replace the native phosphotyrosine residue. The formylglycine probe identified low-affinity fragments with high spatial resolution as substantiated by molecular modelling. The best fragment hit, 4-amino-phenyl-acetic acid, was converted into a cellularly active, nanomolar inhibitor of the protein tyrosine phosphatase SHP2.