Fragility of Surface States in Non-Wigner-Dyson Topological Insulators