Semi-Device-Independently Characterizing Quantum Temporal Correlations