Let's imagine a scenario where you travel back in time to college years and found out that the girl you really wanted during Junior year also really wanted you.
You didn't know it at the time so nothing ever developed of it, so you remained single at the time. You travel back in time, and you are in your Junior year body (so there isn't a younger version and an older version of you that exists simultaneously), but you have the knowledge you have now. Now, you're married, but you weren't during your Junior year. Is it okay to bang that girl? Would you be unfaithful if you do?