Somebody famous once asked “if time travel is possible, where are the time travelers?” Presumably they would disguise themselves to fit in, but you have to assume there would be lapses. Maybe the people we think are crazy are really time travelers. That might explain things. Or maybe the people who think they are time-travelers are crazy.
The most plausible means of time travel was that I’ve read went something like this: if something could go into a black hole and not be destroyed, it could theoretically reappear at any point in the Universe and at any point in time. The problem is, nothing that we know of can survive going into a black hole. (Obviously, I’ve oversimplified a lot here, mostly because I don’t understand it too well myself. This might help.)
Then, of course, there are all the paradoxes that arise with time-travel. They make for good stories, but they also seem to suggest it’s impossible. Oh, well. It’s a question better minds than my own have had difficulty grappling with, I know that.