The Olympic games were actually a tournament held every 4 years on Gallifrey, originally to pit the best of the Timelords against each other in a series of competitive events.
Due to fractures in space time, knowledge of these games was obtained by the ancient Greeks of Earth, who interpreted this information and instructions from the gods, and thus the Olympics as we now know them were born.
Inspired by Doctor Who.