Doctor Who is one of those shows where, truly, anything could happen. Sci-fi shows and movies tend to have carte blanche to go as wacky as they like at the best of times, but Doctor Who’s main focus of time travel means fans never know which period they’ll be in or who they’ll meet in any given episode.

Friends and foes alike are changed by their experiences with the Doctor, as we’ve seen countless times over the show’s long span. Allies have become sworn enemies and vice versa. This gets us wondering, then, what if things had turned out differently? Here are some characters from throughout the show’s history that could have been fantastic villains, if they hadn’t gone in another direction.

Luke Rattigan

We can’t help but feel for Luke Rattigan. The character appeared in just two episodes, “The Sontaran Strategem” and “The Poison Sky” (episodes four and five of Series Four). He was a teenage genius and millionaire who ran the Rattigan Academy for similarly gifted teenagers.

His claim to fame was developing the ATMOS system, which supposedly reduced all CO2 emissions from cars. Sadly, this breakthrough came about in collaboration with the Sontarans, who manipulated the technology to try and choke humanity to birth a new clone world for the Sontarans. Predictably, Rattigan was betrayed by the invaders; later giving his life to remain behind and destroy their ships with an explosive charge (on a one-way teleporter trip). There’s always potential for physically unimposing yet brilliant masterminds to be great villains, so it’s a shame that he was simply a pawn of the true enemies here.

The Ninth Doctor

Those who first jumped on board with the rebooted series would have met the Ninth Doctor, Christopher Eccleston, as their first. You didn’t need to know the full ins-and-outs of the character’s past to understand one thing very quickly: he’s the sole survivor of a catastrophic war (or he isn’t, as we’d find out later in the series) and utterly traumatized by its events.

While David Tennant’s and Matt Smith’s Doctors are known for their upbeat nature and quirkiness, the Ninth Doctor had a dark, damaged personality (as seen in episodes like “Dalek”). There could have been a fascinating arc that saw him go bad before Rose pulled him back from the brink.

Clara Oswald

Poor Clara. The Impossible Girl probably had the most harrowing story arc of all of the Doctor’s recent companions (if not ever). She suffered multiple deaths and was also lost in time on more than one occasion, after her journey through the Doctor’s own history following the encounter with The Great Intelligence.

Companions like Donna Noble suffered great personal tragedies, but the fate of Clara Oswald in terms of its implications for the timeline were beyond anything else. We don’t know where Clara is right now, the trauma of her experiences and the powers at her disposal could have turned her into a formidable vengeful force.

Captain Jack Harkness

One of the most popular supporting characters from the rebooted series, John Barrowman’s Captain Jack Harkness was always bombastic, always larger than life, always a riot to watch. His tragic demise during “The Parting Of The Ways” was blessedly short-lived, as Rose Tyler (having absorbed all the power of the Time Vortex) had the power to bring him back to life.

Inexpertly wielding that power, however, meant that Captain Jack’s life would be almost eternal. At the close of Series Three, he asks the Doctor what’s to become of him, as he’s continuing to age but unable to die. The revelation that he would later become the Face of Boe left fans reeling, but we have to wonder to: what if Jack had taken a darker path, as he went through impossible changes over all those thousands of years?

The Abzorbaloff

In 2005, the BBC’s Blue Peter held a children’s contest to design a Doctor Who monster. The winning entry would have the honor of featuring in the second series of the rebooted show. That entry would be nine-year-old William Grantham’s Abzorbaloff, an alien that could absorb its enemies into its body.

The silly-looking creature gave beloved British comedian Peter Kay a chance to play a pivotal role in the episode “Love And Monsters.” Kay’s Abzorbaloff was the antagonist, slowly and gleefully picking apart and absorbing a band of amateur London Doctor investigators, but the whole thing was played for laughs. With the endless creativity of childrens’ imaginations and Doctor Who’s rich history of ridiculous creatures, there was potential for a much better villain here.

The Autons

As long-time fans will remember, the Autons of “Rose” (Episode One, Season One) were not the first living plastic creatures in the series’ history. They had made a few appearances in the classic series since their debut in 1970’s “Spearhead From Space.”

The trouble with the Autons of “Rose” was that they were simply too easy to dispense with. A vial of anti-plastic is just a little too convenient, and it has even be used on more than one occasion. The Nestene Consciousness which controls the Autons wasn’t quite the top-tier enemy that fans deserved from the first episode of the series’ big return.

Nyssa

The Doctor’s companions are often enthusiastic, wide-eyed humans who idolize the Time Lord. Some, like Donna Noble, challenged and questioned the Doctor and his actions, while others (like Captain Jack Harkness and the short-lived DoctorDonna) have knowledge that rival his own.

Nyssa traveled with the Fourth and Fifth Doctor, proving invaluable to them as a Traken with a brilliant mind and bioelectronics expertise. When first parting ways with the Doctor, she went to the Terminus to perform crucial medical research. Had they parted on bad terms or had Nyssa been fundamentally changed by her experiences, she could have been a formidable foe herself.

Jackson Lake

Jackson Lake is a bit more of an obscure character, for those who don’t have an encyclopedic knowledge of every episode of the show to date. He appeared only once, in the 2008 Christmas special “The Next Doctor.” He is first believed to be (and believes himself to be) a different version of the Doctor, before the reveal that he is actually a humble math teacher from Victorian London, whose memories and knowledge had been tampered with by the Cybermen.

Inspired by all of this information, he made himself a TARDIS (actually a hot-air balloon), has a Sonic Screwdriver and thinks he is a Time Lord. Of course, he fights alongside the Doctor and helps to put an end to the Cybermen’s plans. With everything he knows, though, what a brilliant and unusual villain he could have made.

The Brigadier

As the incidents with Captain America and HYDRA have taught Marvel fans, you can never be quite sure who you can trust or how deeply the treachery runs in an organization. The Brigadier (Alistair Gordon Lethbridge-Stewart) has been a staunch ally of the Doctor’s for decades,  whether as commanding officer and colleague in UNIT or otherwise.

Naturally, though, close friends of the Doctor are always going to be high-priority targets for his many enemies. What if foes managed to turn Lethbridge-Stewart against him? With his influences and the forces he commands, it doesn’t bear thinking about.

John Smith

As we saw recently in “Human Nature” and “The Family Of Blood,” the Doctor is able to change their biology and become human. In this two-part episode, his Time Lord consciousness is stored within a mysterious pocket watch with Gallifreyan symbols on the front (the Master also did this to masquerade as Professor Yana).

Some echoes of his true nature remain, though, which is exactly where the problems could arise. As John Smith, he’s more vulnerable to being corrupted by everything he knows (yet doesn’t quite know), and there could be a possible storyline there.