Do bulls really hate the colour red?
The short answer
False. Cattle have two types of colour-sensing cone, not three, so red does not stand out to a bull the way it does to us. What makes a bull charge is the cape moving, and it charges the pink and yellow cape just as hard.

Verdict: False. Bulls are not enraged by the colour red. Cattle cannot tell red apart from many other colours the way we can, and what sets a bull off in the ring is the cape moving, not its colour. A bull will charge a waved cloth of almost any colour, and it charges the bullfighter's other cape, which is pink and yellow, just as hard as the red one.
The phrase "like a red rag to a bull" works fine as a figure of speech. As a description of bulls, it gets them wrong.
What a bull can actually see
Colour vision depends on cone cells in the retina, each type tuned to a different range of light. Most people have three types, which is what lets us split the world into reds, greens and blues. Cattle have two.
The key study here is "Photopigment basis for dichromatic color vision in cows, goats, and sheep" by Gerald Jacobs, Jess Deegan and Jay Neitz, published in the journal Visual Neuroscience in 1998. Measuring the light-sensitive pigments in the animals' eyes, they found one cone type tuned to short, bluish wavelengths and one tuned to the middle of the spectrum. There is no third cone that separates red from green.
That makes a bull's colour vision broadly similar to a person with red-green colour blindness. Bulls are not seeing in black and white. Blues probably still look distinct to them. But a red cloth would not glow against green grass the way it does for us; to a bull it is more likely a muddy, darkish tone that sits close to the colours around it.

The right-hand panel is only an approximation. Nobody can know exactly what a colour looks like to another animal, and two-cone vision is not identical across species. What the pigment evidence does show clearly is the part that matters here: cattle lack the machinery that makes red pop out for humans.
What makes a bull charge
If not the colour, then what? Movement, mainly, and the situation the animal is in.
A bullfight has two capes. The large capote, used in the early part of the fight, is usually magenta or pink on one side and yellow on the other. The small red muleta only comes out in the final stage. Bulls go for the capote with full force, long before any red appears, which on its own sinks the idea that red is the trigger.
What both capes have in common is how they are used. The bullfighter flicks, sweeps and snaps the cloth, and the bull goes for the thing that moves. A large animal that has been bred for aggression, released into a noisy arena and provoked will charge at a sudden moving shape whatever its colour. Bullfighters also rely on this: a still cloth draws far less attention than a moving body, which is why they keep the cape moving and themselves as still as they can.
So why is the muleta red at all? The explanation usually given is practical and theatrical. Red is a striking colour for a crowd, and it makes bloodstains on the cloth harder to see as the fight goes on. Either way, the colour is chosen for the people in the stands.
Why the myth is so believable
From the stands, the story writes itself. A red cloth waves, a bull charges it, and the eye connects the two things it can see. The bull's own vision is invisible to the audience, and the capote stage, where a pink and yellow cape gets the same treatment, is easy to forget once the red cloth appears for the dramatic finale.
Language did the rest. "Seeing red" and "a red rag to a bull" both tie the colour to anger, and red is the colour we already associate with danger, warnings and rage. A bull that hates red fits neatly into that picture. Like the horned Viking helmet, it is an image shaped by spectacle and then repeated so often that it started to feel like a fact.
None of this makes a bull safe to approach in a red shirt or any other colour. Cattle can be dangerous, especially bulls and cows with calves, and what tends to provoke them is the same thing that provokes them in the ring: something moving quickly into their space. Your choice of jumper has very little to do with it.