I want to say something from Super Circuit, but I actually like how unique a lot of those tracks are (and, in my opinion, Mario Kart 8's version of Cheeseland kind of ruined the aesthetic), and I'm not sure if the problem is with those tracks or with the ridiculously stiff controls and unforgiving AI.
I'm surprised Rainbow Road Wii hasn't been mentioned yet - it's certainly my least favourite out of all the Rainbow Roads.
Besides that, I'd have to say Banshee Boardwalk from 64. Not sure what it is, but the level just... never sat right with me. Maybe it's because the bats just randomly decide whether or not to make you wipe out, or the "ehhhh" aesthetics (I don't care much for most ghost levels in Mario games).
city/highway-themed tracks are annoying because physically touching one of the non-kart vehicles causes you to fuck up to the point of like falling off the edge and whatnot. it's like Rad Racer but MUCH more rage-inducing.