Gregory's Server
So I'm gonna ruin the joke and tell you that you're missing the point:
1. The group Jesus was speaking to (Pharisees) prided themselves on their charity.
2. The first two people--the ones who walked by--were members of sects the Pharisees considered snooty and uncaring.
3. So they're all expecting the punchline of this joke-structured story to be a Pharisee, but no, it's a (slur).
(Rimshot.)
Which is the point of the story: "Love your neighbour" doesn't have exceptions.