It always feels very bad for me to do any kind of help with diversity in conference schedules, because in practical terms it involves tricking people who don't want to do a talk, and who have nothing to gain from doing a talk, to do a talk. I'm not doing them a favor, and I don't feel like it's helping with the overall situation.
You see, it's not that only white males are invited to speak. It's that only white males feel like they are gaining anything from it.