The First Amendment says the government can’t arrest you for peacefully assembling. It doesn’t say that taxpayers have to give funding to old money private schools. Maybe if there were financial consequences for discrimination they would take a more skeptical look at the value of this “cultural transmission gradient”.
There is generally considered rights held by (among other things) highly selective organizations to not have to follow public accommodation laws (which might harm their selectivity).