As the social news site tries to grow up, its managers are struggling to figure out what to do with communities like r/niggers.