Conservative justices seemed more open to the arguments of opponents of the program who say the university is practicing illegal discrimination by considering race at all, especially since the school achieves significant diversity through its race-blind admissions.
Chief Justice John Roberts, Justice Antonin Scalia and Alito raised repeated objections to the affirmative action plan.
Roberts wanted to know how the university would determine when it had a "critical mass" of diversity on campus that would allow it to end the program.
Near the end of the session, he complained: "I'm hearing a lot about what it's not. I would like to know what it is."
Scalia, a dissenter in the 2003 case, mocked the university's efforts to have more minority students not just in the student body as a whole but at the classroom level.
"How do they figure out that particular classes don't have enough? What, somebody walks in the room and looks them over to see who looks Asian, who looks black, who looks Hispanic? Is that how it's done?" Scalia said.
Justice Anthony Kennedy, who is the likely decisive vote, looked skeptically on Texas lawyer Greg Garre's defense of the program. "What you're saying is what counts is race above all," Kennedy said.
Kennedy also was a dissenter in 2003 and never has voted in favor of an affirmative action program, though he has voiced support for diversity in education.
Justice Elena Kagan is not taking part, probably because she worked on the case at the Justice Department before joining the court.
A decision should come by late June.