(CBS) – A visibly upset Earvin “Magic” Johnson offered more pushback Monday night against Donald Sterling, the L.A. Clippers owner allegedly heard in a racist rant posted on the Internet this week.

Johnson said Sterling should lose his NBA franchise in the wake of the controversy.

“I think that he has to lose the team, hopefully,” Johnson told reporters after attending a fundraising benefit for Gary’s Community School Corporation. “The commissioner and his partners have to decide (Sterling’s) fate.”

In the alleged recorded conversation with a mistress, Sterling criticized her for associating with minorities. He also complained that she posted an Instagram picture of her with Johnson and told her, “Don’t bring black people to games.”

Johnson, who said he’s not interested in owning the Clippers, suggested a boycott may be appropriate, depending on what NBA officials decide.

“If nothing happens, then you can boycott in the off-season, and not just the Clippers — the whole league has to,” Johnson said. “If he’s not thrown out, you’ve got to do something.”

The NBA is planning to announce a punishment Tuesday as Sterling’s Clippers head in to Game 5 of their playoff series.

Johnson discussed the controversy over the weekend. He appeared reluctant to comment further Monday when approached by CBS 2’s Brad Edwards on his way into the Gary schools event but later took questions on the topic.

Watch & Listen LIVE