"It's the brain. I look at people, and I think about people at that level and not on a superficial basis," Ben Carson said when asked whether he felt "lonely" being the only African-American member of Trump's cabinet.
"An agency can't just decide one day that it's not going to follow its own regulations," a plaintiff said.
Ben Carson has come under fire for an expensive makeover of his office.
Trump's budget proposal suggests major cuts to housing spending, food stamps and could put financial strain on low-income families.
"He's not here to help," said one of the many housing protesters who gathered at HUD Secretary Ben Carson's meeting with local housing leaders in Chicago.
The housing secretary visited Puerto Rico for the first time since Hurricane Maria hit in September.
An aide to the former presidential contender had said Carson initially removed himself from consideration due to inexperience.
A nationwide proposal could ban cigarettes, cigars and pipes in all living units, indoor common areas and administrative offices.