Matt Hancock sparked a row when he claimed other Conservatives took a "more balanced approach" to the issue of anti-Muslim hate in Tory ranks than former chairman Sayeeda Warsi.
Baroness Warsi, who was the female Muslim to attend cabinet, has been campaigning for a full independent inquiry into Islamophobia.
However, the British prime minister said the Conservatives would be conducting a "general investigation into prejudice" stopping short of Baroness Warsi's calls for an independent probe.
Hancock told the Today program: "Well look, I like Sayeeda (Warsi), she has a particular view on this. There are others who take a more balanced approach."
Asked if he was saying she was "unbalanced", the health secretary said: "No, I'm certainly not saying that. I have an enormous amount of respect for Sayeeda but she does take a particular view."
He added: "There needs to be an inquiry of course, but of course you should look into all kinds of prejudice.
"I think that this is something that any responsible party always needs to be on the look-out for."
Baroness Warsi posted on Twitter: "Oh @MattHancock Thank you for “whitesplaining” this to me.
"I’m so glad I have colleagues like you who can educate me even after my 30 years of experience of work in race relations. "Thousand apologies sir".
Miqdaad Versi, from the Muslim Council of Britain, accused Hancock of trying to pretend Baroness Warsi was "isolated" in her view.
And shadow home office minister Afzal Khan said: "Racism is not a “view”, it is the reality for many black and brown people.
"This gaslighting of Baroness Warsi’s experience and expertise within the Tory Party is a clear example of that racism in action.
“The Tory party has betrayed Muslim communities once again by breaking its promise to begin an inquiry into Islamophobia in the party by the end of the year.”
The prime minister has come under pressure to implement an inquiry into anti-Muslim hate in the Tory party, after committing to a probe during the Tory leadership contest.
Boris Johnson has also attracted controversy for comparing Muslim women wearing the niqab to '"letterboxes" in a newspaper column.
In an interview with BBC Radio Nottinghamshire on Friday, Johnson said: "Saj [Javid] and I are totally in agreement with this, we are doing a general investigation into prejudice of all kinds."
Asked about comments he made in the past about Muslim women wearing the Niqab looking like letterboxes, he added: "I'm very proud of my own Muslim heritage, my great-grandfather could recite the Koran off by heart ... it's absolutely true.
"When I was running London, and indeed in all my time as a politician, I have campaigned for equalities, for inclusiveness, of all kinds."
Source: The Independent