The have cited his "history of anti-Muslim sentiment and discrimination" as the reason for their opposition.
Ellison and Carson are the only two Muslim members of Congress, and Carson served on the House intelligence committee with Pompeo when Pompeo was a member in the House.
"Mr. Pompeo will regularly be required to liaise with Muslim leaders and Muslim communities abroad, especially during this time of conflict, tension, and unprecedented humanitarian need throughout much of the world," Ellison, of Minnesota, and Carson, of Indiana, wrote in a letter to Senate Foreign Relations Committee Chairman Bob Corker, a Tennessee Republican, and the panel's top Democrat, Sen. Bob Menendez of New Jersey, CNN reported.
"We need a credible Secretary of State, not one hobbled by a history of anti-Muslim sentiment and discrimination," they wrote in the letter. "We thus urge you to oppose Mr. Pompeo's confirmation as Secretary of State."
The two lawmakers are not alone in their reservations about US President Donald Trump's nominee.
During Pompeo's confirmation hearing before the Senate Foreign Relations Committee on Thursday, Sen. Cory Booker grilled the nominee about his views on Muslims. The two sparred in an exchange about what Pompeo has claimed is a "special obligation" that falls on Muslim leaders when it comes to preventing terrorist attacks in America.
"Do you think that Muslim Americans in this country who serve in our military, who serve in the state department, their failure to speak up, is that -- are they complicit in terrorist attacks?"
Pompeo, who currently serves as CIA director, responded: "Senator, each and every human — not just Americans — each and every human being has an obligation to push back against the extremist use of violence, from whatever faith."
He added that "when it comes to making sure we don't have terrorists brewing in places where Muslims congregate, there is a special place ... it's more than a duty, it's an opportunity."
Activist groups have also expressed their concern over Pompeo's previous comments about Muslims, including a 2013 speech when he criticized American Islamic leaders for their "silence" after the Boston Marathon bombings. He's also been associated with Frank Gaffney and Brigitte Gabriel, who are known anti-Muslim media figures.
"While the ACLU does not take a position on nominees, we believe nominees to the highest offices of government must uphold the law — which includes upholding religious freedom and preventing discrimination," Manar Waheed, Legislative and Advocacy Counsel for the American Civil Liberties Union, wrote in an ACLU blog post. "Senators must hold Pompeo accountable for his hostility toward Muslims in America and around the world, and ensure that such religious bigotry has no place in shaping State Department policies."
John Bolton, Trump's national security adviser pick, also has a history of history of provocative, often bellicose pronouncements. Bolton has a decade-long association with anti-Islam activist Pamela Geller.