Republicans on the House panel with oversight of the IRS called on President Trump Wednesday to fire Commissioner John Koskinen, citing a loss of trust.

Fifteen members of the House Ways and Means Committee wrote to Trump asking him to sack Koskinen, including the panel's chairman, Republican Rep. Kevin Brady of Texas.

"I have lost all faith in Commissioner Koskinen's ability to lead the IRS," Brady said. "He has been dishonest with Congress, and the IRS — under his watch — has continuously violated taxpayer rights. He has to go."

Conservative members of the House Republican caucus have long sought to impeach Koskinen for mismanagement, in their judgment, of the probe of IRS' targeting of conservative groups. Leadership, including Brady, were cool to that idea under former President Barack Obama.

Now, with a Republican in the White House, Republicans want Koskinen to simply be removed.

Wednesday's letter outlined the complaints about Koskinen. While Koskinen was not in office during the targeting scandal, Republicans fault him for frustrating their efforts to get the emails of former official Lois Lerner. "Not only was key evidence relevant to this committee's investigation destroyed under his watch, but he also misled Congress in the process, intentionally degraded customer service at the agency, and has since lost the trust of the American people," the letter read.

Nine Republican members of the committee did not join in signing the letter.

Koskinen has said that he will step down if asked by Trump. His term runs into November.