Senators on both sides of the aisle were fuming Wednesday over accusations that the Central Intelligence Agency was spying on their computers, and at least one lawmaker believes CIA Director John Brennan should step down if the claim turns out to be true.

Sen. Dianne Feinstein, D-Calif., said in an unprecedented speech on the Senate floor Tuesday that she believes the agency was tapping into the independent computers used by the Senate Intelligence panel, which she chairs. At the time, the committee was conducting an investigation into alleged torture of terror suspects by CIA agents during the height of the war on terror under then-President George W. Bush.

Brennan has adamantly denied Feinstein's spying charges, and the Justice Department is now investigating the matter.

“If this is a misunderstanding, then the CIA and Dianne Feinstein need to get this resolved quickly because it is a chilling story,” Sen. Lindsey Graham, R-S.C., said Wednesday. “If it is true, heads should roll and people should go to jail.”

Sen. Mark Begich, D-Alaska, said the matter has damaged his trust in Brennan.

“I’m concerned about his level of transparency to the committee and that puts in question the information we have received,” he said.

Brennan is already suffering from a trust issue among senators, with some believing he stonewalled them on questions and provided false answers during his nomination hearing over the September 2012 attack on the U.S. consulate in Benghazi, Libya.

Brennan told NBC’s Andrea Mitchell on Tuesday he would not step down unless President Obama made the request.

Graham said if the CIA was indeed hacking into Senate computers, “you would have to reorganize the CIA from top to bottom because the culture would allow this to happen. You’d have to bring in someone new to clean house.”