Sen. Rand Paul, one of the most outspoken critics of the National Security Agency programs revealed to have been spying on innocent Americans, said Sunday that the Fourth Amendment should apply to third-party records, such as phone records.

Paul was responding to “Fox News Sunday” host Chris Wallace's remarks about the latest NSA revelations from Edward Snowden that the agency is collecting 4 billion phone records every day.

“I would like to apply the Fourth Amendment to third-party records,” the Kentucky senator said. “I don’t think you give up your privacy when someone else holds your records.”

Paul was referring to the justification that NSA supporters use that because a third party - in this case, the phone company - actually holds the records, there is no expectation of privacy.

“When I have a contract with a phone company, I think those are still my records,” Paul said. “And you can look at them if you’re from the government, if you ask a judge.”

Paul continued, saying that warrants apply to one person, not every American.

“It’s absolutely against the spirit and the letter of the Fourth Amendment to say that a judge can write one warrant and you can get every phone call in America,” Paul said.

Paul said he would fight all the way to the Supreme Court to protect Americans' privacy and Fourth Amendment rights, but reiterated that he still believes the fight against terrorism is important.

“I'm for going after terrorists with every tool we have,” Paul said. “I'm not opposed to the NSA. I'm not opposed to spying. But I am in favor of the Fourth Amendment.”

It’s the classic liberty vs. security question. Paul believes that a warrant is needed for every phone record in order to comply with the Fourth Amendment.

“We can’t give up the Bill of Rights in order to try to fight terrorism,” Paul said.