House Speaker Paul Ryan said Republicans are "very close" to getting a breakthrough on legislation to repeal Obamacare even after rowdy town halls during the Easter recess sparked more opposition among Republicans.

Lawmakers are "negotiating finishing touches" on a compromise, Ryan told the London think tank Policy Exchange during an overseas trip Wednesday.

"We have a system that is in dire need of reform," Ryan said, referring to Obamacare.

Since lawmakers adjourned for recess nearly two weeks ago, talks have been continuing among conservatives and centrists to let states opt out of some Obamacare insurer mandates. Conservatives in the House Freedom Caucus want states to be able to opt out of requirements for insurers to provide essential health benefits and abide by price controls that prevent them from charging sick people more.

However, centrists are against easing the mandates because of concerns about affordable coverage for people with pre-existing conditions. Recent raucous town halls haven't helped sway centrists to get on board.

Congress is set to return next week.

Ryan also said that Congress hopes to take on tax reform by the end of this year, hopefully by the end of summer. Lawmakers have sought to get Obamacare repeal done first to use the savings for tax reform.