Conservatives in Congress say they're worried that House Speaker John Boehner and other Republican leaders are preparing to cave in to Democrats on "fiscal cliff" negotiations by agreeing to raise taxes, a shift away from the Tea Party principles that swept the party into power in 2010.

But Boehner, R-Ohio, sent a powerful message Wednesday to the right wing of his party that the Tea Party members better not buck the legislation he introduces to avert the fiscal cliff.

"We are watching your votes," Boehner told House Republicans in a closed-door meeting, according to participants. Those who stand against the leadership risk losing plum committee assignments, Boehner warned.

That's exactly what happened to Reps. Justin Amash of Michigan and Tim Huelskamp of Kansas, whom Boehner booted from the Budget and Agriculture committees, respectively.

Both members are backed by the Tea Party and have been outspoken fiscal conservatives. They've also been thorns in Boehner's side as he's struggled to reach compromises with President Obama on budget issues, liked the 2011 debt ceiling deal that Boehner supported and Amash and Huelskamp opposed.

Boehner also removed Rep. Dave Schweikert of Arizona and Walter Jones of North Carolina from the Financial Services Committee. Both of them had also voted against the 2011 debt ceiling deal.

"This is clearly a vindictive move, and a sure sign that the GOP establishment cannot handle disagreement," Huelskamp said.

Boehner is pushing back against conservatives who resist compromise at the very moment the speaker is negotiating with the White House over a deficit reduction deal that needs to be struck by year's end to prevent massive tax increases and spending cuts from taking effect Jan. 1.

Boehner has so far refused a Democratic proposal to raise taxes on the top 2 percent of wage earners. But Boehner's proposal to raise additional tax revenue by closing loopholes in the tax code also offended conservatives.

"In case there was any doubt, Tea Partiers and small government constitutional conservatives should now realize that the real political challenge before us lies not with Nancy Pelosi, Harry Reid and Barack Obama," the Tea Party blog Conservative HQ opined. "It is in the ranks of the Republicans' principle-free leadership, like Speaker of the House John Boehner."

Top Republicans close to Boehner told The Washington Examiner that the GOP will eventually agree to a tax increase as part of a deficit reduction compromise. Many in the party are worried about polls showing that most Americans are ready to blame the Republican Congress if they fail to reach a deal by year's end.

The Tea Party group FreedomWorks, meanwhile, wrote to Boehner on Wednesday warning him not to push back too hard on Tea Party-backed conservatives.

"If the Republican Party wants to become the majority governing party, it must stop alienating the millions of grassroots citizens upon whom it depends and whose principles it claims to share," said FreedomWorks President Matt Kibbe.

Amash, posting on his Facebook page, said his committee ouster was punishment for voting as a conservative.

"It says to the growing number of young believers in liberty that their views are not welcome here," Amash said.