The Congressional Black Caucus has taken Rep. Paul Ryan to task for comments he made about poverty in U.S. "inner cities" and invited the Wisconsin Republican to address the group.

Ryan, while appearing Wednesday on Bill Bennett’s "Morning in America" radio show, complained about a "tailspin of culture, in our inner cities in particular, of men not working and just generations of men not even thinking about working or learning the value and the culture of work."

The lawmaker, who grew up in a small Wisconsin town, immediately was accused of making racially charged and insensitive comments.

In a Friday letter to Ryan, the CBC called his remarks "highly offensive."

Ryan, the 2012 Republican vice presidential nominee and chairman of the House Budget Committee, has walked back his comments, saying he was "not implicating the culture of one community, but of society as a whole."

In further clarifying his remarks, Ryan added, "We have allowed our society to isolate or quarantine the poor rather than integrate people into our communities."

"Government’s response has inadvertently created a poverty trap that builds barriers to work."

But the CBC also wasn't pleased with his full clarification, saying that a "lack of resources," not isolation or government intervention, was at the heart of poverty in the U.S.

"A serious policy conversation on poverty should not begin with assumptions or stereotypes," the group said in its letter to Ryan. "Poverty in our nation is a critical problem that must be approached with diligence and the utmost respect for those who are trapped by poverty’s grasp."

The letter, signed by CBC Chairwoman Marcia Fudge, D-Ohio, and member Rep. Gwen Moore, D-Wis., ended with an invitation for Ryan "to join us at one of our weekly meetings to discuss our perspectives on poverty in search of finding constructive common ground."