The average American family's finances continued to suffer long after the recession ended and the recovery began, new data show.

Median family income fell 5 percent between 2010 and 2013, according to a survey released by the Federal Reserve on Thursday, from $49,000 to $46,700. Median family wealth also fell by 2 percent, to $81,200.

The losses came even though the recession officially ended in 2009 and the country began gaining jobs 2010.

But although the ensuing recovery failed to boost the fortunes of middle-class families, it did aid the rich. Average income grew 4 percent, even though "only families at the very top of the income distribution saw widespread income gains between 2010 and 2013," according to the Fed.

The survey results "confirm that the shares of income and wealth held by affluent families are at modern historically high levels," Fed researchers wrote in a summary of the data.

Unlike other data sources, such as tax data, which show the top 1 percent or even the top 0.1 percent growing apart from the rest of the distribution, the Fed statistics show that rising inequality is being driven by the top 3 percent. That group's share of income has grown from 28 percent to 31 percent since 2010, although it was even higher before the recession.

Meanwhile, wealth inequality has been growing uninterruptedly. The share of wealth by the top 3 percent rose from 45 percent in 1989 to 52 percent in 2007 and then to 54 percent in 2013.

The Fed's data, taken from its Survey of Consumer Finances, is considered the most comprehensive source of information about Americans' wealth. More than 6,000 families were interviewed for the data set.

The data corroborate other indications that the typical American family has failed to recover the wealth lost when the housing bubble burst, although the report's estimate for median family wealth is higher than that from other data.