Profiles of former students struggling to repay their education debt are compelling. Too many students enroll in college but leave school without completing a credential, making even small amounts of debt burdensome. Some students complete their programs only to discover that because of either the institutions they attended or the fields in which they earned their degrees, they can’t find jobs that reward their education. And when the overall unemployment rate is high, even graduates with a good education can struggle to find a well-paying job.

But these problems do not mean that most student loan borrowers are less well off than those without student debt—many of whom never went to college. In fact, most outstanding student debt is held by people with relatively high incomes.

According to the recently released Survey of Consumer Finances for 2016, households in the top quartile of the income distribution, with incomes above $81,144 in 2016, held 34 percent of all outstanding education debt. The top 10 percent of households, with incomes of $144,720 or higher, held 11 percent of the debt. This debt represents a combination of students borrowing for their own education and parents or grandparents borrowing to help their children or grandchildren pay for college. Some of the loans supported students who are still in school, but other loans were taken out years ago.

Households in the lowest income quartile (with household incomes $0 to $22,586) hold only 12 percent of outstanding education debt. In other words, education debt is disproportionately concentrated among the well off.

Information on outstanding debt is based on where borrowers are after they have financed their college education, not where they started out. Those with the highest incomes hold the most education debt because more education frequently means more debt, but it usually also means higher earnings.

Students who complete more years of education tend to borrow more than those who are in college for a short time. As a result, average debt levels increase with level of education.

Average earnings, however, also increase with level of education. In 2016, median earnings ranged from $18,403 for adults who had not completed high school to $66,920 for those whose highest degree was a bachelor’s degree and $113,768 for those with professional degrees.

As you might expect, those who earn more owe more. The average household in the lowest income quartile with education debt owes $23,800. The average household in the top 10 percent of the income distribution with education debt owes $42,700.

People in the lowest income quartile tend to be those with little or no college education. Those in the top income quartile are most likely to have at least a bachelor’s degree. They paid for more education than those who are now less well off. The education many of them borrowed to pay for is also what helped them rise toward the top of the income distribution. In fact, 48 percent of outstanding student debt is owed by households with graduate degrees.

But the fact that average debt levels are lower for low-income households does not mean that none of these households have borrowed. It means that the bottom income quartile includes a larger share of the borrowers than of the debt. Fourteen percent of the households with education debt are in this income bracket. And while they may not hold large amounts of debt, 42 percent of those with education debt have an associate degree or less. For these households, even lower-than-average-debt levels can cause financial strain.

The concentration of education debt among the relatively affluent means that some policies designed to reduce the burden of education debt are actually regressive. Focusing on lowering the interest rate on all outstanding student debt or on forgiving large amounts of that debt would bestow significant benefits on relatively well-off people. Protecting households struggling with student debt through such policies as income-driven loan repayment plans, in which more than a quarter of all student loan borrowers now participate, is important. But targeting the households struggling most financially requires looking beyond who owes the most.