Organizations aimed at helping more black, Hispanic, and poor students get to college are already worried about reports that the Trump administration plans to fight college affirmative action programs.

According to the Washington Post and the New York Times, an office of the Justice Department is looking to investigate and sue colleges and universities whose policies they believe discriminate against white students. Both the Justice Department and Department of Education have yet to comment.

If such investigations begin, Anthony Davis Jr. of the NAACP’s youth and college division said that they could scare some schools away from enforcing affirmative action in their admissions policies and discourage students of color from applying to schools, too.

“We are extremely discouraged by this news coming from the Justice Department,” Davis said. “It shows a lack of support for the African American community,” he added.

The director of KIPP New York City’s Through College program, Jane Dowling, is worried the move will “perpetuate the notion of a student of color or low-income kid getting into college without being qualified.”

“That [perception] is one of the things that these college students struggle with the most,” she said. “It’s just heartbreaking.”

Critics of affirmative action include the head of the Education Department’s Office for Civil Rights, Candice Jackson, as ProPublica has reported. Multiple lawsuits have accused schools of discriminating against Asian-American students using the policy, too.

Dowling noted that other specific admissions policies, like ones that favor children of alumni and athletes, disproportionately help white and affluent students — a point other advocates echoed.

“We would all like to live in a world where affirmative action was not necessary, but unfortunately we do have to live in a world where there is systemic racism, where there is unequal access to quality education, and where African American and Latino students are overrepresented in communities of poverty,” said Rhea Wong, the executive director of Breakthrough New York, which provides educational support to low-income students.