A U.S. circuit court judge who is seen inside the White House as a front-runner for the Supreme Court nomination is drawing criticism from conservatives who fear he would be more likely than others on President Trump’s list to gravitate toward the court’s ideological middle if appointed.

Sources close to the nomination process, including at least two influential conservative groups, have told the White House counsel’s office they are concerned about D.C. Circuit Court Judge Brett Kavanaugh, who Trump interviewed on Monday. The former clerk to retiring Justice Anthony Kennedy was added in November to a list of judges the president has promised to draw from for Supreme Court nominations, and he has since emerged as one of four top contenders for the latest vacancy.