House Speaker Paul Ryan on Tuesday added his name to the growing list of Republican lawmakers calling on Alabama Senate candidate Roy Moore to quit the race in the wake of allegations of sexual misconduct against young women.

"He should step aside," Ryan, R-Wis., said of Moore, who is running as a Republican in the race to fill Alabama's U.S. Senate seat.

"Number one, these allegations are credible," Ryan said. "Number two, he should step aside. If he cares about the values and the people he claims to care about then he should step aside."

That's a shift from Ryan, who said last week that Moore's alleged actions are "disqualifying if true."

Many other Republicans said Moore should now step down in the wake of allegations from a fifth accuser, who said Monday that Moore sexually assaulted her in his car when she was 16 years old. The woman said Moore pushed her out of the car when she resisted after threatening her to not tell anyone about his actions.

Moore has refused to quit the race and Republicans in the Senate are now threatening to expel him if he wins.