Defense Secretary Jim Mattis and John Kelly, who just this week transitioned from being homeland security secretary to White House chief of staff, made a pact in the early weeks of the Trump administration to not leave the country at the same time in order to keep tabs on the White House, according to a report Tuesday.

Mattis and Kelly, both retired generals, were frustrated by the White House's implementation of its initial travel ban and were aiming to monitor a rush of executive orders issued by President Trump, the Associated Press reported.

The AP said that in one Oval Office meeting when Kelly was DHS secretary, he demanded to speak with Trump alone after the president complained the U.S. was admitting travelers from countries he deemed to be high-risk.

Kelly demanded other advisers leave the room, which Trump resisted at first, but later agreed to allow.

Kelly told Trump admitting these travelers was standard procedure and the people had legitimate reasons to visit the country.

The Supreme Court last month allowed the implementation of parts of a revised and limited version of Trump's travel ban temporarily barring foreigners from six Muslim-majority countries until the high court rules on the legality of the order this fall.