Meghan, Harry felt despair while stepping down from royal duties

Meghan Markle and Prince Harry were devastated after deciding they no longer wish to be a part of the royal family.

According to royal author Omid Scobie, who worked with the former royals on their biography Finding Freedom, Meghan and Harry thought royal departure was the hardest thing they had to do.

“The hardest part for them was taking those initial steps away from their royal roles,” Scobie told PEOPLE.

“That was harder than they would ever imagined. They had had it all mapped out in their heads,” he added.

Although the couple initially wanted a “one foot in, one foot out” approach that allowed for both royal duty and private work, that arrangement was ruled out by Queen Elizabeth.

“They knew that they had to change things, but what they had actually planned wasn’t actually what was best,” said the author, who spoke with the couple’s friends and close associates to write Finding Freedom, with Carolyn Durand.

“They tried to find a way to compromise,” continued Scobie. “But would that have enabled them to have that level of happiness and security that they have today? Probably not. Those ties to the institution [of the monarchy] would have still been strong and there would have constantly been issues about financial endeavors and the business decisions they made.”