“Harry is frightened and feels the only person who can do anything about it is his father,” a royal insider tells PEOPLE

Prince Harry’s efforts to safeguard his wife and children have caused a further rift with his father, King Charles.

When the Duke of Sussex, 39, returned to the UK in May for the 10th anniversary of the Invictus Games, one of his most cherished accomplishments, his father was notably absent from the event.

The two failed to meet during Harry’s visit, with the prince’s spokesperson citing the King’s “busy schedule.” However, insiders close to Harry suggest that there’s a deeper reason for the distance. Harry hoped to discuss his ongoing legal battle for security, which he’s been fighting for over four years and believes his father has the authority to address. (Buckingham Palace declined to comment on security matters, but a palace insider tells PEOPLE that the idea that Charles controls Harry’s security is “completely false.”)

The public family tension, which began in 2020 when Harry and Meghan Markle distanced themselves from royal duties, has been marked by harsh criticism and unresolved issues. But security remains the main point of contention, according to multiple sources close to Harry.

Even though the father and son had a positive interaction in February following King Charles’s cancer diagnosis, their relationship has since worsened. Those close to Harry say the King no longer answers his son’s calls or responds to his letters.

“A friend of Harry’s shared, ‘He gets told the King is ‘unavailable right now.’ His calls are ignored. He’s tried reaching out about the King’s health, but those messages go unanswered as well.'”

Amid the growing distance within the family, Prince Harry is deeply worried about the well-being of his wife, Meghan, and their children, Prince Archie and Princess Lilibet. Sources reveal that he has repeatedly reached out to his father, seeking assistance.

“Harry feels scared and believes that his father is the only one who can take action,” a royal insider shared with PEOPLE. Another source close to the matter added, “Harry is resolute in ensuring his family’s safety at any cost.”

The tension has now reached a breaking point. During the ITV documentary Tabloids on Trial, aired on July 25 in the U.K., Prince Harry expressed his fears that the negative media coverage could lead to a violent incident targeting Meghan.

“One individual is all it takes,” he explained, citing these “genuine concerns” as the reason for not bringing Meghan, 42, Archie, 5, and Lilibet, 3, back to the U.K.

The Duke of Sussex has also taken legal action, although he was unsuccessful in his attempt to have his taxpayer-funded security reinstated earlier this year. (He intends to appeal the decision.)

The reason preventing Harry from obtaining the protection he desires remains a source of heated debate. While the monarch holds no political power in the U.K., the responsibility for providing police protection falls to the Executive Committee for the Protection of Royalty and Public Figures (RAVEC), which acts on behalf of the U.K. government.

Harry, who had offered to pay for the police protection himself, also had that proposal turned down in court. He believes that, as King, Charles could step in and ensure the necessary protection is granted.

This disagreement has led to a deep rift between Harry and Charles, with sources claiming the tension has shifted from frustration to total “silence” from the King.

Sources close to the Duke and Duchess of Sussex, who have established a new life in Montecito, California, say the palace is mindful of what happened when Princess Diana left royal life — and instantly eclipsed the institution.

“Charles’ fear is a repeat of the past,” the insider says. “When his wife divorced the institution, she took the headlines, overshadowed his work and became a global superstar. Seeing his son leave with his wife was not part of the plan.”