Prince Harry reportedly considered changing his family’s surname to his late mother Princess Diana’s maiden name, Spencer, but was advised against it by his uncle, Charles Spencer.

According to The Mail on Sunday, the Duke of Sussex, 40, had contemplated abandoning the Mountbatten-Windsor surname — the one currently used by his children, Prince Archie and Princess Lilibet — in favor of his mother’s family name. During a rare visit to the U.K., Harry discussed the possibility with Diana’s brother, the 9th Earl Spencer, but was told it would be legally complicated.

“They had a very amicable conversation and Spencer advised him against taking such a step,” a friend of Harry told the publication.

Representatives for both Prince Harry and Earl Spencer did not comment when contacted by PEOPLE.

Harry and Meghan’s two children, Archie, 6, and Lilibet, who turns 4 on June 4, are officially registered with the surname Mountbatten-Windsor on their birth certificates. However, the family now informally goes by Sussex, following royal tradition — much like Harry was known as “Harry Wales” growing up, based on his father’s title at the time.

In a past interview with PEOPLE, Meghan reflected on how significant the Sussex name had become for their family. “It’s our shared name as a family, and I guess I hadn’t recognized how meaningful that would be to me until we had children,” she said. “I love that that is something that Archie, Lili, H and I all have together. It means a lot to me.”

The family name became a topic of public interest again in March, during Meghan’s Netflix series With Love, Meghan, when she gently corrected friend Mindy Kaling for calling her “Markle” rather than “Sussex.”

Mountbatten-Windsor is the official surname for descendants of Queen Elizabeth II and Prince Philip. Changing it might have been interpreted as a slight toward King Charles III, who reportedly holds the Mountbatten name in high regard, just as his father did.

The tensions between Prince Harry and King Charles remain unresolved since Harry and Meghan stepped back from royal duties in 2020. One lingering issue has been their loss of publicly funded security, which Harry referenced in a BBC interview, stating he “can’t see a world” where he would bring his family back to the U.K. without adequate protection.

The Mountbatten name originates from Prince Philip, who adopted it upon becoming a British citizen in 1947. Windsor, the royal house name adopted by King George V in 1917, was combined with Mountbatten in 1960 when Queen Elizabeth and Prince Philip decided it would be the surname for their direct descendants.

While royals don’t typically use surnames day to day, children often adopt a version of their parents’ titles in school or the military. For example, Harry and his brother William used “Wales” growing up, in reference to their father’s title, Prince of Wales.

When King Charles ascended the throne in September 2022, Archie and Lilibet officially became prince and princess, in line with royal protocols. Their titles were then updated on the royal family’s website to “Prince Archie of Sussex” and “Princess Lilibet of Sussex” in the line of succession. The Duke and Duchess of Sussex titles were originally granted to Harry and Meghan by Queen Elizabeth on their wedding day in May 2018.

Last year, PEOPLE confirmed that the couple had adopted “Sussex” as their family name for their children, maintaining the royal custom of using titles as informal surnames.