Prince Harry has been planning the 10th anniversary of the Invictus Games for quite some time now. However, after weeks of will he or won’t he see his family, a spokesperson confirmed that Harry would not be catching up with his father, King Charles, during his time in the U.K. Earlier this week, the rep told People that their respective schedules made it hard to get something on the calendar, which also meant that the king would not be able to attend the church service Harry hosted in honor of his organization’s momentous milestone.

Instead, Charles and his wife, Queen Camilla, were busy at Buckingham Palace (just down the road from St Paul’s Cathedral where the Service of Thanksgiving took place) hosting a garden party attended by other family members including Princess Anne, Prince Edward and wife Sophie, Duchess of Edinburgh, and the Duke and Duchess of Gloucester.

Harry’s brother, Prince William, and his wife, Kate Middleton, were not at either event, though the Princess of Wales has been sitting out of royal obligations amid her ongoing cancer treatment.

Despite his family’s absence, Harry appeared to be in high spirits as he made his way up the cathedral stairs ahead of the ceremony. He smiled and waved to onlookers outside of the church and greeted the priests before inside. Harry’s wife Meghan Markle also opted out of accompanying her husband on his trip across the pond.

“In response to the many inquiries and continued speculation on whether or not The Duke will meet with his father while in the U.K. this week, it, unfortunately, will not be possible due to His Majesty’s full program,” Harry’s spokesperson told People earlier this week. “The Duke, of course, is understanding of his father’s diary of commitments and various other priorities and hopes to see him soon.”

Ahead of Harry’s trip to the U.K., royal experts and authors speculated that the family’s current crisis (Charles and Kate’s cancer diagnoses) would bring Harry and William together for a brotherly reunion. However, earlier this week, Ingrid Seward called it “unlikely.”

“William has enough going on and doesn’t need the stress,” Seward told The Mirror. “But he might realize Harry will want to see Kate after her cancer diagnosis as they were once so close.”