Born Lady Anne Coke, the eldest daughter of the 5th Earl of Leicester, she grew up at Holkham Hall. Only a boy would be able to inherit and she grew up knowing that Holkham would never be hers.