Colonel Sir Horace was the son of Admiral Lord Hugh Seymour (son of Francis Seymour-Conway, 1st Marquess of Hertford) and Lady Anne Horatia Waldegrave. He married, firstly, Elizabeth Malet Palk, daughter of Sir Lawrence Palk, Bt., on 15 May 1818. He married, secondly, Frances Selina Isabella Poyntz, daughter of William Stephen Poyntz and Hon. Elizabeth Mary Browne, in July 1835.
By his first wife he had three children;