"Whether it was that the Colonial Office had wearied of the long strife between him and Mr. Tonge, or were dissatisfied at the want of judgment shown in disputing the right of the Assembly to deal with the seats of their own members, or to vote the public monies as they judged best, whether they thought that Sir John's age, now verging on 72, was too great for the work required of him, or whether with the imminent prospect of a war with the United States, in addition to the wars with the whole continent of Europe already on the hands of the British Government, there was need of a military man at the head of affairs in Nova Scotia, whether it was one or all of these that influenced the ministry, they came to the decision that it was time for Sir John to retire."43

The Wentworths retired to their lodge on Bedford Basin. Wentworth seemed happy enough, but Francis, who now occupied herself with her dogs and her chickens, was unhappy. Frances' brother, Benning44, died that year, the very year that Wentworth was obliged to retire from office. Benning's widow and children lost little time in removing themselves from Halifax and sailed for England. Francis felt alone and she longed for England. Charles, their only child, had already moved to England in 1805.45 Wentworth came around to thinking it best for them to go London, not only to keep Frances happy, but also to be there personally so to better press the government for the payment of long outstanding accounts. Thus it was that the Wentworths moved to England in 1810, taking up residence at London.46 Charles moved in with them.

In the years at London, Wentworth was to have a bad time of it. The government did not pay the kind of money that he thought was obviously due to him. He fell increasingly into debt, and the prospect of bankruptcy and debtor's prison loomed large. By 1812 the financial problems were that serious that Wentworth was unable to pay his rent or his tradesmen's bills. Just before the bailiff arrived at the doorstep, Wentworth fled. His biographer, Brian Cuthbertson writes of this sad episode in Wentworth's life:

"His attorney, a kindly and generous man, whisked Wentworth off to Liverpool under the assumed name of John Wallace. From Liverpool he took the first ship to Halifax, and the attorney managed to save the Wentworths' plate and diamonds before the bailiffs arrived and seized everything remaining. Charles Mary and Frances went to live in a small cottage at Sunning Hill near Windsor, Berkshire. Frances was not told of her husband's flight from the creditors; her mind was going and she was seriously ill and probably heavily sedated with opium."47