The English merchants trading to Virginia also entered complaint before the Privy Council against Harvey's administration. They sought relief from a duty of two pence per hogshead on all tobacco exported from the colony, from a fee of six pence a head on immigrants, and a requisition of powder and shot laid upon vessels entering the James.[304] The Privy Council, always careful of the welfare of British trade, wrote the Governor and the Council, demanding an explanation of these duties and requiring an account of the powder and shot. Harvey replied at great length, justifying the duties and begging their Lordships not to credit "the malitious untruths of such who by all means do goe about and studie to traduce us".
But the Privy Council, not waiting to receive all of Harvey's defense, decided to remove him and to appoint in his place Sir Francis Wyatt.[305] The new Governor was directed to retain the old Council and to confirm Kemp as Secretary.[306] But he was authorized to restore to Matthews any part of his estate yet withheld from him, and to reopen in the Virginia courts the case against Anthony Panton.[307] The day of reckoning had now arrived. When Wyatt reached Virginia, he lost no time in bringing Harvey to account for his misdeeds. He was arraigned before the courts, where he was forced to answer countless complaints of injustice and oppression, and to restore to their owners his ill gotten gains. Kemp wrote, in March, 1640, that Sir John was being persecuted with great rigor, that most of his estate had been confiscated, and at the next court would assuredly be swept away.[308] A few weeks later Harvey wrote to Secretary Windebank, to relate his misfortunes. "I am so narrowly watched," he complained, "that I have scarce time of priviledge for these few lines, which doe humbly crave of you to acquaint his Majesty how much I groan under the oppressions of my prevayling enemies, by whom the King's honor hath soe much suffered and who are now advanced to be my judges, and have soe farr already proceeded against me as to teare from me my estate by an unusuall way of inviting my creditors to clamour." He wished to return to England, there to repair his fortunes and seek revenge upon his enemies, but for some time he was detained in Virginia. The new Governor thought best to keep him in the colony where it would be difficult for him to plot against the administration. Harvey wrote, "I am denyed my passage for England notwithstanding my many infirmities and weaknesses of body doe crave advice and help beyond the skill and judgment which this place can give."[309]