Iain Duncan Smith has warned the government risks dividing society in his first interview since resigning as work and pensions secretary.
He attacked the "desperate search for savings" focused on benefit payments to people who "don't vote for us".
And he told Andrew Marr his "painful" decision to resign was "not personal" against Chancellor George Osborne.
Climate Change Secretary Amber Rudd said she was "perplexed" at Mr Duncan Smith's "bombshell" resignation.
She said he had urged colleagues to support the controversial change to Personal Independence Payments, which the government is now reconsidering, describing his decision to quit as "really disappointing".
On Friday, Mr Duncan Smith said he was quitting as work and pensions secretary in protest at planned disability benefits cuts.
Prime Minister David Cameron said he had been "puzzled and disappointed" by the resignation, arguing that those reforms had been "collectively agreed" and announced by the Department for Work and Pensions (DWP).
'Narrow attack'
Speaking on the BBC on Sunday, Mr Duncan Smith said he had supported a consultation on the changes but had come under "massive pressure" to deliver the savings ahead of last week's Budget.
The way the cuts were presented in the Budget had been "deeply unfair", he said, because they were "juxtaposed" with tax cuts for the wealthy.
He criticised the "arbitrary" decision to lower the welfare cap after the general election and suggested the government was in danger of losing "the balance of the generations", expressing his "deep concern" at a "very narrow attack on working-age benefits" while also protecting pensioner benefits.
If the focus on the working-age benefit budget continued, he said, "it just looks like we see this as a pot of money, that it doesn't matter because they don't vote for us".
Mr Duncan Smith, who said he felt he had become "semi-detached" from government, said the Conservatives had to return to being a party "that cares about even those who do not vote for us".
He said he cared "passionately" about "people who don't get the choices my children get" and "bringing people back in to arena where we play daily but they do not".
Ministers divided
He suggested the government was in "danger of drifting in a direction that divides society rather than unites it, and that, I think, is unfair".
Mr Duncan Smith's resignation has divided his former ministerial team at the DWP.
Pensions minister Baroness Ros Altmann attacked his tenure, describing him as "exceptionally difficult" to work for, and accused him of using his resignation "to do maximum damage to the party leadership" in order to support the campaign to leave the EU.
But her fellow DWP minister Shailesh Vara said he was "surprised" at Baroness Altmann's comments, saying: "Ros's recollection does not accord with mine and I'm sorry that this has all happened."
Disabilities minister Justin Tomlinson said the former secretary of state had "always conducted himself in a professional, dedicated and determined manner", while employment minister Priti Patel told BBC Radio 5 live it had been a "privilege" to work for him.
'Fundamental unfairness'
Speaking on the BBC's Sunday Politics, Conservative MP Heidi Allen, who rebelled against her party's plans to cut tax credits last year, said Mr Duncan Smith had "reached a point where he had had enough of the purse strings being pulled such that he could not deliver the welfare he wanted to".
But Ms Rudd told Sky News: "I do resent his high moral tone on that when the rest of us are absolutely committed to a one-nation government."
She said she found his "manner and his approach really disappointing".
Owen Smith, Labour's welfare spokesman, said Mr Duncan Smith had been "very honest in explaining how George Osborne could have taken different choices" and had revealed "the fundamental unfairness at the heart of government policy".
