A Tory minister said it was "ridiculous" to ask him to resign if he can't deliver his own promise to improve rape convictions.
In response, Mr Buckland, who secured many headlines by issuing a press release expressing his deep shame at the downward trend in convictions, said calls for him to quit if it doesn't improve was "low politics" and "ridiculous".
Ministers set out plans for a "system and culture change" after convictions for rape and lesser offences in England and Wales hit a record low.
Shadow justice secretary David Lammy accused Mr Buckland of shedding "crocodile tears" this week when he apologised for the downward trends in bringing sexual offenders to justice.
Mr Buckland said sorry over the dire situation and accepted Government cuts to the legal system played a part in plunging conviction rates following the publication of an official review.
But Labour's Shadow Justice Secretary David Lammy said: "The Justice Secretary's crocodile tears will mean nothing if the Government fails to reverse its disastrous failure of rape victims.
"The Conservatives' decade of cuts to the justice system has let rapists and other violent criminals off the hook while denying victims justice.
"Rape convictions and prosecutions have more than halved in three years. If he cannot reverse these figures within a year of his apology, the Justice Secretary should do the honourable thing and resign."
Mr Buckland told Sky's Trevor Phillips On Sunday: "First of all, I think that the nature of the challenge and the low politics of it speaks volumes of the complete failure by the Labour Party to actually come up with anything constructive on this most serious of issues.
"It's also constitutionally illiterate. Decisions made to investigate and prosecute are made by the independent police and their operational work and the CPS, which is independent. If there was any suggestion that prosecutions were being brought about because of political pressure on me frankly that would make convictions unsafe - it's a ridiculous argument."
Pressed again if he would resign if he did not meet a target, Mr Buckland said: "The idea that somehow a resignation or political pressure should be brought to bear on independent prosecutorial decisions is not only bad politics but it's actually dangerous. I'm not going to engage in that level of debate."