Labour have accused the Tory homelessness minister of "incompetence" for suggesting rough sleepers could move back with friends and family.
Luke Hall raised the "half-baked" idea in a letter to council chiefs, urging them to consider "all options" for supporting homeless people in the pandemic.
More than 5,000 rough sleepers were offered emergency accommodation as coronavirus unfolded, including in hotels, as part of a scheme called 'Everyone In'.
Mr Hall has now given English councils until June 11 to produce a "next steps" plan of where people should be housed.
He wrote to town hall chief executives on May 28: "In terms of move-on accommodation, all options need to be considered.
"We ask that you seek to encourage people, where appropriate and possible, to return to friends and family."
The letter went on to advise councils to find "as many sustainable move-on options for people as possible."
This could include working with housing associations to supply more homes, supporting people to move into private rents, or short-term accommodation to stop people returning to the streets.
But Shadow Housing Secretary Thangam Debbonaire slammed the letter - which came after the government pledged to end rough sleeping in the wake of Covid-19.
She said: "This half-baked suggestion shows that this Government is already rowing back from that promise.
"Luke Hall has shown a woeful lack of understanding about the complex reasons why so many people sleep rough in this country.
"If rough sleepers were able to return to their friends and family, they would not be sleeping rough.
"They need somewhere safe to live and the support to make a go of it.
“The Government should be working with councils to fulfil its commitment to end rough sleeping, not washing of its hands of the responsibility to house the homeless."
Coronavirus rules make it illegal in England to stay overnight away from the place where you are living.
A Ministry of Housing, Communities and Local Government spokesman insisted people would return to friends or family "only where appropriate and possible".
He added: "Our new rough sleeping taskforce has one overriding objective: to ensure as many people as possible who have been brought in off the streets in this pandemic do not return to sleeping rough.
"To help achieve this, we have accelerated plans for new rough sleeping services – backed by £433 million – which will ensure 6,000 new housing units will be put into the system, with 3,300 of these becoming available in the next 12 months.
"This is on top of the £3.2 billion we have given councils to meet the immediate pressures they are facing since the beginning of the pandemic.
"We’ve been clear councils must continue to provide safe accommodation to vulnerable rough sleepers and support those moving on from emergency accommodation."