First Minister Mark Drakeford has said there are "too many second homes" in some parts of Wales that are affecting communities and causing housing prices to rise sharply.
This comes after Mr Drakeford signed a Senedd co-operation deal with Plaid Cymru leader Adam Price on Wednesday.
The deal by the Welsh Labour Government includes potentially increasing taxes and more planning controls as part of a wide range of measures to tackle the second homes crisis that has hit rural areas of Wales, including Pen Llŷn, Pembrokeshire, and other communities where the Welsh language is under threat.
According to the Office for National Statistics (ONS) Wales has seen a 15.4% rise in average house prices this year, which is the biggest increase in the UK. In some coastal villages such as Abersoch 46% of properties are second homes and holiday lets.
In an interview with BBC News Mr Drakeford talked about communities where nearly "half of their properties are only occupied for only a part of the year".
However some holiday home owners have said that they are being blamed when the real failure is the shortage of affordable housing.
Jonathan Martin of Home Owners of Wales Group believed that he and other home owners were being used as a "scapegoat".
"We are being used as a scapegoat because of their failed national policies," he said. "If I was a politician do I want to stand up and say: 'I have failed in my duty to nominate the number of homes I have needed 10 years ago'?
"I'll blame someone else. I'll blame the second home owners – that's easy to do."
Mr Drakeford said in reply: "I never use the language of blame. It isn't helpful for anybody and it's not in our business of doing.
"What we are in the business of doing is trying to make sure that local people and young people in particular have a fair chance to go on living, growing up, and contributing in the communities to which they were born."
To get the latest news from WalesOnline sent straight to your email inbox click here .