An application to build a single dwelling at Grade II listed Ryshworth Hall in Crossflatts was submitted to Bradford Council by a Mr Sparow last year.
An outline plan for a home on the site had been approved in 2016, but this latest application was needed to provide more details of the scheme.
The plans are for a two-storey, three bedroom home.
The hall was divided into five homes in the mid 20th Century, and the application says: "These 20th century works fundamentally changed the setting of the listed building and harmed its interest.
"However, the subdivision of the building and its grounds has probably been key to its survival.
Plan for development of canalside homes is revealed
"Other detrimental changes to the west facing gardens include the loss of paths, flower beds and glasshouses, which probably occurred in the early 20th century.
"The development of the former western gardens is the best achievable conclusion to a phase in the history and development of Ryshworth Hall that started in the 20th century."
It says the question of what to do with the garden area of the hall the application relates to went "unanswered in the 20th century" and that this development would be the best way to prevent the site from falling into disrepair.
It adds: "Whilst it is arguable that the loss of a historically open space will cause some further harm to the setting of Ryshworth Hall, in the opinion of the writer, this harm is limited."
Historic England and Bingley Town Council raised concerns about the plans, and how they would impact the building.
But planning officers said: "The proposed dwelling will have an impact on the setting of the Hall, although with careful attention to design and detailing, the impact of a new built form here will be modest."
The plans will also involve the applicant repairing the boundary wall to the site that the application says is in need of repair and maintenance.