The company revealed earlier this year that it will be opening a new, larger store and food hall at the Ravenside Retail Park.
Marks and Spencer says the move will allow it to offer a wider range of products across its food, clothing and home departments.
However, some shoppers say it will leave a big hole in Chesterfield town centre.
Chesterfield and District Civic Society agrees the retailer will be a huge loss.
Chairman Philip Riden said: “This has been spun by the company as an opportunity to have a larger branch in Chesterfield with a wider range of stock.
"This may well be true, and it will be possible for customers to park closer to the store if it moves out of the town centre.
“The downside is that the town centre is losing one of the most iconic High Street names which has for decades been an important status symbol for a place the size of Chesterfield.
"If you’ve got an M&S you’re a proper town where people want to come and shop.
"Now it will have a prominent (and no doubt for a time empty) shop that will look unmistakeably like a former Marks & Spencer store.
"This is not just another empty shop — it’s an empty M&S shop.
“Hopefully, another retailer (or several, if it can be divided) will be persuaded to go into the store and so avoid a long-term void in a very prominent position.
"Hopefully also, M&S’s traditional neighbour, Boots, will not also decide to move out.”
Craig Baldock, regional manager for M&S, previously said he appreciated the move ‘may be disappointing news for some’, but it was ‘the right decision’.
“Shopping habits are changing, so we’re rotating our store estate and this investment in Chesterfield is part of our work to ensure we have the right stores to offer our customers a brilliant shopping experience,” he said.
Councillor Kate Sarvent, Chesterfield Borough Council's cabinet member for town centres and visitor economy, has described announcement as ‘disappointing’ but said the focus remains on plans to regenerate Chesterfield town centre.