The tent that was pitched near the Holywell Cross roundabout yesterday has now moved into the grounds on the Crooked Spire church.
The tent had been on a patch of grass opposite the Blue Bell pub since Thursday, but was moved sometime overnight.
On Sunday morning, the man who is living in the tent with his partner and their dogs said he was genuinely homeless.
He said: "I don't want to be on the streets but I don't have any choice. It's cold and it's hard because we get no help from anybody.
"I get no benefits because I keep getting sanctioned. How are we supposed to get anywhere?
"I can go round robbing or begging but whatever I do I get a bad name.
"Because I have got a dog I am not a priority to them.
"They say I have to get rid of him but he is not a dog to me he is my family."
As well as struggling to find the help he needs, the man alleges that homeless people in the town are being 'bullied' by the authorities.
"They are supposed to be helping us but they are just making it harder.
"They are going round giving people these 48-hour town bans. Where are people supposed to live?
"They have now turned the heating off down by the escalators. We can't find anywhere to sleep.
"They have started enforcing these public space protection orders but they shouldn't be doing that yet as they don't come into force until later in December.
"We are getting p****d on - getting water chucked on us, things thrown from cars."
"I got back from walking the dogs yesterday and there was a man dragging my tent across the road.
"The police said we can't camp there so we came here. They haven't said anything yet."
The man, who did not want to be named, said he did have a flat in Chesterfield last year but lost it.
He added that his partner does have a flat in Birdholme but he can't live there because her landlord won't allow it as he stole some beers from him some years ago.
The man said that while the authorities treat them badly, most of the public are 'wonderful'.
"I can't say enough good things about the public," he said.
"They help us more than any of the official people."
The Derbyshire Times will give the Derbyshire police full right of reply to the man's allegations on Monday.