Rowe was suspended in January amid allegations of misconduct and then departed the club by ‘mutual consent’ in February.
He took Town from the relegation zone to the play-offs in his first season and they were top of the National League when he left earlier this year.
The club has remained silent on the nature of the allegations.
When asked by a fan at last week’s AGM how difficult it was to depart with Rowe, director and vice-chair, Martin Thacker, revealed they had suffered ‘personal abuse’ for making the decision.
“I think everyone knows that James Rowe was a fantastic manager,” Thacker said.
"We all know where we were when we parted company with John Pemberton.
"James Rowe came along and made a difference, the record stands for itself.
"So as a board of directors, you can imagine it would not be easy for us to have a difficult situation presented to us when we were top of the league.
"We would not want to get rid of a manager that has got this team top of the league unless there was good reason to have to part company.
"The difficulty for us is that it is not appropriate for us to disclose the elements of what happened, the allegations, the insinuations, and all the things that were put to us.
“The board had to deal with fact, and at the end of dealing with fact, the agreement came to be that we had to part company and that is where we are at.”
He added: "Some of us have had personal abuse from people who don’t even know us, don’t even know the situation.
"We have been commended by the football authorities for acting commendably and professionally and that is how we want to leave it.
"We have remained professional and we wish him all the best in what he wants to do.”