Mr Mitchell met Police Federation representatives on Friday to discuss a claim – which he denies – that he called officers “plebs” last month.
The local federation’s chairman said the minister’s position was untenable.
He said the federation – which has been at odds with the government over police cuts in England and Wales and changes to members’ pay and conditions – was using the row as an excuse to bring up other grievances.
“We’ve now got other people who were not involved in the incident who seem to be trying to hijack this issue now and take it forward for their own purposes,” he said.
“The man’s apologised, the person he insulted has accepted the apology, let’s draw a line.”
He added: “The House of Commons will be back on Monday – Andrew Mitchell will be performing his duties as chief whip.
“I don’t really buy the argument that he can’t do the job. I think he can do the job.”
Mr Mitchell, who is reported to have sworn at an officer, has apologised for disrespectful remarks made three weeks ago after he was asked to get off his bicycle and use a small pedestrian gate rather than the main Downing Street gate.
But he has maintained he “did not use the words attributed to me”.
West Mercia Police Federation chairman Ken Mackaill – one of three representatives who met the Tory MP for 45 minutes in his Sutton Coldfield constituency office – said Mr Mitchell had “no option but to resign” after refusing to give details of exactly what he said in his outburst.
Mr Mackaill added: “Whilst he has repeated his – to use his words – profound apology for what he did say… he has also repeated his denial of using many of the words reported in the officer’s notes recorded at the time.”
Mr Mackaill said the issue was “about the honesty and accuracy of police records and there are implications for officers giving evidence in court, after all”.
“We take the view that this is a Cabinet minister challenging the accuracy of police records and that, we think, is of interest to all police officers.”
Prime Minister David Cameron and senior officers have said a line should be drawn under the matter following Mr Mitchell’s previous apology, but pressure on Mr Mitchell continues.
His future was the source of speculation during this week’s Tory conference – which he chose not to attend – with ministers repeatedly asked about the incident.
On Saturday, shadow home secretary Yvette Cooper said: “The failure by David Cameron and Andrew Mitchell to take this incident seriously enough and to sort it out straight away means Andrew Mitchell will clearly not be able to instil respect in Parliament or beyond as chief whip, and this will just drag on and on.”
Last month, police officers protested outside Mr Mitchell’s constituency office wearing T-shirt with the words “PC Pleb and Proud” printed on them.
Tory Party members are told simply to say he is “getting on with the job” ahead of a crucial summit with police representatives.
It’s been a very intriguing few weeks for the Conservative officials have attempted to stamp down on rumours about Chief Whip Andrew Mitchell in a bid to save his job.
MPs have been told not to talk to the media about the senior Tory, who is still under pressure after his row with a police officer on September 19.
His position is regarded as increasingly untenable after the mood within the party soured against him, as he stayed away from its conference in Birmingham.
The Chief Whip is to meet with police representatives on Friday in what could be a make-or-break summit.
Meanwhile, Tory HQ has told party members not to discuss the row and just say he is “getting on with the job” if they are pressed to comment.
But he has already been openly mocked by Tory Cabinet ministers amid suggestions senior figures are privately warning that he can no longer do his job effectively.
As Chief Whip, Mr Mitchell is in charge of keeping discipline within the party – a crucial role at any time and even more so as Prime Minister David Cameron works to keep the Tory right on side.
With Parliament due to reconvene on Monday, commentators are openly speculating that this weekend would be a good time for him to walk.
In a highly unusual move, the MP for Sutton Coldfield pulled out of the Tory conference after telling friends he feared he would be a “distraction”.
He is due to meet with members of West Midlands Police Federation on Friday to clear the air but, if the summit unravels, it could be the final straw.
The meeting comes after Work and Pensions Secretary Iain Duncan Smith laughed when it was suggested the MP could be sent to be high commissioner in Rwanda.
The remark, sparked by revelations about the MP’s links to dictator Paul Kagame, prompted Mr Duncan Smith to say: “I hear there’s no gates.”
Former Tory chairman Lord Tebbit told reporters: “I don’t think anybody would shed buckets of tears if he was to fall on his sword.”
Mr Mitchell has denied calling police “plebs” in the argument, which happened when he tried to leave Downing Street through the main gates.
Mr Cameron made him explain the row face-to-face but tried to draw a line under the affair after police said they would not be pursuing it.
If the Chief Whip is forced out now – almost a month after the clash – it will raise new questions about Mr Cameron’s judgment and his political operation at No 10.
If he thought he had toughed it out, Conservative Chief whip Andrew Mitchell may be in for a rude surprise.
There appears to be an increasing number of Tory MPs who want him to go after his clash with a policeman in Downing Street.
They believe David Cameron should have sacked him immediately for being offensively rude to the officer, regardless of whether or not he used the word “pleb”.
And they who do not see how he can possibly continue to exercise any authority or impose discipline after the embarrassing argument last month.
Government source say his position is increasingly perilous and it has been suggested that he could be forced to quit within days.
Intriguingly a political editor Adam Boulton reported from the Tory gathering in Birmingham: “Andrew Mitchell’s position is just about untenable judging by the mood of the conference towards him.”
Mr Mitchell did not travel to Birmingham despite being the city’s only Conservative MP after telling friends he was concerned he would be a “distraction”.
Nicknamed “Thrasher” – a moniker he claims does not come from Rugby School but was invented by Private Eye – the MP has severely damaged the Tory brand.
He reminded people of the “nasty party” label that Mr Cameron and the party leadership are so desperate to eliminate, and underlined the impression that the party is led by a bunch of arrogant toffs.
It would be to Labour’s advantage if he stayed because he would be a constant reminder of those labels and a continuing opportunity to tar the entire Tory party with the same poisoned brush.
Mr Mitchell is also failing the so-called Alistair Campbell test: he has become the story.
This is the worst possible place for a Chief Whip who is supposed to operate behind the scenes, in the shadows even, cajoling and coercing support from wayward MPs.
If Mr Cameron does tell him to go, it could happen very soon.
The Prime Minister surely will not want this to drag on into next week when the House of Commons returns after the party conference season.
Mr Mitchell will no doubt say that he has to resign for the good of the party and that he had become a diversion from its work.
This is a classic case of “if it were done, were best done quickly” – but it probably should have been done well before now.
Recently I was in Birmingham West Midlands walking along Broad Street during the Conservative Conference I came across a lovely bill board which summed up David Cameron which said “Say Hello To David wave the Police Force Good Bye”
And what made my days leading up to the conference was:
Five things that were left out of David Cameron’s speech
There were a few notable omissions from the prime minister’s address – including the Lib Dems and the police
1. The police. With Andrew Mitchell’s alleged “pleb” rant at Downing Street police officers still rather too fresh in everyone’s minds, and police reforms, budget cuts and pay cuts exercising officers up and down the country, maybe it was no surprise that David Cameron steered clear of mentioning this particular group of public servants. A poster outside the conference centre said: “Say hello to Dave, wave goodbye to your police service.”
2. An EU referendum. There was no mention either of a referendum on Europe, which had been clearly signalled by the prime minister in a round of interviews this week. But Cameron knows it’s not just about holding a vote, but what question you are actually asking.
3. The Lib Dems. Cameron focused on lauding the Conservative brand, and panning the Labour one, and left the Liberal Democrats out altogether. While Cameron name-checked Labour several times, and Ed Miliband in person, there was not a peep about his coalition partners or their leader, Nick Clegg.
4. Gay marriage. David Cameron has previously tried to sell gay marriage as consistent with Tory values ie commitment, but it’s not going down well with many grassroots activists who aired their dissent at conference fringes on the issue. So Cameron left it out, and left it to the equalities minister, Maria Miller, to speak up for gay marriage in a speech earlier in the day.
5. The Health Act. Cameron’s claim to be “the party of the NHS” was made without a single mention of the Tory-inspired shakeup of the NHS courtesy of recent legislation pushed through by the former health secretary Andrew Lansley, who has been shipped out of health to make way for Jeremy Hunt.