Downing Street has acknowledged to the BBC that, although he had a permit to Number 10, it was "temporary" and "given back several weeks ago" for prominent Labour contributor Lord Waheed Alli.
Labour has been accused by the Conservatives of "cronyism" when the Sunday Times revealed that the peer, who has given the party hundreds of thousands of pounds, was able to enter Downing Street even though he had no formal position in government.
Minister Pat McFadden of the Cabinet Office stated earlier on Sunday that it was "unusual for people to have passes to attend political meetings".
When Lord Alli did have a pass, it was confirmed by a No 10 source that he attended a garden reception; however, the source did not specify why Alli had a pass, nor when or why he gave it up.
The TV executive was appointed to the House of Lords by Tony Blair in 1998 and has recently led fundraising for the Labour Party, as well as donating to individuals including Prime Minister Sir Keir Starmer.
Asked about the Sunday Times report during an appearance on Sky News on Sunday, Mr McFadden said: "I don't think he's got the pass now but he may have had one in the past."
Tory shadow paymaster general John Glen MP said it was "disappointing" to see Labour "only attempting to act on their culture of cronyism after feeling the pressure in the media".
It comes after Chancellor Rachel Reeves was challenged over Labour donor Ian Corfield's appointment to a civil service position in the Treasury.
Mr Corfield - who has previously donated £20,000 to Labour MPs, including £5,000 to Ms Reeves last year - was given a temporary job focused on organising an international investment summit taking place in October.
Labour had defended the appointment as "perfectly appropriate" but it is understood Mr Corfield has now moved to an unpaid advisory role.
Shadow Treasury minister Laura Trott previously called the appointment a "clear conflict" and urged the prime minister's independent ethics adviser to investigate.
Civil service sources told the BBC "hundreds" of officials are appointed through the post-election "exceptions" process - where time-limited jobs can be filled outside of usual recruitment processes.
They also said Mr Corfield's appointment was signed off by the independent Civil Service Commission.
BBC News has asked the government to comment on Mr Corfield's contract status.
Source: BBC