Police said they detained the men - thought to be linked to the Youth Demand protest group - "within one minute of them entering the grounds". They remain in custody.
Officials from the prime minister's residence have confirmed that four males have been taken into custody on suspicion of aggravated trespassing.
Rishi Sunak was attending festivities in London to commemorate the Japanese state visit when the incident occurred at his constituency residence in Kirby Sigston, North Yorkshire.
A police statement said officers were "with the four men within one minute of them entering the grounds".
The arrests are connected to a protest by campaigners from Youth Demand. It describes itself as a group of young people who want "the Tories and the Labour Party commit to a two-way arms embargo on Israel, and to stop all new oil and gas licences".
A spokesperson for the group said three of those arrested were taking part in the demonstration, while the fourth person was an independent photographer.
A North Yorkshire Police spokesperson said: "They were detained at around 12.40pm before being escorted off the property and arrested on suspicion of aggravated trespass.
"The men, aged 52 from London, 43 from Bolton, 21 from Manchester, and 20 from Chichester, remain in police custody for questioning and enquiries are ongoing."