Camden council leader Sagal Abdi-Wali held the north London seat for Labour but the party's majority was reduced from 11,572 to 3,364.
The by-election, which was triggered after former Prime Minister Sir Keir Starmer stood down as an MP, had been seen as a key test of whether Labour under Andy Burnham can win over voters who have switched to the Greens.
Polanski insisted he would stay on as leader despite the result.
He has also come under pressure over his response to a motion passed by members declaring Zionism a form of racism.
Labour's Abdi-Wali won with 12,292 votes (45%), followed by Polanski with 8,928 (33%).
Ewan Cameron of the Conservatives came a distant third with 1,922 votes.
The inner London constituency has been Labour since it was created in 1983, with the party taking at least 40% of the vote at every general election.
In her victory speech at the count in Camden, Abdi-Wali, who came to the UK as a child refugee from Somalia, said her victory was a testament to the success of diversity in the borough.
"Whether you voted for me or not my commitment to this community runs deep, she said.
"I hope this moment also says something about who gets to belong, for people who look like me, I hope they see themselves reflected in this place and know that their path too belongs here. This is Camden, and this is Britain."
She added: "That is what this victory represents. Winning not by making people afraid of each other, but by bringing people together."
The by-election came after Polanski faced criticism for his response to a motion passed by members declaring Zionism a form of racism.
Earlier, former party leader Caroline Lucas told the BBC Polanski had failed to show leadership over the motion.
Speaking after polls closed but before the result of the by-election was announced, Lucas criticised Polanski for not taking part in the debate or voting on the motion at the Green Party's annual conference in Brighton on Sunday, describing the motion as "dangerous" and "divisive".
Asked if Polanski should stand down, she said: "I think let's just see what happens in the coming days and weeks. Obviously he has been completely taken up with this by-election.
"I want him to speak now about what's happened, I want him to show the leadership that was frankly missing on Sunday.
"I think the jury is out about whether he will do that - but I very much hope he will."
The motion defined Zionism as an ideology supporting "an ethnonationalist Jewish state" and backs a "single democratic Palestinian state in all of historic Palestine", which includes modern-day Israel.
Polanski was not present for the vote on the motion, after leaving the conference early to campaign in Holborn and St Pancras.