In Israel if no one can form a majority then they hold a new election. It’s not working very well. They held elections in April 2019, then September 2019 after no one got a majority. The new election only made it harder for a majority coalition to form. Now they’re holding a third election in March.
In Ireland the major parties have a disincentive to avoid holding new elections: the Sinn Féin gains in this election were limited by the fact that they didn’t run many candidates. They will be able to run more candidates and thus gain more seats if another election is held soon. However, if no one can get a majority, they would have to hold another election.
How would votes of no confidence work with an approval vote-selected prime minister? If no PM candidate can win majority approval, then they may not be able to win a confidence vote, either.
Personally, I would have each MP make an official list the PM candidates whom they would support. The PM is whoever is included on the most lists at the end of some day before the opening of Parliament. The government/confidence and supply partners consist of those MPs who support the PM. The Leader of the Opposition is the candidate on the most lists that do not support the PM. The lists can be updated after a PM has been selected, and if another candidate for PM winds up getting more support, then at the end of the day, that candidate takes over, and a new government forms consisting of that PM’s supporters. (There may need to be some sort of waiting period to allow MPs who are unaware of the impending change to update their lists before this happens.) For early elections: each MP also records whether they would support holding new elections, and if a majority (or perhaps some sort of supermajority early in the session, because going to the polls every 6 months and getting the same outcome over and over like Israel is doing is silly) is recorded as in favor at the end of the day, then the matter is put to a floor vote at the beginning of the next day.