It also suggests that the Palestinians should be offered immediate recognition of statehood within provisional borders as an incentive to drop their UN bid.
Mr Abbas plans to submit a request to the UN General Assembly on 29 November.
Currently, the Palestine Liberation Organisation (PLO), the umbrella group which represents most Palestinian factions and conducts negotiations with Israel, only has "permanent observer" status at the UN.
Mr Abbas, who is chairman of the PLO and president of the Palestinian Authority, wants Palestine to be admitted as a non-member observer state based on the boundaries which existed before Israel occupied the West Bank, East Jerusalem and the Gaza Strip during the 1967 war.
'Extract heavy price'
However, Mr Lieberman was quoted last week by Israeli Channel 10 TV as saying he would ensure that the Palestinian Authority "collapses" if its unilateral UN bid went ahead.
He has also in the past described Mr Abbas, who is also known as Abu Mazen, as an
"obstacle that needs to be removed".
The position paper states that the "main goal of the State of Israel" is to deter the Palestinians from unilaterally seeking non-member observer state status at the UN, which should be seen as "crossing a red line that will require the harshest Israeli response".
"If deterrence efforts do not succeed, Israel must extract a heavy price from Abu Mazen," it says.
The reality of a Palestinian state recognised by the UN is a "unilateral step that will crush Israeli deterrence, completely undermine its credibility and make any future peace deal that could be acceptable to Israel impossible", it adds.
"Even though this would not be a simple step to take because Israel would have to pay the consequences, toppling Abu Mazen's regime is the only option in such a case."
No deadline would be set for negotiations over permanent borders and Israel would not freeze construction in the major Jewish settlement blocs of Ariel, Maale Adumim and Gush Etzion, the paper says.
Israeli foreign ministry spokesman Lior Ben Dor told the BBC that if President Abbas continued to pursue the UN bid, he would be in breach of the 1993 Oslo peace accords, under which the Palestinian Authority was established.
He said Israel was concerned that if Palestine became a UN non-member state, it could ask the International Criminal Court (ICC) to resolve disputes with Israel.
In that case, Mr Ben Dor said, Israel would "take unilateral steps to protect its interests".
He would not elaborate on the measures, but recent reports in the Israeli media suggested they might include halting the transfer of tax revenue to the Palestinian Authority and restricting the movement of Palestinian officials through the West Bank.
The chief Palestinian negotiator, Saeb Erekat, told the BBC: "We take Israel's threats seriously and we do not rule out any attempt on Israel's part to hurt the president."
The Israeli government is intent on persuading other countries to oppose the Palestinian move at the United Nations. But calls for the potential toppling of the Palestinian Authority President, Mahmoud Abbas, raise the inevitable question of who or what would replace him?
Chaos in the West Bank is hardly in Israel's interests. The Palestinian Authority's security forces, for example, have operated reasonably effectively to clamp down on militant groups.
For now, the Israeli threats should be seen as just that - an attempt to influence the debate and to either convince the Palestinians to back down or to thwart their UN bid.