Opera Resyncing: Future Improvements

Over the weekend, there was a period of unresponsiveness from the Opera network. This was caused by some nodes going offline simultaneously, and when restarting they were unable to resync with the network automatically; resyncing needed to be initiated manually.

Since the rewards unlock, there’s been a high level of smart contract interactions. This overloaded a few nodes’ servers and caused them to go offline. As a high level of validating power was affected simultaneously (over ⅓), it then became impossible for the other validators to reach consensus and confirm new blocks until these nodes had resynced and got back up-to-date.

We then paused the PWA Wallet (not the chain as has been circulating – it’s not possible to pause the chain) to limit the number of new transactions being propagated; the PWA Wallet is the most widely used, so this was the most effective way to do this. We did not take this decision lightly – several nodes were close to being pruned based on their reported offline time, so it was critical that they were able to resync as soon as possible. Anyone needing access to funds was still able to interact with the chain via MEW, CLI, or FantomScan – nobody was categorically locked out.

What we’re doing

• Investigating improvements in the efficiency of smart contracts, especially the SFC, building on our research in the area to reduce the server overhead required for interactions and therefore minimize the chance of nodes going offline.

 

• We have already upgraded server specifications for all Foundation nodes.

 

• Investigating methods to increase accessibility for validator nodes, so more members of the community are able to contribute to validating the network and decentralization of validating power.

 

• Committing to an increase in offline time required to be pruned (from 24 hours currently to 72 hours) in the next possible SFC upgrade, in order to give validators more peace of mind.

 

• Investigating methods to reduce the tendency for new delegators to choose large validators without effort to research them. If large validators are actively chosen after due diligence, we don’t see an issue. However, those that are choosing a validator based on the first to appear in a list cause a positive feedback loop to occur.
• Increasing the urgency of allowing delegators to diversify between different validators within the same wallet, and redelegate to different validators instantly. This will reduce friction and inconvenience for delegators to split their delegation among different validators, and should encourage more decentralization of validating power. 

What validators can do

• Consider upgrading your validator nodes to a higher specification, for example, EC2 m5.xlarge.

As all developer resources were moved to addressing this issue and formulating steps to strengthen the network, we’ll be pushing the release of Fantom Finance back by two weeks. While we understand this will be disappointing for the community, as it is for us, it’s critical that we can make progress on the above plan before adding more smart contract throughput and complexity to the ecosystem, which increases risk. We’re committed to a stable, successful launch that fully mitigates these risks and therefore have taken the decision to action a number of the points laid out in this article first.

 

Finally, we’d like to thank the community for being patient during this time. Our efforts have been highly focused on resolving the issue as quickly as possible, as well as the plans outlined above, to push the network forward and grow stronger.