Graceful shutdown of an OnDiskStateMachine

[i512]

Using dragonboat v4.

Is there a way to exit from a StateMachine Update with an error without causing a panic? Is this something you would consider adding? It could be triggered by returning a special kind of error from the Update, for example.

As I understand, the assumption is that a state machine error always indicates a logic error, and we should not continue. I would like to argue that this is not the case for OnDiskStateMachines: in our project, we have multiple replicas per node host, and each replica's SM stores data on its own HDD. The HDDs can enter RO mode or fail unexpectedly. This is not a logic error but a partial hardware fault in this case. We would like to handle such cases gracefully by stopping the failed replica without affecting other replicas on this NodeHost.

We have to handle two scenarios of disk failures:

• RO->RW: FS enters RO mode, an op fixes it, and makes it RW again. In this case, the SM can continue applying updates after a long pause.
• Disk fails completely: Wait for a new replica to be created and become up to date, after this we can safely discard the failed replica.

The simplest way to handle these would be to just stop the replica without applying the last update.

[i512]

We can probably handle the first case by stalling in the Update until the disk becomes RW again. Stall could last a couple of hours. This is something you wouldn't expect an SM to do, so not sure if it will cause other problems.

[kevburnsjr]

Backward compatible proposal:

1. Add new property Recover func(error) to replica config in config.Config
2. If Update returns a non-nil error and Recover is non-nil on replica config, Dragonboat stops the replica and calls Recover(err) out of band rather than panicking

This would allow the recovery function to decide what to do without disrupting other replicas:

• If the disk is full: Reclaim disk space and restart the replica
• If the disk is failed: Cordon the disk, schedule a new replica elsewhere and send an alert
• If the state machine is corrupted: Delete the replica and create a new replica on the same host to pull a fresh snapshot

Adding this as a new optional config field ensures it won't change the behavior of existing applications.