/sys/storage/raft/autopilot

The /sys/storage/raft/autopilot endpoints are used to manage raft clusters using autopilot with Vault's Integrated Storage backend. Refer to the Integrated Storage Autopilot tutorial to learn how to manage raft clusters using autopilot.

Get cluster state

This endpoint is used to retrieve the raft cluster state. See the docs page for a description of the output.

Method	Path
GET	/sys/storage/raft/autopilot/state

Sample request

$ curl \
    --header "X-Vault-Token: ..." \
    http://127.0.0.1:8200/v1/sys/storage/raft/autopilot/state

Sample response

{
  "healthy": true,
  "failure_tolerance": 1,
  "servers": {
    "raft1": {
      "id": "raft1",
      "name": "raft1",
      "address": "127.0.0.1:8201",
      "node_status": "alive",
      "last_contact": "0s",
      "last_term": 3,
      "last_index": 459,
      "healthy": true,
      "stable_since": "2021-03-19T20:14:11.831678-04:00",
      "status": "leader",
      "meta": null
    },
    "raft2": {
      "id": "raft2",
      "name": "raft2",
      "address": "127.0.0.2:8201",
      "node_status": "alive",
      "last_contact": "516.49595ms",
      "last_term": 3,
      "last_index": 459,
      "healthy": true,
      "stable_since": "2021-03-19T20:14:19.831931-04:00",
      "status": "voter",
      "meta": null
    },
    "raft3": {
      "id": "raft3",
      "name": "raft3",
      "address": "127.0.0.3:8201",
      "node_status": "alive",
      "last_contact": "196.706591ms",
      "last_term": 3,
      "last_index": 459,
      "healthy": true,
      "stable_since": "2021-03-19T20:14:25.83565-04:00",
      "status": "voter",
      "meta": null
    }
  },
  "leader": "raft1",
  "voters": ["raft1", "raft2", "raft3"],
  "non_voters": null
}

Enterprise only

Vault Enterprise will include additional output in its API response to indicate the current state of redundancy zones, automated upgrade progress (if any), and optimistic failure tolerance.

Sample response (Enterprise)

{
  "failure_tolerance": 0,
  "healthy": true,
  "leader": "vault_1",
  "optimistic_failure_tolerance": 3,
  "redundancy_zones": {
    "a": {
      "servers": [
        "vault_1",
        "vault_2",
        "vault_5"
      ],
      "voters": [
        "vault_1"
      ],
      "failure_tolerance": 2
    },
    "b": {
      "servers": [
        "vault_3",
        "vault_4"
      ],
      "voters": [
        "vault_3"
      ],
      "failure_tolerance": 1
    }
  },
  "upgrade_info": {
    "other_version_non_voters": [
      "vault_2",
      "vault_4"
    ],
    "other_version_voters": [
      "vault_1",
      "vault_3"
    ],
    "redundancy_zones": {
      "a": {
        "target_version_non_voters": [
          "vault_5"
        ],
        "other_version_voters": [
          "vault_1"
        ],
        "other_version_non_voters": [
          "vault_2"
        ]
      },
      "b": {
        "other_version_voters": [
          "vault_3"
        ],
        "other_version_non_voters": [
          "vault_4"
        ]
      }
    },
    "status": "await-new-voters",
    "target_version": "1.12.0",
    "target_version_non_voters": [
      "vault_5"
    ]
  },
  "voters": [
    "vault_1",
    "vault_3"
  ]
}

Get configuration

This endpoint is used to get the configuration of the autopilot subsystem of Integrated Storage.

Method	Path
GET	/sys/storage/raft/autopilot/configuration

Sample request

$ curl \
    --header "X-Vault-Token: ..." \
    http://127.0.0.1:8200/v1/sys/storage/raft/autopilot/configuration

Sample response

{
  "cleanup_dead_servers": false,
  "dead_server_last_contact_threshold": "24h0m0s",
  "last_contact_threshold": "10s",
  "max_trailing_logs": 1000,
  "min_quorum": 0,
  "server_stabilization_time": "10s",
  "disable_upgrade_migration": true
}

Note that in the above sample response, disable_upgrade_migration is an Enterprise-only field.

Set configuration

This endpoint is used to modify the configuration of the autopilot subsystem of Integrated Storage.

Method	Path
POST	/sys/storage/raft/autopilot/configuration

Parameters

• cleanup_dead_servers (bool: false) - Controls whether to remove dead servers from the Raft peer list periodically or when a new server joins. This requires that min_quorum is also set.

• last_contact_threshold (string: "10s") - Limit on the amount of time a server can go without leader contact before being considered unhealthy.

• dead_server_last_contact_threshold (string: "24h") - Limit on the amount of time a server can go without leader contact before being considered failed. This takes effect only when cleanup_dead_servers is true. This can not be set to a value smaller than 1m.

• max_trailing_logs (int: 1000) - Amount of entries in the Raft Log that a server can be behind before being considered unhealthy.

• min_quorum (int: 3) - Minimum number of servers allowed in a cluster before autopilot can prune dead servers. This should at least be 3. Applicable only for voting nodes.

• server_stabilization_time (string: "10s") - Minimum amount of time a server must be in a stable, healthy state before it can be added to the cluster.

• disable_upgrade_migration (bool: false) - Disables automatically upgrading Vault using autopilot. (Enterprise-only)

Sample request

$ curl \
    --header "X-Vault-Token: ..." \
    --request POST \
    --data @payload.json \
    http://127.0.0.1:8200/v1/sys/storage/raft/autopilot/configuration

Sample payload

{
  "cleanup_dead_servers": true,
  "last_contact_threshold": "10s",
  "dead_server_last_contact_threshold": "24h",
  "max_trailing_logs": "1000",
  "min_quorum": "3",
  "server_stabilization_time": "10s",
  "disable_upgrade_migration": true
}

Note that in the above sample payload, disable_upgrade_migration is an Enterprise-only field.