# Naemon cluster configuration and environment scaling

## Overview

Scaling a Naemon environment means distributing monitoring work across multiple
Naemon instances instead of running every check on a single server. Energy
Monitor uses Merlin to exchange monitoring data between Naemon processes and to
support redundancy and load distribution.

The topology should be selected according to the required outcome:

- **Peer ↔ peer** provides redundancy. Both nodes have equivalent roles,
  exchange monitoring state, and can share responsibility for the same
  configuration.
- **Master ↔ poller** provides horizontal scaling. The master manages the
  monitoring configuration, while the poller executes checks for assigned host
  groups and returns the results.
- **Master + redundant master + poller** combines high availability with
  workload distribution. Two peer nodes act as redundant masters, while a
  poller handles checks assigned to it.

Before starting, make sure that every node can resolve the other nodes by the
hostnames used in the Merlin configuration and that SSH connectivity is
available. Run the following command at any stage to inspect cluster status:

```bash
mon node status
```

> **Warning:** The `asmonitor mon oconf push <node>` command overwrites the
> existing object configuration on the destination node. Back up any local
> configuration that must be retained before running it.

## Peer ↔ peer

### Relationship description

Peer nodes are equal members of the cluster. They exchange monitoring events
and state information and provide redundancy for one another. A peer topology
is useful when monitoring must remain available after one node fails. The peers
must use a consistent object configuration; the procedure below copies the
configuration from `peer01` to `peer02`.

The example uses these nodes:

| Node | IP address | Role |
|---|---|---|
| `peer01` | `192.0.2.11` | Peer |
| `peer02` | `192.0.2.12` | Peer |

> **Note:** The node names must resolve to their corresponding IP addresses.
> Add the entries shown below to `/etc/hosts` only when the names are not
> resolved by DNS. Alternatively, specify a literal IP address with the Merlin
> `address` parameter while retaining a descriptive logical node name.

### Configure `peer01`

1. Add `peer02` to `/etc/hosts` if required:

   ```text
   192.0.2.12 peer02
   ```

2. Add `peer02` to the Merlin node configuration:

   ```bash
   mon node add peer02 type=peer address=192.0.2.12
   ```

3. Push the SSH keys used by the administrative and `asmonitor` accounts:

   ```bash
   mon sshkey push --type=peer
   asmonitor mon sshkey push --type=peer
   ```

4. Restart the monitoring services:

   ```bash
   mon restart
   ```

### Configure `peer02`

1. Add `peer01` to `/etc/hosts` if required:

   ```text
   192.0.2.11 peer01
   ```

2. Add `peer01` to the Merlin node configuration:

   ```bash
   mon node add peer01 type=peer address=192.0.2.11
   ```

3. Push the SSH keys used by the administrative and `asmonitor` accounts:

   ```bash
   mon sshkey push --type=peer
   asmonitor mon sshkey push --type=peer
   ```

4. Restart the monitoring services:

   ```bash
   mon restart
   ```

Verify the status on both nodes:

```bash
mon node status
```

![Peer 1 cluster status](/media/05_00_31_01_peer1.png)
![Peer 2 cluster status](/media/05_00_31_02_peer2.png)

### Synchronize the object configuration

To make both nodes operate on the same monitoring configuration, push the
configuration from one peer to the other. For example, run the following on
`peer01` to replace the configuration on `peer02`:

```bash
asmonitor mon oconf push peer02
```

Restart all nodes after the configuration has been transferred:

```bash
mon node ctrl --self -- mon restart
```

## Master ↔ poller

### Relationship description

In a master-poller topology, the master is the central configuration and
coordination node. A poller is responsible for executing checks for one or more
assigned host groups and sending their results back to the master. This reduces
the check-execution load on the master and makes it possible to monitor more
hosts or remote network segments.

The example uses these nodes:

| Node | IP address | Role |
|---|---|---|
| `master01` | `192.0.2.11` | Master |
| `poller01` | `192.0.2.21` | Poller |

> **Note:** The node names must resolve to their corresponding IP addresses.
> Add the entries shown below to `/etc/hosts` only when the names are not
> resolved by DNS. Alternatively, specify a literal IP address with the Merlin
> `address` parameter while retaining a descriptive logical node name.

### Configure name resolution

On `poller01`, add the master:

```text
192.0.2.11 master01
```

On `master01`, add the poller:

```text
192.0.2.21 poller01
```

### Configure the relationship

Run the following steps on `master01` unless stated otherwise.

1. Confirm that the host group to be assigned to the poller exists:

   ```bash
   mon query ls hostgroups -c members name="$HOSTGROUP_NAME"
   ```

2. Add the poller and assign the host group to it:

   ```bash
   mon node add poller01 type=poller hostgroup="$HOSTGROUP_NAME" takeover=no address=192.0.2.21
   ```

3. Push the required SSH keys to the poller:

   ```bash
   mon sshkey push poller01
   asmonitor mon sshkey push poller01
   ```

4. Add `master01` to the poller's node configuration:

   ```bash
   mon node ctrl poller01 mon node add master01 type=master connect=no address=192.0.2.11
   ```

5. Restart Naemon on the master:

   ```bash
   systemctl restart naemon
   ```

6. Push the object configuration to the poller:

   ```bash
   asmonitor mon oconf push poller01
   ```

7. Restart the poller:

   ```bash
   mon node ctrl poller01 mon restart
   ```

Verify the status on both instances:

```bash
mon node status
```

![Master cluster status](/media/05_00_31_03_master1.png)
![Poller cluster status](/media/05_00_31_04_poller1.png)

## Master + redundant master + poller

### Environment description

This topology combines the two preceding models. `master01` and `master02` form
a peer cluster and therefore provide redundant master functionality. Both masters
know about `poller01`, while the poller knows about both masters. The poller
executes checks for the assigned host group; the peer pair exchanges monitoring
state and keeps the central monitoring tier available if one master fails.

The example uses these nodes:

| Node | IP address | Role |
|---|---|---|
| `master01` | `192.0.2.11` | Master and peer of `master02` |
| `master02` | `192.0.2.12` | Redundant master and peer of `master01` |
| `poller01` | `192.0.2.21` | Poller |

> **Note:** The node names must resolve to their corresponding IP addresses.
> Add the entries shown below to `/etc/hosts` only when the names are not
> resolved by DNS. Alternatively, specify a literal IP address with the Merlin
> `address` parameter while retaining a descriptive logical node name.

First, configure `master01` and `master02` as peers by following the procedure
in [Peer ↔ peer](#peer--peer), using the master hostnames shown above.

### Configure name resolution

On `master01`, add:

```text
192.0.2.12 master02
192.0.2.21 poller01
```

On `master02`, add:

```text
192.0.2.11 master01
192.0.2.21 poller01
```

On `poller01`, add:

```text
192.0.2.11 master01
192.0.2.12 master02
```

### Connect `master01` to the poller

Run the following commands on `master01`:

1. Confirm that the host group exists:

   ```bash
   mon query ls hostgroups -c members name="$HOSTGROUP_NAME"
   ```

2. Add the poller and assign the host group to it:

   ```bash
   mon node add poller01 type=poller hostgroup="$HOSTGROUP_NAME" takeover=no address=192.0.2.21
   ```

3. Push the SSH keys:

   ```bash
   mon sshkey push poller01
   asmonitor mon sshkey push poller01
   ```

4. Register `master01` on the poller:

   ```bash
   mon node ctrl poller01 mon node add master01 type=master connect=no address=192.0.2.11
   ```

5. Restart Naemon on `master01`:

   ```bash
   systemctl restart naemon
   ```

6. Push the object configuration to the poller:

   ```bash
   asmonitor mon oconf push poller01
   ```

7. Restart the poller:

   ```bash
   mon node ctrl poller01 mon restart
   ```

### Connect `master02` to the poller

Run the following commands on `master02`:

1. Confirm that the host group exists:

   ```bash
   mon query ls hostgroups -c members name="$HOSTGROUP_NAME"
   ```

2. Add the poller and assign the host group to it:

   ```bash
   mon node add poller01 type=poller hostgroup="$HOSTGROUP_NAME" takeover=no address=192.0.2.21
   ```

3. Push the SSH keys:

   ```bash
   mon sshkey push poller01
   asmonitor mon sshkey push poller01
   ```

4. Register `master02` on the poller:

   ```bash
   mon node ctrl poller01 mon node add master02 type=master connect=no address=192.0.2.12
   ```

5. Restart Naemon on `master02`:

   ```bash
   systemctl restart naemon
   ```

6. Push the object configuration to the poller:

   ```bash
   asmonitor mon oconf push poller01
   ```

7. Restart the poller:

   ```bash
   mon node ctrl poller01 mon restart
   ```

### Complete the poller configuration

On `poller01`, push the SSH keys to both masters:

```bash
mon sshkey push --type=master
asmonitor mon sshkey push --type=master
```

Finally, run `mon node status` on all three nodes and confirm that both master
connections and the poller connection are active.

- `master01`:

  ![Master 1 status in the redundant master topology](/media/05_00_31_05_peer_master1.png)

- `master02`:

  ![Master 2 status in the redundant master topology](/media/05_00_31_06_peer_master2.png)

- `poller01`:

  ![Poller status in the redundant master topology](/media/05_00_31_07_poller2.png)