Leveraging Configuration Management

Topic Progress:

10.2. Leveraging Configuration Management

With the ability to install Kali on multiple computers very quickly, you will need some help in managing those machines post-installation. You can leverage configuration management tools to manage machines or configure replacement computers to any desired state.

Kali Linux contains many popular configuration management tools that you might want to use (ansible, chef, puppet, saltstack, etc.) but in this section, we will only cover SaltStack.

https://saltstack.com

10.2.1. Setting Up SaltStack

SaltStack is a centralized configuration management service: a salt master manages many salt minions. You should install the salt-master package on a server that is reachable by all the hosts that you want to manage and salt-minion on the hosts that you wish to manage. Each minion must be told where to find their master. Simply edit /etc/salt/minion and set the master key to the DNS name (or IP address) of the Salt master. Note that Salt uses YAML as format for its configuration files.

minion# vim /etc/salt/minion
minion# grep ^master /etc/salt/minion
master: 192.168.122.105

Each minion has a unique identifier stored in /etc/salt/minion_id, which defaults to its hostname. This minion identifier will be used in the configuration rules and as such, it is important to set it properly before the minion opens its connection to the master:

minion# echo kali-scratch >/etc/salt/minion_id
minion# systemctl enable salt-minion
minion# systemctl start salt-minion

When the salt-minion service is running, it will try to connect to the Salt master to exchange some cryptographic keys. On the master side, you have to accept the key that the minion is using to identify itself to let the connection proceed. Subsequent connections will be automatic:

master# systemctl enable salt-master
master# systemctl start salt-master
master# salt-key --list all
Accepted Keys:
Denied Keys:
Unaccepted Keys:
kali-scratch
Rejected Keys:
master# salt-key --accept kali-scratch
The following keys are going to be accepted:
Unaccepted Keys:
kali-scratch
Proceed? [n/Y] y
Key for minion kali-scratch accepted.

10.2.2. Executing Commands on Minions

As soon as minions are connected, you can execute commands on them from the master:

master# salt '*' test.ping
kali-scratch:
    True
kali-master:
    True

This command asks all minions (the '*' is a wildcard targeting all minions) to execute the ping function from the test execution module. This function returns a True value on success and is a simple way to ensure that the connection is working between the master and the various minions.

You can also target a specific minion by giving its identifier in the first parameter, or possibly a subset of minions by using a less-generic wildcard (such as '*-scratch' or 'kali-*'). Here is an example of how to execute an arbitrary shell command on the kali-scratch minion:

master# salt kali-scratch cmd.shell 'uptime; uname -a'
kali-scratch:
     05:25:48 up 44 min,  2 users,  load average: 0.00, 0.01, 0.05
    Linux kali-scratch 4.5.0-kali1-amd64 #1 SMP Debian 4.5.3-2kali1 (2020-05-09) x86_64 GNU/Linux

Salt Module Reference


There are many execution modules available for all sorts of use cases. We won't cover them all here, but the full list is available at https://docs.saltproject.io/en/latest/ref/modules/all/index.html. You can also obtain a description of all the execution modules and their available functions on a given minion with the salt minion sys.doc command. Running this command returns a very long list of functions, but you can filter the list by passing the name of a function or module prefixed by its parent module as a parameter:

master# salt kali-scratch sys.doc disk.usage
disk.usage:

    Return usage information for volumes mounted on this minion

One of the most useful modules is pkg, which is a package manager abstraction relying on the appropriate package manager for the system (apt-get for Debian and its derivatives like Kali). The pkg.refresh_db command updates the package list (that is, it performs apt-get update) while pkg.upgrade installs all the available updates (it performs apt-get upgrade or apt-get dist-upgrade, depending on the options received). The pkg.list_upgrades command lists the pending upgrade operations (that would be performed by the pkg.upgrade dist_upgrade=True command).

The service module is an abstraction of the service manager (systemd in the case of Kali), which lets you perform all the usual systemctl operations: service.enableservice.disableservice.startservice.stopservice.restart, and service.reload:

master# salt '*' service.enable ssh
kali-scratch:
    True
kali-master:
    True
master# salt '*' service.start ssh
kali-master:
    True
kali-scratch:
    True
master# salt '*' pkg.refresh_db
kali-scratch:
    ----------
kali-master:
    ----------
master# salt '*' pkg.upgrade dist_upgrade=True
kali-scratch:
    ----------
    changes:
        ----------
        base-files:
            ----------
            new:
                1:2020.2.1
            old:
                1:2020.2.0
[...]
        zaproxy:
            ----------
            new:
                2.5.0-0kali1
            old:
                2.4.3-0kali3
    comment:
    result:
        True

As a more concrete sample, you could easily set up a distributed Nmap scan with dnmap. After having installed the package on all the minions, you start the server in a first terminal:

server# salt '*' pkg.install dnmap
[...]
server# vim dnmap.txt
server# dnmap_server -f dnmap.txt

Assuming that the server IP is 1.2.3.4, you can next tell all minions to start a client process that connects to the server:

server# salt '*' cmd.run_bg template=jinja 'dnmap_client -s 1.2.3.4 -a {{ grains.id }}'
kali-scratch:
    ----------
    pid:
        17137
[...]

Note that the example uses cmd.run_bg to run the dnmap_client command in the background. Don't wait until it finishes, since it is a long-running process. Unfortunately, it doesn't kill itself properly when you interrupt the server so you might have to clean it up:

server# salt '*' cmd.shell 'pkill -f dnmap_client'

10.2.3. Salt States and Other Features

While remote execution is an important building block, it is only a tiny fraction of what SaltStack can do.

When setting up a new machine, you often run many commands and tests to determine the details of the system prior to installation. These operations can be formalized in re-usable configuration templates called state files. The operations described in state files can then be performed with a single state.apply salt command.

To save some time, you can rely on many ready-to-use state files that have been created by the community and which are distributed in "Salt formulas":

https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html

There are many other features that can be combined:

  • Scheduled execution of actions
  • Defining actions in response to events triggered by minions
  • Collecting data out of minions
  • Orchestration of a sequence of operations across multiple minions
  • Applying states over SSH without installing the salt-minion service
  • Provisioning systems on cloud infrastructures and bringing them under management
  • And more

SaltStack is quite vast and we can't possibly cover all the features here. In fact, there are books dedicated entirely to SaltStack and the online documentation is very extensive as well. Check it out if you want to learn more about its features:

https://docs.saltstack.com/en/latest/

If you manage a significant number of machines, you would be well advised to learn more about SaltStack as you can save a significant amount of time when deploying new machines and you will be able to maintain a coherent configuration throughout your network.

To give you a taste of what it looks like to work with state files, we will cover a simple example: how to enable the APT repository and install a package that you create in Section 10.3.3, "Creating a Package Repository for APT" and Section 10.3.2, "Creating Configuration Packages". You will also register a SSH key in root's account so that you can login remotely in case of problems.

By default, state files are stored in /srv/salt on the master; they are YAML structured files with a .sls extension. Just like for running commands, applying a state relies on many state modules:

https://docs.saltstack.com/en/latest/topics/tutorials/starting_states.html

https://docs.saltstack.com/en/latest/ref/states/all/

Your /srv/salt/offsec.sls file will call three of those modules:

offsec_repository:
  pkgrepo.managed:
    - name: deb http://pkgrepo.offsec.com offsec-internal main
    - file: /etc/apt/sources.list.d/offsec.list
    - key_url: salt://offsec-apt-key.asc
    - require_in:
      - pkg: offsec-defaults

offsec-defaults:
  pkg.installed

ssh_key_for_root:
  ssh_auth.present:
    - user: root
    - name: ssh-rsa AAAAB3NzaC1yc2...89C4N rhertzog@kali

The offsec_repository state relies on the pkgrepo state module. The example uses the managed function in that state module to register a package repository. With the key_url attribute, you let salt know that the (ASCII armored) GPG key required to verify the repository's signature can be fetched from /srv/salt/offsec-apt-key.asc on the salt master. The require_in attribute ensures that this state is processed before the offsec-defaults, since the latter needs the repository correctly configured to be able to install the package.

The offsec-defaults state installs the package of the same name. This shows that the name of the key is often an important value for states, although it can always be overridden with a name attribute (as done for the former state). For simple-cases like this one, this is both readable and concise.

The last state (ssh_key_for_root) adds the SSH key given in the name attribute to /root/.ssh/authorized_keys (the target user is set in the user attribute). Note that we have shortened the key for readability here, but you should put the full key in the name attribute.

This state file can next be applied to a given minion:

server# salt kali-scratch state.apply offsec
kali-scratch:
----------
          ID: offsec_repository
    Function: pkgrepo.managed
        Name: deb http://pkgrepo.offsec.com offsec-internal main
      Result: True
     Comment: Configured package repo 'deb http://pkgrepo.offsec.com offsec-internal main'
     Started: 06:00:15.767794
    Duration: 4707.35 ms
     Changes:
              ----------
              repo:
                  deb http://pkgrepo.offsec.com offsec-internal main
----------
          ID: offsec-defaults
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: offsec-defaults
     Started: 06:00:21.325184
    Duration: 19246.041 ms
     Changes:
              ----------
              offsec-defaults:
                  ----------
                  new:
                      1.0
                  old:
----------
          ID: ssh_key_for_root
    Function: ssh_auth.present
        Name: ssh-rsa AAAAB3NzaC1yc2...89C4N rhertzog@kali
      Result: True
     Comment: The authorized host key AAAAB3NzaC1yc2...89C4N for user root was added
     Started: 06:00:40.582539
    Duration: 62.103 ms
     Changes:
              ----------
              AAAAB3NzaC1yc2...89C4N:
                  New

Summary for kali-scratch
------------
Succeeded: 3 (changed=3)
Failed:    0
------------
Total states run:     3
Total run time:  24.015 s

It can also be permanently associated to the minion by recording it in the /srv/salt/top.sls file, which is used by the state.highstate command to apply all relevant states in a single pass:

server# cat /srv/salt/top.sls
base:
  kali-scratch:
    - offsec
server# salt kali-scratch state.highstate
kali-scratch:
----------
          ID: offsec_repository
    Function: pkgrepo.managed
        Name: deb http://pkgrepo.offsec.com offsec-internal main
      Result: True
     Comment: Package repo 'deb http://pkgrepo.offsec.com offsec-internal main' already configured
     Started: 06:06:20.650053
    Duration: 62.805 ms
     Changes:
----------
          ID: offsec-defaults
    Function: pkg.installed
      Result: True
     Comment: Package offsec-defaults is already installed
     Started: 06:06:21.436193
    Duration: 385.092 ms
     Changes:
----------
          ID: ssh_key_for_root
    Function: ssh_auth.present
        Name: ssh-rsa AAAAB3NzaC1yc2...89C4N rhertzog@kali
      Result: True
     Comment: The authorized host key AAAAB3NzaC1yc2...89C4N is already present for user root
     Started: 06:06:21.821811
    Duration: 1.936 ms
     Changes:

Summary for kali-scratch
------------
Succeeded: 3
Failed:    0
------------
Total states run:     3
Total run time: 449.833 ms