APT package reference

Topic Progress:

8.4. APT Package Reference: Digging Deeper into the Debian Package System

Now it is time to dive really deep into Debian and Kali's package system. At this point, we are going to move beyond tools and syntax and focus more on the nuts and bolts of the packaging system. This behind-the-scenes view will help you understand how APT works at its foundation and will give you insight into how to seriously streamline and customize your Kali system. You may not necessarily memorize all the material in this section, but the walk-through and reference material will serve you well as you grow in your mastery of the Kali Linux system.

So far, you have interacted with APT's package data through the various tools designed to interface with it. Next, we will dig deeper and take a look inside the packages and look at the internal meta-information (or information about other information) used by the package management tools.

This combination of a file archive and of meta-information is directly visible in the structure of a .deb file, which is simply an ar archive, concatenating three files:

$ ar t /var/cache/apt/archives/apt_1.4~beta1_amd64.deb

The debian-binary file contains a single version number describing the format of the archive:

$ ar p /var/cache/apt/archives/apt_1.4~beta1_amd64.deb debian-binary

The control.tar.gz archive contains meta-information:

$ ar p /var/cache/apt/archives/apt_1.4~beta1_amd64.deb control.tar.gz | tar -tzf -

And finally, the data.tar.xz archive (the compression format might vary) contains the actual files to be installed on the file system:

$ ar p /var/cache/apt/archives/apt_1.4~beta1_amd64.deb data.tar.xz | tar -tJf -

Note that in this example, you are viewing a .deb package in APT's archive cache and that your archive may contain files with different version numbers than what is shown.

In this section, we will introduce this meta-information contained in each package and show you how to leverage it.

8.4.1. The control File

We will begin by looking at the control file, which is contained in the control.tar.gz archive. The control file contains the most vital information about the package. It uses a structure similar to email headers and can be viewed with the dpkg -I command. For example, the control file for apt looks like this:

$ dpkg -I apt_1.4~beta1_amd64.deb control
Package: apt
Version: 1.4~beta1
Architecture: amd64
Maintainer: APT Development Team 
Installed-Size: 3478
Depends: adduser, gpgv | gpgv2 | gpgv1, debian-archive-keyring, init-system-helpers (>= 1.18~), libapt-pkg5.0 (>= 1.3~rc2), libc6 (>= 2.15), libgcc1 (>= 1:3.0), libstdc++6 (>= 5.2)
Recommends: gnupg | gnupg2 | gnupg1
Suggests: apt-doc, aptitude | synaptic | wajig, dpkg-dev (>= 1.17.2), powermgmt-base, python-apt
Breaks: apt-utils (<< 1.3~exp2~)
Replaces: apt-utils (<< 1.3~exp2~)
Section: admin
Priority: important
Description: commandline package manager
 This package provides commandline tools for searching and
 managing as well as querying information about packages
 as a low-level access to all features of the libapt-pkg library.
 These include:
  * apt-get for retrieval of packages and information about them
    from authenticated sources and for installation, upgrade and
    removal of packages together with their dependencies
  * apt-cache for querying available information about installed
    as well as installable packages
  * apt-cdrom to use removable media as a source for packages
  * apt-config as an interface to the configuration settings
  * apt-key as an interface to manage authentication keys

In this section, we will walk you through the control file and explain the various fields. Each of these will give you a better understanding of the packaging system, give you more fine-tuned configuration control, and provide you with insight needed to troubleshoot problems that may occur. Dependencies: the Depends Field

The package dependencies are defined in the Depends field in the package header. This is a list of conditions to be met for the package to work correctly—this information is used by tools such as apt in order to install the required libraries, in appropriate versions fulfilling the dependencies of the package to be installed. For each dependency, you can restrict the range of versions that meet that condition. In other words, it is possible to express the fact that you need the package libc6 in a version equal to or greater than "2.15" (written "libc6 (>= 2.15)"). Version comparison operators are as follows:

  • <<: less than;
  • <=: less than or equal to;
  • =: equal to (note that "2.6.1" is not equal to "2.6.1-1");
  • >=: greater than or equal to;
  • >>: greater than.

In a list of conditions to be met, the comma serves as a separator, interpreted as a logical "AND." In conditions, the vertical bar ("|") expresses a logical "OR" (it is an inclusive "OR," not an exclusive "either/or"). Carrying greater priority than "AND," you can use it as many times as necessary. Thus, the dependency "(A OR B) AND C" is written A | B, C. In contrast, the expression "A OR (B AND C)" should be written as "(A OR B) AND (A OR C)", since the Depends field does not tolerate parentheses that change the order of priorities between the logical operators "OR" and "AND". It would thus be written A | B, A | C. See http://www.debian.org/doc/debian-policy/ch-relationships.html for more information.

The dependencies system is a good mechanism for guaranteeing the operation of a program but it has another use with metapackages. These are empty packages that only describe dependencies. They facilitate the installation of a consistent group of programs preselected by the metapackage maintainer; as such, apt install metapackage will automatically install all of these programs using the metapackage's dependencies. The gnome, kali-tools-wireless, and kali-linux-large packages are examples of metapackages. For more information on Kali's metapackages, see https://tools.kali.org/kali-metapackages Pre-Depends, a More Demanding Depends

Pre-dependencies, which are listed in the Pre-Depends field in the package headers, complete the normal dependencies; their syntax is identical. A normal dependency indicates that the package in question must be unpacked and configured before configuration of the package declaring the dependency. A pre-dependency stipulates that the package in question must be unpacked and configured before execution of the pre-installation script of the package declaring the pre-dependency, that is before its installation.

A pre-dependency is very demanding for apt because it adds a strict constraint on the ordering of the packages to install. As such, pre-dependencies are discouraged unless absolutely necessary. It is even recommended to consult other developers on before adding a pre-dependency as it is generally possible to find another solution as a work-around. Recommends, Suggests, and Enhances Fields

The Recommends and Suggests fields describe dependencies that are not compulsory. The recommended dependencies, the most important, considerably improve the functionality offered by the package but are not indispensable to its operation. The suggested dependencies, of secondary importance, indicate that certain packages may complement and increase their respective utility, but it is perfectly reasonable to install one without the others.

You should always install the recommended packages unless you know exactly why you do not need them. Conversely, it is not necessary to install suggested packages unless you know why you need them.

The Enhances field also describes a suggestion, but in a different context. It is indeed located in the suggested package, and not in the package that benefits from the suggestion. Its interest lies in that it is possible to add a suggestion without having to modify the package that is concerned. Thus, all add-ons, plug-ins, and other extensions of a program can then appear in the list of suggestions related to the software. Although it has existed for several years, this last field is still largely ignored by programs such as apt or synaptic. The original goal was to let a package like xul-ext-adblock-plus (a Firefox extension) declare Enhances: firefox, firefox-esr and thus appear in the list of suggested packages associated to firefox and firefox-esr. Conflicts: the Conflicts Field

The Conflicts field indicates when a package cannot be installed simultaneously with another. The most common reasons for this are that both packages include a file of the same name, provide the same service on the same transmission control protocol (TCP) port, or would hinder each other's operation.

If it triggers a conflict with an already installed package, dpkg will refuse to install a package, except if the new package specifies that it will replace the installed package, in which case dpkg will choose to replace the old package with the new one. APT always follows your instructions: if you choose to install a new package, it will automatically offer to uninstall the package that poses a problem. Incompatibilities: the Breaks Field

The Breaks field has an effect similar to that of the Conflicts field, but with a special meaning. It signals that the installation of a package will break another package (or particular versions of it). In general, this incompatibility between two packages is transitory and the Breaks relationship specifically refers to the incompatible versions.

When a package breaks an already installed package, dpkg will refuse to install it, and apt will try to resolve the problem by updating the package that would be broken to a newer version (which is assumed to be fixed and, thus, compatible again).

This type of situation may occur in the case of updates without backwards compatibility: this is the case if the new version no longer functions with the older version and causes a malfunction in another program without making special provisions. The Breaks field helps prevent these types of problems. Provided Items: the Provides Field

This field introduces the very interesting concept of a virtual package. It has many roles, but two are of particular importance. The first role consists in using a virtual package to associate a generic service with it (the package provides the service). The second indicates that a package completely replaces another and that for this purpose, it can also satisfy the dependencies that the other would satisfy. It is thus possible to create a substitution package without having to use the same package name.

Meta-Package and Virtual Package

It is essential to clearly distinguish meta-packages from virtual packages. The former are real packages (including real .deb files), whose only purpose is to express dependencies.

Virtual packages, however, do not exist physically; they are only a means of identifying real packages based on common, logical criteria (for example, service provided, or compatibility with a standard program or a pre-existing package). Providing a Service

Let's discuss the first case in greater detail with an example: all mail servers, such as postfix or sendmail are said to provide the mail-transport-agent virtual package. Thus, any package that needs this service to be functional (e.g. a mailing list manager, such as smartlist or sympa) simply states in its dependencies that it requires a mail-transport-agent instead of specifying a large yet incomplete list of possible solutions. Furthermore, it is useless to install two mail servers on the same machine, which is why each of these packages declares a conflict with the mail-transport-agent virtual package. A conflict between a package and itself is ignored by the system, but this technique will prohibit the installation of two mail servers side by side. Interchangeability with Another Package

The Provides field is also interesting when the content of a package is included in a larger package. For example, the libdigest-md5-perl Perl module was an optional module in Perl 5.6, and has been integrated as standard in Perl 5.8. As such, the package perl has since version 5.8 declared Provides: libdigest-md5-perl so that the dependencies on this package are met if the system has Perl 5.8 (or newer). The libdigest-md5-perl package itself was deleted, since it no longer had any purpose when old Perl versions were removed.

Use of a Provides Field in Order to Not Break Dependencies
Figure 8.3. Use of a Provides Field in Order to Not Break Dependencies

This feature is very useful, since it is never possible to anticipate the vagaries of development and it is necessary to be able to adjust to renaming, and other automatic replacement, of obsolete software. Replacing Files: The Replaces Field

The Replaces field indicates that the package contains files that are also present in another package, but that the package is legitimately entitled to replace them. Without this specification, dpkg fails, stating that it cannot overwrite the files of another package (technically, it is possible to force it to do so with the --force-overwrite option, but that is not considered standard operation). This allows identification of potential problems and requires the maintainer to study the matter prior to choosing whether to add such a field.

The use of this field is justified when package names change or when a package is included in another. This also happens when the maintainer decides to distribute files differently among various binary packages produced from the same source package: a replaced file no longer belongs to the old package, but only to the new one.

If all of the files in an installed package have been replaced, the package is considered to be removed. Finally, this field also encourages dpkg to remove the replaced package where there is a conflict.

8.4.2. Configuration Scripts

In addition to the control file, the control.tar.gz archive for each Debian package may contain a number of scripts (postinst, postrm, preinst, prerm) called by dpkg at different stages in the processing of a package. We can use dpkg -I to show these files as they reside in a .deb package archive:

$ dpkg -I /var/cache/apt/archives/zsh_5.3-1_amd64.deb | head
 new debian package, version 2.0.
 size 814486 bytes: control archive=2557 bytes.
     838 bytes,    20 lines      control
    3327 bytes,    43 lines      md5sums
     969 bytes,    41 lines   *  postinst             #!/bin/sh
     348 bytes,    20 lines   *  postrm               #!/bin/sh
     175 bytes,     5 lines   *  preinst              #!/bin/sh
     175 bytes,     5 lines   *  prerm                #!/bin/sh
 Package: zsh
 Version: 5.3-1
$ dpkg -I zsh_5.3-1_amd64.deb preinst
set -e
# Automatically added by dh_installdeb
dpkg-maintscript-helper symlink_to_dir /usr/share/doc/zsh zsh-common 5.0.7-3 -- "$@"
# End automatically added section

The Debian Policy describes each of these files in detail, specifying the scripts called and the arguments they receive. These sequences may be complicated, since if one of the scripts fails, dpkg will try to return to a satisfactory state by canceling the installation or removal in progress (insofar as it is possible).

The dpkg Database

You can traverse the dpkg database on the filesystem at /var/lib/dpkg/. This directory contains a running record of all the packages that have been installed on the system. All of the configuration scripts for installed packages are stored in the /var/lib/dpkg/info/ directory, in the form of a file prefixed with the package's name:

$ ls /var/lib/dpkg/info/zsh.*

This directory also includes a file with the .list extension for each package, containing the list of files that belong to that package:

$ head /var/lib/dpkg/info/zsh.list

The /var/lib/dpkg/status file contains a series of data blocks (in the format of the famous mail headers request for comment, RFC 2822) describing the status of each package. The information from the control file of the installed packages is also replicated there.

$ more /var/lib/dpkg/status
Package: gnome-characters
Status: install ok installed
Priority: optional
Section: gnome
Installed-Size: 1785
Maintainer: Debian GNOME Maintainers 
Architecture: amd64
Version: 3.20.1-1

Let's discuss the configuration files and see how they interact. In general, the preinst script is executed prior to installation of the package, while the postinst follows it. Likewise, prerm is invoked before removal of a package and postrm afterwards. An update of a package is equivalent to removal of the previous version and installation of the new one. It is not possible to describe in detail all the possible scenarios here but we will discuss the most common two: an installation/update and a removal.

These sequences can be quite confusing, but a visual representation may help. Manoj Srivastava made some diagrams explaining how the configuration scripts are called by dpkgSimilar diagrams have also been developed by the Debian Women project ; they are a bit simpler to understand, but less complete.

Caution: Symbolic Names of the Scripts

The sequences described in this section call configuration scripts by specific names, such as old-prerm or new-postinst. They are, respectively, the prerm script contained in the old version of the package (installed before the update) and the postinst script contained in the new version (installed by the update). Installation and Upgrade Script Sequence

Here is what happens during an installation (or an update):

  1. 1. For an update, dpkg calls the old-prerm upgrade new-version.
  2. 2. Still for an update, dpkg then executes new-preinst upgrade old-version; for a first installation, it executes new-preinst install. It may add the old version in the last parameter if the package has already been installed and removed (but not purged, the configuration files having been retained).
  3. 3. The new package files are then unpacked. If a file already exists, it is replaced, but a backup copy is made and temporarily stored.
  4. 4. For an update, dpkg executes old-postrm upgrade new-version.
  5. 5. dpkg updates all of the internal data (file list, configuration scripts, etc.) and removes the backups of the replaced files. This is the point of no return: dpkg no longer has access to all of the elements necessary to return to the previous state.
  6. 6. dpkg will update the configuration files, prompting you to decide if it is unable to automatically manage this task. The details of this procedure are discussed in Section 8.4.3, "Checksums, Conffiles".
  7. 7. Finally, dpkg configures the package by executing new-postinst configure last-version-configured. Package Removal

Here is what happens during a package removal.

  1. 1. dpkg calls prerm remove.
  2. 2. dpkg removes all of the package's files, with the exception of the configuration files and configuration scripts.
  3. 3. dpkg executes postrm remove. All of the configuration scripts, except postrm, are removed. If you have not used the purge option, the process stops here.
  4. 4. For a complete purge of the package (command issued with dpkg --purge or dpkg -P), the configuration files are also deleted, as well as a certain number of copies (*.dpkg-tmp, *.dpkg-old, *.dpkg-new) and temporary files; dpkg then executes postrm purge.

In some cases, a package might use debconf to require configuration information from you: the four scripts detailed above are then complemented by a config script designed to acquire that information. During installation, this script defines in detail what questions debconf will ask. The responses are recorded in the debconf database for future reference. The script is generally executed by apt prior to installing packages one by one in order to group all the questions together at the beginning of the process. The pre- and post-installation scripts can then use this information to operate according to your wishes.

The debconf Tool

The debconf tool was created to resolve a recurring problem in Debian. All Debian packages unable to function without a minimum of configuration used to ask questions with calls to the echo and read commands in postinst shell scripts (and other similar scripts). This forced the installer to babysit large installations or updates in order to respond to various configuration queries as they arose. These manual interactions have now been almost entirely dispensed with, thanks to debconf.

The debconf tool has many interesting features: It requires the developer to specify user interaction; it allows localization of all the displayed strings (all translations are stored in the templates file describing the interactions); it provides different frontends for questions (text mode, graphical mode, non-interactive); and it allows creation of a central database of responses to share the same configuration with several computers. The most important feature is that all of the questions can be presented in a row, all at once, prior to starting a long installation or update process. Now, you can go about your business while the system handles the installation on its own, without having to stay there staring at the screen, waiting for questions to pop up.

8.4.3. Checksums, Conffiles

In addition to the maintainer scripts and control data already mentioned in the previous sections, the control.tar.gz archive of a Debian package may contain other interesting files:

# ar p /var/cache/apt/archives/bash_4.4-2_amd64.deb control.tar.gz | tar -tzf -

The first—md5sums—contains the MD5 checksums for all of the package's files. Its main advantage is that it allows dpkg --verify to check if these files have been modified since their installation. Note that when this file doesn't exist, dpkg will generate it dynamically at installation time (and store it in the dpkg database just like other control files).

conffiles lists package files that must be handled as configuration files. Configuration files can be modified by the administrator, and dpkg will try to preserve those changes during a package update.

In effect, in this situation, dpkg behaves as intelligently as possible: if the standard configuration file has not changed between the two versions, it does nothing. If, however, the file has changed, it will try to update this file. Two cases are possible: either the administrator has not touched this configuration file, in which case dpkg automatically installs the new version; or the file has been modified, in which case dpkg asks the administrator which version they wish to use (the old one with modifications, or the new one provided with the package). To assist in making this decision, dpkg offers to display a diff that shows the difference between the two versions. If you choose to retain the old version, the new one will be stored in the same location in a file with the .dpkg-dist suffix. If you choose the new version, the old one is retained in a file with the .dpkg-old suffix. Another available action consists of momentarily interrupting dpkg to edit the file and attempt to reinstate the relevant modifications (previously identified with diff).

dpkg handles configuration file updates, but, while doing so, regularly interrupts its work to ask for input from the administrator. This can be time consuming and inconvenient. Fortunately, you can instruct dpkg to respond to these prompts automatically. The --force-confold option retains the old version of the file, while --force-confnew will use the new version. These choices are respected, even if the file has not been changed by the administrator, which only rarely has the desired effect. Adding the --force-confdef option tells dpkg to decide by itself when possible (in other words, when the original configuration file has not been touched), and only uses --force-confnew or --force-confold for other cases.

These options apply to dpkg, but most of the time the administrator will work directly with the aptitude or apt programs. It is, thus, necessary to know the syntax used to indicate the options to pass to the dpkg command (their command line interfaces are very similar).

apt -o DPkg::options::="--force-confdef" -o DPkg::options::="--force-confold" full-upgrade

These options can be stored directly in apt's configuration. To do so, simply write the following line in the /etc/apt/apt.conf.d/local file:

DPkg::options { "--force-confdef"; "--force-confold"; }

Including this option in the configuration file means that it will also be used in a graphical interface such as aptitude.

Conversely, you can also force dpkg to ask configuration file questions. The --force-confask option instructs dpkg to display the questions about the configuration files, even in cases where they would not normally be necessary. Thus, when reinstalling a package with this option, dpkg will ask the questions again for all of the configuration files modified by the administrator. This is very convenient, especially for reinstalling the original configuration file if it has been deleted and no other copy is available: a normal re-installation won't work, because dpkg considers removal as a form of legitimate modification, and, thus, doesn't install the desired configuration file.