Upgrading Puppet Enterprise

Upgrade your PE installation as new versions become available.

Upgrade paths

These are the valid upgrade paths for PE.

If you're on version... Upgrade to... Notes
2021.6 (latest) You're up to date!
2019.8.z latest
2019.y 2019.8.z
2018.1.2 or later

2018.1.3 or later with disaster recovery

 2019.8.z You must have version 2018.1.2 or later in order to complete prerequisites for upgrade to 2019.8.z.

With disaster recovery enabled, you must have version 2018.1.3 in order to upgrade to 2019.8.z. Alternatively, you can forget and then recreate your replica after upgrade.

2018.1.0 or 2018.1.1 2018.1.z

Upgrade cautions

These are the major changes to PE since the last long-term support release, 2019.8. Review these recommendations and plan accordingly before upgrading to this version.

PostgreSQL 14 upgrade in PE 2021.6

PE 2021.6.0 upgrades pe-postgresql from version 11 to version 14. This upgrade involves a datastore migration that requires extra disk space (110% of the current PostgreSQL 11 datastore) and extra time to upgrade (roughly two to four minutes of additional time per 10GB of datastore size). The installer issues a warning and cancels the upgrade if there is insufficient space.

To check your PostgreSQL installation size and the size and number of bytes available for the partition, run facter -p pe_postgresql_info on the node that runs the pe-postgresql service (this is either your primary server or a separate node that manages your PostgreSQL database).

To speed the migration and optimize queries, clean up the PE-PostgreSQL database prior to upgrading by applying the pe_databases module to nodes running the pe-postgresql service. This module is installed with PE versions 2021.3 and later, but you must enable it by setting the puppet_enterprise::enable_database_maintenance parameter to true. For best results, apply the module at least one week prior to upgrade to allow the module's maintenance schedule enough time to clean all databases.

Optionally, after upgrading, you can remove packages and directories from older PostgreSQL versions by running puppet infrastructure run remove_old_postgresql_versions. If applicable, the installer prompts you to complete the cleanup.

CAUTION: Don't use CONCURRENTLY with PostgreSQL 14.0 through 14.3. If you do, make sure to REINDEX without using CONCURRENTLY.
Restriction: If you use a PostgreSQL instance not managed by PE, you must separately upgrade the instance to PostgreSQL 14. For more information, see Upgrade an unmanaged PostgreSQL installation.

Platforms removed in 2021.0

Several agent platforms that were previously deprecated have been removed in PE 2021.0.

Before upgrading to this version, remove the pe_repo::platform class for these operating systems from the PE Master node group in the console, and from your code and Hiera.

  • AIX 6.1
  • Enterprise Linux 4
  • Enterprise Linux 6, 7 s390x
  • Fedora 26, 27, 28, 29
  • Mac OS X 10.9, 10.12, 10.13
  • SUSE Linux Enterprise Server 11, 12 s390x

Puppet upgrade in 2021.0

PE 2021.0 includes a new major version of Puppet, with several changes that might impact your upgrade. For details, see Upgrading from Puppet 6 to Puppet 7.

Test modules before upgrade

To ensure that your modules work with the newest version of PE, update and test them with Puppet Development Kit (PDK) before upgrading.

Before you begin

If you are already using PDK, your modules should pass validation and unit tests with your currently installed version of PDK.

Update PDK with each new release to ensure compatability with new versions of PE.

  1. Download and install PDK. If you already have PDK installed, this updates PDK to its latest version. For detailed instructions and download links, see the installing instructions.
  2. If you have not previously used PDK with your modules, convert them to a PDK compatible format. This makes changes to your module to enable validation and unit testing with PDK. For important usage details, see the converting modules documentation.

    For example, from within the module directory, run:

    pdk convert
  3. If your modules are already compatible with PDK, update them to the latest module template. If you converted modules in step 2, you do not need to update the template. To learn more about updating, see the updating module templates documentation.

    For example, from within the module directory, run:

    pdk update
  4. Validate and run unit tests for each module, specifying the version of PE you are upgrading to. When specifying a PE version, be sure to specify at least the year and the release number, such as 2018.1. For information about module validations and testing, see the validating and testing modules documentation.

    For example, from within the module directory, run:

    pdk validate
pdk test unit
    The pdk test unit command verifies that testing dependencies and directories are present and runs the unit tests that you write. It does not create unit tests for your module.
  5. If your module fails validation or unit tests, make any necessary changes to your code.
Results

After you've verified that your modules work with the new PE version, you can continue with your upgrade.

Upgrade PE

Upgrade PE infrastructure components to get the latest features and fixes. Follow the upgrade instructions for your installation type to ensure you upgrade components in the correct order. Coordinate upgrades to ensure all infrastructure nodes are upgraded in a timely manner, because agent runs and replication fail on infrastructure nodes running a different agent version than the primary server.

Before you begin

Review the upgrade cautions for major changes to architecture and infrastructure components which might affect your upgrade.

Configure non-production environment for infrastructure nodes

If your infrastructure nodes are in an environment other than production, you must manually configure PE to use your chosen environment before you upgrade.

Important: You only need to follow these steps if your infrastructure nodes are in an environment that is not production.
  1. In pe.conf, set both of these parameters to the environment your infrastructure nodes are in:
    • pe_install::install::classification::pe_node_group_environment
    • puppet_enterprise::master::recover_configuration::pe_environment
  2. Run puppet-enterprise-installer -- --pe-environment <env>

Upgrade a standard installation

To upgrade a standard installation, run the PE installer on your primary server, and then upgrade any additional components.

Before you begin

Back up your PE installation.

If you're upgrading a replica, ensure you have a valid admin RBAC token.

In Hiera, pe.conf, or the console (in the PE Master node group), remove any agent_version parameters you set in the pe_repo class that matches your infrastructure nodes. This ensures the upgrade isn't blocked by attempting to download non-default agent versions for your infrastructure OS and architecture.

  1. Download the tarball for your operating system and architecture.
  2. Run tar -xf <TARBALL> to unpack the installation tarball.
    You need about 1 GB of space to untar the installer.
  3. From the installer directory on your primary server, run sudo ./puppet-enterprise-installer -s to start the installer, and then follow the CLI instructions to complete your server upgrade.
    • The -s flag tells the installer to skip database checks.
    • To specify a different pe.conf file than the existing file, use the -c flag, such as:
      sudo ./puppet-enterprise-installer -c <FULL_PATH_TO_pe.conf>

      This flag tells the installer to backup the previous pe.conf file to /etc/puppetlabs/enterprise/conf.d/<TIMESTAMP>.conf and create a new pe.conf file at /etc/puppetlabs/enterprise/conf.d/pe.conf.

  4. Upgrade these additional PE infrastructure components:
    • Agents
    • PE client tools: On unmanaged nodes, you must re-install the client tools version that matches the PE version you upgraded to. On managed nodes and infrastructure nodes, client tools are automatically updated when you upgrade PE.
  5. In disaster recovery installations, upgrade your replica.
    The replica is temporarily unavailable to serve as backup during this step, so time upgrading your replica to minimize risk.
    1. On your primary server logged in as root, run: 
      sudo puppet infrastructure upgrade replica <REPLICA_FQDN>
      If you want to specify an authentication token other than the default, run:
      sudo puppet infrastructure upgrade replica <REPLICA_FQDN> --token-file <PATH_TO_TOKEN>
    2. After the replica upgrade successfully completes, verify that primary and replica services are operational. On your primary server, run: 
      sudo /opt/puppetlabs/bin/puppet-infra status
    3. If your replica reports errors, reinitialize the replica. On your replica, run: 
      sudo /opt/puppetlabs/bin/puppet-infra reinitialize replica -y
  6. Optional: Remove old PE packages from all infrastructure nodes. On your primary server, run: puppet infrastructure run remove_old_pe_packages
    All packages older than the current version are removed by default. To remove specific versions, append pe_version=<VERSION_NUMBER_OR_SHA> to the command.

Upgrade a large installation

To upgrade a large installation, run the PE installer on your primary server, and then upgrade compilers and any additional components.

Before you begin

Back up your PE installation.

Ensure you have a valid admin RBAC token in order to upgrade compilers or a replica.

In Hiera, pe.conf, or the console (in the PE Master node group), remove any agent_version parameters you set in the pe_repo class that matches your infrastructure nodes. This ensures the upgrade isn't blocked by attempting to download non-default agent versions for your infrastructure OS and architecture.

  1. Download the tarball for your operating system and architecture.
  2. Run tar -xf <TARBALL> to unpack the installation tarball.
    You need about 1 GB of space to untar the installer.
  3. From the installer directory on your primary server, run sudo ./puppet-enterprise-installer -s to start the installer, and then follow the CLI instructions to complete your server upgrade.
    • The -s flag tells the installer to skip database checks.
    • To specify a different pe.conf file than the existing file, use the -c flag, such as:
      sudo ./puppet-enterprise-installer -c <FULL_PATH_TO_pe.conf>

      This flag tells the installer to backup the previous pe.conf file to /etc/puppetlabs/enterprise/conf.d/<TIMESTAMP>.conf and create a new pe.conf file at /etc/puppetlabs/enterprise/conf.d/pe.conf.

  4. To upgrade compilers, log in to your primary server as root and run one of these commands:
    • To upgrade specific compilers, run:
      sudo puppet infrastructure upgrade compiler <COMPILER_FQDN-1>,<COMPILER_FQDN-2>
    • To upgrade all compilers simultaneously, run:
      sudo puppet infrastructure upgrade compiler --all
    • To specify an authentication token location other than the default location, include --token-file <PATH_TO_TOKEN> in the command, such as:
      sudo puppet infrastructure upgrade compiler <COMPILER_FQDN> --token-file <PATH_TO_TOKEN>
  5. Upgrade these additional PE infrastructure components:
    • Agents
    • PE client tools: On unmanaged nodes, you must re-install the client tools version that matches the PE version you upgraded to. On managed nodes and infrastructure nodes, client tools are automatically updated when you upgrade PE.
  6. In disaster recovery installations, upgrade your replica.
    The replica is temporarily unavailable to serve as backup during this step, so time upgrading your replica to minimize risk.
    1. On your primary server logged in as root, run: 
      sudo puppet infrastructure upgrade replica <REPLICA_FQDN>
      If you want to specify an authentication token other than the default, run:
      sudo puppet infrastructure upgrade replica <REPLICA_FQDN> --token-file <PATH_TO_TOKEN>
    2. After the replica upgrade successfully completes, verify that primary and replica services are operational. On your primary server, run: 
      sudo /opt/puppetlabs/bin/puppet-infra status
    3. If your replica reports errors, reinitialize the replica. On your replica, run: 
      sudo /opt/puppetlabs/bin/puppet-infra reinitialize replica -y
  7. Optional: Remove old PE packages from all infrastructure nodes. On your primary server, run: puppet infrastructure run remove_old_pe_packages
    All packages older than the current version are removed by default. To remove specific versions, append pe_version=<VERSION_NUMBER_OR_SHA> to the command.
What to do next

Optionally convert legacy compilers to the new style compiler running the PuppetDB service.

Upgrade an extra-large installation

For help upgrading an extra-large installation, reach out to your technical account manager.

Upgrade a standalone PE-PostgreSQL installation

To upgrade a large installation with standalone PE-PostgreSQL, run the PE installer first on your PE-PostgreSQL node, then on your primary server, and then upgrade any additional components.

Before you begin

Back up your PE installation.

Ensure you have a valid admin RBAC token in order to upgrade compilers.

In Hiera, pe.conf, or the console (in the PE Master node group), remove any agent_version parameters you set in the pe_repo class that matches your infrastructure nodes. This ensures the upgrade isn't blocked by attempting to download non-default agent versions for your infrastructure OS and architecture.

  1. Download the tarball for your operating system and architecture.
  2. Run tar -xf <TARBALL> to unpack the installation tarball.
    You need about 1 GB of space to untar the installer.
  3. Upgrade your PostgreSQL node.
    1. Ensure that the pe.conf file on your PostgreSQL node is up to date by running puppet infrastructure recover_configuration on your primary server, and then copying /etc/puppetlabs/enterprise/conf.d to the PostgreSQL node.
    2. Copy the installation tarball to the PostgreSQL node, and from the installer directory, run the installer: sudo ./puppet-enterprise-installer
  4. From the installer directory on your primary server, run sudo ./puppet-enterprise-installer -s to start the installer, and then follow the CLI instructions to complete your server upgrade.
    • The -s flag tells the installer to skip database checks.
    • To specify a different pe.conf file than the existing file, use the -c flag, such as:
      sudo ./puppet-enterprise-installer -c <FULL_PATH_TO_pe.conf>

      This flag tells the installer to backup the previous pe.conf file to /etc/puppetlabs/enterprise/conf.d/<TIMESTAMP>.conf and create a new pe.conf file at /etc/puppetlabs/enterprise/conf.d/pe.conf.

  5. To upgrade compilers, log in to your primary server as root and run one of these commands:
    • To upgrade specific compilers, run:
      sudo puppet infrastructure upgrade compiler <COMPILER_FQDN-1>,<COMPILER_FQDN-2>
    • To upgrade all compilers simultaneously, run:
      sudo puppet infrastructure upgrade compiler --all
    • To specify an authentication token location other than the default location, include --token-file <PATH_TO_TOKEN> in the command, such as:
      sudo puppet infrastructure upgrade compiler <COMPILER_FQDN> --token-file <PATH_TO_TOKEN>
  6. Upgrade these additional PE infrastructure components:
    • Agents
    • PE client tools: On unmanaged nodes, you must re-install the client tools version that matches the PE version you upgraded to. On managed nodes and infrastructure nodes, client tools are automatically updated when you upgrade PE.
  7. Optional: Remove old PE packages from all infrastructure nodes. On your primary server, run: puppet infrastructure run remove_old_pe_packages
    All packages older than the current version are removed by default. To remove specific versions, append pe_version=<VERSION_NUMBER_OR_SHA> to the command.
What to do next

Optionally convert legacy compilers to the new style compiler running the PuppetDB service.

Upgrade an unmanaged PostgreSQL installation

To upgrade a Puppet Enterprise (PE) installation that relies on an unmanaged PostgreSQL database, you must upgrade PostgreSQL to version 14, ensure your PostgreSQL database includes a PuppetDB read user, and then proceed with the PE upgrade.

Before you begin
Back up your PE installation, and, if desired, back up your PostgreSQL databases.

Make sure you have a valid admin RBAC token to upgrade compilers (for example, Generate a token using puppet-access).

In Hiera, pe.conf, or the console (in the PE Master node group), remove any agent_version parameters you set in the pe_repo class that matches your infrastructure nodes. This ensures the upgrade isn't blocked by attempting to download non-default agent versions for your infrastructure OS and architecture.

CAUTION: Don't use CONCURRENTLY with PostgreSQL 14.0 through 14.3. If you do, make sure to REINDEX without using CONCURRENTLY.
  1. If you are upgrading from a PE version earlier than 2021.6, upgrade PostgreSQL to version 14:
    1. On your PostgreSQL node, install these packages:
      • postgresql14
      • postgresql14-libs
      • postgresql14-contrib
      • postgresql14-server
    2. Stop the PostgreSQL 11 service: systemctl stop postgresql-11.service
    3. As the PostgreSQL user, create a new database: 
      su - postgres /usr/pgsql-14/bin/pg_ctl -D /var/lib/pgsql/14/data/ initdb
    4. If you use certificate authentication, note the location of your certificate, private key, and certificate authority files. You might need to move them to the new install directory, and you might need to update postgresql.conf.
    5. Copy your postgresql.conf file from /var/lib/pgsql/11/data/ to /var/lib/pgsql/14/data/ and update the data directory setting. If you moved the certificate, private key, and certificate authority files, update the ssl_cert_file, ssl_key_file, and ssl_ca_file settings.
    6. Migrate databases to PostgreSQL 14: 
      /usr/pgsql-14/bin/pg_upgrade -b /usr/pgsql-11/bin -B /usr/pgsql-14/bin/ -d /var/lib/pgsql/11/data -D /var/lib/pgsql/14/data/
      Tip: This command uses pg_upgrade to upgrade and migrate your unmanaged PostgreSQL instance from version 11 to 14. If you don't want to use this command, you can back up your PostgreSQL 11 databases, manually wipe the PostgreSQL 11 installation, ensure PostgreSQL 14 is installed, and then restore the databases to the PostgreSQL 14 installation. Alternately, you can back up the PostgreSQL 11 databases, set up a new node with PostgreSQL 14, restore databases to the new node, and then reconfigure PE to point to the new database_host node.
    7. Copy your pg_hba.conf and pg_ident.conf files from /var/lib/pgsql/11/data/ to /var/lib/pgsql/14/data/.
    8. In the pg_hba.conf file change all instances of clientcert=1 to clientcert=verify-full.
  2. If you're upgrading from a PE version earlier than 2019.8.7, add the PuppetDB read user. To do this, run the commands corresponding with your authentication method.
    • Password authentication: Run the following commands (make sure you specify a password):
      CREATE ROLE "pe-puppetdb-read"  LOGIN NOCREATEROLE NOCREATEDB NOSUPERUSER  CONNECTION LIMIT -1 PASSWORD '<PUPPETDB_USER_PASSWORD>';
GRANT CONNECT ON DATABASE "pe-puppetdb" TO "pe-puppetdb-read";
ALTER DEFAULT PRIVILEGES FOR USER "pe-puppetdb" IN SCHEMA "public" GRANT SELECT ON TABLES TO "pe-puppetdb-read";
ALTER DEFAULT PRIVILEGES FOR USER "pe-puppetdb" IN SCHEMA "public" GRANT EXECUTE ON FUNCTIONS TO "pe-puppetdb-read";
ALTER DEFAULT PRIVILEGES FOR USER "pe-puppetdb" IN SCHEMA "public" GRANT USAGE ON SEQUENCES TO "pe-puppetdb-read";
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO "pe-puppetdb-read";
GRANT USAGE ON ALL SEQUENCES IN SCHEMA "public" TO "pe-puppetdb-read";
    • Certificate authentication: Edit pg_hba.conf and pg_ident.conf. First, add these lines to pg_hba.conf to create the database user:
      hostssl pe-puppetdb pe-puppetdb-read 0.0.0.0/0 cert map=pe-puppetdb-pe-puppetdb-read-map clientcert=verify-full
hostssl pe-puppetdb pe-puppetdb-read ::/0 cert map=pe-puppetdb-pe-puppetdb-read-map clientcert=verify-full
      Then, add this line to pg_ident.conf to map the database user role:
      pe-puppetdb-pe-puppetdb-read-map <PRIMARY_CERTNAME> pe-puppetdb-read
  3. Start the PostgreSQL 14 service: 
    systemctl start postgresql-14
  4. Optional: As the PostgreSQL user, clean up the database: 
    su - postgres /usr/pgsql-14/bin/vacuumdb --analyze --all
  5. Download the tarball for your operating system and architecture.
  6. Run tar -xf <TARBALL> to unpack the installation tarball.
    You need about 1 GB of space to untar the installer.
  7. From the installer directory on your primary server, run sudo ./puppet-enterprise-installer -s to start the installer, and then follow the CLI instructions to complete your server upgrade.
    • The -s flag tells the installer to skip database checks.
    • To specify a different pe.conf file than the existing file, use the -c flag, such as:
      sudo ./puppet-enterprise-installer -c <FULL_PATH_TO_pe.conf>

      This flag tells the installer to backup the previous pe.conf file to /etc/puppetlabs/enterprise/conf.d/<TIMESTAMP>.conf and create a new pe.conf file at /etc/puppetlabs/enterprise/conf.d/pe.conf.

  8. To upgrade compilers, log in to your primary server as root and run one of these commands:
    • To upgrade specific compilers, run:
      sudo puppet infrastructure upgrade compiler <COMPILER_FQDN-1>,<COMPILER_FQDN-2>
    • To upgrade all compilers simultaneously, run:
      sudo puppet infrastructure upgrade compiler --all
    • To specify an authentication token location other than the default location, include --token-file <PATH_TO_TOKEN> in the command, such as:
      sudo puppet infrastructure upgrade compiler <COMPILER_FQDN> --token-file <PATH_TO_TOKEN>
  9. Upgrade these additional PE infrastructure components:
    • Agents
    • PE client tools: On unmanaged nodes, you must re-install the client tools version that matches the PE version you upgraded to. On managed nodes and infrastructure nodes, client tools are automatically updated when you upgrade PE.
  10. Optional: Remove old PE packages from all infrastructure nodes. On your primary server, run: puppet infrastructure run remove_old_pe_packages
    All packages older than the current version are removed by default. To remove specific versions, append pe_version=<VERSION_NUMBER_OR_SHA> to the command.
  11. Optional: Remove PostgreSQL 11 packages and directories by running puppet infrastructure run remove_old_postgresql_versions. If applicable, the installer prompts you to complete the cleanup.
What to do next

Optionally, convert legacy compilers to the new style compiler running the PuppetDB service.

Migrate PE

As an alternative to upgrading, you can migrate your PE installation. Migrating results in little or no downtime, but it requires additional system resources because you must configure a new primary server.

Note: This procedure is not supported for installations with disaster recovery.

Migrate a standard installation

Migrate a standard installation by standing up a new primary server, restoring it with your existing installation, upgrading it, and then pointing agents at the new primary.

Before you begin

Review the upgrade cautions for major changes to architecture and infrastructure components which might affect your upgrade.

  1. Install your current PE version on a new primary server node.
  2. Back up your existing installation.
  3. Restore your installation on the new primary server using the backup you created.
  4. Upgrade your new primary server to the latest PE version.
  5. Update nodes' puppet.conf server setting to the hostname for your new primary server: 
    sudo puppet config set server <PRIMARY_HOSTNAME>