PE release notes

These are the new features, enhancements, resolved issues, and deprecations in this version of Puppet Enterprise (PE).

Security and vulnerability announcements are posted at https://puppet.com/docs/security-vulnerability-announcements.

PE 2021.7.5

Released September 2023

Important: PE 2021.7 is the current PE LTS series, and PE 2023 is the STS PE series.

For information about upgrading from 2019.8.z to 2021.7 (and earlier 2021.y series release notes), see What's new since PE 2019.8 and Upgrading Puppet Enterprise.

For information about upgrading to 2023, see Upgrading Puppet Enterprise in the 2023.0 documentation.

Enhancements

Classifier service flags unmappable legacy facts in node group rules
Legacy facts are deprecated in Puppet 7, which is the Puppet version included in this release, and are removed in Puppet 8, which is the Puppet version introduced in PE 2023.3. To support the transition away from legacy facts to structured facts, the classifier service in PE 2021.7.5 analyzes your node group rules and generates warning messages in the logs to flag uses of certain legacy facts that do not map to equivalent structured facts in Puppet 8. For more information about the removal of deprecated legacy facts in Puppet 8, see Puppet upgrade in 2023.3.
Orchestrator HTTP-client limits can be configured to match infrastructure requirements
You can now specify HTTP-client connection limit parameters in the puppet_enterprise::profile::orchestrator class. You can set connection limits for authenticated and unauthenticated clients by specifying an integer value for the following parameters:
  • max_connections_per_route_authenticated
  • max_connections_total_authenticated
  • max_connections_per_route_unauthenticated
  • max_connections_total_unauthenticated
Orchestrator socket timeout is configurable
By default, whenever no data is available on the socket, the orchestrator waits for a maximum of 120,000 milliseconds before closing the HTTP connection. Now you can specify the maximum time before socket timeout by changing the default value of the socket_timeout parameter in the puppet_enterprise::profile::orchestrator class.
Improvements to error logging for the puppet backup command
Previously, error messages returned by the puppet backup command were generic in many cases. Now, descriptive error messages are displayed both in the terminal and in the log file, and you can use a --debug flag with puppet backup to extend error logging to all underlying Puppet commands.

Platform support

PE 2021.7.5 adds support for the following operating system platforms.
Added primary server platforms
Red Hat Enterprise Linux (RHEL) 9 x86_64
Ubuntu 22.04 amd64
Added agent platforms
macOS 13 ARM and x86_64
Added client tools platform
macOS 13 ARM and x86_64
Added patch management platforms
Red Hat Enterprise Linux (RHEL) 9 x86_64

With this release, support was removed for several previously deprecated platforms. Before upgrading to PE 2021.7.5, review the following list of removed platforms and the important information about removed platforms in Platforms removed in 2021.0 and later.

Removed agent platforms
CentOS 7 aarch64
macOS 10.15
Oracle Linux 7 aarch64
Red Hat 7 aarch64
Scientific Linux 7 aarch64
Removed client tool platforms
macOS 10.15

Deprecations and removals

Removed platforms
For information about platforms removed in this release, see the Platform support section.

Resolved issues

Installing Windows agent through the console no longer fails when option to test connections is selected
In PE 2021.7.2 and later, when installing Windows agents in the console’s Install agent on nodes screen, checking the Test Connections checkbox before clicking Add nodes caused the process to hang indefinitely. The issue is resolved in PE 2021.7.5.

PE 2021.7.3

Released May 2023

Important: PE 2021.7 is the current PE LTS series, and PE 2023 is the STS PE series.

For information about upgrading from 2019.8.z to 2021.7 (and earlier 2021.y series release notes), see What's new since PE 2019.8 and Upgrading Puppet Enterprise.

For information about upgrading to 2023, see Upgrading Puppet Enterprise in the 2023.0 documentation.

Enhancements

Improved performance when querying PuppetDB
This enhancement helps to improve performance for PuppetDB queries that contain large arrays, for example, if many nodes are enumerated or many terms are joined by a single "and" or "or" element.
Improved performance for the each, map, and filter functions in the Puppet language
Previously, the Puppet language built-in functions each, map, and filter showed poor performance and consumed unnecessary resources when run on JRuby software. The issue was resolved to enhance performance.
Puppet Server provides more reliable warnings when it cannot check for an update
By default, Puppet Server periodically checks whether a new version of Puppet Server is available. Previously, if Puppet Server could not connect to the update server, users were not provided with adequate information about the error. Starting with Puppet Server 7.10.1, a warning about the error is available in the log file.

Deprecations and removals

Deprecated PSON
In previous releases, Pure JavaScript Open Notation (PSON) was used in Puppet to serialize data for transmission.

PSON is deprecated in Puppet 7 and will be removed in Puppet 8.

Resolved issues

Tasks page is available following a software update
After upgrading PE from 2019.8 to 2021.7.1, the Tasks overview page in the PE console sometimes failed to load because of a timeout error. The issue is fixed in PE 2021.7.3 and 2023.1.
Enabling the lockless code deploy feature no longer causes performance issues in PuppetDB catalog compilation
When the versioned_deploys setting is enabled, Puppet previously reported the full directory path to the environment after resolving symbolic links as the source for resources in a catalog. Puppet now reports the path to the resource before resolving symbolic links in the environment path to help prevent instability of the PuppetDB instance.
Performance issue with Puppet agent runtimes is resolved
After an upgrade from PE 2019.8.12 to PE 2021.7.1, some users saw a significant increase in Puppet agent runtimes. The increase was caused by Facter 4, which was not using cached information to resolve facts. As a result, facts were resolved multiple times. The issue is now resolved to normalize the performance of the Puppet agent.
Certificates and keys can be backed up and restored by specifying the certs scope
Previously, if you ran the puppet-backup create command and specified a scope of certs, the command failed to back up the certificate authority root key and certificates. This issue occurred because Puppet 7 introduced a new default path for the certificate authority (CA) directory (/etc/puppetlabs/puppetserver/ca), but the puppet-backup create command failed to locate the new directory. Similarly, if you ran the puppet-backup restore command with a scope of certs, the restore operation failed. The CA directory issue is resolved so that backup and restore operations can run successfully.
Updates implemented to help users enter valid URLs
In previous versions of PE, the role-based access control (RBAC) service permitted the entry of invalid URLs when users specified the Organizational URL setting. Login attempts would then fail with the following error message:
'Invalid settings: organization_not_enough_data'

In PE 2021.7.3 and 2023.1, the RBAC service is updated to enforce valid URLs when users create or update a connection to a Security Assertion Markup Language (SAML) identity provider, and the PE console displays a warning if the user enters an invalid URL for the Organizational URL setting.

Timeouts can be specified for SAML authentication
Previously, when users configured the PE console to specify session-timeout and session-maximum-lifetime values, the settings were applied to Lightweight Directory Access Protocol (LDAP) tokens and local login tokens. However, the specified settings were not applied to SAML tokens, which are used for authentication with a SAML identity provider. The issue is corrected to ensure that the specified settings also apply to SAML session lifetimes.
User-defined temporary directory is honored during PE restore operations
After you back up your PE infrastructure, you can use the puppet-backup restore command to restore the backup. Previously, if you set the —tmpdir flag or the TMPDIR environment variable to specify a temporary directory for restore operations, the directory was not honored, and the default /tmp directory was used in some cases. In addition, some files were not cleaned up after the restore operation. This issue is corrected to ensure that the user-specified directory is used and all temporary files are removed after the restore operation.
Issue that caused an unexpected increase in CPU usage is resolved
In PE 2021.7.1, 2021.7.2, and 2023.0, an issue with Puppet Server caused an unexpected increase in central processing unit (CPU) usage in some environments. CPU usage continued to grow and some operations took longer than expected until the Puppet Server service was restarted. This issue is resolved in PE 2023.1 and 2021.7.3.
Security fixes
Addressed CVE-2023-1894 and CVE-2023-26048.

PE 2021.7.2

Released January 2023

Important: PE 2021.7 is the current PE LTS series, and PE 2023.0 is the new STS PE series.

For information about upgrading from 2019.8.z to 2021.7 (and earlier 2021.y series release notes) go to What's new since PE 2019.8 and Upgrading Puppet Enterprise.

For information about upgrading to 2023.0, go to Upgrading Puppet Enterprise in the 2023.0 documentation.

Enhancements

recover_configuration command recreates nodes files
Previously, the puppet infrastructure recover_configuration command merged new values into the nodes files (at /etc/puppetlabs/enterprise/conf.d/nodes) instead of overwriting the files. This process caused problems if you deleted a value relevant to one or more nodes, because the deleted value would remain in these files and continue to be applied.
Now, the recover_configuration command fully rewrites the nodes files on each invocation. This process matches how the command handles changes to the user_data.conf file.
Improved performance when regenerating agent certificates for multiple agents
The puppet infrastructure run regenerate_agent_certificate action is now faster when you Regenerate agent certificates for multiple agents. You can also now use the agent_pdb_query parameter to run a PDB query to generate a list of agents for which you want to regenerate certificates.
This action now uses the Puppet Server CA API endpoints directly, rather than relying on the puppetserver ca CLI, as it did previously. This process is faster, but, if you encounter problems, you can revert to the previous behavior by including use_puppetserver_cli=true in the command.
Specify Code Manager worker cache cleanup interval
The deploy-pool-cleanup-interval parameter specifies how often workers pause to clean their on-disk caches. Learn more about this setting in Code Manager parameters.

Platform support

This version adds support for the following platforms:
Agent platforms
Solaris 10 (SPARC, i386)
Client tools platforms
Solaris 10 (SPARC, i386)

Resolved issues

Code Manager respects full_deploy setting in Hiera
The full_deploy parameter is now correctly applied when you Customize Code Manager configuration in Hiera.
Previously, the full_deploy parameter value was disregarded when included in a Code Manager configuration in Hiera. As a work-around, you could create a separate .conf file to manually manage this parameter.
Important: If you created a .conf file for the full_deploy parameter, you must remove this file and reconfigure the parameter in Hiera (as described in Configuring module deployment scope).
Certain plans correctly restore puppet service to pre-plan state
Due to a bug introduced in PE 2021.6, some plans that must stop the puppet service while the plans run were not restoring the puppet service to its pre-plan state after the plan finished running.
The four affected plans, and their associated puppet infra commands, are as follows:
  • The secondary_cert_regen plan, which is triggered by puppet infra run regenerate_compiler_certificate and puppet infra run regenerate_replica_certificate
  • The convert_legacy_compiler plan, which is triggered by puppet infra run convert_legacy_compiler
  • The reprovision_replica plan, which is specifically triggered by puppet infra upgrade replica --only-recreate-databases
  • The enable_ha_failover plan, which is triggered by puppet infra run enable_ha_failover
Important: If you ran any of these four plans since upgrading to PE 2021.6, check the state of the puppet service on your infrastructure nodes.
PuppetDB database user can purge reports
An issue was fixed to help ensure that the PuppetDB database user can purge reports.
Corrected fact list handling in some PE console UI components
Some user interface (UI) components in the PE console use fact lists. A recent change caused these components to use the entire list of fact names. This process caused performance problems in environments with many facts. The handling of fact lists was updated to fix this issue and improve performance.
Orchestrator code directories excluded from puppet-backup create --scope=config
When Customizing backup and restore scope, the orchestrator code directories (specifically opt/puppetlabs/server/data/orchestration-services/data-dir and opt/puppetlabs/server/data/orchestration-services/code) are excluded when you specify the config scope.
These directories are included in the code scope.
Garbage collection log fixes
The introduction of Java 11 resulted in two issues pertaining to garbage collection logs. The issues are now fixed:
Dates and times are included in garbage collection logs.
The maximum volume of retained garbage collection logs is 256 MB.
Security fixes
Addressed CVE-2022-41946 and CVE-2022-41404.

PE 2021.7.1

Released October 2022

Important: PE 2021.7 is our new PE LTS series. If you're preparing to upgrade, or looking for earlier 2021.y release notes, go to What's new since PE 2019.8.

For those awaiting the new STS, we're still getting things ready for the first release in that series. We thank you for your patience.

New features

Stop in-progress plans
Use POST /command/stop_plan to stop an orchestrator plan job that is currently running.

Platform support

Deprecated and removed platforms are listed under Deprecations and removals.

This version adds support for these platforms:
Agent platforms
Fedora 36
Patch management platforms
Fedora 36

Deprecations and removals

Deprecated agent platforms
Debian 9
Fedora 32, 34
Deprecated patch management platforms
Debian 9
Fedora 34

Resolved issues

Deactivated scheduled jobs could still run.
If you deactivate a recurring scheduled job, the inactive job no longer continues to run after restarting pe-orchestration-services.
Deactivated jobs aren't visible in the console or in responses from the Scheduled jobs endpoints, but, prior to this fix, some deactivated jobs could run again at their next scheduled interval as if they had not been deactivated. This issue only impacted deactivated scheduled jobs that had run within the job_prune_threshold limit after restarting pe-orchestration-services.
Orchestrator didn't properly periodically prune jobs
Fixed a calculation error introduced in PE 2021.5 that caused job records to be stored beyond the job_prune_threshold limit.
regenerate_agent_certificate couldn't verify node type if client tools were installed through a package resource
When you run the puppet infra run regenerate_agent_certificate command, the plan can now verify that a node isn't an infrastructure node if the pe-client-tools package was installed on the node through a package resource.
RBAC API command/config/remove-disclaimer endpoint erroneously required Content-Type header
The POST /command/config/remove-disclaimer endpoint no longer requires a Content-Type header, because requests to this endpoint have no body content.
Internal task jobs shared primary task thread pool
Internal task jobs (such as tasks that force stop other tasks) no longer run on the same thread pool as your user-initiated tasks. This allows internal tasks to queue separately from other tasks. For example, requests to POST /command/stop don't get stuck waiting if the regular task queue is full.
Improved PuppetDB disaster recovery sync performance
The PuppetDB disaster recovery sync process transferred more reports than necessary when syncing reports, which sometimes caused timeouts.
Empty task metadata files prevented you from running tasks in the console
Loading empty task metadata files no longer cause errors.
Some puppet infrastructure commands failed when restarting the puppet service
Previously, several puppet infrastructure commands failed when restarting the puppet service at the end of the action. While the service had successfully restarted, the effected actions couldn't properly detect the restart, which caused them to fail. This has been fixed.

PE 2021.7.0

Released August 2022

Important: PE 2021.7 is our new PE LTS. Expect to see changes to the documentation ahead of our next STS release. We apologize for any inconvenience.

If you're preparing to upgrade or looking for earlier 2021.y release notes, go to What's new since PE 2019.8.

New features

Force stop in-progress Puppet runs
By default, POST /command/stop prevents new runs from starting, but allows in-progress runs to finish. Now you can use the force option to block new runs and stop in-progress runs. This is useful, for example, if you need to stop a task that is hanging.
pe_status_check module bundled with PE
The pe_status_check module helps keep your PE installation in an ideal state. Read About the pe_status_check module to learn how the module works and how to get the module's reports.
Important: If you have previously specified a version of this module, from the Forge or other sources, in your code, we recommend removing this version before upgrading to allow the version bundled with PE to be asserted.
New Orchestrator scheduling API
This release includes a new scheduling API for the orchestrator, which introduces several new scheduled_jobs endpoints and deprecates the previous scheduling API's endpoints (for a list of deprecated endpoints, see Deprecations and removals for this release, below).
Existing scheduled jobs are automatically migrated to the new scheduling system, and the PE console now uses the new API (but there is no change to the UI).
With the new API, you can edit scheduled jobs; however, this functionality is currently only available through the API (not yet available in the PE console). To learn more about the new endpoints, go to Scheduled jobs endpoints.
Tools that rely on the deprecated endpoints must be upgraded to use the new endpoints.
Use the RBAC API to set the disclaimer text on the console login page
You can use the RBAC API v1 Disclaimer endpoints to configure the disclaimer text that appears on the PE console login page.
Automatically sync LDAP user details and group membership
Prior to this release, user details and group membership for LDAP-based users only refreshed when users logged in. Now, LDAP group bindings, user names, and descriptions update automatically every 30 minutes (by default) for every LDAP user in the system. If a user is no longer present in LDAP or has no group bindings, all user-group associations are removed from the user and all of the user's known tokens are revoked.
You can disable automatic refresh or change the refresh time by changing the puppet_enterprise::profile::console::ldap_sync_period parameter. Learn more about this parameter in Configure RBAC and token-based authentication settings.
Stop LDAP users from logging in if they have no group membership
You can use the exclude-groupless-ldap-users setting to prevent LDAP users with no group memberships from logging in. This setting is off by default. To learn how to enable this setting, go toRequire LDAP group membership to log in.
Metrics API v2 documentation
The Metrics API v2 uses the Jolokia library to query Orchestrator service metrics. This version of the API has been available for some time, but it was only described in the open source Puppet documentation.
Disaster recovery support for FIPS platforms
Disaster recovery is now supported for FIPS 140-2 compliant Red Hat Enterprise Linux (RHEL) 7 and 8.

Enhancements

Orchestrator API endpoints return "total": 0 if there are no jobs
Orchestrator API v1 endpoints that return pagination containing the total number of jobs (such as GET /jobs, GET /scheduled_jobs (deprecated), and GET /plan_jobs) now return "total": 0, instead of "total": null, when there are no jobs.
Activity service API /v2/events endpoint returns more information for orchestrator events
Responses from GET /v2/events containing information about orchestrator events (Puppet agent runs and task runs) now report additional information about the job start time, end time, duration, and status.
Upgraded JRuby
We are now shipping JRuby 9.3.4.0.
Addressed CVEs
We updated the PostgreSQL driver in some PE component to address CVE-2022-31197. The application was not vulnerable to exploit prior to this update.
We also made changes to address CVE-2022-1292 and CVE-2022-2068.

Platform support

Ubuntu 16.04 is no longer a supported agent platform.

This version adds support for these platforms:
Agent
macOS 12 M1
Ubuntu (General Availability kernels) 22.04 x86_64
Microsoft Windows 11 x64
Client tools
Ubuntu (General Availability kernels) 22.04 x86_64
macOS 12 M1, M2
Patch management
Ubuntu (General Availability kernels) 22.04 x86_64
Microsoft Windows 11 x64

Deprecations and removals

Ubuntu 16.04 is no longer a supported agent platform.

The following endpoints are deprecated due to the release of several new Scheduled jobs endpoints for the Orchestrator API. Tools that rely on deprecated endpoints must be upgraded to use the new endpoints. Existing scheduled jobs are automatically migrated to the new scheduling system that uses the new endpoints.
GET /scheduled_jobs (deprecated)
Replaced by GET /scheduled_jobs/environment_jobs and GET /scheduled_jobs/environment_jobs/<job-id>
DELETE /scheduled_jobs/<job-id> (deprecated)
Replaced by PUT /scheduled_jobs/environment_jobs/<job-id>
POST /command/schedule_deploy (deprecated)
Replaced by POST /scheduled_jobs/environment_jobs
POST /command/schedule_plan (deprecated)
Replaced by POST /scheduled_jobs/environment_jobs
POST /command/schedule_task (deprecated)
Replaced by POST /scheduled_jobs/environment_jobs

Resolved issues

full-deploy didn't override --incremental
Code Manager's full-deploy option, used for Configuring module deployment scope, now correctly overrides the default --incremental deploy behavior.
Code Manager couldn't fetch code on FIPS platforms
On FIPS platforms running PE versions 2021.5 or 2021.6, Code Manager and r10k couldn't fetch code from your code repo due to libssh attempting to use algorithms that are not allowed on FIPS. In PE 2021.7, the disallowed algorithms are disabled in libssh, allowing Code Manager and r10k to successfully fetch code.
An unreachable replica consumed all of the primary server's disk space
Previously, if a provisioned replica became unreachable, the associated primary server could quickly run out of disk space, causing a complete interruption to PE services. In larger installations, an outage could occur in under an hour. Excessive disk usage was caused by the PE-PostgreSQL service on the primary server retaining change logs that the replica hadn't acknowledged.
To resolve this, we limited available disk space for the pg_wal directory. To learn more and tune this setting in your installation, refer to PostgreSQL WAL disk space.
Orchestrator ignored _noop when passed to run_task() through a plan
When a plan passed the _noop flag to the run_task() function, the PE Orchestrator now correctly acknowledges the _noop flag.
Some RBAC endpoints returned an incorrect Content-Type
Responses for the following endpoints now return the correct Content-Type: POST /users/<uuid>/password/reset, POST /auth/reset, and PUT /users/current/password.
LDAP with anonymous binding sometimes prevented Console Services from starting or restarting
Previously, if you use anonymous binding, or another configuration with a zero-length password, Console Services sometimes couldn't start or restart. This could cause upgrade failures when upgrading to PE version 2021.4 through 2021.6 from a version earlier than 2021.4. This is resolved.
Orchestrator doesn't restart unexpectedly during the convert_legacy_compiler plan
Previously, when running the enterprise_tasks::convert_legacy_compiler plan, the hosts in the pcp-brokers array could change order. This caused the pe-orchestration-services service to restart (as a result of detecting a presumed configuration change) and, ultimately, caused the plan to fail.
Some SSO configuration fields weren't marked as required
The Organization and Contacts fields on the SSO Configuration page are now correctly marked as required.
Orchestrator couldn't run tasks within modules named tasks or scripts
You can now successfully run tasks that are within modules named tasks or scripts.
Incorrect run-time for splayed agent runs
In previous PE versions, when agent runs were splayed, the run-time reported in the PE console was incorrect.
Sensitive parameters sometimes exposed in cleartext in job results
Sensitive plan parameters from Bolt plans that execute actions over PCP transport are no longer stored in the orchestrator database and, therefore, are properly masked in the job results.