Installing compilers

As your Puppet Enterprise infrastructure scales up to 4,000 nodes and beyond, add load-balanced compilers to your installation to increase the number of agents you can manage.

Each compiler increases capacity by 1,500 to 3,000 nodes, until you exhaust the capacity of PuppetDB or the console.

How compilers work

A single primary server can process requests and compile code for up to 4,000 nodes. When you exceed this scale, expand your infrastructure by adding compilers to share the workload and compile catalogs faster.

Important: Compilers must run the same OS major version, platform, and architecture as the primary server.
Compilers act as PCP brokers, conveying messages between the orchestrator and Puppet Execution Protocol (PXP) agents. PXP agents connect to PCP brokers running on compilers over port 8142. Status checks on compilers must be sent to port 8140, using https://<hostname>:8140/status/v1/simple.
A diagram of compiler connections.

Components and services running on compilers

Compilers typically run Puppet Server and PuppetDB services, as well as a file sync client. Older, legacy-style compilers must be converted in order to add PuppetDB.

When triggered by a web endpoint, file sync takes changes from the working directory on the primary server and deploys the code to a live code directory. File sync then deploys that code to all your compilers. By default, compilers check for code updates every five seconds.

The certificate authority (CA) service is disabled on compilers. A proxy service running on the compiler Puppet Server directs CA requests to the primary server, which hosts the CA in default installations.

Compilers also have:

  • The repository for agent installation, pe_repo
  • The controller profile used with PE client tools
  • Puppet Communications Protocol (PCP) brokers to enable orchestrator scale

Logs for compilers are located at /var/log/puppetlabs/puppetserver/.

Logs for PCP brokers on compilers are located at /var/log/puppetlabs/puppetserver/pcp-broker.log. Logback configuration for PCP broker logs is part of the Orchestration services settings.

Using load balancers with compilers

When using more than one compiler, a load balancer can help distribute the load between the compilers and provide a level of redundancy.

Specifics on how to configure a load balancer infrastructure falls outside the scope of this document, but examples of how to leverage haproxy for this purpose can be found on the HAproxy module Forge page.

Calculating load balancing

PCP brokers run on compilers and connect to PXP agents over port 8142. PCP brokers are built on websockets and require many persistent connections. If you're not using HTTP health checks, we recommend using a round robin or random load balancing algorithm for PXP agent connections to PCP brokers, because PCP brokers don't operate independent of the orchestrator and isolate themselves if they become disconnected. You can check connections with the /status/v1/simple endpoint for an error state.

Configure your load balancer to avoid closing long-lived connections that have little traffic. In the HAproxy module, you can set the timeout tunnel to 6m because PCP brokers disconnect inactive connections after 6 minutes. You can also configure the idle-timeout in the PCP broker trapperkeeper service configuration part of your Orchestration services settings.

Due to the diverse nature of the network communications between the agent and the primary server, we recommend that you implement a load balancing algorithm that distributes traffic between compilers based on the number of open connections. Load balancers often refer to this strategy as "balancing by least connections."

Using health checks

The Puppet REST API exposes a status endpoint that can be leveraged from a load balancer health check to ensure that unhealthy hosts do not receive agent requests from the load balancer.

The primary server service responds to unauthenticated HTTP GET requests issued to https://<hostname>:8140/status/v1/simple. The API responds with an HTTP 200 status code if the service is healthy.

If your load balancer doesn't support HTTP health checks, a simpler alternative is to check that the host is listening for TCP connections on port 8140. This ensures that requests aren't forwarded to an unreachable instance of the primary server, but it does not guarantee that a host is pulled out of rotation if it's deemed unhealthy, or if the service listening on port 8140 is not a service related to Puppet.

Load balancing for multi-region installations

If you have load balancers in multiple regions, use a global DNS proximity-based service address.

Tip: For guidance about deploying PE in global, multi-region, or multi-network segment scenarios, see the Multi-region Reference Architectures article.

When using a centralized Puppet deployment with multiple regional proxies or load balancers, create a global DNS proximity-based service address for Puppet and use that to route agents to the appropriate regional load balancer based on their location. Set the global DNS proximity-based address as the compiler pool address in Hiera.

For example, in your common.yaml file:
pe_repo::compile_master_pool_address: "<PUPPET-GLOBAL-SERVICE-ADDRESS>"
Some suitable global DNS proximity-based service address implementations include:
  • BIG-IP DNS
  • Route 53 Geolocation routing in AWS
  • TCP Proxy Global Load Balancing in GCP
  • Traffic Manager in Azure

Install compilers

Installing a compiler adds the specified node to the PE Infrastructure Agent and PE Compiler node groups and installs the PuppetDB service on the node.

Before you begin
The node you want to provision as a compiler must have a Puppet agent installed, or you must be able to connect to a non-agent node with SSH.

To install a FIPS-compliant compiler, install the compiler on a supported platform with FIPS mode enabled. The node must be configured with sufficient available entropy or the installation process fails.

Important: Contact Support for guidance before installing compilers in multi-region installations. If your primary server and compilers are connected with high-latency links or congested network segments, you might experience better PuppetDB performance with legacy compilers.
  1. Configure the agent on infrastructure nodes to connect to the primary server.
    1. In the console, click Node groups, and in the PE Infrastructure group, select the PE Agent > PE Infrastructure Agent group.
    2. If you manage your load balancers with agents, on the Rules tab, pin load balancers to the group.
      Pinning load balancers to the PE Infrastructure Agent group ensures that they communicate directly with the primary server.
    3. On the Classes tab, find the puppet_enterprise::profile::agent class and specify these parameters:
      Parameter Value
      manage_puppet_conf Specify true to ensure that your setting for server_list is configured in the expected location and persists through Puppet runs.
      pcp_broker_list Hostname for your primary server and replica, if you have one. Hostnames must include port 8142, for example ["PRIMARY.EXAMPLE.COM:8142", "REPLICA.EXAMPLE.COM:8142"].
      primary_uris Hostname for your primary server and replica, if you have one, for example ["PRIMARY.EXAMPLE.COM", "REPLICA.EXAMPLE.COM"]. This setting assumes port 8140 unless you specify otherwise with host:port.
      server_list
    4. Remove any values set for pcp_broker_ws_uris.
    5. Commit changes.
    6. Run Puppet on all agents classified into the PE Infrastructure Agent group.
  2. Pin the node that you want to provision to the PE Infrastructure Agent group and run Puppet on the node: puppet agent -t
  3. On your primary server, logged in as root, run the following command to provision a single compiler: 
    puppet infrastructure provision compiler <COMPILER_FQDN>
    This command accepts a maximum of one compiler FQDN; this command can't provision multiple compilers at once. Additionally, you can specify these optional parameters:
    • dns-alt-names: Comma-separated list of any alternative names that agents use to connect to the compiler. The installation uses puppet by default.
      Important: If your puppet.conf file includes a dns_alt_names entry, you must include the dns-alt-names parameter when provisioning your compiler.
    • no-dns-alt-names: Prevents the installer from setting the default alternative name, puppet. Use this parameter if you don't allow alternative names (as indicated by allow-subject-alt-names: false in your ca.conf file).
    • use-ssh: Enables installing on a node that doesn't have a Puppet agent currently installed. You must be able to connect to the node with SSH. You can pair this flag with additional SSH parameters. Run puppet infrastructure provision --help for details.
What to do next
Configure compilers to appropriately route communication between your primary server and agent nodes.

Configure compilers

Compilers must be configured to appropriately route communication between your primary server and agent nodes.

Before you begin

  • Install compilers and load balancers.

  • If you need DNS altnames for your load balancers, add them to the primary server.

  • Ensure port 8143 is open on the primary server or on any workstations used to run orchestrator jobs.

Note: This procedure is not intended for installations with load balancers in multiple locations. For details about configuring compilers in multi-region installations, see Load balancing for multi-region installations.
  1. Configure pe_repo::compile_master_pool_address to send agent install requests to the load balancer.
    Note: If you are using a single compiler, configure compile_master_pool_address with the compiler's fully qualified domain name (FQDN).
    1. In the console, click Node groups, and in the PE Infrastructure group, select the PE Master group.
    2. On the Configuration data tab, select the pe_repo class and specify parameters:
      Parameter Value
      compile_master_pool_address Load balancer hostname.
    3. Click Add data and commit changes.
  2. Run Puppet on the compiler, and then on the primary server.
  3. Configure agents to connect orchestration agents to the load balancer.
    1. In the console, click Node groups, and in the PE Infrastructure group, select the PE Agent group.
    2. On the Classes tab, find the puppet_enterprise::profile::agent class and specify parameters:
      Parameter Value
      manage_puppet_conf Specify true to ensure that your setting for server_list is configured in the expected location and persists through Puppet runs.
      pcp_broker_list Hostname for your load balancers. Hostnames must include port 8142, for example ["LOADBALANCER1.EXAMPLE.COM:8142","LOADBALANCER2.EXAMPLE.COM:8142"].
      primary_uris Hostname for your load balancers, for example ["LOADBALANCER1.EXAMPLE.COM","LOADBALANCER2.EXAMPLE.COM"]. This setting assumes port 8140 unless you specify otherwise with host:port.
      server_list
    3. Remove any values set for pcp_broker_ws_uris.
    4. Commit changes.
    5. Run Puppet on the primary server, then run Puppet on all agents, or install new agents.

      This Puppet run configures PXP agents to connect to the load balancer.

Convert existing compilers

If you have legacy compilers, you can improve their usability and scalability by adding PuppetDB. In addition to installing the PuppetDB service, converting an existing compiler adds the node to the PE Compiler node group and unpins it from the PE Master node group.

Before you begin
Open port 5432 from compilers to your primary server or, in extra-large installations, your PE-PostgreSQL node.
Important: Contact Support for guidance before converting compilers in multi-region installations. If your primary server and compilers are connected with high-latency links or congested network segments, you might experience better PuppetDB performance with legacy compilers.
On your primary server logged in as root, run: . 
puppet infrastructure run convert_legacy_compiler compiler=<COMPILER_FQDN-1>,<COMPILER_FQDN-2>
Tip: To convert all compilers:
puppet infrastructure run convert_legacy_compiler all=true
What to do next
Run puppet infrastructure tune on your primary server and adjust tuning for compilers as needed.