Guidelines for running Comply at scale
You can run Puppet Comply on a maximum of 25,000 nodes. Before you run Comply at scale, review the guidelines for configuring the environment and running the scan. The process of running Comply at scale was tested by Puppet in a controlled environment. Because many factors affect performance, results in your system environment might vary.
System requirements and configuration for large-scale environments
To support environments with more than 10,000 nodes, your Comply installation needs a total of at least 16GB of memory and 100GB of storage space available.
postgresqlsettings under Additional Support as follows before installation:
- Set Comply PostgreSQL memory to "8Gi".
- Set Comply PostgreSQL capacity to at least "50Gi".
- If you plan to retain historical data for more than 14 complete scans, increase Comply PostgreSQL capacity by approximately 3GB per additional scan.
Configure the scan process
- In Puppet orchestrator, set the
task_concurrencyparameter to a value appropriate for your environment and number of nodes. This value sets the maximum number of task or plan actions that can run concurrently in the orchestrator. If you set the parameter to 250 and run a scan of 5000 nodes, the orchestrator will be fully consumed until the scans are completed on all 5000 nodes. (For more information about optimizing performance, see Tune task and plan performance in Puppet Enterprise (PE).)
- Schedule scans to coincide with periods of minimal workflow to help ensure adequate network throughput.
- Plan adequate time for the initial inventory ingestion from Puppet Enterprise (PE). In lab testing, the ingestion of 25,000 nodes took 30 minutes.
- Plan adequate time for the scan. In lab testing with very high
task_concurrencyvalue, a scan of 25,000 nodes took 30 minutes. In a real world environment the Puppet orchestrator is likely to have lower concurrency limits, which may significantly affect your actual results.
- If you have a large number of nodes, consider configuring scans in smaller batches of up to 10,000 nodes.
Upgrade Comply in a large-scale environment
Before you upgrade Comply in an environment with thousands of nodes, review the limitations and consider the best strategy for your environment.
During the standard upgrade process, a new version of the CIS-CAT Pro Assessor is downloaded to each Puppet-managed node. However, Comply supports a limited number of concurrent downloads of the assessor. In lab testing, a maximum of about 120 concurrent downloads was achieved. Thus, if you initiate an upgrade of thousands of nodes, not all nodes will be updated on the first run.
- Run Puppet manually on a maximum of 120 nodes. Repeat the process until all nodes are updated.
- Configure Comply to host the assessor file on an internal web server and then upgrade Comply. If you choose this option you need to ensure that you host the correct assessor bundle based on your operating system.
- In the Puppet Enterprise (PE) console, click .
- In the Add new class field, select the Comply class.
- In the Parameter name field, select
- Set the value of the scanner source to the URL where the assessor will be
hosted. For example, the URL can have the following structure, where
server-hosting-assessor-ip specifies the IP address
of the server that will host the assessor and
- Commit the changes.
- In the PE console, click .
- Complete the upgrade process by selecting the relevant nodes and running the job.
Optimize scanning and reporting at scale
You can compare the results of your scanning and reporting processes against the results obtained in lab testing. If performance is not adequate in your environment, determine the cause of bottlenecks and address the issues.
Comply has been tested and is able to process reports from up to 25,000 nodes in a single scan. Processing this number of reports can take between 30 and 60 minutes depending on system resources. However, total scan time may be significantly longer based on Puppet orchestrator concurrency limits as well as the amount of time the CIS-CAT Pro Assessor takes to run on individual nodes.
The assessor run times are affected by the host type. In general, scans on Microsoft
Windows systems take longer than scans on
*nix systems. Run times can vary significantly,
depending on many other factors. For example, run times are longer for nodes with
many user accounts and for nodes with many types of software installed. Results
obtained in the lab represent an optimal use case.
To help understand performance issues, you can analyze log files. For more information, see Access logs.