About file sync

File sync helps Code Manager keep your Puppet code synchronized across your primary server and compilers.

When triggered by a web endpoint, file sync takes changes from your working directory on your primary server and deploys the code to a live code directory. File sync then automatically deploys that code to all compilers.

In addition, file sync ensures that your Puppet code is deployed only when it is ready. These deployments ensure that your agents' code do not change during a run. File sync also triggers an environment cache flush when the deployment has finished, to ensure that new agent runs happen against the newly deployed Puppet code.

Important:

File sync is part of Code Manager, and you usually don't need to directly configure or trigger file sync.

Directly accessing the file sync API is unsupported and not recommended.

For information about how code deployments are triggered, refer to How Code Manager works.

Tip: If you want code deployments to run uninterrupted while Orchestrator plans are running, you can enable Running plans alongside code deployments.

File sync terms

There are a few terms that are helpful when you are working with file sync or tools that use file sync.

Live code directory
This is the directory that all of the Puppet environments, manifests, and modules are stored in. This directory is used by Puppet Server for catalog compilation. It corresponds to the Puppet Server master-code-dir setting and the Puppet$codedir setting. The default value is /etc/puppetlabs/code. In file sync configuration, you might see this referred to simply as live-dir. This directory exists on your primary server and compilers.
Staging code directory
This is the directory where you stage your code changes before rolling them out to the live code dir. You can move files into this directory in your usual way. Then, when you trigger a file sync deployment, file sync moves the changes to the live code dir on your primary server and compilers. This directory exists only on the primary server; compilers do not need a staging directory. The default value is /etc/puppetlabs/code-staging.

How file sync works

File sync helps distribute your code to your primary server, compilers, and agents.

By default, file sync is disabled and the staging directory is not created on the primary server. If you're upgrading from 2015.2 or earlier, file sync is disabled after the upgrade. You must enable file sync, and then run Puppet on your primary server and compilers. This creates the staging directory on the primary server, which you can then populate with your Puppet code. File sync can then commit your code; that is, it can prepare the code for synchronization to the live code directory, and then your compilers. Normally, Code Manager triggers this commit automatically, but, only if absolutely necessary, you can trigger a commit by hitting the file sync endpoint.

Important:

File sync is part of Code Manager, and you usually don't need to directly configure or trigger file sync.

File sync endpoint documentation is for informational purposes only. Directly accessing the file sync API is unsupported and not recommended.

For example:

type_header='Content-Type: application/json'
cert="$(puppet config print hostcert)"
cacert="$(puppet config print localcacert)"
key="$(puppet config print hostprivkey)"
uri="https://$(puppet config print server):8140/file-sync/v1/commit"
data='{"commit-all": true}'

curl --header "$type_header" --cert "$cert" --cacert "$cacert" --key "$key" --request POST "$uri" --data "$data"

See Usage notes for curl examples for information about forming curl commands.

The above command is run from the primary server and contains the following variables:

  • fqdn: Fully qualified domain name of the primary server.
  • cert: The ssl cert for the primary server.
  • cacert: The Puppet CA's certificate.
  • key: The private key for the primary server.

This command commits all of the changes in the staging directory. After the commit, when any compilers check the file sync service for changes, they receive the new code and deploy it into their own code directories, where it is available for agents checking in to those compilers. (By default, compilers check file sync every 5 seconds.)

Commits can be restricted to a specific environment and can include details such as a message, and information about the commit author.

Enabling or disabling file sync

File sync is normally enabled or disabled automatically along with Code Manager.

File sync's behavior is linked to that of Code Manager. Because Code Manager is disabled by default, file sync is also disabled. To enable file sync, enable Code Manager You can enable and configure Code Manager either during or after PE installation.

The file_sync_enabled parameter in the puppet_enterprise::profile::master class in the console defaults to automatic, which means that file sync is enabled and disabled automatically with Code Manager. If you set this parameter to true, it forces file sync to be enabled even if Code Manager is disabled. The file_sync_enabled parameter doesn't appear in the class definitions --- you must add the parameter to the class in order to set it.

Resetting file sync

If file sync has entered a failure state, consumed all available disk space, or a repository has become irreparably corrupted, reset the service.

Resetting deletes the commit history for all repositories managed by file sync, which frees up disk space and returns the service to a "fresh install" state while preserving any code in the staging directory.

  1. On the primary server, perform the appropriate action:
    • If you use file sync with Code Manager, ensure that any code ready for deployment is present in the staging directory, and that the code most recently deployed is present in your control repository so that it can be re-synced.
    • If you use file sync with r10k, perform an r10k deploy and ensure that the code most recently deployed is present in your control repository so that it can be re-synced.
    • If you use file sync alone, ensure that any code ready for deployment is present in /etc/puppetlabs/code-staging.
  2. Shut down the pe-puppetserver service: puppet resource service pe-puppetserver ensure=stopped.
  3. Delete the data directory located at /opt/puppetlabs/server/data/puppetserver/filesync/storage.
  4. Restart the pe-puppetserver service: puppet resource service pe-puppetserver ensure=running.
  5. Perform the appropriate action:
    • If you use file sync with Code Manager, use Code Manager to deploy all environments.
    • If you use file sync alone or with r10k, perform a commit.
Results

File sync is now reset. The service creates fresh repositories on each client and the storage server for the code it manages.

Checking your deployments

You can manually look up information about file sync's deployments by using curl commands that hit the status endpoint.

Important:

File sync is part of Code Manager, and you usually don't need to directly configure or trigger file sync.

File sync endpoint documentation is for informational purposes only. Directly accessing the file sync API is unsupported and not recommended.

To look up deployment information, run the following curl command:

curl --insecure "https://$(puppet config print server):8140/status/v1/services/file-sync-storage-service?level=debug"

The curl command returns a list of:

  • All of the clients that file sync is aware of.
  • When those clients last checked in.
  • Which commit they have deployed.
Note: This command returns output in JSON, so you can pipe it to python -m json.tool for better readability. See Using example commands for information about forming curl commands.

You can check if a specific commit was deployed by viewing the latest_commit. This commit is listed as the latest_commit.

Important: The SHA reported as latest_commit is internal to file sync and not from the user's control repository.

Cautions

There are a few things to be aware of with file sync.

File sync API

Because file sync is part of Code Manager, Code Manager handles communication with the file sync API. Information about the file sync API in this documentation is for informational purposes only.

Important: Do not directly access the file sync API.

Where to edit code

With Code Manager, you only modify code in your control repo. Changes made in invalid locations are overwritten by the next deployment. For more information, refer to Understanding file sync and the staging directory.

While extremely uncommon, if you're using file sync without Code Manager, only modify your Puppet code in the staging directory.

The enable-forceful-sync parameter is set to true by default in PE. When false, file sync no longer overwrites changes in the live code directory. Instead, it logs errors to the Puppet Server log (at /var/log/puppetlabs/puppetserver/puppetserver.log).

If you need to set this parameter to false, you must add it with Hiera:
puppet_enterprise::master::file_sync::file_sync_enable_forceful_sync: false

The puppet module command and file sync

The puppet module command doesn't work with file sync. You'll need to specify modules in your environment's Puppetfiles and use Code Manager to handle module code deployments. For information and instructions, refer to:

Permissions

File sync runs as the pe-puppet user. To sync files, file sync must have permission to read the staging directory and to write to all files and directories in the live code directories. You can run the following command to make sure the pe-puppet user owns the required directories:
chown -R pe-puppet /etc/puppetlabs/code /etc/puppetlabs/code-staging

Environment isolation metadata

File sync generates .pp metadata files in your staging code directory and live code directories. These files provide environment isolation for your resource types, which ensures that each environment uses the correct version of the resource type.

CAUTION: Do not delete or modify these files. Do not use expressions from these files in regular manifests.

For more details about these files and how they isolate resource types in multiple environments, refer to Environment isolation in the Puppet documentation. For information about when these files are generated, refer to Environment isolation metadata and Code Manager.