diff --git a/README.md b/README.md index 68c2035..14e399e 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,7 @@ It includes automatic parsing of the `Puppetfile`, `environment.conf` and others - [Puppetfile](#puppetfile) - [Spec testing](#spec-testing) - [Adding your own spec tests](#adding-your-own-spec-tests) + - [Vendored Modules](#vendored-modules) - [Using Workarounds](#using-workarounds) - [Extra tooling](#extra-tooling) - [Plugins](#plugins) @@ -36,6 +37,7 @@ It includes automatic parsing of the `Puppetfile`, `environment.conf` and others - [Ruby Warnings](#ruby-warnings) - [Rake tasks](#rake-tasks) - [generate_fixtures](#generate_fixtures) + - [generate_vendor_cache](#generate_vendor_cache) ## Overview @@ -616,6 +618,30 @@ If you want to see Puppet's output, you can set the `SHOW_PUPPET_OUTPUT` environ `SHOW_PUPPET_OUTPUT=true onceover run spec` +### Vendored Modules + +As of Puppet 6.0 some resource types were removed from Puppet and repackaged as individual modules. These supported type modules are still included in the `puppet-agent` package, so you don't have to download them from the Forge. However, this does not apply to the `puppet` gem used when spec testing. This frequently results in users wondering why their Puppet manifests apply just fine on a node, but their tests fail with messages like `Unknown resource type: cron_core` for example. A common workaround for this problem was to add said modules into your Puppetfile, thus requiring manual management. + +Onceover now has the ability to remove that manual process for you by querying Github's API to determine which versions are in use by the version of the [puppet-agent package](https://github.com/puppetlabs/puppet-agent/tree/main/configs/components) you are testing against. + +This functionality is opt in, so to use it configure the following: + +```yaml +# onceover.yaml +opts: + auto_vendored: true +``` + +or on the cli: + +```shell +bundle exec onceover run spec --auto_vendored=true +``` + +Essentially what this is doing is resolving any of these [supported type modules](https://www.puppet.com/docs/puppet/8/type#supported-type-modules-in-puppet-agent) that are not already specified in your Puppetfile, and adding them to the copy Onceover uses to deploy into its working directory structure. + +CI/CD pipeline users are encouraged to provide Onceover with a cache of the module versions to test against in order to avoid hitting Githubs API ratelimit. To do so, the [generate_vendor_cache](#generate_vendor_cache) rake task can be used to populate the cache into your `spec/vendored_modules` directory. + ## Using workarounds There may be situations where you cannot test everything that is in your puppet code, some common reasons for this include: @@ -890,6 +916,12 @@ fixtures: Notice that the symlinks are not the ones that we provided in `environment.conf`? This is because the rake task will go into each of directories, find the modules and create a symlink for each of them (This is what rspec expects). +#### generate_vendor_cache + +`bundle exec rake generate_vendor_cache` + +This task will query Github's API to determine the versions of the vendored modules in use by the version of the puppet agent you are testing against, and cache that information in `control-repo/spec/vendored_modules`. This way your pipelines won't need to reach out for this information each time Onceover is ran with `auto_vendored` enabled. + ## Developing Onceover Install gem dependencies: diff --git a/features/auto_vendored.feature b/features/auto_vendored.feature index 87b8f69..eddf8dd 100644 --- a/features/auto_vendored.feature +++ b/features/auto_vendored.feature @@ -1,4 +1,4 @@ -@vendored +@vendored @puppet6 Feature: Automatically resolve modules vendored with puppet-agent package Onceover should optionally attempt to resolve these vendored modules so that users do not need to maintain these in their Puppetfile's unless they have a reason diff --git a/lib/onceover/vendored_modules.rb b/lib/onceover/vendored_modules.rb index 96e5f2b..c974a61 100644 --- a/lib/onceover/vendored_modules.rb +++ b/lib/onceover/vendored_modules.rb @@ -26,35 +26,37 @@ class VendoredModules attr_reader :vendored_references, :missing_vendored def initialize(opts = {}) -# def initialize(repo = Onceover::Controlrepo.new, cachedir = nil) @repo = opts[:repo] || Onceover::Controlrepo.new @cachedir = opts[:cachedir] || File.join(@repo.tempdir, 'vendored_modules') @puppet_version = Gem::Version.new(Puppet.version) - @puppet_major_version = Gem::Version.new(@puppet_version).segments[0] + @puppet_major_version = Gem::Version.new(@puppet_version.segments[0]) @force_update = opts[:force_update] || false @missing_vendored = [] + # This only applies to puppet >= 6 so bail early + raise 'Auto resolving vendored modules only applies to puppet versions >= 6' unless @puppet_major_version >= Gem::Version.new('6') + # Create cachedir unless File.directory?(@cachedir) logger.debug "Creating #{@cachedir}" FileUtils.mkdir_p(@cachedir) end - # location of user provided caches: + # Location of user provided caches: # control-repo/spec/vendored_modules/-puppet_agent-.json @manual_vendored_dir = File.join(@repo.spec_dir, 'vendored_modules') - # get the entire file tree of the puppetlabs/puppet-agent repository + # Get the entire file tree of the puppetlabs/puppet-agent repository # https://docs.github.com/en/rest/git/trees?apiVersion=2022-11-28#get-a-tree puppet_agent_tree = query_or_cache( "https://api.github.com/repos/puppetlabs/puppet-agent/git/trees/#{@puppet_version}", { :recursive => true }, component_cache('repo_tree') ) - # get only the module-puppetlabs-_core.json component files + # Get only the module-puppetlabs-_core.json component files vendored_components = puppet_agent_tree['tree'].select { |file| /configs\/components\/module-puppetlabs-\w+\.json/.match(file['path']) } - # get the contents of each component file + # Get the contents of each component file # https://docs.github.com/en/rest/git/blobs?apiVersion=2022-11-28#get-a-blob @vendored_references = vendored_components.map do |component| mod_slug = component['path'].match(/.*(puppetlabs-\w+).json$/)[1] @@ -73,26 +75,34 @@ def component_cache(component) # By default look for any caches created during previous runs cache_file = File.join(@cachedir, desired_name) - # If the user provides their own cache - if File.directory?(@manual_vendored_dir) - # Check for any '-puppet_agent-.json' files - dg = Dir.glob(File.join(@manual_vendored_dir, "#{component}-puppet_agent*")) - # Check if there are multiple versions of the component cache - if dg.size > 1 - # If there is the same version supplied as whats being tested against use that - if dg.any? { |s| s[desired_name] } - cache_file = File.join(@manual_vendored_dir, desired_name) - # If there are any with the same major version, use the latest supplied - elsif dg.any? { |s| s["#{component}-puppet_agent-#{@puppet_major_version}"] } - maj_match = dg.select { |f| /#{component}-puppet_agent-#{@puppet_major_version}.\d+\.\d+\.json/.match(f) } - maj_match.each { |f| cache_file = f if version_from_file(f) >= version_from_file(cache_file) } - # otherwise just use the latest supplied - else - dg.each { |f| cache_file = f if version_from_file(f) >= version_from_file(cache_file) } + unless @force_update + # If the user provides their own cache + if File.directory?(@manual_vendored_dir) + # Check for any '-puppet_agent-.json' files + dg = Dir.glob(File.join(@manual_vendored_dir, "#{component}-puppet_agent*")) + # Check if there are multiple versions of the component cache + if dg.size > 1 + # If there is the same version supplied as whats being tested against use that + if dg.any? { |s| s[desired_name] } + cache_file = File.join(@manual_vendored_dir, desired_name) + # If there are any with the same major version, use the latest supplied + elsif dg.any? { |s| s["#{component}-puppet_agent-#{@puppet_major_version}"] } + maj_match = dg.select { |f| /#{component}-puppet_agent-#{@puppet_major_version}.\d+\.\d+\.json/.match(f) } + maj_match.each do |f| + if (version_from_file(cache_file) == version_from_file(desired_name)) || (version_from_file(f) >= version_from_file(cache_file)) + # if the current cache version matches the desired version, use the first matching major version in user cache + # if there are multiple major version matches in user cache, use the latest + cache_file = f + end + end + # Otherwise just use the latest supplied + else + dg.each { |f| cache_file = f if version_from_file(f) >= version_from_file(cache_file) } + end + # If there is only one use that + elsif dg.size == 1 + cache_file = dg[0] end - # if there is only one use that - elsif dg.size == 1 - cache_file = dg[0] end end @@ -110,15 +120,15 @@ def version_from_file(cache_file) Gem::Version.new(version_regex.match(cache_file)[1]) end - # currently expects to be passed a R10K::Puppetfile object. + # Currently expects to be passed a R10K::Puppetfile object. # ex: R10K::ModuleLoader::Puppetfile.new(basedir: '.') def puppetfile_missing_vendored(puppetfile) puppetfile.load @vendored_references.each do |mod| - # extract name and slug from url + # Extract name and slug from url mod_slug = mod['url'].match(/.*(puppetlabs-\w+)\.git/)[1] mod_name = mod_slug.match(/^puppetlabs-(\w+)$/)[1] - # array of modules whos names match + # Array of modules whos names match existing = puppetfile.modules.select { |e_mod| e_mod.name == mod_name } if existing.empty? # Change url to https instead of ssh to allow anonymous git clones @@ -132,7 +142,7 @@ def puppetfile_missing_vendored(puppetfile) end end - # return json from a query whom caches, or from the cache to avoid spamming github + # Return json from a query whom caches, or from the cache to avoid spamming github def query_or_cache(url, params, filepath) if (File.exist? filepath) && (@force_update == false) logger.debug "Using cache: #{filepath}" @@ -146,7 +156,7 @@ def query_or_cache(url, params, filepath) json end - # given a github url and optional query parameters, return the parsed json body + # Given a github url and optional query parameters, return the parsed json body def github_get(url, params) uri = URI.parse(url) uri.query = URI.encode_www_form(params) if params @@ -168,12 +178,12 @@ def github_get(url, params) end end - # returns parsed json of file + # Returns parsed json of file def read_json_dump(filepath) MultiJson.load(File.read(filepath)) end - # writes json to a file + # Writes json to a file def write_json_dump(filepath, json_data) File.write(filepath, MultiJson.dump(json_data)) end