Clone with NO history (shallow clones)

[daruvin]

Issue:

We are utilizing r10k deploy environment and the modules with files are starting to accumulate history every time the file changes. Since modules now allow files to be placed in the directory, we are seeing the size of a full repo get quite large, very quickly.

Reason:

For example, a 100mb file that has been replaced 10 times within the git repo, now causes a 1gb (100mb*10) file download onto the puppet masters, where the history of the past 9 versions of the file is not really needed.

Impact:

Our deployments that used to be 1 - 4 minutes are now close to 1 hour when introducing a new module.
Keeping all the relevant files in the module repo makes it easy for development sharing and distribution to our 1 dozen Regional masters. Separating the files will require 3 changes. Put files onto MoM, add code to sync down to masters, modules deployment depends on first 2 being complete.
Alternative 1, purge all history and loose ability to revert to older versions.
Alternative 2, maintain 1 repo for development, merge changes to "live" repo, purge all the history in the live repo to keep it shallow.

Proposed solution

Git allows a shallow copy. Is there a way to get r10k to clone modules with no history : where depth is '1'
Would be great to have it as an option, wouldn't mind if it was the default behavior, since there is no development on the master servers, the history is not needed.

git clone --depth <depth> -b <branch> <repo_url>
git clone --depth 1 -b production <repo_url>

Git documentation: https://git-scm.com/docs/git-clone : search for depth

Example use: https://stackoverflow.com/questions/30001304/clone-git-repository-without-history/30001366

#65 : Similar issue, but now with a relevant use case

[akitty]

libgit2 and rugged, the underlying git library r10k uses doesnt support shallow clones. So this will not be an easy change.

[daruvin]

Considering the performance hit, 4 min << 60 mins, I think it would be a good reason to make the necessary changes. As the directories grow, the problem will keep getting bigger.

[radupantiru]

This would be very useful indeed.

[binford2k]

it looks like libgit might be implementing shallow clones soon, libgit2/libgit2#3058

[github-actions]

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

[halradaideh]

it is implemented here libgit2/libgit2#3058
@akitty can we expect it on r10k?

[binford2k]

Shallow cloning is now possible with libgit. We should implement it.

[github-actions]

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

[kjetilho]

Although libgit2 now has support for shallow copies, the Ruby binding Rugged does not. This is issue libgit2/rugged#295