SLEPIC: Issues with Winrepo

[twangboy]

Description of Issue/Question

There are many issues that need to be addressed in winrepo

• How to handle auto updating software (don't downgrade if a newer version is installed unless you want to force it). It seems to happen when there's a newer version that is not defined in the software definition.
• Ability to use a .msp file when installing an .msi

Also look at overriding from a state file: saltstack/salt-winrepo#813 (comment)

Option to remove out of date software (previous versions)
saltstack/salt-winrepo-ng#398
saltstack/salt-winrepo-ng#350
saltstack/salt-winrepo-ng#336
saltstack/salt-winrepo-ng#144

[morganwillcock]

I've spent the last couple of weeks packaging software, the biggest problem I've found so far is incorrect detection of state based on name collisions. As an example, Visual C++ 2005 runtime packages will install together, but state checks will fail based on the identical full_name:

salt-winrepo-ng$ grep -h full_name ms-vcpp-2005*.sls
    full_name: 'Microsoft Visual C++ 2005 Redistributable (x64)'
    full_name: 'Microsoft Visual C++ 2005 Redistributable'
    full_name: 'Microsoft Visual C++ 2005 Redistributable (x64)'
    full_name: 'Microsoft Visual C++ 2005 Redistributable'
    full_name: 'Microsoft Visual C++ 2005 Redistributable (x64)'
    full_name: 'Microsoft Visual C++ 2005 Redistributable'
    full_name: 'Microsoft Visual C++ 2005 Redistributable (x64)'
    full_name: 'Microsoft Visual C++ 2005 Redistributable'

I've found that I can work around the downgrade issue with state dependencies, effectively pinning package versions at a minimum level:

adobe_flash_player_plugin:
  pkg.installed:
    - version: '21.0.0.213'
    - watch:
      - module: adobe_flash_player_plugin_upgrade_available

adobe_flash_player_plugin_upgrade_available:
  module.run:
    - name: pkg.upgrade_available
    - m_name: adobe_flash_player_plugin

For packages upgrades that use a patch file, perhaps the best option would be to allow the cache path to be set manually (to a file or a directory) rather than always inferred from the installer path. Letting you control the caching yourself would let you do whatever you like for any type of installer and I think this would solve a lot of issues where caching is conflicting with install options (while still retaining backwards compatibility):

installer: 'msiexec.exe'
install_flags: '/p MyApp553.msp /qn /norestart'
msiexec: False
cache_dir: True
cache_path: 'salt://win/repo-ng/my_app'

installer: 'msiexec.exe'
install_flags: '/p MyApp553.msp /qn /norestart'
msiexec: False
cache_dir: False
cache_path: 'salt://win/repo-ng/my_app/MyApp553.msp'

Apologies for the lengthy comment...

[damon-atkins]

The pkg manage updates I am doing , have been always striping the version string out of the name. I will add code so that if the package needs the version string like the above C++ examples it will add version missing string to the name, this way you will see all of them installed instead of just one being reported back.

e.g. Microsoft Visual C++ 2005 Redistributable (x64) will become
Microsoft Visual C++ 2005 Redistributable (x64) - 6.0.30729.17

[TheBigBear]

@twangboy what does 'SLEPIC' stand for?

By now I have quite a (mental) long collection of issues with winrepo-ng, is this issue here an attempt to list them in one place and pull these things together?

Or is it only a certain 'SLEPIC' (whatever that means) related category of issues ?

Do you know if the conversations I had with @UtahDave that feels like about a year or so ago but was less in reality, it was actually as "recent" as in the runn-up to releasing 2015.8 with jinja2 winrepo-ng support he was thinking at the time about starting a 'proper' discussion on the future of winrepo-ng and how to add some of the missing pieces etc etc.

Is this conversation ever going to actually happen ...?

[TheBigBear]

@damon-atkins sorry but I can't follow your logic understanding why you want to or need to remove the version numbers from the windows pkg_fullname anywhere. can you explain to me why you think that would be good or what you hope (or need to) gain by this?

[twangboy]

@TheBigBear Git hub started doing epics so I have been trying it out. To keep my epics from getting lost in the other epics, I prefixed all mine with SL (Shane Lee), hence the SLEPIC.

Yeah, I'm trying to group together all the issues. It just seems they keep coming in and I don't want to lose them between now and the time I can actually start working winrepo again.

[TheBigBear]

@twangboy so should I add mine to this list or keep a list of my own ones and at the right time point you to it, or do one link here to my list on winrepo issues ? what would you prefer?
But actually I think we already have at least two (if not more) similar starts of collecting them all in one place, should I try to hunt them all down and link them to this issue here? is that the best place? There are others that have made valueable feedback and started collecting them like @eliasp and others that I can't readily remember, but with a bit of effort on my gray cells could probably track it all down.
Any rough time frame - ball park fiugre only, as to when you might be able or free to work on winrepo and start to get a roadmap or plan of action hashed out?

[twangboy]

@TheBigBear That would be great. Hopefully we can address as much as possible. As far as a time frame here, we will be talking to our project manager to get a handle on priorities and a roadmap. The trick is going to be implementing new features without breaking it for older minions. I don't want to have to create a salt-winrepo-ng-v2. :)

[lorengordon]

please, no, not salt-winrepo-ng-v2! 😛

[damon-atkins]

Version are not required as part of the application name, because they are an attribute not the title. There are a few exceptions such as C++ runtime... As per unix you do not place the version in the name, its an attribute of what version you want to install.

[eliasp]

@twangboy

How to handle auto updating software (don't downgrade if a newer version is installed unless you want to force it). It seems to happen when there's a newer version that is not defined in the software definition.

That should be covered since ~2 years by #12711

[lorengordon]

Oh! I just installed the zenhub.io extension for Chrome. Now I get the epic thing. Nifty!

[morganwillcock]

@eliasp I can see that this would work where the package display name stays the same on upgrade, but if the name changes isn't it going to be seen as a completely different application? I think where installers are known to be inconstant there needs some more generic way of identifying them, maybe options to regex match the name and version, as well as yield control of it? I installed an Adobe Air application today and the version number of the underlying MSI package actually went backwards on upgrade...

[TheBigBear]

please, no, not salt-winrepo-ng-v2!

I agree and understand, but it might be a tad early to promise that, I think.
Probably best to first list the items that were on the list of what a windows pkg manager needs and then add the things we discovered since and then try to work out how this can be done .... , right?

[twangboy]

@TheBigBear We are want to avoid a v2. Everything that happens within the .sls file must be compatible with win_pkg as it stands now.

[morganwillcock]

Sorry to add to the pile, but I think I found the issue that made the version number go backwards...
#33357

[damon-atkins]

So the code I have written is different, but works with existing, as long as you do not manage the same item with both. The only way to solve the problems with existing is to change it......
It also tries to check the contents of the yaml to point out issues in a friendly manner.

e.g. You should not have to edit the file every-time a new version of the software comes out. This is the number 1 priority of the code I am writing. esp. salt/http/etc source has a version number.

[morganwillcock]

#35943
I think this is straightforward to fix, but I thought I should reference it here as it might need a new parameter to be added.

[crashmaster18]

@damon-atkins @morganwillcock "You should not have to edit the file every-time a new version of the software comes out." Exactly! I love the local cache idea/option documented as well. We actually do that here without using salt (we have our own scripts that automatically detect and download the latest versions of windows programs). So we are just starting with saltstack here (we are on the 2016.3.4). It seems that to automate this completely with 2016.3.4 we need to take our downloaded local/cache file://application.msi path, update the application.sls with the corrected local location then run salt '*' pkg.refresh_db. My gut to achieve this for each .sls would be to use simple bash scripting, but I suspect there's a better way that would help other salt users accomplish the same (or perhaps not?). We are very interested in learning how we can help with salt/winrepo to achieve this...

[damon-atkins]

What I was hoping to do is have a slp (Salt Package) file. The file describes how to install the software but also describes how to download it to the salt master (or maybe something else) This is on the TODO list.

Because direct download from the internet onto server is not really best practice and the github sls all do this. And so makes it a bit hard to use the sls "as is".

Currently the sls files are processed once as a template before creating winrepo.

The slp file is effectively stored as a mini template within the winrepo (not template processed first like sls), and when a install or remove is action the version string is provided and the mini template is process on the fly to generate the final yaml optionally using the version passed from pkg.install. So no need to list every version in the file. When the refresh_db it does process the slp as a template but only to check that the slp file is valid with a fake version string (if valid its added to the winrepo [different structure to the existing sls files so they can exist at the same time]) Also does not need a sls or slp definition to give a consistent package name between versions.

Also pkg.list_pkgs output is more like yum version of pkg.list_pkgs, and supports software reporting of software installed by users . It also detects more software and removes the existing hard coded ignoring of few software titles and also tries to detect the installer software (e.g. msi, install shield, etc) I spent a lot of time working out how the software is recorded in the registry and looking at how different installers set values in the registry. (It's painful)

Some of the recent change I did in 2016.11 release was to allow this code to be added with small change to existing code. So that merging forward changes for 2016.11 releases would still be possible and clean.

Need to find the time to merge it into the 2016.11 code which is structure different to previous release I developed the code for.

It's a very very large change I suspect it will be a year before it makes it into a release as anything other than experimental. Assume saltstack accepts the code.

[damon-atkins]

Some of the above has changed.
refresh_db validates the information in the slp. winrepo only records enough information to determine a) the unique package name using regex and b) version number (choice of display name, display version or binary version). The slp template is render again at install and remove time allow the template to be rendered with current information.

[damon-atkins]

Can auto detect/scan available version of software from http/https/ftp/salt:// sources