Enhancement: Perform the Great Fork Merge

[lcn2]

The Great Fork Merge

The Great Fork Merge will occur when the multi-thousand commits that the temp-test-ioccc repo is ahead of the Official IOCCC winner repo are bright back to main Official www.ioccc.org web site.

TODOs

In order to perform the Great Fork Merge, the following tasks need to be completed:

• Revise status.json format.

• Write a new tool to generate the new top level status.html from status.json

NOTE: This may require a rewrite of the bin/ioccc-status.sh tool as well change the format of the from status.json file.

• Fix name www under RHEL 9

Port the bin tools tools to run under RHEL 9 version of Linux. With this port, the IOCCC judges should be able to use these tools on a wide enough variety of systems for their purposes.

• Complete issue Enhancement: Address XXX lines in top level files #2008

• Complete issue Enhancement: improve the consistency of winning directories #3

• Verify that the format of markdown files is good

See also comment 1993767060.

• Complete issue Enhancement: add templates for issue forms #858

• Complete issue Enhancement: Improve the IOCCC manifest #1933

• Move all FAQ content into their own menu section

See comment 9192931.

• Verify that the top level README.md contains suitable content for a GitHub repo README file.

The exception, of course, is the initial text down to the XXX that indicates this is a test repo.

• Complete issue Enhancement: improve markdown links to FAQ entries #2417

• Complete issue Enhancement: improve the look and feel of the constructed HTML files #2006

Improve the look and feel of the constructed HTML files.

• Complete issue Enhancement: Reorganize the FAQ #2682

Reorganize the FAQ.

• Complete issue #979 in the mkiocccentry repo

• Complete issue #955 in the mkiocccentry repo

• Complete issue #956 in the mkiocccentry repo

Prepare the mkiocccentry repo for a code freeze and release.

• Reformat JSON indented with multiples of 4 ASCII spaces and no TABs

See also GH-issue 931 from that other repo.

NOTE: JSONPath.sh with only -S -A and without -T may be helpful with this coversion.

NOTE: The bin/jprint-wrapper.sh tool should also be updated.

• Complete GH-issues/931 from the mkiocccentry repo

This will be in keeping with the completed TODO the JSON indented with multiples of 4 ASCII spaces and no TABs.

• Review and improve bin/README.md

Now that man pages for bin/ tools (issue #2009) are not planned, we need to review and likely improve the bin/README.md documentation.

We need to add some notes about these recent additions:

• bin/all-jfmt.sh
• bin/combine_author_handle.sh
• bin/jval-wrapper.sh
• bin/manifest.csv.entry.awk
• bin/manifest.entry.json.awk

• Create a new issue for recovery of very old entries in news.md

Issue #2686 was created to satisfy this TODO item.

• Build bin tools making manifest.numbers no longer necessary and make .entry.json the primary "entry data truth"

See comment-2189364148.

• Add an FAQ about how to handle de-obfuscated code

See comment 919283.

See comment-2198635072.

This TODO was changed from referring to a "spoiler" into a "de-obfucated" educational emphasis. In that regard we should NOT use terms like "spoiler" in the FAQ as well as NOT using "spoiler" in the entry's README.md file.

At a most, README.md should suggest that the reader might wish to study the prog.c code first before going on to review the de-obfuscated "alt" code.

See also comment-2198807715.

After completing the FAQ entry how to handle de-obfuscated code, consider if needed, revising entries with existing de-obfuscated code AND if needed, fixing cases where the entry README.md makes reference to something being a spoiler.

• While OT, create an issue for a that performs JSON semantical checks on the .entry.json files in mkioccentey repo

See comment-2172105426 and see comment-2174364721.

• While OT, fork the jparse/ directory from mkioccentey repo tree into @xexyl's jparse tree

See comment-2197371351.

• Resolve issue [Bug] bin tools are using jstrencode(1) when they should be decoding JSON encoded strings #2752

• Create an initial text for details of the registration process as a new FAQ 1.4

Add a "see also" link to the new FAQ 1.4 in FAQ 0.0 subsection 2.

If the registration process is not "screenshot ready" by the time this TODO is addressed, put in a "stay tuned for images" placeholder into this new FAQ.

• Create an initial text for details of the submit process as a new FAQ 1.5

Add a "see also" link to the new FAQ 1.4 in FAQ 0.0 subsection 5.

If the ** submit process** is not "screenshot ready" by the time this TODO is addressed, put in a "stay tuned for images" placeholder into this new FAQ.

• Carefully reread and review the next rules and guidelines

• Carefully reread and review the FAQ

In addition to the useful check for typos, wording, and broken links: review for suitability for gong live with on the main web site after the Great Fork Merge. Update faq.md if and as needed.

• Cleanup from 2024 testing

We need to cleanup from testing tools, so we need to remove:

• 2024 from .gitignore
• 2024 from .top
• 2024 from Makefile

These may have added as part of tool testing and need to be removed before we proceed along the Great Fork Merge process below.

• Inspect and sanity check the remaining TODO items

Add, modify and/or remove things from the remaining TODOs if/as needed.

• Determine that we are ready to perform the Great Fork Merge

• Announce a code freeze

In a comment in this issue #2239, announcers that general pull requests for the temp-test-ioccc repo will no be accepted and that we are beginning work on the Near Final TODOs.

Getting ready for the Near Final TODOs

NOTE: Once the above TODOs have been completed, the following last minute TODO actions must be completed in relatively short order (preferably in the same calendar day) before the Great Fork Merge takes place.

• Remove all trailing whitespace from markdown files

Use this to look for any whitespace from markdown files

find . -name '*.md' -print0 | xargs -0 grep -E -l '[[:space:]][[:space:]]*$'

• Verify the markdown code blocks are properly indented

Use:

find . -name '*.md' ! -name markdown.md -print0 | xargs -0 pcregrep -M '^``.*\n[^|`\n \t]'

• Verify that the Nu Html Checker finds no errors, warnings nor info messages for all generated HTML

After making sure that the temp-test-ioccc repo is up to date and related GitHub pages have been rendered, use the ✓ on the navbar to check all generated HTML pages.

Fix any errors, warnings and info messages reported. Update the temp-test-ioccc repo and recheck those pages.

• Update FAQ 1.2 with there date of Great Fork Merge

In FAQ 1.2, look for the <!-- XXX - Fill in the date when Great Fork Merge happens --> and update the date as for that section as needed.

• Change bin tools to use www.ioccc.org

Change default URLs and REPOs to refer to the https://www.ioccc.org web site and winner repo.

• Change references to use www.ioccc.org instead of temp-test-ioccc

Change references as needed given their context.

This includes markdown files, HTML files, var.mk, Makefiles, text files, etc.

Try:

make www
find . \( -name .git -o -name NOTES \) -prune -o -type f -print0 | xargs -0 grep -l -F temp-test-ioccc

Also change the comment in var.mk

• In mkiocccentry repo, Change references to use www.ioccc.org instead of temp-test-ioccc

See GH-#issuecomment-2403175498.

• Re-release the mkiocccentry repo

• Rebuild web tree via make www

• Scan repo to verify that references to temp-test-ioccc have been processed.

The only exceptions should be historic references found in news.md, news.hml, faq.md and faq.html.

• Add news event to news.md announcing the new www.ioccc.org web site.

Edit news.md as needed.

Perform make www and commit any changes.

Near Final TODOs

NOTE: Once the above TODOs have been completed, the following last minute TODO actions must be completed in relatively short order (preferably in the same calendar day) before the Great Fork Merge takes place.

• Verify that the installed location(1) tool is up to date

Be sure the mkiocccentry repo is up to date and perform a make install in that typo.

• Purge repo of previous *.tar.bz2 files and reduce Git repository size

See GH-issuecomment-2430718370 for a summary of the test process.

We will perform the following actions:

git mv 2011/dlowe/dlowe-aux-data.tar.bz2 HOLD
git commit -m'move dlowe-aux-data.tar.bz2 to HOLD'

git remote -v
git filter-repo --path-glob '????/*.tar.bz2' --invert-paths --force
git remote add origin [email protected]:lcn2/trim-temp-test-ioccc.git

git mv HOLD 2011/dlowe/dlowe-aux-data.tar.bz2
git commit -m'move HOLD back to dlowe-aux-data.tar.bz2'

for year in [12][0-9][0-9][0-9]; do
  touch "$year/$year.tar.bz2"; git add "$year/$year.tar.bz2";
done

for year in [12][0-9][0-9][0-9]; do
  for dir in $(< "$year/.year"); do
    entry_id=${dir/\//_};
    touch "$dir/$entry_id.tar.bz2";
    git add "$dir/$entry_id.tar.bz2";
  done;
done

git commit -m'add empty tar.bz2 files until we do make update'

git repack
git prune-packed
git reflog expire --expire=now --all
git gc --aggressive --prune=now
git fsck --full

git push --all --force
git push --tags --force
git push --set-upstream origin master

FYI: See:

• GitHub notes on Removing sensitive data from a repository
• comment-2159159871
• stack overflow replies
• How to clean up the git repo and reduce its disk size
• GH-issuecomment-2430718370

NOTE: The make update in a TODO farther below will add back the needed *.tar.bz2 files.

NOTE: The make gen_years will generate Warnings of the form: "bin/gen-years.sh: Warning: compressed tarball for YYYY: yyyy is empty: yyyy/yyyy.tar.bz2".

This is OK and will be corrected once make update or make tar is done.

• Perform make update on macOS and push out any changes

• Verify that running make update again on macOS changes only:

• sitemap.xml
• status.html
• status.json

If so, then revert those changes by:

git restore sitemap.xml status.html status.json

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

• Verify make www on RHEL 9 changes nothing

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

• update README.md and index.md

Remove the lines between:

<!-- XXX - This entire section goes away during the final stages of the Great Fork Merge -->

and:

<!-- XXX - remove down to here in the final stages of the Great Fork Merge -->

• Verify that the temp-test-ioccc repo is clean

Verify that:

git status

reports:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

• Verify that the temp-test-ioccc repo is up to date

Fetch any last minute changes:

cd docroot/temp-test-iocc

git fetch && git rebase

git clean -f

git status

The last command should report:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

• Perform make update on macOS

• Commit all changes and push the result to the temp-test-ioccc repo

git commit -m'final pre-Great Fork Merge'

git push

• Re-Verify that the temp-test-ioccc repo is up to date

Fetch any last minute changes:

cd docroot/temp-test-iocc

git fetch && git rebase

git clean -f

git status

The last command should report:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

• Re-Verify make update on macOS changes * [ ] Verify that running make update again on macOS changes only:

• sitemap.xml
• status.html
• status.json

If so, then revert those changes by:

git restore sitemap.xml status.html status.json

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

• Re-Verify make www on RHEL 9 changes nothing

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

FINAL REVIEW

• Pause and ask for a review of readiness

We expect this period to be between 2024 November 1 and 2024 November 15.

Please review the effect of all of the above completed to do. Please limit PRs to only critical problems, and broken links, and linked to the wrong file, and failure to the official website and repo, etc.

• Incorporate any PRs as a result of the previously mentioned review

Accept any such PRs, if needed modify the result after accepting, and finally do a make update.

Great Fork Merge

• Perform the Great Fork Merge

Click Create Pull Request for the Official IOCCC winner repo.

Inspect the pull request for the Official IOCCC winner repo.

Accept and complete the pull request.

Post Great Fork Merge TODOs

Once the Great Fork Merge occurs and the official IOCCC winner repo and related Official www.ioccc.org web site has been updated, these TODOs need to be performed on the official IOCCC winner repo:

• Copy issue Enhancement: recover old and historic news items and make them available via news.md #2686 into the official IOCCC winner repo

Close down issue #2686 as "won't fix" in this repo.

• Close issue Question: Issues that aren’t really major issues but are still issues #5

• Transfer any remaining issues to the official IOCCC winner repo

• Remove the website associated with the temp-test-ioccc repo

• Verify the official IOCCC winner website links work

Fix any problems found on the Official www.ioccc.org web site by editing the official IOCCC winner repo as needed.

• Announce on Mastodon that the Great Fork Merge has happened and the official IOCCC web site has been updated

• Mark the temp-test-ioccc repo read-only

From temp-test-ioccc repo settings click on the ((Archive this repository)) button.

This will Freeze the temp-test-ioccc repo but leave it in place for historic purposes.

[xexyl]

Please assign this to me.

And as I noted before: if needed I would like to update my script that updates the status.json file.

Thanks.

I will have to look at this issue later on today as it seems rather important!

[xexyl]

Also I am not sure if I can finish the year README files before the 10th but I should be able to finish them sometime soon anyway.

But for now I must go do other things again. Back in a while.

[xexyl]

In order to perform the Great Fork Merge, the following tasks need to be completed:

• Revise status.json format.
• Write a new tool to generate the new top level status.html from status.json

NOTE: This may require a rewrite of the bin/ioccc-status.sh tool as well change the format of the from status.json file.

It's funny you should say that .. the generating the file. Do you mean creating the file rather than modifying it? I actually had the thought of adding an option to the script to do that and I seem to recall that I almost did do it. It would be easy to do. But since the format is not necessarily the final format I will hold off on that.

• Complete issue Enhancement: improve the consistency of winning directories #3

The anticipated completion date for this issue is 2024 March 10 2:30 (2:30AM) Pacific.

Like I said over there mostly it is done: except for the YYYY/README.md files. But I can do those after 10 March if necessary so I have no problem with it being closed. You were going to take care of the silencing the warnings and right now would be fine as I don't anticipate modifying any Makefiles or if I'm honest for now anything at all (except maybe the FAQ which you asked me to look at). That probably will hold for a day or two at least but even if I do make other changes it'll likely only be the YYYY/README.md files. I say I might not finish it before 10 March but I might yet: I just have to have the energy and time and the former seems to be a problem lately I'm afraid. Even so I might be able to do some of it today: that remains to be seen however.

• Complete issue Enhancement: add templates for issue forms #858

Okay in order to do this I think I even created a new repo to test how this works. I think the other issues should come first though. Is that correct?

• Complete issue Enhancement: Improve the IOCCC manifest #1933

Ah yes ... I forgot. I have to update descriptions and also check the html files! It's been very hectic and I totally forgot about this. I was also holding off as you were planning many changes that would conflict. Those changes are now done so I can look at this sometime soon too. If I don't please remind me.

• Complete issue Enhancement: improve the look and feel of the constructed index.html files #2006

Right ... well this one might be good to look at when I look at the manifest too. After all the result of the manifest is in the index.html files.

• Complete issue Enhancement: Address XXX lines in top level files #2008

Well some of this cannot be done until the submit server is ready, right?

Near Final TODOs

Once the above TODOs have been completed, the following last minute TODO actions must be completed in relatively short order before the Great Fork Merge takes place:

• Change HTML references to use www.ioccc.org

In the (few?) cases where a absolute URL is required, change that URL to be under https://www.ioccc.org and NOT this repo's top level URL.

sgit(1) should be useful for this one! :)

• Add news event announcing the new www.ioccc.org web site.

Edit news.md as needed.

That's a great idea .. looking forward to see what you have in mind! Almost eager enough to try and get myself to work on YYYY/README.md files today but I think that might be a mistake with how tired I feel :(

• Verify that the installed location(1) tool is up to date

How do you mean verify that it's up to date? Do you mean locally installed? But what version should it be then? Did you have other updates in mind? (I in any case have the most recent version at the other repo.) Then again perhaps this is something you have to do so I might not have to worry about it. Or maybe a tool here uses it and I do have to worry about it. But as noted I do have them installed so it's fine.

• Verify manifest.numbers agrees with the local temp-test-ioccc repo

How do you mean verify that it agrees with it when it's in the repo? Though again I guess this might have to be something you do so I don't need to worry about it: updating the manifest I do have to worry about but perhaps not this step?

Perform the 12 steps as noted in comment 2089758002.

Don't forget the script I wrote! It's very useful. By default it does not commit automatically but there is an option to allow it to do so. It even asks if you're on the right branch.

• Remove tmp directory.

Ever notice what tmp spells backwards? :-)) Well okay it is a British English abbreviation but still :-)

Post Great Fork Merge TODOs

Once the Great Fork Merge occurs and the official IOCCC winner repo and related Official www.ioccc.org web site has been updated, these TODOs need to be performed on the official IOCCC winner repo:

• Close issue Question: Issues that aren’t really major issues but are still issues #5

I've enjoyed this one though I must say! A lot of good things, a lot of fun, a lot of laughs and stories and other things ... thanks for the privilege and honour!

• Transfer any remaining issues to the official IOCCC winner repo

Good idea.

• Mark the temp-test-ioccc repo read-only

From temp-test-ioccc repo settings click on the ((Archive this repository)) button.

THANK YOU for going for that (as I suggested) rather than deleting it.

• Verify site links work
  Ah yes .. that'll be fun! :-)

Process TBD.

Fix any problems found as needed.

Of course.

At the other repo I left a comment regarding an issue I have (about finishing something we talked about the other day) but I think that and this comment is probably all I'll manage to do today for the contest. But who knows. I might be able to take a look at some things at this repo too. For now I'll be afk (or I intend to be afk soon). I think today though I will mostly be reading when not doing the things I have to do.

[lcn2]

Incomplete TODO

NOTICE: This TODO list will be built over a few days. Until this notice is removed, consider this list to be incomplete.

:-)

=-=

• Revise status.json format.
• Write a new tool to generate the new top level status.html from status.json

NOTE: This may require a rewrite of the bin/ioccc-status.sh tool as well change the format of the from status.json file.
It's funny you should say that .. the generating the file. Do you mean creating the file rather than modifying it?

The content of status.json is still TBD, and so is the tool that will mange it.

=-=

The anticipate completion date for this issue is 2024 March 10 2:30 (2:30AM) Pacific.

Like I said over there mostly it is done: except for the YYYY/README.md files. But I can do those after 10 March if necessary so I have no problem with it being closed.

Sure.

BTW: issue #2006 does indicate that it covers such files, and the title has been changed to say "constructed HTML files".

[lcn2]

• Complete issue Enhancement: add templates for issue forms #858

Okay in order to do this I think I even created a new repo to test how this works. I think the other issues should come first though. Is that correct?

It is a ((*top priority)) issue and has been for a while. It pertains to something that is much needed after the Great Fork Merge when people start considering opening up issues on the revised web site.

=-=

• Complete issue Enhancement: Address XXX lines in top level files #2008

Well some of this cannot be done until the submit server is ready, right?

We will need to put in some placeholder text that indicates the exact process has not been created, but will be before the IOCCCMOCK is held.

=-=

• Verify that the installed location(1) tool is up to date

How do you mean verify that it's up to date? Do you mean locally installed? But what version should it be then? Did you have other updates in mind? (I in any case have the most recent version at the other repo.) Then again perhaps this is something you have to do so I might not have to worry about it. Or maybe a tool here uses it and I do have to worry about it. But as noted I do have them installed so it's fine.

As we need to be the one be performing these last steps, you @xexyl do not need to worry about the details.

=-=

• Remove tmp directory.

Ever notice what tmp spells backwards? :-)) Well okay it is a British English abbreviation but still :-)

:-)

[xexyl]

• Complete issue Enhancement: add templates for issue forms #858

Okay in order to do this I think I even created a new repo to test how this works. I think the other issues should come first though. Is that correct?

It is a ((*top priority)) issue and has been for a while. It pertains to something that is much needed after the Great Fork Merge when people start considering opening up issues on the revised web site.

Let me reword that. Besides #3 (and I hope I can look at some of it today in a while but we shall see) do you have an order of priority for the remaining issues or should it be 'whatever I have the time and energy and inclination for'?

That idea of whatever I can work on would be useful but if you have an order I could try and work with it.

=-=

• Complete issue Enhancement: Address XXX lines in top level files #2008

Well some of this cannot be done until the submit server is ready, right?

We will need to put in some placeholder text that indicates the exact process has not been created, but will be before the IOCCCMOCK is held.

That all makes sense to me yes. Thanks.

=-=

• Verify that the installed location(1) tool is up to date

How do you mean verify that it's up to date? Do you mean locally installed? But what version should it be then? Did you have other updates in mind? (I in any case have the most recent version at the other repo.) Then again perhaps this is something you have to do so I might not have to worry about it. Or maybe a tool here uses it and I do have to worry about it. But as noted I do have them installed so it's fine.

As we need to be the one be performing these last steps, you @xexyl do not need to worry about the details.

I guessed that but I wanted to be sure.

=-=

• Remove tmp directory.

Ever notice what tmp spells backwards? :-)) Well okay it is a British English abbreviation but still :-)

:-)

:-) :-)

[lcn2]

Let me reword that. Besides #3 (and I hope I can look at some of it today in a while but we shall see) do you have an order of priority for the remaining issues or should it be 'whatever I have the time and energy and inclination for'?

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .

We home this clarifies @xexyl .

We will be mostly offline for about 2-3 days working on issue #2008 as well as prep actions for this issue.

[xexyl]

Let me reword that. Besides #3 (and I hope I can look at some of it today in a while but we shall see) do you have an order of priority for the remaining issues or should it be 'whatever I have the time and energy and inclination for'?

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .

I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

We home this clarifies @xexyl .

Hopefully :-) it does .. yes it does - thanks!

We will be mostly offline for about 2-3 days working on issue #2008 as well as prep actions for this issue.

Best wishes! Today I'm not doing anything for the contest I'm afraid. Having a very off day. Tomorrow i'm sure will be better.

[lcn2]

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .
I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

[xexyl]

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .
I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

Thanks. I want to be sure of something before I finish my part of #3 which I'll ask you there. My part left in that is minimal except of course for checking typos etc. in the FAQ and other like files so that's good. But I won't be doing anything today. Depending on when you get back to me on the concern and the answer too I hopefully can start tomorrow.

[lcn2]

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .
I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

Thanks. I want to be sure of something before I finish my part of #3 which I'll ask you there. My part left in that is minimal except of course for checking typos etc. in the FAQ and other like files so that's good. But I won't be doing anything today. Depending on when you get back to me on the concern and the answer too I hopefully can start tomorrow.

Best wishes, @xexyl

[xexyl]

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .
I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

Thanks. I want to be sure of something before I finish my part of #3 which I'll ask you there. My part left in that is minimal except of course for checking typos etc. in the FAQ and other like files so that's good. But I won't be doing anything today. Depending on when you get back to me on the concern and the answer too I hopefully can start tomorrow.

Best wishes, @xexyl

Should be good: I'm going to soon get back to LR! And you too!

[lcn2]

With commit 449205e the new status.json model is complete.

The first 2 TODO items of this issue #2239 have been completed.

[lcn2]

We will next focus on issue #2008 as that issue is somewhat independent of the other ((Top
Priority)) issues.

[lcn2]

We will next focus on issue #2008 as that issue is somewhat independent of the other ((Top Priority)) issues.

Issue #2008, as per comment 1987654095 and commit f482380 has been completed.

[lcn2]

We made more tweaks to the TODO list for this issue.

We think the TODO list is nearly ready.

UPDATE 0

Comments and corrections welcome.

[lcn2]

We believe we have addressed all of the current questions that still need answering at this time. If we've missed something or something else needs to be clarified, please ask again.

[xexyl]

We believe we have addressed all of the current questions that still need answering at this time. If we've missed something or something else needs to be clarified, please ask again.

I will have to get back to this tomorrow and determine that. I'm sure you did though.

[lcn2]

With commit 4471875 the bin tools have been ported to RHEL 9.

The make quick_www and make www should work well under Linux, assuming that up to date critical tools such as location(1), jparse(1) are installed and assuming that pandoc(1) has ben installed with a minimum 3.1.12.2 version, and when bash(1) found by #!/usr/bin/env bash is version 5.1.8 or later.

UPDATE 0

The TODO list has been updated accordingly.

[xexyl]

With commit 4471875 the bin tools have been ported to RHEL 9.

The make quick_www and make www should work well under Linux, assuming that up to date critical tools such as location(1), jparse(1) are installed and assuming that pandoc(1) has ben installed with a minimum 3.1.12.2 version, and when bash(1) found by #!/usr/bin/env bash is version 5.1.8 or later.

UPDATE 0

The TODO list has been updated accordingly.

Well now I ran into a new problem. Don't you use pandoc from Homebrew? If so: I updated it yesterday and it's not a high enough version for running the make www. Where do you get it from then? Thanks.

[xexyl]

With commit 4471875 the bin tools have been ported to RHEL 9.
The make quick_www and make www should work well under Linux, assuming that up to date critical tools such as location(1), jparse(1) are installed and assuming that pandoc(1) has ben installed with a minimum 3.1.12.2 version, and when bash(1) found by #!/usr/bin/env bash is version 5.1.8 or later.

UPDATE 0

The TODO list has been updated accordingly.

Well now I ran into a new problem. Don't you use pandoc from Homebrew? If so: I updated it yesterday and it's not a high enough version for running the make www. Where do you get it from then? Thanks.

Oh I know why. I have it from MacPorts too. My guess is it'll be a problem to remove it so I have to maybe put the Homebrew path before the MacPorts path.

UPDATE 0

All good it seems. Thanks. I can now look at the comments in the other issue and hopefully get them done in the next few days. Right now though I must go again. It is unclear to me yet if I can do much today or anything at all but at least last night was better than the previous three nights! Hopefully that continues.

[lcn2]

Priorities

Regarding the top priory open issues:

• top: issue Enhancement: Improve the IOCCC manifest #1933
• next to top: issue Enhancement: improve the look and feel of the constructed HTML files #2006

While going thru the IOCCC manifest, reviewing the data fields from the manifest.numers spreadsheet:

• inventory_order
• OK_to_edit
• display_as
• display_via_github
• entry_text

Look at each entry's inventory, as found in index.html#inventory to see how the contents are being expressed. Look at the sorted lines in the manifest.numers spreadsheet that relate to the entry in question.

Because issue #1933 and issue #2006 are somewhat intertwined @xexyl, we suggest you process both them at the same time. Like you did for issue #3, you can announce your progress on an IOCCC year by YEAR basis.

Regarding issue #2006

And since one is looking a an entry's inventory, give a glance over the rest of the index.html page.

Here are the types of questions, @xexyl, to ask yourself, does the index.html page contain:

• Some formatting that looks obnoxious?
• Information displayed in way that is difficult to read?
• Have too much to too little whitespace that makes the web page awkward to read?

We are NOT focusing on the accuracy of the index.html page per se (although if you see something glaring, feel free to fix it under the guide of something that slipped passed the issue #3 effort). We are focusing on HOW it LOOKS with an emphasis on correcting dysfunctional presentations.

In nearly all cases, corrections can be addressed by editing the entry's README.md file. Please modify the entry's README.md file and re-build the entry's index.html in a pull request.

If, however, you see an issue that may be more a matter of changing ioccc.css or the related bin tool, please raise the matter as a comment in this issue #2239 so we can discuss the matter before editing the CSS and/or bin tool. Again, this is less likely to happen, but if it does, we recommend a comment related discussion first. Modification to the entry's README.md file can be done directly via pull requests.

Regarding issue #1933

Here are the types of questions, @xexyl, to ask yourself:

• Are the brief (59 characters or less) descriptions accurate or at least somewhat useful?

See the entry_text field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

• Is the order listed reasonable?

See the inventory_order field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

• The the more useful files to look at in the Primary files section? (see inventory_order)

See the inventory_order field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

• Are files that are built by bin tools marked as false for _ OK_to_edit_?

See the OK_to_edit field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

• Are files that marked as true or _ OK_to_edit_ files not created by bin tools nor Makefile rules?

See the OK_to_edit field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

• Is the general type of file consistent across similar classes of file types?

See the display_as field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

• Do files that need to be dowloaded (instead of viewed), downloading in a browser?

See the display_via_github field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

• Do files that need to be viewed (such as C code, Makefiles, etc.) via GitHub going to the GitHub site?

See the display_via_github field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

• Are files that we OK to be viewed directly in a browser (such as images, photos, etc.) workin in the browser?

See the display_via_github field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the Manifest fields section of the top TODO comment for issue #1933 for more information.

Comments, Suggestions as Questions welcome

Any Comments, Suggestions as Questions you may have about this comment are welcome.

[lcn2]

As per comment 2010767087, this TODO item has been marked complete:

• Verify that the format of markdown files is good

[xexyl]

Yes I would be happy to help out.

Just let me know Leo. I actually thought about emailing you about that bug since it's your entry and maybe you have a way to fix it for modern systems. I managed to fix a lot and Yusuke fixed some too. Landon also fixed some.

But I have been busy with this and other things so I never got to it.

Anyway if you do take a look and resolve the issue if anything needs to be done after the fact I can take care of it. For instance if you fix a bug I can update the appropriate files.

Pretty sure a bug that prevented something from working right was discovered. Not looking at the file now. If this seems wrong I apologise: been a hard long day.

Nonetheless whatever you need help with I would be happy to help as I am quite familiar with the way it all works.

Hope you have a great night. Off to get some sleep shortly.

[xexyl]

FYI about jstrdecode/jstrencode (with a QUESTION)

I just pushed out the change of the swap of names as the terms were wrong, it seems. The new version of the tools is "2.0.0 2024-10-31". In order to be able to sync this to mkiocccentry it requires some code changes which I have already done in my fork. But since this is a major change if you do not want this done then you certainly do not have to sync it until after the merge or maybe even IOCCC28.

If however you do want it done, first of all please advise me, and second of all this means that the scripts that rely on it, here, will have to be updated to use the correct tool and version, which I am happy to do.

QUESTION

What do you prefer?

[xexyl]

There's a really odd problem with the -J option over there. I have to open a new bug report. Might be time to try git bisect to see what happened but too tired for it now. It's not from today at least but that means it's been there for a while.

[lcn2]

QUESTION
What do you prefer?

You make a good case for doing the upgrade in the "other repo".

We would suggest the before you commit that change in the "other repo", that you:

• make www to be sure everything is up to date
• update the "other repo" but don't commit anything to a PR just yet
• do a make install the "other repo", as well as a make install under the jparse sub-directory of that "other repo" and then
• do another make www in this repo

The purpose of that test would be to verify that the installed tools (both a the repo level and the jparse sub-directory) allow a full make www to work well. If all is well then the 2nd make www should change nothing.

If all is well, issue a PR for the "others repo" along and include the results of the above tests.

We can later on process the pull request for the "other repo" if your tests work as expected.

[xexyl]

QUESTION
What do you prefer?

You make a good case for doing the upgrade in the "other repo".

We would suggest the before you commit that change in the "other repo", that you:

• make www to be sure everything is up to date
• update the "other repo" but don't commit anything to a PR just yet
• do a make install the "other repo", as well as a make install under the jparse sub-directory of that "other repo" and then
• do another make www in this repo

The purpose of that test would be to verify that the installed tools (both a the repo level and the jparse sub-directory) allow a full make www to work well. If all is well then the 2nd make www should change nothing.

If all is well, issue a PR for the "others repo" along and include the results of the above tests.

We can later on process the pull request for the "other repo" if your tests work as expected.

I agree. I was thinking it would have to involve make www but I was not sure of the exact procedure that you wanted. I'll look at this sometime tomorrow (I hope).

Thanks! I would start it now but not sure how much time I have and make www takes a fair amount of time. I'm sure it's fine but I agree it's better to verify it. If I don't get to this tomorrow please remind me when you can.

I'l also set a reminder.

[xexyl]

Have a bit of extra time. Doing a make www with the tools from the jparse repo. Unsure if I'll have time to test it with the subdirectory of mkiocccentry but if not I'll do that tomorrow. I'll reply to your comment in the other repo in a few.

[xexyl]

Running make www after installing from the jparse REPO does not modify any html files.

I just did a make install from the mkiocccentry repo's subdirectory jparse. I'm doing make www but do not know if I will finish it today. If not I'll start it over tomorrow and then go from there. But it's of course expected that everything works okay esp as it's the same code.

UPDATE 0

Now I look at the time I certainly won't finish it tonight. After last night I need to get some sleep, soon, so I'll be shutting down soon. Would right now except I have a cat between my legs.

[xexyl]

Running make www ... at the gen-location step.

I have a minor improvement of that script which will be included. That improvement helps with the output. Instead of doing:

bin/gen-authors.sh: debug[1]: Processing author_handles that begin with z for: authors.html

it'll do

bin/gen-authors.sh: debug[1]: Processing author_handles that begin with "z" for: authors.html

to help read the output some.

I'd do it as a separate pull request but since the script also requires the other change, the tool change, ...

After this is all done I"ll try and finish the other issue that Leo opened. Hope travels are well and you got a good sleep!

UPDATE 0

At the readme2index stage. All good so far as expected. In about 40 minutes or so I'll be away a bit of time but I should be done later this morning.

[xexyl]

I've fixed a number of lint problems with html files. Most of them have been fixed. But I ran into a problem that is kind of insidious and I'm not sure what to do about it.

Let's look at an example: 1986/bright. What do we see in the .entry.json for abstract? We see:

"abstract" : "hex dump with cpp compressed that uses lots of << for constants"

But what does this generate in the index.html file? Here we see:

  <h3>hex dump with cpp compressed that uses lots of << for constants</h3>

which is very wrong. Now it displays fine but the lint is not Happy. Nor is it Sleepy or Happy or any other one but Grumpy.

Now without thinking I changed it to:

"abstract" : "hex dump with cpp compressed that uses lots of \u003c\u003c for constants"

but then that's not the problem. The problem is that unless we change the abstract to have the html encoded << (which looks bad as abstract) we cannot resolve this, or so I think.

The question is do you want me to update the abstract to this?

[xexyl]

Oh! I solved it. Code blocks. If we want to do it another way we can but this way it should all be resolved.

[xexyl]

FYI: I wrote a script that runs the Nu html checker on all the html files (that are not problematic, not withstanding the ones with the abstract that has a < - the scripts that extract abstract will have to fix that .. maybe with a sed transforming certain characters to html entities ? Just thought of that idea).

Unfortunately their code uses java and it requires a web server running but that jar file has that as well. I have the jar file and the script on both the laptop and the server but if you would find it of interest I can share it. Not making a pull request in case something more important comes up and it's undesired (which I can very well see).

[xexyl]

Unfortunately the abstract having a ` did not work. I have two ideas of what might solve the problem.

One is that sed(1) is used prior to writing it to the html file and it transforms certain characters like < to the correct html entity. The less ideal one is ugly and it would mean the abstract would have the (for the two cases) < in it which would be a real shame.

Otherwise the script could be modified in some other way I guess. I can do the first one but I would need to know which script is and unfortunately there are a lot of matches in bin/ of abstract.

Let me know what you like when you get a chance. Made a pull request here and also mkiocccentry. Off to do other things .. maybe can do more tomorrow.

[lcn2]

Unfortunately the abstract having a ` did not work. I have two ideas of what might solve the problem.

One is that sed(1) is used prior to writing it to the html file and it transforms certain characters like < to the correct html entity. The less ideal one is ugly and it would mean the abstract would have the (for the two cases) < in it ...

One could create a bin/html.sed file that contains the sed commands needed perform a few HTML conversions of problematic characters such as for example:

• & into &amp;
• < into &lt;
• > into &gt;

The bin/output-index-inventory.sh tool has a sed(1) that is used to strip off the numerically sorted inventory_order values from the output:

Those two sed commands on lines 563 and 567 of bin/output-index-inventory.sh in this way:

From:

    ... | sed -e 's/^[0-9][0-9]* //'

To:

    ... | sed -f "$HTML_SED" -e 's/^[0-9][0-9]* //'

where HTML_SED was set to $BIN_DIR/html.sed.

Then earlier in the bin/output-index-inventory.sh tool, say after the "verify pandoc wrapper tool" test (after line 446), a 3-part test could be added to verify that $HTML_SED exists, is a file, and is a readable file could be done. As well as adding the debug printing of $HTML_SED, say after line 482.

By creating the bin/html.sed file, other HTML conversions could be added to that file as needed, AND if we find another occurrence where HTML filtering is needed then the bin/html.sed file could be used there as well.

We are NOT suggesting that you should go looking for cases where other HTML conversions are possible. We are simply suggest ing that the sed(1) related HTML conversions could be centrally managed and maintained in this way.

We would be happy to do this when we get back, if you prefer.

The above took about 45 minutes to upload/correct as a comment via a "very sincere" internet 🛜 connection where "sincerely" was one of the internet 🛜 connection's more admirable qualities 😉.

[xexyl]

As for the above it's kind of like before jstrencode (then jstrdecode) worked. Okay. I'll have to look at your comment in more detail when more awake. I have a few minor changes in some html files that I'll do a pull request with later, too, but I hope that I can do more than just that today.

Sorry that took so long, the posting that comment! Best wishes for next time and I appreciate all the more your comment and time!

[xexyl]

I wrote the html.sed script. The script you refer to has been modified to use it. But other scripts have to be updated too. I also have to find out a way to test the one you mentioned though I think it's good and it just is a matter of updating/touching a README.md file and making sure it works fine.

[xexyl]

Well the good news is I know the script works. The bad news is this:

bin/md2html.sh: ERROR: in -s token=value, the value may not contain a ; (semicolon) character: HEADER_3

.. so I have to figure out if editing the md2html.sh script to allow for a ; is okay. I guess for HEADER_3 it is.

[xexyl]

Please advise whether that option can contain a ;. Alternatively if you wish to update it when you're back you can.

The script your refer to was updated to use the new sed file (see commit ad90b88) however I'm not committing that yet until I have run make www which as you know takes quite some time. I will be going afk soon and I'll start the process before I leave. I have to take care of other things today too but at least this new file is in.

[xexyl]

QUESTION below.

With commit 7b05c2c output-index-inventory.sh now uses the html.sed file that I added in commit ad90b88.

But I cannot do the other file, the one that needs this, until I know if a ; can be allowed in the field=value part as I referenced earlier.

QUESTION

So with the above in mind can the script that extracts the abstract (and the other one .. can't remember what and too tired to check) allow semicolons in the value? If not then what do you propose I do? If yes I can take care of it though that might not happen until tomorrow.

It feels like I won't do more today here or any other repo but that remains to be seen. Nonetheless I did make a number of fixes including in some html files. I need to do more checks of course but the fixed ones were useful to fix (well one was not really necessary but desired).

[lcn2]

QUESTION
So with the above in mind can the script that extracts the abstract (and the other one .. can't remember what and too tired to check) allow semicolons in the value? If not then what do you propose I do? If yes I can take care of it though that might not happen until tomorrow.

It could be that quoting protects the calling scripts against the presence of a semicolon and that the reference in help messages and code comments was excessive paranoia.

Presumably because you issued a PRs you already know that the semicolons introduced by the new bin/html.sed script have not broken make www. But due to our very sincere Internet 🛜 service at the moment, we cannot verify this against your most recent PRs.

We do recommend forcing a make www to do a full test against the committed master branch just to be sure 👍.

[xexyl]

QUESTION
So with the above in mind can the script that extracts the abstract (and the other one .. can't remember what and too tired to check) allow semicolons in the value? If not then what do you propose I do? If yes I can take care of it though that might not happen until tomorrow.

It could be that quoting protects the calling scripts against the presence of a semicolon and that the reference in help messages and code comments was excessive paranoia.

Presumably because you issued a PRs you already know that the semicolons introduced by the new bin/html.sed script have not broken make www. But due to our very sincere Internet 🛜 service at the moment, we cannot verify this against your most recent PRs.

We do recommend forcing a make www to do a full test against the committed master branch just to be sure 👍.

Oh I did do a make www first. But I had to not have the semicolon etc. update because of that restriction (in fact I didn't realise it and it failed at first). It seems that it'll be fine as make www did fine with the new sed script. If you think the other one should be updated I can do that but probably not until tomorrow. Feeling a bit sleepy and I have other things to do too.

The way I interpret your words at this time is that you wish me to remove that restriction and then test it again with make www. If all is good the only changes should be in the two entries that have in their abstract the character <. These would be changed to <.

QUESTION

Is that what you're suggesting I do? Thanks.

[lcn2]

QUESTION

So with the above in mind can the script that extracts the abstract (and the other one .. can't remember what and too tired to check) allow semicolons in the value? If not then what do you propose I do? If yes I can take care of it though that might not happen until tomorrow.

It could be that quoting protects the calling scripts against the presence of a semicolon and that the reference in help messages and code comments was excessive paranoia.

Presumably because you issued a PRs you already know that the semicolons introduced by the new bin/html.sed script have not broken make www. But due to our very sincere Internet 🛜 service at the moment, we cannot verify this against your most recent PRs.

We do recommend forcing a make www to do a full test against the committed master branch just to be sure 👍.

Oh I did do a make www first. But I had to not have the semicolon etc. update because of that restriction (in fact I didn't realise it and it failed at first). It seems that it'll be fine as make www did fine with the new sed script. If you think the other one should be updated I can do that but probably not until tomorrow. Feeling a bit sleepy and I have other things to do too.

The way I interpret your words at this time is that you wish me to remove that restriction and then test it again with make www. If all is good the only changes should be in the two entries that have in their abstract the character <. These would be changed to <.

QUESTION

Is that what you're suggesting I do? Thanks.

Give that a try and see what happens ..

.. power is going on/off which tends to "cut down" all of the available bandwidth: down below the level of sincerity 🤓

[xexyl]

QUESTION

So with the above in mind can the script that extracts the abstract (and the other one .. can't remember what and too tired to check) allow semicolons in the value? If not then what do you propose I do? If yes I can take care of it though that might not happen until tomorrow.

It could be that quoting protects the calling scripts against the presence of a semicolon and that the reference in help messages and code comments was excessive paranoia.

Presumably because you issued a PRs you already know that the semicolons introduced by the new bin/html.sed script have not broken make www. But due to our very sincere Internet 🛜 service at the moment, we cannot verify this against your most recent PRs.

We do recommend forcing a make www to do a full test against the committed master branch just to be sure 👍.

Oh I did do a make www first. But I had to not have the semicolon etc. update because of that restriction (in fact I didn't realise it and it failed at first). It seems that it'll be fine as make www did fine with the new sed script. If you think the other one should be updated I can do that but probably not until tomorrow. Feeling a bit sleepy and I have other things to do too.

The way I interpret your words at this time is that you wish me to remove that restriction and then test it again with make www. If all is good the only changes should be in the two entries that have in their abstract the character <. These would be changed to <.

QUESTION

Is that what you're suggesting I do? Thanks.

Give that a try and see what happens ..

Will do! More from me tomorrow. Hopefully it will resolve the problems.

.. power is going on/off which tends to "cut down" all of the available bandwidth: down below the level of sincerity 🤓

Oh that's always bloody annoying. Good luck with that!

[xexyl]

Updated the script .. now doing make www. Might try and rest as I have been awake since 4 and I might be able to rest now. Hopefully all will go well with this though!

[xexyl]

So the problem is when escaping it the md2html.sh script turns it into (for example with 1986/bright) s|%%HEADER_3%%|hex dump with cpp compressed that uses lots of \<lt;\<lt; for constants|g but without the escape in the html.sed file it would just have & as the match which is also not going to work. Perhaps another sed command needs to be used? I'll see if that can be done but having a fear it might need an ugly hack.

[xexyl]

Made progress though it still does not work right, seemingly.

s|%%HEADER_3%%|hex dump with cpp compressed that uses lots of \<\< for constants|g

is how it's generated now.

UPDATE 0

And running it on the command line shows correct output, it would seem:

$ echo '%%HEADER_3%%' | sed -e 's|%%HEADER_3%%|hex dump with cpp compressed that uses lots of \<\< for constants|g'
hex dump with cpp compressed that uses lots of << for constants

[xexyl]

I GOT IT! Now I can do a make www again and see how it goes.

[xexyl]

QUESTION

Should make www run the script I wrote bin/format-headers.sh before the html generation?

[xexyl]

The modification of '%' does not work .. have to look at it in a bit.

[xexyl]

Typo in sed script. Fixed .. running make www again.

[xexyl]

Ugh. Another problem now. I have to check if it was a new suggestion in your remarks in the pull request. It does not allow a ; and that's a problem though which I should have foreseen as it has to have one for the html entities.