Skip to content

Publish on-demand log JSON atomically #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
okt-limonikas wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Publish on-demand log JSON atomically #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9f1748c
tools/rgt: publish generated JSON atomically
okt-limonikas Aug 4, 2026
569be08
tools/log_server: serialize on-demand generation
okt-limonikas Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 91 additions & 12 deletions tools/log_server/te-logs-error404.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,40 @@
# HTTP response
#############################
function not_found_exit() {
printf "Status: 404 Not Found\r\n"
printf "Content-Type: text/plain\r\n"
printf "\r\n"
printf "Not Found\r\n"
printf "$*\r\n"
[[ $# -eq 0 ]] || printf "%s\r\n" "$*"
exit 0
}

#############################
# Exit with an internal server error without exposing generator output.
# Arguments:
# None
# Outputs:
# HTTP response
#############################
function internal_error_exit() {

@arybchik arybchik Aug 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dislike that really different changes are mixed in one changeset. It makes the review harder.
Please, factor out introduction of these functions into separate patch.

printf "Status: 500 Internal Server Error\r\n"
printf "Content-Type: text/plain\r\n"
printf "\r\n"
printf "Log generation failed\r\n"
exit 0
}

#############################
# Redirect to the now-generated request path.
# Arguments:
# None
# Outputs:
# HTTP response
#############################
function redirect_to_request() {
printf "Status: 302 Found\r\n"
printf "Location: %s\r\n" "${redirect_uri}"
printf "\r\n"
exit 0
}

Expand All @@ -52,13 +82,61 @@ case "${REQUEST_URI}" in
;;
esac

# Requested file
request_file="${REQUEST_URI/$root_dir_uri/$root_dir}"
# Requested file. Ignore a query string when mapping the URI to the file
# system, but preserve it in redirects.
redirect_uri="${REQUEST_URI}"
request_uri_path="${REQUEST_URI%%\?*}"
request_uri_query=
if [[ "${REQUEST_URI}" == *\?* ]] ; then
request_uri_query="?${REQUEST_URI#*\?}"
fi

# JSON node files belong to the json subdirectory. Normalize a root-level
# request before passing it to rgt-log-get-item, otherwise that tool creates
# the requested file literally in the run root.
request_uri_dir="${request_uri_path%/*}"
request_uri_name="${request_uri_path##*/}"
case "${request_uri_name}" in
node_*.json | tree.json )
if [[ "${request_uri_dir##*/}" != "json" ]] ; then
request_uri_path="${request_uri_dir}/json/${request_uri_name}"
redirect_uri="${request_uri_path}${request_uri_query}"
fi
;;
esac

request_file="${root_dir}${request_uri_path#"${root_dir_uri}"}"
# Substitute %20->'space'
request_file="${request_file//%20/ }"
# Substitute %3A->':'
request_file="${request_file//%3A/:}"

# Reject paths which escape the configured logs root after normalization.
canonical_root=$(realpath -m -- "${root_dir}") || internal_error_exit
request_file=$(realpath -m -- "${request_file}") || internal_error_exit
case "${request_file}" in
"${canonical_root}"/* ) ;;
* ) not_found_exit ;;

@arybchik arybchik Aug 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logically it is a separate enhancement not directly related to locking and serialization

esac

# Another request may have completed generation after Apache selected this
# error handler but before this process started.
[[ -r "${request_file}" ]] && redirect_to_request

# Serialize generation by final output path. The check after flock is
# essential: all waiters should reuse the first request's completed file.
lock_dir="${TMPDIR:-/tmp}/te-log-generation-locks"
mkdir -p -- "${lock_dir}" || internal_error_exit
lock_key=$(printf "%s" "${request_file}" | sha256sum) || internal_error_exit
lock_key="${lock_key%% *}"
# Use a bounded pool: collisions only serialize generation of unrelated files,
# while arbitrary request paths can create at most 256 persistent lock files.
lock_slot="${lock_key:0:2}"
exec {lock_fd}>"${lock_dir}/${lock_slot}.lock" || internal_error_exit
flock -x "${lock_fd}" || internal_error_exit

[[ -r "${request_file}" ]] && redirect_to_request

get_item_cmd=()
# It is recommended to run it under nice since really many requests
# could be generated via Web server
Expand All @@ -73,15 +151,16 @@ status=$("${get_item_cmd[@]}" --req-path="${request_file}" 2>&1)
result=$?

if [[ -r "${request_file}" ]] ; then
# Now requested file exists, redirect
printf "Status: 302 Moved\r\n"
printf "Location: %s\r\n" "${REQUEST_URI}"
printf "\r\n"
elif [[ x"$result" = x"2" ]] ; then
# Internal error
printf "Content-Type: text/plain\r\n"
printf "\r\n"
printf "ERROR: $status\r\n"
redirect_to_request
elif [[ "${result}" -eq 1 ]] ; then
# The source log or requested item does not exist.
not_found_exit
elif [[ "${result}" -ne 0 ]] ; then
# Keep command output in the server log; it may contain host paths or
# parser details which should not be returned to clients.
printf "Failed to generate %s (status %d): %s\n" \
"${REQUEST_URI}" "${result}" "${status}" >&2
internal_error_exit
else
not_found_exit
fi
Loading