Skip to content

Add support for a centos-based distro #202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add support for a centos-based distro #202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 50 additions & 14 deletions bin/gbuild
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,12 @@ def build_one_configuration(suite, arch, build_desc)
ENV["LXC_SUITE"] = suite
end

if ENV["USE_DOCKER"] and build_desc["distro"].eql? "centos"
ontarget_root_extra = "-w /root"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if we don't supply this argument?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On newer versions of docker there appears to be some sort of security setting that causes the command (without the -w root) to fail because of bad permissions on /home/centos (which is weird, of course, since the login is root). This showed up on a coworker's Arch system -- but only when using a centos-based container.

Similar reports:

else
ontarget_root_extra = ""
end

suitearch = "#{suite}-#{arch}"

info "Stopping target if it is up"
Expand Down Expand Up @@ -73,7 +79,7 @@ def build_one_configuration(suite, arch, build_desc)

system! "on-target true"

system! "on-target -u root tee -a /etc/sudoers.d/#{ENV['DISTRO'] || 'ubuntu'} > /dev/null << EOF
system! "on-target -u root #{ontarget_root_extra} tee -a /etc/sudoers.d/#{ENV['DISTRO'] || 'ubuntu'} > /dev/null << EOF
%#{ENV['DISTRO'] || 'ubuntu'} ALL=(ALL) NOPASSWD: ALL
EOF" if build_desc["sudo"] and @options[:allow_sudo]

Expand All @@ -98,29 +104,55 @@ EOF" if build_desc["sudo"] and @options[:allow_sudo]
if build_desc["multiarch"]
info "Adding multiarch support (log in var/install.log)"
for a in build_desc["multiarch"]
system! "on-target -u root dpkg --add-architecture #{a} >> var/install.log 2>&1"
system! "on-target -u root #{ontarget_root_extra} dpkg --add-architecture #{a} >> var/install.log 2>&1"
end
end

info "Updating apt-get repository (log in var/install.log)"
system! "on-target -u root apt-get update >> var/install.log 2>&1"
case build_desc["distro"]
when "centos"
info "Updating yum repository (log in var/install.log)"
system! "on-target -u root #{ontarget_root_extra} yum -y makecache fast >> var/install.log 2>&1"
else
info "Updating apt-get repository (log in var/install.log)"
system! "on-target -u root #{ontarget_root_extra} apt-get update >> var/install.log 2>&1"
end

info "Installing additional packages (log in var/install.log)"
system! "on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install #{build_desc["packages"].join(" ")} >> var/install.log 2>&1"

case build_desc["distro"]
when "centos"
system! "on-target -u root #{ontarget_root_extra} yum -y install #{build_desc["packages"].join(" ")} > var/install.log 2>&1"
else
system! "on-target -u root #{ontarget_root_extra} -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install #{build_desc["packages"].join(" ")} >> var/install.log 2>&1"
end

if build_desc["alternatives"]
info "Set alternatives (log in var/install.log)"
for a in build_desc["alternatives"]
system! "on-target -u root update-alternatives --set #{a["package"]} #{a["path"]} >> var/install.log 2>&1"
system! "on-target -u root #{ontarget_root_extra} update-alternatives --set #{a["package"]} #{a["path"]} >> var/install.log 2>&1"
end
end

if @options[:upgrade] || system("on-target -u root '[ ! -e /var/cache/gitian/initial-upgrade ]'")
if @options[:upgrade] || system("on-target -u root #{ontarget_root_extra} '[ ! -e /var/cache/gitian/initial-upgrade ]'")
info "Upgrading system, may take a while (log in var/install.log)"
system! "on-target -u root bash < target-bin/upgrade-system.sh >> var/install.log 2>&1"
case build_desc["distro"]
when "centos"
system! "on-target -u root #{ontarget_root_extra} mkdir -p /var/cache/gitian"
system! "on-target -u root #{ontarget_root_extra} yum -y update > var/upgrade.log 2>&1"
system! "copy-to-target #{@quiet_flag} var/upgrade.log /var/cache/gitian/upgrade.log"
system! "on-target -u root #{ontarget_root_extra} touch /var/cache/gitian/initial-upgrade"
else
system! "on-target -u root #{ontarget_root_extra} bash < target-bin/upgrade-system.sh >> var/install.log 2>&1"
end
end
info "Creating package manifest"
system! "on-target -u root bash < target-bin/grab-packages.sh > var/base-#{suitearch}.manifest"

case build_desc["distro"]
when "centos"
system! "on-target -u root #{ontarget_root_extra} yumdb get checksum_data | awk '/checksum_data =/ { print $3, package; next } { package=$1 }' | sort --key 2 > var/base-#{suitearch}.manifest"
else
system! "on-target -u root #{ontarget_root_extra} bash < target-bin/grab-packages.sh > var/base-#{suitearch}.manifest"
end

info "Creating build script (var/build-script)"

Expand All @@ -143,7 +175,7 @@ EOF" if build_desc["sudo"] and @options[:allow_sudo]
build_desc["remotes"].each do |remote|
dir = sanitize(remote["dir"], remote["dir"])

author_date = `cd inputs/#{dir} && git log --format=@%at -1 | date +"%F %T" -u -f -`.strip
author_date = `cd inputs/#{dir} > /dev/null && git log --format=@%at -1 | date +"%F %T" -u -f -`.strip
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the output that this is suppressing?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I could figure out, the cd was spitting out information about the new directory -- I think in the case where you/your-distro have your shell configured to update the "window title" to reflect the current working directory. Apparently this isn't an issue on the non-Centos invocations?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't happen in non-interactive shells. Perhaps your personal setup has some interactive shell setup or an alias to cd in the wrong shell startup file?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tracked it down -- it's because I have CDPATH set: https://unix.stackexchange.com/questions/245453/how-to-not-show-path-after-cd-command-with-cdpath-set (note the workaround).

raise "error looking up author date in #{dir}" unless $?.exitstatus == 0

system! "copy-to-target #{@quiet_flag} inputs/#{dir} build/"
Expand Down Expand Up @@ -220,11 +252,13 @@ in_sums = []
build_dir = 'build'
result_dir = 'result'
cache_dir = 'cache'
work_dir = 'var'
enable_cache = build_desc["enable_cache"]

FileUtils.rm_rf(build_dir)
FileUtils.mkdir(build_dir)
FileUtils.mkdir_p(result_dir)
FileUtils.mkdir_p(work_dir)

package_name = build_desc["name"] or raise "must supply name"
package_name = sanitize(package_name, "package name")
Expand Down Expand Up @@ -290,13 +324,15 @@ build_desc["remotes"].each do |remote|
end
system!("cd inputs/#{dir} && git fetch --update-head-ok #{sanitize_path(remote["url"], remote["url"])} +refs/tags/*:refs/tags/* +refs/heads/*:refs/heads/*")
commit = sanitize(remote["commit"], remote["commit"])
commit = `cd inputs/#{dir} && git log --format=%H -1 #{commit}`.strip
commit = `cd inputs/#{dir} > /dev/null && git log --format=%H -1 #{commit}`.strip
raise "error looking up commit for tag #{remote["commit"]}" unless $?.exitstatus == 0
info("commit is #{commit}")
system!("cd inputs/#{dir} && git checkout -q #{commit}")
system!("cd inputs/#{dir} && git submodule update --init --recursive --force")
in_sums << "git:#{commit} #{dir}"
end


base_manifests = YAML::Omap.new

suites.each do |suite|
Expand Down Expand Up @@ -333,7 +369,7 @@ Dir.glob(File.join(out_dir, '**', '*'), File::FNM_DOTMATCH).sort.each do |file_i
next if File.directory?(file_in_out)
file = file_in_out.sub(out_dir + File::SEPARATOR, '')
file = sanitize_path(file, file_in_out)
out_sums[file] = `cd #{out_dir} && sha256sum #{file}`
out_sums[file] = `cd #{out_dir} > /dev/null && sha256sum #{file}`
raise "failed to sum #{file}" unless $? == 0
puts out_sums[file] unless @options[:quiet]
end
Expand All @@ -343,15 +379,15 @@ if enable_cache
next if File.directory?(file_in_out)
file = file_in_out.sub(cache_common_dir + File::SEPARATOR, '')
file = sanitize_path(file, file_in_out)
cache_common_sums[file] = `cd #{cache_common_dir} && sha256sum #{file}`
cache_common_sums[file] = `cd #{cache_common_dir} > /dev/null && sha256sum #{file}`
raise "failed to sum #{file}" unless $? == 0
end

Dir.glob(File.join(cache_package_dir, '**', '*'), File::FNM_DOTMATCH).sort.each do |file_in_out|
next if File.directory?(file_in_out)
file = file_in_out.sub(cache_package_dir + File::SEPARATOR, '')
file = sanitize_path(file, file_in_out)
cache_package_sums[file] = `cd #{cache_package_dir} && sha256sum #{file}`
cache_package_sums[file] = `cd #{cache_package_dir} > /dev/null && sha256sum #{file}`
raise "failed to sum #{file}" unless $? == 0
end
end
Expand Down
10 changes: 9 additions & 1 deletion bin/make-base-vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,12 +184,20 @@ if [ $DOCKER = "1" ]; then
base_image="$DISTRO:$SUITE"
fi

if [ $DISTRO = "centos" ]; then
UPDATE_FROM_REPO="yum -y update && yum -y install epel-release"
INSTALL_FROM_REPO="yum -y install"
else
UPDATE_FROM_REPO="apt-get update"
INSTALL_FROM_REPO="apt-get --no-install-recommends -y install"
fi

# Generate the dockerfile
cat << EOF > $OUT.Dockerfile
FROM $base_image

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get --no-install-recommends -y install $addpkg
RUN $UPDATE_FROM_REPO && $INSTALL_FROM_REPO $addpkg

RUN useradd -ms /bin/bash -U $DISTRO
USER $DISTRO:$DISTRO
Expand Down
8 changes: 7 additions & 1 deletion libexec/copy-to-target
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,13 @@ fi
if [ -n "$USE_DOCKER" ]; then
docker exec -u $TUSER gitian-target mkdir -p "/home/$TUSER/$2"
docker cp "$1" gitian-target:"/home/$TUSER/$2"
docker exec -u root gitian-target chown -R $TUSER:$TUSER "/home/$TUSER/$2"
# newer docker version is strict about non-default-user working directories
if docker exec -u root -w /root gitian-target true > /dev/null 2>&1
then
docker exec -u root -w /root gitian-target chown -R $TUSER:$TUSER "/home/$TUSER/$2"
else
docker exec -u root gitian-target chown -R $TUSER:$TUSER "/home/$TUSER/$2"
fi
elif [ -z "$USE_LXC" ]; then
src="${1%/}" # remove trailing / which triggers special rsync behaviour
rsync --checksum -a $QUIET_FLAG -e "ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT" "${src}" "$TUSER@localhost:$2"
Expand Down
14 changes: 13 additions & 1 deletion libexec/on-target
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,18 @@ if [ $# != 0 ] ; then
ENV="$2"
shift 2
;;
--workdir|-w)
if [ -n "$USE_DOCKER" ]; then
if docker exec -u root -w /root gitian-target true > /dev/null 2>&1; then
# newer docker version is strict about non-default-user working dir
TWORKDIR="-w $2"
fi
shift 2
else
echo "unrecognized option $1"
exit 1
fi
;;
--*)
echo "unrecognized option $1"
exit 1
Expand All @@ -47,7 +59,7 @@ fi
#fi

if [ -n "$USE_DOCKER" ]; then
docker exec -u $TUSER -i gitian-target $*
docker exec -u $TUSER $TWORKDIR -i gitian-target $*
elif [ -z "$USE_LXC" ]; then
ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT $TUSER@localhost $*
else
Expand Down