Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions terragrunt/accounts/docs-rs-staging/docs-rs/terragrunt.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,6 @@ inputs = {
private_subnet_ids = dependency.vpc.outputs.private_subnets
domain = "docs-rs-staging.rust-lang.net"
bastion_security_group_id = dependency.vpc.outputs.bastion_security_group_id
min_num_builder_instances = 1
max_num_builder_instances = 1
Comment on lines +28 to +29
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what the autoscaling does when you've pinned it to always be at one instance?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It assures that there's one healthy instance. So if one instance stops or gets terminated a new one boots.

}
94 changes: 92 additions & 2 deletions terragrunt/modules/docs-rs/builder.tf
Original file line number Diff line number Diff line change
@@ -1,5 +1,57 @@
// The instance profile the builder will assume when communicating with
// other AWS services.
// The autoscaling group for the builder

resource "aws_autoscaling_group" "builder" {
name = "docs-rs-builder"
vpc_zone_identifier = var.cluster_config.subnet_ids
max_size = var.max_num_builder_instances
min_size = var.min_num_builder_instances
# Let the instances get warm
default_instance_warmup = 60

launch_template {
id = aws_launch_template.builder.id
version = "$Latest"
}
}

resource "aws_launch_template" "builder" {
name_prefix = "builder"
image_id = data.aws_ami.builder.id
instance_type = "t2.large"

network_interfaces {
associate_public_ip_address = true
security_groups = [aws_security_group.builder.id]
}

iam_instance_profile {
arn = aws_iam_instance_profile.builder.arn
}

block_device_mappings {
device_name = "/dev/sda1"

ebs {
volume_size = 64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seeing the current filesystem usage (~100 GB) I would prefer something at least double this size.

( of course the current usage also includes the database & some web cache)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if we didn't need so much storage. I believe a large part of that storage is only needed during a single crate's build and afterwards can be deleted, no? Would it be possible to add some clean up to the builder process so that the filesystem usage doesn't grow so large?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm.. we are cleaning up after the build.

Plus the cleanup tasks for docker images, which are in cron right now.
( btw, cc @Nemo157 @jyn514 , these cronjobs would need to be configured in our ansible images too, right? )

Only looking at the above I could totally imagine to just try with the current definition above, let the builder build, and watch how much space is used. ( assuming the big docker image is configured?)

But, we're also planning on adding some build artifact caching: rust-lang/docs.rs#1757
( of course we could increase storage only then, when that feature is finished)

Copy link
Contributor

@Nemo157 Nemo157 Feb 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, we just have a daily cronjob (systemd-timer) running docker container prune --force && docker image prune --force (and cargo-sweep which shouldn't be necessary if we rebuild the image for a new version?).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is the cronjob currently configured? I can add this to the Ansible configuration (though I wouldn't block merging this).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's in

/etc/systemd/system/prune-disk-space.service
/etc/systemd/system/prune-disk-space.timer

delete_on_termination = true
}
}

tag_specifications {
resource_type = "instance"

tags = {
Name = "docs-rs-builder"
}
}
}

data "aws_ami" "builder" {
most_recent = true
name_regex = "^docs-rs-builder-*"
}

// The instance profile the builder will assume when communicating with s3

resource "aws_iam_instance_profile" "builder" {
name = "builder"
Expand Down Expand Up @@ -47,3 +99,41 @@ resource "aws_iam_role_policy" "builder_s3" {
]
})
}

// Security group allowing all egress and ssh ingress from the bastion instance
resource "aws_security_group" "builder" {
vpc_id = var.cluster_config.vpc_id
name = "docs-rs-builder"
description = "Access rules for the docs-rs builder."

// SSH access from the bastion instance
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
description = "SSH access from bastion"
security_groups = [aws_security_group.web.id]
}

// Allow outgoing connections

egress {
from_port = 0
to_port = 0
protocol = -1
cidr_blocks = ["0.0.0.0/0"]
description = "Allow all IPv4 egress traffic."
}

egress {
from_port = 0
to_port = 0
protocol = -1
ipv6_cidr_blocks = ["::/0"]
description = "Allow all IPv6 egress traffic."
}

tags = {
Name = "docs-rs-builder"
}
}
10 changes: 10 additions & 0 deletions terragrunt/modules/docs-rs/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,13 @@ variable "cluster_config" {
})
description = "The configuration for the cluster this is running in"
}

variable "min_num_builder_instances" {
type = number
description = "The minimum number of builder instances there should be"
}

variable "max_num_builder_instances" {
type = number
description = "The maximum number of builder instances there should be"
}