Skip to content

Commit

Permalink
Opportunistically delete ignored documents
Browse files Browse the repository at this point in the history
The ignorelist contents and logic may change over time, and we want to
delete any documents that may exist remotely but are now covered by the
ignorelist.

We've set deletion up as an idempotent operation anyway, so there is no
harm in firing off an opportunistic delete request when a document is
ignored.

- Attempt to delete documents remotely that come through as ignored
- Add an example ignored message to integration tests
- Make documents not existing on deletion an `info` rather than `warn`
  log message, and remove TODO comment now that our approach is final
- Fix incorrectly named mock object
  • Loading branch information
csutter committed Nov 29, 2023
1 parent 3194fa7 commit b1904ea
Show file tree
Hide file tree
Showing 5 changed files with 185 additions and 7 deletions.
5 changes: 4 additions & 1 deletion app/models/publishing_api_document.rb
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,10 @@ def synchronize
log("delete")
delete_service.call(content_id, payload_version:)
else
log("ignore (#{ignore_reason})")
log("ignore and delete (#{ignore_reason})")
# Eagerly attempt to delete the document anyway in case it has been previously synchronised
# (for example, if the ignorelist has changed and it should now no longer be in search)
delete_service.call(content_id, payload_version:)
end
end

Expand Down
7 changes: 2 additions & 5 deletions app/services/discovery_engine/delete.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,9 @@ def call(content_id, payload_version: nil)

log(Logger::Severity::INFO, "Successfully deleted", content_id:, payload_version:)
rescue Google::Cloud::NotFoundError => e
# TODO: Should we eventually send this to Sentry? A document not existing is a relatively
# common error initially as an unpublish request may come through before we've imported the
# document to begin with.
log(
Logger::Severity::WARN,
"Failed to delete document as it doesn't exist remotely (#{e.message})",
Logger::Severity::INFO,
"Did not delete document as it doesn't exist remotely (#{e.message}).",
content_id:, payload_version:,
)
rescue Google::Cloud::Error => e
Expand Down
1 change: 1 addition & 0 deletions config/document_type_ignorelist.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,6 @@ shared:
- worldwide_office

test:
- html_publication
- test_ignored_type
- !ruby/regexp /^another_test_ignored_type/
166 changes: 166 additions & 0 deletions spec/fixtures/files/message_queue/html_publication_message.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
{
"title": "Access Consultation Forum terms of reference",
"public_updated_at": "2022-09-22T13:04:49Z",
"publishing_app": "whitehall",
"rendering_app": "government-frontend",
"update_type": "republish",
"phase": "live",
"analytics_identifier": null,
"document_type": "html_publication",
"schema_name": "html_publication",
"first_published_at": "2019-09-30T10:30:12Z",
"base_path": "/government/publications/access-consultation-forum-terms-of-reference/access-consultation-forum-terms-of-reference",
"description": null,
"details": {
"body": "\u003cdiv class=\"govspeak\"\u003e\u003ch2 id=\"purpose\"\u003e\n\u003cspan class=\"number\"\u003e1. \u003c/span\u003ePurpose\u003c/h2\u003e\n\n\u003cp\u003eA meeting where representatives with a relevant interest discuss the accessibility of regulated qualifications and assessments. The forum shares information and promotes good practice.\u003c/p\u003e\n\n\u003cp\u003eOfqual organises the ACF. Other UK qualification regulators attend the forum (Qualifications Wales (QW) for Wales, Council for the Curriculum, Examinations and Assessment (CCEA) for Northern Ireland, and Scottish Qualifications Authority (SQA) for Scotland). The regulators and other attendees (including representative groups and awarding organisations) can use forum discussions to inform their work.\u003c/p\u003e\n\n\u003ch2 id=\"scope\"\u003e\n\u003cspan class=\"number\"\u003e2. \u003c/span\u003eScope\u003c/h2\u003e\n\n\u003cp\u003eThe main focus of the group is the accessibility of regulated qualifications and assessments for disabled learners, including the use of reasonable adjustments. The forum will consider issues affecting learners with other protected characteristics if appropriate.\u003c/p\u003e\n\n\u003ch2 id=\"aims\"\u003e\n\u003cspan class=\"number\"\u003e3. \u003c/span\u003eAims\u003c/h2\u003e\n\n\u003cp\u003eThe ACF aims to:\u003c/p\u003e\n\n\u003cul\u003e\n \u003cli\u003epromote understanding of the issues faced by disabled learners when taking regulated qualifications and assessments\u003c/li\u003e\n \u003cli\u003epresent research and share expertise\u003c/li\u003e\n \u003cli\u003eprovide an opportunity for attendees to raise questions about current practice\u003c/li\u003e\n \u003cli\u003ebe a platform for regulators to share planned work relating to the accessibility of regulated qualifications or assessments, invite feedback and expertise from attendees, and explore solutions\u003c/li\u003e\n \u003cli\u003eencourage all attendees to share, and consider, good practice\u003c/li\u003e\n \u003cli\u003eraise awareness and understanding of the importance of accessibility to qualification and assessment validity\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003ch2 id=\"meetings\"\u003e\n\u003cspan class=\"number\"\u003e4. \u003c/span\u003eMeetings\u003c/h2\u003e\n\n\u003cp\u003eOfqual will organise and chair ACF meetings at least twice a year, usually in spring and autumn. Ofqual will circulate an agenda at least 2 weeks before the meeting.\u003c/p\u003e\n\n\u003cp\u003eOfqual may arrange additional meetings to discuss particular areas of work, in addition to the biannual meetings. Ofqual may invite other organisations to join those meetings as appropriate.\u003c/p\u003e\n\n\u003cp\u003eOfqual will ensure visibility of relevant areas of work between the ACF and its Vocational and Technical Qualification Awarding Organisation Equalities Forum and its Vocational and Technical Qualification Stakeholder Equalities Forum (mainly attended by centres). Ofqual will consider joint meetings where appropriate. The ACF secretariat will be responsible for such joint meetings.\u003c/p\u003e\n\n\u003ch2 id=\"attendance\"\u003e\n\u003cspan class=\"number\"\u003e5. \u003c/span\u003eAttendance\u003c/h2\u003e\n\n\u003cp\u003eAttendance at the forum is by invitation only. You can register your or your organisation’s interest in attending by emailing \u003ca href=\"mailto:[email protected]\" class=\"govuk-link\"\[email protected]\u003c/a\u003e. Ofqual will consider reasonable requests for attendance.\u003c/p\u003e\n\n\u003cp\u003eOfqual will consider requests for attendance from those who:\u003c/p\u003e\n\n\u003cul\u003e\n \u003cli\u003erepresent the interests of disabled learners and the professionals who work with and support them\u003c/li\u003e\n \u003cli\u003erepresent the interests of organisations approved by a UK qualifications regulator to offer regulated qualifications\u003c/li\u003e\n \u003cli\u003ebecause of their expertise can contribute to the purpose and/or aims of the group or inform a particular agenda item\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003eOfqual will invite attendees to suggest agenda items and will accommodate suggestions where possible. Ofqual expects attendees to present their work to support the learning of the group, to share their expertise and to consider and respect all views expressed by other attendees.\u003c/p\u003e\n\n\u003cp\u003eSlides and papers presented at an ACF meeting will be shared with attendees and/or shared with all regulated awarding organisations, unless Ofqual is asked to treat them confidentially. Attendees should not publish content shared with the ACF without discussion with Ofqual. Ofqual will provide attendees with a brief note of the meeting, including agreed actions.\u003c/p\u003e\n\n\u003cp\u003eOfqual may contact attendees outside of meetings to follow up on contributions or to share additional relevant information.\u003c/p\u003e\n\n\u003ch2 id=\"attendees\"\u003e\n\u003cspan class=\"number\"\u003e6. \u003c/span\u003eAttendees\u003c/h2\u003e\n\n\u003cp\u003eThe UK qualifications regulators (Ofqual, CCEA, QW, SQA)\u003c/p\u003e\n\n\u003cp\u003eThe Department for Education (DfE) and the Standards \u0026amp; Testing Agency (STA). The Welsh, Northern Irish and Scottish governments may attend directly or provide input via the relevant qualifications regulator.\u003c/p\u003e\n\n\u003cp\u003eThe Equality and Human Rights Commission may also attend.\u003c/p\u003e\n\n\u003cp\u003eRepresentative groups:\u003c/p\u003e\n\n\u003cul\u003e\n \u003cli\u003eAce Centre\u003c/li\u003e\n \u003cli\u003eAutism Education Trust (AET)\u003c/li\u003e\n \u003cli\u003eBritish Association for Teachers of the Deaf (BATOD)\u003c/li\u003e\n \u003cli\u003eBritish Dyslexia Association (BDA)\u003c/li\u003e\n \u003cli\u003eCALL Scotland\u003c/li\u003e\n \u003cli\u003eColour Blind Awareness (CBA)\u003c/li\u003e\n \u003cli\u003eCommunicate-ed\u003c/li\u003e\n \u003cli\u003eCentre for Research in Autism and Education at UCL (CRAE)\u003c/li\u003e\n \u003cli\u003eFederation of Awarding Bodies (FAB)\u003c/li\u003e\n \u003cli\u003eJISC\u003c/li\u003e\n \u003cli\u003eJoint Council for Qualifications (JCQ)\u003c/li\u003e\n \u003cli\u003eNational Association for Special Educational Needs (NASEN)\u003c/li\u003e\n \u003cli\u003eNational Deaf Children’s Society (NDCS)\u003c/li\u003e\n \u003cli\u003eRoyal National Institute of Blind People (RNIB)\u003c/li\u003e\n \u003cli\u003eThe Bell Foundation\u003c/li\u003e\n \u003cli\u003eThe Exams Office\u003c/li\u003e\n \u003cli\u003eThomas Pocklington Trust (TPT)\u003c/li\u003e\n \u003cli\u003eUK Association for Accessible Formats (UKAAF)\u003c/li\u003e\n \u003cli\u003eVisual Impairment Educational Workforce (VIEW)\u003c/li\u003e\n \u003cli\u003eWelsh Association of Vision Impairment Educators (WAVIE)\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003eAwarding organisations:\u003c/p\u003e\n\n\u003cul\u003e\n \u003cli\u003eCambridge Assessment International Education\u003c/li\u003e\n \u003cli\u003eCity \u0026amp; Guilds\u003c/li\u003e\n \u003cli\u003eInternational Baccalaureate (IB)\u003c/li\u003e\n \u003cli\u003eOxford, Cambridge and RSA Examinations (OCR)\u003c/li\u003e\n \u003cli\u003ePearson\u003c/li\u003e\n \u003cli\u003eThe Assessment and Qualifications Alliance (AQA)\u003c/li\u003e\n \u003cli\u003eWelsh Joint Education Committee (WJEC)\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003eOfqual will invite additional organisations or individual attendees as necessary according to the agenda.\u003c/p\u003e\n\n\u003ch2 id=\"further-information\"\u003e\n\u003cspan class=\"number\"\u003e7. \u003c/span\u003eFurther information\u003c/h2\u003e\n\n\u003cp\u003eOfqual will publish these terms of reference on its website, and on the Ofqual Portal for regulated Awarding Organisations.\u003c/p\u003e\n\n\u003ch2 id=\"data-protection\"\u003e\n\u003cspan class=\"number\"\u003e8. \u003c/span\u003eData Protection\u003c/h2\u003e\n\n\u003cp\u003eOfqual will retain attendees’ personal data as follows: name, job title, organisation name, and email address.\u003c/p\u003e\n\n\u003cp\u003eA To understand how Ofqual processes your personal data please see our \u003ca href=\"https://www.gov.uk/government/organisations/ofqual/about/personal-information-charter\" class=\"govuk-link\"\u003epersonal information charter\u003c/a\u003e.\u003c/p\u003e\n\n\u003cp\u003eIf you no longer wish to be an attendee of this group, you can withdraw by contacting \u003ca href=\"mailto:[email protected]\" class=\"govuk-link\"\[email protected]\u003c/a\u003e.\u003c/p\u003e\n\n\u003ch2 id=\"review-date\"\u003e\n\u003cspan class=\"number\"\u003e9. \u003c/span\u003eReview date\u003c/h2\u003e\n\n\u003cp\u003eThese terms of reference will be reviewed at least every 2 years. They were last reviewed in June 2022. ACF members are not required to agree to the terms of reference but they will be circulated for information.\u003c/p\u003e\n\u003c/div\u003e",
"political": false,
"public_timestamp": "2022-09-22T14:18:42.000+01:00",
"first_published_version": false
},
"routes": [
{
"path": "/government/publications/access-consultation-forum-terms-of-reference/access-consultation-forum-terms-of-reference",
"type": "exact"
}
],
"redirects": [],
"content_id": "1f1f2c96-5a14-4d2a-9d0c-be6ac6c62c3b",
"locale": "en",
"expanded_links": {
"government": [
{
"content_id": "d4fbc1b9-d47d-4386-af04-ac909f868f92",
"title": "2015 Conservative government",
"locale": "en",
"api_path": "/api/content/government/2015-conservative-government",
"base_path": "/government/2015-conservative-government",
"document_type": "government",
"details": {
"started_on": "2015-05-08T00:00:00+00:00",
"ended_on": null,
"current": true
},
"links": {}
}
],
"organisations": [
{
"content_id": "83f6e93f-bb2c-46ab-9b02-394f972b7030",
"title": "Ofqual",
"locale": "en",
"analytics_identifier": "D109",
"api_path": "/api/content/government/organisations/ofqual",
"base_path": "/government/organisations/ofqual",
"document_type": "organisation",
"schema_name": "organisation",
"withdrawn": false,
"details": {
"acronym": "Ofqual",
"logo": {
"image": {
"url": "https://assets.publishing.service.gov.uk/media/5bba55baed915d23b049e12f/ofqual_gov_uk_logo.png",
"alt_text": "Ofqual"
},
"formatted_title": "Ofqual"
},
"brand": "department-for-education",
"default_news_image": {
"url": "https://assets.publishing.service.gov.uk/media/5bba558ce5274a2252092fcc/s300_Default_News_Image.jpg",
"high_resolution_url": "https://assets.publishing.service.gov.uk/media/5bba558c40f0b664e26be324/s960_Default_News_Image.jpg"
},
"organisation_govuk_status": {
"url": null,
"status": "live",
"updated_at": null
}
},
"links": {}
}
],
"parent": [
{
"content_id": "9fb97baa-180c-4983-8013-d64ea663d6db",
"title": "Access Consultation Forum - terms of reference",
"locale": "en",
"analytics_identifier": null,
"api_path": "/api/content/government/publications/access-consultation-forum-terms-of-reference",
"base_path": "/government/publications/access-consultation-forum-terms-of-reference",
"document_type": "guidance",
"public_updated_at": "2022-09-22T13:18:42Z",
"schema_name": "publication",
"withdrawn": false,
"links": {}
}
],
"primary_publishing_organisation": [
{
"content_id": "83f6e93f-bb2c-46ab-9b02-394f972b7030",
"title": "Ofqual",
"locale": "en",
"analytics_identifier": "D109",
"api_path": "/api/content/government/organisations/ofqual",
"base_path": "/government/organisations/ofqual",
"document_type": "organisation",
"schema_name": "organisation",
"withdrawn": false,
"details": {
"acronym": "Ofqual",
"logo": {
"image": {
"url": "https://assets.publishing.service.gov.uk/media/5bba55baed915d23b049e12f/ofqual_gov_uk_logo.png",
"alt_text": "Ofqual"
},
"formatted_title": "Ofqual"
},
"brand": "department-for-education",
"default_news_image": {
"url": "https://assets.publishing.service.gov.uk/media/5bba558ce5274a2252092fcc/s300_Default_News_Image.jpg",
"high_resolution_url": "https://assets.publishing.service.gov.uk/media/5bba558c40f0b664e26be324/s960_Default_News_Image.jpg"
},
"organisation_govuk_status": {
"url": null,
"status": "live",
"updated_at": null
}
},
"links": {}
}
],
"available_translations": [
{
"title": "Access Consultation Forum terms of reference",
"public_updated_at": "2022-09-22T13:04:49Z",
"analytics_identifier": null,
"document_type": "html_publication",
"schema_name": "html_publication",
"base_path": "/government/publications/access-consultation-forum-terms-of-reference/access-consultation-forum-terms-of-reference",
"api_path": "/api/content/government/publications/access-consultation-forum-terms-of-reference/access-consultation-forum-terms-of-reference",
"withdrawn": false,
"content_id": "1f1f2c96-5a14-4d2a-9d0c-be6ac6c62c3b",
"locale": "en"
}
]
},
"user_journey_document_supertype": "thing",
"email_document_supertype": "other",
"government_document_supertype": "other",
"content_purpose_subgroup": "other",
"content_purpose_supergroup": "other",
"publishing_request_id": "21-1695991152.770-10.13.27.71-698",
"govuk_request_id": null,
"links": {
"government": [
"d4fbc1b9-d47d-4386-af04-ac909f868f92"
],
"organisations": [
"83f6e93f-bb2c-46ab-9b02-394f972b7030"
],
"parent": [
"9fb97baa-180c-4983-8013-d64ea663d6db"
],
"primary_publishing_organisation": [
"83f6e93f-bb2c-46ab-9b02-394f972b7030"
]
},
"payload_version": "12345"
}
13 changes: 12 additions & 1 deletion spec/integration/document_synchronization_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
let(:message) { GovukMessageQueueConsumer::MockMessage.new(payload) }

let(:put_service) { instance_double(DiscoveryEngine::Put, call: nil) }
let(:delete_service) { instance_double(DiscoveryEngine::Put, call: nil) }
let(:delete_service) { instance_double(DiscoveryEngine::Delete, call: nil) }

before do
allow(DiscoveryEngine::Put).to receive(:new).and_return(put_service)
Expand Down Expand Up @@ -253,6 +253,17 @@
end
end

describe "for an 'html_publication' message" do
let(:payload) { json_fixture_as_hash("message_queue/html_publication_message.json") }

it "is ignored and deleted from Discovery Engine through the Delete service" do
expect(delete_service).to have_received(:call).with(
"1f1f2c96-5a14-4d2a-9d0c-be6ac6c62c3b",
payload_version: 12_345,
)
end
end

describe "for a 'gone' message" do
let(:payload) { json_fixture_as_hash("message_queue/gone_message.json") }

Expand Down

0 comments on commit b1904ea

Please sign in to comment.