Merge pull request #3524 from cal-itp/3497-deprecate-ntd-tables

Remove annual_database_service models and manual scrape process

Author: erikamov

airflow/dags/create_external_tables/ntd_data_products/annual_database_service.yml

@@ -13,9 +13,6 @@
 https://www.transit.dot.gov/ntd/data-product/2021-annual-database-agency-information
 Contains basic contact and agency information for each NTD reporter.
 
-https://www.transit.dot.gov/ntd/data-product/2022-annual-database-service
-Contains operating statistics reported by mode and type of service. Categorized by vehicles operated and vehicles available in maximum service by day and time period.
-
 https://www.transit.dot.gov/ntd/data-product/monthly-module-adjusted-data-release
 Monthly ridership
 
@@ -28,7 +25,6 @@
 then
 poetry run python scrape_ntd.py annual-database-agency-information 2021 https://www.transit.dot.gov/sites/fta.dot.gov/files/2022-10/2021%20Agency%20Information.xlsx
 poetry run python scrape_ntd.py monthly-ridership-with-adjustments 2024 https://www.transit.dot.gov/sites/fta.dot.gov/files/2024-04/February%202024%20Complete%20Monthly%20Ridership%20%28with%20adjustments%20and%20estimates%29_240402_0.xlsx
-poetry run python scrape_ntd.py annual-database-service 2022 https://www.transit.dot.gov/sites/fta.dot.gov/files/2024-04/2022%20Service.xlsx
 """
 
 import gzip

script/scrape_ntd.py

@@ -7,7 +7,6 @@ sources:
     schema: external_ntd_data_products
     tables:
       - name: annual_database_agency_information
-      - name: annual_database_service
       - name: monthly_ridership_with_adjustments_upt
       - name: monthly_ridership_with_adjustments_vrm
       - name: monthly_ridership_with_adjustments_vrh

warehouse/models/staging/ntd/_src.yml

@@ -12,7 +12,6 @@ models:
       - name: ntd_id
         tests:
           - not_null
-  - name: stg_ntd__annual_database_service
   - name: stg_ntd__monthly_ridership_with_adjustments_upt
   - name: stg_ntd__monthly_ridership_with_adjustments_vrm
   - name: stg_ntd__monthly_ridership_with_adjustments_vrh