Improve data saving (#1)

* add some config files

* update valid query fields from current docs

https://developers.facebook.com/docs/graph-api/reference/archived-ad/

* timestamp csv files, be more forgiving on input

* add new params, overhaul URL construction

- Adds ad_type and media_type so we can set those values
- changes default API version from v14.0 to v20.0
- renames some variables to match parameters sent to the API
- only sends parameters in the URL if they're set

* add output folder, update gitignores

* add some output, including rate limiting details

* rename get_parser -> argument_parser

* better organization of saved files

save files in a new timestamped folder in the output directory

* load access token from env if it's there

* use csv.writer to write csv, and write as we go

Writing each ad to the csv file as we get it instead of waiting until
we have all the ads.

* update README

* check for different types of rate limit headers

https://developers.facebook.com/docs/graph-api/overview/rate-limiting/

* set batch size default to 250 and update help

* Facebook Ads Library API -> Meta Ad Library API

* improve help

Author: tmaybe

.env-sample

@@ -0,0 +1 @@
+ACCESS_TOKEN=xxx

.gitignore

@@ -0,0 +1,6 @@
+*.csv
+.DS_Store
+.env
+__pycache__
+pyrightconfig.json
+venv

README.md

@@ -1,35 +1,119 @@
 # Ads-Library-API-Script-Repository
-Ads-Library-API-Script-Repository is a set of code examples to help user/researchers understand how the Facebook Ads Library API works. It also provides a simple command-line interface(CLI) for users to easily use the Facebook Ads Library API.
+Ads-Library-API-Script-Repository is a set of code examples to help user/researchers understand how the Meta Ad Library API works. It also provides a simple command-line interface(CLI) for users to easily use the Meta Ad Library API.
 
-## Examples
-Here's an example on how to use the CLI:
+## Setup
 
-    $ python fb_ads_library_api_cli.py -t {access_token} -f 'page_id,ad_snapshot_url,funding_entity,ad_delivery_start_time' -c 'CA' -s '.' -v count
+### Make sure you have Python 3 installed
 
-It would count the number of all polictical ads in CA(Canada);
+This command should show you a path to the executable, like `/usr/bin/python3`
+```bash
+which python3
+```
 
-Note: please replace the '{access_token}' with your [Facebook Developer access token](https://developers.facebook.com/tools/accesstoken/).
+If Python isn't installed and you're on an Apple computer, [install homebrew](https://brew.sh/) and use it to install python3
+```bash
+brew install python
+```
 
-## Requirements
-Ads-Library-API-Script-Repository requires or works with
-* Mac OS X or Linux or Window
-* Python 3.0+
-* Python Requests Library ([installation](https://docs.python-requests.org/en/master/user/install/#install))
-* Python iso3166 Library ([installation](https://pypi.org/project/iso3166/))
+You can check [Python's downloads page](https://www.python.org/downloads/) for instructions on installing on other operating systems.
 
+### Start a virtual environment
+
+Create the environment
+```bash
+python3 -m venv venv
+```
+
+Activate it
+```bash
+source venv/bin/activate
+```
+
+### Install the required packages
+```bash
+python3 -m pip install -r requirements.txt
+```
+
+## Usage
+
+To use these scripts to access the [Meta Ad Library API](https://www.facebook.com/ads/library/api), you must have a Facebook developer account, which will require you to confirm your identity (by uploading identifying documents such as a drivers license or passport) and mailing address (by entering a code that Meta sends you in the physical mail.)
+
+Once those details are confirmed, you can create a new app (an app of type "Business" will work) which will allow you to generate an access token. That token is required by these scripts to authenticate with the API. The access token can be found on the [Graph API Explorer](https://developers.facebook.com/tools/explorer/) or the [Access Token Tool](https://developers.facebook.com/tools/accesstoken/), where it's described as the "User Token".
+
+The access token can be passed to the script using the `-t` flag, or saved in a `.env` file with the key `ACCESS_TOKEN`. You can copy the `.env-sample` file in this repository to `.env` and fill in your token there.
+
+```bash
+cp .env-sample .env
+```
+
+If you choose to save the results of your query to a file, they will be saved in the `output` directory, in a folder time-stamped with the time you started the query.
+
+Here are some examples on how to use the CLI:
+
+### Count the number of political ads in Canada (CA)
+replace `{access_token}` with your token
+```python
+python3 python/fb_ads_library_api_cli.py -t {access_token} -f 'page_id,ad_snapshot_url,funding_entity,ad_delivery_start_time' -c 'CA' -s '.' -v count
+```
+
+### Search US political ads delivered after 7/20 for "coconut" and save them to a CSV file
+Assuming you've put your access token in `.env`
+```python
+python3 python/fb_ads_library_api_cli.py -f 'id,ad_creation_time,ad_creative_bodies,ad_creative_link_captions,ad_creative_link_descriptions,ad_creative_link_titles,ad_delivery_start_time,ad_delivery_stop_time,ad_snapshot_url,age_country_gender_reach_breakdown,beneficiary_payers,bylines,currency,delivery_by_region,demographic_distribution,estimated_audience_size,eu_total_reach,impressions,languages,page_id,page_name,publisher_platforms,spend,target_ages,target_gender,target_locations' -c 'US' --ad-type 'POLITICAL_AND_ISSUE_ADS' -s 'coconut' --batch-size 250 --after-date 2024-07-20 -v save_to_csv coconut_after_07_20
+```
+
+### Options
+
+You can see the available arguments by passing `--help`
+
+```bash
+python3 python/fb_ads_library_api_cli.py --help
+```
+
+```
+The Meta Ad Library API CLI Utility
+
+positional arguments:
+  action                Action to take on the ads, possible values: count,save,save_to_csv,start_time_trending
+  args                  The parameter for the specific action
+
+options:
+  -h, --help            show this help message and exit
+  -t ACCESS_TOKEN, --access-token ACCESS_TOKEN
+                        The Facebook developer access token
+  -f FIELDS, --fields FIELDS
+                        Fields to retrieve from the Ad Library API, comma-separated, no spaces
+  -s SEARCH_TERMS, --search-terms SEARCH_TERMS
+                        The terms you want to search for, space-separated
+  -c COUNTRY, --country COUNTRY
+                        Country code(s), comma-separated, no spaces
+  --search-page-ids SEARCH_PAGE_IDS
+                        A specific Facebook Page you want to search
+  --ad-active-status AD_ACTIVE_STATUS
+                        Filter by the current status of the ads at the moment the script runs, can be ALL (default), ACTIVE, INACTIVE
+  --ad-type AD_TYPE     Return this type of ad, can be ALL (default), CREDIT_ADS, EMPLOYMENT_ADS, HOUSING_ADS, POLITICAL_AND_ISSUE_ADS
+  --media-type MEDIA_TYPE
+                        Return ads that contain this type of media, can be ALL (default), IMAGE, MEME, VIDEO, NONE
+  --after-date AFTER_DATE
+                        Only return ads that started delivery after this date, in the format YYYY-MM-DD
+  --batch-size BATCH_SIZE
+                        Request records in batches of this size, default is 250
+  --retry-limit RETRY_LIMIT
+                        How many times to retry when an error occurs, default is 3
+  -v, --verbose
+```
 
 ## How Ads-Library-API-Script-Repository works
-The script will query the [Facebook Ads library API](https://www.facebook.com/ads/library/api) to get all the Ads Library information on the Facebook platform;
+The script will query the [Meta Ad Library API](https://www.facebook.com/ads/library/api) to get all the Ad Library information on the Facebook platform;
 
-## Full documentation
-You can find the full documentation here: (--to-be-added--)
 
-## More about Facebook Ads Library
+## More about Meta Ad Library
 * Website: https://www.facebook.com/ads/library
 * Report: https://www.facebook.com/ads/library/report
 * API: https://www.facebook.com/ads/library/api
 
 See the [CONTRIBUTING](CONTRIBUTING.md) file for how to help out.
 
+
 ## License
 Ads-Library-API-Script-Repository is licensed under the Facebook Platform License, as found in the LICENSE file.

output/.gitignore

@@ -0,0 +1,3 @@
+# ignore everything in this directory except this file
+*
+!.gitignore

python/fb_ads_library_api.py

@@ -9,6 +9,7 @@
 import json
 import re
 from datetime import datetime
+from json.decoder import JSONDecodeError
 
 import requests
 
@@ -21,52 +22,64 @@ def get_ad_archive_id(data):
 
 
 class FbAdsLibraryTraversal:
-    default_url_pattern = (
-        "https://graph.facebook.com/{}/ads_archive?access_token={}&"
-        + "fields={}&search_terms={}&ad_reached_countries={}&search_page_ids={}&"
-        + "ad_active_status={}&limit={}"
-    )
-    default_api_version = "v14.0"
+    default_url_parameters = [
+        "access_token",
+        "ad_active_status",
+        "ad_reached_countries",
+        "ad_type",
+        "fields",
+        "limit",
+        "media_type",
+        "search_page_ids",
+        "search_terms",
+    ]
+    default_url_pattern = "https://graph.facebook.com/{}/ads_archive?"
+    default_api_version = "v20.0"
 
     def __init__(
         self,
         access_token,
         fields,
-        search_term,
-        country,
+        search_terms,
+        ad_reached_countries,
+        ad_type="ALL",
+        media_type="ALL",
         search_page_ids="",
         ad_active_status="ALL",
         after_date="1970-01-01",
-        page_limit=500,
+        limit=250,
         api_version=None,
         retry_limit=3,
     ):
         self.page_count = 0
         self.access_token = access_token
         self.fields = fields
-        self.search_term = search_term
-        self.country = country
+        self.search_terms = search_terms
+        self.ad_reached_countries = ad_reached_countries
+        self.ad_type = ad_type
+        self.media_type = media_type
         self.after_date = after_date
         self.search_page_ids = search_page_ids
         self.ad_active_status = ad_active_status
-        self.page_limit = page_limit
+        self.limit = limit
         self.retry_limit = retry_limit
         if api_version is None:
             self.api_version = self.default_api_version
         else:
             self.api_version = api_version
 
     def generate_ad_archives(self):
-        next_page_url = self.default_url_pattern.format(
-            self.api_version,
-            self.access_token,
-            self.fields,
-            self.search_term,
-            self.country,
-            self.search_page_ids,
-            self.ad_active_status,
-            self.page_limit,
-        )
+        # construct the URL
+        next_page_url = self.default_url_pattern.format(self.api_version)
+        params_to_add = []
+
+        for param in self.default_url_parameters:
+            param_value = getattr(self, param)
+            if param_value:
+                params_to_add.append(f"{param}={param_value}")
+
+        next_page_url += "&".join(params_to_add)
+
         return self.__class__._get_ad_archives_from_url(
             next_page_url, after_date=self.after_date, retry_limit=self.retry_limit
         )
@@ -75,13 +88,32 @@ def generate_ad_archives(self):
     def _get_ad_archives_from_url(
         next_page_url, after_date="1970-01-01", retry_limit=3
     ):
+        rate_limit_headers = [
+            "x-ad-account-usage",
+            "x-app-usage",
+            "x-business-use-case-usage",
+        ]
         last_error_url = None
         last_retry_count = 0
         start_time_cutoff_after = datetime.strptime(after_date, "%Y-%m-%d").timestamp()
 
         while next_page_url is not None:
+            print(">> requesting page at " + next_page_url)
             response = requests.get(next_page_url)
             response_data = json.loads(response.text)
+            print(">> got response!")
+
+            # get rate limiting details from headers
+            for header in rate_limit_headers:
+                usage = response.headers.get(header)
+                if usage:
+                    try:
+                        print(f">> {header}")
+                        print(json.loads(usage))
+                    except JSONDecodeError as err:
+                        print(">> error trying to get rate limit details from headers!")
+                        print(err)
+
             if "error" in response_data:
                 if next_page_url == last_error_url:
                     # failed again