Skip to content

Commit

Permalink
feat!: rewrite the library as app-store-web-scraper
Browse files Browse the repository at this point in the history
- `app-store-scraper` has been renamed to `app-store-web-scraper`.

- `AppStore` is now `AppStoreEntry`.

- The `review()` method is now named `reviews()` and returns an iterator of
  `AppReview` objects instead of a list of dicts. This makes the API
  more pleasant to use and offers more control for the caller.

- `urllib3` is now used directly (instead of through `requests`).

- HTTP connections are now reused across requests, which makes the
  library more efficient and reduces load on the App Store servers.  In
  addition, an `AppStoreSession` class was introduced that allows
  reusing connections across multiple `AppStoreEntry` instances.

- Python >= 3.10 is now required.

- Support for podcasts has been dropped.
  • Loading branch information
denisw committed Jun 12, 2024
1 parent 3a11b58 commit 9e07c86
Show file tree
Hide file tree
Showing 17 changed files with 741 additions and 659 deletions.
7 changes: 0 additions & 7 deletions .github/PULL_REQUEST_TEMPLATE.md

This file was deleted.

123 changes: 15 additions & 108 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
name: Build
name: CI

on:
push:
branches: [ master ]
branches: [main]
paths-ignore:
- '**.md'
pull_request:
branches: [ master ]
branches: [main]
paths-ignore:
- '**.md'

Expand All @@ -15,117 +15,24 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: [3.6, 3.7, 3.8]
python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install black flake8 pytest
pip install -r requirements.txt
- name: Format with black
run: |
black . --check
- name: Lint with flake8
run: |
flake8 . --ignore=E203 --count --show-source --statistics --max-line-length=90
- name: Test with pytest
run: |
pytest
release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Check version bump and set versions as envvar
if: github.ref != 'refs/heads/master'
run: |
FILES_CHANGED=$(echo -n $(git diff origin/${{ github.base_ref }} --name-only))
if [[ $FILES_CHANGED != *.py ]]; then
echo "No Python scripts are modified"
echo "::set-env name=FILES_CHANGED::false"
exit 0
fi
SEMVER_PATTERN="([0-9]+)\.([0-9]+)\.([0-9]+)"
VERSION=$(echo -n $(git diff origin/${{ github.base_ref }} -G '__version__' app_store_scraper/__version__.py))
if [ "$VERSION" = "" ]; then
echo "Version must be bumped for every PR"
exit 1
fi
VERSION_OLD=$(echo "$VERSION" | sed -E "s/.*\-__version__ = \"($SEMVER_PATTERN).+/\1/")
VERSION_OLD_MAJOR=$(echo "$VERSION_OLD" | sed -E "s/$SEMVER_PATTERN/\1/")
VERSION_OLD_MINOR=$(echo "$VERSION_OLD" | sed -E "s/$SEMVER_PATTERN/\2/")
VERSION_OLD_PATCH=$(echo "$VERSION_OLD" | sed -E "s/$SEMVER_PATTERN/\3/")
VERSION_NEW=$(echo "$VERSION" | sed -E "s/.*\+__version__ = \"($SEMVER_PATTERN).+/\1/")
VERSION_NEW_MAJOR=$(echo "$VERSION_NEW" | sed -E "s/$SEMVER_PATTERN/\1/")
VERSION_NEW_MINOR=$(echo "$VERSION_NEW" | sed -E "s/$SEMVER_PATTERN/\2/")
VERSION_NEW_PATCH=$(echo "$VERSION_NEW" | sed -E "s/$SEMVER_PATTERN/\3/")

echo "::set-env name=VERSION_OLD::$VERSION_OLD"
echo "::set-env name=VERSION_OLD_MAJOR::$VERSION_OLD_MAJOR"
echo "::set-env name=VERSION_OLD_MINOR::$VERSION_OLD_MINOR"
echo "::set-env name=VERSION_OLD_PATCH::$VERSION_OLD_PATCH"
- name: Install Hatch
uses: pypa/hatch@install

echo "::set-env name=VERSION_NEW::$VERSION_NEW"
echo "::set-env name=VERSION_NEW_MAJOR::$VERSION_NEW_MAJOR"
echo "::set-env name=VERSION_NEW_MINOR::$VERSION_NEW_MINOR"
echo "::set-env name=VERSION_NEW_PATCH::$VERSION_NEW_PATCH"
- name: Set up Python environment
run: hatch env create

echo "Old version: $VERSION_OLD"
echo "New version: $VERSION_NEW"
- name: Check for patch version bump
if: ${{ !startsWith(github.head_ref, 'release') && github.ref != 'refs/heads/master' }}
run: |
if [ "$FILES_CHANGED" = false ]; then
echo "No Python scripts are modified"
exit 0
fi
- name: Check code and formatting
run: hatch fmt --check

if [ "$VERSION_OLD_MAJOR" = "$VERSION_NEW_MAJOR" ] &&
[ "$VERSION_OLD_MINOR" = "$VERSION_NEW_MINOR" ]; then
if (($VERSION_OLD_PATCH < $VERSION_NEW_PATCH)); then
echo "Bumped patch version $VERSION_OLD -> $VERSION_NEW"
exit 0
else
echo "Bump patch version in __version__"
exit 1
fi
else
echo "Major / minor version must be bumped in a release branch"
exit 1
fi
- name: Check for major / minor version bump
if: ${{ startsWith(github.head_ref, 'release') && github.ref != 'refs/heads/master' }}
run: |
if (($VERSION_OLD_MAJOR < $VERSION_NEW_MAJOR)); then
echo "Bumped major version $VERSION_OLD -> $VERSION_NEW"
exit 0
elif (($VERSION_OLD_MINOR < $VERSION_NEW_MINOR)); then
echo "Bumped minor version $VERSION_OLD -> $VERSION_NEW"
exit 0
else
echo "Major / minor version must be bumped for release"
exit 1
fi
- name: Create and push tag
if: github.ref == 'refs/heads/master'
run: |
version=$(echo -n $(git diff HEAD^1 -G '__version__' app_store_scraper/__version__.py))
version=$(echo "$version" | sed -E "s/.*\+__version__.*([0-9]+\.[0-9]+\.[0-9]+).+/\1/")
if [ "$version" != "" ]; then
echo "Create and push v$version tag"
git tag -f v"$version"
git push origin v"$version"
fi
- name: Run unit tests
run: hatch test
27 changes: 13 additions & 14 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,18 @@ jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: '3.x'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel twine
- name: Build and publish 🐍 📦 to PyPi
env:
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
run: |
python setup.py sdist bdist_wheel
twine upload dist/*
python-version: 3.12

- name: Install Hatch
uses: pypa/hatch@install

- name: Build package
run: hatch build

- name: Publish package
run: hatch publish
Loading

0 comments on commit 9e07c86

Please sign in to comment.