gym-adserver is an OpenAI Gym environment for reinforcement learning-based online advertising algorithms. gym-adserver is one of the official OpenAI environments.
The AdServer environment implements a typical multi-armed bandit scenario where an ad server agent must select the best advertisement (ad) to be displayed in a web page.
Each time an ad is selected, it is counted as one impression. A displayed ad can be clicked (reward = 1) or not (reward = 0), depending on the interest of the user. The agent must maximize the overall click-through rate.
- Added budget, bid, type as attributes of ads
- Introduced User class to model user preferences
- Changed the click probability to continuous functions
- Implemented Thompson Sampling Agent and Explore-Then-Commit Agent
| Attribute | Value | Notes |
|---|---|---|
| Action Space | Discrete(n) | n is the number of ads to choose from |
| Observation Space | Box(0, +inf, (2, n)) | Number of impressions and clicks for each ad |
| Actions | [0...n] | Index of the selected ad |
| Rewards | 0, 1 | 1 = clicked, 0 = not clicked |
| Render Modes | 'human' | Displays the agent's performance graphically |
You can download the source code and install the dependencies with:
git clone https://github.com/falox/gym-adserver
cd gym-adserver
pip install -e .Alternatively, you can install gym-adserver as a pip package:
pip install gym-adserverYou can test the environment by running one of the built-in agents:
python gym_adserver/agents/ucb1_agent.py --num_ads 10 --impressions 10000Or comparing multiple agents (defined in compare_agents.py):
python gym_adserver/wrappers/compare_agents.py --num_ads 10 --impressions 10000The environent will generate 10 (num_ads) ads with different performance rates and the agent, without prior knowledge, will learn to select the most performant ones. The simulation will last 10000 iterations (impressions).
A window will open and show the agent's performance and the environment's state:
The overall CTR increases over time as the agent learns what the best actions are.
During the initialization, the environment assigns to each ad a "Probability" to be clicked. Such a probability is known by the environment only and will be used to draw the rewards during the simulation. Note that the probability is modeled as a continous function depending on ad type and time (computed based on the number of impressions).
The probability is also afftected by the user ad preference modeled by the User class. Favorable ads for certain group of users will receive bonus click probability.
The bid and budget of a single ad are randomly generated within a range, and the type is also determined at random. The bid is the amount an advertiser is willing to pay per click on an ad; budget is the total amount an advertiser allocates for an ad campaign. Once the budget runs out, the ad should not be displayed again, because the advertiser cannot pay for it any more.
The effective agent will give most impressions to the most performant ads with the budget constraint.
The gym_adserver/agents directory contains a collection of agents implementing the following strategies:
- Random
- epsilon-Greedy
- Softmax
- UCB1
Agents added in this repo:
- Thompson Sampling
- Explore-then-Commit
Each agent has different parameters to adjust and optimize its performance.
You can use the built-in agents as a starting point to implement your own algorithm.
You can run the unit test for the environment with:
pytest -v- Implement Q-learning agents
- Implement a meta-agent that exploits multiple sub-agents with different algorithms
- Implement epsilon-Greedy variants