Skip to content

Commit

Permalink
MAJOR RELEASE: VERSION 1.0. BREAKS BACKWARD COMPATABILITYgit add . SE…
Browse files Browse the repository at this point in the history
…E README.md FOR COMPLETE CHANGELOG
patx committed Jan 10, 2025
1 parent 2dd930d commit d01baec
Showing 10 changed files with 553 additions and 1,190 deletions.
309 changes: 181 additions & 128 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,196 +1,249 @@
![Download badge](http://pepy.tech/badge/pickledb)
# **pickleDB: Your Lightweight, High-Speed Key-Value Store**

# pickleDB
pickleDB is lightweight, fast, and simple database based on the
[json](https://docs.python.org/3/library/json.html) module.
And it's BSD licensed!
## **Fast. Simple. Reliable.**
Unlock the power of effortless data storage with **pickleDB**—the no-fuss, blazing-fast key-value store designed for Python developers. Whether you're building a small script or a performant microservice, pickleDB delivers simplicity and speed with the reliability you can count on.

---

## pickleDB is Fun
```python
>>> import pickledb
## **Why Choose pickleDB?**

>>> db = pickledb.load('test.db', False)
### **Blazing Speed**
Backed by the high-performance [orjson](https://pypi.org/project/orjson/) library, pickleDB handles millions of records with ease. Perfect for applications where every millisecond counts.

>>> db.set('key', 'value')
### **Ridiculously Easy to Use**
With its minimalist API, pickleDB makes adding, retrieving, and managing your data as simple as writing a Python list. No steep learning curves. No unnecessary complexity.

>>> db.get('key')
'value'
### **Rock-Solid Reliability**
Your data deserves to be safe. Atomic saves ensure your database remains consistent—even if something goes wrong.

>>> db.dump()
True
```
### **Pythonic Flexibility**
Store strings, lists, dictionaries, and more—all with native Python operations. No need to learn special commands. If you know Python, you already know pickleDB.

## Installation
- Just add the `pickledb.py` file to your working directory or use `pip install pickledb`.
- pickleDB also includes a simplified and faster version using orjson called `pkldb.py`, to use this add it to your working directory and note the slight difference in setup `from pkldb.py import pkldb` then `db = pkldb('example.json')`
---

# PickleDB Documentation
## **Getting Started**

## Introduction
PickleDB is a lightweight, file-based key-value store with optional support for time-to-live (TTL). It provides a simple and intuitive API for storing and managing data persistently.
### **Install in Seconds**
pickleDB is available on PyPI. Get started with just one command:
```bash
pip install pickledb
```

---
### **Your First pickleDB**
```python
from pickledb import PickleDB

## Table of Contents
1. **Basic Usage**
2. **Key-Value Methods**
3. **List Methods**
4. **Dictionary Methods**
5. **Enhanced Features**
# Initialize the database
db = PickleDB('my_database.db')

---
# Add a key-value pair
db.set('greeting', 'Hello, world!')

## 1. Basic Usage
```python
from pickledb_enhanced import load
# Retrieve the value
print(db.get('greeting')) # Output: Hello, world!

db = load('mydb.json', auto_dump=True, enable_ttl=True)
# Save the data to disk
db.save()
```
- `auto_dump`: Automatically save changes to the file.
- `enable_ttl`: Enable TTL support for expiring keys.
It’s that simple! In just a few lines, you have a fully functioning key-value store.

---

## 2. Key-Value Methods
## **More Examples to Get You Inspired**

### `set(key, value, ttl=None)`
Set a key-value pair in the database.

### `get(key)`
Retrieve the value associated with a key.
### **Store and Retrieve Complex Data**
PickleDB works seamlessly with Python data structures. Example:
```python
# Store a dictionary
db.set('user', {'name': 'Alice', 'age': 30, 'city': 'Wonderland'})

### `exists(key)`
Check if a key exists.
# Retrieve and update it
user = db.get('user')
user['age'] += 1

### `rem(key)`
Remove a key from the database.
# Save the updated data
db.set('user', user)
print(db.get('user')) # Output: {'name': 'Alice', 'age': 31, 'city': 'Wonderland'}
```

### `getall()`
Get all keys in the database.
### **Use Lists for Dynamic Data**
Handle lists with ease:
```python
# Add a list of items
db.set('tasks', ['Write code', 'Test app', 'Deploy'])

### `clear()`
Clear all keys.
# Retrieve and modify
tasks = db.get('tasks')
tasks.append('Celebrate')
db.set('tasks', tasks)

### `deldb()`
Delete the database file.
print(db.get('tasks')) # Output: ['Write code', 'Test app', 'Deploy', 'Celebrate']
```

---
### **Store Configurations**
Create a simple, persistent configuration store:
```python
# Set configuration options
db.set('config', {'theme': 'dark', 'notifications': True})

# Access and update settings
config = db.get('config')
config['notifications'] = False
db.set('config', config)
print(db.get('config')) # Output: {'theme': 'dark', 'notifications': False}
```

## 3. List Methods
### **Session Management**
Track user sessions effortlessly:
```python
# Add session data
db.set('session_12345', {'user_id': 1, 'status': 'active'})

### `lcreate(name)`
Create a new list in the database.
# End a session
session = db.get('session_12345')
session['status'] = 'inactive'
db.set('session_12345', session)

### `ladd(name, value)`
Add a value to an existing list.
print(db.get('session_12345')) # Output: {'user_id': 1, 'status': 'inactive'}
```

### `lgetall(name)`
Retrieve all values from a list.
---

### `lsort(name, reverse=False)`
Sort a list in ascending or descending order.
- `reverse`: Sort in descending order if `True`.
## **Performance Highlights**

### `lremove(name, value)`
Remove a value from a list.
pickleDB demonstrates strong performance for handling large-sized datasets:

### `lgetrange(name, start, end)`
Retrieve a range of values from a list.
- `start`: Start index.
- `end`: End index.
| Entries | Memory Load Time | Retrieval Time | Save Time |
|--------------|------------------|----------------|-----------|
| **1M** | 1.21 sec | 0.90 sec | 0.17 sec |
| **10M** | 14.11 sec | 10.30 sec | 1.67 sec |
| **50M** | 93.79 sec | 136.42 sec | 61.08 sec |

### `llen(name)`
Get the length of a list.
Tests were performed on a StarLabs StarLite Mk IV (Quad-Core Intel® Pentium® Silver N5030 CPU @ 1.10GHz w/ 8GB memory) running elementary OS 7.1 Horus.

---

## 4. Dictionary Methods
## **Minimal, Powerful API**

pickleDB offers a clean and Pythonic API for managing data efficiently:

### `dcreate(name)`
Create a new dictionary in the database.
### **`set(key, value)`**
Add or update a key-value pair:
```python
# Add a new key-value pair
db.set('username', 'admin')

### `dadd(name, key, value)`
Add a key-value pair to a dictionary.
# Update an existing key-value pair
db.set('username', 'superadmin')
print(db.get('username')) # Output: 'superadmin'
```

### `dget(name, key)`
Retrieve a value from a dictionary.
### **`get(key)`**
Retrieve the value associated with a key:
```python
# Get the value for a key
print(db.get('username')) # Output: 'superadmin'

### `dgetall(name)`
Retrieve all key-value pairs from a dictionary.
# Attempt to retrieve a non-existent key
print(db.get('nonexistent_key')) # Output: None
```

### `dremove(name, key)`
Remove a key from a dictionary.
### **`all()`**
Get a list of all keys:
```python
# Add multiple keys
db.set('item1', 'value1')
db.set('item2', 'value2')

### `dmerge(name, other_dict)`
Merge another dictionary into an existing dictionary.
# Retrieve all keys
print(db.all()) # Output: ['username', 'item1', 'item2']
```

### `dkeys(name)`
Get all keys from a dictionary.
### **`remove(key)`**
Delete a key and its value:
```python
# Remove a key-value pair
db.remove('item1')
print(db.all()) # Output: ['username', 'item2']
```

### `dvalues(name)`
Get all values from a dictionary.
### **`purge()`**
Clear all data in the database:
```python
# Clear the database
db.purge()
print(db.all()) # Output: []
```

### **`save()`**
Persist the database to disk:
```python
# Save the current state of the database
db.save()
print("Database saved successfully!")
```

---

## 5. Enhanced Features
## **Key Improvements in Version 1.0**

### **TTL Support**
- Expire keys automatically after a given time.
pickleDB 1.0 is a reimagined version designed for speed, simplicity, and reliability. Key changes include:

### **File Compression**
- Compress the database file to save space.
- **Atomic Saves**: Ensures data integrity during writes, eliminating potential corruption issues.
- **Faster Serialization**: Switched to `orjson` for significantly improved speed.
- **Streamlined API**: Removed legacy methods (e.g., `ladd`, `dmerge`) in favor of native Python operations.
- **Unified Handling of Data Types**: Treats all Python-native types (lists, dicts, etc.) as first-class citizens.
- **Explicit Saves**: The `auto_save` feature was removed to provide users greater control and optimize performance.

### **Automatic Persistence**
- Save changes automatically using `auto_dump`.
If backward compatibility is essential, version 0.9 is still available:
- View the legacy code [here](https://gist.github.com/patx/3ad47fc3814d7293feb902f6ab49c48f).
- Install it by:
```bash
pip uninstall pickledb
```
Then download the legacy file and include it in your project.

---

## Example Usage
## **Limitations**

### **Working with Lists**
```python
# Create a list and add values
db.lcreate('mylist')
db.ladd('mylist', 'item1')
db.ladd('mylist', 'item2')
While pickleDB is powerful, it’s important to understand its limitations:

# Sort the list
db.lsort('mylist') # ['item1', 'item2']
- **Memory Usage**: The entire dataset is loaded into memory, which might be a constraint on systems with limited RAM for extremely large datasets.
- **Single-Threaded**: The program is not thread-safe. For concurrent access, use external synchronization like Python's `RLock()`.
- **Blocking Saves**: Saves are blocking by default. To achieve non-blocking saves, use asynchronous wrappers.
- **Lack of Advanced Features**: pickleDB is designed for simplicity, so it may not meet the needs of applications requiring advanced database features.

# Get a range of values
db.lgetrange('mylist', 0, 1) # ['item1']
For projects requiring more robust solutions, consider alternatives like **[kenobiDB](Https://github.com/patx/kenobi)**, [Redis](http://redis.io/), [SQLite](https://www.sqlite.org/), or [MongoDB](https://www.mongodb.com/).

# Remove an item
db.lremove('mylist', 'item1')
```
---

### **Working with Dictionaries**
```python
# Create a dictionary and add values
db.dcreate('mydict')
db.dadd('mydict', 'key1', 'value1')
db.dadd('mydict', 'key2', 'value2')
## **Asynchronous Saves**
Want non-blocking saves? You can implement an async wrapper to handle saves in the background. This is particularly useful for applications that need high responsiveness without delaying due to disk operations, like small web applications. Check out examples [here](https://gist.github.com/patx/5c12d495ff142f3262325eeae81eb000).

# Merge another dictionary
db.dmerge('mydict', {'key3': 'value3'})
---

# Get all keys and values
db.dkeys('mydict') # ['key1', 'key2', 'key3']
db.dvalues('mydict') # ['value1', 'value2', 'value3']
## **Community & Contributions**

# Remove a key
db.dremove('mydict', 'key1')
```
### **Join the Community**
We’re passionate about making pickleDB better every day. Got ideas, feedback, or an issue to report? Let’s connect:
- **File an Issue**: [GitHub Issues](https://github.com/patx/pickledb/issues)
- **Ask Questions**: Reach out to our growing community of users and developers.

---
### **Contribute to pickleDB**
Want to leave your mark? Help us make pickleDB even better:
- **Submit a Pull Request**: Whether it's fixing a bug, improving the documentation, or adding a feature, we’d love your contributions.
- **Suggest New Features**: Share your ideas to make pickleDB more powerful.

## Notes
- Always ensure proper file permissions for the database file.
- Use thread-safe practices when accessing the database concurrently.
Together, we can build a better tool for everyone.

---

## Changelog
- **Enhanced Features**: Added methods for list sorting, removal, range fetching, and dictionary merging.
## **Documentation**

Explore the full capabilities of pickleDB with our detailed documentation:
- **API Reference**: [Commands and Examples](https://patx.github.io/pickledb/commands.html)
- [GitHub Repository](https://github.com/patx/pickledb)
- [Installation Details (PyPI)](http://pypi.python.org/pypi/pickleDB)

Whether you're a beginner or an experienced developer, these resources will guide you through everything pickleDB has to offer.
Binary file removed docs/.commands.html.swp
Binary file not shown.
131 changes: 77 additions & 54 deletions docs/commands.html
Original file line number Diff line number Diff line change
@@ -1,59 +1,82 @@

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<!DOCTYPE html>
<html lang="en">
<head>
<link rel="shortcut icon" href="http://packages.python.org/pickleDB/favicon.ico" type="image/x-icon">
<link rel="icon" href="http://packages.python.org/pickleDB/favicon.ico" type="image/x-icon">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>pickleDB - simple key-value database</title>
<style type="text/css">
.logo { width: 800px; margin: 40px auto; }
.body { font-family: "helvetica"; width: 540px; margin: 40px auto; }
img.c1 { position: absolute; top: 0; right: 0; border: 0; }
span.c2 { color: #8F5902 }
span.c9 { color: #4E9A06 }
a { color: #008000; border-bottom: 0px dotted white; text-decoration: none; }
a:hover { color: #008000; border-bottom: 0px solid white; }
</style>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="shortcut icon" href="https://packages.python.org/pickleDB/favicon.ico" type="image/x-icon">
<link rel="icon" href="https://packages.python.org/pickleDB/favicon.ico" type="image/x-icon">
<title>pickleDB API Documentation</title>
<style>
.logo { width: 800px; margin: 40px auto; }
.body { font-family: "Helvetica", sans-serif; width: 540px; margin: 40px auto; }
img.c1 { position: absolute; top: 0; right: 0; border: 0; }
span.c2 { color: #8F5902; }
span.c9 { color: #4E9A06; }
a { color: #008000; border-bottom: 0px dotted white; text-decoration: none; }
a:hover { color: #008000; border-bottom: 0px solid white; }
</style>
</head>
<body>
<div class="logo"><a href="index.html"><img src="logo.png" alt="pickleDB logo"></a></div>
<div class="body">
<h1>Current Commands</h1>
<p><code><span class="c2">LOAD</span> <span class="c9">path</span> <span class="c9">auto_dump</span></code> &rarr; Load a database from a path with auto_dump enabled or not <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">SET</span> <span class="c9">key</span> <span class="c9">value</span></code> &rarr; Set the value of a <em>str</em> key <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">GET</span> <span class="c9">key</span></code> &rarr; Get the value of a key <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">GETALL</span></code> &rarr; Return a list of all keys in database <small><em>(available since 0.4)</em></small></p>
<p><code><span class="c2">REM</span> <span class="c9">key</span></code> &rarr; Delete a key <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">APPEND</span> <span class="c9">key</span> <span class="c9">more</span></code> &rarr; Add more to a key's value <small><em>(available since 0.1.3)</em></small></p>
<p><code><span class="c2">EXISTS</span> <span class="c9">key</span></code> &rarr; Determine if a key exists <small><em>(available since 0.7.2)</em></small></p>
<p><code><span class="c2">TOTALKEYS</span> <span class="c9">name</span></code> &rarr; Get the total number of keys in whole database or in a specified (<code>name</code>) dict or list <small><em>(available since 0.7.3)</em></small></p>
<p><code><span class="c2">LCREATE</span> <span class="c9">name</span></code> &rarr; Create a list with <em>str</em> name <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">LADD</span> <span class="c9">name</span> <span class="c9">value</span></code> &rarr; Add a value to a list <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">LGETALL</span> <span class="c9">name</span></code> &rarr; Return all values in a list <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">LEXTEND</span> <span class="c9">name</span> <span class="c9">seq</span></code> &rarr; Extend a list with a sequence <small><em>(available since 0.6)</em></small></p>
<p><code><span class="c2">LGET</span> <span class="c9">name</span> <span class="c9">pos</span></code> &rarr; Return one value in a list <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">LRANGE</span> <span class="c9">name</span> <span class="c9">start</span> <span class="c9">end</span></code> &rarr; Return all the values from a given range in a list <small><em></em></small></p>
<p><code><span class="c2">LREMLIST</span> <span class="c9">name</span></code> &rarr; Remove a list and all of its values <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">LREMVALUE</span> <span class="c9">name</span> <span class="c9">value</span></code> &rarr; Remove a value from list <code>name</code> <small><em>(available since 0.8.2)</em></small></p>
<p><code><span class="c2">LPOP</span> <span class="c9">name</span> <span class="c9">pos</span></code> &rarr; Remove one value in a list <small><em>(available since 0.1)</em></small></p>
<p><code><span class="c2">LLEN</span> <span class="c9">name</span></code> &rarr; Return the length of a list <small><em>(available since 0.6)</em></small></p>
<p><code><span class="c2">LAPPEND</span> <span class="c9">name</span> <span class="c9">pos</span> <span class="c9">more</span></code> &rarr; Add more to a value in a list <small><em>(available since 0.1.3)</em></small></p>
<p><code><span class="c2">LEXISTS</span> <span class="c9">name</span> <span class="c9">value</span></code> &rarr; Determine if a value is in a certain list <small><em>(available since 0.7.2)</em></small></p>
<p><code><span class="c2">DCREATE</span> <span class="c9">name</span></code> &rarr; Create a dict with <em>str</em> name <small><em>(available since 0.2.2)</em></small></p>
<p><code><span class="c2">DADD</span> <span class="c9">name</span> <span class="c9">pair</span></code> &rarr; Add a key-value pair to a dict, <code>pair</code> is a tuple <small><em>(available since 0.2.2)</em></small></p>
<p><code><span class="c2">DGETALL</span> <span class="c9">name</span></code> &rarr; Return all key-value pairs from a dict <small><em>(available since 0.2.2)</em></small></p>
<p><code><span class="c2">DGET</span> <span class="c9">name</span> <span class="c9">key</span></code> &rarr; Return the value for a key in a dict <small><em>(available since 0.2.2)</em></small></p>
<p><code><span class="c2">DKEYS</span> <span class="c9">name</span></code> &rarr; Return all the keys for a dict <small><em>(available since 0.6)</em></small></p>
<p><code><span class="c2">DVALS</span> <span class="c9">name</span></code> &rarr; Return all the values for a dict <small><em>(available) since 0.6)</em></small></p>
<p><code><span class="c2">DEXISTS</span> <span class="c9">name</span> <span class="c9">key</span></code> &rarr; Determine if a key exists <small><em>(available since 0.6)</em></small></p>
<p><code><span class="c2">DREM</span> <span class="c9">name</span></code> &rarr; Remove a dict and all of its pairs <small><em>(available since 0.2.2)</em></small></p>
<p><code><span class="c2">DPOP</span> <span class="c9">name</span> <span class="c9">key</span></code> &rarr; Remove one key-value in a dict <small><em>(available since 0.2.2)</em></small></p>
<p><code><span class="c2">DMERGE</span> <span class="c9">name1</span> <span class="c9">name2</span> <span class="c9">name3</span></code> &rarr; Merge <code>name1</code> and <code>name2</code> into a new dict: <code>name3</code> <small><em>(available since 0.7.3)</em></small>
<p><code><span class="c2">DELDB</span></code> &rarr; Delete everything from the database <small><em>(available since 0.2.1)</em></small></p>
<p><code><span class="c2">DUMP</span></code> &rarr; Save the database from memory to a file specified in <code>LOAD</code> <small><em>(available since 0.3)</em></small></p>
<h1>Suggestions</h1>
<p>If you would like to suggest a command, you can create an <a href="http://github.com/patx/pickledb/issues">issue on GitHub</a>.</p>
</div>
<div class="logo">
<a href="index.html">
<img src="logo.png" alt="pickleDB logo">
</a>
</div>
<div class="body">
<h1>API Documentation</h1>
<p>For examples and help on getting started/installation see the <a href="https://github.com/patx/pickledb/?tab=readme-ov-file#readme">README on GitHub.</a></p>

<p>
<strong>Class Initialization</strong>
</p>

<p><code><span class="c2">PickleDB</span>(<span class="c9">path</span>)</code></p>
<p>Initialize a PickleDB instance with the specified path.</p>
<ul>
<li><span class="c9">path</span>: The path to the database file.</li>
</ul>

<p>
<strong><code><span class="c2">PickleDB</span></code> Class Methods</strong>
</p>

<p><code><span class="c2">set</span>(<span class="c9">key</span>, <span class="c9">value</span>)</code> &rarr; Add or update a key-value pair in the database.</p>
<ul>
<li><span class="c9">key</span>: The key to set. Converted to string if not already.</li>
<li><span class="c9">value</span>: The value to associate with the key. This can be any JSON serializable Python data type.</li>
<li><span class="c9">Returns</span>: <em>True</em>.</li>
</ul>

<p><code><span class="c2">get</span>(<span class="c9">key</span>)</code> &rarr; Retrieve the value associated with a key.</p>
<ul>
<li><span class="c9">key</span>: The key to retrieve.</li>
<li><span class="c9">Returns</span>: The value associated with the key, or <em>None</em> if the key does not exist.</li>
</ul>

<p><code><span class="c2">all</span>()</code> &rarr; Retrieve a list of all keys in the database.</p>
<ul>
<li><span class="c9">Returns</span>: A list of keys.</li>
</ul>

<p><code><span class="c2">remove</span>(<span class="c9">key</span>)</code> &rarr; Delete a key and its value from the database.</p>
<ul>
<li><span class="c9">key</span>: The key to delete.</li>
<li><span class="c9">Returns</span>: <em>True</em> if the key was deleted, or <em>False</em> if the key does not exist.</li>
</ul>

<p><code><span class="c2">purge</span>()</code> &rarr; Clear all keys and values from the database.</p>
<ul>
<li><span class="c9">Returns</span>: <em>True</em>.</li>
</ul>

<p><code><span class="c2">save</span>()</code> &rarr; Save the current state of the database to the file.</p>
<ul>
<li><span class="c9">Returns</span>: <em>True</em> if the operation succeeds, or <em>False</em> otherwise.</li>
<li>For larger datasets consider using a non-blocking wrapper around this method <a href="https://gist.github.com/patx/5c12d495ff142f3262325eeae81eb000">like this</a>.
</ul>

<h1>Suggestions</h1>
<p>If you would like to suggest an improvement or report an issue, please create an <a href="https://github.com/patx/pickledb/issues">issue on GitHub</a>.</p>
</div>
</body>
</html>
126 changes: 71 additions & 55 deletions docs/index.html
Original file line number Diff line number Diff line change
@@ -1,66 +1,82 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<!DOCTYPE html>
<html lang="en">
<head>
<link rel="shortcut icon" href="https://pythonhosted.org/pickleDB/favicon.ico" type="image/x-icon">
<link rel="icon" href="https://pythonhosted.org/pickleDB/favicon.ico" type="image/x-icon">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>pickleDB - simple key-value database</title>

<style type="text/css">
.logo { width: 800px; margin: 40px auto; }
.body { font-family: "helvetica"; width: 540px; margin: 40px auto; }
img.c1 { position: absolute; top: 0; right: 0; border: 0; }
span.c2 { color: #8F5902 }
span.c9 { color: #4E9A06 }
a { color: #008000; border-bottom: 1px dotted #008000; text-decoration: none; }
a:hover { color: #008000; border-bottom: 1px solid #008000; }
</style>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="shortcut icon" href="https://pythonhosted.org/pickleDB/favicon.ico" type="image/x-icon">
<link rel="icon" href="https://pythonhosted.org/pickleDB/favicon.ico" type="image/x-icon">
<title>pickleDB - Simple Key-Value Database</title>
<style>
.logo {
width: 800px;
margin: 40px auto;
}
.body {
font-family: "Helvetica", sans-serif;
width: 540px;
margin: 40px auto;
}
img.c1 {
position: absolute;
top: 0;
right: 0;
border: 0;
}
span.c2 {
color: #8F5902;
}
span.c9 {
color: #4E9A06;
}
a {
color: #008000;
border-bottom: 1px dotted #008000;
text-decoration: none;
}
a:hover {
color: #008000;
border-bottom: 1px solid #008000;
}
</style>
</head>
<body>
<div class="logo"><img src="logo.png" alt="pickleDB logo"></div>
<div class="body">
<h1>Welcome</h1>
<strong>pickleDB is a lightweight and simple key-value store.</strong>
It is built upon Python's <a href="https://docs.python.org/3/library/json.html">json</a>
module and was inspired by <a href="http://redis.io/">redis</a>. It is licensed
with the BSD three-clause license.
<h1>pickleDB is Fun</h1>
<code>
<span class="c2">&gt;&gt;&gt;</span> <span class="c9">import</span> pickledb
<br /><br />
<span class="c2">&gt;&gt;&gt;</span> db = pickledb.load(<span class="c9">'example.db', False</span>)
<br /><br />
<div class="logo">
<img src="logo.png" alt="pickleDB logo">
</div>
<div class="body">
<h1>Welcome</h1>
<p><strong>pickleDB is a lightweight, simple and fast key-value store.</strong>
It is built upon the <a href="https://pypi.org/project/orjson/">orjson</a> module for extremely high performance and was inspired by <a href="http://redis.io/">redis</a>. It is licensed under the BSD three-clause license.</p>

<h2>pickleDB is Fun</h2>
<pre><code>
<span class="c2">&gt;&gt;&gt;</span> <span class="c9">from</span> pickledb <span class="c9">import</span> pickleDB

<span class="c2">&gt;&gt;&gt;</span> db = PickleDB(<span class="c9">'example.json'</span>)

<span class="c2">&gt;&gt;&gt;</span> db.set(<span class="c9">'key', 'value'</span>)
<br />
True
<br /><br />

<span class="c2">&gt;&gt;&gt;</span> db.get(<span class="c9">'key'</span>)
<br />
'value'
<br /><br />
<span class="c2">&gt;&gt;&gt;</span> db.dump()
<br />

<span class="c2">&gt;&gt;&gt;</span> db.save()
True
</code>
<h1>And Easy to Install</h1>
<code>
</code></pre>

<h2>And Easy to Install</h2>
<pre><code>
<span class="c2">$</span> pip install <span class="c9">pickledb</span>
</code>
<h1>More Information</h1>
You can view all of pickleDB's commands and what they do <a href="commands.html">here</a>.<br />
<br />
pickleDB was written by <a href="http://patx.github.io">Harrison Erd</a>. If you would like
to file an issue report or fork the project,
check out the <a href="http://github.com/patx/pickledb">Github project page</a>.
You can also take a look at pickleDB <a href="https://pypi.python.org/pypi/pickleDB">on PyPI</a>.<br />
<br />
pickleDB got its name from Python's "pickle" module, which it previously used.
However, now pickleDB uses the "json" module. It is faster and cleaner.
But the name stuck!
<br />
<br />
<br>
<a href="http://github.com/patx/pickledb"><img style="position: fixed; top: 0; right: 0; border: 0;" src="http://s3.amazonaws.com/github/ribbons/forkme_right_gray_6d6d6d.png" alt="Fork me on GitHub"></a>
</div>
</code></pre>

<h2>More Information</h2>
<p>You can view all of pickleDB's commands and what they do <a href="commands.html">here</a>.</p>
<p>pickleDB was written by <a href="http://patx.github.io">Harrison Erd</a>. If you would like to file an issue report or fork the project, check out the <a href="https://github.com/patx/pickledb">Github project page</a>. You can also take a look at pickleDB <a href="https://pypi.python.org/pypi/pickleDB">on PyPI</a>.</p>

<a href="https://github.com/patx/pickledb">
<img style="position: fixed; top: 0; right: 0; border: 0;" src="https://github.blog/wp-content/uploads/2008/12/forkme_right_green_007200.png" alt="Fork me on GitHub">
</a>
</div>
</body>
</html>

534 changes: 97 additions & 437 deletions pickledb.py

Large diffs are not rendered by default.

169 changes: 0 additions & 169 deletions pkldb/README.md

This file was deleted.

170 changes: 0 additions & 170 deletions pkldb/pkldb.py

This file was deleted.

60 changes: 0 additions & 60 deletions pkldb/pkldb_tests.py

This file was deleted.

45 changes: 22 additions & 23 deletions setup.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@

"""
pickleDB
--------
pickleDB is lightweight, fast, and simple database based on Python's own
json module. And it's BSD licensed!
pickleDB is lightweight, fast, and simple database based on the orjson module. And it's BSD licensed!
pickleDB is Fun
```````````````
@@ -13,7 +12,7 @@
>>> import pickledb
>>> db = pickledb.load('test.db', False)
>>> db = pickledb.load('test.db')
>>> db.set('key', 'value')
@@ -31,33 +30,31 @@
$ pip install pickledb
Links
`````
* `website <https://patx.github.io/pickledb>`_
* `documentation <http://patx.github.io/pickledb/commands.html>`_
* `pypi <http://pypi.python.org/pypi/pickleDB>`_
* `github repo <https://github.com/patx/pickledb>`_
* `Website <https://patx.github.io/pickledb>`_
* `Documentation <http://patx.github.io/pickledb/commands.html>`_
* `PyPI <http://pypi.python.org/pypi/pickleDB>`_
* `Github Repo <https://github.com/patx/pickledb>`_
Latest Release Notes (version: 0.9)
```````````````````````````````````
* Now load() uses *'rt'* mode instead of 'rb' (0.9.2)
* Change lrem(name) to *lremlist(name)* (0.9)
* Add *lremvalue(name, value)* (0.9)
* Add load() option to use sigterm handler or not (0.9)
* All *keys* must now be strings (0.8)
* All *names* for lists must now be strings (0.8)
* All *names* for dicts must now be strings (0.8)
* The get(key) function now returns *False* instead of None if there is no key (0.8)
* Switched to Python's built in json module from simplejson (0.8.1)
Key Improvements in Version 1.0
```````````````````````````````
*pickleDB 1.0 is a reimagined version designed for speed, simplicity, and reliability. This version is NOT backwards compatible. Key changes include:
* Atomic Saves: Ensures data integrity during writes, eliminating potential corruption issues.
* Faster Serialization**: Switched to `orjson` for significantly improved speed.
* Streamlined API**: Removed legacy methods (e.g., `ladd`, `dmerge`) in favor of native Python operations.
* Unified Handling of Data Types**: Treats all Python-native types (lists, dicts, etc.) as first-class citizens.
* Explicit Saves**: The `auto_save` feature was removed to provide users greater control and optimize performance.
"""
from distutils.core import setup
"""

setup(name="pickleDB",
version="0.9.3",
version="1.0",
description="A lightweight and simple database using json.",
long_description=__doc__,
author="Harrison Erd",
@@ -69,5 +66,7 @@
"License :: OSI Approved :: BSD License",
"Intended Audience :: Developers",
"Topic :: Database" ],
py_modules=['pickledb'],)
py_modules=['pickledb'],
install_requires=['orjson'],
)

199 changes: 105 additions & 94 deletions tests.py
Original file line number Diff line number Diff line change
@@ -1,115 +1,126 @@

import unittest
import os
import time
from pickledb import load
import signal
from pickledb import PickleDB # Adjust the import path if needed

class TestPickleDBEnhanced(unittest.TestCase):

class TestPickleDB(unittest.TestCase):
def setUp(self):
self.db_file = "test_db.json"
self.db = load(self.db_file, auto_dump=True, enable_ttl=True)
"""Set up a PickleDB instance with a real file."""
self.test_file = "test_pickledb.json"
self.db = PickleDB(self.test_file, auto_dump=False)

def tearDown(self):
if os.path.exists(self.db_file):
os.remove(self.db_file)
if os.path.exists(f"{self.db_file}.gz"):
os.remove(f"{self.db_file}.gz")

# Enhanced List Features
def test_lsort(self):
self.db.lcreate("test_list")
self.db.ladd("test_list", 3)
self.db.ladd("test_list", 1)
self.db.ladd("test_list", 2)
self.assertEqual(self.db.lsort("test_list"), [1, 2, 3])
self.assertEqual(self.db.lsort("test_list", reverse=True), [3, 2, 1])

def test_lremove(self):
self.db.lcreate("test_list")
self.db.ladd("test_list", "item1")
self.db.ladd("test_list", "item2")
self.assertTrue(self.db.lremove("test_list", "item1"))
self.assertEqual(self.db.lgetall("test_list"), ["item2"])
self.assertFalse(self.db.lremove("test_list", "nonexistent"))

def test_lgetrange(self):
self.db.lcreate("test_list")
self.db.ladd("test_list", "a")
self.db.ladd("test_list", "b")
self.db.ladd("test_list", "c")
self.assertEqual(self.db.lgetrange("test_list", 0, 2), ["a", "b"])

def test_llen(self):
self.db.lcreate("test_list")
self.db.ladd("test_list", "a")
self.db.ladd("test_list", "b")
self.assertEqual(self.db.llen("test_list"), 2)

# Enhanced Dictionary Features
def test_dremove(self):
self.db.dcreate("test_dict")
self.db.dadd("test_dict", "key1", "value1")
self.assertTrue(self.db.dremove("test_dict", "key1"))
self.assertFalse(self.db.dremove("test_dict", "key2"))

def test_dmerge(self):
self.db.dcreate("test_dict")
self.db.dadd("test_dict", "key1", "value1")
self.db.dmerge("test_dict", {"key2": "value2", "key3": "value3"})
self.assertEqual(self.db.dgetall("test_dict"), {
"key1": "value1",
"key2": "value2",
"key3": "value3"
})

def test_dkeys(self):
self.db.dcreate("test_dict")
self.db.dadd("test_dict", "key1", "value1")
self.db.dadd("test_dict", "key2", "value2")
self.assertEqual(set(self.db.dkeys("test_dict")), {"key1", "key2"})

def test_dvalues(self):
self.db.dcreate("test_dict")
self.db.dadd("test_dict", "key1", "value1")
self.db.dadd("test_dict", "key2", "value2")
self.assertEqual(set(self.db.dvalues("test_dict")), {"value1", "value2"})

# Additional Tests
def test_persistence(self):
"""Clean up after tests."""
if os.path.exists(self.test_file):
os.remove(self.test_file)

def _timeout_handler(self, signum, frame):
"""Handle timeouts for stress tests."""
raise TimeoutError("Test exceeded the timeout duration")

# Original Stress Test
def test_stress_operation(self):
"""Stress test: Insert and retrieve a large number of key-value pairs, then dump."""
timeout_duration = 600 # Timeout in seconds (10 minutes)

# Set a signal-based timeout
signal.signal(signal.SIGALRM, self._timeout_handler)
signal.alarm(timeout_duration)

try:
num_docs = 20_000_000

# Measure memory loading time
start_time = time.time()
for i in range(num_docs):
self.db.set(f"key{i}", f"value{i}")
mem_time = time.time()
mem_duration = mem_time - start_time
print(f"\n{num_docs} stored in memory in {mem_duration:.2f} seconds")

# Measure retrieval performance before dumping
start_time = time.time()
retrieved_docs = [self.db.get(f"key{i}") for i in range(num_docs)]
retrieval_time = time.time() - start_time
print(f"Retrieved {num_docs} key-value pairs in {retrieval_time:.2f} seconds")

# Measure dump performance
start_time = time.time()
self.db.dump()
dump_time = time.time() - start_time
print(f"Dumped {num_docs} key-value pairs to disk in {dump_time:.2f} seconds")

finally:
signal.alarm(0) # Cancel the alarm after the test

# Functional Tests
def test_set_and_get(self):
"""Test setting and retrieving a key-value pair."""
self.db.set("key1", "value1")
del self.db
db = load(self.db_file, auto_dump=True)
self.assertEqual(db.get("key1"), "value1")

def test_invalid_ladd(self):
with self.assertRaises(TypeError):
self.db.ladd("nonexistent_list", "item")
self.assertEqual(self.db.get("key1"), "value1")

def test_invalid_dadd(self):
with self.assertRaises(TypeError):
self.db.dadd("nonexistent_dict", "key", "value")
def test_get_nonexistent_key(self):
"""Test retrieving a key that does not exist."""
self.assertIsNone(self.db.get("nonexistent"))

def test_compress(self):
def test_remove_key(self):
"""Test removing a key-value pair."""
self.db.set("key1", "value1")
self.assertTrue(self.db.compress())
self.assertTrue(os.path.exists(f"{self.db_file}.gz"))

def test_ttl_expiry(self):
self.db.set("key1", "value1", ttl=1)
time.sleep(2)
self.assertTrue(self.db.remove("key1"))
self.assertIsNone(self.db.get("key1"))

def test_clear(self):
def test_remove_nonexistent_key(self):
"""Test removing a key that does not exist."""
self.assertFalse(self.db.remove("nonexistent"))

def test_purge(self):
"""Test purging all keys and values."""
self.db.set("key1", "value1")
self.db.set("key2", "value2")
self.db.clear()
self.assertEqual(len(self.db.getall()), 0)
self.db.purge()
self.assertEqual(self.db.all(), [])

def test_deldb(self):
def test_all_keys(self):
"""Test retrieving all keys."""
self.db.set("key1", "value1")
self.db.deldb()
self.assertFalse(os.path.exists(self.db_file))
self.db.set("key2", "value2")
self.assertListEqual(sorted(self.db.all()), ["key1", "key2"])

def test_dump_and_reload(self):
"""Test dumping the database to disk and reloading it."""
self.db.set("key1", "value1")
self.db.dump()
reloaded_db = PickleDB(self.test_file, auto_dump=False)
self.assertEqual(reloaded_db.get("key1"), "value1")

def test_invalid_file_loading(self):
"""Test initializing a database with a corrupt file."""
with open(self.test_file, 'w') as f:
f.write("corrupt data")
db = PickleDB(self.test_file, auto_dump=False)
self.assertEqual(db.all(), [])

def test_auto_dump(self):
"""Test the auto-dump functionality."""
db = PickleDB(self.test_file, auto_dump=True)
db.set("key1", "value1")
reloaded_db = PickleDB(self.test_file, auto_dump=False)
self.assertEqual(reloaded_db.get("key1"), "value1")

def test_set_non_string_key(self):
"""Test setting a non-string key."""
self.db.set(123, "value123")
self.assertEqual(self.db.get("123"), "value123")

def test_remove_non_string_key(self):
"""Test removing a key that was stored as a non-string key."""
self.db.set(123, "value123")
self.assertTrue(self.db.remove(123))
self.assertIsNone(self.db.get("123"))


if __name__ == "__main__":
unittest.main()

0 comments on commit d01baec

Please sign in to comment.