Skip to content
This repository was archived by the owner on Apr 2, 2025. It is now read-only.

Commit c29875c

Browse files
bosdbosd
committed
🧑‍🎓 add parser comparison notebook ✨
1 parent 668d3c2 commit c29875c

File tree

3 files changed

+272
-0
lines changed

3 files changed

+272
-0
lines changed

Diff for: ‎README.md

+2
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,8 @@ pypdf_table_extraction also comes packaged with a [command-line interface](https
5050

5151
Refer to the [QuickStart Guide](https://github.com/py-pdf/pypdf_table_extraction/blob/main/docs/user/quickstart.rst#quickstart) to quickly get started with pypdf_table_extraction, extract tables from PDFs and explore some basic options.
5252

53+
**Tip:** Visit the `parser-comparison-notebook` to get an overview of all the packed parsers and their features. [![image](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/py-pdf/pypdf_table_extraction/blob/main/examples/parser-comparison-notebook.ipynb)
54+
5355
**Note:** pypdf_table_extraction only works with text-based PDFs and not scanned documents. (As Tabula [explains](https://github.com/tabulapdf/tabula#why-tabula), "If you can click and drag to select text in your table in a PDF viewer, then your PDF is text-based".)
5456

5557
You can check out some frequently asked questions [here](https://pypdf-table-extraction.readthedocs.io/en/latest/user/faq.html).

Diff for: ‎docs/user/how-it-works.rst

+7
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@ This part of the documentation includes a high-level explanation of how pypdf_ta
88
You can choose between the following table parsing methods, *Stream*, *Lattice*, *Network* and *Hybrid*.
99
Where *Hybrid* is a combination of the *Network* and *Lattice* parser.
1010

11+
.. tip::
12+
For a side-by-side visual comparison of the parser use the `parser-comparison-notebook`:
13+
14+
.. image:: https://colab.research.google.com/assets/colab-badge.svg
15+
:target: https://colab.research.google.com/github/py-pdf/pypdf_table_extraction/blob/main/examples/parser-comparison-notebook.ipynb
16+
17+
1118
.. _stream:
1219

1320
Stream

Diff for: ‎examples/parser-comparison-notebook.ipynb

+263
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,263 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Parser comparison\n",
8+
"\n",
9+
"This notebook lets you visualize side-by-side how each parser analyzes a document, and compare the resulting tables.\n"
10+
]
11+
},
12+
{
13+
"cell_type": "markdown",
14+
"metadata": {},
15+
"source": [
16+
"# Setup pypdf_table_extraction"
17+
]
18+
},
19+
{
20+
"cell_type": "code",
21+
"execution_count": null,
22+
"metadata": {},
23+
"outputs": [],
24+
"source": [
25+
"import os\n",
26+
"os.getcwd()\n",
27+
"# Install from source\n",
28+
"!git clone -b main https://github.com/py-pdf/pypdf_table_extraction.git src\n",
29+
"%cd src\n",
30+
"\n",
31+
"\n",
32+
"!pip install -e .\n",
33+
"\n",
34+
"# Optionally you can Install ghostscript as the imageconversion backend.\n",
35+
"# uncomment the following lines\n",
36+
"# !apt-get install -y ghostscript\n",
37+
"# !pip install ghostscript"
38+
]
39+
},
40+
{
41+
"cell_type": "code",
42+
"execution_count": null,
43+
"metadata": {
44+
"tags": []
45+
},
46+
"outputs": [],
47+
"source": [
48+
"# Bootstrap and common imports\n",
49+
"import sys, time\n",
50+
"sys.path.insert(0, os.path.abspath('')) # Prefer the local version of pypdf_table_extraction if available\n",
51+
"import pypdf_table_extraction\n",
52+
"\n",
53+
"print(f\"Using pypdf_table_extraction v{pypdf_table_extraction.__version__} from file {pypdf_table_extraction.__file__}.\")"
54+
]
55+
},
56+
{
57+
"cell_type": "markdown",
58+
"metadata": {},
59+
"source": [
60+
"## Select a PDF file to review\n",
61+
"\n",
62+
"You can modify the section below to point to a pdf or your choice to visualize the results. By default, it points to one of the test .pdfs included with pypdf_table_extraction.\n",
63+
"This is seeded with the unit test files for convenience."
64+
]
65+
},
66+
{
67+
"cell_type": "code",
68+
"execution_count": null,
69+
"metadata": {},
70+
"outputs": [],
71+
"source": [
72+
"kwargs = {}\n",
73+
"data = None\n",
74+
"# pdf_file = \"vertical_header.pdf\" # test_network_vertical_header\n",
75+
"# pdf_file, kwargs = \"vertical_header.pdf\", {\"pages\": \"all\"} # test_network_vertical_headerpages\n",
76+
"# pdf_file, kwargs = \"background_lines_1.pdf\", {\"process_background\": True} # {\"process_background\": True} # test_lattice_process_background\n",
77+
"\n",
78+
"# pdf_file, kwargs, data = \"superscript.pdf\", {\"flag_size\": True}, data_stream_flag_size # test_network_flag_size\n",
79+
"# pdf_file, kwargs = \"superscript.pdf\", {\"flag_size\": True} # , data_stream_flag_size # test_network_flag_size\n",
80+
"# pdf_file = \"health.pdf\" # test_network\n",
81+
"# pdf_file = \"clockwise_table_2.pdf\"\n",
82+
"# pdf_file = \"clockwise_table_1.pdf\"\n",
83+
"# pdf_file = \"foo.pdf\"\n",
84+
"# pdf_file, kwargs = \"saturation_threshold.pdf\", {\"process_color_background\": False, \"process_background\": True, \"saturation_threshold\": 5, \"threshold_blocksize\": 25} # \"process_background\": True,\n",
85+
"\n",
86+
"# pdf_file = \"birdisland.pdf\"\n",
87+
"# pdf_file, kwargs = \"diesel_engines.pdf\", {\"pages\": \"4-5\"} # containing multiple pages 2-4 = hybrid error same for 3-4,2-3\n",
88+
"\n",
89+
"# pdf_file, kwargs = \"column_span_1.pdf\", {\"copy_text\": \"h\"}\n",
90+
"# pdf_file = \"tabula/12s0324.pdf\" # interesting because contains two separate tables\n",
91+
"# pdf_file, kwargs = \"tabula/12s0324.pdf\", {\"strip_text\": \" ,\\n\"} # interesting because contains two separate tables\n",
92+
"# pdf_file, kwargs = \"tabula/us-007.pdf\", {\"table_regions\": [\"320,335,573,505\"]} # test_network_table_regions\n",
93+
"# pdf_file, kwargs = \"tabula/us-007.pdf\", {\"table_areas\": [\"320,500,573,335\"]} # test_network_table_areas\n",
94+
"# pdf_file, kwargs = \"detect_vertical_false.pdf\", {\"strip_text\": \" ,\\n\"} # data_stream_strip_text\n",
95+
"# pdf_file = \"detect_vertical_false.pdf\" #\n",
96+
"# pdf_file, kwargs, data = \"tabula/m27.pdf\", {\"columns\": [\"72,95,209,327,442,529,566,606,683\"], \"split_text\": True, }, data_stream_split_text # data_stream_split_text\n",
97+
"# pdf_file, kwargs= \"tabula/m27.pdf\", {\"columns\": [\"72,95,209,327,442,529,566,606,683\"], \"split_text\": True, } # , data_stream_split_text # data_stream_split_text\n",
98+
"\n",
99+
"# pdf_file = \"clockwise_table_2.pdf\" # test_network_table_rotated / test_stream_table_rotated\n",
100+
"# pdf_file, kwargs = \"clockwise_table_2.pdf\", {\"edge_tol\": 10} # configurable vgap header search not working\n",
101+
"# edge_tol 0 gives an error\n",
102+
"pdf_file = \"vertical_header.pdf\"\n",
103+
"\n",
104+
"# pdf_file = \"twotables_2.pdf\"\n",
105+
"# pdf_file = \"camelot-issue-132-multiple-tables.pdf\"\n",
106+
"# pdf_file = \"multiple_tables.pdf\" # fixes issue 132\n",
107+
"# pdf_file, kwargs, data = \"edge_tol.pdf\", {\"edge_tol\": 500}, data_stream_edge_tol\n",
108+
"# pdf_file, kwargs = \"edge_tol.pdf\", {\"edge_tol\": 500} # , data_stream_edge_tol\n",
109+
"# pdf_file, kwargs, data = \"edge_tol.pdf\", {}, data_stream_edge_tol\n",
110+
"\n",
111+
"# pdf_file = \"tabula/arabic.pdf\"\n",
112+
"# pdf_file = \"tabula/indictb1h_14.pdf\" # interesting mixed type table\n",
113+
"# pdf_file = \"tabula/m27.pdf\" # one table spanning multiple pages\n",
114+
"# pdf_file = \"tabula/mednine.pdf\" # one table spanning multiple pages\n",
115+
"\n",
116+
"# pdf_file = \"tabula/spreadsheet_no_bounding_frame.pdf\n",
117+
"# pdf_file, kwargs = \"diesel_engines.pdf\", {\"pages\": \"4-5\"} # containing multiple pages\n",
118+
"\n",
119+
"# pdf_file, kwargs = \"tabula/schools.pdf\", {\"pages\": \"all\"} # network parser hangs on contour plot\n",
120+
"\n",
121+
"filename = os.path.join(\n",
122+
" os.path.dirname(os.path.abspath('.')),\n",
123+
" \"src/tests/files\",\n",
124+
" pdf_file\n",
125+
")\n"
126+
]
127+
},
128+
{
129+
"cell_type": "code",
130+
"execution_count": null,
131+
"metadata": {
132+
"tags": []
133+
},
134+
"outputs": [],
135+
"source": [
136+
"FLAVORS = [\"stream\", \"lattice\", \"network\", \"hybrid\"]\n",
137+
"tables_parsed = {}\n",
138+
"parses = {}\n",
139+
"max_tables = 0\n",
140+
"for idx, flavor in enumerate(FLAVORS):\n",
141+
" timer_before_parse = time.perf_counter()\n",
142+
" error, tables = None, []\n",
143+
" try:\n",
144+
" tables = pypdf_table_extraction.read_pdf(filename, flavor=flavor, debug=True, **kwargs)\n",
145+
" except ValueError as value_error:\n",
146+
" error = f\"Invalid argument for parser {flavor}: {value_error}\"\n",
147+
" print(error)\n",
148+
" timer_after_parse = time.perf_counter()\n",
149+
" max_tables = max(max_tables, len(tables))\n",
150+
"\n",
151+
" parses[flavor] = {\n",
152+
" \"tables\": tables,\n",
153+
" \"time\": timer_after_parse - timer_before_parse,\n",
154+
" \"error\": error\n",
155+
" }\n",
156+
"\n",
157+
" print(f\"##### {flavor} ####\")\n",
158+
" print(f\"Found {len(tables)} table(s):\")\n",
159+
" for idx, table in enumerate(tables):\n",
160+
" flavors_matching = []\n",
161+
" for previous_flavor, previous_tables in tables_parsed.items():\n",
162+
" for prev_idx, previous_table in enumerate(previous_tables):\n",
163+
" if previous_table.df.equals(table.df):\n",
164+
" flavors_matching.append(\n",
165+
" f\"{previous_flavor} table {prev_idx}\")\n",
166+
" print(f\"## Table {idx} ##\")\n",
167+
" if flavors_matching:\n",
168+
" print(f\"Same as {', '.join(flavors_matching)}.\")\n",
169+
" else:\n",
170+
" display(table.df)\n",
171+
" print(\"\")\n",
172+
" tables_parsed[flavor] = tables\n"
173+
]
174+
},
175+
{
176+
"cell_type": "markdown",
177+
"metadata": {},
178+
"source": [
179+
"## Show tables layout within original document"
180+
]
181+
},
182+
{
183+
"cell_type": "code",
184+
"execution_count": null,
185+
"metadata": {
186+
"tags": []
187+
},
188+
"outputs": [],
189+
"source": [
190+
"\n",
191+
"# Set up plotting options\n",
192+
"import matplotlib.pyplot as plt\n",
193+
"%matplotlib inline\n",
194+
"PLOT_HEIGHT = 12\n",
195+
"\n",
196+
"row_count = max(max_tables, 1)\n",
197+
"plt.rcParams[\"figure.figsize\"] = [PLOT_HEIGHT * len(FLAVORS), PLOT_HEIGHT * row_count]\n",
198+
"fig, axes = plt.subplots(row_count, len(FLAVORS))\n",
199+
"plt.subplots_adjust(wspace=0, hspace=0) # Reduce margins to maximize the display zone\n",
200+
"\n",
201+
"fig.suptitle('Side-by-side flavor comparison', fontsize=24, fontweight='bold')\n",
202+
"for idx, flavor in enumerate(FLAVORS):\n",
203+
" parse = parses[flavor]\n",
204+
" tables = parse[\"tables\"]\n",
205+
" top_ax = axes.flat[idx]\n",
206+
" title = f\"{flavor}\\n\" \\\n",
207+
" f\"Detected {len(tables)} table(s) in {parse['time']:.2f}s\"\n",
208+
" if parse['error']:\n",
209+
" title = title + f\"\\nError parsing: {parse['error']}\"\n",
210+
" top_ax.set_title(title, fontsize=12, fontweight='bold')\n",
211+
" for table_idx, table in enumerate(tables):\n",
212+
" if max_tables > 1:\n",
213+
" ax = axes[table_idx][idx]\n",
214+
" else:\n",
215+
" ax = axes[idx]\n",
216+
" # Check if the table has data before attempting to plot it\n",
217+
" if table.shape[0] > 0 and table.shape[1] > 0: # Check if table has rows and columns\n",
218+
" fig = camelot.plot(table, kind='grid', ax=ax)\n",
219+
" ax.text(\n",
220+
" 0.5, -0.1,\n",
221+
" \"{flavor} table {table_idx} - {rows}x{cols}\".format(\n",
222+
" flavor=flavor,\n",
223+
" table_idx=table_idx,\n",
224+
" rows=table.shape[0],\n",
225+
" cols=table.shape[1],\n",
226+
" ),\n",
227+
" size=14, ha=\"center\",\n",
228+
" transform=ax.transAxes\n",
229+
" )\n",
230+
" else:\n",
231+
" print(f\"Skipping plotting for empty table {table_idx} in {flavor}\") # Inform user about the skipped table\n",
232+
" timer_after_plot = time.perf_counter()\n"
233+
]
234+
}
235+
],
236+
"metadata": {
237+
"file_extension": ".py",
238+
"kernelspec": {
239+
"display_name": "Python 3 (ipykernel)",
240+
"language": "python",
241+
"name": "python3"
242+
},
243+
"language_info": {
244+
"codemirror_mode": {
245+
"name": "ipython",
246+
"version": 3
247+
},
248+
"file_extension": ".py",
249+
"mimetype": "text/x-python",
250+
"name": "python",
251+
"nbconvert_exporter": "python",
252+
"pygments_lexer": "ipython3",
253+
"version": "3.12.5"
254+
},
255+
"mimetype": "text/x-python",
256+
"name": "python",
257+
"npconvert_exporter": "python",
258+
"pygments_lexer": "ipython3",
259+
"version": 3
260+
},
261+
"nbformat": 4,
262+
"nbformat_minor": 4
263+
}

0 commit comments

Comments
 (0)