Skip to content

Commit ab13873

Browse files
chaokunyangchaokunyang
authored
docs(python): refine python code and add api doc (#2816)
## Why? <!-- Describe the purpose of this PR. --> ## What does this PR do? refine python code and add api doc ## Related issues <!-- Is there any related issue? If this PR closes them you say say fix/closes: - #xxxx0 - #xxxx1 - Fixes #xxxx2 --> ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fory/issues/new/choose) describing the need to do so and update the document if necessary. Delete section if not applicable. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark <!-- When the PR has an impact on performance (if you don't know whether the PR will have an impact on performance, you can submit the PR first, and if it will have impact on performance, the code reviewer will explain it), be sure to attach a benchmark data here. Delete section if not applicable. -->
1 parent a0c107b commit ab13873

18 files changed

+950
-384
lines changed

‎AGENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -389,14 +389,14 @@ Fory python has two implementations for the protocol:
389389

390390
Code structure:
391391

392-
- `python/pyfory/_serialization.pyx`: Core serialization logic and entry point for cython mode based on `xlang serialization format`
392+
- `python/pyfory/serialization.pyx`: Core serialization logic and entry point for cython mode based on `xlang serialization format`
393393
- `python/pyfory/_fory.py`: Serialization entry point for pure python mode based on `xlang serialization format`
394394
- `python/pyfory/_registry.py`: Type registry, resolution and serializer dispatch for pure python mode, which is also used by cython mode. Cython mode use a cache to reduce invocations to this module.
395395
- `python/pyfory/serializer.py`: Serializers for non-internal types
396396
- `python/pyfory/includes`: Cython headers for `c++` functions and classes.
397397
- `python/pyfory/resolver.py`: resolving shared/circular references when ref tracking is enabled in pure python mode
398398
- `python/pyfory/format`: Fory row format encoding and decoding, arrow columnar format interoperation
399-
- `python/pyfory/_util.pyx`: Buffer for reading/writing data, string utilities. Used by `_serialization.pyx` and `python/pyfory/format` at the same time.
399+
- `python/pyfory/_util.pyx`: Buffer for reading/writing data, string utilities. Used by `serialization.pyx` and `python/pyfory/format` at the same time.
400400

401401
#### Go
402402

‎BUILD‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -51,11 +51,11 @@ pyx_library(
5151
)
5252

5353
pyx_library(
54-
name = "_serialization",
54+
name = "serialization",
5555
srcs = glob([
5656
"python/pyfory/includes/*.pxd",
5757
"python/pyfory/_util.pxd",
58-
"python/pyfory/_serialization.pyx",
58+
"python/pyfory/serialization.pyx",
5959
"python/pyfory/__init__.py",
6060
]),
6161
cc_kwargs = dict(
@@ -96,7 +96,7 @@ genrule(
9696
":python/pyfory/_util.so",
9797
":python/pyfory/lib/mmh3/mmh3.so",
9898
":python/pyfory/format/_format.so",
99-
":python/pyfory/_serialization.so",
99+
":python/pyfory/serialization.so",
100100
],
101101
outs = [
102102
"cp_fory_py_generated.out",
@@ -111,12 +111,12 @@ genrule(
111111
cp -f $(location python/pyfory/_util.so) "$$WORK_DIR/python/pyfory/_util.pyd"
112112
cp -f $(location python/pyfory/lib/mmh3/mmh3.so) "$$WORK_DIR/python/pyfory/lib/mmh3/mmh3.pyd"
113113
cp -f $(location python/pyfory/format/_format.so) "$$WORK_DIR/python/pyfory/format/_format.pyd"
114-
cp -f $(location python/pyfory/_serialization.so) "$$WORK_DIR/python/pyfory/_serialization.pyd"
114+
cp -f $(location python/pyfory/serialization.so) "$$WORK_DIR/python/pyfory/serialization.pyd"
115115
else
116116
cp -f $(location python/pyfory/_util.so) "$$WORK_DIR/python/pyfory"
117117
cp -f $(location python/pyfory/lib/mmh3/mmh3.so) "$$WORK_DIR/python/pyfory/lib/mmh3"
118118
cp -f $(location python/pyfory/format/_format.so) "$$WORK_DIR/python/pyfory/format"
119-
cp -f $(location python/pyfory/_serialization.so) "$$WORK_DIR/python/pyfory"
119+
cp -f $(location python/pyfory/serialization.so) "$$WORK_DIR/python/pyfory"
120120
fi
121121
echo $$(date) > $@
122122
""",

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ cd python
155155
python setup.py develop
156156
```
157157

158-
- Use `cython --cplus -a pyfory/_serialization.pyx` to produce an annotated HTML file of the source code. Then you can analyze interaction between Python objects and Python's C API.
158+
- Use `cython --cplus -a pyfory/serialization.pyx` to produce an annotated HTML file of the source code. Then you can analyze interaction between Python objects and Python's C API.
159159
- Read more: https://cython.readthedocs.io/en/latest/src/userguide/debugging.html
160160

161161
```bash

‎python/CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ cd python
5252
python setup.py develop
5353
```
5454

55-
- Use `cython --cplus -a pyfory/_serialization.pyx` to produce an annotated HTML file of the source code. Then you can
55+
- Use `cython --cplus -a pyfory/serialization.pyx` to produce an annotated HTML file of the source code. Then you can
5656
analyze interaction between Python objects and Python's C API.
5757
- Read more: <https://cython.readthedocs.io/en/latest/src/userguide/debugging.html>
5858

‎python/pyfory/__init__.py‎

Lines changed: 115 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18-
from pyfory import lib # noqa: F401 # pylint: disable=unused-import
19-
from pyfory._fory import ( # noqa: F401 # pylint: disable=unused-import
18+
from pyfory.lib import mmh3
19+
from pyfory._fory import (
2020
Fory,
2121
Language,
2222
ThreadSafeFory,
@@ -26,16 +26,43 @@
2626
XLANG = Language.XLANG
2727

2828
try:
29-
from pyfory._serialization import ENABLE_FORY_CYTHON_SERIALIZATION
29+
from pyfory.serialization import ENABLE_FORY_CYTHON_SERIALIZATION
3030
except ImportError:
3131
ENABLE_FORY_CYTHON_SERIALIZATION = False
3232

3333
from pyfory._registry import TypeInfo
3434

3535
if ENABLE_FORY_CYTHON_SERIALIZATION:
36-
from pyfory._serialization import Fory, TypeInfo # noqa: F401,F811
36+
from pyfory.serialization import Fory, TypeInfo # noqa: F401,F811
3737

38-
from pyfory.serializer import * # noqa: F401,F403 # pylint: disable=unused-import
38+
from pyfory.serializer import ( # noqa: F401 # pylint: disable=unused-import
39+
Serializer,
40+
XlangCompatibleSerializer,
41+
BooleanSerializer,
42+
ByteSerializer,
43+
Int16Serializer,
44+
Int32Serializer,
45+
Int64Serializer,
46+
Float32Serializer,
47+
Float64Serializer,
48+
StringSerializer,
49+
DateSerializer,
50+
TimestampSerializer,
51+
CollectionSerializer,
52+
ListSerializer,
53+
TupleSerializer,
54+
StringArraySerializer,
55+
SetSerializer,
56+
MapSerializer,
57+
EnumSerializer,
58+
SliceSerializer,
59+
DataClassSerializer,
60+
FunctionSerializer,
61+
TypeSerializer,
62+
MethodSerializer,
63+
ReduceSerializer,
64+
StatefulSerializer,
65+
)
3966
from pyfory.type import ( # noqa: F401 # pylint: disable=unused-import
4067
record_class_factory,
4168
get_qualified_classname,
@@ -47,23 +74,98 @@
4774
float32,
4875
float64,
4976
# Int8ArrayType,
50-
Int16ArrayType,
51-
Int32ArrayType,
52-
Int64ArrayType,
53-
Float32ArrayType,
54-
Float64ArrayType,
77+
int16_array,
78+
int32_array,
79+
int64_array,
80+
float32_array,
81+
float64_array,
5582
dataslots,
5683
)
5784
from pyfory.policy import DeserializationPolicy # noqa: F401 # pylint: disable=unused-import
5885
from pyfory._util import Buffer # noqa: F401 # pylint: disable=unused-import
5986

87+
__version__ = "0.13.0.dev"
88+
89+
__all__ = [
90+
# Core classes
91+
"Fory",
92+
"Language",
93+
"ThreadSafeFory",
94+
"TypeInfo",
95+
"Buffer",
96+
"DeserializationPolicy",
97+
# Language constants
98+
"PYTHON",
99+
"XLANG",
100+
# Type utilities
101+
"record_class_factory",
102+
"get_qualified_classname",
103+
"TypeId",
104+
"int8",
105+
"int16",
106+
"int32",
107+
"int64",
108+
"float32",
109+
"float64",
110+
"int16_array",
111+
"int32_array",
112+
"int64_array",
113+
"float32_array",
114+
"float64_array",
115+
"dataslots",
116+
# Serializers
117+
"Serializer",
118+
"XlangCompatibleSerializer",
119+
"BooleanSerializer",
120+
"ByteSerializer",
121+
"Int16Serializer",
122+
"Int32Serializer",
123+
"Int64Serializer",
124+
"Float32Serializer",
125+
"Float64Serializer",
126+
"StringSerializer",
127+
"DateSerializer",
128+
"TimestampSerializer",
129+
"CollectionSerializer",
130+
"ListSerializer",
131+
"TupleSerializer",
132+
"StringArraySerializer",
133+
"SetSerializer",
134+
"MapSerializer",
135+
"EnumSerializer",
136+
"SliceSerializer",
137+
"DataClassSerializer",
138+
"FunctionSerializer",
139+
"TypeSerializer",
140+
"MethodSerializer",
141+
"ReduceSerializer",
142+
"StatefulSerializer",
143+
"mmh3",
144+
# Version
145+
"__version__",
146+
]
147+
148+
# Try to import format utilities (requires pyarrow)
60149
import warnings
61150

62151
try:
63152
with warnings.catch_warnings():
64153
warnings.filterwarnings("ignore", category=RuntimeWarning)
65-
from pyfory.format import * # noqa: F401,F403 # pylint: disable=unused-import
154+
from pyfory.format import ( # noqa: F401 # pylint: disable=unused-import
155+
create_row_encoder,
156+
RowData,
157+
encoder,
158+
Encoder,
159+
)
160+
161+
__all__.extend(
162+
[
163+
"format",
164+
"create_row_encoder",
165+
"RowData",
166+
"encoder",
167+
"Encoder",
168+
]
169+
)
66170
except (AttributeError, ImportError):
67171
pass
68-
69-
__version__ = "0.13.0.dev"

0 commit comments

Comments
 (0)