Reading and Splitting large json files from S3.

[1996sajal]

Issue:
We have more 10Gigs Json files stored in s3. These files need to be retrieved from s3 and split based on chunks. But as json has objects we need to divide those based on number of objects per chunk.

Question
We're thinking to use awswrangler to read from s3 and divide them in chunks.
Does awswrangler divide the file based on number of lines?
What are another good alternatives?

[1996sajal]

Any inputs? @jaidisido

[1996sajal]

def s3():
    session = boto3.Session(profile_name='PROFILE')
    dev_s3_client = session.client('s3')
    dfs = awswrangler.s3.read_json(path=['s3://test-large-json-bucket/split6.json'], chunksize=100, orient='records', lines=True, boto3_session=session)
    print(next(dfs))

Getting below issue:
File "/Users/AL49723/sandbox/lib/python3.11/site-packages/pandas/io/json/_json.py", line 1047, in __next__ obj = self._get_object_parser(lines_json) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/AL49723/sandbox/lib/python3.11/site-packages/pandas/io/json/_json.py", line 1001, in _get_object_parser obj = FrameParser(json, **kwargs).parse() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/AL49723/sandbox/lib/python3.11/site-packages/pandas/io/json/_json.py", line 1134, in parse self._parse() File "/Users/AL49723/sandbox/lib/python3.11/site-packages/pandas/io/json/_json.py", line 1347, in _parse loads(json, precise_float=self.precise_float), dtype=None ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: Expected object or value