Support for feather or parquet format?

[lambertsbennett]

Currently we are evaluating using Argo Workflows to carry out ETL operations at my company. We are using pandas to manipulate tabular data and when I tried to create an output artifact by running df.to_feather('/tmp/df.feather') I received an error that the path/file was not in the artifact store. When I switch the code to output JSON files everything runs smoothly. Does Argo Workflows not support formats like parquet and feather? For our applications the write time to a feather file is incredibly fast and it would be ideal to be able to use feather file artifacts.
I'm relatively new to Argo Workflows, so I don't know if it doesn't support these formats or if I am fundamentally doing something wrong.

Thanks for the help.

[terrytangyuan]

I don't think Argo Workflows has any limitations on the file format. As long as the file is in the artifact storage, it should be able to retrieve the files. Is it possible that you specified the wrong key/path?

[terrytangyuan]

@lambertsbennett Do you have any reproducible example that could illustrate the behavior?

[lambertsbennett]

This is embarrassing, but in trying to recreate the problem I made an image, script, and template to reproduce the problem but it works. So now I need to understand what was wrong in my original setup. Thanks for the help.

[lambertsbennett]

Just for closure here is the script, dockerfile, and template that works.

import argparse
import pandas as pd

parser = argparse.ArgumentParser()

parser.add_argument("--name", action='store', dest='name', help='filename')

df_dict = {
    "Timestamp": ["2021-15-6T00:00:00Z", "2021-15-6T00:05:00Z", "2021-15-6T00:10:00Z", "2021-15-6T00:15:00Z", "2021-15-6T00:20:00Z"],
    "SensorValue": [1,2,3,4,5]
}

df = pd.DataFrame.from_dict(df_dict)

if __name__ == '__main__':
    args = parser.parse_args()
    df.to_feather(args.name)

#syntax=docker/dockerfile:1.2
FROM python:3.8.10-slim-buster
COPY ./requirements.txt ./requirements.txt
RUN --mount=type=secret,id=proxy,dst=proxy pip install -r requirements.txt --proxy $(cat proxy)
COPY ./feather_test.py ./feather_test.py
ENTRYPOINT ["python", "./feather_test.py"]

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: feather-test-

spec:
  entrypoint: dag-template
  arguments:
    parameters:
    - name: filepath
      value: /tmp/test.feather
    
  templates:
  - name: dag-template
    inputs:
      parameters:
      - name: filepath
    dag:
      tasks:
      - name: feather-save-template
        template: feather-save-template
        arguments:
          parameters:
          - name: filepath
            value: "{{inputs.parameters.filepath}}"
  
  - name: feather-save-template
    inputs:
      parameters:
      - name: filepath
    container:
      image: lambertsbennett/feathertest
      args: ["--name", "{{inputs.parameters.filepath}}"]
    outputs:
      artifacts:
      - name: featherfile
        path: /tmp/test.feather

and the requirements.txt

pandas==1.1.4
pyarrow

[terrytangyuan]

Cool good to know. Please mark this as resolved then.

[lambertsbennett]

Thanks for the quick response. I don't think it's a path problem, but I am going to keep at it. In particular I don't think it's a path problem because when I keep everything the same, but switch to JSON everything works.

…

On Tue, Jun 15, 2021, 15:53 Yuan Tang ***@***.***> wrote: I don't think Argo Workflows has any limitations on the file format. As long as the file is in the artifact storage, it should be able to retrieve the files. Is it possible that you specified the wrong key/path? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6155 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALBGTAQWK6RCXI2FE3ZR4FLTS5LLRANCNFSM46XGJGHQ> .