UofT-DSI | Production - Assignment 1

[brianna-lowe]

UofT-DSI | Production - Assignment 1

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)

added feature engineering steps to the price data using Dask, including lagged values for Close and Adjusted Close, daily returns, and the daily high–low price range. I then converted the Dask DataFrame to a pandas DataFrame and added a 10-day moving average of returns, as required by the assignment.

What did you learn from the changes you have made?

I learned how Dask handles lazy evaluation differently from pandas, particularly when working with groupby, shift, and rolling operations. I also gained a better understanding of when it makes sense to stay in Dask versus when converting to pandas can simplify debugging and downstream calculations.

Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?

An alternative approach would have been to compute the rolling moving average entirely in Dask. While this would be more scalable for very large datasets, it requires careful partitioning to ensure correct results across tickers and time. Given the assignment requirements and dataset size, converting to pandas was a more straightforward solution.

Were there any challenges? If so, what issue(s) did you face? How did you overcome it?

The main challenge was encountering compute-time errors related to index alignment and duplicate labels when converting from Dask to pandas. I (hopefully) resolved this by resetting the index appropriately and simplifying the pipeline so that more complex rolling calculations were handled in pandas, where the behavior is more predictable. I look forward to comments on how I could improve or avoid this in the future. I probably did something silly.

How were these changes tested?

The changes were tested by running the full pipeline end to end, checking intermediate outputs (such as lagged values and returns), and verifying that the final pandas DataFrame contained the expected features and no runtime errors.

A reference to a related issue in your repository (if applicable)

N/A

Checklist

• [ YES] I can confirm that my changes are working as intended

[github-actions]

Hello, thank you for your contribution. If you are a participant, please close this pull request and open it in your own forked repository instead of here. Please read the instructions on your onboarding Assignment Submission Guide more carefully. If you are not a participant, please give us up to 72 hours to review your PR. Alternatively, you can reach out to us directly to expedite the review process.