Go to file
Matt 1f13224c4f add hydra config.
remove click.
add launch script.
add test dir.
switch from fashion mnist to generic.
2023-01-26 07:25:07 -08:00
src add hydra config. 2023-01-26 07:25:07 -08:00
test add hydra config. 2023-01-26 07:25:07 -08:00
.gitignore add hydra config. 2023-01-26 07:25:07 -08:00
Makefile add hydra config. 2023-01-26 07:25:07 -08:00
README.md add hydra config. 2023-01-26 07:25:07 -08:00
environment.yml move files to src dir. add env.yml 2022-11-05 07:37:44 -07:00
launch.sh add hydra config. 2023-01-26 07:25:07 -08:00

README.md

Mimimal Viable Deep Learning Infrastructure

Deep learning pipelines are hard to reason about and difficult to code consistently.

Instead of remembering where to put everything and making a different choice for each project, this repository is an attempt to standardize on good defaults.

Think of it like a mini-pytorch lightening, with all the fory internals exposed for extension and modification.

Usage

Install:

Install the conda requirements:

make install

Which is a proxy for calling:

conda env updates -n ml_pipeline --file environment.yml

Run:

Run the code on MNIST with the following command:

make run

Tutorial

The motivation for building a template for deep learning pipelines is this: deep learning is hard enough without every code baase being a little different.

Especially in a research lab, standardizing on a few components makes switching between projects easier.

In this template, you'll see the following:

  • src/model, src/config, storage, test dirs.
  • if __name__ == "__main__" tests.
  • Hydra config.
  • dataloader, optimizer, criterion, device, state are constructed in main, but passed to an object that runs batches.
  • tqdm to track progress.
  • debug config flag enables lots breakpoints.
  • python type hints.
  • a launch.sh script to dispatch training.
  • a Makefile to install and run stuff.
  • automatic linting with the black package.
  • collate functions!