Data Loading is a Huge Deal

I've been thinking about the work I am doing and have to do in my role at Cat, in Cat Autonomy, building Forge (see forge-ahead). I feel like I have little revelations almost every day now, not that it means I'm writing something amazing and producing it really fast but there's just a whole suite of problems that different technologies solve at different levels and the more I become aware of the problems that exist, the more the existence of some solutions makes sense.

The one problem I'm really fixated on right now is data loading, the problem I need to solve is accessing data from a wide-variety of scripts/tools, without a framework or standard method/library of accessing data in the first place. I've seen lots of projects on GitHub claim to make data loading easier and I didn't quite understand what problem they were solving... one example to name is Data Load Tool. I've seen similar ones I don't have the name for right now that claim to make it easy/fast to load data from s3, or ways to make s3 and a database both abstract in the user-experience for loading data. But I hadn't really understood why these tools existed. I have a lot of experience with Kedro and their DataCatalog which provides a python object over a set of yamlfiles that makes it pretty simple to load and save data in a way that isolates I/O from the business logic. But what I didn't realize at the time how powerful that catalog was, the power of standard patterns and shared libraries. Now that I don't have it available to me, I'm quite aware of the absence.

In my new role something I'm realizing is that for all the developers my team now supports, there isn't a canonical way to access data. When I was in Reman and working with kedro, the DataCatalog was the access pattern and so when I was developing a platform I never really had to think about it - it was an established pattern that I treated as a constraint and then built processes around it. I've been battling some mental block for weeks on Forge because of the lack of that canonical pattern, and as I've talked with other engineers it seems like the baseline assumption is that data is just available on a filesystem, but everyone's code loads data in different ways. On my small Reman team, with common patterns to build on, it was easy to make things cloud-native or shim in some devops to improve people's lives. But when everyone's doing their own thing, and everyone's "own thing" is very much built-on some rigid tribal patterns then it's hard to really move fast cause everyone isn't already moving in the same direction.

That made me realize that the first problem Forge needed to solve was in providing a way for engineers to have filesystem-native data access in the Cloud, where we are S3-first in our storage philosophy. I didn't need to figure out a way for everyone to name a dataset, define the dataset in the first place, and give a nice my_dataset.load that worked in python, bash, cpp, and who knows what else.... I reframed the problem from "how do engineers load up the data" to "how do engineers have access to the data". The requirements of the Cat Autonomy group was pretty simple: POSIX-compliant storage.

My pathway to solving this problem is initially underway, I can't imagine it'll be too difficult to setup for FSx instances for teams and give them an api to run a Batch Job with the FSx mounted. From there, their code can load data from /mnt/fsx/<whatever> just like they otherwise could be doing locally. Or maybe FSx will let us setup mounts to very flexible mount points and their local scripts will "just work" :shrugs:. I don't know the exact shape, but after realizing the loading data is a big deal, I'm thankful I have a narrower problem to solve first.