So you've been asked to "take over" some old data pipeline

So you've been asked to "take over" some old data pipeline... Apr 21, 2026 Work anywhere long enough and you're going to receive "gifts" from various people at work in the form of "here's a data processing thing, it is yours now". The gift giver is gone for whatever reason. Maybe there is documentation, but at best it's incomplete or outdated if it ever existed to begin with. It very likely has broken because people who aren't the owner likely wouldn't have remembered it existed to give it to you. Whatever the reasons or situation for being handed this data pipeline-like thing, you will have the same set of questions as everyone else. What is this thing? What is it supposed to do? How does it work? How do people use it?These situations come up all the time. I'm sure everyone reading this can probably think of some situation, maybe big, maybe very tiny, where they were just handed a data process and were told to make sense of it. But what I find very fascinating about this situation is that it is wide open to interpretation. Depending on the exact specifics of the situation, you have the freedom to do some really interesting work based on what lenses you choose to apply to the problem.Today's post is a bit about various techniques that are common brought to bear to approach the problem. It is also a bit of an attempt to remind people who've done repeatedly before that the paths we actually do take in the end are probably not the most obvious ones.Accessing the situationNo matter what happens the first thing is to gather the basic info about what you just got. So the first order of business is always going to be basic gathering of context. The vast majority of this information is not technical. While you might be holding a blob of code in your hands, very little of that critical context has to do with the code. You've got to first figure out who wanted to use this data, and for what purpose. Unless you know this thing was supposed to count widgets in a certain way for a very specific report used by this one specific person, it'll make no sense when you delve into the code.While this is the very first step, the amount of time different people spend at this initial step tends to be indicative of the amount of experience you've spent working with data. People who are fairly new tend to gloss over this step because it takes time. You have to track down people and ask questions that may not have an answer. Meanwhile the code is right there for inspection, so why can't we just dive in first to help formulate what questions we want to ask?Meanwhile, experienced folk spend a LOT of time in this step. They'll take a cursory scan of the code to see if there's any documentation written down, but even it exists they won't 100% take it at face value. Instead they want to track down the backstory of the code – who, exactly, was using this output. What specifically was it used for? More importantly, why do we even care now? The recognition here is that the code blob is a relic of the past. It existed in a previous context, and given the ever-changing nature of the world, it is probably ill suited to the current context. The reason why we care any bit about the history behind a blob of arbitrary data processing code is because it helps us set the stage for what this code should be in the future – and that includes whether the future should be the trash bin. If you ask around and find out the pipeline created was measuring the wrong thing – trash bin.…

This excerpt is published under fair use for community discussion. Read the full article at Counting Stuff.