Being able to prioritize is undoubtedly one of the most important soft skills for managers and ICs alike. Working in a fast-paced company means you never just have one task on hand; even the most junior member on my team has multiple things they are working on at any given time. At the same time, there’s always pressure to deliver quickly since leadership and stakeholders want everything yesterday.

So knowing how to stack rank all the things on your plate in terms of priority is extremely important at all levels of the team. And if you want to grow into a more senior role or become a manager, it becomes absolutely essential.

However, based on my experience leading data science teams, this skill is not part of any formal training and it takes most people a long time to learn it through trial and error. I regularly see people working hard and delivering high-quality outputs and still get the feedback from cross-functional stakeholders that they are not doing things “fast enough” or “driving enough impact”.

When I sit down with them to look through their backlog of things, it’s clear that they have been prioritizing the wrong things.

You might think "Prioritization is easy, shouldn’t I just work on the things my manager/stakeholders asked me to work on?”. While that should definitely be part of your consideration, you will soon realize that most stakeholders have a pretty low bar for “asking for analysis” and your manager will not be able to stay on top of everything on everyone’s plate on the team. So it’s eventually part of your job to decide whether something should be prioritized.

The key to this is to understand which initiative drives the most impact. And “impact” is different from “output” or “input”:

• I noticed that the most junior DS on the team talk about “inputs” – “I spent X hours working on these analyses”

• More senior DS learn to uplevel from “inputs” to “outputs”; they’ll say “I helped with 5 experiments and built two dashboards”

• But only a handful are able to clearly articulate the business impact they drove; they’ll focus on the money they helped the company save, or the growth in important metrics they helped drive

Even though it’s rarely an explicitly stated skill requirement, knowing how to prioritize is nevertheless a “hidden” requirement for anyone’s career progress because it’s a pre-requisite for driving impact. Working hard and delivering high-quality work is important, but it only matters if it's on the right projects.

In this article, I will convince you why being able to judge priorities is crucial across all roles (DS, PM, engineering, etc.), break down the mental prioritization framework you can immediately put to use, and walk you through a hands-on example of how to apply it.

To achieve any goal, prioritization is key; and to prioritize, you need to know the ROI

If somebody gave your team a big goal (say, growing the monthly active member base from 10M to 50M), would you know:

1. Whether it’s a realistic goal you can hit?

2. How can you hit the goal, i.e. what do you need to do exactly?

If you answered “No”, you’re not alone; nobody can answer these questions off the bat. The reason is that businesses are complex; you’ll need many different initiatives to hit an ambitious goal like that, and you cannot assume that what worked in the past will get you to the next level as well.

In other words, you need to break this big, abstract goal into smaller, concrete work streams and then come up with ideas that will help you move the needle on your target metrics. And then, once you have a list of things you could do, you need to figure out which ones you should actually prioritize. You don’t have unlimited resources to do everything (especially not in a single quarter), so you need to select the initiatives with the biggest potential.

That’s where ROI estimation comes in. ROI stands for “return on investment”:

• Investment: How much of your limited resources you put in (usually time or money)

• Return: How much business impact you (expect to) get as a result

In other words, you’re looking for the biggest “bang for your buck”.

Estimations like this are essential during planning cycles to make sure teams are working on the most impactful things. Here is what that would look like in an ideal state:

1. Leadership decides on top-down goals for the company (e.g. grow revenue by $XM)

2. The goal is distributed across the different organizations by assigning team-level goals (e.g. product team 1 needs to contribute 20% of the goal, product team 2 30% etc.)

3. Teams brainstorm initiatives that can help them hit their goal

4. Teams estimate the impact of each initiative they came up with and stack-rank them; the most promising ones become the roadmap for the quarter, and the rest goes into the backlog

5. [Optional] Based on steps 1 - 4 above, teams should have an idea of whether they’ll be able to hit their goal if they successfully executive their roadmap. If they won’t, they either need to come up with more (or better) initiatives, or push back on the goal they’ve been given

Besides helping teams prioritize, the impact estimates also give you goals for each initiative that you can track against throughout the quarter. Then, after the quarter is over, you can do a retrospective and compare actuals against the plan. If any initiatives missed their goal due to overly optimistic or pessimistic estimates rather than execution issues, you can use those insights to refine your estimates during future planning exercises.

Unfortunately, this process doesn’t always get the attention it deserves. Some companies and teams run very structured planning processes and loop in the data science team (or business operations) early to help with these calculations, but other teams rely on experience to come up with these numbers, or simply “make an educated guess” when they run out of time.

If you notice a gap like this, it’s a great opportunity for you to step up and offer to help your stakeholders to level up the process. And don’t worry if you don’t have experience in this regard: This guide will give you the toolkit you need to do that.

In addition to planning, impact estimation is also helpful day-to-day when you are trying to decide 1) whether you should take on a new project or 2) which project on your plate you should focus on. This is important not just for managers that are trying to figure this out for their teams; even as an IC you should constantly make sure you’re working on high-impact things.

Like it or not, when it comes to promotions, the most important thing is whether you helped drive business impact. It doesn’t matter how hard you work, if the project goes nowhere or doesn’t move the needle, it’s not going to be helpful for your promo packet.

That’s not to say that you need to do an elaborate estimation for every single project you work on; once you’re familiar with the technique and have practiced it a bit, you’ll start doing it subconsciously and will quickly develop a gut feeling about a project’s potential. However, learning the “proper” way of calculating ROI can help you build the foundation for this mental model and refine it.

While the “I” can be estimated from experience, the “R” needs to be deduced based on first principles

So how to estimate ROI? There are two components in ROI, Return and Investment.

I won’t spend too much time on the investment piece in this article because it’s less generalizable. If a project is similar to work you’ve done in the past, you will likely have a pretty good idea of how much effort it’ll be. And if it’s something new, your estimate will likely be very off anyways, so don’t spend too much time on it. According to Hofstadter’s Law, things always take longer than expected, so whatever your initial estimated effort is, double it and call it a day.

The piece that is much more important to get right — but that a lot of people struggle to estimate — is the “return” component. Luckily, this is the part that you can actually estimate somewhat accurately by taking a first-principles approach, and the framework is generalizable for different domains.

Step 1. Define the “return”

When estimating the impact, you first need to decide which metric you want to anchor the impact on. For example, for a planned feature that is supposed to drive user engagement, should you estimate the % increase of total time spent on site or the amount of additional revenue?

This is important because it facilitates a “common currency” and enables an apples-to-apples comparison. Without anchoring to key metrics, you might have one initiative sized to drive “15% incremental time spent on site” and another sized to “increase revenue by 5%”. How do you know which one is more impactful?

Besides making sure you’re using the same impact metric for all estimates, it’s also important that you’re choosing one that teams think and care about (e.g. because they’ve set goals against it); otherwise, it’s hard to convey the impact because you won’t get their attention.

Which metric you ultimately choose depends highly on what your company cares about. In most companies, there are a handful of metrics that everyone rallies around, be it a north star metric or a set of topline KPIs around growth and revenue or profitability. In addition, each org may have a couple of key metrics it’s responsible for driving, and staying close to the planning process will help you understand what those are.

To have a tangible example for the rest of this post, let’s say your company ultimately wants to know the impact of any initiative in terms of revenue.

Step 2. Have clarity about which lever you are moving

Once you decide what impact metric you are doing the sizing on, the problem becomes much clearer. You’re basically asking:

“How much can I lift X (your “return” or impact metric) by improving Y (the lever that can be improved by the initiative you have in mind) by Z%?”

You already know your X (the ultimate business impact you’re sizing; in our example, that’s revenue) from the previous section, so the next step is to figure out Y (the immediate improvement your initiative can drive). This means you have to translate the qualitative thing you plan to do to the immediate quantitative impact you expect.

To continue our example, let’s say you are a DS working with the feed product team at a social media company. And one of the initiatives the team has in mind is to improve the feed ranking algorithm so that users see more relevant content and ultimately scroll more and consume more content units.

In this case, our Y (the quantitative impact from our initiative) would be “# content consumed / visitor” .

Then it comes to the difficult part: Translating this improvement into the ultimate business impact (i.e. revenue increase) you care about. This requires establishing a lineage between the metrics, and the best way to do that is through building a driver tree.

Which brings us to the next essential step.

Step 3. Understand other levers that affect your target metric

Building out the driver tree has two key advantages:

1. Establishing a lineage between your initiative impact Y and your target metric X (as mentioned above)

2. Providing a comprehensive list of levers for X

At this point you might ask, why is the second piece important if I already know which lever I want to move?

The reason is that there are typically a lot of ways (levers) to move your target metric, and any given initiative will only affect a subset of them. It’s essential to have visibility into all the other moving pieces, because by asking “How much can this initiative improve X?”, we are really asking “How much can this improve X, holding everything else equal?”.

Much like when running a regression, you need to control for confounding factors to establish the relationship between X and Y.

I have covered driver trees extensively in my previous articles, so I will not repeat everything here, but the general idea is to break down the target metric into drivers that are MECE (Mutually Exclusive and Collectively Exhaustive). What this means is best explained by applying this principle to our example.

As a recap, your ultimate goal is to increase revenue. The company likely has more than one revenue stream; but as a social media company, any improvement on feed will likely mostly impact the revenue stream coming from ads. This means we can focus on ads revenue as the top of the driver tree.

Let’s also assume ads revenue is click-based (i.e. advertisers are charged, and you make money, per click). We can then decompose this mathematically:

This breakdown is MECE because 1) Any change in ad revenue is explained by a change in one of these three drivers (i.e. “collectively exhaustive”), and 2) every driver represents a distinct component with no overlap (i.e. “mutually exclusive”): Volume, conversion and price.

Then, “# ad impressions” can be further broken down:

You need to keep on building out the tree (by breaking down the components into more granular subcomponents like shown above) until the metric you plan to move through your initiative (Y) shows up as a branch in the tree; then you can figure out the quantitative relationship by tracing the path all the way up to your target metric X.

And all the other branches on the tree are the additional levers you need to control for in your sizing.

Step 4. Use the lineage to build the sizing and make implicit assumptions explicit

The previous steps were about understanding the relationships between what you can control and the impact metric you are ultimately trying to size. Now, it’s time to start creating the actual structure of your sizing calculation.

This step should be pretty easy once you’ve done all of the prep work in the previous steps; you have all the pieces and you just need to put them together. The only tricky part here is thinking through how to set this up in a spreadsheet. Ideally you want to make this sizing not only a hacky one-off effort that lands in the trash after, but rather something that can be leveraged in the future for similar initiatives that aim to drive the same impact metric.

One tip for enabling this scalability is to separate the structure of the calculation from the assumptions. In other words, list out the relevant variables from your driver tree and link them, and then have a separate section where you list out all of your assumptions like the expected lift from your initiative (instead of hardcoding them into the formulas).

This has a few key advantages besides making the sizing re-usable:

• Transparency: It’s easy for others to see in one spot what assumptions you’re making. If anyone questions the end result, you can quickly figure out which exact assumption they disagree with

• Scenario modeling: You can easily tweak each assumption to see how the outcome changes. For example, you could figure out what the upper and lower bounds of possible impact are by plugging in the most ambitious or conservative assumptions for each driver

• Diagnosing underperformance: If the initiative gets launched and performance is not meeting the projected impact, it’s easier to root cause why you’re missing the target, i.e. which specific assumption was off. This then allows you to refine your projections in the future

Step 5. Plug numbers into your sizing

Now that you have all the levers listed out, how do you know what values to plug in?

To start, you need baseline numbers for all of your metrics; this can be something like the average value over the last three months, and you can usually pull it from a dashboard or directly from a table in the data warehouse.

Then, you need to apply the anticipated % improvement to the lever you plan to move (the “Z” from our equation under Step 2). I.e. in our example, we need to plug in the % increase in feed content consumption per user that we expect from our ranking change.

You can estimate this anticipated lift by benchmarking against similar past initiatives. It’s likely not the first time the team has made improvements to the ranker to improve relevance, so you can reference those experiments or launches to get a rough idea of what to expect this time.

You’ll often find that there are trends you can forecast out; for example, you might find that the team has tackled all of the low-hanging fruit already, and the most recent improvements were mostly smaller fine-tuning changes with diminishing returns. Taking these trends into account will get you a more accurate range for where the impact of your upcoming initiative might land.

Step 6. Consider and model out any ripple effect of this change

Until this step, we have been working off of one big assumption: our initiative will move certain levers and the rest of them will be constant. This is essentially assuming that there’s no interaction effect between different levers. In reality though, when one lever is moved, there’s almost always ripple effects on other ones, and it’s important to anticipate these.

For the sizing to be as accurate as possible, it’s crucial to model out the major interactions. In our example, when we increase relevance of feed content, members will consume more content when they are on site. And in turn they will consume more ads (assuming the ads share of contents is kept the same).

It would be naive to assume, however, that members will keep the same level of click-through-rate on ads when they are shown more of them.

In addition, if the ranking change is expected to affect some user groups more than others (e.g. because it’s leveraging a signal that is more relevant for certain users), those users will start accounting for a larger share of content and ad impressions, and your average cost per click might change as a result (since not all users are equally valuable to advertisers).

Again, past experiments will be the best data source to estimate this impact. Once you have enough data, you can start developing simple rules of thumb such as “A 1% increase in the number of ads shown per session results in a X% decrease in CTR”. This will make it easier to do these sizing exercises in the future since you won’t have to look at raw experiment data every time.

Lastly, these ripple effects should be part of the “Assumptions” section of the sizing spreadsheet so it’s easy to tweak them and model out different scenarios when needed.

Remember, sizing is an art as much as it’s a science.

Of course you can always make the sizing more accurate; but at the end of the day, it’s important to remember that nobody has a crystal ball that predicts the future. The sizing is just an attempt to estimate the order of magnitude of what you expect to happen, so you need to be comfortable with “good enoughs” in this process.

Remember, while “good enough” is not perfect, it’s better than flying blind and having no estimation at all.

Making all the assumptions explicit and tweak-able is enabling us to iterate on the sizing when we have learnings in the process (e.g. in the online experiment, we might notice that we have under-estimated some ripple effect) so our modeling can get closer to reality as we iterate on it with new information.

“He’s not ready to be promoted. Every time I ask him a question, I get this long-winded answer that’s loaded with unnecessary details… ”

This manager was saying out loud what a lot of managers and executives are thinking (but rarely give as direct feedback). Too much detail doesn’t make you seem more capable — on the contrary, it makes you look junior.

As a manager, I have led teams and coached a lot of bright DS in my career. Part of a manager’s job is to determine, among a team of bright people, who to pull in for what work.

Managers can usually quickly form a “gut feeling” about who they can bring in to lead cross-functional work streams independently versus who can execute but isn’t ready for leadership yet — whether formal (e.g. as a tech lead) or informal (leading a project).

So what are the key distinguishers between these two groups of individuals? After talking to a lot of managers, I’ve noticed there is a consensus:

It’s usually not the ability to do technical work (surprising, I know), but rather certain behavioral/communication patterns that make people perceive you as a lead, take you seriously and pay attention to what you have to say.

In this article, I will break down some of the key things that will make you appear junior, convince you why it’s important to improve on them and teach you how to do exactly that.

Four behaviors that make you seem junior — and what to do instead

Junior trait #1: Providing too much details/over-explaining

This is innate to analytics folks since the field requires us to pay attention to details. As a result, a lot of junior DS think providing a lot of details will showcase that they have considered a lot of the edge cases and give the audience confidence that they have a lot of knowledge about the field.

In reality, though, too much detail will often confuse your audience and make you look like you don’t know how to synthesize information. As a result, you will lose people in your communication and cause them to miss your key points since they are buried in all the details (think the VP who starts to scroll on his phone when you get into the nitty gritty of how you pulled the data in SQL).

I have literally had stakeholders telling me “Don’t put this person in front of execs because they will confuse everyone”.

This might sound harsh, but everyone who has ever been in a large corporate meeting knows how inefficient they are even when things are going well. That means as a manager you cannot afford to add additional friction and confusion by having a team member lead the discussion who is not ready.

Don’t get me wrong, it is crucial to consider all the edge cases and pay attention to details during your analysis. But that doesn’t mean all those details and edge cases are worthy of being communicated to others when you present your work (unless they specifically ask about them).

What to do about it

This should be practiced in both written and verbal form.

In written form, try to summarize your work in a TL;DR (a brief blurb highlighting the most important takeaways) that you put at the top of your document or email.

Ideally write your TL;DR using the Pyramid Principle: Start with the conclusion related to the business question you want to answer, and then add key supporting evidence as needed.

Remember, being able to decide what NOT to communicate is as important as deciding what to mention.

For everything you want to communicate, ask yourself if it’s “important enough”. If not, put it in the appendix in case someone asks but remove it from the summary.

For example, imagine that during your analysis you discover a data issue that applies to 0.1% of the member base. Ask yourself: “Does this data error fundamentally change the conclusion of my analysis and/or my recommendation?”

The answer is most likely “No”, so only bring it up when asked.

In verbal form, practice talking about your project with peers who are not deep in the weeds of the work with you to see if your summary confuses them. And when crafting this verbal summary in your head, imagine you are doing an “elevator pitch” to the CEO – giving him/her a good enough description of your work in less than 30 seconds that makes them deeply understand the “why” of your work and gives just enough context on the “what” so that they can form an opinion.

Junior trait #2: Not having an opinion or recommendation

When asked “Based on your analysis, what would be your recommendation?”, one of my former team members loved using the phrase “I have no dog in this fight”. This might not seem like a big deal, but it’s actually an example of one of the most common career-limiting traits of data scientists.

The more senior you become as a data scientist (and the more AI replaces the “execution” part of the job), the more important it becomes to translate your analysis into a recommendation for your stakeholders.

But it’s not surprising that junior DS struggle with this part. Forming an opinion or recommendation takes more than just pulling the data; it requires digesting and understanding the data and connecting it to the needs of the business. And let’s face it, when you’re deep in the weeds of a 500-line query or a complex data pipeline, it’s easy to lose sight of the bigger picture. So a lot of people do a great job executing the data pull and then leave it to their audience to make sense of it.

The other reason a lot of data scientists or analysts are hesitant to give a clear recommendation is the perceived risk involved. Hedging is deeply wired into our risk-averse analytical brains.

This makes sense, since data is nuanced and nothing is ever completely black-and-white. But this can backfire when you’re working in a fast-moving environment like a startup and becoming a blocker because you’re drowning your audience — who are trying to make a decision — in caveats.

Every decision comes with risks, and every recommendation has caveats. But putting too much emphasis on them results in analysis paralysis — gathering more and more data and doing more and more analysis, hoping the decision will make itself clear in the process.

Ultimately, not giving recommendations shows a lack of ownership of the problem and lack of the ability or willingness to make sense of the data.

And you’ll be limited to simple “execution” work as a result.

What to do about it

First and foremost, adopt an ownership mindset.

Most of us are able to make decisions and have strong opinions when it comes to things we care about. For example, you were able to decide where to travel last time you planned a trip, right?

You probably gathered information about weather, flight tickets, reviews etc. to decide between a couple of options. And despite all that complex information, you landed on a clear decision in the end. Why would business decisions be different?

Imagine that you’re the decision-maker and have to make this call:

• What data would you need?

• And once you pull those data points, ask yourself: Would you convinced by the data presented?

• If not, what else can possibly convince you?

Just don’t fall into the trap of thinking that gathering more and more data is the solution when you’re stuck. In a lot of cases, what we actually need is to digest and make sense of the data we already have. 1,000 different data cuts will only confuse people and take them further away from reaching a decision since each cut is telling a slightly different story and it’s difficult to make sense of them all.

And if the fear of risk is what’s stopping you from having a recommendation, remember this: The only way to generate above-average returns in financial markets is by taking risks; risk-free strategies only yield the low risk-free rate of return. The same applies at work; if the decision were crystal clear with no uncertainty or risk involved, nobody would need you to make a recommendation. The value you add comes from giving robust recommendations despite nuance and ambiguity.

Don’t get me wrong, you should definitely list out the caveats of your recommendation. But trust me: Most people prefer that you have a recommendation that they don’t agree with instead of having no recommendation at all. Because if you do have a recommendation and list out how you arrived at it, they can dig in to see which of your assumptions and decisions they agree with and which ones they don’t.

At the end of the day, data teams are not paid to only pull and present data; we are paid to help drive business decisions with data.

So if you always leave the mental burden of digesting data and making the decision to your audience or others on your team, you limit yourself to being a junior analyst instead of a true thought partner.

Junior trait #3: Not being clear about the “why” behind the analysis

I can’t count how many times — after I talked to my team about an analysis — I felt like we were solving the wrong problem.

When I ask the individual “Why are we doing this analysis? What’s the business decision behind this that we are trying to drive?”, the answer I usually get from junior DS is “XYZ stakeholder asked for this”.

That should never be the only reason you are working on an analysis.

The interesting thing is, one of the most common complaints I hear from data scientists is being treated as a “data puller”; but when involved in an analysis, a lot of junior analysts voluntarily demote themselves to “data pullers”.

They are so laser-focused on finding out what data they need or what data cuts they should produce that they leave the big picture questions like “What problem are we trying to solve?” and “Why is it important to solve this problem?” to their managers or tech leads to figure out.

Whenever this happens, it serves as a signal to me, the manager, that this person is not ready to run projects independently because I need to go back to the stakeholder to inquire about the motivation of the analysis.

So why is this so problematic, especially if you want to grow into a lead role?

It goes back to the ownership mindset. Not knowing the “why” behind the analysis means you are not taking ownership of the business problem. More importantly, not being clear about the why means not being clear about the problem space, which severely hinders your ability to deliver the most effective solutions.

People who don’t have a deep understanding of the problem space often conduct analyses that don't answer the exact question stakeholders have in mind. These stakeholders will then ask for additional data, hoping that it will help them answer the original question. As a result, the analyst gets frustrated because the “data requests” constantly change, and a vicious cycle starts.

What to do about it

Similar to my suggestion in the last section — start by owning the problem. If your stakeholder tells you “I’m trying to understand how many people used our search function in the past month”, instead of directly jumping into pulling the number, you should try to find out why they need that data. What decision are they trying to make?

Once you understand the ultimate problem your business partners have in mind, you can help them brainstorm the best data that can be used to solve that problem (and trust me, in a lot of cases it’s not the original data they asked you to pull).

This way, you are truly elevating yourself to be a thought partner and owner of the problem, not just somebody who executes data pulls. If you find yourself struggling with this, The Operator’s Handbook has an in-depth guide on how to deeply understand a problem space and find the most effective solution from there.

Junior trait #4: Not having the basics down to be able to stay “one step ahead”

Imagine an economist is giving an interview about their opinion on the tariffs, but stumbles when asked “Roughly what % of American goods are imported?”. Would you trust anything they say afterwards?

Probably not. Similarly, the easiest way to lose credibility as a data scientist is when people feel like you don’t really know the data you’re working with or the area of the business you’re covering. For example, if you’re doing a deep-dive analysis into user behavior on your company’s platform, you should know at the top of your head roughly how many active users the platform has.

If you want to establish yourself as an expert in an area, you need to stay “one step ahead” of your stakeholders. If you are the POC for a topic, you are expected to be the most familiar with the data (even compared to your manager) and have an answer ready for the most common questions people might have.

Don’t fret, I’m not saying you need to be able to anticipate every single “follow up” question that people may have. But imagine you’re presenting an analysis and share the fact that more than 80% of your product’s users are based in the US.

Most of your audience will naturally wonder things like:

• “What are the other major countries besides the US?”

• “Does the US also account for 80% of DAUs and revenue, or a larger or smaller share?”

If you don’t have an answer for any of the “natural follow-up questions”, it will make people wonder whether you really spent enough time to understand and explore the data.

What to do about it

Try to be curious about the data you’re working with. Start with a basic question, and then explore the data from there.

For example, imagine you work at Facebook and are analyzing marketplace data. The first thing you check is how big the product is and you find that “10% of all Facebook DAUs use FB Marketplace”.

What natural follow-up questions could you explore from here?

For example, you might want to check how this number has changed over time, and how that 10% compares to other product surfaces. Let your curiosity carry you a little bit and jot down the answers to these questions. The key is to deeply understand the context of the data you plan to present without wandering too far from the key question you’re trying to answer for the business. It’s a tricky balance, I know.

Generally, you can expect follow-up questions from stakeholders to fall into three buckets:

1. Foundational knowledge: Different stakeholders will have different levels of pre-existing knowledge on the topic you’re presenting. Some of them might even have very basic questions (e.g. how the product works, what steps the onboarding flow has, how many users there are etc.)

2. Your analysis: You should always share only the most important insights from your analysis rather than dumping everything you did on your stakeholders. However, stakeholders might ask for more detail on certain aspects, and you should be prepared to elaborate when needed

3. Next steps: Stakeholders will often be curious about what your analysis means for them. What are the recommended next steps? What changes should they make based on your findings?

Think through these three areas before you present an analysis, and you’ll be able to anticipate many of the questions you’ll get.

Lastly: Get a second pair of eyes on your work (or use ChatGPT) before you present to a broader audience. Ideally, this person is someone who’s not as deep in the weeds of the analysis as you are; this will help you find any obvious things you missed because you were too close to the data.

Conclusion

A lot of the “symptoms” mentioned in this article represent a common theme — lacking an ownership mindset. What establishes you as a lead in an area is not the title you have or how long you have been working in that area; it’s the mindset you adopt and how you take responsibility for owning and solving a problem.

So if you want to be perceived as a trusted thought partner instead of a junior IC who can only help pull data, put yourself in the shoes of the decision makers and treat the decisions with the same level of rigor as the ones you care deeply about.

“What are the drivers for churn?”

As data scientists, we encounter questions like these every day. A stakeholder comes across something they want to understand better, and they look to us to take an open-ended question like this and make sense of it.

However, even seasoned data scientists are sometimes stunned by just how vague or open-ended these questions are. As a result, they don’t know where to start or how to carry these analyses forward.

In fact, a lot of analyses either circle high level at 30,000 feet and only scratch the surface, or get stuck in a rabbit hole that’s very far from the original question we hoped to answer. 

Like mentioned at the beginning, there are generally two categories of open-ended business questions data scientists encounter: 

1. Questions that are more investigative and aim to identify drivers of a past event; most companies/people refer to this as root cause analysis (e.g. “XX metrics decreased week over week, what happened?”)

2. Questions that are of exploratory nature and focus on identifying drivers of improvements (e.g. “we want to improve XX metric, what are the things we can do?”)

There are a lot of similarities between how you would carry out these types of analyses, but there are also slight differences.

In this article, I will focus on investigative root cause analysis and:

• Share general tips for how to conduct analysis for open-ended questions

• Introduce the high level steps for this type of analysis

• Go through a hands-on guide for each detailed step

• Apply the hands-on guide on a realistic example so you can see how it can be carried out in practice

I will get to exploratory analysis in another article.

General advice & pitfalls to avoid when answering open ended business questions

Have a framework

From my last article, you should already know how much I love frameworks and hopefully learned a little about the common frameworks that DS should apply in their analyses.

When it comes to investigative work, the most relevant framework is the “hypothesis tree”, where you try to list out all the probable causes of a problem in a MECE way and rule them out one by one until you find the culprit(s). 

Don’t expect to get there in a straight line

These types of analysis are hard to do because the process sometimes can be more of an art than strict science. It’s a creative process and it’s guided by curiosity and quality questions/hypotheses.

Because it’s open-ended, there isn’t one predetermined “correct” way to get to the answer. In fact, the steps that will eventually lead us to a conclusion cannot all be planned in advance.

The “hypothesis tree” can only be built out a couple of levels at a time, because you don’t know what insights or patterns you will find in the data and what hypotheses you can reject with the data.

These insights will largely dictate what your next move will be.

This means you will explore and reject a lot of “not-it” hypotheses along the way. However, these detours are not a waste of time. Doing data-related investigation is not that different from investigations in police work — ruling out “suspects” is a key step in having a thorough investigation, so expect to chase down a lot of dead-end leads before getting to the “lead suspect”.

Have your eyes on the goal at all times

This is super important. When I was a junior DS, I sometimes found myself elbow deep in my analysis and only after a couple of hours of exploring found myself wondering “How did I end up here? What am I looking for? How does this help answer my original question?” 

It’s important to ask yourself these questions frequently in your process. There will likely be a lot of people with different opinions about what you should look into, and you will have a million data points you could be investigating.

One way to help decide what is worth digging deeper into is to see whether it has a clear lineage to trace back to the ultimate question you want to answer.

Act on curiosity, but don’t let it take you too far down a rabbit hole

Going back to my comparison between data investigation and detective work:

More often than not, when you come across something unexpected, it can lead to a big discovery that will change the course of your investigation. So it’s important to act on curiosity when it comes to this type of analysis — if you see something weird or interesting, dig in and see if there’s anything valuable you haven’t considered. 

But remember to utilize the tip above to know when it’s time to call it quits and stop the descent into a rabbit hole. Think about the hypothesis you’re investigating and ask yourself:

• Have you seen enough evidence to prove or reject it?

• If not, what else do you need to close the loop?

The 5 steps of an analytical investigation

Step 1: Scope the analysis 

It’s crucial to understand the business problem at hand so you can set the scope of your analysis. Let’s say your stakeholders asked the following question: “XX metric is down 10%, what caused it?”

In this case, it’s important to understand what the benchmark is (i.e. is it down 10% compared to forecast? Or is it down 10% compared to a previous period?) and what time period is the observation based on.

This will help you decide which time period of data you should be observing, what else you need to understand (for example, if the comparison is to a forecast, you need to understand how the forecast was created) and what hypothesis you can quickly rule out even without data. 

Step 2: Generate hypotheses (using a hypothesis tree)

You should always rely on two things in hypothesis generation — your curiosity and a framework. 

A framework will get you started with a basic set of hypotheses that can represent the most common issues based on logical reasoning. Your curiosity will then lead you to observe “interesting” things in the data (e.g. counter intuitive findings) that can inspire new hypotheses.

In other words, finding the root cause will be an iterative process. Even if it’s tempting, don’t try to map out the whole hypothesis tree at the beginning (trust me, I have tried, it will quickly fall apart).

As a reminder, this is what a hypothesis tree looks like:

Step 3: Decide what data you need to test the hypotheses, retrieve and QA the data

Once you have a new set of hypotheses developed, the goal is to quickly find ways to validate or reject them so you can move forward to the next iteration of the cycle.

The very key to this is securing the right data and making sure it can be trusted.

Once you get your hands on the data, the first step should never be to jump into hypotheses testing or insights generation — it should be to QA your data.

Because simply put, you should never blindly trust the data you work with. The data gathered in real life is never as clean and structured as Kaggle datasets. The work also doesn’t go to waste; a lot of these QA visualizations and summary statistics can be reused when you pull all the insights and story together in the end.

Some typical QA questions to look into: 

Are there any missing/NULL data?

If there are missing/NULL values, is it big enough to be worried about? If it’s a small amount, should you simply ignore those records or delete them if they distort the conclusion you are trying to draw.

Speed often matters more in investigations than perfection.

If it’s a significant amount, did you just discover a data issue? In this case, you should aim to understand why the values are NULL. Is there a legitimate reason why this field wouldn’t have a value in many cases, or is there a problem with the data pipeline? 

Once you understand the “why”, you can decide whether it makes sense to replace the NULL values or not.

How is the data distributed, and does it align with your expectation?

It’s a good habit to always pull some useful summary stats of the data you are looking into :

• What’s the mean, median, percentiles?

• Are there any outliers?

• Does the distribution align with your understanding / expectation?

For example, let’s say one of your hypotheses is:

“The Growth team told me we recently increased our email send frequency from once a week to 3 times a week. That might have led to more email unsubscribes, and in turn led to fewer people coming to our website.”

To validate this hypothesis, you pulled email sent/unsubscribe data for the past weeks. Instead of directly looking at whether unsubscribe quantity increased, you might want to check the distributions of emails sent as well.

You might find that some people are receiving way more than the “3 emails per week” you have been told and assumed. That might send your investigation in a completely new direction:

Is this expected behavior? Are the email sequences correctly configured, or could excessive email volume (e.g. because a suppression rule is not working) explain the metric drops you saw?

Step 4: Test hypotheses

Now that you know you have trust-worthy data, you can use it to test the hypotheses you have in mind.

The key here is  to decide ahead of time what you need to accept/reject a hypothesis. This can prevent you from randomly exploring and getting distracted, or continuing down a branch of your investigation when you should have already moved on.

There are two components to what’s “sufficient”, one from an academic angle, one from a storytelling angle.

• For the academic side, it’s usually best to stick to the standard suite of statistical tools for hypothesis testing (t-test, z-test etc.); I will not cover them in detail here, but likely provide a separate deep dive on experimentation in the future.

• The harder piece for most people to figure out is the storytelling angle. What data do you need to create a convincing narrative about what has happened?

Proving something academically through data is one thing; getting all stakeholders on the same page with regards to what’s going on is a whole other challenge.

And in order to convince others, YOU need to form an opinion first. Like I mentioned in my article about the key mindset data scientists should possess, a data scientist’s job is not to simply present data, but to use data to guide decisions.

If you can’t even convince yourself, how can you convince others? 

Step 5: Generate and present insights with ranked importance

Once you identify the “smoking gun(s)” and can exit the “hypothesis generation → hypothesis testing” loop, you need to organize your findings and tell a compelling story.

Storytelling with data is another key craft for data scientists to master; it deserves its own article and I will cover it in detail in the future. In the meantime, here are some high-level tips:

My high level suggestion is to ONLY present relevant data points to avoid confusion and distraction. Ask yourself: “What is the simplest, most direct way I can explain what happened?” and then create the story by either writing the headlines for a slide deck or by drafting the outline of a summary document.

Only once you have the storyline in place, you should start to fill in the charts and tables supporting that narrative. This ensures that every piece of data you are showing is crucial for the explanation.

With that being said, all the “dead-end” leads you explored are not wasted. They go into the appendix, and you can point to them when people inevitably ask “have you considered…” / “what about…”.

And all of your data QA? That also deserves a place in the appendix so when people ask questions about the data and its integrity, you can show your in-depth knowledge and the thoroughness of your analysis.

This helps build trust in you and your analysis: Because if you can’t even answer the most basic question about the data you are using, how can I trust the insights you developed?

In this process, it’s important to ask yourself: “What questions would others ask to challenge and poke holes in the narrative?”. If you want to go deeper on this topic, check out the articles on sanity checking your work and anticipating questions from stakeholders by Torsten Walbaum.

Case Study: Investigating why total video view time is down on YouTube

Disclaimer: This is a simplified, made up example for demonstration purposes; all data is fabricated by the author

It’s time to put all of this into practice.

Let’s say you work at YouTube and someone from leadership asks you to dig into why the total video view time is down.

Step 1: Understand the concrete question/issue at hand

First, ask clarifying questions so you are crystal clear about the question you’re answering. For example:

• "What time period are we talking about?”

• “What’s the benchmark we compare it to when we say the view time is down?”

For demonstration purposes, let’s assume we found out that the weekly total video view time is down by 10% compared to last week. 

Step 2: Generate hypotheses

Start by laying out the debugging structure (AKA the hypothesis tree) to break the problem into smaller pieces you can tackle. As much as possible, you should follow the MECE principle.

In this case, the first layer of the hypothesis tree is very simple:

Total watch time = number of unique watchers * average watch time

Can we isolate the issue to one of part of that equation?

For this example, let’s assume we can: Average watch time is flat, but the number of unique watchers is down by 10%. Congrats, you just narrowed down your scope for the analysis. 

Then one level down, one (of many) MECE way to break down all the factors is “internal factor” (e.g. data error, engineering change etc.) vs. “external factor” (e.g. seasonality, macroeconomic factor etc.). However, you need to be a little more concrete than this to have testable/verifiable hypotheses. 

In reality (not in an interview setting), some of the low hanging fruit hypotheses can even be verified without much data retrieval. So it’s crucial to quickly identify and test them.

The hypotheses listed below are some easy-to verify ones that we can quickly rule out or accept:

Step 3 — Step 4: Decide what data you need to test the hypotheses, QA the data if needed, test the hypothesis and attribute the change

You should easily be able to verify if any major initiatives (e.g. new launches) happened in the relevant time frame that could be responsible for tanking the metric.

Most companies A/B test big changes, so you should look through the data for recent experiments to see if any can explain the change. Let’s say we see that one of the changes we recently rolled out affected the SEO ranking of our videos on Google, and it caused a reduction in SEO traffic and fewer unique watchers as a result. 

We can also quickly verify whether there’s any seasonality at play. In this case, we are looking at weekly aggregated data, so we can disregard intra-week seasonality. The best way to check this is to look at weekly data trends over long time periods (e.g. by plotting different years on top of each other) and see if there are any repeating patterns.

Judging macroeconomic factors is more of an art than science; they are typically not recurring, so you won’t have historical data to use as a benchmark. One way to attribute/estimate the effect of these factors to look at industry or competitor data since macroeconomic developments will affect the market broadly.

For example, during COVID, many retail stores experienced similar dips in foot traffic and revenue.

Repeat steps 2–4 until you can attribute all the changes

While practice problems might make it seem like that, in reality, it’s usually not just one or two factors that caused the change you’re observing. Often, many factors combined explain the metric movement, although some contribute more than others:

So it’s important to treat “hypothesis generation -> hypothesis testing” as a recurring and iterative process until we can explain the whole magnitude of change we observed. 

Final Thoughts

Using data to answer business questions is never an easy process. The questions are usually open-ended, and it’s up to you to develop hypotheses and eventually identify the relevant driver(s).

The most important part is to have a framework that you can follow when exploring different possibilities. Otherwise, it’s easy to get lost in an ocean of random guesses and hunches.

And without a way to systematically rule out hypotheses one by one, working groups often forget what they have already concluded and what they still need to test, causing you to go in circles.

Hopefully this article provides you a helpful guide for structured investigations.

Since the final (and arguably most important) step is to package the findings into a convincing story, I will cover data storytelling in my next post. Stay tuned!

As a manager, part of my job is to coach data scientists on the team when it comes to projects and career growth in general. I realized what junior data scientists struggle with the most is usually not the technical/execution part of the job — that’s the easy to teach/easy to learn part.

It’s usually the more abstract/soft-skill-related part of the job that most people don’t know how to navigate — things like how to break down an abstract business problem into smaller, clearly defined analyses that can eventually lead to concrete business impact.

These are the things I got to practice day in day out as a consultant, and I think the learnings carry over to data science very well.

To help my fellow data scientists, I want to summarize my learnings from my consulting days so you can benefit from them without going through the grind.

In this article, I will:

• Convince you why consulting-style training can immensely benefit data scientists of any level

• Walk you in detail through the most valuable frameworks I learned at McKinsey that you can apply in your day-to-day work

Why I think consulting experience can benefit every data scientist

Reason 1: It grows your ability to learn about an area and make impact fast.

Because consulting projects are always strapped for time (consultants are billed by the hour), consultants don’t have the luxury to spend months to learn about all the background, context and client-specific subject matter in depth before being pressured to deliver solutions.

So consultants are trained to learn about an area efficiently and make impact along the way. There are a lot of skills involved in this process:

• Asking the right questions to the right people; what is the information you really need to know to solve the problem?

• Discovering gaps and figuring out short-term solutions to plug the gap

• Turning short-term solutions into long-term ones and identifying the right stakeholders to help pushing things forward.

These skills are essential because while “great things take time”, people still expect you to deliver incremental “good things” on the way of delivering great things

Delivering the (albeit not perfect) MVP in a short timeframe that makes a big (enough) impact is the goal. This forces consultants to constantly deploy the 80/20 rule:

With 20% of the effort you can get 80% of the impact.

There are two reasons for this:

1. Every activity quickly hits diminishing returns. For example, building the first version of a dashboard when the team previously had no way of looking at data has a huge impact. As you keep refining the dashboard, you quickly get to the point where a new filter or toggle only adds a small benefit for individual users

2. You don’t need perfect accuracy to make most decisions. For example, to make a “Go” or “No-Go” decision, you often just need to know whether a number will land in a certain ballpark range (e.g. $150M — $200M), not what the exact number will be.

This will be completely out of the comfort zone of a lot of engineers and data scientists. But remember that at the end of the day we are trying to run a business, not write a white paper; it’s crucial to build this mental flexibility.

Reason 2: It teaches you to be a full-stack data scientist when needed.

Most consulting teams are not staffed with data engineers + data scientists + ML engineers + PMs + biz ops.

So if you need raw data cleaned and built into pipelines to run an analysis, you don’t have data engineers to count on, you have to pick it up; if you need to estimate the impact of an initiative, put on your biz ops hat and do the back-of-the-envelop calculation yourself.

Most data scientists in consulting are full-stack data scientists; they can wear different hats when needed. For one project, we built an end-to-end AB test analytics solution in a month for an e-commerce company; for another project, we built a prediction model for movies’ box office performance in a few weeks so the team can make fast decisions about which screenplays to fund.

• Are these solutions as good as a proper SaaS product from a big tech company? Definitely not.

• But are they good enough to solve a business problem timely? Almost certainly.

And most importantly, I learned a ton about how to deliver solutions end-to-end wearing different hats. This makes my collaboration nowadays with partner teams like ML engineers, data engineers and PMs that much smoother because I know a little about their jobs.

Reason 3: It “forces” you to see the big picture and communicate in a succinct but clear way (both written and verbal communication).

While in bigger tech companies you won’t get a lot of visibility in front of executives as a junior IC, consultants can usually bypass the hierarchy and get the opportunity to work directly with C-level executives.

These executives usually have a million things on their plates and they are constantly context switching — so communication with them needs to be high level and effective.

Junior ICs in the analytics realm usually struggle with this type of communication because the analytics work itself requires you to be deep in the weeds and it’s hard to then to zoom out when communicating your findings.

Consultants spends hours honing every client-facing deck and documentation to nail this skill. Every piece of material goes through peer review, manager review and review from partners so people at different levels of familiarity with the project can help judge whether the material is easy to digest.

Compared to junior ICs in the big tech companies who may only get a handful of opportunities per year to practice this type of communication, consultants’ frequent communication with execs forces them to practice and master the ability to not only see but also communicate the big picture.

McKinsey’s “Secret Sauce”: The most useful consulting frameworks and how to apply them to analytics

But not everyone will get a chance or is interested in being in consulting during their career.

So let me introduce you to some of the most important frameworks that can help you get the essence of the consulting way of thinking without going through the grind.

MECE framework

If there’s one thing you take away from this article, it should be this framework.

This is the first framework new consultants are introduced to and it’s a framework that can help you think through just about any problem in life.

MECE stands for mutually exclusive and collectively exhaustive.

It’s a way to break down something big or abstract into smaller buckets to make the problem more approachable while making sure that there’s no duplicated effort (“mutually exclusive”) or missed area (“collectively exhaustive”) in the process.

Let me give an example: Let’s say you want to segment the member base of Instagram by age. It’s not MECE to have groups like “underage (<18)”, “teenagers (13–19)” because there’s duplication.

Similarly, segmenting the data into groups like “underage (<18)” and “age 18–60” is not MECE; while there is no overlap, it doesn’t cover the whole universe of possibilities. What about our friendly senior citizens above 60?

Not all problem can be neatly broken down in the MECE way; but if you apply concerted effort to solve problems with this framework, your approach will be a lot more comprehensive.

MECE is the foundation of most other frameworks; in other words, any good framework that involves any segmentation or categorization should be mutually exclusive and collectively exhaustive.

Issue Trees

An issue tree is the best embodiment of the MECE framework. It is commonly used to decompose a complicated problem and show how different factors contribute to the whole.

For example, the question “How can an e-commerce company increase its profit?” seems like a daunting one without a framework. A lot of people start to brainstorm without any structure:

• “Let’s do more marketing on social media”, or

• “Let’s move our factories to lower-cost countries”

These might be solid ideas, but where did they come from and how can you make sure you explored all the levers at your disposal? The truth is if you brainstorm like this, you will very likely only explore a couple of options and miss many others.

But if you can break down the problem more systematically, you can make sure you explore all avenues and it can even help you delegate since each sub-tree is a smaller-scale problem that can be solved individually.

Let me show you how:

The beauty of issue tree is it breaks down a huge problem into smaller ones so it’s easier to digest — thinking about ways to “increase product variety” is a lot more concrete and digestible than thinking about ways to “increase profit” directly.

Hypothesis Tree

A typical DS interview question is “XX metric is down, how would you go about investigating what’s causing it?” A lot of candidates again start grabbing hypothesis out of thin air. What interviewers (or your stakeholders and managers when it comes to your day-to-day work) want to see is that you can generate and test hypotheses in a structured way.

Hypothesis trees are a variant of issues trees and another great way of utilizing the MECE framework as data scientists is building hypothesis trees for those types of questions. Let’s say we work at or interview with a food delivery marketplace and want to investigate “Why did the # of deliveries in NY go down by 10% in the last week?”

We can start generating hypotheses in a MECE way like below:

Once you have this framework, you can quickly verify or reject the hypotheses one by one and not have that nagging feeling “have I missed anything”.

Of course there are shortcuts you can take once you have accumulated some domain knowledge on the job — for example, if you know that every year during Chinese New Year delivery numbers go down (maybe because the majority of the customers are Chinese and they prefer to have home-cooked meals during Chinese New Year), then you can quickly check that hypothesis first without having to go through an entire framework.

But if you deal with a new problem, hypothesis trees can save the day.

2x2 Matrix

The two-by-two matrix might be the most well-known consulting framework of all.

It’s sometimes frowned upon because it’s extremely simple; but in my experience, this simplicity is often helpful because it forces you to cut through all the (sometimes unnecessary) complexity and distill the issue down to its core.

A two-by-two matrix helps you categorize something into four categories across two dimensions. Let’s look at a simple example; let’s say you are evaluating analytics projects for the next quarter.

Instead of assessing projects across many different factors, which risks dragging out the planning process, you could just focus on two key factors:

1. Does the proposed project support any of the key company priorities?

2. What’s the expected business impact?

Each combination requires a different approach. Laying the problem out in this simple grid can help facilitate discussions and move things forward, which is often more valuable than 100% accuracy.

Minto Pyramid

This one is mentioned in my articles countless times already so you know how important I think it is. It’s THE framework anyone should adopt when it comes communication.

Adopting it takes practice because it requires you to communicate in almost the opposite order compared to how we actually do the work. Most analytics work is open-ended and explorative in nature; so it’s not until the later stages of a project that we have some findings and eventually a conclusion.

But when it comes to communication, it’s crucial that we focus on the conclusion FIRST, then the supporting arguments, and lastly the supporting evidence. This requires you to structure your thoughts in advance rather than just walking through a recap of your analysis.

I have drawn the pyramid itself in previous articles so I will not repeat it here, but it might be helpful to demonstrate the framework using a more concrete example:

Contrary to how a lot of data scientists’ default storytelling mode “we did A, then we discovered B … and here’s the conclusion”, the pyramid framework captures the audience’s attention a lot better; and if they don’t have time to read the whole email, they won’t miss the conclusion.

Conclusion: To have a framework at all

The best framework in my opinion is the one that works for you. Having a framework at all is always better than jumping in with no structure.

The frameworks above are not supposed to be the “end all be all” as there are so many more frameworks out there developed to solve different problems; they are just supposed to get you started in building this muscle.

The most important thing is to develop a way of thinking — the structured way of thinking. Without a framework, you are essentially letting your intuition guide you and hoping it will lead to the right solution by chance. But with a structured framework, on the other hand, problem solving becomes a repeatable and scalable skill.

“Why is my manager nit picking my write-up? What difference does it make changing the wording from X to Y?”

You have probably caught yourself thinking this when you see your managers’ numerous suggestions all over your document; I know I have. In fact, I used to think that writing is the most trivial part of the job of a data scientist; because the analyses and numbers should speak for themselves right? Wrong!

Over the last years, I have realized that writing is an essential skill for data scientists, and that the ability to write well is one of the key things that sets high-impact data scientists apart from their peers.

In this article, I will first convince you that writing is at least as important as your technical skills, and then give you concrete tips to help you improve your writing.

Why is writing so important for data scientists?

1. It’s used everywhere in the corporate world — I have highlighted the importance of communication in my previous articles and like it or not, majority of communication in the corporate world happens in a written form. From project-scoping documents to weekly updates, analysis and experiment write-ups, feedback and performance reviews, JIRA tickets and wiki pages, everything counts on effective written communication to get the message across. 

2. Writing helps to bring clarity to your thinking process — Paul Graham, cofounder of the famous startup accelerator Y Combinator (who’s a computer scientist AND writer) famously said in one of his memos:

If writing down your ideas always makes them more precise and more complete, then no one who hasn’t written about a topic has fully formed ideas about it. And someone who never writes has no fully formed ideas about anything nontrivial. 

— Paul Graham

Very often, when you start writing things down, you realize how little you know about a subject and the potential gaps in your thinking/analysis. 

3. Writing is the “last mile” of your data science work. None of your stakeholders will read your SQL query or look at your Jupyter Notebook (a lot of engineers and data scientists would like believe the opposite but trust me, they likely won’t). If you want your work to be understood by others and influence decisions, then you need to do the final step of packaging it in an effective write-up. If you skip this step, it’s like leaving the package in the warehouse instead of delivering it to the customer.

What does “good” writing look like in data science?

Be clear about your audience. If you are writing for everyone, you are writing for no one. Be very specific about who this particular piece of writing is for, and tailor it to that audience and their needs.

Focus on the “so what”; the sausage-making goes in the appendix. As data scientists, we love to talk about the complex analysis we did or how we designed the experiment. Because we put in all that work it feels so wasteful to NOT talk about it. But the harsh truth is, most of the time, our audience does not care; they just want to understand the takeaways.

You can describe the technical details of your work in the appendix in case someone wants to go deep, but the main part should focus on the insights and recommendations. 

Have a clear storyline. Fiction or not, every piece of (long-form) writing should be a story. Because that’s how humans communicate and that how our brains process information. Usually the storyline for analysis goes like this:

⮕ We found out about something interesting and this is why you should care about it / what you should do (summary to get your readers hooked, including a recommendation if applicable)

⮕ Here’s how we arrived at these insights (analysis details for the curious explorers)

⮕ Here are the caveats and alternative paths forward (optionality in case someone challenges the recommendation)

⮕ Here are additional resources you might find interesting (appendix for those that really want to go deep on the topic)

It might help to build the skeleton first before adding the details. If the story depends on how the analysis goes (which is often the case for DS analysis since the nature is more exploratory), at least figure out the structure of the doc before diving into the details.

If you are building a deck/presentation, I have a little trick that I learned from my consulting days — orchestrate your slide titles as the main storyline. Imagine the reader flipping through the deck ONLY reading each slide’s title, they should get a pretty good idea about the key takeaways.

Have a clear summary. If you remember the pyramid principle I mentioned in my previous post about communication, it’s especially important to written communication. Because the summary is your first touch point with your readers, it should be interesting enough to capture the their attention so they want to read on; at the same time, it should capture all the essence so if they decide to stop reading after the summary, they got all the most crucial information they need to know. 

Be succinct. When it comes to writing, less is more.

Keep it simple. We work in a technical field and use technical jargon all the time. Often, data scientists think it makes them seem more competent if they use technical language. If you look closely, though, you will notice that the more senior people become, the simpler their choice of words. VPs and C-Level executives can explain complex topics in language that anyone can understand, regardless of their (technical) background. You can use tools like the Hemingway app to check if your writing is too complex.

Use signposting. Signposting is a technique that makes it easier for the reader to understand your document. The core idea is to use words and phrases that make it immediately clear what the sentence or section is about, so that readers can quickly skim the text and make sense of it. For example:

• Using the phrase “for example” before you give an example

• Writing “in conclusion” before you summarize

• Labeling sequences of arguments with “Firstly / secondly / finally”

Add visualizations. It’s a cliche for a reason: “A picture says more than a thousand words.” When you are trying to communicate dense technical content, a crisp diagram, framework or flowchart can help a lot to get your point across. For example, illustrating what “pyramid principle” means like the graph below will hopefully give you a better idea about how to carry it out in your own writing.

How can you improve your writing?

Read a lot. This includes both guides on how to write well (by reading this post, you’ve made the first step!) as well as strong technical writing that you can imitate (you can find some examples here).

If you want to dig deeper into the science of writing well, I recommend you take a look at “On Writing Well” by William Zinsser.

Practice, practice, practice. As with everything else, practice makes perfect. Here are a few concrete things you can do to practice your writing:

1. Document your work in a personal wiki. Few data scientists do this in my experience, but it’s a very useful resource to have and a great way to get more writing practice.

2. Write structured Slack messages. Most of the Slack messages we send and receive all day feel like a stream of consciousness (or worse, like teenagers’ text messages). People tend to type what comes to their mind and hit “Send” without taking the time to structure the message in a way that makes it easy for the reader to understand it. Writing succinct, structured Slack messages using the principles discussed above is a great way to stand out.

3. Write online. Writing these posts on Medium is ongoing writing practice for me. Try it out; you might even enjoy it and find an audience that enjoy your insights. 

   

Challenge yourself. “You are your own worst enemy” might not be a bad thing when it comes to writing. You need to be able to read your own writing like it’s your first time seeing it so you can be objective about what’s missing, what’s confusing and what needs to be shortened.

Ask others to be your devil’s advocate. Being your own devil’s advocate can be extremely hard sometimes, because true objectivity requires you to abandon your current knowledge about the topic and your ego. It’s sometimes just easier to find another challenger for your work. Ideally this is someone who truly knows nothing about the subject matter and is willing to be very honest with you about their opinion. 

What are some good examples of strong technical writing?

I described above what good writing looks like in theory but it’s easier to understand once you see a few examples of it. Here I’m providing some concrete examples for some of the points I mentioned above so you can have a better idea about how to put those suggestions into practice. 

Clear audience

The Data-Driven VC newsletter is targeted specifically towards Venture Capitalists and startup founders who want to take a data-driven approach to investing in and growing companies. While this creates a niche blog that might not appeal to everyone, picking this specific target audience makes it easier to provide value for them.

Strong visualizations

For a crash-course on how to visualize complex systems and technical subject matter in general, check out ByteByteGo. Their diagrams make it super easy to understand things that would take multiple paragraphs of jargon to describe accurately.

SeattleDataGuy is also using plenty of visualizations, but typically in a slightly less serious way (e.g. see his post on Apache Iceberg here).

Keeping it simple

Gergely Orosz, who writes The Pragmatic Engineer, does a good job summarizing complex topics in relatively simple terms. E.g. check out his post on how AI Software Engineering Agents work.

Combining best practices: Simple, succinct language with clear visualizations

Daily Dose of Data Science is a prime example for how to combine multiple best practices to produce easy-to-understand but still insightful data science content.

For example, check out their recent post on Confidence Intervals and Prediction Intervals. Or their super brief, but informative post on Cross-Validation Techniques.

In conclusion

Being able to write (well) is crucial for your work, even (or, you could argue, especially) for technical folks. Being able to succinctly communicate your thoughts on paper takes practice. Reading a lot, writing a lot and being open to feedback are the keys to getting better at this craft.

Unfortunately, in reality I have observed a lot of data scientists (especially junior ones) struggling with the story-telling part of the job. They have all the data in hand, but for some reason the message just doesn’t seem to get through to the audience and the analysis ends up being another pretty report on the bookshelf collecting dust with no impact. So what exactly went wrong?

Here are some tips that I have learned over the years that will help you improve your communication as a data scientist.

Always use the pyramid principle

A lot of data scientists communicate in a linear manner (what is demonstrated on the left in the graph below). This is understandable because that’s how we experience things, and also how we tell stories in our everyday lives. But it’s not the most effective way to get important information across because audience can easily get lost or bored in this type of communication.

The gold standard when it comes to communication in business is the pyramid principle — the one demonstrated in the graph on the right. To explain it simply, always start with the most important message — the insights; then drill down to the details.

Similar to headlines of newspaper articles, the insights will grab the audience attention. They are what you ultimately want to get across, so you need to put them front and center. If the audience is interested in the details, they will read (listen) on; if not, they got the most important message.

Clean up the formatting and grammar

This may seem like a nit pick, but let me tell you why it might be more important than you think.

Most of us know Albert Mehrabian’s “7–38–55” rule in verbal communication, which highlights that only 7% of the communication is about the actual content, the rest is composed of tonality (38%) and body language, facial expression etc. (55%). When it comes to written communication, it might not be as extreme, but you can bet that the visual presentation of your work has already communicated the quality of the work in silence.

When people open a doc with numbers and paragraphs that are in different font, not with the right indentation, full of underlines for misspelled words etc., they will likely start doubting the amount of diligence that went into the analysis, and by extension the quality of the conclusions as well.

It’s also disrespectful to the audience; messy formatting puts an additional mental burden on the reader that has to make sense of it all.

So what can you do about it?

The general rule of thumb is make the numbers as digestible as possible and get rid of all the useless details.

Let me use an example to demonstrate what I mean: Let’s say you want to convey insights about the composition of user base in terms of country.

Some data scientists would present something like this:

A couple of very low effort things you can do off the bat:

• Change the number format to separate groups of thousand (in Excel / Google Sheets, change to “Number” format and remove the decimal points)

• Rank in descending order so that the information is easier to digest

Detailed numbers are harder to grok. Depending on the audience, they likely don’t need that level of detail, so you can even simplify more and express the numbers in thousands instead:

Choose the right chart to get your point across

The type of chart you choose can carry or bury your insight; the chart should act as a visual aid to the audience. Being able to judge what type of chart/table can communicate insights more effectively is a key ability data scientists should possess.

In the example above, likely your ultimate goal is to convey the member base breakdown by country. A pie chart will provide more direct visuals than a table of raw numbers or even a column chart and makes it easier to tell the relative size of each bucket.

Looking at the chart below, even without any narrative, I can visually deduce insights such as “more than half of our member base resides in the USA” or “Germany, UK and Italy have similar amount of users”.

Focus on the important numbers instead of presenting ALL of the numbers

Communication is all about prioritization. You don’t have to include all the numbers just because you have them. You want to help your readers focus instead of distracting them with irrelevant information.

To continue our example, is there really a point of showing the numbers of the long tail? Most likely not. Note that the auto-generated pie chart above hid some of the countries for us already for that exact reason — they are too small to matter, so there’s no point cram them onto the chart.

So a better way to present the data is either to group the long tails together (with a footnote describing what is included in the grouping) or getting rid of it completely (if it’s another type of chart like a distribution and the long tail doesn’t offer any value).

Keep in mind that the ability to tell stories effectively with data doesn’t come naturally without practice. But it is one of the key skills that distinguishes a great DS from good ones.

The best way to practice is by doing. When there’s an opportunity to present, seize it, do dry runs with a teammate or your manager and get feedback before and after the presentation.

At the time, being one of a few data scientists in a sea of consulting generalists, a lot of this feedback initially felt like fuzzy consulting jargon to me and I would have rather had someone critique my code. Now being a manager myself and looking back, I realize that these seemingly disconnected points are all related to one thing — the mindset I was missing back when I was a junior IC focused on polishing my technical skillset.

I was purely focused on execution instead of acting like an owner of the problem; however, thinking that my job was only to perform a task well was a mistake in hindsight. Over the years since then, I’ve grown convinced that an ownership mentality is one of the key things that sets high performers apart from their peers.

If you are thinking to yourself “this sounds very vague and abstract”, you are not alone. From my observation, most ICs struggle a lot with the feedback “you should adopt more ownership mindset”; which is probably why most managers prefer to give more tactical feedback about the “symptoms” mentioned above and not the core issue itself.

To make it less abstract and more actionable, I will attempt to break down the “ownership mindset” (or lack thereof) into the three most common manifestations I have observed among junior DS and talk about mindset shifts that can help you act more like an owner.

From jumping into “How we should do this?” to starting with “Why are we doing this?”

In order to come up with the best “How”, you need to first understand the “Why”. So instead of directly jumping into the solution, you should dwell a little longer in the problem space and truly understand why this work matters and what decisions or business outcomes it is supposed to influence.

Remember, every analytics problem started as an abstract business problem. But very likely, as a junior IC, when a project is communicated to you from your manager or stakeholder, some thinking has already been done on your behalf (i.e. your manager already did some scoping based on his/her understanding of the business ask). I would encourage you to probe for the original motivation behind the scoped-out analysis task. Understanding this will help you to make a couple of crucial decisions — how to prioritize this piece of work against everything else on your plate, who your key partner is, and whether you can propose a more efficient way to solve the original question.

For example, very often data scientists get what I call “data Siri questions” like “how many active users do we have?”. Once you pull that stat, you often get follow up questions (e.g. geographic breakdown of the users etc.). Before you know it, what seemed to be a 5-minute ad hoc data pull turns into 100 seemingly disconnected data questions.

The most efficient way to deal with these situations is actually to start from the “why”. When you understand that the PMs are trying to launch a new feature on IOS and need data to decide which countries to prioritize based on engagement, then you will be able to make suggestions on exactly what data cuts will be most insightful to answer that question; or, if you dig even one level deeper, you might pose the question whether engagement is even the right metric to prioritize countries. As a result of this kind of approach, you will also be viewed more like a partner who should be involved in the original scoping of the project instead of someone who only helps to pull data.

From “What do I NEED to do?” to “What CAN I do to help?”

Most junior ICs are passively waiting to be assigned to a project with a clear a scope and outlined steps so they can focus on executing. It’s not surprising because most of us grow up having someone telling us what to do, usually parents and teachers in school. Even in college, there’s a syllabus outlining the exact things we need to do to get an A. The good news is that managers usually don’t expect junior ICs to proactively scout and scope projects independently. But if you want to stand out from your peers and grow to the next level, that’s something you should be doing.

Instead of passively waiting for an assignment and only doing what is required of you, go above and beyond by paying attention in cross functional meetings and conversations you are involved in. If you tune your senses to spot gaps and needs proactively, you will notice the lack of data is frequently the blocker of progress and decision making, and that’s something the data science team is in a unique position to help with.

Spotting gaps and figuring out a solution to plug the gap is one of the crucial skills you need to possess if you want to be a tech lead or manager one day. Building up this skill requires getting rid of the attitude of “this is not my problem” or “I have no dog in this fight” and instead acting as a true stakeholder in the company’s success. This may sometime require you to step out of your comfort zone and extend your current skillset, but that’s exactly how you grow your skillset proactively.

From “This is the number” to “This is the takeaway”

Many ICs struggle to distill succinct insights from analyses they do; raw data dumped in a spreadsheet or document is rarely a satisfactory deliverable. To quote from a leader I respect a lot:

Everything we do is for the benefit of others. If others are not understanding or seeing the benefit of the work we do, our work is incomplete. Hence, communicating our work is as important as writing code and building the models and products to get the job done.

A data scientist’s job is to “drive business decisions through analytical insights” not “to pull the numbers and leave it to others to distill the insights”. Not to mention, the “pulling numbers” piece is the easiest part for AI to replace.

Imagine you are the reader/consumer of the analysis instead of the one conducting it. Without any in-the-weeds knowledge about the analysis, can you quickly distill what it’s trying to do and what the takeaway is? Is it clear why engineers, PMs etc. should care about this work? If not, you have not succeeded at communicating insights through your analysis.

Conclusion

Remember you are the sole owner of your career and every project that powers it. So taking the initiative to make it more successful is your responsibility and yours alone. Adopting this mindset definitely takes time and practice; but once you master it, it will benefit not only your career, but also your personal life once you start applying the same mindset to everything you do.

What tech layoffs taught me in 2023

Just when we thought that life is going back to normal after the pandemic outbreak, a wave of tech layoff caught all of us tech workers by surprise. In 2023, more than 240K tech workers were laid off across more than 1,000 companies; the most recent layoff at Google and Discord indicates that it’s continuing in 2024. Both my current company and my pervious company had more than one round of layoff during the last year, with some of my friends impacted.

Even though this is the not the first time layoffs happened in the industry, the cuts were deeper than in the past. Past layoffs had often focused on functions such as Sales and Recruiting that were immediately affected by slowdowns in the business and hiring. This time, however, the impact was across the board including technical functions like software engineers and data scientists.

It’s human nature wanting to distill patterns and learnings from things, especially as a data scientist (after all, that’s one of the key aspects of your job). So like a good data scientist, I sat down at the beginning of the year to conduct a “post mortem”.

While it’s hard to predict beforehand which team/who will be impacted by a layoff (unless you are one of the decision makers), I think as a data scientist, there are things you can do to make yourself more “layoff-proof”. And I will share a few here:

Stay up to date about the company and org’s strategic direction

Why? When it comes to layoffs, the decision about which teams/individuals to let go is usually highly dependent on the company’s strategy and prioritization at the moment. For example, if a company for several quarters has been trying to make dashboard building and metric reporting more self-serviceable, when a round of layoff comes, any team that focuses on dashboard making and metric reporting will be considered.

Remember, things don’t happen overnight; there are signs that you can take as a heads up. Priority shifts usually can’t be implemented overnight or even in one quarter. In the example above, before letting go of the whole team of DS who are building dashboards/reporting metrics, the company needs to make sure the self-serviceable vision it has in mind is actually happening. So usually it will take a couple of months or quarters to build tools and set up workshops to democratize the work.

Hence it’s important to NOT just keep your head down and focus on things in your scope, but to understand what direction the company is heading in terms of its strategy and prioritization to deduce whether your work is, and will remain, important for the company. If the answer is no, you should really think long and hard about why you are still working on it and what you should be working on instead and have a discussion with your manager about it.

How? There are countless ways to stay close to strategic decisions and priorities. If you have a good manager, they should be sharing organizational context with you and the team; if they don’t do this, ask them proactively to share relevant strategic context. In situations where your manager is not capable of doing so (which is more common than you think), there are other options available; for example, All Hands meetings at all levels (company wide or org wide) are one of the best ways to get this information so you should definition try to attend those. Other ways (depending on your company’s culture) include asking in relevant Slack channels (e.g. leadership Q&A), talking to other managers in the org or your skip-level manager (use those skip-level 1v1s wisely instead of just treating them as a coffee chat!).

Make sure your scope is not too narrow and you work with different teams

Why? Continuing with the example above, if you are on the team that ONLY works on dashboard building and metric reporting, when those are democratized, your team’s value is hard to justify. But if the scope of the team is broader, it can adjust and shift its focus to one of the remaining areas it covers. At the very least, it will give the decision makers pause before completely cutting the team.

How? If you are in a position to influence the team’s scope and roadmap, make sure the team is not ONLY focusing on one thing, especially not on things that are easy to democratize or automate. Note that I’m not saying you should spread the team or yourself thin to work on EVERYTHING; focus is still important. But unless you only have one or two people on the team, the team should be able to take on a slightly broader scope and get more opportunities to pivot in dire situations. If you are not able to influence the team’s scope and it is already very narrow, you can branch out and establish relationships with other teams to get opportunities, either officially or unofficially, to work on things that might be slightly outside of your team’s immediate scope.

Consider staying close to the core business

Why? When it comes to a business, there is usually the core business (think about what a company is known for, e.g. Search for Google, Rides and Eats for Uber) and then there are the shiny new things or bets. Take Uber as an example; it started with Uber Rides then expanded into Uber Eats, which is a fairly direct extension of the core business and closely intertwined (e.g. shared supply as many Uber drivers are active on both platforms). Both businesses achieved relevant market share and became core revenue and profit drivers for the company

Uber also launched a number of new business lines that were much more experimental such as self-driving, freight, Uber Works (a staffing marketplace) and Uber Copter (helicopter rides). When those initiatives were announced, I have no doubt they were painted as top priorities and important bets for the company. However, compared to Uber Eats, which quickly became a core pillar, 1) none of these new bets contributed meaningfully to the top or bottom line within the first few years of their existence, and 2) they were much less intertwined with the core business and those easier to spin off or wind down. At the end of the day, businesses have stakeholders to answer to (e.g. Wall Street in the case of public companies), and when times get tough, non-core activities are the first ones to be scaled back; so it’s not surprising that over time, Uber ultimately shut down or scaled back most of these bets, while Rides and Eats jobs were less affected besides a temporary COVID cut-back.

How? This one is a personal choice; of course it’s exciting to work on new initiatives and the sexy new things, and these can sometimes offer career-defining growth opportunities, but if you want to increase your job security, it can be beneficial to stay close to the cash cow of your company.

Make sure you advertise for your own work

Why? In most layoff situations, the decision and information is kept confidential and only a few key individuals are in the loop. So maybe your manager is blindsided, maybe your skip-level manager as well. But most likely SOMEONE in the command chain IS helping with the decision making. Imagine you are that someone and you look at a team in your org and find yourself asking, “hmm.. what does this person do again exactly?”; I bet you will put them on the top of the cut list.

How? I know this doesn’t come naturally to most people, especially people in more technical roles. But I would really suggest that you take any opportunity you get to showcase your work to leadership. Every organization provides different opportunities of doing so — special topics during All Hands, deep dive on certain topics etc. Seize those opportunities when you can to make people remember your work and how your work is contributing to the company’s top priorities.

But remember, while everything I mentioned above aims to reduce the chance of you getting impacted by a layoff, NOTHING can guarantee that you won’t be impacted. Even if you prepare as much as possible in advance and tried everything to de-risk yourself, life happens and you might find yourself out of a job. So I have one last piece of advice I would like to give:

Never turn down an opportunity

Why? You may think you love your job, you love the company and there’s no chance you will look for a new job. But sometimes it’s not 100% within your control; when a layoff hits out of the blue, you might look back and wish you had entertained the recruiter outreach a while back.

Plus, even though you don’t want to always go through the whole painful recruiting process, it never hurts to at least take the intro calls to understand what opportunities are out there and how much your current skillset is valued in the market.

How? Pretty simple, when recruiters reach out on LinkedIn and the opportunity looks interesting, just reply to them, take an intro call and see what they say. It also doesn’t hurt to keep in touch with folks in your network so you hear about new opportunities as they become available.

I hope these lessons I learned from the last few years of layoffs can help you in some way. If you are interested in learning more about data science related topics, check out some of my other articles.

Soft Skills Is What Sets You Apart in Your Data Science Interviews
How to up-level your structured problem solving skills and communication skillstowardsdatascience.com

5 Mistakes I Wish I Had Avoided in My Data Science Career
I learned these lessons the hard way so you don’t have totowardsdatascience.com

5 Lessons McKinsey Taught Me That Will Make You a Better Data Scientist
towardsdatascience.com

How to up-level your structured problem solving skills and communication skills

So you have brushed up on ML concepts, practiced Python and SQL for months, you think you are done with interview prep. But you might be missing the most important and hardest-to-prepare-for part of the interview — problem-solving skills. And based on my experience of interviewing others as well as going through interviews myself, I can confidently tell you, this part is more than often what makes or breaks your interviews.

This is the fifth article for the data science interview guide, in previous articles, I have mostly touched on technical concepts that commonly show up in interviews. For reference, the previous articles are listed below:

1. Part I. Distribution

2. Part II. Probability

3. Part III. Basic Supervised Learning Models

4. Part IV. Random Forest

But contrary to most people, I don’t believe technical portion of the interview is the most difficult part; I think the soft skills are the hardest to grasp, learn or teach when it comes to interview prep. As a manager, that’s the most important thing I look for when I’m hiring for my team. Because let’s face it, technical skills are easy to brush up on and learn (not only for humans, but also for machines — ChatGPT can implement code pretty well these days), but the ability to understand, solve business problems and communicate effectively is not something you can build up in a day or two.

This portion of the interview is also the most unpredictable as there’s less of a template interviewers follow; instead, a lot of times the interview would go in the direction you take it. Before you get discouraged, I have summarized a couple of ways the soft skills are tested in interviews and tips that can help you prepare for this part of the interview.

There are in general two modules of the interview that fit into the soft-skill category — case study (which is to test your ability to carry out structured problem solving) and behavioral interview (which tests your communication and again, structured thinking ability).

Ability to translate business problems into data science problems — case study

This portion of the interview is usually a case that’s very similar to the consulting case interview, with a slight data science flavor added on top. It’s usually a RCA (root cause analysis) type of question for a metric movement or business decision. Examples are questions like “our daily active user decreased by 10% in the past week, how to debug what the cause is?” OR “we want to put electric chargers for our fleet around SF, how should we make the decision about where to put them?”

What is it testing? These questions are testing your ability to come up with a problem-solving framework and your ability to explain the framework. It’s extremely important to have one in your day-to-day work as a data scientist so you can make sure that your solution is thorough, comprehensive and can easily make sense to others.

What do interviewers want to see? Keep in mind there’s no right or wrong answer for this kind of questions, only organized or unorganized ones. The interviewers really want to see how you organize your thoughts and walk people through the problem-solving process. So make sure you verbalize all the assumptions you are making, all the details you are exploring and ask all the clarifying questions you might need to ask.

It’s also important to show your ability to take feedback and work with others. Sometimes your interviewer will give you some guidance or hint about where they want you to take the case

How to prepare? The best way to prepare for a case interview is practice, practice and practice. I suggest getting yourself one or two books related to consulting case interviews or PM case interviews, books like cracking the PM interview and case interview secrets come to mind as recommendations. But just reading through the examples the trying to force memorize the frameworks in the books won’t help as much. The most effective thing to do in my opinion is to take sample cases in the book and mock the case with a friend. Very likely your answer will not be identical as the “sample solution”, but ask your friend to check “is my thinking process easy to follow?”, “was it clear how to arrived at each step in the process?”.

Cultural fit — communication skills and working style

What is it testing? Interviewers want to see that you have the qualities that the team is looking for aside from hard skills. And if the interviewer is a hiring manager, they would want to see that they can effectively communicate and work with you.

What do interviewers want to see? Depending on the team, interviewers/hiring managers may care about/want to test different things. But in most cases, I have seen interview questions aiming to prob about your ability to learn, to adapt to changing scope, to collaborate with others and to manage stakeholders.

How to prepare? A couple things can be helpful for this portion of the interview. 1) Make sure you are super familiar with any project you have on your resume, because the interviewer will likely ask about details of different projects on your resume to get a sense of your working style. And 2) make sure you practice the top-down communication style and follow the STAR framework when talking about your projects.

Finally, always remember, interviews are a two-way assessment; you are gauging the company’s fit for you as much as they you, and the behavioral portion of the interview is the best way to test whether YOU like THEM. The people who interview you will likely be the group of people you work really closely with; so if you feel like the “vibe is off” (which trust me, happens more than you would expect), maybe it’s a good signal you should take into the consideration when deciding between offers.

Preparing for a data science interview and want to read more about interview preparation? Here are some articles you might enjoy!

Acing the ML Portion of McKinsey Data Science Interview
A detailed guide for the what, why, and how of the ML part of consulting interviewstowardsdatascience.com

Productivity Tips for Data Scientists
How to work better, smarter but not necessarily harder as a data scientisttowardsdatascience.com

A detailed guide for the what, why, and how of the ML part of consulting interviews

After I wrote about my decision to join McKinsey as well as the most valuable lessons I learned there about data science, I have also shared the reasons for my decision to eventually leave the firm, I have gotten a lot of messages from readers of those articles asking about HOW to prepare for data science interviews for consulting firms. So I thought I would address all of these questions with several articles talking about my consulting interview process and my preparation for it.

My interview with McKinsey was 3 years ago. As the practice evolve, some of the interview processes might have changed a little bit. But the general idea should remain the same and the preparation process shouldn’t differ that much.

So let’s talk about what consulting firms usually look for in candidates and how to prepare for the interviews. There are different sections of the interview — general topics are ML knowledge, case study and cultural fit. I will address them in separate articles to take care of everyone’s attention span. You might be wondering how come SQL and Python (which are commonly tested by other companies for DS interviews) are no on the list. I still strongly recommend knowing at least the basics about SQL and Python before your interview in case the interview format has changed since mine. But McK generally believes that the super technical skills (like coding) can be learned on the job as long as you have some coding experience. What they rather spend time testing are your “soft skills” like ability to learn, structured problem solving and the analytical way of thinking.

Why It Is Tested

In the tech world, usually Machine Learning Engineers (MLE) are the ones that build models and data scientist work mostly on analyses and insight generation. But as a DS consultant, you are viewed as a “full stack” DS, meaning you need to be able to cover things from data pipelining to ML modeling, all the way to insights generation and “storytelling” with data.

So consulting firms want to make sure that you have enough ML knowledge to work on or even lead modeling projects.

How It Is Tested

Like I mentioned in my interview series about ML, there are generally two ways to test ML knowledge I haven seen — resume based or theory based.

Most consulting interviews are resume based from what I have heard (and experienced). That means two things — you need to have some modeling experience and you need to know how to talk about it.

How to Prepare for It

In order to be able to talk about modeling experience, you need to, well, have some modeling experience. You can get modeling experience through your current work by getting on modeling-intensive projects; if that’s not an option, you can always utilize websites like Kaggle to get some modeling experience through side projects.

When it come to learning the basic theoretical ML knowledge. The best book to use in my opinion is The Analytics Edge written by MIT operations research professor Dimitris Bertsimas.

It covers things from basic concepts like linear regression, CART model all the way to more complicated models like the random forest. It doesn’t cover the super advanced models like neural networks, reinforcement learning etc. But based on my experience, I rarely used those models in my consulting life, so I will be surprised if consulting interviews put a lot of weight on those.

If I were to re-read this book from scratch for an interview prep, I would start from Chapter VIII, where each model is dissected and explained in details, before moving onto some of the previous chapters which demonstrate case studies of the models.

If you lack model-building experience outside of academia and Kaggle environment, you might need to also get some knowledge about the operation side of things of ML. I personally found Educative.io’s Grokking ML Interview course very helpful in that regard. It gets into the details about operational aspects of ML like how to choose metrics and online/offline model evaluation.

If you are relatively familiar with machine learning foundations and just need a brush up on it before the interview, I have used Springboard’s 51 Essential Machine Learning Interview Questions and Answers as a quick refresher.

Final Advice

Like I mentioned in my previous articles, one thing consulting firms REALLY care about is your ability to communicate, especially when it comes to complicated analytical concepts. Data scientist in consulting firms more than often need to work with clients who are NOT from an analytics background; so it’s essential that you showcase your ability to do just that. When explaining ML concepts and models, try to use as much plain English as possible instead of obscure jargons and make sure you can really explain them without getting into the details of the calculation; because you will encounter clients who ask you to “explain how clustering works without the math”.

Want to Read More About DS Interviews? Here Are Some Recommendations:

Concepts You Have to Know for Data Science Interviews — Part I: Distribution
Most frequently asked questions in data scientist interviewstowardsdatascience.com

Concepts You Have to Know for Data Science Interviews — Part II. Probability
Most frequently asked questions in data scientist interviewstowardsdatascience.com

Concepts You Have to Know for Data Science Interviews — Part III. Basic Supervised Learning Models
Most frequently asked questions in data scientist interviews for modelingtowardsdatascience.com

Concepts You Have to Know for Data Science Interviews — Part IV. Random Forest
Most frequently asked questions in data scientist interviewstowardsdatascience.com

Most frequently asked questions in data scientist interviews

This is the 4th article in the interview series. I’m hoping this series will function as a centralized starting point for aspiring data scientists in terms of interview preparation. So far, we have talked about the following concepts:

1. Part I: Distribution

2. Part II. Probability

3. Part III. Basic Supervised Learning Models

In this article, I want to continue the journey down the ML lane and talk about advanced supervised learning models. More specifically, I will be focusing on Random Forest since it is probably the most commonly used and commonly asked in DS interviews among the more complicated/advanced ML models. In fact, I was asked about Random Forest in my interview with McKinsey as a data scientist; knowing how to explain the algorithm on the high level and intuitively will definitely help you stand out from the rest of the interviewees if there’s a modeling component in your interviews.

What is Random Forest

We talked about CART model in the previous article. As a reminder, CART stands for Classification and Regression Tree; and Random Forest, well, is a forest. The descriptive naming convention reveals the clear relationship between the two — Random Forest consists multiple decision trees.

Random forest is an ensemble method (ensemble methods is a category of ML methods that combines multiple learning algorithms to obtain better results, achieving that 1+1>2 effect), utilizing multiple trees to avoid overfitting (decision trees are prone to overfit). Imagine each tree in the forest casting a vote and having a say in the final decision of the whole model; the final decision/prediction is achieved through the forest taking the majority vote.

In order to avoid overfitting, the trees shouldn’t be correlated. How does the model avoid trees being correlated? You might ask. Random Forest makes sure of that by doing two things — when building each tree, randomly selecting a subset of training samples, and randomly selecting a subset of features. This “random subsetting” is often used in ensemble models and is commonly referred as “bagging” and is often used to reduce variance in trained models.

A little digression here since we are talking about “bagging”. “Boosting” is another method commonly used in ML models; in fact, Gradient Boosted Decision Trees is a high-performing cousin in the decision-tree family. I’m mentioning this because “Bagging” and “boosting” are often compared; I was asked in different interviews about the differences between them. What’s special about “boosting” is it can improve weak learners. Different from “bagging”, which builds trees in parallel and separately, the “boosting” process builds trees sequentially; so each tree can “learn” from the previous one’s mistakes and improve.

There are a lot of detailed explanations about “boosting” on Medium so I won’t go into any technicalities here. But it’s worth noting that due to this “sequential” nature, “boosting” algorithms train more slowly and are more prone to overfitting comparing to their “bagging” counterparts.

Back to Random Forest. If you remember what we talked about in the previous article, CART’s biggest advantage is arguably its interpretability. Even though random forest is usually an improvement from CART in terms of performance (especially on test sets since it is less prone to overfit), it sacrifices interpretability to some extend in the process. As the number of trees grows, it will be harder and harder to plot each tree and see the features they used in splitting the data; thus it becomes harder to understand exactly how each tree was built. But it’s still possible to generate a plot for feature importance (across the trees in the forest). Most random forest packages come with such a plot that’s easy to access.

How are these tested and what to watch out for

I have already went through in the previous post about how ML concepts are usually tested and the most important things to remember when you answer ML modeling questions. Click to that post (linked below) if you want to read more.

Concepts You Have to Know for Data Science Interviews — Part III. Basic Supervised Learning Models
Most frequently asked questions in data scientist interviews for modelingtowardsdatascience.com

The only thing to add about random forest, or any complicated ML algorithms in general, is it’s important to be able to explain the algorithms in layman terms. What I mean by that is the interviewers usually are interested in testing your understanding of the algorithm, but are not looking for a memorized version of the wikipedia page of it.

Little tip at the end, the best audience for practicing your intuitive explanation of ML algorithms is friends who are not in the analytics field; you can quickly tell if your description of the algorithm is making any sense.

Interested in reading more about data science career tips? I might have something for you:

5 Lessons McKinsey Taught Me That Will Make You a Better Data Scientist
towardsdatascience.com

Why I Left McKinsey as a Data Scientist
Things you should consider before starting as a data science consultanttowardsdatascience.com

The Ultimate Interview Prep Guide for Data Scientists and Data Analysts
What helped me interview successfully with FANG as well as unicornstowardsdatascience.com

Most frequently asked questions in data scientist interviews for modeling

This is the 3rd article in my interview series. I’m hoping this series will function as a centralized starting point for aspiring data scientists in terms of interview preparation. The first two articles are here if you are interested:

1. Concepts You Have to Know for Data Science Interviews — Part I: Distribution

2. Concepts You Have to Know for Data Science Interviews — Part II. Probability

In this article, I want to talk about arguably the most interesting part for a lot of aspiring data scientists — ML. ML is a super complicated topic, so I won’t even attempt to cover EVERYTHING about it in one post. In fact, I will separate the ML part into several posts and talk about different aspects of ML and different categories of models in each of them. In this post, I will focus only on the basic supervised learning models.

There are countless types of models out there when it comes to ML, from simple models like linear and logistic regression all the way to convoluted ones like deep learning and reinforcement learning. I will post by post cover the important ones that will show up in interviews.

First things first, it’s important to know that unless you are targeting super modeling-intensive roles such as machine learning engineer (MLE) or research scientists, ML usually is NOT the biggest focus for interviews. The interviewers usually ONLY want to see that you have a basic level of understanding of different models and modeling techniques; you WON’T be asked to whiteboard proofs or defend the choice for the number of layers in a neural network you built.

In general, ML models/techniques can be separated into two categories— Supervised vs. unsupervised learning.

The difference between supervised and unsupervised learning is one of the most fundamental concepts data scientists should be familiar with. Supervised learning is the general category of all the ML techniques that utilizes labeled datasets, whereas unsupervised learning techniques work with unlabeled datasets. A concrete example: If you want to build a model to distinguish dogs from cats, you would be using supervised learning because there should be a clearly correct answer to each data point whether it is a dog or a cat (the categories you want to predict). Whereas if you just want to cluster house pets together, you would use unsupervised learning. There’s not a clearly “correct” answer for each data point’s label — a small dog may be grouped into the same group as other small cats instead of big dogs. This article will focus on most commonly tested supervised learning techniques; I will go into more advanced supervised learning models and unsupervised learning in the following articles.

Linear regression

This is arguably the most rudimentary model used in ML (if you think it “counts” as ML). Linear regression is the most common method of supervised learning that’s most suitable to predict a continuous variable (as opposed to categorical variables).

Linear regression essentially is trying to fit the best line through all training data points. The definition of “the best line” varies depending on the choice of the loss function. The most commonly used one is “ordinary least squares (OLS)”, which finds the best line by minimizing the sum of squared errors. The error is defined as rᵢ = yᵢ -y^, where y^ is the predicted value by the model and yᵢ is the observed (true) value. The value you want to minimize is the sum of the aforementioned error terms

Logistic regression

The biggest difference between logistic regression and linear regression is that while linear regression models a continuous variable, logistic regression models the probability of an event (an event with binary outcomes), hence the output is between 0 and 1.

It’s worth noting that, even though commonly used for classification, the logistic regression itself is NOT a classifier. But it can be turned into a classifier when layered with a threshold on top (usually 0.5). With a cutoff determined, the probability can be turned into a binary output. As an example, if you are trying to predict whether an e-mail is spam or not, a logistic regression with a threshold of 0.5 would classify any e-mail with a predicted “probability of being spam” ≥ 50% as spam.

Understanding that the classifier consists of two parts — a threshold and the logistic regression — can help you understand why choosing a proper cutoff is an essential but usually overlooked part of model tuning when it comes to classifiers powered by logistic regression.

CART

CART is short for categorization and regression trees; as the name suggest, it is a supervised learning technique that can be used to predict categorical or continuous variables. CART is usually talked about together with Random Forest, because CART is the simpler version; it’s a single tree instead of, well, a forest (I will cover Random Forest in the next article).

CART’s biggest advantage when compared to regressions, or even random forest, is its interpretability, because it’s possible to plot the feature split used to build the tree. The illustration below showcases a super simple CART model. Illustrations like this can be easily plotted by most CART packages you use to fit the model and it can help you visualize how the model is built in the background. Another neat thing is that it shows you what features were used first to split the model; so it also illustrates feature importance.

How are these tested and what to watch out for

In general, there are two approaches to test these fundamental ML modeling concepts — resume-driven or theory-driven.

• Resume-driven: the interview questions will be based on your resume. So make sure you take a couple of hours to take a trip down memory lane and refresh your understanding of the modeling experiences you mention in your resume. Depending on the interviewer and his/her background in machine learning, they might ask in-depth questions about certain algorithms you mention.

• Theory-driven: If you lack modeling experience, or modeling shows up in a case study portion of the interview, the questions will be theoretical and hypothetical. The interviewers will ask you what you WOULD DO in a certain situation. And they might throw you a curve ball by planting some problems with the dataset and see if you know how to deal with difficult data problems in modeling (we will cover that in detail in my upcoming article talking about model training).

• Things to remember: When talking about your modeling experience, make sure you showcase that you understand interpretability is the most important thing when building a model. There are countless ways to visualize model performance and feature importance for almost every commonly-used model. Stay tuned for an article on that too. Another thing to remember is that data cleaning and feature engineering are as important, if not more so than the modeling itself. Otherwise your model will suffer from the infamous “garbage in garbage out” symptom.

Interested in reading more about data science career tips? I might have something for you:

5 Lessons McKinsey Taught Me That Will Make You a Better Data Scientist
towardsdatascience.com

Why I Left McKinsey as a Data Scientist
Things you should consider before starting as a data science consultanttowardsdatascience.com

And maybe you can do the same

Let me explain the subtitle first. This is not one of those feel-good posts where I PROMISE new writers that the success can be replicated; I personally hate those because they ignore the survivorship bias and the luck factor in success and suffer from the “fair world” fallacy. But before you lose hope and click away, while the tips here CANNOT guarantee views and income on your article, they DO increase your CHANCE of being successful on the platform.

So, I have been writing on Medium since the beginning of 2021. My very first article was born out of the frustration that there was no good tutorial online about how to make an API call to get data, even though I have personally run into this problem multiple times as a data scientist. So I decided to take the issue into my own hands and wrote an article documenting my learnings, and the rest is history. My best performing article to this day is my article about the data science lessons McKinsey has taught me during my time working there. It made me over $1000 since I published it in May last year. In fact, it generated over $1000 in the first 2 months after it was published.

So what have I learned from writing in the past year with several “viral” articles?

I. Write consistently

I know this one is a cliché, but I still want to repeat it because it’s definitely true. The internet is a weird place, you can never know for certain which article will be popular as an author. I have had articles that I was very excited about sink like a stone with absolutely no ripple; and vice versa, some articles that I thought would only speak to a small audience found broad success. So the best way to write a viral article is to write a lot of good articles and see which one will get the attention it deserves. It’s almost impossible for your very first article to go viral; it usually takes a bit of work to accumulate readership so your article gets the baseline level of views as a start and has a fair chance at getting more attention from there. Looking at stats of the articles right before the viral article, there’s undeniably a build up and it takes time and consistency.

But talk is cheap; the harder problem is HOW to actually write consistently.

With English as my second language, I have always found writing hard. There’s a mental hurdle for me when it comes to writing because we are naturally scared of things we are not good at or familiar with. What I have found helpful is to open my laptop everyday to write SOMETHING, ANYTHING to get over that mental hurdle. If you have great ideas that day, write a paragraph or two, or even a whole article; if your writer’s juice is not flowing that day, write once sentence or two, or even just outline some ideas. ANYTHING counts.

Just remember, you don’t need to finish each article in a single setting, so write down any piecemeal inspiration you have WHEN you have them. Getting started is the hardest part; building the habit of writing into your daily routine is the key to having successful article(s).

II. Get your article the initial exposure

I publish with several different publications and occasionally self-publish as well. Depending on which editor/publication picks up the article, it might get very different treatment in the publishing process and trust me, it makes a noticeable difference.

I have noticed that if the article gets an initial boost from the editor’s applause or immediate posting across social media, it gets more attention faster than if the social media post only follows after a week of the article’s posting date, or if there’s no social media promotion at all. It makes sense, that snow ball effect is real. Having built several machine learning models myself, I wouldn’t be surprised if Medium’s decision algorithm for whether to promote an article places a high weight on the article’s initial traction (views and reactions like applause or comment). The more exposure your article gets at the beginning (getting posted on not only Medium, but also other platforms like LinkedIn), the more traffic it will receive, and the more signal it will send to Medium’s algorithm: “Hey, push me to more people on the platform because I’m a good article and people will read me”.

Now you understand the why, let’s talk about the how. If you publish with publications, try to work with the editors who help you publish and remind them to promote your article through their social media platforms if there are any. Also try to promote the article yourself on different platforms like LinkedIn, Facebook, Twitter etc.

III. Find your niche and your publication(s)

By finding your niche, I don’t just mean finding the general area you want to write about; of course, that’s the first step. I mean within that general area, find the gap that nobody has written a lot about that fits your expertise and/or interest.

In my case, I knew data science is the general area I want to write about because of my interest and background. There is a fair amount of authors writing about ML, AI and the technical side of data science, but I realized that career development and interview guides in data science are lesser touched areas. Since I have been through several rounds of recruiting in data science myself that happens to be the area where I have a good amount of experience to share and help my readers. As you write more articles in the niche you find, you will continue attracting readers who are interested in the topic and gain followers that way. Which means your future articles in the same area will get a certain baseline level of reads from your followers.

Equally importantly, finding one or more publications that you frequently work with will help you leverage their existing reader bases (usually a lot bigger than your own). I published most of my articles through Towards Data Science since it fits my focus area, and was able to get good traction on my initial articles despite having virtually no followers at the time. Going back to the point mentioned above, having publication(s) you are familiar with will also help you establish and maintain a relationship with editors that can help you promote your articles.

IV. Name dropping still works, for better or for worse

Why do you think so many YouTubers and Medium writers use prestigious companies’ names in their titles (Google, Amazon, Uber, McKinsey, you name it)? Because it gets views. Not only does the name dropping get people’s attention right away, it helps the articles get picked up by search engines and other curations mechanisms. Believe or not, Google will even curate and push it to potentially interested parties if the articles gets enough attention. I know this because one McKinsey alumni I have no connection with read the article in question from his Google Chrome recommendation section and reached out to me (it’s probably because he indicated his interest in McKinsey related topics somehow).

So far, I have written 3 McKinsey-related articles and on average, they outperformed my other articles. So if you have any ideas in the pipeline that are related to your experience with big companies, don’t be shy to use the name in the title. With that being said, don’t be one of those annoying people who use big names as click baits when the content has nothing to do with those companies. You don’t want to throw away your integrity and credibility just to get a couple more clicks.

I hope these tips are helpful to new writers out there and don’t be shy to reach out and connect or

If you are interested reading the articles that performed well for me, here are some:

5 Lessons McKinsey Taught Me That Will Make You a Better Data Scientist
towardsdatascience.com

The Ultimate Interview Prep Guide for Data Scientists and Data Analysts
What helped me interview successfully with FANG as well as unicornstowardsdatascience.com

Why I Left McKinsey as a Data Scientist
Things you should consider before starting as a data science consultanttowardsdatascience.com

So you want to be promoted? So does everyone else. How can you stand out from your peers?

On top of my passion for data science, I have always been interested in people management. I have always wanted to help others achieve their career goals (hence the Medium account), and I love to get the big picture and learn about/help decide how a team fits into the company’s broader ecosystem.

When I started my new job a year and half ago, I was hoping for a promotion to manager soon since I had a couple of years of individual contributor (IC) experience under my belt and I thought I would be able to find some general guidance online about how to make this transition possible. But interestingly enough, while the internet is filled with interview guides and general career advice, there’s not much written about how to make the IC → manager transition. Everyone’s path to manager will be slightly different since there are a lot of variables in play here. But thanks to my manager’s coaching and guidance, I was able to distill some general lessons from my experience and hopefully will be able to help out some aspiring DS managers.

Take initiative

One of the most valuable lessons I have learned in this process is — you don’t get a promotion and THEN start to perform at the next level; you perform at the next level IN ORDER TO get a promotion.

So when you notice a gap somewhere, even if it doesn’t necessarily fall into your current role description, don’t be afraid to bring it up to your manager and discuss whether you can/should take initiative to help plug the gap. You manager will for sure appreciate ICs who can notice those things and can help brainstorm solutions.

The best way to notice gaps and in turn take initiative is to be a good listener and constantly communicate with your partners & stakeholders about their teams’ work and pain points. Generally, the more you know about other teams’ work and how they collaborate with each other, the easier it is for you to identify ways to improve things.

Mentor a peer

One of the biggest responsibilities of a manager is to help other team members in terms of prioritization, guidance and general mentorship. So you should start practicing these when you are thinking about making the transition.

If you have new members joining the team, great, offer to be an onboarding buddy to guide them through their first few weeks. See if you are able to provide general context about stakeholders, tech stack and projects of the team and help with more detailed questions. This will also help you get a better understanding of your knowledge gaps and help you improve as an IC.

If there are no new members joining the team in the near term, you can still help out current peers on the team when they need someone to brainstorm a solution or when they don’t know which team to reach out to about a question.

Step out of your immediate scope

As a manager, you will need to elevate from a single IC’s scope and instead be in charge of several IC’s work streams on a higher level. Take this potential transition period as your risk-free practice.

If you have bandwidth outside of your day-to-day work, talk to your teammates to learn more about their projects. Occasionally you will find opportunities for collaboration between team members to improve team efficiency; these would be great suggestions to bring to your manager.

The same thing is also possible on an inter-team level. If you are able to pay more attention to conversations that are seemingly unrelated to your immediate scope, you will be able to quickly make connections between different projects and initiatives and help improve the efficiency of collaboration across teams.

Get involved in team-level activities

Managers always appreciate a helping hand in team-level activities such as sprint planning, quarterly planning or even just timely suggestions. This will be a great opportunity to test your knowledge about other team members’ work and other teams’ requests for your team. It will also give you some exposure to the manager’s plan and vision for the team. Similarly, volunteering for culture initiatives is a great way to practice thinking about the team as a whole and start building a muscle that you will need to flex as a manager.

I have seen a lot of ICs with great ideas that they implement in their own work; whether it’s having a product design doc for the data product they develop, or having an SLA agreement with their partner teams. What would be even better is to codify the best practices you use in your own work so the whole team could benefit from it. This is in the same spirit as “step out of your own scope”; adopt a manager mindset and help out the whole team.

Have open, timely feedback conversations with your manager

As a data scientist, you should know the importance of metrics to track your progress. So how do you establish a metric for your transition to a manager? Having open and timely conversations with your manager is key here.

Every company has slightly different leveling guides for different roles. So it’s very important to understand the criteria you need to meet as a manager. Ask for the leveling guide when you have the initial career development conversation with your manager. And make sure you mention your aspiration to be a manager as soon as possible (don’t be shy) as well as your aspired timeline that you are working towards. At the same time, ask your manager for candid feedback with regards to their assessment of your readiness to become a manager, and any gaps that they think you need to address. Similar to the “build in public” mentality, don’t be afraid to put yourself out there for those type of things; even if you fail, it’s a great learning experience.

In followup career development check-ins, ask your manager to provide feedback for you against the leveling guide. This will provide clarity for both you and your manager in terms of your progress toward the goal.

Summary

Not sure if you have noticed, but communication is key in a lot of the things mentioned above. Communicate openly and often with your teammates, your stakeholders and your manager. It’s also important to get your manager’s support in this whole process. I wouldn’t have been able to do any of these without my awesome manager Dennis’ help and mentorship. Remember, your manager will be your biggest advocate in all of those promo conversations, so it’s important to bring them on the journey and get their honest feedback along the way.

Besides that, remember to step out of your immediate scope to so you can get involved with more team-level work, mentor a peer, or just take more initiative in general.

Want to read more about data science career? Here are some great articles for you.

Productivity Tips for Data Scientists
How to work better, smarter but not necessarily harder as a data scientisttowardsdatascience.com

5 Mistakes I Wish I Had Avoided in My Data Science Career
I learned these lessons the hard way so you don’t have totowardsdatascience.com

How To Pick The Right Career In The Data World
Data Scientist, Data Analyst, or Data Engineer? How do you know which one is right for you?towardsdatascience.com

Most frequently asked questions in data scientist interviews

In the last article in the series (Concepts You Have to Know for Data Science Interviews — Part I: Distribution), I touched on the basics of distributions — the most important distributions and their characteristics that might show up in data science interviews. In this article, I want to continue the tutorial with common probability questions that companies like to ask DS candidates.

Probability is a complicated subject that might be hard to master in a short period of time if you really want to understand it in depth. In fact, it is a required course that spans a whole semester for math majors in college. So this article will definitely NOT make you an expert in probability, but instead will give you an idea of the most commonly tested areas within this topic.

Conditional Probability/ Bayes’ Theorem

Conditional probability is THE most tested type of probability problems during DS interviews. It’s a helpful skill because in our day-to-day job, a lot of analytics questions we run into involve conditional probability — for example, what’s the probability of having a disease given the test of the disease is positive. Conditional probability is usually calculated with Bayes’ Theorem, the formula is shown below:

A,B → Events

P(A|B) → probability of A given B is true

P(B|A) → probability of B given A is true

P(A), P(B) → the independent probabilities of A and B

Independence is an important concept when learning about conditional probability. It’s worth noting that if P(A|B) = P(A), meaning “the probability of A given B” is the same as “probability of A”, then the two events A and B are independent.

A good example of independent events is the result of each coin flip; for a fair coin:

A: flip a head in the 2nd flip → P(A) = 1/2

B: flip a tail in the 1st flip → P(B) = 1/2

A|B: flipping a tail in the 2nd flip given flipped a head in the 1st flip → P(A|B) = 1/2

Knowing that you flipped a head in the 1st flip does NOT change the probability of flipping a tail next, so those two events are independent.

In interviews, a strong hint that you should be considering using Bayes’ Theorem is the key phrases like “given that” or “conditioning on”.

Total Probability

The law of total probability is stated as below:

“Bx” and “By” are disjoint events that in combination represent the whole selection universe. Suppose a group consist of people from either SF or Seattle. We know that 40% of the group are from SF (P(Bx)) and 60% are from Seattle (P(By)); the probability of Seattle being rainy today is 70% (P(A|By)) and SF being rainy today is 20% (P(A|Bx)). What is the probability that we randomly call a person within the group and their city is rainy today (P(A))?

Using the formula above, we can easily get P(A) = 0.4*0.2+0.6*0.7 = 0.5

The total probability is usually used in calculation for conditional probability because for the conditional probability shown below, often P(B) is not directly given but has to be calculated using the law of total probability.

Binomial Probability

We have talked about the binomial distribution in my previous article about distributions. Binomial probability is using the binomial distribution to calculate the probability of having exactly n success among Y trials of an experiment that only has two outcomes (a good example is flipping a coin).

k: number of trials with success

n: number of total trials

p: probability of success in a single trial

n!: n factorial, calculation for this is n*(n-1)*(n-2)… *3*2*1

k!: k factorial, calculation similar to above

(n-k)!: you get the point

Recognizing that the question involves an event with only two outcomes is the key to deciding when to use the binomial formula to calculate probabilities.

One good example of this is: the probability of getting 4 heads out of 10 coin flips (assuming it’s an unfair coin, and for every flip, there’s a 1/4 probability of getting a head and 3/4 probability of getting a tail).

In this example:

n = 10, k=4, p=1/4

The calculation for this would be 10!/(4!*6!)*(1/4)⁴*(3/4)⁶ = 210*1/4⁴*(3/4)⁶= 0.15

How are these tested and how to prepare for them

These are tested in a lot of different ways. But the general theme is, instead of being in isolation, they are usually tested in combination. Like mentioned above, the Law of Total Probability can be easily incorporated into the Bayes’ Theorem when asked in interviews.

It’s important to note that just memorizing the formulas for these will not be enough for interviews. Because in interviews, nobody will tell you when to use which formula. Determining which is the correct one to use is arguably the hardest part. The best way to master it is looking at a large number of sample questions and try NOT to look at the solution but try to solve them yourself instead. Getting familiar with the type of questions that require the use of these formulas is key. For sample questions, a simple Google search will give you plenty of results for each. If you want to have a more centralized “database” for probability questions, I highly recommend the Ace the Data Science Interview book by Nick Singh and Kevin Huo.

Looking for more interview guide? Here are some articles that might be helpful!

Concepts You Have to Know for Data Science Interviews — Part I: Distribution
Most frequently asked questions in data scientist interviewstowardsdatascience.com

Take-Home Exercises can Make or Break Your DS Interviews
How to tackle your take-home exercise — arguably the most important part of the interview and the part that’s most in…towardsdatascience.com

The Ultimate Interview Prep Guide for Data Scientists and Data Analysts
What helped me interview successfully with FANG as well as unicornstowardsdatascience.com

The Ultimate Interview Prep Guide for Your Next Dream Data Job
What helped me interview successfully with FANG and unicorns for jobs ranging from product manager to data scientisttowardsdatascience.com

Things you should consider before starting as a data science consultant

In my previous articles about McKinsey, I talked about why I decided to join McKinsey as a data science consultant and why I think it’s a great career move for aspiring data scientists; I have also talked about valuable lessons I have learned from being a data science consultant.

Given the prestige of the company and the great things I mentioned in those articles, it was a surprise to a lot of people that I chose to leave after two years. In this article, I want to share what drove my decision to leave and what you should know and take into consideration if you are also thinking about joining consulting as a data scientist.

And just to steal the thunder of the negative voices that appear every single time I mention McKinsey in my articles — the reasons I will be mentioning here are solely out of the consideration for personal and career development; as for debates about “whether the firm is ethical”, “whether consultants are useful for companies” etc., everyone is entitled to their own opinion and those are out of the scope of my discussion here.

So with that out of the way, let’s talk about why I left:

It’s hard to feel the long-term impact of your work as a DS

You probably hear A LOT OF consultants saying this is THE reason why they have left consulting. I used to think it’s just a convenient reason people use in interviews but it turned out to be one of the main reasons that drove me to leave in the end.

This is partially determined by the nature of consultancy. Consultants provide expertise and advice in certain fields to companies. At the end of each engagement, the proposed solution (usually a model or an analysis for data science engagements) will be wrapped in a bow and handed to the client to implement, test and/or maintain. Imagine you built a demand-forecasting model for a client as a data science consultant: You can assess the model’s performance using historic data before handing it off to the client, and if the partner manages to convince the client to extend the project to test the model, you might get several weeks worth of new data to assess impact. But most likely you won’t know its performance after it’s officially deployed by the client (or if it will be deployed and used at all), let alone its continued impact several years down the road.

Until this day, it’s still a debate whether the implementation phase should be part of consulting firms’ scope. Some people believe if consultants start to implement the solutions proposed, they are taking on the role of managers while others think proposal of solutions without implementation is a waste of time and money. McKinsey started taking implementation more and more seriously during my tenure; but at the current stage of the industry, aspiring data science consultants should not get their hopes up that they will see their models in production and be able to observe longer-term impact.

Not many chances to work with cutting-edge DS concepts

Companies hire consultants for the expertise they DON’T have. So naturally, most clients that work with consultants for analytics needs are not the most advanced on the data science front. As a data science consultant, you are most likely NOT going to work on building a recommendation engine for Amazon or search optimization for Google.

There are projects here and there focusing on relatively cutting-edge techniques such as dynamic pricing, but the majority of the engagements will be utilizing simpler methods. Not to mention that clients who are just starting out on developing their analytics practice usually have more distrust for black-box models and prefer interpretability over it. As a data science consultant, you will have more opportunity to learn how to explain analytics concepts to non-analytical audiences (which in my opinion is an extremely valuable skill), but less opportunity to implement the newest practice in the DS field.

This is a more relevant concern for some than for others. If your interest as a data scientist lies in machine learning, AI and/or other advanced analytics areas, it’s worth keeping this in mind before deciding which projects you want to work on in consulting, or if consulting is the right industry to enter at all. However, if you are more interested in solving real-world business problems through data and/or modeling, consulting is the perfect place to start learning and practicing those skills.

Scalability is not always top of mind and it’s hard to think about/take on long-term initiatives

Since consulting engagements are usually short-term, even squeezed for time occasionally, clients expect to see deliverables within weeks. Due to short timelines and the fact that most times consultants are not held accountable for the sustainability of the deliverables after the handoff, efficiency usually takes priority over scalability when it comes to choosing approaches. A lot of the analytics deliverables, models or analyses end up being more of a “proof of concept” or “prototype” than a real fully-fledged product.

Ever since working in the tech industry, especially as a manager, I have learned the importance of long-term initiatives. These type of projects are usually initiated by managers and leaders who elevate themselves from their teams’ immediate day-to-day scope and proactively address gaps in realizing the whole company’s vision. Consultants hardly ever need to think about long-term initiatives because they are hired to solve a (or a set of) problem(s); so the nature of the work is reactive instead of proactive and the scope of the work is normally pre-defined. Imagine you are hired to figure out how to place a piece of the puzzle, but you don’t have a say in the how the rest of the puzzle can be moved around; it can get frustrating and you don’t get to develop an understanding of the rest of the puzzle and thus never develop the skillset to move those pieces around.

The long hours are real and the traveling can get frustrating

The final reason is more on the personal level than on the professional level. Due the nature of the work and the industry (like lawyers, consultants bill their clients by the hour), consulting is not known for a great work-life balance. Churning out analyses/decks until midnight is the norm for most weekdays.

On top of the long hours, pre-pandemic, consultants need to travel to a different city from Monday to Thursday for most projects. You might be thinking to yourself, “that sounds exciting, I get to visit different cities!”, and that’s what I thought, too, at the beginning. However, the excitement of travel for work wears off pretty quickly, especially since you have limited time to explore the city and spend most nights working late in similar-looking business hotels.

Summary:

Like I mentioned in my previous article about why I decided to join McKinsey as a data scientist, even though consulting is a great place to start your career as an aspiring data scientist, it might not be for everyone. Things to keep in mind about being a data scientist in consulting:

• Consulting might be a better fit for you if you are more of an “explorer” than a “builder” because for most projects, you will hand off the prototype to clients and won’t see it grow

• In general, you will acquire more data story-telling skills than cutting-edge modeling skills

• You will get more opportunities to engage in efficient problem-solving with short-term analytics projects; but less chance to do long-term planning for robust analytics ones

Interested in reading more about data science career? Here are some articles that you might be interested in:

Why I Joined McKinsey as a Data Scientist
Should you ever be a data science consultant despite the 80-hour work weeks?towardsdatascience.com

5 Lessons McKinsey Taught Me That Will Make You a Better Data Scientist
towardsdatascience.com

5 Mistakes I Wish I Had Avoided in My Data Science Career
I learned these lessons the hard way so you don’t have totowardsdatascience.com

How to improve productivity and be happier when WFH

By now I’m sure everyone has heard extroverts’ complaints about the whole WFH situation — the fun of meetings and being surrounded by people is taken away from them. But since introverts gain energy from the solitude, that means WFH should be easy for them, right?

Well, not really.

Like a lot of people, at the beginning of the pandemic I thought that WFH would be more introvert-friendly and less draining in comparison to an in-person working environment, and as an introvert myself, I was looking forward to that. Man, was I wrong: The virtual work environment can be very loud, arguably louder than being in the office at times.

Work from home (WFH) I hoped for:

9am-11am: A couple of important meetings with small groups

11am-12pm: Lunch time

12pm on: Heads down coding/modeling/analyses time without interruption or distraction from office noise etc.

Work from home in reality:

9am-forever: Meetings to eternity with 10+ people, with a couple of last-minute “can we hop on a call real quick” requests sprinkled on top.

10-minute-intervals in between meetings: Trying to quickly get some heads-down work done

Remote work during COVID has unfortunately resulted in more frequent meetings and emails. It definitely took a toll on my mental health when my calendar was suddenly filled with huge meetings where it was hard to get points cross without cutting people off; my slack beeped every 30 seconds; and the little red circle on my email app increased counts every minute. The virtual working environment quickly became too “loud” for me to be productive.

Know what energizes you and what drains your energy as an introvert:

Even though extroversion and and introversion are not binary, if you self-identify as an introvert, it’s likely that big groups drain your energy to some extent and you prefer smaller, more intimate interactions. It shouldn’t be a surprise that you need time and space to reflect and/or work on things that involve deep thinking and analysis thus it’s helpful to find ways to eliminate distractions.

For the past several months, I experimented with different ways to improve the remote-work situation trying to reduce the distraction and noise so I can be productive and keep my mental health in check.

Meetings

Control your schedule by grouping all your meetings together

For introverts, what’s more annoying that having tons of meetings everyday is having tons of meetings spanning the whole day with 30-minute breaks in between. Because those 30-minute breaks are not enough to do any deep work before you get pulled into another meeting, they are essentially “wasted”. And introverts get energy from being able to dive into and focus on a topic.

What works better is trying to group all of your meetings together. So instead of having meetings filling up your whole day, have them fill your whole morning (or afternoon, or early morning and late afternoon, your choice) so you will still have chunks of time to do in-depth work without getting distracted outside of those meeting blocks. In order to make sure that nobody schedules over your focus time, block if off on your calendar as “DNS” (Do Not Schedule) or “Please ask before scheduling”.

And instead of having back-to-back 30-minute meetings, I would usually try to schedule 25-minute meetings: end the meeting 5 minutes early or start the meeting invite 5-minutes after the clock. This way you have enough time to take a break between meetings and reset.

Have a pre-read and agenda for meetings

It’s always hard to get your point across in big meetings, especially as an introvert. Not to mention you will often have participants in the meetings ask a marginally-related question and take everyone down a rabbit hole; as an introvert, I find it hard to then drag everyone back to the original topic.

To avoid situations like these, for the meetings I run, I prefer to send pre-reads and an agenda before the meetings detailing the topics we want to talk about and providing some background/context of the meeting. This will help decrease the amount of digression and help drive the points home. When the discussions get out of hands, it’s also easier to have the agenda serving as a lighthouse for people to re-focus on the agreed-upon goal of the meeting.

Keep your meeting size small and focus on small-group interactions

Giant meetings and 100-people slack channels can be a nightmare for introverts.

I prefer to keep my meetings small so I can ask targeted questions and make sure everyone’s opinion gets factored into the discussion. When having cross-functional discussions with other teams, instead of inviting everyone, I only invite one or two key people on each team and count on them for intra-team communication from there.

For giant virtual meetings, I find it easier to ask questions through the chat function of the meeting tool than trying to talk over people.

Other working tools

Turn off your slack notification and check it on your own terms

Instead of getting distracted by the sound of slack every 5 minutes, try turning off the notification and only check it when you want to take a break from the analysis you are working on. Even if you can’t help yourself but to check the message right away when a little red bubble with number “1” shows up on top of Slack, the absence of the notification sound will make you less stressed.

Only dedicate certain times for checking/processing emails

Depending on your company culture and people’s working habit around you, you may get tons of emails on top of slack messages. I find it the easiest to make sure people know that emails should be used for non-urgent communications whereas Slack could handle the more urgent ones. This way I can bulk check my emails once a day and not get distracted by them during my heads-down time.

Hopefully some of the tips here are helpful to my fellow introverts out there — the virtual world can be loud and noisy but you have the power to turn down its volume and find your own rhythm in it.

Don’t know what to read next? The following might be interesting to you:

Why I Joined McKinsey as a Data Scientist
Should you ever be a data science consultant despite the 80-hour work weeks?towardsdatascience.com

Productivity Tips for Data Scientists
How to work better, smarter but not necessarily harder as a data scientisttowardsdatascience.com

Why I Won’t Participate in the ‘Great Resignation’ For My Side Hustle
and why you should reconsider, too, if you are thinking about itmedium.com

Office Hours

Most frequently asked questions in data scientist interviews

It’s the beginning of the year and a lot of people are looking into new opportunities and new challenges. I remember from my multiple rounds of recruiting in the past that I regularly struggled to find a good comprehensive resource to prepare for my data science interviews, so I decided to put one together myself.

If you are starting a new round of recruiting or looking to up-skill your data science knowledge for the new year, hopefully you will find this cheatsheet/crash course helpful as a starting point. This is the first article of the data science interview series where I want to mention some basic distributions and basic concepts related to distributions that get asked frequently in data science interviews.

The concepts I will touch on in this series is by no means exhaustive, it’s an aggregated list for the most asked questions in data science interviews, or concepts good data scientists should be familiar with, based on my experience.

Basic Statistics Concepts Related to Distributions

This might seem too basic for some of you, but it’s important to make sure you truly understand the definitions, differences and use cases for simple statistic concepts like mean, median and mode. Several key concepts I see coming up time and time again in interviews:

1. Mean is different from the median in the sense that the mean is more easily influenced by outliers and the skewness of the distribution. The mean will be the same as the median for symmetric distributions. The mean will be smaller than the median for left skewed distributions and will be bigger than the median for right skewed distributions.

2. Mode represents the “peak” of the distribution and it overlaps with both the mean and the median for symmetric distributions. For left-skewed distributions, the mode is bigger than both the median and the mean; and the mode is smaller than both for right-skewed distributions.

Distributions

Distributions are a very important concept in statistics that gets tested in interviews a lot. There are a lot of distributions out there, but several important and common ones get mentioned and used over and over again in interviews:

1. The Normal Distribution

This is THE most important distribution to know as many things in life follow the Normal Distribution — height of adults, IQ of adults etc. It’s important to know that the mean, median and mode are the same for the Normal Distribution as it is symmetric and NOT skewed.

The Normal Distribution has symmetric variation around the mean that is defined as standard deviation. This means it’s equally likely for values to be higher or lower than the mean. But it’s less likely for values to vary from the mean by a larger magnitude than a smaller magnitude (regardless of the direction). In fact, for the Normal Distribution, 68.3% of values lie within 1 standard deviation away from the mean; 95.5% lie within 2 standard deviations away from the mean and 99.7% values are within 3 standard deviations from the mean.

The Normal Distribution is determined by only two parameters — the mean and the standard deviation. The mean determines the center of the distribution — where the peak (mean, median and mode) lies whereas the standard deviation determines the shape/spread of the distribution — the smaller the standard deviation, the tighter the distribution.

2. Bernoulli Distribution

The Bernoulli Distribution describes a single experiment that has ONLY two outcomes. We usually describe these two outcomes as success-1 and failure-0 for simplicity. The most common examples of this distribution are things like a flip of a coin, where a head would be a “success” and a tail would be a “failure”; or the weather outcome in terms of whether it will rain, where rain would be a “success” and no rain will be a “failure” (or the other way around if you really hate rain).

There is only one parameter for this distribution — the probability of success p. In the case of a fair coin, p=50%.

3. Binomial distribution

The Binomial Distribution is closely related to the Bernoulli distribution as it models the number of successes in repeated Bernoulli experiments.

The Binomial Distribution takes two parameters — the probability of success p and the number of repetition n. When p is greater than 0.5, the distribution is left-skewed; when it’s smaller than 0.5, the distribution is right-skewed and when p = 0.5, you guessed it, the distribution is symmetric.

If you think the symmetric Binomial Distribution above looks a lot of a Normal Distribution, you are right. In fact, the Binomial Distribution can be approximated by the Normal Distribution when the number of experiments is big enough (with n being the number of experiment repetition and p being probability of success, the rule of thumb is when both np and n(1-p) are bigger than 10, the Binomial Distribution can be approximated by the Normal Distribution).

Other less common distributions that you might get asked about are the Poisson Distribution (it describes the number of events in a certain timeframe; the most common example is the number of customers arriving to a restaurant within a fixed time frame), Exponential Distribution (it is related to the Poisson Distribution since it describes the time interval between the events; in the customer example, it describes the time passed between each customer’s arrival within a fixed time window) and Gamma Distribution (it is similar to the Exponential Distribution in the sense that it describes the the total time until reaching a fix number of events; to continue the example, it describe the time you need to wait until having a certain number of customers arrived at the restaurant).

How these are tested

These concepts are usually NOT tested separately in the format of a school test, because interviewers want to see you can actually use those concepts to solve real world problems instead of memorizing the wikipedia pages about them. So the common questions asked are along the lines of:

• “What kind of distribution does XX (e.g. the length of Facebook Marketplace page visits etc.) have?”

• Follow up question: “What’s the relationship of mean and median of this distribution?” or “Do you think the mean will be bigger or smaller than median for this distribution?”

Hope this summary is helpful to your recruiting process. Keep in mind that you won’t be a stats expert by just reading several articles about it; it takes practice to master stats knowledge. So the best way to familiarize yourself with this is to utilize this knowledge whenever you can; for example, when you research about the company you are interested in, ask yourself what the most important metrics for this company would be and what you think their distribution looks like.

Don’t know what to read next? Here are some suggestions:

Why I Joined McKinsey as a Data Scientist
Should you ever be a data science consultant despite the 80-hour work weeks?towardsdatascience.com

Productivity Tips for Data Scientists
How to work better, smarter but not necessarily harder as a data scientisttowardsdatascience.com

Top Qualities Hiring Managers Look For In Data Scientist Candidates
Some of these are arguably more important than writing efficient codetowardsdatascience.com

Should you ever be a data science consultant despite the 80-hour work weeks?

Time really flies; we said goodbye to the crazy and unexpected 2021 and stepped into a new year….more than a month ago! Reflecting on 2021, I realized that I have left the consulting life behind for over a year now.

Looking back, joining McKinsey was one of the best moves I made for my career (so far) and I truly learned a ton about data science and career development in general from the experience (if you want to read about the lessons I learned about data science, read my previous article here).

So as a memo to myself and people who are interested in the consulting world, I want to share some thoughts in this article about why I initially joined McKinsey; in my next article, I will then share why I ultimately decided to leave. Hopefully, in combination these articles will provide some insights for those who are considering data science consulting as a career move.

1. Consulting is a great place to look for your passion and learn about different industries and from different industry experts

After being a quant researcher in the finance industry for almost two years, I realized that I loved the analytics and data aspect of my job, but I was not a huge fan of the finance industry.

So I decided to take my transferrable data skills somewhere else. The question was, where? I only knew what I DIDN’T want to do, but was clueless about what I DID want to do. Did I want to work with geo-spatial data? Optimize consumer marketing campaigns or B2B sales funnels?

Consulting is the perfect industry to go if you are in the same spot as I was — knowing that you want to work with data but don’t know what KIND of data you want to work with and/or which INDUSTRY you are interested in working in.

Because consulting firms serve a wide range of clients, no matter where your interests and passion lie, you WILL find something that’s for you. Even within the same industry, you will get to work with a wide range of companies in terms of size, maturity, culture and other dimensions.

Because experienced consultants have worked with dozens of companies in the past, they often have developed best practices/playbooks in a certain area. So as someone who’s new to the data science field, consulting is one of the best places to “learn the ropes”.

2. Consulting is a good way to pivot your career

The transition from a quant finance researcher to a data scientist was not an easy one despite the fact that there is a lot of overlap in terms of skillset; a lot of data science recruiters are unfamiliar with other fields (in my case the quant finance industry) and often don’t know how to correctly evaluate non-traditional backgrounds. Consulting hires from more diverse backgrounds than typical industry jobs, so it’s the perfect stepping stone and pivot opportunity if you want to make a career shift into data science.

Having a consulting background is definitely a plus if you are later applying for industry jobs; partially because of the prestige of most consulting firms and partially because of the skills and experience you get in a consultant role.

3. You learn to be agile, learn quickly and be a full-stack data scientist

In consulting, every project will be different and every new engagement is like getting a new job. As a result, you learn to be extremely agile and adapt to different working situations, managers, teammates and stakeholders quickly. Some clients use Python and some prefer R; some clients’ data is in databases and can be extracted through SQL, some have data only in CSVs or even PDFs. As a data science consultant, you often need to find creative ways to solve the seemingly impossible and up-skill yourself through learning on the job.

Because every project’s setup and team varies, as a data scientist, sometimes (if you are lucky) you get staffed with a whole team of data engineers; sometimes you have to BE the data engineer and try to process client’s “perfect” data from PDFs. Because of this, most data science consultants are “full-stack” data scientists who can work on the whole end-to-end process of data science projects covering data engineering, model building, all the way to tool building and insights generation.

These skills are extremely valuable for companies these days, especially for startups or companies that are just starting to build up their data science teams. Additionally, it was the perfect way for me to test out which career path in the data world I was actually interested in and wanted to specialize in (if you are not familiar with different data careers, read my previous article here).

4. Exit options are great and seeking them is even encouraged

Unlike most industry companies, which try to avoid churn like the plague, consulting companies aren’t too worried about their employees leaving. It might be obvious why this is the case — when consulting alumni leave to join industry companies, they become potential clients for the consulting company.

Every industry company hires consultants for something at some point; who better to vouch for you over your competitors than your own alumni?

Because consulting firms embrace the ultimate departure of the employee, they create a lot of channels like internal job boards, newsletter etc. for their alumni to share job opportunities outside of consulting.

Also most companies LOVE to hire ex-consultants because they have experience adapting and deploying data science solutions in different companies in various industries. And consultants are used to being thrown into the deep end and learn things from the scratch fast.

For me, consulting was a great way to figure out what I wanted to do next while keeping my options open, and once I had figured that out, the McKinsey brand, network and support (e.g. paid “search” time to find a new job) helped me line up my next opportunity.

5. The network you build is incredible

Since a lot of people leave consulting eventually, there are a lot of consulting alumni in industry companies’ management and leadership teams. So it’s likely that there’s already a McK alum working at your dream company when you are thinking about leaving consulting; and networking with alumni is always easier than cold-emailing.

What’s also common for ex-consultants is to eventually start their own companies, building on their experiences across various industries and opportunities they discovered along the way. If your dream is to eventually build a company of your own, it’s also likely to find like-minded co-founders in consulting (or you can find and join an early-stage venture that matches your interests through the McKinsey network).

Conclusion:

I think you should absolutely join consulting (McK or other companies) as a data scientist if any of the following applies to you:

• You don’t know yet what kind of data you are passionate about as a data scientist

• You don’t know which industry is right for you

• You are hoping to pivot your career into data science from another field

• You want to learn the tried and true best practices for data science (notice best practice doesn’t necessarily mean cutting edge; in fact, it’s almost always the contrary)

Consulting might NOT be for you if:

• You want to go deep and specialize in a certain area of data science (e.g. you know want to ONLY focus on building ML models and be an expert in the area); this is possible in consulting, but might be easier to achieve in an industry role

• You want to work on cutting edge methodologies in data science (I will explain why it’s hard for you to achieve this in consulting in my next article where I explain why I left McKinsey)

Don’t know what to read next? I might have some recommendations for you:

Productivity Tips for Data Scientists
How to work better, smarter but not necessarily harder as a data scientisttowardsdatascience.com

Top Qualities Hiring Managers Look For In Data Scientist Candidates
Some of these are arguably more important than writing efficient codetowardsdatascience.com

Avoid These Five Behaviors That Make You Look Like A Data Novice
And be a trustworthy, likable data partnermedium.com

How to work better, smarter but not necessarily harder as a data scientist

I have always been a firm believer of “work smarter not harder” and working in fast-paced companies in the past several years reinforced this belief. Based on my observations as a data scientist, there seems to be a permanent discrepancy between how “slow” analytics (or engineering) work goes and how fast businesses need solutions.

Even though this tension is mainly due to the fact that highly technical work has the level of difficulty and needs the level of precision that takes time, it can partially be alleviated through proper stakeholder management (which I have covered in several other articles) and by making yourself a more effective and efficient data scientist. There are several lessons I learned in the past several years that I hope can help you make your work as a data scientist “smarter”.

Help others to help you

If you have ever gotten questions along the lines of “how many XX (transactions, trips, etc.) did we have last month?” (what I call “data Siri questions”), you know how much time you spend on those seemingly small asks and how frustrating they are.

The best way to NOT let these small asks drag down your productivity (and mood) is to up-skill your stakeholders so they can be self-sufficient to some extent.

Most stakeholders I have worked with are more than happy to learn the basics about analytics in a weekly or monthly office hour or training session the data team hosts, because nobody loves to rely on other people all the time to get their jobs done.

Be that “annoying” data scientist and ask “where does the data go”

Time and time again I observe the data team scrambling to get data when people ask “can we track XXX (related to a product or process recently launched)?” and the data team realizes the data is not being tracked correctly or at all.

My solution to avoid this kind of situation is to “butt in” the conversations early. When you are in a meeting where people talk about building a new product and/or process, if you know you will eventually be responsible for analyzing the data (that’s probably why you are in the meeting as a data scientist), ask “dumb” questions as early as possible “where will the data be stored?”, “have we decided the schemas of the tables yet?”…

I used to assume someone must have already thought of those questions. Don’t make those assumptions. You will be surprised by how often people forget about the data aspect of things, even in companies that pride themselves for being data-driven.

And trust me, people will appreciate you for asking those “dumb” questions early on as it will avoid bottlenecks down the road.

Pay attention to “irrelevant conversations”

Being able to absorb information about things that don’t seems to fall directly in your immediate scope is an underrated ability in my opinion. Time and time again I realize that conversations I read in Slack threads and group chats that didn’t seem to concern me at the moment contain useful information for my future work.

It’s really not surprising if you think about it. Every piece of work that happens in the company is intertwined and connected because at the end of the day, everyone is working towards the same goal. It’s only a matter of time before things that are not relevant for you become relevant; and when they do, you have some basic context and know what’s already been done or at least where to start looking for information.

So be curious and inquisitive of things others are working on and don’t be tunnel visioned by your scope. The future you will thank the past you for catching certain pieces of “irrelevant” information and saving a huge amount of time because of it.

Be a sponge and a dot connector

This one ties back directly to the one above because you can only connect the dots if you paid attention to things that are happening around you.

Being able to absorb information and connect the dots is one of the most valuable abilities I look for in people I hire. I can’t even start to describe how much I appreciate team members who can tell me “XX mentioned that he/she did YY that I think is very similar to this new ask”.

Like I mentioned in my previous post, when it comes to high impact projects, it’s likely that someone has thought about and done something similar in the past; so being able to NOT reinvent the wheel but build upon previous work can save a lot of time and create a ton of synergy. And if another team is currently actively working on a similar project, you can either join forces or simply reprioritize and work on something else while they solve the problem for you (assuming that the timeline and output align with your needs).

Batch your work and utilize productivity-hacking

Unless you have an extremely micromanaging manager, you usually have some control over how you prioritize your work. The best tricks for being more productive that I have learned over the years are targeting low hanging fruits at the right time and decreasing switching cost by batching similar work.

Even if you successfully up-skill all of your stakeholders, you will inevitably get some “data Siri” questions that shouldn’t take much of your brain space, so use those to fill the hours when you feel like your brain is fried and you can’t really concentrate, or when you are in a boring meeting and you can multi-task.

Batching your work is another way to improve productivity because switching back and forth between different topics can be distracting and result in inefficiency and slow progress. So I usually group similar tasks together and knock as many of them out as possible in one sitting. Schedule all of your meetings back to back and reserve several hours afterwards for headphones-on, heads-down, productive coding time. Your brain will thank you for not pulling it in all directions in a short span of time.

Never dive into a problem without taking a step back first — question everything

Don’t let others’ approach for a problem be the “box” for your thinking process. Think outside of the box. What I mean by this is if your stakeholder has an analytics request that they have some ideas about how you can approach it, that’s great, hear them out; but don’t let that be the Bible for how you MUST approach it.

As a data scientist, you should have the data expertise in the room and can help people decide what the most efficient way to approach a problem is from a data perspective. Don’t be afraid of challenging the approach and making suggestions.

Key Takeaways:

There are several ways to make you more productive and more effective as a data scientist:

• Up-skill your stakeholders to lighten your ad-hoc requests load

• Be more proactive in data-related conversations

• Take more initiative in /pay more attention to conversations that are seemingly not in your scope

• Learn to connect the dots

• Prioritize by batching your work and breaking down big tasks

• Don’t take anything for granted, always look for more efficient approaches

Enjoyed the article? Subscribe to my email list so you won’t miss an article in the future! Don’t know what to read next? Here are some suggestions:

How to Deal With Frustrating Stakeholder Situations as a Data Scientist
And turn them into opportunitiestowardsdatascience.com

Top Qualities Hiring Managers Look For In Data Scientist Candidates
Some of these are arguably more important than writing efficient codetowardsdatascience.com

Should You Join a Big Corporation or a Small Startup As a Data Scientist?
How do you know which one is for you and what to expect from eachtowardsdatascience.com