Benjamin Rogojan

Can data analysts switch and become data engineerings.

The short answer is, yes!

The longer answer...

Working as a data analysts gives you exposure to several skills and infrastructure that data engineers work on.

This includes:

- Working with data warehouses and data lakes
- SQL
- Automating processes with Python and VBA
- Data

Once you've got a baseline, then you just need to:

- Learn about distributed systems
- Practice data modeling for analytics and transaction systems
- Read up on various software design principles and paradigms like OOP and functional programing
- Ask your manager to take on small DE projects at work
- Learn about data pipelines, ETLs/ELTs and other workflows
- Subscribe to the youtube channels in the comments
- Start interviewing for data engineering roles!

I actually put together a video on how I went from a data analyst to data engineer.

If you're working as a data engineer, a data analyst, or a data scientist there are many lessons you probably wish you would have known before you started.

Here are some things I wish I knew before I became a data engineer.

1. Don’t let the hype distract you - There are a lot of fancy terms, tools, and job titles floating around in the data world. Yeah, you do need to spend some time learning about a new tool here and there. However, I would say focusing on basic technical skills will often take you farther and make it easier for you to pick up new tools in the future.

2. Create Maintainable Systems - It's easy to build out a solution that pulls data from 8 different sources at different times and utilizes different tools. This isn't sustainable and likely if you do build a system that is overly complex it will be left behind in the future.

3. Source Of Truth Is Less Of A Destination And More Of A Process - The phrase “Source Of Truth“ has been around for decades. I heard at my first job and quickly found out I was lied to. Creating a source of truth is often more of a process and less of a final destination. Your company will constantly be adding in new systems, switching out ERPs, and changing business workflows. All of which means you will constantly need to improve and change your “Source Of Truth“.

4. Save Your SQL - Whether you're an analyst or a data engineer you will likely be asked to run some ad-hoc SQL. If no one tells you or you don't have a process set up to save your SQL, you may lose it or change it to the point where you don't recall what you did. So make sure your SQL logic is being saved. Whether that is via your SQL engine or better yet through some form of version control.

5. Don’t Say Yes To Every Request - I still struggle with this issue which is saying yes to everyone's request. It's just so easy to keep saying yes but you only have so much time in the day. You can't deal with every ad-hoc data request and data pipeline. So practice saying no or at least prioritizing.

What was one lesson you wish you would have known before you started in data?

#datascientist #dataengineer

Interviewing for any technical position generally requires preparing, studying, and long, all-day interviews.

This is why I put together a data engineer interview study guide to help keep track of my progress when I was interviewing(shared below).

There are a number of subjects that need to be covered in order to ensure you are ready for back-to-back questions.

Some interviews will ask you about Spark, others SQL and still others..both.

I recommend asking the recruiter if you aren’t sure which type of interview you will be facing. Some companies are very good at keeping interviews consistent, but even then, teams can deviate depending on what they are looking for.

Here are some examples of what I have noticed about some companies' data engineering interviews.

Amazon — SQL and database-design heavy as well as general ETL design. Surprisingly, coding is some what team dependent. I have talked with several people and done the interview myself and currently some have had coding rounds others have not.

Expedia — Big Data questions, like what is Spark and RDDs, as well as SQL and Python.

Meta - SQL, data warehouse/ETL design and Python focused.

What have your experiences interviewing been for DE positions?

https://lnkd.in/gHgSq2yD

#dataengineering #data

Are you trying to learn more about data engineering in 2024?

Then here is a free crash course with +25 free resources you can use to get up to speed fast.

It's broken down by the basics, then goes into higher level concepts like OLTP vs OLAP, data modeling, and finally goes into a few solutions like Airflow, Snowflake, etc.

The Basics (Programming, SQL, The Cloud, etc)

SQL For Beginners By Alex Freberg
1. https://lnkd.in/gAXcqn8W

Data Engineering Vocab 101
2. https://lnkd.in/gRQagRax

What Is Data Engineering - Why Is Data Engineering Important?
3. https://lnkd.in/gFmtavpu

Python Tutorial - Python Full Course for Beginners Mosh Hamedani
4. https://lnkd.in/gpdCy4mf

Python Libraries You Should Know As A Data Engineer - Python For Beginners
5. https://lnkd.in/gzupGDh4

Data Warehouse Tool Kit
6. https://lnkd.in/gaVw_hgg

Becoming A Better Data Engineer - Tips On Translating Business Requirements
7. https://lnkd.in/gnQNyZbs

Normalization Vs Denormalization
8. https://lnkd.in/gf_ivben

Data Modeling - Walking Through How To Data Model As A Data Engineer - Dimensional Modeling 101
9. https://lnkd.in/g4QcjgXy

Data Modeling Where Theory Meets Reality - How Different Companies I Worked At Modeled Their Data
10. https://lnkd.in/gMmVmU8B

Using The Cloud As A Data Engineer
11.https://https://lnkd.in/gqKfD_Rc

Higher Level Concepts

What Is A Data Pipeline
12. https://lnkd.in/ginyTQYM

Transactional Databases Vs Data Warehouses Vs Data Lakes
13. https://lnkd.in/gxpbCRPi

Why Is Data Modeling So Challenging – How To Data Model For Analytics
14. https://lnkd.in/gd5CQrZs

Workflow Orchestration With DataTalksClub
15. https://lnkd.in/gRxvuuQZ

OLTP vs OLAP
16. https://lnkd.in/gfMqKQHB

How to design resilient and large scale data systems by Zach Wilson
17. https://lnkd.in/gWXzeqre

Setting Standards For Your Data Team
18. https://lnkd.in/gSpjWpvt

How To Come Up With A Data Engineering Project?
19. https://lnkd.in/gZTGK2Cz

7 Data Engineering Projects To Put On Your Resume
20. https://lnkd.in/gPtU5TSm

🚖 Uber Data Analytics | End-To-End Data Engineering Project by Darshil Parmar
21. https://lnkd.in/gAy7mGjJ

Some technologies data engineers might need to know.

Intro To Spark by Darshil Parmar
22. https://lnkd.in/g3QBdzGa

The Realities Of Airflow - The Mistakes New Data Engineers Make Using Apache Airflow
23. https://lnkd.in/gRRnpCjh

What Is Docker - Docker Intro And Tutorial On Setting Up Airflow
24. https://lnkd.in/g3fPuz8r

Spark vs Polars. Real-life Test Case by Daniel Beach
25. https://lnkd.in/gDEeHn5h

Spark Vs Flink
https://lnkd.in/gC_8EvuK

Snowflake Vs Databricks - 🏃‍♂️ A Race To Build THE Cloud Data Platform 🏃‍♂️
26. https://lnkd.in/gfy-8gDs

What are your favorite free resources to learn data engineering?


#dataengineering

When I quit Facebook I assumed all of my time would be spent consulting on data infra and engineering problems.

But recently I have had the opportunity to take on a new set of challenges as an Advisor for a few start-ups.

With that I am happy to share that I’m starting a new position as an Advisor at Estuary!

We've been talking a lot about working together for the past few months and after seeing Estuary in action, I decided it was time to sign on! I am looking forward to working with David Yaffe and the rest of his awesome team!