What cloud data engineering actually looks like on the job

Cloud data engineering still starts with the same job as plain data engineering: move data from where it gets generated to where someone can query it. The cloud part changes the setup. You stop running scripts on your 8 GB laptop and hoping it survives a ten GB CSV. You use managed services for storage, compute and scheduling, and those services come with bills and failure modes you have to learn.

Some days it is SQL until your eyes hurt. The rest is moving files and fixing schemas. And opening a DAG that failed at 3 AM. When you join your first team, you'll hear ETL, ELT, partition, and scan cost in the same week. Under all that, it's one pipeline: pull data from an API, clean it, load it into a warehouse.

That's ETL. The newer flavour is ELT. So what changes there? You load the raw data first, then transform it with SQL inside the warehouse. Why? Cloud warehouses have enough compute to do the transforms themselves, so you don't need a separate machine for every cleanup step.

Some pipelines run as real-time streams. Most juniors still start with batch jobs.

Storage comes first because data has to sit somewhere before you process it. On the cloud that's usually object storage: Amazon S3 or Google Cloud Storage. Think of object storage as a giant folder on the internet where you drop CSV files, JSON logs, parquet files.

Beginners often dump everything into flat folders with no partitioning. If you put ten gigabytes of daily transaction logs in a single folder, your query engine has to read every byte each time, even when you only want one day.

s3://logs/transactions/year=2023/month=10/day=25/

That's partitioning. The year=... folders let the query engine skip the days you didn't ask for. It also keeps queries from reading every day when you only want one.

Processing data at scale

Processing is where Spark shows up. Apache Spark is the big one here. You write Python or Scala, then submit the job to a cluster. AWS EMR and Databricks spin up multiple nodes and run the work in parallel. You don't manage the hardware. You do have to write code that doesn't fight the cluster.

The classic beginner mistake is treating PySpark like a normal Python list. You loop over rows, call collect()and process each row on the driver. If the table is big, collect() can kill the driver with java.lang.OutOfMemoryError before your loop even starts. And if it doesn't, you've pulled all the data onto one machine, which is the opposite of distributed computing.

# Avoid this row-by-row loop in PySpark
for row in df.collect():
    process(row)

# Do this instead using dataframe transformations
df_transformed = df.withColumn("status", expr("val > 10"))

The second version stays inside Spark. Spark can run it across the nodes, and you don't move the whole table to one machine. That one change usually turns a job that never ends into one that finishes.

After processing, data goes into a cloud data warehouse. Snowflake and BigQuery are common. Redshift is the AWS one. These warehouses store data by column instead of by row, so analytical queries can skip columns you didn't ask for. That's why a query over billions of rows can come back in seconds.

The bill is the catch. Some cloud warehouses, such as BigQuery, charge by how much data your query scans. Run SELECT * on a huge table and you're scanning every column, including the ones you'll never look at. A beginner sees a slow query and adds a filter. The bill still goes up because the filter didn't cut the columns.

-- This scans every column in the table
SELECT * FROM sales WHERE order_date = '2023-10-25';

-- This scans only the columns you need
SELECT order_id, amount FROM sales WHERE order_date = '2023-10-25';

Filter early and select only the columns you actually use. If the bill still looks wrong, open the query plan and check what got scanned.

Keeping pipelines running smoothly

Airflow is the most common one you'll meet for orchestration. You define your pipeline as Python code, called a DAG (Directed Acyclic Graph, if you want the full form). Each task is a step, and dependencies make sure your transform doesn't start before your extract finishes. If a task fails at 3 AM, Airflow retries it or alerts you, and the logs tell you which step broke.

But Airflow only schedules the work. It won't tell you that your Spark job is leaking memory until the job dies. Watch the metrics for your jobs. If a Spark job keeps failing with ExecutorLostFailurecheck memory and shuffle settings before rerunning it. The logs usually point to the stage that's struggling.

Security and compliance are part of the pipeline. Don't treat them as a cleanup task after launch. If you work with digital personal data in India, the Digital Personal Data Protection Act applies. The guidelines are on meity.gov.in. In practice, that means role-based access control and encryption for data at rest and in transit. An S3 bucket open to the public is a data leak.

Our Cloud Data Engineering course at Rise Institute covers these pipeline patterns and the security practices in more detail. We spend time on partitioned writes and on PySpark jobs that don't collect everything to the driver. The SELECT * bill gets its own section.

Getting started doesn't need every cloud service on day one. Pick one provider. Learn how to move a CSV file from object storage into a data warehouse with Python and SQL. Then build a small Airflow DAG that runs the same steps on a schedule. Once storage and compute make sense, and Airflow is running the steps, a new tool is mostly new syntax.