Home » Apache Spark Fundamentals

Apache Spark Fundamentals

Leer data te verwerken met PySpark op Apache Spark
Studie-investeringsformulier
Terug naar overzicht

Cursusbeschrijving

Met de opkomst van cloud computing, gedistribueerde opslag en (big) data verwerking, beginnen veel organisaties Apache Spark te gebruiken voor hun dataprocessen. Of het nu gaat om data science, data-analyse of data engineering, Apache Spark kan het juiste gereedschap zijn voor de klus. Het vormt een basis onder Azure Synapse Analytics, Microsoft Fabric en Databricks.

Deze training gaat je de grondbeginselen bijbrengen van het werken met Apache Spark, te beginnen met wat het is en hoe het werkt. Vervolgens ga je verder met het lezen, transformeren en schrijven van gegevens met behulp van PySpark.

Ten slotte zal er extra aandacht zijn voor het gebruik van development best practices om ervoor te zorgen dat je code veilig in productie kan worden genomen.

Onderwerpen

  • 1: About Spark
  • What is Spark, where did it come from, why was it created? And how does it work?
  • Lessons
  • History of Apache Spark
  • Technical Architecture (Driver, Cluster Manager, Executors)
  • RDD and Dataframe
  • Pyspark
  • Benefits of using Spark
  • Running Spark locally
  • After completing this module, students will be able to:
  • Explain how Spark works
  • ### 2: Reading Data
  • To work with data, we first need to retrieve it from wherever it is located. This is done through spark.read.
  • Lessons
  • spark.read
  • read options
  • read modes
  • Using regex in the filepath(s)
  • Lab
  • Read your first files in Spark
  • After completing this module, students will be able to:
  • Read data using PySpark
  • ### 3: Transforming Data
  • After retrieving our data we need to perform transformations on it. Operations such as joins, filters, grouping, aggregating, splitting and renaming are necessary in most data pipelines. How do they work in Spark?
  • Lessons
  • Filtering
  • Narrow and broad transformations
  • Column operations
  • JSON transformations
  • Window functions
  • UDF and Lambdas
  • Lab
  • Perform transformations with PySpark
  • After completing this module, students will be able to:
  • Transform data using PySpark
  • ### 4: Writing Data
  • After completing the necessary transformations in memory, it is time to write our data to our target location. This may sound like a plain operation, but there are things to consider such as file formats and partitioning.
  • Lessons
  • Common file formats
  • Apache Parquet
  • Delta Lake
  • Data partitioning
  • Bucketing
  • Lab
  • Write data with PySpark, with partitions and buckets
  • After completing this module, students will be able to:
  • Write data using PySpark
  • ### 5: Development Best Practices
  • All we need to do with data is reading, transforming and writing it. But the code we use to do that needs to be maintained. For this, we need to use development best practices. Some of them are general, others are specific to Apache Spark.
  • Lessons
  • Notebooks for Development, python files for production
  • Modularization
  • Logging
  • Error Handling
  • Testing
  • Continuous Integration
  • Lab
  • Read, clean, transform and write data using development best practices for production ready code
  • After completing this module, students will be able to:
  • Write PySpark code following development best practices

    • Materiaal

    Digitaal Info Support cursusmateriaal

    Voorkennis

    Python development