Introduction :
I built this project to process hotel-booking CSV data on Databricks using a three-layer Medallion Architecture. The pipeline incrementally ingests raw files into Bronze, applies cleaning and data-quality checks in Silver, and creates analytical datasets in Gold. I also added automated tests and a deployment workflow so that code changes can be validated and deployed through GitHub Actions.
Data flow: Bronze, Silver, and Gold :
Bronze – Incremental ingestion : A Databricks job runs SrcParameters.ipynb to provide the configured source names, then runs 01_bronze_ingestion.ipynb for each source. The ingestion code uses Auto Loader to read CSV files and append the incoming data as Delta. Schema, checkpoint, and data locations are maintained for the configured hotel_bookings source.

Figure 1. Databricks notebooks for source configuration (SrcParameters) and Bronze ingestion (01bronzeingestion).
Silver – Cleaning and validation : The Silver pipeline reads Bronze data, casts booking columns to their intended types, removes rescued data, and adds a modification timestamp. Data-quality expectations drop records with invalid cancellation flags, guest counts, or ADR values. The resulting table is silver.silver_hotel_bookings.
Gold – Analytics : The Gold pipeline derives arrival dates, total nights, and estimated room revenue. It creates three analytical tables: gold_hotel_booking_summary, gold_monthly_booking_trends, and gold_monthly_room_revenue.
Orchestrating the complete pipeline :
A parent job named Hotel Bookings BSG Job coordinates the three stages in dependency order: Bronze ingestion, Silver transformation, and Gold transformation.
In the captured run, all three tasks succeeded and the complete job finished in approximately 3 minutes and 9 seconds.

Figure 2. Successful end-to-end Databricks job run: Bronze → Silver → Gold.
Automated testing with PyTest :
The project includes PyTest tests that run with local PySpark.
The tests cover:
– Bronze source-name validation and path generation,
– Silver casting, cleaning and data-quality expectations,
– Gold derived values and aggregations.
GitHub Actions sets up Python and Java, installs the test dependencies, and runs the test suite before the production deployment job can proceed.

Figure 3. GitHub Actions test job completing successfully.
CI/CD with Databricks Asset Bundles :
Databricks Asset Bundles define the project’s jobs, pipelines, variables, and deployment targets in YAML. The GitHub Actions workflow runs tests for pushes to dev and main and for pull requests targeting those branches. On a push to main, and only after the test job succeeds, the workflow validates the production bundle, deploys it, and triggers Hotel_Bookings_BSG_Job.

Figure 4. Successful production workflow: bundle validation, deployment, and orchestration-job execution.
Resulting datasets in Unity Catalog :
After execution, the processed datasets are available through Unity Catalog. The captured catalog view shows the Silver schema and its tables, alongside the Gold schema and its three analytical tables.

Figure 5. Silver and Gold schemas and tables visible in Unity Catalog.
Running the project :
The test suite can be run from the k_hotels_dab directory after installing Java 17, Python 3.12, pytest, and PySpark 3.5.0:
python -m pip install pytest pyspark==3.5.0
python -m pytest tests/ -v
To validate and deploy the production bundle and run the orchestration job:
databricks bundle validate -t prod
databricks bundle deploy -t prod
databricks bundle run -t prod Hotel_Bookings_BSG_Job
Conclusion:
This project brought together ingestion, transformation, data-quality validation, automated testing, deployment, and orchestration in one Databricks workflow. The dev and prod bundle targets in this implementation point to the same Databricks workspace, so they provide separate deployment configurations rather than fully isolated environments.
You may also like : Data Engineering for Growing Companies: How to Turn Scattered Data Into Business Intelligence
Written By
