YAVASA hotel management software dashboard for managing room bookings, guest records, billing, GST invoices, and real-time room availability.

Building an End-to-End Project on Databricks 

Share this post on:

Introduction : 

I built this project to process hotel-booking CSV data on Databricks using a three-layer Medallion Architecture. The pipeline incrementally ingests raw files into Bronze, applies cleaning and data-quality checks in Silver, and creates analytical datasets in Gold. I also added automated tests and a deployment workflow so that code changes can be validated and deployed through GitHub Actions. 

Data flow: Bronze, Silver, and Gold : 

Bronze – Incremental ingestion : A Databricks job runs SrcParameters.ipynb to provide the configured source names, then runs 01_bronze_ingestion.ipynb for each source. The ingestion code uses Auto Loader to read CSV files and append the incoming data as Delta. Schema, checkpoint, and data locations are maintained for the configured hotel_bookings source. 

Figure 1. Databricks notebooks for source configuration (SrcParameters) and Bronze ingestion (01bronzeingestion).

Figure 1. Databricks notebooks for source configuration (SrcParameters) and Bronze ingestion (01bronzeingestion). 

Silver – Cleaning and validation : The Silver pipeline reads Bronze data, casts booking columns to their intended types, removes rescued data, and adds a modification timestamp. Data-quality expectations drop records with invalid cancellation flags, guest counts, or ADR values. The resulting table is silver.silver_hotel_bookings. 

Gold – Analytics : The Gold pipeline derives arrival dates, total nights, and estimated room revenue. It creates three analytical tables: gold_hotel_booking_summary, gold_monthly_booking_trends, and gold_monthly_room_revenue. 

Orchestrating the complete pipeline : 

A parent job named Hotel Bookings BSG Job coordinates the three stages in dependency order: Bronze ingestion, Silver transformation, and Gold transformation. 
 
In the captured run, all three tasks succeeded and the complete job finished in approximately 3 minutes and 9 seconds. 

Figure 2. Successful end-to-end Databricks job run: Bronze → Silver → Gold. 

Figure 2. Successful end-to-end Databricks job run: Bronze → Silver → Gold. 

Automated testing with PyTest : 

The project includes PyTest tests that run with local PySpark. 
The tests cover: 
    – Bronze source-name validation and path generation, 
    – Silver casting, cleaning and data-quality expectations, 
    – Gold derived values and aggregations.  

GitHub Actions sets up Python and Java, installs the test dependencies, and runs the test suite before the production deployment job can proceed. 

Figure 3. GitHub Actions test job completing successfully. 

CI/CD with Databricks Asset Bundles : 

Databricks Asset Bundles define the project’s jobs, pipelines, variables, and deployment targets in YAML. The GitHub Actions workflow runs tests for pushes to dev and main and for pull requests targeting those branches. On a push to main, and only after the test job succeeds, the workflow validates the production bundle, deploys it, and triggers Hotel_Bookings_BSG_Job. 

Figure 4. Successful production workflow: bundle validation, deployment, and orchestration-job execution. 

Resulting datasets in Unity Catalog : 

After execution, the processed datasets are available through Unity Catalog. The captured catalog view shows the Silver schema and its tables, alongside the Gold schema and its three analytical tables. 

Figure 5. Silver and Gold schemas and tables visible in Unity Catalog. 

Running the project : 

The test suite can be run from the k_hotels_dab directory after installing Java 17, Python 3.12, pytest, and PySpark 3.5.0: 

python -m pip install pytest pyspark==3.5.0 
python -m pytest tests/ -v 

To validate and deploy the production bundle and run the orchestration job: 

databricks bundle validate -t prod 
databricks bundle deploy -t prod 
databricks bundle run -t prod Hotel_Bookings_BSG_Job 

Conclusion: 

This project brought together ingestion, transformation, data-quality validation, automated testing, deployment, and orchestration in one Databricks workflow. The dev and prod bundle targets in this implementation point to the same Databricks workspace, so they provide separate deployment configurations rather than fully isolated environments. 

You may also like : Data Engineering for Growing Companies: How to Turn Scattered Data Into Business Intelligence 

Written By

Databricks Registered Partner

Bronze Partner Network
Avatar photo

Piyush Solanki

PHP Tech Lead & Backend Architect

10+ years experience
UK market specialist
Global brands & SMEs
Full-stack expertise

Core Technologies

PHP 95%
MySQL 90%
WordPress 92%
AWS 88%
  • Backend: PHP, MySQL, CodeIgniter, Laravel
  • CMS: WordPress customization & plugin development
  • APIs: RESTful design, microservices architecture
  • Frontend: React, TypeScript, modern admin panels
  • Cloud: AWS S3, Linux deployments
  • Integrations: Stripe, SMS/OTP gateways
  • Finance: Secure payment systems & compliance
  • Hospitality: Booking & reservation systems
  • Retail: E-commerce platforms & inventory
  • Consulting: Custom business solutions
  • Food Services: Delivery & ordering systems
  • Modernizing legacy systems for scalability
  • Building secure, high-performance products
  • Mobile-first API development
  • Agile collaboration with cross-functional teams
  • Focus on operational efficiency & innovation

Piyush Solanki is a seasoned PHP Tech Lead with 10+ years of experience architecting and delivering scalable web and mobile backend solutions for global brands and fast-growing SMEs.

He specializes in PHP, MySQL, CodeIgniter, WordPress, and custom API development, helping businesses modernize legacy systems and launch secure, high-performance digital products.

He collaborates closely with mobile teams building Android & iOS apps, developing RESTful APIs, cloud integrations, and secure payment systems. With extensive experience in the UK market and across multiple sectors, Piyush Solanki is passionate about helping SMEs scale technology teams and accelerate innovation through backend excellence.

    Reach Out Us


    Your name

    Your email

    Subject

    Your message