{"id":5545,"date":"2026-11-20T06:46:33","date_gmt":"2026-11-20T06:46:33","guid":{"rendered":"https:\/\/www.200oksolutions.com\/blog\/?p=5545"},"modified":"2026-11-20T06:46:34","modified_gmt":"2026-11-20T06:46:34","slug":"building-end-to-end-project-databricks","status":"publish","type":"post","link":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/","title":{"rendered":"Building an End-to-End Project on Databricks\u00a0"},"content":{"rendered":"\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe title=\"Building an End-to-End Project on Databricks\u00a0I 200OK SOLUTIONS\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/QOF0SsFRQQk?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Introduction :<\/strong>\u00a0<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I built this project to process hotel-booking CSV data on Databricks using a three-layer Medallion Architecture. The pipeline incrementally ingests raw files into Bronze, applies cleaning and data-quality checks in Silver, and creates analytical datasets in Gold. I also added automated tests and a deployment workflow so that code changes can be validated and deployed through GitHub Actions.\u00a0<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Data flow: Bronze, Silver, and Gold :<\/strong>\u00a0<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Bronze &#8211; Incremental ingestion : <\/strong>A Databricks job runs SrcParameters.ipynb to provide the configured source names, then runs 01_bronze_ingestion.ipynb for each source. The ingestion code uses Auto Loader to read CSV files and append the incoming data as Delta. Schema, checkpoint, and data locations are maintained for the configured hotel_bookings source.\u00a0<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img fetchpriority=\"high\" decoding=\"async\" width=\"996\" height=\"207\" src=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-2.png\" alt=\"Figure 1. Databricks notebooks for source configuration (SrcParameters) and Bronze ingestion (01bronzeingestion).\" class=\"wp-image-5546\" style=\"aspect-ratio:4.811594202898551;width:664px;height:auto\" srcset=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-2.png 996w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-2-300x62.png 300w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-2-765x159.png 765w\" sizes=\"(max-width: 996px) 100vw, 996px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 1. Databricks notebooks for source configuration (SrcParameters) and Bronze ingestion (01bronzeingestion).<\/em>\u00a0<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Silver &#8211; Cleaning and validation :<\/strong> The Silver pipeline reads Bronze data, casts booking columns to their intended types, removes rescued data, and adds a modification timestamp. Data-quality expectations drop records with invalid cancellation flags, guest counts, or ADR values. The resulting table is silver.silver_hotel_bookings.\u00a0<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Gold \u2013 Analytics :<\/strong> The Gold pipeline derives arrival dates, total nights, and estimated room revenue. It creates three analytical tables: gold_hotel_booking_summary, gold_monthly_booking_trends, and gold_monthly_room_revenue.\u00a0<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Orchestrating the complete pipeline :<\/strong>\u00a0<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A parent job named Hotel Bookings BSG Job coordinates the three stages in dependency order: Bronze ingestion, Silver transformation, and Gold transformation.\u00a0<br>\u00a0<br>In the captured run, all three tasks succeeded and the complete job finished in approximately 3 minutes and 9 seconds.\u00a0<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img decoding=\"async\" width=\"1024\" height=\"599\" src=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-6-1024x599.png\" alt=\"Figure 2. Successful end-to-end Databricks job run: Bronze \u2192 Silver \u2192 Gold.\u00a0\n\" class=\"wp-image-5550\" style=\"aspect-ratio:1.7111650485436893;width:705px;height:auto\" srcset=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-6-1024x599.png 1024w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-6-300x175.png 300w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-6-768x449.png 768w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-6.png 1057w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 2. Successful end-to-end Databricks job run: Bronze \u2192 Silver \u2192 Gold.<\/em>\u00a0<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Automated testing with PyTest :<\/strong>\u00a0<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The project includes PyTest tests that run with local PySpark.\u00a0<br><strong>The tests cover:\u00a0<br><\/strong>\u00a0\u00a0\u00a0 &#8211; Bronze source-name validation and path generation,\u00a0<br>\u00a0\u00a0\u00a0 &#8211; Silver casting, cleaning and data-quality expectations,\u00a0<br>\u00a0\u00a0\u00a0 &#8211; Gold derived values and aggregations.\u00a0\u00a0<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">GitHub Actions sets up Python and Java, installs the test dependencies, and runs the test suite before the production deployment job can proceed.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" width=\"957\" height=\"433\" src=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-4.png\" alt=\"\" class=\"wp-image-5548\" style=\"aspect-ratio:2.207612456747405;width:638px;height:auto\" srcset=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-4.png 957w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-4-767x347.png 767w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-4-300x136.png 300w\" sizes=\"(max-width: 957px) 100vw, 957px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 3. GitHub Actions test job completing successfully.<\/em>\u00a0<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>CI\/CD with Databricks Asset Bundles :<\/strong>\u00a0<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Databricks Asset Bundles define the project&#8217;s jobs, pipelines, variables, and deployment targets in YAML. The GitHub Actions workflow runs tests for pushes to dev and main and for pull requests targeting those branches. On a push to main, and only after the test job succeeds, the workflow validates the production bundle, deploys it, and triggers Hotel_Bookings_BSG_Job.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"957\" height=\"394\" src=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-3.png\" alt=\"\" class=\"wp-image-5547\" style=\"aspect-ratio:2.425855513307985;width:638px;height:auto\" srcset=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-3.png 957w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-3-300x124.png 300w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-3-768x316.png 768w\" sizes=\"(max-width: 957px) 100vw, 957px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 4. Successful production workflow: bundle validation, deployment, and orchestration-job execution.<\/em>&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Resulting datasets in Unity Catalog :<\/strong>\u00a0<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">After execution, the processed datasets are available through Unity Catalog. The captured catalog view shows the Silver schema and its tables, alongside the Gold schema and its three analytical tables.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"967\" height=\"562\" src=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-5.png\" alt=\"\" class=\"wp-image-5549\" style=\"aspect-ratio:1.72;width:645px;height:auto\" srcset=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-5.png 967w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-5-300x174.png 300w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/image-5-767x446.png 767w\" sizes=\"(max-width: 967px) 100vw, 967px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\"><em>Figure 5. Silver and Gold schemas and tables visible in Unity Catalog.<\/em>\u00a0<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Running the project :<\/strong>\u00a0<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The test suite can be run from the k_hotels_dab directory after installing Java 17, Python 3.12, pytest, and PySpark 3.5.0:&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>python -m pip install pytest pyspark==3.5.0&nbsp;<br>python -m pytest tests\/ -v&nbsp;<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">To validate and deploy the production bundle and run the orchestration job:&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>databricks bundle validate -t prod&nbsp;<br>databricks bundle deploy -t prod&nbsp;<br>databricks bundle run -t prod Hotel_Bookings_BSG_Job&nbsp;<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Conclusion:<\/strong>&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This project brought together ingestion, transformation, data-quality validation, automated testing, deployment, and orchestration in one Databricks workflow. The dev and prod bundle targets in this implementation point to the same Databricks workspace, so they provide separate deployment configurations rather than fully isolated environments.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You may also like :\u00a0<a href=\"https:\/\/www.200oksolutions.com\/blog\/data-engineering-for-growing-companies-how-to-turn-scattered-data-into-business-intelligence\/\" target=\"_blank\" rel=\"noreferrer noopener\">Data Engineering for Growing Companies: How to Turn Scattered Data Into Business Intelligence<\/a>\u00a0<\/p>\n\n\n<div class=\"is-default-size wp-block-site-logo\"><a href=\"https:\/\/www.200oksolutions.com\/blog\/\" class=\"custom-logo-link light-mode-logo\" rel=\"home\"><img loading=\"lazy\" decoding=\"async\" width=\"484\" height=\"191\" src=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/01\/cropped-200ok_logo.png\" class=\"custom-logo\" alt=\"200OK Solutions Blog\" srcset=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/01\/cropped-200ok_logo.png 484w, https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/01\/cropped-200ok_logo-300x118.png 300w\" sizes=\"(max-width: 484px) 100vw, 484px\" \/><\/a><\/div>","protected":false},"excerpt":{"rendered":"<p>Introduction :\u00a0 I built this project to process hotel-booking CSV data on Databricks using a&hellip;<\/p>\n","protected":false},"author":5,"featured_media":5551,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2341],"tags":[2604,2601,2606,2598,1438,2605,1370,2599,1553,1217,2600,2602,2603,2521],"class_list":["post-5545","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-engineering","tag-auto-loader","tag-automated-testing","tag-bronze-silver-gold-architecture","tag-ci-cd-pipeline","tag-data-engineering","tag-data-pipeline","tag-databricks","tag-databricks-asset-bundles","tag-delta-lake","tag-github-actions","tag-medallion-architecture","tag-pyspark","tag-pytest","tag-unity-catalog"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Building an End-to-End Project on Databricks<\/title>\n<meta name=\"description\" content=\"Build an end-to-end Databricks project using Medallion Architecture, PySpark, Delta Lake, automated testing, and GitHub Actions CI\/CD.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Building an End-to-End Project on Databricks\" \/>\n<meta property=\"og:description\" content=\"Build an end-to-end Databricks project using Medallion Architecture, PySpark, Delta Lake, automated testing, and GitHub Actions CI\/CD.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/\" \/>\n<meta property=\"og:site_name\" content=\"200OK Solutions Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-20T06:46:33+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-11-20T06:46:34+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2240\" \/>\n\t<meta property=\"og:image:height\" content=\"1260\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Piyush Solanki\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Piyush Solanki\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":[\"Article\",\"BlogPosting\"],\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/\"},\"author\":{\"name\":\"Piyush Solanki\",\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/person\/e07f6b8e3c9a90ce7b3b09427d26155e\"},\"headline\":\"Building an End-to-End Project on Databricks\u00a0\",\"datePublished\":\"2026-11-20T06:46:33+00:00\",\"dateModified\":\"2026-11-20T06:46:34+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/\"},\"wordCount\":565,\"publisher\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png\",\"keywords\":[\"Auto Loader\",\"Automated Testing\",\"Bronze Silver Gold Architecture\",\"CI\/CD Pipeline\",\"Data Engineering\",\"Data Pipeline\",\"Databricks\",\"Databricks Asset Bundles\",\"Delta Lake\",\"github actions\",\"Medallion Architecture\",\"PySpark\",\"PyTest\",\"Unity Catalog\"],\"articleSection\":[\"Data Engineering\"],\"inLanguage\":\"en-US\"},{\"@type\":[\"WebPage\",\"SearchResultsPage\"],\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/\",\"url\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\",\"name\":\"Building an End-to-End Project on Databricks\",\"isPartOf\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png\",\"datePublished\":\"2026-11-20T06:46:33+00:00\",\"dateModified\":\"2026-11-20T06:46:34+00:00\",\"description\":\"Build an end-to-end Databricks project using Medallion Architecture, PySpark, Delta Lake, automated testing, and GitHub Actions CI\/CD.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#primaryimage\",\"url\":\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png\",\"contentUrl\":\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png\",\"width\":2240,\"height\":1260,\"caption\":\"YAVASA hotel management software dashboard for managing room bookings, guest records, billing, GST invoices, and real-time room availability.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.200oksolutions.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Blog >Data Engineering > Building an End-to-End Project on Databricks\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#website\",\"url\":\"https:\/\/www.200oksolutions.com\/blog\/\",\"name\":\"Web Development, Software, and App Blog | 200OK Solutions\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.200oksolutions.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#organization\",\"name\":\"200OK Solutions\",\"url\":\"https:\/\/www.200oksolutions.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2025\/09\/200ok_logo-CGzMrWDu.png\",\"contentUrl\":\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2025\/09\/200ok_logo-CGzMrWDu.png\",\"width\":500,\"height\":191,\"caption\":\"200OK Solutions\"},\"image\":{\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.instagram.com\/200ok_solutions\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/person\/e07f6b8e3c9a90ce7b3b09427d26155e\",\"name\":\"Piyush Solanki\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/05\/cropped-piyush-solanki-96x96.jpg\",\"contentUrl\":\"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/05\/cropped-piyush-solanki-96x96.jpg\",\"caption\":\"Piyush Solanki\"},\"description\":\"Piyush is a seasoned PHP Tech Lead with 10+ years of experience architecting and delivering scalable web and mobile backend solutions for global brands and fast-growing SMEs. He specializes in PHP, MySQL, CodeIgniter, WordPress, and custom API development, helping businesses modernize legacy systems and launch secure, high-performance digital products. He collaborates closely with mobile teams building Android &amp; iOS apps , developing RESTful APIs, cloud integrations, and secure payment systems using platforms like Stripe, AWS S3, and OTP\/SMS gateways. His work extends across CMS customization, microservices-ready backend architectures, and smooth product deployments across Linux and cloud-based environments. Piyush also has a strong understanding of modern front-end technologies such as React and TypeScript, enabling him to contribute to full-stack development workflows and advanced admin panels. With a successful delivery track record in the UK market and experience building digital products for sectors like finance, hospitality, retail, consulting, and food services, Piyush is passionate about helping SMEs scale technology teams, improve operational efficiency, and accelerate innovation through backend excellence and digital tools.\",\"url\":\"https:\/\/www.200oksolutions.com\/blog\/author\/piyush\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Building an End-to-End Project on Databricks","description":"Build an end-to-end Databricks project using Medallion Architecture, PySpark, Delta Lake, automated testing, and GitHub Actions CI\/CD.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/","og_locale":"en_US","og_type":"article","og_title":"Building an End-to-End Project on Databricks","og_description":"Build an end-to-end Databricks project using Medallion Architecture, PySpark, Delta Lake, automated testing, and GitHub Actions CI\/CD.","og_url":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/","og_site_name":"200OK Solutions Blog","article_published_time":"2026-11-20T06:46:33+00:00","article_modified_time":"2026-11-20T06:46:34+00:00","og_image":[{"width":2240,"height":1260,"url":"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png","type":"image\/png"}],"author":"Piyush Solanki","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Piyush Solanki","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["Article","BlogPosting"],"@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#article","isPartOf":{"@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/"},"author":{"name":"Piyush Solanki","@id":"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/person\/e07f6b8e3c9a90ce7b3b09427d26155e"},"headline":"Building an End-to-End Project on Databricks\u00a0","datePublished":"2026-11-20T06:46:33+00:00","dateModified":"2026-11-20T06:46:34+00:00","mainEntityOfPage":{"@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/"},"wordCount":565,"publisher":{"@id":"https:\/\/www.200oksolutions.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#primaryimage"},"thumbnailUrl":"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png","keywords":["Auto Loader","Automated Testing","Bronze Silver Gold Architecture","CI\/CD Pipeline","Data Engineering","Data Pipeline","Databricks","Databricks Asset Bundles","Delta Lake","github actions","Medallion Architecture","PySpark","PyTest","Unity Catalog"],"articleSection":["Data Engineering"],"inLanguage":"en-US"},{"@type":["WebPage","SearchResultsPage"],"@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks\/","url":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks","name":"Building an End-to-End Project on Databricks","isPartOf":{"@id":"https:\/\/www.200oksolutions.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#primaryimage"},"image":{"@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#primaryimage"},"thumbnailUrl":"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png","datePublished":"2026-11-20T06:46:33+00:00","dateModified":"2026-11-20T06:46:34+00:00","description":"Build an end-to-end Databricks project using Medallion Architecture, PySpark, Delta Lake, automated testing, and GitHub Actions CI\/CD.","breadcrumb":{"@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#primaryimage","url":"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png","contentUrl":"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/11\/How-to-Migrate-to-Cloud-Native-Architectures-Using-Microservices-1-2026-11-20T110920.090.png","width":2240,"height":1260,"caption":"YAVASA hotel management software dashboard for managing room bookings, guest records, billing, GST invoices, and real-time room availability."},{"@type":"BreadcrumbList","@id":"https:\/\/www.200oksolutions.com\/blog\/building-end-to-end-project-databricks#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.200oksolutions.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Blog >Data Engineering > Building an End-to-End Project on Databricks"}]},{"@type":"WebSite","@id":"https:\/\/www.200oksolutions.com\/blog\/#website","url":"https:\/\/www.200oksolutions.com\/blog\/","name":"Web Development, Software, and App Blog | 200OK Solutions","description":"","publisher":{"@id":"https:\/\/www.200oksolutions.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.200oksolutions.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.200oksolutions.com\/blog\/#organization","name":"200OK Solutions","url":"https:\/\/www.200oksolutions.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2025\/09\/200ok_logo-CGzMrWDu.png","contentUrl":"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2025\/09\/200ok_logo-CGzMrWDu.png","width":500,"height":191,"caption":"200OK Solutions"},"image":{"@id":"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.instagram.com\/200ok_solutions\/"]},{"@type":"Person","@id":"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/person\/e07f6b8e3c9a90ce7b3b09427d26155e","name":"Piyush Solanki","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.200oksolutions.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/05\/cropped-piyush-solanki-96x96.jpg","contentUrl":"https:\/\/www.200oksolutions.com\/blog\/wp-content\/uploads\/2026\/05\/cropped-piyush-solanki-96x96.jpg","caption":"Piyush Solanki"},"description":"Piyush is a seasoned PHP Tech Lead with 10+ years of experience architecting and delivering scalable web and mobile backend solutions for global brands and fast-growing SMEs. He specializes in PHP, MySQL, CodeIgniter, WordPress, and custom API development, helping businesses modernize legacy systems and launch secure, high-performance digital products. He collaborates closely with mobile teams building Android &amp; iOS apps , developing RESTful APIs, cloud integrations, and secure payment systems using platforms like Stripe, AWS S3, and OTP\/SMS gateways. His work extends across CMS customization, microservices-ready backend architectures, and smooth product deployments across Linux and cloud-based environments. Piyush also has a strong understanding of modern front-end technologies such as React and TypeScript, enabling him to contribute to full-stack development workflows and advanced admin panels. With a successful delivery track record in the UK market and experience building digital products for sectors like finance, hospitality, retail, consulting, and food services, Piyush is passionate about helping SMEs scale technology teams, improve operational efficiency, and accelerate innovation through backend excellence and digital tools.","url":"https:\/\/www.200oksolutions.com\/blog\/author\/piyush\/"}]}},"_links":{"self":[{"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/posts\/5545","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/comments?post=5545"}],"version-history":[{"count":2,"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/posts\/5545\/revisions"}],"predecessor-version":[{"id":5553,"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/posts\/5545\/revisions\/5553"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/media\/5551"}],"wp:attachment":[{"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/media?parent=5545"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/categories?post=5545"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.200oksolutions.com\/blog\/wp-json\/wp\/v2\/tags?post=5545"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}