SITUATION

Innovation at our client's organization was being hampered by data silos, Information Technology/Operational Technology convergence challenges, insufficient data governance over asset-related data, and proprietary systems across the entire oil and gas supply chain. Additionally, their central data warehousing team struggled with governance and cost optimization for all Databricks instances, leading to a higher administrative burden. To address these issues, our client enlisted Everforth Apex to assess and implement a migration to Databricks Unity Catalog (UC), focusing on performance, cost, and benefits. As a trusted advisor and preferred UC implementation partner of Databricks, Everforth Apex was chosen as the sole system integrator to transition the client's enterprise Databricks workspaces to UC.

The team, assembled from both Everforth Apex and Databricks, collaborated seamlessly to provide our mutual client with top-tier consultation and engineering services. This partnership allowed us to approach the client with one voice and minimize risks during the implementation. Workspaces were prioritized based on their business value, high Databricks Unit (DBU) usage, and readiness of the client project team. ​​

Engagement Highlights: ​​

• Everforth Apex’s Data Solution Leader spearheaded the engagement with a hybrid US & Mexico-based team  ​​

• Unity Catalog eXtensions (UCX) assessments identified common themes​ across workspaces​

• Infrastructure and data engineering workshops laid the groundwork for UC onboarding​​

• Multiple business lines (Commercial, Upstream, Midstream) were engaged simultaneously​​

• A Data Mesh was created to safely and efficiently share information ​​

• Data pipelines were classified by source, target and type of processing​​

• UCX migration tools and custom notebooks assisted in table and workflow migration​​

• Automation was leveraged to onboard workspaces and develop re-usable patterns

RESULT

By leveraging industry best practices and our strong partnership with Databricks, the Everforth Apex team successfully migrated 21 workspaces, including 5,000 tables, 600 job workflows, and 25 models. The Everforth Apex team achieved 90% of attainable DBU targeted for migration, resulting in a 30% productivity gain for our client by centrally governing the platform and reducing the need for data architects. Additionally, our client matured their data and AI governance, reduced costs, and enabled scalable data processing, eliminating the barriers to innovation.