Winter Is So Over: Quick Guide on the 18 Big Announcements
Introduction
The Databricks Summit has ended, and if we were hyped before, after this particular one, we’re now 10x more excited. Before Summit, Databricks was already recognized as the most integrated and cost-effective platform. This Summit just proved once again that it is also the most future-proof platform!
While we read, research, learn and test all of them, here’s a roundup of the groundbreaking features and tools announced:
GENERATIVE AI
1. Mosaic AI Agent Framework
• Description: A toolkit and workflow designed to build high-quality retrieval-augmented generation (RAG) applications, helping users create advanced AI systems efficiently.
• Link: Mosaic AI Agent Framework
2. LLM Evaluation
• Description: Tools and methodologies for evaluating large language models, ensuring they meet desired performance and accuracy standards.
• Link: LLM Evaluation
3. Mosaic AI Model Training for Fine-tuning
• Description: Enables fine-tuning of open-source foundation models using private data, allowing customization of models to specific use cases and data sets.
• Link: Mosaic AI Model Training for Fine-tuning
4. Shutterstock ImageAI, Powered by Databricks
• Description: Leverages AI to create new high-resolution, photorealistic images that are safe for corporate use, enhancing creative capabilities.
• Link: Shutterstock ImageAI, Powered by Databricks
BUSINESS INTELLIGENCE
5. AI/BI Dashboards
• Description: Allows users to quickly build dashboards and visualizations using natural language, simplifying the data analysis process.
• Link: AI/BI Dashboards
6. AI/BI Genie
• Description: Provides a chat-like experience to ask questions of your data not answered by existing dashboards, making data exploration more intuitive.
• Link: AI/BI Genie
UNITY CATALOG
7. Unity Catalog Open Source
• Description: The industry’s only open catalog for data and AI, allowing seamless integration and governance of data assets.
• Link: Unity Catalog Open Source
8. Lakehouse Federation GA
• Description: Enables discovery, governance, and querying of data no matter where it lives, integrating diverse data sources into a single platform.
• Link: Lakehouse Federation GA
9. Lakehouse Monitoring GA
• Description: Monitors all data pipelines, from data to features to ML models, without needing additional tools, simplifying oversight.
• Link: Lakehouse Monitoring GA
DATA ENGINEERING
10. LakeFlow Connect
• Description: A no-code method of ingesting data from popular SaaS applications and databases, simplifying data integration tasks.
• Link: LakeFlow Connect
11. Serverless Compute
• Description: Offers hands-off, auto-optimized compute managed by Databricks, lowering customers’ total cost of ownership.
• Link: Serverless Compute
COLLABORATION
12. Clean Rooms
• Description: Facilitates secure collaboration with customers and partners on any cloud in a privacy-safe environment.
• Link: Clean Rooms
13. Delta Sharing D20
• Description: An open solution to securely share live data from your lakehouse to any computing platform, enhancing data accessibility.
• Link: Delta Sharing D20
DATA WAREHOUSING
14. AI Functions
• Description: Built-in SQL functions that allow users to apply AI directly on their data using SQL, streamlining AI integration.
• Link: AI Functions
15. Open Variant Data Type
• Description: Introduces a flexible data type to handle diverse data formats, enhancing data processing capabilities.
• Link: Open Variant Data Type
DATA FORMAT
16. Delta Lake 4.0
• Description: The biggest Delta release yet, offering a range of interoperability, reliability, and ease-of-use features to make data handling more convenient.
• Link: Delta Lake 4.0
OPTIMIZATIONS
17. Predictive Optimization GA
• Description: Provides autonomous optimizations based on data and usage patterns to offer the best price-performance.
• Link: Predictive Optimization GA
18. Liquid Clustering GA
• Description: Delivers a flexible data layout that autonomously adapts to data and usage patterns, optimizing performance.
• Link: Liquid Clustering GA
Closing
The DAIS Summit 2024 was a testament to Databricks’ commitment to innovation and excellence. The announcements made at the Summit highlight how Databricks continues to push the envelope in data engineering, machine learning, and analytics. These new features not only enhance the platform’s capabilities but also ensure that it remains the go-to solution for businesses aiming to harness the full potential of their data.
We at SunnyData are incredibly excited about these developments. Our heartfelt appreciation goes out to our customers, partners, and friends who joined us at the Summit. Your support and collaboration are what make these innovations possible. Here’s to a future of continued success and groundbreaking achievements in the world of data!
