Databricks AI/BI Series: A Technical Overview of AI/BI Genie
Introduction
Previously, we wrote about how the AI/BI Dashboards from Databricks caught our eye as an important development at the 2024 Databricks’ Data + AI Summit. Also, we’ve written about the data governance capabilities offered by Databricks’ Unity Catalog here (UC: A Practical Exploration) and also here (UC: Enhancing Security and Productivity).
Today, we will be talking about AI/BI Genie, the LLM & Unity Catalog powered sibling to AI/BI Dashboards that also made a splash at the summit. Genie is Databricks’ way of connecting your Unity Catalog’s data to your business questions, leveraging natural language in a ChatGPT-like experience, providing insight beyond what dashboards have to offer. The best part? It is affordable and part of the built-in Databricks features.
Here is a concise, but wide and deep technical dive into this new offering. We will cover multiple areas using a question-based approach.
What’s Wrong With Dashboards?
Dashboards (can be) fantastic, and are not going away anytime soon, however, there are at least three different scenarios where they are not enough:
There are times when dashboards are not sufficient and business users resort to reaching out to analysts for time-consuming ad-hoc requests.
Other times, these business users export dashboard data into Excel, where they create siloed business knowledge in a non-governed and often non-reproducible way.
Lastly, on other occasions, the data has never been available inside of a dashboard, even though the right data exists somewhere. Perhaps no one has sought to answer the question at hand with a dashboard or worse, the business need has been identified, but the team doesn’t have enough time to see it through.
All the scenarios above have 2 common threads: (1) There is a business need not addressed and (2) there are time constraints. Enters Databricks, offering to alleviate some of these burdens with AI/BI Genie.
Takeaway: While dashboards provide a visual overview, they often fall short when exploring new data and conducting ad-hoc analysis, pushing users towards time-consuming manual data manipulation and ungovernable shortcuts.
What It Is and What Is It Not?
Genie is not ChatGPT + your data. In fact, It is not intended:
As a black box solution. At the very least, for it to work as intended, training through conversations and good Unity Catalog metadata is required.
To be a general large language model. It is not meant to help you write a business proposal or to help you code a pipeline.
Here is what it actually is intended for:
Answering business-specific questions by incorporating user queries, instructions (user-specific training), and Unity Catalog metadata.
Empowering business users to self-serve analytics through the use of contextually-aware natural language in a governed environment.
Go beyond the dashboard without needing to be a technical person
Takeaway: AI/BI Genie is designed to enhance business-specific decision-making through tailored conversations that connect non-technical users to the answers they need that aren’t being addressed by dashboards today.
What Model Is It Using?
Instead of relying on a single model, like for example, GPT or DBRX, it actually relies on multiple models. As a compound AI based offering, leveraging multiple models allows Databricks to pick the right model(s) for each task that needs to be completed. Different tasks can be SQL generation, visualization creation, summarization, deciding when to ask clarification questions, etc. It is dynamically evaluating which one is better for the task(s) at hand. One conversation could leverage multiple tasks and perhaps, one prompt could even leverage multiple models.
Takeaway: Leveraging a composite AI architecture, AI/BI Genie dynamically selects the most appropriate model for each query, ensuring both flexibility and precision in responses.
How Much Does It Cost?
When I started using it, my assumption was that you pay for both the SQL Serverless Warehouse you connect to your Genie space, as well as per tokens in and out. To my surprise, at least as of now, for the web based interface, you only pay for compute. In the future, we might see the release of an API for Genie, though I would expect that the way that billing is handled for the API usage will be different.
Takeaway: AI/BI Genie proves cost-effective by charging only for compute usage.
What are the different type of Instructions, its purposes and differences?
There are currently 3 different types of “Instructions”. They help train and shape how Genie answers your questions and are key to successfully building out a Genie space and answering questions to your business needs. Here is an explanation of what each of them is used for.
1. General Instructions: Use this for general preferences on how you’d like a Genie space to behave. For example, you might pass instructions to always format numbers with 2 decimal places, to not be very wordy in response, or hints as to abbreviation you might use and their meaning. General instructions are also good for context/semantics that are applicable across multiple tables/columns. One important call-out is that as of the time of writing, Genie does sometimes ignore some general instructions.
2. Example SQL Queries: SQL Queries are SQL scripts that provide starting points for Genie to provide certain answers. There are two ways you can configure them.
One method is to leverage Genie’s chat interface to build some reporting, tailoring it through one or many prompts. Once you get a desired answer, you can store the underlying SQL and name it with the question that it answers.
The other method is simply copy-pasting a pre-built SQL script into the Genie’s configuration which, when paired up with a corresponding label (question), Genie will be able to answer that and closely related questions.
3. Trusted Assets: They are user-defined table functions that are used to answer commonly asked questions in a highly repeatable and consistent format. They are not used to address any other questions, unlike general instructions or SQL queries. With Trusted Assets, the only room for interpretation you are giving Genie is determining whether it should use a Trusted Asset to answer a question, as well as the parameters you pass to it, such as a state or date range you want to evaluate.
Takeaway: AI/BI Genie provides multiple tools to help you provide the training it needs to deliver business value to end-users.
How Is Access To The Data Managed?
There are two different access control options, depending on your industry, business needs, and data governance policies.
Unity Catalog Permissions: Governance that your organization has set up at the Unity Catalog level will be enforced. Row level security is enforced as well. If a user wouldn’t be able to query the data outside of a Genie space, they won’t be able to query it through Genie either, even though it is part of the established tables for the Genie space. They will get a warning instead if they are missing access.
Embedded Credentials: If the Genie space developer has access to the data and it is available inside of the Genie space, then access is granted to the business users that are authorized to use the space. The end-user might not be able to query a table outside of the Genie space, but inside of it, it has full access to anything in the space. Please be aware, this is in private preview, and you will need to talk to your account team to get access to this option.
If you are at a company with very mature data governance, you will find that the first method is better fitted for business alignment. Other companies with a less mature or simplified governance environment might benefit from the embedded credentials approach. The same can also be said for users coming from the BI world as it mirrors the most common practices there as well. For example, a Genie space could be created for operations, another for middle management, while a Genie space could be created for the C-Suite.
Takeaways: There are two methods of access control: Unity Catalog permissions for stringent data governance and embedded credentials (Genie space level access) for more flexible data access.
How Is Performance Monitored?
By going to the “Monitoring” tab, you are able to see a history of queries generated from questions, along with whether they were deemed as good or bad answers (upvoted or downvoted) by those utilizing Genie.
Some practical ways to leverage this information:
For upvoted results, you might consider leveraging the different instruction options to ensure the quality of the results remain consistently good or simply call it a day, knowing Genie is working as intended.
For downvoted results, evaluate whether bad results stem from wrong data availability, bad metadata in the Unity Catalog, or simply Genie getting it wrong, and make an action with that information.
Consider evaluating the prompts users are making and consider whether Genie is the best delivery method, or whether you spot a common theme that might be better addressed through other methods, such as dashboards.
Takeaway: Monitoring AI/BI Genie’s performance allows users to refine and optimize its functionality, ensuring that the insights provided align with business objectives and enhance decision-making efficiency.
Should My Organization Be Using It?
We love the vision for AI/BI Genie. That said, at this moment, getting Genie to give consistent & accurate answers requires a lot of training by technical users that are also SMEs before non-technical users can leverage a Genie space. For some organizations, this in itself might diminish the short term ROI expected from Genie and make Genie a no-go for now. In the future, the aspiration is that training will require less technical knowledge to ultimately help Genie become truly accessible to almost anyone, regardless of technical expertise. Additionally, it is currently in Public Preview and I do not believe it will be truly GA until at least a year from now.
Some of the weakness areas we are keeping our eyes on:
Instructions still have many quirks that need to be worked out in terms of both bugs and making the behavior of the instructions more predictable when shaping results.
Visualizations are a hit-or-miss, with more hits than misses, especially as it relates to labels and customization.
I expect that there will be many changes between what we have available today and one year from now, and building fully-decked out Genie spaces that might have to be re-made in the future is a significant risk.
There is currently no version control or cloning, which is critically needed for this to be a production level product.
The above said, let me be clear about something: if you are set on using LLMs with your data, it will be hard to find something better than Genie in the market right now. Also, building in-house is unlikely to yield a worthwhile ROI, when the Databricks team has already done so much of the hard work. So if that sounds like where you are right now, then by all means start using Genie to help your business while sharing your feedback with the Genie team to continue to make the product and your experience better.
Takeaway: Training Genie requires technical resources for it to be successful with non-technical audiences. While it is still in public preview and with many imperfections, Genie is the best option we have in the market for an LLM-like self-service analytics experience.
Closing Thoughts…
AI/BI Genie simplifies access to data analytics beyond the dashboard, making it easier for business users to get the answers they need without needing to be technical. Some additional things to remember:
It follows a compound AI framework to accurately process various types of queries.
Leverages Databricks' Unity Catalog for secure and straightforward data access.
Translates natural language queries into data insights.
Only charges for compute usage, keeping costs predictable.
Learns from user interactions to enhance its accuracy and relevance.
Requires thorough training to improve accuracy and usability, as it is not intended to be a black-box.
Takeaway: Much like an eager intern, Genie learns from every interaction, enhancing its ability to translate natural language into precise data insights. With each query, it grows more adept, delivering increasing value as it matures within its role in your organization.
I encourage you to check out my conversation with Chao below, as well as our article on AI/BI Dashboards here.
Conversation with Databricks Product Team
For a more in-depth look at these features, check out this 10’ video: