Databricks Compute Types: A Performance & Cost Analysis

Here are practical Databricks compute type selection insights, based on some recent tests I did so you can save the most amount of money and/or have the desired performance.

Quick Notes On Tests

1. Classic Compute & SQL Serverless Warehouses were configured to be at similar price points to the $56 per hour of a SQL Serverless Warehouse, up or down roughly 10-15% depending on the set of tests.

2. My first set of tests contained 9 compute-intensive queries, including window functions on a flat table which included about 100B rows. The second set contained 10 compute-intensive queries, including window functions, where clauses, joins, and write operations on roughly 80B rows.

Recommendations For Interactive Workloads...

1. SQL Serverless Warehouses give you the best value on interactive SQL workloads. In terms of $, I would simply not use anything else. Best value for interactive SQL needs. In terms of speed, across 19 queries, it was a virtual split between SQL Serverless & "Serverless", so provided you are not under-sizing the SQL Serverless Warehouse, this is a great option.

2. Serverless, while at a discounted price, like it is right now, through the end of April 2025, has a lot of value. In terms of costs, the discounted rates set it at about average on my first set of tests, and 2nd best in the second set of tests that I ran. As I already mentioned, speed-wise, it was a virtual split with SQL Serverless. It beat SQL Serverless in the overall performance on both, but marginally, and upsizing the SQL Serverless Warehouse would have likely made Serverless 2nd place.

3.  Classic Compute, within the context of SQL and interactive workloads, was my least favorite option. Cost-wise, it was generally down the middle and included a lot of trial and error that was costly for my configuration testing. Performance-wise, I could have potentially just given it a lot more workers or sized up the computer hunting for marginal gains at the exponential cost of my time. You want your developers to build, not be spending time trying to figure out the best configurations WHILE still developing the code itself, especially during the early trial & error periods of development.

For Workflows/Jobs

1. Serverless Jobs, at a discounted rate, are generally one of the more affordable paths to take at this point in time. Price-wise, it was generally in the top 3 affordability out of about 8 configurations in my first set of tests, and number 1 in my second set with 5 configurations. In terms of speed, see my previous notes above, with the added note that if you want the fastest performance, subject to the hourly rate limits on Serverless not being crossed, Serverless is a good choice.

2. SQL Serverless Warehouses provide the best value for SQL jobs, though the gaps with Serverless at the discounted rate are marginal and marginal with some of the better classic compute configurations that I tried. If I was trying to have near absolute control of costs while being able to try out configurations easily & have solid performance, this would be my go-to.

3. Classic Compute in jobs was generally affordable, and very close to the Serverless compute (at a discount) and SQL Serverless cost points. The best performance generally came from trusting autoscale with a higher number of max workers, in addition to Photon.

Other Important Notes

1. Photon is generally worth it. In my first set of tests, Photon made everything go faster vs non-photon. The price did go up, but in a way that justifies the increased speed. In the second set of tests, Photon made everything go faster AND considerably cheaper. I will add, that I wasn't a fan of Photon, but it is actually good most of the time now.

2. Autoscaling is worth it for performance, and likely worth it for costs. Autoscaling generally made performance faster while maintaining price parity with fixed when paired up with Photon vs a fixed with Photon test.

3. A Note On Interactive Serverless: I mentioned discounted rates on Serverless a few times and that is intentional. At the listed price, I would personally feel comfortable using Serverless interactive on smaller, everyday compute OR if my team couldn't afford the time tinkering with compute configurations OR was less experienced with Databricks. For larger-scale, SQL interactive workloads, it would not be my first choice. SQL Serverless Warehouses would be.
   I know the Databricks team is working on some upcoming performance improvements of Serverless, which should continue to improve this side of their offering. Two of the queries that I ran in particular had edge cases that the Databricks team was already planning on addressing in upcoming releases. I will be re-running these tests on Serverless every month until April to measure performance over time.

4. A Note On Serverless Jobs: Similar to the above, but a little bit of a different take. I would still consider serverless jobs as an option even without the discount, especially if I wasn't too experienced with configuring compute. It would be more expensive, but I would balance that against the workloads running a lot faster on Serverless. 

Key Takeaways

1. SQL Serverless Warehouses are powerhouses & safe choices for getting the right cost, and performance while maintaining cost controls for both everyday queries & workflows/jobs, as long as you are comfortable with right-sizing compute yourself.

2. Jobs running on "Serverless" at the current discounted rate is a good deal in terms of performance & cost, without having to spend time trying to get the right configuration.

3. Photon & Autoscaling are musts when using Classic Compute for interactive & jobs SQL workloads. During testing, I generally had better performance & costs when setting a higher number for max autoscale vs when setting a lower limit. Even more surprising to me was Photon consistently having a positive impact on performance, when in the early days of Photon, the team at Kythera Labs wasn't a fan of it. It has come a long way!

Josue is a Technical Advisor to SunnyData, an Architect @Kythera Labs, as well as an advisor to Sigma. You can follow more of his content here on our blog, on LinkedIn.com/in/JosueBogran or on YouTube at YouTube.com/@JosueBogranChannel .