When you upgrade to a modern version of SQL Server, there are some critical things you should do to help avoid any surprise SQL Server performance issues. I define a modern version of SQL Server as SQL Server 2016 or later. Doing the necessary planning and work beforehand is a crucial part of your efforts in preventing SQL Server upgrade performance issues.
SQL Server 2008 and SQL Server 2008 R2 fell out of Extended support from Microsoft on July 9, 2019. In addition, SQL Server 2014 fell out of Mainstream support on July 9, 2019. Finally, SQL Server 2012 fell out of Mainstream support on July 11, 2017. Because of this, I am seeing an increasing number of organizations that have been migrating to a modern version of SQL Server.
I see this as a positive development overall, since SQL Server 2016 and newer actually have many useful new features that make them much better products than their predecessors. Migrating to a modern version of SQL Server also usually means running on a current version of Windows Server. It also usually means new, faster hardware and storage, which is also very beneficial, as long as you choose your new hardware and storage wisely.
SQL Server Upgrade Performance Issues
Despite all of this, I have seen a decent number of cases where organizations have migrated from a legacy version of SQL Server to a modern version of SQL Server on new hardware and a new operating system, and then be unpleasantly surprised by SQL Server performance issues once they are in Production. How can these performance regressions be occurring, and what steps can you take to help prevent them?
The main culprit in most of these performance regressions is a combination of lack of knowledge, poor planning, and inadequate performance testing. Unlike legacy versions of SQL Server, modern versions of SQL Server have several important performance-related configuration options that you need to be aware of, understand, and actually test with your workload. Most people who run into performance regressions have done what I call a “blind migration” where they simply restore their databases from the older version to the new version of SQL Server, with no meaningful testing of the performance effects of these different configuration options.
With this method, maybe they get lucky preventing SQL Server upgrade performance issues, and maybe they don’t…
Key Configuration Options
So, what are these key configuration options that you need to be concerned with from a performance perspective? The most important ones include your database compatibility level, the cardinality estimator version that you are using, your database-scoped configuration options, and what trace flags you are using. Since SQL Server 2014, the database compatibility level affects the default cardinality estimator that the query optimizer will use. Since SQL Server 2016, the database compatibility level also controls other performance related behavior by default. I have written more about this subject here:
You have the ability override many of these database compatibility level-related changes with database scoped configuration options and query hints. There are actually a rather large number of different combinations of settings that you have to think about and test. So, what are you supposed to do?
Microsoft Database Experimentation Assistant
In my ideal scenario, you would use the free Microsoft Database Experimentation Assistant (DEA) to capture a relevant production workload. This involves taking a full production database backup, then capturing a production trace that covers representative high priority workloads. While this is going on, I would run some of my SQL Server Diagnostic Information Queries to get some baseline metrics from your legacy instance.
Once you have done that, you can then restore that backup to your new environment, and replay the production trace multiple times in your new environment. Each time you do this (which also includes a fresh restore from that original full production database backup), you will use a different combination of these key configuration settings. You have to make the database configuration/property changes after each restore, but before you replay the DEA trace.
Configuration Combinations to Test
The idea here is to see which combination of these configuration settings yields the best performance with your workload. Here are some relevant, likely combinations:
- Use the existing database compatibility level of the old version
- Use the existing database compatibility level of the old version and use the query optimizer hotfixes database-scoped configuration option
- Use the default native database compatibility level of the new version
- Use the default native database compatibility level of the new version and use the query optimizer hotfixes database-scoped configuration option
- Use the default native database compatibility level of the new version and use the legacy cardinality estimator database-scoped configuration option
- Use the default native database compatibility level of the new version and use the legacy cardinality estimator database-scoped configuration option and use the query optimizer hotfixes database-scoped configuration option
This level of DEA testing may not be practical if you have a large number of databases, but you should really try to do it on your most mission critical databases. Barring that, I would try to do as much testing of your most important stored procedures and queries as possible, using these different configuration settings.
Microsoft’s Recommended Upgrade Sequence
Finally, if no adequate testing is possible you can follow Microsoft’s recommended upgrade sequence (in your new production environment, after you go live), which is:
- Upgrade to the latest SQL Server version and keep the source (legacy) database compatibility level
- Enable Query Store, and let it collect a baseline of your workload
- Change the database compatibility level to the native level for the new version of SQL Server
- Use Query Store (and Automatic Plan Correction on SQL Server 2017 Enterprise Edition) to fix performance regressions by forcing the last known good plan
You also have all of the other new “knobs” of database-scoped configuration options, query-level hints, and trace flags available to you. You may have to do some additional work on some queries with USE HINT query hints. Ideally, you would have done enough testing so that you already have a pretty good idea of the “best” combination of these settings for your workload, but many organizations don’t actually do that.
Keep in mind that for each of the new QP features over the last two versions (Adaptive Query Processing in SQL Server 2017 and Intelligent Query Processing in SQL Server 2019), Microsoft exposes the ability to disable specific behavior at the database scoped configuration or query USE HINT scope. Microsoft generally recommends that if you do find regressions related to a specific feature, try disabling it at lower granularities first, so you can still benefit from all of the rest of the improvements you get from the latest database compatibility level.
Query Tuning Assistant
Microsoft shipped a free tool called Query Tuning Assistant (QTA) in SSMS 18.0. QTA can guide you through the recommended database compatibility level upgrade process in a wizard-fashion, collecting the baseline workload in Query Store, bumping up the database compatibility level, and then comparing performance with the post-upgrade workload collection. At the end of this process, if performance regressions are detected, rather than moving back to the previously known good plan, the QTA will actually suggest hint-based improvements that can be deployed for individual queries (using plan guides), without having to necessarily move back to the legacy CE. It will also gives you some ideas (indirectly) for how you can modify problematic queries that have CE-related regression issues, when you have that option.
Choosing Your Hardware Wisely
Unfortunately, I still see many organizations make terrible hardware choices that cause poor performance or increased SQL Server license costs. For SQL Server, making a wise processor selection is extremely important.
Make sure you do some analysis here, and don’t just let your server admin make the choice with no input from you. You also should make sure your hardware is configured correctly, especially your power management.
You can also reach out to me on a paid consulting basis for help with all of this.
Being aware of these different combinations of configuration options and how they will affect your workload is critical in preventing SQL Server upgrade performance issues. Please take the time to understand and test different configurations before you go live on a modern version of SQL Server.
If you have any questions about this post, please ask me here in the comments or on Twitter. I am pretty active on Twitter as GlennAlanBerry. Thanks for reading!