Parallel programming in Sitecore7.2, yes Sitecore 7.2
announced a new feature called Parallel Publishing
Enabling Parallel Publishing with Sitecore 7.2
The new Parallel Publishing functionality is by default disabled in 7.2. This document provides a practical walkthrough showing how to enable the new features and optimizations to dramatically increase performance of publishing operations.
Inspect the current state of your environment
The first thing you need to do is to make sure your environment is ready for parallel publishing.
This is the Sitecore instance where the publishing activity is happening. In most cases it is happening on the same environment as content authoring/management (CM).
If your CM server runs minimum hardware recommendation configuration (4 core CPU / 4 GB of RAM), you will definitely need more resources to run parallel publishing comfortably without affecting ongoing authoring activities.
Once that bridge is crossed, gather avg. CPU and memory metrics from the server throughout peak times. Make sure you have plenty of room there.
It is highly recommended to consider setting up a dedicated Sitecore publishing instance to reduce the impact of parallel publishing operations on the authoring server. Think about such instance as another CM instance with content authoring activity disabled. To set it up, consult the Scaling Guide’s section 4.3 on “How to: Configure the Name of the Publishing Instance”.
It is important to note that from the licensing perspective, setting up this instance means having another license to run. So if in doubt, please consult with your regional Sitecore Sales Manager.
Similar to the web server running publishing, enabling parallel publishing significantly increases the need for disk, CPU and memory resources on your database server. So your database server must have capacity to handle much, much more throughput than before during publishing.
Besides CPU and memory, disk IO becomes a big factor. Things like splitting data and log files to separate disks could be one of the action items.
Most likely, your Sitecore database server running content databases is currently overprovisioned and you have plenty of room for maximizing publishing.
In any case, it is highly recommended to involve your DBA in this process, and monitor the database activity closely after this change.
Parallel publishing means that more data needs to get from the publishing instance to the database server. Most on premise environments are expected to have plenty of network throughput capacity, however, especially for the hybrid deployments involving cloud / hosted, it is worth getting a better understanding of available throughput and potential impact on operational costs if the traffic between publishing server and database server increases 10-25x.
Enabling Parallel Publishing and related optimizations
Now that your environment is ready, follow these easy steps to enable Parallel Publishing.
1. Enable the following /App_Config/Include file by renaming its extension to .config:
This file patches a very important setting controlling how many parallel publishing tasks will be running by the .NET Task Parallel Library: Publishing.MaxDegreeOfParallelism
This setting defaults to 4. The optimal value for this setting obviously depends on your solution and on the CPU capacity of the server that runs the publishing operations.
On 8 and 16 core servers increasing this value to 8 and 16 correspondingly had positive effects on publishing performance. If you have 8 cores or more and your publishing instance is not utilizing the CPU capacity, you can start increasing this setting gradually (8, 12, 16, 24, etc.) if you don’t notice any abnormalities (SQL deadlocks or other errors in the logs).
2. Enable the following /App_Config/Include file by renaming its extension to .config:
This file enables a set of optimizations, some of them can be considered experimental.
Please inspect the contents of this file carefully, as you may need to tweak it to accommodate the specifics of your environment to make sure you don’t break your custom code.
2.1 The group of 3 Caching.CacheKeyIndexingEnabled.* settings for various cache types are meant to reduce the impact of cache eviction on the CPU of the publishing instance. Though these patches help tremendously, especially during republish jobs, these are considered to be experimental.
As a trade-off, enabling this setting on content management servers with many concurrent editors and many content items can degrade performance.
2.3 Caching.DisableCacheSizeLimits – does exactly what is says.
You should comment out this element if you are not running 64-bit systems, or if your servers do not have enough memory to cache the data that is requested from the database.
This can grow to tens of Gibabytes if you let it to, so you gotta be extra careful here.
You should comment out this element if you use Fast Query to perform descendant lookups.
Disables incremental updates to the link database during publishing operations as an experimental optimization to speed up publishing (25% on average).
You should comment out this element if you use the LinkDatabase API on your content delivery instances.
Another set of experimental optimizations targeting the execution of rules via item event handlers. Most solutions don’t use this feature. Disabling it reduced the CPU spikes on all servers, including Content Delivery, during publishing.
You should comment out this element if there are rules in any of the following folders that must be executed during publishing:
This include file replaces the standard EventProvider implementation with AsyncEventProvider.
The only difference between the two implementations is that the AsyncEventProvider processes queued events from various databases asynchronously.
When the system is busy, processing the event queue of one database can block event processing from all other databases. The AsyncEventProvider solves this problem.
This problem can occur, for example, during large publishing operations when the Sitecore.Publishing.Parallel.config include file is enabled. This can result in significant delays in updating the processed items counter in the Publishing Wizard.
4. Tune your SQL Server or Oracle database settings.
Here is a specific configuration change you must do in order to prevent SQL timeouts on SQL server.
By default, the Autogrowth setting on the data and log files on all content databases is set to 10%:
After you enable parallel publishing and republish a sizable master database (this scenario was tested on 1M items), the target web database starts growing fast and at a specific point the 10% become too big for SQL Server to grow data/log files, which results in SQL Server timeouts.
To address this, change this setting on all web databases to a specific value in MB (30MB-50MB to start with).
If you are running Oracle for your content databases, our tests indicate that the number of defaultProcesses and Sessions must be increased from default values to accommodate the values of Publishing.MaxDegreeOfParallelism setting of 2 or more. Please consult official Oracle documentation on how to do it.
It is highly recommended to involve your DBA in this process, and monitor the database activity closely after this change.