One of my favourite ways of multi-threading on Windows

Having blogged about a few bugs in multi-threading libraries recently, I want to show an easy and convenient alternative. It's minimalistic but it works.

On Windows, the I/O completion port offers a way to use a thread pool for your multi-threaded application. Designed primarily for efficient processing of asynchronous I/O, it supports files, named pipes, sockets and device control.

In addition to that, you can post your own packets to the port. Quoting from the documentation:

The PostQueuedCompletionStatus function allows an application to queue its own special-purpose completion packets to the I/O completion port without starting an asynchronous I/O operation.

An example of this is shown in the worker demo project (which uses no async I/O at all).

Another Bug

So you've put in the time and effort to refactor your code and data for parallel execution and are eager to see some parallel action. Unfortunately, you might be disappointed; in some cases your tasks might get serialized, performed sequentially in a single thread. With the overhead you've just introduced, your code probably performs a little bit worse than before.

If you thought TParallel.Join was a nasty bug, things can apparently get even worse. Reading further in Primož Gabrijelčič's book Delphi High Performance, Chapter 7, "Exploring Parallel Practices":

There's a nasty bug in the System.Threading code that was introduced in Delphi 10.2 Tokyo. I certainly hope that it will be fixed in the next release, as it makes the Parallel Programming Library hard to use. It sometimes causes new threads not to be created when you start a task. That forces your tasks to execute one by one, not in parallel.

(Interestingly, I'm able to reproduce it reliably in XE7, too.)

The ParallelTasks sample project demonstrates the issue.
If you run the "Check primes 2" code with four tasks first, then two tasks, and finally one task, you'll get the expected result:

However, running the code with one task first, then two tasks, and finally four tasks will give you this:

The author offers two different workarounds:
- insert a little delay after each call to TTask.Run (in the sample code, uncomment the Sleep(1) call in btnCheckPrimes2Click method), or
- create your own thread pool and limit the minimum number of running threads in it (in the sample code, see btnCustomThreadPoolClick method).

Don't lose time with a known Delphi bug affecting TParallel.Join

Writing multi-threaded code is hard and takes a lot of time. That's why it's especially annoying to waste time with bugs like this.

Reading (and enjoying) Primož Gabrijelčič's book Delphi High Performance, I've come across this paragraph in Chapter 7: Exploring Parallel Practices:

There's not much to say about Join, except that in current Delphi it doesn't work correctly. A bug in the 10.1 Berlin and 10.2 Tokyo implementations causes Join to not start enough threads. For example, if you pass in two tasks, it will only create one thread and execute tasks one after another. If you pass in three tasks, it will create two threads and execute two tasks in one and one in another.

The code accompanying the book is available on Github: PacktPublishing/Delphi-High-Performance

Related:
G+ discussion
TParallel,Join does not create enough threads (Embarcadero's Quality Portal RSP-19557)

The author offers a simple workaround by starting a dummy task (which does nothing) first; you can see an example here.

Some time ago (using Delphi XE7), I wrote a library with my own TThreadPool class using and encapsulating the Windows IOCP. The result turned out well, it was rock-solid. Reading about bugs like this in the latest-and-greatest Delphi version makes me glad I chose to write my own implementation from scratch and saved a lot of time hunting for bugs like this in Delphi's runtime library (in addition to my own).

Potential Deadlocks in Parallel.For

Recently, I've come across some C# code deadlocking quite reproducibly while executing some tasks using Parallel.For method. The seemingly innocuous code lead to an "obscure situation" exactly as described in this blog post by Stephen Toub:
Does Parallel.For use one Task per iteration?

...iterations are handed out in indivisible chunks, and only one thread is involved in the processing of a particular chunk. This has implications for interdependencies between iterations. If iteration i blocks waiting for iteration i + 1 to be completed, and iterations i and i + 1 are both allocated to the same chunk, the loop will likely deadlock. The thread processing those iterations will block processing iteration i, but as that thread is also responsible for processing iteration i + 1, iteration i + 1 will never get processed and iteration i will never unblock.

The problem in this case was exactly as described above: there were blocking wait dependencies between the tasks in a producer/consumer pattern. The solution (once the problem was clear) was relatively simple: don't rely on the default partitioning of Parallel.For; provide your own to avoid the potential deadlock.

A good framework or library can provide you with a good-enough solution in a large-enough percentage of possible use cases (probably making some compromises to achieve that goal). Don't expect a pre-fabricated solution to solve all your problems out of the box; There is no silver bullet.

Here's some interesting reading about the Parallel.For implementation in .NET and trade-offs between simplicity, overheads, and load balancing:
Patterns of Parallel Programming (Understanding and Applying Parallel Patterns with the .NET Framework and Visual C#)

The strange limitation of 64 threads

When using the Windows I/O Completion Port (IOCP), people seem to limit their thread pools to a maximum of 64 threads.

This is probably caused by the fact that WaitForMultipleObjects limits the number of input handles with the nice magic constant MAXIMUM_WAIT_OBJECTS (which happens to be 64).

Here are a few examples:

• Creating a Thread Pool for MFC-Based ISAPI Extensions - this might be where it all started...
  
• The venerable IsapiThreadpool unit which originally appeared in Delphi 6. You only had to add it to your ISAPI DLL project to enjoy a nice performance boost.
• Many others...

The (anti-)pattern is related to the process of shutting down the thread pool: to do this cleanly, the threads in the pool should be allowed to finish what they're doing (or just wake up if they're idle at the moment), perform any cleaning up as necessary and terminate correctly. The shutdown is usually performed in two steps:

1. Send a shutdown signal (completion key) to each thread.
Each thread in the pool calls GetQueuedCompletionStatus in a loop and checks for the special (application-defined) shutdown completion key to which it responds by breaking out of the loop and terminating. The shutdown procedure can therefore simply send the shutdown completion key to the IOCP as many times as there are threads, relying on the fact that exactly one thread will respond to exactly one such signal.
2. Wait for all threads to terminate.
The shutdown is not complete before all threads actually had a chance to receive the signal and terminate. Only then it's safe to continue closing the IOCP, freeing memory, etc. So we absolutely have to wait for the threads to terminate. The reasoning here seems to be: Since WaitForMultipleObjects can only handle up to 64 threads, we can't allow more threads to be associated with the pool in the first place, can we?

Well, there's no need to use WaitForMultipleObjects in Step 2. It's fairly easy to keep a counter of active threads in the pool (interlocked-incremented when a thread starts, interlocked-decremented when a thread is finished). When the counter reaches zero (no more active threads), signal an event. With only one event to wait for, you can use WaitForSingleObject in Step 2.