Colours on my keyboard

A few days ago I had to consider reinstalling Windows on my notebook. Instead I decided to switch to Linux Mint. I've been using Linux Mint in virtual machines for a few years and it worked very well for me. This is the first time I run Linux directly on the host. For development I keep a few Windows virtual machines. I'm very happy with it so far. Everything just worked without much ado, except one detail: The notebook comes with a backlit keyboard with configurable colours. Under Windows I got used to setting it to a dim white colour which works best for me. Since the default colour on boot is dark blue I was looking for a way to change it in Linux. Luckily, I've found a driver for it.

Big kudos to Steven Seeger (klystron) for his efforts!

I can enjoy my comfortable dim white colour again. Or play around; see below. ;-)

Happy holidays and a great year 2017 to everyone!

Delphi Perspectives: Kick-Start or the Last Kicks?

I've received another e-mail offer from Embarcadero to upgrade my Delphi 10.1 Berlin Starter Edition to Professional for €749 (claiming to save over €2000). The offer is valid this weekend only and includes:

1. Delphi 10.1 Berlin Professional upgrade
2. Mobile Add-On Pack
3. "Bonus Packet" with VCL and FireMonkey Styles, Konopka VCL Controls and Marco Cantù's Object Pascal Handbook

After Embarcadero shut down their R&D centers (with no public statement) and let go of people like Allen Bauer and David Intersimone I start wondering: Is this the beginning of the final sell-out? On a more positive note, the new General Manager mentioned in an interview (PDF) that they wanted to attract new developers and get back into education. Hints of this happening are already here.

Let's see what happens...

Potential Deadlocks in Parallel.For

Recently, I've come across some C# code deadlocking quite reproducibly while executing some tasks using Parallel.For method. The seemingly innocuous code lead to an "obscure situation" exactly as described in this blog post by Stephen Toub:
Does Parallel.For use one Task per iteration?

...iterations are handed out in indivisible chunks, and only one thread is involved in the processing of a particular chunk. This has implications for interdependencies between iterations. If iteration i blocks waiting for iteration i + 1 to be completed, and iterations i and i + 1 are both allocated to the same chunk, the loop will likely deadlock. The thread processing those iterations will block processing iteration i, but as that thread is also responsible for processing iteration i + 1, iteration i + 1 will never get processed and iteration i will never unblock.

The problem in this case was exactly as described above: there were blocking wait dependencies between the tasks in a producer/consumer pattern. The solution (once the problem was clear) was relatively simple: don't rely on the default partitioning of Parallel.For; provide your own to avoid the potential deadlock.

A good framework or library can provide you with a good-enough solution in a large-enough percentage of possible use cases (probably making some compromises to achieve that goal). Don't expect a pre-fabricated solution to solve all your problems out of the box; There is no silver bullet.

Here's some interesting reading about the Parallel.For implementation in .NET and trade-offs between simplicity, overheads, and load balancing:
Patterns of Parallel Programming (Understanding and Applying Parallel Patterns with the .NET Framework and Visual C#)

No runtime type information on interface properties

You might have tried to decorate an interface property in Delphi with a custom attribute like in this question on StackOverflow.

Unfortunately, that won't work, as I explained in my answer. No runtime type information is generated for interface properties so your attribute will have no effect, even though there's no compiler warning. You can try the code below which demonstrates it.

Of course, if you can prove me wrong or have another idea please let me know in a comment here. Thanks!

Here's the code:

program Test;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils,
  System.Rtti;

type
  TTestAttribute = class(TCustomAttribute)
  private
    FName: string;
  public
    constructor Create(const AName: string);
    property Name: string read FName;
  end;

  [TTestAttribute('AtInterface')]
  ITestInterface = interface(IInvokable)
    ['{92A8107E-B40E-4786-A02E-251EA0DB3937}']
    [TTestAttribute('AtMethod')]
    function GetValue: string;
    [TTestAttribute('AtMethod')]
    procedure SetValue(const AValue: string);
    [TTestAttribute('AtProperty')]
    property Value: string read GetValue write SetValue;
  end;

constructor TTestAttribute.Create(const AName: string);
begin
  inherited Create;
  FName := AName;
end;

procedure ShowAttributes(const Attributes: TArray<TCustomAttribute>);
var
  Attr: TCustomAttribute;
begin
  for Attr in Attributes do
    if Attr is TTestAttribute then
      Writeln(Format('%s(''%s'')', [Attr.ClassName, TTestAttribute(Attr).Name]))
    else
      Writeln(Attr.ClassName);
end;

procedure Main;
var
  Ctx: TRttiContext;
  AType: TRttiInterfaceType;
  Attributes: TArray<TCustomAttribute>;
  Method: TRttiMethod;
  Prop: TRttiProperty;
begin
  Ctx := TRttiContext.Create;
  AType := Ctx.GetType(TypeInfo(ITestInterface)) as TRttiInterfaceType;
  Writeln(Format('Attributes on the interface ''%s'':', [AType.Name]));
  ShowAttributes(AType.GetAttributes);
  Writeln;
  for Method in AType.GetMethods do
  begin
    Writeln(Format('Attributes on method ''%s'':', [Method.Name]));
    ShowAttributes(Method.GetAttributes);
  end;
  Writeln;
  for Prop in AType.GetProperties do
  begin
    Writeln(Format('Attributes on property ''%s'':', [Prop.Name]));
    ShowAttributes(Prop.GetAttributes);
  end;
end;

begin
  try
    Main;
    Readln;
  except
    on E: Exception do
    begin
      ExitCode := 1;
      Writeln(Format('[%s] %s', [E.ClassName, E.Message]));
    end;
  end;
end.

This produces the following output:

Attributes on the interface:
TTestAttribute('AtInterface')

Attributes on method 'GetValue':
TTestAttribute('AtMethod')
Attributes on method 'SetValue':
TTestAttribute('AtMethod')

Already the call to AType.GetProperties returns an empty array; there is no RTTI for interface properties.

Free Delphi Starter Edition

Interesting offer by Embarcadero:
Delphi Starter Edition Special Offer

Using RemObjects Hydra to consume a .NET WCF service from Delphi

Using RemObjects Hydra is probably the easiest way to consume a WCF service from Delphi in case you need to support .NET-specific bindings which might require a lot of work to implement on your side but are already available for .NET.

This series of videos shows how you can write a managed Hydra plugin module to consume a WCF service and expose it to a Delphi Hydra host application as a custom interface, delegating the method calls to the WCF service.

01. WCF Service Library

• Create a new WCF Service Library project:
• File\New\Project...
• Templates\Visual C#\WCF\WCF Service Library
• Add a new method to the service contract (interface)
• Implement the new method
• Test the service with the WCF test client

02. WCF Service Client Library

• Create a new class library project:
• Right-click solution, Add\New Project...
• Installed\Visual C#\Class Library
• Add service reference (to the service created in the previous step)
• Delete the default empty class module

03. WCF Service Client Console Application

• Create a new console project:
• Right-click solution, Add\New Project...
• Installed\Visual C#\Console Application
• Add System.ServiceModel reference
• Add service client library reference
• Add code to call the service methods
• Run the console application

04. Hydra C# Plugin Module

• Create a new Hydra Plugin Library project:
• Right-click solution, Add\New\Project...
• Installed\Visual C#\Hydra\Plugin Module
• Create a new Hydra Plugin:
• Right-click project, Add\New Item...
• Installed\Visual C# Items\Non-Visual Plugin
• Add System.ServiceModel reference
• Add service client library reference
• Create a COM-compatible interface
• Right-click project, Add\New Item...
• Installed\Visual C# Items\Code\Interface
• Declare interface IService1Client (public)
• Inherit from IHYCrossPlatformInterface (RemObjects.Hydra.CrossPlatform)
• Add IID
• Add methods
• Add CompositeType struct
• Add StructLayout attribute
• Add MarshalAs attributes
• Declare data contract parameter ref on the C# side, const on Delphi side
• Make the plugin implement the interface by delegating to the WCF service client library

05. Delphi Hydra Host Console Application

• Create a console application
• Add Vcl to unit scope names
• Change the output directory to be the same as the plugin output directory
• Write code to load the plugin and call its interface
• Copy and patch uHYCLRWrappers.pas unit to load configuration from a .config file
• Create the configuration file

You can find the source code here:
https://github.com/tondrej/hydra-wcf-delphi

Include resource files in your Delphi build process

Quite often people don't know about this neat little feature, as recently in this g+ discussion, too.

You can simply make any .rc file a part of your project and its build process. There's a BrccCompile target defined in your $(BDS)\bin\CodeGear.Delphi.Targets file which is included in every Delphi project by default and it takes care of rebuilding the resources with the brcc32 resource compiler.

One easy way to do this is to select "File\New\Other..." menu item to open the "New Items" dialog and select "Text file" under "Other Files":

In the following "New file" prompt, select ".rc Resource Script" and check the "Add new file to current project" checkbox:

Alternatively, you can create your .rc file in an external editor and drag-and-drop it from the Windows Explorer onto your project's node in the Project Manager treeview.

In either case your new .rc file becomes a part of the project:

At the same time a $R directive similar to this is added to your project's source file (.dpr):

{$R 'New1.res' 'New1.rc'}

That's it. Any modifications to your .rc files will now be recompiled automatically into corresponding .res files and linked into your executable, as a part of the build process. And you can open and edit them easily in the IDE directly, too.

Embracing .NET with RemObjects Hydra

If you're a Delphi shop and find yourself (sometimes) jealous of all the .NET goodies have a look at RemObjects Hydra (more details here). It's a framework which allows you to mix and use VCL, FireMonkey and .NET code modules in a single mixed-mode application. (They've recently started the "Escape from Delphi" offer.)

Your host application can be an unmanaged 32-bit or 64-bit Delphi VCL or FireMonkey application (written in Delphi 7 or later), or a managed WinForms .NET application (written in any CLR-compliant language like C#, Oxygene, VB.NET, etc).
Your plugin libraries (loadable by the host dynamically at runtime) can implement non-visual or visual plugin modules which can be unmanaged Delphi VCL or FireMonkey, or managed .NET WinForms, WPF or Silverlight DLLs. The host and the plugins can communicate with each other through interfaces. IDE support with templates and wizards to create the hosts and the plugins, for both Visual Studio and Delphi, is included.

If your trusty old Delphi codebase comprises millions of lines of code the idea of porting that code over to .NET might be intimidating. It probably seems completely out of question, an investment of huge amount of time and effort into a project with uncertain benefits. The development of new features during the transition might be stalled or limited, and you're going to have a period of time when nothing works. In short, probably a nightmare.

With Hydra now, you can think about it again: You can start with a skeleton host application, quite simply rebuilding your existing Delphi units into plugin modules, keeping all your existing code working as before. You can then port the modules to .NET one by one, at your own pace, if at all. Simultaneously, you can develop new modules in .NET already. All of a sudden, it's not so bad anymore, it's definitely doable, and probably fun.

Below is a screenshot of a very quick exercise in rebuilding the SearchBox VCL sample as a Hydra plugin. It took me only a few minutes to resolve some issues: to include the VCL styles, the DLL must link in the sample's resource file, and use the VCL.Styles unit. I also had to adjust the form slightly: remove border icons and make it visible. Here's the result:

Free Pascal and Lazarus foundation

The Free Pascal and Lazarus foundation is a non-profit organisation dedicated to the promotion and sponsoring of the open source Free Pascal and Lazarus projects.

Escape from Delphi

Interesting: "Escape from Delphi" offer by RemObjects.

SQL Server Bulk Copy using Delphi

When inserting a lot of data into a SQL Server table, the memory-based bulk copy exposed by the IRowsetFastLoad interface in Microsoft's OLE DB providers is the fastest way.

Here are the steps listed in the example above (with some comments added by me):

1. Establish a connection to the data source.
In Delphi, you can simply set up a TADOConnection component and set its Connected property to True:

    Connection.Provider := 'SQLOLEDB'; // or 'SQLNCLI11'
    Connection.ConnectionString := 'Integrated Security=SSPI;Data Source=localhost;';
    Connection.LoginPrompt := False;
    Connection.KeepConnection := True;
    Connection.Connected := True;

2. Set the SQLOLEDB provider-specific data source property SSPROP_ENABLEFASTLOAD to VARIANT_TRUE. This allows the newly created session to access the IRowsetFastLoad interface.
In fact, from my experiments, this step seems to be unnecessary, as the bulk copy works without setting this property, too. (Or perhaps it's set already by default.)
3. Create a session requesting the IOpenRowset interface, and
4. Call IOpenRowset.OpenRowset to open a rowset that includes all the rows from the target table:

function OpenFastLoad(Connection: TADOConnection;
  const TableName: WideString): IRowsetFastLoad; overload;
var
  ConnectionConstruction: ADOConnectionConstruction;
begin
  SetProperty(Connection, DBPROPSET_SQLSERVERDATASOURCE, SSPROP_ENABLEFASTLOAD, True);
  ConnectionConstruction := Connection.ConnectionObject as ADOConnectionConstruction;
  Result := OpenFastLoad(ConnectionConstruction.Get_DSO as IDBCreateSession, TableName);
end;

function OpenFastLoad(const DBCreateSession: IDBCreateSession;
  const TableName: WideString): IRowsetFastLoad; overload;
var
  OpenRowSet: IOpenRowset;
  TableID: TDBID;
begin
  OleDbCheck(DBCreateSession.CreateSession(nil, IID_IOpenRowset, IUnknown(OpenRowSet)),
    DBCreateSession, IID_IDBCreateSession);
  TableID.eKind := DBKIND_NAME;
  TableID.uName.pwszName := PWideChar(TableName);
  OleDbCheck(OpenRowSet.OpenRowset(nil, @TableID, nil, IID_IRowsetFastLoad, 0, nil,
    @Result), OpenRowSet, IID_IOpenRowset);
end;

5. Do the necessary bindings and create an accessor using IAccessor.CreateAccessor:
The bindings are an array of TDBBinding records (declared in Winapi.OleDb) describing each inserted column and their offsets in the buffer:

procedure InitializeBinding(Field: TField; var Binding: TDBBinding; var Offset: Integer);
begin
  Binding.iOrdinal := Field.FieldNo; // column ordinal position
  Binding.wType := FieldTypeToOleDbType(Field.DataType); // column data type
  if Field.IsBlob then
    Binding.wType := Binding.wType or DBTYPE_BYREF; // pointer to external blob data
  Binding.eParamIO := DBPARAMIO_NOTPARAM;
  Binding.dwMemOwner := DBMEMOWNER_CLIENTOWNED; // we are releasing the memory
  Binding.obLength := Offset; // length field offset (starts with 0 for the first column)
  Binding.obStatus := Binding.obLength + SizeOf(DBLENGTH); // status field offset
  Binding.obValue := Binding.obStatus + SizeOf(DBSTATUS); // value offset
  Binding.dwPart := DBPART_LENGTH or DBPART_STATUS or DBPART_VALUE; // included parts
  case Field.DataType of
    ftDate:
      Binding.cbMaxLen := SizeOf(TDBDate); // OLE DB date
    ftTime:
      Binding.cbMaxLen := SizeOf(TDBTime); // OLE DB time
    ftDateTime, ftTimeStamp:
      Binding.cbMaxLen := SizeOf(TDBTimeStamp); // OLE DB timestamp
    else
      Binding.cbMaxLen := Field.DataSize;
  end;

  Inc(Offset, SizeOf(TColumnData) + Binding.cbMaxLen - 1); // next column's offset...
  Align(Offset); // ...aligned to 8 bytes
end;

  ...
  OleDbCheck(FastLoad.QueryInterface(IID_IAccessor, Accessor), FastLoad, IID_IRowsetFastLoad);
  OleDbCheck(Accessor.CreateAccessor(DBACCESSOR_ROWDATA, Dataset.FieldCount, Bindings, BufferSize,
    AccessorHandle, StatusCodes), Accessor, IID_IAccessor);

6. Set up the memory buffer from which the data will be copied to the table.
The record buffer is a sequence of TColumnData records (of variable size):

type
  DBLENGTH = ULONGLONG;
  PColumnData = ^TColumnData;
  TColumnData = record
    Length: DBLENGTH; // data length
    Status: DBSTATUS; // null or has a value
    Data: array[0..0] of Byte; // value data
  end;

For each column, fill in the length, status and data fields within the buffer (code simplified):

procedure GetFieldValue(Field: TField; const Binding: TDBBinding; Buffer: Pointer);
var
  Column: PColumnData;
begin
  Column := Pointer(NativeUInt(Buffer) + Binding.obLength);
  if Field.IsNull then
  begin
    Column^.Status := DBSTATUS_S_ISNULL;
    Column^.Length := 0;
  end
  else
  begin
    Column^.Status := DBSTATUS_S_OK;
    case Field.DataType of
      ftDate:
        with PDBDate(@Column^.Data[0])^ do
          DecodeDate(Field.AsDateTime, Word(year), month, day);
      ftTime:
        with PDBTime(@Column^.Data[0])^ do
          DecodeTime(Field.AsDateTime, hour, minute, second, MSec);
      ftDateTime, ftTimeStamp:
        with PDBTimeStamp(@Column^.Data[0])^ do
        begin
          DecodeDate(Field.AsDateTime, Word(year), month, day);
          DecodeTime(Field.AsDateTime, hour, minute, second, MSec);
          fraction := MSec * 1000000;
        end;
      else
        Field.GetData(@Column^.Data[0], False);
    end;
    case Field.DataType of
      ftString, ftMemo:
        Column^.Length := StrLen(PAnsiChar(@Column^.Data[0]));
      ftWideString, ftWideMemo:
        Column^.Length := StrLen(PWideChar(@Column^.Data[0])) * SizeOf(WideChar);
      else
        Column^.Length := Field.DataSize;
    end;
  end;
end;

7. Call IRowsetFastLoad.InsertRow to bulk copy the data in to the table.
(Repeat the previous and this step for each row you want to insert.)
8. Call IRowsetFastLoad.Commit to commit all changes.

Since you explicitly control in your code how the buffers are allocated and populated, this approach offers some ways to optimize for performance especially when inserting large amounts of data. It's also possible to set up bulk insert sessions inserting into the same table in parallel from multiple threads.
As a starting point, you can find the example source code here.

The strange limitation of 64 threads

When using the Windows I/O Completion Port (IOCP), people seem to limit their thread pools to a maximum of 64 threads.

This is probably caused by the fact that WaitForMultipleObjects limits the number of input handles with the nice magic constant MAXIMUM_WAIT_OBJECTS (which happens to be 64).

Here are a few examples:

• Creating a Thread Pool for MFC-Based ISAPI Extensions - this might be where it all started...
  
• The venerable IsapiThreadpool unit which originally appeared in Delphi 6. You only had to add it to your ISAPI DLL project to enjoy a nice performance boost.
• Many others...

The (anti-)pattern is related to the process of shutting down the thread pool: to do this cleanly, the threads in the pool should be allowed to finish what they're doing (or just wake up if they're idle at the moment), perform any cleaning up as necessary and terminate correctly. The shutdown is usually performed in two steps:

1. Send a shutdown signal (completion key) to each thread.
Each thread in the pool calls GetQueuedCompletionStatus in a loop and checks for the special (application-defined) shutdown completion key to which it responds by breaking out of the loop and terminating. The shutdown procedure can therefore simply send the shutdown completion key to the IOCP as many times as there are threads, relying on the fact that exactly one thread will respond to exactly one such signal.
2. Wait for all threads to terminate.
The shutdown is not complete before all threads actually had a chance to receive the signal and terminate. Only then it's safe to continue closing the IOCP, freeing memory, etc. So we absolutely have to wait for the threads to terminate. The reasoning here seems to be: Since WaitForMultipleObjects can only handle up to 64 threads, we can't allow more threads to be associated with the pool in the first place, can we?

Well, there's no need to use WaitForMultipleObjects in Step 2. It's fairly easy to keep a counter of active threads in the pool (interlocked-incremented when a thread starts, interlocked-decremented when a thread is finished). When the counter reaches zero (no more active threads), signal an event. With only one event to wait for, you can use WaitForSingleObject in Step 2.