FastText follow-up

Just a quick follow-up for those of you who showed interest.

My proof-of-concept with FastText has been a success. The C# class library has been integrated with a product and staged for a future release.

The product provides a web-based conversational user interface, accepting natural language as user input and performing configurable actions, possibly interacting with other (external or internal) software products and services. For example, "How's the weather in Tokyo?" will send a request to a weather service and return the response, or "Show me the current sequence of application xyz" will query another product's database and display the results (prompting the user in case of uncertainty). Therefore the initial use case is supervised text classification, and FastText has proven to be fit for the purpose.

The FastText functionality I've exposed to .NET so far includes:
- all training methods (cbow, skipgram, supervised) and their parameters
- file persistence
- text classification
- partial word representation, "nearest neighbour" queries, "analogies"
- accessing dictionary and model data

The top two points open the possibility to re-train based on new user input, ie. continuous learning.

Additionally, I've implemented database persistence of the dictionaries and trained models so they can be queried directly in SQL code. For example, the classic "king - man + woman" (using cosine similarity):

	-- king - man + woman
	with q (
	id,
	w
	) as (
	select w.id, +1.0 from fasttext.word as w where (w.word = 'king')
	union
	select w.id, -1.0 from fasttext.word as w where (w.word = 'man')
	union
	select w.id, +1.0 from fasttext.word as w where (w.word = 'woman')
	),
	qv (
	i,
	value
	) as (
	select
	v.i,
	sum(v.value * q.w) as value
	from q
	join fasttext.vector as v on
	(v.modelId = 1) and
	(v.wordId = q.id) and
	(v.entryType = 0)
	group by
	v.i
	),
	tt (
	wordId,
	sim
	) as (
	select top 5
	v2.wordId,
	sum(v1.value * v2.value) / (sqrt(sum(v1.value * v1.value)) * sqrt(sum(v2.value * v2.value))) as sim
	from qv as v1
	join fasttext.vector as v2 on
	(v2.modelId = 1) and
	(v2.wordId not in (select id from q)) and
	(v2.i = v1.i) and
	(v2.entryType = 0)
	group by
	v2.wordId
	order by
	2 desc
	)
	select
	w.word,
	tt.sim
	from tt
	join fasttext.word as w on
	(w.id = tt.wordId)
	order by
	tt.sim desc;
	/*------------------------
	word sim
	-------------------------------------------------- ----------------------
	queen 0.779682014249644
	regnant 0.755401848735527
	consort 0.743375493476356
	daughter 0.723103235814238
	throne 0.721994349834325
	(5 rows affected)
	------------------------*/

view raw analogies.sql hosted with ❤ by GitHub