Lisa Hoek

Structure optimalisation for the development of Open.Amsterdam (ongoing)

About Municipality of Amsterdam
More information about this coming soon.

Task
More information about this coming soon.

Metadata optimalisation for the digital infrastructure (ongoing)

About City Archives of Amsterdam
More information about this coming soon.

Task
More information about this coming soon.

Smart Rule implementation using SpaCy

About Textmetrics
Textmetrics is a company that specializes in text optimization and content creation tools. They provide software that helps businesses improve the quality and effectiveness of their written content, e.g., job advertisements, by analyzing and offering suggestions for enhancements. Their platform is often used to ensure that content is clear, inclusive, and aligned with brand and SEO strategies.

Task
When a new customer starts using Textmetrics' text optimization software, I implement custom rules in the system that align with the customer's writing guide. For example, some businesses would like to write numbers below twenty in digits, other prefer its written out form. I make sure Textmetrics' tool makes correct suggestions to the customer's text. Other examples include specific formats for time, dates, valuta and phone numbers, or avoiding certain words or phrases.

Method
Part of the backend that makes these customisations possible is a Python application that heavily relies on SpaCy and extends upon this. It uses SpaCy's doc type with its tokenization and many more of its features. There are multiple NLP pipelines running that analyse the text and create the optimization suggestions to the user.

Detecting Age Discrimination in Job Advertisements using GPT and BERT

About Textmetrics
Textmetrics is a company that specializes in text optimization and content creation tools. They provide software that helps businesses improve the quality and effectiveness of their written content, e.g., job advertisements, by analyzing and offering suggestions for enhancements. Their platform is often used to ensure that content is clear, inclusive, and aligned with brand and SEO strategies.

Task
One of the components for an inclusive job advertisement is to avoid age discrimination. Textmetrics currently detects this with a set of regular expressions. However, these expressions do not match the full variety of sentences, i.e., its recall is low. Also, it struggles at sentences with a justification. For example, 'You need to be at least 18' is discriminating, but it can be justified when it is 'because you need to work late' / 'in order to serve liquor'. So, our goal is to find a method and create an age discrimination detection that has a much higher recall, while preserving a good precision.

Method
We first explored whether GPT can detect age discrimination with the help of an extensive prompt. In a set of 1,000 job advertisements (22,366 sentences), it found 599 discriminative sentences against 71 from the RegExes. We manually removed the false positives to 418 discriminations to build a fairly correctly labeled dataset.

On the left, the age discrimination found by the RegExes. On the right, the age discrimination found by GPT.

We used this dataset to fine-tune GPT, in order to create an even larger labeled dataset. This resulted in 100,000 sentences of which 2,504 with a positive label. At this point, we divided age discrimination in 4 different categories:

• Age discrimination (2,100), consisting of direct phrases like 'you are at least ... old', but mostly indirect phrases like 'we are looking for a young person'.
• Health discrimination (122), consisting of sentences like 'you need to be fit' without specifying why.
• Tone of voice (221), using slang to attract youth and not appeal to older people.
• Justified discrimination (60), sentences that ask the applicant to be of a certain age or be fit, because of the specific type of job.

After creating this dataset, many iterations followed manually improving the data. We mainly focused on the positive labels, because the volume of non-discriminative sentences is too big to manually go through them. Textmetrics does not have the resources to create labeled data in an annotation project using many annotators. Using GPT to create a labeled dataset is better feasible and we compared its performance to the original RegExes to validate its results. A disadvantage of this technique is that we are unaware of false negatives, and not all false positives are corrected. In our next step, we try to further eliminate those false positives.

Results
We trained 5 BERT models using cross-validation on a representative set of 5,000 non-discriminative sentences and 1,945 discriminative sentences. By looking at the wrong predictions of all the models, we were able to revise our labeled dataset more. Below is a confusion matrix of the 5 models combined:

In the end, we decided to split the workflow in two BERT models: first, detecting age discrimination (along with health and tone of voice), second, a model that decides whether the discriminative sentences are justified or not. Note that our method is on a sentence-level, so a disadvantage is that it does not take surrounding sentences into account; a justification needs to be in the same sentence. Below are the final confusion matrices:

Discussion
This project shows that finetuning a BERT model achieves adequate results for a sentence prediction task such as detecting age discrimination. It does not tie you to keep using a paid LLM. We did, however, use GPT to our advantage by using it once to create a labeled dataset, instead of spending time and effort into a big annotation project.

Transcribing 20th century Surinamese death certificates using Transkribus and GPT

About the HDSC
The 'Historical Database of Suriname and the Caribbean' (HDSC) is dedicated to making historical documents (slave registers, civil status records, and more) available online. This allows for the tracking of individuals' life histories, especially following significant events such as the abolition of slavery in 1863. Thanks to the work of many volunteers and sponsors, since 2018 the slave registers of Suriname have been viewable online, followed by those from Curaçao in 2020. Additionally, as of 2023, the birth certificates of Paramaribo are searchable online on the website of the National Archive of Suriname. Currently, more civil status records from Suriname and the (former) Dutch Caribbean islands are being transcribed.

Project
In this project, we adjusted our HTR+GPT pipeline to process 40,000 20th century Surinamese death certificates. With the help of 10 dedicated volunteers, we transcribed 500 scans in Transkribus. These served as training data for two custom Field models, a Baseline model and HTR model.

We developed a Python script to reduce Transkribus costs. It made Field detection (1 credit per scan) redundant for the first half of the certificates. Baseline detection with automatic region detection (0.25 credit per scan) suffices for the majority of the certificates. For a small percentage, when the certificate contains marginalia text, baseline detection fails. Our Python code calculated the average position the main certificate text starts to split overlapping baselines between the marginalia and the certificate text. This adjusted 668 of 20,000 scans and saved 20,000 Transkribus credits.

We observed that the HTR quality of the scans is higher than those of Curaçao. We suspect due to easier handwriting and higher scan quality. We did not evaluate the HTR quality per entity. The HTR training achieved a CER of 2.97% on the validation set (mainly due to the printed text in the certificate that is easily read). The handwritten text has a higher CER, e.g., names of the deceased in the curaçaoan data was 10%. To ensure quality, all entities are checked by volunteers in a review project after extraction.

To extract all entities, we tested OpenAI's newest model GPT-4o and got satisfactory results (similar performance but cheaper processing compared to GPT-4). After a couple of iterations improving the prompt, GPT-4o ran over all 20,000 HTR-ed transcriptions. Then, colleagues splitted and standardized names for the first half of the dataset. For the second half, only standardization was performed as GPT could split the names into first and last names.

We do not have a final evaluation of the performance. These entities are now corrected by volunteers in a review project to ensure quality. We suspect quality is higher than the Curaçaoan entities, due to a higher scan quality, easier handwriting, larger training data and a newer GPT model. For the Curaçaoan evaluation, we refer to our previous project.

GPT Prompt revision for kids' magic pen pal

About Magie Post
'Magie Post' is a Dutch start-up company writing letters to children in an interactive way. Children together with their caregiver write to 'Bibi the flower fairy' who lives in the magical 'Twinkelrijk' together with her bunny and other friends. When children write something about themselves, Bibi answers the children's message in a positive, encouraging way. In the second part of the letter, she tells about her own adventures in Twinkelrijk and needs help from the children who then craft magical objects that help Bibi on her adventure.

The adventure is a standard format consisting of 3-9 letters. However, the beginning of each letter is custom made and uniquely responds to each child's input. For example, they talk about what happened at school or what they did during the weekend. One of OpenAI's GPT-models is prompted to act as Bibi and respond to the child.

Project
The previous set-up had multiple problems, such as Bibi not answering all questions from the child and Bibi closing her letter while GPT was prompted not to, because the output is still followed by the predetermined adventure.

We fixed these issues by evolving from GPT-3.5 to GPT-4o. This model has a higher intelligence and is able to react to all questions from the input letter. The problem of closing the letter in the middle of the text is fixed by forcing GPT to give an introduction, response to the child and closure in separate JSON values. This way, we can add the predetermined adventure text between the response and closure.

Furthermore, it may happen that a child talks about something specific that GPT does not know, such as the name of a specific restaurant or person. For example, for 'Monkey Town', we saw GPT-4o respond correctly most of the time that it is a playground for children, but there was also the probability it responds as if it is a zoo where you can visit monkeys. To avoid this behaviour, we added to the prompt that Bibi the flower fairy should react interested and ask what it is. This also encourages more conversation between the child and Bibi.

Outlook
After revising the prompt to meet the requirements, we gave Magie Post several possibilities to grow their product:

• Integrating clients' feedback into the GPT-responses after the company has launched.
• GPT receiving the entire conversation, so when multiple letters between the child and Bibi are sent, GPT has the whole context and can adapt to this. This relates to the improvement of adding the predetermined adventures to the chat with GPT. The child crafts some things for Bibi to help her on her adventure, e.g., a magic wand. Then, it should not happen that GPT responds in the letter, because Bibi already reacts in the predetermined adventure.
• Exploring if adding pictures of these childrens' crafts is possible and analysing how GPT responds to them. It would be a fun addition when a child makes a purple magic wand, that Bibi can create a tailored response to this.
• Implementing OpenAI's Batch API to lower costs. This halves costs, but it takes longer to respond (a maximum of 24 hours). This is easily done, because an instant reaction is not necessary.

Transcribing 19-20th century Curaçaoan death certificates using Transkribus and GPT

About the HDSC
The 'Historical Database of Suriname and the Caribbean' (HDSC) is dedicated to making historical documents (slave registers, civil status records, and more) available online. This allows for the tracking of individuals' life histories, especially following significant events such as the abolition of slavery in 1863. Thanks to the work of many volunteers and sponsors, since 2018 the slave registers of Suriname have been viewable online, followed by those from Curaçao in 2020. Additionally, as of 2023, the birth certificates of Paramaribo are searchable online on the website of the National Archive of Suriname. Currently, more civil status records from Suriname and the (former) Dutch Caribbean islands are being transcribed.

Project
In this project, we pursued developing the HTR+RegEx model from our MSc thesis for death certificates from Curaçao between 1879-1949. We first evaluated the HTR+RegEx model on 1534 records from 1879-1881.

The names and occupations were transcribed by two volunteers (since it had already become apparent that the HTR+RegEx performed insufficiently in this regard), followed by a review project where a final decision was made for the spelling of names and professions, and other fields e.g., dates were checked (these were the extracted entities from the HTR+RegEx model).

We concluded that the model too often failed to find a match due to errors in the HTR. These empty fields are undesirable for citizen scientists in a review project. There were actually too many errors to consider it a review project and to ensure quality. Therefore, we experimented with GPT-4 instead of regular expressions to see if it was better capable of extracting entities from the HTR-processed text.

We successfully increased the scores. GPT-4 always provides an answer, compared to static regular expressions which often fail. This slightly increased the number of incorrect entities but also increased the number of correct entities.

In the end, we processed ~24,000 death certificates using the HTR+GPT model. We refer to our report at the top of this page for more information. We now started processing Surinamese death certificates. A similar HTR+GPT model will be built to match the layout and text of these scans.

Extracting Entities from Handwritten Civil Records using HTR and RegExes (MSc thesis)

About the HDSC
The 'Historical Database of Suriname and the Caribbean' (HDSC) is dedicated to making historical documents (slave registers, civil status records, and more) available online. This allows for the tracking of individuals' life histories, especially following significant events such as the abolition of slavery in 1863. Thanks to the work of many volunteers and sponsors, since 2018 the slave registers of Suriname have been viewable online, followed by those from Curaçao in 2020. Additionally, as of 2023, the birth certificates of Paramaribo are searchable online on the website of the National Archive of Suriname. Currently, more civil status records from Suriname and the (former) Dutch Caribbean islands are being transcribed.

Project
We investigated whether computer models can speed up the transcribing process and/or assist volunteers in transcribing. Focusing on death certificates from Curaçao, we built an HTR-RegEx model that automatically extracts fields from a scan. The first step takes place on Transkribus, a platform for Handwritten Text Recognition (HTR), where we determine the regions and baselines on each scan and then employ our HTR model to obtain the transcriptions. Step 2 consists of a Python script with regular expressions (RegExes) to extract entities from the text.

Original layout by Transkribus (left); Layout from our two models (right).

Our custom layout models are trained to create a region for the marginalia on the left and a region with the certificate itself on the right. Our baseline model is trained to create sentences fully from left to right. This prevents incorrect sentence order. Also, sentences from the marginalia do not continue into the certificate text.

Our HTR model achieved an average Character Error Rate (CER) of 5%. Unfortunately, names are much harder to read (CER=16%), which means at least one-third of all names contained at least one spelling error. This makes linking people across different databases difficult.

In the image above, our HTR model returned "Martha Seratina" instead of the correct "Martha Serafina". We also perceived disagreements in transcriptions among citizen scientists.

We conclude our thesis with advice on potential integrations of the HTR-RegEx model in citizen science projects. Currently, the HTR is not good enough to be implemented independently, thus a combination of the HTR-RegEx model and the contribution of volunteers is a more realistic option. They could, for instance, revise pre-filled in entities, or, the review project could only evaluate certificates where model output differed from independent human output.

Continuation
In the 'HANDS-RX' project, we improve and eventually execute the HTR-RegEx model on 24,000 records from the Curaçaoan death certificate database. We ultimately choose GPT for Entity Extraction, as it is more flexible to incorrect HTR (spelling errors) than static RegExes.

For more information, we refer to the buttons at the top of this page for our MSc thesis, GitHub repository, and the follow-up 'HANDS-RX' project.

Pull-Factor Regular Expressions in Job Advertisements

Introduction
Every job advertisement contains features of the job that are intended to make the position more attractive to job seekers. Examples include a company car/laptop, good pension scheme, or flexibility in working hours. These are known as "pull factors."

Research has shown that job seekers from different sectors value different pull factors. The company Textmetrics has developed a tool for recruiters to make writing job advertisements easier with a focus on inclusivity and diversity. One of its functionalities is a checklist to see if important pull factors have been mentioned in the job advertisement.

Task
In Textmetrics’ application, pull factors are automatically recognized. This was previously done using a list of words/sentences. Our task was to create regular expressions so that more mentions of pull factors could be identified. Regular expressions are more flexible and allow for more variation in the phrasing of a pull factor. For example, a company car can be referred to in many different ways in the job advertisement:

• We offer the possibility of using a lease car
• You get a company car
• A car is available

It was also important to minimise the number of false positives. Classifying all sentences with the word "car" as pull factor would create too many false positives, but "company car" could, for example, be expanded to include "a car from us" and "a car from our company."

Method
Our method consisted of the following two steps:

• Step 1: Creating extensive regular expressions using the old lists of pull factors + 500 job advertisements annotated by colleagues.
• Step 2: Balancing between correct and incorrect detections by refining the expressions.

Evaluation
We evaluated by running the regular expressions over 10,000 job advertisements. For each pull factor, we then used word embeddings to select a good, representative subset to manually review. This way, we were able to refine the regular expressions and balance between 1) retaining correct detections, and 2) removing false positives as much as possible. For an end-user of the Textmetrics app, nothing is more frustrating than writing a pull factor in the job advertisement and not seeing a check mark in the checklist, but conversely, you also don't want a check mark when the pull factor has not yet been mentioned.

Outcome
Upon completion of the project, Textmetrics has a more comprehensive pull factor detection model that goes beyond a basic list of words and sentences and is more flexible with different phrasings of pull factors. We achieved a proper balance in our evaluation step, allowing for more variation in sentence fragments while keeping false positives as low as possible.

Named Entities in Job Advertisements
(Internship Data Science)

Introduction
Textmetrics is a company operating in multiple countries, offering a tool to write better job advertisements. Their tool detects age and gender discrimination but also examines readability (e.g., word difficulty) and more. In this project, we explore whether Named Entity Recognition (NER) can provide solutions to multiple issues within the tool.

One functionality in Textmetrics’ application is suggesting synonyms for words. This is desirable for adjectives like 'new' (= unused, freshly made, inexperienced). However, in the context of place names like 'Nieuw-Beijerland,' synonyms are incorrect and misleading. Also, in the Italian version of the tool, many locations are labeled as 'difficult word,' even though locations often cannot be simplified. So, for words that are entities, suggesting synonyms and labeling as difficult is unwanted behaviour.

Task
The goal is to label certain words in job advertisements so that they no longer appear as difficult words or with synonyms. For this, we created a custom NER model specialised in job advertisements and evaluated its performance.

The labels we defined include: names of organisations, locations, persons, job titles, education (e.g., studies), tools, and an additional category for miscellaneous terms that often do not have synonyms (e.g., scrum, individual choice budget, driver's licence B).

Results
We created a dataset of 12,200 sentences from job advertisements and annotated these with the help of colleagues. We then fine-tuned RobBERT, a BERT model for the Dutch language. The confusion matrix below shows how well the model can classify sentence fragments into one of our categories (O = no label).

In an additional analysis, we examined how well the model performs on completely new entities, i.e., words the model has not seen during training. This resulted in scores dropping from between 0.7-0.97 to 0.40-0.95 (table below).

Discussion
We conclude that some entities are very well recognized by the model, even completely new entities. The model has therefore learned from the sentence' context whether the term belongs to one of the categories. Unfortunately, the scores are poor for the miscellaneous category, because it is unclear to the model what terms this category entails. To conclude, the results are promising, and we expect that the scores could be further improved by, for example, providing more training data with active learning.