Text categorization problems (e.g., indexing, filtering, routing) are recurrent in various natural language processing tasks. This presentation focuses on the text style to provide answers to various questions (author profiling (determining the author gender, age or psychological traits) or author verification, etc.). In this talk, we will cover the following aspects. First, according to the target application, a text representation must be determined. We will show that a relatively large variability does exist to achieve a good text representation. Some of them could be viewed as weird but could be useful for cross-lingual or cross-domain applications. Second, a feature selection could be applied to extract the best subset of predictors. Third, a machine learning (ML) model must be selected and employed with the choice between a simple ML model or a deep learning solution. Four, an evaluation procedure is used to measure the effectiveness of different solutions. But this is not the final point. Then the data analyst must provide an answer to questions such as: which predictors are the most influential for the model? How would the predictions change if a given predictor is rising? What are features that differentiate the writing style of men vs. women? To provide an answer to these questions, we will show that explainable machine learning models could provide a comprehensive response.

Large models have drastically changed the natural language processing landscape. Nowadays, pre-trained language models represent the de-facto standard to implement NLP models even in areas where labelled data is scarce and hard to obtain. Although this works fine for general-purpose applications for the NLP domain, it proves to be complex in highly-specific domains. Additionally, using those models for server-side inferencing is challenging and deploying them for on-device applications in constrained devices is completely impractical due to their size and inference cost. In this talk, we focus on alternatives to transformer-based architectures and efficient NLP and show that weight-efficient models can reach competitive performance in several tasks with model sizes in the order of a few megabytes.

Handwriting on digital surfaces, done either with a finger or a stylus, is one of the many ways to use digital ink. In this talk we focus on two tasks – handwriting recognition and digital ink synthesis that together give stylus users access to functionality that matches and exceeds that of keyboard users. We show how handwriting recognition is far from a solved task and discuss complexities in solving it in resource constrained environments and in multilingual settings. Furthermore, we discuss the recent advances in both tasks and connections between. Finally, we explore the connections to the specificities of the Swiss language ecosystem.

This talk will provide an overview of the the evolving adoption of natural language processing techniques by Swiss companies and other organizations during the last 15 years. Indeed, while the first encounters with NLP in the 2000’s were mainly in terms of efficient document management (information extraction and classification), several organizations took an interest in conversational interfaces around the 2010’s. Chatbots have been seen as an entry point into deeper customer insights and more efficient knowledge access, but somehow addressed only part of the business needs – often times, with disappointing results.

During the last five years, the advent of large language models and efficient representation learning have offered new impact opportunities – even in very human-centered fields such as the job marketplace and humanitarian activity.
The most recent renaissance of conversational language models, joining key aspects of previous promising technologies, offers yet new perspectives that the Swiss market is eager to take. This comes in hand with an increased awareness of the value of explainability and the issue of trust.

The continued growth of Large Language Models (LLMs) and their wide-scale adoption in commercial applications such as chatGPT make it increasingly important to (a) develop ways to source their training data in a more transparent way, and (b) to investigate it, both for research and for ethical issues. This talk will discuss the current state of affairs, the ongoing developments in LLM regulation and data documentation, and also present some data governance lessons learned from Big Science, an open-source effort to train a multilingual LLM.

Newspapers or magazines currently often publish articles in print or online in almost identical versions, maybe just adding links to related content or a user forum online. For further enriching the original article, we investigated methods to create different types of content automatically with state-of-the-art generative language models: (1) Keynotes, (2) a summary, and (3) a content-related quiz (i.e., a question with answers).

Our study is based on 2.5k German articles from the Swiss daily newspaper “Tages-Anzeiger”. Our evaluation with a professional editor showed that content generated by ChatGPT from OpenAI outperformed traditional NLP methods as well as human-generated content. At the venue, we plan to showcase a Web-based demo application and report on our experience with using the OpenAI API for automatic text generation in comparison with other NLP methods and human performance.

Public services rely on data to perform their functions and to document their decisions. They are legally required to archive such data. Archives, in return, are legally required to make these data accessible to the public, while protecting the privacy of people and legal entities. Anonymization provides a well-established trade-off between transparent governance and the right to privacy.
Depending on the data type, anonymization presents various levels of difficulty. It can be achieved with relative ease when dealing with structured statistical tables. When dealing with textual data, elements identifying people and legal entities are not immediately available, but must be located among innocuous words and phrases prior to any attempt at redaction. To ensure economic efficiency, public services seek to automatize this process. Available solutions resort to named entity recognition (NER) driven by machine learning (ML). Such tools have shown their efficiency in many cases, and many pre-trained NER models already exist. But it is difficult to use such models on metadata such as document titles. In effect, such titles rarely form natural sentences; which means that they cannot be processed by pretrained models, while they can still contain sensitive personal information. For instance, if a person’s name occurs in a title of a document whose position in the archival tree implies legal prosecution, this mere appearance already gives away sensitive information; even without access to the document itself. Hence the need for the Swiss Federal Archives (SFA) to anonymize a substantial part of its metadata in order to be able to publish it, in concordance with the federal law.
The SFA has commissioned the Text Crunching Center (TCC) of the University of Zurich to develop a system capable of detecting such sensitive metadata. The experience has shown that a satisfying level of automatic detection can be trained with less than 2000 manually annotated metadata entries. This by a combination of rule-based detection and of an Ensemble System consisting of three NER algorithms: Conditional Random Fields, Support Vector Machines and a Multilayer Perceptron. This tool allowed the SFA to publish metadata for 1.4 million extra records on its web portal.
In our presentation, we will discuss the legal context, as well as the training and the evaluation process of this tool. We will also make a live demonstration. The tool is now published on GitHub (https://github.com/SwissFederalArchives/tcc-metadata-anonymization) in concordance with the open-source software development policy of the SFA.

VIAN-DH is a recent web application developed by Linguistic Research Infrastructure, University of Zurich, intended for the integrated analysis of audiovisual elements of human communication, such as speech, gesture, facial expressions and more. VIAN-DH aims at bridging two approaches to the analysis of human communication: conversation analysis/interactional linguistics (IL), so far a dominantly qualitative field, and computational/corpus linguistics and its quantitative and automated methods.

Contemporary IL investigates the systematic organization of conversations and interactions composed of speech, gaze, gestures, body positioning among others. These highly integrated multimodal behavior is analyzed based on video data aimed at uncovering so called “multimodal gestalts”, patterns of linguistic and embodied conduct that reoccur in specific sequential positions employed for specific purposes.

Multimodal analyses (and other disciplines using videos) are so far dependent on time and resource intensive processes of manual transcription of each component from video materials. Automating these tasks requires advanced programming skills, which is often not in the scope of IL. Moreover, the use of different tools makes the integration and the analysis of different formats challenging. Consequently, IL research often deals with relatively small samples of annotated data which are suitable for qualitative analysis, but not enough for making generalized empirical claims derived quantitatively.

VIAN-DH aims to create a workspace where many annotation layers required for multimodal analysis of videos can be created, processed, and combined. VIAN-DH will provide a graphical interface that operates state-of the-art tools for automating parts of the data processing. The integration of tools from computational linguistics and computer vision, facilitates data processing, speeds up the overall research process, and enables processing of large amounts of data. The main features to be introduced are automatic speech recognition for transcription of language, pose estimation for extraction of gestures and other visual cues, as well as grammatical annotation for adding morphosyntactic information to the verbal content.

In order to view and search the data, VIAN-DH will provide a unified format and enable the import of the main existing formats of annotated video data and the export to other formats used in the field, while integrating different data source formats in a way that they can be combined in research. VIAN-DH will adapt querying methods from corpus linguistics to enable parallel search of many annotation levels, combining token-level and chronological search for various types of data.

VIAN-DH strives to bring crucial innovation to the fields analyzing human communicative behavior from video materials. It will allow processing of large amounts of data automatically and, implementation of quantitative analyses, combining it with the qualitative approach. It will facilitate the investigation of correlations between linguistic patterns (lexical or grammatical) with conversational aspects (turn-taking or gestures). Users will be able to automatically transcribe and annotate visual, spoken and grammatical information from videos, and to correlate those different levels and perform queries and analyses.

With the rise of Generative Large Language Models (LLM), companies across a variety of sectors are looking into the opportunities proffered by this new technology, in its multifaceted incarnations. One area of particular interest is the automated handling of customer requests –received through channels such as email, chat, social media, etc.– using the institutional knowledge at hand. Such knowledge is typically represented in a mix of documents, and, to respond to requests with accuracy, a system often needs to combine information from multiple sources, with the entire knowledge base as a “context”. An implication of the above is that, when using a LLM for that purpose, one needs to “adapt” it to the domain at hand, and to the factual knowledge of the individual organization. It also means that the model will need to have access, in one way or another, to privileged, non-public information in the company’s knowledge base. To satisfy these requirements, one would need to have models that: (a) can be trained / fine-tuned / otherwise adapted with reasonable resources; and, (b) can be prepared and used on-premise, to avoid sending private information outside the company’s infrastructure. Towards these goals, we are exploring how open-source language models can be customized for handling customer requests by utilizing prompt engineering, the ReAct paradigm, and model fine-tuning. Our primary focus is on the German language. At present, we are evaluating the BLOOM and LLaMA models and their derivatives. Given the rapid advances in the field, the scope of the work may be adjusted between the time of writing and the SwissText conference to take into consideration new developments. In this talk we will discuss the different approaches we have investigated, the necessary data and hardware requirements for implementing them, and share the results of our research.

New drugs are risky and costly to develop. “Drug repositioning” or “drug repurposing” describes the well-known practice in identifying new uses for already existing drugs or active compounds. Using a case study, this paper describes ongoing research about the exploration of the potential in using NLP techniques on publicly available data sources to identify drugs for glioblastoma therapy not documented in established standardized databases.

In this work, a pretrained instance of RoBERTa is used to extract named entities from resumes. These named entities can be used to form a knowledge base, summarize the resumes, evaluate whether the resumes contain the prerequisites mentioned in the job offers and finally to pre-fill application forms during a job application. After transfer learning performed on a corpus of 112 resumes, the model achieves an average F1-Score of 0.72 across the 22 named entity classes.

Transformers pretrained on unfiltered corpora are known to contain biases. The biases presented in this work appear when the model is used to extract the first name contained in an artificially generated sentence. These first names are associated with ethnic groups and genders (male or female). Especially, the model more easily extracts first names associated with white men. Some suggestions to mitigate the bias are presented in the discussion section.

Recent breakthroughs in NLP largely increased the presence of ASR systems in our daily lives. How- ever, for many low-resource languages, ASR models still need to be improved due in part to the difficulty of acquiring pertinent data. This project aims to help advance research in ASR models for Swiss German dialects, by providing insights about the performance of state-of-the-art ASR models on recently published Swiss German speech datasets. We propose a novel loss that takes into account the semantic distance between the predicted and the ground-truth labels. We outperform current state-of-the-art results by fine-tuning OpenAI’s Whisper model on Swiss German datasets.

The growing capabilities of transformer models pave the way for solving increasingly complex NLP tasks. A key to supporting application-specific requirements is the ability to fine-tune. However, compiling a fine-tuning dataset tailored to complex tasks is tedious and results in large datasets, limiting the ability to control transformer output. We present an approach in which complex tasks are divided into simpler subtasks. Multiple transformer models are fine-tuned to one subtask each, and lined up to accomplish the complex task. This simplifies the compilation of fine-tuning datasets and increases overall controllability. Using the example of reducing gender bias as a complex task, we demonstrate our approach and show that it performs better than using a single model.

Text summarization is an important downstream natural language processing (NLP) task that challenges both the understanding and generation capabilities of language models. Thanks to large language models (LLMs) and techniques for fine-tuning models in machine learning, significant progress has been made in automatically summarizing short texts such as news articles, often leading to very satisfactory machine-generated results. In contrast, summarizing long documents still remains a major challenge. This is partly due to the complex nature of contextual information in long texts, but also due to the lack of open-source benchmarking datasets and the corresponding evaluation frameworks that can be used to develop and test model performance. In this work, we use ChatGPT, the latest breakthrough in the field of NLP and LLMs, together with the extractive summarization model C2F-FAR (Coarse-to-Fine Facet-Aware Ranking) to propose a hybrid extraction and summarization pipeline for long documents such as business articles and books. We work with the world-renowned company getAbstract AG and leverage their expertise and experience in professional book summarization. A practical study has shown that machine-generated summaries can perform at least as well as human-written summaries when evaluated using current automated evaluation metrics. However, a closer examination of the texts generated by ChatGPT through human evaluations has shown that there are still critical issues in terms of text coherence, faithfulness, and style. Overall, our results show that the use of ChatGPT is a promising but not yet mature approach for summarizing long documents and can at best serve as an inspiration for human editors. We anticipate that our work will inform NLP researchers about the extent to which ChatGPT’s capabilities for summarizing long documents overlap with practitioners’ needs. Further work is needed to test such a hybrid summarization pipeline, in particular involving GPT-4, and to propose a new evaluation framework tailored to the task of summarizing long documents.

Prompting is used to guide or steer a language model in generating an appropriate response that is consistent with the desired outcome.
Chaining is a concept that is applied to break down more complex tasks into smaller, manageable tasks.
We use prompt chaining for long legal document classification tasks, since they are difficult to process at once due to the complex domain-specific language as well as the length.
First, we create a concise summary of the original document.
Second, we use semantic search to retrieve related exemplar texts with corresponding annotations from a training corpus.
Finally, we prompt for a label – based on the task – to assign, by leveraging the in-context learning from the few-shot prompt.
We show that we can improve the performance via prompt chaining over zero-shot and even outperforming ChatGPT zero-shot with smaller models in terms of the micro-f1 score.

We present SwissBERT, a masked language model created specifically for processing Switzerland-related text. SwissBERT is a pre-trained model that we adapted to news articles written in the national languages of Switzerland – German, French, Italian, and Romansh. We evaluate SwissBERT on natural language understanding tasks related to Switzerland and find that it tends to outperform previous models on these tasks, especially when processing contemporary news and/or Romansh Grischun. Since SwissBERT uses language adapters, it may be extended to Swiss German dialects in future work. The model and our open-source code are publicly released at <url anonymized for review>.

We test similarity-based word alignment models (SimAlign and awesome-align) in combination with word embeddings from mBERT and XLM-R on parallel sentences in German and Romansh. Since Romansh is an unseen language, we are dealing with a zero-shot setting. Using embeddings from mBERT, both models reach an alignment error rate of 0.22, which outperforms fast_align, a statistical model, and is on par with similarity-based word alignment for seen languages. We interpret these results as evidence that mBERT contains information that can be meaningful and applicable to Romansh.

To evaluate performance, we also present a new trilingual corpus, which we call the DERMIT (DE-RM-IT) corpus, containing press releases made by the Canton of Grisons in German, Romansh and Italian in the past 25 years. The corpus contains 4 547 parallel documents and approximately 100 000 sentence pairs in each language combination. We additionally present a gold standard for German-Romansh word alignment. The data is available at https://github.com/eyldlv/DERMIT-Corpus.

Automatic Text Simplification (ATS) aims at simplifying texts by reducing their linguistic complexity while retaining their meaning. While being an interesting task from a societal and computational perspective, the lack of monolingual parallel data prevents an agile implementation of ATS models, especially in less resource-rich languages than English. For these reasons, this paper investigates how to create a general-language parallel simplification dataset for French using a method to extract complex-simple sentence pairs from comparable corpora like Wikipedia and its simplified counterpart, Vikidia. By using a two-step automatic filtering process, we sequentially address the two primary conditions that must be satisfied for a simplified sentence to be considered valid: (1) preservation of the original meaning, and (2) simplicity gain with respect to the source text. Using this approach, we provide a dataset of parallel sentence simplifications (WiViCo) that can be later used for training French sequence-to-sequence general-language ATS models.

We introduce Parallel Paraphrasing (Para-both), an augmentation method for translation metrics making use of automatic paraphrasing of both the reference and hypothesis. This method counteracts the typically misleading results of speech translation metrics such as WER, CER, and BLEU if only a single reference is available. We introduce two new datasets explicitly created to measure the quality of metrics intended to be applied to Swiss German speech-to-text systems. Based on these datasets, we show that we are able to significantly improve the correlation with human quality perception if our method is applied to commonly used metrics.

In Switzerland, two thirds of the population speak Swiss German, a primarily spoken language with no standardised written form. It is widely used on Swiss TV, for example in news reports, interviews and talk shows, and captions are required for people who do not understand this spoken language. This paper focuses on the second part of a cascade approach for the automatic Standard German captioning of spoken Swiss German. We apply a multilingual pre-trained model to translate automatic speech recognition of Swiss German into Standard German suitable for captioning. Results of several evaluations, both human and automatic, show that the system succeeds in improving the content, but is currently not capable of producing entirely correct Standard German.

We present 20 Minuten, a dataset for abstractive text summarisation of German news articles. The corpus fills a gap in summarisation resources for the German language and includes multiple professionally written and stylistically distinct summaries, along with captioned images and document-level reading times. In this paper, we conduct baseline experiments with mT5 and compare the performance of fine-tuning in both single and multi-task settings on six different downstream tasks supported by our dataset. Our results reveal that dedicated models are preferable, especially for the more distinct tasks. We make our dataset available for research and provide the code to facilitate handling and model training.

The Battle for NLP Ideas is a collaborative events where participants group together to brainstorm innovative ideas based on the latest NLP technologies. The best ideas are presented on stage and shared with the whole conference.