Structure Discovery in Natural Language

Book Preface

After 60 years of attempts to implement natural language competence in machines, there is still no automatic language processing system that comes even close to human language performance. The fields of Computational Linguistics and Natural Language Processing predominantly sought to teach machines a variety of subtasks of language understanding either by explicitly stating processing rules or by providing annotations they should learn to reproduce. In contrast to this, human language acquisition largely happens in an unsupervised way — the mere exposure to numerous language samples triggers acquisition processes that imprint the generalisation and abstraction needed for understanding and speaking that language.

Exactly this strategy is pursued in this work: rather than telling machines how to process language, one instructs them how to discover structural regularities in text corpora. Shifting the workload from specifying rule-based systems or manually annotating text to creating processes that employ and utilise structure in language, one builds an inventory of mechanisms that — once they have been verified on a number of datasets and applications — are universal in a way that allows their application to unseen data with similar structure. This enormous alleviation of what is called the “acquisition bottleneck of language processing” gives rise to a unified treatment of language data and provides accelerated access to this part of our cultural memory.

Now that computing power and storage capacities have reached a sufficient level for this undertaking, we for the first time find ourselves able to leave the bulk of the work to machines and to overcome data sparseness by simply processing larger batches of data.