Babyberta 1

🚀 BabyBERTA

BabyBERTa is a light - weight version of RoBERTa, trained on 5M words of American - English child - directed input. It's designed for language acquisition research and can run on a single desktop with a single GPU, eliminating the need for high - performance computing infrastructure.

🚀 Quick Start

Loading the tokenizer

BabyBERTa was trained with add_prefix_space=True, so it won't work properly with the tokenizer defaults. For example, to load the tokenizer for BabyBERTa - 1, use the following code:

tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa-1",
                                                 add_prefix_space=True)

✨ Features

Hyper - Parameters

All provided models were trained for 400K steps with a batch size of 16. Importantly, during training, BabyBERTa never predicts unmasked tokens (unmask_prob is set to zero). For detailed hyper - parameter information, refer to the paper.

Performance

BabyBerta was developed to learn grammatical knowledge from child - directed input. Its grammatical knowledge was evaluated using the Zorro test suite.

The best model achieves an overall accuracy of 80.3, comparable to RoBERTa - base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021). There are two reasons for the slight differences from the values reported in the CoNLL 2021 paper:

1. The performance of RoBERTa - base is slightly better because the authors previously lower - cased all words in Zorro before evaluation. Lower - casing proper nouns is detrimental to RoBERTa - base as it's likely been trained on title - cased proper nouns. In contrast, since BabyBERTa is not case - sensitive, its performance isn't affected by this change.
2. The latest version of Zorro no longer contains ambiguous content words like "Spanish" (which can be both a noun and an adjective), resulting in a small reduction in BabyBERTa's performance.

Overall Accuracy on Zorro:

📚 Documentation

Additional Information

This model was trained by Philip Huebner, currently at the UIUC Language and Learning Lab.

More info can be found here.

[link - BabyBERTa - 1]: https://huggingface.co/phueb/BabyBERTa - 1 [link - BabyBERTa - 2]: https://huggingface.co/phueb/BabyBERTa - 2 [link - BabyBERTa - 3]: https://huggingface.co/phueb/BabyBERTa - 3