đ roberta-news
This model is a fine - tuned version of [roberta - base](https://huggingface.co/roberta - base) for unmasking news.
đ Quick Start
The model can be used with the HuggingFace pipeline. Here is an example:
>>> from transformers import pipeline
>>> unmasker = pipeline('fill - mask', model='andyreas/roberta - gen - news')
>>> print(unmasker("The weather forecast for <mask> is rain.", top_k = 5))
[{'score': 0.06107175350189209,
'token': 1083,
'token_str': ' Friday',
'sequence': 'The weather forecast for Friday is rain.'},
{'score': 0.04649643227458,
'token': 1359,
'token_str': ' Saturday',
'sequence': 'The weather forecast for Saturday is rain.'
},
{'score': 0.04370906576514244,
'token': 1772,
'token_str': ' weekend',
'sequence': 'The weather forecast for weekend is rain.'},
{'score': 0.04101456701755524,
'token': 1133,
'token_str': ' Wednesday',
'sequence': 'The weather forecast for Wednesday is rain.'},
{'score': 0.03785591572523117,
'token': 1234,
'token_str': ' Sunday',
'sequence': 'The weather forecast for Sunday is rain.'}]
⨠Features
The model is fine - tuned from [roberta - base](https://huggingface.co/roberta - base) to specialize in unmasking news.
đĻ Installation
No specific installation steps are provided in the original README.
đģ Usage Examples
Basic Usage
>>> from transformers import pipeline
>>> unmasker = pipeline('fill - mask', model='andyreas/roberta - gen - news')
>>> print(unmasker("The weather forecast for <mask> is rain.", top_k = 5))
đ Documentation
Model Description
The model is [roberta - base](https://huggingface.co/roberta - base) fine - tuned to unmask news.
Training Data
The model's training data consists of almost 13,000,000 English articles from ~90 outlets, which each consists of a headline (title) and a subheading (description). The articles were collected from the Sciride News Mine, after which some additional cleaning was performed on the data, such as removing duplicate articles and removing repeated "outlet tags" appearing before or after headlines such as "| Daily Mail Online".
The cleaned dataset can be found on huggingface [here](https://huggingface.co/datasets/AndyReas/frontpage - news). roberta - gen - news was pre - trained on a large subset (12,928,029 / 13,118,041) of the linked dataset, after repacking the data a bit to avoid abrupt truncation.
Training
Training ran for 1 epoch using a learning rate of 2e - 6 and 50K warm - up steps out of ~800K total steps.
Bias
Like any other model, roberta - gen - news is subject to bias according to the data it was trained on.
đ§ Technical Details
Training ran for 1 epoch using a learning rate of 2e - 6 and 50K warm - up steps out of ~800K total steps.
đ License
This project is licensed under the MIT license.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
