IELTS Essay Scoring

🚀 雅思作文自动评分模型

我们利用大量由人工评分的训练数据集，训练了一个语言模型，用于自动为雅思（International English Language Testing System）作文评分。

训练数据集包含18000篇真实的雅思考试作文及其官方分数。我们的模型评分结果按照雅思官方标准，从任务完成情况、连贯与衔接、词汇、语法和整体这五个维度进行衡量。整体（OVERALL） 是雅思作文的综合得分。

测试数据集的结果令人印象深刻：准确率 = 0.82，F1分数 = 0.81。就目前的结果而言，我们的模型在一定程度上大致可以替代人工评阅雅思作文，但我们将继续对其进行优化，以提高其准确性和有效性。

如果您使用此模型，请引用以下论文：

@article{sun2024automatic,
  title={Automatic Essay Multi-dimensional Scoring with Fine-tuning and Multiple Regression},
  author={Kun Sun and Rong Wang},
  year={2024},
  journal={ArXiv},
  url={https://arxiv.org/abs/2406.01198}
}

🚀 快速开始

以下是使用该模型为新的雅思作文评分的代码。在下面的示例中，一篇作文取自测试数据集，其整体得分为8.0分。我们的模型为这篇作文打出了8.5分，与人工评分非常接近。

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
import numpy as np

# Load the pre-trained model and tokenizer
model_path = "KevSun/IELTS_essay_scoring"
model = AutoModelForSequenceClassification.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Example text to be evaluated, the essay with the score by human rater (= 8.5) in the test dataset.

new_text = (
    "It is important for all towns and cities to have large public spaces such as squares and parks. "
    "Do you agree or disagree with this statement? It is crucial for all metropolitan cities and towns to "
    "have some recreational facilities like parks and squares because of their numerous benefits. A number of "
    "arguments surround my opinion, and I will discuss it in upcoming paragraphs. To commence with, the first "
    "and the foremost merit is that it is beneficial for the health of people because in morning time they can "
    "go for walking as well as in the evenings, also older people can spend their free time with their loved ones, "
    "and they can discuss about their daily happenings. In addition, young people do lot of exercise in parks and "
    "gardens to keep their health fit and healthy, otherwise if there is no park they glue with electronic gadgets "
    "like mobile phones and computers and many more. Furthermore, little children get best place to play, they play "
    "with their friends in parks if any garden or square is not available for kids then they use roads and streets "
    "for playing it can lead to serious incidents. Moreover, parks have some educational value too, in schools, "
    "students learn about environment protection in their studies and teachers can take their pupils to parks because "
    "students can see those pictures so lively which they see in their school books and they know about importance "
    "and protection of trees and flowers. In recapitulate, parks holds immense importance regarding education, health "
    "for people of every society, so government should build parks in every city and town."
)


encoded_input = tokenizer(new_text, return_tensors='pt', padding=True, truncation=True, max_length=512)


model.eval()

# Perform the prediction
with torch.no_grad():
    outputs = model(**encoded_input)

predictions = outputs.logits.squeeze()


predicted_scores = predictions.numpy()  

# Normalize the scores
normalized_scores = (predicted_scores / predicted_scores.max()) * 9  # Scale to 9


rounded_scores = np.round(normalized_scores * 2) / 2

item_names = ["Task Achievement", "Coherence and Cohesion", "Vocabulary", "Grammar", "Overall"]


for item, score in zip(item_names, rounded_scores):
    print(f"{item}: {score:.1f}")

##the output:
#Task Achievement: 9.0
#Coherence and Cohesion: 7.5
#Vocabulary: 8.0
#Grammar: 7.5
#Overall: 8.5

📄 许可证

本项目采用MIT许可证。